AWS Deployment Enhancements

[elizoller]

A follow up from: #77
Goal state: https://www.lucidchart.com/invitations/accept/8a83a394-5cf6-48c8-9434-6803456c283a
still need to watch for:
Islandora-Devops/islandora-playbook#162 (needed to separate FITS from webserver)
islandora-deprecated/ansible-role-crayfish#31 (needed to separate Gemini to RDS)

Need to set up

• Gemini on RDS
• S3 for Fedora
• More than 2 servers

[elizoller]

see changing this: https://github.com/Islandora-Devops/ansible-role-fcrepo/blob/master/templates/jdbc-mysql-repository.json to something closer to this: https://github.com/fcrepo4/fcrepo4/blob/master/fcrepo-configs/src/main/resources/config/jdbc-mysql-s3/repository.json
just not sure what the impact of this would be - should probably be done PRIOR to actually putting content in to fedora.

[elizoller]

ELB provisioning added with asulibraries/islandora-playbook@f728015

[elizoller]

added a new template at Islandora-Devops.fcrepo/templates/jdbc-mysql-s3-repository.json in order to accommodate the s3 connection - branch here: https://github.com/asulibraries/ansible-role-fcrepo/tree/s3 and i'll be submitting a PR to push this back into the islandora repo soon

{
"name" : "repo",
"jndiName" : "",
"workspaces" : {
"predefined" : ["default"],
"default" : "default",
"allowCreation" : true,
"cacheSize" : 10000
},
"storage" : {
"persistence": {
"type" : "db",
"connectionUrl": "jdbc:mysql://{{ fcrepo_db_host }}:{{ fcrepo_db_port }}/{{ fcrepo_db_name }}?createDatabaseIfNotExist=true",
"driver" : "com.mysql.jdbc.Driver",
"username" : "{{ fcrepo_db_user }}",
"password" : "{{ fcrepo_db_password }}"
},
"binaryStorage" : {
"type" : "s3",
"username" : "{{aws_access_key}}",
"password" : "{{aws_secret_key}}",
"bucketName" : "{{fcrepo_aws_bucket}}"
}
},
"security" : {
"anonymous" : {
"roles" : ["readonly","readwrite","admin"],
"useOnFailedLogin" : false
},
"providers" : [
{ "classname" : "org.fcrepo.auth.common.BypassSecurityServletAuthenticationProvider" }
]
},
"garbageCollection" : {
"threadPool" : "modeshape-gc",
"initialTime" : "00:00",
"intervalInHours" : 24
},
"node-types" : ["fedora-node-types.cnd", "file:/opt/fcrepo/configs/claw.cnd"]
}

[elizoller]

Also worth noting that the keys pointed to by the syn-settings.xml files, such as /opt/keys/claw/public.key must MATCH on the two (or more) servers that are expected to communicate. and that key is proliferated in other places during deployment (ie within crayfish)

[elizoller]

I also had to modify the syn-settings for both crayfish and fedora.
In fedora, this was accomplished by modifying the fcrepo_syn_sites variable in the /inventory/stage/group_vars/tomcat.yml file (see asulibraries/islandora-playbook@18d9098#diff-dbbeb387f76a3db9bec2cd1fabeea514R48 )
In crayfish, this was accomplished by modifying the roles/internal/Islandora-Devops.crayfish/templates/syn-settings.xml.jp2 I modified it to include:

<?xml version="1.0" encoding="UTF-8"?>
<!-- managed by Ansible -->
<config version='1' header='X-Islandora'>
	<site url='https://35.166.140.245' algorithm='RS256' encoding='PEM' anonymous='true' default='true' path='{{ crayfish_install_dir }}/public.key'/>
    <token user='admin' roles='admin,fedoraAdmin'>
      {{ crayfish_syn_token }}
    </token>
    <site url='https://35.163.77.215' algorithm='HS256' encoding='PEM' anonymous='true' default='true' path='{{ crayfish_install_dir }}/public.key' />
</config>

This could potentially be rolled back up into a PR for upstream - it would require using a variable, much like the way it is handled in the fcrepo-syn role here: https://github.com/Islandora-Devops/ansible-role-fcrepo-syn/blob/master/templates/syn-settings.xml

[elizoller]

Eli to do: add and configure https://www.drupal.org/project/flysystem_s3 to write drupal files directly to s3. Ideally we probably need to be using private files for all of the derivatives and such since some of those files will be limited access.

[elizoller]

went the route of https://www.drupal.org/project/s3fs which uses the AWS SDK for communicating with S3 for drupal public and private files instead of have to use s3fuse due to known performance issues

[elizoller]

This issue is still pending for separation of FITS and moving Gemini to RDS

[elizoller]

FITS has been provisioned to the services server.
Still waiting on Gemini to RDS

[elizoller]

in order to accomplish this, due to pending PRs, i applied those prs to local copies of the ansible roles and deployed.