Elasticsearch Migration / Upgrade

[oalders]

We can use this discussion to start working out what's required to update Elasticsearch to a modern version.

[haarg]

Likely the largest change we need to account for is the removal of types. We need to separate out each of the types we are using to separate indexes.

Removing types will likely cause issues with ElasticSearchX::Model. We may be able to patch the module enough to work around any problems. Removing the module entirely would be much preferable but is not trivial.

We may still have some queries using older syntax. And we may need to audit anyone using the Elasticsearch passthrough. I know that perlbrew does this in at least one place, which should be converted to use download_url. But download_url doesn't currently work for perl releases.

The mappings similarly will need some updates. These are likely not difficult.

We have several scripts installed on the servers, using the "groovy" language. That is no longer supported Elasticsearch 6+. As far as I can tell, only one of these scripts is in use, as part of search. I believe we should be able to rewrite this using "expression". We allow the other scripts to be used via the Elasticsearch passthrough. That ability should just be removed.

All of these changes can be done with our current infrastructure and without moving to a new ES version.

[mickeyn]

Instead of dealing with ElasticSearchX::Model, I think we should try writing a new indexing suite (preferably in a separate repo from the API) that's not relying on it (or any specific storage in the data processing step)

[haarg]

Elasticsearch 6:

• integer values no longer supported for boolean fields

[haarg]

Elasticsearch 5:

• search_type=scan removed
• fields parameter has been replaced by stored_fields
• Deprecated queries removed: filtered, missing, query
• Removed support for the deprecated filter fields in nested query, in favour of query.
• Removed support for the top level filter element in function_score query, replaced by query.
• Removed support for the deprecated top level filter in the search api
• string fields replaced by text/keyword fields
• The _timestamp and _ttl fields were deprecated and are now removed.
• the index property now only accepts true/false instead of not_analyzed/no

[haarg]

Elasticsearch 7:

• Negative boosts are not allowed
• hits.total is now an object in the search response

[haarg]

Elasticsearch 8:

• The nested_filter and nested_path properties have been removed from the search API’s sort request body parameter. Was deprecated in 6.

[oalders]

It looks like the fastest way to move to the cloud would be by taking a snapshot, storing it in S3 and then re-indexing from the snapshot: https://www.elastic.co/guide/en/elasticsearch/reference/2.4/modules-snapshots.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html

type: "s3"

[oalders]

Further cloud notes:

• we can also use terraform to manage the hosted instances
• each cluster is behind a load balancer, so it has one endpoint to communicate with rather than 3
• backups happen automatically every 30 minutes
• we can spin up a 2nd cluster from backup if we need to
• as noted above, we'd use s3 buckets for migration

[ranguard]

Migration is tracked in metacpan/metacpan-puppet#240 (I've put in puppet as we can look at migrating before we switch to K8s).

Also of note we already snapshot to AWS

https://github.com/metacpan/metacpan-api/blob/master/lib/MetaCPAN/Script/Snapshot.pm

https://github.com/metacpan/metacpan-puppet/blob/master/hieradata/nodes/bm-mc-01.yaml#L22-L31

[ranguard]

The ingrest project is at https://github.com/metacpan/metacpan-ingest