Document Elasticsearch for Arches 7

[mradamcox]

describe the issue

Arches 7 upgrades to version 8 of Elasticsearch. I believe there are some significant changes to how people need to set up ES 8 as compared to past versions, so this really needs to be evaluated as soon as possible.

Here is a forum post with a little more detail on what users have encountered: https://community.archesproject.org/t/elastic-user-authentication-fail/1700

please add links to existing docs or code (if relevant)

Software dependencies:
https://arches.readthedocs.io/en/stable/requirements-and-dependencies/#software-dependencies
Arches and Elasticsearch:
https://arches.readthedocs.io/en/stable/arches-and-elasticsearch/

It may be worth adding a section to the installation documentation... installing Elasticsearch for Arches hasn't really changed for years, so this may be a significant difference.

which release does this issue concern?

Arches >= 7

[mradamcox]

This seems to be the correct configuration that we should put in the documentation: https://community.archesproject.org/t/elastic-user-authentication-fail/1700/8

[aarongundel]

@mradamcox it's worth noting that xpack security shouldn't be disabled if you're running elastic in a production setup. If you're running in development, it's fine to disable. A lot of this is covered in the 7.x release notes. It would be nice if we had a way to get the release notes into the documentation. But the elastic setup is more involved for production-worthy version of es (you need to set a password, setup SSL, etc). Elasticsearch (especially if it's exposed to the internet) is dangerous to run without security. We've seen it compromised on more than one occasion.

[mradamcox]

@aarongundel thanks for the note about dev vs. production. As for release notes, completely agree. I mentioned that here #272, but probably we should just make a new ticket specifically for their inclusion in the docs.

[ekansa]

@aarongundel

I wanted to revisit your comment about securing production deployments of Elastic. Do you have an example working config you can share that can be good guidance for making xpack (https) secured Elastic work? Perhaps in the context of a Docker deployment?

I'm looking at fleshing out this point made in the release notes you referenced in the documentation:

Elasticsearch 8 (ES 8) enables a feature called xpack security by default. See the elasticsearch 8 documentation for details on the new features and how you can use them to secure your elasticsearch instance.

By default, xpack security turns on SSL with a self-signed certificate and will expect requests to come from an ES user. Elasticsearch users can be created via the elasticsearch-users command. Uncomment and override ELASTICSEARCH_CONNECTION_OPTIONS in settings.py line as needed (with an appropriate password and certificate settings) in your project settings or settings_local files.
If you choose to turn off xpack security in your ES 8 instance (not recommended) you will need to override the ELASTICSEARCH_HOSTS setting and change the scheme to "http".

[aarongundel]

@ekansa you need to set ELASTICSEARCH_HOSTS with connection info like this... ELASTICSEARCH_HOSTS = [{"scheme": "https", "host": "your_host_here", "port": your_port_here}] (this is already the default in the stock arches settings.py, though it points to localhost) and ELASTICSEARCH_CONNECTION_OPTIONS with the appropriate elastic login info ELASTICSEARCH_CONNECTION_OPTIONS = {"timeout": 30, "basic_auth": ("username_here", "password_here")}

I'm currently doing this locally with afs and it works OK, although python complains about not having an actual cert for ES quite a bit. This is pretty straightforward to correct using an actual (not a self signed) cert, but that's not something I've budgeted any time for in the dev setup, and is OK as far as a dev setup is concerned.