Upgrade to Lucene 9.x?

[reckart]

I kind of expected that RDF4J 5.x would already be using Lucene 9, but looks like it still uses Lucene 8. Are there any plans to upgrade?

[hmottestad]

I believe that our main issue is our compatibility with Elasticsearch. Because of how Elasticsearch designed their code we need to use the same version of Lucene. We have an issue for migrating away from the old Elasticsearch approach, but it's a lot of work. We also can't really upgrade Elasticsearch since they changed their license to an incompatible one.

#4776

#4777

We need to migrate a client code to the new client, which uses a completely different api design. We also need to migrate some of our test code that uses a rather old approach of integrating into Elasticsearch's own test framework, which might have been a good idea at the time but is at least now not recommended. That test framework also locks us into a tight integration with Elasticsearch which we really don't want anymore.

[reckart]

How about having a Lucene 9 sail maybe in parallel to a Lucene 8 sail and let ElasticSearch stay on Lucene 8? Personally, I don't use ElasticSearch, only the Lucene Sail.

[hmottestad]

I know that Apache HttpClient chose a new namespace for v5 because of how pervasive v4 is in Java that it becomes hard for a project to upgrade before all their dependencies have upgraded. If Lucene did something similar we would have more control. At the moment I think that we wouldn't be able to provide a LuceneSail with v9 and the two Elasticsearch sails in the same RDF4J Server & Workbench. The JDK would arbitrarily load Lucene v8 or v9 whenever a class was needed and there would be a huge risk of MethodNotFound exceptions for our users.

Personally I'm really quite annoyed at Elastic for changing up the client three times in six years and not caring when their patch releases contain breaking changes.

[reckart]

Server & Workbench is only one use-case. I'd assume there are other users like me that are using RDF4J as a library and who do not care about ElasticSearch. Such users would be able to upgrade their applications to Lucene 9.

I am considering preparing a Lucene 9 version of the SAIL. A few weeks back, I prepared a Lucene 9 PR for another library that I use and that worked out quite ok. In the worst case, I would fork the RDF4J SAIL into my own codebase in order to do that upgrade, but it might as well live in the RDF4J codebase as a submodule that is not used by the server/workbench.

[JervenBolleman]

We need to investigate if elastic with changing their license again, is an option for upgrading which might make our live simpler.