Use Weaviate to improve Proceedings search and linking data

[laura-ham]

This discussion topic is around a potential solution to #113. In this issue, a proceedings title parser is proposed. The current implementation looks for exact matches of the words entered in the form and the keywords of the proceedings title. If a similar name is used but the same proceeding was intended to be found, this is not possible with the current implementation. A potential solution is to use Weaviate to store the proceedings data. Searches can then be done based on concept or semantically, rather than only with exact matching keywords. 

Additionally, Weaviate could solve data enrichment by making relations between various existing (meta) data sources (e.g. proceedings, conferences, papers, authors, countries, etc).

An example of a successful project with Weaviate and scientific articles from Arxiv can be found here. 2.7 GB of scholarly articles with cross references to authors, journals and categories are present in a Weaviate instance. This enables semantic search through these publications and articles, overcoming the exact keyword matching-based search problem. You can try it out yourself with newsarticles in a live demo setting here (not Arxiv since it's being updated as I'm typing this). 

First step would be to create a Weaviate data schema based on the available (meta) data of this project.

Read here on how to get started with a Weaviate instance running locally or in the cloud: https://www.semi.technology/developers/weaviate/current/getting-started/installation.html

I would like to hear your opinions on this idea. I could help with the ideation and implementation if there are questions. 

[WolfgangFahl]

Hi Laura - welcome to our discussions and thanks for all the effort you and Bob have put into this in recent months. I am positive that Weaviate might be part of our solution in the mid- to long term. Unfortunately, I could not work on my research on that matter in January. Therefore, I might have to postpone things further depending on the priorities of the project. I'd love to get the project team acquainted to your offer and would appreciate if we could schedule an online meeting. Currently, we are pretty busy so we'll have to see when this will be possible, yet let's hope for the best.
Yours faithfully,
Wolfgang