Feature/ml pipeline

[nmenezes0]

Context

Add the ML pipeline to generate the themes/topics and save them to the DB. For each question in a consultation, we need to classify the free text responses for each question into topics (also called "themes") and save these info to the DB.

Out of scope for this PR:

• Adding the LLM calls to generate the theme summaries will be done separately.
• Evaluation/improvement of the modelling approach - at the moment, we just want to implement the existing workflow that was used for previous consultations.
• Dealing with outliers - at the moment they are just saved like any other topic (as we do still want them).

Changes proposed in this pull request

• Adding BERTopic and other ML packages.
• Adding a question field to the Theme model. Whilst it feels like a bit of duplication as there is also a question field on the Answer model - this allows us to add uniqueness criteria, simplifies logic, and the question is only assigned to the answer and theme once (i.e. it isn't going to change and get out of date).
• Adding the ML pipeline to assign topics ("themes") for each question - using BERTopic.

ML pipeline overview for a given question (with free text that needs to be classified):

• Get the answers for a given question i.e. the free text responses corresponding to that question.
• Get embeddings for the list of free text responses i.e. representations of each response as vectors.
• Using these embeddings, generate a topic model (i.e. this classifies the responses into different topics i.e. clusters the vectors above).
• Save each of these topics/themes to the DB, and save the theme for each answer.

Follows the first part of this: https://github.com/i-dot-ai/ova-consultation/blob/main/run_full_analysis_pipeline.py (not the LLM summary bit). BERTopic docs are pretty good too: https://maartengr.github.io/BERTopic/.

Guidance to review

Check the pipeline runs and things are saved to the DB as appropriate - this is covered in the tests (but check you are happy the tests check the entire pipeline runs, and checks themes are saved to DB).

Check the ML pipeline gets topics as per Jonah's code: https://github.com/i-dot-ai/ova-consultation/blob/main/run_full_analysis_pipeline.py

Link to JIRA ticket

https://technologyprogramme.atlassian.net/browse/CON-47

Things to check

• I have added any new ENV vars in all deployed environments and updated the .env.example and .env.test files in the repo

[rachaelcodes]

I've a couple of clarifying questions, but otherwise LGTM