Ragaaf (RAG assessment annotation free)

[adkakne]

Description

Adding annotation free RAG evaluation feature using LLM-as-a-judge methodology.

Issues

n/a.

Type of change

List the type of change like below. Please delete options that are not relevant.

• New feature (non-breaking change which adds new functionality)

Dependencies

jsonlines, openai

Tests

Added ./tests/test_ragaaf.py for testing.

[adkakne]

Test is working with local endpoint for mistralai/Mixtral-8x7B-Instruct-v0.1 sharded on 8 HPUs.

[adkakne]

Hi @XinyuYe-Intel @lkk12014402 @minmin-intel,

After your review, I made updates to RAGAAF codebase, our final offering is -

1. Data - can accept unit test cases, local datasets or benchmarking datasets
2. Models - can accept all OpenAI models or a endpoint.
3. Hardware - endpoint can run on HPU/GPU/CPU, local testing supports GPU/CPU, OpenAI testing uses their remote server.
4. Metrics - we offer factualness, correctness, relevance and readability. User can add their own metrics easily.

Thank you for your final review and your insights for improvements in the code.

[lkk12014402]

LGTM