Getting Started

Installation

The source code is currently hosted on GitHub at: llminspector

pip install notebook==5.2.2
pip install git+https://github.com/michelin/LLMInspector.git

The list of changes to LLMInspector between each release can be found here.

Usage

from llm_inspector.llminspector import llminspector
import pandas as pd

obj = llminspector(config_path=<path-to-cofig.ini>, env_path=<path-to-.env>)

# Reads the golden dataset and saves a excel file with all the perturbations and augmentations applied.
obj.alignment() 

# Reads the adversarial dataset provided and helps filter only the capabilities that are mentioned in the config.
obj.adversarial()

# Reads all the documents from the document's directory provided in the config file and creates test questions and ground truths.
obj.rag_testset_gen()

# Reads the excel file containing the questions, ground truth, actual answer and context provided by RAG and evaluates it based on retrieval and generation metrics.
obj.rag_evaluate()

# It takes in data frame containing the questions, ground truth and answer and perform accuracy testing, sentiment analysis, emotion analysis, PII detection and much more.
df = pd.read_excel(<path-to-input-csv>)
obj.evaluate(df) 

Setting up config.ini and .env

The package has been designed to work based on the information provided in the config.ini file, thus it is crucial to set up the config file with relevant information.

Example content of the config.ini file:

#------------------------------
# Configuration File
#-----------------------------
[Alignment_File]

Alignment_input_FilePath = User/input_file_directory/
Alignment_GoldenDataset_FileName= input_file_name.xlsx

Alignment_output_path = User/output_file_directory/
Alignment_Output_fileName =  /output_file_name_
paraphrase_count = 2
augmentations = {
    'uppercase': ('Robustness', 1),
    'lowercase': ('Robustness', 1),
    'typo': ('Robustness', 1),
    'add_punctuation': ('Robustness', 1),
    'strip_punctuation': ('Robustness', 1),
    'context': ('Robustness', 1),
    'titlecase': ('Robustness', 1),
    'contract': ('Robustness', 1),
    'abbreviate': ('Robustness', 1),}

[Insights_File]
Insights_output_path = User/insights_output_file_directory/
Insights_Output_fileName = /insights_output_file_name_
output_lang = french
; fullmetric_list = ["rouge_score", "bert_score", "answer_similarity", "answer_correctness", "question_toxicity", "answer_toxicity", "pii_detection", "readability", "translate", "question_language", "answer_language", "question_sentiment", "answer_sentiment", "question_emotion", "answer_emotion"]
Metrics =  ["pii_detection"]
question_col = question
answer_col = answer
ground_truth_col = ground_truth
threshold = 0.9

[RAG_File]
RAG_testset_document_directory = Users/kiran.prasath_ext/llm_inspector-dev-opensource/input
RAG_testset_input_directory = Users/kiran.prasath_ext/llm_inspector-dev-opensource/dataset
RAG_testset_input_filename = /processed_en_output.xlsx

RAG_testset_Output_fileName = /Rag_testset_Output_
RAG_eval_Output_fileName = /RAG_eval_Output_
RAG_output_directory = Users/kiran.prasath_ext/llm_inspector-dev-opensource/output
testset_size = 10

[Adversarial_File]
Adversarial_input_FilePath = User/adversarial_input_file_directory/
Adversarial_FileName= file_name.xlsx

Adversarial_output_path = User/adversarial_outut_file_directory/
Adversarial_Output_fileName = /Adversarial_Output_
capability = all
subCapability = all

[Tag_Augmentation_Key]
tag_keyword_dict = {
    '{greeting}' : ['Hi','Hey', 'Hola', 'Namaste'],
    '{seasonality}' : ['summer', 'winter', 'rain', 'all season'],
    '{road_condition}' : ['dry', 'off-road', 'wet', 'snow'],
    '{brand_name}': ['Apple', 'Tesla','Tata'],
    '{product_name}': ['Macbook Pro', 'Apple Watch', 'Macbook Mini', 'Model X'],
    '{company_&_location}': ["Apple, California"],
    '{department}': ['Engineering', 'Design', 'IT', 'Manufacturing'],
    '{country_name}': ['India', 'United States', 'China'],
    '{city_name}': ['New York', 'Chicago', 'Houston', 'Los Angeles'],
    '{nationality}': ['Indian', 'American', 'French'],
    '{language}': ['English', 'French', 'Hindi'],
    '{activity}': ['Photography', 'Trekking', 'Singing'],
    '{mood_positive}': ['happy', 'excited', 'amazing'],
    '{mood_negative}': ['sad', 'updset', 'angry'],
    }

[Augmentation_Type]
augmentation_dict = {
    'Greetings' : ['greeting'],
    'Seasons' : ['seasonality'],
    'Rating': ['Season', 'Rating'],
    'Road Conditions':['road_condition'],
    'Brands' : ['brand_name'],
    'Products': ['product_name'],
    'Countries':['country_name'],
    'Cities': ['city_name'],
    'Nationalities': ['nationality'],
    'Languages': ['language'],
    'Activities': ['activity'],
    'Moods': ['mood_positive', 'mood_negative'],
    'Locations': ['company_&_location],
    'Job Departments': ['department']
    }

The paths, augmentation tags and other configuration parameters needs to be changed based on the requirements of the project.

Sensitive information like API keys are to be given in .env file.

Example content of .env file:

api_version = <OPENAI-API-VERSION>
azure_endpoint = <AZURE-ENDPOINT>
api_key = <AZURE-OPENAI-API-KEY>