Dataprep updated readme and parametrized prepare_doc_arango.py

[ajaykallepalli]

Changes Made

• Updated Dockerfile to include git for dependency installation
• Updated Docker compose with required environment variables

Testing Done

• Running service with Docker compose and python files.
• Tested pdf with table parsing
  -- takes 25-30 mins with 500 chunk size and 50 overlap, 10 mins with 2000 chunk size and 200 overlap.
• Tested time required for parsing sample pdf files
• Embeddings working, with ability to set embeddings to False

Setup Instructions

1. Set environment variables:

export ARANGO_URL="http://localhost:8529"
export ARANGO_USERNAME="root"
export ARANGO_PASSWORD="password"
export ARANGO_DB_NAME="opea"
export PYTHONPATH='{path to comps file}'

2. Start services:

python prepare_doc_arango.py

3. Curl command to run:

curl -X POST \
    -H "Content-Type: multipart/form-data" \
    -F "files=@./your_file.pdf" \
    -F "graph_name=${your_graph_name}" \
    http://localhost:6007/v1/dataprep
    
curl -X POST \
    -H "Content-Type: multipart/form-data" \
    -F "files=@./file1.txt" \
    -F "graph_name=${your_graph_name}" \
    http://localhost:6007/v1/dataprep

Notes

• None of the services that required TGI Gaudi service tested, if required commented out before running.
• More detailed instructions in the readme

Testing

• Tested standalone deployment
• Tested docker-compose deployment

[ajaykallepalli]

Will be merging changes into Arango branch first, apologies