Build embeddings with the load data script. #4402
Conversation
keyurva
commented
Jun 26, 2024
keyurva
commented
- Ran the script with the floret dataset and here are the build embeddings results.
| # TODO: try using the large model | ||
| -f https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.1/en_core_web_sm-3.7.1-py3-none-any.whl | ||
| en_core_web_sm==3.7.1 |
There was a problem hiding this comment.
@shifucun - Needed to add this to the embeddings requirements to eliminate installing the nl_server requirements. Let me know if this is ok.
There was a problem hiding this comment.
I was even thinking get rid of the lib here and make a shared requirements.txt. Duplicating the lib makes the version diverge very easily. Saw an bug before due to this.
| ) | ||
| local start_ts=$(date +%s) | ||
| set -x | ||
| python -m tools.nl.embeddings.build_embeddings \ |
There was a problem hiding this comment.
@ajaits - The run_cmd function did not work for argument values with spaces so calling the script directly here. Let me know if there's a better way or we can go with this?
There was a problem hiding this comment.
if there are no spaces within the json values in CUSTOM_CATALOG_DICT, you can try:
cmd=(python -m tools.nl.embeddings.build_embeddings)
cmd+=(--embeddings_name "$CUSTOM_EMBEDDING_INDEX")
cmd+=(--output_dir "$NL_EMBEDDINGS_DIR")
cmd+=(--catalog "$(echo $CUSTOM_CATALOG_DICT | sed -e 's/ //g')")
run_cmd ${cmd[@]}
| --output_dir "$NL_EMBEDDINGS_DIR" \ | ||
| --catalog "$CUSTOM_CATALOG_DICT" >> $LOG 2>&1 | ||
| set +x | ||
| status=$? |
There was a problem hiding this comment.
move this before set, right after the python command so we get the status of python (not set).
|
Added @pradh to also take a look. We had some offline discussion about maintaining and testing the script. Would like to get more opinions. |
