New workflow to generate embeddings in a single workflow #1296
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
natoverse
reviewed
Oct 21, 2024
|
Please remember to rename the new workflow to remove "create_final" so it is more clear that it doesn't have a direct output. |
|
Please add a test under test_verbs that confirms all selected embeddings tables are created. You can test this by providing a storage instance in the test, and then confirming that all requested tables exist. You can also read any of the tables and assert that they have mock embeddings values. |
AlonsoGuevara
requested changes
Oct 21, 2024
natoverse
requested changes
Oct 31, 2024
natoverse
previously approved these changes
Oct 31, 2024
AlonsoGuevara
approved these changes
Nov 1, 2024
xxll88
reviewed
Nov 2, 2024
There was a problem hiding this comment.
"Column(s) ['description_embedding'] do not exist"
3 tasks
prasodani
reviewed
Nov 10, 2024
| if embed_column not in input.columns: | ||
| msg = f"Column {embed_column} not found in input dataframe with columns {input.columns}" | ||
| raise ValueError(msg) | ||
| title = title_column or embed_column |
There was a problem hiding this comment.
While trying to index, embed_text fails due to this change.
title is re-initialised in the while loop below leading to a key error. Using some other variable name will fix it.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
New workflow to separate text embedding generation in a new single workflow to generate new parquet files with just id/embedding columns.
Proposed Changes
New workflow called: create_final_embeddings.
Remove all embeddings configuration in all other workflows.
Standardize text embed configurations for all embeddings.
New standard for parquet embeddings files:
<original_table_name>_<source_column>_embeddings.parquetwith just id/embedding columns.