Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Image Type and support in sem_operators #33

Merged
merged 32 commits into from
Nov 28, 2024
Merged

Image Type and support in sem_operators #33

merged 32 commits into from
Nov 28, 2024

Conversation

harshitgupta412
Copy link
Collaborator

@harshitgupta412 harshitgupta412 commented Nov 13, 2024

Changes:

  • Added ImageArray type for storing images
    • loads the image lazily from the stored data. Caution: There are no explicit checks on what is stored; the only way to confirm whether the data is proper is by accessing it.
  • Updated sem_ops mentioned in the title to allow multimodal data.
    • Major updates in lotus/templates/task_instructions.py and retriever_models to support more general types.
      • Added df2multimodal_info to load multimodal data properly
      • Clip model can be used through sentence transformer.
      • updated the user prompts to send images properly (user_message_formatter and context_formatter)
      • index and call can now take pd.Series or Images as input.
  • Tests in .github/tests/multimodality_tests.py
  • Example in examples/op_examples/multimodal_ops

.github/tests/multimodality_tests.py Outdated Show resolved Hide resolved
lotus/utils.py Outdated Show resolved Hide resolved
requirements.txt Outdated Show resolved Hide resolved
lotus/models/qwen2_model.py Outdated Show resolved Hide resolved
.github/workflows/tests.yml Show resolved Hide resolved
lotus/sem_ops/sem_filter.py Outdated Show resolved Hide resolved
lotus/templates/task_instructions.py Outdated Show resolved Hide resolved
requirements.txt Outdated Show resolved Hide resolved
lotus/utils.py Outdated Show resolved Hide resolved
lotus/templates/task_instructions.py Outdated Show resolved Hide resolved
lotus/models/__init__.py Outdated Show resolved Hide resolved
@harshitgupta412 harshitgupta412 changed the title Image Type and support in filter Image Type and support in sem_operators: filter, map, join, top_k and extract Nov 19, 2024
@harshitgupta412 harshitgupta412 changed the title Image Type and support in sem_operators: filter, map, join, top_k and extract Image Type and support in sem_operators Nov 26, 2024
@liana313 liana313 merged commit c7ed69a into main Nov 28, 2024
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants