Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[STACKED #1255] [ENH] Add multitenancy #1244

Merged
merged 23 commits into from
Oct 24, 2023
Merged

[STACKED #1255] [ENH] Add multitenancy #1244

merged 23 commits into from
Oct 24, 2023

Conversation

HammadB
Copy link
Collaborator

@HammadB HammadB commented Oct 13, 2023 •
edited
Loading

Description of changes

Summarize the changes made by this PR.

  • Improvements & Bug fixes
    • ...
  • New functionality
    • Adds multitenancy and databases as first class concepts by migrating the db and plumbing it through the API into the sysdb
    • We now treat the "System" as a singleton-per-path and create a wrapper API object that proxies to it with context on the tenant/database. In this way the server is context-unaware about the connection.

Test plan

How are these changes tested?
Unit Tests were added for new client tenant/database behavior
Property tests were added for the new tenant/database behavior by subclassing the collection state machine and switching the tenant/database as a state machine transition.

  • Tests pass locally with pytest for python, yarn test for js

Documentation Changes

I will add a section to the docs about multitenancy and how to use it.
We can remove warnings about the client being a singleton.

This was referenced Oct 13, 2023
Merged
Merged
Merged
Merged
@HammadB Graphite App
Copy link
Collaborator Author

HammadB commented Oct 13, 2023 •
edited
Loading

Current dependencies on/for this PR:

This comment was auto-generated by Graphite.

@github-actions
Copy link

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

  • Can you think of any use case in which the code does not behave as intended? Have they been tested?
  • Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
  • If appropriate, are there adequate property based tests?
  • If appropriate, are there adequate unit tests?
  • Should any logging, debugging, tracing information be added or removed?
  • Are error messages user-friendly?
  • Have all documentation changes needed been made?
  • Have all non-obvious changes been commented?

System Compatibility

  • Are there any potential impacts on other parts of the system or backward compatibility?
  • Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

  • Is this code of a unexpectedly high quality (Readbility, Modularity, Intuitiveness)

@HammadB HammadB force-pushed the hammad/multitenancy branch from d21e6c1 to dda4a61 Compare October 13, 2023 07:31
@HammadB HammadB force-pushed the hammad/get_or_create_sysdb branch from 47c73e9 to ee2849b Compare October 16, 2023 05:57
@HammadB HammadB changed the base branch from hammad/get_or_create_sysdb to main October 17, 2023 23:44
@HammadB HammadB force-pushed the hammad/multitenancy branch from dda4a61 to 2ae2b51 Compare October 17, 2023 23:44
@HammadB HammadB changed the base branch from main to hammad/remove_3.7 October 17, 2023 23:57
@HammadB HammadB force-pushed the hammad/multitenancy branch from 2ae2b51 to 99b8b75 Compare October 17, 2023 23:57
@HammadB HammadB mentioned this pull request Oct 17, 2023
Merged
1 task
@HammadB HammadB changed the title [WIP] [ENH] Add multitenancy [WIP] [STACKED #1255] [ENH] Add multitenancy Oct 17, 2023
@HammadB HammadB force-pushed the hammad/multitenancy branch from 99b8b75 to 340c956 Compare October 18, 2023 00:01
@HammadB HammadB force-pushed the hammad/multitenancy branch from 340c956 to 0321fe8 Compare October 18, 2023 00:24
@HammadB HammadB mentioned this pull request Oct 19, 2023
Closed
1 task
@mattbit mattbit mentioned this pull request Oct 20, 2023
Closed
@HammadB HammadB requested a review from beggers October 23, 2023 18:33
HammadB
HammadB commented Oct 23, 2023
View reviewed changes
chromadb/api/__init__.py Outdated Show resolved Hide resolved
HammadB
HammadB commented Oct 23, 2023
View reviewed changes
chromadb/api/client.py Outdated Show resolved Hide resolved
HammadB
HammadB commented Oct 23, 2023
View reviewed changes
chromadb/config.py Outdated Show resolved Hide resolved
@HammadB HammadB requested a review from jeffchuber October 23, 2023 20:57
jeffchuber
jeffchuber approved these changes Oct 23, 2023
View reviewed changes
Copy link
Contributor

@jeffchuber jeffchuber left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you for walking me through this

HammadB
HammadB commented Oct 23, 2023
View reviewed changes
chromadb/api/client.py Outdated Show resolved Hide resolved
@HammadB HammadB changed the base branch from hammad/remove_3.7 to main October 23, 2023 22:01
atroyn
atroyn reviewed Oct 23, 2023
View reviewed changes
chromadb/config.py Outdated Show resolved Hide resolved
chromadb/__init__.py Show resolved Hide resolved
chromadb/api/__init__.py Show resolved Hide resolved
chromadb/api/client.py Outdated Show resolved Hide resolved
chromadb/api/client.py
raise ValueError("Chroma API implementation must be set in settings")
elif api_impl == "chromadb.api.segment.SegmentAPI":
if settings.is_persistent:
identifier = settings.persist_directory
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

somewhat iffy on relying on these as unique id's; why not generate uuids? is it just so the user is able to uniquely address them?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the path has to be a unique database, I don't see a way around that

@HammadB HammadB mentioned this pull request Oct 24, 2023
Open
@HammadB HammadB merged commit 0552704 into main Oct 24, 2023
85 checks passed
@HammadB HammadB mentioned this pull request Oct 24, 2023
Merged
1 task
HammadB added a commit that referenced this pull request Oct 24, 2023
@HammadB
[BUG] [TST] Multitenancy bug fix (#1276)
0b0b785 
## Description of changes

*Summarize the changes made by this PR.*
 - Improvements & Bug fixes
- Fixes a small bug that was introduced by a bad merge in #1244 .
Get_coll should look at the metadata in the model, not from the input
collection.
 - New functionality
	 - None.

## Test plan
*How are these changes tested?*
These are test changes.
- [x] Tests pass locally with `pytest` for python, `yarn test` for js

## Documentation Changes
None required.
@HammadB HammadB mentioned this pull request Oct 25, 2023
Closed
@jeffchuber jeffchuber mentioned this pull request Oct 30, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@atroyn atroyn atroyn left review comments

@beggers beggers beggers approved these changes

@jeffchuber jeffchuber jeffchuber approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@HammadB @jeffchuber @atroyn @beggers