-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add import bulk tuples #217
add import bulk tuples #217
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Small change regarding prefixing relations before creating them
Co-authored-by: Jonathan Marcantonio <[email protected]>
LGTM |
/lgtm - @akoserwal dont mind the failing github action of IQE - feel free to merge it |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
internal/data/spicedb.go
Outdated
tuple.Relation = addRelationPrefix(tuple.Relation, relationPrefix) | ||
batch = append(batch, createSpiceDbRelationship(tuple)) | ||
} | ||
client, err := s.client.ImportBulkRelationships(context.Background()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If I am reading this right, it looks like this method...
- Gets a batch from the input stream
- Starts a streaming RPC with SpiceDB
- Sends one batch on that stream
- Closes that stream
And then repeats for each batch.
However what we want is:
- Start the spicedb stream
- For each batch, send that batch to SpiceDB
- Then close the stream after there are no more input batches (the client closes the stream on Relations end)
In other words, if a client has one streaming RPC with Relations, there should only be one equivalent streaming RPC with SpiceDB.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch, thank you for the suggest. I update the code.
930eb93
to
d75289e
Compare
internal/data/spicedb.go
Outdated
if !errors.Is(streamErr, io.EOF) { | ||
if streamErr = stream.SendAndClose(&apiV1beta1.ImportBulkTuplesResponse{NumImported: totalImported}); err != nil { | ||
return status.Errorf(codes.Internal, "failed to send response: %v", streamErr) | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure about this one. It's setting streamErr
but checking err
? And looking at example gRPC client streaming server docs, it looks like they show if the error from the client Recv()
is not EOF, they return that rather than SendAndClose. What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, I agree.
d75289e
to
ed8fe09
Compare
ed8fe09
to
db04c5f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Thanks for iterating on it.
PR Template:
Describe your changes
response
Ticket reference (if applicable)
Fixes #
Checklist
Are the agreed upon acceptance criteria fulfilled?
Was the 4-eye-principle applied? (async PR review, pairing, ensembling)
Do your changes have passing automated tests and sufficient observability?
Are the work steps you introduced repeatable by others, either through automation or documentation?
The Changes were automatically built, tested, and - if needed, behind a feature flag - deployed to our production environment. (Please check this when the new deployment is done and you could verify it.)
Are the agreed upon coding/architectural practices applied?
Are security needs fullfilled? (e.g. no internal URL)
Is the corresponding Ticket in the right state? (should be on "review" now, put to done when this change made it to production)
For changes to the public API / code dependencies: Was the whole team (or a sufficient amount of ppl) able to review?