Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add import bulk tuples #217

Merged
merged 6 commits into from
Oct 7, 2024

Conversation

akoserwal
Copy link
Contributor

@akoserwal akoserwal commented Oct 4, 2024

PR Template:

Describe your changes

  • Add Bulk Import tuples
grpcurl -plaintext -d @ localhost:9000 kessel.relations.v1beta1.KesselTupleService.ImportBulkTuples <<EOM
  {
  "tuples": [
     {
    "resource": {
      "type": {
        "namespace": "rbac",
        "name": "group"
      },
      "id": "bobclub"
    },
    "relation": "member",
    "subject": {
      "subject": {
        "type": {
          "namespace": "rbac",
          "name": "user"
        },
        "id": "bob"
      }
    }
  }
  ]
}
EOM

response

{
  "numImported": "1"
}
  • ...

Ticket reference (if applicable)

Fixes #

Checklist

  • Are the agreed upon acceptance criteria fulfilled?

  • Was the 4-eye-principle applied? (async PR review, pairing, ensembling)

  • Do your changes have passing automated tests and sufficient observability?

  • Are the work steps you introduced repeatable by others, either through automation or documentation?

    • If automation is possible but not done due to other constraints, a ticket to the tech debt sprint is added
    • An SOP (Standard Operating Procedure) was created
  • The Changes were automatically built, tested, and - if needed, behind a feature flag - deployed to our production environment. (Please check this when the new deployment is done and you could verify it.)

  • Are the agreed upon coding/architectural practices applied?

  • Are security needs fullfilled? (e.g. no internal URL)

  • Is the corresponding Ticket in the right state? (should be on "review" now, put to done when this change made it to production)

  • For changes to the public API / code dependencies: Was the whole team (or a sufficient amount of ppl) able to review?

@akoserwal akoserwal marked this pull request as ready for review October 4, 2024 11:54
Copy link
Contributor

@lennysgarage lennysgarage left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small change regarding prefixing relations before creating them

internal/data/spicedb_test.go Outdated Show resolved Hide resolved
internal/data/spicedb.go Show resolved Hide resolved
@merlante
Copy link
Contributor

merlante commented Oct 4, 2024

LGTM

@Rajagopalan-Ranganathan
Copy link
Contributor

/lgtm - @akoserwal dont mind the failing github action of IQE - feel free to merge it

Copy link
Contributor

@lennysgarage lennysgarage left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

tuple.Relation = addRelationPrefix(tuple.Relation, relationPrefix)
batch = append(batch, createSpiceDbRelationship(tuple))
}
client, err := s.client.ImportBulkRelationships(context.Background())
Copy link
Member

@alechenninger alechenninger Oct 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I am reading this right, it looks like this method...

  1. Gets a batch from the input stream
  2. Starts a streaming RPC with SpiceDB
  3. Sends one batch on that stream
  4. Closes that stream

And then repeats for each batch.

However what we want is:

  1. Start the spicedb stream
  2. For each batch, send that batch to SpiceDB
  3. Then close the stream after there are no more input batches (the client closes the stream on Relations end)

In other words, if a client has one streaming RPC with Relations, there should only be one equivalent streaming RPC with SpiceDB.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thank you for the suggest. I update the code.

@akoserwal akoserwal force-pushed the RHCLOUD-35457-bulk-imports branch from 930eb93 to d75289e Compare October 4, 2024 15:50
if !errors.Is(streamErr, io.EOF) {
if streamErr = stream.SendAndClose(&apiV1beta1.ImportBulkTuplesResponse{NumImported: totalImported}); err != nil {
return status.Errorf(codes.Internal, "failed to send response: %v", streamErr)
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure about this one. It's setting streamErr but checking err? And looking at example gRPC client streaming server docs, it looks like they show if the error from the client Recv() is not EOF, they return that rather than SendAndClose. What do you think?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, I agree.

@akoserwal akoserwal force-pushed the RHCLOUD-35457-bulk-imports branch from d75289e to ed8fe09 Compare October 4, 2024 16:08
@akoserwal akoserwal force-pushed the RHCLOUD-35457-bulk-imports branch from ed8fe09 to db04c5f Compare October 4, 2024 16:28
Copy link
Member

@alechenninger alechenninger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for iterating on it.

@akoserwal akoserwal merged commit fb97833 into project-kessel:main Oct 7, 2024
9 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants