Kafka Connect: Runtime distribution with integration tests #10739
Conversation
Can we please split into two PRs. one for adding runtime distributions and one for integration tests? |
|
The integration tests depend on the runtime. |
I know. We can review and merge the runtime PR first and then rebase the integration test PR? |
|
The integration tests are testing the runtime, so I feel they belong together. Also, the runtime build is just the |
bryanck
changed the title
bryanck
changed the title
bryanck
force-pushed
the
kc-runtime
branch
3 times, most recently
from
224700b to
39d0310
Compare
| entrypoint: mc mb /data/bucket | ||
|
|
||
| iceberg: | ||
| image: tabulario/iceberg-rest |
There was a problem hiding this comment.
Should we really add this dependency?
I am not sure about the future roadmap of this project.
Can we use some OSS implementation of the catalog?
There was a problem hiding this comment.
I'm not sure if there is a better alternative but I'm happy to switch if there is one. @danielcweeks do you have concerns relying on this image?
There was a problem hiding this comment.
This is currently used by a number of projects as a test depending (python,rust,trino) and is fine for now. We've discussed moving this into the test fixtures in core and I'll work with @Fokko to get that done and we can move the test dependencies separately.
There was a problem hiding this comment.
Thanks for the update.
It would be nice to maintain and publish this image under OSS iceberg account 👍
There was a problem hiding this comment.
I can help on the DockerHub "seat" for apache/iceberg (I just did that for the new openserverless podling). Just let me know 😄
There was a problem hiding this comment.
Yes.
With that account we can even publish some more test image like Trino, kafaka-connect, Spark etc.
And we can maintain these images in Iceberg repo test module or as a separate subproject "Iceland - A playground for Iceberg"
There was a problem hiding this comment.
Thanks @jbonofre! That was one of the biggest outstanding issues because we don't currently produce any images as a project.
There was a problem hiding this comment.
@danielcweeks happy to help ! 😄
|
Thanks for the reviews @danielcweeks , @fqaiser94 , and @ajantha-bhat !! |
This PR is the next stage in submitting the Kafka Connect Iceberg sink connector, and is a follow up to #8701, #9466, #9641, and #10351.
This adds building of runtime distributions plus some integration tests. There are two runtime distributions, one that bundles the HMS client and related libraries, and one that does not. The one without the HMS client is much smaller and also doesn't have dependencies with critical security vulnerabilities. Repositories like Confluent Hub require connectors be free of critical vulnerabilities.
Remaining tasks are listed in #10740.