[QUERY] Azure Schema registry + Spark Structured Streaming + Kafka/EventHub compatibility

[scaladevspark]

Is there a way to use AZURE Schema Registry with Spark Structured Streaming? (not the opensource schema registry).

Ideally I would use some high level library like Abris (https://github.com/AbsaOSS/ABRiS), to read and write avro from/to Kafka with automatically registering/ reading the df schema in the schema registry. However, I have not managed to find anything at all that could enable spark to do to_avro or from_avro against an Azure Schema Registry.

There is an open PR that could add the functionality for some column types Azure/azure-event-hubs-spark#615

There seems to be an unmainained repo that at least tried to do the from_avro part (https://github.com/arerlend/azure-schemaregistry-spark)

Anybody knows about a good way how to interact with Azure Schema Registry with Spark Structured Streaming?

[joshfree]

@conniey could you please follow up with @scaladevspark?

[conniey]

Unfortunately, I have no insights into Spark Structured Streaming. Pinging @hmlam who may have some insights.

[hmlam]

@yamin-msft is actually working on getting the EventHub Spark connector to work with Azure Schema Registry, which I think is what you want in your scenario. once we have a PR we would post it at https://github.com/Azure/azure-event-hubs-spark

[github-actions]

Hi @scaladevspark, we deeply appreciate your input into this project. Regrettably, this issue has remained unresolved for over 2 years and inactive for 30 days, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support.