How can I queue or asynchronously generate certificate and get a callback once certificate is generated?

[aks30]

Is there an alternative approach to generate certificates apart from calling the generate API synchronously and waiting for the response so as to not worry about the Sunbird RC service being not available or delaying in giving a response in case of heavy load.

Cc - @aks30 @vijiurs @kiranharidas187 @varsha-sl @Prateek-slokam @aishwaryashikshalokam @priyanka-TL

[pramodkvarma]

Yes, that's how it was implemented in CoWin using Kafka. @dileepbapat can shed some light.

[aks30]

@tejash-jl this one is for an entity where else we were looking for the certificate async and sync generation?

[aks30]

Hi, @tejash-jl is the async API presently available only for entities and not certificates?

[tejash-jl]

@aks30
When an entity gets created, if there is a credential subject configured in the schema, we create a VC for that entity. The generated VC can be fetched as JSON/certificate (PDF/HTML/SVG). The complete flow is sync.
In async mode, when the create entity api is triggered, it pushes the entity to Kafka and the rest of the functionality gets completed by the consumer.

[aks30]

@tejash-jl What we were looking for is creating certificates against the existing entities in async mode. For example in a training certificate entity as and when different users complete their training for various things we want to generate certificates for them in async mode. Is that possible right now? Is it the same as an entity like at present i.e. at a time it can be working on sync or async mode? If so how do we make it work in both modes at the same time.

[sankarshanmukhopadhyay]

@tejash-jl What we were looking for is creating certificates against the existing entities in async mode. For example in a training certificate entity as and when different users complete their training for various things we want to generate certificates for them in async mode. Is that possible right now? Is it the same as an entity like at present i.e. at a time it can be working on sync or async mode? If so how do we make it work in both modes at the same time.

I've been following this very interesting conversation (especially the use of a message queue etc) However, this model seems to be more of a choice between batch mode and just-in-time mode of generating and issuing certificates, is it not? A learner might complete training A today and receive a certificate and training B next week and receive another certificate then.

[tejash-jl]

@aks30,
yes, we have async functionality with webhook callback funcationality,
to enable async you need to set the below environment variables,
https://github.com/tejash-jl/ref-sunbird-certificate-demo/blob/async_mode/docker-compose.yml#L56-L60

also find the jupyter playbook for testing the apis,
https://github.com/tejash-jl/ref-sunbird-certificate-demo/blob/async_mode/async-mode-api.ipynb

[aks30]

@tejash-jl Can you suggest why for the statement - "I would suggest to use the API, instead of pushing directly to Kafka topic", as it would mean we would have to handle failure and overload scenario in our side.

[tejash-jl]

just for clarification,
I meant to use the API with async_mode enabled so that the Sunbird-RC internally handles pushing it to the queue and also it calls the webhook for both success & failure events.

The other things that will be handled using the api:

• the input will be validated against the schema definition
• creation of user entities in keycloak (optional)

[pramodkvarma]

Definitely better to use API as it abstracts away queue/async tech from consumer. If we later allow adapters to Rabbit MQ etc, you will need to change code.

[aks30]

@pramodkvarma , @tejash-jl the benefit of using Kafka directly would be to handle the very scenario we mentioned in the first post i.e. in case RC service is down or under heavy load the message is there in the queue and RC can start processing once its available.

[pramodkvarma]

Yes, then you are making assumption about the actual format and Kafka as the pipe. If this is what you desire, you are better off managing Kafka on your side, push the message, and have a service that takes each message and call RC in async mode with retries etc. Using internal message format and bypassing RC async API is a bad idea from upgradeability etc.