OrmLite MQ

[DeonHeyns]

Background MQ and the other MQ providers are really solid. However, it would be great if we could have an OrmLite MQ similar to Solid Queue from Rails. I know implementation wise this could be optimized for the different types of database providers supported by OrmLite but something simple would do and over a few iterations we could create optimized versions for Postresql etc.

[mythz]

Thanks for opening this feature request, we're actually looking into creating a Backgrounds Jobs feature to be able to execute APIs or Commands using OrmLite, but we're only looking at supporting SQLite in order to avoid needing any external infrastructure dependencies and also to be able to create naturally archivable monthly databases.

This will be a good thread to gather requirements for this feature for executing persistent background jobs.

• Execute ServiceStack APIs or Commands in managed background threads
  • Support for push-based notification to execute callback URL on completion + APIs to query completed jobs
• Uses SQLite
  • Primary database maintains active queue
  • After completion (success of failure) job gets archived in monthly databases
• Admin UI to monitor active and completed jobs

Let me know if there's other specific features you would like in v1 and what use-cases you would intend to use this for.

[DeonHeyns]

That would be a solid v1.

I’m not too concerned about archiving the SQLite DBs. I would probably have logic around the Background jobs to store the initial request and result. The push-based notifications are going to be great.

Main use case is going to be offloading any long running process from the request:
ETL of data
Sending emails
Number crunching

Features available to see or action in the Admin-ui:
Retry count
Last executed
Purge/move to archive

[mythz]

ok cool sounds like we're on the right track them.

Using SQLite means there wouldn't be any need to archive/purge since completed/failed tasks would naturally be written into they're own monthly db, e.g. /2024-06/app.db.

[DeonHeyns]

Definitely on the right track. Are those archives viewable from the Admin-ui? Can I see the physical DBs and the past jobs?

[mythz]

Yeah you'd be able to view each the completed and failed jobs for each month.

[sbosell]

In the past I've always used Hangfire with Servicestack for offline processing. Building something more integrated would be welcome. It might be beyond the scope here but being able to do some kind of rate limiting over a sliding time period would be awesome. An example would be If you think about sending email marketing across various clients you don't want one campaign or client to monopolize the sending in a given time frame. This could be applied to handing different scaling tasks so that competing work can be distributed in without having to handle it "manually".

[mythz]

FYI we've developed CreatorKit which we use to send our newsletter emails. They're all sent in the background using Background MQ, which is also used to send emails in all new Identity Auth templates.

I'm looking at implementing a "logical thread" feature where you can optionally queue jobs to be executed by a named thread, so you could queue client tasks to be executed and throttled by their own thread. But that's about it, any other rate limiting would need to be handled within the custom command or API that's executed.

[sbosell]

That's awesome. I'll check it out.

I figured it would be beyond the scope of what you might be building. That being said you might consider providing some hooks that would allow some custom code to determine which jobs to pick up.

[mythz]

This has now been implemented with our new Background Jobs Feature in ServiceStack v8.4