Ability to receive a notification/check when a queue item has been completed/abandoned(or removed).

[niemyjski]

Currently we are abandoning queue items when there is an error which is great. But in some scenarios this doesn't work the greatest for example. When the queue item has been abandoned more times then the retry limit, there is no way to see when the queue item will be removed. As such we can't clean up data data payloads on disk that the queue item points to.

[ejsmith]

I am going to add a FileQueue where you can store the queue payload in file storage and then put a file marker item in the queue. This is a common pattern and we should make it easier.

[niemyjski]

Sounds good 👍

[niemyjski]

Do we want to make any options available to say move any non processed items to a error folder? It might be nice to take a look at adbandoned items to figure out why they were discarded.. thoughts? In our case an event might not be able to be processed due to a serialization error. It would suck to discard them and never be able to reproduce.

[ejsmith]

Yeah, we would pass in an IFileStorage impl and we would also need paths for where you want to store the files. We also allow archiving of completed items. So it would need to specify a queue folder, archive folder and deadletter folder. Probably have a deadletter TTL policy on it as well where it would cleanup deadletter files after a certain time. I think queues have a deadletter ttl already so the file storage should probably just honor that setting.

[niemyjski]

yeah, I kind of agree. But how is it supposed to know what folder it's supposed to go into cause that's dynamically changing.. It's almost like this thing should be dumb. Maybe we call a method to calculate it and pass the queue item??

[ejsmith]

Yeah, I think the concrete queue would have to provide logic to figure out an archive folder.

[niemyjski]

@ejsmith status on this issue?