refactor to get rid of persistent queues and change to singular lookup

[DGaffney]

Description

Queues could theoretically get wonky with long-held state, and also we could be more specific when selecting queues by using singular access instead of filters.

Reference: CV2-5589

How has this been tested?

Not tested locally yet

Are there any external dependencies?

Nothing changes in this PR

Have you considered secure coding practices when writing this code?

Nothing changes in this PR

[sonoransun]

Looks good!

[skyemeedan]

does it not break when run locally with elasticmq?

[DGaffney]

does it not break when run locally with elasticmq?

I believe so, yes:

app-1        | 2024-11-14 15:07:17,451 - INFO - lib.logger - Queue dev_mean_tokens__Model_output doesn't exist - creating
app-1        | 2024-11-14 15:07:17,451 - INFO - lib.logger - Using ElasticMQ Interface
elasticmq-1  | 15:07:17.465 [elasticmq-akka.actor.default-dispatcher-7] INFO  o.elasticmq.actor.QueueManagerActor - Creating queue CreateQueueData(dev_mean_tokens__Model_output,None,None,None,None,None,None,false,false,None,None,Map())
app-1        | 2024-11-14 15:07:17,481 - INFO - lib.logger - Processor listening to queues of {'dev_mean_tokens__Model_output': sqs.Queue(url='http://localhost:9324/000000000000/dev_mean_tokens__Model_output')}
app-1        | 2024-11-14 15:07:17,481 - INFO - lib.logger - Beginning callback loop...

And when pushing a message,

>>> message = {"id": "123"}
>>> queue.push_message(f"{queue_prefix}{process_name}{queue_suffix}", schemas.parse_input_message({"body": message, "model_name": process_name}))
Message(body=GenericItem(id='123', content_hash=None, callback_url=None, url=None, text=None, raw={}, parameters={}, result=MediaResponse(hash_value=None)), model_name='image__Model', retry_count=0)

[sentry-io]

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

• ‼️ KeyError: 'endpoint_resolver' /process_item/{process_name} View Issue

_{Did you find this useful? React with a 👍 or 👎}