Handeling time expensive tasks with Celery

[timatooth]

Hi there,

I'm thinking of extending photologue so that time intensive tasks such as uploading zips, image resampleing or applying watermarks gets done in a Celery worker backed by a message broker such as rabbitMQ. Would you be interested in merging something like that into the project if I tried to keep it as an optional thing to configure?

[celiao]

Hi Tim --

I implemented a Celery worker to generate the additional image sizes in my
fork of Photologue: https://github.com/celiao/django-photologue. Richard
only included the portion of my pull request implementing the Django File
Storage API (allowing storage on Amazon S3 for example). You might be able
to generalize my Celery implementation for all the time intensive tasks.

-- Celia

On Sat, Aug 30, 2014 at 1:16 AM, Tim Sullivan [email protected]
wrote:

Hi there,

I'm thinking of extending photologue so that time intensive tasks such as
uploading zips, image resampleing or applying watermarks gets done in a
Celery worker backed by a message broker such as rabbitMQ. Would you be
interested in merging something like that into the project if I tried to
keep it as an optional thing to configure?

—
Reply to this email directly or view it on GitHub
#100.

[richardbarran]

Hi,
It's an interesting thought - the question had already been brought up by another contributor - see #90.

I think that it's something that should stay out of the core Photologue code, as for most users, it's not necessary - and, if performance is an issue and suggests the use of asynchronous Qs, Celery is not the only option anyway. Personally, I've tested both Celery and django-rq, and I far prefer django-rq! But that's just a personal opinion.
My point is, there is more than one asynchronous Q solution out there, and it's often not needed anyway.

I had previously suggested to Celiao to fork Photologue and implement this feature separately.

However - if there is a strong demand for this feature and if it could be implemented in a pluggable way, I would be open to adding it.

By a 'pluggable way', I mean that if the code that generates thumbnails could easily be moved into separate methods, and the calls to those methods could then be overridden by custom settings - then it becomes straightforward, on a per-project basis, to switch the thumbnail generator methods to some custom code - for example, something that uses Celery. And this keeps the 'core' Photologue codebase clean and simple.

Such an approach is fairly common in both Django itself, and in 3rd party application - just yesterday I was looking at an example http://django-embed-video.readthedocs.org/en/v0.11/examples.html#custom-backends

Thoughts/suggestions welcome!

[timatooth]

Thanks @celiao and @richardbarran

I haven't looked at django-rq before so will look at that some time this week. Totally agree about the need for it to be pluggable since setting up a broker and async queue library hookups becomes quite a chore.

I might try adapt the idea of 'Backends' as you sugest. That way there could be a RQBackend and CeleryBackend plus of course the existing PhotologueBackend could all co-exist without enforcing more dependicies unless they were enabled right?

Additionally, would it also be ideal to display a progress bar to the Django Admin interface using AJAX polling so in the case of uploading a large zip of say; 400x 20+ Megapixel images you see progress of the precaching process in visual form somewhere? If doing that in Django Admin isn't suitable how about a Drag n Drop clone of Facebooks/Flickr album uploader using a HTML5 drag n drop zone at a URL like /gallery/album/new or gallery/album/some-existing-album-name/upload

[richardbarran]

Hi @timatooth,

Pluggable backends - totally agree with what you say. However I haven't looked at the code to see how easy it would be to implement!

About a progress bar: that's a part of Django that I really haven't explored at all. If you have any links to examples/blog posts/snippets, could you please post them so I that I can read up on the subject?

Likewise for HTML5 drag'n'drop zones - I'm more of a backend coder :-)