Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate Celery support for laborious, asynchronous tasks. #21

Closed
haxys opened this issue Mar 2, 2018 · 1 comment
Closed

Integrate Celery support for laborious, asynchronous tasks. #21

haxys opened this issue Mar 2, 2018 · 1 comment
Labels
enhancement New feature or request

Comments

@haxys
Copy link
Contributor

haxys commented Mar 2, 2018

In order to lighten the burden of the backend, let's offload some of the more processing-intensive and repetitive tasks to Celery:

1. Populating the url_queue table.
2. Rebuilding the top-20 lists so they're not constantly being refreshed live.
3. Rebuilding the various page indexes and search tables.
4. Eventually, building a daily snapshot of the node links for the force-directed graph.
5. Processing newly-submitted .onions from the public.

We'll use RabbitMQ as the broker.

@haxys haxys added the enhancement New feature or request label Mar 2, 2018
@haxys haxys added this to the Stable Version 1.0 milestone Mar 2, 2018
@haxys
Copy link
Contributor Author

haxys commented Mar 3, 2018

We could also use Celery to generate a database dump of interesting data for people to download while search functionality is not yet available.

@artagel artagel mentioned this issue Mar 4, 2018
artagel added a commit that referenced this issue Mar 4, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant