Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Transfer the majority of information processing from the spider to the backend. #40

Open
haxys opened this issue Mar 17, 2018 · 2 comments
Open

Transfer the majority of information processing from the spider to the backend. #40

haxys opened this issue Mar 17, 2018 · 2 comments
Labels
enhancement New feature or request
Milestone
Stable Version 1.0

Comments

@haxys
Copy link
Contributor

haxys commented Mar 17, 2018

Offloading the heavy lifting to the backend allows us to use only two spider-side API queries for processing each url rather than the hundreds it takes currently. All we'll need is one query to get the url to scan, and one query to add the discovered information to the database.
@haxys haxys added the enhancement New feature or request label Mar 17, 2018
@haxys haxys added this to the Stable Version 1.0 milestone Mar 17, 2018
@haxys
Copy link
Contributor Author

haxys commented Mar 17, 2018

See: TorSpider/TorSpider#35

@haxys
Copy link
Contributor Author

haxys commented Apr 5, 2018

This should be pretty much complete at this point, with the offload_to_backend pull request. Just test and commit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
Stable Version 1.0
Development

No branches or pull requests
1 participant
@haxys