Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix polling order in contrib modules #349

Closed
wants to merge 1 commit into from

Conversation

my8100
Copy link
Collaborator

@my8100 my8100 commented Jul 18, 2019

This PR use project_priority_map to store (priority, -timestamp) as value,
so as to respect both job priorities across projects and the FIFO principle.

See also issue #187 and PR #344.

@codecov
Copy link

codecov bot commented Jul 18, 2019

Codecov Report

Merging #349 into master will increase coverage by 1.08%.
The diff coverage is 85.48%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #349      +/-   ##
==========================================
+ Coverage   68.37%   69.45%   +1.08%     
==========================================
  Files          17       20       +3     
  Lines         860      920      +60     
  Branches      104      115      +11     
==========================================
+ Hits          588      639      +51     
- Misses        242      246       +4     
- Partials       30       35       +5
Impacted Files Coverage Δ
scrapyd/utils.py 77.5% <100%> (+0.57%) ⬆️
scrapyd/contrib/fix_poll_order/spiderqueue.py 100% <100%> (ø)
scrapyd/app.py 81.81% <40%> (-4.46%) ⬇️
scrapyd/contrib/fix_poll_order/poller.py 75% <75%> (ø)
scrapyd/contrib/fix_poll_order/sqlite.py 92.85% <92.85%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4520c45...bcfb8f7. Read the comment docs.

@my8100 my8100 requested a review from Digenis July 19, 2019 02:15
@my8100
Copy link
Collaborator Author

my8100 commented Jul 22, 2019

@Digenis
How about this contrib implementation?

@Digenis
Copy link
Member

Digenis commented Jul 23, 2019

I was busy, I'll review it in a day or two

@my8100
Copy link
Collaborator Author

my8100 commented Aug 30, 2019

Any progress?

Comment on lines +27 to +31
if config.getboolean('fix_poll_order', False):
log.msg("Activating contrib modules for fix_poll_order")
poller = FixQueuePoller(config)
else:
poller = QueuePoller(config)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should do something similar to Launcher, etc. so that users can select any module (including their own), not just between these two modules.

@jpmckinney
Copy link
Contributor

This looks similar to #344 except is puts the new logic in a contrib module, and users opt-in to using that module.

As mentioned there, this multi-queue prioritization approach seems more complicated than a single queue, discussed in #187. I've now made the poller configurable, so a user could copy this code into their own project, and load it, if they prefer this approach.

@jpmckinney jpmckinney closed this Jul 19, 2024
@jpmckinney jpmckinney added status: invalid not a Scrapyd issue, an accidental PR or an improper approach and removed topic: scheduling labels Jul 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status: invalid not a Scrapyd issue, an accidental PR or an improper approach
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants