Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

POC - batched queue processing #99

Closed
wants to merge 12 commits into from
Closed

POC - batched queue processing #99

wants to merge 12 commits into from

Commits on Mar 16, 2020

  1. Duplicate worker code

    @gregplaysguitar
    gregplaysguitar committed Mar 16, 2020
    Configuration menu
    Copy the full SHA
    4adc26c View commit details
    Browse the repository at this point in the history

  2. Batched job processing POC

    @gregplaysguitar
    gregplaysguitar committed Mar 16, 2020
    Configuration menu
    Copy the full SHA
    32f0787 View commit details
    Browse the repository at this point in the history

  3. Return early if no jobs found (as before), update tests

    @gregplaysguitar
    gregplaysguitar committed Mar 16, 2020
    Configuration menu
    Copy the full SHA
    04de4b5 View commit details
    Browse the repository at this point in the history

  4. cleanup

    @gregplaysguitar
    gregplaysguitar committed Mar 16, 2020
    Configuration menu
    Copy the full SHA
    ba6e034 View commit details
    Browse the repository at this point in the history

  5. Fix dupe filter

    @gregplaysguitar
    gregplaysguitar committed Mar 16, 2020
    Configuration menu
    Copy the full SHA
    6779565 View commit details
    Browse the repository at this point in the history

  6. optimise complete_batch function a little

    @gregplaysguitar
    gregplaysguitar committed Mar 16, 2020
    Configuration menu
    Copy the full SHA
    28bc394 View commit details
    Browse the repository at this point in the history

  7. Tiny optimisation

    @gregplaysguitar
    gregplaysguitar committed Mar 16, 2020
    Configuration menu
    Copy the full SHA
    251c5f1 View commit details
    Browse the repository at this point in the history

  8. optimise get_jobs 

    Master branch is getting ~2150 jobs/s for me

The original batched get_jobs was performing really badly at ~680 jobs/s. Using a composite type instead of json internally increased that to ~840

Replacing the select array … into with a CTE got it right back up to ~1840 jobs/s at batch size 1. With batch size 4 that jumps to ~4400.
    @gregplaysguitar
    gregplaysguitar committed Mar 16, 2020
    Configuration menu
    Copy the full SHA
    863184c View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2020

  1. Merge branch 'main' of https://github.com/graphile/worker into batche… 

    …d-queue-processing
    @gregplaysguitar
    gregplaysguitar committed Sep 21, 2020
    Configuration menu
    Copy the full SHA
    3ace8a6 View commit details
    Browse the repository at this point in the history

Commits on Sep 22, 2020

  1. Pacify prettier

    @gregplaysguitar
    gregplaysguitar committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    8f6c6f8 View commit details
    Browse the repository at this point in the history

  2. Migration bump

    @gregplaysguitar
    gregplaysguitar committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    c51f088 View commit details
    Browse the repository at this point in the history

  3. Update db dump

    @gregplaysguitar
    gregplaysguitar committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    f51638f View commit details
    Browse the repository at this point in the history