Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update rq to 2.0.0 #3118

Closed
wants to merge 1 commit into from
Closed

Update rq to 2.0.0 #3118

wants to merge 1 commit into from

Conversation

pyup-bot
Copy link
Collaborator

This PR updates rq from 0.12.0 to 2.0.0.

Changelog

2.0

New Features:
* Multiple job executions: a job can now have multiple executions running simultaneously. This will enable future support for long running scheduled jobs. Thanks selwin!
* `Worker(default_worker_ttl=10)` is deprecated in favor of `Worker(worker_ttl=10)`. Thanks stv8!
* Added a `cleanup` parameter to `registry.get_job_ids()` and `registry.get_job_count()`. Thanks anton-daneyko-ultramarin!
* Added support for AWS Elasticache Serverless Redis. Thanks bobbywatson3!
* You can now specify TTL for deferred jobs. Thanks hberntsen!
* RQ's code base is now typed (mostly). Thanks terencehonles!
* Other minor fixes and improvements. Thanks hongquan, rbange, jackkinsella, terencehonles, wckao, sim6!

Breaking Changes:
* Dropped support for Redis server < 4
* `RoundRobinWorker` and `RandomWorker` are deprecated. Use  `--dequeue-strategy <round-robin/random>` instead.
* `Job.__init__` requires both `id` and `connection` to be passed in.
* `Job.exists()` requires `connection` argument to be passed in.
* `Queue.all()` requires `connection` argument.
* `job` decorator now requires `connection` argument.
* Built in Sentry integration has been removed. To use Sentry with RQ, please refer to [Sentry's docs](https://docs.sentry.io/platforms/python/integrations/rq/).

Bug Fixes:
* Fixed an issue where abandoned jobs are sometimes not enqueued. Thanks Marishka17!
* Fixes an issue where Redis connection does not expose `name` attribute. Thanks wckao!
* `job.get_status()` will now always return `JobStatus` enum. Thanks indepndnt!
* Queue key should always be created even if jobs are deferred. Thanks sim6!
* RQ's pubsub thread will now attempt to reconnect on Redis connection errors. Thanks fcharlier!

1.16.2

* Fixed a bug that may cause jobs from intermediate queue to be moved to FailedJobRegistry. Thanks selwin!

1.16.1

* Added `worker_pool.get_worker_process()` to make `WorkerPool` easier to extend. Thanks selwin!

1.16

* Added a way for jobs to wait for latest result `job.latest_result(timeout=60)`. Thanks ajnisbet!
* Fixed an issue where `stopped_callback` is not respected when job is enqueued via `enqueue_many()`. Thanks eswolinsky3241!
* `worker-pool` no longer ignores `--quiet`. Thanks Mindiell!
* Added compatibility with AWS Serverless Redis. Thanks peter-gy!
* `worker-pool` now starts with scheduler. Thanks chromium7!

1.15.1

* Fixed a bug that may cause a crash when cleaning intermediate queue. Thanks selwin!
* Fixed a bug that may cause canceled jobs to still run dependent jobs. Thanks fredsod!

1.15

* Added `Callback(on_stopped='my_callback)`. Thanks eswolinsky3241!
* `Callback` now accepts dotted path to function as input. Thanks rishabh-ranjan!
* `queue.enqueue_many()` now supports job dependencies. Thanks eswolinsky3241!
* `rq worker` CLI script now configures logging based on `DICT_CONFIG` key present in config file. Thanks juur!
* Whenever possible, `Worker` now uses `lmove()` to implement [reliable queue pattern](https://redis.io/commands/lmove/). Thanks selwin!
* Require `redis>=4.0.0`
* `Scheduler` should only release locks that it successfully acquires. Thanks xzander!
* Fixes crashes that may happen by changes to `as_text()` function in v1.14. Thanks tchapi!
* Various linting, CI and code quality improvements. Thanks robhudson!

1.14.1

* Fixes a crash that happens if Redis connection uses SSL. Thanks tchapi!
* Fixes a crash if `job.meta()` is loaded using the wrong serializer. Thanks gabriels1234!

1.14.0

* Added `WorkerPool` (beta) that manages multiple workers in a single CLI. Thanks selwin!
* Added a new `Callback` class that allows more flexibility in declaring job callbacks. Thanks ronlut!
* Fixed a regression where jobs with unserializable return value crashes RQ. Thanks tchapi!
* Added `--dequeue-strategy` option to RQ's CLI. Thanks ccrvlh!
* Added `--max-idle-time` option to RQ's worker CLI. Thanks ronlut!
* Added `--maintenance-interval` option to RQ's worker CLI. Thanks ronlut!
* Fixed RQ usage in Windows as well as various other refactorings. Thanks ccrvlh!
* Show more info on `rq info` CLI command. Thanks iggeehu!
* `queue.enqueue_jobs()` now properly account for job dependencies. Thanks sim6!
* `TimerDeathPenalty` now properly handles negative/infinite timeout. Thanks marqueurs404!

1.13.0

* Added `work_horse_killed_handler` argument to `Worker`. Thanks ronlut!
* Fixed an issue where results aren't properly persisted on synchronous jobs. Thanks selwin!
* Fixed a bug where job results are not properly persisted when `result_ttl` is `-1`. Thanks sim6!
* Various documentation and logging fixes. Thanks lowercase00!
* Improve Redis connection reliability. Thanks lowercase00!
* Scheduler reliability improvements. Thanks OlegZv and lowercase00!
* Fixed a bug where `dequeue_timeout` ignores `worker_ttl`. Thanks ronlut!
* Use `job.return_value()` instead of `job.result` when processing callbacks. Thanks selwin!
* Various internal refactorings to make `Worker` code more easily extendable. Thanks lowercase00!
* RQ's source code is now black formatted. Thanks aparcar!

1.12.0

* RQ now stores multiple job execution results. This feature is only available on Redis >= 5.0 Redis Streams. Please refer to [the docs](https://python-rq.org/docs/results/) for more info. Thanks selwin!
* Improve performance when enqueueing many jobs at once. Thanks rggjan!
* Redis server version is now cached in connection object. Thanks odarbelaeze!
* Properly handle `at_front` argument when jobs are scheduled. Thanks gabriels1234!
* Add type hints to RQ's code base. Thanks lowercase00!
* Fixed a bug where exceptions are logged twice. Thanks selwin!
* Don't delete `job.worker_name` after job is finished. Thanks eswolinsky3241!

1.11.1

* `queue.enqueue_many()` now supports `on_success` and on `on_failure` arguments. Thanks y4n9squared!
* You can now pass `enqueue_at_front` to `Dependency()` objects to put dependent jobs at the front when they are enqueued. Thanks jtfidje!
* Fixed a bug where workers may wrongly acquire scheduler locks. Thanks milesjwinter!
* Jobs should not be enqueued if any one of it's dependencies is canceled. Thanks selwin!
* Fixed a bug when handling jobs that have been stopped. Thanks ronlut!
* Fixed a bug in handling Redis connections that don't allow `SETNAME` command. Thanks yilmaz-burak!

1.11

* This will be the last RQ version that supports Python 3.5.
* Allow jobs to be enqueued even when their dependencies fail via `Dependency(allow_failure=True)`. Thanks mattchan-tencent, caffeinatedMike and selwin!
* When stopped jobs are deleted, they should also be removed from FailedJobRegistry. Thanks selwin!
* `job.requeue()` now supports `at_front()` argument. Thanks buroa!
* Added ssl support for sentinel connections. Thanks nevious!
* `SimpleWorker` now works better on Windows. Thanks caffeinatedMike!
* Added `on_failure` and `on_success` arguments to job decorator. Thanks nepta1998!
* Fixed a bug in dependency handling. Thanks th3hamm0r!
* Minor fixes and optimizations by xavfernandez, olaure, kusaku.

1.10.1

* **BACKWARDS INCOMPATIBLE**: synchronous execution of jobs now correctly mimics async job execution. Exception is no longer raised when a job fails, job status will now be correctly set to `FAILED` and failure callbacks are now properly called when job is run synchronously. Thanks ericman93!
* Fixes a bug that could cause job keys to be left over when `result_ttl=0`. Thanks selwin!
* Allow `ssl_cert_reqs` argument to be passed to Redis. Thanks mgcdanny!
* Better compatibility with Python 3.10. Thanks rpkak!
* `job.cancel()` should also remove itself from registries. Thanks joshcoden!
* Pubsub threads are now launched in `daemon` mode. Thanks mik3y!

1.10.0

* You can now enqueue jobs from CLI. Docs [here](https://python-rq.org/docs/#cli-enqueueing). Thanks rpkak!
* Added a new `CanceledJobRegistry` to keep track of canceled jobs. Thanks selwin!
* Added custom serializer support to various places in RQ. Thanks joshcoden!
* `cancel_job(job_id, enqueue_dependents=True)` allows you to cancel a job while enqueueing its dependents. Thanks joshcoden!
* Added `job.get_meta()` to fetch fresh meta value directly from Redis. Thanks aparcar!
* Fixes a race condition that could cause jobs to be incorrectly added to FailedJobRegistry. Thanks selwin!
* Requeueing a job now clears `job.exc_info`. Thanks selwin!
* Repo infrastructure improvements by rpkak.
* Other minor fixes by cesarferradas and bbayles.

1.9.0

* Added success and failure callbacks. You can now do `queue.enqueue(foo, on_success=do_this, on_failure=do_that)`. Thanks selwin!
* Added `queue.enqueue_many()` to enqueue many jobs in one go. Thanks joshcoden!
* Various improvements to CLI commands. Thanks rpkak!
* Minor logging improvements. Thanks clavigne and natbusa!

1.8.1

* Jobs that fail due to hard shutdowns are now retried. Thanks selwin!
* `Scheduler` now works with custom serializers. Thanks alella!
* Added support for click 8.0. Thanks rpkak!
* Enqueueing static methods are now supported. Thanks pwws!
* Job exceptions no longer get printed twice. Thanks petrem!

1.8.0

* You can now declare multiple job dependencies. Thanks skieffer and thomasmatecki for laying the groundwork for multi dependency support in RQ.
* Added `RoundRobinWorker` and `RandomWorker` classes to control how jobs are dequeued from multiple queues. Thanks bielcardona!
* Added `--serializer` option to `rq worker` CLI. Thanks f0cker!
* Added support for running asyncio tasks. Thanks MyrikLD!
* Added a new `STOPPED` job status so that you can differentiate between failed and manually stopped jobs. Thanks dralley!
* Fixed a serialization bug when used with job dependency feature. Thanks jtfidje!
* `clean_worker_registry()` now works in batches of 1,000 jobs to prevent modifying too many keys at once. Thanks AxeOfMen and TheSneak!
* Workers will now wait and try to reconnect in case of Redis connection errors. Thanks Asrst!

1.7.0

* Added `job.worker_name` attribute that tells you which worker is executing a job. Thanks selwin!
* Added `send_stop_job_command()` that tells a worker to stop executing a job. Thanks selwin!
* Added `JSONSerializer` as an alternative to the default `pickle` based serializer. Thanks JackBoreczky!
* Fixes `RQScheduler` running on Redis with `ssl=True`. Thanks BobReid!

1.6.1

* Worker now properly releases scheduler lock when run in burst mode. Thanks selwin!

1.6.0

* Workers now listen to external commands via pubsub. The first two features taking advantage of this infrastructure are `send_shutdown_command()` and `send_kill_horse_command()`. Thanks selwin!
* Added `job.last_heartbeat` property that's periodically updated when job is running. Thanks theambient!
* Now horses are killed by their parent group. This helps in cleanly killing all related processes if job uses multiprocessing. Thanks theambient!
* Fixed scheduler usage with Redis connections that uses custom parser classes. Thanks selwin!
* Scheduler now enqueue jobs in batches to prevent lock timeouts. Thanks nikkonrom!
* Scheduler now follows RQ worker's logging configuration. Thanks christopher-dG!

1.5.2

* Scheduler now uses the class of connection that's used. Thanks pacahon!
* Fixes a bug that puts retried jobs in `FailedJobRegistry`. Thanks selwin!
* Fixed a deprecated import. Thanks elmaghallawy!

1.5.1

* Fixes for Redis server version parsing. Thanks selwin!
* Retries can now be set through job decorator. Thanks nerok!
* Log messages below logging.ERROR is now sent to stdout. Thanks selwin!
* Better logger name for RQScheduler. Thanks atainter!
* Better handling of exceptions thrown by horses. Thanks theambient!

1.5.0

* Failed jobs can now be retried. Thanks selwin!
* Fixed scheduler on Python > 3.8.0. Thanks selwin!
* RQ is now aware of which version of Redis server it's running on. Thanks aparcar!
* RQ now uses `hset()` on redis-py >= 3.5.0. Thanks aparcar!
* Fix incorrect worker timeout calculation in SimpleWorker.execute_job(). Thanks davidmurray!
* Make horse handling logic more robust. Thanks wevsty!

1.4.3

* Added `job.get_position()` and `queue.get_job_position()`. Thanks aparcar!
* Longer TTLs for worker keys to prevent them from expiring inside the worker lifecycle. Thanks selwin!
* Long job args/kwargs are now truncated during logging. Thanks JhonnyBn!
* `job.requeue()` now returns the modified job. Thanks ericatkin!

1.4.2

* Reverted changes to `hmset` command which causes workers on Redis server < 4 to crash. Thanks selwin!
* Merged in more groundwork to enable jobs with multiple dependencies. Thanks thomasmatecki!

1.4.1

* Default serializer now uses `pickle.HIGHEST_PROTOCOL` for backward compatibility reasons. Thanks bbayles!
* Avoid deprecation warnings on redis-py >= 3.5.0. Thanks bbayles!

1.4.0

* Custom serializer is now supported. Thanks solababs!
* `delay()` now accepts `job_id` argument. Thanks grayshirt!
* Fixed a bug that may cause early termination of scheduled or requeued jobs. Thanks rmartin48!
* When a job is scheduled, always add queue name to a set containing active RQ queue names. Thanks mdawar!
* Added `--sentry-ca-certs` and `--sentry-debug` parameters to `rq worker` CLI. Thanks kichawa!
* Jobs cleaned up by `StartedJobRegistry` are given an exception info. Thanks selwin!
* Python 2.7 is no longer supported. Thanks selwin!

1.3.1

- Run `rqworker` with `--sentry_dsn=""` to disable Sentry integration. Thanks Bolayniuss!
- Support for `SSL` Redis kwarg. Thanks ajknv!
- `rqworker`and `rqscheduler` management commands now uses RQ's built in `setup_loghandlers` function. Thanks Paulius-Maruska!
- Remove the use of deprecated `admin_static` template tag. Thanks lorenzomorandini!

1.3.0

* Support for infinite job timeout. Thanks theY4Kman!
* Added `__main__` file so you can now do `python -m rq.cli`. Thanks bbayles!
* Fixes an issue that may cause zombie processes. Thanks wevsty!
* `job_id` is now passed to logger during failed jobs. Thanks smaccona!
* `queue.enqueue_at()` and `queue.enqueue_in()` now supports explicit `args` and `kwargs` function invocation. Thanks selwin!

1.2.2

* `Job.fetch()` now properly handles unpickleable return values. Thanks selwin!

1.2.1

* `enqueue_at()` and `enqueue_in()` now sets job status to `scheduled`. Thanks coolhacker170597!
* Failed jobs data are now automatically expired by Redis. Thanks selwin!
* Fixes `RQScheduler` logging configuration. Thanks FlorianPerucki!

1.2.0

* This release also contains an alpha version of RQ's builtin job scheduling mechanism. Thanks selwin!
* Various internal API changes in preparation to support multiple job dependencies. Thanks thomasmatecki!
* `--verbose` or `--quiet` CLI arguments should override `--logging-level`. Thanks zyt312074545!
* Fixes a bug in `rq info` where it doesn't show workers for empty queues. Thanks zyt312074545!
* Fixed `queue.enqueue_dependents()` on custom `Queue` classes. Thanks van-ess0!
* `RQ` and Python versions are now stored in job metadata. Thanks eoranged!
* Added `failure_ttl` argument to job decorator. Thanks pax0r!

1.1.0

- Added `max_jobs` to `Worker.work` and `--max-jobs` to `rq worker` CLI. Thanks perobertson!
- Passing `--disable-job-desc-logging` to `rq worker` now does what it's supposed to do. Thanks janierdavila!
- `StartedJobRegistry` now properly handles jobs with infinite timeout. Thanks macintoshpie!
- `rq info` CLI command now cleans up registries when it first runs. Thanks selwin!
- Replaced the use of `procname` with `setproctitle`. Thanks j178!

1.0.1

- Django 2.0 compatibility fixes.
- Minor bug fixes

1.0

Backward incompatible changes:

- `job.status` has been removed. Use `job.get_status()` and `job.set_status()` instead. Thanks selwin!

- `FailedQueue` has been replaced with `FailedJobRegistry`:
* `get_failed_queue()` function has been removed. Please use `FailedJobRegistry(queue=queue)` instead.
* `move_to_failed_queue()` has been removed.
* RQ now provides a mechanism to automatically cleanup failed jobs. By default, failed jobs are kept for 1 year.
* Thanks selwin!

- RQ's custom job exception handling mechanism has also changed slightly:
* RQ's default exception handling mechanism (moving jobs to `FailedJobRegistry`) can be disabled by doing `Worker(disable_default_exception_handler=True)`.
* Custom exception handlers are no longer executed in reverse order.
* Thanks selwin!

- `Worker` names are now randomized. Thanks selwin!

- `timeout` argument on `queue.enqueue()` has been deprecated in favor of `job_timeout`. Thanks selwin!

- Sentry integration has been reworked:
* RQ now uses the new [sentry-sdk](https://pypi.org/project/sentry-sdk/) in place of the deprecated [Raven](https://pypi.org/project/raven/) library
* RQ will look for the more explicit `RQ_SENTRY_DSN` environment variable instead of `SENTRY_DSN` before instantiating Sentry integration
* Thanks selwin!

- Fixed `Worker.total_working_time` accounting bug. Thanks selwin!

1.0.0

-   You can now view worker information
-   Detailed worker statistics such as failed/completed job count are
 now shown (requires RQ >= 0.9.0). Thanks seiryuz!
-   `rqstats` management command now allows you to monitor queue stats
 via CLI. Thanks seiryuz!
-   Added `/stats.json` endpoint to fetch RQ stats in JSON format,
 useful for monitoring purposes. Thanks seiryuz!
-   Fixed a crash when displaying deferring jobs. Thanks Hovercross!
-   Added `sentry-dsn` cli option to `rqworker` management command.
 Thanks efi-mk!
-   Improved performance when requeueing all jobs. Thanks
 therefromhere!

0.13.1

* Support for RQ 1.14. Thanks HerrTxbias!
* Added `result_ttl` and `ttl` arguments to `scheduler.cron()`. Thanks marcelblijleven!

0.13.0

- Compatibility with Redis 3.0. Thanks dash-rai!
- Added `job_timeout` argument to `queue.enqueue()`. This argument will eventually replace `timeout` argument. Thanks selwin!
- Added `job_id` argument to `BaseDeathPenalty` class. Thanks loopbio!
- Fixed a bug which causes long running jobs to timeout under `SimpleWorker`. Thanks selwin!
- You can now override worker's name from config file. Thanks houqp!
- Horses will now return exit code 1 if they don't terminate properly (e.g when Redis connection is lost). Thanks selwin!
- Added `date_format` and `log_format` arguments to `Worker` and `rq worker` CLI. Thanks shikharsg!
Links

@pyup-bot pyup-bot mentioned this pull request Oct 28, 2024
@pyup-bot
Copy link
Collaborator Author

Closing this in favor of #3150

@pyup-bot pyup-bot closed this Dec 23, 2024
@ednilson ednilson deleted the pyup-update-rq-0.12.0-to-2.0.0 branch December 23, 2024 13:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant