Release v2.6.0

CHANGELOG.md

@@ -1,6 +1,76 @@
 # Changelog
 
-All notable changes to `Oban` are documented in this file.
+All notable changes to `Oban` are documented here.
+
+## [2.6.0] — 2021-04-02
+
+_🌟 Web and Pro users should check the [v2.6 upgrade guide][v26ug] for a
+complete walkthrough._
+
+### Pluggable Queue Engines
+
+Queue producers now use an "engine" callback module to manage demand and work
+with the database. The engine provides a clean way to expand and enhance the
+functionality of queue producers without modifying the solid foundation of queue
+supervision trees. Engines make enhanced functionality such as global
+concurrency, local rate limiting and global rate limiting possible!
+
+The `BasicEngine` is the default, and it retains the _exact_ functionality of
+previous Oban versions. To specify the engine explicitly, or swap in an
+alternate engine, set it in your configuration:
+
+```elixir
+config :my_app, Oban,
+  engine: Oban.Queue.BasicEngine,
+  ...
+```
+
+See the [v2.6 upgrade guide][v26ug] for instructions on swapping in an alternate
+engine.
+
+### Recursive Queue Draining
+
+The ever-handy `Oban.drain_queue/1,2` function gained a new `with_recursion`
+option, which makes it possible to test jobs that _insert more jobs_ when they
+execute. When `with_recursion` is enabled the queue will keep executing until no
+jobs are available. It even composes with `with_scheduled`!
+
+In practice, this is especially useful for testing dependent workflows.
+
+### Gossip Plugin for Queue Monitoring
+
+The new gossip plugin uses PubSub to efficiently exchange queue state
+information between all interested nodes. This allows Oban instances to
+broadcast state information regardless of which engine they are using, and
+without storing anything in the database.
+
+See the [v2.6 upgrade guide][v26ug] for details on switching to the gossip
+plugin.
+
+[v26ug]: v2-6.html
+
+### Added
+
+- [Oban.Job] Inject the current conf into the jobs struct as virtual field,
+  making the complete conf available within `perform/1`.
+
+- [Oban.Notifier] Add `unlisten/2`, used to stop a process from receiving
+  notifications for one or more channels.
+
+### Changed
+
+- [Oban.Telemetry] Stop emitting circuit events for queue producers. As
+  producers no longer poll, the circuit breaker masked real errors more than it
+  guarded against sporatic issues.
+
+- [Oban.Telemetry] Delay `[:oban, :supervisor, :init]` event until the complete
+  supervision tree finishes initialization.
+
+- [Oban.Migration] Stop creating an `oban_beats` table entirely.
+
+### Fixed
+
+- [Oban.Plugins.Cron] Prevent schedule overflow right before midnight
 
 ## [2.5.0] — 2021-02-26
 
@@ -755,7 +825,8 @@ No changes from [2.0.0-rc.3][].
 
 For changes prior to 2.0 see the [1.2 branch][1.2]
 
-[Unreleased]: https://github.com/sorentwo/oban/compare/v2.5.0...HEAD
+[Unreleased]: https://github.com/sorentwo/oban/compare/v2.6.0...HEAD
+[2.6.0]: https://github.com/sorentwo/oban/compare/v2.5.0...v2.6.0
 [2.5.0]: https://github.com/sorentwo/oban/compare/v2.4.3...v2.5.0
 [2.4.3]: https://github.com/sorentwo/oban/compare/v2.4.2...v2.4.3
 [2.4.2]: https://github.com/sorentwo/oban/compare/v2.4.1...v2.4.2

README.md

@@ -152,7 +152,7 @@ dependencies in `mix.exs`:
 ```elixir
 def deps do
   [
-    {:oban, "~> 2.5.0"}
+    {:oban, "~> 2.6.0"}
   ]
 end
 ```

guides/installation.md

@@ -6,7 +6,7 @@ dependencies in `mix.exs`:
 ```elixir
 def deps do
   [
-    {:oban, "~> 2.5.0"}
+    {:oban, "~> 2.6.0"}
   ]
 end
 ```

guides/upgrading/v2.0.md

@@ -18,13 +18,13 @@ Update Oban to the latest version:
   end
 ```
 
-If you're an Oban Web+Pro user you'll also need to bump ObanWeb and add ObanPro:
+If you're an Oban Web+Pro user you'll also need to bump Oban.Web and add Oban.Pro:
 
 ```elixir
   defp deps do
     [
-      {:oban, "~> 2.0.0"}
-      {:oban_web, "~> 2.0.0", organization: "oban"}
+      {:oban, "~> 2.0.0"},
+      {:oban_web, "~> 2.0.0", organization: "oban"},
       {:oban_pro, "~> 0.3.0", organization: "oban"}
       ...
     ]
@@ -87,7 +87,7 @@ config :my_app, Oban,
   ...
 ```
 
-🌟 ObanPro users may opt to use the [DynamicPruner](dynamic_pruning.html)
+🌟 Oban.Pro users may opt to use the [DynamicPruner](dynamic_pruning.html)
 instead for finer control. For example, to set per-state retention periods:
 
 ```elixir
@@ -117,7 +117,7 @@ config :my_app, Oban,
   ...
 ```
 
-🌟 ObanPro users may use the [Lifeline](lifeline.html) plugin to retain
+🌟 Oban.Pro users may use the [Lifeline](lifeline.html) plugin to retain
 automatic orphaned job rescue with lightweight heartbeat recording:
 
 ```elixir
@@ -186,13 +186,13 @@ Here is a conversion chart for any other handlers you may have:
 [:oban, :open_circuit] -> [:oban, :circuit, :open]
 ```
 
-## Update ObanWeb (Optional)
+## Update Oban.Web (Optional)
 
-ObanWeb is streamlined to share configuration with Oban and use the new plugin
-system. Remove any ObanWeb specific configuration:
+Oban.Web is streamlined to share configuration with Oban and use the new plugin
+system. Remove any Oban.Web specific configuration:
 
 ```diff
-- config :my_app, ObanWeb, repo: MyApp.Repo
+- config :my_app, Oban.Web, repo: MyApp.Repo
 ```
 
 Next, add the necessary plugins to the Oban config:

guides/upgrading/v2.6.md

@@ -0,0 +1,141 @@
+# Upgrading to v2.6
+
+For Oban OSS users the v2.6 upgrade is a drop in replacement—there isn't
+anything to do! However, Web+Pro users will need to make some changes to unlock
+the goodness of engines.
+
+## Bump Your Deps
+
+Update Oban, Web and Pro to the latest versions:
+
+```elixir
+[
+  {:oban, "~> 2.6"},
+  {:oban_web, "~> 2.6", organization: "oban"},
+  {:oban_pro, "~> 0.7", organization: "oban"}
+  ...
+]
+```
+
+Be sure to specify **both `:oban_web` and `:oban_pro`** if you use them both.
+There aren't any dependencies between Web and Pro now. That means you're free to
+use Pro for workers and only include Web for Phoenix servers, etc.
+
+## Switch to the SmartEngine
+
+The `SmartEngine` uses centralized records to track and exchange state globally,
+enabling features such as global concurrency.
+
+First, create a migration to add the new `oban_producers` table:
+
+```bash
+$ mix ecto.gen.migration add_oban_producers
+```
+
+Within the migration module:
+
+```elixir
+use Ecto.Migration
+
+defdelegate change, to: Oban.Pro.Migrations.Producers
+```
+
+If you have multiple Oban instances or use prefixes, you can specify the prefix
+and create multiple tables in one migration:
+
+```elixir
+use Ecto.Migration
+
+def change do
+  Oban.Pro.Migrations.Producers.change()
+  Oban.Pro.Migrations.Producers.change(prefix: "special")
+  Oban.Pro.Migrations.Producers.change(prefix: "private")
+end
+```
+
+Next, update your config to use the `SmartEngine`:
+
+```elixir
+config :my_app, Oban,
+  engine: Oban.Pro.Queue.SmartEngine,
+  ...
+```
+
+If you have multiple Oban instances you need to configure each one to use the
+`SmartEngine`, otherwise they'll default to the `BasicEngine`.
+
+## Start Gossiping
+
+The `Lifeline` plugin from Pro no longer writes heartbeat records to `oban_beats`.
+Instead, queues can use the `Gossip` plugin to exchange their status via PubSub.
+
+To start, include the `Gossip` plugin in your Oban config:
+
+```elixir
+config :my_app, Oban,
+  plugins: [
+    Oban.Plugins.Gossip
+    ...
+```
+
+With the default configuration the plugin will broadcast every 1 second. If that
+is too frequent you can configure the interval:
+
+```elixir
+  plugins: [
+    {Oban.Plugins.Gossip, interval: :timer.seconds(5)}
+    ...
+```
+
+## Remove the Workflow Manager
+
+Due to an improvement in how configuration is passed to workers the
+`WorkflowManager` plugin is no longer needed. You can remove it from your list
+of plugins:
+
+```diff
+plugins: [
+  Oban.Pro.Plugins.Lifeline,
+- Oban.Pro.Plugins.WorkflowManager,
+  ...
+```
+
+## Remove Extra Lifeline Options
+
+The `Lifeline` plugin is simplified and doesn't accept as many configuration
+options. If you previously configured the `record_interval` or `delete_interval`
+you can remove them:
+
+```diff
+plugins: [{
+  Oban.Pro.Plugins.Lifeline,
+- delete_interval: :timer.minutes(10),
+- record_interval: :timer.seconds(10),
+  rescue_interval: :timer.minutes(5)
+}]
+```
+
+## Drop the Beats Table
+
+Once you've rolled out the switch to producer records, the smart engine and the
+gossip plugin you are free to remove the `oban_beats` table at your discretion
+(preferrably in a follow up release, to prevent errors):
+
+```bash
+$ mix ecto.gen.migration drop_oban_beats
+```
+
+Within the generated migration module:
+
+```elixir
+use Ecto.Migration
+
+def up do
+  drop table("oban_beats")
+  drop table("oban_beats", prefix: "private") # If you have any prefixes:
+end
+
+def down do
+  # No going back!
+end
+```

mix.exs

@@ -2,7 +2,7 @@ defmodule Oban.MixProject do
   use Mix.Project
 
   @source_url "https://github.com/sorentwo/oban"
-  @version "2.5.0"
+  @version "2.6.0"
 
   def project do
     [
@@ -69,6 +69,7 @@ defmodule Oban.MixProject do
       "guides/release_configuration.md",
       "guides/writing_plugins.md",
       "guides/upgrading/v2.0.md",
+      "guides/upgrading/v2.6.md",
 
       # Recipes
       "guides/recipes/recursive-jobs.md",
@@ -84,12 +85,14 @@ defmodule Oban.MixProject do
       [
         "../oban_pro/guides/pro/overview.md": [filename: "pro_overview"],
         "../oban_pro/guides/pro/installation.md": [filename: "pro_installation"],
-        "../oban_pro/guides/plugins/lifeline.md": [title: "Lifeline Plugin"],
+        "../oban_pro/guides/queue/smart_engine.md": [title: "Smart Engine"],
         "../oban_pro/guides/plugins/dynamic_cron.md": [title: "Dynamic Cron Plugin"],
         "../oban_pro/guides/plugins/dynamic_pruner.md": [title: "Dynamic Pruner Plugin"],
+        "../oban_pro/guides/plugins/lifeline.md": [title: "Lifeline Plugin"],
         "../oban_pro/guides/plugins/relay.md": [title: "Relay Plugin"],
         "../oban_pro/guides/plugins/reprioritizer.md": [title: "Reprioritizer Plugin"],
         "../oban_pro/guides/workers/batch.md": [title: "Batch Worker"],
+        "../oban_pro/guides/workers/chunk.md": [title: "Chunk Worker"],
         "../oban_pro/guides/workers/workflow.md": [title: "Workflow Worker"],
         "../oban_pro/CHANGELOG.md": [filename: "pro-changelog", title: "Changelog"]
       ]