diff --git a/_images/components/messenger/basic_cycle.png b/_images/components/messenger/basic_cycle.png new file mode 100644 index 00000000000..a0558968cbb Binary files /dev/null and b/_images/components/messenger/basic_cycle.png differ diff --git a/_images/components/scheduler/scheduler_cycle.png b/_images/components/scheduler/scheduler_cycle.png new file mode 100644 index 00000000000..18addb37d91 Binary files /dev/null and b/_images/components/scheduler/scheduler_cycle.png differ diff --git a/scheduler.rst b/scheduler.rst new file mode 100644 index 00000000000..aecf327ef44 --- /dev/null +++ b/scheduler.rst @@ -0,0 +1,534 @@ +Scheduler +========= + +.. versionadded:: 6.3 + + The Scheduler component was introduced in Symfony 6.3 + +The Symfony Scheduler is a powerful and flexible component designed to manage tasks scheduling within your PHP application. + +This document focuses on using the Scheduler component in the context of a full stack Symfony application. + +Installation +------------ + +In applications using :ref:`Symfony Flex `, run this command to +install the scheduler component: + +.. code-block:: terminal + + $ composer require symfony/scheduler + +Debugging the Schedule +~~~~~~~~~~~~~~~~~~~~~~ + +The ``debug:scheduler`` command provides a list of schedules along with their recurring messages. + +You can narrow down the list to a specific schedule, or even specify a date to determine the next run date using the ``--date`` option. +Additionally, you have the option to display terminated recurring messages using the ``--all`` option. + +.. code-block:: terminal + + $ php bin/console debug:scheduler + + Scheduler + ========= + + default + ------- + + ------------------- ------------------------- ---------------------- + Trigger Provider Next Run + ------------------- ------------------------- ---------------------- + every 2 days App\Messenger\Foo(0:17..) Sun, 03 Dec 2023 ... + 15 4 */3 * * App\Messenger\Foo(0:17..) Mon, 18 Dec 2023 ... + -------------------- -------------------------- --------------------- + +Introduction to the case +------------------------ + +Embarking on a task is one thing, but often, the need to repeat that task looms large. +While one could resort to issuing commands and scheduling them with cron jobs, this approach involves external tools and additional configuration. + +The Scheduler component emerges as a solution, allowing you to retain control, configuration, and maintenance of task scheduling within our PHP application. + +At its core, the principle is straightforward: a task, considered as a message needs to be managed by a service, and this cycle must be repeated. +Does this sound familiar? Think :doc:`Symfony Messenger docs `. + +But while the system of Messenger proves very useful in various scenarios, there are instances where its capabilities +fall short, particularly when dealing with repetitive tasks at regular intervals. + +Let's dive into a practical example within the context of a sales company. + +Imagine the company's goal is to send diverse sales reports to customers based on the specific reports each customer chooses to receive. +In constructing the schedule for this scenario, the following steps are taken: + +#. Iterate over reports stored in the database and create a recurring task for each report, considering its unique properties. This task, however, should not be generated during holiday periods. + +#. Furthermore, you encounter another critical task that needs scheduling: the periodic cleanup of outdated files that are no longer relevant. + +On the basis of a case study in the context of a full stack Symfony application, let's dive in and explore how you can set up your system. + +Symfony Scheduler basics +------------------------ + +Differences and parallels between Messenger and Scheduler. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The primary goal is to generate and process reports generation while also handling the removal of outdated reports at specified intervals. + +As mentioned, this component, even if it's an independent component, it draws its foundation and inspiration from the Messenger component. + +On one hand, it adopts well-established concepts from Messenger (such as message, handler, bus, transport, etc.). +For example, the task of creating a report is considered as a message that will be directed, and processed by the corresponding handler. + +However, unlike Messenger, the messages will not be dispatched in the first instance. Instead, the aim is to create them based on a predefined frequency. + +This is where the specific transport in Scheduler, known as the :class:`Symfony\\Component\\Scheduler\\Messenger\\SchedulerTransport`, plays a crucial role. +The transport autonomously generates directly various messages according to the assigned frequencies. + +From (Messenger cycle): + +.. image:: /_images/components/messenger/basic_cycle.png + :alt: Symfony Messenger basic cycle + +To (Scheduler cycle): + +.. image:: /_images/components/scheduler/scheduler_cycle.png + :alt: Symfony Scheduler basic cycle + +In essence, it is crucial to precisely define the message to be conveyed and processed. In Scheduler, the concept of a message takes on a very particular characteristic; +it should be recurrent: It's a :class:`Symfony\\Component\\Scheduler\\RecurringMessage`. + +Clarifying the need to define and attach RecurringMessages to a Schedule. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In order to generate various messages based on their defined frequencies, configuration is necessary. +The heart of the scheduling process and its configuration resides in a class that must extend the :class:`Symfony\\Component\\Scheduler\\ScheduleProviderInterface`. + +The purpose of this provider is to return a schedule through the method :method:`Symfony\\Component\\Scheduler\\ScheduleProviderInterface::getMethod` containing your different recurringMessages. + +The :class:`Symfony\\Component\\Scheduler\\Attribute\\AsSchedule` attribute, which by default references the ``default`` named schedule, allows you to register on a particular schedule:: + + // src/Scheduler/MyScheduleProvider.php + namespace App\Scheduler; + + class MyScheduleProvider implements ScheduleProviderInterface + { + public function getSchedule(): Schedule + { + // ... + } + } + +.. tip:: + + This becomes important when initiating the ``messenger:consume`` command, especially when specifying one or more specific transports. + In Scheduler, the transport is named ``scheduler_nameofyourschedule``. + +.. tip:: + + It is a good practice to memoize your schedule to prevent unnecessary reconstruction if the ``getSchedule`` method is checked by another service or internally within Symfony + + +The Concept of RecurringMessage in Scheduler +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A message associated with a Trigger +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +First and foremost, a RecurringMessage is a message that will be associated with a trigger. + +The trigger is what allows configuring the recurrence frequency of your message. Several options are available to us: + +#. It can be a cron expression trigger: + +.. configuration-block:: + + .. code-block:: php + + RecurringMessage::cron(‘* * * * *’, new Message()); + +.. tip:: + + `dragonmantank/cron-expression`_ is required to use the cron expression trigger. + + Also, `crontab_helper`_ is a good tool if you need help to construct/understand cron expressions + +.. tip:: + + It's possible to add and define a timezone as a 3rd argument + +#. It can be a periodicalTrigger through various frequency formats (string / integer / DateInterval) + +.. configuration-block:: + + .. code-block:: php + + RecurringMessage::every('10 seconds', new Message()); + RecurringMessage::every('3 weeks', new Message()); + RecurringMessage::every('first Monday of next month', new Message()); + + $from = new \DateTimeImmutable('13:47', new \DateTimeZone('Europe/Paris')); + $until = '2023-06-12'; + RecurringMessage::every('first Monday of next month', new Message(), $from, $until); + +#. It can be a custom trigger implementing :class:`Symfony\\Component\\Scheduler\\TriggerInterface` + +If you go back to your scenario regarding reports generation based on your customer preferences. +If the basic frequency is set to a daily basis, you will need to implement a custom trigger due to the specific requirement of not generating reports during public holiday periods:: + + // src/Scheduler/Trigger/NewUserWelcomeEmailHandler.php + namespace App\Scheduler\Trigger; + + class ExcludeHolidaysTrigger implements TriggerInterface + { + public function __construct(private TriggerInterface $inner) + { + } + + public function __toString(): string + { + return $this->inner.' (except holidays)'; + } + + public function getNextRunDate(\DateTimeImmutable $run): ?\DateTimeImmutable + { + if (!$nextRun = $this->inner->getNextRunDate($run)) { + return null; + } + + while (!$this->isHoliday($nextRun) { // loop until you get the next run date that is not a holiday + $nextRun = $this->inner->getNextRunDate($nextRun); + } + + return $nextRun; + } + + private function isHoliday(\DateTimeImmutable $timestamp): bool + { + // app specific logic to determine if $timestamp is on a holiday + // returns true if holiday, false otherwise + } + } + +Then, you would have to define your RecurringMessage + +.. configuration-block:: + + .. code-block:: php + + RecurringMessage::trigger( + new ExcludeHolidaysTrigger( // your custom trigger wrapper + CronExpressionTrigger::fromSpec('@daily'), + ), + new SendDailySalesReports(), + ); + +The RecurringMessages must be attached to a Schedule:: + + // src/Scheduler/MyScheduleProvider.php + namespace App\Scheduler; + + #[AsSchedule('uptoyou')] + class MyScheduleProvider implements ScheduleProviderInterface + { + public function getSchedule(): Schedule + { + return $this->schedule ??= (new Schedule()) + ->with( + RecurringMessage::trigger( + new ExcludeHolidaysTrigger( // your custom trigger wrapper + CronExpressionTrigger::fromSpec('@daily'), + ), + new SendDailySalesReports()), + RecurringMessage::cron(‘3 8 * * 1’, new CleanUpOldSalesReport()) + + ); + } + } + +So, this recurringMessage will encompass both the trigger, defining the generation frequency of the message, and the message itself, the one to be processed by a specific handler. +But what is interesting to know is that it also provides you with the ability to generate your message(s) dynamically. + +A dynamic vision for the messages generated +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This proves particularly useful when the message depends on data stored in databases or third-party services. + +Taking your example of reports generation, it depends on customer requests. +Depending on the specific demands, any number of reports may need to be generated at a defined frequency. +For these dynamic scenarios, it gives you the capability to dynamically define our message(s) instead of statically. +This is achieved by wrapping it in a :class:`Symfony\\Component\\Scheduler\\Trigger\\CallbackMessageProvider`. + +Essentially, this means you can dynamically, at runtime, define your message(s) through a callback that gets executed each time the scheduler transport checks for messages to be generated:: + + // src/Scheduler/MyScheduleProvider.php + namespace App\Scheduler; + + class MyScheduleProvider implements ScheduleProviderInterface + { + public function getSchedule(): Schedule + { + return $this->schedule ??= (new Schedule()) + ->with( + RecurringMessage::trigger( + new ExcludeHolidaysTrigger( // your custom trigger wrapper + CronExpressionTrigger::fromSpec('@daily'), + ), + // instead of being static as in the previous example + new CallbackMessageProvider([$this, 'generateReports'], 'foo')), + RecurringMessage::cron(‘3 8 * * 1’, new CleanUpOldSalesReport()) + + ); + } + + public function generateReports(MessageContext $context) + { + // ... + yield new ReportSomething(); + yield new ReportSomethingReportSomethingElse(); + .... + } + } + +Exploring alternatives for crafting your Recurring Messages. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There is also another way to build a recurringMessage, and this can be done simply by adding an attribute above a service or a command: +:class:`Symfony\\Component\\Scheduler\\Attribute\\AsPeriodicTask` attribute and :class:`Symfony\\Component\\Scheduler\\Attribute\\AsCronTask` attribute. + +For both of these attributes, you have the ability to define the schedule to roll with using the ``schedule``option. By default, the ``default`` named schedule will be used. +Also, by default, the ``__invoke`` method of your service will be called but, it's also possible to specify the method to call via the ``method``option and you can define arguments via ``arguments``option if necessary. + +The distinction between these two attributes lies in the options pertaining to the trigger: + +#. :class:`Symfony\\Component\\Scheduler\\Attribute\\AsPeriodicTask` attribute: + + #. You can configure various options such as ``frequencies``, ``from``, ``until`` and ``jitter``, encompassing options related to the trigger. + +#. :class:`Symfony\\Component\\Scheduler\\Attribute\\AsCronTask` attribute: + + #. You can configure various options such as ``expression``, ``jitter``, encompassing options related to the trigger. + +By defining one of these two attributes, it enables the execution of your service or command, considering all the options that have been specified within the attributes. + +Managing Scheduled Messages: +---------------------------- + +Modifying Scheduled Messages in real time +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +While planning a schedule in advance is beneficial, it is rare for a schedule to remain static over time. +After a certain period, some recurringMessages may become obsolete, while others may need to be integrated into our planning. + +As a general practice, to alleviate a heavy workload, the recurringMessages in the schedules are stored in memory to avoid recalculation each time the scheduler transport generates messages. +However, this approach can have a flip side. + +In the context of our sales company, certain promotions may occur during specific periods and need to be communicated repetitively throughout a given timeframe +or the deletion of old reports needs to be halted under certain circumstances. + +This is why the Scheduler incorporates a mechanism to dynamically modify the schedule and consider all changes in real-time. + +Exploring Strategies for adding, removing, and modifying entries within the Schedule +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The schedule provides you with the ability to :method:`Symfony\\Component\\Scheduler\Schedule::add`, :method:`Symfony\\Component\\Scheduler\Schedule::remove`, or :method:`Symfony\\Component\\Scheduler\Schedule::clear` all associated recurringMessages, +resulting in the reset and recalculation of the in-memory stack of recurringMessages. + +For instance, for various reasons, if there's no need to generate a report, a callback can be employed to conditionally skip generating of some or all reports. + +However, if the intention is to completely remove a recurringMessage and its recurrence, +the :class:`Symfony\\Component\\Scheduler\Schedule` offers a :method:`Symfony\\Component\\Scheduler\Schedule::remove` or a :method:`Symfony\\Component\\Scheduler\Schedule::removeById` method. +This can be particularly useful in your case, especially if you need to halt the generation of the recurringMessage, which involves deleting old reports. + +In your handler, you can check a condition and, if affirmative, access the :class:`Symfony\\Component\\Scheduler\Schedule` and invoke this method:: + + // src/Scheduler/MyScheduleProvider.php + namespace App\Scheduler; + + class MyScheduleProvider implements ScheduleProviderInterface + { + public function getSchedule(): Schedule + { + $this->removeOldReports = RecurringMessage::cron(‘3 8 * * 1’, new CleanUpOldSalesReport()); + + return $this->schedule ??= (new Schedule()) + ->with( + // ... + $this->removeOldReports; + ); + } + + // ... + + public function removeCleanUpMessage() + { + $this->getSchedule()->getSchedule()->remove($this->removeOldReports); + } + } + + // src/Scheduler/Handler/.php + namespace App\Scheduler\Handler; + + #[AsMessageHandler] + class CleanUpOldSalesReportHandler + { + public function __invoke(CleanUpOldSalesReport $cleanUpOldSalesReport): void + { + // do what you have to do + + if ($isFinished) { + $this->mySchedule->removeCleanUpMessage(); + } + } + } + +Nevertheless, this system may not be the most suitable for all scenarios. Also, the handler should ideally be designed to process the type of message it is intended for, +without making decisions about adding or removing a new recurringMessage. + +For instance, if, due to an external event, there is a need to add a recurrent message aimed at deleting reports, +it can be challenging to achieve within the handler. This is because the handler will no longer be called or executed once there are no more messages of that type. + +However, the Scheduler also features an event system that is integrated into a Symfony full-stack application by grafting onto Symfony Messenger events. +These events are dispatched through a listener, providing a convenient means to respond. + +Managing Scheduled Messages via Events: +--------------------------------------- + +A strategic event handling +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The goal is to provide flexibility in deciding when to take action while preserving decoupling. +Three primary event types have been introduced types + + #. PRE_RUN_EVENT + + #. POST_RUN_EVENT + + #. FAILURE_EVENT + +Access to the schedule is a crucial feature, allowing effortless addition or removal of message types. +Additionally, it will be possible to access the currently processed message and its message context. + +In consideration of our scenario, you can easily listen to the PRE_RUN_EVENT and check if a certain condition is met. + +For instance, you might decide to add a recurringMessage for cleaning old reports again, with the same or different configurations, or add any other RecurringMessage(s). + +If you had chosen to handle the deletion of the recurringMessage, you could have easily done so in a listener for this event. + +Importantly, it reveals a specific feature :method:`Symfony\\Component\\Scheduler\\Event\\PreRunEvent::shouldCancel` that allows you to prevent the message of the deleted recurringMessage from being transferred and processed by its handler:: + + // src/Scheduler/MyScheduleProvider.php + namespace App\Scheduler; + + class MyScheduleProvider implements ScheduleProviderInterface + { + public function getSchedule(): Schedule + { + $this->removeOldReports = RecurringMessage::cron(‘3 8 * * 1’, new CleanUpOldSalesReport()); + + return $this->schedule ??= (new Schedule()) + ->with( + // ... + ); + ->before(function(PreRunEvent $event) { + $message = $event->getMessage(); + $messageContext = $event->getMessageContext(); + + // can access the schedule + $schedule = $event->getSchedule()->getSchedule(); + + // can target directly the RecurringMessage being processed + $schedule->removeById($messageContext->id); + + //Allow to call the ShouldCancel() and avoid the message to be handled + $event->shouldCancel(true); + } + ->after(function(PostRunEvent $event) { + // Do what you want + } + ->onFailure(function(FailureEvent $event) { + // Do what you want + } + } + } + + +Efficient management with Symfony Scheduler: +-------------------------------------------- + +However, if your worker becomes idle, since the messages from your schedule are generated on-the-fly by the schedulerTransport, +they won't be generated during this idle period. + +While this might not pose a problem in certain situations, consider the impact for your sales company if a report is missed. + +In this case, the scheduler has a feature that allows you to remember the last execution date of a message. +So, when it wakes up again, it looks at all the dates and can catch up on what it missed. + +This is where the ``stateful`` option comes into play. This option helps you remember where you left off, which is super handy for those moments when the worker is idle and you need to catch up:: + + // src/Scheduler/MyScheduleProvider.php + namespace App\Scheduler; + + class MyScheduleProvider implements ScheduleProviderInterface + { + public function getSchedule(): Schedule + { + $this->removeOldReports = RecurringMessage::cron(‘3 8 * * 1’, new CleanUpOldSalesReport()); + + return $this->schedule ??= (new Schedule()) + ->with( + // ... + ); + ->stateful($this->cache) + } + } + +To scale your schedules more effectively, you can use multiple workers. +In such cases, a good practice is to add a lock for some job concurrency optimization. It helps preventing the processing of a task from being duplicated:: + + // src/Scheduler/MyScheduleProvider.php + namespace App\Scheduler; + + class MyScheduleProvider implements ScheduleProviderInterface + { + public function getSchedule(): Schedule + { + $this->removeOldReports = RecurringMessage::cron(‘3 8 * * 1’, new CleanUpOldSalesReport()); + + return $this->schedule ??= (new Schedule()) + ->with( + // ... + ); + ->lock($this->lockFactory->createLock(‘my-lock’) + } + } + +.. tip:: + + The processing time of a message matters. + If it takes a long time, all subsequent message processing may be delayed. So, it's a good practice to anticipate this and plan for frequencies greater than the processing time of a message. + +.. tip:: + + A lock is essential if there are several workers, and a cache is good if it's important not to miss certain tasks when the worker is out of service. + +Additionally, for better scaling of our schedules, you have the option to wrap your message in a :class:`Symfony\\Component\\Messenger\\Message\\RedispatchMessage`. +This allows you to specify a transport on which your message will be redispatched before being further redispatched to its corresponding handler:: + + // src/Scheduler/MyScheduleProvider.php + namespace App\Scheduler; + + class MyScheduleProvider implements ScheduleProviderInterface + { + public function getSchedule(): Schedule + { + return $this->schedule ??= (new Schedule()) + ->with(RecurringMessage::every('5 seconds’), new RedispatchMessage(new Message(), ‘async’)) + ); + } + } + +.. _dragonmantank/cron-expression: https://packagist.org/packages/dragonmantank/cron-expression +.. _crontab_helper: https://crontab.guru/