Beyond Batch and Queue: Temporal integration with Drupal

Background

Beyond serving pages, a lot of websites have background processes they need to run. Importing third party data, indexing changes, generating reports and so on. Once a site becomes so large it is no longer feasible for a single PHP process to do this for every entity it cares about, things become more difficult.

What Drupal Offers

Drupal provides two APIs for long-running processes: batch and queue. A batch runs an operation, saves state and then runs it again until the operation says "stop". However, if an operation fails then the entire batch aborts. Queue lines up operations and runs them independently of each other so the error handling is somewhat better. If an operation fails then the rest of the operations can run but there is very little control over retries. These two are good for the basics, they are good enough to be included in Drupal core, but when you are dealing with larger sites they are woefully inadequate. I should know: I wrote the queue API originally.

What if we had an ability to run these processes without having to worry about crashes, retries, or scaling across machines with extensive reporting about what has happened? That system is Temporal, and this post will walk through how it integrates with Drupal.

Temporal Basics

In Temporal, your Drupal code is an activity. Activities are a single unit of work. It doesn't matter whether it takes a short or long time, for example transcoding media might take a long time, but it's still a single unit. A workflow tells Temporal which activities to run and how to retry them. It has a rich selection of retry strategies. It also controls various timeouts. One of the more interesting timeouts is the heartbeat for long-running activities: if the activity doesn't send a heartbeat within the specified time, it will be cancelled. Heartbeats can also carry progress information.

The workflow is a long-running process and if it stops for any reason, Temporal makes it resume where it stopped. When I first read this I thought "huh, maybe it somehow saves the memory state but that'd be very fugly" and no, that's not what it does. Instead, this magic is achieved by saving the inputs and outputs of every activity call in its own durable event log. There is a UI to see the log which also includes workflow events besides these activity events.

The Magic

Thanks to this event log, when a workflow is restarted the activities do not need to be rerun, the workflow fast-forwards to the point where it stopped based on the event log replay. I can't emphasize enough how important this is: no matter what crashes and when, the system completely transparently handles it. The activity calls a third party service that is temporarily down and so it needs to return with an error? PHP crashed with an out of memory error because Drupal leaks memory like a sieve? No need to worry about any of this, no need to write elaborate retry strategies for the remote call, no need to try to patch up the leaking sieve. Temporal will retry the activity, the workflow will continue and neither needs to care about errors and crashes.

A Little Theory

For this to work well, workflows need to be deterministic: given the same series of events, a workflow must always make the same decisions. For this reason, workflows should not consult databases, file systems, clocks, random numbers, and the like. That's a job for activities. Most workflows will even avoid logging to prevent any side effects, especially since Temporal already records all activity inputs and outputs.

And activities are recommended to be idempotent: running them multiple times should have the same result as running them once. This is important because they can be re-tried if they fail. A classic example of an idempotent operation is the stop button on a media player: no matter how many times you tap it the music will not play. The play/pause button, on the other hand, is the classic example for a non-idempotent operation. Within PHP, writing to a stream opened with fopen('filename', 'w') is idempotent: the contents of the file become the data written. On the other hand, if a stream is opened with fopen('filename', 'a') then the writes are not idempotent: the data is appended over and over again.

To further highlight the difference between the two, consider a database MERGE: the operation will want to report back whether the row was inserted or updated, so it is not deterministic, but it is idempotent because the database row ends up with the same data either way.

Enough of the theoretical talk, let's talk code!

Coding a Workflow

A workflow is a PHP class. To make it easy for Drupal to discover them, they are in the Drupal\mymodule\Temporal\Workflow namespace and the class has the Temporal\Workflow\WorkflowInterface attribute. This pattern should be familiar from writing plugins. This attribute is enough for Temporal to recognize this as a workflow class. Temporal also requires the workflow method to have the Temporal\Workflow\WorkflowMethod attribute.

While a workflow looks like a Drupal plugin, it is not. As we discussed workflows need to be deterministic and due to the complexity of Drupal it is almost impossible to guarantee any call into Drupal to be deterministic so it's best if workflows do not talk to Drupal at all. The integration encourages this: workflow classes are instantiated by Temporal directly without passing any arguments to the constructor.

The most important thing a workflow does is calling an activity. The Temporal PHP SDK's mechanism for this is a bit unusual at first, but it's the same pattern as mocks/stubs in phpunit: activity classes get a stub in the phpunit sense and the methods defined in the activity class are called on this stub.

For example:

/** @var \Drupal\temporal\Temporal\GenericActivity $activity */ 
$activity = Workflow::newActivityStub($activityClass);
$ids = yield $activity->getIds();

(Irrelevant arguments are cut from this example, see the GenericWorkflowBase class shipping with the module for the rest.)

Calling a method on the activity stub returns a promise (from the ReactPHP package) that encapsulates this method invocation. Then yield hands back control to the Temporal PHP SDK, which resolves this promise by sending the activity to the Temporal server as a gRPC request. (Yes, yield can have a value, see the documentation for the rarely used Generator::send() for more.) It is not necessary to yield after every call: see ParallelGenericWorkflow for an example on how to instruct Temporal to run multiple activities in parallel.

Once you are used to this calling convention this is much easier to read than a traditional request builder. Now you can see why the code uses the old /** @var */ convention instead of asserting the type directly: as far as the IDE and the developer is concerned, $activity can be treated as an instance of $activityClass. But in reality, it's an ActivityProxy class.

When the Temporal server gets the request to call an activity, it might just send the relevant answer immediately if it is replaying the event log. Otherwise, it logs the activity inputs and puts the request in a task queue. The task queues are processed by workers, we will get back to them after discussing activities. First let's mention the two example workflows shipped with the module. Both call an activity for a large list of IDs (by default 1000), then small chunks of these (by default 20) are sent back to the activity for processing. One workflow launches a chunk worth of activities in parallel, the other sends the chunk in a single call. The former is good for something like search indexing; the latter is good for anything that writes the database and wants to keep database load lower by keeping many writes in a single transaction. A lot of tasks can be accomplished by writing an activity for one of these two, so writing a workflow is not always necessary.

Coding an Activity

Writing activities is much easier: the code does not need to talk to Temporal, these are normal Drupal plugins containing ordinary Drupal code and writing Drupal code is very easy ;). As usual for plugins, they need to be within a specific namespace, Drupal\mymodule\Temporal\Activity with the Temporal\Activity\ActivityInterface attribute on the class which, again, is enough for Temporal as well to recognize it as an activity. Methods are marked with the ActivityMethod attribute for Temporal.

While the code doesn't need to contain calls to Temporal, there are some considerations knowing they will be used by Temporal:

• As activity calls are remote calls in disguise, both the arguments and the return values need to be serializable.
• If there is an error – typically in calling third party services – that warrants a retry then the activity can simply let the exception propagate. The SDK will catch it and pass it to Temporal. For more fine-grained control the activity can throw an ApplicationFailure, which, among others, can tell Temporal whether the failure can be retried at all.
• Heartbeats are super easy to send: Activity::heartbeat($progress); It really must be noted how easy it is to work with the Temporal PHP SDK. We have a complex server-client architecture but most of the complexity is not visible at all: activity calls are hidden behind a proxy class and a simple yield, progress reporting is a single static method call and simple error handling is completely automatic.

An example activity is shipped with the module which re-saves every entity of an entity type.

Actually Trying It

Before we can get to trying it, there's one more thing we need to introduce: Workers. These are long-running processes that poll the Temporal task queues and run workflows and activities. To better support their long-running nature, Temporal uses the RoadRunner application server for them. The Drupal integration ships this worker as a Drush command and supplies a sample .rr.yaml RoadRunner configuration to run this command. Most of the time simply copying the configuration to the project root and running rr serve is all you need to do. Read the module README.md for more. Besides a Temporal server instance at least one worker is needed for Temporal to work. But you can run as many as the workload warrants.

To round it off, there's a Drush command to start workflows and another to send signals and queries to them.

To make local development easier, a DDEV add-on (chx/ddev-temporalio) has been developed as well, this spins up a Temporal server and a Temporal web UI. So in a ddev project, you can run

ddev get chx/ddev-temporalio
ddev restart
ddev composer require 'drupal/temporal:^2.1'
ddev drush en -y temporal
ddev drush temporal:workflow:start 'Drupal\temporal\Temporal\Workflow\GenericWorkflow' 'Drupal\temporal\Temporal\Activity\EntityResave' user

This will re-save every user entity. The first argument of the Drush temporal:workflow:start command is the name of the workflow. In turn, the first argument of this particular workflow is the activity class. This is not a Temporal convention or even a convention of the Drupal-temporal integration, it's simply convenient for such a generic workflow. The rest of the arguments are passed to the activity and the entity resave activity needs the entity type.

We started with talking about batch, let's finish with it, too: we actually integrated the Drupal batch system with temporal. In the next blog post we will talk about that. Teaser: you can start the batch and close the browser tab.

Running into the limits of Drupal's batch and queue? See how Tag1 scales Drupal.

• Architecture
• Drupal
• Platforms
• Scalability
• Web Development