<!-- Source: docs/introduction.md -->

# Introduction

Durable Workflow 2.0 keeps workflow state and history outside short-lived
application processes so PHP, Python, and Rust workers can resume safely after
a restart. If this is your first visit, take the
[Durable Workflow 2.0 Quickstart](/docs/quickstart/) first: it gets one workflow
to `completed` before you enter the reference-heavy
[Capability Index](/docs/capabilities/).

## Choose a deployment model

### Service mode

Applications use a remote durable runtime through first-party SDKs. Choose who
operates that runtime:

- **[Durable Workflow Cloud](/docs/polyglot/cloud-control-plane/)** is the
  managed choice. Durable Workflow operates orchestration, persistence, and
  Managed Waterline; your team runs SDK clients and workers against its
  provisioned namespace. **Cloud users do not install or run Durable Workflow
  Server or a separate Waterline service.**
- **[Self-hosted Server](/docs/polyglot/server/)** gives your team the same
  service boundary while you deploy, secure, scale, back up, and upgrade the
  runtime. Waterline observation is a separate service you may deploy against
  the Server-owned namespace.

Both choices expose the same versioned HTTP+JSON control plane, worker
protocol, namespace model, and language-neutral payload envelope. See
[Deployment Modes](/docs/polyglot/deployment-modes/) for their ownership
boundaries.

### Embedded Laravel

Embedded mode is a separate deployment model for a Laravel application that
wants workflow state, queues, configuration, and operator tooling inside its
own infrastructure. It installs `durable-workflow/workflow` and does not
connect to Cloud or require a separate Server. The embedded Waterline package
reads that application-owned state in process.

Start with [Embedded Installation](/docs/installation/) only when that
in-application ownership model is intentional.

Laravel teams coming from stable v1—or reconsidering an existing 2.0 embedded
deployment—should use the
[Laravel adoption and runtime transition guide](/docs/laravel-adoption/)
to compare the executable embedded and PHP SDK paths before changing traffic.

## Choose a service-mode SDK

- **[PHP SDK](/docs/polyglot/php/):** install `durable-workflow/sdk` in a
  framework-neutral PHP application or remote worker.
- **[Python SDK](/docs/polyglot/python/):** author deterministic workflows
  and activities and use the async control-plane client. The
  stable release manifest as the Server quickstart.
- **[Rust SDK](/docs/polyglot/rust/):** author deterministic workflows and
  activities and run native worker services.

All three are first-party implementations of the same public service boundary.
The [Client and Worker Capabilities](/docs/polyglot/cli-python-parity/)
guide makes supported and intentionally different client and worker surfaces
explicit.

## Your first completed workflow

The [Quickstart](/docs/quickstart/) states the goal, runtime choice,
prerequisites, time, and expected outcome up front. It keeps PHP, Python, and
Rust equally available while showing one runnable language path at a time.
Use the local self-hosted path for a source-free published-artifact exercise,
or use the Cloud onboarding values for a managed namespace without running
Server.

## How service mode fits together

A service-mode deployment has three parts:

- **The runtime** owns durable state, command and history recording, task
  matching, timers, schedules, namespaces, and authenticated protocols. Cloud
  operates it for managed namespaces; your team operates it when self-hosting.
- **Application workers** run workflow and activity code through the PHP,
  Python, or Rust SDK. They can deploy with an application or as independent
  services and scale separately from the runtime.
- **Clients and operational tools** start, inspect, and command the same
  runtime-owned state through SDK clients, the `dw` CLI, HTTP APIs,
  machine-readable schemas, Waterline, and agent interfaces.

## One public durable-execution contract

The first-party SDKs share registered string workflow and activity type names
and a public payload envelope. The envelope identifies its codec and carries
portable values instead of PHP serialization, Python pickles, or Rust
implementation types.

Workflow workers reconstruct decisions from durable command and history
records. Activity and child-workflow input and results can cross language
boundaries when workers advertise the same public codec and register matching
type names. Consult the [Capability Index](/docs/capabilities/) and runtime
discovery before depending on a specific SDK surface.

## Learn from the matching examples

- **Service mode and polyglot:** use the
  [Quickstart](/docs/quickstart/) and the PHP, Python, or Rust SDK guide.
- **Embedded Laravel:** use the [Sample App](/docs/sample-app/) gallery to
  explore Laravel-native workflow patterns and Waterline evidence.

The embedded gallery is not a universal starting point for Cloud or
self-hosted service-mode readers.

## Agent-operable by contract

Human operators and autonomous agents use the same machine-readable contract.
The testable loop is **Discover -> Change -> Run -> Diagnose -> Repair**:
version and capability manifests, explicit workflow commands, structured
results, typed history and worker/queue diagnostics, safe mutations, and
post-change verification. See the
[Agent Operating Loop](/docs/agent-operating-loop/) and the direct
[AI-agent evaluator](/docs/ai-agent-workflow-engine/).

## Do you need a workflow?

You probably need a workflow if:

- The process spans minutes, hours, or days
- You need to wait for a human approval step
- You need to wait for a webhook or other external event
- You need to pause and continue later without keeping a process running
- You need to be able to restart after a crash without causing bugs or duplicating work

If your task is "run five queued jobs in order and bail on the first failure,"
a job chain is usually a better fit. Durable Workflow is for cases where the
next step depends on an external event, a wait, or a decision that cannot be
known up front.

<!-- Source: docs/capabilities.md -->

# Durable Workflow 2.0 Capability Index

This is the capability index for the stable Durable Workflow 2.0 release line.

## Supported installable channel

Exact artifact identities belong to package registries and release metadata,
not this capability summary.

| Surface | Supported channel | Role in this index |
| --- | --- | --- |
| CLI | 2.0 stable | Machine-readable operator and diagnostic client. |
| PHP SDK | 2.0 stable | First-party service-mode `durable-workflow/sdk` client and remote workflow/activity worker for a Cloud namespace runtime or self-hosted Server. |
| Embedded Laravel engine | 2.0 stable | Separate in-application `durable-workflow/workflow` deployment mode with its own authoring runtime, persistence engine, queues, and replay. |
| Python SDK | 2.0 stable | First-party service-mode deterministic workflow/activity SDK plus operational and control-plane client for a Cloud namespace runtime or self-hosted Server. Requires Python 3.10 or newer. |
| Rust SDK | 2.0 stable | First-party service-mode deterministic workflow/activity SDK, worker service, and control-plane client for a Cloud namespace runtime or self-hosted Server. Requires Rust 1.86 or newer. |
| Server | 2.0 stable | Self-hosted, PHP-implemented language-neutral runtime for the v2 control-plane and worker protocols. |
| Waterline | 2.0 stable | Cloud includes Managed Waterline; self-hosted operators separately deploy the service image against a Server namespace; embedded Laravel installs the Composer package and reads application-owned state in process. |

PHP, Python, and Rust are the three first-party service-mode SDK languages.
Each can target a provisioned Durable Workflow Cloud namespace runtime or a
self-hosted Server where runtime discovery reports the published capability.
They share stable string type names, one durable command/history model, control
plane version `2`, worker protocol major `1`, and the public payload envelope.
Rust is a workflow SDK: it runs deterministic workflow code, activities, and
worker services, not only raw protocol calls. Python combines workflow
authoring with namespace, schedule, worker, queue, history, repair, and other
operational client surfaces.

Embedded Laravel remains a separate in-application PHP deployment mode; it is
not required for PHP service-mode clients or workers. The framework-neutral PHP
SDK is versioned independently from the embedded Laravel engine.

Use [Deployment Modes](/docs/polyglot/deployment-modes/) to choose the
ownership boundary, then continue with
[Cloud Managed Runtime](/docs/polyglot/cloud-control-plane/) or the
[self-hosted Server](/docs/polyglot/server/). The capability evidence below
applies only where the selected runtime advertises the required contract.

PHP SDK 2.3.2, Python SDK 2.5.7 and Rust SDK 3.5.0 default to worker protocol
`1.20`. Default workers require a Server advertising `1.20` or newer within
major `1`. Each SDK also supports explicitly selecting `1.19` for an older
compatible Server. Cooperative cancellation requires `1.20` and its explicit
capability opt-in. Runtime discovery and the selected SDK's published metadata
decide compatibility. See [Version Compatibility](/docs/compatibility/) and
[Worker Protocol](/docs/polyglot/worker-protocol/).

## Capability and SDK floors

“Service mode” means either Cloud or a self-hosted Server owns the remote
runtime. “Embedded” means a Laravel application owns the runtime in process.
The SDK column deliberately records differences; a blank or limited SDK is not
implied to have parity.

| Capability | Current 2.0 contract and first-party floor | Explicit 2.0 evidence |
| --- | --- | --- |
| Workflows | Remote PHP, Python, and Rust workers author deterministic workflows; embedded Laravel authoring uses the Workflow package. | [Workflow authoring](/docs/defining-workflows/workflows/), [PHP SDK](/docs/polyglot/php/), [Python SDK](/docs/polyglot/python/), [Rust SDK](/docs/polyglot/rust/) |
| Activities and services | The embedded engine and all three service-mode SDKs author activities. Rust workflows and activities run in first-party worker services; Python and PHP also expose first-party control/service clients. | [Activities](/docs/defining-workflows/activities/), [PHP SDK](/docs/polyglot/php/), [Activity execution](/docs/features/activity-execution-model/), [External execution](/docs/polyglot/external-execution/) |
| Cross-namespace service calls | Embedded PHP, PHP SDK and Python SDK author durable registered service operations with completion or admission-only waiting and recorded results. PHP service-mode target namespace currently uses the array `namespace` option. Authored Rust calls are unavailable pending [sdk-rust #86](https://github.com/durable-workflow/sdk-rust/issues/86). | [Embedded authoring](/docs/features/service-operations/), [Shared catalog and SDK support](/docs/polyglot/service-operations/) |
| Signals | Durable, asynchronous mutation is available to PHP, Python, and Rust workflow surfaces at their current floors. | [Signals](/docs/features/signals/), [Python SDK](/docs/polyglot/python/), [Rust SDK](/docs/polyglot/rust/) |
| Queries | Read-only replay queries are supported. PHP, Python, and Rust expose query handlers/clients; Rust replayed query tasks require worker protocol `1.8` or newer. | [Queries](/docs/features/queries/), [Python SDK](/docs/polyglot/python/), [Rust SDK](/docs/polyglot/rust/) |
| Updates | PHP, Python, and Rust expose client and worker-handler update surfaces. Python declared validators use synchronous pre-accept validation when Server discovery advertises that exact contract: acceptance follows approval, while rejection, worker loss, timeout, and fenced completion fail explicitly. PHP and Rust do not expose validator authoring and declare no validators. | [Updates](/docs/features/updates/), [Python SDK](/docs/polyglot/python/), [Rust SDK](/docs/polyglot/rust/#workflow-updates) |
| Timers | Service-runtime-backed durable timers replay in PHP, Python, and Rust. | [Timers](/docs/features/timers/), [Rust SDK](/docs/polyglot/rust/) |
| Retries | Durable activity retry policy is recorded with the command. PHP, Python, and Rust expose activity retry options at the current floors. | [Failures and recovery](/docs/failures-and-recovery/), [Python SDK](/docs/polyglot/python/), [Rust SDK](/docs/polyglot/rust/) |
| Timeouts | Service-mode runtimes in Cloud and self-hosted Server enforce activity and workflow timeout families. Rust includes activity options and workflow execution/run start deadlines. | [Timeouts](/docs/features/timeouts/), [Rust SDK](/docs/polyglot/rust/) |
| Child workflows | PHP, Python, and Rust start and await durable children. Cross-language type identity is a stable string and child payloads use the shared envelope. | [Child workflows](/docs/features/child-workflows/), [Python SDK](/docs/polyglot/python/) |
| Deterministic parallel composition | PHP `all()`/`parallel()`, Python nested list-yield, and Rust `WorkflowContext::parallel()`/`join()` schedule activity, child-workflow, timer, mixed, and nested groups through ordinary commands. All three publish the shared group identity/path metadata and restore results in nested input order. | [Concurrency](/docs/features/concurrency/), [Python SDK](/docs/polyglot/python/#fan-out), [Rust SDK](/docs/polyglot/rust/#deterministic-parallel-groups) |
| <span id="durable-first-completion-selection-capability">Durable first-completion selection</span> | Embedded PHP and the PHP, Python, and Rust service SDKs start independent activity, child-workflow, timer, external-input wait, and nested-group members. One winner is persisted with stable member identity; non-winners continue and retain await/cancel handles. | [Concurrency](/docs/features/concurrency/#first-completion-selection), [Python SDK](/docs/polyglot/python/#first-completion-selection), [Rust SDK](/docs/polyglot/rust/#durable-first-completion-selection) |
| Saga compensation | PHP's Saga helpers, Python `WorkflowContext.saga()`, and Rust `WorkflowContext::saga()` register durable compensations after forward success and unwind sequentially in reverse order by default. Python and Rust stop at the first compensation failure and preserve the initiating plus compensation failures; PHP additionally exposes opt-in parallel and continue-on-error policies. | [Sagas](/docs/features/sagas/), [Python SDK](/docs/polyglot/python/#saga-compensation), [Rust SDK](/docs/polyglot/rust/#saga-compensation) |
| Cancellation | Embedded Laravel and service-mode PHP, Python, and Rust support cooperative cancellation requests with one root identity, an immutable deadline, and bounded cleanup on explicitly enabled workers. Immediate terminal cancellation remains available with selected-run safety and typed outcomes. | [Cancel and terminate](/docs/features/cancel-and-terminate/), [Service-mode cancellation](/docs/polyglot/cancellation/), [Rust SDK](/docs/polyglot/rust/) |
| Termination | Termination closes a run immediately with a distinct terminal outcome; it is exposed through Cloud and self-hosted Server service runtimes, CLI, PHP, Python, and Rust control surfaces. | [Cancel and terminate](/docs/features/cancel-and-terminate/), [Server API](/docs/polyglot/server-api-reference/) |
| Side effects | PHP, Python, and Rust record non-deterministic values exactly once and decode the recorded value on replay. | [Side effects](/docs/features/side-effects/), [Python SDK](/docs/polyglot/python/), [Rust SDK](/docs/polyglot/rust/) |
| Workflow Streams | PHP, Python, and Rust expose typed list, describe, offset subscription/resume, append, close, and error operations for Server-owned run-scoped output streams. All three author replay-safe emits from a durable command identity. Python resolves external payloads with its configured storage driver; PHP and Rust preserve opaque references. | [Workflow Streams](/docs/polyglot/workflow-streams/), [exact SDK matrix](/workflow-stream-capabilities.json), [runtime scenarios](/platform-conformance/workflow-stream-runtime-scenarios.json) |
| Version markers | PHP, Python, and Rust expose durable version markers for compatible code evolution. | [Versioning](/docs/features/versioning/), [Python SDK](/docs/polyglot/python/), [Rust SDK](/docs/polyglot/rust/) |
| Deterministic replay | PHP, Python, and Rust workflow workers reconstruct decisions from durable history and report typed non-determinism rather than re-running external work. | [Execution guarantees](/docs/constraints/execution-guarantees/), [Platform conformance](/docs/platform-conformance/), [Rust SDK](/docs/polyglot/rust/) |
| Schedules | Cloud and self-hosted Server service runtimes own durable schedules. PHP, Python, and CLI expose schedule operations. Rust does not claim a schedule-management API. | [Schedules](/docs/features/schedules/), [CLI reference](/docs/polyglot/cli-reference/), [Python SDK](/docs/polyglot/python/) |
| Namespaces | Service-mode runtimes are namespace-scoped. PHP/Python clients and CLI manage namespaces where the selected runtime exposes that operation; Rust workers/clients target a namespace but do not claim namespace administration at the current floor. | [Namespace, auth, and workers](/docs/polyglot/namespace-auth-workers/), [Server API](/docs/polyglot/server-api-reference/) |
| Search attributes | Cloud and self-hosted Server service runtimes index typed search attributes. PHP and Python expose authoring/control surfaces; CLI and operator APIs expose structured discovery and filtering. | [Search attributes](/docs/features/search-attributes/), [Python SDK](/docs/polyglot/python/) |
| Worker compatibility | SDKs register runtime, SDK version, build ID, supported types, protocol version, and capacity; the server publishes accepted versions and routing facts. | [Compatibility](/docs/compatibility/), [Worker compatibility and routing](/docs/polyglot/worker-compatibility-routing/) |
| Local activities | Available in embedded Laravel, PHP and Python service SDKs, and Rust SDK 3.1.0 with explicit worker opt-in. | [Embedded activity execution](/docs/features/activity-execution-model/), [Service-mode support matrix](/docs/polyglot/portable-worker-affinity/) |
| Worker sessions | Available in embedded Laravel, PHP and Python service SDKs, and Rust SDK 3.2.0 with explicit worker opt-in. | [Worker Sessions](/docs/features/worker-sessions/), [Service-mode support matrix](/docs/polyglot/portable-worker-affinity/) |
| Sticky execution | Available in embedded Laravel and the PHP service SDK. Python and Rust also provide optional bounded history caches with explicit worker opt-in. Every worker must retain correct cold replay behavior. | [Sticky Execution](/docs/features/sticky-execution/), [Service-mode support matrix](/docs/polyglot/portable-worker-affinity/) |
| Codec interoperability | PHP, Python, and Rust use the public `codec` + `blob` envelope and one fixed recursive Avro Value schema. Named branches preserve integers versus doubles, text versus bytes, booleans versus integers, and lists versus maps. | [Avro Value protocol](/docs/polyglot/avro-value-protocol/), [Worker protocol](/docs/polyglot/worker-protocol/), [Rust SDK](/docs/polyglot/rust/) |
| Diagnostics | Service runtimes and CLI publish version, protocol, worker, task-queue, replay, history, typed failure, and repair facts as JSON. Managed Waterline, a separately deployed self-hosted service, or the embedded package presents evidence from its owning runtime. | [Monitoring](/docs/monitoring/), [CLI reference](/docs/polyglot/cli-reference/), [Waterline operator API](/docs/waterline-operator-api/) |
| Agent tooling | Discover -> Change -> Run -> Diagnose -> Repair is available through public manifests, schemas, HTTP operations, CLI JSON, SDK clients, typed history/diagnostics, and safe mutations. MCP is one optional interface, not the definition. | [Agent tooling contract](/docs/agent-tooling-contract/), [Agent operating loop](/docs/agent-operating-loop/) |

## Payload interoperability boundary

Cross-language workflow and activity calls do not pass serialized PHP objects,
Python pickles, or Rust implementation types. They pass registered string type
names and a public payload envelope. With `avro`, each first-party SDK uses the
official language implementation and the same fixed Value schema; decoded
maps, lists, strings, bytes, integers, doubles, booleans, and nulls preserve
their primitive type. A
language-specific class is reconstructed only inside its owning SDK.

That contract applies to workflow input and result, child workflows, activity
input and result, signals, queries, updates, and external execution. Deployments
must still verify the codecs advertised by `GET /api/cluster/info` before
admitting a worker.

## Release and maturity boundary

The platform contracts indexed here are the stable 2.0 product surface.
Use the [AI-agent evaluator](/docs/ai-agent-workflow-engine/) for a concise
fit decision.

<!-- Source: docs/quickstart.md -->

# Durable Workflow 2.0 Quickstart

## Before you begin

**Goal:** run one service-mode workflow and read its completed durable result
from PHP, Python, or Rust.

**Expected time:** about 15 minutes after your runtime is available.

**Completed outcome:** the selected SDK starts a worker and workflow, then
prints a workflow ID, `status=completed`, and `Hello, <language>!`.

**Prerequisites:**

- `curl` and a terminal
- Docker for the self-hosted local path, or a provisioned Durable Workflow
  Cloud namespace
- one language toolchain: PHP 8.1+ with Composer, Python 3.10+, or Rust 1.86+

You do not need Laravel for service mode. The embedded Laravel path is separate
at the end of this guide.

## 1. Choose your service-mode runtime

| Runtime | Choose it when | Next action |
| --- | --- | --- |
| Durable Workflow Cloud | You want Durable Workflow to operate the runtime, persistence, and Managed Waterline. | Follow the executable [Cloud first workflow](/docs/polyglot/cloud-control-plane/#cloud-first-workflow), which maps PHP, Python, and Rust complete sources to provisioned credentials and a `completed` result. **Do not run Server or a separate Waterline service.** |
| Self-hosted Server | You want to operate the runtime yourself or run this exact local published-artifact exercise. | Continue below with Docker and `curl`; deploy Waterline separately only when you want its operator UI. |

The runnable source below uses a local self-hosted Server so it can be exercised
without an account or source checkout. Cloud uses the same SDK and worker model;
replace the local development connection with the provisioned values shown in
[Cloud Managed Runtime](/docs/polyglot/cloud-control-plane/).

## 2. Start the local Server

Skip this action when you chose Cloud. For the self-hosted path, expand and run
the exact pinned setup. It starts a source-free native Server with SQLite and a
development token.

## Start the pinned Server image

<!-- docs-example id="quickstart.server.setup" -->
```bash
export DW_SERVER_IMAGE=durableworkflow/server:3.10.0
export DW_AUTH_TOKEN=dev-token

docker volume create durable-workflow-quickstart

docker rm -f durable-workflow-server >/dev/null 2>&1 || true
docker run -d --name durable-workflow-server \
  -p 127.0.0.1:8080:8080 \
  -v durable-workflow-quickstart:/app/database \
  -e DW_AUTH_TOKEN="$DW_AUTH_TOKEN" \
  "$DW_SERVER_IMAGE"

until curl -sf http://localhost:8080/api/ready >/dev/null; do sleep 1; done
curl -H "Authorization: Bearer $DW_AUTH_TOKEN" \
  http://localhost:8080/api/cluster/info
```

The API is bound to this machine's loopback address. `dev-token` is only for
this local exercise. For deliberate remote access, use the [Compose deployment](/docs/polyglot/server/#published-image--compose)
with your own credentials, TLS and network access controls.

The native Server is one process. On first start it creates its SQLite schema
and the `default` namespace in the mounted volume, and it fires durable timers
and schedules and runs its maintenance passes itself, even while SDK workers
are stopped. There is no bootstrap command, queue worker or scheduler to run.
The Server refuses to start without `DW_AUTH_TOKEN`.

**Expected result:** the readiness request succeeds and cluster info reports
`"implementation":"rust"` with version `3.10.0`. Keep it
running while you complete one language route.

## 3. Choose one language {#choose-one-language}

All three first-party SDKs are available at the same level. Only the selected
tab is shown, so you can follow one path without scrolling past two other
programs.

## 4. Clean up the local Server

Cloud users have no local Server to remove. For the self-hosted exercise:

```bash
docker rm -f durable-workflow-server
docker volume rm durable-workflow-quickstart
```

## Separate path: embedded Laravel

Embedded Laravel is a separate first-party PHP deployment mode for
applications that want workflow state, queue execution, configuration, and
operator tooling inside their existing Laravel infrastructure. It installs
`durable-workflow/workflow`; it does not use the standalone server or
`durable-workflow/sdk`.

Start a fresh embedded application with the published package:

```bash
composer create-project laravel/laravel durable-workflow-laravel-quickstart
cd durable-workflow-laravel-quickstart
composer require durable-workflow/workflow:2.28.1
php artisan migrate
php artisan queue:work
```

When you want the operator UI inside that same Laravel application, add the
qualified embedded Waterline Composer package:

```bash
composer require durable-workflow/waterline:2.18.0
php artisan waterline:install
```

This Composer package is not the install identity for the separately deployed
self-hosted Waterline service. Embedded Laravel does not run Server or install
one of the service-mode SDKs.

Continue with [Embedded Installation](/docs/installation/) to configure a
non-`sync` Laravel queue, then [define](/docs/defining-workflows/workflows/)
and [start](/docs/defining-workflows/starting-workflows/) an embedded
workflow. [Deployment Modes](/docs/polyglot/deployment-modes/) compares
this specialized route with the service-mode platform.

## Next steps

- Use the [Capability Index](/docs/capabilities/) to check the supported
  surface for your selected runtime and SDK.
- Continue with the service-mode [PHP SDK](/docs/polyglot/php/),
  [Python SDK](/docs/polyglot/python/), or
  [Rust SDK](/docs/polyglot/rust/) guide.
- Compare lifecycle, messages, schedules, visibility, and worker execution in
  [Client and Worker Capabilities](/docs/polyglot/cli-python-parity/).
- Operate the matching runtime through
  [Cloud Managed Runtime](/docs/polyglot/cloud-control-plane/) or the
  [self-hosted Server](/docs/polyglot/server/), then add the
  [CLI](/docs/polyglot/cli/) when shell automation is useful.
- Plan safe service-worker deployment with
  [worker compatibility and routing](/docs/polyglot/worker-compatibility-routing/)
  and [build-ID rollout](/docs/polyglot/worker-build-id-rollout/).

For embedded Laravel authoring features such as timers, signals, queries,
activities, and child workflows, use the separate
[Embedded documentation](/docs/category/embedded/).

Release qualification is intentionally separate from this first-success
guide. The [Platform Conformance Suite](/docs/platform-conformance/)
contains the exact artifact matrix, public-source checks, full execution
transcripts, wall-clock criteria, teardown, and machine-readable quickstart
contract used for certification.

<!-- Source: docs/installation.md -->

# Installation

This guide covers installing the Durable Workflow PHP package for Laravel applications.

If you are deciding between package embedding and the standalone server, start
with [Deployment Modes](/docs/polyglot/deployment-modes). This page covers
the embedded Laravel path.

> **Prefer to start from a working app?** The
> [Sample App](/docs/sample-app) is a runnable Laravel 13 project on
> Durable Workflow 2.0 with one workflow per pattern surface, a Codespaces
> flow, and a `docker compose` flow. Clone it, run `php artisan app:init`,
> and you have the same install applied for you. Come back here when you
> are ready to add Durable Workflow to your own Laravel application.

## Requirements

- PHP 8.2 or later in the supported PHP 8.x line
- Laravel 12.72 or later in 12.x, or Laravel 13.41 or later in 13.x
- PHP 8.3 or later for Laravel 13

These requirements apply to the current embedded release. For an older
application, choose the appropriate [published support line](./compatibility.md#laravel-support)
before updating Composer. The bare service-mode PHP SDK has its own requirements.

Check [Database Compatibility](./polyglot/database-support.md) for the embedded
and Server declarations, exact tested configurations and qualification limits.

Durable Workflow can be used with any queue driver that Laravel supports (except the `sync` driver), including:

- Amazon SQS
- Beanstalkd
- Database
- Redis

Each queue driver has its own [prerequisites](https://laravel.com/docs/12.x/queues#driver-prerequisites).

Durable Workflow also requires a cache driver that supports [locks](https://laravel.com/docs/12.x/cache#atomic-locks).

## Installing Durable Workflow

Durable Workflow is installable via Composer:

```bash
composer require durable-workflow/workflow:2.28.1
```

Composer pins the published embedded release above. Review the
[Laravel support and upgrade boundary](./compatibility.md#laravel-support)
before widening that constraint or upgrading an older Laravel application.

The package auto-loads its migrations, so a normal migrate run is enough after install:

```bash
php artisan migrate
```

## Running Workers

Durable Workflow uses queues to run workflows and activities in the background. You will need to either run the `queue:work` [command](https://laravel.com/docs/12.x/queues#the-queue-work-command) or use [Horizon](https://laravel.com/docs/12.x/horizon) to run your queue workers. Without a queue worker, workflows and activities will not be processed. To run workflows and activities in parallel, you will need more than one queue worker.

<!-- Source: docs/laravel-adoption.md -->

# Laravel Adoption and Runtime Transition

Laravel applications have two first-party Durable Workflow 2.0 destinations.
Embedded mode keeps durable execution inside the application. Service mode
keeps Laravel as the application and worker framework while Cloud or a
self-hosted Server owns durable state.

This is the decision and transition page for three journeys:

1. Stable v1 Laravel to 2.0 embedded.
2. Stable v1 Laravel to the 2.0 PHP SDK in service mode.
3. 2.0 embedded Laravel to the 2.0 PHP SDK in service mode.

If your immediate goal is a package-level v1 upgrade, keep the detailed
[2.0 migration guide](/docs/migration/) open beside this page. If you have
already selected a self-hosted service runtime, continue with the
[embedded-to-Server runbook](/docs/polyglot/embedded-to-server/) after
choosing the transition policy here.

## Pick the destination

| Starting point | Destination | Choose it when | First transition boundary |
| --- | --- | --- | --- |
| Stable v1 Laravel | 2.0 embedded | The Laravel app should continue to own workflow persistence, queues, execution, and embedded Waterline. | Upgrade the maintained `durable-workflow/workflow` package to the 2.0 release line. Existing v1 runs finish through the package's v1 compatibility path; new starts use v2. |
| Stable v1 Laravel | PHP SDK service mode | Laravel should keep dependency injection, configuration, Artisan, logging, events, and test fakes, but Cloud or Server should own durable state. | Install `durable-workflow/sdk` as a separate service client/worker boundary. v1 history and durable Laravel queue jobs stay on the v1 runtime until terminal. |
| 2.0 embedded Laravel | PHP SDK service mode | The application should stop owning the orchestration runtime or needs a shared/polyglot runtime boundary. | Route new starts to the service runtime after its workers are ready. Drain embedded runs where they started, or use the explicit eligible embedded-v2 import procedure for self-hosted Server. |

Check the [published Laravel support lines](./compatibility.md#laravel-support)
before choosing a destination. The current embedded package requires Laravel
12.72 or 13.41 and compatible PHP. The service-mode Laravel bridge follows its
[own PHP SDK requirements](./polyglot/php.md#requirements). The bare
service-mode client keeps PHP 8.1 support. PHP SDK 3.9.x keeps its earlier
Laravel 10–13 bridge contract. An older v1
application may need a framework upgrade before moving to either current line.
Preserve its v1 runtime and open runs during that upgrade.

## Compare the ownership boundary

| Concern | Stable v1 Laravel | 2.0 embedded Laravel | 2.0 PHP SDK service mode |
| --- | --- | --- | --- |
| Runtime owner | The Laravel application owns orchestration and replay. | The Laravel application owns the v2 engine, history, matching, timers, and replay. | Cloud operates the namespace runtime, or your team operates a self-hosted Server. Laravel runs clients and remote workers, not the runtime. |
| Composer package | `durable-workflow/workflow` on the stable 1.x line. | `durable-workflow/workflow` from the 2.0 artifact authority. | `durable-workflow/sdk` from the independently published PHP SDK authority. It does not depend on the embedded engine. |
| Required processes | Laravel web/CLI processes plus Laravel queue workers or Horizon. | Laravel web/CLI processes plus queue workers or Horizon; scheduler and repair roles follow the embedded deployment configuration. | Laravel application processes plus one or more `php artisan durable-workflow:worker` processes. Cloud supplies the runtime. Self-hosting additionally requires Server and its persistence. |
| Queues and storage | Workflow rows and PHP-oriented history live in configured Laravel databases; workflow, activity, retry, and timer jobs live in Laravel queue backends. | Event history and projections live in the application's configured database; embedded tasks normally use Laravel queues. | Durable history and task state live in Cloud or Server persistence. `DURABLE_WORKFLOW_TASK_QUEUE` is a remote task-queue identity polled by the SDK worker, not a Laravel queue connection. |
| Workflow/activity identity | PHP workflow and activity classes are durable identities. | Configure stable aliases under `workflows.v2.types`; classes remain local implementations. | `#[Workflow('orders.fulfill')]` and `#[Activity('orders.reserve')]` publish stable string type keys during worker registration. Do not use PHP FQCNs as the cross-runtime contract. |
| Credentials | Laravel application, database, cache, and queue credentials; the original `APP_KEY` remains part of v1 recovery. | Laravel application credentials plus the database, cache, queue, and any embedded Waterline authentication owned by the host app. | Cloud application processes receive only a control credential and worker processes only a worker credential. A self-hosted deployment may use one shared token or equivalent scoped credentials. Credentials are process inputs and are deliberately absent from cached Laravel configuration. |
| Operational tooling | Laravel logs/events, queue tooling, Horizon, v1 Waterline, and `workflow:v1:list` where available. | Laravel logs/events, queue tooling, `workflow:v2:doctor`, history/repair commands, and embedded Waterline. | SDK diagnostics flow through Laravel's PSR logger and `WorkerDiagnosticEvent`. Cloud includes Managed Waterline. Self-hosted operators use Server APIs/CLI and may deploy Waterline separately against the Server namespace. |
| Cutover and rollback owner | Your team preserves the v1 database, queue state, configuration, and secrets as one recovery set. | Your team separates v1 and v2 starts, retains v1 wake paths, and decides when old tables/queues can be retired. | Your team routes each command to the runtime that owns its run. Cloud owns runtime rollback inside the managed service; self-hosted operators own Server persistence and rollback. Application rollback must keep both old and new runtime routes until their runs are terminal. |

Cloud and self-hosted Server are alternatives inside service mode. Cloud users
do not install Server or a separate Waterline service. Self-hosted operators
deploy, secure, back up, scale, and upgrade Server; Waterline is a separate
optional deployment against the Server-owned namespace.

## One Laravel use case in all three modes

The examples reserve inventory for `orders.fulfill`. Map the stable v1 class to
the v2 type keys deliberately, then keep those v2 type keys and the configured
task queue stable. Inventory reservation and the business event consumer must
be idempotent because an activity can execute more than once after a retry or
lost completion acknowledgement.

The application action remains the same in production and tests:

| Mode | Production entry point | Entry point after enabling its fake |
| --- | --- | --- |
| Stable v1 Laravel | `app(StartOrderFulfillment::class)->start('1001')` | `app(StartOrderFulfillment::class)->start('1001')` |
| 2.0 embedded Laravel | `app(StartOrderFulfillment::class)->start('1001')` | `app(StartOrderFulfillment::class)->start('1001')` |
| PHP SDK service mode | `app(StartOrderFulfillment::class)->start('1001')` | `app(StartOrderFulfillment::class)->start('1001')` |

Laravel constructor injection remains available at the application boundary in
every mode. Stable v1 injects activity dependencies into `execute()`. Embedded
v2 and service mode both ask Laravel's container to construct workflow and
activity objects, so their application dependencies use ordinary constructors.
Keep anything read by a replayed workflow deterministic and config-backed;
database, network, clock, and other side effects belong in activities.

### Stable v1 Laravel

Keep the [stable v1 installation](/docs/1.x/installation/) and its real queue-driver
requirements. This representative implementation uses a constructor-injected
application starter and method-injected activity services:

<!-- docs-example id="laravel.adoption.v1.handlers" -->
```php
<?php

use App\Contracts\InventoryGateway;
use App\Contracts\OrderPolicy;
use App\Events\InventoryReserved;
use Illuminate\Contracts\Events\Dispatcher;
use Psr\Log\LoggerInterface;
use Workflow\Activity;
use Workflow\Workflow;
use Workflow\WorkflowOptions;
use Workflow\WorkflowStub;
use function Workflow\activity;

final class FulfillOrderWorkflow extends Workflow
{
    public function execute(string $orderId): \Generator
    {
        return yield activity(ReserveInventoryActivity::class, $orderId);
    }
}

final class ReserveInventoryActivity extends Activity
{
    public function execute(
        OrderPolicy $orders,
        InventoryGateway $inventory,
        LoggerInterface $logger,
        Dispatcher $events,
        string $orderId,
    ): array {
        $orders->assertFulfillable($orderId);
        $reservation = $inventory->reserve($orderId);
        $logger->info('Inventory reserved', ['order_id' => $orderId]);
        $events->dispatch(new InventoryReserved($orderId));

        return $reservation;
    }
}

final class StartOrderFulfillment
{
    public function __construct(private readonly LoggerInterface $logger)
    {
    }

    public function start(string $orderId): void
    {
        $workflow = WorkflowStub::make(FulfillOrderWorkflow::class);
        $workflow->start($orderId, new WorkflowOptions('redis', 'orders'));
        $this->logger->info('Order workflow started', ['order_id' => $orderId]);
    }
}
```

`config/queue.php` and the deployment environment still select the real
Laravel queue connection. Run a worker for the configured queue:

<!-- docs-example id="laravel.adoption.v1.worker" -->
```bash
php artisan queue:work redis --queue=orders
```

The stable fake runs the same injected application action used in production:

<!-- docs-example id="laravel.adoption.v1.fake" -->
```php
<?php

use Workflow\WorkflowStub;

WorkflowStub::fake();
WorkflowStub::mock(ReserveInventoryActivity::class, ['status' => 'reserved']);

app(StartOrderFulfillment::class)->start('1001');

WorkflowStub::assertDispatched(
    ReserveInventoryActivity::class,
    fn (string $orderId): bool => $orderId === '1001',
);
```

Laravel event assertions remain available when testing the real activity. Do
not replace v1 workers merely because the v2 SDK has been added elsewhere in
the application.

### 2.0 embedded Laravel

The stable install uses the same Workflow package as the embedded quickstart:

<!-- docs-example id="laravel.adoption.v2-embedded.install" -->
```bash
composer require durable-workflow/workflow:2.28.1
```

Publish or update `config/workflows.php`, then give the same business contract
stable aliases. The published file contains additional required v1 and v2
settings; merge these values into its existing arrays instead of replacing the
complete file with this focused excerpt:

<!-- docs-example id="laravel.adoption.v2-embedded.config" -->
```php
<?php

return [
    'v2' => [
        'namespace' => env('DW_V2_NAMESPACE', 'production'),
        'types' => [
            'workflows' => [
                'orders.fulfill' => App\Workflows\FulfillOrderWorkflow::class,
            ],
            'activities' => [
                'orders.reserve' => App\Activities\ReserveInventoryActivity::class,
            ],
        ],
    ],
];
```

The authoring API changes to `handle()` and no longer yields activity results.
Laravel constructs both objects, while the activity remains the right place
for logging and business events that represent side effects:

<!-- docs-example id="laravel.adoption.v2-embedded.handlers" -->
```php
<?php

use App\Contracts\InventoryGateway;
use App\Contracts\OrderPolicy;
use App\Events\InventoryReserved;
use Illuminate\Contracts\Events\Dispatcher;
use Psr\Log\LoggerInterface;
use Workflow\V2\Activity;
use Workflow\V2\Workflow;
use Workflow\V2\WorkflowStub;
use function Workflow\V2\activity;

final class FulfillOrderWorkflow extends Workflow
{
    public ?string $connection = 'redis';
    public ?string $queue = 'orders';

    public function __construct(private readonly OrderPolicy $orders)
    {
    }

    public function handle(string $orderId): array
    {
        $this->orders->assertFulfillable($orderId);

        return activity(ReserveInventoryActivity::class, $orderId);
    }
}

final class ReserveInventoryActivity extends Activity
{
    public ?string $connection = 'redis';
    public ?string $queue = 'orders';

    public function __construct(
        private readonly OrderPolicy $orders,
        private readonly InventoryGateway $inventory,
        private readonly LoggerInterface $logger,
        private readonly Dispatcher $events,
    ) {
    }

    public function handle(string $orderId): array
    {
        $this->orders->assertFulfillable($orderId);
        $reservation = $this->inventory->reserve($orderId);
        $this->logger->info('Inventory reserved', ['order_id' => $orderId]);
        $this->events->dispatch(new InventoryReserved($orderId));

        return $reservation;
    }
}

final class StartOrderFulfillment
{
    public function __construct(private readonly LoggerInterface $logger)
    {
    }

    public function start(string $orderId): void
    {
        WorkflowStub::make(FulfillOrderWorkflow::class, "order-{$orderId}")
            ->start($orderId);
        $this->logger->info('Order workflow started', ['order_id' => $orderId]);
    }
}
```

Run the Laravel queue worker exactly as an embedded deployment requires:

<!-- docs-example id="laravel.adoption.v2-embedded.worker" -->
```bash
php artisan queue:work redis --queue=orders
```

Use the embedded fake through the same application action. It executes ready
workflow tasks inline and records activity dispatches:

<!-- docs-example id="laravel.adoption.v2-embedded.fake" -->
```php
<?php

use Workflow\V2\WorkflowStub;

WorkflowStub::fake();
WorkflowStub::mock(ReserveInventoryActivity::class, ['status' => 'reserved']);

app(StartOrderFulfillment::class)->start('1001');
while (WorkflowStub::runReadyTasks() > 0) {
}

WorkflowStub::assertDispatched(
    ReserveInventoryActivity::class,
    fn (string $orderId): bool => $orderId === '1001',
);
```

This fake exercises embedded durable rows and history. It is not the SDK client
fake and does not stand in for a service runtime.

### PHP SDK service mode in Laravel

Install the independently published SDK package. This command is rendered from
the published PHP SDK artifact authority, not copied from a point release:

<!-- docs-example id="laravel.adoption.service.install" -->
```bash
composer require durable-workflow/sdk:3.18.2
```

Supported Laravel versions auto-discover the provider. On an unsupported or
unknown Laravel or PHP version, the provider refuses before registering any
configuration, clients or workers. Publish the package configuration and list
the attributed handler classes:

<!-- docs-example id="laravel.adoption.service.config" -->
```php
<?php

return [
    'runtime_url' => env('DURABLE_WORKFLOW_RUNTIME_URL', 'http://localhost:8080'),
    'namespace' => env('DURABLE_WORKFLOW_NAMESPACE', 'production'),
    'task_queue' => env('DURABLE_WORKFLOW_TASK_QUEUE', 'orders'),
    'handlers' => [
        App\Workflows\FulfillOrderWorkflow::class,
        App\Activities\ReserveInventoryActivity::class,
    ],
    'poll_timeout_seconds' => 5,
];
```

The service workflow uses the same public strings as embedded v2. Laravel
container-constructs both attributed service classes:

<!-- docs-example id="laravel.adoption.service.handlers" -->
```php
<?php

use App\Contracts\InventoryGateway;
use App\Contracts\OrderPolicy;
use App\Events\InventoryReserved;
use DurableWorkflow\Attribute\Activity;
use DurableWorkflow\Attribute\Workflow;
use DurableWorkflow\Worker\ActivityContext;
use DurableWorkflow\Worker\WorkflowContext;
use Illuminate\Contracts\Events\Dispatcher;
use Psr\Log\LoggerInterface;

final class FulfillOrderWorkflow
{
    public function __construct(private readonly OrderPolicy $orders)
    {
    }

    #[Workflow('orders.fulfill')]
    public function run(WorkflowContext $context, string $orderId): array
    {
        $this->orders->assertFulfillable($orderId);

        return $context->activity('orders.reserve', [$orderId]);
    }
}

final class ReserveInventoryActivity
{
    public function __construct(
        private readonly OrderPolicy $orders,
        private readonly InventoryGateway $inventory,
        private readonly LoggerInterface $logger,
        private readonly Dispatcher $events,
    ) {
    }

    #[Activity('orders.reserve')]
    public function run(ActivityContext $context, string $orderId): array
    {
        $this->orders->assertFulfillable($orderId);
        $reservation = $this->inventory->reserve($orderId);
        $this->logger->info('Inventory reserved', ['order_id' => $orderId]);
        $this->events->dispatch(new InventoryReserved($orderId));

        return $reservation;
    }
}
```

Application code injects the Laravel-native SDK interface instead of reaching
through a facade or assembling a transport client. The attributed class
supplies the workflow type, and published configuration supplies the default
task queue:

<!-- docs-example id="laravel.adoption.service.client" -->
```php
<?php

use DurableWorkflow\Bridge\Laravel\LaravelWorkflowClientInterface;
use DurableWorkflow\WorkflowHandleInterface;

final class StartOrderFulfillment
{
    public function __construct(
        private readonly LaravelWorkflowClientInterface $workflows,
    ) {
    }

    public function start(string $orderId): WorkflowHandleInterface
    {
        return $this->workflows->start(
            FulfillOrderWorkflow::class,
            [$orderId],
            workflowId: "order-{$orderId}",
        );
    }

    public function handle(string $orderId): WorkflowHandleInterface
    {
        return $this->workflows->handle(
            FulfillOrderWorkflow::class,
            "order-{$orderId}",
        );
    }
}
```

After publishing the configuration, cache only non-secret settings. Set
`RUNTIME_URL` to the complete provisioned Cloud namespace runtime URI or to the
self-hosted Server origin; do not append `/api`.

Start the worker with only its worker-role credential:

<!-- docs-example id="laravel.adoption.service.worker" -->
```bash
php artisan vendor:publish --tag=durable-workflow-config
env -u DURABLE_WORKFLOW_TOKEN \
  -u DURABLE_WORKFLOW_CLIENT_TOKEN \
  -u DURABLE_WORKFLOW_WORKER_TOKEN \
  DURABLE_WORKFLOW_RUNTIME_URL="$RUNTIME_URL" \
  DURABLE_WORKFLOW_NAMESPACE=production \
  DURABLE_WORKFLOW_TASK_QUEUE=orders \
  php artisan config:cache
env -u DURABLE_WORKFLOW_CLIENT_TOKEN \
  DURABLE_WORKFLOW_WORKER_TOKEN="$WORKER_SECRET" \
  php artisan durable-workflow:worker
```

With no `--queue` override, the Artisan worker polls the configured `orders`
task queue. Its registered-and-polling diagnostic names the runtime, namespace,
queue, attributed workflow/activity types, and worker credential role before
normal polling begins.

Application/web/queue processes instead receive
`DURABLE_WORKFLOW_CLIENT_TOKEN` and not the worker token. Self-hosted operators
may use `DURABLE_WORKFLOW_TOKEN` when one credential is intentionally authorized
for both roles.

The SDK worker writes lifecycle, retry, handler-failure, and shutdown
diagnostics through Laravel's PSR logger and dispatches
`DurableWorkflow\Bridge\Event\WorkerDiagnosticEvent` through Laravel events.
Business activities can keep using the injected logger and event dispatcher as
shown above.

The shipped facade fake replaces the same injected interface used by
application code:

<!-- docs-example id="laravel.adoption.service.fake" -->
```php
<?php

use DurableWorkflow\Bridge\Laravel\Facades\DurableWorkflow;

$workflows = DurableWorkflow::fake()
    ->setWorkflowResult('order-1001', ['status' => 'fulfilled']);

app(StartOrderFulfillment::class)->start('1001');
$result = app(StartOrderFulfillment::class)->handle('1001')->result();

$this->assertSame(['status' => 'fulfilled'], $result);
$workflows->assertWorkflowStarted(
    FulfillOrderWorkflow::class,
    ['1001'],
    workflowId: 'order-1001',
);
$workflows->assertResultRequested('order-1001');
```

This fake proves the Laravel application interaction without requiring Cloud or
Server. Use the SDK's worker test harness when testing workflow/activity handler
commands; the client fake does not execute remote worker code.

Durations carry over from embedded workflows. From PHP SDK 3.9.0, service-mode
timers, condition-wait timeouts and activity and child workflow timeouts accept
the same `'30 minutes'` strings as embedded `timer()` and `await()`, with the
same rounding to whole seconds, as well as `DateInterval` and `CarbonInterval`
values. Rewrite ISO 8601 strings as `new DateInterval(...)`, and keep activity
and child workflow timeouts at one second or more. See
[PHP SDK durations](/docs/polyglot/php/#durations).

## Drain, cut over, and roll back

Changing a Composer requirement does not move an in-flight workflow, its
history, a delayed timer, an activity retry, or a ready/reserved queue job.
Treat the old and new runtimes as separate durable systems throughout the
transition.

### 1. Establish the recovery cut

Before changing traffic:

- Inventory the v1/embedded database connection, every configured workflow and
  activity queue, queue backend account/region/prefix, cache locks, and the
  secret-manager reference for the original `APP_KEY`.
- Block new starts and message ingress long enough to quiesce the source. Stop
  every worker after its current job boundary; `queue:restart` alone is not a
  quiesce when a supervisor immediately replaces the process.
- Either drain all source runs, or capture an application-consistent recovery
  cut containing SQL plus every ready, delayed, and reserved queue job. A
  database-only backup does not recreate queue-backed timers, retries, or
  activities.
- Record which runtime owns each nonterminal workflow ID. Signals, queries,
  updates, cancellation, termination, repair, and result reads must continue to
  follow that ownership record.

### 2. Apply the path-specific rule

| Transition | Existing nonterminal runs | New runs after cutover | Rollback boundary |
| --- | --- | --- | --- |
| v1 to 2.0 embedded | Finish through the v1 compatibility path. Retain their v1 tables, PHP decoding requirements, Laravel queue consumers, and queue state until terminal. | Start through `Workflow\V2\WorkflowStub` only after the v2 canary passes. | Restore the matched v1 SQL/queue recovery cut and exact app configuration, or fix forward. Do not restore SQL over newer queue state. |
| v1 to Cloud or self-hosted service mode | Stay on v1. v1 history cannot be imported into the service runtime or replayed by remote workers. | Start with `LaravelWorkflowClientInterface` after a registered SDK worker advertises the expected type keys. | Route new starts back to v1 only if the v1 runtime remains healthy. Runs already accepted by service mode stay there. |
| Embedded v2 to Cloud | Drain embedded runs in place; Cloud onboarding does not turn a package change into history import. | Start in the provisioned Cloud namespace after client/worker credential and task-queue checks pass. | Route new starts back to embedded if needed; keep commands for Cloud-accepted runs pointed at Cloud. |
| Embedded v2 to self-hosted Server | Drain in place, or separately perform the supported export, dry-run, and atomic import for an eligible quiesced embedded-v2 run. Never infer eligibility from package installation. | Start through the SDK after Server discovery, namespace, worker registration, and queue checks pass. | A failed import writes no partial run. After a committed import, Server owns that run; do not resume the embedded copy. |

For the self-hosted import option, use the complete
[embedded-to-Server migration procedure](/docs/polyglot/embedded-to-server/#phase-e-import-eligible-embedded-v2-runs).
It rejects v1, redacted history, leased tasks, running activity attempts, and
other unsafe snapshots. History export by itself is audit/debugging data; only
that explicit validated import operation creates Server-owned state.

### 3. Prove the destination before switching traffic

1. Configure one namespace, task queue, and stable workflow/activity type set.
2. Start the destination runtime and its SDK workers beside the old runtime.
3. Verify Server discovery or Cloud provisioning, role credentials, worker
   registration, advertised type keys, task-queue visibility, and Laravel
   diagnostics.
4. Send a uniquely identified shadow/canary order and wait for a terminal
   result in the destination's operator surface.
5. Switch new starts one workflow family at a time. Prevent the same business
   key from starting in both runtimes.
6. Keep source workers, queue state, credentials, logs/events, and the old
   operator surface until every source-owned run is terminal or has completed a
   supported self-hosted import.

Retiring an old package, queue, table, secret, or Waterline deployment is the
last step, never the cutover mechanism.

For moves between service-mode runtimes, use [moving between runtimes](/docs/polyglot/runtime-migration/). Server-sourced history export is not live-state import.

## Continue with the chosen path

- [Stable v1 installation](/docs/1.x/installation/)
- [Stable v1 migration planning](/docs/migration/)
- [2.0 embedded installation](/docs/installation/)
- [Detailed v1-to-v2 package migration](/docs/migration/)
- [PHP SDK service mode](/docs/polyglot/php/)
- [Cloud managed runtime](/docs/polyglot/cloud-control-plane/)
- [Self-hosted Server](/docs/polyglot/server/)
- [Embedded v2 to self-hosted Server](/docs/polyglot/embedded-to-server/)

<!-- Source: docs/defining-workflows/workflows.md -->

# Workflows

Workflows and activities are defined as classes that extend the base `Workflow` and `Activity` classes provided by the framework. A workflow is a class that defines a sequence of activities that run in parallel, series or a mixture of both.

You may use the `make:workflow` artisan command to create a new workflow:

```php
php artisan make:workflow MyWorkflow --v2
```

It is defined by extending the `Workflow` class and implementing the `handle()` method.

```php
use function Workflow\V2\activity;
use Workflow\V2\Workflow;

class MyWorkflow extends Workflow
{
    public function handle()
    {
        return activity(MyActivity::class);
    }
}
```

<!-- Source: docs/defining-workflows/activities.md -->

# Activities

An activity is a unit of work that performs a specific task or operation (e.g. making an API request, processing data, sending an email) and can be executed by a workflow.

:::note Durable Execution Contract
Ordinary v2 activities run as durable queued tasks. Local activities are an
explicit same-process primitive for short activity work that still records
durable activity history. Worker sessions are available when multiple durable
activity steps need one worker lease, and sticky execution is a supported
replay optimization, not a correctness contract. See [Activity Execution Model](/docs/features/activity-execution-model),
[Local Activities](/docs/features/local-activities), [Worker Sessions](/docs/features/worker-sessions),
and [Sticky Execution](/docs/features/sticky-execution) for the exact
contracts.
:::

You may use the `make:activity` artisan command to create a new activity:

```php
php artisan make:activity MyActivity --v2
```

It is defined by extending the `Activity` class and implementing the `handle()` method.

```php
use Workflow\V2\Activity;

class MyActivity extends Activity
{
    public function handle()
    {
        // Perform some work...
        return $result;
    }
}
```

## Execution Contract

Every `activity(...)` call records an activity command on workflow history and
creates a durable queued task for a worker to claim. Ordinary activities may
be claimed by any compatible worker, while [worker sessions](/docs/features/worker-sessions)
can pin a sequence of activity attempts to one worker-session lease when the
workflow explicitly opts into that contract.

If you need a replay-safe value without scheduling queued work, use
[`sideEffect(...)`](/docs/features/side-effects) instead. If you need a
short retryable same-process activity with timeout, heartbeat, and activity
history semantics, use [Local Activities](/docs/features/local-activities).
For the full placement model, see [Activity Execution Model](/docs/features/activity-execution-model).
For sticky replay-cache behavior, see
[Sticky Execution](/docs/features/sticky-execution).

## Idempotency and Durable Identity

Activity execution is at-least-once. Retries, lease expiry, and redelivery can
cause the same logical activity to be observed more than once, so the side
effect or remote target must be safe to repeat.

Inside the activity, use the runtime's durable identifiers when you need
correlation or remote dedupe:

- `activityId()` identifies one logical activity execution across retries and
  is the default remote idempotency key.
- `attemptId()` identifies one concrete try of that execution.
- `attemptCount()` tells you which try is currently running.

Prefer `activityId()` when the remote system should treat retries as the same
logical request. Reach for `attemptId()` only when the remote system truly
needs to distinguish separate tries. If a worker finishes remote work, loses
its lease, and reports late, the engine may reject that late completion because
another worker already won the durable race, but the remote side effect may
still have happened.

See [Execution Guarantees and Idempotency](/docs/constraints/execution-guarantees),
[Heartbeats](/docs/features/heartbeats), and
[Failures and Recovery](/docs/failures-and-recovery) for the operational
model behind those identifiers.

## Per-Call Overrides

Routing and retries default to the activity class's own `$connection`, `$queue`, `$tries`, and `backoff()` properties. When a single call needs to override those — for example, routing one call to a higher-priority queue or giving it more retry attempts — pass an `ActivityOptions` instance:

```php
use function Workflow\V2\activity;
use Workflow\V2\Support\ActivityOptions;

$result = activity(
    MyActivity::class,
    new ActivityOptions(queue: 'high-priority', maxAttempts: 5),
    'Taylor',
);
```

See [Activity options](/docs/configuration/options#activityoptions) for the full list of fields, including timeouts and heartbeats.

## Local Activity Calls

Use `localActivity(...)` when a short idempotent side effect should run inside
the current workflow worker process but still needs activity retry, timeout,
heartbeat, cancellation, and history semantics:

```php
use Workflow\V2\Support\LocalActivityOptions;
use function Workflow\V2\localActivity;

$receipt = localActivity(
    SendReceiptActivity::class,
    new LocalActivityOptions(maxAttempts: 3, startToCloseTimeout: 10),
    $orderId,
);
```

Local activities reject `connection`, `queue`, worker-session routing, and
schedule-to-start options because they do not create ordinary activity tasks.
See [Local Activities](/docs/features/local-activities) for the complete
contract.

<!-- Source: docs/defining-workflows/starting-workflows.md -->

# Starting Workflows

To start a workflow, create a workflow instance and then call the `start()` method on it.

```php
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::make(MyWorkflow::class);
$workflow->start();
```

Once a workflow has been started, it will be executed asynchronously by a queue worker. The `start()` method returns immediately and does not block the current request.

Pass arguments to the workflow's `handle()` method through `start()`:

```php
$workflow->start($orderId);
```

You can attach visibility labels, a business key, memo, or timeouts through `StartOptions`. See [Start Options](/docs/configuration/options#startoptions) when you need them.

<!-- Source: docs/defining-workflows/passing-data.md -->

# Passing Data

You can pass data into a workflow via the `start()` method.

```php
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::make(MyWorkflow::class);
$workflow->start('world');
```

Arguments are passed to the workflow's `handle()` method.

Similarly, you can pass data into an activity via the `activity()` helper function.

```php
use function Workflow\V2\activity;
use Workflow\V2\Workflow;

class MyWorkflow extends Workflow
{
    public function handle($name)
    {
        return activity(MyActivity::class, $name);
    }
}
```

Arguments are passed to the activity's `handle()` method.

```php
use Workflow\V2\Activity;

class MyActivity extends Activity
{
    public function handle($name)
    {
        return "Hello, {$name}!";
    }
}
```

In general, you should only pass small amounts of data in this manner. Rather than passing large amounts of data, you should write the data to the database, cache or file system. Then pass the key or file path to the workflow and activities. The activities can then use the key or file path to read the data.

When the application genuinely needs to carry large bytes through history — documents, media blobs, serialized exports — enable [External Payload Storage](../features/external-payload-storage.md) on the namespace. The runtime offloads over-threshold payloads to a configured object store and records a verifiable reference envelope in history, keeping replay integrity while staying under the [`payload_size_bytes` structural limit](../constraints/structural-limits.md#payload-size).

## Output

Once the workflow has completed, you can retrieve the output using the `output()` method.

```php
$workflow->output();
=> 'Hello, world!'
```

## Models

V2 payloads use a portable value format, so an Eloquent model can't be a workflow or activity argument or result. Pass the model's key and load the model inside the activity that uses it.

```php
use function Workflow\V2\activity;
use Workflow\V2\Workflow;

class MyWorkflow extends Workflow
{
    public function handle(int $userId)
    {
        return activity(MyActivity::class, $userId);
    }
}
```

```php
use App\Models\User;
use Workflow\V2\Activity;

class MyActivity extends Activity
{
    public function handle(int $userId)
    {
        $user = User::findOrFail($userId);

        return "Hello, {$user->name}!";
    }
}
```

Load models in activities, not in workflow code. Workflow code is replayed, and a database query can return different rows on a later replay.

Passing a model to `start()` throws an `InvalidArgumentException`. Passing a model to `activity()` fails the workflow run. Both errors start with `unsupported_value_type`.

The same rule applies to Eloquent collections. Pass an array of keys instead. If the activity needs only a few attributes, pass just those values as an array. Workflow history keeps them, so never include credentials or other secrets.

In v1, workflows and activities received models through `SerializesModels`, which stored a model identifier and reloaded the model when the job ran. V2 doesn't do this.

## Dependency Injection

In addition to passing data, you are able to type-hint dependencies on the workflow or activity `handle()` methods. The Laravel service container will automatically inject those dependencies.

```php
use Illuminate\Contracts\Foundation\Application;
use Workflow\V2\Workflow;

class MyWorkflow extends Workflow
{
    public function handle(Application $app)
    {
        if ($app->runningInConsole()) {
            // ...
        }
    }
}
```

Dependency injection does not relax workflow determinism. A workflow may
type-hint stable services such as the Laravel application container or
configuration readers, but injected dependencies that do work still need the
right durable boundary. Put external I/O, database access, service calls, or
other work that can fail or should retry in `activity(...)` or
`localActivity(...)`. Use `sideEffect()` only for a one-time replay-safe value
snapshot that should be frozen in history, such as a wall-clock read, generated
randomness, or a config value captured at decision time.

<!-- Source: docs/defining-workflows/workflow-status.md -->

# Workflow Status

`running()` observes the handle's loaded state. Call `refresh()` before observing
a workflow that another queue worker is executing. Use a bounded wait with a
pause between reads:

```php
$deadline = hrtime(true) + 20_000_000_000;
do {
    $workflow->refresh();
    if (! $workflow->running()) {
        break;
    }
    usleep(100_000);
} while (hrtime(true) < $deadline);

if ($workflow->running()) {
    throw new RuntimeException('Workflow is still running; observation deadline expired.');
}
if (! $workflow->completed()) {
    throw new RuntimeException('Workflow ended with status '.$workflow->status());
}
$result = $workflow->output();
```

An observation timeout stops this caller's wait. It does not cancel the workflow.
Keep the workflow ID so another observer can load its current state. A terminal
failure, cancellation or termination must be handled before reading a successful
output.

An instance handle from `Workflow\V2\WorkflowStub::load($workflowId)` follows the
current run when refreshed, including across `continueAsNew()`. A handle from
`loadRun()` or `loadSelection()` keeps observing the selected run, whose completion
can mean it continued into a successor. See [Workflow API](./workflow-api.md) for
run selection and closure details.

## Status Values

The `status()` method returns string statuses:

```text
reserved
pending
running
waiting
cancelled
terminated
completed
failed
```

- `reserved` means an instance id has been created but the first start command has not been accepted yet
- `pending` means the run exists and has work ready to be claimed
- `running` means a workflow task is actively leased to a worker
- `waiting` means the run is blocked on a durable resume source such as an activity, timer, or named signal
- `cancelled` means an accepted engine-level cancel command closed the current run
- `terminated` means an accepted engine-level terminate command force-closed the current run
- `completed`, `failed`, `cancelled`, and `terminated` are terminal run states

When a run uses `continueAsNew()`, the old run ends with `status = completed` and `closed_reason = continued`. Waterline keeps that run in the completed bucket while still surfacing the exact `closed_reason` so operators can see that the instance rolled forward into a newer run.

`running()` returns `true` for `pending`, `running`, and `waiting`.

## State Machine

This is the state machine for a workflow status.

<!-- Source: docs/defining-workflows/workflow-api.md -->

# Workflow API

This page is the complete reference. For an easier to read version, please look at the individual feature page.

- [Workflows](./workflows.md) for the workflow class shape and deterministic
  orchestration model.
- [Activities](./activities.md) for side effects, retries, and routing.
- [Local Activities](../features/local-activities.md) for short same-process
  activity work with activity history and retry semantics.
- [Signals](../features/signals.md), [Updates](../features/updates.md), and
  [Queries](../features/queries.md) for workflow-facing contracts.
- [Timers](../features/timers.md),
  [Condition Waits](../features/condition-waits.md), and
  [Continue As New](../features/continue-as-new.md) for long-running control
  flow.
- [Message Streams](../features/message-streams.md) for repeated ordered
  messages with cursor semantics.

## More Info for AI

Most readers can skip the disclosure below. Open it when you need exact
signatures, return contracts, machine-operable notes, or the full API
surface in one place.

## More Info for AI

**Base workflow object**

```php
use Workflow\V2\Workflow;

abstract class Workflow
{
    public ?string $connection = null;
    public ?string $queue = null;

    public function workflowId(): string;
    public function runId(): string;
    public function lastChild(): ?ChildWorkflowHandle;
    public function children(): array;
    public function historyLength(): int;
    public function historySize(): int;
    public function shouldContinueAsNew(): bool;
    public function historyBudgetAtStep(): array;
    public function shouldContinueAsNewAtStep(): bool;
}
```

| Member | Use when | Return contract |
| --- | --- | --- |
| `workflowId()` | The workflow needs its stable public instance id. | Instance id string, unchanged across continue-as-new. |
| `runId()` | The workflow needs the currently executing run id. | Run id string for the selected execution. |
| `lastChild()` | The workflow needs to signal the most recently spawned child. | `ChildWorkflowHandle` or `null`. |
| `children()` | The workflow needs handles for children visible to the current replay sequence. | List of `ChildWorkflowHandle`. |
| `historyLength()` | The workflow needs a count-based history budget signal. | Current history event count. |
| `historySize()` | The workflow needs a byte-based history budget signal. | Approximate persisted history size in bytes. |
| `shouldContinueAsNew()` | The workflow should rotate before history becomes expensive. | `true` when configured history budgets recommend rotation. |
| `historyBudgetAtStep()` | The workflow reads the budget more than once in a run, for example in a loop (Workflow 2.26+). | Budget counters as of the current step, the same on every replay. |
| `shouldContinueAsNewAtStep()` | A loop decides whether to rotate (Workflow 2.26+). | `true` when the history before the current step crosses a configured budget. |

## Durable commands

The static facade delegates to namespaced helpers in `Workflow\V2`. The two
forms are equivalent:

```php
use Workflow\V2\Workflow;
use function Workflow\V2\activity;

$resultFromFacade = Workflow::activity(SendReceipt::class, $orderId);
$resultFromHelper = activity(SendReceipt::class, $orderId);
```

<span id="durable-command-surface"></span>

| Facade | Helper | Signature | Durable effect |
| --- | --- | --- | --- |
| `Workflow::activity()` | `activity()` | `activity(string $activity, mixed ...$arguments): mixed` | Schedules an activity and waits for its result. |
| `Workflow::executeActivity()` | `activity()` | `executeActivity(string $activity, mixed ...$arguments): mixed` | Alias for `activity()`. |
| `Workflow::localActivity()` | `localActivity()` | `localActivity(string $activity, mixed ...$arguments): mixed` | Runs a short activity in the current workflow worker process and records activity history with `execution_mode=local`. |
| `Workflow::executeLocalActivity()` | `localActivity()` | `executeLocalActivity(string $activity, mixed ...$arguments): mixed` | Alias for `localActivity()`. |
| `Workflow::child()` | `child()` | `child(string $workflow, ChildWorkflowOptions? $options = null, mixed ...$arguments): mixed` | Starts a child workflow and waits for its result. Pass a `ChildWorkflowOptions` as the first argument to set the parent-close policy (default `ParentClosePolicy::Abandon`) or override child routing. |
| `Workflow::executeChildWorkflow()` | `child()` | `executeChildWorkflow(string $workflow, ChildWorkflowOptions? $options = null, mixed ...$arguments): mixed` | Alias for `child()`. |
| `Workflow::async()` | `async()` | `async(callable $callback): mixed` | Runs a callable as an auto-generated child workflow. |
| `Workflow::all()` | `all()` | `all(iterable $calls): mixed` | Waits for concurrent calls and returns results in iteration order. |
| `Workflow::parallel()` | `all()` | `parallel(iterable $calls): mixed` | Alias for `all()`. |
| `Workflow::select()` | `select()` | `select(iterable $calls): SelectionResult` | Starts independent durable calls and returns the first committed winner plus stable handles for every member. |
| `Workflow::await()` | `await()` | `await(callable\|string $condition, int\|string\|CarbonInterval\|null $timeout = null, ?string $conditionKey = null): mixed` | Waits for a named signal or replay-safe condition. |
| `Workflow::awaitWithTimeout()` | `await()` | `awaitWithTimeout(int\|string\|CarbonInterval $timeout, callable\|string $condition, ?string $conditionKey = null): mixed` | Waits for a signal or condition with an explicit timeout. |
| `Workflow::awaitSignal()` | `await()` | `awaitSignal(string $name): mixed` | Waits for a named signal. |
| `Workflow::timer()` | `timer()` | `timer(int\|string\|CarbonInterval $duration): mixed` | Suspends until durable time advances. |
| `Workflow::sideEffect()` | `sideEffect()` | `sideEffect(callable $callback): mixed` | Records a non-deterministic result in history and replays it. |
| `Workflow::uuid4()` | `uuid4()` | `uuid4(): mixed` | Generates a replay-stable UUIDv4. |
| `Workflow::uuid7()` | `uuid7()` | `uuid7(): mixed` | Generates a replay-stable UUIDv7. |
| `Workflow::continueAsNew()` | `continueAsNew()` | `continueAsNew(mixed ...$arguments): mixed` | Ends the current run and starts a new run for the same instance. |
| `Workflow::getVersion()` | `getVersion()` | `getVersion(string $changeId, int $minSupported = WorkflowStub::DEFAULT_VERSION, int $maxSupported = 1): mixed` | Negotiates a replay-safe workflow-code version. |
| `Workflow::patched()` | `patched()` | `patched(string $changeId): mixed` | Returns whether the run crossed a named patch marker. |
| `Workflow::deprecatePatch()` | `deprecatePatch()` | `deprecatePatch(string $changeId): mixed` | Keeps a patch marker alive after old code is removed. |
| `Workflow::upsertMemo()` | `upsertMemo()` | `upsertMemo(array $entries): void` | Updates non-indexed run metadata. |
| `Workflow::upsertSearchAttributes()` | `upsertSearchAttributes()` | `upsertSearchAttributes(array $attributes): void` | Updates indexed operator-visible metadata. |
| `Workflow::now()` | `now()` | `now(): CarbonInterface` | Reads deterministic workflow time. |

`activity()` and `executeActivity()` schedule durable queued activity tasks.
`localActivity()` and `executeLocalActivity()` run in the current workflow
worker process and record normal activity history with the local marker. Use
`sideEffect()` for replay-safe snapshots that should not use activity retry,
timeout, heartbeat, or cancellation semantics. Use `Workflow::workerSession()`
or `Workflow\V2\workerSession()` when multiple ordinary activity steps need
the supported worker-session affinity contract. See
[Activity Execution Model](/docs/features/activity-execution-model),
[Local Activities](/docs/features/local-activities), and
[Worker Sessions](/docs/features/worker-sessions) for the full execution
contract.

**Timer helpers**

Timer helpers are shorthand for `timer()` and `Workflow::timer()`:

```php
use Workflow\V2\Workflow;

Workflow::seconds(30);
Workflow::minutes(5);
Workflow::hours(2);
Workflow::days(1);
Workflow::weeks(1);
Workflow::months(1);
Workflow::years(1);
```

| Helper | Equivalent |
| --- | --- |
| `seconds(int $seconds)` | `timer($seconds)` |
| `minutes(int $minutes)` | `timer($minutes * 60)` |
| `hours(int $hours)` | `timer($hours * 3600)` |
| `days(int $days)` | `timer($days * 86400)` |
| `weeks(int $weeks)` | `timer($weeks * 604800)` |
| `months(int $months)` | `timer("{$months} months")` |
| `years(int $years)` | `timer("{$years} years")` |

Use explicit `timer()` calls when the duration comes from configuration or
workflow input. Use timer helpers when the source code should read as a fixed
business wait.

**Message streams**

Open durable message streams from the workflow instance:

```php
use Workflow\V2\MessageStream;
use Workflow\V2\Workflow;

final class AssistantWorkflow extends Workflow
{
    public function handle(string $targetWorkflowId): array
    {
        $message = $this->inbox('ai.user')->receiveOne();

        if ($message === null) {
            return ['status' => 'waiting'];
        }

        $reply = $this->outbox('ai.assistant')->sendReference(
            targetInstanceId: $targetWorkflowId,
            payloadReference: 'app://payloads/reply-123',
            correlationId: $this->workflowId(),
            idempotencyKey: 'reply-123',
            metadata: ['kind' => 'assistant_reply'],
        );

        return [
            'status' => 'sent',
            'stream' => $reply->stream_key,
            'sequence' => $reply->sequence,
        ];
    }
}
```

| Method | Signature | Contract |
| --- | --- | --- |
| `$this->messages()` | `messages(?string $streamKey = null, ?MessageService $messages = null): MessageStream` | Opens the stream for reading or sending. |
| `$this->inbox()` | `inbox(?string $streamKey = null, ?MessageService $messages = null): MessageStream` | Alias for inbound authoring code. |
| `$this->outbox()` | `outbox(?string $streamKey = null, ?MessageService $messages = null): MessageStream` | Alias for outbound authoring code. |
| `MessageStream::key()` | `key(): string` | Returns the stream key. |
| `MessageStream::cursor()` | `cursor(): int` | Returns the durable cursor position for this run. |
| `MessageStream::hasPending()` | `hasPending(): bool` | Returns whether unconsumed messages exist on the stream. |
| `MessageStream::pendingCount()` | `pendingCount(): int` | Returns the number of unconsumed messages on the stream. |
| `MessageStream::peek()` | `peek(int $limit = 100): Collection` | Reads pending messages without consuming them. |
| `MessageStream::receive()` | `receive(int $limit = 1, ?int $consumedBySequence = null): Collection` | Reads and consumes messages, recording cursor advancement. |
| `MessageStream::receiveOne()` | `receiveOne(?int $consumedBySequence = null): ?WorkflowMessage` | Reads and consumes one message. |
| `MessageStream::sendReference()` | `sendReference(string $targetInstanceId, ?string $payloadReference = null, MessageChannel\|string $channel = MessageChannel::WorkflowMessage, ?string $correlationId = null, ?string $idempotencyKey = null, array $metadata = [], ?DateTimeInterface $expiresAt = null): WorkflowMessage` | Sends an ordered payload-reference message to another workflow instance. |

Use message streams for repeated ordered messages with cursor semantics. Use
[Signals](../features/signals.md) for one-shot external events and
[Updates](../features/updates.md) for request/return mutations.

**Attributes and public contracts**

```php
use Workflow\QueryMethod;
use Workflow\UpdateMethod;
use Workflow\V2\Attributes\Signal;
use Workflow\V2\Attributes\Type;
use Workflow\V2\Workflow;

#[Type('order-approval')]
#[Signal('approved-by', [
    ['name' => 'approvedBy', 'type' => 'string', 'allows_null' => false],
])]
final class OrderApprovalWorkflow extends Workflow
{
    private string $stage = 'waiting';

    public function handle(): void
    {
        $this->stage = Workflow::awaitSignal('approved-by');
    }

    #[QueryMethod('current-stage')]
    public function currentStage(): string
    {
        return $this->stage;
    }

    #[UpdateMethod('mark-ready')]
    public function markReady(): string
    {
        return $this->stage = 'ready';
    }
}
```

| Attribute | Target | Stable contract |
| --- | --- | --- |
| `#[Type('type-key')]` | Workflow or activity class | Declares the language-neutral durable type key. |
| `#[Signal('signal-name', [...])]` | Workflow class, repeatable | Declares accepted signal names and optional ordered parameter contracts. |
| `#[QueryMethod('query-name')]` | Workflow method | Declares a replay-safe query name. Omit the name to use the PHP method name. |
| `#[UpdateMethod('update-name')]` | Workflow method | Declares a replay-safe update name. Omit the name to use the PHP method name. |

Signals, queries, and updates are public workflow contracts. Prefer explicit
names so PHP method renames do not become API breaks.

**Failure surface**

Authoring API failures are durable workflow failures unless the command is
rejected before execution:

| Surface | Typical failure | Operator meaning |
| --- | --- | --- |
| `activity()` | Activity throws, times out, or exhausts retry policy. | The run records activity failure history and follows workflow error handling. |
| `child()` | Child workflow fails, cancels, terminates, or times out. | The parent observes a child failure outcome at the waiting command. |
| `await()` | Timeout elapses before the condition or signal is satisfied. | The wait returns or fails according to the selected await form. |
| `timer()` | Invalid duration input after normalization. | Authoring code should pass positive durations or explicit zero-duration waits. |
| `continueAsNew()` | New run cannot be created. | Current run remains the evidence point for the failed transition. |
| `upsertSearchAttributes()` | Attribute key, count, or total size exceeds limits. | The run fails before invalid indexed metadata is persisted. |
| `MessageStream::receive()` | No positive workflow history sequence is available. | Receive must occur from workflow execution, not direct service code. |
| `MessageStream::sendReference()` | Payload reference, route, or storage contract is invalid downstream. | Message ordering remains separate from payload-store integrity. |

For payload and history limits, see [Structural Limits](../constraints/structural-limits.md).
For command rejection responses outside PHP workflow code, see
[Server API Reference](../polyglot/server-api-reference.md).

**Determinism rules**

Workflow code must be replay-safe. Keep irreversible or non-deterministic work
behind durable commands:

```php
use Workflow\V2\Workflow;

final class DeterministicWorkflow extends Workflow
{
    public function handle(): array
    {
        $workflowTime = Workflow::now();
        $stableId = Workflow::uuid7();
        $remoteQuote = Workflow::activity(FetchQuote::class);

        return [
            'time' => $workflowTime->toIso8601String(),
            'id' => $stableId,
            'quote' => $remoteQuote,
        ];
    }
}
```

- Use `Workflow::now()` instead of wall-clock time in workflow branches.
- Use `Workflow::uuid4()` or `Workflow::uuid7()` instead of direct randomness.
- Put network calls, filesystem writes, email sends, and external side effects
  in activities.
- Use `Workflow::sideEffect()` only when the value must be captured in history
  and the side effect itself is not the business action.
- Use `Workflow::getVersion()`, `Workflow::patched()`, and
  `Workflow::deprecatePatch()` to evolve workflow code without breaking replay.

<!-- Source: docs/defining-workflows/workflow-id.md -->

# Workflow ID

## Instance IDs and Run IDs

A durable workflow has two identifiers with different lifecycles:

- The **instance id** (`id()`, `workflowId()`) is the stable public handle for the workflow. It is generated when the workflow is first started — or supplied by the caller — and never changes for the lifetime of that workflow, including across continue-as-new chains.
- The **run id** (`runId()`) identifies one concrete execution generation of that workflow. It is newly allocated every time a new run is created, which happens on the first start and when the workflow continues as new.

Treat the instance id as the public, callable handle. Treat the run id as a pointer to a specific generation that is useful for inspection, debugging, or audit — not as something long-lived control-plane callers should track.

```php
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::make(MyWorkflow::class);

$id = $workflow->id();

$workflow->start();

$runId = $workflow->runId();
```

Use the id when you want the stable public handle for the workflow. Use the run id when you need to inspect one concrete execution.

## Run ID Lifecycle

A run id is an execution-generation identifier. Every v2 workflow generation has its own run id, and a single instance id can span many generations over time:

- A fresh `start()` creates the first run and its first run id.
- `continueAsNew(...)` closes the current run as `completed` / `continued` and starts a new run with a fresh run id under the same instance id. See [Continue As New](/docs/features/continue-as-new).

Because a run id belongs to one generation, callers that want durable behavior across those transitions should resolve by instance id:

- Signals, queries, updates, and operator commands always resolve the newest durable run for an instance. Callers do not need to track run ids across continue-as-new boundaries.
- Drill-down tools that pin a specific generation — for example Waterline's run-detail view, a history export, or a replay debugger — accept an explicit run id so they can inspect one execution in isolation.

The run id surfaces as `runId()` inside workflow code and activities, and as the `workflow_run_id` / `run_id` field on history, projections, and exports. It is always the run id of the generation that owns the event or row. Do not expect a run id recorded on older history events to equal the current run id of the same instance.

## User-defined Instance IDs

When starting a workflow you may optionally specify the id. User-defined ids must be URL-safe strings up to 191 characters using only letters, numbers, `.`, `_`, `-`, and `:`.

```php
$workflow = WorkflowStub::make(MyWorkflow::class, 'order-123');
```

Later when you want to reference the workflow, you can load it with your custom id.

```php
$workflow = WorkflowStub::load('order-123');
```

### Choosing a caller-supplied ID

Choose an ID that identifies one business operation consistently. Retrying the
same start should reuse that ID; generate a new ID for a different operation.
Include a workflow-type prefix and, when needed, an opaque tenant reference to
avoid collisions between applications sharing a store. For example,
`order:tenant-7f2:01J9Z6K6Z2EPV5G0C3R8F4M9AN` combines those prefixes with a
collision-resistant identifier. A tenant prefix is a naming convention, not an
authorization boundary. Configure namespace isolation separately.

The ID does not override the [duplicate-start policy](../configuration/options.md).
The default rejects a duplicate start. `ReturnExistingActive` can return the
existing active workflow without starting another run or replacing its original
arguments. A completed instance's ID remains reserved; changing the workflow
type does not make that ID available again.

### IDs are visible metadata

Instance IDs appear in plaintext in Waterline, URLs, CLI output, logs, history
exports and the database. Avro encoding does not encrypt payloads or protect
these identifiers. Released packages have no supported client-side payload
encryption hook. [Shared #217](https://github.com/durable-workflow/.github/issues/217)
tracks that feature. Use opaque references instead of email addresses,
names, access tokens or other personal or secret data. Keep sensitive business
details in payload fields protected by your application's data-handling policy.
An unguessable ID does not replace authorization.

## Accessing IDs Inside Activities and Workflows

Inside an activity:

```php
use Workflow\V2\Activity;

class MyActivity extends Activity
{
    public function handle(): void
    {
        $id = $this->workflowId();
        $runId = $this->runId();
    }
}
```

Inside a workflow, `$this->workflowId()` returns the instance id and `$this->runId()` returns the selected run id for the currently executing run.

<!-- Source: docs/constraints/overview.md -->

# Overview

Workflows and activities live on opposite sides of the durable boundary and carry different constraints. Workflow code is **replayed**; activity code is **retried**. Those are not the same operation, and the rules flow from that split.

Start with [Idempotent vs. Deterministic Workflows](/docs/constraints/idempotent-vs-deterministic/) for a direct comparison and concrete examples showing why neither property implies the other.

- **Workflow authoring code must be deterministic.** The engine replays history to rebuild a workflow's state whenever it resumes — on another worker, after a restart, after a deployment, or during a long-running execution. Replay re-invokes the workflow body, not activities, so the workflow must produce the same decisions in the same order every time it sees the same history. Wall-clock reads, live cache reads, random numbers, network calls, and other sources of change are prohibited inside the workflow body; use [`Workflow::now()`](../defining-workflows/workflow-api.md), [`sideEffect(...)`](../features/side-effects.md), activities, and similar helpers to cross the durable boundary.

- **Activity code must be idempotent.** Activity attempts are **at-least-once**. Retries, lease expiry, and redelivery can all cause the same logical work to be observed more than once, and that is first-class behavior — not a bug condition. The framework records at most one terminal outcome per attempt at the durable state layer, but an activity body may start executing more than once before the engine sees the winning report. Treat repeat observation as normal and use an idempotency key, a deterministic target resource, or a naturally idempotent operation when the external side effect must not duplicate.

- **Event sourcing persists history, not rerun side effects.** The engine writes each durable step — activity completion, timer fired, signal received, side effect recorded — as a typed history event. Replay reads that history and hands cached results back to the workflow body; it does not re-dispatch activities, timers, or signals. Durable state events for a given identifier are exactly-once at the history layer even when the transport behind them delivered work more than once.

Together, determinism and idempotency let the engine resume workflows that span deployments, worker restarts, and distributed retries without losing place and without duplicating external side effects the application has already made safe to repeat.

See [Execution Guarantees and Idempotency](./execution-guarantees.md) for the
public v2 contract around replay, redelivery, lease expiry, and exactly-once
durable history. Then use [Workflow Constraints](./workflow-constraints.md)
for the specific authoring rules that keep replay deterministic, and
[Activity Constraints](./activity-constraints.md) for the idempotency guidance
that keeps at-least-once activity execution safe.

<!-- Source: docs/constraints/idempotent-vs-deterministic.md -->

# Idempotent vs. Deterministic Workflows: What's the Difference?

**Deterministic behavior makes the same decisions when it is given the same inputs and history. Idempotent behavior has the same intended effect whether an operation runs once or several times.** Determinism makes one execution predictable; idempotency makes repeated executions safe. Neither property implies the other.

| Question | Deterministic | Idempotent |
| --- | --- | --- |
| What stays the same? | The decisions made from the same inputs and history | The intended external effect after one or many calls |
| Why does Durable Workflow need it? | Replay must reconstruct the same workflow path | Retries and redelivery must not duplicate side effects |
| Where does it belong? | Workflow and orchestration code | Activities and the systems they call |
| Does it prohibit IO or randomness? | Yes, inside replayed workflow code unless the result is recorded | No; an activity may use IO, time, or randomness and still make its effect safe to repeat |

## The Properties Are Independent

### Deterministic but not idempotent

Consider an operation that increments a counter:

```text
increment(counter):
    counter.value = counter.value + 1
```

Given the same starting counter value, this operation always makes the same calculation, so it is deterministic. It is not idempotent: calling it twice increments the counter twice, while calling it once increments it once.

### Idempotent but not deterministic

Now consider an activity that sets an order to a known state but uses randomized backoff while contacting the service:

```text
archive(order_id):
    wait(random_backoff())
    PUT /orders/{order_id}/status {"status": "archived"}
```

The timing and number of internal attempts can differ, so this implementation is not deterministic. The operation can still be idempotent: after one successful call or several, the order is archived. The response, latency, and internal path do not have to be identical; the intended external effect does.

## Why Workflow Code Must Be Deterministic

Durable Workflow rebuilds orchestration state by replaying committed history. With the same workflow input and the same history, workflow code must schedule the same activities, timers, and waits in the same order. A wall-clock read, random value, live database query, or network response inside the workflow body can choose a different branch during replay and make the new decisions disagree with history.

Move those operations into activities, or use a workflow helper that records the value in history. Replay then returns the recorded activity or side-effect result instead of performing the external operation again. Determinism does not require every activity invocation to produce a universal, timeless answer; it requires replayed orchestration to make decisions that agree with its recorded history.

See [Workflow Constraints](./workflow-constraints.md) for the authoring rules and [How It Works](../how-it-works.md) for the replay model.

For Rust-specific `HashMap`, clock, randomness and future-ordering rules, see the [durable replacements](../polyglot/rust.md#rust-determinism-rules). Rust SDK 4.5.0 adds [Server-free activity/workflow tests and recorded-history replay](../polyglot/rust.md#offline-testing-and-replay); retain integration tests for its documented execution limits.

## Why Activities and Side Effects Must Be Idempotent

Activities cross the durable boundary into databases, payment APIs, email providers, object storage, and other systems. They can be delivered again after a timeout, retry, worker failure, or expired lease. A worker may even complete the external change and then fail before Durable Workflow receives its report.

Design the activity's intended effect so another attempt is safe:

- send a stable idempotency key to the remote API
- upsert by a durable business or execution identifier
- write to a deterministic object or resource name
- use a naturally idempotent operation such as setting a value or deleting a known resource

An idempotency key does not make activity code deterministic, and it does not prevent another attempt from starting. It lets the external system recognize that repeated attempts represent the same logical operation.

See [Activity Constraints](./activity-constraints.md), [Activity Execution Model](../features/activity-execution-model.md), and [Execution Guarantees and Idempotency](./execution-guarantees.md) for the retry, redelivery, and durable-history contracts.

## One Boundary, Across Every SDK

The same design applies whether the worker is written in PHP, Python, or Rust:

1. Keep orchestration replay-safe: given the same input and history, issue the same durable commands.
2. Put network calls, clocks, randomness, and mutable external state behind an activity boundary.
3. Give each side-effecting activity a stable identity that the target system can deduplicate.

The language syntax changes. The determinism-versus-idempotency boundary does not.

<!-- Source: docs/constraints/workflow-constraints.md -->

# Workflow Constraints

The determinism constraints for workflow classes dictate that a workflow class must not depend on external state or services that may change over time. This means that a workflow class should not perform any operations that rely on the current date and time, the current user, external network resources, or any other source of potentially changing state.

Here are some examples of things you shouldn't do inside of a workflow class:

- Don't use the `Carbon::now()` method to get the current date and time, as this will produce different results each time it is called. Instead, use `Workflow\V2\Workflow::now()` (or the `Workflow\V2\now()` helper), which returns replay-safe workflow time.
- Don't use the `Auth::user()` method to get the current user, as this will produce different results depending on who is currently logged in. Instead, pass the user as an input to the workflow when it is started.
- Don't make network requests to external resources, as these may be slow or unavailable at different times. Instead, pass the necessary data as inputs to the workflow when it is started or use an activity to retrieve the data.
- Don't use random number generators (unless using a side effect) or other sources of randomness, as these will produce different results each time they are called. Instead, pass any necessary randomness as an input to the workflow when it is started.

## Logging from workflow code {#workflow-logging}

A worker reconstructs committed workflow history after activations and cold replacement, so ordinary application logging can repeat earlier messages. Use the replay-aware logger documented for your binding when available. Rust service mode provides the [Rust SDK 4.17 workflow logger](../polyglot/rust.md#workflow-logging); other bindings retain their separately documented contracts.

Logging is a process observation. Keep durable workflow commands independent of replay-state observations and log delivery. Even a replay-aware logger can repeat a message from an uncommitted attempt after a crash or retry; it is not an exactly-once outbox. Keep host handlers bounded and nonblocking so logging does not stall the worker. Put durable business effects behind the appropriate activity or recorded workflow operation instead.

## Boot-Time Guardrails

When workflow classes are registered under `workflows.v2.types.workflows`, the package scans them at boot time for obvious replay-unsafe calls (`Carbon::now()`, `Auth::user()`, `DB::`, `Http::`, `random_int()`, and similar) and surfaces findings according to `workflows.v2.guardrails.boot`:

| Mode | Behavior |
|------|----------|
| `warn` (default) | Logs a warning for each detected finding. Does not block application boot. |
| `silent` | Skips boot-time scanning entirely. |
| `throw` | Throws a `LogicException` on the first finding. Useful for CI pipelines. |

```php
// config/workflows.php
'v2' => [
    'guardrails' => [
        'boot' => env('DW_V2_GUARDRAILS_BOOT', 'warn'),
    ],
],
```

Set `DW_V2_GUARDRAILS_BOOT=throw` in CI to fail builds that introduce new replay-unsafe calls; keep `warn` in production so rollouts are not blocked by a latent finding.

### What the scan checks {#scan-coverage}

The scan is lexical. It reads only the workflow class, its parent classes and its traits, and flags:

- wall-clock reads: `time()`, `date()`, `microtime()`, `hrtime()`, `today()`, static calls such as `Carbon::now()`, and `new DateTime()`, `new DateTimeImmutable()`, `new Carbon()` or `new CarbonImmutable()` with no argument or a relative time such as `'now'` or `'+1 day'`. A fixed date is not flagged.
- randomness: `random_int()`, `random_bytes()`, `rand()`, `mt_rand()`, `uniqid()`, `shuffle()`, `array_rand()`, `str_shuffle()` and `Str::uuid()`-style helpers.
- live state: `DB::`, `Cache::`, `Http::`, `Auth::`, `auth()` and `request()`.

It can't see calls made through helper or service classes, values that depend on process state such as static properties or the order of unordered data, or anything decided at runtime. A clean scan is not proof of determinism. Run [replay verification](../testing.md) against recorded histories to catch the rest.

First-release scope: boot-time scanning is the only blocking workflow-mode guardrail. The runtime does not rerun determinism diagnostics again at workflow-task claim time. That deferral is intentional for 2.0: boot scanning catches locally registered PHP workflows before rollout, and Waterline surfaces definition-fingerprint drift for long-lived runs without turning cross-build claims into a new source of deploy failures.

Pre-fingerprint runs also follow a conservative first-release policy. If a run reaches a new `getVersion()` branch and its `WorkflowStarted` history predates the fingerprint snapshot, the runtime keeps that run on `WorkflowStub::DEFAULT_VERSION` instead of assuming the current definition is safe. See [Versioning](../features/versioning.md).

<!-- Source: docs/constraints/activity-constraints.md -->

# Activity Constraints

Activities are where workflow code crosses the durable boundary into IO and
side effects. They are not replayed like workflow code. They are **executed
at-least-once**, which means retries, lease expiry, and redelivery can all
cause the same logical work to be observed more than once.

That behavior is first-class, not an error path. If your activity creates a
charge, sends an email, writes to another system, or mutates an external
resource, the operation must be safe to repeat.

## What This Means In Practice

- The default idempotency surface for one logical activity execution is
  `activity_execution_id`.
- Each retry attempt also gets its own `activity_attempt_id`.
- A worker can finish external work, lose its lease, and then report late.
  The engine may reject that late report because another worker already won
  the durable race, but the remote side effect may still have happened.
- Activity code is allowed to use IO, wall-clock time, and mutable process
  state. Workflow code is not. Keep that boundary clear.

## Preferred Idempotency Patterns

Many external APIs support passing an `Idempotency-Key`. Use the workflow
runtime's logical activity identity when the remote service supports it.

- Prefer `activity_execution_id` when the remote system should treat retries
  as the same logical request.
- Use `activity_attempt_id` only when the remote system must distinguish
  separate tries of the same logical work.

Other good patterns:

- Write to a deterministic external resource name or natural key.
- Use upserts or dedupe tables keyed by a durable identifier.
- Make the operation naturally idempotent so the second call becomes a no-op.

Many operations are naturally idempotent. If you encode a video twice, you
still end up with the same video. If you delete the same file twice, the
second deletion does nothing.

Some operations are not inherently idempotent, but duplication may still be
the safer failure mode. If you are unsure whether an email actually left the
provider, a duplicate email may be preferable to silently dropping the
notification. Make that trade-off deliberately.

## What Not To Assume

- Do not assume one activity attempt only ever runs on one worker.
- Do not assume a retry means the previous external side effect failed.
- Do not assume late completion implies the activity never ran.
- Do not move side effects into workflow code to avoid retries; that only
  turns an idempotency problem into a determinism bug.

See [Execution Guarantees and Idempotency](./execution-guarantees.md) for the
full contract and [Failures and Recovery](../failures-and-recovery.md) for the
operator-facing recovery model.

<!-- Source: docs/constraints/execution-guarantees.md -->

# Execution Guarantees and Idempotency

Durable Workflow v2 draws a hard line between **workflow replay** and
**activity execution**:

- Workflow code is replayed from committed history and must be deterministic.
- Activity code performs side effects and is **at-least-once**.
- The durable history layer records committed workflow and activity outcomes
  exactly once for a given durable identifier, even when transport delivered
  work more than once.

Those guarantees are what let the engine survive worker restarts, lease
expiry, queue redelivery, and rolling deploys without losing the run's place.

## Replay Is Not Retry

Workflow tasks rebuild state by replaying committed history and then deciding
what to do next. Replay re-invokes the workflow body, but it does **not**
re-run activities, re-send signals, or repeat side effects that were already
recorded on history.

That is why workflow authoring code must stay deterministic. Use workflow-safe
helpers such as [`Workflow::now()`](../defining-workflows/workflow-api.md),
[`sideEffect(...)`](../features/side-effects.md), queries, updates, activity
results, memos, and search attributes when you need to cross the durable
boundary.

## Activity Execution Is At-Least-Once

Activities are the side-effecting part of the system, so the contract is
different:

- An activity attempt can be claimed more than once.
- Lease expiry can cause redelivery to another worker.
- A worker can finish external work, lose its lease, and still report late.
- A retry schedules a new durable attempt for the same logical activity
  execution.

Duplicate observation is therefore **first-class behavior, not a bug
condition**. The application author must make the activity body or the remote
system it calls safe to repeat.

See [Activity Constraints](./activity-constraints.md) for authoring guidance
and [Failures and Recovery](../failures-and-recovery.md) for operator-facing
recovery behavior.

## What Is Exactly Once

Durable Workflow does **not** promise that a worker process sees side-effecting
work only once. It does promise that committed durable facts are authoritative
and do not duplicate for the same durable identifier.

In practice that means:

- A committed workflow decision is persisted once in typed history for the
  durable command or step id it represents.
- A committed terminal outcome for one activity attempt is persisted once for
  that `activity_attempt_id`.
- Replay reads those committed facts back and rebuilds workflow state from
  them instead of re-running external work.

That split is the core mental model:

- **Transport and workers are at-least-once.**
- **Committed durable history is exactly-once per durable identifier.**

## Lease Expiry and Redelivery

Lease expiry is a normal distributed-systems recovery path:

- A claimed task carries a lease owner and expiry time.
- If the lease expires before the worker reports progress or completion, the
  task becomes eligible for redelivery.
- A different worker may then claim the same logical work.

Redelivery does not mean the engine forgot what already committed. It means
the engine is recovering from uncertainty at the worker or transport layer.

When you see duplicate execution symptoms, ask two questions separately:

1. Did the side effect happen more than once?
2. Did the durable state record more than one committed outcome for the same
   durable identifier?

The first question is solved with idempotent activity design. The second is
the engine contract.

## Default Idempotency Surfaces

These identifiers are the stable places to dedupe work:

| Surface | What it identifies | Typical use |
| --- | --- | --- |
| `workflow_instance_id` | One public workflow instance | Duplicate-start handling and business-level run identity |
| `workflow_run_id` | One specific durable run | Pinning one selected run for queries, export, or diagnostics |
| `workflow_command_id` | One mutating external command | Client-side request retry dedupe |
| `activity_execution_id` | One logical activity execution across retries | Default remote idempotency key for external side effects |
| `activity_attempt_id` | One concrete attempt of that activity | Correlation when a remote system must distinguish separate tries |
| `schedule_id` | One schedule definition | Dedupe around schedule ownership and trigger identity |
| message-stream `idempotencyKey` | One retried logical message send | Prevent duplicate message ingestion when a sender retries |

When in doubt, use `activity_execution_id` as the default idempotency key for
an external operation. Reach for `activity_attempt_id` only when the external
target truly needs every retry attempt to be distinguishable.

```php
use Workflow\V2\Activity;

final class ChargeCard extends Activity
{
    public function handle(array $payload): string
    {
        return app(PaymentGateway::class)->charge(
            $payload,
            idempotencyKey: $this->activityId(),
            attemptCorrelation: $this->attemptId(),
        );
    }
}
```

### Service-mode activity identity {#service-activity-identity}

For Rust service-mode activities, SDK **4.9.0 or newer** exposes the original optional `activity_execution_id()`, `workflow_id()` (instance) and `run_id()` through `ActivityContext`. Use the logical activity ID across retries; missing claim fields stay `None`, and a per-attempt task/attempt ID is not a replacement. The [Rust handler recipe](../polyglot/rust.md#activity-identity) derives a downstream key and explains older-Server, local-activity and Server-free-test limits. A key still needs an idempotent downstream API or durable effect ledger; these accessors do not provide resumable checkpoints.

Python service-mode activities expose the read-only optional
`activity.context().info.activity_execution_id`, with `workflow_id` and `run_id`
for the owning workflow instance and run. Use the logical execution ID across
retries. Missing identity stays `None`; never substitute a task, attempt or
session ID. The [Python identity guide](../polyglot/python.md#activity-identity)
links the SDK-owned service example and defines legacy, session, prepared-local,
inline-local and Server-free behavior. Its named published-package/native
Server receipt proves ordinary retry identity only. The ID is public metadata,
not claim authority, and does not make an external effect exactly once. Keep
downstream idempotency, distinct keys for different effects and outcome
reconciliation.

PHP service-mode activities expose the read-only nullable
`ActivityContext::activityExecutionId` from the received claim, with `workflowId`
and `runId` for the owning workflow and run. Keep the logical execution ID
across retries. Missing identity stays `null`; a task, attempt or session ID is
not a fallback. The [PHP identity guide](../polyglot/php.md#activity-identity)
links the SDK-owned service-key example and defines absent, session,
historical prepared-local, inline-local and Server-free cases. These accessors
are public metadata, not claim authority. External effects still need
downstream idempotency, distinct keys for different effects and outcome
reconciliation.

## What Developers Must Make Idempotent

You do **not** need to make workflow replay itself idempotent. The framework
handles replay by rebuilding state from committed history.

You **do** need to make external effects safe to repeat, including:

- payment or billing calls
- emails, texts, and webhooks
- writes to another database or service
- file creation or upload
- any command that creates or mutates state outside the workflow history

Common approaches:

- Pass the remote API an idempotency key.
- Write to a deterministic target resource such as a known object key.
- Use an upsert or transaction keyed by a durable identifier.
- Make the action naturally repeatable so the second call is a no-op.

## Operator Guidance

When diagnosing a run in Waterline, the CLI, or server logs:

- Treat duplicate activity observation after lease expiry as expected until the
  durable attempt outcome shows otherwise.
- Treat late completion or failure reports as a race the engine resolves, not
  as proof that the remote side effect never happened.
- Treat workflow-task replay as recovery, not as workflow-level retry.
- Treat missing compatible workers, stuck leases, or repeated repair as
  operational signals that need investigation, not as a reason to assume the
  workflow body should re-run side effects.

The most important operator distinction is between **transport uncertainty**
and **durable outcome**. Durable Workflow surfaces both so you can tell the
difference.

## Related Guides

- [Overview](./overview.md) introduces the workflow/activity split.
- [Workflow Constraints](./workflow-constraints.md) covers deterministic
  authoring rules.
- [Activity Constraints](./activity-constraints.md) covers side-effect safety
  and idempotency techniques.
- [Failures and Recovery](../failures-and-recovery.md) covers retries,
  timeout enforcement, and repair.
- [Activity Execution Model](../features/activity-execution-model.md) explains
  how queued activities, local activities, worker sessions, and sticky
  execution fit together.
- [Local Activities](../features/local-activities.md) explains same-process
  activity attempts, workflow-task heartbeats, retries, and cold replay.
- [Sticky Execution](../features/sticky-execution.md) explains sticky replay
  caches and why cold replay remains the correctness fallback.

<!-- Source: docs/constraints/constraints-summary.md -->

# Constraints Summary

| Constraint | Workflow code | Activity code |
| --- | --- | --- |
| IO | Not allowed | Allowed |
| Mutable global variables | Not allowed | Allowed |
| Non-deterministic functions | Not allowed | Allowed |
| `Carbon::now()` | Not allowed | Allowed |
| `sleep()` | Not allowed | Allowed |
| External side effects | Move them into activities | Allowed only when safe to repeat |

Workflow code must be deterministic because the engine rebuilds a workflow's state by replaying its history of durable steps — it re-invokes the workflow body, not activities. If the body produces different decisions on replay, the engine cannot reconstruct the workflow's state and continue execution correctly. The workflow body does not need to be idempotent; determinism is the replay requirement.

Activities must be idempotent because activity execution is at-least-once. Retries, lease expiry, and redelivery can cause the same attempt to be observed more than once, and that is first-class behavior rather than an error. Making the activity body or its external target safe to repeat — with an idempotency key, a deterministic target resource, or a naturally idempotent operation — is what keeps duplicate execution from producing duplicate side effects.

See [Idempotent vs. Deterministic Workflows](/docs/constraints/idempotent-vs-deterministic/)
for a direct comparison and examples that separate the two properties.

See [Execution Guarantees and Idempotency](./execution-guarantees.md) for the
public v2 contract behind those terms.

<!-- Source: docs/constraints/structural-limits.md -->

# Structural Limits

Structural limits cap the resource consumption of a single workflow run. When an operation would exceed a configured limit, the engine records a typed failure with a machine-readable `structural_limit` failure category and the specific limit kind, then fails the run. This protects the system from unbounded fan-out, oversized payloads, and metadata bloat.

## Limit kinds

| Limit kind | Default | What it caps |
|---|---|---|
| `pending_activity_count` | 2,000 | Non-terminal activity executions open simultaneously |
| `pending_child_count` | 1,000 | Non-terminal child workflows open simultaneously |
| `pending_timer_count` | 2,000 | Pending timers open simultaneously |
| `pending_signal_count` | 5,000 | Unprocessed signals pending simultaneously |
| `pending_update_count` | 500 | Unresolved updates pending simultaneously |
| `command_batch_size` | 1,000 | Items in a single parallel fan-out (`all()`) |
| `payload_size_bytes` | 2 MiB | Serialized size of a single argument payload |
| `memo_size_bytes` | 64 KiB | Serialized size of non-indexed memo metadata |
| `search_attribute_size_bytes` | 40 KiB | Serialized size of indexed search-attribute metadata |
| `history_transaction_size` | 5,000 | History events produced by a single workflow task execution |

All limits are enforced at the point of scheduling, recording, or command intake. A value of `0` disables the check for that limit kind.

## Soft-limit warnings

Before a hard limit terminates a run or rejects a command, the engine can warn you that a resource is approaching its ceiling. When a count-based resource (pending activities, children, timers, signals, updates, command batch size, or history transaction events) crosses a configurable percentage of the hard limit, the engine logs a structured warning.

The default warning threshold is **80%**. For example, with the default `pending_activity_count` limit of 2,000, a warning is logged when a run reaches 1,600 pending activities. The run continues executing normally — the warning gives operators time to react (scale workers, trigger continue-as-new, raise the limit) before the hard guard fails the run.

Configure the threshold via `workflows.v2.structural_limits.warning_threshold_percent`:

```env
DW_V2_LIMIT_WARNING_THRESHOLD_PERCENT=80
```

Set to `0` to disable soft-limit warnings entirely.

Warning log entries include structured context:

```
[Durable Workflow] Run 42 approaching structural limit [pending_activity_count]: 1620 / 2000 (81% utilization, warning at 80%).
```

The structured log context includes `workflow_run_id`, `workflow_type`, `limit_kind`, `current`, `limit`, and `utilization_percent` for integration with log aggregation and alerting tools.

## Configuration

Override any limit through `workflows.v2.structural_limits` in your config or via environment variables:

```php
// config/workflows.php
'v2' => [
    'structural_limits' => [
        'pending_activity_count' => (int) env('DW_V2_LIMIT_PENDING_ACTIVITIES', 2000),
        'pending_child_count' => (int) env('DW_V2_LIMIT_PENDING_CHILDREN', 1000),
        'pending_timer_count' => (int) env('DW_V2_LIMIT_PENDING_TIMERS', 2000),
        'pending_signal_count' => (int) env('DW_V2_LIMIT_PENDING_SIGNALS', 5000),
        'pending_update_count' => (int) env('DW_V2_LIMIT_PENDING_UPDATES', 500),
        'command_batch_size' => (int) env('DW_V2_LIMIT_COMMAND_BATCH_SIZE', 1000),
        'payload_size_bytes' => (int) env('DW_V2_LIMIT_PAYLOAD_SIZE_BYTES', 2097152),
        'memo_size_bytes' => (int) env('DW_V2_LIMIT_MEMO_SIZE_BYTES', 65536),
        'search_attribute_size_bytes' => (int) env('DW_V2_LIMIT_SEARCH_ATTRIBUTE_SIZE_BYTES', 40960),
        'history_transaction_size' => (int) env('DW_V2_LIMIT_HISTORY_TRANSACTION_SIZE', 5000),
        'warning_threshold_percent' => (int) env('DW_V2_LIMIT_WARNING_THRESHOLD_PERCENT', 80),
    ],
],
```

## Enforcement points

### Pending count limits (executor-side)

Before the executor schedules an activity, child workflow, or timer, it counts the currently non-terminal items of that type on the run. If the count is already at or above the configured limit, the run fails immediately with a `StructuralLimitExceededException`.

This protects against patterns like unbounded parallel fan-out loops that accumulate thousands of pending operations:

```php
// This will fail if $items exceeds the pending_activity_count limit
$calls = [];
foreach ($items as $item) {
    $calls[] = fn () => activity(ProcessItemActivity::class, $item);
}
return all($calls); // Also checked against command_batch_size
```

To handle large batches within the limits, process items in bounded chunks:

```php
foreach (array_chunk($items, 500) as $chunk) {
    $calls = [];
    foreach ($chunk as $item) {
        $calls[] = fn () => activity(ProcessItemActivity::class, $item);
    }
    all($calls);
}
```

### Pending count limits (intake-side)

When a signal or update command arrives via the control plane, webhook, or `WorkflowStub`, the engine checks the count of unprocessed signals (`received` status) or unresolved updates (`accepted` status) on the target run before accepting the command.

If the count is at or above the configured limit, the command is **rejected** with reason `structural_limit_exceeded`. The rejection response includes machine-readable metadata (`structural_limit_kind`, `structural_limit_value`, `structural_limit_configured`) so callers can identify the root cause.

Unlike executor-side limits (which fail the run), intake-side limits reject the individual command without terminating the workflow. The run remains active, and the caller can retry once pending items have been processed.

For a Server update with `request_id`, the keyed rejection is retained. Reusing
that key returns the original rejection even after capacity or settings change.
After confirming the rejection and restoring capacity, use a new key for a new
admission attempt. An uncertain response is different: recover its original
receipt before deciding whether another admission is safe.

```php
// If 5,000 signals are already pending, this will be rejected:
$result = $workflow->attemptSignal('process-item', $data);
if ($result->rejected()) {
    // $result->rejectionReason() === 'structural_limit_exceeded'
    // back off and retry later
}
```

### Command batch size

The `all()` function checks the total number of leaf operations in a single fan-out group against `command_batch_size`. This is checked before any individual activities or children are scheduled, so the run fails cleanly rather than partially scheduling a batch.

### Payload size

When the executor schedules an activity or child workflow, it serializes the argument payload and checks the byte length against `payload_size_bytes`. If the serialized payload exceeds the limit, the run fails before any database rows are created for the operation.

This applies to:

- **Activity arguments** — checked at the point `scheduleActivity` serializes the `ActivityCall` arguments.
- **Child workflow arguments** — checked at the point `scheduleChildWorkflow` serializes the child's start arguments, before creating the child instance or run rows.

```php
// A 3 MiB payload will fail with the default 2 MiB limit
activity(ProcessDocumentActivity::class, $threeMegabyteBlob);
```

To work within the limit, either enable
[External Payload Storage](../features/external-payload-storage.md) on the
namespace so the runtime transparently offloads over-threshold payloads to a
configured object store, or store the bytes yourself and pass an
application-level reference:

```php
$ref = Storage::put('docs/incoming.pdf', $blob);
activity(ProcessDocumentActivity::class, $ref);
```

External payload storage preserves replay integrity by recording a hashed
`durable-workflow.v2.external-payload-reference.v1` envelope in history, so
the reference envelope becomes the payload the limit sees — not the bytes.

### Memo size

Memo has fixed caps: each value up to 10,240 bytes when encoded (`memo_value_bytes`), at most 100 keys (`memo_keys`) and 65,536 bytes in total (`memo_total_bytes`). `memo_size_bytes` checks the encoded size of the merged memo map and can tighten the total but not raise it: a larger setting acts as 65,536. Setting it to `0` turns this check off; the fixed caps stay.

Start, `upsertMemo()` and service-worker memo upserts all check the memo that the write would leave on the run before anything is written. Going over a limit raises `StructuralLimitExceededException` with a `limitName`. An upsert from workflow code fails the run, and its `WorkflowFailed` event records `structural_limit_name`.

These rules apply from Workflow 2.25.0. Earlier releases defaulted `memo_size_bytes` to 256 KiB, which the fixed total made unreachable, and raised `InvalidArgumentException` for the fixed caps.

### History transaction size

Each workflow task execution (a single "turn" of replay and forward progress) may produce new history events — activity scheduling, timer creation, side-effect recording, search-attribute upserts, and so on. The `history_transaction_size` limit caps the total number of new events a single task can produce.

This catches runaway loops that create unbounded events in a single task without yielding control:

```php
// If a workflow schedules thousands of operations in one task,
// the history transaction limit prevents the task from growing
// without bound. Process large batches in bounded chunks instead.
foreach (array_chunk($items, 500) as $chunk) {
    $calls = [];
    foreach ($chunk as $item) {
        $calls[] = fn () => activity(ProcessItemActivity::class, $item);
    }
    all($calls);  // Each chunk is a separate task execution
}
```

The check runs at the top of each iteration of the executor's main loop. Events created during replay (reading existing history) do not count toward the limit — only new events written during the current task contribute.

### Search attribute size

Search attributes have fixed caps: strings up to 2,048 characters (`search_attribute_string_characters`), keywords and keyword-list entries up to 255 characters (`search_attribute_keyword_characters`), at most 100 attributes (`search_attribute_count`) and 65,536 bytes in total (`search_attribute_total_bytes`). `search_attribute_size_bytes` checks the JSON size of the merged attribute map and can tighten the total but not raise it. Setting it to `0` turns this check off; the fixed caps stay.

Start, `upsertSearchAttributes()` and service-worker upserts apply the same limits before anything is written, and fail the same way as memo limits.

## Failure taxonomy

When a structural limit is exceeded, the engine records:

- A `WorkflowFailure` row with `failure_category = structural_limit`
- A `WorkflowFailed` history event with:
  - `failure_category = structural_limit`
  - `structural_limit_kind` — the specific limit that was exceeded (e.g. `pending_activity_count`, `command_batch_size`)
  - `structural_limit_value` — the current count or size that triggered the limit
  - `structural_limit_configured` — the configured ceiling

This metadata is machine-readable, so operators, Waterline, and external tooling can identify the root cause without parsing free-text messages.

## Health check

The current structural limits configuration is included in the v2 health check snapshot under `structural_limits`, making the active ceilings visible to operators:

```json
{
  "structural_limits": {
    "pending_activity_count": 2000,
    "pending_child_count": 1000,
    "pending_timer_count": 2000,
    "pending_signal_count": 5000,
    "pending_update_count": 500,
    "command_batch_size": 1000,
    "payload_size_bytes": 2097152,
    "memo_size_bytes": 65536,
    "search_attribute_size_bytes": 40960,
    "history_transaction_size": 5000,
    "warning_threshold_percent": 80
  }
}
```

## Backend-dependent limits

The backend capabilities snapshot publishes the full structural-limit contract adjusted for the current infrastructure. Most limits are backend-independent configuration values, but certain backends impose additional constraints:

- **SQS queue**: [Standard SQS message delays](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-timers.html) are capped at 900 seconds. Longer timer, activity-retry and workflow-task delays are chunked by the transport layer. The capability snapshot retains `max_single_timer_delay_seconds: 900` for compatibility.
- **SQLite database** — SQLite serializes writes, so the snapshot notes `concurrent_write_safety: limited`. High pending-count limits may cause lock contention under concurrent worker load.

An early SQS delivery schedules another capped message without leasing the
durable task, spending an execution attempt, writing execution history or running
user code. The original `available_at` stays unchanged. Once due, the task follows
normal execution and duplicate-delivery fencing. Overall workflow and activity
deadlines keep running during the wait, so choose a backoff that fits those
budgets. Queue drivers without a delay cap retain the exact requested delay.

The full contract is available in the `structural_limits` section of the backend capabilities response:

```json
{
  "structural_limits": {
    "configured": { "pending_activity_count": 2000, "..." : "..." },
    "backend_adjustments": { "max_single_timer_delay_seconds": 900 },
    "effective": { "pending_activity_count": 2000, "max_single_timer_delay_seconds": 900, "..." : "..." },
    "issues": [
      {
        "component": "structural_limits",
        "severity": "info",
        "code": "queue_max_delay_constraint",
        "message": "The [sqs] queue driver limits delayed dispatch to 900 seconds; longer task delays are chunked by the transport layer."
      }
    ]
  }
}
```

## Waterline

Waterline surfaces structural-limit failures in the exceptions table with the `structural_limit` failure category. The timeline failure details include the limit kind, current value, and configured ceiling.

## Service Mode limits {#server-request-boundary-limits}

Server and Cloud HTTP request limits are separate from these embedded Laravel controls. See the [Server request reference](../polyglot/server-config-reference.md#server-request-boundary-limits).

For native worker-command names and IDs, see the [field bounds](../polyglot/server-config-reference.md#native-worker-command-name-limits).

### Workflow-task completion size {#workflow-task-completion-size}

Service Mode completion caps and SDK refusal behavior are covered in the [Server reference](../polyglot/server-config-reference.md#workflow-task-completion-size).

<!-- Source: docs/features/signals.md -->

# Signals

Signals allow you to trigger events in a workflow from outside the workflow. This can be useful for reacting to external events, enabling *human-in-the-loop* interventions, or for signaling the completion of an external task. For repeated ordered inputs with durable cursor advancement, use [Message Streams](./message-streams.md).

## Namespace boundary {#namespace-boundary}

An ordinary signal targets a workflow in the client's namespace. A child
handle also retains its original namespace. Use a
[service operation](./service-operations.md) for an operation published in
another namespace; changing a workflow id or queue is not cross-namespace
authority.

## Acceptance and closure

`signal_received` confirms durable acceptance. It does not confirm that workflow
code processed the signal. Inspect the signal's retained status and history when
the business operation needs confirmation.

With Core 2.7.0 or a compatible Server, eligible signals accepted during a remote
workflow task get a delivery turn before successful closure. Received signals
addressed to the instance and declared by its command contract carry across [continue-as-new](./continue-as-new.md)
with their original identity, arguments and ordering. A signal addressed to a
specific run never moves to another run.

Runs without a durable signal declaration do not gain one from a worker's
later registration. If the runtime cannot establish delivery, the accepted
signal closes with the explicit rejected disposition below.

If a run closes without applying a signal, its status becomes `rejected` with
`run_closed_before_signal_applied`, a closed timestamp and a `SignalRejected`
history event. This includes an unused buffered signal at ordinary completion
and signals that cannot be applied before failure, cancellation or timeout.
The earlier acceptance receipt remains valid. Use an [update](./updates.md) when
the caller needs a retained processing result, rather than an acceptance alone.

Before upgrading a writer to Core 2.7.0, drain and upgrade every embedded or
direct-database reader, including Waterline when it reads the same database.
Older readers cannot decode `SignalRejected`; do not leave them running against
the new writer or roll back a reader after that event has been stored.

## Named Signal Waits

A workflow calls `await('signal-name')` directly. The next accepted signal command with that name resumes the run and returns a deterministic value to the suspended workflow.

```php
use Workflow\V2\Attributes\Signal;
use Workflow\V2\Attributes\Type;
use Workflow\V2\Workflow;
use function Workflow\V2\await;

#[Type('order-approval')]
#[Signal('approved-by', [
    ['name' => 'approvedBy', 'type' => 'string'],
])]
final class OrderApprovalWorkflow extends Workflow
{
    public function handle(): array
    {
        $approvedBy = await('approved-by');

        return [
            'approved_by' => $approvedBy,
            'workflow_id' => $this->workflowId(),
            'run_id' => $this->runId(),
        ];
    }
}
```

Trigger the signal from PHP by addressing the public instance id:

```php
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::load('order-123');

$result = $workflow->attemptSignalWithArguments('approved-by', [
    'approvedBy' => 'Taylor',
]);

$result->accepted();    // true
$result->outcome();     // "signal_received"
$result->commandId();   // Durable signal-command id
$result->instanceId();  // "order-123"
```

Signal behavior:

- Declare each external signal name up front with a repeatable `#[Signal('signal-name')]` class attribute. Optionally include an ordered parameter contract.
- Signal commands target the public workflow instance id — not a run id — so continue-as-new chains keep the same public signal route.
- With a parameter contract, intake rejects invalid payloads as `rejected_invalid_arguments` with machine-readable `validation_errors`.
- Without a contract, `await('name')` returns `true` when no arguments were sent, the single argument when one was sent, or the full argument array when several were sent.
- Unknown signal names reject as `rejected_unknown_signal`; signals against a closed or unstarted instance reject as `rejected_not_active` or `rejected_not_started`.

## Evolve a signal payload {#evolve-a-signal-payload}

Use one declared map argument for an application payload that needs compatible
additions. This example keeps `schema_version: 1` for an optional, informational
`note`. An old sender omits it; a newer reader uses `null`. An older reader can
ignore it without changing the approval decision. Do not use an ignored field
for a change that must affect authorization or business behavior.

```php
use Workflow\V2\Attributes\Signal;
use Workflow\V2\Attributes\Type;
use Workflow\V2\Workflow;
use function Workflow\V2\await;

#[Type('map-approval')]
#[Signal('reviewed', [
    ['name' => 'payload', 'type' => 'array', 'allows_null' => false],
])]
final class MapApprovalWorkflow extends Workflow
{
    public function handle(): array
    {
        $payload = await('reviewed');
        if (! is_array($payload) || array_is_list($payload)
            || ($payload['schema_version'] ?? null) !== 1) {
            return ['status' => 'rejected_payload', 'reason' => 'unsupported_shape_or_version'];
        }
        if (! in_array($payload['decision'] ?? null, ['approved', 'rejected'], true)) {
            return ['status' => 'rejected_payload', 'reason' => 'invalid_decision'];
        }
        $note = $payload['note'] ?? null;
        if ($note !== null && ! is_string($note)) {
            return ['status' => 'rejected_payload', 'reason' => 'invalid_note'];
        }

        return ['status' => 'processed', 'decision' => $payload['decision'], 'note' => $note];
    }
}
```

Both senders use the same declared `payload` parameter:

```php
use Workflow\V2\WorkflowStub;

$oldPayload = ['schema_version' => 1, 'decision' => 'approved'];
$newPayload = [...$oldPayload, 'note' => 'review λ'];

$receipt = WorkflowStub::load('review-123')->attemptSignalWithArguments(
    'reviewed',
    ['payload' => $newPayload], // An old sender passes $oldPayload here.
);
```

The declaration checks the outer argument. A scalar in place of `payload`
rejects at admission. Its `array` type does not validate nested fields or
distinguish a PHP list from a string-keyed map. The workflow checks those
application rules before doing business work. An unsupported version or invalid
`note` can therefore be durably admitted, then produce `rejected_payload` as the
workflow result. That result is separate from the `signal_received` receipt.

Keep required fields and their meanings stable. Define whether missing and
explicit `null` mean the same thing; they do for `note` here. Preserve old
decoders for recorded values. A breaking shape or meaning needs a new
application version and an explicit compatibility plan, not just another
optional field.

A run keeps the signal declarations and defaults recorded when it started.
Changing a current handler signature or default does not rewrite that contract.
Adding another top-level signal argument is different from adding a key inside
the existing map. Do not assume that an old run accepts it or adopts the new
default. New runs can record a new declaration while old runs retain theirs.

This application format is separate from the fixed
[Avro Value wire schema](/docs/polyglot/avro-value-protocol). Service-mode PHP,
Python and Rust clients encode supported map/list/scalar values through their
SDKs. A Python dataclass, PHP domain object or Rust struct needs the relevant
explicit value adapter; an application version field is not a schema registry.
See the [SDK references](/docs/sdk-neutrality) for the service-mode APIs.

Payload compatibility also does not authorize a changed workflow definition to
claim an existing run. Keep the required compatible worker/build available.
Server 2.9.7's source-fingerprint restriction is tracked separately in
[Server #518](https://github.com/durable-workflow/server/issues/518). This recipe
does not bypass it or promise that a current handler replaces the old run's code.
Use [workflow versioning](./versioning.md) for replay-safe code changes within
the runtime's supported routing contract.

The same recorded-value boundary applies to activity arguments and results:
retries and replay can encounter older shapes. Keep their adapters compatible
with those values rather than rewriting history or relying on current model
defaults.

## After an uncertain response {#after-an-uncertain-response}

A timeout or lost response can happen after a signal has been durably admitted.
A later closed-run or not-found response describes that later request; it does
not establish whether the original signal was applied. A status check before
sending also leaves a gap in which the workflow can close or continue as new.

Keep the original namespace, workflow identity, signal name, arguments and
application operation key when investigating an uncertain delivery. Repeating
a signal must follow the target's actual duplicate-handling contract. Do not
start a replacement workflow merely because the retry was refused.

See [recovering an uncertain command](./updates.md#recover-after-an-uncertain-response)
for retained update inspection and application-owned reconciliation.

## Sender identity and authorization

`approvedBy` is application data. The string `Taylor` in the example is not
proof of an authenticated sender. A `#[Signal]` contract validates the signal
name and arguments; your Laravel application must authenticate the caller and
authorize the operation before admitting the signal.

For example, an approval endpoint can use the authenticated user and an order
policy. This application defines an `Order` model with stored
`workflow_instance_id` and `workflow_namespace` fields, and registers an
`approve` policy that checks the user's access to that order. The endpoint
does not take the actor or workflow routing values from the request body:

```php
use App\Models\Order;
use Illuminate\Http\Request;
use Illuminate\Support\Facades\Gate;
use Illuminate\Support\Facades\Route;
use Workflow\V2\CommandContext;
use Workflow\V2\WorkflowStub;

Route::post('/orders/{order}/approve', function (Request $request, Order $order) {
    $user = $request->user();
    Gate::forUser($user)->authorize('approve', $order);

    $actorId = (string) $user->getAuthIdentifier();
    $context = CommandContext::phpApi()->withPrincipal('user', $actorId);

    $result = WorkflowStub::load(
        $order->workflow_instance_id,
        $order->workflow_namespace,
    )->withCommandContext($context)->attemptSignalWithArguments('approved-by', [
        'approvedBy' => $actorId,
    ]);

    return response()->json([
        'accepted' => $result->accepted(),
        'outcome' => $result->outcome(),
        'command_id' => $result->commandId(),
    ], $result->accepted() ? 202 : 409);
})->middleware('auth');
```

The policy check happens before loading or signaling the workflow. Every
approval entry point must enforce the same application policy, including any
webhook or administrative endpoint that can send this signal. Inside the
workflow, `await('approved-by')` receives the recorded argument; it does not
return an authenticated-sender object or perform another permission check.

The command's audit identity is separate from that argument.
`withCommandContext()` records the context on the `WorkflowCommand` identified
by `commandId()`. Its `context.principal` and `principalType()`, `principalId()`
and `principalLabel()` accessors support host-side audit inspection; linked
history and operator views can display the recorded command attribution.
Missing attribution does not establish a sender identity.

`CommandContext::withPrincipal()` records what trusted host code supplies. It
does not authenticate a user or authorize a signal. Keep that context outside
caller-controlled input, and do not use a live HTTP session or a database
lookup of current permissions inside deterministic workflow replay.

Service-mode SDK calls use the Server's HTTP authorization boundary instead
of this embedded route. See
[Signal authorization and attribution](/docs/polyglot/namespace-auth-workers#signal-authorization-and-attribution)
for credentials, trusted gateways and audit identity.

**Important:** The `await()` function should only be used in a workflow, not an activity.

For condition waits — waiting until a predicate over durable state becomes true — see [Condition Waits](./condition-waits.md). For a timeout-backed `await`, see [Signal + Timer](./signal+timer.md). For repeated inbox/outbox flows, see [Message Streams](./message-streams.md).

## Run this pattern

The webhook-started workflow in the
[Sample App](/docs/sample-app) is the runnable reference for the
named-signal-wait pattern on this page:

```bash
php artisan app:webhook
```

`App\Workflows\Webhooks\WebhookWorkflow` starts from an HTTP webhook
ingress and parks on `await('ready')` until the matching signal lands.
Open Waterline while it is parked and you will see the
`SignalReceived` event materialize the moment the signal
is accepted.

<!-- Source: docs/features/queries.md -->

# Queries

Queries allow you to retrieve information about the current state of a workflow without affecting its execution. This is useful for monitoring and debugging purposes.

## Replay-Safe Query Methods

Queries replay committed history for the current selected run and then invoke the annotated method on the hydrated workflow object. They do not apply accepted-but-not-yet-applied signal or update commands implicitly.

```php
use Workflow\QueryMethod;
use Workflow\V2\Workflow;
use function Workflow\V2\await;

final class ApprovalWorkflow extends Workflow
{
    private string $stage = 'booting';

    public function handle(): void
    {
        $this->stage = 'waiting-for-approval';

        await('approved-by');

        $this->stage = 'approved';
    }

    #[QueryMethod('current-stage')]
    public function currentStage(): string
    {
        return $this->stage;
    }

    #[QueryMethod('starts-with')]
    public function startsWith(string $prefix): bool
    {
        return str_starts_with($this->stage, $prefix);
    }
}
```

Address the current run through the public instance id:

```php
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::load('order-123');

$workflow->currentStage(); // "waiting-for-approval"
$workflow->query('starts-with', 'waiting'); // true
$workflow->queryWithArguments('starts-with', ['prefix' => 'waiting']); // true
```

When you want to pin one historical or selected run explicitly, query through `loadRun($runId)` instead:

```php
$selectedRun = WorkflowStub::loadRun($runId);

$selectedRun->currentStage();
```

Query behavior:

- Queries are replay-safe: they observe committed history only and do not mutate workflow state.
- Arguments are forwarded to the annotated method. Declare a stable public name with `#[QueryMethod('public-name')]` so the callable survives PHP method renames.
- `load($instanceId)` queries the newest durable run for that instance; `loadRun($runId)` targets one specific run (useful for pre–continue-as-new queries).
- Accepted-but-not-yet-applied signals and updates are visible in command history but do not count as applied state until the worker records `SignalApplied` / `UpdateApplied`.

To define a query method on a workflow, use the `QueryMethod` annotation. The optional string argument lets you freeze a public durable query name that survives PHP method renames:

```php
use Workflow\QueryMethod;
use Workflow\V2\Workflow;

final class MyWorkflow extends Workflow
{
    private bool $ready = false;

    #[QueryMethod('is-ready')]
    public function getReady(): bool
    {
        return $this->ready;
    }
}
```

To query a workflow, call the method on the workflow instance. The query method will return the data from the workflow.

```php
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::load('order-123');

$ready = $workflow->getReady();
$sameReady = $workflow->query('is-ready');
```

Use `queryWithArguments()` when your caller already has one positional list or a named parameter map:

```php
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::load('order-123');

$workflow->queryWithArguments('starts-with', [
    'prefix' => 'wait',
]);
```

Waterline uses the selected-run query contract surface. The dashboard exposes `declared_query_targets[*]` alongside signals and updates, but only shows query execution when `can_query = true`. The selected-run or current-run query operator posts JSON `arguments` to `/waterline/api/instances/{instanceId}/queries/{query}` or `/waterline/api/instances/{instanceId}/runs/{runId}/queries/{query}`. Query execution still requires a loadable workflow definition because the selected run must be replayed before the query method can run; when durable query targets exist but that definition is unavailable, selected-run detail reports `can_query = false` with `query_blocked_reason = workflow_definition_unavailable`, and query POSTs return HTTP `409 Conflict` with `blocked_reason = workflow_definition_unavailable`. If the run only has an incomplete snapshot and the current build can no longer finish backfilling it, detail reports `declared_contract_source = unavailable`; surviving query targets remain visible as diagnostic metadata, but named query arguments still reject with `422` until a compatible build persists the missing contract.

The public webhook bridge exposes that same replay-safe query surface outside Waterline:

```text
POST /webhooks/instances/{instanceId}/queries/{query}
POST /webhooks/instances/{instanceId}/runs/{runId}/queries/{query}
```

Those webhook routes accept the same JSON `arguments` field as Waterline, return a typed JSON `result` on success, and use the same error shape for invalid arguments (`422` with `validation_errors`) and replay blocks (`409` with `blocked_reason`). When the current workflow definition is still loadable, callers may address the query by either its durable `#[QueryMethod('public-name')]` target or the underlying PHP method name; successful HTTP responses normalize `query_name` back to the durable public target so external callers do not harden on method-renaming details.

**Important:** Querying a workflow does not advance its execution, unlike signals.

<!-- Source: docs/features/updates.md -->

# Updates

Updates allow you to retrieve information about the current state of a workflow and mutate the workflow state at the same time. They are essentially both a query and a signal combined into one.

## Explicit Update Commands

Each accepted update:

- Is a request/response call against a running workflow. `attemptUpdate*()` waits for the handler to finish; `submitUpdate*()` returns as soon as the command is durably accepted and exposes `inspectUpdate($updateId)` to poll later.
- Uses the declared durable update name in command history and webhook routing — `#[UpdateMethod('mark-approved')]` keeps the public callable name stable across PHP method renames.
- Rejects against historical or already-closed runs rather than silently mutating the current run.
- Rejects undeclared method names as `rejected_unknown_update` and contract-invalid arguments as `rejected_invalid_arguments` (with machine-readable `validation_errors`) before the handler runs.

Like queries, updates replay committed history first. Unlike queries, updates are allowed to mutate replay-safe workflow state and return a value.

To define an update method on a workflow, use the `UpdateMethod` annotation. The optional string argument lets you freeze a public durable name that survives PHP method renames:

```php
use Workflow\UpdateMethod;
use Workflow\V2\Workflow;

final class MyWorkflow extends Workflow
{
    private bool $ready = false;

    #[UpdateMethod('mark-ready')]
    public function updateReady(bool $ready): bool
    {
        $this->ready = $ready;

        return $this->ready;
    }
}
```

Call the update method directly when you want the raw return value:

```php
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::load('order-123');

$ready = $workflow->updateReady(true);
```

That direct PHP call still uses the method name. The durable command target remains `mark-ready`.

Use `attemptUpdate()` when you want the durable command outcome as well. Pass the durable update name there:

```php
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::load('order-123');

$result = $workflow->attemptUpdate('mark-ready', true);

$result->accepted();        // true
$result->completed();       // true when the update body ran successfully
$result->updateStatus();    // "accepted", "completed", "failed", or "rejected"
$result->updateId();        // Durable update lifecycle id
$result->result();          // Raw update return value when completed
$result->failureMessage();  // Failure message when the update body threw
```

`attemptUpdate()` records the accepted update first, then waits for the workflow worker to apply it and close the update lifecycle before returning. The wait is time-bounded by `workflows.v2.update_wait.completion_timeout_seconds`; if the worker has not closed the update when the budget expires, `attemptUpdate()` returns the accepted lifecycle with `waitTimedOut() === true` and `updateStatus() === 'accepted'`.

Use `withUpdateWaitTimeout()` when one call needs a different completion budget:

```php
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::load('order-123')
    ->withUpdateWaitTimeout(5);

$result = $workflow->attemptUpdate('mark-ready', true);
```

Use `attemptUpdateWithArguments()` when the caller already has a positional list or named parameter map:

```php
$result = $workflow->attemptUpdateWithArguments('mark-ready', [
    'ready' => true,
]);
```

Named maps are validated against the durably snapped update contract and normalized into declaration order before acceptance.

Use `submitUpdate()` or `submitUpdateWithArguments()` when the caller only needs durable acceptance and is fine with the workflow worker applying the update later:

```php
$accepted = $workflow->submitUpdate('mark-ready', true);

$accepted->accepted();     // true
$accepted->completed();    // false
$accepted->updateStatus(); // "accepted"
$accepted->result();       // null until the worker records UpdateCompleted
```

Use `inspectUpdate()` when you already have an `update_id` and want to read the stored lifecycle later without waiting again:

```php
$latest = $workflow->inspectUpdate($accepted->updateId());

$latest->updateStatus(); // "accepted"
$latest->closedAt();     // null until the lifecycle closes
```

`inspectUpdate()` does not wait for workflow execution. It reloads the stored durable lifecycle and returns the current `UpdateResult`.

## Recover after an uncertain response {#recover-after-an-uncertain-response}

Durable admission, handler completion and an external business effect are
separate facts. A caller timeout can leave an accepted update still running,
and a lost response can hide a result that was already committed. A later
refusal to submit a fresh update does not resolve the original outcome.

When your application saved an update ID, inspect that retained lifecycle
instead of submitting another update. Keep its original run ID so that a
continue-as-new transition does not redirect the investigation:

```php
use Workflow\V2\WorkflowStub;

$latest = WorkflowStub::loadRun($savedRunId, $savedNamespace)->inspectUpdate($savedUpdateId);

$latest->updateStatus();
$latest->result(); // Available when that retained update completed.
```

The saved IDs come from the original operation. Inspection reads the retained
record and can still work after that run closes or becomes historical. Missing
or pruned evidence leaves the outcome unresolved; it is not permission to
repeat a business effect. Preserve an application-owned operation record for
reconciliation when the original response and generated update ID were lost.

### Request IDs and retained receipts

Save the namespace, instance ID, original run ID, update ID, operation name and
intent in your application record. An accepted receipt proves durable admission.
A completed lifecycle supplies the handler's recorded outcome. Neither proves
that a separate payment, email or database write happened exactly once.

Embedded Laravel update webhooks take the caller key from `X-Request-Id`.
Preserve that header through proxies. Standalone Server HTTP updates instead
take the key only from the JSON body's `request_id`; their `X-Request-Id` header
is correlation context. Omitting the body key leaves an update unkeyed, even
when requests share that header.

`request_id` is not a portable, run-scoped Update-ID contract. The qualified PHP
paths can return the first retained update when the same key is submitted with
different arguments or update names within that instance. Reusing a key after
continue-as-new can return a predecessor's result
without admitting an update to the successor. A retained keyed receipt can also
be returned after closure, while a fresh update is refused.

Use one key for one logical operation and preserve its original intent. Do not
change the arguments, reuse it for another update name or assume it starts a new
operation in the next run. The [published PHP qualification](https://github.com/durable-workflow/server/issues/505#issuecomment-6287102328)
covers caller keys independently of proxy headers, including retained receipts
after closure. This recovery behavior does not promise portable same-key,
different-intent conflict detection. That broader contract is tracked in
[shared issue 216](https://github.com/durable-workflow/.github/issues/216).
Native Server and other SDK paths require their own qualification.

Receipt recovery depends on retained command, update and history records. There
is no application-independent deduplication lifetime promised here. Review
[history retention](../configuration/pruning-workflows.md) and keep your business ledger
for the lifetime required by your application. Missing evidence remains an
unknown outcome.

### Protect the business effect

For an operation such as charging an invoice:

1. Persist an application operation key, such as `tenant-1:charge:invoice-123`,
   and its immutable intent before sending the command. Reuse that key during
   recovery.
2. In the activity that applies the effect, use a unique database constraint on
   the tenant and operation key. Atomically check the stored intent, apply the
   local database effect and store its result in one transaction. Return the
   stored result for a duplicate with the same intent. Refuse a different intent.
3. For an external provider, use its supported idempotency contract or an outbox
   and reconciliation process. A transaction in your database cannot make a
   separate provider call atomic.
4. After an uncertain response, reconcile that operation record, the original
   update lifecycle and the provider's evidence. Record whether the effect is
   pending, committed, failed or still unknown before deciding another action.

A query followed by a command leaves a race. A duplicate signal can trigger
another activity even when the original was admitted. The application guard
must tolerate repeated delivery. The qualified example executes two activities
for the same key but commits one local database effect. It does not establish
exactly-once execution for arbitrary external systems.

Update rules:

- `load($instanceId)` updates the newest durable run for that instance (including after continue-as-new).
- Fresh updates through `loadRun($runId)` reject with `rejected_not_current` once that selected run is historical. Retained receipt inspection is separate.
- Fresh updates against closed runs reject with `rejected_not_active`. Replaying a retained keyed receipt is separate.
- An update accepted while the workflow task that completes the run is running gets one delivery turn before the run closes, as a signal does. Its handler runs and the caller receives its result. Updates still pending when a run fails, is cancelled or is terminated close with an explicit failure.
- Failed update bodies do not close the workflow run; they are recorded as update-scoped failures and leave the run open for the next replay task.
- Queries replay completed updates, but rejected or failed updates remain non-replayable command and history facts only.

<!-- Source: docs/features/timers.md -->

# Timers

The framework provides the ability to suspend the execution of a workflow and resume at a later time. These are durable timers, meaning they survive restarts and failures while remaining consistent with workflow replay semantics. This can be useful for implementing delays, retry logic, or timeouts.

To use timers, call `timer($duration)` within your workflow:

```php
use function Workflow\V2\timer;
use Workflow\V2\Workflow;

class MyWorkflow extends Workflow
{
    public function handle(): string
    {
        timer(30);

        return 'The workflow waited 30 seconds.';
    }
}
```

## Delay resolution

Durable timer delays resolve to whole seconds. Pass integer seconds such as
`timer(1)`. When using a PHP interval string or `CarbonInterval`, a positive
fractional duration is rounded up to the next second. A 100 ms interval therefore
becomes one second. `timer(0)` still fires inline.

Service-mode PHP workflows use `WorkflowContext::sleep()` and `deferTimer()`.
From PHP SDK 3.9.0 they accept the same strings and intervals with the same
rounding; see [PHP SDK durations](../polyglot/php.md#durations).

Becoming due makes a timer eligible to fire. Queue latency and worker load can
delay its actual processing, so this resolution is not an exact wake-up guarantee.
It does not reduce the precision of recorded timestamps. See
[testing](../testing.md#keeping-delay-tests-fast) for tests that avoid real waits.

Timer behavior:

- each `timer()` call creates a durable timer row plus typed `TimerScheduled`, `TimerFired`, and, when superseded, `TimerCancelled` history events
- delayed timers run through a dedicated timer task before the workflow task is resumed
- `timer(0)` fires inline during the workflow task and does not create a timer task
- replay and query paths treat typed `TimerScheduled`, `TimerFired`, and `TimerCancelled` history as authoritative for timer lifecycle, so pure timers stay blocked until the committed fire event arrives and selected-run detail can rebuild open, fired, or cancelled timer waits from history
- Waterline surfaces timer waits in run detail and dashboard payloads
- engine-level `cancel()` and `terminate()` commands supersede open timer waits durably, and late timer jobs no-op instead of reopening the run

Timers of any duration work with every Laravel queue driver, including Amazon SQS — the engine transparently chunks long delays over the driver's per-message limit.

`sideEffect()` is available for replay-safe snapshots such as randomness or one-time branch inputs.

## Reading deterministic time

Inside a workflow body, use `Workflow::now()` (or `Workflow\V2\now()`) instead of Laravel's `now()` helper or `Carbon::now()` when you need the current time:

```php
use function Workflow\V2\activity;
use function Workflow\V2\now;
use Workflow\V2\Workflow;

class DurationAwareWorkflow extends Workflow
{
    public function handle(string $name): array
    {
        $startedAt = Workflow::now();
        $greeting = activity(GreetingActivity::class, $name);
        $finishedAt = Workflow::now();

        return [
            'greeting' => $greeting,
            'took_ms' => $finishedAt->getTimestampMs() - $startedAt->getTimestampMs(),
        ];
    }
}
```

`Workflow::now()` advances as the executor replays history events — it returns `recorded_at` of the last activity completion, timer fire, signal receipt, condition resolution, or child workflow completion the replay has consumed. Before any event is consumed, it returns the run's `started_at`. Outside a workflow fiber (for example in an activity or a query method that forwards to a non-workflow helper), it falls back to wall-clock `now()`.

Using `Workflow::now()` keeps your workflow deterministic: two replays of the same history produce the same time values, even if wall-clock time has advanced between them.

<!-- Source: docs/features/signal+timer.md -->

# Signal + Timer

`Workflow\V2` supports both `await($condition, timeout: $seconds, conditionKey: $key)` for timeout-backed condition waits and `await('signal-name', timeout: $seconds)` for timeout-backed named signal waits.

Use it when the workflow should continue as soon as some durable replayed state becomes true, but should also unblock after a deadline if that state never changes.

Service-mode PHP workflows use the `timeout` of `WorkflowContext::waitCondition()` and `deferCondition()`. From PHP SDK 3.9.0 it accepts the same duration strings and intervals as embedded `await()`; see [PHP SDK durations](../polyglot/php.md#durations).

```php
use Workflow\UpdateMethod;
use function Workflow\V2\await;
use Workflow\V2\Attributes\Type;
use Workflow\V2\Workflow;

use function Workflow\V2\minutes;

#[Type('approval-with-timeout')]
class MyWorkflow extends Workflow
{
    private bool $ready = false;

    public function handle(): string
    {
        $approved = await(fn () => $this->ready, timeout: minutes(5), conditionKey: 'approval.ready');

        return $approved ? 'approved' : 'timed out';
    }

    #[UpdateMethod]
    public function markReady(bool $ready = true): array
    {
        $this->ready = $ready;

        return ['ready' => $this->ready];
    }
}
```

`await()` with a `timeout:` parameter works like this:

- The predicate must depend only on replayed durable inputs (activity results, update mutations, child-workflow results). Non-durable state will not wake the wait.
- The optional condition key is a stable operator label for the wait. Replay validates the key against prior history so a redeployment cannot silently reuse the same workflow step for a different predicate.
- If the predicate becomes true before the timer fires, `await()` returns `true`.
- If the deadline wins first, `await()` returns `false`.

Timer sugar helpers are available for readable timeout values: `seconds()`, `minutes()`, `hours()`, `days()`, `weeks()`, `months()`, `years()`.

To change the predicate durably from application code, call the declared update:

```php
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::load('approval-with-timeout');

$workflow->markReady();
```

The return value is `true` when the condition becomes true before the timeout task fires and `false` when the timeout wins.

For named signals, `await('name', timeout: minutes(5))` returns the signal payload when the signal arrives and `null` when the timeout wins. `null` is reserved for timeout: no-argument signals resolve to `true`, one argument resolves to that value, and multiple arguments resolve to an array. `Workflow\V2` does not use legacy `#[SignalMethod]` mutator methods to flip workflow state.

## `awaitWithTimeout()` Helper

`Workflow::awaitWithTimeout($timeout, $condition, $conditionKey = null)` is a sugar wrapper for `await($condition, timeout: $timeout, conditionKey: $conditionKey)`. The argument order puts the deadline first to make the timeout intent obvious at the call site.

```php
use function Workflow\V2\{await, minutes};
use Workflow\V2\Workflow;

#[Type('approval-with-timeout')]
class MyWorkflow extends Workflow
{
    private bool $ready = false;

    public function handle(): string
    {
        $approved = Workflow::awaitWithTimeout(
            minutes(5),
            fn () => $this->ready,
            'approval.ready',
        );

        return $approved ? 'approved' : 'timed out';
    }

    #[UpdateMethod]
    public function markReady(bool $ready = true): array
    {
        $this->ready = $ready;

        return ['ready' => $this->ready];
    }
}
```

Return semantics are the same as `await()` with a `timeout:` parameter: the call returns `true` when the condition wins, `false` when the deadline wins for a callable predicate, the signal payload when a named signal wins, and `null` when the deadline wins for a named signal.

`awaitWithTimeout()` reads "wait at most this long for a condition" rather than "wait for a condition, with a timeout option". Use whichever spelling reads more clearly at the call site — the runtime treats them identically and they record the same workflow history.

<!-- Source: docs/features/condition-waits.md -->

# Condition Waits

`await($condition, $conditionKey = null)` provides replay-safe condition waits. Use it when the predicate depends only on workflow state that was already derived from durable inputs such as updates, activity results, or child results. If you want one named external signal value directly, call `await('name')` — see [Signals](./signals.md).

Condition waits are driven by an update method instead of signal mutators:

```php
use Workflow\UpdateMethod;
use function Workflow\V2\await;
use Workflow\V2\Attributes\Type;
use Workflow\V2\Workflow;

#[Type('approval-workflow')]
class MyWorkflow extends Workflow
{
    private bool $ready = false;

    public function handle(): void
    {
        await(fn () => $this->ready, 'approval.ready');
    }

    #[UpdateMethod]
    public function setReady(bool $ready = true): array
    {
        $this->ready = $ready;

        return ['ready' => $this->ready];
    }
}
```

The optional condition key is a stable, URL-safe operator label for the wait. Replay validates the previously recorded key so a later deployment cannot accidentally reuse the same workflow step for a different predicate — see [Versioning](./versioning.md) for how to evolve a workflow safely across deployments.

For a condition wait with a timeout, see [Signal + Timer](./signal+timer.md).

**Important:** `await()` should only be used in a workflow, not an activity.

<!-- Source: docs/features/side-effects.md -->

# Side Effects

A side effect is a closure containing non-deterministic code. The closure is only executed once and the result is saved. It will not execute again if the workflow is retried. Instead, it will return the saved result. This makes the workflow deterministic because replaying the workflow will always return the same stored value rather than re-running the non-deterministic code.

```php
use function Workflow\V2\await;
use function Workflow\V2\sideEffect;
use Workflow\V2\Attributes\Signal;
use Workflow\V2\Workflow;

#[Signal('finish')]
class MyWorkflow extends Workflow
{
    public function handle(): array
    {
        $token = sideEffect(fn () => random_int(1000, 9999));
        $finish = await('finish');

        return compact('token', 'finish');
    }
}
```

The workflow will only call `random_int()` once and save the result, even if the workflow later fails and is retried.

## When to use side effects

Use `sideEffect()` when you need a non-deterministic value that:

- is computed locally without external I/O (random numbers, UUIDs, timestamps)
- should never change once recorded, even across replays
- does not need retry semantics — the closure runs exactly once

```php
// Generate a correlation token for downstream systems.
$correlationId = sideEffect(fn () => (string) Str::uuid());

// Snapshot the current time for a business rule.
$decidedAt = sideEffect(fn () => now()->toIso8601String());
```

## When to use an activity instead

If the code can fail, talks to an external service, or needs retry/timeout semantics, use an [activity](../defining-workflows/activities.md) instead of a side effect:

| Scenario | Use |
|---|---|
| Generate a random token | `sideEffect()` |
| Read a config value at decision time | `sideEffect()` |
| Call an external API | `activity()` |
| Write to a database | `activity()` |
| Send an email or notification | `activity()` |
| Compute an expensive value that can throw | `activity()` |

The rule of thumb: if the closure can throw an exception that you would want to retry, it belongs in an activity.

## How it works

- each `sideEffect()` call appends a typed `SideEffectRecorded` history event with the workflow step sequence
- workflow replay and query replay both reuse that committed value instead of re-running the closure
- Waterline surfaces the side-effect snapshot as a typed history entry in the selected run timeline
- side effects are still for replay-safe snapshots only, not for work that can fail or that needs retry semantics

## Anti-patterns

**Do not call external services inside a side effect.** If the service call fails, the side effect will not be retried and the workflow will fail permanently:

```php
// BAD: HTTP calls can fail and side effects do not retry.
$price = sideEffect(fn () => Http::get('/api/price')->json('amount'));

// GOOD: Use an activity for external calls.
$price = activity(FetchPriceActivity::class);
```

**Do not put slow or blocking operations inside a side effect.** The closure runs on the workflow task thread. Long-running work delays the entire workflow task:

```php
// BAD: Expensive computation blocks the workflow task.
$hash = sideEffect(fn () => bcrypt($largePayload));

// GOOD: Offload heavy work to an activity.
$hash = activity(ComputeHashActivity::class, $largePayload);
```

**Do not rely on mutable external state.** The closure is executed exactly once. If you read a value that changes over time, the snapshot is frozen at the moment of first execution — not at replay time:

```php
// The cached value is whatever it was during the first execution.
// If the cache changes later, this workflow still sees the old value.
$setting = sideEffect(fn () => cache('feature.flag'));
```

This is by design — the snapshot is intentionally frozen for determinism. If you need a value that updates over the lifetime of the workflow, use a signal or an activity.

## Run this pattern

The elapsed-time workflow in the
[Sample App](/docs/sample-app) is the runnable reference for
keeping clock reads behind `sideEffect()`:

```bash
php artisan app:elapsed
```

`App\Workflows\Elapsed\ElapsedTimeWorkflow` records start and end
timestamps as integer values inside `sideEffect()` callbacks so the
recorded value survives Avro payload decoding on replay. The
Waterline run detail shows two `MarkerRecorded` events bracketing the
timer fire — that pair of markers is the on-disk evidence that the
clock reads stayed deterministic.

<!-- Source: docs/features/heartbeats.md -->

# Heartbeats

Heartbeats let a long-running activity report that its current attempt is still alive.

`heartbeat()` updates the currently claimed `activity_attempts` row with the heartbeat time and latest progress, mirrors the latest heartbeat onto the live activity execution, and renews that activity task's lease. It writes an `ActivityHeartbeatRecorded` history event only when progress changed, at most once per recording interval (see below). Waterline selected-run detail and history exports read the latest heartbeat and progress from the attempt row, and rebuild attempt status, task id, worker, lease, cancellation, and close timestamps from typed `ActivityStarted`, `ActivityHeartbeatRecorded`, `ActivityRetryScheduled`, `ActivityCompleted`, `ActivityFailed`, and `ActivityCancelled` history, with attempt rows as fallback for older data.

```php
use Workflow\V2\Activity;

final class PollRemoteJob extends Activity
{
    public function handle(string $jobId): array
    {
        do {
            sleep(1);

            $status = RemoteService::status($jobId);

            $this->heartbeat([
                'message' => 'Polling remote job',
                'current' => $status['completed'] ?? null,
                'total' => $status['total'] ?? null,
                'unit' => 'steps',
                'details' => [
                    'remote_state' => $status['state'] ?? 'running',
                ],
            ]);
        } while ($status['state'] === 'running');

        return $status;
    }
}
```

Inside the activity, `activityId()`, `attemptId()`, and `attemptCount()` expose the durable execution and current-attempt identity if you need correlation keys for external work. Use `activityId()` as the default remote idempotency key; use `attemptId()` only for systems that need to distinguish separate tries of the same durable activity execution.

Polyglot workers can heartbeat via the HTTP worker bridge without constructing the PHP activity class — see the [worker protocol](/docs/polyglot/worker-protocol) for the endpoints.

A single blocking call may need a separate heartbeat helper. This reports
liveness, not progress: give the call its own finite deadline and stop the
helper when the call finishes or loses authority. Python workers should follow
the [captured-context and cancellation recipe](../polyglot/python.md#heartbeat-helper-thread).

## Cadence and receipts {#heartbeat-cadence-and-receipts}

For a queued activity, the first heartbeat deadline starts when the runtime claims it, before task delivery and user code. A recorded heartbeat sets the next deadline to its runtime timestamp plus `heartbeatTimeout`. Sending the request or receiving its response does not start another full window. See [Heartbeat timeout](./timeouts.md#heartbeat-timeout).

Choose a cadence well inside that window and bound the work between calls. With a 15-second timeout, a target around five seconds can be a starting point only if delivery, each batch or I/O call, scheduling pauses, network latency and transport retries fit the remaining budget. Measure those delays and leave margin. This is planning guidance, not a guaranteed safe interval. An expired or replaced attempt cannot recover its lease by sending a late heartbeat.

There is no automatic progress-heartbeat schedule or SDK batching guarantee. Worker-registration heartbeats and ownership-status probes do not publish activity progress. Every accepted heartbeat renews the lease and moves the heartbeat deadline. A history event is written only when the progress differs from the last recorded progress and at least the larger of the heartbeat timeout and 60 seconds has passed since that event. A heartbeat without progress only proves liveness and writes no history. Progress held back by the interval is recorded once when the attempt closes. A heartbeat every second for an hour with unchanged progress writes one event. Older histories keep the per-heartbeat events they already recorded.

Embedded `Activity::heartbeat()` returns `void`. Invalid progress and storage failures can throw. A normal return is not an acknowledgement object: a closed, expired or replaced attempt can return without recording another heartbeat. Service-mode error behavior differs between direct client calls and supervised workers. See the [PHP](../polyglot/php.md#activity-heartbeats), [Python](../polyglot/python.md#activity-heartbeat-requests) and [Rust](../polyglot/rust.md#activity-heartbeats) guides.

Heartbeat progress does not atomically commit an external batch or fence external effects. A retry-safe application checkpoint needs its own durable commit, stable logical job key and idempotency or external fencing. A replacement attempt's acknowledged-checkpoint accessor is tracked separately in [the checkpoint feature](https://github.com/durable-workflow/.github/issues/218).

## Progress Snapshots

Both `Activity::heartbeat()` and `ActivityTaskBridge::heartbeat()` accept an optional bounded progress snapshot:

- `message`
  - non-empty string up to 280 characters
- `current`
  - non-negative integer or float
- `total`
  - non-negative integer or float
- `unit`
  - non-empty string up to 64 characters
- `details`
  - flat map of up to 20 `key => scalar|null` entries, where keys match `[A-Za-z0-9_.:-]{1,64}`

Use that payload for operator-facing progress like `"Downloading chunk"` or `2 / 5 chunks`, not for large logs or arbitrary nested state. Waterline selected-run detail, the timeline heartbeat entry, and history export surface it back as `last_heartbeat_progress` on the activity and current attempt, while the raw typed `ActivityHeartbeatRecorded` history event keeps the same normalized payload under `progress`.

## Durable Attempt Tracking

The runtime records one first-class durable row per activity attempt, including attempt number, status, started time, latest heartbeat and progress, lease expiry, and close time. It also records typed attempt history as activities start, heartbeat, retry, complete, fail, or cancel, so historical attempt detail survives if mutable attempt or task rows drift later. A recorded heartbeat adds a compact `ActivityHeartbeatRecorded` timeline point with the activity execution id, activity attempt id, heartbeat timestamp, lease expiry, progress, and event-era activity snapshot. If a cancel or terminate command closes the run, the bridge records `ActivityCancelled` when the worker observes the stop and rejects later completion or failure as stale instead of turning cancellation into a result.

Heartbeats are operational liveness signals. Heartbeat as often as your timeout and recovery needs require. Liveness-only heartbeats add no history, and changing progress is recorded at most once per interval. Keep the optional progress snapshot compact enough for operator diagnostics instead of treating it like a streaming event feed.

<!-- Source: docs/features/child-workflows.md -->

# Child Workflows

The current `Workflow\V2` slice supports durable child workflows through straight-line `child()` calls and through `all([...])` fan-in barriers that can be child-only, mixed with activities, or nested inside larger `all([...])` groups. Together, these give a parent workflow a durable way to schedule one or more sub-workflows and wait for their outcomes without keeping the parent process alive.

A child run is still a normal workflow run with its own workflow instance id and run id. What makes it a child is the durable linkage back to the parent run plus one stable parent-issued `child_call_id` for that invocation.

## Namespace boundary {#namespace-boundary}

A child and signals sent through its handle stay in the parent's namespace.
Selecting another workflow type, language or queue does not change that
namespace. Use a [service operation](./service-operations.md) for a durable
call to an operation published in another namespace of the same cluster.

```php
use function Workflow\V2\child;
use Workflow\V2\Workflow;

final class ParentWorkflow extends Workflow
{
    public function handle(string $name): array
    {
        $child = child(ChildWorkflow::class, $name);

        return [
            'parent_workflow_id' => $this->workflowId(),
            'parent_run_id' => $this->runId(),
            'child' => $child,
        ];
    }
}
```

## Search attributes and memo {#child-metadata-inheritance}

A new child run starts with copies of its parent's current search attributes and memo. This is a snapshot at child creation, not a live link. The child can upsert inherited keys or add its own; later parent upserts do not change the child, and child upserts do not change the parent. Untouched copied search attributes are marked `inherited_from_parent`.

For example, a child initially inherits `customer=customer-42` and the parent's memo note. Changing the child's stage or note leaves the parent's values intact. A customer filter can therefore match both parent and child while both retain that customer value; see [parent/child list filters](./search-attributes.md#parent-child-list-filters).

The [published-artifact checks](https://github.com/durable-workflow/durable-workflow.github.io/issues/368#issuecomment-6362355504) verify the snapshot and independent upserts in embedded Workflow 2.10.1 and in service mode with Python SDK 2.9.0 / Server 2.11.0. These are named metadata checks, not a change to the selected installation tuple.

## Current Behavior

- Import the helper with `use function Workflow\V2\child;` and call `child(ChildWorkflow::class, ...)`. The call suspends the parent, creates a durable child run, and returns the child's result when the child closes.
- A failed child throws an exception in the parent; a successful child returns its output. Cancellation and termination surface as distinct exception types.
- `$this->child()` returns the most recent child handle; `$this->children()` returns every child handle in workflow-step order. Handles expose `id()`, `runId()`, `callId()`, and signal helpers such as `signal()` and `signalWithArguments()`.
- If a child uses `continueAsNew()`, the parent transparently follows the newest run — `runId()` moves forward while `id()` stays stable.
- `all()` can combine `fn () => child(...)` and `fn () => activity(...)` closures into one barrier; the parent wakes on the first child failure but waits for every successful branch to close before resuming.

## Parallel Child Barrier

Import the helpers with `use function Workflow\V2\all;` and `use function Workflow\V2\child;` when you want one parent step to wait on several child workflows together.

```php
use function Workflow\V2\all;
use function Workflow\V2\child;
use Workflow\V2\Workflow;

final class ParentWorkflow extends Workflow
{
    public function handle(): array
    {
        $children = all([
            fn () => child(FirstChildWorkflow::class),
            fn () => child(SecondChildWorkflow::class),
        ]);

        return $children;
    }
}
```

That child-only form is still useful when every parallel member is a child workflow:

- it waits for the whole child group as one parent step
- it preserves the original array order in the returned results
- it throws the selected non-successful child closure back into the parent workflow body using the same earliest-close-time, lowest-index tie break

If you need to combine child workflows with activities in the same fan-in step, use one mixed `all([...])` barrier instead. The child waits and activity waits share the same `parallel_group_id`, and Waterline labels that shared barrier as `parallel_group_kind = mixed`. If one mixed or child-only subgroup sits inside a larger `all([...])`, the innermost group still drives `parallel_group_id` while `parallel_group_path` preserves the full outer-to-inner path.

### Cost of wide fan-out {#fan-out-cost}

Each child close locks the parent run while it records the child's outcome, checks the group barrier and updates the parent's projections. In Workflow 2.27 and later, that work doesn't grow with the number of siblings: a close reads no other child's history and runs a fixed number of queries. Closes of one parent's children still take turns on the parent's lock, and the parent can't run its own workflow task while a close holds it.

Before 2.27, each close also re-read every sibling's history and run, so closing all children of a wide fan-out cost work that grew with the square of the number of children.

For fan-outs in the hundreds, or close to the default limit of 1,000 pending children (`pending_child_count`, see [Structural limits](../constraints/structural-limits.md)), split the work:

- **Batches:** start a group of children, wait for it with `all([...])`, then start the next group.
- **A tree:** let each child start its own children, so no single parent supervises more than a few hundred at once.

## Identity Contract

- `child()->id()` and Waterline `target_name` refer to the child workflow instance id, which stays stable across that child instance's run chain.
- `child_call_id` refers to the parent-issued invocation itself, which stays stable even if the child later uses `continueAsNew()`.
- `resume_source_id`, `child_workflow_run_id`, and selected-run routes refer to one concrete child run in that invocation chain.

## Child Handles

Once the parent has durably reached a child step, the workflow can inspect that invocation through a `ChildWorkflowHandle`.

```php
use Workflow\UpdateMethod;
use function Workflow\V2\child;
use Workflow\V2\Workflow;

final class ParentWorkflow extends Workflow
{
    public function handle()
    {
        return child(ApprovalWorkflow::class);
    }

    #[UpdateMethod('approve-child')]
    public function approveChild(string $approvedBy): void
    {
        $this->child()?->signal('approved-by', $approvedBy);
    }
}
```

- `$this->child()` returns the latest visible child handle or `null`.
- `$this->children()` returns every visible handle in workflow-step order.
- During `all([fn () => child(...)])` barriers, those handles appear in the same step order as the child calls inside the barrier, and `$this->child()` returns the last visible one.
- A handle's `runId()` follows the newest parent-recorded `ChildRunStarted` in that child invocation chain, so a parent waiting on a child that used `continueAsNew()` still sees the latest child run id without losing the original `callId()`.

## Handle Availability

Child handles are history-backed, not speculative:

- Before the parent has durably reached a child step, `$this->child()` returns `null` and `$this->children()` returns an empty list.
- Once the parent has recorded child scheduling or child-start history for that step, the handle becomes visible to queries, updates, and later workflow replay.
- Query replay never dispatches child signals from recorded updates; it only restores the in-memory state implied by committed history.

## Waterline Visibility

When the parent is blocked on a child, Waterline shows it as a child wait rather than as a separate task type. That means:

- the parent run summary reports `wait_kind = child` and `liveness_state = waiting_for_child`
- `waits` includes `kind = child`, the stable `child_call_id`, the child workflow instance id in `target_name`, and the child run id in `resume_source_id`
- when several child waits are open at once, detail also exposes `open_wait_count`
- child waits created by one child-only or mixed `all([...])` barrier share `parallel_group_id` and expose `parallel_group_kind`, `parallel_group_base_sequence`, `parallel_group_size`, and `parallel_group_index`
- nested child waits also expose `parallel_group_path`, ordered from the outermost enclosing barrier to the innermost one
- `timeline` includes typed child events such as `ChildWorkflowScheduled`, `ChildRunStarted`, `ChildRunCompleted`, and `ChildRunFailed`, with the same stable `child_call_id`
- open child waits and lineage now prefer the parent's typed `ChildWorkflowScheduled` / `ChildRunStarted` history, keep one logical child entry per stable `child_call_id`, and follow the newest parent-recorded child run even if copied `workflow_links` rows disappear or the child instance's mutable `current_run_id` drifts after history has already been committed
- once the parent has recorded a typed child-resolution event, selected-run detail keeps that child wait resolved from parent history even if the mutable child run row later drifts; child terminal history or legacy link data can enrich lineage and diagnostics, but cannot replace missing parent typed child step history for replay
- once that typed child-resolution event exists, selected-run lineage and history export also keep the child workflow type, class, run number, status bucket, and closed reason from parent history instead of re-reading the mutable child run row for those resolved-child fields
- if a terminal child row or link exists but the parent has no typed `ChildWorkflowScheduled`, `ChildRunStarted`, `ChildRunCompleted`, `ChildRunFailed`, `ChildRunCancelled`, or `ChildRunTerminated` history for that child step, worker and query replay block with `history_shape_mismatch` and recorded events `no typed history`; Waterline marks the selected child wait `status = unsupported`, exposes `history_authority = unsupported_terminal_without_history` and `history_unsupported_reason = terminal_child_link_without_typed_parent_history`, and surfaces child identity from the blocked task or lineage fallback when available
- if that parent resume workflow task row is lost after the child-resolution event, selected-run detail stays `repair_needed`, exposes a synthetic missing workflow task with `workflow_wait_kind = child`, `child_call_id`, and `child_workflow_run_id`, and manual `repair()`, `workflow:v2:repair-pass`, or worker-loop repair recreates the same child-resolution task from typed parent history
- lineage arrays expose the durable parent/child relationship alongside continue-as-new links, and child-workflow entries now carry the same `child_call_id`

## Parent-Close Policy

When a parent workflow closes — whether by completing, failing, timing out, being cancelled, or being terminated — each open child workflow is affected according to its **parent-close policy**. The policy is set per child call and controls what happens to that child when the parent run exits.

| Policy | Value | Behavior |
|---|---|---|
| Abandon | `abandon` | The child continues running independently. This is the default. |
| Request Cancellation | `request_cancellation` | The child receives a cooperative cleanup request with the parent's original cancellation lineage and deadline. |
| Request Cancel | `request_cancel` | A cancel command is sent to the child when the parent closes. |
| Terminate | `terminate` | A terminate command is sent to the child when the parent closes. |

### Default policy

A `child()` call that does not pass a `ChildWorkflowOptions` always runs under `ParentClosePolicy::Abandon`. The source-compatible helpers — `Workflow\V2\child()`, `Workflow\V2\Workflow::child()`, and `Workflow\V2\Workflow::executeChildWorkflow()` — all construct the default options when the first argument is not a `ChildWorkflowOptions`, which sets `parentClosePolicy` to `ParentClosePolicy::Abandon`. The same default applies to every `child()` closure inside an `all([...])` barrier that omits an options argument. Override the default by passing an explicit `ChildWorkflowOptions` as shown below.

### Setting the policy

Pass a `ChildWorkflowOptions` as the first argument to `child()`:

```php
use function Workflow\V2\child;
use Workflow\V2\Enums\ParentClosePolicy;
use Workflow\V2\Support\ChildWorkflowOptions;
use Workflow\V2\Workflow;

final class ParentWorkflow extends Workflow
{
    public function handle(): array
    {
        $options = new ChildWorkflowOptions(
            parentClosePolicy: ParentClosePolicy::RequestCancel,
        );

        return child(ChildWorkflow::class, $options, 'argument1');
    }
}
```

The same pattern works with closures inside `all()` for parallel barriers:

```php
use function Workflow\V2\all;
use function Workflow\V2\child;
use Workflow\V2\Enums\ParentClosePolicy;
use Workflow\V2\Support\ChildWorkflowOptions;

$cancelOptions = new ChildWorkflowOptions(
    parentClosePolicy: ParentClosePolicy::RequestCancel,
);

$results = all([
    fn () => child(FirstChild::class, $cancelOptions),
    fn () => child(SecondChild::class, $cancelOptions, 'arg'),
]);
```

### How it works

- The policy is recorded on the `workflow_links` row (`parent_close_policy`) and in the `ChildWorkflowScheduled` history event payload.
- When the parent run closes for any reason, the engine queries open child links with a non-abandon policy and sends the appropriate command (request cancellation, cancel, or terminate) to each open child.
- If the child has already closed by the time the policy is enforced, no action is taken — the command is silently skipped.
- Policy enforcement is best-effort: if a child command is rejected (e.g. the child is already terminal), the parent's closure is not affected. When enforcement succeeds, a `ParentClosePolicyApplied` history event is recorded on the parent run. When enforcement fails, a `ParentClosePolicyFailed` history event is recorded instead, so operators can distinguish successful enforcement from silent failures.
- Continue-as-new does **not** trigger parent-close policy, because the workflow instance remains active under a new run. From Workflow 2.26, the chain's final run applies it instead. See [Continue-as-new and open children](#continue-as-new-and-open-children).

### Parent disposition matrix

Parent-close policy fires on every terminal parent disposition, and stays inert for runs that are still live or handed off to a continued run. The engine's behavior is the same for every disposition that marks the parent terminal — the policy is applied to any non-abandon, still-open child link.

| Parent disposition | Policy enforced? | Notes |
|---|---|---|
| Completed | Yes | Fires after `WorkflowCompleted` is recorded. A straight-line `child()` call always waits for the child, so a natural completion normally has no open children; the enforcer is still called as a safety pass. |
| Failed | Yes | Fires after `WorkflowFailed` is recorded for any terminal workflow-task failure. |
| Timed out | Yes | Fires after `WorkflowTimedOut` is recorded for run-timeout and execution-timeout closures. |
| Cancelled | Yes | Fires after the accepted `CancelRequested` command closes the parent run. |
| Terminated | Yes | Fires after the accepted `TerminateRequested` command closes the parent run. |
| Continue-as-new | No | The workflow instance stays active under the new run. From Workflow 2.26, the continued run takes over supervising the open children, and the chain's final run applies their policy. |
| Reset | Not currently applicable | The v2 runtime does not expose a standalone reset command or reset terminal disposition. A future reset or repair operation that closes or replaces a parent run must define whether child ownership transfers; if the old run becomes terminal, it must apply parent-close policy before it stops owning open children. |

### Continue-as-new and open children {#continue-as-new-and-open-children}

In Workflow 2.26 and later, a parent that continues as new while children it started are still open hands them to the continued run, in the same transaction. Embedded and service-mode runs behave the same:

- Each open child gets a row in the `workflow_child_supervisions` table, which the 2.26 migration creates. A child that has already closed is not handed on.
- The child link, child call, sequence, deadlines and parent-close policy stay as the starting run recorded them. When the continued run continues as new again, it hands the children on again.
- When the chain's final run completes, fails, times out, is cancelled or is terminated, it applies each child's policy and records `ParentClosePolicyApplied` or `ParentClosePolicyFailed` in its own history. `abandon` stays `abandon`. The closed earlier run gets no new history.
- The continued run has no handle on these children, and their results are not delivered to it. Pass anything the next run needs in the arguments to `continueAsNew()`.
- A handed-on child that fails is retried under the retry policy the starting run recorded, with the same child call identity and attempt count, while a run of the chain still supervises it. The retry run's `WorkflowStarted` names the starting run as its parent, and the chain's final run applies the parent-close policy to the retry run. The failure is final once the attempts are spent, the failure is non-retryable, or the chain has closed.

Before 2.26, these children kept running after the chain closed and their policy was never applied, and a child that failed after its parent continued as new was not retried. See the [2.26.0 release notes](https://github.com/durable-workflow/workflow/releases/tag/2.26.0).

### When to use each policy

**Abandon** (default) is correct when children represent independent work that should complete regardless of the parent's fate — for example, a notification workflow or a cleanup task that must finish.

**Request Cancellation** uses `ParentClosePolicy::RequestCancellation` for cooperative cleanup. It preserves an existing parent cancellation request's root identity and immutable deadline. If the parent closes without such a request, the policy records one shared cleanup budget of 600 seconds from the parent's recorded closure time. Repeated enforcement does not grant more time. See [Cooperative Cancellation](/docs/polyglot/cancellation) for the service-mode worker capability and cleanup contract.

**Request Cancel** applies the child's terminal `cancel()` command when the parent closes. Despite the policy name, it does not deliver the separate cooperative `requestCancellation()` request. The child cannot run workflow-code cleanup or compensation after this policy closes its run. Complete any required compensation before closing the parent, or use **Abandon** for independent cleanup work.

**Terminate** is correct when children must stop immediately. Use this for children that are purely auxiliary to the parent and have no independent value after the parent closes.

### Waterline visibility

Waterline shows the `parent_close_policy` in:

- the child link entry on the parent run's lineage view
- the `ChildWorkflowScheduled` timeline entry payload
- the child's `CancelRequested` or `TerminateRequested` history event when policy enforcement fires, with a reason that names the parent closure

## Current Limitations

The current surface does not include:

- built-in bounded-concurrency helpers beyond the current `all([fn () => child(...)])`, `all([fn () => activity(...)])`, and mixed `all([fn () => child(...), fn () => activity(...)])` barriers

In other words, durable child handles are supported for already-reached child steps, but it does not yet include higher-level launch-handle or bounded-concurrency APIs beyond today's `child(...)` and `all([...])` surface.

## Run this pattern

The microservice coordination workflow in the
[Sample App](/docs/sample-app) is the runnable reference for
parent–child orchestration across application boundaries:

```bash
php artisan app:microservice
```

`App\Workflows\Microservice\MicroserviceWorkflow` runs the parent on
the main Laravel app and dispatches child work to the bundled
microservice worker. Open Waterline while the run is in flight to see
the child links populated under the parent run, with the close policy
visible on the lineage view exactly as this page describes.

<!-- Source: docs/features/concurrency.md -->

# Concurrency

This guide covers concurrency in embedded Laravel workflows using
`Workflow\V2\Workflow`. For service-mode workers, use the
[PHP SDK guide](https://php.durable-workflow.com/build/workflows-activities/),
[Python SDK guide](https://python.durable-workflow.com/sdk-reference/#deterministic-parallel-groups), or
[Rust SDK reference](https://rust.durable-workflow.com/durable_workflow/struct.WorkflowContext.html#method.parallel).

`all([...])` describes the complete durable group before suspension. The
embedded runtime schedules its activity and child-workflow commands and returns
results in the original nested input shape.

Use a selection group when progress depends on the first completed member
instead of the whole barrier. Selection is also durable: it starts every member,
records one winner, and leaves every non-winner running until workflow code
awaits or explicitly cancels it.

## Series

This example will execute 3 activities in series, waiting for the completion of each activity before continuing to the next one.

```php
use function Workflow\V2\activity;
use Workflow\V2\Workflow;

class MyWorkflow extends Workflow
{
    public function handle()
    {
        return [
            activity(MyActivity1::class),
            activity(MyActivity2::class),
            activity(MyActivity3::class),
        ];
    }
}
```

## Parallel

This example will execute 3 activities in parallel, waiting for the completion of all activities and collecting the results.

```php
use function Workflow\V2\{all, activity};
use Workflow\V2\Workflow;

class MyWorkflow extends Workflow
{
    public function handle()
    {
        return all([
            fn () => activity(MyActivity1::class),
            fn () => activity(MyActivity2::class),
            fn () => activity(MyActivity3::class),
        ]);
    }
}
```

The main difference between the serial example and the parallel execution example is where suspension happens. In the serial example, each `activity()` call suspends and resumes the workflow directly. In the parallel example, the closures describe the whole barrier first and `all()` suspends once for the whole group, so every member can run in parallel before the workflow resumes.

## First-completion selection

`select([...])` starts independent activities, child workflows, timers, signal
waits, condition waits, or nested ordinary barriers and resumes when one member
commits an eligible result or typed failure. Give members stable application
keys when later code needs to distinguish or revisit them.
Member keys must be a non-empty string or a non-negative integer.

The following coordinator starts its deadline at the same durable step as the
resolver. Input processing and resolver progress cannot reset or postpone that
deadline:

```php
use function Workflow\V2\{activity, await, select, timer};
use Workflow\V2\Workflow;

final class ResolveInRealTime extends Workflow
{
    public function handle(string $requestId): array
    {
        $selected = select([
            'resolved' => fn () => activity(ResolveRequest::class, $requestId),
            'manual' => fn () => await('resolution.received'),
            'deadline' => fn () => timer('2 seconds'),
        ]);

        if ($selected->key === 'deadline') {
            $selected->handles['resolved']->cancel();

            return ['status' => 'timed_out'];
        }

        // The deadline is not cancelled just because another member won.
        $selected->handles['deadline']->cancel();

        return [
            'status' => 'resolved',
            'source' => $selected->key,
            'value' => $selected->result(),
        ];
    }
}
```

The result contains the stable member key and index, operation kind and durable
identity, result or typed failure, the winner handle, and a handle for every
member. A non-winning handle can be awaited later with `await()` or cancelled
with `cancel()`. Selection never implicitly discards or cancels a sibling.
`cancel()` is a void/unit request and never reports whether cancellation won.
`SelectionOperationCancelled` history is the outcome authority. If completion
commits first, the runtime writes no cancellation marker, replay advances past
the cancel call only after a committed workflow-task boundary, successor
command, or workflow terminal event proves the no-op committed. Query replay
stops at an in-flight cancel instead of exposing speculative state after it.
Awaiting the handle after the committed no-op returns the earlier completion.

The runtime records the first eligible resolution while holding the parent-run
commit lock. Cold restart, persisted-history reload, query replay, and later
loser completions therefore reuse the recorded winner instead of racing local
language futures or reinterpreting history order. Exact duplicate delivery is
idempotent. Concurrent inputs are ordered by their committed durable history;
an input that arrives while an activity runs is visible on the next workflow
task, and a late or duplicate input cannot replace an already recorded winner.

### Losers when the run closes {#selection-losers-at-close}

From Workflow 2.28, if the run closes while a non-winning member is still open, the close cancels it in the same transaction. This applies when the run completes, fails or continues as new. An open activity gets `ActivityCancelled` with `cancellation_reason: run_closed`, and a pending timer gets `TimerCancelled`, both recorded before the terminal event. Nothing is appended after the terminal event, and a late result from the cancelled activity is refused. Replay ignores these cancellations, because the workflow never saw them.

If the workflow needs a loser's effect, await the loser before returning. In service mode, an activity scheduled with `cancellation_policy: abandon` keeps running instead. The terminal event lists it under `abandoned_activities`, and its outcome is never recorded on the closed run. Child workflows follow their parent-close policy.

## Nested Barriers

Nested `all([...])` groups let one workflow step express a tree of durable fan-out and fan-in work. The runtime schedules every activity or child workflow as a durable leaf sequence, records the leaf's full `parallel_group_path`, waits until every enclosing barrier can make progress, and then rebuilds the original nested result shape before resuming the workflow body. During replay, an activity or child leaf from an `all([...])` step must still match that recorded group path; typed leaf history that has no group metadata is treated as incompatible older preview history instead of being guessed into the current barrier.

```php
use function Workflow\V2\{all, activity};
use Workflow\V2\Workflow;

final class NestedWorkflow extends Workflow
{
    public function handle(): array
    {
        return all([
            fn () => activity(BuildSummary::class),
            fn () => all([
                fn () => activity(BuildInvoice::class),
                fn () => activity(BuildShipment::class),
            ]),
        ]);
    }
}
```

In that example, Waterline exposes three open leaf waits, not one synthetic "nested" wait. The first leaf belongs only to the outer barrier, while the second and third leaves expose a two-entry `parallel_group_path` so operators can see both the outer group and the inner subgroup that is still open.

The outer group size counts durable leaves, not nested arrays.
Each nested leaf carries an outer-to-inner `parallel_group_path`; every path
entry preserves the same durable workflow position. The group schedules all
leaves before it suspends, then assembles successful values by input position.
Worker restart and completed-history replay rebuild the same group identity.
Exact duplicate terminal delivery is ignored, and a late sibling completion
can enrich partial diagnostics without changing the failed member already
selected by the embedded runtime's deterministic policy.

The embedded runtime raises the typed leaf failure from `all()` and retains
the barrier's durable group metadata in history and operator views.

## Async Callback

`async(...)` runs a serializable callback as a durable child workflow with the system type `durable-workflow.async`. Async callbacks use the same straight-line-only helper contract as named v2 workflows, so `activity()`, `await()`, `timer()`, `sideEffect()`, and the other single-step helpers suspend directly inside the callback body without forcing `yield`.

```php
use function Workflow\V2\{activity, async};
use Workflow\V2\Workflow;

final class CustomerWorkflow extends Workflow
{
    public function handle(string $customerId): array
    {
        $profile = async(static function () use ($customerId): array {
            $customer = activity(LoadCustomer::class, $customerId);

            return [
                'customer' => $customer,
                'score' => activity(ScoreCustomer::class, $customer['id']),
            ];
        });

        return ['profile' => $profile];
    }
}
```

The parent run sees the callback as a child wait, so command history, lineage, and Waterline detail use the same `child_call_id`, child run id, and child outcome history as an explicit `child(...)` call. The callback is serialized with Laravel's serializable-closure support, so keep it app-local and deployment-local. Use a named `child(SomeWorkflow::class, ...)` call when the work needs a stable public workflow type for cross-service routing or long-lived code evolution. `async(...)` callbacks are now straight-line only in v2, so call helpers like `activity()`, `child()`, `await()`, `timer()`, and `all([...])` directly without `yield`.

## Mixed Activity + Child Barriers

The same `all()` helper can also fan in a mixed group of activities and child workflows. Results still come back in the original array order, successful members still wait for the rest of the group, and the first failed member still wakes the parent immediately.

When more than one barrier member has already closed unsuccessfully by the time the parent replays, the parent receives the failure with the earliest recorded close time. If two failures have the same recorded time, the lower barrier leaf index wins, so workflow resume and query replay select the same exception. Later sibling failures do not replace the exception that has already been thrown into the parent step.

```php
use function Workflow\V2\{all, activity, child};
use Workflow\V2\Workflow;

final class OrderWorkflow extends Workflow
{
    public function handle(): array
    {
        [$charge, $shipment] = all([
            fn () => activity(ChargeCustomer::class),
            fn () => child(ShipOrderWorkflow::class),
        ]);

        return compact('charge', 'shipment');
    }
}
```

## Current Limits

The current concurrency surface does not yet include:

- built-in bounded-concurrency helpers beyond explicit nested `all([...])` groups

## Child Workflows in Parallel

Child workflows can also run in their own `all([...])` barrier. It works the same way as parallel activity execution, but for child workflows: the parent fans out several child runs durably and resumes only when the whole child barrier can make progress.

```php
use function Workflow\V2\{all, child};
use Workflow\V2\Workflow;

final class ParentWorkflow extends Workflow
{
    public function handle(): array
    {
        return all([
            fn () => child(MyChild1::class),
            fn () => child(MyChild2::class),
            fn () => child(MyChild3::class),
        ]);
    }
}
```

This makes it easy to build hierarchical parallelism into your workflows, including nested child-only or mixed child-plus-activity groups when one parent step needs more than one fan-in layer.

<!-- Source: docs/features/sagas.md -->

# Sagas

This guide covers compensation in embedded Laravel workflows using
`Workflow\V2\Workflow`. For service-mode workers, use the
[PHP SDK guide](https://php.durable-workflow.com/build/workflows-activities/),
[Python SDK guide](https://python.durable-workflow.com/sdk-reference/#saga-compensation), or
[Rust SDK reference](https://rust.durable-workflow.com/durable_workflow/struct.Saga.html).

Sagas are an established design pattern for managing complex, long-running operations:

- A saga manages distributed transactions using a sequence of local transactions.
- A local transaction is a work unit performed by a saga participant (an activity).
- Each operation in the saga can be reversed by a compensatory activity.
- Workflow code decides when to invoke compensation. Compensation can fail and must have time and working workers to complete.

```php
use function Workflow\V2\activity;
use Workflow\V2\Attributes\Type;
use Workflow\V2\Workflow;

#[Type('booking-saga')]
class BookingSagaWorkflow extends Workflow
{
    public function handle(): array
    {
        try {
            $flightId = activity(BookFlightActivity::class);
            $this->addCompensation(fn () => activity(CancelFlightActivity::class, $flightId));

            $hotelId = activity(BookHotelActivity::class);
            $this->addCompensation(fn () => activity(CancelHotelActivity::class, $hotelId));

            $carId = activity(BookRentalCarActivity::class);
            $this->addCompensation(fn () => activity(CancelRentalCarActivity::class, $carId));

            return compact('flightId', 'hotelId', 'carId');
        } catch (\Throwable $e) {
            $this->compensate();

            throw $e;
        }
    }
}
```

When the workflow catches an exception, `$this->compensate()` runs every registered compensation in **reverse order**. In the example above, if `BookRentalCarActivity` fails, the engine cancels the hotel first and then the flight — unwinding the saga from the most recent step backward.

Activities run at least once, so a forward call can take effect and still fail from the workflow's point of view, for example after a timeout or a lost completion. Register an idempotent compensation, keyed by a stable business id, before the forward call, so it runs whether or not the forward step's outcome is known. Registering after the forward call succeeds is fine when the compensation needs the forward result and the forward step has no external effect until it succeeds, as in the example above. Replay reconstructs the registration order and reuses recorded activity results, including completed compensations.

## Deadline-driven compensation

An engine [workflow timeout](./timeouts.md#timeout-outcome-and-cleanup) closes
the run before workflow code can run compensation. A `catch` or `finally` block
does not change that behavior. Immediate `cancel()` and `terminate()` commands
also provide no cleanup window.

Use an earlier application deadline when timeout should undo completed work.
This embedded workflow reserves a resource, then waits up to 30 seconds for a
confirmation signal. Expiry resumes workflow code, which releases the
reservation before returning an `expired` business result:

```php
use function Workflow\V2\{activity, await};
use Workflow\V2\Attributes\Signal;
use Workflow\V2\Attributes\Type;
use Workflow\V2\Support\ActivityOptions;
use Workflow\V2\Workflow;

#[Type('reservation-deadline')]
#[Signal('confirm')]
final class ReservationDeadlineWorkflow extends Workflow
{
    public function handle(string $orderId): array
    {
        $reservationId = activity(
            ReserveResourceActivity::class,
            new ActivityOptions(scheduleToCloseTimeout: 30),
            $orderId,
        );
        $this->addCompensation(fn () => activity(
            ReleaseResourceActivity::class,
            new ActivityOptions(scheduleToCloseTimeout: 30),
            $reservationId,
        ));

        if (await('confirm', timeout: 30) === null) {
            $this->compensate();

            return ['status' => 'expired'];
        }

        return ['status' => 'confirmed'];
    }
}
```

`ReserveResourceActivity` and `ReleaseResourceActivity` are your application's
activities. Make both idempotent, with a stable business key for the reservation.
The wait starts after the reservation succeeds. Send `confirm` without
arguments so its successful result is `true`; `null` means the wait expired.

Start it with a larger engine limit as a backstop:

```php
use Workflow\V2\StartOptions;
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::make(ReservationDeadlineWorkflow::class, 'order-123');
$workflow->start(
    'order-123',
    StartOptions::rejectDuplicate()->withRunTimeout(300),
);

// From an authenticated application action, before the wait expires:
WorkflowStub::load('order-123')->signal('confirm');
```

An expired wait with successful compensation completes normally with
`status: expired` in its output. It is not an engine-timed-out run. If the release
activity fails, `compensate()` propagates that failure. If the engine deadline
arrives first, the run closes and remaining compensation cannot execute.
Allow enough time for scheduling and retries, and account for an earlier
execution deadline if one is configured.

The separate cooperative `requestCancellation()` operation is available for
caller-requested cleanup while a run is active. Use its bounded cancellation
shield for durable cleanup under a cancellation request. It is not triggered
automatically by a workflow timeout. See
[Cooperative Cancellation](../polyglot/cancellation.md).

`compensate()` itself runs under a cancellation shield, so a cancellation request that arrives during compensation does not skip the remaining compensations. See [Protected cleanup](./cancel-and-terminate.md#protected-cleanup).

## Compensation ordering

By default, compensations execute **sequentially in reverse registration order**. This is the safest default because later steps may depend on earlier ones.

By default, `compensate()` stops at the first compensation failure and
propagates it to the caller.

## Parallel compensation

To run compensations in parallel, use `setParallelCompensation(true)`. When parallel compensation is enabled, each compensation closure should return a started (but not awaited) activity call so the engine can execute them concurrently:

```php
use function Workflow\V2\activity;

use Workflow\V2\Attributes\Type;
use Workflow\V2\Workflow;

#[Type('parallel-saga')]
class ParallelSagaWorkflow extends Workflow
{
    public function handle(): void
    {
        $this->setParallelCompensation(true);

        try {
            $flightId = activity(BookFlightActivity::class);
            $this->addCompensation(fn () => activity(CancelFlightActivity::class, $flightId));

            $hotelId = activity(BookHotelActivity::class);
            $this->addCompensation(fn () => activity(CancelHotelActivity::class, $hotelId));

            activity(ChargePaymentActivity::class);
        } catch (\Throwable $e) {
            $this->compensate();

            throw $e;
        }
    }
}
```

When parallel compensation is enabled, compensation closures return activity calls that the engine collects and runs through `all()`.

## Continue with error

By default, if a compensation activity throws an exception, the remaining compensations are skipped and the error propagates. To run all compensations regardless of individual failures, use `setContinueWithError(true)`:

```php
$this->setContinueWithError(true);
```

When enabled, the engine catches and discards exceptions from each compensation closure and continues to the next one. This is useful when compensations are independent and you want a best-effort cleanup even if some steps fail.

### Combining both flags

`setParallelCompensation(true)` and `setContinueWithError(true)` can be used together. When both are enabled, all compensations run concurrently through `all()`, and if any compensation throws, the error is caught so the remaining compensations still complete. Without `setContinueWithError(true)`, a parallel compensation failure propagates immediately and the workflow fails.

```php
$this->setParallelCompensation(true);
$this->setContinueWithError(true);
```

## How it works

- `addCompensation()` registers a callable that will be invoked during `compensate()`
- `compensate()` iterates the registered compensations in reverse order
- each compensation closure is a normal V2 workflow step — the activities it calls produce durable history events just like any other activity
- compensation activities are visible in Waterline's timeline and history export
- if the workflow succeeds, compensation closures are never invoked and produce no history

## Run this pattern

The signal-driven travel-agent saga in the
[Sample App](/docs/sample-app) is the runnable reference for this
page. Clone the sample app, set `OPENAI_API_KEY`, and run:

```bash
php artisan app:ai
```

`App\Workflows\Ai\AiWorkflow` registers compensations on every booking
activity, so a flight failure after a successful hotel booking unwinds
the hotel through the compensation list — the same pattern this page
describes, with the events visible in Waterline's run timeline.

<!-- Source: docs/features/events.md -->

# Events

Lifecycle events are dispatched at key stages of workflow and activity execution to notify your application of progress, completion, or failures. These are standard Laravel events — register listeners in your `EventServiceProvider` or with `Event::listen()`.

`WorkflowCancelled` and `WorkflowTerminated` wait for the transaction on the run's storage connection to commit. Rejected commands and rolled-back storage writes emit neither event. Other engine transaction and lifecycle-ordering gaps remain tracked in [workflow#893](https://github.com/durable-workflow/workflow/issues/893). A transaction on a separate application database does not make workflow storage writes atomic.

## Event Identity

Every V2 event carries durable identity fields:

| Field | Description |
|---|---|
| `instanceId` | The workflow instance ID (stable across continue-as-new). |
| `runId` | The specific execution run ID. |
| `workflowType` | The durable type key registered via `#[Type('...')]`. |
| `workflowClass` | The PHP class name of the workflow. |
| `committedAt` | ISO 8601 wall-clock timestamp captured when preparing the lifecycle notification. |

Activity events additionally include:

| Field | Description |
|---|---|
| `activityExecutionId` | The durable activity execution ID. |
| `activityType` | The durable type key of the activity. |
| `activityClass` | The PHP class name of the activity. |
| `sequence` | The position of the activity within the workflow execution. |
| `attemptNumber` | The attempt number (starts at 1). |

## Workflow Events

### WorkflowStarted

Dispatched when a workflow start is durably committed — meaning the first run has been created and the `WorkflowStarted` history event has been recorded.

```php
use Workflow\V2\Events\WorkflowStarted;

Event::listen(WorkflowStarted::class, function (WorkflowStarted $event) {
    Log::info('Workflow started', [
        'instance_id' => $event->instanceId,
        'run_id' => $event->runId,
        'type' => $event->workflowType,
    ]);
});
```

This event also fires when a new run begins via continue-as-new.

### WorkflowCompleted

Dispatched when a workflow run completes successfully.

```php
use Workflow\V2\Events\WorkflowCompleted;

Event::listen(WorkflowCompleted::class, function (WorkflowCompleted $event) {
    Log::info('Workflow completed', [
        'instance_id' => $event->instanceId,
        'run_id' => $event->runId,
    ]);
});
```

### WorkflowFailed

Dispatched when a workflow run fails terminally.

Run timeouts also dispatch `WorkflowFailed`. Expiry of a cooperative cancellation cleanup deadline dispatches `WorkflowCancelled` instead.

Additional fields:
- `exceptionClass`: The PHP exception class name.
- `message`: The exception message.

```php
use Workflow\V2\Events\WorkflowFailed;

Event::listen(WorkflowFailed::class, function (WorkflowFailed $event) {
    Log::error('Workflow failed', [
        'instance_id' => $event->instanceId,
        'exception' => $event->exceptionClass,
        'message' => $event->message,
    ]);
});
```

### WorkflowCancelled {#workflowcancelled}

Dispatched when immediate cancellation closes a run, or when cooperative cancellation closes it after cleanup completes or its deadline expires. A cancellation request alone does not dispatch this event.

In addition to the common identity fields:

| Field | Description |
|---|---|
| `reason` | The original cancellation command reason, or `null`. Cleanup completion and deadline messages do not replace it. |
| `closedAt` | The persisted run close time as a UTC ISO 8601 string. |

```php
use Workflow\V2\Events\WorkflowCancelled;

Event::listen(WorkflowCancelled::class, function (WorkflowCancelled $event) {
    Log::info('Workflow cancelled', [
        'instance_id' => $event->instanceId,
        'run_id' => $event->runId,
        'reason' => $event->reason,
        'closed_at' => $event->closedAt,
    ]);
});
```

### WorkflowTerminated {#workflowterminated}

Dispatched when termination closes a run. It carries the same identity fields, `reason` and `closedAt`, using the termination command's reason. Termination does not execute workflow cleanup code.

```php
use Workflow\V2\Events\WorkflowTerminated;

Event::listen(WorkflowTerminated::class, function (WorkflowTerminated $event) {
    Log::info('Workflow terminated', [
        'instance_id' => $event->instanceId,
        'run_id' => $event->runId,
        'reason' => $event->reason,
        'closed_at' => $event->closedAt,
    ]);
});
```

These are in-process Laravel notifications. A listener failure cannot undo a committed close. The events grant no authority to resume a closed run and do not guarantee durable delivery to an external system.

## Activity Events

### ActivityStarted

Dispatched when an activity task is claimed and execution begins.

```php
use Workflow\V2\Events\ActivityStarted;

Event::listen(ActivityStarted::class, function (ActivityStarted $event) {
    Log::info('Activity started', [
        'activity' => $event->activityType,
        'sequence' => $event->sequence,
        'attempt' => $event->attemptNumber,
    ]);
});
```

### ActivityCompleted

Dispatched when an activity completes successfully.

```php
use Workflow\V2\Events\ActivityCompleted;

Event::listen(ActivityCompleted::class, function (ActivityCompleted $event) {
    Log::info('Activity completed', [
        'activity' => $event->activityType,
        'execution_id' => $event->activityExecutionId,
    ]);
});
```

### ActivityFailed

Dispatched when an activity fails terminally (all retries exhausted or non-retryable exception). Retryable failures that will be retried do **not** trigger this event.

Additional fields:
- `exceptionClass`: The PHP exception class name.
- `message`: The exception message.

```php
use Workflow\V2\Events\ActivityFailed;

Event::listen(ActivityFailed::class, function (ActivityFailed $event) {
    Log::error('Activity failed', [
        'activity' => $event->activityType,
        'exception' => $event->exceptionClass,
        'message' => $event->message,
    ]);
});
```

## Failure Events

### FailureRecorded

Dispatched whenever a durable failure record is committed — for both workflow and activity terminal failures. This is the single hook for error-reporting integrations like Sentry or Bugsnag.

| Field | Description |
|---|---|
| `failureId` | The durable failure record ID. |
| `sourceKind` | `"workflow_run"` or `"activity_execution"`. |
| `sourceId` | The ID of the source (run ID or activity execution ID). |
| `exceptionClass` | The PHP exception class name. |
| `message` | The exception message. |

```php
use Workflow\V2\Events\FailureRecorded;

Event::listen(FailureRecorded::class, function (FailureRecorded $event) {
    // Report to Sentry, Bugsnag, etc.
    report(new \RuntimeException(
        "[{$event->sourceKind}] {$event->exceptionClass}: {$event->message}"
    ));
});
```

## Timestamp Semantics

The engine captures `committedAt` when preparing a lifecycle notification. It is a wall-clock timestamp, distinct from:

- **Workflow virtual time**: The logical time inside the workflow execution (used by timers).
- **Attempt time**: When a specific activity attempt started or finished.
- **Resume latency**: How long after a timer was due the workflow actually resumed.

For cancellation and termination, `closedAt` is the persisted run close timestamp in UTC. An enclosing storage transaction can commit later than when the event fields were captured. Record listener receipt time separately when measuring delivery latency.

## Lifecycle

A typical successful workflow lifecycle:

```
Workflow\V2\Events\WorkflowStarted
Workflow\V2\Events\ActivityStarted
Workflow\V2\Events\ActivityCompleted
Workflow\V2\Events\WorkflowCompleted
```

A workflow lifecycle with a terminal activity failure:

```
Workflow\V2\Events\WorkflowStarted
Workflow\V2\Events\ActivityStarted
Workflow\V2\Events\ActivityFailed
Workflow\V2\Events\FailureRecorded    (source: activity_execution)
Workflow\V2\Events\WorkflowFailed
Workflow\V2\Events\FailureRecorded    (source: workflow_run)
```

A cancelled run ends with:

```
Workflow\V2\Events\WorkflowStarted
Workflow\V2\Events\WorkflowCancelled
```

Cooperative cleanup can emit activity events between these two notifications.

A terminated run ends with:

```
Workflow\V2\Events\WorkflowStarted
Workflow\V2\Events\WorkflowTerminated
```

## Event Namespace

V2 events live in the `Workflow\V2\Events` namespace. If you are upgrading an existing app, see [Migration](../migration.md) for the V1 compatibility event mapping.

<!-- Source: docs/features/webhooks.md -->

# Webhooks

The framework provides webhooks that allow external systems to start workflows and send signals dynamically. This feature enables seamless integration with external services, APIs, and automation tools.

## Enabling Webhooks
To enable webhooks, register the webhook routes in your application’s routes file (`routes/web.php` or `routes/api.php`):

```php
use Workflow\V2\Webhooks;

Webhooks::routes([
    App\Workflows\OrderWorkflow::class,
    'manual-invoice' => App\Workflows\InvoiceWorkflow::class,
]);
```

Pass the explicit map of workflow classes you want to expose. Each alias becomes the public route segment, and each workflow class must carry a stable durable type key via `#[Type(...)]` or a registered entry in `workflows.v2.types.workflows`. See the [Explicit Command And Query Webhooks](#explicit-command-and-query-webhooks) section for the full route matrix.

## Visibility Metadata
When you register routes through `Workflow\V2\Webhooks::routes(...)`, the start route also accepts a reserved `visibility` object. Those fields are stored as workflow visibility metadata and are not passed to the workflow `handle()` method.

```bash
curl -X POST "https://example.com/webhooks/start/order-workflow" \
     -H "Content-Type: application/json" \
     -d '{
           "workflow_id": "order-123",
           "orderId": 123,
           "visibility": {
             "business_key": "order-123",
             "labels": {
               "tenant": "acme",
               "region": "us-east"
             },
             "memo": {
               "customer": {
                 "id": 42,
                 "name": "Taylor"
               },
               "source": "checkout"
             }
           }
         }'
```

`business_key` and `visibility.labels` are copied onto the instance, run, run summary, typed start history, selected-run detail, and history export. Waterline can filter list screens by those fields with exact-match query parameters.

`visibility.memo` is copied onto the instance, run, typed start history, selected-run detail, and history export, and later `continueAsNew()` runs inherit the same memo by default. It must be a JSON object at the top level, with nested scalars, `null`, arrays, or objects. `memo` is returned-only metadata, not a list-filter or run-summary search field.

## Webhook Authentication
By default, webhooks don't require authentication, but you can configure one of several strategies in `config/workflows.php`.

**Important:** If webhook URLs are shared with external parties or exposed publicly, enable authentication (token or HMAC signature) to prevent unauthorized access.

### Authentication Methods
It supports:
1. No Authentication (none)
2. Token-based Authentication (token)
3. HMAC Signature Verification (signature)
4. Custom Authentication (custom)

### Token Authentication
For token authentication, webhooks require a valid API token in the request headers. The default header is `Authorization` but you can change this in the configuration settings.

#### Example Request
```bash
curl -X POST "https://example.com/webhooks/start/order-workflow" \
     -H "Content-Type: application/json" \
     -H "Authorization: your-api-token" \
     -d '{"orderId": 123}'
```

### HMAC Signature Authentication
For HMAC authentication, it verifies requests using a secret key. The default header is `X-Signature` but this can also be changed.

#### Example Request
```bash
BODY='{"orderId": 123}'
SIGNATURE=$(echo -n "$BODY" | openssl dgst -sha256 -hmac "your-secret-key" | awk '{print $2}')

curl -X POST "https://example.com/webhooks/start/order-workflow" \
     -H "Content-Type: application/json" \
     -H "X-Signature: $SIGNATURE" \
     -d "$BODY"
```

### Custom Authentication
To use a custom authenticator, create a class that implements the `WebhookAuthenticator` interface:

```php
use Illuminate\Http\Request;
use Workflow\Auth\WebhookAuthenticator;

class CustomAuthenticator implements WebhookAuthenticator
{
    public function validate(Request $request): Request
    {
        $allow = true;

        if ($allow) {
            return $request;
        } else {
            abort(401, 'Unauthorized');
        }
    }
}
```

Then configure it in `config/workflows.php`:

```php
'webhook_auth' => [
    'method' => 'custom',
    'custom' => [
        'class' => App\Your\CustomAuthenticator::class,
    ],
],
```

The `validate()` method should return the `Request` if valid, or call `abort(401)` if unauthorized.

## Configuring Webhook Routes
By default, webhooks are accessible under `/webhooks`. You can customize the route path in `config/workflows.php`:

```php
'webhooks_route' => 'workflows',
```

After this change, webhooks will be accessible under:
```
POST /workflows/start/order-workflow
POST /workflows/signal/order-workflow/{workflowId}/mark-as-shipped
POST /workflows/instances/{workflowId}/queries/{query}
POST /workflows/instances/{workflowId}/runs/{runId}/queries/{query}
POST /workflows/instances/{workflowId}/signals/{signal}
POST /workflows/instances/{workflowId}/runs/{runId}/signals/{signal}
POST /workflows/instances/{workflowId}/repair
POST /workflows/instances/{workflowId}/cancel
POST /workflows/instances/{workflowId}/terminate
GET  /workflows/instances/{workflowId}/describe
POST /workflows/instances/{workflowId}/runs/{runId}/repair
POST /workflows/instances/{workflowId}/runs/{runId}/cancel
POST /workflows/instances/{workflowId}/runs/{runId}/terminate
GET  /workflows/instances/{workflowId}/runs/{runId}/describe
GET  /workflows/workflow-tasks/poll
GET  /workflows/activity-tasks/poll
POST /workflows/control-plane/start
```

## Explicit Command And Query Webhooks

Durable start, signal, update, repair, cancel, and terminate commands are exposed over HTTP, plus replay-safe query routes for current-run and selected-run reads.

You register an explicit alias map and route only the workflows you want to expose.

```php
use Workflow\V2\Webhooks;

Webhooks::routes([
    App\Workflows\OrderWorkflow::class,
    'manual-invoice' => App\Workflows\InvoiceWorkflow::class,
]);
```

When you register a workflow class directly, it must define a stable type key with `#[Type(...)]` or be registered under `workflows.v2.types.workflows`:

```php
use Workflow\V2\Attributes\Type;
use Workflow\V2\Workflow;

#[Type('order-workflow')]
class OrderWorkflow extends Workflow
{
    public function handle(int $orderId)
    {
        // ...
    }
}
```

Equivalent config registration:

```php
// config/workflows.php
'v2' => [
    'types' => [
        'workflows' => [
            'order-workflow' => App\Workflows\OrderWorkflow::class,
        ],
    ],
],
```

The resulting start route is:

```
POST /webhooks/start/order-workflow
```

Example request:

```bash
curl -X POST "https://example.com/webhooks/start/order-workflow" \
     -H "Content-Type: application/json" \
     -d '{"workflow_id":"order-123","orderId":123}'
```

The `workflow_id` is the opaque public workflow instance id. It is not a run id and should be treated as an opaque string.

Caller-supplied `workflow_id` values must be non-empty URL-safe strings up to 191 characters using only letters, numbers, `.`, `_`, `-`, and `:`. Blank, overlong, or unsupported-character ids are rejected at webhook validation time with HTTP `422`.

The same durable type map is also the current worker fallback when a stored workflow class name drifts after a refactor. If you keep `order-workflow` stable and repoint the config registration to the new class, queued v2 work can still resolve the durable type key even when the original stored PHP class name is no longer loadable.

You can also request explicit duplicate-start behavior:

```bash
curl -X POST "https://example.com/webhooks/start/order-workflow" \
     -H "Content-Type: application/json" \
     -d '{"workflow_id":"order-123","orderId":123,"on_duplicate":"return_existing_active"}'
```

Supported `on_duplicate` values are:

- `reject_duplicate`
- `return_existing_active`

All command webhooks return the same JSON envelope:

```json
{
  "outcome": "started_new",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "command_source": "webhook",
  "rejection_reason": null
}
```

Accepted response when a new run is created:

```json
{
  "outcome": "started_new",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "command_source": "webhook",
  "rejection_reason": null
}
```

Accepted response when the caller requested reuse of the existing active run:

```json
{
  "outcome": "returned_existing_active",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "command_source": "webhook",
  "rejection_reason": null
}
```

Rejected duplicate-start response:

```json
{
  "outcome": "rejected_duplicate",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "requested_run_id": null,
  "resolved_run_id": "01J...",
  "command_id": "01J...",
  "workflow_type": "order-workflow",
  "command_status": "rejected",
  "command_source": "webhook",
  "rejection_reason": "instance_already_started"
}
```

Response fields:

- `workflow_id` is the public workflow instance id.
- `run_id` is the first active run created by the accepted start command.
- `requested_run_id` is only set when the caller explicitly addressed one selected run.
- `resolved_run_id` is the run the engine actually resolved for this command; for instance-targeted starts it matches `run_id`.
- `command_id` is the durable start-command id.
- `command_source` is `webhook` for the webhook routes.
- instance-targeted signal, update, repair, cancel, and terminate routes resolve the newest durable run for that instance instead of trusting only the mutable current-run pointer, so continue-as-new chains stay addressable through the same public id even if that column drifts.
- run-targeted command routes pin one selected run under the same public instance id and reject with `target_scope = run`, `outcome = rejected_not_current`, and `rejection_reason = selected_run_not_current` if that run is no longer current; in that case `run_id` and `requested_run_id` stay on the historical selection while `resolved_run_id` points at the current run that callers should address next.
- payload keys are matched to the workflow `handle()` parameter names in declaration order.
- Workflows define their start arguments on `handle()`.
- Workflow classes that do not declare `handle()` are rejected with HTTP `422` validation errors before a run is created.
- missing required payload keys are rejected with HTTP `422` validation errors instead of being silently dropped.
- blank, overlong, or non-route-safe `workflow_id` values are rejected with HTTP `422` validation errors instead of falling through duplicate-start handling.
- invalid `on_duplicate` values are rejected with HTTP `422` validation errors.
- `on_duplicate = return_existing_active` returns HTTP `200` when the current active run is reused.
- duplicate starts that are not reused return HTTP `409` with `outcome = rejected_duplicate`, `command_status = rejected`, and `rejection_reason = instance_already_started`.
- the durable `workflow_commands` row also stores webhook ingress context separately from business arguments, including the caller label, auth method or outcome, request route name, request path, and a request fingerprint derived from the normalized payload plus selected request headers such as `X-Request-Id` and `X-Correlation-Id`

Current HTTP response matrix for the start webhook:

- `202` with `outcome = started_new` when a new run is created
- `200` with `outcome = returned_existing_active` when the existing active run is reused
- `409` with `outcome = rejected_duplicate` when the duplicate start is rejected
- `401` for webhook auth failure
- `404` for an unknown alias
- `422` for payload validation failure

### Signal Command Webhooks

Both instance-targeted and run-targeted signal commands are available:

```text
POST /webhooks/instances/{workflowId}/signals/{signal}
POST /webhooks/instances/{workflowId}/runs/{runId}/signals/{signal}
```

`workflowId` is the public workflow instance id. The instance-targeted route resolves the current active run at apply time. The run-targeted route also takes one selected `runId` and rejects if that selected run is historical. The signal name comes from the route parameter. The request body currently accepts one optional top-level field:

```json
{
  "arguments": ["Taylor"]
}
```

The `arguments` field must be an array. When it is omitted, a workflow waiting with `await('signal-name')` resumes with `true`.
The targeted workflow class must also declare the signal name with `#[Workflow\V2\Attributes\Signal('...')]`; undeclared names are rejected durably instead of being buffered blindly.

Signal example:

```bash
curl -X POST "https://example.com/webhooks/instances/order-123/signals/approved-by" \
     -H "Content-Type: application/json" \
     -d '{"arguments":["Taylor"]}'
```

Accepted signal response:

```json
{
  "outcome": "signal_received",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "requested_run_id": null,
  "resolved_run_id": "01J...",
  "command_id": "01J...",
  "command_sequence": 2,
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "command_source": "webhook",
  "rejection_reason": null
}
```

Accepted and rejected signal commands also get one first-class durable `workflow_signal_records` lifecycle row linked back to the originating command. Signal command responses still return the command id as the HTTP correlation id; Waterline selected-run detail and history exports expose the lifecycle row as `signals[*].id`, together with the signal name, `signal_wait_id`, command sequence, workflow sequence once applied, status, outcome, validation errors, and stored arguments.

Rejected response when the instance exists but has not started a run yet:

```json
{
  "outcome": "rejected_not_started",
  "workflow_id": "order-123",
  "run_id": null,
  "command_id": "01J...",
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "rejected",
  "rejection_reason": "instance_not_started"
}
```

Rejected response when the current run is already closed:

```json
{
  "outcome": "rejected_not_active",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "rejected",
  "rejection_reason": "run_not_active"
}
```

Rejected response when the route targets an undeclared signal name:

```json
{
  "outcome": "rejected_unknown_signal",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "rejected",
  "rejection_reason": "unknown_signal"
}
```

Rejected response when a run-targeted signal route addresses a historical run:

```json
{
  "outcome": "rejected_not_current",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "requested_run_id": "01J...",
  "resolved_run_id": "01J...",
  "command_id": "01J...",
  "target_scope": "run",
  "workflow_type": "order-workflow",
  "command_status": "rejected",
  "rejection_reason": "selected_run_not_current"
}
```

Current HTTP response matrix for the signal webhook:

- `202` with `outcome = signal_received` when the signal command is accepted
- `401` for webhook auth failure
- `404` for an unknown workflow instance id
- `404` with `outcome = rejected_unknown_signal` when the workflow does not declare that signal name; when the run already carries a typed `WorkflowStarted` contract snapshot, or when an older `WorkflowStarted` event can be backfilled on first compatible intake, that rejection no longer depends on reflecting the live workflow class first
- `409` with `outcome = rejected_not_started` when the instance has no current run yet
- `409` with `outcome = rejected_not_active` when the targeted current run is already closed
- `409` with `outcome = rejected_not_current` when the run-targeted route addresses a historical selected run
- `422` when `arguments` is present but not an array

### Signal-With-Start Webhooks

One instance-targeted linked-intake route starts a new run or reuses the current active run before recording a signal:

```text
POST /webhooks/start/{alias}/signals/{signal}
```

`alias` is the webhook workflow alias you registered in `Workflow\V2\Webhooks::routes([...])`. `signal` is the durable signal name declared by `#[Signal(...)]`.

The request body combines the normal start fields plus one reserved `signal_arguments` field for the attached signal payload:

```json
{
  "workflow_id": "order-123",
  "signal_arguments": ["Taylor"],
  "visibility": {
    "business_key": "order-123"
  }
}
```

`signal_arguments` must be an array. The route defaults `on_duplicate` to `return_existing_active`, and rejects any other duplicate policy value for this linked-intake route.

Accepted response when the route starts a new run:

```json
{
  "outcome": "signal_received",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "requested_run_id": null,
  "resolved_run_id": "01J...",
  "command_id": "01J...",
  "command_sequence": 2,
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "command_source": "webhook",
  "start_command_id": "01J...",
  "start_command_sequence": 1,
  "start_outcome": "started_new",
  "start_command_status": "accepted",
  "intake_group_id": "01J...",
  "rejection_reason": null
}
```

When the workflow instance already has an active current run, the same route returns `start_outcome = returned_existing_active` and keeps the signal command as the top-level `command_*` response object. The linked start and signal commands share the same `intake_group_id`, request route metadata, and request correlation headers, and the runtime records both commands in one transaction before the worker can run the first user workflow step.

Selected-run detail and history export freeze that same compound intake under `linked_intakes[*]`. The durable authority is the shared `workflow_commands.context.intake.group_id` plus `mode` recorded on each accepted command row. For the `signal_with_start` mode, the grouped row exposes `start_command_*`, the non-start `primary_command_*`, `complete`, `missing_expected_command_types`, and the ordered nested `commands[*]` snapshots.

If the signal name is unknown or the signal payload fails durable contract validation, the runtime rejects the signal command before creating a new run. In that case the response leaves `run_id`, `start_command_id`, and `start_outcome` as `null`.

Current HTTP response matrix for the signal-with-start webhook:

- `202` with `outcome = signal_received` and `start_outcome = started_new` when the request creates a run and records the signal
- `202` with `outcome = signal_received` and `start_outcome = returned_existing_active` when the request reuses the current active run and records the signal
- `401` for webhook auth failure
- `404` for an unknown webhook workflow alias
- `404` with `outcome = rejected_unknown_signal` when the workflow does not declare that signal name
- `422` with `outcome = rejected_invalid_arguments` when `signal_arguments` is an array but does not satisfy the durable signal contract
- `422` when `signal_arguments` is present but not an array, or when `on_duplicate` is set to any value other than `return_existing_active`

### Query Webhooks

Both instance-targeted and run-targeted query routes are available:

```text
POST /webhooks/instances/{workflowId}/queries/{query}
POST /webhooks/instances/{workflowId}/runs/{runId}/queries/{query}
```

`workflowId` is the public workflow instance id. The instance-targeted route resolves the newest durable run for that instance after refresh. The run-targeted route pins one selected `runId`, including historical or already-closed runs, because queries are read-only replay operations.

`query` is the public query target declared by `#[QueryMethod]`. When the workflow definition is still loadable, the route also accepts the underlying PHP method name, but successful responses normalize `query_name` back to the durable public target.

The request body accepts either a positional `arguments` list or a named `arguments` map keyed by the declared query parameters:

```json
{
  "arguments": ["start"]
}
```

```json
{
  "arguments": {
    "prefix": "start"
  }
}
```

Unlike the mutating webhook routes, query webhooks do not append a durable command row. They replay the selected run and return a serialized result immediately.

Query example:

```bash
curl -X POST "https://example.com/webhooks/instances/order-123/queries/events-starting-with" \
     -H "Content-Type: application/json" \
     -d '{"arguments":{"prefix":"start"}}'
```

Accepted query response:

```json
{
  "query_name": "events-starting-with",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "target_scope": "instance",
  "result": "i:1;"
}
```

Rejected response when the query arguments do not match the declared contract:

```json
{
  "query_name": "events-starting-with",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "target_scope": "instance",
  "message": "Workflow query [events-starting-with] received invalid arguments.",
  "validation_errors": {
    "prefix": [
      "The prefix argument is required."
    ]
  }
}
```

Rejected response when the selected run's workflow definition can no longer be replayed:

```json
{
  "query_name": "events-starting-with",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "target_scope": "instance",
  "blocked_reason": "workflow_definition_unavailable",
  "message": "Workflow 01J... [order-123] cannot execute query [events-starting-with] because the workflow definition is unavailable for durable type [order-workflow]."
}
```

Current HTTP response matrix for the query webhook:

- `200` with a serialized `result` when the query replay succeeds
- `401` for webhook auth failure
- `404` for an unknown workflow instance id or selected run id
- `409` when the selected instance has not started yet, when the requested query is not declared on that selected run, or when replay is blocked by `blocked_reason = workflow_definition_unavailable`
- `422` when the declared query contract rejects the provided `arguments`

### Update Command Webhooks

Instance-targeted and run-targeted update POST routes plus durable lifecycle lookup routes are available:

```text
POST /webhooks/instances/{workflowId}/updates/{update}
POST /webhooks/instances/{workflowId}/runs/{runId}/updates/{update}
GET /webhooks/instances/{workflowId}/updates/{updateId}
GET /webhooks/instances/{workflowId}/runs/{runId}/updates/{updateId}
```

`workflowId` is the public workflow instance id. `update` is the durable update name declared by `#[UpdateMethod]`. `updateId` is the durable lifecycle id returned by update responses. The instance-targeted POST route resolves the current active run at apply time. The run-targeted POST route also takes one selected `runId` and rejects if that selected run is historical. The instance-targeted GET route reads the stored lifecycle for that workflow instance, while the run-targeted GET route narrows lookup to the selected run.

If you use `#[UpdateMethod('mark-approved')]`, the public webhook target is `mark-approved` even if the underlying PHP method is named differently.

The request body accepts either a positional `arguments` list or a named `arguments` map keyed by the declared update parameters:

```json
{
  "arguments": [true, "api"]
}
```

```json
{
  "arguments": {
    "approved": true
  }
}
```

Set `wait_for` to `accepted` when the caller only needs the durable update command to be accepted and wants the workflow worker to apply the update later:

```json
{
  "arguments": {
    "approved": true
  },
  "wait_for": "accepted"
}
```

When `wait_for` is omitted or set to `completed`, the webhook records the accepted update first, then waits up to the configured completion budget for the workflow worker to apply it. Override that budget per request with `wait_timeout_seconds`:

```json
{
  "arguments": {
    "approved": true
  },
  "wait_timeout_seconds": 5
}
```

If the worker completes within that budget, the webhook returns the normal completed or failed update response. If the wait budget expires first, the webhook returns HTTP `202` with the still-open accepted lifecycle instead of blocking indefinitely. When `wait_for` is `accepted`, the webhook records the command row, the first-class update lifecycle row, and typed `UpdateAccepted` history, then schedules or re-dispatches a workflow task and returns HTTP `202` without waiting for application.

Every update POST and GET response includes the normal command fields plus `update_id`, `update_name`, `update_status`, `workflow_sequence`, `accepted_at`, `applied_at`, `rejected_at`, and `closed_at`. Accepted-but-open lifecycles return `workflow_sequence = null`, `result = null`, `applied_at = null`, `rejected_at = null`, and `closed_at = null` until the worker closes the lifecycle.

When a named map is accepted, the engine normalizes it into the declared parameter order before it appends typed `UpdateAccepted` or `UpdateApplied` history. Optional parameters also fill from their PHP defaults during that normalization step.

Every accepted or rejected update is backed by one durable `workflow_updates` row keyed by `update_id`, so callers can POST once and later GET the stored lifecycle without inferring state from generic command rows.

Update example:

```bash
curl -X POST "https://example.com/webhooks/instances/order-123/updates/mark-ready" \
     -H "Content-Type: application/json" \
     -d '{"arguments":{"approved":true}}'
```

Completed update response:

```json
{
  "outcome": "update_completed",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "update_id": "01J...",
  "command_sequence": 2,
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "command_source": "webhook",
  "update_name": "mark-ready",
  "update_status": "completed",
  "workflow_sequence": 1,
  "accepted_at": "2026-04-10T12:00:00.000000Z",
  "applied_at": "2026-04-10T12:00:01.000000Z",
  "rejected_at": null,
  "closed_at": "2026-04-10T12:00:01.000000Z",
  "wait_for": "completed",
  "wait_timed_out": false,
  "wait_timeout_seconds": 10,
  "rejection_reason": null,
  "validation_errors": [],
  "result": {
    "approved": true
  },
  "failure_id": null,
  "failure_message": null
}
```

Accepted-only update response:

```json
{
  "outcome": null,
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "update_id": "01J...",
  "command_sequence": 2,
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "command_source": "webhook",
  "update_name": "mark-ready",
  "update_status": "accepted",
  "workflow_sequence": null,
  "accepted_at": "2026-04-10T12:00:00.000000Z",
  "applied_at": null,
  "rejected_at": null,
  "closed_at": null,
  "wait_for": "accepted",
  "wait_timed_out": false,
  "wait_timeout_seconds": null,
  "rejection_reason": null,
  "validation_errors": [],
  "result": null,
  "failure_id": null,
  "failure_message": null
}
```

Lifecycle lookup example:

```bash
curl "https://example.com/webhooks/instances/order-123/updates/01J..." \
     -H "Content-Type: application/json"
```

Lookup response while the update is still open:

```json
{
  "outcome": null,
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "update_id": "01J...",
  "command_sequence": 2,
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "command_source": "webhook",
  "update_name": "mark-ready",
  "update_status": "accepted",
  "workflow_sequence": null,
  "accepted_at": "2026-04-10T12:00:00.000000Z",
  "applied_at": null,
  "rejected_at": null,
  "closed_at": null,
  "wait_for": "status",
  "wait_timed_out": false,
  "wait_timeout_seconds": null,
  "rejection_reason": null,
  "validation_errors": [],
  "result": null,
  "failure_id": null,
  "failure_message": null
}
```

Lookup response after the lifecycle closes:

```json
{
  "outcome": "update_completed",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "update_id": "01J...",
  "command_sequence": 2,
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "command_source": "webhook",
  "update_name": "mark-ready",
  "update_status": "completed",
  "workflow_sequence": 1,
  "accepted_at": "2026-04-10T12:00:00.000000Z",
  "applied_at": "2026-04-10T12:00:01.000000Z",
  "rejected_at": null,
  "closed_at": "2026-04-10T12:00:01.000000Z",
  "wait_for": "status",
  "wait_timed_out": false,
  "wait_timeout_seconds": null,
  "rejection_reason": null,
  "validation_errors": [],
  "result": {
    "approved": true
  },
  "failure_id": null,
  "failure_message": null
}
```

Lookup routes do not wait for execution. They only read the stored durable lifecycle, so they always return `wait_for = status`, `wait_timed_out = false`, and `wait_timeout_seconds = null`.

When the selected run already has an earlier accepted signal waiting to be applied, the later POST rejects as `rejected_pending_signal` instead of running that workflow task inline on the caller path. Invalid named or positional payloads reject as `rejected_invalid_arguments` with `validation_errors`. Historical run-targeted POST routes reject as `rejected_not_current`.

Current HTTP response matrix for the update POST routes:

- `200` with `outcome = update_completed` when the update command is accepted and completed
- `202` with `update_status = accepted` when `wait_for = accepted` durably accepts the update command and leaves application to the workflow worker
- `202` with `update_status = accepted`, `wait_for = completed`, and `wait_timed_out = true` when the webhook waited up to `wait_timeout_seconds` or the configured default and the worker still had not closed the update lifecycle
- `401` for webhook auth failure
- `404` for an unknown workflow instance id
- `404` with `outcome = rejected_unknown_update` for an unknown durable update name; when the run already carries a typed `WorkflowStarted` contract snapshot, or when an older `WorkflowStarted` event can be backfilled on first compatible intake, that rejection can come from durable run metadata before any live-definition fallback
- `409` with `outcome = rejected_not_started` when the instance has no current run yet
- `409` with `outcome = rejected_not_active` when the current run is already closed
- `409` with `outcome = rejected_not_current` when the run-targeted route addresses a historical selected run
- `409` with `outcome = rejected_pending_signal` when an earlier accepted signal still has to be applied before the update can run
- `422` with `outcome = rejected_invalid_arguments` when the update payload is an array but does not satisfy the declared update contract, including missing arguments, unknown arguments, type mismatches, or nullability violations
- `422` with `outcome = update_failed` when the update body throws
- `422` when `arguments` is present but not an array; that request-shape failure is rejected before a durable command row is created

Current HTTP response matrix for the update GET lookup routes:

- `200` once the lifecycle is closed and `update_status` is `completed`, `failed`, or `rejected`
- `202` while `update_status = accepted`
- `401` for webhook auth failure
- `404` for an unknown workflow instance id, selected run id, or `update_id` within the requested scope

### Repair And Terminal Command Webhooks

Both instance-targeted and run-targeted repair and terminal command routes are available:

```text
POST /webhooks/instances/{workflowId}/repair
POST /webhooks/instances/{workflowId}/cancel
POST /webhooks/instances/{workflowId}/terminate
POST /webhooks/instances/{workflowId}/runs/{runId}/repair
POST /webhooks/instances/{workflowId}/runs/{runId}/cancel
POST /webhooks/instances/{workflowId}/runs/{runId}/terminate
```

`workflowId` here is the public workflow instance id. The instance routes always resolve the current active run. The run routes also take one selected `runId` and reject historical selections with the durable `rejected_not_current` outcome. The same `workflows.webhook_auth` configuration used by the start route also applies to these terminal command routes.

Repair example:

```bash
curl -X POST "https://example.com/webhooks/instances/order-123/repair" \
     -H "Content-Type: application/json"
```

Cancel example:

```bash
curl -X POST "https://example.com/webhooks/instances/order-123/cancel" \
     -H "Content-Type: application/json"
```

Terminate example:

```bash
curl -X POST "https://example.com/webhooks/instances/order-123/terminate" \
     -H "Content-Type: application/json"
```

Accepted repair response when the runtime restores durable progress for the current run:

```json
{
  "outcome": "repair_dispatched",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "rejection_reason": null
}
```

Accepted repair response when the current run already has a healthy durable resume path:

```json
{
  "outcome": "repair_not_needed",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "rejection_reason": null
}
```

Accepted cancel response:

```json
{
  "outcome": "cancelled",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "rejection_reason": null
}
```

Accepted terminate response:

```json
{
  "outcome": "terminated",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "rejection_reason": null
}
```

Rejected response when the instance exists but has not started a run yet:

```json
{
  "outcome": "rejected_not_started",
  "workflow_id": "order-123",
  "run_id": null,
  "command_id": "01J...",
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "rejected",
  "rejection_reason": "instance_not_started"
}
```

Rejected response when the current run is already closed:

```json
{
  "outcome": "rejected_not_active",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "rejected",
  "rejection_reason": "run_not_active"
}
```

Rejected response when a run-targeted repair, cancel, or terminate route addresses a historical run:

```json
{
  "outcome": "rejected_not_current",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "requested_run_id": "01J...",
  "resolved_run_id": "01J...",
  "command_id": "01J...",
  "target_scope": "run",
  "workflow_type": "order-workflow",
  "command_status": "rejected",
  "rejection_reason": "selected_run_not_current"
}
```

Current HTTP response matrix for the repair, cancel, and terminate webhooks:

- `200` with `outcome = repair_dispatched` when repair re-dispatches an overdue ready task, reclaims an expired lease, or recreates a missing durable task that has not started running yet, including accepted-update and accepted-signal workflow application tasks whose rows were lost before the command could be applied
- `200` with `outcome = repair_not_needed` when the run already has a healthy durable resume path or when the selected run is already inside an in-flight activity with no task row
- `200` with `outcome = cancelled` or `outcome = terminated` when the current run is closed successfully
- `401` for webhook auth failure
- `404` for an unknown workflow instance id
- `409` with `outcome = rejected_not_started` when the instance has no current run yet
- `409` with `outcome = rejected_not_active` when the current run is already closed
- `409` with `outcome = rejected_not_current` when the run-targeted route addresses a historical selected run

### Describe Webhooks

Both instance-targeted and run-targeted describe routes are available for inspecting workflow state over HTTP:

```text
GET /webhooks/instances/{workflowId}/describe
GET /webhooks/instances/{workflowId}/runs/{runId}/describe
```

`workflowId` is the public workflow instance id. The instance-targeted route returns the current run's state. The run-targeted route describes a specific run by id, including historical or already-closed runs.

Describe is a read-only inspection operation. It does not create a durable command row and does not replay the workflow. It returns instance metadata, current or selected run state, summary fields, and action availability from committed projection data.

Describe example:

```bash
curl "https://example.com/webhooks/instances/order-123/describe"
```

Response for an active workflow:

```json
{
  "found": true,
  "workflow_instance_id": "order-123",
  "workflow_type": "order-workflow",
  "workflow_class": "App\\Workflows\\OrderWorkflow",
  "business_key": "order-123",
  "run": {
    "workflow_run_id": "01J...",
    "run_number": 1,
    "is_current_run": true,
    "status": "waiting",
    "status_bucket": "running",
    "closed_reason": null,
    "compatibility": "build-a",
    "connection": "redis",
    "queue": "default",
    "started_at": "2026-04-12T12:00:00+00:00",
    "closed_at": null,
    "last_progress_at": "2026-04-12T12:00:01+00:00",
    "wait_kind": "signal",
    "wait_reason": "Waiting for signal [approved-by]"
  },
  "run_count": 1,
  "actions": {
    "can_signal": true,
    "can_query": true,
    "can_update": true,
    "can_cancel": true,
    "can_terminate": true
  },
  "reason": null
}
```

Response for a terminated workflow:

```json
{
  "found": true,
  "workflow_instance_id": "order-123",
  "workflow_type": "order-workflow",
  "workflow_class": "App\\Workflows\\OrderWorkflow",
  "business_key": "order-123",
  "run": {
    "workflow_run_id": "01J...",
    "run_number": 1,
    "is_current_run": true,
    "status": "terminated",
    "status_bucket": "failed",
    "closed_reason": "terminated",
    "compatibility": "build-a",
    "connection": "redis",
    "queue": "default",
    "started_at": "2026-04-12T12:00:00+00:00",
    "closed_at": "2026-04-12T12:01:00+00:00",
    "last_progress_at": "2026-04-12T12:01:00+00:00",
    "wait_kind": null,
    "wait_reason": null
  },
  "run_count": 1,
  "actions": {
    "can_signal": false,
    "can_query": false,
    "can_update": false,
    "can_cancel": false,
    "can_terminate": false
  },
  "reason": null
}
```

Action availability reflects whether the operation can succeed right now: closed runs cannot accept commands, remote-only workflows cannot serve queries or updates locally, and non-current runs cannot receive signals, updates, cancellations, or terminations.

Run-targeted describe example:

```bash
curl "https://example.com/webhooks/instances/order-123/runs/01J.../describe"
```

The run-targeted response includes the same payload but `is_current_run` will be `false` when the selected run is historical, and all mutating actions will show `false`.

Response when the instance is not found:

```json
{
  "found": false,
  "workflow_instance_id": "order-123",
  "workflow_type": null,
  "workflow_class": null,
  "business_key": null,
  "run": null,
  "run_count": 0,
  "actions": {
    "can_signal": false,
    "can_query": false,
    "can_update": false,
    "can_cancel": false,
    "can_terminate": false
  },
  "reason": "instance_not_found"
}
```

Response when the instance exists but the selected run is not found:

```json
{
  "found": true,
  "workflow_instance_id": "order-123",
  "workflow_type": "order-workflow",
  "workflow_class": "App\\Workflows\\OrderWorkflow",
  "business_key": "order-123",
  "run": null,
  "run_count": 1,
  "actions": {
    "can_signal": false,
    "can_query": false,
    "can_update": false,
    "can_cancel": false,
    "can_terminate": false
  },
  "reason": "run_not_found"
}
```

Current HTTP response matrix for the describe webhooks:

- `200` when the instance is found, whether the run is active, closed, or historical
- `200` when the instance is found but the selected run does not exist (`reason = run_not_found`)
- `401` for webhook auth failure
- `404` when the workflow instance id does not exist (`reason = instance_not_found`)

### Control-Plane Start Webhook

A control-plane start route accepts a durable workflow type key directly, without requiring the workflow class to be locally resolvable. This is the preferred start path for an external consumer that drives workflows through type keys rather than PHP class aliases.

```text
POST /webhooks/control-plane/start
```

The request body accepts:

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| `workflow_type` | string | yes | Durable workflow type key |
| `instance_id` | string | no | Caller-supplied public workflow instance id |
| `arguments` | string | no | Codec-tagged serialized arguments |
| `connection` | string | no | Queue connection override |
| `queue` | string | no | Queue name override |
| `business_key` | string | no | Caller-supplied business key |
| `labels` | object | no | Visibility labels |
| `memo` | object | no | Non-indexed metadata |
| `duplicate_start_policy` | string | no | `reject_duplicate` or `return_existing_active` |

Example request:

```bash
curl -X POST "https://example.com/webhooks/control-plane/start" \
     -H "Content-Type: application/json" \
     -d '{
           "workflow_type": "order-workflow",
           "instance_id": "order-123",
           "arguments": "{\"orderId\":123}",
           "connection": "redis",
           "queue": "default",
           "business_key": "order-123",
           "labels": {"tenant": "acme"}
         }'
```

Response for a newly started workflow:

```json
{
  "started": true,
  "workflow_instance_id": "order-123",
  "workflow_run_id": "01J...",
  "workflow_type": "order-workflow",
  "outcome": "started_new",
  "task_id": "01J...",
  "reason": null
}
```

When `instance_id` is omitted, the engine generates a ULID-based instance id automatically.

When the workflow type key maps to a locally resolvable class through `workflows.v2.types.workflows` config or a `#[Type(...)]` attribute, the full command-contract snapshot and routing are applied at start time. When the class is not locally available, the instance is created with the type key and explicit routing from options; the command-contract snapshot and definition fingerprint are deferred to the worker that claims the first task. This means an external consumer can start workflows for type keys that only the worker fleet can resolve.

Current HTTP response matrix for the control-plane start webhook:

- `202` with `outcome = started_new` when a new run is created
- `200` with `outcome = returned_existing_active` when the existing active run is reused
- `409` with `outcome = rejected_duplicate` when the duplicate start is rejected
- `401` for webhook auth failure
- `422` when `workflow_type` is missing or `instance_id` is invalid

The `WorkflowControlPlane` contract is also available as a container-resolvable singleton for programmatic use. Resolve `Workflow\V2\Contracts\WorkflowControlPlane` from the Laravel container to call `start()`, `signal()`, `query()`, `update()`, `cancel()`, `terminate()`, and `describe()` directly from PHP code, tests, or Artisan commands.

The webhook system covers explicit alias-based start intake plus both instance-targeted and run-targeted signal, update, query, describe, repair, cancel, and terminate command webhooks, task poll routes for workflow and activity tasks, and a control-plane start route for type-key-based workflow creation.

<!-- Source: docs/features/continue-as-new.md -->

# Continue As New

The **Continue As New** pattern allows a running workflow to restart itself with new arguments.
This is useful when you need to:

* Prevent unbounded workflow history growth.
* Model iterative loops or recursive workflows.
* Split long-running workflows into smaller, manageable executions while preserving continuity.

## Using `continueAsNew`

To restart a workflow as new, call the helper function `continueAsNew(...)` from within the workflow’s `handle()` method. `continueAsNew()` is a first-class run transition.

```php
use function Workflow\V2\activity;
use function Workflow\V2\continueAsNew;
use Workflow\V2\Workflow;

class CounterWorkflow extends Workflow
{
    public function handle(int $count = 0, int $max = 3)
    {
        $result = activity(CountActivity::class, $count);

        if ($count >= $max) {
            return [
                'count' => $result,
                'workflow_id' => $this->workflowId(),
                'run_id' => $this->runId(),
            ];
        }

        return continueAsNew($count + 1, $max);
    }
}
```

In this example:

* The workflow executes an activity each iteration.
* If the maximum count has not been reached, it continues as new with incremented arguments.
* The final result is returned only when the loop completes.

When a workflow continues as new:

- The instance id stays stable across the chain; `WorkflowStub::runId()` moves forward to the newest run.
- The continued run closes as `completed` / `continued` and the next run starts immediately with a fresh run id.
- Instance-scoped signals, queries, updates, and operator commands resolve the newest durable run. Explicit run-targeted commands remain bound to their selected run.

Core 2.7.0 and compatible Servers preserve received instance-scoped signals whose
handler is declared by the run's command contract and which the old run has not
applied. They transfer with pending accepted updates in
their original command order. Signal and command IDs, argument bytes and receipt
times stay unchanged. An already applied signal is not delivered again to the
successor. Keep business state needed by the next run in the arguments you pass
to `continueAsNew()`; arbitrary local state is not copied automatically.

A run-targeted signal is never retargeted. If it cannot be applied before its
run closes, it gets the explicit rejected disposition described under
[signal acceptance and closure](./signals.md#acceptance-and-closure).
Missing legacy declarations do not authorize handoff; those signals close
explicitly on the predecessor.

Long-lived loops can shed history without inventing a new public workflow id every time the run rolls forward.

From Workflow 2.28, activities and timers the closing run never awaited are cancelled when it continues as new; children are handed to the continued run instead. See [Losers when the run closes](./concurrency.md#selection-losers-at-close).

## History Budget {#history-budget}

Workflow code can observe its current history length and a continue-as-new suggestion flag through explicit runtime context. This lets long-lived loops hand off deliberately before they hit history-budget cliffs.

In Workflow 2.26 and later, use `shouldContinueAsNewAtStep()` and `historyBudgetAtStep()`. They count only the history recorded before the current workflow step, so the same line returns the same answer on every replay. A loop that checks the budget on each iteration continues as new once, at the first iteration that crosses the threshold.

The older methods read the whole current history. Inside a loop, an earlier iteration can see a later total on replay and take a different branch than it took the first time. They are unchanged, so runs that already decided with them keep replaying.

```php
use function Workflow\V2\activity;
use function Workflow\V2\continueAsNew;
use Workflow\V2\Workflow;

class PollingWorkflow extends Workflow
{
    public function handle(int $iteration = 0)
    {
        $result = activity(PollActivity::class);

        if ($this->shouldContinueAsNew()) {
            return continueAsNew($iteration + 1);
        }

        return ['result' => $result, 'iteration' => $iteration];
    }
}
```

### Available methods

| Method | Return type | Description |
| --- | --- | --- |
| `$this->historyLength()` | `int` | Number of history events in the current run |
| `$this->historySize()` | `int` | Total size of history events in bytes |
| `$this->shouldContinueAsNew()` | `bool` | `true` when the run exceeds the configured event count or byte size budget |
| `$this->historyBudgetAtStep()` | `array` | The budget as of the current step: `history_event_count`, `history_size_bytes`, `history_fan_out`, `continue_as_new_recommended`, `pressure`, `pressure_dimensions` and `observation` (`step_prefix.v1`). Workflow 2.26 and later. |
| `$this->shouldContinueAsNewAtStep()` | `bool` | `continue_as_new_recommended` from `historyBudgetAtStep()`. Use it in loops. Workflow 2.26 and later. |

### Configuration

The continue-as-new recommendation thresholds are configurable:

```php
// config/workflows.php
'v2' => [
    'history_budget' => [
        'continue_as_new_event_threshold' => 10000,
        'continue_as_new_size_bytes_threshold' => 5242880, // 5 MB
    ],
],
```

`shouldContinueAsNew()` and `shouldContinueAsNewAtStep()` return `true` when either threshold is crossed. Run summaries also track `history_event_count`, `history_size_bytes`, and `continue_as_new_recommended` so Waterline can flag long-lived runs in fleet views.

## Metadata Carry-Forward

When a workflow continues as new, the new run inherits the previous run's metadata automatically:

| Metadata | Carry-forward behavior |
| --- | --- |
| Search attributes | Full merged map carries forward, including mid-run upserts |
| Memo | Full merged map carries forward, including mid-run upserts |
| Visibility labels | Carried forward as-is (set once at start time) |
| Business key | Carried forward as-is |
| Compatibility marker | Carried forward so the new run targets the same worker build |
| Message cursor position | Carried forward so the new run knows which inbound messages were already consumed through [Message Streams](./message-streams.md) |

The new run starts with the inherited metadata and can continue upserting search attributes and memo from there. This means a long-lived polling workflow can accumulate metadata across generations without losing state when it sheds history.

```php
use function Workflow\V2\{activity, continueAsNew, upsertMemo, upsertSearchAttributes};
use Workflow\V2\Workflow;

class PollingWorkflow extends Workflow
{
    public function handle(int $iteration = 0)
    {
        $result = activity(PollActivity::class);

        upsertSearchAttributes(['iteration' => (string) $iteration, 'status' => 'polling']);
        upsertMemo(['last_result' => $result, 'iteration' => $iteration]);

        if ($this->shouldContinueAsNew()) {
            // The new run inherits all search attributes and memo
            return continueAsNew($iteration + 1);
        }

        return ['result' => $result, 'iteration' => $iteration];
    }
}
```

A parent that continues as new while child workflows it started are still open hands them to the continued run. See [Continue-as-new and open children](./child-workflows.md#continue-as-new-and-open-children).

For long-lived human-input or assistant loops, prefer the first-class
`$this->inbox()` / `$this->outbox()` [Message Streams](./message-streams.md)
facade. Pending stream messages and the cursor position move to the continued
run, while consumed messages remain on the original run as history evidence.
Workflow 2.20.1 and earlier left pending stream messages on the closed run
([workflow#1018](https://github.com/durable-workflow/workflow/issues/1018),
fixed in 2.20.2); see [Message Streams](./message-streams.md#continue-as-new).

<!-- Source: docs/features/versioning.md -->

# Versioning

## Replay before changing builds {#replay-before-changing-builds}

Before promoting a candidate, replay representative open and recently closed histories against its exact application code. Before rollback, replay histories written by the newer build against the older artifact. Follow the [replay testing gate](../testing.md#replay-before-deployment); a compatible routing cohort or successful integrity check alone does not prove replay compatibility.

Since workflows can run for long periods, sometimes months or even years, it's common to need to make changes to a workflow definition while executions are still in progress. Without versioning, modifying workflow code that affects the execution path would cause non-determinism errors during replay.

The `getVersion()` helper function allows you to safely introduce changes to running workflows by creating versioned branch points.

## Evolving Recorded Entry Arguments {#evolving-recorded-entry-arguments}

A run keeps the arguments recorded when it started. Deploying a new definition does not rewrite that input. On replay, the runtime binds those recorded values to today's workflow entry method **before any code inside that method runs**. A `getVersion()` or `patched()` branch inside the body cannot rescue an incompatible signature: the runtime has to enter the body first.

Keep existing positional arguments in the same order and accept their recorded types. A PHP default fills an omitted argument; it does not replace an incompatible value already recorded at that position. For example, a run started with `[7]` still supplies the integer `7` after deployment:

| Entry change | Effect on the recorded input `[7]` |
| --- | --- |
| `handle(int $amount)` → `handle(int $amount, string $currency = 'USD')` | The old argument still binds; the new trailing argument can use its default. Keep replayed commands compatible too. |
| Add a required trailing argument without a default | The old run has no value for it. |
| Insert, remove or reorder leading arguments | Existing values can move into the wrong parameter. Do not assume matching parameter names repair positional input. |
| Change `int $amount` to `string $amount` | The recorded integer has not become a string. |
| Change `int $amount` to `PaymentRequest $request` | Dependency injection is not an input migration. A newly constructed DTO must not replace the recorded amount with constructor defaults. |
| Remove only a trailing parameter | PHP can accept the call, but verify that dropping its meaning does not change replayed commands or business behavior. |

Prefer an additive, optional trailing parameter or a new workflow type for a different input contract. If you need a new DTO internally, construct it explicitly from compatible recorded arguments **inside** the workflow body and version any change to durable commands there. Keep code able to replay every supported input shape until those runs no longer need it. Test the actual stored inputs against the new definition, including a cold replay after a durable wait; testing only new starts misses this deployment boundary.

### Workflow 2.7.2 Behavior {#released-embedded-entry-behavior}

In Workflow **2.7.2**, an entry arity or type error can become a terminal `WorkflowFailed` event. Some dependency-resolution errors instead leave the run open with a repeatedly failing task. A scalar changed to a newly injected DTO with default constructor values can even run with the wrong data. These are product defects tracked in [Workflow #866](https://github.com/durable-workflow/workflow/issues/866), not a safe versioning strategy.

Restoring compatible code can recover an open run that can be redelivered, but it does not reopen a run already made terminal. Neither `repair()` nor a new version branch promises to revive such a failed original. Inspect its actual status and failure first; preserve its input and history while deciding on recovery.

### Released Embedded Safeguard {#embedded-entry-argument-safeguard}

Workflow **2.7.3** validates recorded entry arguments before entering application code. Incompatible binding becomes a typed `WorkflowEntryArgumentsMismatchException` and a replay-blocked **workflow task**, with `replay_blocked_reason = workflow_entry_arguments_mismatch`; the **run stays nonterminal**. Diagnostic fields identify the workflow, entry method, parameter and expected/actual types without copying argument values. A newly injected DTO that would discard recorded input is refused too. Check the actual Workflow package installed in the embedded application. This safeguard does not reopen terminal runs created by an older release.

With that released safeguard, recover the same still-open original as follows:

1. Inspect the selected run's task diagnostic and confirm `workflow_entry_arguments_mismatch`, rather than assuming every task failure has this cause.
2. Restore a definition that accepts the original recorded arguments, or deploy a compatible adapter in the workflow body. Keep the original workflow identity, input, history and deadlines; do not edit storage to make the new signature fit.
3. Use the supported repair action for that same still-open run after compatible code is available, then verify its original work resumes and its terminal outcome is recorded once. Repair does not extend the original deadline; a deadline that has elapsed still applies.

The safeguard does **not** reinterpret exceptions thrown by the workflow body as deployment mismatches. A genuine application `TypeError` or other unhandled body failure still follows normal workflow failure behavior. It also does not supply the fleet-level legacy-branch retirement check tracked in [shared #220](https://github.com/durable-workflow/.github/issues/220).

### Service SDKs Have Separate Boundaries {#service-sdk-entry-boundaries}

An embedded Core fix does not change the SDK process that executes service-mode workflow code. The following older published SDKs predate the service entry safeguard tracked in [shared #221](https://github.com/durable-workflow/.github/issues/221):

| Published SDK | Entry-binding behavior |
| --- | --- |
| [PHP 2.5.3](https://github.com/durable-workflow/sdk-php/tree/43d39bff50ccace0a309de41306cd350751ec575) | Calls the registered handler with context followed by recorded positional input. An entry `ArgumentCountError` or `TypeError` can follow the generic workflow-failure path. |
| [Python 2.6.1](https://github.com/durable-workflow/sdk-python/tree/150c6f6bc14e3516f75cc615b2f75f093ed84ea1) | Calls `run(ctx, *start_input)`. Changed arity can fail the workflow; type annotations alone do not enforce argument types or convert recorded values. Generator-body errors are application failures too. |
| [Rust 3.5.5](https://github.com/durable-workflow/sdk-rust/tree/4d5047ba9da39785512340b0975ce28da09f0cee) | The typed adapter deserializes input before calling its fixed-signature handler. A changed request type can fail that decoding step. This differs from PHP/Python positional binding; an incidental unconsumed-history refusal is not a promised recoverable signature safeguard. |

Keep service input compatibility and actual SDK qualification separate from the embedded release. Do not catch every `TypeError` or every Rust handler error and label it replay-safe: that would hide real application failures.

### Service Entry Safeguard {#service-sdk-entry-safeguard}

**Released support.** Server 2.9.9 and PHP SDK 2.5.4, Python SDK 2.6.2 and Rust SDK 3.5.6 introduced the entry safeguards and routing/recovery behavior below. These are minimum feature versions; install current stable releases for security fixes.

The prepared SDKs validate recorded entry input before application workflow code runs. Incompatible input reports `workflow_entry_arguments_mismatch` as a failed, replay-blocked workflow **task**, without issuing `fail_workflow` or closing the business **run**. Input, workflow/run identity, committed history, build pin and original deadlines remain unchanged. PHP validates required positional arguments and declared PHP types. Python uses signature binding for arity and defaults; annotations do not become runtime type checks. Rust's typed adapters validate input decoding before calling the handler or its state factory. Raw Rust handlers remain responsible for their own application input handling.

The Server's specific task `failure_reason` is `workflow_entry_arguments_mismatch`; `replay_blocked_reason` remains the generic `worker_reported_replay_failure`. Inspect `latest_workflow_task_failure.reason` in the selected run's debug response. Do not confuse these fields with the embedded Core diagnostic above. Successfully bound application code that throws a `TypeError`, returns a Rust handler error or otherwise fails still follows ordinary business-failure behavior.

Restore code that accepts the original recorded input, then use the [supported same-original recovery procedure](../failures-and-recovery.md#service-entry-recovery). Do not replace input with DTO defaults, alter history or reset a deadline. This safeguard does not reopen an original that an older worker already made terminal.

Safe replacement routing also needs the [entry-validation worker capability](../polyglot/worker-compatibility-routing.md#definition-fingerprint-evolution). A changed fingerprint is neither proof of replay compatibility nor a reason to rewrite a run's original fingerprint. Version durable-command changes inside a compatible entry method and test cold replay before promotion.

## Evolving Activity Arguments and Results {#evolving-activity-arguments-and-results}

An activity has two stored contracts: the arguments recorded when it was scheduled and the result recorded when it completed. A deployment does not migrate either. This applies to embedded PHP and service workers using the PHP, Python and Rust SDKs.

A pending attempt or retry receives the original scheduled arguments. It runs the activity implementation selected by the worker's registration and build-routing policy, which may still be an older build. Deploying a new activity implementation does not guarantee every attempt uses it, and does not update queued arguments.

When a completed activity is replayed, the current workflow code receives its recorded result without running the activity again. Matching an activity step and type is not a schema check of its arguments or result. A changed payload shape can therefore fail application code without a history mismatch.

### Add Optional Input Without Reinterpreting Old Values {#additive-activity-input}

Keep existing positional arguments and their meaning. In embedded PHP, an optional trailing parameter lets an old scheduled input such as `[7]` continue to bind:

```php
use Workflow\V2\Activity;

final class PriceActivity extends Activity
{
    public function handle(int $amount, string $currency = 'USD'): array
    {
        return ['amount' => $amount, 'currency' => $currency];
    }
}
```

The default fills a missing argument. It does not convert an incompatible recorded value or repair a reordered argument. Keep the activity's external effects compatible too: a retry can happen after the deployment and must remain idempotent.

PHP service handlers receive their activity context followed by recorded positional arguments. Python handlers receive the recorded positional arguments directly. Both need defaults for omitted arguments. Python annotations alone do not validate or convert values. Rust typed activities deserialize one request value; add an optional field or a field with an explicit Serde default while preserving the existing representation. A newly required field can fail decoding before the Rust handler runs. See the [PHP](https://php.durable-workflow.com/), [Python](https://python.durable-workflow.com/) and [Rust](https://rust.durable-workflow.com/) SDK guides for their activity APIs.

### Read Older Results With a Fallback {#additive-activity-result}

When a new activity result adds `currency`, workflows must still accept older completed results that contain only `amount`:

```php
$result = activity(PriceActivity::class, 7);
$currency = $result['currency'] ?? 'USD';
```

Choose a fallback that preserves the old business meaning. Retain it for every history that may be replayed, including queries and recovery. Deploy compatible readers before writers that emit a changed representation, and verify rollback readers against newly recorded results too. A default on the activity's input does not add a field to an already recorded result.

### Use a New Type for a Different Contract {#incompatible-activity-contract}

For a renamed field, a different representation or a changed business meaning, keep the old activity type available and introduce a new one. Use a workflow `getVersion()` branch at the scheduling call when old histories must keep their old activity step and new runs should schedule the new type. A branch inside the activity cannot fix arguments that fail before its body runs, and a version marker does not rewrite results already in history.

Test two cold deployment boundaries against the actual installed packages: schedule with old code and execute with candidate activity code; then complete with old activity code and replay with candidate workflow code. Include incompatible inputs and results as failure controls. Do not assume the workflow-entry safeguards above also protect activity signatures or application result access. The confirmed embedded activity DTO defect is tracked in [Workflow #910](https://github.com/durable-workflow/workflow/issues/910).

### Verified Failure Boundaries {#activity-compatibility-failures}

The [published-package checks in #363](https://github.com/durable-workflow/durable-workflow.github.io/issues/363) used fresh processes across each deployment boundary. These are observed baselines, not minimum supported versions:

| Published package | Incompatible activity input | Incompatible result reader |
| --- | --- | --- |
| Embedded Workflow 2.11.2 | A missing required argument produces `ArgumentCountError` and an activity failure. | Accessing a missing array field under Laravel error handling produces `ErrorException` and workflow failure. |
| PHP SDK 3.2.0 with Server 2.11.0 | A missing required positional argument is reported as `ArgumentCountError` in the activity failure. | Passing a missing field to a required string reader produces `TypeError` and workflow failure. |
| Python SDK 2.9.0 with Server 2.11.0 | A missing required positional argument produces `TypeError` and an activity failure. | Indexing a missing result key produces `KeyError` and workflow failure. |
| Rust SDK 4.2.2 with Server 2.11.0 | A newly required request field fails typed input decoding before the activity handler runs. | A newly required result field fails typed result decoding in the workflow. |

The service checks recorded `max_attempts = 1`; the unhandled activity failure then failed the workflow. In general, activity failures follow their recorded retry and timeout policy. A workflow can handle an exhausted activity failure. A result-reader error is an application workflow failure, not a request to rerun the completed activity. All additive controls completed with the original amount `7` and default currency `USD`, with one activity attempt and no repairs. In result controls, the completed activity stayed completed.

Embedded Workflow 2.11.2 also accepts a changed activity signature that replaces recorded scalar `7` with an injected DTO whose default amount is `0`, and can incorrectly complete with `0`. [Workflow #910](https://github.com/durable-workflow/workflow/issues/910) tracks that product defect. Do not use dependency injection as an activity-input migration or assume the workflow-entry safeguard covers it. Restoring compatible code does not reopen an already terminal run; inspect the actual run and follow the [recovery guide](../failures-and-recovery.md) without changing recorded input, history or deadlines.

Review [recorded workflow start arguments](#evolving-recorded-entry-arguments) and the [signal payload evolution recipe](signals.md#evolve-a-signal-payload) alongside these activity contracts.

### Released embedded activity guard {#embedded-activity-argument-safeguard}

Workflow **2.11.3 is the fixed minimum** for the embedded PHP guard against activity binding that displaces recorded arguments. In queued and local activity paths, replacing recorded scalar `7` with an injected DTO whose default is `0` is refused before the activity body runs, with `ActivityEntryArgumentsMismatchException`. The [published, installed cold-process receipt](https://github.com/durable-workflow/workflow/issues/910#issuecomment-6362080363) verifies refusal and recovery with the original input. The older 2.11.2 behavior above remains affected. This guard does not add fields to older activity results; compatible readers still need their fallback.

The refusal follows the activity's **recorded attempt, backoff and timeout policy**. With one allowed attempt, it exhausts the activity and an unhandled failure fails the workflow. If authored attempts remain and compatible code returns before the permitted retry and deadlines, the same original run/activity can complete with its recorded input. The receipt verifies two authored attempts with zero backoff and output `7`; it does not promise an indefinite hold for a future deployment or grant extra attempts.

Restoring code or installing the package does not reopen terminal originals, reset deadlines or restore consumed attempts. Inspect the actual failure and preserve recorded input/history. Check the Workflow package installed in the embedded application: this library release does not update existing Server images or establish service-worker/native acceptance.

## Using `getVersion`

`getVersion()` is a replay-safe straight-line helper. Each change point records a durable version marker for the run on first execution, and every later replay reuses that committed value instead of recalculating the branch from today's code. Old runs that predate a newly introduced branch point conservatively receive `DEFAULT_VERSION`.

```php
use Workflow\V2\Workflow;
use Workflow\V2\WorkflowStub;
use function Workflow\V2\{activity, getVersion};

final class MyWorkflow extends Workflow
{
    public function handle(): void
    {
        $version = getVersion(
            'my-change-id',
            WorkflowStub::DEFAULT_VERSION,
            1
        );

        if ($version === WorkflowStub::DEFAULT_VERSION) {
            activity(OldActivity::class);
        } else {
            activity(NewActivity::class);
        }
    }
}
```

## How It Works

The `getVersion()` method takes three parameters:

- **changeId** - A unique identifier for this change point
- **minSupported** - The minimum version this code still supports
- **maxSupported** - The maximum (current) version for new executions

When a workflow encounters `getVersion()`:

- **New executions** append a typed `VersionMarkerRecorded` history event with the `change_id`, chosen `version`, and supported range, then return `maxSupported`
- **Replaying executions** return the previously recorded version from that history marker
- **Runs without a marker yet** fall back to `WorkflowStub::DEFAULT_VERSION` when the runtime determines the branch predates the current workflow definition. It first checks the run's compatibility marker against the current worker, then compares the run's durably snapped `workflow_definition_fingerprint` to the current loadable workflow class. If the fingerprints match, the run is on the same definition and the marker is recorded fresh; if they differ, the run predates the change and the fallback fires. Runs whose `WorkflowStarted` history predates the fingerprint snapshot (no recorded fingerprint) conservatively receive `DEFAULT_VERSION` because the runtime cannot prove the definition has not changed. This conservative fallback is the frozen 2.0 policy for pre-fingerprint runs: the runtime does not block claim solely because an older run lacks a recorded fingerprint. The fallback does not append a synthetic marker or consume a new workflow step
- **Query methods** replay the same committed version marker before invoking the annotated method, so queries see the same branch the workflow task saw
- **Waterline** exposes recorded markers in the selected-run timeline with `version_change_id`, `version`, `version_min_supported`, and `version_max_supported`, and selected-run detail now also surfaces `workflow_definition_fingerprint`, `workflow_definition_current_fingerprint`, and `workflow_definition_matches_current` so operators can see when a long-lived run started on an older definition even before a version marker was committed

This allows new workflows to use the latest code path while existing workflows continue using their original path.

## Adding a New Version

Suppose you have an existing workflow that calls `prePatchActivity`:

```php
use Workflow\V2\Workflow;
use function Workflow\V2\activity;

final class MyWorkflow extends Workflow
{
    public function handle()
    {
        $result = activity(PrePatchActivity::class);

        return $result;
    }
}
```

To replace it with `postPatchActivity` without breaking running workflows:

```php
use Workflow\V2\Workflow;
use Workflow\V2\WorkflowStub;
use function Workflow\V2\{activity, getVersion};

final class MyWorkflow extends Workflow
{
    public function handle()
    {
        $version = getVersion(
            'activity-change',
            WorkflowStub::DEFAULT_VERSION,
            1
        );

        $result = $version === WorkflowStub::DEFAULT_VERSION
            ? activity(PrePatchActivity::class)
            : activity(PostPatchActivity::class);

        return $result;
    }
}
```

When you roll out a deployment that introduces a new `getVersion()` branch point, keep rotating `DW_V2_CURRENT_COMPATIBILITY` for that build wave. The runtime uses the run's start-time `workflow_definition_fingerprint` as its primary authority when deciding whether a missing version marker belongs to a fresh execution or to an older run that should stay on `DEFAULT_VERSION`. The compatibility marker still matters for compatibility-aware worker routing and fires before the fingerprint check. Runs that predate the fingerprint snapshot (no recorded fingerprint) conservatively stay on `DEFAULT_VERSION` since the runtime cannot verify the definition has not changed.

## Adding More Versions

When you need to make additional changes, increment `maxSupported`:

```php
$version = getVersion(
    'activity-change',
    WorkflowStub::DEFAULT_VERSION,
    2
);

$result = match($version) {
    WorkflowStub::DEFAULT_VERSION => activity(PrePatchActivity::class),
    1 => activity(PostPatchActivity::class),
    2 => activity(AnotherPatchActivity::class),
};
```

## Deprecating Old Versions

After all workflows using an old version have completed, you can drop support by increasing `minSupported`. This removes the need to maintain old code paths.

```php
// After all DEFAULT_VERSION workflows have completed:
$version = getVersion(
    'activity-change',
    1,  // No longer supporting DEFAULT_VERSION
    2
);

$result = match($version) {
    1 => activity(PostPatchActivity::class),
    2 => activity(AnotherPatchActivity::class),
};
```

If a workflow with a version older than `minSupported` tries to replay, it will throw a `VersionNotSupportedException`. That includes older-compatibility runs whose safe fallback is still `WorkflowStub::DEFAULT_VERSION`.

If you accidentally reuse the same `changeId` for a different branch point later, the runtime treats that as a determinism error instead of silently accepting the mismatch. Keep one stable `changeId` per logical code change.

## Multiple Change Points

You can use multiple `getVersion()` calls in the same workflow for independent changes:

```php
use Workflow\V2\Workflow;
use Workflow\V2\WorkflowStub;
use function Workflow\V2\getVersion;

final class MyWorkflow extends Workflow
{
    public function handle(): void
    {
        $version1 = getVersion('change-1', WorkflowStub::DEFAULT_VERSION, 1);
        $version2 = getVersion('change-2', WorkflowStub::DEFAULT_VERSION, 1);

        // Each change point is tracked independently
    }
}
```

**Important:** Each `changeId` should be unique within a workflow. The chosen version is recorded as typed workflow history and replayed deterministically on later workflow tasks, queries, and Waterline detail views. When Waterline shows no `VersionMarkerRecorded` entry for a change point on an older run, that means replay stayed on the legacy `DEFAULT_VERSION` path without backfilling a new marker into existing history. The selected-run detail fields `workflow_definition_fingerprint`, `workflow_definition_current_fingerprint`, and `workflow_definition_matches_current` tell you whether that run started on a different workflow definition than the one your current build can load today.

## `patched()` Shorthand

`patched($changeId)` is a two-state shorthand for the common "did this run cross a one-time code change?" question. It records the same durable `VersionMarkerRecorded` history event as `getVersion()` and resolves to a boolean instead of an integer.

```php
use Workflow\V2\Workflow;
use function Workflow\V2\{activity, patched};

final class MyWorkflow extends Workflow
{
    public function handle(): string
    {
        if (patched('use-new-payment-activity')) {
            return activity(NewPaymentActivity::class);
        }

        return activity(LegacyPaymentActivity::class);
    }
}
```

Behavior:

- New runs commit the marker, take the new branch, and `patched()` returns `true`.
- Runs that started before this `changeId` was added stay on `DEFAULT_VERSION` under the same fingerprint fallback `getVersion()` uses, and `patched()` returns `false`.
- Replays of either kind read the previously committed marker and return the same value, so the branch decision is durable.

`patched()` is exactly equivalent to `getVersion($changeId, DEFAULT_VERSION, 1) === 1`. Use it when you only have two branches and you do not need to keep the legacy branch around forever — the boolean spelling reads cleaner at the call site than a `match` over `DEFAULT_VERSION` and `1`.

If you need more than two branches, or you plan to add another version of the same logical change later, use `getVersion()` with an explicit `maxSupported`. Switching from `patched()` to `getVersion()` for the same `changeId` is a determinism error because the existing marker was recorded with `maxSupported = 1`.

## Removing the Legacy Branch with `deprecatePatch()`

When every legacy run for a `patched()` change point has finished, you can delete the legacy branch from your workflow code. The compatible way to do that is to replace the `patched()` call with `deprecatePatch()` rather than removing the call entirely.

```php
use Workflow\V2\Workflow;
use function Workflow\V2\{activity, deprecatePatch};

final class MyWorkflow extends Workflow
{
    public function handle(): string
    {
        deprecatePatch('use-new-payment-activity');

        return activity(NewPaymentActivity::class);
    }
}
```

`deprecatePatch()` keeps the change point on the workflow timeline so already-committed `VersionMarkerRecorded` history events still match a known call, but it returns `null` and unconditionally takes the new path. Once you ship this version:

- New runs commit a `deprecate_patch` marker, take the new branch, and the `deprecatePatch()` call returns `null`.
- Existing runs that already committed a `patched` marker for this `changeId` keep replaying that marker and resolve through the deprecated branch — which is now the only branch — without erroring.
- Existing runs that committed `false` for `patched()` (the legacy path) are no longer in flight by assumption. If one is still running and tries to replay, the workflow will still read the legacy marker but execute the new branch — which is why the safe lifecycle is "wait for legacy runs to drain before deploying `deprecatePatch()`".

Two-phase patch lifecycle and placement rules:

1. **Introduce `patched()`** at the change point. Both branches stay in the workflow code. New runs take the new branch; older runs keep taking the legacy branch.
2. **Wait for every legacy run to finish.** Use Waterline run search or `dw workflow:list` to confirm there are no open runs that could replay through the legacy branch.
3. **Replace `patched()` with `deprecatePatch()`** at the same call site, with the same `changeId`, and delete the legacy branch from the function body. Keep the call there — do not remove it — so durable history still matches a known change point.
4. **Optional: remove `deprecatePatch()` entirely** once you are also confident no historical replay (queries, exports, audits) will ever read history that committed this marker. That is an irreversible step; most workflows can leave `deprecatePatch()` in place indefinitely without paying any runtime cost.

Placement rules:

- `patched()` and `deprecatePatch()` must be called from the workflow function body, not from activities, signal handlers, or query methods. They are workflow steps and must replay deterministically.
- Each `changeId` is one logical change point. Reusing the same `changeId` for a different branch is a determinism error.
- A `changeId` can appear in `patched()` or `deprecatePatch()` form during its lifetime, but never both at the same time. The deploy that swaps the call must replace, not coexist.
- `getVersion()` and `patched()` cannot coexist for the same `changeId`. Pick one model when you introduce the change point.

<!-- Source: docs/features/cancel-and-terminate.md -->

# Cancel and Terminate

This guide covers the embedded Laravel `Workflow\V2\WorkflowStub` API. For
service-mode clients and workers, use the
[PHP SDK guide](https://php.durable-workflow.com/),
[Python cancellation reference](https://python.durable-workflow.com/reference/errors/#durable_workflow.errors.ActivityCancelled), or
[Rust lifecycle API](https://rust.durable-workflow.com/durable_workflow/struct.Client.html#method.cancel_workflow).

Cancel and terminate are first-class durable commands that close a running workflow. Both are recorded in command history, appear in typed history events, and surface in Waterline.

In the current embedded API, both commands close the run immediately. **Cancel**
records a `cancelled` outcome; **terminate** records a `terminated` outcome.
Neither command schedules cleanup inside the closed workflow. Embedded Laravel
also provides `requestCancellation()` for bounded cooperative cleanup.
Server and the PHP, Python and Rust SDKs support the separate cooperative
request with workers that explicitly enable it. See
[Cooperative Cancellation](/docs/polyglot/cancellation) for the capability,
deadline, cleanup and recovery contract.

## Cancel

Cancel immediately transitions the run to `cancelled` and records durable history.

```php
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::load('order-123');

$result = $workflow->cancel();

$result->accepted();   // true
$result->outcome();    // "cancelled"
$result->commandId();  // Durable command id
$result->reason();     // null (no reason provided)
```

### Cancel with reason

You can provide a structured reason to distinguish user-driven, policy-driven, and operator-driven cancellation in audit trails, command history, and Waterline.

```php
$result = $workflow->cancel('Customer requested cancellation');

$result->reason(); // "Customer requested cancellation"
```

The reason is persisted on the durable command record and in both the `CancelRequested` and `WorkflowCancelled` typed history events, so it survives replay, export, and offline analysis.

### What cancel does

When a cancel command is accepted, the engine closes any open activity executions and pending timers, transitions the run to `cancelled`, and resumes a parent workflow waiting on the cancelled child so it can observe the outcome.

### Cancel is rejected when

- The instance has not started yet: `rejection_reason = instance_not_started`
- The current run is already closed: `rejection_reason = run_not_active`
- A run-targeted cancel addresses a historical (non-current) run: `rejection_reason = selected_run_not_current`

## Terminate

Terminate immediately closes a running workflow without scheduling further workflow code. Like cancel, it does not provide a cleanup window; use it when the distinct `terminated` outcome is appropriate.

```php
$result = $workflow->terminate();

$result->accepted();   // true
$result->outcome();    // "terminated"
```

### Terminate with reason

```php
$result = $workflow->terminate('Operator emergency shutdown');

$result->reason(); // "Operator emergency shutdown"
```

### What terminate does

Terminate follows the same transactional steps as cancel, but:

- Records `TerminateRequested` and `WorkflowTerminated` history events instead.
- Creates the `WorkflowFailure` row with `failure_category = terminated` and `propagation_kind = terminated`.
- Sets `closed_reason = terminated` on the run.
- Does not schedule any further workflow-code execution.

### Terminate is rejected when

Same rejection conditions as cancel.

## Run-targeted commands

Both cancel and terminate can target a specific run instead of the instance's current run.

```php
$selectedRun = WorkflowStub::loadRun($runId);
$result = $selectedRun->attemptCancel('Draining old run');

$result->targetScope(); // "run"
```

Run-targeted commands reject with `selected_run_not_current` when the addressed run is no longer the instance's current run. The response includes `requested_run_id` (the run the caller addressed) and `resolved_run_id` (the current run that should be used next).

## Non-throwing API

Use `attemptCancel()` and `attemptTerminate()` when you want to handle rejection without exceptions:

```php
$result = $workflow->attemptCancel('Duplicate order');

if ($result->rejected()) {
    $result->rejectionReason(); // e.g. "run_not_active"
}
```

`cancel()` and `terminate()` throw `LogicException` on rejection.

## Webhooks

Cancel and terminate are available through webhook routes:

```text
POST /webhooks/instances/{workflowId}/cancel
POST /webhooks/instances/{workflowId}/terminate
POST /webhooks/instances/{workflowId}/runs/{runId}/cancel
POST /webhooks/instances/{workflowId}/runs/{runId}/terminate
```

Pass the reason in the request body:

```json
{
  "reason": "Operator: duplicate order"
}
```

The response includes the command outcome, the public instance id, and the reason:

```json
{
  "outcome": "cancelled",
  "workflow_id": "order-123",
  "run_id": "01J10000000000000000000021",
  "reason": "Operator: duplicate order",
  "command_id": "01J40000000000000000000021",
  "command_status": "accepted"
}
```

## Protected cleanup after a cancellation request {#protected-cleanup}

`requestCancellation()` keeps the run active. Workflow code receives the request at its next durable call as a `Workflow\V2\Exceptions\WorkflowCancellationRequestedException`. Like `WorkflowCancelledException`, it extends `\Error`, so `catch (\Exception $e)` does not catch it. Catch it by name or use `finally`.

Run cleanup that needs durable calls inside `cancellationShield()`. Inside the shield, activities, timers and child workflows run normally instead of receiving the request again:

```php
use function Workflow\V2\{activity, cancellationShield};
use Workflow\V2\Exceptions\WorkflowCancellationRequestedException;
use Workflow\V2\Workflow;

final class ShipOrderWorkflow extends Workflow
{
    public function handle(string $orderId): string
    {
        $reservationId = activity(ReserveStockActivity::class, $orderId);

        try {
            activity(ShipOrderActivity::class, $orderId);
        } catch (WorkflowCancellationRequestedException $request) {
            cancellationShield(fn () => activity(ReleaseStockActivity::class, $reservationId));

            throw $request;
        }

        return 'shipped';
    }
}
```

Rethrowing the request, or returning normally, closes the run as `cancelled`. The shield does not extend the cleanup deadline set by `requestCancellation()`, and termination and the run and execution timeouts still close the run. Saga `compensate()` runs under a shield automatically. See [Sagas](./sagas.md). Service-mode SDKs have the same shield. See [Cooperative Cancellation](/docs/polyglot/cancellation) and each SDK site.

## Cancellation is not an error you catch by accident

Cancellation is an explicit lifecycle outcome, not an unexpected application
error. The embedded package's
`Workflow\V2\Exceptions\WorkflowCancelledException` extends `\Error`, not
`\Exception`. A `catch (\Exception $e)` block will not catch it; a
`catch (\Throwable $t)` block will.

When reading a cancelled workflow's result, catch the exception by name if you
need to distinguish cancellation from other failures. Catching a result-side
exception does not reopen the cancelled run or schedule cleanup inside it.

## Waterline

Waterline exposes cancel and terminate as operator actions on the selected-run detail view. The detail payload includes `can_cancel` and `can_terminate` flags driven from durable state.

The command history view shows each cancel or terminate command with its reason, caller identity, and outcome. The `commands[*].reason` field carries the operator- or caller-provided reason when one was supplied.

## Status mapping

Both `cancelled` and `terminated` runs land in the `failed` status bucket for list routing. The raw `status` and `closed_reason` fields distinguish them:

| Status | Status bucket | Closed reason |
| --- | --- | --- |
| `cancelled` | `failed` | `cancelled` |
| `terminated` | `failed` | `terminated` |

## Typed history

A cancelled run produces this history sequence:

```
StartAccepted
WorkflowStarted
... (workflow progress) ...
CancelRequested       <- reason field present when supplied
TimerCancelled        <- for each open timer
ActivityCancelled     <- for each open activity
WorkflowCancelled     <- failure_id, failure_category, reason when supplied
```

A terminated run produces:

```
StartAccepted
WorkflowStarted
... (workflow progress) ...
TerminateRequested    <- reason field present when supplied
TimerCancelled        <- for each open timer
ActivityCancelled     <- for each open activity
WorkflowTerminated    <- failure_id, failure_category, reason when supplied
```

## Cancel vs. terminate

| | Cancel | Terminate |
| --- | --- | --- |
| Further workflow code is scheduled | No | No |
| Open activities cancelled | Yes | Yes |
| Open timers cancelled | Yes | Yes |
| Reason metadata | Yes | Yes |
| Durable command history | Yes | Yes |
| Parent receives child outcome | Yes | Yes |
| Failure row recorded | Yes (`cancelled`) | Yes (`terminated`) |
| Status bucket | `failed` | `failed` |
| Closed reason | `cancelled` | `terminated` |

<!-- Source: docs/features/search-attributes.md -->

# Search Attributes

Search attributes are typed, indexed key-value pairs that a workflow can upsert at any point during execution. Unlike visibility labels (which are set once at start time), search attributes can be updated as the workflow progresses, making them ideal for tracking workflow status, customer identifiers, or any operator-visible metadata that changes over the lifetime of a run.

## Upserting Search Attributes

`upsertSearchAttributes()` is a durable straight-line helper. Each call records a typed `SearchAttributesUpserted` history event and merges the new attributes into the run's persisted search attributes.

```php
use Workflow\V2\Workflow;
use function Workflow\V2\{activity, upsertSearchAttributes};

final class OrderWorkflow extends Workflow
{
    public function handle(string $orderId, string $customer): array
    {
        upsertSearchAttributes([
            'status' => 'processing',
            'customer' => $customer,
            'order_id' => $orderId,
        ]);

        $result = activity(ProcessOrderActivity::class, $orderId);

        upsertSearchAttributes([
            'status' => 'completed',
            'result' => $result->outcome,
        ]);

        return $result->toArray();
    }
}
```

## Setting Search Attributes at Start Time

Search attributes can also be set when starting a workflow via `StartOptions::withSearchAttributes()`. Start-time attributes are recorded on the run immediately and appear in the `StartAccepted` and `WorkflowStarted` history events. Any subsequent `upsertSearchAttributes()` calls merge on top of them.

```php
use Workflow\V2\StartOptions;
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::make(OrderWorkflow::class, 'order-42');
$workflow->start(
    $orderId,
    $customer,
    StartOptions::rejectDuplicate()->withSearchAttributes([
        'env' => 'production',
        'region' => 'us-east',
        'priority' => 'high',
    ]),
);
```

You can chain `withSearchAttributes()` with other `StartOptions` builders:

```php
$options = StartOptions::rejectDuplicate()
    ->withBusinessKey('order-42')
    ->withLabels(['tenant' => 'acme'])
    ->withMemo(['note' => 'VIP order'])
    ->withSearchAttributes(['priority' => 'high']);

$workflow->start($orderId, $customer, $options);
```

Start-time search attributes follow the same key rules and limits as `upsertSearchAttributes()`. Null values are dropped (not persisted). A start over a limit is refused before anything is written.

### Control plane

The control plane `start()` method also accepts search attributes:

```php
$controlPlane->start('orders.process-order', 'order-42', [
    'arguments' => $serializedArgs,
    'search_attributes' => ['env' => 'production', 'priority' => 'high'],
]);
```

## How It Works

The `upsertSearchAttributes()` function accepts an associative array of key-value pairs:

- **Keys** must be 1-64 characters from letters, digits, `.`, `_`, `-` and `:`
- **Values** are a string, int, float, bool, list of strings, or null. See [Privacy and Limits](#privacy-and-limits) for size limits
- Passing `null` as a value removes that key from the search attributes

Each call:

1. Suspends the workflow fiber and yields an `UpsertSearchAttributesCall` command
2. The executor validates and normalizes the attributes (keys are sorted alphabetically)
3. A `SearchAttributesUpserted` history event is appended with the upserted `attributes` and the full `merged` result
4. The run's `search_attributes` column is updated with the merged map
5. The run summary projection is refreshed so Waterline reflects the change immediately

On replay, recorded search attribute events are reused without re-executing the upsert, preserving determinism.

## Registering attributes {#registering-attributes}

On the Server you can register a search attribute with a type through `POST /api/search-attributes`. A registered key must then carry a value of its type, and a start or upsert with the wrong type is refused.

Registration only constrains the keys you register. From native Server 3.5.0, a key that is not registered is still accepted and its type is inferred, as in a namespace with no registered attributes. Registering one attribute does not break running workflows that upsert other keys. PHP Server 2.x and native Server before 3.5.0 refuse every unregistered key once any attribute is registered, so register all the keys your workflows use before registering the first one there.

Deleting a registration stops checking that key. Stored values and running workflows are not changed, and later upserts of the key are inferred again. On native Server, a list query that filters on a key that is not registered, including one whose registration you deleted, is refused because the key is not defined in the namespace.

To change an attribute's type, register a new attribute name with the new type, update your workflows to write it, and filter on the new name once existing runs have caught up. Deleting and re-registering the same name with a different type leaves earlier values in the old type.

## Merging Behavior

Search attributes merge across multiple upserts within the same run. Each upsert overlays new keys on top of existing ones:

```php
// First upsert
upsertSearchAttributes(['status' => 'processing', 'customer' => 'Taylor']);
// search_attributes = { customer: Taylor, status: processing }

// Second upsert
upsertSearchAttributes(['status' => 'completed', 'result' => 'success']);
// search_attributes = { customer: Taylor, result: success, status: completed }
```

To remove a key, set it to `null`:

```php
upsertSearchAttributes(['temporary_flag' => null]);
```

## Visibility and Filtering

Search attributes appear in:

- **Run detail view** — the merged search attributes map is displayed alongside labels and memo
- **Run summary projection** — search attributes are included in the denormalized summary for fast reads
- **Visibility filters** — operators can filter workflow runs by search attribute values in Waterline saved views
- **History timeline** — each `SearchAttributesUpserted` event appears as a typed entry in the run timeline
- **History export** — search attributes are included in JSON history exports

For Server-side typed comparisons, time ranges, membership, quoting and HTTP pagination, use the [Visibility Query Language](../polyglot/visibility-query-language.md). Current embedded listing and Waterline saved-view filters do not implement that full grammar. Shared parity remains pending in [the visibility-query feature](https://github.com/durable-workflow/.github/issues/224).

### Parent and child list filters {#parent-child-list-filters}

Children [inherit search attributes and memo at creation](./child-workflows.md#child-metadata-inheritance), then can upsert their own values independently. Filtering an inherited attribute does not implicitly exclude children. For a registered keyword attribute, the Server query `customer = 'customer-42'` can return the parent and every child still carrying that value. Memo remains unindexed metadata; it is not another search-attribute filter.

The qualified Server 2.11.0 list API has **no built-in roots-only or children-only predicate**. `ParentWorkflowId`, `RootWorkflowId` and `IsChildWorkflow` are not automatic relationship fields in its query language; the published checks reject those unregistered names. Registering a custom attribute with such a name does not populate parent/child identity. A workflow-type filter selects a type, not every root or child across types.

Inspect the run's lineage when you need its actual parent/child relationship. Broader shared visibility filtering remains [product work](https://github.com/durable-workflow/.github/issues/224); this limitation and a user-authored metadata convention do not deliver that capability. The [verification receipt](https://github.com/durable-workflow/durable-workflow.github.io/issues/368#issuecomment-6362355504) preserves the observed parent-and-child result and rejected predicates.

## Repairing datetime search attributes {#repair-datetime-search-attributes}

The history-backed repair command requires an actually installed **Workflow 2.12.0 or newer**. Check `composer.lock` and `php artisan help workflow:v2:repair-search-datetimes` in the Laravel application that will run it. A source checkout or Composer constraint alone does not establish the installed command.

Datetime projections store UTC instants with microsecond precision: `2026-10-08T12:34:56.123456+02:00` is `2026-10-08T10:34:56.123456Z`. Earlier writers could persist the offset’s local wall time while retaining its offset in typed history. Upgrading prevents new writer errors but does not identify or rewrite old rows. Never infer an original timezone from the stored wall time.

Upgrade every projection writer and back up the workflow database first. Use the exact **run ID**, not a workflow instance ID. Start with a preview:

```sh
php artisan workflow:v2:repair-search-datetimes RUN_ID --namespace=production --json
php artisan workflow:v2:repair-search-datetimes RUN_ID --namespace=production --key=SubmittedAt --json
```

`--namespace=production` is an exact equality guard; use the run’s actual namespace. Omit it only when the exact run ID is sufficient, including legacy null namespaces; an empty explicit namespace is refused. Without `--key`, the selection covers that run’s datetime attributes. Repeat the option for known keys, such as `--key=SubmittedAt --key=ApprovedAt`.

Preview is the default. Review `examined`, `would_repair` and the exact run/key selection before explicitly applying the same selection:

```sh
php artisan workflow:v2:repair-search-datetimes RUN_ID --namespace=production --key=SubmittedAt --apply --json
```

Read the selected attributes and run summary back, then preview again. With unchanged evidence and already-correct values, a repeat apply reports zero repairs. A repaired subset does not establish that unselected keys are correct.

Repair requires the latest retained, explicitly typed `SearchAttributesUpserted` evidence for each key, with its original logical command sequence matching the projection. Pruned history, missing/stale evidence, timezone-ambiguous values, wrong namespace and selected keys no longer retained as datetime attributes refuse the operation. One unprovable key refuses the entire selection; inspect the history and select only proven keys instead of guessing an offset.

The command locks and validates the complete selection before updating derived attributes and their summary in one storage transaction. Authoritative history, business commands, run/task identities and leases remain unchanged. JSON failures distinguish `refused` from `unverified`: after an unexpected storage error, inspect the rows before retrying because an uncertain commit is not a guaranteed rollback.

This repairs derived search attributes and their summary; it does not restore pruned history or change authoritative input. The separate Server visibility-query binding path and Server image adoption require their own qualified delivery; a Core package tag does not establish an HTTP fix. See the [canonical recovery procedure](https://github.com/durable-workflow/workflow/blob/b3bbfdb5258c68546ae38ca1fde52818e4b05d12/docs/architecture/projection-timestamp-storage.md#datetime-search-attributes).

## Continue-as-New

When a workflow continues as new, the current search attributes are carried forward to the new run automatically. The new run starts with the full merged attribute map from the previous run and can continue upserting from there.

## Privacy and Limits

Search attributes are **plain-text operator metadata** — their values are visible to anyone with Waterline access and are stored unencrypted. Never store secrets, passwords, tokens, or personally identifiable information (PII) in search attributes.

Constraints:

- Key names: 1-64 characters from letters, digits, `.`, `_`, `-` and `:`. On Server a search attribute must be registered first, and registered names start with a letter and use only letters, digits and `_` (up to 128 characters). Use that form for keys that work in both modes.
- Values: strings up to 2,048 characters, keywords up to 255 characters, lists of keywords, ints, floats, bools and datetimes.
- Per run: at most 100 attributes and 65,536 bytes in total. The [`search_attribute_size_bytes`](../constraints/structural-limits.md#search-attribute-size) limit (default 40 KiB) can tighten the total but not raise it.
- Going over a limit raises `StructuralLimitExceededException`, whose `limitName` says which limit. A start is refused without writing anything, and an upsert from workflow code fails the run as a `structural_limit` failure.
- Each upsert appends a `SearchAttributesUpserted` history event, so upserting in a loop grows the run's history.
- Setting a value to `null` removes that key

## Search Attributes vs. Memo vs. Visibility Labels

| | Search Attributes | Memo | Visibility Labels |
|---|---|---|---|
| **Set when** | Start time or any time during execution | Start time or any time during execution | Start time only |
| **Mutable** | Yes, via `upsertSearchAttributes()` | Yes, via `upsertMemo()` | No |
| **Value types** | Scalars and lists of strings | Any JSON-serializable | Scalar only |
| **Indexed** | Yes | No | Yes |
| **Filterable** | Yes | No | Yes |
| **Use case** | Dynamic status, progress tracking | Rich metadata, notes, context | Static classification |
| **History events** | `SearchAttributesUpserted` per upsert | `MemoUpserted` per upsert | None (set on start) |

For rich, structured metadata that does not need to be filtered or sorted, use [memo](./memo.md) instead.

<!-- Source: docs/features/memo.md -->

# Memo

Memos are non-indexed key-value metadata that a workflow can read and update at any point during execution. Unlike [search attributes](./search-attributes.md) (which are indexed and filterable), memos are designed for richer, structured metadata that appears in detail views and history exports but is excluded from fleet-wide filtering and sorting by contract.

## Upserting Memos

`upsertMemo()` is a durable straight-line helper. Each call records a typed `MemoUpserted` history event and merges the new entries into the run's persisted memo.

```php
use Workflow\V2\Workflow;
use function Workflow\V2\{activity, upsertMemo};

final class OrderWorkflow extends Workflow
{
    public function handle(string $orderId, string $customer): array
    {
        upsertMemo([
            'customer_name' => $customer,
            'order_id' => $orderId,
            'status' => 'processing',
            'line_items' => [
                ['sku' => 'WIDGET-1', 'qty' => 2],
                ['sku' => 'GADGET-3', 'qty' => 1],
            ],
        ]);

        $result = activity(ProcessOrderActivity::class, $orderId);

        upsertMemo([
            'status' => 'completed',
            'result_summary' => $result->outcome,
        ]);

        return $result->toArray();
    }
}
```

Memos can also be set at start time via `StartOptions`:

```php
use Workflow\V2\StartOptions;
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::make(OrderWorkflow::class, 'order-123');
$workflow->start(
    'ORD-456',
    'Taylor',
    StartOptions::rejectDuplicate()->withMemo([
        'source' => 'api',
        'priority' => 'high',
    ]),
);
```

## How It Works

The `upsertMemo()` function accepts an associative array of key-value pairs:

- **Keys** must be 1-64 characters from letters, digits, `.`, `_`, `-` and `:`, and can't be an integer such as `42`
- **Values** can be any JSON-serializable data: scalars, null, arrays, or nested objects
- Passing `null` as a value removes that key from the memo

Each call:

1. Suspends the workflow fiber and yields an `UpsertMemoCall` command
2. The executor validates and normalizes the entries (keys are sorted alphabetically)
3. A `MemoUpserted` history event is appended with the upserted `entries` and the full `merged` result
4. The run's `memo` column is updated with the merged map

On replay, recorded memo events are reused without re-executing the upsert, preserving determinism.

## Merging Behavior

Memos merge across multiple upserts within the same run. Each upsert overlays new keys on top of existing ones:

```php
// First upsert
upsertMemo(['status' => 'processing', 'customer' => 'Taylor']);
// memo = { customer: Taylor, status: processing }

// Second upsert
upsertMemo(['status' => 'completed', 'result' => 'success']);
// memo = { customer: Taylor, result: success, status: completed }
```

To remove a key, set it to `null`:

```php
upsertMemo(['temporary_note' => null]);
```

## Visibility

Memos appear in:

- **Run detail view** — the full merged memo is displayed in Waterline's workflow detail page
- **History timeline** — each `MemoUpserted` event appears as a typed entry in the run timeline
- **History export** — memos are included in JSON history exports
- **Describe** — the control plane `describe` response includes the current memo

Memos are **not** included in run summary projections and are **not** available for filtering, sorting, or saved views. Use [search attributes](./search-attributes.md) for indexed, filterable metadata.

## Continue-as-New

When a workflow continues as new, the current memo is carried forward to the new run automatically. The new run starts with the full merged memo from the previous run and can continue upserting from there.

## Nested Structures

Unlike search attributes (which are limited to scalar values), memos support nested JSON structures:

```php
upsertMemo([
    'order' => [
        'id' => 'ORD-456',
        'items' => [
            ['sku' => 'WIDGET-1', 'qty' => 2],
            ['sku' => 'GADGET-3', 'qty' => 1],
        ],
    ],
    'metadata' => [
        'source' => 'api',
        'version' => 2,
    ],
]);
```

The key rule applies to top-level memo keys. Nested values count toward the memo value size limit.

## Constraints

- Key names: 1-64 characters from letters, digits, `.`, `_`, `-` and `:`, and not an integer such as `42`. The rule applies to top-level keys.
- Size: each value up to 10,240 bytes when encoded, at most 100 keys, and 65,536 bytes in total per run. The [`memo_size_bytes`](../constraints/structural-limits.md#memo-size) limit (default 64 KiB) can tighten the total but not raise it.
- Going over a limit raises `StructuralLimitExceededException`, whose `limitName` says which limit. A start is refused without writing anything, and an upsert from workflow code fails the run as a `structural_limit` failure.
- Each upsert appends a `MemoUpserted` history event, so upserting in a loop grows the run's history.
- Values: any JSON-serializable type (scalars, null, arrays, nested objects)
- Setting a value to `null` removes that key
- Memos are eventually consistent metadata, not replay authority — do not branch workflow logic based on memo values

## Memo vs. Search Attributes vs. Visibility Labels

| | Memo | Search Attributes | Visibility Labels |
|---|---|---|---|
| **Set when** | Start time or any time during execution | Any time during execution | Start time only |
| **Mutable** | Yes, via `upsertMemo()` | Yes, via `upsertSearchAttributes()` | No |
| **Value types** | Any JSON-serializable | Scalars and lists of strings | Scalar only |
| **Indexed** | No | Yes | Yes |
| **Filterable** | No | Yes | Yes |
| **Use case** | Rich metadata, notes, context | Dynamic status, progress tracking | Static classification |
| **History events** | `MemoUpserted` per upsert | `SearchAttributesUpserted` per upsert | None (set on start) |

<!-- Source: docs/features/message-streams.md -->

# Message Streams

Message streams are the v2 authoring surface for repeated workflow messages:
human input loops, assistant replies, workflow-to-workflow notifications, and
other ordered messages that must survive replay and continue-as-new.

This page describes the embedded Laravel inbox/outbox API. Server-owned
[Workflow Streams](/docs/polyglot/workflow-streams/) reuse names such as
stream, offset, pending count, lifecycle, and error where their semantics
match, but they are output-only. They do not add inbound workflow messaging or
continue-as-new cursor transfer to service mode.

Workflow authors should use the first-class facade:

- `$this->inbox('stream-key')`
- `$this->outbox('stream-key')`
- `$this->messages('stream-key')`
- `Workflow\V2\MessageStream`

Do not write `workflow_messages` rows or cursor rows directly from application
workflow code. The facade is the stable contract; lower-level message services
exist for package/runtime integration.

:::warning Message streams before Workflow 2.20.3

Workflow code replays on every later workflow task and on every query. In
Workflow releases through 2.20.0, `outbox()->sendReference()` writes a new
outbound and inbound message on each replay, and `inbox()->receive()` and
`receiveOne()` can return a different message, or none, on replay
([workflow#1010](https://github.com/durable-workflow/workflow/issues/1010)).
Workflow 2.20.1 records sends and receives for replay and prevents queries
from writing or consuming messages. Empty receives and streams kept across
steps need the additional replay fixes in 2.20.3, described below.

- Rows that earlier releases duplicated are not removed. Give each logical
  send a stable `idempotencyKey`, such as `reply:{turn}`, and deduplicate by
  `idempotency_key` when consuming a stream that an earlier release wrote.
  Messages that an earlier release consumed under the wrong step stay
  consumed.
- Signals, updates and all streams share one message cursor. Through 2.20.2,
  a signal, update or receive on another stream can move it past pending
  messages and hide them
  ([workflow#1011](https://github.com/durable-workflow/workflow/issues/1011)).
  For the signal-ordering case on those releases, send the signal before the
  message and receive only after a signal or condition says a message is there.

:::

## Stream upgrades and rollback {#stream-upgrades}

In Workflow 2.20.3, the shared cursor no longer hides pending messages from
`receive()`, `receiveOne()`, `peek()`, `pendingCount()` or `hasPending()`.
Receives keep sequence order and their batch limit. After upgrading, a receive
loop can return older pending messages that the shared cursor previously hid,
before newer messages.

A replayed receive returns what its original step consumed. If the original
call found nothing, replay returns nothing again. An update handler running
for the first time can still receive. A stream kept in a property now records
receives at the current workflow step, rather than the step where the stream
was opened. Receives that earlier releases recorded at the opening step still
replay.

Before rolling an embedded application back below 2.20.3, follow the
[replay testing gate](../testing.md#replay-before-deployment) with histories
written by the newer package. Earlier versions look up kept-stream receives
at the opening step and can replay those runs differently. Rollback also
restores the shared-cursor, empty-receive replay and fake task-drain precision
bugs. See the [tagged compatibility notes](https://github.com/durable-workflow/workflow/blob/2.20.3/CHANGELOG.md#2203---2026-11-04).

The query limitation in
[workflow#1024](https://github.com/durable-workflow/workflow/issues/1024)
remains: a query can return nothing for a message an update handler received
because it replays the handler at the wrong step. The query changes no message
rows. The workflow's own replay and completed output still return what the
handler received.

## Receiving Messages

Use `peek()` when a workflow needs to inspect pending inbound messages without
advancing its durable cursor. Use `receive()` or `receiveOne()` when the
workflow is ready to consume messages and record cursor advancement in history.

```php
use Workflow\V2\Workflow;

final class ApprovalInboxWorkflow extends Workflow
{
    public function handle(): array
    {
        $pending = $this->inbox('approval.requests')->peek(limit: 10);

        if ($pending->isEmpty()) {
            return ['status' => 'waiting'];
        }

        $message = $this->inbox('approval.requests')->receiveOne();

        return [
            'status' => 'received',
            'payload_reference' => $message?->payload_reference,
            'correlation_id' => $message?->correlation_id,
        ];
    }
}
```

`peek()` is read-only. `receive()` and `receiveOne()` mark messages consumed and
advance the durable message cursor for the current run. Cursor advancement is a
history fact, so replay, Waterline exports, and continue-as-new handoff can all
reason about which inbound messages were already consumed.

## Sending References

Use `outbox(...)->sendReference(...)` when a workflow needs to publish an
ordered outbound message. The message table stores routing metadata and a
payload reference; large or application-owned content stays in the app's
payload store.

```php
use Workflow\V2\Workflow;

final class AssistantReplyWorkflow extends Workflow
{
    public function handle(string $targetWorkflowId, string $replyReference): array
    {
        $message = $this->outbox('ai.assistant')->sendReference(
            targetInstanceId: $targetWorkflowId,
            payloadReference: $replyReference,
            correlationId: $this->workflowId(),
            metadata: ['kind' => 'assistant_reply'],
        );

        return [
            'status' => 'sent',
            'stream' => $message->stream_key,
            'sequence' => $message->sequence,
        ];
    }
}
```

This keeps the workflow history small and lets consumers validate the payload
store separately from durable message ordering.

## Continue-As-New

Message streams are instance-first. When a workflow continues as new, pending
inbound messages move to the continued run, the cursor position carries
forward, and consumed messages remain attached to the original run as
historical evidence. Workflow 2.20.1 and earlier moved only the cursor:
messages still pending when the run continued as new stayed on the closed run,
and the continued run never received them
([workflow#1018](https://github.com/durable-workflow/workflow/issues/1018),
fixed in 2.20.2). On those releases, receive pending messages before you call
`continueAsNew()`, or send input that must survive the handoff as a
[signal](./signals.md): accepted signals that the run has not applied move to
the continued run.

That means a long-lived workflow can shed history and keep the same public
message route:

```php
use function Workflow\V2\continueAsNew;
use Workflow\V2\Workflow;

final class HumanInputLoopWorkflow extends Workflow
{
    public function handle(int $iteration = 0): array
    {
        $message = $this->inbox('human.replies')->receiveOne();

        if ($message === null) {
            return ['status' => 'waiting', 'iteration' => $iteration];
        }

        if ($this->shouldContinueAsNew()) {
            return continueAsNew($iteration + 1);
        }

        return [
            'status' => 'processed',
            'iteration' => $iteration,
            'payload_reference' => $message->payload_reference,
        ];
    }
}
```

The caller keeps addressing the workflow instance id. It does not need to know
which run currently owns the stream cursor.

## Authoring Rules

- Use semantic stream keys such as `ai.assistant`, `human.replies`, or
  `approval.requests`; treat them as part of the workflow's public contract.
- Use `peek()` for inspection and `receive()` / `receiveOne()` for durable
  consumption.
- Use `sendReference()` for outbound messages; store large payloads in an
  application payload store and pass a reference.
- Keep application code on `Workflow::inbox()`, `Workflow::outbox()`, and
  `MessageStream`; do not depend on `MessageService`, `WorkflowMessage`, or
  cursor-row writes as the authoring pattern.
- For one-shot external events, use [Signals](./signals.md). For request/return
  state changes, use [Updates](./updates.md). Use message streams when repeated
  ordered messages need cursor semantics.

## Run this pattern

The travel-agent workflow in the
[Sample App](/docs/sample-app) is the runnable reference for the
inbox/outbox shape this page describes:

```bash
php artisan app:ai
```

`App\Workflows\Ai\AiWorkflow` stores large assistant payloads in the
app-owned `ai_workflow_messages` table and publishes only a durable
reference on the `ai.assistant` outbox. Replies arrive through the
matching inbox stream via the `receive` update. Read the workflow class
next to this page to see the same `inbox()` / `outbox()` /
`sendReference()` calls the snippets above use, end to end.

<!-- Source: docs/features/timeouts.md -->

# Timeouts

Workflow-level timeouts let you bound how long a workflow is allowed to run. There are two timeout scopes:

- **Execution timeout** — caps the total wall-clock time across all runs of a workflow instance, including continue-as-new transitions. The deadline is computed once at start and carried forward unchanged.
- **Run timeout** — caps a single run. The deadline is recomputed each time a new run begins (including continue-as-new).

Both are optional and can be combined.

## Configuration

Timeouts are configured through `StartOptions` when starting a workflow:

```php
use Workflow\V2\StartOptions;
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::make(MyWorkflow::class, 'order-123');

$workflow->start(
    $orderId,
    StartOptions::rejectDuplicate()
        ->withExecutionTimeout(7200)   // 2 hours across all runs
        ->withRunTimeout(3600),        // 1 hour per run
);
```

### Execution timeout

The execution timeout spans the entire workflow instance. If the workflow uses continue-as-new, the deadline stays the same across every run in the chain.

```php
StartOptions::rejectDuplicate()->withExecutionTimeout(86400); // 24 hours
```

The timeout value is stored on the `WorkflowInstance` model and the computed deadline is snapped on every `WorkflowRun`.

### Run timeout

The run timeout applies to a single run. When a workflow continues as new, the new run gets a fresh deadline computed from the current time plus the configured run timeout.

```php
StartOptions::rejectDuplicate()->withRunTimeout(1800); // 30 minutes per run
```

### Validation

Timeout values must be at least 1 second. Passing zero or a negative value throws a `LogicException`:

```php
// Throws LogicException: "Workflow v2 execution timeout must be at least 1 second."
StartOptions::rejectDuplicate()->withExecutionTimeout(0);
```

### Control plane

Timeouts can also be set when starting a workflow through the control plane:

```php
$controlPlane->start('my-app.order-workflow', 'order-123', [
    'execution_timeout_seconds' => 7200,
    'run_timeout_seconds' => 3600,
]);
```

The `describe` response includes timeout and deadline fields:

```php
$description = $controlPlane->describe('order-123');

$description['execution_timeout_seconds']; // 7200
$description['run']['run_timeout_seconds']; // 3600
$description['run']['execution_deadline_at']; // ISO 8601 timestamp
$description['run']['run_deadline_at'];       // ISO 8601 timestamp
```

### Waterline

When timeouts are configured, the run detail view in Waterline displays the timeout durations and their computed deadlines.

### History

The `WorkflowStarted` history event payload includes timeout and deadline fields when configured:

```json
{
    "execution_timeout_seconds": 7200,
    "run_timeout_seconds": 3600,
    "execution_deadline_at": "2026-04-12T14:00:00+00:00",
    "run_deadline_at": "2026-04-12T13:00:00+00:00"
}
```

### Continue-as-new

When a workflow continues as new:

- The **execution deadline** is carried forward unchanged from the previous run.
- The **run timeout** value is carried forward, but the **run deadline** is recomputed from the current time.

This means the execution timeout always measures from the original start, while each new run gets its own fresh run-timeout window.

## Enforcement

Workflow-level timeouts are enforced at two points:

1. **At workflow task start** — every workflow task checks `deadlineExpired()` before executing the workflow. If the deadline has passed, the run is immediately timed out.
2. **By the TaskWatchdog** — the watchdog scans for non-terminal runs whose execution or run deadline has passed but that have no open workflow task. When found, it creates a deadline-expired workflow task and dispatches it, which triggers the timeout on the next task execution.

When a timeout fires, the engine:

- Cancels all open tasks (activity, timer, workflow) except the current one
- Cancels all open activity executions with `ActivityCancelled` history events
- Cancels all pending timers with `TimerCancelled` history events
- Records a `WorkflowFailure` with `failure_category = timeout` and a `WorkflowTimeoutException`
- Records a terminal `WorkflowTimedOut` history event with `timeout_kind` set to `execution_timeout` or `run_timeout`
- Applies parent-close policy to any open child workflows
- Notifies parent workflows if this was a child run

The failure row stores `Workflow\V2\Exceptions\WorkflowTimeoutException` as the exception class, carrying the `timeout_kind` and the deadline timestamp for programmatic inspection.

### Timeout outcome and cleanup

A workflow deadline closes the run with status `failed` and
`closed_reason = timed_out`. `WorkflowTimedOut` is the terminal history event.
The runtime records `WorkflowTimeoutException` as failure data. It does **not**
throw that exception into the workflow body or resume workflow code for cleanup.
Workflow `catch` and `finally` blocks and registered saga compensations do not
run when the engine enforces this deadline.

Cancelling an activity's durable execution does not undo an external side
effect or prove that its callback stopped. Use idempotent operations and
application-owned reconciliation for effects that may already have committed.

When cleanup must follow an application deadline, use a shorter durable timer
or timeout-backed wait that lets workflow code run
[compensation before closure](./sagas.md#deadline-driven-compensation).
Leave time for activity retries, worker scheduling and cleanup before either
the run or execution deadline. The engine timeout remains the final backstop.
A delayed worker can reach that backstop before compensation finishes.

The immediate `cancel()` and `terminate()` commands also close the run without
a workflow cleanup window. The separate cooperative `requestCancellation()`
operation can deliver bounded cleanup to a compatible worker while the run is
still active. A workflow timeout does not automatically become a cooperative
cancellation request, and a cancellation shield does not extend an execution
or run deadline. See [Cancel and Terminate](./cancel-and-terminate.md) and
[Cooperative Cancellation](../polyglot/cancellation.md).

## Workflow-task lease {#workflow-task-lease}

A workflow task is one worker turn that replays committed history and produces durable commands. It is not the entire workflow. Waiting on a durable timer, queued activity or signal does not hold one workflow task for the whole wait. A later task resumes the run.

| Deployment | Default seconds | Setting |
| --- | --- | --- |
| Embedded Laravel | `300` | `DW_V2_WORKFLOW_TASK_LEASE_SECONDS` |
| Server | `60` | `DW_WORKFLOW_TASK_TIMEOUT` |

The embedded setting maps to `workflows.v2.workflow_task_lease_seconds`. Server maps its own setting into that package key at boot, so use the Server setting on Server. This is runtime configuration, not a `StartOptions` or per-start activity timeout. See [Publishing Config](../configuration/publishing-config.md#workflow-task-lease-v2) and the [Server configuration reference](../polyglot/server-config-reference.md#worker-protocol-and-query-transport).

### Heartbeats and execution {#workflow-task-heartbeats}

Service-mode workers use task heartbeats to renew an in-progress workflow-task claim. Server validates the task, attempt and lease owner. An acknowledged renewal uses the effective lease duration, subject to applicable cancellation-cleanup bounds. It does not change the original run/execution deadlines or activity timeouts. Worker-registration heartbeats keep the registration fresh. A stale registration can invalidate a Server claim before the task lease expires. These heartbeats do not renew that task lease. A queued activity's heartbeat reports activity progress and does not replace a workflow-task renewal.

Synchronous embedded workflow code cannot periodically renew while blocked. Keep workflow code deterministic and short. Put long I/O or CPU work in activities. The lease is not a hard process or CPU execution limit.

Embedded local activities retain the hosting workflow task. Their `$this->heartbeat()` can also renew its lease while the original activity attempt and workflow-task claim remain valid.

### Lease expiry {#workflow-task-lease-expiry}

An expired, unfinished claim becomes eligible for recovery. Matching/watchdog or repair clears the expired claim and redispatches work so a compatible worker can replay committed history. Recovery requires running recovery processes, an available database and an eligible worker. Expiry does not kill the old process or guarantee immediate redelivery. On Server, an expired or replaced claim cannot submit new commands; a rejected renewal must be treated as lost authority.

Repair does not reopen a closed run, grant extra business retries or reset original deadlines. Activity `startToCloseTimeout` instead bounds one activity attempt under its recorded retry policy. Run and execution timeouts bound the workflow lifetime. Workflow-task renewal or replay does not extend them.

## Activity timeouts

Activity timeouts let you bound how long individual activity executions are allowed to take. There are four activity timeout scopes:

- **Schedule-to-start** — caps the time from scheduling to the first worker claim. Enforced while the activity is `Pending`.
- **Start-to-close** — caps the time from when a worker claims the activity to when it must complete. Resets on each retry attempt.
- **Schedule-to-close** — caps the total wall-clock time from scheduling to completion across all retry attempts. This is always terminal — retrying would not help because the overall deadline has passed.
- **Heartbeat** — caps the time between heartbeats. For long-running activities that call `$this->heartbeat()`, the engine requires a heartbeat within the configured interval or the activity is considered unresponsive.

All are optional and can be combined. Configure them through `ActivityOptions` when calling an activity:

```php
use function Workflow\V2\activity;
use Workflow\V2\Support\ActivityOptions;

$result = activity(
    LongRunningActivity::class,
    new ActivityOptions(
        scheduleToStartTimeout: 30,     // must be claimed within 30s
        startToCloseTimeout: 300,       // each attempt has 5 minutes
        scheduleToCloseTimeout: 600,    // total 10 minutes across all retries
        heartbeatTimeout: 15,           // heartbeat well before this 15-second window expires
        maxAttempts: 3,
    ),
    $input,
);
```

In service mode, PHP SDK 3.9.0 and later accept a string such as `'5 minutes'` or a `DateInterval` or `CarbonInterval` for activity timeouts and for child workflow execution and run timeouts, as well as seconds. Those timeouts must be at least one second. See [PHP SDK durations](../polyglot/php.md#durations).

In service mode, closing an activity attempt does not forcibly stop SDK application code. Python SDK 2.9.0 reports a refusal at the next heartbeat; code that never heartbeats can run until it returns, and a physical thread slot remains occupied until exit. See [Python heartbeat refusals and callback exit](../polyglot/python.md#heartbeat-refusals-and-callback-exit).

### Schedule-to-start timeout

The deadline is computed when the activity is scheduled. If no worker claims the activity before the deadline, the `TaskWatchdog` enforces the timeout. If retry attempts remain, a new activity task is scheduled with the snapped backoff and the schedule-to-start deadline is recomputed relative to the retry's available-at time — each retry gets a fresh window. If no `scheduleToStartTimeout` was configured, the deadline is cleared on retry. If all attempts are exhausted, a terminal `ActivityTimedOut` history event is recorded and the workflow is woken.

### Start-to-close timeout

The deadline is computed when a worker claims the activity task. Each retry gets a fresh start-to-close deadline. If the activity does not complete before the deadline, the current attempt is closed. If retry attempts remain, the execution returns to `Pending` with a reset schedule-to-start deadline (if configured) so the retried task is not immediately re-timed-out. If all attempts are exhausted, a terminal timeout is recorded.

### Schedule-to-close timeout

The deadline is computed once at scheduling time and never resets. When it expires, the activity is immediately failed as terminal — even if retry attempts remain, because the total allowed wall-clock time has passed. This is useful for bounding the overall cost of a flaky activity that might otherwise retry indefinitely within its per-attempt limits.

### Heartbeat timeout

For a queued activity, the first deadline starts when the runtime claims it, before task delivery and user code. Each accepted heartbeat sets the deadline to its recorded runtime timestamp plus the configured timeout. It does not add time to the previous deadline or start a fresh window when the caller receives the response. Leave room for delivery, bounded work, scheduling pauses, request latency and transport retries. Send heartbeats well before the window expires. An expired or replaced attempt cannot regain authority by sending a late heartbeat. If retry attempts remain, a new attempt is scheduled with a reset schedule-to-start deadline (if configured). Otherwise a terminal timeout is recorded.

Moving the deadline adds no history. A heartbeat writes an `ActivityHeartbeatRecorded` event only when its progress changed, at most once per recording interval, so you can heartbeat as often as this timeout needs. See [Heartbeats](./heartbeats.md).

```php
use Workflow\V2\Activity;

class LongRunningActivity extends Activity
{
    public function handle($input)
    {
        foreach ($items as $item) {
            $this->heartbeat(['processed' => $count]);
            // ... process item ...
        }
    }
}
```

### Enforcement

Activity timeouts are enforced by the `TaskWatchdog` on each worker-loop pass. The watchdog scans for executions whose deadline columns have passed and delegates to `ActivityTimeoutEnforcer`. Each enforcement records:

- A terminal `ActivityTimedOut` history event with `timeout_kind` set to `schedule_to_start`, `start_to_close`, `schedule_to_close`, or `heartbeat`
- A `WorkflowFailure` row with `failure_category = timeout` and `propagation_kind = timeout`
- A workflow resume task to wake the parent workflow so it can observe the failure

Waterline displays the retry policy including all configured timeout types in the activity detail view. The timeline shows the timeout kind in the activity timed-out event message.

### What is not yet covered

The following are planned but not yet implemented:

- Retry policies at the workflow level

### Structural limits

Typed structural-limit failures for payload size, pending fan-out counts, and metadata size ceilings are enforced by the engine. See [Structural Limits](../constraints/structural-limits.md) for the full limit contract, configuration, and failure taxonomy.

<!-- Source: docs/features/schedules.md -->

# Schedules

Schedules let you start workflow runs on a recurring basis using cron expressions. Each schedule is a named, durable entity that the engine evaluates on every tick to determine whether a new run should be triggered.

## Creating a schedule

Use `ScheduleManager::create()` to define a named schedule:

```php
use Workflow\V2\Enums\ScheduleOverlapPolicy;
use Workflow\V2\Support\ScheduleManager;

$schedule = ScheduleManager::create(
    scheduleId: 'daily-invoice-sync',
    workflowClass: InvoiceSyncWorkflow::class,
    cronExpression: '0 2 * * *',
    arguments: ['nightly'],
    timezone: 'America/New_York',
    overlapPolicy: ScheduleOverlapPolicy::Skip,
    labels: ['team' => 'billing'],
    memo: ['origin' => 'scheduled'],
    searchAttributes: ['tenant_id' => '42'],
    notes: 'Runs every night at 2 AM ET.',
    executionTimeoutSeconds: 120,
    runTimeoutSeconds: 60,
);
```

The `scheduleId` is a unique, user-chosen identifier for the schedule. Each triggered run gets a deterministic workflow instance ID derived from the schedule ID and trigger timestamp.

### Parameters

| Parameter | Type | Default | Description |
|---|---|---|---|
| `scheduleId` | `string` | required | Unique identifier for the schedule |
| `workflowClass` | `string` | required | The workflow class to start |
| `cronExpression` | `string` | required | Standard cron expression (5 fields) |
| `arguments` | `array` | `[]` | Arguments passed to the workflow's `handle()` method |
| `timezone` | `string` | `'UTC'` | Timezone for evaluating the cron expression |
| `overlapPolicy` | `ScheduleOverlapPolicy` | `Skip` | What to do when the previous run is still active |
| `labels` | `array` | `[]` | Visibility labels applied to each triggered run |
| `memo` | `array` | `[]` | Memo fields applied to each triggered run |
| `searchAttributes` | `array` | `[]` | Search attributes applied to each triggered run |
| `jitterSeconds` | `int` | `0` | Maximum random delay in seconds added to each fire time (thundering-herd mitigation) |
| `maxRuns` | `int\|null` | `null` | Maximum number of runs before auto-deleting the schedule |
| `connection` | `string\|null` | `null` | Queue connection for triggered runs (overrides the workflow class default) |
| `queue` | `string\|null` | `null` | Queue name for triggered runs (overrides the workflow class default) |
| `notes` | `string\|null` | `null` | Free-form operator notes |
| `namespace` | `string\|null` | `null` | Namespace for the schedule (defaults to the configured `workflows.v2.namespace` or `'default'`) |
| `executionTimeoutSeconds` | `int\|null` | `null` | Timeout for each logical workflow execution across retries and continue-as-new |
| `runTimeoutSeconds` | `int\|null` | `null` | Timeout for each individual run, resetting on continue-as-new |

### Time zones and daylight saving time

Cron expressions select local times in the schedule's `timezone`. Clock
changes follow these rules:

- When the hour field selects fewer than all 24 hours, a repeated local time
  fires once, at its first occurrence. In New York, `30 1 * * *` fires at
  01:30 EDT on the day clocks go back.
- A local time skipped when clocks go forward fires once, shifted by the
  size of the gap. New York `30 2 * * *` fires at 03:30 on that day.
- An hour field selecting all 24 hours, including `*`, `*/1` and `0-23`,
  keeps elapsed-time behavior through the repeated hour. For example,
  `*/15 * * * *` keeps firing every 15 real minutes through both passes of
  New York's repeated hour.

Use `timezone: 'UTC'` to avoid local clock changes. Use an interval spec when
a fixed elapsed duration matters more than local wall-clock time.

### Workflow timeouts

The execution timeout covers one logical execution across retries and
continue-as-new. The run timeout limits an individual run and resets on
continue-as-new. A run cannot exceed the remaining execution budget. Each
scheduled occurrence starts a new execution budget, measured from the actual
workflow start rather than its scheduled fire time.

Both options accept positive integer seconds or `null`. Omitted or `null`
options leave the corresponding limit unset. Manual triggers, automatic ticks,
buffered starts and backfills apply the stored limits through the default PHP
starter.

For `createFromSpec()`, put the limits in the action:

```php
$schedule = ScheduleManager::createFromSpec(
    scheduleId: 'hourly-invoice-sync',
    spec: ['intervals' => [['every' => 'PT1H']]],
    action: [
        'workflow_class' => InvoiceSyncWorkflow::class,
        'input' => ['hourly'],
        'execution_timeout_seconds' => 120,
        'run_timeout_seconds' => 60,
    ],
);
```

Creation and action updates validate PHP workflow timeouts using the same
`StartOptions` rules as direct starts. No custom starter is needed.

Change limits for future starts by updating the action:

```php
$schedule = ScheduleManager::update($schedule, action: [
    ...$schedule->action,
    'execution_timeout_seconds' => 240,
    'run_timeout_seconds' => 90,
]);
```

The action is replaced, so preserve its workflow class and input as shown.
Previously started runs keep their original deadlines. Set a timeout field to
`null` to remove that limit from subsequent starts.

## Advanced scheduling

The sections below cover scheduling features you will reach for when the basic cron pattern is not enough: fixed-interval firing, mixing cron and interval specs, and overlap policies. Skip ahead unless you need one.

### Interval-based schedules

In addition to cron expressions, schedules support interval-based firing using ISO 8601 duration syntax. Use `ScheduleManager::createFromSpec()` for full control over the schedule spec:

```php
use Workflow\V2\Support\ScheduleManager;

$schedule = ScheduleManager::createFromSpec(
    scheduleId: 'health-check-30m',
    spec: [
        'intervals' => [
            ['every' => 'PT30M'],
        ],
    ],
    action: [
        'workflow_type' => 'health-check',
        'workflow_class' => HealthCheckWorkflow::class,
        'input' => ['region' => 'us-east-1'],
    ],
);
```

### Interval spec fields

| Field | Type | Description |
|---|---|---|
| `every` | `string` | ISO 8601 duration (e.g., `PT30M` for 30 minutes, `PT1H` for 1 hour, `P1D` for 1 day) |
| `offset` | `string\|null` | Phase offset as ISO 8601 duration — shifts the alignment point of the interval |

The offset parameter controls where in the interval cycle the schedule fires. For example, an hourly interval with a 5-minute offset fires at `:05`, `:05+1h`, etc.:

```php
$schedule = ScheduleManager::createFromSpec(
    scheduleId: 'offset-hourly',
    spec: [
        'intervals' => [
            ['every' => 'PT1H', 'offset' => 'PT5M'],
        ],
    ],
    action: [
        'workflow_type' => 'sync-workflow',
        'workflow_class' => SyncWorkflow::class,
        'input' => [],
    ],
);
```

### Mixed cron and interval specs

A single schedule can combine cron expressions and intervals. The engine evaluates all specs and uses the earliest upcoming fire time:

```php
$schedule = ScheduleManager::createFromSpec(
    scheduleId: 'mixed-schedule',
    spec: [
        'cron_expressions' => ['0 12 * * *'],  // noon daily
        'intervals' => [['every' => 'PT6H']],  // every 6 hours
        'timezone' => 'America/Chicago',
    ],
    action: [
        'workflow_type' => 'report-workflow',
        'workflow_class' => ReportWorkflow::class,
        'input' => [],
    ],
);
```

The `createFromSpec` method accepts all the same lifecycle parameters as `create()` (`overlapPolicy`, `jitterSeconds`, `maxRuns`, `connection`, `queue`, `namespace`, etc.) as separate named arguments.

## Overlap policies

When a schedule fires and the previous run is still active, the overlap policy controls behavior:

| Policy | Behavior |
|---|---|
| `Skip` | Do not start a new run (default) |
| `BufferOne` | Buffer one pending trigger; skip further triggers until the buffer is drained. On the next `tick()` after the active run completes, the buffered trigger fires automatically. |
| `BufferAll` | Buffer all pending triggers with no cap; each buffered trigger drains sequentially as previous runs complete. |
| `AllowAll` | Start the new run regardless of the previous run's state |
| `CancelOther` | Cancel the previous run, then start the new run |
| `TerminateOther` | Terminate the previous run, then start the new run |

## Managing schedules

### Pause and resume

```php
ScheduleManager::pause($schedule);

// The schedule will not trigger while paused.

ScheduleManager::resume($schedule);
// next_fire_at is recalculated from now.
```

### Update

```php
ScheduleManager::update(
    $schedule,
    cronExpression: '30 3 * * *',
    timezone: 'America/Chicago',
    overlapPolicy: ScheduleOverlapPolicy::AllowAll,
    notes: 'Moved to 3:30 AM CT.',
);
```

Only a change to the spec (cron expressions, intervals or timezone) or to the jitter recalculates `next_fire_at`. Other updates keep it, so they cannot skip a due fire. The `ScheduleUpdated` event lists in `changed_fields` only the fields whose stored value changed.

An update that changes nothing records no event and keeps `next_fire_at`, including its jitter. On a schedule loaded fresh, `wasChanged()` on the returned model is `false`.

Changing the action (workflow type, arguments, queue routing or timeouts) drops triggers buffered under `BufferOne` or `BufferAll`, because they were captured for the old action. Each dropped trigger is recorded as a `ScheduleTriggerSkipped` event with reason `action_changed` and its `occurrence_time`.

### Delete

```php
ScheduleManager::delete($schedule);
```

Deleting is soft — the row remains with status `deleted` and a `deleted_at` timestamp. A deleted schedule cannot be paused, resumed, updated, or triggered. Its id stays taken. See [Apply from configuration](#apply-from-configuration).

### Apply from configuration

Deploy tooling can apply the same schedule definitions on every deploy. Create each schedule, and update it when its id is taken:

```php
use Workflow\V2\Exceptions\ScheduleAlreadyExistsException;
use Workflow\V2\Support\ScheduleManager;

try {
    ScheduleManager::create(
        scheduleId: 'daily-invoice-sync',
        workflowClass: InvoiceSyncWorkflow::class,
        cronExpression: '0 2 * * *',
        timezone: 'America/New_York',
    );
} catch (ScheduleAlreadyExistsException $e) {
    if ($e->existingStatus === 'deleted') {
        throw $e; // The id belonged to a deleted schedule. Pick a new id.
    }

    ScheduleManager::update(
        ScheduleManager::findByScheduleId($e->scheduleId, $e->namespace),
        cronExpression: '0 2 * * *',
        timezone: 'America/New_York',
    );
}
```

Re-applying an unchanged definition records nothing, so this is safe to run on every deploy.

Schedule ids are not reusable. A deleted schedule, including one deleted when its `max_runs` ran out, keeps its id. Creating a schedule with that id raises `ScheduleAlreadyExistsException` with `existingStatus` `deleted`. Use a new id instead.

Native Server applies the same id rule: creating a schedule with a taken id returns `409 schedule_already_exists`. Since native Server 3.6.0, re-applying an unchanged schedule is also a no-op: it records no `ScheduleUpdated` event and keeps `next_fire_at`. The response outcome is still `updated`. Native Server releases before 3.6.0 record `ScheduleUpdated` and recompute `next_fire_at` on every update, including an unchanged one.

### Describe

```php
$description = ScheduleManager::describe($schedule);

$description->scheduleId;      // 'daily-invoice-sync'
$description->namespace;       // 'default'
$description->status;          // ScheduleStatus::Active
$description->spec;            // ['cron_expressions' => ['0 2 * * *'], 'timezone' => 'America/New_York']
$description->overlapPolicy;   // ScheduleOverlapPolicy::Skip
$description->firesCount;      // 47
$description->nextFireAt;      // DateTimeInterface|null
$description->lastFiredAt;     // DateTimeInterface|null
$description->latestInstanceId; // 'schedule:daily-invoice-sync:...'
$description->jitterSeconds;   // 0
$description->note;            // 'Runs every night at 2 AM ET.'
$description->toArray();       // full array representation
```

### Find by schedule ID

```php
$schedule = ScheduleManager::findByScheduleId('daily-invoice-sync');
```

## Triggering schedules

### Manual trigger

```php
$instanceId = ScheduleManager::trigger($schedule);
```

This immediately evaluates the overlap policy and, if allowed, starts a new workflow run. Returns the instance ID of the started workflow, or `null` if the trigger was skipped.

### Tick (evaluate all due schedules)

```php
$results = ScheduleManager::tick();

// Returns rows with schedule_id, instance_id, outcome, occurrence_time,
// last_fired_at, and next_fire_at when those fields apply.
```

`tick()` triggers due schedules, which are active schedules whose `next_fire_at` is in the past. Each call starts at most `$limit` of them (`tick(int $limit = 100)`), oldest first and shared round-robin across namespaces. Due schedules beyond the limit stay due for the next call. The `occurrence_time` field is the due fire time the scheduler observed. After each trigger, `next_fire_at` advances from the current clock to the next cron or interval occurrence.

### Missed-fire policy

Live schedule evaluation uses **fire once, then resume** semantics. If the scheduler is down across one or more nominal fire times, the first tick after recovery starts one workflow for the overdue `next_fire_at`, records that due time as `occurrence_time`, and then advances `next_fire_at` from the current clock. Additional nominal occurrences that passed while the scheduler was down are skipped by live evaluation.

The same rule applies when the scheduler is running but cannot keep up. A schedule that is still waiting for a pass when its next nominal time arrives fires once and skips the occurrences in between. See [Pass size and bursts](#pass-size-and-bursts).

Use `backfill()` when every missed occurrence must run. Backfill enumerates the requested window explicitly and records each occurrence separately.

### Artisan command

Run a single tick from the command line:

```bash
php artisan workflow:v2:schedule-tick
php artisan workflow:v2:schedule-tick --json
php artisan workflow:v2:schedule-tick --limit=500
```

To evaluate schedules continuously, call this command from Laravel's task scheduler:

```php
// app/Console/Kernel.php
$schedule->command('workflow:v2:schedule-tick')->everyMinute();
```

### Pass size and bursts

Each `workflow:v2:schedule-tick` pass starts at most `limit` due schedules and,
separately, drains at most `limit` buffered actions. `limit` comes from the
`--limit` option, otherwise from `workflows.v2.schedules.tick_limit`
(`DW_V2_SCHEDULE_TICK_LIMIT`), and defaults to 100. A value that is not a
positive integer makes the command fail without evaluating any schedule; it is
never replaced by another size. The setting and the option need Workflow 2.16.0
or later.

Both batches are shared round-robin across namespaces, oldest first, so one busy
namespace cannot hold back another. Due schedules beyond `limit` stay due and are
picked up by the next pass. When `N` schedules fall due together, nothing older
is due and every start succeeds, all of them fire within `ceil(N / limit)`
passes. With the every-minute recipe, the last one starts up to
`ceil(N / limit) - 1` minutes after the first. Set `limit` at or above the
largest burst you expect.

The bound is longer when:

- a start fails: the occurrence stays due and is retried on later passes, so a
  schedule that keeps failing keeps taking a slot while it is the oldest;
- schedules that became due earlier, or more schedules falling due before the
  burst clears, are selected first;
- ticks run late or are skipped.

Under sustained overload, a schedule that waits past its next nominal time
fires once and skips the occurrences in between, as described in
[Missed-fire policy](#missed-fire-policy). Use `backfill()` if those occurrences
must run.

Alert on the operator metrics `schedules.missed`, `schedules.oldest_overdue_at`
and `schedules.max_overdue_ms`, and on increases in `schedules.failures_total`.

:::note Upgrading an app that published its config before 2.16.0

Laravel merges a package's config into a published config file one top-level key
at a time. An app that published `config/workflows.php` before 2.16.0 keeps its
own `v2` array, so `DW_V2_SCHEDULE_TICK_LIMIT` is not read: the command uses 100
and still accepts `--limit`. To use the setting, re-publish the config file, or
add this entry to the `v2` array of your published file:

```php
'schedules' => [
    'tick_limit' => env('DW_V2_SCHEDULE_TICK_LIMIT', 100),
],
```

:::

### Native Server

Native Server 3.x evaluates schedules itself, so there is no command to run or
schedule. Each Server process runs a maintenance loop: it waits 250 ms, then
drains up to 100 buffered actions and starts up to 100 due schedules, ordered by
`next_fire_at`, then waits again. The batch size is fixed; there is no setting
for it. Schedule passes serialize across Server nodes, so adding nodes does not
raise throughput. A burst of `N` due schedules fires within `ceil(N / 100)`
passes, in due order.

A start that Server refuses is recorded as a failure, and the schedule moves on
to its next occurrence, so a failing schedule does not keep its slot. A schedule
whose spec cannot be evaluated stays due and is retried on every pass. The
`schedule:evaluate --limit` command of the frozen PHP Server does not apply to
native Server.

## Max runs

When `maxRuns` is set, the schedule tracks `remaining_actions`. After the last allowed trigger, the schedule is automatically soft-deleted.

```php
$schedule = ScheduleManager::create(
    scheduleId: 'one-shot-retry',
    workflowClass: RetryWorkflow::class,
    cronExpression: '*/5 * * * *',
    maxRuns: 3,
);

// After 3 triggers, the schedule status becomes 'deleted'.
```

## Backfill

Backfill triggers workflows for past cron occurrences that were missed (e.g., after a schedule was paused, a deployment outage, or late creation):

```php
$results = ScheduleManager::backfill(
    $schedule,
    from: new DateTimeImmutable('2026-04-10 00:00:00'),
    to: new DateTimeImmutable('2026-04-14 00:00:00'),
);

// Returns: [['schedule_id' => '...', 'instance_id' => '...|null', 'cron_time' => '...'], ...]
```

Each missed cron occurrence is triggered sequentially. The schedule's overlap policy applies to each occurrence, with one exception: **buffer policies (`BufferOne`, `BufferAll`) are treated as `AllowAll` during backfill**. Buffering is a real-time flow-control mechanism that has no meaning for catch-up operations — backfill should start every missed occurrence, not queue them into a buffer that will never drain. You can also override the policy explicitly:

```php
$results = ScheduleManager::backfill(
    $schedule,
    from: new DateTimeImmutable('2026-04-10 00:00:00'),
    to: new DateTimeImmutable('2026-04-14 00:00:00'),
    overlapPolicyOverride: ScheduleOverlapPolicy::AllowAll,
);
```

Backfill respects `maxRuns` — if the schedule's remaining actions are exhausted mid-backfill, the operation stops and the schedule is auto-deleted.

Backfill instance IDs are deterministic: `schedule:{scheduleId}:backfill:{timestamp}`.

## Queue routing

When `connection` or `queue` is set on a schedule, triggered workflows dispatch to that connection and queue instead of the workflow class default:

```php
$schedule = ScheduleManager::create(
    scheduleId: 'priority-sync',
    workflowClass: InvoiceSyncWorkflow::class,
    cronExpression: '0 * * * *',
    connection: 'redis',
    queue: 'high-priority',
);

// Every triggered run dispatches to redis/high-priority,
// regardless of InvoiceSyncWorkflow's default routing.
```

The routing precedence is: schedule fields → workflow class defaults → global queue config.

## Jitter

When multiple schedules share the same cron expression, they all fire at the exact same instant, creating a thundering-herd spike. The `jitterSeconds` parameter spreads triggers across a random window to smooth the load.

```php
$schedule = ScheduleManager::create(
    scheduleId: 'hourly-report',
    workflowClass: ReportWorkflow::class,
    cronExpression: '0 * * * *',
    jitterSeconds: 300, // fire within 0–300 seconds after the top of the hour
);
```

When `jitterSeconds` is set, each computed `next_fire_at` is offset by a random value between 0 and `jitterSeconds` (inclusive). The jitter is re-rolled every time the next fire time is calculated — after a trigger, after a resume, or after an update.

Jitter applies only to the tick-evaluation fire time stored in the database. Backfill enumeration always uses canonical (unjittered) cron times so that backfilled occurrences land on exact cron boundaries.

Setting `jitterSeconds` to `0` (the default) disables jitter entirely — fire times are exact cron matches.

## History event types

Schedule lifecycle events are recorded on two separate event streams:

- **Workflow-run lineage.** When a schedule triggers a workflow, a
  `ScheduleTriggered` event is appended to the started run's history
  (`workflow_history_events`). This gives the run a verifiable link back
  to the schedule that started it.
- **Schedule audit stream.** Every schedule lifecycle transition is
  recorded in the per-schedule audit log
  (`workflow_schedule_history_events`) under a monotonically increasing
  `sequence` field. The audit stream is authoritative for "what happened
  to this schedule"; the run history is authoritative for "why this run
  was started".

Both streams use the same `HistoryEventType` enum and the same
`HistoryEventPayloadContract` payload-key registry, so event names and
payload shapes stay in sync across streams.

### Run-lineage event (on the started workflow run)

`ScheduleTriggered` is appended to the triggered workflow run's history
with the following payload keys:

- `schedule_id` — schedule's user-facing identifier.
- `schedule_ulid` — schedule's internal ULID primary key.
- `cron_expression`, `timezone`, `overlap_policy` — the schedule's
  primary cron expression, IANA timezone, and active overlap policy at
  trigger time.
- `trigger_number` — which trigger this was (1-indexed).
- `occurrence_time` — the schedule fire time the scheduler evaluated.
  Tick-driven triggers record the due `next_fire_at`; backfill triggers
  record the enumerated backfill occurrence. With jitter enabled, the
  tick-driven value includes the jittered fire time stored on the
  schedule.

### Schedule audit stream (on the schedule itself)

Every schedule lifecycle transition is recorded on the schedule's own
audit stream. Sequences start at `1` for `ScheduleCreated` and increment
monotonically per schedule.

| Event | When recorded | Payload keys |
| --- | --- | --- |
| `ScheduleCreated` | Schedule was created | `spec`, `action`, `overlap_policy`, `next_fire_at`, `command_context` |
| `SchedulePaused` | Schedule was paused | `reason`, `paused_at`, `command_context` |
| `ScheduleResumed` | Schedule was resumed | `next_fire_at`, `command_context` |
| `ScheduleUpdated` | Schedule cron, timezone, spec, action, or policy was changed | `changed_fields`, `spec`, `action`, `overlap_policy`, `next_fire_at`, `command_context` |
| `ScheduleTriggered` | A workflow run was started from the schedule | `workflow_instance_id`, `workflow_run_id`, `outcome`, `effective_overlap_policy`, `trigger_number`, `occurrence_time`, `command_context` |
| `ScheduleTriggerSkipped` | A trigger was skipped due to overlap policy, non-triggerable status, or exhausted actions | `reason`, `skipped_trigger_count`, `last_skipped_at`, `occurrence_time`, `command_context` |
| `ScheduleDeleted` | Schedule was soft-deleted — either by an explicit delete call or by exhausting `max_runs` (`reason: max_runs_exhausted`) | `reason`, `deleted_at`, `command_context` |

`command_context` carries the principal, request id, source, and reason
attributes recorded by `ScheduleManager` when the caller passes a
`CommandContext`. It is optional — events recorded without a context
omit the key rather than writing a blank value.

### Payload contract stability

The payload keys in both tables above are declared in
`Workflow\V2\Support\HistoryEventPayloadContract`, asserted on every
write, and pinned by test coverage in the workflow package. Adding a
new key to any schedule event is a wire-format change and follows the
history-event change rules in the
[version compatibility contract](../compatibility.md).

### Retention

The audit stream is retained for the life of the schedule row. Schedule
deletion is a soft delete that writes a final `ScheduleDeleted` event;
the audit rows themselves are not cascaded when a schedule row is
removed. Operators that purge historical schedules must choose an
explicit retention strategy for their deployment — the package ships no
built-in TTL for audit events.

### Visibility

The audit stream is exposed through every operator surface:

- **In-process (Eloquent).** `WorkflowSchedule::historyEvents()`
  (resolved through `ConfiguredV2Models::query(
  'schedule_history_event_model', ...)`) returns the stream for a
  schedule from within the host application.
- **Waterline HTTP.**
  `GET /waterline/api/v2/schedules/{scheduleId}/history` returns the
  stream with `limit` (1–500, default 100) and `after_sequence`
  cursor pagination. The response is scoped to the Waterline
  namespace so multi-tenant deployments only see their tenant's
  audit rows.
- **Waterline UI.** The **History** action on each row in the
  Waterline schedule registry opens a modal that renders the stream
  for that schedule. Events are shown with their sequence, recorded
  timestamp, event type, linked workflow instance and run IDs, and
  formatted payload, with a **Load more** control that advances the
  cursor in batches of 100.
- **Standalone server HTTP.**
  `GET /api/schedules/{scheduleId}/history` on the standalone
  server returns the same stream with the same pagination contract
  and the same `X-Namespace` scoping rules. History remains
  available after a schedule is soft-deleted, since the audit trail
  is what operators reach for to reconstruct a removed schedule.
- **CLI.** `dw schedule:history <schedule-id>` prints the stream
  as a table (`Seq`, `Event`, `Recorded At`, `Workflow Refs`) with
  a **More events available** hint when `has_more` is true.
  `--limit` and `--after-sequence` forward to the server endpoint,
  `--all` pages through every remaining event, and
  `--output=json` / `--output=jsonl` emit structured output
  (`jsonl` drops the cursor envelope so each line is a
  self-contained event).
- **Python SDK.** `Client.get_schedule_history(schedule_id, *, limit=None,
  after_sequence=None)` returns a single `ScheduleHistoryPage`, and
  `Client.iter_schedule_history(schedule_id, *, limit=None, after_sequence=None)`
  is an `AsyncIterator[ScheduleHistoryEvent]` that pages through the full
  stream with the same keyword arguments. `ScheduleHandle` exposes
  matching `.history(...)` and `.iter_history(...)` convenience methods.
  `limit` is clamped server-side between 1 and 500 (default 100) and
  `after_sequence` is a non-negative cursor obtained from the previous
  page's `next_cursor`.

All of these surfaces read from the same `workflow_schedule_history_events`
table; the payload-key contract in
[Payload contract stability](#payload-contract-stability) applies
regardless of which surface the operator uses.

### Migration behavior for pre-audit schedules

The schedule audit stream was introduced together with the
`workflow_schedule_history_events` table. Schedules that were
created before that migration ran have no retroactive
`ScheduleCreated` event — `ScheduleManager` records audit events
only as lifecycle transitions happen, and the migration does not
synthesize a backfilled event for existing schedules.

What this means for operators:

- A pre-existing schedule's stream is empty until its next
  lifecycle transition. Pausing, resuming, updating, triggering,
  skipping a trigger, or deleting the schedule appends events as
  normal from that point on.
- Sequence numbers still start at `1` and increment monotonically
  per schedule. A pre-existing schedule whose first recorded event
  is a `SchedulePaused` will have `sequence = 1` on that row; the
  absence of a preceding `ScheduleCreated` is expected.
- `ScheduleTriggered` and `ScheduleTriggerSkipped` events are
  written every time a tick evaluates the schedule, so any schedule
  that fires on a cron cadence after the migration will accumulate
  audit rows without operator intervention.
- No operator action is required to opt a schedule into the audit
  stream. The stream is always on; it simply has no rows until the
  first post-migration event is written.

## Skip tracking

When a trigger is skipped (due to overlap policy, non-triggerable status, or exhausted actions), the schedule tracks the skip:

- `last_skip_reason` — why the most recent trigger was skipped (e.g., `overlap_policy_skip`, `status_not_triggerable`, `remaining_actions_exhausted`)
- `last_skipped_at` — when the skip occurred
- `skipped_trigger_count` — cumulative number of skipped triggers

These fields are included in `ScheduleManager::describe()` and the Waterline schedule detail API.

## Namespace scoping

Schedules belong to a namespace. When `namespace` is passed to `create()` or `createFromSpec()`, the schedule is scoped to that namespace. When omitted, the schedule inherits the configured `workflows.v2.namespace` (defaulting to `'default'`).

Schedule IDs are unique within a namespace — the same `scheduleId` can exist in different namespaces without conflict.

The standard PHP schedule starter passes the schedule's stored namespace to the
workflow instance, run and durable tasks. A later change to the application's
default namespace does not move an existing schedule's executions. Use Workflow
2.4.2 or newer for this namespace propagation.

```php
$schedule = ScheduleManager::create(
    scheduleId: 'daily-sync',
    workflowClass: SyncWorkflow::class,
    cronExpression: '0 2 * * *',
    namespace: 'billing',
);

// Find by schedule ID within a namespace:
$found = ScheduleManager::findByScheduleId('daily-sync', namespace: 'billing');
```

When Waterline is configured with a namespace (`waterline.namespace`), the schedule list and detail endpoints automatically scope to that namespace. This ensures multi-tenant deployments show only the schedules belonging to the operator's namespace.

## Database

The schedule table (`workflow_schedules`) is created by migration `2026_04_14_000157`. The model class is configurable via `workflows.v2.schedule_model`.

If your deployment runs package migrations alongside application migrations, migration 157 detects a pre-existing `workflow_schedules` table and handles it gracefully: if the table already matches the package schema it is left as-is; if it was created by an earlier shim migration with a different schema, it is replaced.

<!-- Source: docs/features/external-payload-storage.md -->

# External Payload Storage

In embedded Laravel, external payload storage replaces large encoded values in
workflow history with verified references to application-managed storage. This
page covers that embedded storage boundary. Service-mode clients and workers
use the [Server runtime transport](../polyglot/server-api-reference.md#runtime-external-payload-transport)
instead: the Server holds provider credentials and SDKs exchange opaque reference
ids over authenticated HTTP. A provider URI is not a valid Server payload envelope.

## When To Use It

Use external storage for large documents, media, reports, activity arguments or
results that genuinely need to pass through durable execution. Small values stay
inline and require no object-store write. Offload does not remove payload limits,
history-event limits or the need to bound the number of objects a run retains.

## How Offload Works

The application's namespace policy supplies a storage driver and a positive
`threshold_bytes`. Encoded payloads whose size is **greater than** the threshold
are stored through the driver. Values equal to or below it remain inline.
A disabled policy or absent driver leaves values inline.

The embedded provider reference uses schema
`durable-workflow.v2.external-payload-reference.v1` with `uri`, `sha256`,
`size_bytes`, `codec` and optional `expires_at`. The codec is `avro`.
`Workflow\V2\Support\ExternalPayloads` stores the encoded envelope with the
`dw-external-payload:v1:` prefix. When the runtime needs the value, it fetches the
encoded bytes and verifies size and SHA-256 before decoding. Missing or changed
bytes are failures, never a substitute `null` value.

This provider reference differs from the Server's
`durable-workflow.v2.runtime-external-payload-reference.v1` schema. See the
[mode support table](../polyglot/server-api-reference.md#external-payload-support-by-mode)
before moving a driver helper or stored envelope between deployment modes.

## Decode Trust Boundary

Storage and decoding are separate trust boundaries. A provider can hold encoded
bytes while a worker, codec server or export tool that decodes them can see
application values. Choose where decoders run, which keys they can access and
how previews and logs are redacted. Hashes, sizes and codec names are integrity
metadata, not an end-to-end encryption guarantee. Provider URIs may also contain
sensitive object names.

## Driver Choices

| Driver | URI scheme | Embedded requirement |
| --- | --- | --- |
| Local filesystem | `file://` | Every embedded process that resolves a reference must reach the same storage root. |
| S3-compatible storage | `s3://` | Application-managed driver or filesystem disk with the appropriate bucket, prefix and credentials. |
| Google Cloud Storage | `gs://` | Application-managed driver or filesystem disk and provider client. |
| Azure Blob Storage | `azure://` | Application-managed driver or filesystem disk and provider client. |

A Server can itself use local storage while remote SDK workers fetch through
HTTP. Those workers do not need a shared filesystem or provider credentials.

## Configuring A Namespace

Embedded applications configure their namespace storage policy and
`ExternalPayloadStorageDriver` in their Laravel runtime. Keep credentials in
application configuration, not workflow arguments, history or reference URIs.
Use one policy consistently across the processes that execute and replay a run.

For self-hosted Server, the admin
[namespace storage API](../polyglot/server-api-reference.md#namespace-and-storage)
and [CLI storage commands](../polyglot/cli-reference.md#namespace-and-search-attribute-commands)
configure Server-owned policy. They do not give SDK workers direct provider access.

### With The CLI

Use `dw namespace:set-storage-driver` to configure a Server namespace, not to
configure an unrelated embedded application's Laravel container. See the CLI
reference for the driver's required disk, bucket and prefix options.

### With The Server API

`PUT /api/namespaces/{namespace}/external-storage` is an admin policy operation.
Normal SDK clients discover the resulting runtime transport through
`GET /api/cluster/info`; they do not copy its provider configuration into workers.

## Verifying The Policy

Exercise a value above the threshold through the actual configured driver and
the processes that will replay it. Verify the exact encoded byte length and
SHA-256 after fetch. For Server policy, `dw storage:test --namespace=billing --json`
checks the configured storage path. A storage diagnostic is not proof of every
SDK's workflow execution, backup restoration or all provider adapters.

## Picking A Threshold

Choose a threshold with headroom under the ordinary request and payload limits.
It applies to **encoded bytes**, not the original file size or business JSON
length. Very low thresholds add storage requests and retained references for
small values. The runtime transport may also externalize values to fit an
aggregate HTTP request within its limit. Use the discovered Server limits for
service-mode clients rather than treating an embedded threshold as a wire limit.

## Replay, Retention, And Cleanup

- **Replay integrity.** A value is decoded only after fetching and verifying its
  referenced bytes. Preserve objects for every retained run that still needs them.
- **Verified-fetch cache.** `ExternalPayloadStorage` bounds its verified-byte
  cache. Clearing a cache does not delete the backing object or change ownership.
- **Retention.** Runtime cleanup accounts for retained owners and shared objects.
  Storage failures, backup holds or remaining references can delay deletion.
  Running retention is not a guarantee that no orphan objects remain.
- **History export.** An export can preserve reference metadata without fetching
  the object. Its checksum or `history_complete` flag does not prove that external
  bytes exist or that the archive can be replayed. Follow the
  [backup and restore boundary](../configuration/pruning-workflows.md#export-before-pruning)
  and verify objects independently. Self-contained archives remain tracked in
  [Server #520](https://github.com/durable-workflow/server/issues/520).

## Using It From Code

Embedded PHP uses `Workflow\V2\Support\ExternalPayloadStorage` with an
`ExternalPayloadStorageDriver`. `LocalFilesystemExternalPayloadStorage` handles
local roots. Provider adapters own their own access configuration and URI rules.
Applications normally let the runtime apply their namespace policy rather than
constructing stored envelopes themselves.

In service mode, PHP, Python and Rust SDK clients and workers automatically
upload and fetch runtime payloads when the Server advertises an available
transport. Do not submit a provider `external_storage` envelope to Server or
delete Server-managed objects through a direct provider helper. See the
[runtime upload and fetch contract](../polyglot/server-api-reference.md#runtime-external-payload-transport)
and the [Python helper boundary](../polyglot/python.md#external-payload-storage).

## See Also

- [Passing Data](../defining-workflows/passing-data.md)
- [Structural Limits: Payload Size](../constraints/structural-limits.md#payload-size)
- [Server API: Runtime External Payload Transport](../polyglot/server-api-reference.md#runtime-external-payload-transport)
- [Python SDK: External Payload Storage](../polyglot/python.md#external-payload-storage)

<!-- Source: docs/features/activity-execution-model.md -->

# Activity Execution Model

For service-worker support and fail-closed capability negotiation across PHP,
Python, and Rust, see the service-mode
[Portable Worker Affinity](/docs/polyglot/portable-worker-affinity) support matrix.

Durable Workflow v2 now has explicit primitives for the common activity
placement choices:

- ordinary queued activities for durable, independently leased work;
- [local activities](/docs/features/local-activities) for short
  same-process activity work that still records activity history;
- [worker sessions](/docs/features/worker-sessions) for durable activity
  affinity across multiple queued activity steps;
- [sticky execution](/docs/features/sticky-execution) for workflow replay
  cache affinity, with ordinary replay as the correctness fallback.

## Ordinary Queued Activities

`activity(...)` and `Workflow::activity(...)` are ordinary queued activities.
They remain the default activity primitive.

1. Workflow code calls `activity(MyActivity::class, ...)`.
2. The workflow task records `ActivityScheduled` and creates a durable activity
   task on the configured connection and queue.
3. A compatible worker claims that activity task under a lease.
4. The worker runs the activity class and reports completion, failure,
   cancellation, heartbeat, or timeout.
5. The engine records the activity outcome on workflow history and resumes the
   workflow from durable state.

Ordinary activity attempts may run on any compatible worker. A retry may land
on a different process, host, or build. Activity code must be idempotent across
duplicate delivery, retries after lease expiry, and late completion races. Use
`activity_execution_id` as the default remote idempotency key and
`activity_attempt_id` only when a downstream system needs per-attempt
correlation.

```php
use function Workflow\V2\activity;

$quote = activity(FetchQuoteActivity::class, $customerId);
$invoice = activity(CreateInvoiceActivity::class, $quote['id']);
```

## Local Activities

`localActivity(...)`, `Workflow::localActivity(...)`, and
`Workflow::executeLocalActivity(...)` run an activity class inside the workflow
worker process that is currently executing the workflow task. They do not
create an ordinary activity task.

Local activities still record normal activity history:

- `ActivityScheduled`
- `ActivityStarted`
- `ActivityHeartbeatRecorded`
- `ActivityRetryScheduled`
- `ActivityCompleted`
- `ActivityFailed`
- `ActivityCancelled`
- `ActivityTimedOut`

Each local activity event carries `execution_mode=local` and
`local_activity=true`, and the activity execution snapshot stores
`activity_options.execution_mode=local`. Activity heartbeats renew the owning
workflow task lease while the local attempt runs.

Use local activities for short, idempotent side effects that need activity
retry, timeout, heartbeat, cancellation, and visibility semantics but do not
need queue routing or an independent activity worker fleet. See
[Local Activities](/docs/features/local-activities) for the API,
timeouts, retry, shutdown, cold-replay, routing, and metrics contract.

```php
use Workflow\V2\Support\LocalActivityOptions;
use function Workflow\V2\localActivity;

$receipt = localActivity(
    SendReceiptActivity::class,
    new LocalActivityOptions(maxAttempts: 3, startToCloseTimeout: 10),
    $orderId,
);
```

## Worker Sessions

Worker sessions pin a sequence of ordinary queued activity attempts to one
worker-session lease. Use them when multiple durable steps must reuse
worker-local resources such as GPU memory, a mounted filesystem, or a loaded
model.

Worker sessions do not make an activity local. Each in-session activity is
still a durable activity task with its own lease, heartbeat, timeout, retry,
and terminal history. The session adds explicit affinity and admission rules on
top of normal queue routing.

See [Worker Sessions](/docs/features/worker-sessions) for the session API,
lease lifecycle, routing, failure handling, shutdown behavior, and operator
diagnostics.

## Sticky Execution

Sticky execution is a workflow-task replay optimization. A worker may keep a
warm process-local replay cache after completing a workflow task. If it claims
a later task for that run, it may reuse its valid cache. Affinity does not
reserve ready tasks or give the cache owner matching priority; another eligible
worker can claim the task and replay its history cold.

Sticky execution does not make workflow progress process-local. Correctness
always falls back to ordinary cold replay from durable history after cache
misses, worker restart, drain, rollout, or eviction. Workflow code must not
rely on in-memory state outside history.

See [Sticky Execution](/docs/features/sticky-execution) for the sticky
cache lifecycle, routing identity, fallback rules, deployment controls, and
metrics.

## Choosing The Right Primitive

Use workflow code directly for deterministic branching and calculations.

Use [`sideEffect(...)`](/docs/features/side-effects) for a replay-safe
snapshot of a non-deterministic value that does not need activity retry,
timeout, heartbeat, or cancellation semantics.

Use [local activities](/docs/features/local-activities) for short,
retryable, same-process side effects that should be visible as activity
attempts but should not enter normal task matching.

Use ordinary [activities](/docs/defining-workflows/activities) for remote
calls, slow I/O, CPU-heavy work, queue backpressure, dedicated worker fleets,
or work that should continue through a separately leased activity task.

Use [worker sessions](/docs/features/worker-sessions) when multiple
ordinary activity steps need explicit worker-local affinity.

<!-- Source: docs/features/local-activities.md -->

# Local Activities

Local activities are the v2 primitive for short activity work that should run
inside the workflow worker process currently executing the workflow task. They
preserve activity retry, timeout, heartbeat, cancellation, history, and
operator visibility semantics, but they bypass ordinary activity-task queueing.

Use local activities when the work is low-latency, idempotent, and appropriate
for the workflow worker itself. Use ordinary queued activities when the work
needs independent worker scaling, queue routing, long execution, or a separate
activity-task lease.

## Contract Summary

- `localActivity(...)` and `Workflow::localActivity(...)` execute the activity
  class in the same process as the current workflow task.
- The runtime creates an `activity_executions` row and normal activity history
  events marked with `execution_mode=local` and `local_activity=true`.
- No ordinary `TaskType::Activity` task is created, so `connection`, `queue`,
  worker-session, and schedule-to-start routing options are rejected.
- The local attempt owns the workflow task lease. Activity `heartbeat()` renews
  that workflow task lease and records `ActivityHeartbeatRecorded`.
- Retries are durable workflow tasks with backoff, not hidden loop retries.
- Cold replay reads committed activity history. If a started local attempt has
  no terminal event after worker loss, the next attempt records
  `retry_reason=cold_replay`.
- Run detail, history export, timelines, and operator metrics expose local
  attempts separately from ordinary queued attempts.

## Authoring API

Use the namespaced helper:

```php
use Workflow\V2\Support\LocalActivityOptions;
use function Workflow\V2\localActivity;

$receipt = localActivity(
    SendReceiptActivity::class,
    new LocalActivityOptions(
        maxAttempts: 3,
        startToCloseTimeout: 10,
        scheduleToCloseTimeout: 30,
        heartbeatTimeout: 5,
    ),
    $orderId,
);
```

Or use the static workflow facade:

```php
use Workflow\V2\Workflow;

$receipt = Workflow::localActivity(SendReceiptActivity::class, $orderId);
$receipt = Workflow::executeLocalActivity(SendReceiptActivity::class, $orderId);
```

`LocalActivityOptions` accepts retry and timeout fields:

- `maxAttempts`
- `backoff`
- `startToCloseTimeout`
- `scheduleToCloseTimeout`
- `heartbeatTimeout`
- `nonRetryableErrorTypes`

It rejects `connection`, `queue`, worker-session routing, and
`scheduleToStartTimeout` because a local activity does not enter normal task
matching.

## Execution And History

When workflow replay reaches a local activity, the current workflow task:

1. creates the activity execution with `activity_options.execution_mode=local`;
2. records `ActivityScheduled` and `ActivityStarted` with the local marker;
3. instantiates and runs the activity class in the workflow worker process;
4. records `ActivityCompleted`, `ActivityFailed`, `ActivityTimedOut`, or
   `ActivityCancelled`;
5. resumes workflow code from the recorded activity event.

Replay does not rerun a completed local activity. Query replay and cold replay
read the same activity history events that ordinary activities use.

## Heartbeats

A local activity does not own an activity task lease. It owns the workflow task
lease that is currently executing the workflow. At attempt start, the runtime
renews the workflow task lease. When activity code calls `$this->heartbeat()`,
the runtime records progress, updates the activity attempt, and renews the
workflow task lease.

Long-running local activities must heartbeat often enough to keep both the
local activity heartbeat timeout and the workflow task lease healthy:

```php
use Workflow\V2\Activity;

final class PollShortJobActivity extends Activity
{
    public function handle(string $jobId): array
    {
        $state = $this->fetch($jobId);

        $this->heartbeat([
            'message' => 'Polling remote job',
            'job_id' => $jobId,
            'state' => $state['status'],
        ]);

        return $state;
    }
}
```

## Timeouts And Retries

`startToCloseTimeout` limits one attempt. `scheduleToCloseTimeout` limits the
whole local execution across retries. `heartbeatTimeout` limits the gap between
recorded local activity heartbeats.

When a retryable failure or timeout occurs, the runtime records
`ActivityRetryScheduled` and creates a workflow task that becomes available
after the retry backoff. That retry task replays workflow history, reaches the
same local activity sequence, and starts the next local attempt.

Each local attempt is a new `activity_attempts` row. A retry records
`retry_reason` as `failure`, `timeout`, or `cold_replay`.

## Cancellation And Worker Loss

Cancellation is cooperative. A local activity observes cancellation at
heartbeat, timeout enforcement, and attempt-completion boundaries. A cancelled
local attempt records `ActivityCancelled` with the local marker.

If a worker exits before committing a terminal local activity event, the
workflow task lease expires and normal task repair reclaims the workflow task.
Cold replay then reads committed history. If history contains a started local
attempt without a terminal event, the runtime schedules a retry with
`retry_reason=cold_replay`.

## Visibility

Operators can distinguish local activities everywhere activity state is
reported:

- history payloads include `execution_mode=local` and `local_activity=true`;
- `activity_executions.activity_options.execution_mode` is `local`;
- run detail and history export include `execution_mode` and `local_activity`;
- operator metrics expose `activities.local`, `activities.local_open`,
  `activities.local_attempts`, and queued-vs-local activity counters.

The runtime manifest is published at
`worker_protocol.server_capabilities.local_activities` in
`GET /api/cluster/info`. The machine-readable contract is
[`local-activity-runtime.schema.json`](/platform-protocol-specs/local-activity-runtime.schema.json)
and is indexed by the
[Platform Protocol Specs](/docs/platform-protocol-specs#local-activity-runtime-notes)
catalog.

The event names remain normal activity event names so timelines and replay
tools preserve ordering without a parallel event family.

## Choosing The Right Primitive

Use a local activity for short, retryable, idempotent side effects that are
best executed by the workflow worker process and do not need queue routing.

Use an ordinary [activity](/docs/defining-workflows/activities) for remote
calls, slow I/O, CPU-heavy work, dedicated worker fleets, backpressure, or
work that should keep making progress through a separately leased activity
task after workflow worker loss.

Use [worker sessions](/docs/features/worker-sessions) when multiple
ordinary activity steps must reuse worker-local resources such as GPU memory
or a mounted filesystem.

Use [`sideEffect(...)`](/docs/features/side-effects) only for replay-safe
snapshots that do not need activity retry, timeout, heartbeat, or cancellation
semantics.

<!-- Source: docs/features/worker-sessions.md -->

# Worker Sessions

Worker sessions are the v2 activity-affinity primitive. Use them when several
durable activity steps must reuse process-local state, GPU memory, a mounted
filesystem, or another worker-local resource.

## Contract Summary

- A session has one lease owner at a time.
- The first admitted in-session activity creates or reacquires the lease.
- Activity heartbeats renew both the activity attempt lease and the session
  lease.
- `POST /api/worker/sessions`, `POST /api/worker/sessions/{sessionId}/heartbeat`,
  and `DELETE /api/worker/sessions/{sessionId}` expose explicit lifecycle
  verbs for external workers.
- Queue routing still runs first; worker-session admission runs after the
  worker is registered for the task queue.
- Worker registration `capabilities` must satisfy every session requirement.
- Expired and orphaned sessions are visible to operators and may be reacquired
  when the session allows failure recovery.

## Authoring API

PHP workflow authors create a session handle with `Workflow::workerSession()`
or `Workflow\V2\workerSession()` and schedule activities through that handle:

```php
use Workflow\V2\Support\WorkerSessionOptions;
use Workflow\V2\Workflow;

$session = Workflow::workerSession(
    'gpu-render',
    new WorkerSessionOptions(
        queue: 'gpu-activities',
        requirements: ['gpu:nvidia-l4'],
        leaseSeconds: 120,
        ttlSeconds: 1800,
        maxConcurrentActivities: 1,
    ),
);

$frames = $session->activity(RenderFramesActivity::class, $videoId);
$manifest = $session->activity(AssembleManifestActivity::class, $frames['path']);
```

The activity option snapshot stores the same contract under
`activity_options.worker_session`. External runtimes use the same JSON shape
on a `schedule_activity` workflow-task command under `worker_session`.

```json
{
  "type": "schedule_activity",
  "activity_type": "media.render-frames",
  "worker_session": {
    "session_id": "gpu-render",
    "queue": "gpu-activities",
    "requirements": ["gpu:nvidia-l4"],
    "lease_seconds": 120,
    "ttl_seconds": 1800,
    "max_concurrent_activities": 1,
    "create_if_missing": true,
    "allow_reacquire_after_failure": true
  }
}
```

## Python service workers {#python-service-workers}

The [Python worker-session example](../polyglot/python.md#worker-sessions) creates `WorkerSessionOptions`, schedules two sequential remote activities, declares the holder capability and handles an exhausted activity failure. It also shows rebuilding disposable worker-local state after holder loss and explains why explicit close or TTL expiry needs a fresh session id. The recipe is checked with published Python SDK 2.9.0 and Server 2.11.0, including the Server's scheduler/maintenance role.

## Lifecycle

Session creation is lazy. The matching layer creates or reacquires a session
when the first in-session activity task is admitted to a capable worker. A
worker may also create the session explicitly before polling by calling
`POST /api/worker/sessions`.

An active session ends when the holder closes it, the session lease expires,
the absolute TTL expires, or the holding worker is detected as failed. TTL
expiry and explicit close are terminal for that session id. Lease expiry and
orphan detection may be reacquired when `allow_reacquire_after_failure` is
true.

## Lease And Ownership

At most one worker owns a session lease at a time. The server admits
in-session activity tasks only when one of these is true:

- no session exists and `create_if_missing` is true;
- the active session is already owned by the polling worker;
- the session is expired, failed, or orphaned and reacquisition is allowed.

Session ownership does not replace the per-attempt activity lease. Each
activity attempt still has its own `activity_attempt_id`, lease owner,
heartbeat, completion, failure, timeout, and cancellation path.

## Admission And Routing

Worker sessions participate in normal queue routing. `WorkerSessionOptions`
may set `connection` and `queue`; per-call `ActivityOptions` can still
override them. The server enforces session-specific admission after normal
task-queue admission:

- worker registration `capabilities` must cover every session requirement;
- `max_concurrent_activities` caps leased activity attempts inside the
  session;
- `max_concurrent_worker_sessions` caps active session leases held by one
  worker registration.

Fleet-specific routing uses plain capability strings such as `gpu:nvidia-l4`,
`gpu:a100`, `fs:/mnt/models`, or `zone:us-east-1a`.

## Failure, Cancellation, And Shutdown

Activity heartbeats renew the activity attempt lease and the session lease.
Workers may also renew the session directly through the worker protocol. If
the session lease expires, the session becomes `expired`; if the holding
worker registration heartbeat is stale or missing, the session becomes
`orphaned`.

When the holding worker dies mid-sequence, in-flight activities keep ordinary
at-least-once behavior: their attempt leases expire, repair makes them
claimable again, and stale completion may be rejected. A capable worker may
reacquire the session when the contract allows it. Workflow authors must
expect process-local state to be rebuilt after reacquisition.

Workflow cancellation still propagates through activity heartbeat responses.
A session lease never authorizes an in-session activity to ignore
`cancel_requested`. Planned worker shutdown should close held sessions through
`DELETE /api/worker/sessions/{sessionId}` before stopping the process;
in-flight activities still finish, fail, cancel, or expire under their own
attempt leases.

## Visibility

Worker-protocol capabilities advertise `worker_session_verbs` and the
`worker_sessions` runtime contract through `GET /api/cluster/info` and every
worker-plane response. The machine-readable contract is published in
[`worker-sessions-runtime.schema.json`](/platform-protocol-specs/worker-sessions-runtime.schema.json)
and is indexed by the
[Platform Protocol Specs](/docs/platform-protocol-specs#worker-session-runtime-notes)
catalog.

Operators can list active, closed, expired, failed, and orphaned sessions at
`GET /api/worker-sessions`. The detail surface includes session id, holder,
queue, requirements, lease expiry, TTL expiry, active activity count, and
failure reason. System operator metrics include counts for each session
status.

## Choosing Worker Sessions

Prefer ordinary queued activities when each step is independent. Prefer one
larger activity when the whole operation is one atomic side effect. Use a
worker session only when multiple durable activity steps must reuse a
worker-local resource and the workflow can tolerate rebuilding that resource
after worker failure.

See [Activity Execution Model](/docs/features/activity-execution-model)
for how worker sessions relate to ordinary queued activities, local
activities, and sticky execution.

<!-- Source: docs/features/sticky-execution.md -->

# Sticky Execution

Sticky execution is a supported Durable Workflow v2 replay optimization. A
worker can keep a warm, process-local workflow cache after it completes a
workflow task. If that worker claims a later task for the same run, it may
reuse its valid cache. Affinity does not reserve the task for that worker.

Sticky execution is not a correctness feature. Workflow progress is still
committed only through durable history, and ordinary cold replay from history
is always valid. Workflow code must not rely on process-local state for
correctness.

## Contract Summary

Sticky execution has four guarantees:

- Sticky caches are owned by worker processes, not by the server or database.
- Sticky affinity uses the worker protocol `worker_id` as the cache-owner identity.
- Sticky affinity is advisory and expires; cold replay is the mandatory
  fallback for cache misses, worker restart, drain, rollout, or eviction.
- Operators have named controls and diagnostics for enablement, TTL, capacity,
  hit rate, miss rate, forced cold replay, and capacity pressure.

The durable affinity fields are `sticky_worker_id` and `sticky_until` on runs
and workflow tasks. The workflow-task diagnostic fields are
`sticky_replay_mode` and `sticky_claimed_at`.

## Sticky-Cache Lifecycle

A sticky cache is a local worker data structure containing replayed workflow
state for one or more runs. When a sticky-capable worker completes a workflow
task, the server may record that worker as the run's sticky owner until
`sticky_until`. Follow-up workflow tasks inherit that affinity when they are
created.

The worker owns cache contents and eviction policy. A worker may evict cached
runs when it reaches capacity, begins draining, restarts, changes build,
detects unsafe cached state, or chooses to free memory. The server never treats
cache contents as durable state.

If a worker receives a sticky-routed task but no longer has a valid cache entry,
the worker must perform cold replay from durable history. The task lease remains
the only authority for committing workflow progress.

## Routing Identity

The routing identity is the worker protocol `worker_id`. Workers opt into
sticky routing by registering with sticky cache enabled and by continuing to
heartbeat as active workers on the task queue.

Sticky affinity does not add an owner preference or an owner-only hold to
ready-task matching. The PHP runtime selects candidates by priority,
`available_at`, then task ID. Server can also rebalance the candidate batch
across fairness keys within a priority tier. Ordinary compatibility, workflow
type, queue, namespace, capacity, and lease checks still apply.

Any eligible polling worker can claim the task before `sticky_until`, even
when another live worker owns the affinity. If the owner has no free workflow
slots, another eligible worker can claim it immediately and cold replay. The
task does not wait for the owner's cache or for affinity expiry. Raising the
TTL does not create a longer owner reservation.

`sticky_until` bounds when the stored affinity is active and when an owner
claim can be labelled `sticky_hit_expected`. It does not set task availability,
extend a lease, or guarantee that the owner's local cache still contains the
run.

## Fallback Semantics

Cold replay is mandatory fallback. It happens when:

- sticky execution is disabled.
- the worker did not register sticky-cache support.
- the task has no active `sticky_worker_id` and `sticky_until`.
- the sticky owner is stale, missing, draining, restarted, or rolled out.
- the sticky owner evicted the run.
- another worker claims the task, including while the owner's affinity is active.
- the affinity expired before the task was claimed.

The replay-mode diagnostics are:

- `sticky_hit_expected` - the sticky owner claimed the task before expiry. The
  worker must still validate its local cache before using it.
- `cold_replay` - no sticky affinity applied.
- `forced_cold_replay` - a sticky owner was recorded, but a different worker
  claimed the task or the affinity had expired. This can occur while the
  original owner is healthy.

`forced_cold_replay` is not a correctness failure. It means sticky execution
did not deliver its intended replay-speed benefit for that task.

## Deployment, Drain, and Rollout

Sticky execution follows the worker lifecycle. During drain, a worker should
stop claiming new workflow tasks while completing, heartbeating, failing, or
letting existing leases expire under the normal lease contract. Other
compatible workers can claim ready tasks and cold replay them without waiting
for the affinity to expire.

Replacement workers do not inherit process-local caches. A rollout can
therefore increase `forced_cold_replay` until new workers warm their own
caches. Use unique `worker_id` values per worker process or restart so
operator diagnostics can distinguish an old cache owner from a new process.

Build-id compatibility and workflow-definition fingerprinting still decide
whether a worker may execute a workflow task. Sticky execution is never a way to
route incompatible code to a run.

## Operator Controls

Sticky execution is controlled by workflow/runtime configuration and worker
capabilities, not by standalone server-image environment variables. The
workflow package configuration exposes the enablement flag and affinity TTL:

```php
'workflows' => [
    'v2' => [
        'sticky_execution' => [
            'enabled' => true,
            'ttl_seconds' => 300,
        ],
    ],
],
```

Worker cache capacity is advertised by each worker at registration and on
heartbeat. Disable sticky routing in runtime configuration, or run workers
without sticky-cache support, without changing workflow semantics. Existing
runs continue by ordinary cold replay.

## Worker Protocol Fields

Sticky-capable workers advertise support at registration:

```json
{
  "worker_id": "orders-worker-01",
  "task_queue": "orders",
  "runtime": "python",
  "sticky_cache_enabled": true,
  "sticky_cache_capacity": 100
}
```

Workers report cache diagnostics on heartbeat:

```json
{
  "worker_id": "orders-worker-01",
  "sticky_cache": {
    "enabled": true,
    "capacity": 100,
    "size": 72,
    "hit_count": 940,
    "miss_count": 31,
    "forced_cold_replay_count": 8,
    "eviction_count": 15
  }
}
```

Workflow-task poll responses include `task.sticky_execution` with the
`sticky_worker_id`, `sticky_until`, `replay_mode`, and `cache_directive`.
Workers should treat `resume_if_present` as permission to use a valid warm
cache and should cold replay if the cache entry is absent or invalid.

## Metrics and Diagnostics

Operator metrics include:

- `sticky_execution.active_sticky_runs`
- `sticky_execution.ready_sticky_tasks`
- `sticky_execution.leased_sticky_tasks`
- `sticky_execution.hit_expected_last_minute`
- `sticky_execution.miss_last_minute`
- `sticky_execution.forced_cold_replay_last_minute`
- `sticky_execution.cold_replay_last_minute`
- `sticky_execution.hit_rate_last_minute`
- `sticky_execution.miss_rate_last_minute`
- `sticky_execution.capacity_pressure_tasks`

The standalone server also reports worker cache capacity, cache size, cache hit
count, cache miss count, forced cold replay count, cache eviction count, and
capacity-pressure worker count under `sticky_execution_workers`.

Use these diagnostics this way:

- `ready_sticky_tasks` counts ready workflow tasks whose recorded affinity has
  not expired. It does not mean those tasks are held for their owners. The PHP
  runtime's `capacity_pressure_tasks` currently reports that same count.
- Low expected-hit rate can result from another worker claiming tasks, TTL
  expiry, worker replacement or cache eviction. Compare claimant IDs and
  worker-reported cache metrics before changing TTL or capacity.
- High miss rate or high forced cold replay means correctness is protected, but
  sticky execution is not improving replay cost.
- Worker cache-capacity pressure reports workers near or above their advertised
  cache capacity. It is distinct from free workflow execution slots and from
  the ready-task count.

## Replay-Safe Code

Workflow code must behave identically under sticky and cold-replay execution.
Workflow decisions must use durable history: workflow inputs, activity results,
timers, signals, updates, side effects and version markers. Memo and search
attribute projections are operator metadata; do not branch workflow logic on
their mutable values.

Do not rely on mutable globals, local files, open sockets, object identity,
random values, wall-clock reads, or any other process-local state for
correctness. Sticky execution may preserve those values by accident on one
task, then lose them on a cache miss, worker restart, rollout, or eviction.

Use [`sideEffect(...)`](/docs/features/side-effects) for non-deterministic
values that must be recorded once, and use ordinary activities for external
side effects. See
[Execution Guarantees and Idempotency](/docs/constraints/execution-guarantees)
for the replay and durable-history contract.

<!-- Source: docs/features/service-operations.md -->

# Service operations {#service-operations}

Use a service operation when a workflow needs a durable result from a service
owned by another namespace. The caller addresses a registered endpoint, service
and operation. The runtime records the call and its result so a replacement
worker can replay the same outcome.

Child workflows and signals stay in their original namespace. Changing a child
type name or a task queue does not cross that boundary. Use a service operation
for work published in another namespace of the same Durable Workflow cluster.

## Call from embedded PHP {#call-from-embedded-php}

Register the target catalog and its handler before starting the caller. The
[service-mode guide](../polyglot/service-operations.md) explains the shared
catalog, boundary policy and independently published SDK surfaces.

The embedded helper uses the application runtime’s own service catalog and control plane. It does not connect itself to a remote Server. Use the PHP SDK when the caller is a remote service-mode worker.

```php
use Workflow\V2\Support\ServiceOperationOptions;
use Workflow\V2\Workflow;

final class CreateInvoice extends Workflow
{
    public function handle(int $invoiceId): mixed
    {
        $call = self::serviceOperation(
            'billing',
            'invoices',
            'create',
            [$invoiceId],
            new ServiceOperationOptions(
                targetNamespace: 'billing',
                modeOverride: 'sync',
                waitFor: 'completed',
            ),
        );

        return $call->responsePayload;
    }
}
```

`serviceOperation()` returns `ServiceOperationResult`, including the durable
`serviceCallId`, lifecycle status, outcome and response payload. A terminal
service failure or cancellation is restored as a typed workflow exception at
the original call. Keep the application failure mappings available for replay.

## Choose what the caller waits for {#choose-what-the-caller-waits-for}

Use `modeOverride: 'sync'` and `waitFor: 'completed'` when the next workflow
decision depends on the service result. The workflow suspends durably while
the call remains nonterminal.

Use `modeOverride: 'async'` or `waitFor: 'accepted'` when admission is enough.
The returned record identifies the original call. Acceptance is not completed
business work, and a later target result does not rewrite the result already
recorded for an admission-only caller. Choose completion waiting initially when
the workflow needs that terminal result.

## Preserve the recorded contract {#preserve-the-recorded-contract}

Keep target namespace, endpoint/service/operation, request and waiting options
stable for an open run. Use workflow version branches for incompatible authoring
changes. Replay reuses the recorded call identity and result, rather than
requesting another execution.

An explicit idempotency key is a durable business identity. Reuse it only for
the same intended operation and request. Do not change it just because a
response was lost or a wait expired. Inspect the original call before deciding
whether another business operation is safe.

Set the operation's catalog `deadline_policy` to bound service execution. Its
`timeout_seconds` is resolved once from original acceptance into
`call_deadline_at`; an offset-bearing `deadline_at` is the alternative. Worker
replacement and replay do not renew that deadline. `waitTimeoutSeconds` is not
the service execution deadline. Keep workflow and service budgets consistent,
and inspect the target after a timeout instead of assuming its external effects
did not happen.

## Related guides {#related-guides}

- [Cross-namespace service calls](../polyglot/service-operations.md)
- [Child workflows](./child-workflows.md)
- [Versioning](./versioning.md)
- [Failure and recovery](../failures-and-recovery.md)

<!-- Source: docs/configuration/publishing-config.md -->

# Publishing Config

This will create a `workflows.php` configuration file in your `config` folder.

```bash
php artisan vendor:publish --provider="Workflow\Providers\WorkflowServiceProvider" --tag="config"
```

## Workflow-task lease (v2) {#workflow-task-lease-v2}

Configure the embedded workflow-task lease with a positive integer number of seconds:

```env
DW_V2_WORKFLOW_TASK_LEASE_SECONDS=300
```

This maps to `workflows.v2.workflow_task_lease_seconds` in `config/workflows.php` and defaults to `300`. Claims and task renewals read the effective configuration. Zero does not disable the lease. Refresh cached configuration and restart long-running workers after changing deployment configuration.

Server uses `DW_WORKFLOW_TASK_TIMEOUT` instead and maps that value into the package setting. This recovery lease is separate from activity and workflow deadlines. See [Workflow-task lease](../features/timeouts.md#workflow-task-lease).

## Changing Workflows Folder

By default, the `make` commands will write to the `app/Workflows` folder.

```php
php artisan make:workflow MyWorkflow
php artisan make:activity MyActivity
```

This can be changed by updating the `workflows_folder` setting.

```php
'workflows_folder' => 'Workflows',
```

## Using Custom Models (Legacy v1)

:::note Legacy

These `stored_workflow_*` keys configure the v1 `Workflow\Models\StoredWorkflow*` classes. Workflow v2 uses the durable model overrides below — `instance_model`, `run_model`, `task_model`, and so on. Keep the v1 keys only if you are still running v1 workflows during migration.

:::

In the published `workflows.php` config file you can update the v1 model classes to use your own subclasses.

```php
'stored_workflow_model' => App\Models\StoredWorkflow::class,

'stored_workflow_exception_model' => App\Models\StoredWorkflowException::class,

'stored_workflow_log_model' => App\Models\StoredWorkflowLog::class,

'stored_workflow_signal_model' => App\Models\StoredWorkflowSignal::class,

'stored_workflow_timer_model' => App\Models\StoredWorkflowTimer::class,
```

## Using Model Overrides (v2)

The runtime also exposes model overrides for the durable instance, run, task, history, and projection tables:

```php
'v2' => [
    'instance_model' => App\Models\WorkflowInstance::class,
    'run_model' => App\Models\WorkflowRun::class,
    'task_model' => App\Models\WorkflowTask::class,
    'history_event_model' => App\Models\WorkflowHistoryEvent::class,
    'run_summary_model' => App\Models\WorkflowRunSummary::class,
    'run_wait_model' => App\Models\WorkflowRunWait::class,
    'run_timeline_entry_model' => App\Models\WorkflowTimelineEntry::class,
    'run_timer_entry_model' => App\Models\WorkflowRunTimerEntry::class,
    'run_lineage_entry_model' => App\Models\WorkflowRunLineageEntry::class,
],
```

Those overrides are not limited to reads or projection rebuilds. `WorkflowStub::make()`, `load()`, `loadSelection()`, instance reservation, run selection, and workflow-task execution all use the configured `instance_model`, `run_model`, and `task_model`, and Waterline detail plus history export read through the same configured classes. One app-level override therefore governs both the core runtime path and operator-facing reads.

Keep custom subclasses schema-compatible with the built-in models. If a subclass also changes table names or other Eloquent conventions that the package models normally infer, override the affected relations on that subclass as well so `currentRun()`, `runs()`, and similar lookups stay aligned with your custom schema.

See the [Customization Matrix](./customization-matrix.md) for the frozen v2
support contract, including which overrides are safe as inherited subclasses,
which ones need explicit relation overrides, and how serializer, repository,
health, import, migration, and write-side behavior fit together.

## Payload Codec

v2 uses `avro` for new workflow payloads:

```php
'serializer' => 'avro',
```

- **`avro`** (default, required for new v2 workflows) — Apache Avro binary encoding. Compact on the wire and in storage, and faster to encode/decode for large payloads than the legacy PHP serializer.

If a published v1 config still sets `serializer`, final v2 keeps reading the value for `workflow:v2:doctor` diagnostics, but new v2 payloads still resolve to Avro.

### Legacy codecs (v1 migration only)

Two PHP-only codecs remain available for reading v1 history during migration:

- `workflow-serializer-y` — PHP `SerializableClosure` with byte-escape encoding (the v1 default).
- `workflow-serializer-base64` — PHP `SerializableClosure` with base64 encoding.

Setting `serializer` to a legacy codec will be flagged by `php artisan workflow:v2:doctor`. New v2 workflows still resolve to Avro; keep a legacy codec setting only while you are finishing or importing v1 runs that need PHP-native payload decoding.

The internal v1 import/drain reader recognizes legacy fully-qualified class names (for example, `Workflow\Serializers\Y::class`). They are not public v2 codec aliases and cannot be selected by a new v2 run or SDK.

## Compatibility Markers

The runtime can stamp each new run with a compatibility marker and let workers advertise which markers they can execute safely. This is the runtime fence that keeps long-lived runs on compatible builds during deliberate worker build rollouts.

Set the marker for new runs on the current build:

```env
DW_V2_CURRENT_COMPATIBILITY=build-2026-04
```

Optionally tell a worker to accept more than one marker during a rollout:

```env
DW_V2_SUPPORTED_COMPATIBILITIES=build-2026-04,build-2026-03
```

Tune how long one worker heartbeat snapshot stays visible in the database-backed fleet view:

```env
DW_V2_COMPATIBILITY_HEARTBEAT_TTL=30
```

Optionally scope that fleet view to one app or deployment namespace when several apps share the same workflow database:

```env
DW_V2_COMPATIBILITY_NAMESPACE=sample-app
```

The published `workflows.php` config maps those values here:

```php
'v2' => [
    'compatibility' => [
        'current' => env('DW_V2_CURRENT_COMPATIBILITY'),
        'supported' => env('DW_V2_SUPPORTED_COMPATIBILITIES'),
        'namespace' => env('DW_V2_COMPATIBILITY_NAMESPACE'),
        'heartbeat_ttl_seconds' => (int) env('DW_V2_COMPATIBILITY_HEARTBEAT_TTL', 30),
    ],
],
```

If `supported` is omitted, workers default to the single `current` marker. Older task and run-summary rows that were created before task-level compatibility markers existed are backfilled from their run marker during migration and again on the runtime claim or recovery path if needed. Tasks and runs that truly have no marker anywhere still remain claimable by any worker.

The `getVersion()` fallback prefers the run's start-time `workflow_definition_fingerprint` when a replay reaches a newly introduced branch point that does not have a typed `VersionMarkerRecorded` event yet. That lets a same-compatibility run keep the `DEFAULT_VERSION` branch when it clearly started on an older workflow definition. Older runs whose `WorkflowStarted` history predates the fingerprint snapshot still fall back to the start-time compatibility marker and occupied-sequence checks, so you should keep rotating `DW_V2_CURRENT_COMPATIBILITY` for deployment waves that introduce new versioned workflow code and temporarily list both the old and new markers in `DW_V2_SUPPORTED_COMPATIBILITIES` while the older worker cohort is draining.

Each queue worker also records a database-backed compatibility heartbeat snapshot during `Looping` and task handling. Waterline and the detail helpers expose both the local-build view (`compatibility_supported`, `compatibility_reason`) and the fleet view (`compatibility_namespace`, `compatibility_supported_in_fleet`, `compatibility_fleet_reason`, `compatibility_fleet`). When `DW_V2_COMPATIBILITY_NAMESPACE` is set, database-backed heartbeat rows must match that namespace, and each database snapshot reports its own `namespace` alongside `worker_id`, queue scope, supported markers, and `source = database`. During a rolling upgrade, that fleet view also reads the older cache heartbeat format from workers that have not restarted onto the new snapshot table yet; those legacy cache rows remain visible as rollout fallback even under a configured namespace, but they surface with `namespace = null` until the older workers restart onto the namespaced snapshot path. In other words, strict namespace isolation only becomes complete after every worker has restarted onto the database-backed heartbeat path. Transport-level recovery such as re-dispatching an overdue task or recreating a missing task no longer depends on the scanning worker being able to execute that task; only the eventual claim step stays compatibility-fenced.

When an open task already exists but neither the current build nor any active worker heartbeat snapshot advertises its marker, the run stays visible as waiting for a compatible worker instead of surfacing a false `repair_needed` state on that build.

## History Budgets

Waterline uses the run-summary projection to report how large a selected run's typed history has become. These thresholds control when the projection flips `continue_as_new_recommended`:

```env
DW_V2_CONTINUE_AS_NEW_EVENT_THRESHOLD=10000
DW_V2_CONTINUE_AS_NEW_SIZE_BYTES_THRESHOLD=5242880
```

The published `workflows.php` config maps those values here:

```php
'v2' => [
    'history_budget' => [
        'continue_as_new_event_threshold' => (int) env('DW_V2_CONTINUE_AS_NEW_EVENT_THRESHOLD', 10000),
        'continue_as_new_size_bytes_threshold' => (int) env('DW_V2_CONTINUE_AS_NEW_SIZE_BYTES_THRESHOLD', 5242880),
    ],
],
```

Set either threshold to `0` to disable that side of the recommendation. The flag is advisory; use it to plan `continueAsNew()` boundaries before replay cost grows without changing the selected run's runtime behavior.

## Update Wait Policy

Completion-waiting update APIs such as `attemptUpdate()`, the webhook update routes, and Waterline's update controls wait only up to a bounded budget before they fall back to the still-accepted update lifecycle. Configure that default budget here:

```env
DW_V2_UPDATE_WAIT_COMPLETION_TIMEOUT_SECONDS=10
DW_V2_UPDATE_WAIT_POLL_INTERVAL_MS=50
```

The published `workflows.php` config maps those values here:

```php
'v2' => [
    'update_wait' => [
        'completion_timeout_seconds' => (int) env('DW_V2_UPDATE_WAIT_COMPLETION_TIMEOUT_SECONDS', 10),
        'poll_interval_milliseconds' => (int) env('DW_V2_UPDATE_WAIT_POLL_INTERVAL_MS', 50),
    ],
],
```

`completion_timeout_seconds` controls how long `attemptUpdate*` and HTTP completion waits try to get a worker-applied result before they return an accepted lifecycle instead of blocking indefinitely. `poll_interval_milliseconds` only tunes how often the caller checks that durable update row while waiting; it does not change worker execution order or replay behavior. Waterline's operator metrics expose the active `update_wait` values next to the repair policy so operators can see the effective default without opening config files.

## History Export Redaction

History exports include stored workflow, command, activity, update, task, and failure data by design, because replay-debug and archive handoff need durable facts. If those artifacts can leave a protected environment, configure a redactor before exposing the export endpoint or CLI output broadly.

Create a redactor that implements `Workflow\V2\Contracts\HistoryExportRedactor`:

```php
namespace App\Support;

use Workflow\V2\Contracts\HistoryExportRedactor;

final class WorkflowHistoryExportRedactor implements HistoryExportRedactor
{
    public function redact(mixed $value, array $context): mixed
    {
        return [
            'redacted' => true,
            'path' => $context['path'],
        ];
    }
}
```

Then register it in `config/workflows.php`:

```php
'v2' => [
    'history_export' => [
        'redactor' => App\Support\WorkflowHistoryExportRedactor::class,
        'signing_key' => env('DW_V2_HISTORY_EXPORT_SIGNING_KEY'),
        'signing_key_id' => env('DW_V2_HISTORY_EXPORT_SIGNING_KEY_ID'),
    ],
],
```

The redactor receives the current value plus context such as `path`, `category`, `workflow_instance_id`, `workflow_run_id`, and `workflow_type`. The export calls it for workflow argument/output payloads, history-event payloads, command payload/context, update payloads, task payloads, activity payloads, and failure message/file/trace diagnostics. The resulting bundle includes `redaction.applied`, `redaction.policy`, and `redaction.paths` so downstream tooling can tell which policy shaped the artifact.

Every export also includes an `integrity` block computed after redaction. The checksum uses `canonicalization = json-recursive-ksort-v1` and `checksum_algorithm = sha256`; when `signing_key` is configured, the same canonical payload is signed with `signature_algorithm = hmac-sha256` and the optional `signing_key_id` is reported as `key_id`. Keep the signing key outside the exported artifact and rotate the key id when downstream verifiers need to distinguish keys.

<!-- Source: docs/configuration/options.md -->

# Options

There are various options available when defining your workflows and activities. These options include the number of times a workflow or activity may be attempted before it fails, the connection and queue, and the maximum number of seconds it is allowed to run.

```php
use Workflow\V2\Activity;

class MyActivity extends Activity
{
    public ?string $connection = 'default';
    public ?string $queue = 'default';

    public int $tries = 3;

    public function backoff(): array
    {
        return [1, 2, 5, 10, 15, 30, 60, 120];
    }
}
```

The `$connection` and `$queue` properties on `Workflow\V2\Workflow` and `Workflow\V2\Activity` are declared as `public ?string` and default to `null`. Subclass overrides must keep the nullable type so PHP's invariant public-property typing rules accept the redeclaration. Use `null` when you want to inherit the application's default connection or queue instead of hard-coding a value.

Activity timeouts are not configured through a class property. Use [`ActivityOptions`](#activityoptions) per-call (for example `startToCloseTimeout`) or the activity retry policy snapshot taken at schedule time. See also [Task Repair Policy](/docs/polyglot/server-config-reference#workflow-package-controls) for worker-loop timing settings.

## StartOptions

`Workflow\V2\StartOptions` carries visibility, deduplication, and execution timeout configuration at workflow start time. It does not select a queue — queue routing is driven by the workflow and activity class `$connection` and `$queue` properties plus per-call `ActivityOptions`.

```php
use Workflow\V2\StartOptions;
use Workflow\V2\WorkflowStub;
use Workflow\V2\Enums\DuplicateStartPolicy;

$workflow = WorkflowStub::make(MyWorkflow::class);

$workflow->start(
    'arg1',
    new StartOptions(
        duplicateStartPolicy: DuplicateStartPolicy::ReturnExistingActive,
        businessKey: 'order-12345',
        labels: ['tenant' => 'acme'],
        executionTimeoutSeconds: 3600,
    ),
);
```

`StartOptions` are consumed by the workflow engine and are not passed as arguments to your workflow `handle()` method. They are persisted with the workflow and used for subsequent workflow/activity dispatching (including replay and continue-as-new behavior).

### Run priority and fairness {#run-priority-and-fairness}

In embedded Workflow 2.15.14 and newer, set dispatch priority and a workload class when starting the run:

```php
$options = StartOptions::rejectDuplicate()
    ->withPriority(1)
    ->withFairness('tenant:acme', 3);

$workflow = WorkflowStub::make(MyWorkflow::class, 'order-12345');
$workflow->start($orderId, $options);
```

Lower priority numbers are selected first under contention. The normal application range is 1–9, with a default of 5; 0 is reserved for urgent control-plane work. Within a priority tier, the fairness key identifies a workload class, such as a tenant. Its weight sets a relative dispatch share (1–1000, default 1). Runs without a key share one class.

The run records these values. Its first workflow task, timer and signal resume tasks, and activity tasks inherit them. Continue-as-new and retry runs keep the predecessor's recorded settings. Explicit priority or fairness values in `ActivityOptions` override the inherited values for that activity. When `signalWithStart()` creates a new run, it records the same `StartOptions` scheduling values.

## ActivityOptions

`Workflow\V2\Support\ActivityOptions` provides per-call overrides for routing, retries, and timeouts when invoking an activity, without requiring changes to the activity class itself:

```php
use function Workflow\V2\activity;
use Workflow\V2\Support\ActivityOptions;

$result = activity(
    ChargeCard::class,
    new ActivityOptions(
        connection: 'redis',
        queue: 'critical',
        maxAttempts: 5,
        startToCloseTimeout: 30,
    ),
    $orderId,
);
```

Routing overrides in `ActivityOptions` apply when a new activity execution is scheduled. They do not move an activity already recorded in history. See [recorded activity queues](../polyglot/task-matching-dispatch.md#recorded-activity-queues) before changing worker pools.

Retry and timeout fields override the activity class for this call only:

- `maxAttempts`: total attempts, counting the first one. It must be at least 1. A new activity scheduled with 0 or less fails the run. Omit it to use the class `$tries`, where `0` retries forever.
- `backoff`: seconds to wait before each retry, as a list or a single value. It overrides the class `backoff()`. When the list runs out, repeat its final entry. An empty list adds no delay. See [Backoff](#backoff).
- `nonRetryableErrorTypes`: error type or class names that fail the activity without another attempt.
- `startToCloseTimeout`: maximum seconds for one attempt.
- `scheduleToStartTimeout`: maximum seconds an activity may wait for a worker before it starts.
- `scheduleToCloseTimeout`: maximum seconds from scheduling to completion, across all attempts.
- `heartbeatTimeout`: maximum seconds between heartbeats before an attempt is treated as unresponsive.

`LocalActivityOptions` accepts the same retry fields plus `startToCloseTimeout`, `scheduleToCloseTimeout` and `heartbeatTimeout`. It has no routing or `scheduleToStartTimeout` fields.

## Connection

The `$connection` setting is used to specify which queue connection the workflow or activity should be sent to. By default, the `$connection` value is not set which will use the default connection. This can be overridden by setting the `$connection` property on the workflow or activity class.

## Queue

The `$queue` setting is used to specify which queue the workflow or activity should be added to. By default, the `$queue` value is not set which uses the default queue for the specified connection. This can be overridden by setting the `$queue` property on the workflow or activity class.

An activity stores its resolved connection and queue when it is scheduled. Its retries and repaired tasks reuse that recorded routing, even after class defaults or per-call options change. Existing workflow runs also retain their recorded workflow queue. Keep compatible workers on the old queues while those dependencies remain. Follow the [queue transition procedure](../polyglot/task-matching-dispatch.md#recorded-activity-queues).

## Retries

The `$tries` setting is used to control the total number of attempts for an activity before it is considered failed. By default, `$tries` is `1` (a single attempt, no automatic retries). Set `$tries` to a value greater than `1` to allow retries, or set it to `0` to retry forever. This can be overridden per call through `ActivityOptions::$maxAttempts`. `maxAttempts` counts the first attempt and must be at least 1.

## Timeout

The v2 `Activity` base class has no `$timeout` class property. Configure activity timeouts per call through [`ActivityOptions`](#activityoptions) using `startToCloseTimeout`, `scheduleToStartTimeout`, `scheduleToCloseTimeout`, or `heartbeatTimeout`. The runtime snapshots the resulting retry policy onto the activity execution when it is scheduled, so the timeout is stable for an already scheduled attempt even if a later deploy changes the activity class or options. Worker-loop level dispatch timing is controlled through [Task Repair Policy](/docs/polyglot/server-config-reference#workflow-package-controls).

## Backoff

The `backoff()` method returns retry delays in seconds as an integer or a list. The default list reaches 120 seconds (2 minutes). Override it by implementing `backoff()` on the activity class.

When retries outnumber the list entries, the final delay repeats. For `maxAttempts: 5` and `backoff: [1, 2]`, the four retry delays are `[1, 2, 2, 2]` seconds. An empty list, `backoff: []`, requests immediate retries (zero backoff), subject to the attempt and timeout limits.

The retry policy is recorded when the activity is scheduled. Already recorded policies keep their recorded delays when later code changes the class defaults or options.

## Namespace

Workflows can be scoped to a namespace for multi-namespace isolation. The configured namespace applies to both the control-plane start API and `Workflow\V2\WorkflowStub::make()->start()`. It is persisted on the new instance, run, durable tasks and run-summary projection. Task polling and Waterline visibility filters can then restrict results to a single namespace. Use Workflow 2.4.2 or newer for consistent namespace assignment through both PHP start APIs.

Namespace names must contain only lowercase alphanumeric characters, dots, underscores, and hyphens (matching `[a-z0-9._-]+`, max 128 characters).

Set the default namespace via environment variable:

```env
DW_V2_NAMESPACE=production
```

Or in `config/workflows.php`:

```php
'v2' => [
    'namespace' => env('DW_V2_NAMESPACE'),
    // ...
],
```

An explicit namespace overrides the configured default. The stub accepts it as its third `make()` argument:

```php
$workflow = \Workflow\V2\WorkflowStub::make(
    OrderProcessingWorkflow::class,
    'order-12345',
    'staging',
);
$workflow->start();
```

The control plane accepts it in the `start()` options:

```php
$controlPlane->start('order-processing', 'order-12345', [
    'namespace' => 'staging',
    // ...
]);
```

When no namespace is configured and none is passed explicitly, new instances retain a `null` namespace. A namespace-scoped observer does not include those runs. Changing configuration does not reassign existing instances. Starting an existing instance through a different namespace is rejected.

### Which namespaces an app runs {#namespace-allowlist}

From Workflow 2.27, an embedded app repairs and runs work only in the namespaces it serves. This covers the watchdog, task repair, run and activity deadlines, `workflow:v2:schedule-tick`, and the queued workflow, activity, timer and service-call jobs. A queued job for another namespace is skipped before it has any effect, and a warning is logged.

```env
DW_V2_NAMESPACE=production
DW_V2_NAMESPACES=production,billing
```

- When `DW_V2_NAMESPACES` is unset, the app serves exactly `DW_V2_NAMESPACE`, so a single-namespace app needs no change.
- When it is set, the app serves exactly the listed namespaces, so include `DW_V2_NAMESPACE` itself. `default` names the unscoped namespace. An empty value serves none, and `*` is refused.
- A process that repairs or ticks every namespace in the database, such as a standalone server, sets `DW_V2_ALL_NAMESPACES=true` instead.

List every namespace the app starts, schedules or signals workflows in. Otherwise their tasks and schedules wait for an app that serves them. The allowlist keeps one app from running another app's work with the wrong code or credentials. It isn't tenant isolation: apps that share a database can still read each other's rows. Refused work is counted in `namespace_scope_refusals`; see [Monitoring](../monitoring.md#stuck-run-metrics).

### Waterline namespace scoping

When Waterline is deployed against a shared database with multiple namespaces, set `WATERLINE_NAMESPACE` to restrict all list views to one namespace:

```env
WATERLINE_NAMESPACE=production
```

This injects a namespace filter into every visibility query so Waterline only shows workflows belonging to the configured namespace. When set, Waterline also scopes all command operations (cancel, signal, terminate, update, repair, archive, and queries) to the configured namespace — a command targeting an instance or run that belongs to a different namespace will return a 404 instead of executing.

### Command namespace scoping

`WorkflowStub::load()`, `loadSelection()`, and `loadRun()` accept an optional `namespace` parameter:

```php
use Workflow\V2\WorkflowStub;

// Load only if the instance belongs to the given namespace
$stub = WorkflowStub::load('order-12345', namespace: 'production');

// Load a specific run, scoped to namespace
$stub = WorkflowStub::loadRun($runId, namespace: 'production');

// Load a specific selection, scoped to namespace
$stub = WorkflowStub::loadSelection('order-12345', $runId, namespace: 'production');
```

When `namespace` is `null` (the default), loading is unscoped and works against all namespaces — this preserves backward compatibility. When a namespace is provided, the query filters by namespace at the database level and throws `ModelNotFoundException` if the workflow does not exist in that namespace.

The control plane command methods (`signal`, `cancel`, `terminate`, `update`, `repair`, `archive`) also accept `namespace` in their options array:

```php
$controlPlane->cancel('order-12345', [
    'namespace' => 'production',
]);
```

### Task bridge namespace filtering

Both the workflow and activity task bridges accept an optional `namespace` parameter on `poll()`:

```php
$tasks = $bridge->poll('redis', 'default', limit: 10, namespace: 'production');
```

When omitted, `poll()` returns tasks from all namespaces (backward-compatible with pre-namespace installations).

## Durable Type Aliases

Durable type keys for workflows and activities are stored when you register them under `workflows.v2.types`. Failure payloads can use the same pattern for exception classes:

```php
'v2' => [
    'types' => [
        'workflows' => [
            'billing.invoice-sync' => App\Workflows\InvoiceSyncWorkflow::class,
        ],
        'activities' => [
            'payments.capture' => App\Activities\CapturePaymentActivity::class,
        ],
        'exceptions' => [
            'billing.invoice-declined' => App\Exceptions\InvoiceDeclined::class,
        ],
        'exception_class_aliases' => [
            App\Exceptions\LegacyInvoiceDeclined::class => App\Exceptions\InvoiceDeclined::class,
        ],
    ],
],
```

When an activity, update, child, or workflow failure is recorded with an exception alias, the engine stores that alias in typed history as `exception_type` and inside the failure payload as `type`. Replay resolves the alias before falling back to the recorded PHP class, so a later class move can keep workflow `catch` semantics stable as long as the alias still points at the current throwable class.

For imported v1 failures that were recorded before an exception alias existed, `workflows.v2.types.exception_class_aliases` can map the recorded legacy exception FQCN to the current throwable class. Durable `exceptions` type aliases still win first. The class-alias map is only a refactor bridge for already-recorded payloads with no durable `type`; new workflows should use durable exception type aliases so history is independent from PHP class names.

Final v2 writes durable exception aliases when the failure is recorded, so configure stable aliases before recording failures whose throwable classes may move later.

If a replayed failure cannot be resolved through the durable `exceptions` map, the class-alias map, or the recorded class, the engine does not fall back to a generic runtime exception inside workflow code. Query replay raises `UnresolvedWorkflowFailureException`, Waterline marks the failure with `exception_replay_blocked = true`, and a worker task that hits the same gap is left failed while the run stays open. Fix the mapping and repair the run rather than relying on broad `catch (RuntimeException)` blocks to handle renamed historical failures.

## Legacy workflow overlap delay {#legacy-overlap-release-delay}

For applications using `Workflow\Workflow`, set `overlapReleaseAfter` to a nonnegative integer number of seconds. The default is `0`. This setting applies to contended workflow, activity, signal and child-completion queue jobs. It does not change lock expiry or activity exception backoff. `Workflow\V2\Workflow` uses task claims and does not use this middleware.

```php
use Workflow\WorkflowOptions;
use Workflow\WorkflowStub;

$workflow = WorkflowStub::make(MyLegacyWorkflow::class);
$workflow->start('argument', new WorkflowOptions(overlapReleaseAfter: 5));
```

`WorkflowOptions::set(['overlapReleaseAfter' => 5])` accepts the same setting. The value is persisted with the run and inherited by children and continue-as-new when no explicit `WorkflowOptions` is supplied. An explicit options object replaces inherited options, including resetting this delay to zero if omitted.

Laravel counts queue reservations as attempts even when middleware releases a job before its handler runs. A positive delay reduces rapid retries but cannot guarantee that a low `$tries`, `--tries`, or database queue attempt limit is safe. Choose the delay and retry budget for the expected contention duration.

<!-- Source: docs/configuration/ensuring-same-server.md -->

# Worker Placement and Affinity

A queue name is a routing boundary, not a physical-host affinity guarantee. Setting
the `$queue` property on a workflow or activity restricts which workers may claim
the task, but any compatible worker polling that queue can receive it. Do not pass
data between ordinary activities through a worker's local filesystem unless the
data is also stored in a durable shared location.

## Route Work to a Dedicated Worker Pool

Use a dedicated queue when a class of work needs a particular worker pool, such
as workers with a mounted shared volume or a GPU capability:

```php
use Workflow\V2\Activity;

final class RenderVideo extends Activity
{
    public ?string $queue = 'gpu-activities';
}
```

Start only the intended Laravel workers on that queue:

```bash
php artisan queue:work --queue=gpu-activities
```

Laravel Horizon can supervise the same queue. In either case, the guarantee is
"one of the workers assigned to this queue," not "the same process or host that
ran the previous step." Put handoff data in durable workflow payloads, external
payload storage, or another shared store available to every eligible worker.

## Choose the Affinity Contract You Need

- Use an [ordinary queued activity](/docs/defining-workflows/activities)
  when the work can run on any compatible worker in its queue.
- Use a [local activity](/docs/features/local-activities) when one short,
  retryable activity must execute inside the workflow worker process that is
  currently handling the workflow task.
- Use a [worker session](/docs/features/worker-sessions) when several durable
  activity steps must reuse one worker-session lease for process-local state,
  GPU memory, or a worker-local mounted filesystem.

Worker sessions still apply queue routing first. They add an explicit lease and
capability contract after routing, including expiry and optional reacquisition;
they do not make an individual machine immortal. If state must survive worker
loss or session reacquisition, keep it in durable shared storage.

See [Activity Execution Model](/docs/features/activity-execution-model) for
the placement comparison and [Options](/docs/configuration/options#queue)
for class-level and per-call queue configuration.

<!-- Source: docs/configuration/customization-matrix.md -->

# Customization Matrix

This page freezes the supported Durable Workflow v2 customization contract.
Use it when you need to move v2 durable models onto a different connection,
swap model subclasses, replace the PHP operator-observability repository, or
decide whether an older serializer setting is still valid during migration.

## Supported override matrix

| Surface | Supported contract | Notes |
| --- | --- | --- |
| `workflows.v2.instance_model` with a schema-compatible subclass that keeps the basename `WorkflowInstance` | Supported | Use this shape when you only need a different connection, casts, or app namespace. The inherited relations keep the package's `workflow_instance_id` foreign-key contract. |
| `workflows.v2.instance_model` with a table-swapped or basename-changing subclass | Supported with explicit relation overrides | Override `runs()`, `commands()`, and `updates()` so they keep `workflow_instance_id`. The package now validates this at boot and rejects inherited relation inference that would change those keys. `currentRun()` already pins `current_run_id` explicitly and does not need an override. |
| Other `workflows.v2.*_model` overrides (`run_model`, `task_model`, `history_event_model`, projections, schedules, activity rows, failures, messages, memos, search attributes, child calls) | Supported for schema-compatible subclasses | Keep the package column names, primary keys, and foreign keys. These relations already pin the keys they use, so table-swapped subclasses can stay on the inherited relation methods when the schema stays compatible. |
| Custom table names plus custom foreign-key column names | Unsupported | The v2 runtime and operator surfaces assume the package column and key names. If you change them, you are outside the supported contract. |
| `OperatorObservabilityRepository` replacement | Supported by container binding | Bind your implementation to `Workflow\V2\Contracts\OperatorObservabilityRepository`. This is a PHP runtime integration contract for Waterline and `WorkflowStub::historyExport()`, not a stable cross-language API. The repository receives package model objects. |
| `serializer` | Supported only for `avro` and legacy drain/import codecs | New v2 runs always use Avro semantics. `workflow-serializer-y` and `workflow-serializer-base64` remain valid only for finishing or importing v1 history that still needs PHP-native decoding. Custom serializer classes from v1 are unsupported in v2. |
| Health and doctor diagnostics | Supported | `php artisan workflow:v2:doctor` and Waterline v2 health/stats reflect the configured model classes and flag stale serializer settings as migration debt. |
| Package migrations on a non-default connection | Supported with published migrations | Auto-loaded package migrations run on Laravel's default connection. If your durable models use another connection, publish the migrations and set the migration `$connection` explicitly. |
| Core write-side runtime (`WorkflowStub::make()`, `load()`, task creation, current-run resolution) | Supported through the configured durable models | The runtime honors the configured `instance_model`, `run_model`, and `task_model`. Keep write-side tables schema-compatible with the package columns and keys. |

## Exact v2 model keys

The published `workflows.php` config exposes these durable model keys under
`workflows.v2`:

- `instance_model`
- `run_model`
- `history_event_model`
- `task_model`
- `command_model`
- `link_model`
- `activity_execution_model`
- `activity_attempt_model`
- `timer_model`
- `failure_model`
- `run_summary_model`
- `run_wait_model`
- `run_timeline_entry_model`
- `run_timer_entry_model`
- `run_lineage_entry_model`
- `schedule_model`
- `schedule_history_event_model`

Use subclasses of the package models for those keys. Keep the package's column
names and foreign keys unless a page in this docs set says otherwise.

## Instance-model override rule

The only current v2 override that needs extra relation work is
`workflows.v2.instance_model` when your subclass changes the inferred foreign
key away from `workflow_instance_id`.

Typical safe example:

```php
namespace App\Models\V2;

use Workflow\V2\Models\WorkflowInstance as BaseWorkflowInstance;

final class WorkflowInstance extends BaseWorkflowInstance
{
    protected $connection = 'workflow';
}
```

Because the subclass basename is still `WorkflowInstance`, Eloquent keeps the
same inferred foreign key and the inherited relations remain aligned.

If you use a different basename or a table-swapped storage model, override the
affected relations explicitly:

```php
namespace App\Workflow\Storage;

use Illuminate\Database\Eloquent\Relations\HasMany;
use Workflow\V2\Models\WorkflowCommand;
use Workflow\V2\Models\WorkflowInstance as BaseWorkflowInstance;
use Workflow\V2\Models\WorkflowRun;
use Workflow\V2\Models\WorkflowUpdate;
use Workflow\V2\Support\ConfiguredV2Models;

final class TenantWorkflowInstance extends BaseWorkflowInstance
{
    protected $table = 'tenant_workflow_instances';

    public function runs(): HasMany
    {
        return $this->hasMany(
            ConfiguredV2Models::resolve('run_model', WorkflowRun::class),
            'workflow_instance_id',
        );
    }

    public function commands(): HasMany
    {
        return $this->hasMany(
            ConfiguredV2Models::resolve('command_model', WorkflowCommand::class),
            'workflow_instance_id',
        )->oldest('created_at');
    }

    public function updates(): HasMany
    {
        return $this->hasMany(
            ConfiguredV2Models::resolve('update_model', WorkflowUpdate::class),
            'workflow_instance_id',
        )
            ->orderBy('command_sequence')
            ->oldest('accepted_at')
            ->oldest('created_at')
            ->oldest('id');
    }
}
```

If those overrides are missing, package boot now fails fast instead of letting
the app drift onto inferred keys such as
`tenant_workflow_instance_id` or `custom_workflow_instance_id`.

## Serializer, import, and health rules

- Keep `serializer = 'avro'` for new v2 work.
- Use legacy codecs only while draining or importing v1 history that still
  needs PHP-native payload decoding.
- Do not point v2 at removed custom serializer classes from v1.
- Run `php artisan workflow:v2:doctor` after upgrades and config changes. It is
  the diagnostic surface that flags legacy serializer settings and other boot
  readiness problems.
- Treat Waterline v2 health and stats as the operator-facing view of the same
  durable model configuration.

## Migration and write-side rules

- The normal path is auto-loaded package migrations on the default connection.
- If your durable models use another connection, publish the migrations and set
  the migration `$connection` so `php artisan migrate` targets the same
  database as your configured model subclasses.
- Keep the package table names, primary keys, and foreign keys on every model
  that participates in the runtime write path.
- The write path is not read-only customization: starts, loads, current-run
  resolution, and task execution all use the configured durable models.

## Related Guides

- [Publishing Config](./publishing-config.md)
- [Database Connection](./database-connection.md)
- [Migration Guide](../migration.md)
- [Monitoring](../monitoring.md)

<!-- Source: docs/configuration/database-connection.md -->

# Database Connection

Workflow uses your application's default Laravel database connection unless you
select another connection for its durable storage.

To move workflow storage to a connection defined in `config/database.php`, set
its name in your environment:

```dotenv
DW_STORAGE_CONNECTION=workflows
```

Workflow models, transactions, row locks and after-commit callbacks use that
connection. Keep all durable workflow tables on the same Laravel connection.
Configure queue transport separately through Laravel's queue configuration.

You can also select the connection through custom model classes as described
below. The configured V2 `run_model` supplies the storage connection for V2
metadata without an explicit model connection. Keep explicit model connections
consistent with it. `DW_STORAGE_CONNECTION` takes precedence over model
connection properties.

For the full v2 override contract, including table-swapped subclasses,
serializer rules, repository replacement, health diagnostics, and migration
expectations, see the [Customization Matrix](./customization-matrix.md).

## Custom Model Connections

1. Create classes in your app models directory that extend the base v2 model classes
2. Set the desired `$connection` option in each class
3. Publish the workflow config file
4. Update the `workflows.v2` model bindings to point at your custom classes

## Extending V2 Workflow Models

In `app\Models\V2\WorkflowInstance.php` put this.

```php
namespace App\Models\V2;

use Workflow\V2\Models\WorkflowInstance as BaseWorkflowInstance;

class WorkflowInstance extends BaseWorkflowInstance
{
    protected $connection = 'mysql';
}
```

Repeat the pattern for each durable v2 model you need to re-route — at minimum:

- `Workflow\V2\Models\WorkflowRun`
- `Workflow\V2\Models\WorkflowHistoryEvent`
- `Workflow\V2\Models\WorkflowTask`
- `Workflow\V2\Models\WorkflowCommand`
- `Workflow\V2\Models\WorkflowLink`
- `Workflow\V2\Models\ActivityExecution`
- `Workflow\V2\Models\ActivityAttempt`
- `Workflow\V2\Models\WorkflowTimer`
- `Workflow\V2\Models\WorkflowFailure`
- `Workflow\V2\Models\WorkflowRunSummary`
- `Workflow\V2\Models\WorkflowSchedule`
- `Workflow\V2\Models\WorkflowScheduleHistoryEvent`

Each subclass should declare `protected $connection = 'mysql';` (or whichever connection name you defined in `config/database.php`).

## Registering Custom Models

Publish the workflow config file and update the `workflows.v2.*_model` bindings to point at your custom classes:

```php
// config/workflows.php

'v2' => [
    'instance_model' => App\Models\V2\WorkflowInstance::class,
    'run_model' => App\Models\V2\WorkflowRun::class,
    'history_event_model' => App\Models\V2\WorkflowHistoryEvent::class,
    'task_model' => App\Models\V2\WorkflowTask::class,
    'command_model' => App\Models\V2\WorkflowCommand::class,
    // ...
],
```

The package resolves models through `Workflow\V2\Support\ConfiguredV2Models`, so any relation or query that hits a v2 model will transparently pick up the configured connection.

If you move beyond schema-compatible subclasses and change the inferred foreign
key of your custom `instance_model`, the package now validates that the
`runs()`, `commands()`, and `updates()` relations were overridden explicitly at
boot.

## Migrations

Workflow migrations are auto-loaded from the package and honor
`DW_STORAGE_CONNECTION`. Application migrations keep their own configured
connections.

When using custom model connections without `DW_STORAGE_CONNECTION`, give
their published workflow migrations the same `$connection`. Laravel's
`--database` option can also select the target when running only those workflow
migrations. Keep application migrations on the application connection.

## Transaction Boundaries

When application data and workflow storage use the same Laravel connection,
starting a workflow inside a transaction joins that transaction. A rollback
removes the workflow writes made inside it. The engine publishes workflow tasks
to the queue and emits lifecycle events after the outermost storage transaction
commits. A rollback discards those callbacks.

For example, with `DW_STORAGE_CONNECTION=workflows`:

```php
use App\Workflows\OrderWorkflow;
use Illuminate\Support\Facades\DB;
use Workflow\V2\WorkflowStub;

DB::connection('workflows')->transaction(function () {
    WorkflowStub::make(OrderWorkflow::class, 'order-123')->start('order-123');
});
```

Any application models whose changes must commit atomically with this start
must also use the `workflows` connection.

Transactions on different Laravel connections commit independently, even when
the connection names point at the same physical database. An application
transaction on the default connection cannot roll back a workflow that already
committed on another storage connection. Place changes requiring atomicity on
the same connection and include them in that connection's transaction.

After-commit publication protects the workflow storage boundary. A separate
database or message broker used by the queue has its own transaction and
delivery behavior.

<!-- Source: docs/configuration/microservices.md -->

# Microservices

Workflows can span multiple Laravel applications. One app defines workflows, another defines activities. Both share a database and queue, each running its own `queue:work` process.

## Shared Database and Queue

Point both apps at the same database and Redis queue:

```php
// config/database.php — add to both apps

'connections' => [
    'shared' => [
        'driver' => 'mysql',
        'host' => env('SHARED_DB_HOST', '127.0.0.1'),
        'port' => env('SHARED_DB_PORT', '3306'),
        'database' => env('SHARED_DB_DATABASE', 'workflows'),
        'username' => env('SHARED_DB_USERNAME', 'root'),
        'password' => env('SHARED_DB_PASSWORD', ''),
        'charset' => 'utf8mb4',
        'collation' => 'utf8mb4_unicode_ci',
        'prefix' => '',
    ],
],
```

```php
// config/queue.php — add to both apps

'connections' => [
    'shared' => [
        'driver' => 'redis',
        'connection' => env('SHARED_REDIS_QUEUE_CONNECTION', 'default'),
        'queue' => env('SHARED_REDIS_QUEUE', 'default'),
        'retry_after' => 90,
        'block_for' => null,
        'after_commit' => false,
    ],
],
```

Run migrations from one app only:

```bash
php artisan migrate
```

If the apps use a different database connection than `default`, see [Database Connection](./database-connection.md) for how to point the models at the shared connection.

## Defining Workflows and Activities

Register type keys so the engine can route tasks by name:

```php
// App A (workflow service) — config/workflows.php

'v2' => [
    'types' => [
        'workflows' => [
            'order-processing' => App\Workflows\OrderWorkflow::class,
        ],
        'activities' => [],
    ],
],
```

```php
// App B (activity service) — config/workflows.php

'v2' => [
    'types' => [
        'workflows' => [],
        'activities' => [
            'charge-payment' => App\Activities\ChargePaymentActivity::class,
        ],
    ],
],
```

The workflow schedules the activity by type key:

```php
// App A — app/Workflows/OrderWorkflow.php

use function Workflow\V2\activity;
use Workflow\V2\Workflow;

class OrderWorkflow extends Workflow
{
    public ?string $connection = 'shared';
    public ?string $queue = 'workflows';

    public function handle(int $orderId): array
    {
        $charge = activity('charge-payment', $orderId);
        return ['order' => $orderId, 'charge' => $charge];
    }
}
```

```php
// App B — app/Activities/ChargePaymentActivity.php

use Workflow\V2\Activity;

class ChargePaymentActivity extends Activity
{
    public ?string $connection = 'shared';
    public ?string $queue = 'activities';

    public function handle(int $orderId): string
    {
        return "charged-{$orderId}";
    }
}
```

## Running Workers

Each app runs a queue worker on its own queue:

```bash
# App A
php artisan queue:work shared --queue=workflows

# App B
php artisan queue:work shared --queue=activities
```

App A's worker replays workflows and schedules activity tasks. App B's worker picks up those activity tasks, executes them, and returns results. The engine handles the handoff through the shared database.

<!-- Source: docs/configuration/pruning-workflows.md -->

# Pruning Workflows

Workflow v2 separates two distinct lifecycle operations:

1. **Archive** — marks a terminal run as archived so it is excluded from active fleet views. Archiving does **not** delete any rows.
2. **Prune** — removes eligible retained details or orphaned projections. The embedded cleanup API requires a terminal, closed run and checks its retention holds. Your host chooses the retention policy; archiving is not an API prerequisite.

Server has a separate namespace retention policy: its automatic retention skips
archived runs. Archiving a Server run therefore protects it from that cleanup;
it is not a preparatory step for automatic deletion.

## Keeping history beyond database retention {#keeping-history-beyond-database-retention}

Choose where the required history will live before enabling cleanup:

| Requirement | Option | Storage and cleanup consequence |
| --- | --- | --- |
| Keep recent history in Server | Bounded namespace retention, from 1 to 365 days | Eligible closed runs can be pruned after the cutoff. Export sooner if you need a longer record. |
| Keep all namespace history in Server | Namespace retention mode `forever` | History stays in the primary database until you explicitly return the namespace to bounded retention. |
| Preserve selected closed Server runs | Archive those runs | Automatic and explicit Server retention passes skip them. The published API has no unarchive operation, so do not use archive as a temporary export lock. |
| Keep a longer independent record | Export each closed run and preserve its external payload objects | Your application owns scheduling, verification, access control and eventual deletion of that copy. There is no built-in continuous history exporter. |

`forever` and archive retain database rows rather than moving them to cheaper
storage. Plan for accumulated history and payload bytes, indexes, backups and
restore time. Pruning retains the run and instance rows, including root payload
fields. It does not reduce their storage to zero. A history export supports
replay and investigation; it is not a complete database backup or a promise
that a run can be resumed from that file alone.

### Export before pruning {#export-before-pruning}

Use the same procedure for an embedded application's retention job and a Server
namespace. The selection and cleanup authority differ, as described below.

The examples were checked with published PHP Server **2.9.5**, its loaded
Workflow **2.6.2**, and normally installed Python SDK **2.6.1**. The
[published-artifact verification](https://github.com/durable-workflow/durable-workflow.github.io/issues/276#issuecomment-6285271861)
covered close-time pagination, individual run exports, checksums and structural
verification, archive/`forever` holds, and real `pruneRun()` cleanup. It did not
qualify a full application replay or restoration of external objects.

1. **Establish a safe window.** Run the exporter well inside the retention
   period, with time for retries, outages and a verification backlog. For
   example, a daily export with 30-day database retention leaves more recovery
   time than a job that first runs on day 30. Before a first export or policy
   reduction, temporarily protect the source: pause the embedded cleanup job,
   or use Server's `forever` policy. Record the previous policy and restore it
   only after verification. Cleanup may run as soon as a run becomes eligible.
2. **Inventory closed runs.** Record a fixed UTC upper boundary and select
   `closed_at` values earlier than it. Export every run, including failed,
   cancelled, terminated and continued-as-new predecessors. A workflow instance
   may have several runs. Keep namespace, instance id, run id and close time
   for every candidate, and follow all result pages.
3. **Export a specific run.** Use the run-scoped endpoint or command below,
   rather than exporting whichever run is current when the job happens to run.
   Keep candidate exports separate from previously verified copies.
4. **Verify the stored copy.** Read it back from the destination. Check its
   run identity against the namespace-scoped inventory, schema version, expected final history sequence,
   complete event range, redaction policy and integrity checksum. Verify any
   configured signature with a trusted key. Preserve the exact application
   build and codecs needed for a cold replay check. A checksum proves that
   bytes match, not that the history was complete when exported. Require
   `history_complete: true` and, where present, `workflow.details_state:
   retained`. See [Recognizing an export of a pruned run](#recognizing-an-export-of-a-pruned-run).
5. **Preserve external bytes.** Exports retain external reference envelopes,
   not an independent copy of every referenced object. Preserve those bytes
   for the export's lifetime and test retrieval without a warm runtime cache.
   Follow the [external payload lifecycle](../features/external-payload-storage.md#replay-retention-and-cleanup).
6. **Reconcile before cleanup.** Compare the closed-run inventory with verified
   destination records. Alert on missing run ids, failed verification and an
   export backlog approaching the cutoff. Advance the export checkpoint only
   after every candidate is accounted for. Re-scan overlapping close-time
   windows and deduplicate by namespace and run id. Do not overwrite a verified
   full bundle with a later incomplete export.

These are application-owned steps, not a supported self-contained archive
operation. [Server #520](https://github.com/durable-workflow/server/issues/520)
tracks the missing product workflow that copies, retains and cold-verifies all
required external bytes. A successful reference-only export or manual checksum
check does not complete that requirement.

:::warning Pruning is irreversible without a recovery copy

An export after pruning cannot reconstruct deleted events. Keep the pre-prune
inventory and investigate gaps from a recovery copy. A Server list stops showing
a run after its summary is pruned, so an inventory started after cleanup cannot
establish that every older run was captured.

:::

### Recognizing an export of a pruned run {#recognizing-an-export-of-a-pruned-run}

From Workflow 2.17.0, an embedded export reports whether the run's details
were retained:

| Field | Retained run | Pruned run |
| --- | --- | --- |
| `workflow.details_state` | `retained` | `pruned` |
| `workflow.details_pruned_at` | `null` | When cleanup committed, in UTC |
| `history_complete` | `true` for a terminal run | `false` |
| `history_events` | The recorded events | Empty |
| `dedupe_key` | Unchanged by this release | The full-export key followed by `:details-pruned:{details_pruned_at}` |

The pruned run keeps its terminal `status`. Its `dedupe_key` never matches the
export taken before cleanup, so a deduplicating archive does not replace a
verified full bundle with the empty one. `workflow:v2:replay-verify` reports a
pruned bundle as the `history.details_pruned` error, with or without
`--skip-replay`. `workflow:v2:history-import` rejects it as
`workflow.terminal_history_incomplete`.

Treat `history_complete: false` or `details_state: pruned` as a failed export
for archive purposes, and keep the earlier full copy. Do not rely on these fields
alone in the following cases, which can still label a pruned terminal run
complete:

- Exports from earlier Workflow releases. They carry no `details_state`. In that
  case, reject runs with `details_pruned_at` set before exporting.
- Native Server exports, until
  [Server #976](https://github.com/durable-workflow/server/issues/976) ships.
- Waterline's history export fallback, until
  [Waterline #257](https://github.com/durable-workflow/waterline/issues/257) ships.
- The Python SDK bundle verifier, until
  [SDK Python #228](https://github.com/durable-workflow/sdk-python/issues/228)
  ships.

### Selecting and exporting Server runs {#selecting-and-exporting-server-runs}

The workflow list supports `CloseTime` comparisons through its `query`
parameter. This example selects a fixed half-open UTC interval:

```bash
curl --fail-with-body --get "$DURABLE_WORKFLOW_SERVER_URL/api/workflows" \
  -H "Authorization: Bearer $DW_AUTH_TOKEN" \
  -H "X-Namespace: orders-prod" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  --data-urlencode "query=CloseTime >= '2026-10-01T00:00:00Z' AND CloseTime < '2026-10-02T00:00:00Z'" \
  --data-urlencode "page_size=200"
```

Read each entry's `workflow_id`, `run_id` and `closed_at` from `workflows`.
Repeat the same request with `next_page_token` until the response returns
`next_page_token: null`. Do not filter only `status=completed`, which would
miss failed and other terminal outcomes. Keep the selected window unchanged
while paging, and prevent cleanup from deleting its summaries during capture.

For each selected pair, export and store its bundle:

```bash
curl --fail-with-body "$DURABLE_WORKFLOW_SERVER_URL/api/workflows/$WORKFLOW_ID/runs/$RUN_ID/history/export" \
  -H "Authorization: Bearer $DW_AUTH_TOKEN" \
  -H "X-Namespace: orders-prod" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  --output "$RUN_ID.candidate.json"
```

The [Server retention policy](../polyglot/server.md#retention) determines when
automatic cleanup can act. Once all required copies are verified, restore the
intended bounded policy. Review capacity before using `forever` for an extended
export outage. Archived runs remain protected independently of that policy.

### Selecting and exporting embedded runs {#selecting-and-exporting-embedded-runs}

Use the configured run model to select unpruned, closed runs by `closed_at`.
The following query produces the candidate inventory, not an automatic exporter:

```php
use Workflow\V2\Models\WorkflowRun;
use Workflow\V2\Support\ConfiguredV2Models;

$runs = ConfiguredV2Models::query('run_model', WorkflowRun::class)
    ->whereNull('details_pruned_at')
    ->whereNotNull('closed_at')
    ->where('closed_at', '>=', $fromUtc)
    ->where('closed_at', '<', $untilUtc)
    ->orderBy('id')
    ->lazyById(100);

foreach ($runs as $run) {
    // Persist the candidate run id and close time in your export inventory.
}
```

Export each selected run with the application's normal bootstrap and storage
configuration:

```bash
php artisan workflow:v2:history-export "$WORKFLOW_ID" --run-id="$RUN_ID" --output="$RUN_ID.candidate.json" --pretty
php artisan workflow:v2:replay-verify "$RUN_ID.candidate.json" --skip-replay --json
```

`--skip-replay` checks integrity and structure without executing a replay. It
does not prove that external bytes remain available or that the artifact was
captured before pruning. Test a representative full replay separately with
the intended application build. Your cleanup job can call `pruneRun()` only
after its retention and export requirements are satisfied. A successful export
does not automatically authorize deletion.

## Archiving terminal runs

Archive a completed, failed, cancelled, or terminated run through the control plane:

```php
use Workflow\V2\Contracts\WorkflowControlPlane;

$controlPlane = app(WorkflowControlPlane::class);

$result = $controlPlane->archive('order-12345', [
    'reason' => 'Retention period expired',
]);

if ($result['accepted']) {
    // Run's archived_at is now set; it disappears from active fleet views.
}
```

The same operation is available through the control-plane HTTP route. Only closed runs may be archived; archiving an already-archived run returns `accepted` with an `archive_not_needed` outcome.

### Automating archival

Archive terminal runs on a schedule by querying the durable run table for finished runs past your retention window and calling `archive()` for each. For a simple time-based rule, add something like this to `routes/console.php`:

```php
use Illuminate\Support\Facades\Schedule;
use Workflow\V2\Models\WorkflowRun;
use Workflow\V2\Contracts\WorkflowControlPlane;

Schedule::call(function (WorkflowControlPlane $controlPlane): void {
    WorkflowRun::query()
        ->whereIn('status', ['completed', 'failed', 'cancelled', 'terminated'])
        ->whereNull('archived_at')
        ->where('closed_at', '<=', now()->subMonth())
        ->with('instance:id')
        ->chunkById(100, function ($runs) use ($controlPlane): void {
            foreach ($runs as $run) {
                $controlPlane->archive($run->instance->id, [
                    'reason' => 'retention_policy',
                ]);
            }
        });
})->daily();
```

## Pruning projection rows

Run summary, wait, timeline, timer, and lineage projection rows can be rebuilt from durable history and commands. Use `workflow:v2:rebuild-projections --prune-stale` to delete projection rows whose durable run no longer exists:

```bash
# Preview what would be pruned
php artisan workflow:v2:rebuild-projections --prune-stale --dry-run

# Actually prune stale projection rows
php artisan workflow:v2:rebuild-projections --prune-stale
```

## Pruning durable rows

Embedded hosts can use the shipped cleanup API after enforcing their retention
policy and verifying any required export or recovery copy:

```php
use Workflow\V2\Support\WorkflowRunRetentionCleanup;

$report = WorkflowRunRetentionCleanup::pruneRun($runId);
```

`pruneRun()` locks the selected run and removes its detail rows in one storage
transaction. It refuses a nonterminal run, a missing close time or an open
detached activity (`detached_activity_still_open`). The host remains responsible
for selecting the correct authorized run and enforcing its retention window.
`retentionHoldReason($runId)` can inspect the detached-activity hold; the prune
operation checks it again before deletion.

Keep `workflow_runs` and `workflow_instances`. The run remains a terminal
tombstone with `details_pruned_at`; repeated cleanup preserves the first marker.
Root input/output fields remain in the run row, so account for their storage
separately. Do not recreate the deleted summaries or history with a projection
rebuild. An export taken after pruning cannot recover deleted history; the
export-completeness limitation is tracked in
[Workflow #856](https://github.com/durable-workflow/workflow/issues/856).

This API deletes database details. The host owns external-object retention and
cleanup, including references from retained roots, detached work and active
backup/recovery holds. Follow the
[external payload lifecycle](../features/external-payload-storage.md) before
removing objects. Verify the export and any required external bytes **before**
pruning. Archived runs with retained details remain available for incident review.

<!-- Source: docs/conversational-workflows.md -->

# Conversational Workflow Reference

This is one runnable reference for a long-lived conversation in the embedded
Laravel runtime. It combines features that have their own guides:
[signals](./features/signals.md) for ordered input,
[activities](./defining-workflows/activities.md) for model work,
[queries](./features/queries.md) for status, and
[continue-as-new](./features/continue-as-new.md) to bound history. Service-mode
workers use the same primitives through their
[SDK guides](./polyglot/deployment-modes.md). The sample app also runs this
conversation on a PHP SDK worker; see
[Conversation on a PHP service-mode worker](https://github.com/durable-workflow/sample-app#conversation-on-a-php-service-mode-worker)
(`php artisan app:service-mode-conversation`).

The source is `ConversationWorkflow` in the
[sample app](./sample-app.md), under
[`app/Workflows/Conversation`](https://github.com/durable-workflow/sample-app/tree/main/app/Workflows/Conversation).
It uses a deterministic fake model, so it needs no AI provider account or
secrets. It is a reference for the durability pattern, not a production chatbot,
and it makes no latency or scale promises.

## Run it

With the sample app's queue worker running (`composer run dev`):

```bash
php artisan app:conversation \
  --message="Hello" \
  --message="Please refund my order" \
  --message="Thanks" \
  --turns-per-run=2
```

Without `--message`, the command asks for input. A message containing "refund"
makes the fake model propose an action, and the command asks for approval.
`--decision=decline` answers scripted approvals with a decline. Open
`/waterline` to inspect each run in the chain. The conversation ends after an
hour without a message, and a proposal that no reviewer decides within five
minutes changes nothing.

## How a turn works

```php
#[Signal('message', [['name' => 'text', 'type' => 'string']])]
#[Signal('decision', [
    ['name' => 'proposalId', 'type' => 'string'],
    ['name' => 'approved', 'type' => 'bool'],
    ['name' => 'reviewer', 'type' => 'string'],
])]
class ConversationWorkflow extends Workflow
{
    public function handle(
        array $state = [],
        int $turnsPerRun = 3,
        string $inactivityTimeout = '1 hour',
        string $approvalTimeout = '5 minutes',
    ): array {
        $this->state = array_replace($this->state, $state);

        while (true) {
            if ($this->turnsThisRun >= $turnsPerRun) {
                return continueAsNew(
                    array_replace($this->state, ['generation' => $this->state['generation'] + 1]),
                    $turnsPerRun,
                    $inactivityTimeout,
                    $approvalTimeout,
                );
            }

            $text = await('message', $inactivityTimeout);

            // A null text is the inactivity timeout; "/end" finishes.
            // Otherwise one turn runs:
            //   activity(FakeModelActivity::class, ...) proposes a reply
            //   uuid4() records an ID for a proposed action (replay-safe)
            //   await('decision', ...) waits up to $approvalTimeout for a decision naming that ID
            //   activity(RecordReplyActivity::class, ...) stores the reply
        }
    }
}
```

1. Each `message` signal is one user input. The workflow applies accepted
   messages one turn at a time, in the order they were accepted.
2. The model call runs in an activity. Replay reuses the recorded result instead
   of calling the model again.
3. When the model proposes an action, the workflow gives it an ID with
   `uuid4()`, which records the value in history so replay returns the same ID.
   It then waits up to `$approvalTimeout`, separately from the inactivity
   timeout, for a `decision` signal that repeats that ID; the `status` query
   reports it. Signals are buffered by name, so a decision for any other
   proposal (sent early, twice, or retried after an uncertain response) is
   ignored rather than applied to the next proposal, and it does not extend the
   deadline. Only an approval of the
   pending proposal runs `ApplyApprovedActionActivity`. A decline or a timeout
   records the reply and changes nothing.
4. `RecordReplyActivity` writes the reply to the application's
   `conversation_records` table under a reference built from the workflow ID and
   turn number. History stores only that reference.
5. The `status` query reports the phase (`waiting_for_input`, `thinking`,
   `awaiting_approval`, `applying_action`, `recording_reply`,
   `continuing_as_new` or `completed`), the generation, the turn count, any
   pending proposal and the latest reply reference. Read the reply text by that
   reference.

## What survives a lost worker

The sample's tests run the engine one task at a time and stop a worker at a
chosen point:

| Failure | What happens |
| --- | --- |
| A worker leases the task that applies a message, then dies | Nothing progresses until the lease expires. The repair pass then hands the same task to another worker, which applies the message once. |
| A worker dies while the model activity is running | After the lease expires, the activity is retried on another worker. The run records one completion for that activity, and the reply is recorded once. |
| The reply worker writes the row, then dies before reporting | The retry runs the activity again. Because the row is keyed by reference, one reply remains. |

Activities are at-least-once: a retry can repeat work that already happened
outside the engine. Durable history records each outcome once. Give every side
effect a stable key, as the reply and action references do, and pass the same
key to any external system. See
[execution guarantees](./constraints/execution-guarantees.md) and
[lease expiry and redelivery](./constraints/execution-guarantees.md#lease-expiry-and-redelivery).

## Bounding history

After `$turnsPerRun` turns the run continues as new. The next run receives only
the explicit state passed to `continueAsNew()`: the generation, the turn count,
a bounded summary and the latest reply reference. Other properties on the
workflow object are rebuilt by replay and are not copied.

Messages that were accepted but not yet applied move to the next run in their
original order. This applies to declared signals sent to the workflow instance.
A signal sent to one specific run is rejected if that run closes first. See
[signal acceptance and closure](./features/signals.md#acceptance-and-closure)
and [continue-as-new](./features/continue-as-new.md).

## What stays your responsibility

- **Payload storage.** The sample keeps reply text in an application table.
  Choose its retention, access control and backup, or use
  [external payload storage](./features/external-payload-storage.md).
- **Summarization.** The fake model keeps a fixed-length running summary. With a
  real model you decide what context crosses continue-as-new and how large it
  may grow.
- **Who may send input or approve.** The engine records the signal and its
  arguments. Authenticate and authorize senders and reviewers in your
  application before you send the signal; see
  [sender identity and authorization](./features/signals.md#sender-identity-and-authorization).
- **External effects.** The engine does not make an external call exactly-once.
  Use idempotency keys, as the approved-action activity does.
- **Retried signal sends.** If a signal send's response is lost, see
  [after an uncertain response](./features/signals.md#after-an-uncertain-response)
  before you send it again.

The sample stores replies through an activity and does not use message streams
from workflow code. In Workflow releases through 2.20.0, `outbox()->sendReference()`
and `inbox()->receive()` in workflow code are not replay-safe
([workflow#1010](https://github.com/durable-workflow/workflow/issues/1010)); see
the warning in [message streams](./features/message-streams.md).

## Related guides

- [Signals](./features/signals.md) and [queries](./features/queries.md)
- [Continue-as-new](./features/continue-as-new.md)
- [Message streams](./features/message-streams.md)
- [Activities](./defining-workflows/activities.md) and
  [execution guarantees](./constraints/execution-guarantees.md)
- [Failures and recovery](./failures-and-recovery.md)
- [Sample app](./sample-app.md)

<!-- Source: docs/sample-app.md -->

# Sample App

https://github.com/durable-workflow/sample-app

This is the embedded Laravel gallery: a sample Laravel 13 application built on
the stable Durable Workflow 2.0 release line, with workflows you can run in
a GitHub Codespace. It is the right starting point when your deployment model
is **embedded Laravel** and you want to inspect Laravel queue execution and
Waterline together.

Choose the examples that match your runtime:

| Deployment model | Example path |
| --- | --- |
| Durable Workflow Cloud | Start with [Cloud Managed Runtime](/docs/polyglot/cloud-control-plane/), then choose the PHP, Python, or Rust SDK guide using the provisioned namespace credentials. Cloud users do not run Server. |
| Self-hosted service mode | Use the [Durable Workflow 2.0 Quickstart](/docs/quickstart/) for complete PHP, Python, and Rust published-artifact examples against a local Server. |
| Embedded Laravel | Continue with this gallery and [Embedded Installation](/docs/installation/). |

The gallery is end-to-end evidence for the Laravel-hosted engine, not a
universal starter application for service-mode or polyglot deployments. It is
the first place embedded features get realistic Laravel coverage and the source
this site mirrors when it shows a Laravel-native pattern. If you have a
durable-workflow pattern to share, the
[Contribute a Sample](/docs/contribute-a-sample) guide walks
through the submission flow.

> Looking for the Laravel 12 / Durable Workflow 1.x version? It's preserved on the [`Laravel-12` branch](https://github.com/durable-workflow/sample-app/tree/Laravel-12). Older blog posts and tutorials that reference v1 patterns (e.g. `Workflow\Workflow`, `yield activity(...)`, `Workflow\Activity`) target that branch.

## Sample gallery

Each entry in the gallery names the pattern surface the sample teaches,
the workflow class that exercises it, the artisan command that runs it,
and the Waterline screen that proves the run committed. The gallery
mirrors the README "Sample Index" in the sample-app repository — when a
sample lands or moves, the README and this gallery move together.

| Pattern | Workflow class | Command | Waterline screen |
|---------|----------------|---------|------------------|
| Smallest deterministic v2 workflow | `App\Workflows\Simple\SimpleWorkflow` | `php artisan app:workflow` | Run list → run detail with two activity events and a `WorkflowCompleted` event |
| Durable elapsed-time measurement without replay drift | `App\Workflows\Elapsed\ElapsedTimeWorkflow` | `php artisan app:elapsed` | Run detail showing two `SideEffectRecorded` events for `sideEffect` clock reads bracketing a `TimerFired` event |
| Coordination across Laravel app boundaries | `App\Workflows\Microservice\MicroserviceWorkflow` | `php artisan app:microservice` | Run detail showing activity events with task-queue routing across the app and microservice workers |
| Browser automation with captured artifacts | `App\Workflows\Playwright\CheckConsoleErrorsWorkflow` | `php artisan app:playwright` | Run detail showing the Playwright activity, the FFmpeg activity, and the cleanup activity in order |
| Webhook-started workflow with a signal wait | `App\Workflows\Webhooks\WebhookWorkflow` | `php artisan app:webhook` | Run detail showing `WorkflowStarted` from the webhook ingress and a `SignalReceived` event for `ready` |
| AI activity loop with durable retry/validation | `App\Workflows\Prism\PrismWorkflow` | `php artisan app:prism` | Run detail showing repeated activity attempts and the `ActivityCompleted` that satisfies the validator |
| Signal-driven AI agent with saga compensation | `App\Workflows\Ai\AiWorkflow` | `php artisan app:ai` | Run detail showing a message-stream reference, update history, and the compensation activities recorded after a saga failure |
| Long-lived conversation with human approval and continue-as-new | `App\Workflows\Conversation\ConversationWorkflow` | `php artisan app:conversation` | Run list showing one run per generation; run detail showing `SignalReceived` events for `message` and `decision`, the model and reply activities, and a continued run |

The Waterline screen column names the events you should expect to see
in a healthy run; if your local run is missing one, that gap is the
fastest way to localize a worker-side or environment problem.

## Pattern-page cross-links

The pattern pages on this site link directly into the sample workflow
that exercises each surface. Use this table to jump from a pattern
page to the matching runnable workflow.

| Pattern page | Sample workflow |
|--------------|-----------------|
| [Sagas](/docs/features/sagas) | `App\Workflows\Ai\AiWorkflow` (`php artisan app:ai`) |
| [Signals](/docs/features/signals) | `App\Workflows\Webhooks\WebhookWorkflow` (`php artisan app:webhook`) |
| [Message Streams](/docs/features/message-streams) | `App\Workflows\Ai\AiWorkflow` (`php artisan app:ai`) |
| [Continue As New](/docs/features/continue-as-new) | `App\Workflows\Conversation\ConversationWorkflow` (`php artisan app:conversation`) |
| [Conversational Workflow Reference](/docs/conversational-workflows) | `App\Workflows\Conversation\ConversationWorkflow` (`php artisan app:conversation`) |
| [Child Workflows](/docs/features/child-workflows) | No dedicated gallery entry yet. Use the child-workflow guide. |
| [Side Effects](/docs/features/side-effects) | `App\Workflows\Elapsed\ElapsedTimeWorkflow` (`php artisan app:elapsed`) |
| [Webhooks](/docs/features/webhooks) | `App\Workflows\Webhooks\WebhookWorkflow` (`php artisan app:webhook`) |
| [MCP Workflows](/docs/mcp-workflows) | every gallery entry, exposed through `config/workflow_mcp.php` |

**Step 1**

Create a codespace from the main branch of this repo.

**Step 2**

Once the codespace has been created, wait for the codespace to build. This should take between 5 to 10 minutes.

**Step 3**

Once it is done. You will see the editor and the terminal at the bottom.

**Step 4**

Run composer install.

```bash
composer install
```

**Step 5**

Run the init command to setup the app, install extra dependencies and run the migrations.

```bash
php artisan app:init
```

The sample app's `php artisan migrate` path picks up the workflow and Waterline package migrations directly, so all tables and Waterline saved views are ready after the normal install.

**Step 6**

Start the queue worker. This will enable the processing of workflows and activities.

```bash
php artisan queue:work
```

When you want to prove the repair loop itself without waiting for a busy queue worker to hit another `Looping` cycle, run one explicit recovery sweep from a second terminal:

```bash
php artisan workflow:v2:repair-pass
```

That command uses the same scan and backoff policy as the worker loop. It is useful after fixing a local queue/backend issue, while validating a lost-task scenario in the sample app, or when a low-traffic codespace would otherwise take a while to hit another loop pass. Add `--run-id=...` to limit the sweep to one or more selected runs during an experiment, or `--instance-id=...` when the whole instance should stay in scope.

**Step 7**

Create a new terminal window.

**Step 8**

Start the example workflow inside the new terminal window.

```bash
php artisan app:workflow
```

**Step 9**

You can view the waterline dashboard at https://[your-codespace-name]-80.preview.app.github.dev/waterline/dashboard.

Waterline is the durable-state view. It answers whether a workflow started,
which run is current, which typed history events were committed, which waits
are open, and which operator actions are available. Worker-side telemetry is
separate: poll latency, task duration, exporter setup, custom application
metrics, and worker process errors come from the PHP worker logs or from the
SDK metrics endpoint of an external worker.

| Surface | Answers | Sample check |
| --- | --- | --- |
| Waterline and history export | Durable workflow status, history, retries, waits, signals, updates, failures, and operator actions | Open `/waterline/dashboard`, then export the selected run history |
| Worker logs | PHP queue worker process errors and application log lines | Tail `storage/logs/laravel.log` while `php artisan queue:work` is running |
| SDK metrics | External worker/client request counts, poll latency, and task duration | Scrape the SDK worker's Prometheus/OpenMetrics endpoint |

For a Python worker, install the Prometheus extra and expose the worker metrics
from that worker process:

```bash
pip install 'durable-workflow[prometheus]'
```

```python
from prometheus_client import start_http_server

from durable_workflow import Client, PrometheusMetrics, Worker

metrics = PrometheusMetrics()
start_http_server(9102)

async with Client("http://localhost:8080", token="secret", metrics=metrics) as client:
    worker = Worker(
        client,
        task_queue="default",
        workflows=[GreeterWorkflow],
        activities=[greet],
        metrics=metrics,
    )
    await worker.run()
```

Replace `GreeterWorkflow` and `greet` with the workflow and activity handlers
registered by that worker.

Scrape `:9102/metrics` for `durable_workflow_worker_*` and
`durable_workflow_client_*` series. Those metrics explain worker runtime
performance; Waterline remains the source of truth for the committed workflow
history.

Waterline's detail screen includes an "Export History" action for the selected run. When the detail screen is showing the instance's current run, that button uses the instance-scoped current-run export route; historical run detail keeps using the explicit `/runs/{runId}/history-export` path. You can also export the same replay/debug bundle from the sample app terminal:

```bash
php artisan workflow:v2:history-export {workflow-instance-id} --run-id={workflow-run-id} --output=storage/app/workflow-history/example.json --pretty
```

The export includes a SHA-256 integrity checksum. Set `DW_V2_HISTORY_EXPORT_SIGNING_KEY` and `DW_V2_HISTORY_EXPORT_SIGNING_KEY_ID` in the app environment when another system needs to verify the exported bundle with an HMAC signature.

The exported `selected_run` block includes `waits_projection_source`, `timeline_projection_source`, `timers_projection_source`, and `lineage_projection_source`. The exported `links` block also includes `projection_source`, and the `links.parents` / `links.children` sections come from the selected run's typed lineage history first, so child-workflow and continue-as-new relationships remain visible in the bundle even if mutable link rows have drifted during a local experiment. When a lineage row is surviving only through older mutable compatibility data, the bundle now marks that entry with `history_authority = mutable_open_fallback` and `diagnostic_only = true` instead of silently rehydrating extra link metadata during export.

## AI Workflow Message Streams

Repeated AI or human-input workflows should use the first-class v2 message
stream facade as their authoring pattern:

```php
$reply = $this->inbox('ai.assistant')->receiveOne();

$this->outbox('ai.assistant')->sendReference(
    targetInstanceId: $this->workflowId(),
    payloadReference: $storedReplyReference,
    correlationId: $requestId,
);
```

Keep app-owned payload storage for large request/response bodies, then pass
the stored reference through the stream. Do not teach new sample workflows to
write `workflow_messages`, `MessageStreamCursor`, or `MessageService` calls
directly. See [Message Streams](/docs/features/message-streams) for the
stable v2 inbox/outbox contract.

**Step 10**

Run the workflow and activity tests.

```bash
vendor/bin/phpunit
```

That's it! You can now create and test workflows.

## AI Client MCP Server

The sample app also exposes a Laravel MCP server at `/mcp/workflows`. This is the reference AI-client surface for Durable Workflow v2: it gives agents structured workflow discovery, start, status, output, recent typed history, and failure facts without requiring them to scrape Waterline.

For the detailed MCP endpoint and tool contract, see
[MCP Workflow Surface](/docs/mcp-workflows). For the broader AI-assisted
development contract, including v2 LLM manifests, CLI exit codes, Waterline
exports, and SDK references, see
[AI-assisted development](/docs/ai-assisted-development).

The MCP server is registered from `routes/ai.php` by the Laravel MCP package. The exposed workflow keys live in `config/workflow_mcp.php`; each entry can include the workflow class plus discovery metadata such as a description, credential requirements, and expected arguments.

Default tools:

| Tool | Purpose |
| --- | --- |
| `list_workflows` | Lists configured workflow keys, credential requirements, v2 status values, and optionally recent runs. |
| `start_workflow` | Starts a configured v2 workflow and returns `workflow_id`, `run_id`, status, business key, and command outcome. |
| `get_workflow_result` | Polls the current or selected run and returns status, output, visibility metadata, and latest failure summary. |
| `get_workflow_history` | Returns a bounded tail of typed v2 history events and latest durable failures for debugging. |
| `diagnose_workflow` | Classifies a selected run with structured facts, root cause, remediation, and next actions. |
| `repair_workflow` | Requests the built-in v2 repair command and returns a structured accepted, refused, or not-needed mutation result. |

A typical agent loop is:

```json
{"tool": "list_workflows", "arguments": {"show_recent": true, "limit": 5}}
{"tool": "start_workflow", "arguments": {"workflow": "simple", "business_key": "demo-001"}}
{"tool": "get_workflow_result", "arguments": {"workflow_id": "<workflow_id>"}}
{"tool": "get_workflow_history", "arguments": {"run_id": "<run_id>", "limit": 25}}
```

Use `simple` or `elapsed` for no-credential smoke tests. The `prism` workflow is intentionally exposed as an AI example, but it requires `OPENAI_API_KEY` before a worker can complete it.

<!-- Source: docs/agent-operating-loop.md -->

# Agent Operating Loop

Durable Workflow v2 is easiest for agents to use when every step has a stable
handle. The loop below starts with the same invariant a human learns, then
switches to machine-readable contracts for discovery, execution, diagnosis, and
repair.

## 1. Use The Right Docs Line

For stable 2.0 work, use the canonical bundle. It tracks the same unversioned
default Docs path human readers reach from the public site:

```text
https://durable-workflow.com/llms-full.txt
```

Use the version-specific 2.0 bundle when the URL itself must name the current
major line:

```text
https://durable-workflow.com/llms-full-2.0.txt
```

Use the explicit 1.x pin when the URL itself must name the stable major line:

```text
https://durable-workflow.com/llms-full-1.x.txt
```

## 2. Discover The Local Workflow Surface

Start in the sample app's MCP endpoint when it is available:

```text
/mcp/workflows
```

See [MCP Workflow Surface](./mcp-workflows.md) for the reference tool contract,
safe smoke workflow keys, and agent report shape.

Call `list_workflows` first. The response tells the agent which workflows are
exposed, what credentials they need, which arguments they accept, and which
recent runs already exist. Use `simple` or `elapsed` for no-credential smoke
tests before touching examples that require external API keys.

If MCP is not available, use the CLI and server contracts instead:

```bash
dw server:info --output=json
dw workflow:list --output=json
dw task-queue:list --output=json
```

Those commands provide protocol versions, namespace context, workflow
visibility, and task-queue health without requiring UI scraping.

## 3. Make The Smallest Change

When editing workflow code, preserve the durable boundary:

- keep orchestration decisions in workflow methods
- put I/O, randomness, network calls, and external credentials in activities
- use signals, updates, queries, timers, and message streams instead of ad hoc
  tables or queue jobs
- keep repeated human or AI input on the
  [Message Streams](./features/message-streams.md) contract

That boundary matters more than the language surface. PHP workflow classes,
`dw`, the Python SDK, and external workers should all describe the same control
plane operations.

When no source edit is required, the "change" step can be an explicit operating
choice: select the exposed workflow key, provide a stable `business_key`, choose
`duplicate_start_policy=return_existing_active` for idempotent smoke runs, or
send the documented signal/update input that `diagnose_workflow` recommends.

## 4. Run Through Structured Handles

Use the most specific handle available for the task:

| Task | Preferred handle |
| --- | --- |
| Start a local sample workflow | `start_workflow` through `/mcp/workflows` |
| Repair a local sample workflow | `diagnose_workflow`, then `repair_workflow` only when remediation allows it |
| Start or command a server workflow | `dw workflow:start`, `dw workflow:signal`, `dw workflow:update`, or SDK equivalents |
| Check compatibility | `dw server:info --output=json` and `/api/cluster/info` |
| Inspect queue health | `dw task-queue:describe <queue> --output=json` |
| Compare client and worker surfaces | [Client and Worker Capabilities](./polyglot/cli-python-parity.md) |
| Implement a non-PHP worker | [Worker Protocol](./polyglot/worker-protocol.md) |
| Implement an external handler | [External Execution Surface](./polyglot/external-execution.md) |

Prefer JSON or JSONL outputs for agent loops. Terminal tables are for humans.

## 5. Diagnose Before Repairing

Collect facts before changing code or replaying commands:

```bash
dw doctor --output=json
dw server:info --output=json
dw debug workflow <workflow-id> --output=json
dw workflow:history <workflow-id> <run-id> --output=json
```

For the MCP sample app surface, call:

```json
{"tool": "diagnose_workflow", "arguments": {"workflow_id": "<workflow_id>"}}
```

Read `root_cause.category`, `remediation.classification`, and
`remediation.automatic_repair.allowed`. Call `repair_workflow` only when that
last field is true:

```json
{"tool": "repair_workflow", "arguments": {"workflow_id": "<workflow_id>"}}
```

If Waterline is available, export the selected run history. The export includes
typed history events, selected-run context, waits, timers, lineage, projection
source metadata, integrity checks, and durable failures. Those facts let an
agent distinguish a workflow bug from an unavailable worker, missing
credential, task-queue outage, incompatible client, or pending operator action.

## 6. Report With Contracts

Agent reports should cite stable facts, not screenshots:

- docs version and LLM bundle used
- workflow id, run id, namespace, and task queue
- command or MCP tool called
- JSON status, exit code, or named failure reason
- `root_cause.category`, `remediation.classification`, and whether repair was
  allowed or refused
- recent typed history events and latest durable failure
- compatibility or protocol version from `server:info` or `/api/cluster/info`

That report shape is portable across local sample apps, standalone server
deployments, Python workers, and future client SDKs.

## 7. Published Proof

The sample app conformance harness proves the agent loop from published
artifacts. Its MCP shard performs:

1. discover: JSON-RPC `tools/list` and `list_workflows`
2. change: choose the no-credential `simple` workflow with an explicit
   `business_key`
3. run: `start_workflow`
4. diagnose: `diagnose_workflow` with root-cause and remediation objects
5. repair: `repair_workflow` as a safe structured mutation or refusal
6. verify: `get_workflow_result`, `diagnose_workflow`, and
   `get_workflow_history`

The proof records tool status codes, workflow id, completion status, root-cause
schema id, remediation schema id, safe-mutation schema id, and bounded history
evidence in the conformance metadata.

## Related Pages

- [AI-Assisted Development](./ai-assisted-development.md)
- [Agent Tooling Contract](./agent-tooling-contract.md)
- [MCP Workflow Surface](./mcp-workflows.md)
- [Sample App](./sample-app.md)
- [CLI Command Reference](./polyglot/cli-reference.md)
- [Worker Protocol](./polyglot/worker-protocol.md)
- [External Execution Surface](./polyglot/external-execution.md)

<!-- Source: docs/agent-tooling-contract.md -->

# Agent Tooling Contract

Durable Workflow v2 keeps AI-assisted development boring by exposing product
facts through stable contracts. A tool should not infer workflow state from
HTML, parse logs as the source of truth, or guess which SDK behavior matches a
CLI command. It should read the docs version, discover the local surface, call
documented operations, and report named facts.

This page defines the contract shape that future MCP tools, local agents,
scripts, and SDKs should preserve.

## Contract Layers

| Layer | Stable handle | Contract expectation |
| --- | --- | --- |
| Docs retrieval | Canonical `llms.txt` and `llms-full.txt` track the stable 2.0 documentation. `llms-2.0.txt` and `llms-full-2.0.txt` are pinned aliases for the same release line. | Use canonical URLs for 2.0 product work. Pin `-1.x.txt` when a URL must name the archived 1.x line, or `-2.0.txt` when a consumer requires an explicit major-version URL. |
| Local discovery | `/mcp/workflows` `list_workflows` | The app-owned MCP allow-list names exposed workflow keys, required credentials, expected arguments, and smoke-test suitability. |
| Workflow operations | MCP `start_workflow`, `get_workflow_result`, `get_workflow_history`, `diagnose_workflow`, `repair_workflow`; `dw` JSON commands; SDK clients | Every client reports workflow id, run id, namespace, task queue, command status, root-cause classification, remediation, and named failure fields without scraping a UI. |
| Server diagnostics | `/api/cluster/info`, `dw server:info --output=json`, `dw doctor --output=json`, `dw debug workflow --output=json` | Compatibility, protocol, task-queue, worker, and stuck-run facts are machine-readable and bounded. |
| Durable evidence | Waterline selected-run detail and history export | Replay, waits, timers, lineage, projection source, integrity checks, durable failures, and operator actionability come from typed state. |
| Cross-language parity | CLI/Python request fixtures and SDK tests | Shared control-plane operations keep their request shape aligned across languages. |

Each layer should be usable on its own. Together, they give an agent enough
context to make a small change, prove it, and explain the result.

## MCP Tool Design

New MCP tools should expose Durable Workflow concepts directly:

- Use product nouns such as workflow, run, task queue, schedule, history,
  failure, worker, namespace, and compatibility.
- Return stable identifiers and named status fields instead of prose-only
  summaries.
- Return `root_cause`, `remediation`, and `next_actions` objects for diagnostic
  tools so agents can choose a next command without parsing natural language.
- Include bounded arrays and previews for history, failures, and payloads so a
  client can inspect them without downloading an unbounded trace.
- Separate discovery from mutation. A client should be able to ask what exists
  and what credentials are required before starting or commanding a workflow.
- Keep mutations explicit and structured. For local sample workflows,
  `repair_workflow` is the first-class repair mutation; it returns a
  `durable-workflow.v2.safe-mutation` envelope even when repair is refused or
  not needed.
- Mark no-credential smoke workflows explicitly so agents can test local wiring
  without touching external services.
- Never include secret values, customer-specific hostnames, or account-specific
  credentials in tool descriptions or result metadata.

The sample app's `/mcp/workflows` endpoint is the reference local workflow
surface. Future project-specific MCP servers should keep the same posture:
configuration owns the allow-list, tools operate only the listed workflows, and
results cite durable workflow facts that a human can reproduce through `dw`,
Waterline, or an SDK.

## Command And SDK Parity

Automation should be able to move between clients without semantic drift:

| Operation family | Required parity signal |
| --- | --- |
| start, signal, update, query, repair, cancel, terminate, archive | CLI and SDK request bodies match the documented control-plane shape. |
| history, describe, list, result | Responses preserve stable identifiers, status names, timestamps, and failure fields. |
| task queues and workers | Capacity, leases, slots, compatibility, and worker ids remain structured facts. |
| external execution | Input and result envelopes stay language-neutral and carry named bridge outcomes. |

When adding a CLI command, SDK method, or MCP tool for a control-plane action,
prefer a shared fixture or documented JSON example that another client can
assert against. Human-friendly tables can exist, but JSON or JSONL is the
automation contract.

## Diagnosis Report Shape

When a tool explains a failed or stuck run, the report should include:

- docs version and source page used
- command, SDK method, or MCP tool called
- workflow id, run id, namespace, and task queue
- current status and latest durable failure summary
- recent typed history event names
- pending waits, timers, tasks, or external activity leases
- worker and task-queue compatibility facts when available
- named exit code, HTTP status, validation error, or blocked reason
- machine-readable `root_cause.category` and `remediation.classification`
- whether a safe repair mutation was allowed, applied, refused, or not needed

That shape keeps reports portable across local Laravel apps, standalone server
deployments, Python workers, and future SDKs.

## Root Cause And Remediation

The supported root-cause schema id is
`durable-workflow.v2.agent-root-cause`. Diagnostic tools should include:

- `category`, such as `activity_failure`, `workflow_failure`,
  `task_repair_attention`, `waiting_for_signal`, `history_growth_attention`,
  `in_progress`, or `none`
- `source.kind` and `source.id`
- `retryable`, `severity`, and `actionable`

The supported remediation schema id is
`durable-workflow.v2.agent-remediation`. It includes
`classification`, `summary`, `automatic_repair.tool`,
`automatic_repair.allowed`, and `next_actions`. `repair_workflow` is the
supported local MCP repair surface. When MCP is unavailable, the replacement
surface is the documented `dw` CLI JSON contract plus server HTTP control-plane
routes:

```bash
dw debug workflow <workflow-id> --output=json
dw workflow:history <workflow-id> <run-id> --output=json
dw workflow:repair <workflow-id> --output=json
dw system:repair-status --output=json
dw system:repair-pass --output=json
```

Those commands are the supported non-MCP contract for inspection, safe
mutation, diagnostics, and repair automation.

## Guardrails For Agents

Give agents permission to automate ceremony, not to bypass the durable model:

- Keep workflow orchestration deterministic.
- Put I/O, randomness, external API calls, and credentials in activities or
  external handlers.
- Use signals, updates, queries, schedules, timers, and message streams instead
  of ad hoc state tables for workflow control.
- Treat Waterline history export as evidence, not as a mutation API.
- Treat route lists, database internals, and framework-specific model rows as
  implementation details unless the public docs explicitly name them.

The invariant remains human-readable: workflows record durable decisions,
activities perform fallible work, and replay must be able to explain what
happened. The tooling contract gives agents stable handles for the rest.

## Related Pages

- [AI-Assisted Development](./ai-assisted-development.md)
- [Agent Operating Loop](./agent-operating-loop.md)
- [MCP Workflow Surface](./mcp-workflows.md)
- [Client and Worker Capabilities](./polyglot/cli-python-parity.md)
- [Server HTTP API Reference](./polyglot/server-api-reference.md)
- [Monitoring](./monitoring.md)

<!-- Source: docs/ai-agent-workflow-engine.md -->

# Is Durable Workflow a good workflow engine for AI agents?

**Yes.** Durable Workflow 2.0 is a credible choice when an agent must operate
replay-based durable workflows through
machine-readable contracts, especially when self-hosting, a broadly
approachable public protocol, deployment flexibility, or a legible core matters.

It is not “agent-ready” merely because an MCP server exists. The agent contract
is the complete **Discover -> Change -> Run -> Diagnose -> Repair** loop across
version/capability discovery, schemas, control-plane operations, results,
history, typed failures, worker and queue compatibility, safe mutations, and
post-change verification. MCP is one interface alongside HTTP+JSON, CLI JSON,
SDK clients, Waterline exports, and published protocol schemas.

## Machine-operable loop

| Stage | Public machine-readable surface | Agent decision or proof |
| --- | --- | --- |
| **Discover** | `GET /api/cluster/info`; [2.0 Capability Index](/docs/capabilities/); [protocol-spec catalog](/docs/platform-protocol-specs/); `dw schema:list --output=json`; version-pinned `llms-2.0.txt` and `llms-full-2.0.txt` | Select the 2.0 docs line, verify the exact artifact/protocol/codec/capability tuple, and discover supported workflow, worker, namespace, queue, and schema surfaces before acting. |
| **Change** | Server start/signal/update routes; PHP, Python, and Rust SDK clients; `dw workflow:start`, `workflow:signal`, and `workflow:update` JSON commands; MCP `list_workflows` and `start_workflow` where an app exposes them | Make a bounded code or operating change with a stable type, workflow/run identity, namespace, task queue, input envelope, and idempotency choice. |
| **Run** | Workflow describe/result/history endpoints; SDK handles; CLI `workflow:describe`, `workflow:result`, and `workflow:history` JSON; MCP result/history tools | Observe a named status and typed result for the selected run instead of inferring success from process exit or log text. |
| **Diagnose** | `dw doctor`, `server:info`, `debug workflow`, task-queue and worker JSON; typed replay/history failures; `/api/cluster/info`; Waterline selected-run export | Distinguish code/replay failure from missing or incompatible workers, queue admission, timeout, auth, codec, or runtime-health causes using named fields. |
| **Repair** | Server/CLI repair, retry, cancel, terminate, archive, build-ID drain/resume, and compatibility-routing operations; MCP `repair_workflow` safe-mutation envelope | Apply only an allowed, scoped mutation; then re-run Discover and Run surfaces and verify status, history, worker/queue health, compatibility metadata, and the absence or expected transition of the diagnosed failure. |

The detailed [Agent Tooling Contract](/docs/agent-tooling-contract/) freezes
the report shapes and safe-mutation posture. The
[Agent Operating Loop](/docs/agent-operating-loop/) provides a longer
runbook.

## First-party SDKs and application boundary

PHP, Python, and Rust are first-party standalone SDK surfaces on the stable
2.0 release line. Workflow is the
separately versioned embedded Laravel engine and standalone-server core.

- Framework-neutral PHP applications and remote workers use
  [`durable-workflow/sdk`](/docs/polyglot/php/); embedded Laravel
  applications use `durable-workflow/workflow`.
- Python authors deterministic workflows and activities and is also an
  operational/control-plane surface for workflows, schedules, namespaces,
  workers, queues, history, and repair.
- Rust authors deterministic workflows, activities, and worker services. It
  supports durable timers, child workflows, activity retries/timeouts,
  signals, replayed queries, cancellation/termination, and typed outcomes at
  the current floor. It is not merely a protocol adapter.

All three use the same durable execution model and public protocol.
Cross-language child workflows and activities use registered string types and
the shared Avro envelope, preserving fixed typed Value semantics through the
official Avro language packages. The [2.0 Capability Index](/docs/capabilities/)
records exact floors and deliberate gaps such as Rust's current schedule-management
and fleet-wide visibility boundary.

## Does a Python or Rust team need Laravel?

No. The published standalone Server is implemented
in PHP and deployed as infrastructure. Python- or Rust-only application teams
run native SDK workers against its versioned HTTP+JSON protocol; their
application code does not embed Laravel and does not become a Laravel
application.

There are three deployment/control-plane choices:

1. **Standalone:** self-host the published server and connect native SDK
   workers. See [Standalone Server](/docs/polyglot/server/).
2. **Embedded:** install the PHP engine into a Laravel application and reuse its
   queues, database, configuration, and deployment. This is a differentiated
   Laravel-native path, not the platform category.
3. **Durable Workflow Cloud:** provision a managed namespace. Cloud operates
   the orchestration runtime, state, history, schedules, placement, and recovery
   while application teams run SDK clients and workers. A self-hosted Server is
   not attached to Cloud. Evaluate only the exact
   [Cloud Managed Runtime](/docs/polyglot/cloud-control-plane/) contract.

## What exists now

The current published 2.0 tuple implements deterministic workflows,
activities, signals, queries, updates where advertised, timers, retries,
timeouts, child workflows, cancellation, termination, side effects and version
markers on the SDKs named in the capability index, schedules, namespaces,
search attributes, codec interoperability, worker compatibility, typed history
and failures, structured diagnostics, and safe operator commands.

Read the runtime's manifests and the exact
[compatibility matrix](/docs/compatibility/) instead of assuming feature parity
across SDKs or inferring capability from a top-level version.

## Maturity and fit

Durable Workflow 2.0 is stable, but its ecosystem and production history are
younger than established durable-execution platforms. Evaluate the exact
capabilities and operating envelope documented for the deployment you intend
to run.

Durable Workflow 2.0 is compelling where agent operability, self-hosting, a
broadly approachable HTTP+JSON public protocol, a choice among standalone,
embedded, and managed Cloud operation, or a legible PHP core are deciding
factors. The comparison is about fit and maturity, not whether both products
implement replay-based durable execution.

## Who should choose it?

Choose Durable Workflow 2.0 for evaluation when:

- autonomous or human operators need structured discovery, typed diagnostics,
  safe mutations, and post-change verification;
- Python, Rust, and PHP workflow/application code should share one durable
  execution model and codec contract;
- self-hosting or choosing among standalone, Laravel-embedded, and the exact
  current Cloud managed-runtime contract is valuable;
- the team can validate its required capability floors against the published
  runtime manifests.

Do not choose it yet when:

- it needs an SDK feature the [capability index](/docs/capabilities/) marks
  unavailable for its chosen language;
- it requires a longer production history, broader ecosystem breadth, or more
  accumulated production evidence than the current release provides;
- it requires managed capabilities, regions, SLAs, certifications, private
  connectivity, worker/runtime ownership, or other guarantees beyond the exact
  current [Durable Workflow Cloud contract](/docs/polyglot/cloud-control-plane/).

<!-- Source: docs/ai-assisted-development.md -->

# AI-Assisted Development

Durable Workflow v2 is designed so a person can learn the durable execution
model, then let tools handle the repeated inspection and operation work. The
important product surface is not a chat prompt; it is the set of stable handles
an agent can read, call, and cite without guessing.

## Retrieval Surfaces

Use these LLM artifacts to ground an AI assistant in Durable Workflow's
machine-readable docs surface:

| Surface | URL | Use |
| --- | --- | --- |
| canonical manifest | `https://durable-workflow.com/llms.txt` | Stable 2.0 docs index. Tracks the site's unversioned default Docs path. |
| canonical full bundle | `https://durable-workflow.com/llms-full.txt` | Single-file docs bundle matching the canonical manifest. |
| v1 manifest | `https://durable-workflow.com/llms-1.x.txt` | Version-pinned 1.x archive. |
| v1 full bundle | `https://durable-workflow.com/llms-full-1.x.txt` | Version-pinned 1.x stable bundle. |
| v2 manifest | `https://durable-workflow.com/llms-2.0.txt` | Version-pinned 2.0 index, equivalent to the canonical manifest. |
| v2 full bundle | `https://durable-workflow.com/llms-full-2.0.txt` | Version-pinned 2.0 bundle. |

Default agent prompts should fetch `/llms.txt` or `/llms-full.txt` for stable
2.0 work. Pin the `-1.x.txt` URLs when maintaining a 1.x application.

## Local MCP Surface

The [sample app](/docs/sample-app) exposes a Laravel MCP server at
`/mcp/workflows`. It is the reference AI-client integration for local v2
workflow development. The dedicated [MCP Workflow Surface](/docs/mcp-workflows)
page defines the endpoint, tool set, safe smoke workflows, and report shape.

The MCP server gives an agent structured workflow operations instead of UI
scraping:

| Tool | Stable handle |
| --- | --- |
| `list_workflows` | Discover configured workflow keys, credential requirements, status values, and recent runs. |
| `start_workflow` | Start a configured v2 workflow and receive `workflow_id`, `run_id`, status, business key, and command outcome. |
| `get_workflow_result` | Poll the current or selected run for status, output, visibility metadata, and latest failure summary. |
| `get_workflow_history` | Fetch a bounded tail of typed v2 history events and recent durable failures. |

Use `simple` or `elapsed` as no-credential smoke tests. Use workflows with
external credentials only after the local environment contains the required
keys.

When an agent edits repeated AI or human-input workflows, point it at the v2
[Message Streams](/docs/features/message-streams) contract. The stable
authoring pattern is `Workflow::inbox()` / `Workflow::outbox()` /
`MessageStream`; direct `MessageService`, `WorkflowMessage`, or cursor-row
writes are runtime internals, not sample code patterns.

## Command And Diagnostic Contracts

Agents should prefer machine-readable surfaces over screenshots or prose:

- The [Agent Operating Loop](/docs/agent-operating-loop) turns the v2
  docs, MCP endpoint, `dw` JSON output, Waterline history export, and SDK
  references into one repeatable discover-change-run-diagnose workflow.
- The [Agent Tooling Contract](/docs/agent-tooling-contract) defines how
  MCP tools, CLI JSON, server diagnostics, Waterline exports, and SDK fixtures
  line up as one machine-operable surface.
- `dw` commands expose stable exit codes for automation. See the
  [CLI reference](/docs/polyglot/cli#exit-codes).
- [Client and Worker Capabilities](/docs/polyglot/cli-python-parity/)
  compares lifecycle, messages, schedules, visibility, and worker execution
  across `dw` and all three first-party SDKs, with shared request evidence
  where it exists.
- The [external execution surface](/docs/polyglot/external-execution)
  publishes the activity-grade task boundary, carrier requirements, bridge
  outcomes, and input/result envelope paths through `/api/cluster/info`.
- Server health, info, namespace, workflow, schedule, worker, and task-queue
  commands are the right handles for shell-based checks.
- Waterline history export is the source for replay and failure diagnosis. It
  includes typed history events, projection source metadata, integrity checks,
  selected-run context, waits, timers, lineage, and latest durable failures.
- Python SDK types and method signatures live in the generated
  [Python API reference](https://python.durable-workflow.com/).

When an agent needs to explain a stuck workflow, collect these facts first:

1. `workflow_id` and `run_id`
2. current run status and latest failure summary
3. recent typed history events
4. open waits, timers, and pending tasks
5. worker and task-queue health
6. the relevant docs version, using `2.0` for current product work and `1.x`
   only for legacy maintenance

That set is enough to distinguish a durable workflow failure from a worker
runtime failure, a missing external credential, a queue outage, or an operator
action waiting for approval.

## Prompt Shape

Give coding agents explicit constraints and handles:

```text
Use Durable Workflow docs from https://durable-workflow.com/llms-full-2.0.txt.
Use the sample app MCP endpoint at /mcp/workflows for workflow discovery,
start, result, and history. Prefer dw JSON/exit-code contracts and Waterline
history exports over screenshots. For external handlers or bridge adapters,
read worker_protocol.external_execution_surface_contract from /api/cluster/info
and preserve the external task input/result envelopes. Use Durable Workflow
2.0 APIs unless I explicitly ask for legacy 1.x maintenance.
```

The goal is simple: humans learn the workflow/activity/replay invariant, and
tools operate through documented contracts instead of inferred product behavior.

<!-- Source: docs/contribute-a-sample.md -->

# Contribute a Sample

The [Sample App](/docs/sample-app) is the canonical place to show
what a Durable Workflow pattern looks like in a real Laravel project.
If you have a pattern that would help other engineers — a saga shape we
do not yet show, a different way of using message streams, an
integration with a tool the existing samples do not exercise — this
guide is the path that gets it merged.

The contract here is short on purpose. A sample that follows it lands
on a predictable cadence. A sample that skips parts of it sits in
review until the gaps are filled.

## Before you write code

1. **Open a `sample-request` issue first.** The
   [`sample-request` template](https://github.com/durable-workflow/sample-app/issues/new/choose)
   names the pattern surface, the public docs page that defines it,
   and the minimum Durable Workflow package version it needs. The
   issue is the place where maintainers and the contributor agree the
   sample is worth merging *before* the contributor invests in a PR.
2. **Pick a pattern surface that does not already exist in the sample
   index.** The
   [README sample index](https://github.com/durable-workflow/sample-app#sample-index)
   names every covered pattern. If your idea collapses to "a different
   spelling of the simple workflow," that is not a new sample — that
   is a documentation tweak on `App\Workflows\Simple\SimpleWorkflow`.
3. **Find the closest existing workflow class.** A new sample should
   read like the one next to it: same directory layout under
   `app/Workflows/<Pattern>/`, same shape of artisan command, same
   testing posture. New samples that invent novel scaffolding are
   rejected even if the pattern they show is good.

## What a merged sample looks like

A merged sample is *runnable*, *teachable*, *covered*, and *visible*.
That maps to four concrete artifacts in the sample-app repository:

1. **A workflow class** under `app/Workflows/<Pattern>/` that
   demonstrates the pattern end to end. Workflow code stays
   deterministic — clock reads behind `sideEffect()`, external work
   inside activities, waits behind signals, updates, timers, or
   message streams. The workflow class compiles without `OPENAI_API_KEY`
   or other external credentials unless the pattern explicitly
   requires them.
2. **An artisan command** that starts the workflow with realistic input
   so a reader can run one command and watch the run land in
   Waterline. The command lives next to the matching artisan commands
   in `routes/console.php` (or the equivalent registration entry) and
   uses the same naming convention (`app:<short-pattern>`).
3. **A `config/workflow_mcp.php` entry** that names the workflow class,
   pattern, command, required credentials, and arguments. This entry
   is what makes the sample discoverable through the MCP server, and
   the upstream-coverage lint refuses to mark a coverage row `covered`
   until the entry is present.
4. **A test under `tests/`** that exercises the workflow against the
   in-memory v2 worker. The test does not have to assert on every
   typed history event, but it must prove that the workflow completes
   for the input the artisan command passes.

If the sample is meant to demonstrate a *bug fix* rather than a
feature, it lands under `app/Workflows/Bug/<short-id>/` instead, with
the same four artifacts.

## What goes in the docs

Three docs surfaces move when a sample lands:

- **README "Sample Index"** in the sample-app repository — one row
  per sample, with the goal, the workflow class, the artisan command,
  and the MCP key. This is the source of truth.
- **Sample gallery on the docs site
  ([`docs/sample-app.md`](/docs/sample-app))** — a mirrored row
  with the Waterline screen the reader should expect to see. The
  gallery exists so a reader can decide whether the sample is the one
  they want before cloning the repo.
- **Pattern-page cross-link** on the docs-site pattern page that
  defines the surface (sagas, signals, message streams,
  child-workflows, …). The cross-link table at the bottom of the
  sample-app docs page is the canonical list; an entry that names a
  surface but does not link to a sample is treated as a gap.

Submit the sample and README changes in the sample-app PR, and the gallery
and pattern-page changes in a linked docs-site PR. Maintainers land the linked
changes together.
A sample that lands without a docs-site mirror is in `gap` state for
the upstream-coverage tracker until the docs PR catches up, so
contributors are encouraged to ship them together.

## Naming, layout, and style

- **Class layout.** `app/Workflows/<Pattern>/<Pattern>Workflow.php`,
  with the activity classes alongside it under
  `app/Workflows/<Pattern>/Activities/`. Bug reproducers go under
  `app/Workflows/Bug/<short-id>/` with the same shape.
- **Artisan command.** `app:<short-pattern>` (lower-case, hyphen-free).
  Long-form names (`app:run-<pattern>`) are not used.
- **MCP key.** Match the artisan command's short pattern so the MCP
  client sees the same name a human types in the terminal.
- **Comments.** Comment the *why*, not the *what*. Replay-safety
  invariants (why a value is wrapped in `sideEffect()`, why a wait
  uses a timer instead of `usleep()`) are worth a comment; the same
  comment on every activity invocation is not.
- **Waterline screenshots.** The docs-site sample-app gallery names
  the Waterline screen the sample is expected to produce; add or
  refresh that screen as part of the sample's PR.

## What gets a sample rejected

A sample is sent back when:

- the workflow code uses `now()`, `random_*()`, or other
  non-deterministic calls outside `sideEffect()` or an activity;
- the activities do durable bookkeeping the engine already owns —
  writing `workflow_messages` rows directly, manually advancing stream
  cursors, or persisting Waterline state from inside workflow code;
- the artisan command requires external credentials without saying so
  in the MCP entry's `requires` field;
- the README sample-index, docs-site gallery row, and pattern-page
  cross-link do not all land in the same change;
- the sample reproduces an existing pattern surface without showing
  something new (a different shape, a different failure mode, a
  different integration). "Same pattern, different prose" is a docs
  change, not a sample.

These are the same things the upstream-coverage lint and the
sample-app review checklist look for, so catching them yourself before
opening the PR is the fastest path to merge.

## Quick checklist

Use this list when you open the PR:

- [ ] `sample-request` issue exists and links to the public pattern
      docs page.
- [ ] Workflow class under `app/Workflows/<Pattern>/`.
- [ ] Artisan command registered with the `app:<short-pattern>` name.
- [ ] `config/workflow_mcp.php` entry with class, pattern, command,
      requires, and arguments.
- [ ] Test that exercises the workflow end to end.
- [ ] README "Sample Index" row.
- [ ] Docs-site gallery row in
      [`docs/sample-app.md`](/docs/sample-app).
- [ ] Cross-link from the matching pattern page on the docs site.
- [ ] Public-boundary scan clean (`scripts/check-public-boundary.sh`).

Maintainers run the same list during review, so a PR that ticks every
box should land within one release cycle.

<!-- Source: docs/mcp-workflows.md -->

# MCP Workflow Surface

The sample app exposes the reference Durable Workflow v2 MCP server at:

```text
/mcp/workflows
```

Use it when an AI client needs to inspect or operate a local workflow app
without scraping Waterline or guessing Laravel internals. The endpoint is a
structured development surface: it names the workflows the app chooses to
expose, describes credential requirements, starts runs, polls results, and
returns bounded history facts.

## Server Contract

The sample app registers the server from `routes/ai.php` and configures exposed
workflow keys in `config/workflow_mcp.php`. Treat that configuration as the
public allow-list for AI clients. A workflow is MCP-operable only when it is
listed there with enough metadata for a client to decide whether it can run the
workflow safely.

Each configured workflow should describe:

- a stable workflow key
- the workflow class behind that key
- required arguments and optional arguments
- credential requirements
- whether it is safe for no-credential local smoke tests
- output and history expectations that an agent can cite

Keep secrets out of tool descriptions. Say that a workflow requires a
credential, but do not include the credential value or account-specific details.

## Tools

The reference server exposes six workflow tools:

| Tool | Use |
| --- | --- |
| `list_workflows` | Discover configured workflow keys, descriptions, credential requirements, v2 statuses, and recent runs. |
| `start_workflow` | Start a configured workflow and return `workflow_id`, `run_id`, status, business key, and command outcome. |
| `get_workflow_result` | Poll the current or selected run for status, output, visibility metadata, and latest failure summary. |
| `get_workflow_history` | Fetch a bounded tail of typed v2 history events and recent durable failures. |
| `diagnose_workflow` | Classify a selected run with structured facts, root cause, remediation, and next actions. |
| `repair_workflow` | Request the built-in v2 repair command and return a structured accepted, refused, or not-needed mutation result. |

Call `list_workflows` before any start. It is the agent's compatibility check:
the response tells the client which workflow keys exist and whether a workflow
can run in the current environment.

Discovery envelopes use the normative schema id
`durable-workflow.v2.mcp-discovery`. Agents can use it to recognize the
published tool list shape, parameter schemas, discovery hints, and
`payload_preview_limit_bytes` semantics before calling a tool.

## Tool Input Contract

The MCP tool schemas intentionally use Durable Workflow terms instead of
Laravel internals. Keep these fields stable when extending the sample app
server:

| Tool | Stable inputs |
| --- | --- |
| `list_workflows` | `show_recent`, `limit`, and optional `status` for recent-run discovery. |
| `start_workflow` | `workflow`, ordered `arguments`, optional `instance_id`, `business_key`, `visibility_labels`, `memo`, `search_attributes`, and `duplicate_start_policy`. |
| `get_workflow_result` | `workflow_id`, optional `run_id`, `include_recent_history`, and `history_limit`. |
| `get_workflow_history` | `workflow_id` or `run_id`, `limit`, and `include_payloads`. |
| `diagnose_workflow` | `workflow_id` or `run_id`, plus optional `history_limit`. |
| `repair_workflow` | `workflow_id` or `run_id`. |

Prefer `arguments` over the legacy `args` input for new agents. `arguments`
maps directly to the workflow `handle()` argument order, while `args` exists
only for older object-shaped callers.

Use caller-supplied `instance_id` only when an agent needs idempotency. Pair it
with `duplicate_start_policy=return_existing_active` when a repeated smoke run
should attach to the active workflow instead of failing as a duplicate.

## Tool Result Contract

Tool-result envelopes use the normative schema id
`durable-workflow.v2.mcp-tool-results`. Agents can use it to parse result
status, payload preview truncation, error fields, root-cause and remediation
objects, safe-mutation envelopes, and schema/version markers.

Agents should treat the following result fields as the durable handles for
cross-tool correlation:

| Tool | Stable result fields |
| --- | --- |
| `list_workflows` | `available_workflows`, `allow_fqcn`, `workflow_id_kind`, `run_id_kind`, `status_values`, and optional `recent_workflows`. |
| `start_workflow` | `workflow_id`, `run_id`, `workflow`, `workflow_class`, `workflow_type`, `status`, `running`, `business_key`, `duplicate_start_policy`, and `command`. |
| `get_workflow_result` | `found`, `workflow_id`, `run_id`, `current_run_id`, `current_run_is_selected`, `status`, `running`, `output`, `error`, visibility metadata, and timestamps. |
| `get_workflow_history` | `found`, `workflow_id`, `run_id`, `current_run_id`, `status`, `history_event_count`, `returned_event_count`, `events_are_most_recent`, `payloads_included`, `events`, and `failures`. |
| `diagnose_workflow` | `found`, `workflow_id`, `run_id`, `diagnosis`, `facts`, `latest_failure`, `recent_history`, `root_cause`, `remediation`, and `next_actions`. |
| `repair_workflow` | `found`, `workflow_id`, `run_id`, `accepted`, `status`, `mutation`, `command`, `remediation`, and `next_actions`. |

History payload previews are intentionally bounded. When `include_payloads` is
true, each event preview reports `payload_preview_limit_bytes`, `size_bytes`,
`preview_bytes`, and `truncated` so an agent can cite whether it saw the full
payload or only a preview.

## Failure And Remediation Taxonomy

`diagnose_workflow` is the machine-readable root-cause surface. Its
`root_cause` object uses the schema id
`durable-workflow.v2.agent-root-cause` and includes:

- `category`, such as `activity_failure`, `workflow_failure`,
  `task_repair_attention`, `waiting_for_signal`, `history_growth_attention`,
  `in_progress`, or `none`
- `source.kind` and `source.id` for the workflow, activity, wait, task queue,
  or history family that produced the classification
- `retryable`, `severity`, and `actionable`
- failure details such as `failure_category`, `exception_class`, and `handled`
  when a durable failure row exists

The companion `remediation` object uses the schema id
`durable-workflow.v2.agent-remediation`. It includes
`classification`, a short `summary`, `automatic_repair.tool`, and
`automatic_repair.allowed`, plus `next_actions` entries for the supported
follow-up commands. Agents should call `repair_workflow` only when
`automatic_repair.allowed` is true. Other classifications tell the agent to
wait, send expected input through the documented workflow command surface,
inspect history, change workflow or activity code, or plan Continue-As-New.

`repair_workflow` returns a `durable-workflow.v2.safe-mutation` envelope. The
`mutation.applied` field says whether a repair command was accepted, while
`command.outcome` distinguishes `repair_dispatched`, `repair_not_needed`, and
structured refusals such as a terminal or non-current run.

## Safe Agent Loop

Use no-credential workflows first:

```json
{"tool": "list_workflows", "arguments": {"show_recent": true, "limit": 5}}
{"tool": "start_workflow", "arguments": {"workflow": "simple", "business_key": "demo-001"}}
{"tool": "diagnose_workflow", "arguments": {"workflow_id": "<workflow_id>"}}
{"tool": "repair_workflow", "arguments": {"workflow_id": "<workflow_id>"}}
{"tool": "get_workflow_result", "arguments": {"workflow_id": "<workflow_id>"}}
{"tool": "get_workflow_history", "arguments": {"run_id": "<run_id>", "limit": 25}}
```

The `simple` and `elapsed` workflow keys are the preferred smoke surfaces. Use
credentialed examples only after `list_workflows` reports the requirement and
the local environment has the needed keys.

## Report Shape

An AI client should report MCP results with stable facts:

- docs version used, normally `2.0`
- MCP endpoint and tool name
- workflow key, `workflow_id`, and `run_id`
- status and latest failure summary
- `root_cause.category` and `remediation.classification`
- whether `remediation.automatic_repair.allowed` was true before a repair
  mutation was attempted
- bounded history event names and timestamps
- whether the run used a no-credential smoke workflow or a credentialed example

Those facts line up with the CLI, Python SDK, and Waterline history-export
surfaces, so a human can reproduce the same run from another client.

## Related Pages

- [AI-Assisted Development](./ai-assisted-development.md)
- [Agent Operating Loop](./agent-operating-loop.md)
- [Agent Tooling Contract](./agent-tooling-contract.md)
- [Sample App](./sample-app.md)
- [Client and Worker Capabilities](./polyglot/cli-python-parity.md)
- [Message Streams](./features/message-streams.md)

<!-- Source: docs/testing.md -->

# Testing

## Replay before deployment or rollback {#replay-before-deployment}

Keep selected-run exports from representative open and recently closed runs. Include the activity, retry, signal, update, child, timer and version-marker paths that the candidate may encounter. Keep each required continuation generation. Use unpruned histories with available original payload bytes and codecs. An export flag alone does not prove completeness. See the [retention limits](./configuration/pruning-workflows.md).

### Embedded PHP {#embedded-php-replay}

Use Workflow **2.11.4 or newer** for `workflow:v2:history-export` to preserve whole/nested floating-point kinds and signed zero, such as `1.0` and `-0.0`, in new JSON bundles. Check `composer.lock` in the artifact that actually runs `artisan`; a separately selected installation pin may be older. The [normal installed writer qualification](https://github.com/durable-workflow/workflow/issues/912#issuecomment-6364964505) covers compact/pretty output for waiting/completed runs with the original checksum untouched. This qualifies the writer and integrity check, not your candidate workflow code.

Older lossy JSON remains invalid: 2.11.4 does not reinterpret or repair it. If the authoritative store and original payload bytes are available, re-export using the fixed producer. Keep the generated bundle unchanged; never coerce numbers or edit a checksum/version to force acceptance. Still run candidate replay and enforce the promotion gate below.

Export from the application's owning store. Set `WORKFLOW_ID` and `RUN_ID` to the exact selected instance and run:

```bash
php artisan workflow:v2:history-export "$WORKFLOW_ID" \
  --run-id="$RUN_ID" --output=replay-bundles/run.json
```

Run verification inside the candidate application artifact, with its workflow classes and configuration loaded. Keep reports outside the bundle directory. Use an isolated CI environment without production side-effect credentials:

```bash
php artisan workflow:v2:replay-verify replay-bundles/run.json \
  --json --output=replay-one.json
php artisan workflow:v2:replay-simulate replay-bundles \
  --json --output=replay-report.json
```

A nonzero exit blocks promotion. An exit of zero can still require review. Save this batch-report gate as `gate-replay.php` and run `php gate-replay.php replay-report.json` in CI:

```php
<?php
$report = json_decode(file_get_contents($argv[1]), true, 512, JSON_THROW_ON_ERROR);
$evidence = $report['evidence'] ?? [];
$count = $evidence['bundle_count'] ?? 0;
$allowed = ($report['promotion_decision'] ?? null) === 'safe_to_promote'
    && $count > 0
    && ($evidence['missing_bundle_count'] ?? 1) === 0
    && ($evidence['replay_checked_count'] ?? 0) === $count;
exit($allowed ? 0 : 1);
```

`--skip-replay` checks integrity only. Even a clean result cannot authorize replay-safe promotion. A pass covers only recorded paths and the visible prefix of an open run, not future branches, domain correctness or network behavior. Before rollback, test the older artifact against histories already written by newer workers that it could receive. Keep unit and real-runtime tests as well.

### Service SDKs {#service-sdk-replay}

Server images, their bundled CLI and `dw`/SDK HTTP exports depend on the actual Server producer. The Core 2.11.4 release alone does not upgrade or qualify those artifacts. Follow the separate [Server #692 package-pin, image and installed-transport qualification](https://github.com/durable-workflow/server/issues/692); do not infer an HTTP fix from the Core tag. Keep refusing invalid bundles.

**PHP:** use `DurableWorkflow\Worker\Replayer::replay()` with the application's codec, callable, event list, original input, queue and recorded task identity. Let `NonDeterministicWorkflow` fail the test and assert the expected commands/result. See [PHP replay testing](https://php.durable-workflow.com/build/testing/).

**Python:** use `Replayer.replay()` or `verify_replay()` against registered candidate workflows and assert the expected outcome. A returned object alone is not a passing assertion. Supply the original decoded `start_input` when events omit it: Core exports store it under `payloads.arguments`. Preserve recorded type, IDs and codec. See [Python replay APIs](https://python.durable-workflow.com/sdk-reference/#replay-captured-histories). `--simulate-bundles` checks integrity, not candidate code; current schema-2 compatibility is tracked in [Python #173](https://github.com/durable-workflow/sdk-python/issues/173). Do not rewrite a version or checksum to bypass refusal.

**Rust:** SDK **4.5.0 or newer** provides `testing::WorkflowEnvironment` and `WorkflowHistory` for Server-free activity/workflow tests and replay of self-contained exports. Reuse candidate registrations, preserve original input/all history pages, and assert completion or the expected pending prefix. Divergence returns typed `Error::NonDeterministicReplay` with its durable sequence and expected/actual commands. New select races and other unsupported execution paths still need integration tests. See the [Rust tests, determinism rules and replay examples](./polyglot/rust.md#offline-testing-and-replay).

## Keeping delay tests fast

Timer delays and durable retry backoff use whole seconds. A 100 ms fractional
interval accepted by a rounding SDK API becomes one second, so sub-second
values are not a way to speed up retry-heavy tests.

For embedded tests, use `WorkflowStub::fake()`, mock activity outcomes and
advance Laravel's test clock by the configured delay. Then call
`WorkflowStub::runReadyTasks()` to drain work that is due. Keep the retry budget
finite. A test-specific zero-second backoff can exercise bounded retry outcomes
without a wall-clock wait; retain a separate test of the real backoff schedule.
Changing a local test clock does not advance a remote Server's clock.

## `Workflow\V2`

`Workflow\V2\WorkflowStub::fake()` provides a deterministic inline test path for the supported `V2` fake surface.

- ready workflow and activity tasks execute inline instead of waiting for a queue worker
- nested child workflows execute as real nested `V2` runs under that same fake mode
- activity mocks still write durable `activity_executions`, task, and history rows
- backend capability checks are bypassed for the fake path, so `sync` remains usable in tests
- `WorkflowStub::assertDispatched()`, `assertDispatchedTimes()`, `assertNotDispatched()`, and `assertNothingDispatched()` cover `V2` activity dispatches
- `WorkflowStub::assertSignalSent()`, `assertSignalSentTimes()`, and `assertSignalNotSent()` cover `V2` signals sent through `WorkflowStub::signal()` or `signalWithStart()`
- `WorkflowStub::assertUpdateSent()`, `assertUpdateSentTimes()`, and `assertUpdateNotSent()` cover `V2` updates sent through `WorkflowStub::update()`, `attemptUpdate()`, or `submitUpdate()`
- delayed timer tasks stay durable queued work until they become due; after you advance time, `WorkflowStub::runReadyTasks()` drains already-due tasks inline

### First-release testing scope

The following are **not** part of the first-release `V2` fake surface and may be added as future additive contracts:

- there is no `V2` equivalent of the legacy `resume()` bridge
- there is no dedicated child-workflow mock or dispatch-assert helper — child workflows execute as real nested `V2` runs under fake mode, so test them through their observable output rather than mocking
- there is no delayed-callback hook for injecting signals or updates at virtual time offsets during a single fake execution — advance time and call `WorkflowStub::runReadyTasks()` between explicit signal/update calls instead

`WorkflowStub::mock()` enforces this scope at runtime: passing a `Workflow` subclass throws a `LogicException` with a clear message directing you to test child workflows through their observable output instead. Only `Activity` classes (and unresolved string keys) are accepted as mock targets.

```php
use function Workflow\V2\activity;
use Workflow\V2\Workflow;

final class MyWorkflow extends Workflow
{
    public function handle(): array
    {
        return [
            'result' => activity(MyActivity::class, 'Taylor'),
            'workflow_id' => $this->workflowId(),
            'run_id' => $this->runId(),
        ];
    }
}
```

```php
use Workflow\V2\Testing\ActivityFakeContext;
use Workflow\V2\WorkflowStub;

public function testWorkflow(): void
{
    WorkflowStub::fake();

    WorkflowStub::mock(MyActivity::class, function (ActivityFakeContext $context, string $name): string {
        $this->assertSame('Taylor', $name);
        $this->assertSame('my-workflow-id', $context->workflowId());

        return "Hello, {$name}!";
    });

    $workflow = WorkflowStub::make(MyWorkflow::class, 'my-workflow-id');
    $workflow->start();

    $this->assertTrue($workflow->refresh()->completed());
    $this->assertSame('Hello, Taylor!', $workflow->output()['result']);

    WorkflowStub::assertDispatched(MyActivity::class, function (string $name): bool {
        return $name === 'Taylor';
    });
}
```

Use `WorkflowStub::assertDispatched()`, `assertDispatchedTimes()`, `assertNotDispatched()`, and `assertNothingDispatched()` to assert the recorded `V2` activity dispatches.

For timer-backed workflows, advance time with Laravel's travel helpers and then drain the due task queue explicitly:

```php
use function Workflow\V2\timer;
use Workflow\V2\Workflow;
use Workflow\V2\WorkflowStub;

final class MyTimerWorkflow extends Workflow
{
    public function handle(): array
    {
        timer(60);

        return ['done' => true];
    }
}

public function testTimerWorkflow(): void
{
    WorkflowStub::fake();

    $workflow = WorkflowStub::make(MyTimerWorkflow::class, 'timer-workflow');
    $workflow->start();

    $this->travel(60)->seconds();

    WorkflowStub::runReadyTasks();

    $this->assertTrue($workflow->refresh()->completed());
    $this->assertSame(['done' => true], $workflow->output());
}
```

`WorkflowStub::runReadyTasks()` only drains tasks that are already due. It does not fast-forward wall clock time or force future-delayed timers to fire early.

Workflow 2.20.3 drains fake tasks at their exact due time, including fractional
seconds, using the configured task model's persisted timestamp precision.
Earlier versions could skip SQLite tasks due at that instant. A future
fractional deadline stays pending until it is due. Timer delays and retry
backoff still use whole seconds. See the
[stream upgrade and rollback limits](./features/message-streams.md#stream-upgrades)
before rolling an embedded application back to an older package.

### Sending Signals in Fake Mode

Signal-waiting workflows can receive signals in fake mode. When you call `$workflow->signal(...)`, the engine records the durable signal command and creates a workflow task. In fake mode that task executes inline, so the workflow resumes synchronously:

```php
use function Workflow\V2\activity;
use Workflow\V2\Attributes\Signal;
use function Workflow\V2\await;
use Workflow\V2\Workflow;
use Workflow\V2\WorkflowStub;

#[Signal('name-provided')]
final class ApprovalWorkflow extends Workflow
{
    public function handle(): array
    {
        $name = await('name-provided');
        $greeting = activity(GreetingActivity::class, $name);

        return ['name' => $name, 'greeting' => $greeting];
    }
}

public function testSignalWorkflow(): void
{
    WorkflowStub::fake();
    WorkflowStub::mock(GreetingActivity::class, 'Hello, Taylor!');

    $workflow = WorkflowStub::make(ApprovalWorkflow::class, 'approval-1');
    $workflow->start();

    // Workflow suspends at await('name-provided')
    $this->assertSame('waiting', $workflow->refresh()->status());

    // Send the signal — resumes the workflow inline
    $workflow->signal('name-provided', 'Taylor');

    $this->assertTrue($workflow->refresh()->completed());
    $this->assertSame('Taylor', $workflow->output()['name']);

    WorkflowStub::assertSignalSent('name-provided');
    WorkflowStub::assertSignalSentTimes('name-provided', 1);
    WorkflowStub::assertSignalNotSent('other-signal');
}
```

Use `WorkflowStub::assertSignalSent()`, `assertSignalSentTimes()`, and `assertSignalNotSent()` to verify which signals were sent during the test. The callback form receives the instance id and signal arguments:

```php
WorkflowStub::assertSignalSent(
    'name-provided',
    fn (string $instanceId, string $name): bool =>
        $instanceId === 'approval-1' && $name === 'Taylor'
);
```

### Sending Updates in Fake Mode

Updates also apply inline in fake mode. When you call `$workflow->attemptUpdate(...)` or `$workflow->update(...)`, the engine records the durable update command, creates a workflow task that applies the update, and executes it inline:

```php
use Workflow\UpdateMethod;
use Workflow\V2\Attributes\Signal;
use function Workflow\V2\await;
use Workflow\V2\Workflow;
use Workflow\V2\WorkflowStub;

#[Signal('done')]
final class SettingsWorkflow extends Workflow
{
    private bool $enabled = false;

    public function handle(): array
    {
        await('done');

        return ['enabled' => $this->enabled];
    }

    #[UpdateMethod]
    public function toggle(bool $enabled): array
    {
        $this->enabled = $enabled;

        return ['enabled' => $this->enabled];
    }
}

public function testUpdateWorkflow(): void
{
    WorkflowStub::fake();

    $workflow = WorkflowStub::make(SettingsWorkflow::class, 'settings-1');
    $workflow->start();

    $this->assertSame('waiting', $workflow->refresh()->status());

    $result = $workflow->attemptUpdate('toggle', true);

    $this->assertTrue($result->accepted());

    WorkflowStub::assertUpdateSent('toggle');
    WorkflowStub::assertUpdateSentTimes('toggle', 1);
    WorkflowStub::assertUpdateNotSent('other-update');

    // Complete the workflow
    $workflow->signal('done');
    $this->assertTrue($workflow->refresh()->completed());
    $this->assertTrue($workflow->output()['enabled']);
}
```

Use `WorkflowStub::assertUpdateSent()`, `assertUpdateSentTimes()`, and `assertUpdateNotSent()` to verify which updates were sent during the test. The callback form receives the instance id and update arguments:

```php
WorkflowStub::assertUpdateSent(
    'toggle',
    fn (string $instanceId, bool $enabled): bool =>
        $instanceId === 'settings-1' && $enabled === true
);
```

### Testing Start Outcomes and Duplicate-Start Policy

Use `attemptStart()` to verify duplicate-start behavior without throwing exceptions. The `StartResult` exposes typed outcome helpers:

```php
use Workflow\V2\StartOptions;
use Workflow\V2\WorkflowStub;

public function testRejectDuplicateStart(): void
{
    WorkflowStub::fake();
    WorkflowStub::mock(MyActivity::class, 'result');

    $workflow = WorkflowStub::make(MyWorkflow::class, 'order-123');

    $first = $workflow->start('Taylor');
    $this->assertTrue($first->startedNew());

    $second = WorkflowStub::load('order-123');
    $duplicate = $second->attemptStart('Taylor');

    $this->assertTrue($duplicate->rejected());
    $this->assertTrue($duplicate->rejectedDuplicate());
    $this->assertSame('instance_already_started', $duplicate->rejectionReason());
}
```

To test the return-existing-active policy, pass `StartOptions::returnExistingActive()`:

```php
public function testReturnExistingActiveStart(): void
{
    WorkflowStub::fake();

    $workflow = WorkflowStub::make(MySignalWorkflow::class, 'order-123');
    $workflow->start();

    $this->assertSame('waiting', $workflow->refresh()->status());

    $second = WorkflowStub::load('order-123');
    $result = $second->attemptStart(StartOptions::returnExistingActive());

    $this->assertTrue($result->accepted());
    $this->assertTrue($result->returnedExistingActive());
    $this->assertSame($workflow->runId(), $result->runId());
}
```

Both policies record durable command and history events, so you can also assert on `WorkflowCommand` and `WorkflowHistoryEvent` rows for deeper verification.

### Testing History Budget

The `HistoryBudget` fields (`history_event_count`, `history_size_bytes`, `continue_as_new_recommended`) are surfaced on the run summary projection and are available through the `RunDetailView`. In your workflow code, the `Workflow` base class exposes these as `$this->historyLength()`, `$this->historySize()`, and `$this->shouldContinueAsNew()`. Inside a loop, use `$this->shouldContinueAsNewAtStep()` (Workflow 2.26 and later), which gives the same answer on every replay:

```php
use Workflow\V2\Workflow;

final class LongRunningWorkflow extends Workflow
{
    public function handle(): void
    {
        while (true) {
            // ... process work ...

            if ($this->shouldContinueAsNewAtStep()) {
                Workflow::continueAsNew($this->carryForwardState());
            }
        }
    }
}
```

To test history budget thresholds, configure the thresholds low and verify the recommendation:

```php
public function testHistoryBudgetRecommendsContinueAsNew(): void
{
    config()->set('workflows.v2.history_budget.continue_as_new_event_threshold', 5);

    WorkflowStub::fake();

    // Run a workflow that produces enough history events to trip the threshold
    $workflow = WorkflowStub::make(ManyActivitiesWorkflow::class, 'budget-test');
    $workflow->start();

    $summary = $workflow->summary();
    $this->assertTrue($summary->continue_as_new_recommended);
}
```

## Legacy `Workflow\WorkflowStub`

:::warning Legacy v1

The examples in this section use the v1 runtime (`Workflow\Workflow`, `use function Workflow\activity`, generator-style `execute()` with `yield`, `Workflow\Models\StoredWorkflow`). Keep these patterns for tests that still exercise v1 workflows during migration. New tests should use `Workflow\V2\WorkflowStub::fake()` with `Workflow\V2\Workflow` and straight-line `handle()` methods as shown above.

:::

### Workflows

You can execute workflows synchronously in your test environment and mock activities and child workflows to define expected behaviors and outputs without running the actual implementations.

```
use function Workflow\activity;
use Workflow\Workflow;

class MyWorkflow extends Workflow
{
    public function execute()
    {
        $result = yield activity(MyActivity::class);

        return $result;
    }
}
```

The above workflow can be tested by first calling `WorkflowStub::fake()` and then mocking the activity.

```
public function testWorkflow()
{
    WorkflowStub::fake();

    WorkflowStub::mock(MyActivity::class, 'result');

    $workflow = WorkflowStub::make(MyWorkflow::class);
    $workflow->start();

    $this->assertSame($workflow->output(), 'result');
}
```

You can also provide a callback instead of a result value to ` WorkflowStub::mock()`.

The workflow `$context` along with any arguments for the current activity will also be passed to the callback.

```
public function testWorkflow()
{
    WorkflowStub::fake();

    WorkflowStub::mock(MyActivity::class, function ($context) {
        return 'result';
    });

    $workflow = WorkflowStub::make(MyWorkflow::class);
    $workflow->start();

    $this->assertSame($workflow->output(), 'result');
}
```

You can assert which activities or child workflows were dispatched by using the `assertDispatched`, `assertNotDispatched`, and `assertNothingDispatched` methods:

```
WorkflowStub::assertDispatched(MyActivity::class);

// Assert the activity was dispatched twice...
WorkflowStub::assertDispatched(MyActivity::class, 2);

WorkflowStub::assertNotDispatched(MyActivity::class);

WorkflowStub::assertNothingDispatched();
```

You may pass a closure to the `assertDispatched` or `assertNotDispatched` methods in order to assert that an activity or child workflow was dispatched that passes a given "truth test". The arguments for the activity or child workflow will be passed to the callback.

```
WorkflowStub::assertDispatched(TestOtherActivity::class, function ($string) {
    return $string === 'other';
});
```

### Skipping Time

By manipulating the system time with `$this->travel()` or `$this->travelTo()`, you can simulate time-dependent workflows. This strategy allows you to test timeouts, delays, and other time-sensitive logic within your workflows.

```
use function Workflow\{activity, timer};
use Workflow\Workflow;

class MyTimerWorkflow extends Workflow
{
    public function execute()
    {
        yield timer(60);

        $result = yield activity(MyActivity::class);

        return $result;
    }
}
```

The above workflow waits 60 seconds before executing the activity. Using `$this->travel()` and `$workflow->resume()` allows us to skip this waiting period in the legacy runtime.

```
public function testTimeTravelWorkflow()
{
    WorkflowStub::fake();

    WorkflowStub::mock(MyActivity::class, 'result');

    $workflow = WorkflowStub::make(MyTimerWorkflow::class);
    $workflow->start();

    $this->travel(120)->seconds();

    $workflow->resume();

    $this->assertSame($workflow->output(), 'result');
}
```

The helpers `$this->travel()` and `$this->travelTo()` methods use `Carbon::setTestNow()` under the hood.

### Activities

Testing activities is similar to testing Laravel jobs. You manually create the activity and then call the `handle()` method.

```
$workflow = WorkflowStub::make(MyWorkflow::class);

$activity = new MyActivity(0, now()->toDateTimeString(), StoredWorkflow::findOrFail($workflow->id()));

$result = $activity->handle();
```

Notice that legacy activities still execute through the runtime `handle()` job method. In `Workflow\V2`, user-defined workflow and activity code also lives in `handle()`.

<!-- Source: docs/how-it-works.md -->

# How It Works

Durable Workflow uses Laravel's queued jobs and event-sourced persistence to create durable coroutines. Workflows suspend through Fiber-backed helper calls for a durable replay contract.

## Runtime

A workflow is a class whose `handle()` method calls straight-line helpers such as `activity()`, `await()`, `timer()`, `sideEffect()`, `child()`, and `all([...])`. Each helper call suspends the workflow until the corresponding durable step completes, then resumes from where it left off with the recorded result.

Every step produces a durable history event. The engine replays that history whenever the workflow wakes up, rebuilding state from the event stream before running the next unexecuted step. That replay is what lets a workflow survive worker restarts, deployments, and machine failures without losing its place.

`WorkflowStub::make()` reserves a public workflow instance id. Starting the workflow creates the first run and the first workflow task. Each run has its own run id; operations such as `signal()`, `cancel()`, and `terminate()` target the current instance run.

## Event Sourcing

Event sourcing builds up the current state from a sequence of saved events rather than saving the state directly. This has several benefits: it provides a complete history of the execution events, and it can be used to resume a workflow if the worker crashes.

## Coroutines

Coroutines are functions whose execution can be suspended and resumed. Durable suspension points are expressed as straight-line Fiber-backed helper calls such as `activity()`, `await()`, `timer()`, and `sideEffect()`.

User workflow code lives in `handle()`, which is an ordinary method that calls those helpers directly. The runtime first checks whether the step already completed durably. If so, the cached result is replayed from history instead of running the step a second time. Otherwise, the runtime queues the next activity, timer, or child work and suspends until that durable step completes or fails.

## Activities

By calling multiple activities, a workflow can orchestrate the results between each of them. The execution of the workflow and the durable steps it schedules are interleaved: the workflow reaches an activity call, suspends until that activity completes, and then continues execution from where it left off.

If a workflow worker crashes, committed events are replayed to rebuild the current state. This allows the workflow to pick up where it left off, with the same inputs and outputs as before, ensuring determinism. An unhandled workflow failure makes the run terminal; replay does not retry a failed run.

Ordinary activities are durable queued work in v2. Explicit
[local activities](./features/local-activities.md) run short activity work in
the workflow worker process while preserving durable history and retry semantics.
Ordinary activities can run on any compatible worker, and
[worker sessions](./features/worker-sessions.md) add an explicit lease when a
sequence of activity steps needs the same worker-local resource. If you need a
replay-safe one-shot value without queueing an activity, use
[`sideEffect(...)`](./features/side-effects.md). For the full contract, see
[Activity Execution Model](./features/activity-execution-model.md).

## Execution Guarantees

Workflow code and activity code have different repeat-execution semantics:

- **Workflow code is replayed.** Re-delivering a workflow task rebuilds state from durable history and re-runs deterministic authoring code. Replay does not re-run external side effects.
- **Activities are at-least-once queued work.** A logical activity may be retried, redelivered after lease expiry, or re-observed after worker loss. Duplicate delivery is a normal distributed-systems condition, not a bug by itself.
- **Activity identity is durable.** `activity_execution_id` identifies one logical activity execution across retries and redelivery, while `activity_attempt_id` identifies an individual try. Use `activity_execution_id` as the default remote idempotency key; use `activity_attempt_id` only when a downstream system needs per-attempt correlation.

See [Execution Guarantees and Idempotency](./constraints/execution-guarantees.md),
[Activity Execution Model](./features/activity-execution-model.md), and
[Failures and Recovery](./failures-and-recovery.md) for the full v2 contract.

## Queues

Queued jobs are background processes that run at a later time. Laravel supports queues via Amazon SQS, Redis, or a relational database. Workflows and activities are both queued jobs, but they behave a little differently. A workflow is dispatched multiple times during normal operation: it runs, dispatches one or more activities, and then exits until the activities complete. An activity is an at-least-once queued task: the common case is one successful attempt, but retry, lease expiry, or worker loss can cause the same logical activity execution to be delivered more than once.

## Example

```php
use Workflow\V2\Workflow;
use function Workflow\V2\{activity, all};

class MyWorkflow extends Workflow
{
    public function handle(): array
    {
        return [
            activity(TestActivity::class),
            activity(TestOtherActivity::class),
            all([
                fn () => activity(TestParallelActivity::class),
                fn () => activity(TestParallelOtherActivity::class),
            ]),
        ];
    }
}
```

## Sequence Diagram

This sequence diagram shows how a workflow progresses through a series of activities, both serial and parallel.

1. The workflow starts by getting dispatched as a queued job.
2. The first activity, `TestActivity`, is then dispatched as a queued job. The workflow job then exits. Once `TestActivity` has completed, it saves the result to the database and returns control to the workflow by dispatching it again.
3. At this point, the workflow enters the event sourcing replay loop. This is where it goes back to the database and looks at the event stream to rebuild the current state. This is necessary because the workflow is not a long running process. The workflow exits while any activities are running and then is dispatched again after completion.
4. Once the event stream has been replayed, the workflow continues to the next activity, `TestOtherActivity`, and starts it by dispatching it as a queued job. Again, once `TestOtherActivity` has completed, it saves the result to the database and returns control to the workflow by dispatching it as a queued job.
5. The workflow then enters the event sourcing replay loop again, rebuilding the current state from the event stream.
6. Next, the workflow starts two parallel activities, `TestParallelActivity` and `TestParallelOtherActivity`. Both activities are dispatched. Once they have completed, they save the results to the database and return control to the workflow.
7. Finally, the workflow enters the event sourcing replay loop one last time to rebuild the current state from the event stream. This completes the execution of the workflow.

## Determinism

Because history is replayed on every wake-up, workflow code must produce the same commands given the same history. Read [Constraints](./constraints/overview.md) for the authoring rules and the helpers Durable Workflow exposes (`Workflow::now()` on `Workflow\V2\Workflow`, `sideEffect()`, `getVersion()`, and similar) for situations where code would otherwise be non-deterministic.

<!-- Source: docs/failures-and-recovery.md -->

# Failures and Recovery

Before you triage a failure, keep the core execution contract in mind:

- workflow tasks recover by replaying committed history
- activity execution is at-least-once and may be observed more than once
- lease expiry and redelivery are normal recovery paths, not proof that the
  previous worker never executed the side effect

Read [Execution Guarantees and Idempotency](./constraints/execution-guarantees.md)
first when you need the precise semantics behind retries, redelivery, and
durable outcomes.

## Handling Exceptions

`Workflow\V2\Activity` defaults to `$tries = 1`: one attempt, with no automatic retry after an exception. The first activity failure is sent back to the workflow. Workflow code can catch it, as below; if the exception is not handled, the run fails and becomes terminal.

Retries are opt-in. Set `$tries` above `1` to allow more attempts, or `0` for unlimited attempts, and choose a `backoff()` policy. A retryable failure is delivered to workflow code only after the configured attempt or timeout budget is exhausted. A non-retryable failure bypasses the remaining retries.

```php
use Exception;
use Workflow\V2\Activity;

class MyActivity extends Activity
{
    public int $tries = 1;

    public function handle(): void
    {
        throw new Exception();
    }
}
```

```php
use Exception;
use function Workflow\V2\activity;
use Workflow\V2\Workflow;

class MyWorkflow extends Workflow
{
    public function handle(): void
    {
        try {
            $result = activity(MyActivity::class);
        } catch (Exception) {
            // handle the exception here
        }
    }
}
```

## Non-retryable Exceptions

In certain cases, you may encounter exceptions that should not be retried. These are referred to as non-retryable exceptions. When an activity throws a non-retryable exception, the workflow will immediately mark the activity as failed and stop retrying.

```php
use Workflow\V2\Activity;
use Workflow\Exceptions\NonRetryableException;

class MyNonRetryableActivity extends Activity
{
    public function handle(): void
    {
        throw new NonRetryableException('This is a non-retryable error');
    }
}
```

## Recovery Process

First inspect the selected run's durable status and activity retry policy. With the default single attempt, an uncaught activity failure has already made the run terminal. Deploying a fix or calling `repair()` does not reopen that failed run.

Fixing code while the original run waits is possible only while the activity still has an eligible retry. Configure `$tries` above `1`, or `0` for unlimited attempts, with `backoff()` before the activity is scheduled. Optionally set `ActivityOptions::$scheduleToCloseTimeout` to bound the total activity duration across retries; workflow deadlines still apply. See [Retries](./configuration/options.md#retries), [Backoff](./configuration/options.md#backoff), and [ActivityOptions](./configuration/options.md#activityoptions). The retry policy is snapshotted when the activity is scheduled, so a later deployment does not extend an existing activity's attempt or timeout budget.

For a run that is still waiting on a retryable activity:

1. Check the logs for the activity that is failing and look for any errors or exceptions that are being thrown.
2. Identify the source of the error and fix it in the code.
3. Deploy the fix to the server where the queue is running.
4. Restart or roll the relevant workers so they pick up the new code and can safely reclaim work.
5. Let the next eligible retry run on a healthy worker. Repair/redelivery can recover lost task transport; it does not grant another business retry after the activity or run has closed.
6. Verify the durable outcome in Waterline, history export, or the server API instead of assuming that one worker log line is the authority.
7. If the activity continues to fail, check the remaining attempt and time budgets before repeating the process.

A run with an eligible activity retry remains waiting and can continue after the corrected code succeeds, provided its original budgets have not expired. This does not guarantee that every failing workflow can be fixed in place: an unhandled final activity failure, a workflow-code failure, a structural limit, or a workflow timeout can make the run terminal.

For a terminal failed run, preserve its history and reconcile any external side effects before deciding whether the application should start a new workflow. Use application idempotency for repeated business operations. If an uncaught activity failure, a workflow-code failure or a structural limit ended the run, check whether you can [redrive](#redrive) it after the fix is deployed. Redrive continues in a new run that reuses the completed steps. A workflow-code failure needs corrected code and Workflow 2.19.0 or later; see [Redrive onto corrected workflow code](#redrive-corrected-definition). There is no automatic workflow-level retry or terminal-run reopening; see [Workflow-Level Retry](#workflow-level-retry).

## Redrive a Failed Run {#redrive}

Redrive continues a failed run in a new run of the same workflow instance. The new run reuses the recorded results of the steps that completed before the failure and runs the rest of the workflow again. Use it after you fix the cause of the failure, so that completed work such as a payment or an email is not repeated.

The failed run stays closed and unchanged. The new run, called the successor, gets the next run number and becomes the instance's current run. Its `continued_from_run_id` names the failed run, and its `WorkflowStarted` event records `recovery_kind: redrive` and `resume_step_sequence`, the step where execution resumes.

This section describes redrive in Workflow 2.20.1 and later. Earlier Workflow releases, and redrive in the native Server, accept fewer runs. See [Earlier releases and the native Server](#redrive-earlier-releases).

The successor replays workflow code from the start:

- Each activity call that completed in the failed run returns its recorded result. The activity does not run again.
- A side effect or version marker before the failed step is reused in step order. The side effect does not run again, and a version check returns the recorded version.
- A `timer()` that fired before the failed step is recorded in the successor as already fired. The successor does not wait for it again.
- A memo or search attribute that workflow code upserted before the failed step is reused in step order, and the successor keeps the same values.
- A signal wait (`await('name')`) that received and applied its signal is reused with the signal's value. The signal is not delivered again.
- An update that was accepted, applied and completed before the failed step is recorded again at the same position. Replay runs the update handler from its recorded arguments, so the workflow's state matches. The update is not delivered again. A rejected update, and one the failure closed before it ran, are left out.
- In Workflow 2.26 and later, a child workflow call that completed before the failed step returns its recorded result. The child is not started again. The successor records the call's `ChildWorkflowScheduled`, `ChildRunStarted` and `ChildRunCompleted` events and keeps the original child instance, child run, link id and call id, but gets no child link of its own.
- `outbox()->sendReference()` and `inbox()->receive()` in reused workflow code return the message the failed run sent or consumed. They don't send or consume it again. A replayed receive whose original call found nothing reads the stream again.
- In Workflow 2.20.2 and later, inbound stream messages still pending on the failed run move to the successor. Workflow 2.20.1 and earlier left them on the failed run, so the successor never received them ([workflow#1013](https://github.com/durable-workflow/workflow/issues/1013)).
- If an activity failure ended the run, the failed activity call is scheduled as a new activity execution, using the retry policy in the workflow code. Its attempts start again.
- If the run failed in workflow code or on a structural limit, the successor resumes at the step the workflow code never reached.
- Everything after the failed step runs as it would in a new run.
- Memos and search attributes set when the run started carry over to the successor, as they do for continue-as-new.
- The successor keeps the failed run's execution deadline. Its run timeout starts again at the redrive.
- Each reused step carries `reused_from_run_id` and `reused_recorded_at`. On a reused update or signal event, the two always come as a pair.
- A successor that reused steps can be redriven again. A reused signal or update payload keeps the run that first recorded it in `workflow_run_id`, `reused_from_run_id` names the run it was redriven from, and every reused step keeps the time it first ran, so a chain of redrives reuses each step once.
- In embedded mode, `Workflow::now()` in the successor starts at the failed run's start time. Replay uses `reused_recorded_at`, so `Workflow::now()` after a reused step equals the time in the failed run. After a reused child workflow call, that is the time the child first finished. A step reused by a second redrive keeps the time it first ran.

### Request a redrive {#request-a-redrive}

In embedded mode, resolve the `Workflow\V2\Contracts\WorkflowControlPlane` contract from the Laravel container and call `redrive()` with the workflow instance id and the failed run id:

```php
use Workflow\V2\Contracts\WorkflowControlPlane;

$result = app(WorkflowControlPlane::class)->redrive($instanceId, $failedRunId, [
    'request_id' => 'incident-42',
]);

if ($result['accepted']) {
    $successorRunId = $result['workflow_run_id'];
} else {
    $reason = $result['reason'];
}
```

In service mode, send `POST /api/workflows/{workflowId}/runs/{runId}/redrive` with an optional `request_id` in the body. The clients wrap the same route:

| Surface | Call |
| --- | --- |
| CLI | `dw workflow:redrive <workflow-id> <run-id> --request-id <id>` |
| PHP SDK | `Client::redriveWorkflow($workflowId, $failedRunId, $requestId)` |
| Python SDK | `await client.redrive_workflow(workflow_id, failed_run_id, request_id=...)` or `await handle.redrive()` |
| Rust SDK | `client.redrive_workflow_run(workflow_id, failed_run_id, request_id)` or `handle.redrive(request_id)` |

An accepted redrive returns the successor's run id and `resume_step_sequence`. A refused redrive returns a `reason` and changes nothing. Redriving a run that was already redriven returns the existing successor instead of creating another one. Pass a `request_id` so that a retried request is safe. A `request_id` already used for another redrive of the same instance is refused with `request_id_already_used`.

Service workers report a workflow definition fingerprint, and redrive compares it with the fingerprint recorded when the run started. In the Rust SDK, call `Worker::set_workflow_definition_sources()` for each workflow you want to redrive. A workflow registered without source identity still runs, but redrive refuses it.

### Redrive onto corrected workflow code {#redrive-corrected-definition}

When a run failed because workflow code threw, the same code would throw again at the same step, so redrive refuses it with `workflow_code_failure_requires_corrected_definition`. Deploy the fix, then request the redrive with `allow_definition_change`:

```php
$result = app(WorkflowControlPlane::class)->redrive($instanceId, $failedRunId, [
    'request_id' => 'incident-42',
    'allow_definition_change' => true,
]);
```

Before it writes anything, redrive replays the completed steps with the new code in memory. The new code must take the same steps in the same order and reach the step where the old code failed. If it can't, redrive refuses with `definition_incompatible_with_prefix`, and `definition_mismatch` names the first step that differs (`sequence`) and a `kind` such as `step_shape` or `version_range`. If the completed history is larger than the history budget, redrive refuses with `definition_check_over_budget`.

An accepted change runs the corrected code and records the replaced definition's fingerprint as `definition_changed_from` on the successor's `WorkflowStarted`. A later redrive on the same definition records none.

`allow_definition_change` applies to embedded runs. Redrive refuses a definition change for a run a service worker started with `definition_change_requires_replay_check`. Without `allow_definition_change`, a changed definition is refused with `workflow_definition_unavailable_or_changed`.

### When a failed run can be redriven {#redrive-eligibility}

Redrive accepts a failed run when all of these are true:

- The run is the instance's current run.
- The run failed because an activity failure reached workflow code and was not caught, or because workflow code threw or a structural limit stopped it. A run that failed on a workflow timeout is refused.
- Every step before the failed step is a completed activity, a side effect, a version marker, a `timer()` that fired, a memo or search-attribute upsert, a signal wait that received and applied its signal, an update that was accepted, applied and completed with the same arguments, or, in Workflow 2.26 and later, a child workflow call that completed. A rejected update, and one the failure closed before it ran, are allowed and left out. Activity, side effect, signal, update and child result payloads must be stored inline. A payload held in external payload storage is refused.
- The workflow definition matches the fingerprint recorded when the run started, or you passed `allow_definition_change` and the corrected code replays the completed steps. A workflow-code failure on the same definition is refused.
- The execution deadline has not passed.
- The run has no parent workflow and no local activities, and its input is stored inline. Before Workflow 2.26, it also has no child workflows. A run that is itself a child is never redriven, because its parent has already recorded the child's outcome.
- Each child workflow call before the failed step completed on its own, outside a parallel group or cancellation scope, without a retry or continue-as-new.
- If a service worker started the run, the steps before the failure are only activities, side effects, version markers and fired timers. Memo and search-attribute upserts, signal waits, applied updates and completed child workflow calls are refused for service runs.

A refusal names the first rule that failed:

| Reason | Meaning |
| --- | --- |
| `run_not_failed` | The run is not in the `failed` state. |
| `source_run_not_current` | A newer run of the instance exists. |
| `unrecorded_failure_boundary` | The run's failure has no recorded step, such as a workflow-code failure recorded by a release before 2.19.0. |
| `workflow_code_failure_requires_corrected_definition` | Workflow code threw, and the definition hasn't changed, so the same code would throw again. |
| `unsupported_history` | The history before the failure contains a step that redrive cannot reuse, such as a cancelled timer. Before Workflow 2.26, any child workflow. |
| `unsupported_open_child` | A child workflow call before the failure never closed. Workflow 2.26 and later. |
| `unsupported_child_outcome` | A child workflow before the failure failed, was cancelled or was terminated. Workflow 2.26 and later. |
| `unsupported_child_workflow` | A child workflow call is in a parallel group or cancellation scope, was retried or continued as new, or its events don't name one child call. Workflow 2.26 and later. |
| `external_child_result_not_supported` | A completed child workflow's result is stored in external payload storage. Workflow 2.26 and later. |
| `unavailable_child_result` | A completed child workflow's result cannot be decoded. Workflow 2.26 and later. |
| `unsupported_local_activity` | The history contains a local activity. |
| `unsupported_timer` | The history contains a condition or signal timeout, a cancellation cleanup timer, or a timer inside a cancellation scope or parallel group. |
| `incomplete_timer_step` | A timer step does not have both its scheduled and fired events. |
| `unsupported_signal_wait` | A signal wait has a timeout, or is inside a cancellation scope or parallel group. |
| `unsupported_pending_signal` | The run received a signal that it never applied. |
| `incomplete_signal_step` | A signal wait does not have its receipt and its application, or the application has no recorded value. |
| `mismatched_signal_receipt` | A signal receipt names another run, instance, command or wait, one receipt is applied to two waits, or a reused signal event has only one of `reused_from_run_id` and `reused_recorded_at`, or names another source. |
| `unsupported_pending_update` | The run accepted an update that it never applied or closed. |
| `mismatched_update_history` | An update's events are not one of the accepted shapes, name another run, update, command or handler, change the arguments between acceptance and application, or have a broken `reused_from_run_id` and `reused_recorded_at` pair. |
| `incomplete_update_step` | An applied update's completion records a failure or has no result. |
| `invalid_step_sequence` | A reused step does not have a valid step number. |
| `duplicate_step_sequence` | Two reused steps share one step number. |
| `external_activity_result_not_supported` | A completed activity result is stored in external payload storage. |
| `external_side_effect_result_not_supported` | A side effect result is stored in external payload storage. |
| `external_signal_payload_not_supported` | A signal's arguments or applied value are stored in external payload storage. Any `external_storage` key counts, even a null one. |
| `external_update_payload_not_supported` | An update's arguments or result are stored in external payload storage. |
| `unavailable_update_payload` | An update's inline arguments or result cannot be decoded. |
| `missing_activity_result`, `unavailable_activity_result` | A completed activity result is missing or cannot be decoded. |
| `unsupported_visibility_metadata` | A memo or search attribute was written by workflow code at a step the successor would not reuse. |
| `unsupported_source_metadata` | The run was started as a child of another workflow. |
| `unsupported_source_link_or_payload` | The run is linked as a child of another workflow, or its input is in external payload storage. |
| `execution_deadline_elapsed` | The execution deadline has passed. |
| `workflow_definition_unavailable_or_changed` | The definition fingerprint is missing, or differs from the one recorded at start and `allow_definition_change` was not passed. |
| `definition_incompatible_with_prefix` | The corrected code can't replay the completed steps. `definition_mismatch` names the first step that differs. |
| `definition_check_over_budget` | The completed history is larger than the history budget the replay check allows. |
| `definition_change_requires_replay_check` | A service worker started the run, so redrive can't check a changed definition. |
| `service_redrive_history_not_qualified` | A service worker started the run, and the steps before the failure include a memo upsert, search-attribute upsert, signal wait, applied update or completed child workflow call. Redrive refuses these steps for service runs until it can check each SDK runtime's replay capability for them. |
| `request_id_already_used` | The `request_id` was already used for another redrive of the instance. |
| `already_redriven` | Reported by run detail and `describe()` for a run that was already redriven. A redrive request for it returns the existing successor instead. |
| `invalid_request_id` | The `request_id` is longer than 191 characters. |
| `run_not_found`, `instance_not_found` | The run or instance does not exist in the namespace. |

The remaining reasons report a history that does not have the expected shape: `invalid_history`, `events_after_failure`, `duplicate_start`, `missing_start`, `unexpected_start`, `missing_terminal_history`, `invalid_activity_sequence`, `activity_sequence_gap`, `missing_activity_type`, `activity_type_mismatch`, `duplicate_activity_completion`, `incomplete_activity_prefix` and `invalid_failed_step`. Export the run's history and inspect it before you decide how to recover.

When redrive refuses a run, preserve its history, reconcile external side effects and decide whether the application should start a new workflow.

### Check whether a run can be redriven {#redrive-check}

In Workflow 2.18.0 and later, run detail reports `can_redrive` and `redrive_blocked_reason`. Waterline's selected-run detail reads these fields. `WorkflowControlPlane::describe()` reports the same fields under `actions`. They come from the same check `redrive()` applies, so you can see whether a run can be redriven, and why not, before you request it.

A run that is not failed reports `run_not_failed`. A run that was already redriven reports `already_redriven`. For a service-mode run, the report only checks that a fingerprint was recorded. `redrive()` still compares it with the worker's fingerprint. The report doesn't run the corrected-code replay check, so a changed definition shows `workflow_definition_unavailable_or_changed` until you request the redrive with `allow_definition_change`.

In Workflow 2.19.0 and later, a successor that runs corrected code reports the replaced fingerprint as `workflow_definition_changed_from` in run detail and as `run.definition_changed_from` in `describe()`.

In Workflow 2.20.0 and later, run detail and `describe()` report the redrive link as typed lineage. The successor's `parents` and the failed run's `continuedWorkflows` list it with `link_type: redrive` and `history_authority: typed_history`, taking the step and time from the successor's `WorkflowStarted`. A link that history doesn't confirm is listed once with `history_authority: mutable_open_fallback` and `diagnostic_only: true`.

### Earlier releases and the native Server {#redrive-earlier-releases}

Workflow releases before 2.26.0 refuse any child workflow in the history with `unsupported_history`. They ignore `reused_recorded_at` on child workflow events, so a 2.26.0 successor that reused a child call can replay there with a different workflow time, and different deadlines, after the call. They also refuse to redrive it again. Once such a successor exists, keep workers on 2.26.0 or later, or confirm that no such history can replay, before you roll back.

Workflow 2.20.0 refuses nothing that 2.20.1 accepts. In 2.20.0, `outbox()->sendReference()` in reused workflow code sends again, and a replayed `inbox()->receive()` can return different messages or none, or consume a newer message ([workflow#1010](https://github.com/durable-workflow/workflow/issues/1010)).

Workflow 2.19.0 refuses an applied update (`unsupported_history`). It also refuses to redrive a successor that reused an update (`unsupported_history`) or a signal wait (`mismatched_signal_receipt`). Run detail and `describe()` show redrive links as diagnostic fallbacks.

Workflow 2.18.0 reuses completed activities, side effects, version markers and fired timers. It refuses a memo or search attribute that workflow code upserted (`unsupported_history`), a signal wait (`unsupported_history`), a run that failed in workflow code (`unrecorded_failure_boundary`) and a changed definition (`workflow_definition_unavailable_or_changed`).

Workflow releases before 2.18.0 accept a narrower set of runs. Every step before the failed activity must be a completed activity. A side effect, version marker or timer in that history is refused with `unsupported_history`, and any memo or search attribute is refused with `unsupported_visibility_metadata`. Run detail and `describe()` do not report `can_redrive` or `redrive_blocked_reason`.

Releases before 2.19.0 ignore `reused_recorded_at` on reused signal, memo and search-attribute steps. After a Workflow 2.19.0 or later successor reuses those steps, keep 2.19.0 or later workers, or confirm that no such history can replay, before you roll back. In embedded mode, Workflow 2.19.0 replays, imports and resumes a 2.20.0 successor that reused an update or a chained signal wait the same way 2.20.0 does ([qualification receipt](https://github.com/durable-workflow/workflow/issues/864#issuecomment-6617902412)).

From 3.7.0, native Server redrive carries memos and search attributes set at start to the successor, and `describe()` reports `can_redrive` and `redrive_blocked_reason` under `actions`. From 3.8.0 it also reuses side effects and version markers before the failed step, recording each with its original payload, `reused_from_run_id` and `reused_recorded_at`, so workflow time after them matches the failed run. A 3.8.0 Server refuses to redrive a run whose workflow code upserted a memo or search attribute (`service_redrive_history_not_qualified`) or whose side effect result is stored externally (`external_side_effect_result_not_supported`). Otherwise it still applies the rules from before 2.18.0, so a fired timer is still refused with `unsupported_history`. Native Server support for the remaining rules is tracked in [server#980](https://github.com/durable-workflow/server/issues/980).

## Service Entry Refusal and Recovery {#service-entry-recovery}

**Released support.** Server 2.9.9 and PHP SDK 2.5.4, Python SDK 2.6.2 and Rust SDK 3.5.6 introduced the entry safeguards and routing/recovery behavior below. These are minimum feature versions; install current stable releases for security fixes.

An entry mismatch is a deployment/replay refusal before application workflow code runs, not an activity retry or a terminal business failure. The worker reports `failure_reason = workflow_entry_arguments_mismatch`, and the Server retains the failed workflow task with `replay_blocked = true` instead of automatically creating another attempt. Its generic `replay_blocked_reason` is `worker_reported_replay_failure`. The run remains nonterminal, with its recorded input, history and deadlines intact. This differs from an exception thrown by a successfully entered workflow body, which can still fail the run.

Recover only after checking the actual durable state:

1. Inspect the selected run with `GET /api/workflows/{workflowId}/runs/{runId}/debug`. Confirm `latest_workflow_task_failure.reason` is `workflow_entry_arguments_mismatch` and that the original run is still open. A terminal run is not reopened by this procedure.
2. Deploy code that accepts the recorded positional arguments or typed input. Preserve the original workflow/run id, namespace, queue, build compatibility, history and deadlines. Never edit stored input or substitute new DTO defaults.
3. Confirm an eligible worker is available using [definition routing](./polyglot/worker-compatibility-routing.md#definition-fingerprint-evolution). Repair does not create a worker or prove replay compatibility.
4. Use `POST /api/workflows/{workflowId}/runs/{runId}/repair` with the normal authorized operator credential and namespace. The selected-run route refuses a historical target instead of silently repairing a newer run. PHP also exposes `Client::repairWorkflow($workflowId, $runId)`; Python's `repair_workflow(workflow_id)` targets the current instance, and Rust has no equivalent repair convenience method.
5. Verify the same original resumes with its recorded input and history prefix, then records its expected outcome once. An elapsed original deadline still applies. If the replacement remains incompatible, inspect the new refusal rather than repeatedly issuing repair.

Older Server releases may store the SDK's specific reason without classifying it as a replay block, so reporting the failure alone does not establish the no-automatic-retry contract. Use the qualified runtime/SDK combination. See [Versioning](./features/versioning.md#service-sdk-entry-safeguard) for language boundaries and [Workflow Commands](./polyglot/server-api-reference.md#workflow-commands) for the authenticated selected-run routes.

## Workflow Timeout Enforcement

When `StartOptions::withExecutionTimeout()` or `StartOptions::withRunTimeout()` is set, the engine records a deadline on the workflow run. The execution deadline spans the entire logical workflow (including continue-as-new runs), while the run deadline resets with each new run.

If a deadline has passed when the engine starts a workflow task, the run is closed immediately:

- All open activity executions, timers, and outstanding tasks are cancelled with typed history events (`ActivityCancelled`, `TimerCancelled`).
- A `WorkflowFailure` row is recorded with `failure_category = timeout` and `propagation_kind = timeout`.
- A `WorkflowTimedOut` history event is recorded with `timeout_kind` set to `execution_timeout` or `run_timeout`.
- The run status becomes `failed` with `closed_reason = timed_out`.
- Parent workflows waiting on the timed-out child are notified.

The background task watchdog also scans for non-terminal runs with expired deadlines that have no open workflow task (for example, a run waiting on an activity or timer when the deadline passes). When it finds one, it creates a workflow task so the executor can detect and enforce the timeout on the next pass.

Waterline surfaces `failure_category` in the exceptions table as a dedicated **Category** column and in timeline failure detail entries. History exports include `failure_category` in the `failures[*]` array. Final v2 writes this classification when the failure is recorded; imported v1 rows that cannot be classified remain visible as unclassified diagnostics.

## Activity Retries

For a shared attempt budget across workflows and queues, combine authored retry limits with [Server admission and an application breaker](./polyglot/task-queue-admission.md#retry-load-and-downstream-budgets). First attempts and retries both consume the cap; local/embedded execution has different boundaries.

`Workflow\V2\Activity` defaults to `$tries = 1`, so an activity failure is sent back to the workflow immediately unless the activity opts into retry attempts.

```php
use RuntimeException;
use Workflow\V2\Activity;

class ChargeCard extends Activity
{
    public int $tries = 3;

    public function backoff(): array
    {
        return [5, 30];
    }

    public function handle(): string
    {
        throw new RuntimeException('temporary gateway failure');
    }
}
```

From workflow code, bound the total activity time, including queue waits and backoff:

```php
use Workflow\V2\Support\ActivityOptions;
use function Workflow\V2\activity;

$result = activity(ChargeCard::class, new ActivityOptions(
    scheduleToCloseTimeout: 120,
));
```

Together these settings permit at most three attempts, with backoffs of 5 and 30 seconds, under the original 120-second schedule-to-close deadline. An earlier workflow deadline still applies. Enable retries only for operations that can safely be repeated, using idempotency guards for external effects.

When a retryable activity throws before `$tries` is exhausted, the engine closes the current `activity_attempts` row as runtime state, returns the `activity_executions` row to `pending`, records a typed `ActivityRetryScheduled` history event for the failed try, and creates a new durable activity task with `available_at` set from the `backoff()` policy. The workflow stays waiting on that same activity execution and is not resumed with the exception until the final retryable attempt fails.

The retry task records `retry_of_task_id`, `retry_after_attempt_id`, `retry_after_attempt`, and `retry_backoff_seconds` in its payload so Waterline can explain why the task is scheduled. Selected-run detail rebuilds the failed attempt in `activities[*].attempts` from typed activity history first, shows `ActivityRetryScheduled` in the timeline, and reports retrying activity counts through `operator_metrics.activities.retrying`, `operator_metrics.activities.failed_attempts`, and `operator_metrics.backlog.retrying_activities`.

`Workflow\Exceptions\NonRetryableExceptionContract` still short-circuits the retry policy: throwing a non-retryable exception fails the activity execution immediately and resumes the workflow with the exception.

### Retry delay resolution

Activity `backoff()` values are integer seconds. Durable retry intervals have
whole-second resolution. Do not use a fractional value such as `0.1` to request
a 100 ms retry. SDK APIs that generate intervals from fractional values round
them up to the next second. The Python and Rust SDK references describe their
accepted parameter types and conversions.

A retry becomes eligible after its delay, but queue latency and worker load can
make the next attempt start later. For fast tests, use
[controlled time or bounded zero-delay fixtures](./testing.md#keeping-delay-tests-fast).

### Activity execution identity and idempotency

Retry is not the only reason the same logical activity can be observed more than once. Lease expiry, worker loss, delayed completion reporting, and redelivery can all produce another attempt or a stale completion report for the same durable activity execution.

- `activity_execution_id` identifies the logical activity across retries and redelivery. Use it as the default idempotency key for remote side effects.
- `activity_attempt_id` identifies one specific try of that logical activity. Use it only when a downstream system must distinguish separate attempts.
- A late completion or failure report from a superseded attempt is normal stale-attempt behavior, not proof that the engine committed the same attempt twice.

When operators investigate a late completion after lease expiry:

- trust Waterline, history export, or the server API for which attempt won the
  durable race
- do not assume a rejected late completion means the remote side effect did not
  happen
- check the external system by its idempotency key before forcing manual retry
  or repair

The safest default is to make the remote side effect idempotent under
`activity_execution_id`, then let the durable outcome tell you whether the
engine accepted the report for that specific attempt.

### Non-retryable failure markers

When an activity or workflow throws an exception that implements `Workflow\Exceptions\NonRetryableExceptionContract`, the engine records a `non_retryable = true` flag on the `WorkflowFailure` row and in the typed history event payload (`ActivityFailed`, `WorkflowFailed`, `UpdateCompleted`). This durable marker communicates to operators, external workers, and tooling that the failure is permanent — retrying the same operation will not succeed.

The flag flows through the full visibility stack:

- **Failure rows**: `workflow_failures.non_retryable` boolean column.
- **History events**: `non_retryable` field in the typed event payload.
- **Failure snapshots**: `non_retryable` included in `FailureSnapshots::forRun()`.
- **Run detail view**: `non_retryable` in the exceptions array.
- **Timeline entries**: `non_retryable` in failure detail metadata.
- **History exports**: `non_retryable` in the `failures[*]` array.
- **Waterline**: a "non-retryable" badge next to the failure category in the exceptions table and timeline.
- **External worker bridge**: the `complete()` command payload accepts `non_retryable` so external workflow workers can report non-retryable failures without requiring the host process to resolve the throwable class.

For failures that do not implement the contract, `non_retryable` is `false` by default. Final v2 records that durable marker at failure time, so declare the contract before the failure is written when operators or SDKs need to distinguish permanent failures from retryable ones.

```php
use Workflow\Exceptions\NonRetryableExceptionContract;

class PaymentDeclinedException extends \RuntimeException implements NonRetryableExceptionContract
{
    // This failure will be marked as non-retryable in the durable record.
}
```

## Workflow-Level Retry

Durable Workflow v2 does **not** support automatic workflow-level retry. When a workflow run fails — whether from an unhandled exception, a structural limit, or a timeout — the run is terminal. The engine does not automatically start a new run of the same workflow instance.

This is an intentional design choice:

- **Activities already have retry.** Activity retry policies with configurable `$tries`, `backoff()`, and non-retryable exceptions handle transient failures at the right granularity.
- **Workflow replay is the recovery primitive.** If a workflow task encounters a transient infrastructure failure (database error, worker crash), the durable task system re-dispatches the task, and replay resumes from committed history — no new run needed.
- **Continue-as-new handles long-lived workflows.** Workflows that need fresh state or history compaction use `continueAsNew()` as an explicit workflow-level restart.
- **Repair handles stuck runs.** The `repair()` command and automatic worker-loop repair recover runs where durable task transport was lost.
- **Redrive is an explicit operator action.** [Redrive](#redrive) continues a failed run in a new run only when you request it, and it reuses completed activity results.

If your application needs workflow-level retry semantics, model them explicitly:

```php
use function Workflow\V2\activity;
use Throwable;
use Workflow\V2\Workflow;

class RetryableWorkflow extends Workflow
{
    public function handle(string $orderId): void
    {
        try {
            activity(ProcessOrderActivity::class, $orderId);
        } catch (Throwable $e) {
            // Record the failure, then start a new workflow
            // for retry-at-workflow-level scenarios.
            activity(NotifyFailureActivity::class, $orderId, $e->getMessage());
        }
    }
}
```

## Core timeline summary bounds {#core-timeline-summary-bounds}

Workflow **2.12.0 or newer** bounds the complete derived timeline summary to 65,535 UTF-8 bytes, including labels, punctuation, the ` [truncated]` marker and any callback-recovery notice. It preserves the authoritative recorded failure, original type and non-retryable decision. Check the receiving runtime's actual `composer.lock`. The [normal installed-package receipt](https://github.com/durable-workflow/workflow/issues/915#issuecomment-6372539577) verifies the summary boundaries independently of SDK reporting limits.

The SDKs' 62 KiB reporting headroom does not fix an older Core summary writer. An oversized activity failure is recorded end to end ([.github#225](https://github.com/durable-workflow/.github/issues/225)) with PHP SDK 3.5.0, Python SDK 2.11.0 or Rust SDK 4.10.1 against native Server 3.5.0 or PHP Server 2.11.3 or newer, and with Workflow 2.13.0 or newer in embedded mode. Embedded mode has no request limit, so Workflow records the failure whole and the workflow receives it with its original type and complete message. A message beyond MySQL `TEXT` capacity, or one containing a NUL byte, is kept in full in the additive `message_full` column; `WorkflowFailure::message`, retained history and history exports return the complete message, while the legacy `workflow_failures.message` column and recent-failure lists hold a labelled, bounded preview. Workflow 2.12.0 bounds the timeline summary, 2.12.2 adds the full-message storage and PostgreSQL reads of NUL-bearing messages, and 2.13.0 adds bounded readers for rejected-outcome diagnostics. Run the package migrations after upgrading; they add the column without backfilling existing rows. Native Server 3.5.0 records a failure message over 64 KiB that earlier releases refused with `503 storage_unavailable` on MySQL and MariaDB: its failure row keeps a labelled preview, and retained history carries the full failure. Waterline 2.16.0 marks each shortened message in run details. The [history-backed datetime repair](./features/search-attributes.md#repair-datetime-search-attributes) also requires installed Workflow 2.12.0 or newer. It repairs derived search rows, not failed timeline writes or missing authoritative history.

## PHP service-mode failure report limits {#php-failure-report-limits}

PHP SDK **3.5.0 or newer** bounds outgoing service-mode failure reports to 62 KiB UTF-8 messages, 128 KiB stacks, 64 KiB combined optional metadata and a 1 MiB complete encoded body, including markers. Lower known transport limits apply. The [PHP reporting guide](./polyglot/php.md#bounded-failure-reports) explains configuration, omission diagnostics, preserved retry semantics and local first-execution/cold-replay equality. These are SDK-specific reporting bounds; embedded recording and Core summary storage retain their separate contracts.

## Python service-mode failure report limits {#python-failure-report-limits}

Python SDK **2.11.0 or newer** bounds outgoing service-mode failure reports to 62 KiB UTF-8 messages, 128 KiB stacks, 64 KiB combined optional metadata and a 1 MiB complete encoded body, including markers. Lower known transport limits apply. The [Python reporting guide](./polyglot/python.md#bounded-failure-reports) explains configuration, omission diagnostics, preserved retry semantics and the same local observation during first execution and cold replay. These limits apply to SDK reporting. Embedded recording and Core summary storage retain their separate contracts.

## Rust service-mode failure report limits {#rust-failure-report-limits}

For service-mode workers actually running Rust SDK **4.10.1 or newer**, outgoing failure messages are at most 62 KiB UTF-8 including marker, stacks 128 KiB where supported, combined serialized optional metadata 64 KiB, and the full encoded body 1 MiB or a lower known transport cap. Original type, retry decision and claim identity stay intact; impossible mandatory envelopes refuse explicitly. The [Rust reporting guide](./polyglot/rust.md#bounded-failure-reports) explains omission markers, whole Avro details and local-result boundaries. These limits do not qualify old SDKs, embedded mode or the separate Core summary-storage correction.

## Related Guides

- [Execution Guarantees and Idempotency](./constraints/execution-guarantees.md)
  explains the replay, retry, lease-expiry, and redelivery contract that shapes
  every recovery path on this page.
- [Monitoring](./monitoring.md) explains where Waterline, history export,
  worker logs, and runtime telemetry surface the failure facts described here.

<!-- Source: docs/monitoring.md -->

# Monitoring

[Waterline](https://github.com/durable-workflow/waterline) is the operator UI
and API for workflow state. Its delivery follows the selected runtime boundary:
Cloud includes Managed Waterline, while a self-hosted operator deploys
Waterline separately when that surface is wanted.

Waterline is one operator product with three consumption surfaces:

- **Embedded mode** installs the Composer package in the Laravel application
  that owns the workflows and reads that application's durable state in
  process.
- **Self-hosted service mode** runs the published
  `durableworkflow/waterline` image and reads state owned by the
  [standalone server](./polyglot/server.md) through the PHP SDK and public
  server API. The Server distribution does not bundle or operate Waterline.
- **Cloud Managed Waterline** is the namespace-scoped operator experience
  included with [Durable Workflow Cloud](./polyglot/cloud-control-plane.md).
  Cloud operates Managed Waterline and the namespace runtime behind it.

The embedded and self-hosted service surfaces expose the same core Waterline UI
and `/waterline/api/...` operator route families. Cloud provides the operator
capabilities through its managed surface instead of a customer-deployed
Waterline route host. None of the surfaces merge runtime state: each view is
limited to runs owned by its runtime and namespace. For self-hosted Server
operations, the native API, CLI, and SDK operator surfaces remain available
whether or not you deploy Waterline. Use the
[Server API Reference](./polyglot/server-api-reference.md) for those native
routes and the
[Waterline Operator API Reference](./waterline-operator-api.md) for
Waterline's routes and response contracts.

Durable Workflow has two observability planes:

| Plane | Source of truth | Typical questions |
| --- | --- | --- |
| Durable state | The owning runtime's workflow database and API, Waterline projections, and history export | Did the workflow start? Which run is current? Which signal, update, timer, activity, retry, or failure was committed? Which operator action is safe now? |
| Worker/runtime telemetry | Queue worker logs, SDK metrics recorders, Prometheus/OpenMetrics endpoints, and application traces | Are workers polling? How long do tasks take? Is an exporter configured? Did custom application metrics leave the worker process? |

Waterline intentionally does not replace worker metrics. If a custom metric
was recorded in activity or worker code, scrape the worker's telemetry
endpoint. Use Waterline to correlate that runtime signal with the durable
workflow history and current run state.

Worker and client setup also remains separate from the operator surface. Use
the generated [PHP SDK API reference](https://php.durable-workflow.com/api/),
[Python SDK API reference](https://python.durable-workflow.com/), or
[Rust SDK API reference](https://rust.durable-workflow.com/) alongside the
language guides when connecting application clients and workers to the runtime
that owns the namespace.

When worker telemetry shows repeated claims, late completion races, or stuck
leases, read [Execution Guarantees and Idempotency](./constraints/execution-guarantees.md)
alongside this guide. That contract separates at-least-once transport
uncertainty from duplicate durable outcomes so duplicate-looking evidence does
not turn into the wrong operational conclusion.

Rust SDK **4.7.0 or later** provides pluggable metrics recorders and an optional `metrics-prometheus` adapter. The default is no-op; applications own the scrape endpoint and exporter lifecycle. See [Rust metrics setup and the released example](./polyglot/rust.md#rust-sdk-metrics) for exact names, bounded tags and process-observation limits. This does not configure Server or Cloud metrics.

## OpenTelemetry tracing {#opentelemetry-tracing}

Connect application requests, workflow tasks and activity attempts across processes with optional OpenTelemetry tracing. Choose the setup for the runtime that owns your runs.

### Embedded Laravel {#embedded-tracing}

To correlate application logs with a workflow run and activity attempt, configure
the optional [embedded tracing integration](https://github.com/durable-workflow/workflow/blob/main/docs/embedded-tracing.md).
It persists W3C context across queue workers, retries and recovery and uses your
application's OpenTelemetry provider and Laravel log channels. Follow the guide
to enable tracing, add the log tap, run the schema migrations and configure
bounded exports. Review its span diagram and runtime/SDK capability matrix
before rollout.

In the application’s `config/workflows.php`, map an application-owned environment flag to the existing tracing configuration:

```php
'tracing' => [
    'enabled' => (bool) env('DW_TRACING_ENABLED', false),
    'baggage_allowlist' => [],
],
```

This example starts with no baggage. Install the optional API and configure your provider, log tap, migrations and bounded exporter as described in the linked guide, then restart workers.

### Service-mode tracing

Configure the application's provider, sampling, propagators and bounded exporter before enabling SDK telemetry. Use the [PHP tracing guide](https://php.durable-workflow.com/operate/tracing/), [Python telemetry guide](https://python.durable-workflow.com/telemetry/) or [Rust telemetry API](https://rust.durable-workflow.com/durable_workflow/telemetry/).

| Runtime | Propagation boundary |
| --- | --- |
| Embedded Laravel | The application's optional Core integration persists context across its queue workers. |
| PHP Server | The [Server trace-context guide](https://github.com/durable-workflow/server/blob/2.10.0/docs/observability-context.md) requires an application image with the optional API, enabled flag and actual nullable schema. |
| PHP, Python and Rust SDKs | Opt-in task/attempt spans and scoped log correlation. Durable caller propagation requires the exact advertised Server capability. Missing or malformed capability uses independent task traces. |
| Native Rust Server | This PHP tracing release does not qualify native carrier support or catalog conversion. |
| Cloud | This release includes no Cloud deployment. Confirm support with the managed runtime. |

Trace IDs, durable run/activity/attempt IDs and application request IDs remain distinct. Baggage is allowlisted and empty by default; links carry no baggage. Providers and exporters remain application-owned. Abrupt termination can lose telemetry, and replay does not promise exactly-once export. Use durable history for business outcomes.

## Waterline views {#waterline-views}

### Dashboard View

The dashboard shows running totals, recent-run counters, and fleet-wide
metrics so you can tell at a glance whether work is flowing, stalling, or
failing.

Use the [Operator Operating Envelope](./operator-operating-envelope.md) when
you need the rollout and runbook contract for those facts: which diagnostics
block traffic, which are advisory, how queue-health facts split between
Waterline and worker telemetry, and how to verify rebuild, export, and archive
paths.

### Workflow View

The workflow detail view shows the durable timeline for a single run: the
activities, signals, timers, and child workflows that happened in order,
each with its inputs, outputs, and timing.

## Stuck runs

Every open run has a `liveness_state` that says what it is waiting for, and a `liveness_reason` sentence. Both appear on run detail, list rows and history exports.

| State | Meaning |
| --- | --- |
| `workflow_task_ready`, `activity_task_ready`, `timer_task_ready` | A task is ready for a worker. |
| `workflow_task_leased`, `activity_task_leased`, `timer_task_leased` | A worker holds the task. |
| `workflow_task_claim_failed`, `activity_task_claim_failed`, `timer_task_claim_failed` | A worker tried to claim the task and was refused, for example by a backend capability check. |
| `workflow_task_waiting_for_compatible_worker`, `activity_task_waiting_for_compatible_worker`, `timer_task_waiting_for_compatible_worker` | No registered worker is compatible with the run. |
| `timer_scheduled` | Waiting for a timer. |
| `waiting_for_signal`, `waiting_for_condition`, `waiting_for_child` | Waiting for a signal, a condition change or a child workflow. |
| `activity_running_without_task` | An activity is running without an open task. Repair waits so the work is not run twice. |
| `workflow_replay_blocked` | The workflow task cannot replay the recorded history, for example after an incompatible code change. The run stays open until the code is fixed. |
| `repair_needed` | The run has no trusted way to continue, for example after an expired lease, a failed dispatch or a missing task. This includes service-mode workflow tasks that workers keep failing. |
| `closed` | The run is closed. |

### Task problem filter

Runs with workflow-task trouble have `task_problem = true`. Filter run lists on `task_problem` (Task Problem in Waterline). `task_problem_badge.code` says which kind:

- `replay_blocked` (Task Failure): the workflow task cannot replay.
- `active` (Task Problem): workflow-task work is missing, retrying or cannot be delivered right now.
- `history` (Task History): the run needed workflow-task repair or replay recovery earlier.

### Metrics and health {#stuck-run-metrics}

Embedded operator metrics (`operator_metrics` in Waterline stats) count open runs in each problem state. Closed runs are not counted.

| Field | Meaning |
| --- | --- |
| `operator_metrics.runs.replay_blocked` | Open runs in `workflow_replay_blocked`. |
| `operator_metrics.runs.oldest_replay_blocked_at`, `operator_metrics.runs.max_replay_blocked_age_ms` | The last update of the stalest replay-blocked run, and its age. |
| `operator_metrics.runs.repair_needed`, `operator_metrics.runs.oldest_repair_needed_at`, `operator_metrics.runs.max_repair_needed_age_ms` | The same for `repair_needed`. |

The advisory `replay_blocked_runs` health check (category `correctness`) reports `warning` while any open run is replay-blocked.

Native Server reports `replay_blocked` and the `replay_blocked_runs` check from 3.6.0.

From Workflow 2.27, `operator_metrics.namespace_scope_refusals` counts work the app's [namespace allowlist](./configuration/options.md#namespace-allowlist) refused in the last 15 minutes, by namespace and path. `job` counts skipped queued jobs. `repair`, `timeout` and `schedule_tick` count repair candidates, expired deadlines and due schedules left in a namespace outside the allowlist for more than two minutes. The advisory `namespace_scope_refusals` health check warns while refusals keep arriving and clears 15 minutes after the last one. The counts live in the default cache store, so use a shared store (Redis, database or Memcached) so that the web process serving health sees the workers' counts.

### Server workflow-task failures

Server reports failed workflow tasks in `GET /api/system/metrics` under `metrics.dw_workflow_task_consecutive_failures`:

- `max_consecutive_failures`: the highest attempt count among failed workflow tasks in the namespace. The first failure counts as one.
- `failed_task_count`: how many workflow tasks are failed.
- `by_workflow_type`: up to 20 workflow types, highest `max_consecutive_failures` first, each with its own `max_consecutive_failures` and `failed_task_count`. `suppressed_workflow_type_count` and `suppressed_failed_task_count` cover the rest.

Alert when `max_consecutive_failures` keeps rising: a workflow task keeps failing on workers instead of completing.

### Repair

- `workflow_replay_blocked`: deploy workflow code that can replay the recorded history (see [Versioning](./features/versioning.md)), then repair the run.
- `repair_needed`: repair the run, or let the repair pass do it (`workflow:v2:repair-pass` embedded, `dw system:repair-pass` on Server). Server also accepts `POST /api/workflows/{workflowId}/runs/{runId}/repair`.

See [Failures and Recovery](./failures-and-recovery.md) for what repair does.

## Waterline deployment and access

### Embedded Laravel

Install Waterline into your Laravel application alongside the workflow
package and run its migrations. See
[durable-workflow/waterline](https://github.com/durable-workflow/waterline)
for the full installation and configuration guide.

Embedded mode uses the host application's database connection, route
middleware, authentication gate, and workflow package. Continue to operate the
Laravel application, its queue workers, scheduler, migrations, and Waterline
assets as one deployment boundary. The
[Waterline Operator API Reference](./waterline-operator-api.md#installation)
includes the current Composer command and asset-publishing step.

### Embedded operator permissions {#embedded-waterline-operator-permissions}

These optional controls require **Waterline 2.13.0**. Keep the host's `viewWaterline` viewing gate: action gates grant no viewing access. By default, users who can view embedded Waterline can use its operator actions; the new settings are opt-in and preserve that behavior when unset. Check `composer.lock`: this feature minimum does not update the separately selected installation tuple, whose commands may still pin an older package without these controls.

To block operator writes for every viewer, set `'read_only' => true` in `config/waterline.php`, or map `WATERLINE_READ_ONLY=true` through the host configuration before rebuilding its cache. Viewing, history exports, queries and personal saved views/preferences keep their existing authorization. This does not stop the host's workflow workers or change direct SDK/Server permissions.

For separate host-user permissions, define Laravel gates in a service provider and map every action group you want to restrict. The application ability names below belong to your host policy:

```php
use Illuminate\Support\Facades\Gate;

Gate::define('waterline-messages', fn ($user) => $user->can('support-workflows'));
Gate::define('waterline-operations', fn ($user) => $user->can('operate-workflows'));
```

Merge this configuration into `config/waterline.php`:

```php
'authorization' => [
    'gates' => [
        'messages' => 'waterline-messages',
        'repair' => 'waterline-operations',
        'cancel' => 'waterline-operations',
        'terminate' => 'waterline-operations',
        'archive' => 'waterline-operations',
        'schedules' => 'waterline-operations',
    ],
],
```

`messages` covers signals and updates; `schedules` covers pause, resume, trigger, backfill and delete. Each gate receives the current host user and optionally the HTTP request as its second argument. A null or omitted mapping preserves default action access. A configured gate must exist and authorize the user; invalid settings refuse the action. Normal Laravel guest and `Gate::before` rules apply, but global read-only takes precedence even over administrator overrides. No gate bypasses namespace or runtime action restrictions.

Deploy the host's resolved configuration and rebuild its configuration cache with `php artisan config:cache`; an environment change alone does not update cached or literal settings. Install the qualified package and publish its matching assets through the ordinary [installation steps](./waterline-operator-api.md#installation). Denied writes return HTTP 403 with `reason=waterline_action_forbidden` or `reason=waterline_read_only` and a `capability`. The UI hides unavailable run controls and explains disabled actions/schedule controls; backend enforcement also covers legacy and exact-run mutation URLs.

These are **embedded host-user controls**. Service mode keeps `WATERLINE_ACCESS_MODE=read_only` and its upstream token permissions rather than Laravel host gates. This feature changes no SDK/Core dependency floor and does not configure Cloud Managed Waterline.

### Waterline service

The published image contains its own PHP and Laravel runtime. It does not need
PHP, Composer, or the workflow package on the container host, and it never
connects to the standalone server's database.

This example binds Waterline to loopback on host port `8080`, persists its own
UI state in a named volume, and connects it to one server namespace:

```bash
export WATERLINE_SERVER_ENDPOINT=https://workflow.example.com
export WATERLINE_SERVER_TOKEN=replace-with-a-server-token

docker run --detach \
  --name waterline \
  --restart unless-stopped \
  --publish 127.0.0.1:8080:8080 \
  --volume waterline-data:/data \
  --env WATERLINE_SERVER_ENDPOINT \
  --env WATERLINE_SERVER_TOKEN \
  --env WATERLINE_NAMESPACE=orders \
  --env WATERLINE_ACCESS_MODE=read_only \
  --env WATERLINE_ALLOW_UNAUTHENTICATED=true \
  --env APP_URL=https://waterline.example.com \
  durableworkflow/waterline:2.18.0
```

Run image 2.16.0 or later. The 2.14.0 image, and the 2.15.0 image that was
published to Docker Hub but never announced, ship Symfony 7.4.21, which is
affected by CVE-2026-118514 and CVE-2026-118516. An embedded Waterline uses
your application's own Symfony packages, so update them there.

Open `/waterline` through the URL represented by `APP_URL`. The image listens
on container port `8080`; set `PORT` only when you intentionally change that
internal port. `WATERLINE_PATH` changes the default `waterline` URL prefix.
The repository also publishes a
[Docker Compose service definition](https://github.com/durable-workflow/waterline/blob/main/deploy/docker-compose.service.yml)
with the same connection boundary.

Service mode has two independent authentication layers:

1. `WATERLINE_SERVER_TOKEN` is the bearer credential Waterline uses when the
   PHP SDK calls `WATERLINE_SERVER_ENDPOINT`. Workflow, worker, queue, and
   schedule observation needs a server operator role; server health and
   operator metrics may need an admin role.
2. Browser and Waterline API access is the deployment's front-door boundary.
   The self-contained image has no host Laravel user directory.
   `WATERLINE_ALLOW_UNAUTHENTICATED=true` therefore belongs only behind an
   authenticating reverse proxy or on a private interface such as the loopback
   binding above. Keep it `false` unless that surrounding authentication
   boundary is in place.

Set `WATERLINE_NAMESPACE` to the same namespace operators intend to inspect.
Waterline sends it on every SDK request. `WATERLINE_ACCESS_MODE=read_only` is
the default and blocks mutating actions in Waterline; use `operator` only with
a server token authorized for the required commands.

The `/data` volume holds Waterline-owned saved views, display preferences, and
Laravel runtime state. With the default settings it contains a file-backed
SQLite database at `/data/waterline.sqlite`. It never contains the server's
workflow history. Use `DATABASE_URL` or the ordinary `DB_*` settings when
Waterline's own state must live in MySQL or PostgreSQL.

#### Health and metrics

Use these probes for different questions:

| Surface | What it proves |
| --- | --- |
| `GET /up` | The Waterline HTTP process started and can answer requests. The image's Docker health check uses this route. |
| `GET /waterline/api/v2/health` | Waterline can assemble namespace-scoped server health, worker registration, and task-queue evidence through the PHP SDK. |
| `GET /waterline/api/stats` | Dashboard totals and the server-backed operator summary used by Waterline. |
| `GET /api/system/health` and `GET /api/system/operator-metrics` on the server | The native server health and operator contracts, independent of Waterline. |

Treat `/waterline/api/stats` as operator JSON, not as a Prometheus scrape
endpoint. Worker SDK metrics, logs, traces, and custom application telemetry
still come from worker processes. A healthy `/up` with an unavailable
Waterline health or stats response points to the server connection,
authorization, namespace, or SDK capability boundary rather than a failed
Waterline process.

#### Workflow visibility and operator actions

In service mode, list views, selected-run detail, history export, schedules,
worker status, task-queue evidence, signals, updates, queries, repair, cancel,
terminate, and archive are projected from the configured standalone server
through the PHP SDK. The Waterline route shapes remain the ones documented in
the [Waterline Operator API Reference](./waterline-operator-api.md); the
underlying server contracts remain documented in the
[Server API Reference](./polyglot/server-api-reference.md).

Waterline only shows the configured namespace and the runs owned by the
connected server. Embedded runs remain visible through their embedded
Waterline deployment, while server-managed runs remain visible through the
service deployment or the server's native surfaces. Changing the Waterline
backend does not migrate or combine runs.

#### Troubleshooting boundaries

| Symptom | Boundary to check |
| --- | --- |
| Container exits before serving `/up` | Check the required `WATERLINE_SERVER_ENDPOINT`, writable `/data`, valid `PORT`, database settings, and bounded startup migration logs. |
| Waterline returns `401` or `403` from a server-backed view | Check `WATERLINE_SERVER_TOKEN` and its server role. A Waterline front-door denial is a separate proxy or `WATERLINE_ALLOW_UNAUTHENTICATED` issue. |
| A mutating route returns `waterline_read_only` | Keep the deployment read-only or explicitly set `WATERLINE_ACCESS_MODE=operator`; the server token must still authorize the command. |
| Expected workflows are absent | Confirm `WATERLINE_NAMESPACE`, the server endpoint, and which runtime accepted the workflow start. Waterline does not search other namespaces or embedded runtimes. |
| `/up` passes but health, stats, or a view reports an unavailable capability | Verify the server endpoint directly with its API or CLI, then check token roles and that the published Waterline/PHP SDK tuple exposes the required method. |
| A service-catalog route reports `backend_capability_unavailable` | Service mode does not mirror Waterline's embedded cross-namespace service catalog. Use the connected server's native service endpoints and API for that capability. |
| A custom metric is missing from Waterline | Inspect the worker's metrics exporter. Waterline reports durable operator facts, not arbitrary process metrics. |

### Environment banner {#waterline-environment-banner}

Label production and non-production Waterline tabs with the environment banner. These variables work in embedded Laravel and standalone service mode:

```dotenv
WATERLINE_ENV_NAME=production
WATERLINE_ENV_COLOR="#dc3545"
```

`WATERLINE_ENV_COLOR` selects a fixed CSP-compatible palette:

| Accepted value | Banner color |
| --- | --- |
| `#0d6efd` or `#2563eb` | Blue |
| `#198754` or `#28a745` | Green |
| `#6f42c1` or `#7746ec` | Purple |
| `#dc3545` | Red |
| `#fd7e14` | Orange |
| `#ffc107` | Yellow |

Other values render neutral gray, including valid hex colors such as `#ff0000`. Use a listed palette value for the intended color.

In embedded Laravel, the resolved `env_name` and `env_color` keys in `config/waterline.php` are authoritative. A published configuration can override the environment-based values. After changing environment-based configuration, rebuild the host application's configuration cache through its normal deployment:

```bash
php artisan config:cache
```

Inspect the resolved Waterline configuration when the banner differs from `.env`; cached configuration remains in effect until rebuilt. In standalone service mode, update the Waterline deployment environment and recreate or redeploy the Waterline container so it loads the new values. These are Waterline display settings; apply them to the Waterline service described above.

### Cloud Managed Waterline

Cloud customers open Cloud Managed Waterline from the namespace-scoped managed
operator surface. They do not deploy or configure Waterline, Server, PHP, or
internal endpoints, and they do not maintain a second Waterline login. The
self-hosted image, `WATERLINE_*` settings, Server token, database, and
front-door authentication instructions above do not apply to this path.

The customer's Cloud authentication establishes the operator identity.
Authorization across the Cloud organization, project, environment, and
namespace determines which managed Waterline scope that identity can open.
Within that boundary, operators can use workflow lists and search, run detail
and durable history, namespace health, and the operator actions supported for
their role. The managed surface remains limited to the selected namespace; it
does not combine data from other environments or namespaces.

Mutations are role-gated. Cloud attributes each supported operator mutation to
the authenticated Cloud actor in its audit data, while the resulting durable
workflow transition remains visible in workflow history. For example, a
successful archive is attributed in Cloud audit data and records the durable
`WorkflowArchived` history event. Operators never need a private runtime
credential or knowledge of the managed runtime's internal deployment to use
this surface.

## List and detail API

Waterline's list views (`/waterline/api/flows/{bucket}`) and selected-run
detail endpoint (`/waterline/api/flows/{id}`) return typed JSON contracts
that you can consume directly from your own dashboards or scripts. The
[Waterline Operator API Reference](./waterline-operator-api.md) documents the
endpoint list, selected-run field families, history export, actionability,
schedules, saved views, preferences, and operator-action contract.

### Actionability Contract

Waterline annotates list rows, selected-run detail responses, and history
exports with a versioned actionability contract. Consumers should treat
`actionability_contract.schema = waterline.actionability` and
`actionability_contract.version = 1` as the contract identifier for the fields
below.

Run-level `actionability` answers whether the selected run can be repaired:

| Field | Meaning |
| --- | --- |
| `repair_state` | One of `repairable`, `blocked`, `not_needed`, or `unknown`. |
| `repairable` | Boolean shorthand for `repair_state = repairable`. |
| `blocked_reason` | Stable reason code when `repair_state = blocked`. |
| `status_bucket` | The Waterline bucket that shaped the run-level decision. |
| `closed_reason` | Durable close reason when the run is closed. |
| `task_problem` | Whether Waterline saw a task-level problem on the run. |
| `diagnostic_only_evidence` | True when at least one child evidence row is informative but not a resume source. |

Evidence rows under `activities`, `waits`, `timers`, `exceptions`, `logs`, and
timeline/export entries can also include their own `actionability` block:

| Field | Meaning |
| --- | --- |
| `state` | `actionable` when the row is a valid repair source, otherwise `diagnostic_only`. |
| `repair_source` | True only for rows backed by a repairable source authority. |
| `diagnostic_only` | True when the row must not be used as a resume source. |
| `history_authority` | Source authority, such as `typed_history`, `mutable_open_fallback`, `failure_row_fallback`, or `unsupported_terminal_without_history`. |
| `history_unsupported_reason` | Stable reason code for unsupported fallback history. |

Automation should gate repair, resume, and replay affordances from
`actionability.repair_state`, `actionability.repairable`, and row-level
`actionability.repair_source`. A row with `diagnostic_only = true` is never a
durable resume source, even when it contains useful failure or fallback
metadata. Rows with `history_authority = unsupported_terminal_without_history`
are diagnostic evidence only; they explain why a run is blocked, but they do
not prove enough typed history to rebuild progress safely.

## Control-plane actions from Waterline

Operators can cancel, terminate, repair, and archive workflows directly
from the detail view. Each action maps to a `POST` on the same run id and
returns either `200` with the resulting state or `409` when the action is
not valid for the run's current state.

In service mode, Waterline forwards supported commands through the PHP SDK.
Both `WATERLINE_ACCESS_MODE=operator` and a server credential authorized for
the command are required for mutations. Operators can always use the server
API or CLI directly when Waterline is not deployed.

In Cloud Managed Waterline, Cloud role and namespace authorization gate each
supported mutation, and the authenticated Cloud identity supplies its audit
attribution. Customers do not configure a Waterline-to-Server credential for
the managed surface.

## Related Guides

- [Execution Guarantees and Idempotency](./constraints/execution-guarantees.md)
  explains the replay, retry, lease-expiry, and durable-outcome contract that
  shapes operator evidence.
- [Operator Operating Envelope](./operator-operating-envelope.md) ties health,
  queue state, rebuild, export, archive, and topology expectations into one
  operator contract.
- [Failures and Recovery](./failures-and-recovery.md) explains retry exhaustion,
  non-retryable failures, timeouts, and repair behavior behind the dashboard
  facts.
- [AI-Assisted Development](./ai-assisted-development.md) names the Waterline,
  CLI, MCP, and LLM-readable contracts that agents should use when diagnosing
  workflow state.

<!-- Source: docs/deployment.md -->

# Self-Hosting Deployments

For self-hosted or Cloud runtime changes, follow [moving between runtimes](/docs/polyglot/runtime-migration/). Keep existing runs at their source while new starts move.

Durable Workflow v2 supports several self-hosted shapes. Pick the smallest path
that matches the environment you operate, then keep the image, database, cache,
auth, readiness, and upgrade contract explicit.

This guide covers the standalone server distribution. If you run the Laravel
package embedded in your own app, use the package installation and configuration
pages instead. Durable Workflow Cloud is a separate managed-service choice in
which Cloud operates the runtime, persistence, placement, and recovery; a
self-hosted Server is never attached to Cloud. See
[Cloud Managed Runtime](/docs/polyglot/cloud-control-plane). The current
Cloud contract is a single-region managed runtime and recovery boundary; it
does not promise multi-region replication, automatic regional failover, or
failback.

The self-hosted Server distribution does not bundle or operate Waterline.
Deploy Waterline separately only when you want its operator UI, and connect it
to a Server-owned namespace using the
[Waterline deployment and monitoring guide](/docs/monitoring/#waterline-service).
Cloud instead includes Managed Waterline for its managed namespace.

The current Server is the native 3.x Server (`durableworkflow/server:3.10.0`).
It runs as one process per node: it creates and checks its own database schema
at startup and fires timers and schedules and runs its maintenance passes
itself, so there is no bootstrap job, queue worker, scheduler or Redis. PHP
Server 2.x is frozen at 2.11.8 and receives no further releases.

:::note Existing PHP Server 2.x installations

PHP Server 2.11.8 is retired and unpatched for the November Laravel advisories. For pinned PHP deployments, follow the [conditional exposure and ingress/signed-link actions](./polyglot/server.md#retired-php-server-security) before using the legacy recipes below.

The PHP 2.x stack (server, queue worker, scheduler, bootstrap job and Redis)
remains available. At the 2.11.8 release tag it is
[`docker-compose.published.yml`](https://github.com/durable-workflow/server/blob/2.11.8/docker-compose.published.yml);
from 3.0.0 the Server repository keeps the same stack as
[`docker-compose.php.yml`](https://github.com/durable-workflow/server/blob/main/docker-compose.php.yml).
Existing PHP 2.x installations can move to native with `upgrade-takeover` by
following the [native takeover runbook](https://github.com/durable-workflow/server/blob/a051f57fcc7e3605b301581ee9105461b3ef5815/docs/rust-server-upgrade-takeover.md).
It is qualified on the published 3.0.4 image (arm64) for databases written by
PHP 2.11.8 on SQLite, MySQL 8.0 and PostgreSQL 17. MariaDB, amd64 and databases
from other PHP releases (layouts 2.8.0–2.11.8) have source-CI coverage only.
Takeover is one-way and backup-first, and PHP must be stopped. It refuses
databases and state it cannot carry over; the runbook's
[refusal tables](https://github.com/durable-workflow/server/blob/a051f57fcc7e3605b301581ee9105461b3ef5815/docs/rust-server-upgrade-takeover.md#refusals)
list each case. Rehearse it on a restored copy of your backup first. There is
no PHP rollback after takeover, and takeover is planned for deprecation in the
3.1.x line. Sections below that mention Redis, a
queue worker, a scheduler or bootstrap describe the PHP 2.x topology.

:::

## Deployment support matrix

| Path | Start from | Supported for | Not promised by this path | Commercial support starts when |
| --- | --- | --- | --- | --- |
| Local development and internal non-production | [`docker-compose.published.yml`](https://github.com/durable-workflow/server/blob/3.10.0/docker-compose.published.yml) with `DW_SERVER_IMAGE=durableworkflow/server:3.10.0` | One developer machine, LAN demos, shared staging, SDK and worker integration tests | Internet-facing production, durable backup guarantees, strict secret rotation, multi-node failover | You want help turning a working dev stack into a production runbook |
| Single-node production | [`docker-compose.published.yml`](https://github.com/durable-workflow/server/blob/3.10.0/docker-compose.published.yml) with a production env file, MySQL and Server storage volumes, a strong shared token plus role-scoped tokens, backups, TLS through a reverse proxy, and pinned image tags or digests | One VM, VPS, or internal Docker host with persistent workflow state and a simple operational model | Host-level HA, automatic database failover, multi-region recovery, zero-downtime major topology changes | The deployment carries production traffic and you want review of backup, restore, auth, TLS, upgrade, or rollback procedures |
| Small clustered deployment (PHP 2.x contract) | Published PHP 2.x `durableworkflow/server` or `ghcr.io/durable-workflow/server` images using the [PHP 2.11.8 Compose recipe](https://github.com/durable-workflow/server/blob/2.11.8/docker-compose.published.yml) as the container/process template, with 2-3 API nodes, shared external MySQL/PostgreSQL, shared Redis, independently scaled workers, and exactly one scheduler/maintenance runner | Horizontal API and worker capacity when one node is no longer enough; rolling upgrades when every guarantee in the [rolling-upgrade contract](/docs/rolling-upgrades) holds | SQLite clustering, Redis-less multi-node mode, duplicate schedulers as a steady-state topology, active/active multi-writer databases, self-hosted hands-free regional failover, broad "five-nines" or "zero-downtime" SLA promises, and self-serve single-region HA failover until its [release-evidence gate](#release-evidence-status) passes | You need sizing, failure-domain, rollout, or recovery planning across more than one host, or you intend to claim the gated single-region HA behavior |
| Helm chart for Kubernetes | The [Server-owned Helm chart](https://github.com/durable-workflow/server/tree/main/k8s/helm/durable-workflow) from `oci://ghcr.io/durable-workflow/charts/durable-workflow`, with your external MySQL or PostgreSQL database, ingress, and secret management | A repeatable production install and upgrade path. Native mode (the default) renders one Server Deployment with service, probes, and policy resources; `runtime=php` keeps an existing PHP 2.x install's worker, singleton scheduler, bootstrap job and Redis | Bundled persistence, provider-managed infrastructure, active/active multi-region, custom operators, and self-serve single-region HA failover until its [release-evidence gate](#release-evidence-status) passes | You need provider-specific architecture, capacity, recovery, or changes outside the chart's published values contract |
| Raw Kubernetes manifests (PHP 2.x shape) | The server repository [`k8s/`](https://github.com/durable-workflow/server/tree/main/k8s) manifests, using published PHP 2.x server images and your existing database, Redis, ingress, and secret management | Teams that already operate Kubernetes and want inspectable manifests for API, worker, scheduler, bootstrap, service, probes, config, and secrets | The separate Helm lifecycle, managed-Kubernetes provider validation, active/active multi-region, custom operators, environment-specific storage/networking/security decisions, and self-serve single-region HA failover until its [release-evidence gate](#release-evidence-status) passes | You need overlays, managed-cluster validation, provider-specific production planning, or intend to claim the gated single-region HA behavior |
| Active/passive multi-region evaluation | A validated single-node or small-cluster deployment per region, plus asynchronous database replication from active to standby and a reviewed failover/failback runbook | Support-led architecture evaluation and environment-specific rehearsal for regional disaster recovery | A proven self-serve 2.0 contract, active/active multi-region, automatic or hands-free regional failover, synchronous cross-region replication (RPO=0), cross-region active visibility or federated search, or region-pinned task queues as an engine-enforced routing axis | Before relying on cross-region replication, failover, failback, RPO, or RTO in production |
| Support-led topologies | A reviewed design based on your environment | Self-hosted active/passive or active/active multi-region, hands-free regional failover, RPO=0 cross-writer replication, duplicate scheduler runners as a steady-state topology, bespoke security/networking, private SLOs, custom overlays, migration planning | Self-serve copy/paste operation | The topology itself is part of the product risk |

The public distribution is intentionally optimized for local development,
single-node production, and small clustered deployments. Kubernetes manifests
are provided for teams that already operate Kubernetes. Active/passive
multi-region material is available as a support-led evaluation guide (see
[Active/passive multi-region](#activepassive-multi-region) below), not as a
proven self-serve 2.0 contract.
Single-region HA failover — managed-database failover, managed-Redis failover,
API-node loss, worker loss, and scheduler-runner restart inside one region —
is currently support-led while its exact-release evidence gate remains closed
(see
[Single-region high availability and failover](#single-region-high-availability-and-failover)
below). For self-hosted server deployments, active/active multi-region,
automatic regional failover, duplicate scheduler runners as a steady-state
topology, and provider-specific managed-Kubernetes validation remain
support-led because they depend on your database, cache, networking, security,
runner, and upgrade choices. The published Helm chart is a self-serve packaging
path within those same operator-owned boundaries. Cloud currently provides
single-region managed operation and recovery, not multi-region replication or
regional failover; see
[Cloud Managed Runtime](/docs/polyglot/cloud-control-plane). See the
[support boundary](/docs/support) for the commercial support model.

## Production recovery deliverables

Calling a topology "production-ready" means publishing the recovery packet for
that topology, not just starting the containers. Keep the latest evidence in
the same runbook as the deployment commands:

| Path | Minimum published recovery packet |
| --- | --- |
| Single-node production | Backup schedule, pinned image or digest, env/config snapshot location, maximum accepted restore lag, and the latest successful restore rehearsal timestamp plus verification evidence. |
| Small clustered deployment | The single-node packet plus the expected impact of losing one API node, one worker node, the scheduler/maintenance runner, Redis, or the shared database; the worker re-registration steps after restore; and the documented rolling-upgrade or stop-the-world posture for the current release. |
| Helm chart for Kubernetes | The clustered packet plus chart version, values revision, image digest, cluster-specific ingress and secret owners, and the latest chart upgrade/rollback rehearsal. |
| Raw Kubernetes manifests | The clustered packet plus the cluster-specific storage, secret, ingress, and rollout owners that must be restored or re-applied before traffic is declared healthy again. |
| Single-region HA evaluation (support-led while the release-evidence gate is closed) | The clustered or raw-manifest packet plus rehearsal evidence for managed-database failover, managed-Redis failover, API-node loss, worker loss, and scheduler-runner restart, each completing within the recovery target published in the [Single-region HA contract](#single-region-high-availability-and-failover) without acknowledged-write loss. |
| Active/passive multi-region evaluation (support-led) | The per-region packet plus the database replication and fencing design, measured replication lag, operator-owned RPO/RTO, promotion and failback runbooks, and environment-specific rehearsal evidence. |

If you cannot produce that packet on demand, treat the environment as staging
until the recovery contract is written down and rehearsed. The
[Operator Operating Envelope](/docs/operator-operating-envelope) defines
the restore order, verification pass, and rehearsal cadence those packets must
follow.

## Security, Data, And Audit Posture

Self-hosted Durable Workflow deployments inherit most security controls from
the environment you operate. Publish these facts in the same release or runbook
packet as the image tag, migration plan, and recovery evidence:

| Posture area | Honest release statement |
| --- | --- |
| Data handling | Workflow arguments, results, history, memos, search attributes, visibility labels, command context, audit rows, exception messages, and operator notes can contain customer application data. [Workflow IDs](./defining-workflows/workflow-id.md#ids-are-visible-metadata), search attributes and labels are operator-visible metadata, not secret storage. Workflow IDs remain plaintext and are not covered by payload codecs. |
| Encryption | Use TLS for every production HTTP surface and configure [database TLS](./polyglot/server-config-reference.md#database-tls) on every Server role, including migrations. At-rest encryption comes from your database, object storage, filesystem, queue, cache, and secret manager. The workflow package and server do not automatically encrypt each payload field. |
| Compliance | The open-source package and self-hosted server provide controls and audit evidence, not a compliance certification by themselves. Claims such as SOC 2, HIPAA, PCI, ISO, or FedRAMP belong to your own program unless a hosted offering documents otherwise. |
| Audit logs | Workflow commands, schedule audit events, history export metadata, and service-call records provide durable operational evidence. They are not a complete SIEM, DLP system, immutable external ledger, or legal-hold system unless you add those components. |
| Support | Role-scoped credentials, TLS termination, backups, restore rehearsal, and narrow self-serve topologies are documented here. Advanced identity, mTLS rollout, private networking, custom policy engines, provider compliance, and bespoke topology review are support-led unless public docs say otherwise. |

Network posture must be explicit:

- **Webhook ingress:** document the public endpoint, auth method, replay or
  idempotency strategy, timeout, payload limit, trusted proxy-header
  configuration, and secret rotation plan.
- **Worker-to-backend traffic:** use TLS verification, role-scoped worker
  credentials, namespace headers, private networking or mTLS when workers cross
  an untrusted network, and rotation that does not grant operator capabilities
  to worker tokens.
- **Operator surfaces:** place Waterline, standalone-server operator APIs, CLI
  automation endpoints, and custom admin panels behind authenticated sessions
  or role-scoped service credentials, with CSRF protection for browser
  sessions and documented proxy/TLS boundaries.

## Published images

Use published images for self-hosted server deployments:

- Docker Hub: `durableworkflow/server:3.10.0`
- GitHub Container Registry: `ghcr.io/durable-workflow/server:3.10.0`
- Digest pinning: `durableworkflow/server@sha256:...` or
  `ghcr.io/durable-workflow/server@sha256:...`

Use mutable tags only for local experiments. Production env files should pin a
specific version tag or digest so upgrade and rollback steps are auditable.
Native releases are major version 3; the rolling `3` and `latest` tags move to
each new native release, and `2` stays on the last PHP 2.x release.

## Local development and internal non-production

Use the published-image Compose recipe when you want a source-free stack: the
native Server and MySQL. Fetch the file from the selected release and pin the
image with `DW_SERVER_IMAGE`:

```bash
curl -fsSLO https://raw.githubusercontent.com/durable-workflow/server/3.10.0/docker-compose.published.yml

export DW_SERVER_IMAGE=durableworkflow/server:3.10.0
export DW_AUTH_TOKEN=dev-token

docker compose -f docker-compose.published.yml up -d --wait
```

Compose refuses to start until `DW_AUTH_TOKEN` is set. The file publishes the
API on `127.0.0.1:8080` only and does not publish MySQL. Set
`SERVER_BIND_ADDRESS` to a protected private interface for remote access.

Verify the API, readiness, cluster discovery, and worker registration:

```bash
curl http://localhost:8080/api/health
curl http://localhost:8080/api/ready
curl -H "Authorization: Bearer $DW_AUTH_TOKEN" \
  http://localhost:8080/api/cluster/info

curl -X POST http://localhost:8080/api/worker/register \
  -H "Authorization: Bearer $DW_AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -H "X-Namespace: default" \
  -H "X-Durable-Workflow-Protocol-Version: 1.0" \
  -d '{"worker_id":"compose-worker","task_queue":"compose","runtime":"python"}'
```

This path is safe for development and internal staging. It is not a production
security boundary: the example uses one shared token, default service
passwords, local named volumes, and no TLS.

## Single-node production

Use the same Compose artifact with production configuration outside source
control:

```env
DW_SERVER_IMAGE=durableworkflow/server:3.10.0
SERVER_PORT=8080
# Loopback by default. Use a protected private interface for the reverse proxy.
SERVER_BIND_ADDRESS=127.0.0.1

DB_DATABASE=durable_workflow
DB_USERNAME=workflow
DB_PASSWORD=replace-with-random-password
DB_ROOT_PASSWORD=replace-with-random-root-password

DW_AUTH_DRIVER=token
# Required. Native accepts the shared token for every role; keep it secret.
DW_AUTH_TOKEN=replace-with-random-shared-token
DW_WORKER_TOKEN=replace-with-worker-token
DW_OPERATOR_TOKEN=replace-with-operator-token
DW_ADMIN_TOKEN=replace-with-admin-token
```

The native Server requires `DW_AUTH_TOKEN` and refuses to start with
`DW_AUTH_BACKWARD_COMPATIBLE=false`, so the shared token cannot be disabled.
Treat it like the admin token, and give workers and applications the narrower
role-scoped tokens.

Start with that env file:

```bash
docker compose --env-file durable-workflow.prod.env \
  -f docker-compose.published.yml up -d --wait
```

Operate the host as a production service:

- Put TLS, public routing, request logging, and IP allow lists in a reverse
  proxy in front of the API container.
- Do not expose MySQL publicly.
- Use `DW_WORKER_TOKEN` for workers, `DW_OPERATOR_TOKEN` for application and
  operator traffic, and `DW_ADMIN_TOKEN` for namespace and administrative work.
- Back up the MySQL volume before every image upgrade and on a regular
  schedule; it is the durable workflow-history source of truth. The
  `server_storage` volume holds the Server's local external payload objects:
  back it up with MySQL, because a recovery point needs both. See the
  [self-hosted backup and restore procedure](https://github.com/durable-workflow/server/blob/main/docs/self-hosted-backup-and-restore.md).
- Keep the exact env file, image tag or digest, and database backup together for
  restores.

Upgrade order:

1. Back up MySQL and `server_storage`, and record the current image reference.
2. Change only `DW_SERVER_IMAGE`.
3. Pull the new image with `docker compose --env-file durable-workflow.prod.env -f docker-compose.published.yml pull`.
4. Run `docker compose --env-file durable-workflow.prod.env -f docker-compose.published.yml up -d --wait`.
5. Confirm `/api/ready`, `/api/cluster/info`, and worker registration before
   shifting external traffic.

The native Server checks its schema at startup. Patch releases within a native
minor line keep the schema, so changing the image is enough. A release whose
notes announce a schema change also needs the image's `schema-upgrade` command,
run after a database backup while every Server is stopped. The Server reference keeps the latest
command-level Compose examples in the
[Official Image + Compose](https://github.com/durable-workflow/server/blob/main/docs/server-reference.md#official-image--compose)
section.

### Upgrading to Server 3.10.0 {#native-schema-upgrade-3-10}

Native 3.10.0 requires an explicit additive schema upgrade from 3.9.x:

| Backend | Schema transition from 3.9.x |
|---|---|
| SQLite | 13 → 14 |
| PostgreSQL | 14 → 15 |
| MariaDB | 14 → 15 |
| MySQL | 15 → 16 |

Stop every Server replica, keep the database running, and back up the database
and external-payload storage. Select the 3.10.0 image and run its
`schema-upgrade` once before restarting every replica on the same version.
Check readiness, reported version and worker registration. Startup does not
upgrade an old schema automatically. Older versions require the intervening
steps too. The [published 3.10.0 upgrade instructions](https://github.com/durable-workflow/server/releases/tag/3.10.0#upgrade-and-rollback)
provide the Compose commands. Worker protocol 1.20 and control plane 2 are unchanged.

The nullable schedule summary is not backfilled from legacy action arrays.
Existing schedules remain unavailable until an owning transition records a
valid summary; see [dispatch-summary authority](./polyglot/server-api-reference.md#schedule-dispatch-summary).
MySQL and MariaDB resume interrupted upgrades; SQLite and PostgreSQL apply the
step transactionally. Rollback to an older binary requires restoring the
pre-upgrade database and storage backup. Do not run an older binary against
the newer schema.

### Upgrading to Server 3.9.2 {#native-schema-upgrade-3-9}

Upgrading from native 3.8.x or earlier requires a schema upgrade on every
backend. Server 3.9.1 adds the empty `dw_command_intents` table:

| Backend | Schema transition from 3.8.x |
|---|---|
| SQLite | 12 → 13 |
| PostgreSQL | 13 → 14 |
| MariaDB | 13 → 14 |
| MySQL | 14 → 15 |

Stop every Server replica, keep the database running, back up the database and
Server storage, then select and pull the 3.9.2 image. Run that image's
`schema-upgrade` once before restarting every replica on the same version.
Server does not upgrade an existing schema at startup: it refuses an old
schema with `native_schema_upgrade_required`. Verify readiness, reported
version and worker registration after restart. The
[qualified 3.9.2 release instructions](https://github.com/durable-workflow/server/releases/tag/3.9.2#upgrade-and-rollback)
provide the source-free Compose commands.

MySQL and MariaDB upgrades resume after interruption; SQLite and PostgreSQL
apply the step transactionally. Rollback to 3.8.x requires restoring the backup:
3.8.0 refuses the upgraded database with `native_schema_newer_than_binary`.
From 3.9.0 or 3.9.1, no further schema step is needed: stop replicas, back up,
select the new image, restart and verify readiness, version and worker
registration. Physical schema and existing command/intent rows are unchanged.
Rollback to 3.9.1 uses the same schema but restores the Unicode-key/receipt
metadata defects and removes observation-owner hardening. From older versions,
`schema-upgrade` applies the earlier steps too, including the
[3.7 JSON rebuild](#native-schema-upgrade-3-7) where applicable. Worker protocol
1.20 and control plane 2 are unchanged.

### Upgrading to Server 3.8.0 {#native-schema-upgrade-3-8}

Server 3.8.0 needs a native schema upgrade on every backend. It adds one
native-only table, `dw_task_queue_pauses`, which is created empty:

| Backend    | Schema  |
|------------|---------|
| SQLite     | 11 → 12 |
| PostgreSQL | 12 → 13 |
| MariaDB    | 12 → 13 |
| MySQL      | 13 → 14 |

The step only creates the table; it rewrites no rows and takes seconds. A
Server never upgrades its schema at startup: a 3.8.0 Server started on the
earlier schema refuses with `native_schema_upgrade_required`. For the
single-node Compose deployment:

1. Stop the Server and leave MySQL running:

   ```bash
   docker compose --env-file durable-workflow.prod.env \
     -f docker-compose.published.yml stop server
   ```
2. Back up MySQL and `server_storage`, and record the current image
   reference. This backup is the only way back to 3.7.x.
3. Change `DW_SERVER_IMAGE` to the 3.8.0 image, pull it and run
   `schema-upgrade` once:

   ```bash
   docker compose --env-file durable-workflow.prod.env \
     -f docker-compose.published.yml pull server
   docker compose --env-file durable-workflow.prod.env \
     -f docker-compose.published.yml run --rm --no-deps server schema-upgrade
   ```

   From 3.7.x on MySQL it reports
   `"from_schema_version":13,"schema_version":14,"status":"upgraded"`. Run
   again on a current database, it reports `"status":"current"`.
4. Start the Server with
   `docker compose --env-file durable-workflow.prod.env -f docker-compose.published.yml up -d --wait`,
   then confirm `/api/ready`, `/api/cluster/info` (`version` is `3.8.0`) and
   worker registration.

With several Servers on one database, stop every one of them before step 3
and start them all on 3.8.0 afterwards. On MySQL and MariaDB an interrupted
upgrade resumes when you run `schema-upgrade` again; SQLite and PostgreSQL
upgrade in one transaction. Rolling back to 3.7.x needs the backup: a 3.7.0
Server refuses the upgraded database with `native_schema_newer_than_binary`.
The worker protocol stays at `1.20` and the control plane at version `2`.

From an earlier release, the 3.8.0 image's `schema-upgrade` applies every
earlier step in the same run. On MySQL from 3.5.x or 3.6.x (schema 12), that
includes the [3.7.0 JSON rebuild](#native-schema-upgrade-3-7), with its
downtime of about 2 minutes per million history rows. From 3.4.x or earlier,
it includes the [3.5.0 schema step](#native-schema-upgrade-3-5).

On MySQL, MariaDB and PostgreSQL, each pool connection now caches up to 200
prepared statements. On MySQL, every client's prepared statements count toward
the server-wide `max_prepared_stmt_count` (default 16382). A native node has 4
pool connections, so keep `nodes × 4 × 200` plus other clients' statements
within that limit; a default MySQL server fits about 20 native nodes. For more,
raise `max_prepared_stmt_count` or lower the cache with
`DW_NATIVE_DB_STATEMENT_CACHE_CAPACITY` (or `statement-cache-capacity` in
`DB_URL`), a whole number from 0 to 4095. A Server given an invalid value, or
two different values, refuses to start. `Prepared_stmt_count` in
`SHOW GLOBAL STATUS` reports current use.

Server 3.8.0 also lets an operator pause and resume activity dispatch on one
task queue, advertised as `capabilities.task_queue_activity_dispatch_pause`
(see the
[3.8.0 release](https://github.com/durable-workflow/server/releases/tag/3.8.0)).
Native redrive now reuses side effects and version markers recorded before the
failed step, with their original payloads and times.

### Upgrading to Server 3.7.0 {#native-schema-upgrade-3-7}

Server 3.7.0 stores MySQL JSON documents as exact text, so doubles keep every
digit. Only MySQL changes: its native schema moves from 12 to 13, and its 75
`json` columns in 36 tables become `LONGTEXT` (`utf8mb4_bin`) with an enforced
`json_valid` check, one `ALTER TABLE` per table. Documents already stored keep
MySQL's earlier rendering. A 3.7.0 Server started on MySQL schema 12 refuses
with `native_schema_upgrade_required`.

MariaDB and PostgreSQL stay at schema 12 and SQLite at 11, so a 3.5.0 or 3.6.0
database on them runs on 3.7.0 as it is: change only `DW_SERVER_IMAGE` and
follow the upgrade order above. Rolling back to 3.6.0 is an image change. From
3.4.x or earlier, every database also needs the
[3.5.0 schema step](#native-schema-upgrade-3-5); run it with the 3.7.0 image,
whose `schema-upgrade` applies the earlier steps too.

On MySQL the Server is down for the whole upgrade. Measured on MySQL 8.0 with
2 CPUs and the default buffer pool, it takes about 2 minutes per million
history rows. For the single-node Compose deployment:

1. Stop the Server and leave MySQL running:

   ```bash
   docker compose --env-file durable-workflow.prod.env \
     -f docker-compose.published.yml stop server
   ```
2. Back up MySQL and `server_storage`, and record the current image
   reference. This backup is the only way back to 3.6.0 on MySQL.
3. Change `DW_SERVER_IMAGE` to the 3.7.0 image, pull it and run
   `schema-upgrade` once:

   ```bash
   docker compose --env-file durable-workflow.prod.env \
     -f docker-compose.published.yml pull server
   docker compose --env-file durable-workflow.prod.env \
     -f docker-compose.published.yml run --rm --no-deps server schema-upgrade
   ```

   On MySQL it prints the schema versions before and after, for example
   `"from_schema_version":12,"schema_version":13,"status":"upgraded"`. Run
   again on a current database, it reports `"status":"current"`.
4. Start the Server with
   `docker compose --env-file durable-workflow.prod.env -f docker-compose.published.yml up -d --wait`,
   then confirm `/api/ready`, `/api/cluster/info` (`version` is `3.7.0`) and
   worker registration.

An interrupted upgrade resumes table by table when you run `schema-upgrade`
again. Rolling back to 3.6.0 on MySQL needs the backup: a 3.6.0 Server refuses
the upgraded database with `native_schema_newer_than_binary`. The worker
protocol stays at `1.20` and the control plane at version `2`.

Server 3.7.0 also changes native redrive (see the
[3.7.0 release](https://github.com/durable-workflow/server/releases/tag/3.7.0)):
memos and search attributes set at start carry to the successor, and
`describe` reports `actions.can_redrive` and `actions.redrive_blocked_reason`.

### Upgrading to Server 3.5.0 {#native-schema-upgrade-3-5}

Server 3.5.0 adds one native-only table, `dw_run_command_contracts`. The
native schema moves from 11 to 12 on MySQL, MariaDB and PostgreSQL, and from
10 to 11 on SQLite. The upgrade creates the table and its index and rewrites
no rows. Server 3.4.x has the same schema as 3.3.x, so this is the only schema
step from 3.3.x or 3.4.x. A 3.5.0 Server started on the earlier schema refuses
with `native_schema_upgrade_required`.

For the single-node Compose deployment:

1. Stop the Server and leave MySQL running:

   ```bash
   docker compose --env-file durable-workflow.prod.env \
     -f docker-compose.published.yml stop server
   ```
2. Back up MySQL and `server_storage`, and record the current image
   reference. This backup is the only way back to 3.4.x.
3. Change `DW_SERVER_IMAGE` to the 3.5.0 image, pull it and run
   `schema-upgrade` once:

   ```bash
   docker compose --env-file durable-workflow.prod.env \
     -f docker-compose.published.yml pull server
   docker compose --env-file durable-workflow.prod.env \
     -f docker-compose.published.yml run --rm --no-deps server schema-upgrade
   ```

   It prints the schema versions before and after, for example
   `"from_schema_version":11,"schema_version":12,"status":"upgraded"`. Run
   again on a current database, it reports `"status":"current"`.
4. Start the Server with
   `docker compose --env-file durable-workflow.prod.env -f docker-compose.published.yml up -d --wait`,
   then confirm `/api/ready`, `/api/cluster/info` (`version` is `3.5.0`) and
   worker registration.

On MySQL and MariaDB, an interrupted upgrade resumes when you run
`schema-upgrade` again. Rolling back to 3.4.x needs the backup: a 3.4.x Server
refuses the upgraded database with `native_schema_newer_than_binary`. From
3.0.x through 3.2.x, the same command also applies the earlier steps described
in the next section. The worker protocol stays at `1.20` and the control plane
at version `2`.

What changes for running workflows is in the
[3.5.0 release notes](https://github.com/durable-workflow/server/releases/tag/3.5.0):
updates for runs started before their worker registered now bind that worker's
contract instead of being rejected, sticky execution with history suffixes,
and expired runs pruned every minute by the maintenance loop.

### Upgrading from Server 3.0.x to 3.3.2 {#native-schema-upgrade-3-0-to-3-3}

These are the earlier schema steps. If you go from 3.0.x straight to 3.5.0,
use the 3.5.0 image in step 3; its `schema-upgrade` applies these steps and the
3.5.0 step together, and the notes on PostgreSQL index builds and MySQL
resumption below still apply.

Server 3.3.2 crosses three minor releases from 3.0.x and changes the native
schema. On MySQL, MariaDB and PostgreSQL it moves from version 8 (3.0.x and
3.1.0) through 10 (3.2.0) to 11 (3.3.0 to 3.3.2). On SQLite it moves from 8
to 10. A Server never upgrades its schema at startup: 3.3.2 started on a
version 8 database refuses with `native_schema_upgrade_required`, so changing
`DW_SERVER_IMAGE` alone leaves the new Server unable to start. The 3.3.2
image's `schema-upgrade` command upgrades a version 8 database directly. The
[3.3.0 upgrade notes](https://github.com/durable-workflow/server/blob/3.3.2/docs/releases/3.3.0.md#upgrade)
describe the step, and the Helm chart's
[Chart 1.1.0 / Server 3.3.2 notes](https://github.com/durable-workflow/server/blob/ac22629c078e8f1b240ead07b1561c99077753ac/k8s/helm/durable-workflow/docs/UPGRADING.md#chart-110--server-332-native)
list the schema steps and the runtime changes from 3.0.5 to 3.3.2.

For the single-node Compose deployment:

1. Stop the Server and leave MySQL running:

   ```bash
   docker compose --env-file durable-workflow.prod.env \
     -f docker-compose.published.yml stop server
   ```
2. Back up MySQL and `server_storage`, and record the current image
   reference. This backup is the only way back to 3.0.x.
3. Change `DW_SERVER_IMAGE` to the 3.3.2 image, pull it and run
   `schema-upgrade` once:

   ```bash
   docker compose --env-file durable-workflow.prod.env \
     -f docker-compose.published.yml pull server
   docker compose --env-file durable-workflow.prod.env \
     -f docker-compose.published.yml run --rm --no-deps server schema-upgrade
   ```

   It prints the backend and the schema versions before and after, for
   example `"from_schema_version":8,"schema_version":11,"status":"upgraded"`.
   On a database that is already current it only verifies the schema.
4. Start the Server with
   `docker compose --env-file durable-workflow.prod.env -f docker-compose.published.yml up -d --wait`,
   then confirm `/api/ready`, `/api/cluster/info` (`version` is `3.3.2`) and
   worker registration.

- **PostgreSQL:** the version 10 and 11 steps build their indexes with a plain
  `CREATE INDEX`, which blocks writes to `workflow_tasks` until the build
  finishes. On a large `workflow_tasks` table, plan the downtime for it.
- **MySQL and MariaDB:** DDL commits on its own there, so the upgrade marks the
  database as upgrading until it finishes, and every Server refuses it
  meanwhile with `native_schema_upgrade_incomplete`. If an upgrade is
  interrupted, run `schema-upgrade` again to resume it.
- **Rollback needs the backup.** There is no schema downgrade. A 3.0.x Server
  refuses an upgraded database with `native_schema_newer_than_binary`. To go
  back, restore the backup taken in step 2, which discards everything written
  after it, and then start the 3.0.x image.

The worker protocol stays at `1.20` and the control plane at version `2`.

## Small clustered deployments

:::note PHP Server 2.x contract

The validated small-cluster contract and its CI harness use the PHP 2.x
Server: shared Redis, a queue worker and exactly one scheduler. Native 3.x has
none of those roles, and no native Compose small-cluster recipe or harness has
been published. For more than one native Server replica, use the
[Helm chart's native mode](#helm-chart-for-kubernetes) with an external
MySQL or PostgreSQL database and a ReadWriteMany volume for payload storage.

:::

A small cluster is a modest extension of the single-node model. The validated
self-serve contract is intentionally narrow:

- Run 2-3 stateless API containers behind a load balancer. Health,
  readiness, cluster discovery, worker registration, workflow-task polling, and
  workflow-task completion must work without sticky sessions.
- Use one shared external MySQL or PostgreSQL database for durable history.
  SQLite is single-node only and is not a clustered persistence backend.
- Use shared Redis for cache, long-poll wake signals, query-task queue locks,
  task-queue admission locks, and queue state. Redis-less multi-node mode is
  not a supported clustered contract.
- Check `GET /api/cluster/info` on each API node during rollout.
  `topology.current_shape` should remain `standalone_server`,
  `topology.current_roles` should include `api_ingress`, `control_plane`,
  `matching`, and `history_projection`, and `topology.execution_mode` should
  remain `remote_worker_protocol` for standalone server nodes. Use
  `topology.matching_role` to confirm the matching path you actually deployed:
  default nodes report `queue_wake_enabled: true`, `shape: "in_worker"`, and
  `wake_owner: "worker_loop"`, while dedicated matching rollouts flip nodes
  with `DW_V2_MATCHING_ROLE_QUEUE_WAKE=0` to `queue_wake_enabled: false`,
  `shape: "dedicated"`, and `wake_owner: "dedicated_repair_pass"`. The same
  block should continue to advertise the intended `task_dispatch_mode`, the
  frozen routing axes in `partition_primitives`, and the current
  `backpressure_model`. See
  [Server Role Topology](/docs/polyglot/server-role-topology) for the
  field-by-field meaning of the topology manifest.
- Scale external SDK workers independently from API nodes. Workers can run on
  separate hosts or processes, but they should talk to the load-balanced API
  endpoint rather than to one sticky node.
- Configure [task queue admission](/docs/polyglot/task-queue-admission)
  for queues that protect a tenant, external API, database pool, or other
  shared downstream dependency.
- Run exactly one scheduler or maintenance process for schedule evaluation,
  activity-timeout enforcement, and history pruning.
- Run bootstrap/migrations once per rollout before new API and worker
  containers accept traffic.
- Choose a rollout posture for this release: stop-the-world (drain workers,
  stop scheduler/maintenance, replace API nodes, run bootstrap/migrations,
  then restart workers and the scheduler) or
  [rolling upgrades](/docs/rolling-upgrades) when every guarantee on
  that contract holds.
- Treat the database and Redis as the primary failure domains. The server
  containers are replaceable; the persistence and coordination layers are not.

The published self-serve recipes start in the `standalone_server` shape even
though cluster discovery also names `split_control_execution` as a supported
product topology. Treat that as one contract with different role assignments,
not as a second server product. If you pilot a more explicit role split later,
keep reading `topology.current_shape`, `topology.current_roles`, and
`topology.matching_role` from `/api/cluster/info` instead of inferring duties
from hostnames or container names. The
[Server Role Topology](/docs/polyglot/server-role-topology) page explains
those role assignments and the migration path in one place.

Every API node should use the same auth tokens or signature keys, app version,
workflow package version, payload-codec configuration, database connection, and
Redis connection. Give each API node a unique `DW_SERVER_ID` so cluster
discovery and logs can distinguish the nodes.

The unsupported boundaries are explicit: SQLite clustering, Redis-less
multi-node mode, duplicate schedulers as a steady-state topology, active/active
multi-writer databases, self-hosted hands-free regional failover,
and broad "five-nines" or "zero-downtime" SLA promises need separate
validation or support-led design before you rely on them. Active/passive
multi-region guidance is also support-led evaluation material; see
[Active/passive multi-region](#activepassive-multi-region) below.
Single-region HA failover — managed-database failover, managed-Redis failover,
API-node loss, worker loss, and scheduler-runner restart inside one region —
is a support-led evaluation contract until the release-evidence gate passes;
see
[Single-region high availability and failover](#single-region-high-availability-and-failover)
below for the engine recovery targets, the readiness rules during a failover,
and the evidence required to open that gate.

This path is self-serve when your team already has a clear VM, network,
database, cache, backup, and load-balancer model. It becomes support-led when
you need help deciding those boundaries, capacity, rollout order, or recovery
procedures.

## Worker long-poll timeout budgets {#worker-long-poll-timeout-budgets}

A healthy worker poll can be silent until its server wait ends. Budget for that silence across the worker's HTTP client, outbound proxy, service mesh, public load balancer, ingress and any reverse proxy before the API. Worker polling uses the shared API ingress. A separate matching role does not bypass those hops.

| Budget | What it limits |
| --- | --- |
| Server poll duration | How long the runtime waits for work before returning an empty response. A request's `timeout_seconds` overrides `DW_WORKER_POLL_TIMEOUT`; the variable is the fallback when the request omits it, not a cap on SDK polls. The protocol permits up to 60 seconds. |
| Client connect / pool timeout | Establishing a connection or acquiring one from the client pool. These need not last for the whole quiet poll. |
| Client read / total timeout | Waiting for response bytes, or the complete operation if the transport has a total limit. Allow the server wait plus connection, scheduling and response margin. A read timeout alone may measure inactivity between chunks, not the whole request. |
| Intermediary idle / upstream read / request timeout | Every active-request limit on every hop must permit a legitimate quiet poll and its response. An idle connection keepalive setting between requests is a different limit. |
| Shutdown / caller deadline | The worker's shutdown budget and any original caller or query authority remain hard bounds. Cancel or drain within those bounds; do not extend a query deadline to accommodate a proxy or a retry. |

Check the installed SDK's arithmetic rather than copying a timeout from another engine. In the verified Python SDK baseline below, a positive poll requests at most 60 seconds on the Server and gets at least five additional seconds of HTTP phase timeout. Its default Worker poll is 35 seconds, which normally selects 40 seconds for those HTTP phases. A 30-second intermediary read limit can therefore interrupt a legitimate default poll. PHP and Rust have their own defaults and retry policies; neither Python's margin nor a gRPC keepalive setting is a portable rule for all SDKs. See the [PHP](https://php.durable-workflow.com/), [Python](https://python.durable-workflow.com/) and [Rust](https://rust.durable-workflow.com/) SDK guides and [task matching reference](./polyglot/task-matching-dispatch.md).

PHP SDK **3.2.0** ordinary control-plane `Client` calls do not supply a default total HTTP deadline; see the [verified timeout gap in PHP #203](https://github.com/durable-workflow/sdk-php/issues/203). Configure a bounded HTTP client for those calls and preserve the original caller deadline. A Worker poll timeout is not a general bound on every SDK request.

PHP SDK **3.4.0 or newer** adds a **60-second default total deadline** for SDK-owned ordinary control HTTP requests; configure a finite `requestTimeoutSeconds` value of at least `0.001` seconds. These SDK-owned timeouts carry `reason=request_timeout` and `status=0`. The SDK makes one attempt: a timed-out mutation may already have been accepted, so inspect its original identity/outcome before resubmitting. Injected clients/transports keep their own timeout policy and cannot be combined with this option. This per-request bound does not replace worker long-poll, payload-transfer or workflow/caller deadlines. Check `composer.lock`; this feature minimum does not update the selected installation tuple. See [HTTP client setup](https://php.durable-workflow.com/getting-started/client-setup/) and the [normal installed 3.4.0 receipt](https://github.com/durable-workflow/sdk-php/issues/203#issuecomment-6363909756).

### Verified short-poll example {#verified-poll-timeout-example}

An isolated test used published Python SDK **2.9.0**, PHP Server **2.11.0** and Nginx **1.30.5**, with an empty task queue. Both cells kept the SDK poll duration at **3 seconds**, HTTPX connect/read/write/pool timeouts at **8 seconds**, and an outer caller bound of **10 seconds**. Only Nginx's upstream read timeout changed:

| Intermediary setting | Observed response |
| --- | --- |
| `proxy_read_timeout 1s` | Nginx returned HTTP **504** after about one second while waiting for the upstream response header. The SDK surfaced `ServerError`. |
| `proxy_read_timeout 8s` | The same legitimate quiet poll completed after about three seconds with HTTP **200**, `task: null` and `poll_status: empty`. |

The Server fallback was deliberately one second in both cells; the request's explicit three-second wait still applied. HTTP retries were limited to one attempt to expose the first failure. This is a functional timeout example, not a capacity or production sizing result. The [qualification in #267](https://github.com/durable-workflow/durable-workflow.github.io/issues/267) records the exact images, requests and proxy logs.

For Nginx deployments, apply an appropriate upstream read budget to the existing worker-poll location. The eight-second value below belongs to this three-second example:

```nginx
proxy_connect_timeout 2s;
proxy_read_timeout 8s;
```

[Nginx measures `proxy_read_timeout` between upstream reads](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout). [HTTPX also distinguishes connect, read, write and pool timeouts](https://www.python-httpx.org/advanced/timeouts/); the eight-second phase settings above are not an eight-second total deadline. Keep an explicit outer deadline where required. Other proxies and providers use different names and semantics. Verify their active-request limits and the complete path with the actual SDK. Increasing one ingress limit does not fix a shorter limit elsewhere.

### Diagnose the response before restarting workers {#diagnose-poll-disconnects}

- **Normal empty poll:** the runtime returns a successful response with no task. In the verified baseline, the raw response is `poll_status: empty`. Poll again normally.
- **Intermediary disconnect:** a transport exception, connection reset, or proxy-generated 502/504 may occur before the legitimate server wait finishes. Correlate the SDK error and elapsed time with proxy/upstream logs. A healthy readiness check and later successful polling can coexist with a misconfigured intermediary.
- **Runtime unavailable:** inspect the runtime's structured reason and readiness checks for database, cache or admission failures. A gateway's HTML error page is not the runtime's typed refusal.
- **Namespace or poll admission refusal:** inspect the structured reason, namespace/queue and any retry delay, including HTTP 429. Raising an idle timeout does not grant admission capacity or fix authorization.

A worker retrying its connection does not prove the original proxy configuration was healthy. Check whether polling and task completion actually continue, and retain the failure class, queue, time and frequency when failures persist. A sporadic disconnect alone is not a reason to restart the process. The Python client can also synthesize `poll_status: timeout` after its HTTP timeout retries; that value alone is not proof that a runtime empty response traversed the proxy. Keep transport retries separate from activity retries, run deadlines and query authority.

## SQL connection budgets {#sql-connection-budgets}

Budget database sessions before adding API replicas or overlapping old and new
processes during a [rolling upgrade](/docs/rolling-upgrades). A database can
reach its connection limit while CPU is still available. This is separate from
write/commit latency, HTTP request capacity and retained storage growth.

### Count every SQL-owning process {#count-every-sql-owning-process}

A native 3.x Server process opens at most four database connections, and a
`schema-upgrade` or takeover command opens one more while it runs. Budget four
sessions per native Server replica at the highest replica count, plus the
operational tools below. The rest of this section describes PHP 2.x.

The published PHP Compose deployment has these possible SQL clients:

| Role | What to count |
| --- | --- |
| API | Every concurrently active PHP request process that can use SQL, including readiness, operator requests and held polls. The published Apache prefork configuration permits 24 request processes per API container. That is an HTTP process ceiling, not a SQL pool or a measurement of 24 open sessions. |
| Laravel queue workers | Each `php artisan queue:work` process. The published Compose recipe runs one; extra worker containers add processes. These are separate from external SDK workers. |
| Scheduler/maintenance | Concurrent PHP commands, not just the scheduler container. The Compose scheduler runs schedule evaluation, repair, activity timeout and pruning commands sequentially, so its ordinary loop has one command active at a time. Keep the singleton scheduler rule. |
| Bootstrap and operational tools | Migration/bootstrap commands, projection rebuilds, retention jobs, backups, monitoring and interactive SQL clients that can overlap the deployment. Count any deliberately concurrent commands separately. |
| SDK clients and workers | Normally zero direct Server-database sessions because PHP, Python and Rust service-mode SDKs use HTTP. Add connections only if your activity code or other application code actually connects to that database. |

Count **distinct SQL handles used by each process**, not just containers or
hostnames. Laravel's default connection and `workflows.storage.connection`
may share one configured connection name. A different storage name, database
queue/cache/session connection, or read/write/direct connection handle can
open an additional session even when it reaches the same database endpoint.
Redis connections are not SQL sessions. Inspect the effective cached
configuration and the connection names each role actually uses.

For a custom FPM deployment, count PHP-FPM execution children, not nginx or
Apache event threads. For a custom Octane deployment, inventory its request
workers, task workers and actual database clients. Do not reuse Apache's
24-process figure as a pool size for either frontend. Synchronous long polls
occupy a PHP execution worker while waiting and may retain a database session;
an idle HTTP poll must not automatically be budgeted as zero SQL usage.

### A conservative rolling-upgrade example {#a-conservative-rolling-upgrade-example}

This is a planning example, **not measured capacity**. Assume a database limit
of 120 client sessions, ten sessions for other applications and fifteen
reserved for recovery/admin access. That leaves 95 for this deployment.
Assume Redis-backed queue/cache and at most one SQL handle per PHP process.

| Peak overlap | Calculation | Planned sessions |
| --- | --- | ---: |
| API replicas, including one rollout surge | `(2 + 1) × 24 × 1` | 72 |
| Laravel queue workers | `2 × 1` | 2 |
| Singleton scheduler/maintenance command | `1 × 1` | 1 |
| Bootstrap/migration | `1 × 1` | 1 |
| Backup, monitoring and other approved tools | Explicit allowance | 4 |
| **Deployment subtotal** | | **80** |
| Unallocated deployment margin | `95 − 80` | 15 |

At these assumptions, a fourth API replica would raise the deployment subtotal
to 104 and exceed its 95-session allocation. Two SQL handles per API process
would already require 144 sessions for the three API replicas alone. Recompute
the budget when changing storage connections, worker counts or rollout surge.
Lower the process/replica ceiling or stage replacement rather than assuming
the database will absorb the overlap. Raising `max_connections` also needs
database memory and workload review.

### Observe and enforce the budget {#observe-and-enforce-the-budget}

1. Record the effective database connection limit, unrelated clients and
   reserved recovery access. Set explicit API replica/process maxima and
   rollout surge. Include overlapping queue-worker generations and tools.
2. On an isolated copy using your exact published image and configuration,
   sample sessions while idle, during held polls, while completing real work
   and during a bounded old/new overlap. Include idle sessions and the monitor
   itself. PostgreSQL exposes `pg_stat_activity`; MySQL exposes
   `information_schema.PROCESSLIST`. Use a suitably scoped monitoring role and
   avoid exporting SQL text or payloads.
3. Compare the observed sessions by application user/database with the
   declared per-process allowance. A short sample validates that fixture's
   assumptions, not a saturation limit or a guarantee for every workload.
4. Watch session headroom together with connection refusals, transaction/commit
   latency, lock waits, held HTTP requests and storage growth. Readiness alone
   does not prove a rollout has spare capacity.
5. Near the limit, reduce the surge or drain and retire an old API/worker group
   before starting its replacement. Keep recovery access available and run
   bootstrap only in the planned migration window. If safe overlap cannot fit,
   use the documented stop-the-world upgrade posture.

On PHP 2.x, [task-queue admission](/docs/polyglot/task-queue-admission) limits
selected dispatch and lease activity, and `DW_WORKER_LONG_POLL_MAX_CONCURRENT`
and `DW_QUERY_TASK_POLL_MAX_CONCURRENT` bound held poll waits on a node. None is
a global SQL-session, database-byte or commit-latency limit; keep capacity for
completions, heartbeats, operator requests and health checks as well. Native
3.x refuses to start with task-queue admission caps and ignores both poll caps;
it has no native equivalent yet. Its fixed pool of four connections per process
bounds its SQL sessions; size SDK worker slots and worker counts for task
concurrency.

Use a direct writable database endpoint as the baseline. A connection proxy must be qualified with the exact Server release, pool mode and settings for prepared statements, transaction boundaries, session state and locking. **PgBouncer transaction pooling requires fixed Server 2.9.8 or later.** The [published 2.9.8 acceptance](https://github.com/durable-workflow/server/issues/524#issuecomment-6293185092) qualifies PostgreSQL 16.15 with PgBouncer 1.26.1: direct PostgreSQL, session pooling, and transaction pooling with either `max_prepared_statements=0` or `max_prepared_statements=200`. This is a compatibility floor and an exact tested configuration, not a latest-version pin or capacity guarantee; qualify other combinations separately. Before admitting traffic, run PHP 2.x's ordinary `server-bootstrap` initialization against the selected endpoint, or let native 3.x check its schema at startup through that endpoint.

`GET /api/ready` exercises a parameter-bound database read (`bound_read_query`) and configured readiness checks; it explicitly leaves write capacity unverified. A successful TCP connection, migration or readiness response does not prove every workflow SQL operation or commit capacity. Read replicas must not replace the authoritative writer used for claims and durable transitions.

## Helm chart for Kubernetes

The Server repository owns the
[chart source and release guidance](https://github.com/durable-workflow/server/tree/main/k8s/helm/durable-workflow).
Install the published chart from the OCI registry:

```bash
helm install durable-workflow \
  oci://ghcr.io/durable-workflow/charts/durable-workflow \
  --version 2.0.0 \
  --namespace durable-workflow --create-namespace \
  -f my-values.yaml
```

Chart 1.x runs the native Server by default (`runtime: native`): one Server
Deployment, with no worker, scheduler, migration Job or Redis. Your values file
must point to an external MySQL, MariaDB or PostgreSQL database, provide
`DW_AUTH_TOKEN` and the role-scoped tokens through existing Kubernetes Secrets,
and configure ingress/TLS for your environment. The chart refuses to render
without the shared `DW_AUTH_TOKEN`, and it refuses SQLite. A native install runs
one replica by default; more than one replica, or autoscaling, requires
`native.storage.existingClaim` to name a ReadWriteMany volume for external
payload objects. The chart does not create persistence or provider-managed
infrastructure. Pin the Server image by digest for a reproducible deployment.

An existing PHP 2.x release does not move to native on an ordinary upgrade:
the chart refuses until you set `runtime` explicitly. Set `runtime: php` to keep
the PHP 2.x worker, scheduler, bootstrap job and Redis. To move to
`runtime: native`, follow the
[native takeover runbook](https://github.com/durable-workflow/server/blob/a051f57fcc7e3605b301581ee9105461b3ef5815/docs/rust-server-upgrade-takeover.md)
and keep `runtime: php` until the takeover is done. See the chart's
[UPGRADING notes](https://github.com/durable-workflow/server/blob/ac22629c078e8f1b240ead07b1561c99077753ac/k8s/helm/durable-workflow/docs/UPGRADING.md).

Chart 1.1.0 changes the default image from Server 3.0.5 to 3.3.2, which needs
the [native schema upgrade](#native-schema-upgrade-3-0-to-3-3) from 8 to 11.
An ordinary `helm upgrade` from chart 1.0.x changes only the image, and the new
pods refuse to start with `native_schema_upgrade_required`. Follow the chart's
[Chart 1.1.0 / Server 3.3.2 steps](https://github.com/durable-workflow/server/blob/ac22629c078e8f1b240ead07b1561c99077753ac/k8s/helm/durable-workflow/docs/UPGRADING.md#chart-110--server-332-native):
scale the Server to zero with autoscaling off, back up the database, then
upgrade to chart 1.1.0 with `native.schemaUpgrade.enabled=true`. Its
`pre-upgrade` hook runs `schema-upgrade` once before the new pods start, and
only when Helm's `lookup` shows every Server stopped. Argo CD, Flux and
`helm template` renders refuse the hook; use the chart README's manual Job
instead. A `helm rollback` to a chart 1.0.x revision does not bring the Server
back, because 3.0.x refuses the upgraded database: restore the backup instead.

Chart 1.2.0 moves the default image to Server 3.4.0 with no schema change, so
a plain `helm upgrade` from chart 1.1.0 is enough. Native pods also gain a
`preStop` sleep (`native.preStopSleepSeconds`, default 5) and a 70-second
termination grace period. Chart 1.3.0 moves the default image to Server 3.5.0,
which needs the [schema upgrade from 11 to 12](#native-schema-upgrade-3-5):
scale the Server to zero, back up the database, and run `helm upgrade` with
`native.schemaUpgrade.enabled=true` (or the manual Job), then scale back up.
See the chart's
[Chart 1.3.0 / Server 3.5.0 notes](https://github.com/durable-workflow/server/blob/ac22629c078e8f1b240ead07b1561c99077753ac/k8s/helm/durable-workflow/docs/UPGRADING.md#chart-130--server-350-native).
Going back to chart 1.2.0 means restoring that backup.
Chart 1.4.0 moves the default image to Server 3.6.0 with no schema change, so
a plain `helm upgrade` from chart 1.3.0 is enough, and going back to 1.3.0 is an
image change. See the chart's
[Chart 1.4.0 / Server 3.6.0 notes](https://github.com/durable-workflow/server/blob/ac22629c078e8f1b240ead07b1561c99077753ac/k8s/helm/durable-workflow/docs/UPGRADING.md#chart-140--server-360-native).
Chart 1.5.0 moves the default image to Server 3.7.0. On MySQL it needs the
[schema upgrade from 12 to 13](#native-schema-upgrade-3-7): scale the Server
to zero, back up the database, and run `helm upgrade` with
`native.schemaUpgrade.enabled=true` (or the manual Job), then scale back up.
Going back to chart 1.4.0 on MySQL means restoring that backup. On MariaDB and
PostgreSQL a plain `helm upgrade` is enough, and going back is an image change.
See the chart's
[Chart 1.5.0 / Server 3.7.0 notes](https://github.com/durable-workflow/server/blob/ac22629c078e8f1b240ead07b1561c99077753ac/k8s/helm/durable-workflow/docs/UPGRADING.md#chart-150--server-370-native).
Chart 1.6.0 moves the default image to Server 3.8.0, which needs the
[schema upgrade](#native-schema-upgrade-3-8) on every backend: scale the
Server to zero, back up the database, and run `helm upgrade` with
`native.schemaUpgrade.enabled=true` (or the manual Job), then scale back up.
Going back to chart 1.5.0 means restoring that backup. The chart pins the
image by tag. For change-controlled installs, also set `image.digest` to the
3.8.0 multi-arch index,
`sha256:7988a23525d9e50e997a9f6f02e801300ed4f626de3cf6977033657ae7e9f58e`,
which both `durableworkflow/server:3.8.0` and
`ghcr.io/durable-workflow/server:3.8.0` resolve to. See the chart's
[Chart 1.6.0 / Server 3.8.0 notes](https://github.com/durable-workflow/server/blob/ac22629c078e8f1b240ead07b1561c99077753ac/k8s/helm/durable-workflow/docs/UPGRADING.md#chart-160--server-380-native).

### Choosing a chart release {#helm-chart-selection}

One chart release records the Server image it was tested against in `Chart.yaml`'s `appVersion`; the chart and Server have independent version numbers. The selected [compatibility tuple](./compatibility.md#last-qualified-reproducible-tuple) pairs chart **2.0.0** with Server **3.10.0**, from the same release metadata as the table and install command. Its [published OCI/package receipt](https://github.com/durable-workflow/server/actions/runs/43487220214) verifies that pairing, not a fresh Kubernetes installation.

For another Server version, inspect the candidate published chart's `Chart.yaml` (the [selected chart source](https://github.com/durable-workflow/server/blob/ac22629c078e8f1b240ead07b1561c99077753ac/k8s/helm/durable-workflow/Chart.yaml) is an example) and select an exact chart version whose `appVersion` matches your intended Server. Read the selected release's [UPGRADING notes](https://github.com/durable-workflow/server/blob/ac22629c078e8f1b240ead07b1561c99077753ac/k8s/helm/durable-workflow/docs/UPGRADING.md), including every chart version crossed. Do not infer a chart version from the Server version or select an unpublished chart from current source.

Overriding `image.tag` changes the image without changing the chart's recorded `appVersion`. That produces a custom chart/image tuple until it receives its own rendered-manifest, bootstrap, rollout and runtime qualification. Keep ordinary database backups and rollback boundaries in the upgrade plan; a successful render alone does not prove the deployment works.

The published history includes an exception: chart 0.2.2 crossed a Server minor under a chart patch number. The later [published correction](https://github.com/durable-workflow/server/blob/2.11.0/k8s/helm/durable-workflow/docs/UPGRADING.md#chart-022--server-210) acknowledges that exception and resumes the stated minor-version policy. Historical versions stay unchanged; this note does not select the later Server/chart tuple.

## Kubernetes manifests

The server repository includes raw manifests under
[`k8s/`](https://github.com/durable-workflow/server/tree/main/k8s) for teams
that already operate Kubernetes. They describe the PHP 2.x shape; for the
native Server, use the Helm chart's native mode or adapt its single Server
Deployment:

- Namespace and shared labels
- ConfigMap and Secret split
- Bootstrap/migration Job
- API Deployment and Service
- Worker Deployment
- Scheduler CronJob
- PodDisruptionBudget
- `/api/health` liveness and `/api/ready` readiness probes
- Conservative resource requests and limits

Before applying the manifests, replace the image tag with a specific published
version or digest, provide real database and Redis credentials, and wire the
ConfigMap values to the services your cluster already operates. The manifests
are intentionally raw and inspectable; they are not a Helm chart and do not
promise generic managed-Kubernetes behavior.

For a Kubernetes production rollout, prove at minimum:

```bash
kubectl -n durable-workflow wait --for=condition=complete job/durable-workflow-migrate --timeout=180s
kubectl -n durable-workflow rollout status deploy/durable-workflow-server
kubectl -n durable-workflow rollout status deploy/durable-workflow-worker
kubectl -n durable-workflow port-forward svc/durable-workflow-server 8080:8080
curl http://localhost:8080/api/ready
curl -H "Authorization: Bearer $DW_ADMIN_TOKEN" http://localhost:8080/api/cluster/info
```

Single-region HA failover — managed-database failover, managed-Redis failover,
API-node loss, worker loss, and scheduler-runner restart inside one region —
remains support-led on the raw-manifest path just as it does on the small-cluster
path while the release-evidence gate is closed. The intended contract requires
the readiness, singleton-scheduler, and shared-substrate rules in
[Single-region high availability and failover](#single-region-high-availability-and-failover)
below. Provider-specific load balancers, storage classes, network policies,
active/active multi-region, and self-hosted hands-free regional failover remain
support-led or tracked separately from the raw-manifest contract. Use the
separately versioned Helm path above when you want Helm-managed installation
and upgrades.

## Running SDK workers on Kubernetes {#sdk-workers-on-kubernetes}

Deploy your application workers separately from the Server and, on PHP 2.x, from its Laravel queue worker. Build an image containing your registered PHP, Python or Rust handlers and its locked SDK dependencies. Use an exec-form entrypoint so the worker process receives `SIGTERM`. Replace the image below with that build's immutable digest.

For workers in the same cluster, use the Server Service's actual name, namespace and port. The raw manifests expose `http://durable-workflow-server.durable-workflow.svc.cluster.local:8080`. A Helm release can use a different Service name. Workers outside the cluster need the reachable ingress or load balancer's HTTPS URL and normal certificate verification. Keep long-poll and transport budgets aligned with [the timeout guidance](#worker-long-poll-timeout-budgets).

Create a Secret in the worker Pod's Kubernetes namespace from your existing worker credential. Keep the token out of the manifest and image. The Server namespace is a separate product boundary from the Kubernetes namespace.

```bash
kubectl -n durable-workflow create secret generic orders-worker-auth --from-literal=token="$DW_WORKER_TOKEN"
```

Save this as `orders-worker.yaml`. These environment names are application configuration: your entrypoint must pass them to the SDK client and Worker. Give each live process its own worker ID and register the handlers for the configured queue.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-worker
  namespace: durable-workflow
spec:
  replicas: 2
  selector:
    matchLabels:
      app: orders-worker
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    metadata:
      labels:
        app: orders-worker
    spec:
      automountServiceAccountToken: false
      terminationGracePeriodSeconds: 75
      containers:
        - name: worker
          image: registry.example.com/acme/orders-worker:release-id
          env:
            - name: DW_SERVER_URL
              value: http://durable-workflow-server.durable-workflow.svc.cluster.local:8080
            - name: DW_NAMESPACE
              value: default
            - name: DW_TASK_QUEUE
              value: orders
            - name: DW_WORKER_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.uid
            - name: DW_WORKER_TOKEN
              valueFrom:
                secretKeyRef:
                  name: orders-worker-auth
                  key: token
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: "1"
              memory: 512Mi
```

Apply the Deployment and wait for its rollout:

```bash
kubectl apply -f orders-worker.yaml
kubectl -n durable-workflow rollout status deployment/orders-worker
```

Replica and resource values are starting examples, not capacity guarantees. Tune them with actual task duration, memory use, downstream limits and [worker admission budgets](./polyglot/task-queue-admission.md). A Deployment becoming available does not prove SDK registration or useful work. Verify fresh worker registrations and the expected build/queue, then start and complete a representative workflow. An SDK worker does not acquire an HTTP health endpoint merely by running in Kubernetes. Use an application-owned probe when you have one, and do not use Server health as a worker liveness check.

### Delivering SIGTERM and waiting for cleanup {#sdk-worker-pod-shutdown}

| SDK | Worker process shutdown |
| --- | --- |
| PHP | Use the managed `Worker::run()` loop with `pcntl` installed. Its signal handler requests shutdown, then the active synchronous task reaches a boundary before managed cleanup and deregistration. A custom loop around `tick()` does not supply that lifecycle. See [PHP graceful shutdown](https://php.durable-workflow.com/operate/graceful-shutdown/). |
| Python | Wire `SIGTERM` to the same Worker's shutdown path and await both `Worker.stop()` and its run task. Set an explicit `shutdown_timeout`. Accepted work shares that drain deadline. A running synchronous thread cannot be forcibly stopped by cancelling its await, and an unconfirmed stop remains an error. Use the [separate-process example and signal wiring](https://python.durable-workflow.com/guides/separate-workers/) and [lifecycle contract](https://python.durable-workflow.com/reference/worker/). |
| Rust | Pass a future that resolves on Unix `SIGTERM` to `Worker::run_until(...)` and await its result. A Ctrl-C-only future does not handle the Pod's termination signal. Keep asynchronous handlers yielding and bound blocking work separately. See [Worker::run_until](https://rust.durable-workflow.com/durable_workflow/struct.Worker.html#method.run_until). |

Give `terminationGracePeriodSeconds` enough time for the installed SDK's poll, accepted-task shutdown and HTTP/session/deregistration cleanup. The example's 75 seconds must be checked against your application's actual bounds. Kubernetes counts any `preStop` hook within that same grace period and force-stops remaining processes after expiry. A sleep hook alone neither asks the SDK to drain nor extends its deadline. See the [Kubernetes Pod termination flow](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination-flow).

Rehearse termination while idle and during bounded work. Treat cleanup errors, a retained registration or a forced kill as incomplete shutdown. A replacement worker continues durable work according to its recorded leases, retries and deadlines. Stopping a Pod does not cancel a workflow or undo an external side effect. Before changing builds, follow [worker build-ID rollout](./polyglot/worker-build-id-rollout.md) and retain its compatibility, replay and rollback checks.

## Single-region high availability and failover

Single-region HA failover is currently an **unverified, support-led** contract
candidate layered on the small-cluster shape and the raw Kubernetes shape. It
targets the failure modes that the engine is designed to survive **inside one
region**: managed-database failover (RDS
Multi-AZ, Aurora cluster failover, Cloud SQL HA, Patroni promotion, etc.),
managed-Redis failover (Sentinel, Elasticache replication-group failover,
Memorystore HA, etc.), API-node loss, worker loss, and
scheduler/maintenance runner restart. Cross-region active/passive recovery is
a different, support-led evaluation; see
[Active/passive multi-region](#activepassive-multi-region) below.

The intended contract — engine behavior, readiness rules, the per-event
recovery targets, the split-brain prevention rules, and the rehearsal
acceptance test — lives in the workflow library at
[`docs/deployment/ha-failover.md`](https://github.com/durable-workflow/workflow/blob/main/docs/deployment/ha-failover.md)
and the standalone server at
[`docs/ha-failover-validation.md`](https://github.com/durable-workflow/server/blob/main/docs/ha-failover-validation.md).
This section is the public surface of those documents.

### Release-evidence status

No passing exact-release result is linked for any public server release. The
candidate topology, runner and recovery bounds below were written for the PHP
2.x Server (shared Redis, a queue worker and a singleton scheduler). The
recorded PHP released-image attempt stopped during `topology_start` while
waiting for API readiness, before any failure-matrix or recovery-bound claim
could be established. No failover runner has been published for the native
3.x Server, and its `GET /api/cluster/info` reports
`single_region_failover_contract: false`. Do not treat the runner's presence, its scenario
manifest, or a deployment-specific rehearsal as evidence that the released
image has passed the full matrix. Single-region HA therefore remains
support-led.

Self-serve wording is authorized only after a source-free run uses the exact
documented server release for the image, runner, and Compose topology and
publishes a `single-region-failover-result.json` with `outcome: "pass"`,
`runner_blocked: false`, every entry in `phase_outcomes` at `status: "pass"`,
every recovery-bound verdict passing, and evidence for every required scenario
in the public
[single-region failover scenario manifest](/platform-conformance/single-region-failover-scenarios.json).
That public passing result must be linked from this status section and must name
the same server image as `durableworkflow/server:3.10.0`. Until all of those
conditions hold, the recovery bounds below are evaluation targets rather than
released-image validation claims.

### Candidate topology

The intended HA contract applies when the deployment matches the small-cluster
shape or the raw-manifest shape and the operator preserves three rules:

- **One writable workflow database endpoint, always.** Managed failover
  (RDS Multi-AZ, Aurora cluster failover, Cloud SQL HA, Patroni, etc.) is
  permitted on the rule that the previous primary is fenced — revoke the
  write user, demote with `read_only=on`, sever replication, or restore from
  a known-good snapshot — before it can re-attach. A connection proxy must
  preserve the database semantics the Server uses and pass exact-release
  qualification for its mode and settings. See
  [SQL connection budgets](#sql-connection-budgets) for the PgBouncer
  transaction-pooling version requirements and tested modes.
- **One Redis endpoint at a time, with a documented promotion path.**
  Managed-Redis failover (Sentinel, Elasticache replication-group failover,
  Memorystore HA, etc.) is permitted. Redis is the acceleration layer, not
  the correctness substrate, so a Redis failover is a latency event, never a
  correctness event.
- **One scheduler/maintenance runner, always.** The orchestrator (Compose
  service with `deploy.replicas: 1`, systemd unit guarded by a host-level
  lease, or Kubernetes `Deployment` with `replicas: 1` and
  `RollingUpdate.maxSurge: 0`) is responsible for keeping the singleton
  invariant during restart. Duplicate scheduler runners as a steady-state
  topology are not in this contract.

### Per-event behavior and recovery targets

The intended engine contract defines bounded recovery targets for each event
class. These are not released-image guarantees while the release-evidence gate
is closed. The wall-clock recovery time the operator observes is the engine
target plus the managed service's own promotion latency.

| Event | Candidate engine behavior | Recovery target (after substrate / runner is back) |
| --- | --- | --- |
| Managed-database failover | Writes return errors and are not silently buffered. Reads return errors. `/api/ready` fails on every API node and on the scheduler. No acknowledged work is lost. | One connection-pool reconnect, plus one `task_repair` cadence (default 3s), plus one long-poll timeout (default 30s, max 60s) for in-flight pollers. |
| Managed-Redis failover | Wake signals dropped → discovery falls back to long-poll timeout. Acceleration-layer health checks go to **warning**, not error. `/api/ready` typically stays green. | Redis client reconnect interval. |
| API node loss (1 of N) | Load balancer removes the failed node within its readiness interval. In-flight requests against it fail at the LB and are retried by the client. | Load-balancer readiness interval (operator-controlled, typically 5–10s). |
| Worker loss | Tasks held by the failed worker pause until lease expiry. Other workers continue claiming their own tasks. | Lease expiry (5 min for activity tasks), plus one `task_repair` cadence. |
| Scheduler/maintenance restart | Schedule fires pause; activity-timeout enforcement pauses; history pruning pauses. All three resume on the next tick after restart. No duplicate fires occur because the runner is a singleton. | Orchestrator restart latency, plus one scheduler tick. |

### Load-balancer, readiness, and traffic-shift rules

The load balancer in front of the API nodes is the single decision point
for traffic admission during a failover:

- Wire the load balancer to **`GET /api/ready`**, not `/api/health` alone.
  `/api/health` only proves the process is serving HTTP; `/api/ready` reports
  database connectivity and configured readiness checks, including whether
  Redis wake acceleration is healthy or degraded. It does not exercise every
  workflow SQL operation or verify write capacity. During a database outage `/api/ready`
  correctly fails on every node — the load balancer MUST tolerate the all-down
  state rather than fall back to a stale "last known good" roster.
- Use a check interval of 5–10 seconds and a removal threshold of 2–3
  consecutive failures.
- Do not require sticky sessions; the small-cluster smoke proves an
  external worker can poll `server-a` and complete on `server-b`.
- During a Redis-only failover, readiness remains green on every node while
  the database-backed durable paths are available. Acceleration-layer
  degradation surfaces as `checks.cache.status=warning` with
  `checks.cache.degraded_capability=long_poll_wake_acceleration`; do not
  configure the load balancer to remove nodes on that warning.

After substrate recovery, the recommended verification sequence is to wait
for at least one node's `/api/ready` to return 200, curl `/api/cluster/info`
through the load balancer with an admin token to confirm the topology
manifest, issue `POST /api/worker/register` for a probe worker through the
load-balanced endpoint, confirm exactly one scheduler runner is alive in
its orchestrator, and resume external traffic.

### Run the exact-artifact rehearsal

PHP Server 2.x releases include a reusable baseline rehearsal intended to
exercise the full failure matrix without building product code from a
checkout. Its Compose topology starts PHP-only roles (the bootstrap job, queue
worker and scheduler), so it cannot run the native 3.x image. The example below
uses the final PHP release, 2.11.8. The runner and Compose topology must come
from the same release tag as the server image; do not combine a moving
default-branch checkout with an arbitrary image. On a clean host with Docker
Engine, Docker Compose v2, Python 3.11 or newer, and public registry access,
run:

<!-- docs-example id="single-region-failover-exact-release" -->
```bash
export DW_SERVER_RELEASE=2.11.8
git clone --depth 1 --single-branch --branch "$DW_SERVER_RELEASE" \
  https://github.com/durable-workflow/server.git "server-$DW_SERVER_RELEASE"
cd "server-$DW_SERVER_RELEASE"
export DW_SERVER_IMAGE="durableworkflow/server:$DW_SERVER_RELEASE"
scripts/conformance/single-region-failover-published-artifacts.sh \
  --result-dir ./failover-result
```

This path deliberately derives the checkout and `DW_SERVER_IMAGE` from one
release variable. Do not override the image independently; select a different
release by changing `DW_SERVER_RELEASE` before cloning. The runner requires a
concrete public server tag or digest, pulls every
supporting image, resolves all runtime images to repository digests, and
rejects Compose build sections, product-source bind mounts, and local or
rolling server references. It starts exactly two API nodes behind one nginx
endpoint, one MySQL database, one Redis service, and one scheduler/maintenance
runner.

The resulting `single-region-failover-result.json` uses schema
`durable-workflow.v2.single-region-failover.result`. It records exact artifact
and tool versions, normalized topology, readiness transitions, measured
recovery times and bound verdicts, workflow/run/task/schedule identities, and
duplicate/loss assertions for cross-node completion, API-node loss, database
interruption, Redis interruption, worker lease loss, and singleton-scheduler
restart. External runners discover the invocation and public scenario manifest
from `GET /api/cluster/info` at `single_region_failover_contract`.

A result validates engine-visible interruption and recovery against the
released image only when it passes the release-evidence gate above. A failed,
partial, or runner-blocked result is diagnostic evidence, not validation. Even
a passing baseline does not turn a local MySQL or Redis container restart into
evidence for a cloud provider's promotion mechanism. Keep provider-native
promotion, fencing, RPO, and elapsed-time evidence alongside the baseline
result before claiming managed-service HA.

### Recovery packet additions

A small-cluster or raw-manifest deployment that claims this contract MUST
extend its recovery packet (per the
[Operator Operating Envelope](/docs/operator-operating-envelope)) with
rehearsal evidence for each event class:

- a managed-database failover that completes without acknowledged-write
  loss and within the bounded recovery time above;
- a managed-Redis failover that does not flap the load-balancer rotation,
  does not lose any acknowledged work, and surfaces
  `checks.cache.status=warning` with `long_poll_wake_acceleration` as the
  degraded capability;
- an API-node loss event that the load balancer absorbs within the
  configured readiness interval, with no acknowledged-write loss;
- a worker-loss event that preserves the durable run through lease expiry,
  reclaims it after the configured repair bound, and completes it exactly once;
- a scheduler-runner restart on a different host that fires no duplicate
  schedules and leaves no schedule unevaluated past its `next_fire_at`
  plus one tick.

While the release-evidence gate is closed, every deployment remains support-led
even if its own rehearsal passes. After a public all-phase result opens the
release gate, a deployment becomes self-serve under this contract only after
its environment-specific rehearsal evidence is recorded in the operator's
recovery packet and refreshed on the cadence the Operator Operating Envelope
publishes.

### Boundary against unsupported HA claims

The single-region HA contract is intentionally narrow. The following remain
**outside** it and continue to require a support-led design pass; the
topology itself is part of the product risk:

- active/active multi-writer database topologies;
- active/passive multi-region and automatic or hands-free regional failover
  for self-hosted topologies (the
  [next section](#activepassive-multi-region) preserves evaluation guidance);
- synchronous cross-region database replication (RPO=0);
- duplicate scheduler/maintenance runners as a steady-state topology;
- engine-enforced region-pinned task queues as a routing axis;
- provider-specific managed-Kubernetes validation beyond the published
  chart's packaging contract;
- broad "five-nines" or "zero-downtime" SLA promises beyond the bounded
  recovery times above.

The contract is *bounded recovery during named events*, not an uptime
promise that depends on the operator's database, network, and orchestrator
choices. Marketing or SLA language for self-hosted deployments MUST NOT
cross that line without dedicated validation.

## Active/passive multi-region

Active/passive multi-region is **support-led evaluation guidance**, not a
proven self-serve contract in the supported 2.0 operating envelope. Use this
section to review a candidate architecture and build an environment-specific
rehearsal with support before relying on it in production. Each region still
starts from a documented single-node, small-cluster, or raw Kubernetes path,
but those single-region contracts do not establish cross-region replication,
failover, failback, RPO, RTO, or split-brain behavior. The deeper design
material — data authority, replication assumptions,
namespace/task-queue/worker behavior, the failover and failback runbook,
fencing, and the consistency/latency tradeoffs — lives in the workflow library
at
[`docs/deployment/multi-region.md`](https://github.com/durable-workflow/workflow/blob/main/docs/deployment/multi-region.md)
and the standalone server at
[`docs/multi-region-validation.md`](https://github.com/durable-workflow/server/blob/main/docs/multi-region-validation.md).
That material and the candidate shape below were written for the PHP 2.x
topology. The native 3.x Server has no Redis or separate scheduler, and no
native multi-region rehearsal has been published.

The candidate shape to evaluate:

- One **active region** running the validated single-node or small-cluster
  contract: API container(s) behind a load balancer, shared external MySQL or
  PostgreSQL as the writable durable database, shared Redis, exactly one
  scheduler/maintenance runner, and external workers.
- One **standby region** holding an asynchronously replicated standby of the
  workflow database, optional standby Redis, no scheduler/maintenance
  process, and zero or more pre-provisioned API/worker containers that are
  idle until promotion.
- A **regional failover** that is explicit operator work: stop write traffic
  to the failed region, confirm replication state against the published
  RPO, promote the standby database, run any release-required migrations on
  the new primary, start the singleton scheduler/maintenance runner in the
  new active region, switch worker endpoints, switch external traffic, and
  rebuild any derived projections. There is no automatic cross-region
  cutover.
- A **failback** that runs the same sequence in reverse once the original
  region returns to service, with the recovered primary fenced (revoke
  write user, demote with `read_only=on`, sever replication, or restore
  from a known-good snapshot) before re-attaching as a standby.

Data authority and replication assumptions:

- The workflow database is the single durable source of truth and is
  region-bound: exactly one region writes to it at any given time. The
  standby region's database is a read replica until promotion. Recovery
  point objective (RPO) is the asynchronous replication lag; recovery time
  objective (RTO) is operator runbook execution time.
- Redis is region-local acceleration. Wake signals, query-task queue
  locks, and admission locks do not propagate across regions and must not
  be expected to. Each region runs its own Redis; the standby's cache is
  cold or warm at the operator's discretion and correctness does not
  depend on it being preserved across the failover.
- Visibility is served by the active region's database. Promote first,
  then read.

Namespace, task-queue, and worker-registration behavior:

- Namespaces and task queues are stored in the workflow database and
  survive promotion exactly as they were at the last replicated commit.
  They are not regionally partitioned by the engine.
- Workers in the new active region register against the local API
  endpoint after promotion. Pre-existing registrations from the failed
  region remain in the database and expire through the normal
  worker-expiry path.
- Build-id rollouts and deployment-lifecycle state survive failover
  because they live in the workflow database.

Consistency and latency tradeoffs in steady state:

- Workflow starts, signals, and updates commit against the active
  region's database; their latency is the active-region commit latency,
  and they are refused while authority is being withdrawn during a
  failover.
- Workflow-task and activity-task delivery follow the single-region
  acceleration contract inside the active region: sub-second when Redis
  is healthy, durable poll cadence otherwise.
- Schedules fire from the singleton scheduler in the active region and
  pause while no scheduler is running; fires resume from durable
  schedule rows after promotion.
- Visibility reads are read-after-write within the active region only;
  the engine does not provide cross-region read-your-writes or RPO=0.

The disaster-recovery boundary is explicit: this candidate design is not a
substitute for backups or evidence. The recovery packet documented in
[Operator Operating Envelope](/docs/operator-operating-envelope)
remains required, with replication-lag SLO, promotion-runbook latency,
last successful failover rehearsal date, and the fencing procedure for
the recovered primary added on top.

For self-hosted deployments, active/passive and active/active multi-region,
automatic regional failover, synchronous cross-region replication (RPO=0),
cross-region active visibility, and region-pinned task queues as an
engine-enforced routing axis remain
[support-led](/docs/support) because the topology itself is part of the
product risk. The current
[Cloud managed-runtime contract](/docs/polyglot/cloud-control-plane) is
single-region and does not supply those multi-region guarantees.

## Readiness contract

Use both health and readiness checks:

- `GET /api/health` proves the process is serving HTTP.
- `GET /api/ready` reports configured dependency checks, including database
  connectivity, migrations and default namespace readiness. It does not prove
  every workflow SQL operation or available write capacity.
- `GET /api/cluster/info` proves an authenticated client can discover build
  identity, control-plane protocol, worker protocol, payload codecs, and server
  capabilities.
- `POST /api/worker/register` proves workers can authenticate into the expected
  namespace and task queue.

Do not shift traffic based on `/api/health` alone.

<!-- Source: docs/migration.md -->

# Migrating to 2.0

This guide covers the key changes when upgrading an existing Laravel v1 application to v2.
First choose whether Laravel will keep owning the runtime or connect to Cloud or
a self-hosted Server through the PHP SDK in the
[Laravel adoption and runtime transition guide](/docs/laravel-adoption/).
This page then provides the detailed v1-to-v2 embedded package procedure.

## Upgrade procedure

### Before upgrading

**1. Inventory every v1 execution store**

v1 execution state is not necessarily confined to the workflow database. In
addition to the workflow rows and history, Laravel queue jobs can be ready,
delayed, or reserved in Redis, a queue database, SQS, or another queue
backend. Record all of the following before changing code:

- the workflow storage connection and the actual v1 model/table mappings
- every queue connection and queue name used by v1 workflows and activities,
  including per-workflow or per-activity overrides
- the queue backend's database, key prefix, region, account, and other routing
  settings needed to restore the same queues
- the secret-manager reference and immutable version that retrieve the exact
  `APP_KEY`, plus any cache store used for v1 unique-job locks

Do not copy the `APP_KEY` value, secret-manager recovery credentials, database
credentials, or queue-provider credentials into this inventory. Encrypted jobs
and serialized workflow arguments require the original key, but the recovery
manifest should contain only its secret-manager reference and version. Keep the
credentials that can retrieve that key separately access-controlled from the
SQL and queue backups.

The default v1 tables are `workflows`, `workflow_logs`, `workflow_signals`,
`workflow_timers`, `workflow_exceptions`, and `workflow_relationships`.
Published `config/workflows.php` files may replace the
`stored_workflow_model`, `stored_workflow_log_model`,
`stored_workflow_signal_model`, `stored_workflow_timer_model`, or
`stored_workflow_exception_model`. A replacement model may also override its
Eloquent `$table`; `workflow_relationships_table` separately controls the
relationship table. Back up the tables and storage connection resolved by
those configured models, not only the default names.

**2. Choose a rollback boundary and quiesce v1 work**

Prevent new workflow starts and signals while taking the recovery cut. Pause
schedulers and stop every worker that can consume the inventoried queues after
its current job reaches a boundary. For example:

```bash
# Horizon
php artisan horizon:pause

# Supervisor or systemd (use the names from your deployment)
sudo supervisorctl stop <your-worker-group>:*
sudo systemctl stop laravel-worker
```

`php artisan queue:restart` alone is not a quiesce operation when Supervisor,
systemd, Kubernetes, or another process manager immediately starts a
replacement worker. Confirm that no consumer can reserve another job before
capturing state.

Choose and record one of these policies:

- **Drain v1:** block new v1 work, leave v1 workers running until
  `php artisan workflow:v1:list` (when available) or an equivalent query of the
  configured stored-workflow table reports no nonterminal workflows, then stop
  the workers and confirm the relevant queues have no ready, delayed, or
  reserved v1 jobs. SQL-only rollback can cover these terminal v1 workflows.
- **Preserve in-flight v1:** stop workers at job boundaries and take an
  application-consistent snapshot of both SQL and every durable queue backend.
  The queue snapshot must include ready, delayed, and reserved work and must be
  from the same recovery cut as SQL. Wait for a reserved job to finish before
  the cut unless the backend documents how to restore its message and lease
  safely.
- **Accept in-flight loss:** if the backend cannot provide a restorable queue
  cut, record the affected workflow IDs and explicitly accept that
  queue-dependent nonterminal executions cannot be recovered by this rollback.
  Exclude an eligible `pending` row from that disposition only after proving
  the Watchdog path below, and exclude a genuine signal-only wait only after
  testing its retained ingress. Do not present any other SQL row as recoverable
  work.

A signal-only wait may legitimately have no queued job if its external signal
ingress remains available after rollback. Activity retries and timers are
queue-backed: a `workflow_timers` row or a waiting `workflows` row does not
recreate a missing delayed job.

Supported v1.0.77 also starts its enabled-by-default `Workflow\Watchdog` from
the queue worker loop. The Watchdog can find a `pending` workflow whose
`updated_at` is at least five minutes old and whose serialized `arguments` are
present, then redispatch that workflow to its recorded connection and queue.
This is a bounded, pending-only wake path, not a replacement for queue backup.
You may rely on it only after staging evidence shows the restored v1 worker
loop dispatching the Watchdog, the Watchdog redispatching an eligible stale
`pending` row, and that workflow advancing on the expected queue. It does not
recreate activity retries or timer jobs and does not recover `waiting` or
`running` rows. Those states still need their preserved queue job or a valid
external signal path where the workflow is genuinely waiting for a signal.

**3. Back up the rollback recovery set**

Create a full database backup before upgrading:

```bash
# MySQL/MariaDB
mysqldump -u root -p your_database > backup-v1-$(date +%Y%m%d-%H%M%S).sql

# PostgreSQL
pg_dump -U postgres your_database > backup-v1-$(date +%Y%m%d-%H%M%S).sql

# Laravel backup package (if installed)
php artisan backup:run --only-db
```

Then preserve queue state according to the backend:

- **Database queue:** include the queue tables and their connection in the same
  recovery cut. They may be outside the workflow database.
- **Redis queue:** use a restorable Redis snapshot or backup that includes the
  exact queue database/prefix and its ready, delayed, and reserved keys. If the
  same Redis database holds unique-job locks needed by the cut, preserve those
  keys too. Prefer a dedicated queue database or instance; restoring a shared
  Redis snapshot can rewind unrelated application data.
- **SQS or another managed queue:** use a provider-supported, point-in-time
  message restore only if it preserves available, delayed, and in-flight
  messages consistently. SQS does not provide an arbitrary queue snapshot for
  this procedure, so drain v1 work or classify the remaining queue-dependent
  v1 work as unrecoverable. The only SQL-only exceptions are a proven
  Watchdog-eligible `pending` row and a tested signal-only wait.

Store the SQL backup, queue backup, non-secret inventory, and recovery timestamp
together. That recovery manifest may name the `APP_KEY` secret-manager
reference and version, but must not contain the key or any credentials that can
retrieve it. Keep secret-manager, database, queue-provider, and backup recovery
credentials separately access-controlled. Restoring SQL from one cut and queue
state from another is not a supported in-flight rollback.

**4. Test in staging first**

**Do not upgrade production without testing in staging.** The upgrade includes:

- Database schema changes (the v2 durable kernel adds new tables; the
  exact count is whatever `php artisan migrate` applies, and a future
  squashed migration may consolidate the per-feature files)
- Namespace changes requiring code updates
- Queue worker restart (brief interruption)
- Backend capability validation

**Staging test checklist:**

- [ ] Deploy v2 code to staging environment
- [ ] Run migrations against staging database
- [ ] Restart queue workers
- [ ] Run `php artisan workflow:v2:doctor --strict`
- [ ] Start a new v2 workflow and verify it completes
- [ ] Verify v1 workflows (if any) still complete
- [ ] Check Waterline shows both v1 and v2 workflows
- [ ] Run your application's test suite
- [ ] Verify no errors in logs

Only proceed to production after staging validation passes.

### Upgrade steps

**1. Update composer dependency**

```bash
composer require durable-workflow/workflow:2.28.1
```

The maintained Composer package is `durable-workflow/workflow` for both v1 and
v2; this command changes its version constraint to the current public v2
artifact pin. The old `laravel-workflow/laravel-workflow` name is only a
compatibility alias for dependency graphs created before the package rename.
Use the maintained name for new requirements and rollback. The current public
artifact pin names the stable 2.0 package. Use
`durable-workflow/workflow:^2.0` when you want Composer to accept compatible
2.x updates automatically.

**2. Run database migrations**

```bash
php artisan migrate
```

v2 adds the durable-kernel tables that back the v2 feature contract. The
current per-feature migrations create:

- Core: `workflow_instances`, `workflow_runs`, `workflow_history_events`, `workflow_tasks`, `workflow_commands`
- Activity: `activity_executions`, `activity_attempts`
- Features: `workflow_updates`, `workflow_signal_records`, `workflow_run_waits`, `workflow_run_timeline_entries`, `workflow_run_lineage_entries`, `workflow_schedules`, `workflow_schedule_history_events`
- Observability: `workflow_run_summaries`, `workflow_failures`, `workflow_links`, `worker_compatibility_heartbeats`
- Timers: `workflow_run_timers`, `workflow_run_timer_entries`
- Search / memo / message / child: `workflow_search_attributes`, `workflow_memos`, `workflow_messages`, `workflow_child_calls`
- Service catalog: `workflow_service_endpoints`, `workflow_services`, `workflow_service_operations`, `workflow_service_calls`

A future squashed migration may consolidate these into fewer files
without changing the durable contract. The supported way to know what
your database actually has is to run `php artisan migrate:status` after
the upgrade.

The default v1 tables (`workflows`, `workflow_logs`, `workflow_signals`,
`workflow_timers`, `workflow_exceptions`, and `workflow_relationships`) are
preserved for finish-on-v1 execution. If the configured v1 models override
their tables or storage connection, those configured tables are the v1 source
of truth instead.

**3. Update configuration (if needed)**

v2 configuration is backward compatible. If you published `config/workflow.php` in v1, it will continue to work. New v2 options include:

- `durable_types` — type aliases for language-agnostic workflow references
- `task_repair_policy` — how to handle stuck tasks
- `backend_capability_check` — strict vs. permissive validation
- `projection_rebuild` — history rebuild strategies
- `history_budget` — event count limits for continue-as-new

These have sensible defaults. Only configure them if you need non-default behavior. See [Configuration](/docs/configuration/options/) for details.

**Payload codec default changed to `avro`.** v1 defaulted to the PHP-only `Workflow\Serializers\Y::class`; v2 defaults to the language-neutral `avro` codec so Python, Go, and TypeScript workers can decode payloads without a shared PHP runtime. `avro` is the only supported codec for new v2 workflows. New v2 workflows you start will be tagged with `payload_codec = "avro"`.

If you have a published `config/workflows.php` from v1 with `'serializer' => Workflow\Serializers\Y::class`, v2 still reads that value so migration diagnostics can flag it, but new v2 workflow payloads resolve to Avro. Run `php artisan workflow:v2:doctor` after upgrading — it will flag a legacy codec setting as a v1 drain/import concern.

To accept the v2 default explicitly, leave `serializer` unset, or pin it:

```php
// config/workflows.php — v2 default (language-neutral, compact binary)
'serializer' => 'avro',
```

Keep a legacy codec (`'workflow-serializer-y'` or `'workflow-serializer-base64'`) only if you need to finish draining v1 runs that share PHP-native values between a server and PHP-only workers. Legacy class names (`Workflow\Serializers\Y::class`, etc.) are still accepted as aliases for decoding v1 runs.

**Custom serializer classes from v1 are unsupported in v2.** The public v2 registry resolves only `avro`. The legacy `workflow-serializer-y` and `workflow-serializer-base64` readers are confined to the internal v1 import/drain path and cannot be selected for a new v2 run or SDK payload. If you had a custom serializer, drain v1 runs before upgrading or re-encode historical payloads into `avro` — the custom class is not consulted. `php artisan workflow:v2:doctor` flags any other `workflows.serializer` value as migration debt; new-run codec omission resolves to `avro`.

**Eloquent models are not v2 payload values.** v1 passed models to workflows and activities through `SerializesModels`. In v2, pass the model key and load the model inside the activity. A model argument fails with an `unsupported_value_type` error. See [Passing Data](/docs/defining-workflows/passing-data/#models).

**Custom model subclasses are supported only when they keep the package's column and key contract.** The frozen support matrix in
[Customization Matrix](/docs/configuration/customization-matrix/) is
authoritative: subclassing the v2 instance, run, task, history-event,
projection, schedule, activity, failure, link, message, memo, search-attribute,
and child-call models is supported when the subclass keeps the package
table names, primary keys, and foreign keys. Custom table names with
custom foreign-key column names are out of contract. Waterline reads the
v2 projections through the
`Workflow\V2\Contracts\OperatorObservabilityRepository` contract, so a
schema-compatible subclass does not require a Waterline change.

**Environment variables:**

v2 does not introduce new required environment variables. Existing `QUEUE_CONNECTION`, `CACHE_DRIVER`, and `DB_CONNECTION` continue to work.

**4. Restart queue workers**

Queue workers must be restarted to load v2 code:

```bash
# If using Laravel queue workers
php artisan queue:restart

# If using Supervisor
sudo supervisorctl restart <your-worker-group>:*

# If using systemd
sudo systemctl restart laravel-worker

# If using Horizon
php artisan horizon:terminate
```

Workers will:
1. Finish their current job
2. Exit gracefully
3. Restart with v2 code loaded

**Workers must restart before processing v2 workflows.** v1 workflows can complete with old or new workers (finish-on-v1 compatibility).

### After upgrading

**1. Verify backend capability**

```bash
php artisan workflow:v2:doctor --strict
```

Expected output:

```
✓ Database driver supports required features
✓ Queue driver supports required features
✓ Cache driver supports locks
✓ All backend capabilities present
```

If any check fails, see [Backend Requirements](/docs/installation/#requirements) for driver prerequisites.

**2. Verify v2 workflows start successfully**

Start a test workflow using v2 API:

```php
use Workflow\V2\WorkflowStub;
use Workflow\V2\StartOptions;

$workflow = WorkflowStub::make(TestWorkflow::class, 'test-upgrade');
$result = $workflow->start(['test' => true], new StartOptions());
$runId = $result->runId();
```

`WorkflowStub::start()` returns a `StartResult` object. Pull the run id off it
with `runId()` before comparing against database rows — treating the return
value as a scalar string will silently compare an object against a column.

Check that:

- Workflow appears in Waterline
- `workflow_instances` table has a row with matching `instance_id` (`$workflow->id()`)
- `workflow_runs` table has a row with matching `run_id` (`$result->runId()`)
- Workflow completes or progresses as expected

**3. Check v1 workflows (if any)**

If you have in-flight v1 workflows:

```bash
php artisan workflow:v1:list
```

Verify they continue to progress. v1 workflows should complete on the v1 engine without errors.

**4. Monitor logs for errors**

Watch application logs for workflow-related errors:

```bash
tail -f storage/logs/laravel.log | grep -i workflow
```

Common issues:

- Namespace errors: code still using `Workflow\Workflow` instead of `Workflow\V2\Workflow`
- Method errors: v2 workflow or activity classes that still need to rename their entry method to `handle()`
- Queue driver errors: using `sync` driver in queue mode (not supported); in poll mode (`workflows.v2.task_dispatch_mode=poll`) the queue is unused for task delivery and `sync` is acceptable

**5. Verify Waterline observability**

Open Waterline (default: `/waterline`) and verify:

- v1 workflows (if any) appear with their original data
- v2 workflows appear with full run/history/activity detail
- No errors in Waterline rendering

### Rollback procedure

Rollback is a recovery-set operation. Restoring the workflow SQL database alone
does not restore in-flight v1 execution when its Laravel queue state lives in
Redis, SQS, another database, or another external backend.

If the upgrade fails in production, roll back only to the policy and recovery
cut selected before the upgrade:

**1. Quiesce application ingress, schedulers, and queue workers**

Use the same stop procedure as the pre-upgrade cut. Confirm that no worker can
reserve a job and no request can start or signal a workflow while state is
being restored.

**2. Validate the recovery set**

- A SQL-only recovery set is supported only when the selected cut has no
  nonterminal v1 execution that depends on preserved queued work. An eligible
  stale `pending` row may instead use the proven v1.0.77 Watchdog path described
  above, and a genuine signal-only wait may use a tested external signal
  ingress. Do not extend either exception to other states.
- An in-flight recovery set must contain SQL plus restorable ready, delayed,
  and reserved queue state from the same cut for retries, timers, activities,
  and `waiting` or `running` work. Signal-only waits must retain a working
  signal ingress; timer waits must retain their delayed queue jobs.
- If neither condition holds, stop here. Fix forward, reconcile the recorded
  workflows manually, or proceed under the previously documented acceptance
  that the affected executions are unrecoverable.

**3. Restore the database backup**

```bash
# MySQL/MariaDB
mysql -u root -p your_database < backup-v1-YYYYMMDD-HHMMSS.sql

# PostgreSQL
psql -U postgres -d your_database < backup-v1-YYYYMMDD-HHMMSS.sql
```

Restore every configured custom v1 model table and the
`workflow_relationships_table`, even when they live on another connection.

**4. Restore the queue recovery cut, when required**

Keep consumers stopped. Follow the queue provider's restore procedure and
restore the exact connections, queues, prefixes, ready jobs, delayed jobs, and
reserved jobs captured with SQL. Restore required unique-job lock state when
it was part of the recovery cut. Do not substitute an empty queue with the
same name: it cannot wake restored retries, timers, or `waiting`/`running`
work. An empty but writable queue is sufficient only for an eligible `pending`
row after you prove that the v1.0.77 Watchdog bootstraps and redispatches it.

**5. Revert the Composer dependency**

```bash
composer require durable-workflow/workflow:^1.0 --with-all-dependencies
```

`durable-workflow/workflow` is the maintained package name for both supported
v1 releases and v2. `laravel-workflow/laravel-workflow` is only a legacy alias
declared by the maintained package for compatibility with older dependency
graphs; do not use it in rollback requirements.

Retrieve the exact `APP_KEY` through the inventoried secret-manager reference
and version, using the separately controlled recovery credentials. Restore it
together with the workflow storage connection, queue connections, queue names,
Redis prefixes, and custom model configuration before starting a consumer.
Encrypted jobs and serialized model references depend on that configuration
matching the recovery cut.

**6. Restart queue workers and reopen ingress**

```bash
# Use the start command for your worker system, for example:
sudo supervisorctl start <your-worker-group>:*
sudo systemctl start laravel-worker
php artisan horizon:continue
```

**7. Verify v1 execution reachability**

Waterline visibility proves that a row was restored; it does not prove that a
worker can resume it. Complete this checklist before declaring rollback
successful:

- [ ] `composer show durable-workflow/workflow` reports the intended v1
  release and workers loaded that code.
- [ ] The actual configured v1 tables and `workflow_relationships_table` are
  present on the expected storage connections.
- [ ] Every nonterminal row in the configured stored-workflow table
  (`workflows` by default) is classified by its next wake path: a
  ready/delayed/reserved queue job, a preserved timer job, or a tested external
  signal ingress. An eligible `pending` row may instead cite the proven
  v1.0.77 Watchdog redispatch evidence below.
- [ ] Queue-provider inspection confirms that each queue-backed retry, timer,
  activity, and `waiting` or `running` workflow wake identified above exists on
  the recorded connection and queue. A SQL timer row by itself does not satisfy
  this check, and the Watchdog is not a substitute for these jobs.
- [ ] If a restored `pending` row has no preserved workflow job, a v1.0.77
  worker loop is observed enqueueing the Watchdog; after the row is stale for
  the five-minute bound, the Watchdog enqueues that workflow on its recorded
  connection and queue and the row advances. Without that end-to-end proof,
  treat the row as stranded even if Waterline displays it.
- [ ] No restored retry, timer, `waiting`, or `running` row lacks its required
  runnable queue job or a usable external signal path for a genuine signal
  wait. Treat such a row as stranded; pending-only Watchdog evidence does not
  make it recoverable.
- [ ] A restored retry or timer advances past its pre-cut marker, and a
  controlled signal-wait can enqueue and consume its signal wake.
- [ ] A new v1 canary workflow completes, and logs contain no missing-job,
  decryption, model-table, connection, or queue-routing errors.

With default tables, this query is a starting inventory; substitute the table
resolved by your configured stored-workflow model when it is customized:

```sql
SELECT id, class, status
FROM workflows
WHERE status NOT IN ('completed', 'failed', 'cancelled');
```

**Important rollback notes:**

- Rollback discards any v2 workflows started after upgrade (they exist only in v2 tables)
- SQL-only rollback restores v1 workflow records, not external queue messages
- In-flight v1 execution is supported only when SQL and durable queue state are
  restored from one application-consistent recovery cut, or when a documented
  signal-only wait retains a usable ingress. The additional SQL-only exception
  is an eligible stale `pending` row with proven v1.0.77 Watchdog redispatch;
  it does not cover retries, timers, or `waiting`/`running` work
- Replaying a pre-upgrade recovery cut can repeat external side effects; review
  application idempotency before restoring queued activity work
- If you must preserve v2 workflows started during the upgrade window, do not restore the database — instead fix the upgrade issue forward

## Code changes

The sections below detail the code-level changes needed when migrating from v1 to v2 APIs.

### Namespace change

All v2 classes live under `Workflow\V2`. Update your imports:

```php
// v1
use Workflow\Workflow;
use Workflow\Activity;
use Workflow\WorkflowStub;

// v2
use Workflow\V2\Workflow;
use Workflow\V2\Activity;
use Workflow\V2\WorkflowStub;
```

### Entry method

v2 workflows and activities use `handle()` as the entry method. Rename v1 `execute()` methods to `handle()` as part of the v2 code migration:

```php
// v1
class MyWorkflow extends Workflow
{
    public function execute($input)
    {
        $result = yield ActivityStub::make(MyActivity::class, $input);
        return $result;
    }
}

// v2
use function Workflow\V2\activity;

class MyWorkflow extends Workflow
{
    public function handle($input)
    {
        return activity(MyActivity::class, $input);
    }
}
```

Do not leave `execute()` as the entry method on a v2 workflow or activity — the runtime rejects it.

### Activity calls

v2 replaces `ActivityStub::make()` and `yield` with direct function helpers:

```php
// v1
$result = yield ActivityStub::make(MyActivity::class, $arg1, $arg2);

// v2
use function Workflow\V2\activity;

$result = activity(MyActivity::class, $arg1, $arg2);
```

Activities now have durable identity. Each scheduled activity gets an `activity_executions` row with a stable execution id, and each concrete attempt gets an `activity_attempts` row with typed history.

### Workflow identity

v2 splits identity into instance id and run id:

- `id()` — the public workflow instance id (same across continue-as-new)
- `runId()` — the id of the current run

In v1, these were the same concept.

### Signals

v2 uses named signal waits instead of `#[SignalMethod]` attribute-based mutators:

```php
// v1
#[SignalMethod]
public function approve()
{
    $this->approved = true;
}

// v2
use function Workflow\V2\await;

$approved = await('approve');
```

Named signals support `await('name')` for blocking workflow-code waits and `signal()` / `attemptSignal()` for external input. Cancellation and termination are not modeled as signals — they remain explicit runtime commands.

### Queries

v2 uses replay-safe query methods instead of reading workflow properties directly:

```php
// v1
#[QueryMethod]
public function getStatus(): string
{
    return $this->status;
}

// v2
use function Workflow\V2\query;

// Queries are defined as named, replay-safe accessors
```

### Timers and side effects

The function-based helpers replace the v1 static methods:

```php
// v1
yield Timer::make(60);
$value = yield SideEffect::make(fn() => random_int(1, 100));

// v2
use function Workflow\V2\timer;
use function Workflow\V2\sideEffect;

timer(60);
$value = sideEffect(fn() => random_int(1, 100));
```

### Timeouts

v2 adds workflow-level timeouts through `StartOptions`:

```php
use Workflow\V2\StartOptions;
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::make(MyWorkflow::class, 'order-123');

$workflow->start(
    $orderId,
    StartOptions::rejectDuplicate()
        ->withExecutionTimeout(7200)   // 2 hours across all runs
        ->withRunTimeout(3600),        // 1 hour per run
);
```

- **Execution timeout** spans the entire instance, including continue-as-new transitions.
- **Run timeout** applies to a single run and resets on continue-as-new.

### Database migrations

v2 adds new tables and columns. The package auto-loads its migrations, so after updating:

```bash
composer update durable-workflow/workflow
php artisan migrate
```

The 2.0.0 release includes clean base table migrations. The normal path is to
let Laravel auto-load the package migrations and run `php artisan migrate`.

If you previously published Durable Workflow migrations into your application,
choose one migration source and keep it current:

- **Auto-loaded package migrations**: remove old published Durable Workflow
  migration files from `database/migrations` and run `php artisan migrate`.
- **Published migrations**: publish the current set before migrating:

  ```bash
  php artisan vendor:publish \
    --provider="Workflow\Providers\WorkflowServiceProvider" \
    --tag=migrations \
    --force

  php artisan migrate
  ```

Do not keep stale published files while also relying on newly auto-loaded
package files; that can leave your app missing newer v2 tables or repair
migrations.

If you customized migration files, diff your local copies against the package's
`src/migrations` directory during each upgrade. Keep the table names, columns,
indexes, and nullable/default contracts schema-compatible with the package
models. A customized install that routes workflow tables to a non-default
connection should publish the migrations, set the migration `$connection`, and
then continue carrying forward every new package migration in timestamp order.

For pre-release v2 adopters, `workflow_run_summaries.memo` is repaired
idempotently if an older published summary-table migration created
`workflow_run_summaries` without that column. Fresh installs already create the
column in the base summary-table migration.

### Backend capability check

v2 validates that your queue, database, and cache drivers meet its requirements. Run the doctor command after upgrading:

```bash
php artisan workflow:v2:doctor --strict
```

### Configuration

v2 introduces several new configuration options. See the [Configuration](/docs/configuration/options/) section for details on:

- Durable type aliases
- Task repair policy
- Backend capability checks
- Projection rebuilds
- History budgets and export redaction

### Waterline

Waterline (the monitoring UI) has been updated for v2 with:

- Run detail views showing timeout durations and deadlines
- Activity attempt tracking with durable ids
- Updated workflow status displays

### Continue-as-new

v2 adds history budgets that can automatically trigger continue-as-new when the event count exceeds a threshold. Metadata (memo, search attributes, timeouts) is carried forward across transitions.

## Existing workflows

### Finish-on-v1 strategy

**Workflows started under v1 will continue to execute through v1 compatibility paths.** New workflows started after upgrading will use v2 semantics. You do not need to migrate running workflow instances.

When you upgrade to 2.0:

1. **v1 data is preserved** — The default `workflows`, `workflow_logs`, `workflow_signals`, `workflow_timers`, `workflow_exceptions`, and `workflow_relationships` tables remain intact; configured v1 model/table overrides remain authoritative
2. **v1 workflows complete using v1 engine** — In-flight v1 workflows continue executing using the v1 replay engine until they reach a terminal state
3. **v2 workflows use v2 engine** — All workflows started after upgrade use the v2 schema (`workflow_instances`, `workflow_runs`, `workflow_history_events`, etc.)
4. **Waterline shows both** — The monitoring UI displays v1 and v2 workflows side-by-side

Finish-on-v1 also depends on the Laravel queue backend. Retain every v1 queue
connection and queue until its workflows are terminal. Database rows preserve
history and status; they do not reconstruct lost ready, delayed, or reserved
jobs from an external backend. The v1.0.77 Watchdog can redispatch an eligible
stale `pending` workflow from SQL, but that bounded recovery path does not
recreate retries, timers, or `waiting`/`running` work.

### Tracking v1 workflow completion

To see which v1 workflows are still active after upgrading:

```bash
php artisan workflow:v1:list
```

This command lists all v1 workflows that have not yet reached a terminal state (completed, failed, cancelled). Use it to track v1 workflow completion over time.

Sample output:

```
+--------------------------------------+---------------------+-----------+------------+
| ID                                   | Class               | Status    | Created    |
+--------------------------------------+---------------------+-----------+------------+
| 01J1234567890ABCDEFGHIJK             | App\OrderWorkflow   | running   | 2 days ago |
| 01J9876543210ZYXWVUTSRQP             | App\InvoiceWorkflow | pending   | 1 day ago  |
+--------------------------------------+---------------------+-----------+------------+
```

#### Waterline visibility

After upgrading to 2.0, Waterline automatically shows workflows from both engines:

- **v1 workflows** appear with their original `StoredWorkflow` data (class, status, logs, signals, exceptions)
- **v2 workflows** appear with full v2 detail (runs, history events, timers, activities, search attributes)

No configuration is needed — Waterline reads from both table sets and presents a unified view.

### When to clean up v1 tables

Once all v1 workflows have completed (confirmed by `workflow:v1:list` showing zero active workflows), you may optionally drop the v1 tables:

```sql
DROP TABLE IF EXISTS workflow_relationships;
DROP TABLE IF EXISTS workflow_exceptions;
DROP TABLE IF EXISTS workflow_timers;
DROP TABLE IF EXISTS workflow_signals;
DROP TABLE IF EXISTS workflow_logs;
DROP TABLE IF EXISTS workflows;
```

**Important:** Do not drop these tables while any v1 workflows remain active. Doing so will cause v1 replay to fail and leave workflows stuck.

### Why finish-on-v1?

The finish-on-v1 strategy avoids forcing a data migration at upgrade time. v1 and v2 use fundamentally different storage models:

- **v1** stores workflow state as a denormalized `workflows` row with related logs, signals, and timers
- **v2** stores workflow state as event-sourced history with projections (`workflow_instances`, `workflow_runs`, `workflow_history_events`)

Converting in-flight v1 workflows to v2 history would require reconstructing event sequences from v1 logs, which risks data loss and replay inconsistencies. The finish-on-v1 approach lets v1 workflows complete safely on their original engine while new work moves to v2 immediately.

<!-- Source: docs/waterline-operator-api.md -->

# Waterline Operator API Reference

Waterline is the durable-state operator surface for both embedded Laravel and
standalone-server deployments. The UI uses the same HTTP+JSON API documented
here, so scripts, dashboards, and agents can read durable workflow facts
without scraping HTML.

Use this reference when you need typed operator evidence: selected-run detail,
history export, actionability, command affordances, saved views, preferences,
and schedule visibility. Use [Monitoring](./monitoring.md) for the conceptual
split between Waterline durable state and worker/runtime telemetry.

## Installation

Install Waterline in the same embedded Laravel application that installs the
workflow package:

```bash
composer require \
  durable-workflow/waterline:2.18.0 \
  durable-workflow/workflow:2.28.1
php artisan waterline:install
```

The generated pins select stable 2.x releases of Waterline and Workflow. Embedded
mode reads the application's workflow state directly and needs no service SDK.
After upgrading Waterline, publish the current assets:

```bash
composer update durable-workflow/waterline
php artisan waterline:publish
```

For service-mode observation inside a Laravel host, also install the PHP SDK:

```bash
composer require durable-workflow/sdk:3.18.2
```

The self-contained Waterline service image already includes that SDK. Its
connection, authentication and persistence inputs are described in
[Monitoring](./monitoring.md#waterline-service).

### Interface Language

Waterline supports English, Brazilian Portuguese, Simplified Chinese, French, German, Japanese, Spanish and Ukrainian in both
embedded and service mode. Set the language in the Laravel application's environment or the service
container's environment:

```dotenv
WATERLINE_LOCALE=es
```

| Language | Setting |
| --- | --- |
| English | `en` |
| Brazilian Portuguese | `pt-BR` |
| Simplified Chinese | `zh-Hans` |
| French | `fr` |
| German | `de` |
| Japanese | `ja` |
| Spanish | `es` |
| Ukrainian | `uk` |

Spanish regional aliases such as `es-ES`, `es-MX` and `es-419` select the shared
Spanish interface. `pt`, `pt_br` and case variants select Brazilian Portuguese
(`pt-BR`). Other Portuguese regions fall back to English. Counts, chart labels
and relative durations follow the selected interface language.
Use `zh-Hans` for Simplified Chinese. `zh`, `zh-CN`, `zh-SG` and explicit
Simplified script tags such as `zh-Hans-CN` select the same interface.
Underscores and case variants are accepted. Traditional Chinese tags such as
`zh-Hant`, `zh-TW` and `zh-HK` fall back to English.
Use `ja` for Japanese. Regional aliases such as `ja-JP`, underscores and case
variants select the same Japanese interface.
Use `fr` for French. Regional aliases such as `fr-FR` and `fr-CA`, underscores
and case variants select the shared French interface.
Use `de` for German. Regional aliases such as `de-DE`, `de-AT` and `de-CH`,
underscores and case variants select the shared German interface.

English is the default and fallback. Embedded applications can also set
`waterline.locale` in their resolved configuration. Rebuild a cached Laravel
configuration after changing the setting. Waterline keeps the host application's
locale, workflow names, identifiers, payloads and original exception text
unchanged. JSON API field names and machine states retain their existing values.

## Deployment Boundary

Waterline has two backend adapters:

- Embedded mode runs inside the Laravel host and reads the workflow package's
  durable state in process.
- Service mode runs the published Waterline image and reads one standalone
  server namespace through `durable-workflow/sdk` and the server's public API.

Both adapters expose the Waterline routes documented on this page. Service
mode translates those routes to the corresponding server contracts:

| Waterline route family | Standalone-server contract used by service mode |
| --- | --- |
| `GET /waterline/api/v2/health` | Server health, worker registration, and task-queue visibility. |
| `GET /waterline/api/stats` | The namespace-scoped operator dashboard and metrics snapshot. |
| Flow lists, selected-run detail, and history export | Workflow list, detail, runs, history, and diagnostics from the [Server API Reference](./polyglot/server-api-reference.md). |
| Signal, update, query, cancel, terminate, repair, and archive actions | The matching server workflow command and repair contracts. |
| Routes under `/waterline/api/v2/schedules` | The server schedule list, detail, history, and mutation contracts. |

The server API, CLI, and SDK surfaces remain available directly when
Waterline is deployed. Do not assume cross-mode or cross-namespace visibility:
runs stay readable and actionable from their owning runtime and namespace.
[Deployment Modes](./polyglot/deployment-modes.md) defines that shared product
boundary.

## Base Path And Scope

Waterline mounts under the Laravel app's configured Waterline base path. The
examples below use `/waterline`; if your app publishes Waterline under a
different prefix, keep the `/api/...` suffixes the same.

When `WATERLINE_NAMESPACE` is configured, every list, detail, schedule, and
operator-action route is namespace scoped. Cross-namespace reads and commands
return not-found semantics instead of leaking another namespace's workflow
state.

```bash
curl -sS "$APP_URL/waterline/api/instances/order-1001" \
  -H "Accept: application/json" | jq '.status, .run_id, .actionability'
```

## Authentication

Every Waterline UI and JSON API controller passes through Waterline's access
check. In embedded mode, the Laravel host defines that check and decides which
guards, route middleware, gates, policies, SSO/OIDC/SAML sessions,
directory-backed groups, service tokens, and rate limits apply.

The current JSON operator API group excludes Laravel's CSRF middleware so a
host can apply a session or service-token boundary consistently to both reads
and commands. The host remains responsible for authenticating those requests
and for documenting any browser-session CSRF protections it adds around the
API.

The self-contained service image has no host-application users. Put it behind
an authenticating reverse proxy or on a private interface before setting
`WATERLINE_ALLOW_UNAUTHENTICATED=true`; that setting delegates the Waterline
front-door check to the surrounding deployment. Separately,
`WATERLINE_SERVER_TOKEN` authenticates Waterline's PHP SDK calls to the
standalone server. Never expose that server credential to the browser.

`WATERLINE_ACCESS_MODE=read_only` blocks mutating Waterline routes locally.
`operator` enables them, but the connected server still applies its own token
role and namespace authorization. See
[Monitoring](./monitoring.md#waterline-service) for the deployment example and
the two authentication boundaries.

## Dashboard And Health

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/waterline/api/stats` | Dashboard totals, backlog counters, repair policy, operator metrics, and fleet trend data. |
| `GET` | `/waterline/api/v2/health` | Waterline package/runtime health and v2 data-source readiness. |

`/waterline/api/stats` is the dashboard's summary surface. Treat
`operator_metrics` as Waterline-owned JSON diagnostics, not as a Prometheus
scrape surface. Use worker SDK metrics for runtime latency and custom
application telemetry.

The [Operator Operating Envelope](./operator-operating-envelope.md) defines how
to interpret those diagnostics during rollouts and incident response. In
particular:

| Field family | Meaning |
| --- | --- |
| `operator_metrics.backlog.*` | Durable runnable, delayed, leased, unhealthy, repair-needed, claim-failed, and compatibility-blocked work counts, plus the fleet-level `tasks_added_last_minute` and `tasks_dispatched_last_minute` queue-flow facts. |
| `operator_metrics.matching_role.*` | The node-local matching/dispatch contract for the process serving the request: `queue_wake_enabled`, deployment `shape`, `task_dispatch_mode`, frozen `partition_primitives`, and current `backpressure_model`. |
| `operator_metrics.repair.*` | Repair-loop sweep footprint, including selected candidates, candidate age, and scan pressure. |
| `operator_metrics.projections.*` | Projection-drift counts for run summaries, waits, timelines, timers, and lineage. |
| `operator_metrics.command_contracts.*` | Legacy WorkflowStarted contract snapshots that still need backfill. |
| `operator_metrics.history.*` | History-size and event-count pressure plus continue-as-new recommendations. |
| `queue_visibility.available`, `queue_visibility.reason` | Whether namespace-scoped queue visibility is available on this host app, and the reason when it is not. |
| `queue_visibility.task_queues[].stats.*` | Queue-local backlog, backlog age, poller counts, workflow/activity ready-versus-leased counts, and per-queue `tasks_added_last_minute` / `tasks_dispatched_last_minute` flow facts. |
| `queue_visibility.task_queues[].repair.*` | Queue-local repair pressure, including candidates, dispatch failures, expired leases, dispatch-overdue counts, and the oldest age for each condition. |
| `coordination_alerts[]` | Roll-up warnings and errors derived from the health checks plus queue-visibility risks such as backlog without pollers, stale pollers, or aged repair candidates. |
| `checks[]`, `categories.*` | Blocking versus advisory v2 health checks, with per-check `category = correctness \| acceleration` and the category rollups Waterline uses on the workers surface. |
| `engine_source`, `readiness_contract` | Whether Waterline is actively using the v2 operator bridge and which readiness contract governs that state. |

The queue-flow fields answer a specific operator question: is durable queue
input arriving faster than the system is dispatching it? `tasks_added_last_minute`
counts distinct durable task rows created in the trailing 60 seconds, and
`tasks_dispatched_last_minute` counts distinct durable task rows whose latest
successful dispatch landed in that same window.

Use the queue-flow totals with the server task-queue visibility routes rather
than by themselves. Waterline tells you whether durable inflow is outrunning
dispatch, while `/api/task-queues` and `/api/task-queues/{taskQueue}` tell you
whether the hot queue is `saturated`, intentionally `throttled`,
`no_active_workers`, or otherwise short on healthy pollers or slots.

`GET /waterline/api/v2/health` exposes the same queue-local evidence without
leaving the Waterline surface. Scripts and dashboards that need one namespace's
current queue posture should read `queue_visibility.task_queues[]` for
per-queue stats and `coordination_alerts[]` for the summary of which queue or
health-check condition currently needs operator attention.

The `matching_role` fields answer a different question: which matching shape is
this node currently serving? `shape` distinguishes `in_worker` from
`dedicated`, `partition_primitives` freezes the routing axes as
`connection`/`queue`/`compatibility`/`namespace`, and `backpressure_model`
currently reports `lease_ownership`. Treat that block as process-local. During
mixed-shape rollouts, compare it across nodes before assuming backlog or poll
differences mean worker trouble.

`GET /waterline/api/v2/health` uses the same distinction: `error` is blocking,
`warning` is advisory, and `ok` means the current v2 operator bridge is ready.

## List Views

List views are bucketed by durable status:

| Method | Path | Bucket |
| --- | --- | --- |
| `GET` | `/waterline/api/flows/running` | Open or actively blocked runs. |
| `GET` | `/waterline/api/flows/completed` | Runs closed successfully, including continued runs with `closed_reason = continued`. |
| `GET` | `/waterline/api/flows/failed` | Failed runs. |
| `GET` | `/waterline/api/flows/cancelled` | Cancelled runs. |
| `GET` | `/waterline/api/flows/terminated` | Terminated runs. |

List rows are compact operator summaries. Stable fields include workflow
identity (`id`, `instance_id`, `run_id`, `workflow_type`), state (`status`,
`closed_reason`, `archived_at`), timing, `history_event_count`,
`history_size_bytes`, `continue_as_new_recommended`, and repair/actionability
badges such as `repair_blocked`.

## Selected-Run Detail

Waterline has two addressing modes:

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/waterline/api/instances/{instanceId}` | Current selected run for a workflow instance. |
| `GET` | `/waterline/api/instances/{instanceId}/runs/{runId}` | Explicit selected run. |
| `GET` | `/waterline/api/flows/{id}` | Legacy/detail lookup by Waterline row or run id. |

Selected-run detail is the authoritative JSON contract for operator screens.
It contains durable state and derived diagnostics for:

| Field family | Meaning |
| --- | --- |
| `activities`, `timers`, `waits`, `children` | Current and historical wait state rebuilt from typed history first. |
| `signals`, `updates`, `declared_signals`, `declared_updates`, `declared_queries` | Command lifecycle rows plus loadable workflow contract targets. |
| `timeline` | Ordered durable events and diagnostic entries. |
| `exceptions`, `logs` | Failure facts and replay/debug context. |
| `can_signal`, `can_update`, `can_query`, `can_cancel`, `can_terminate`, `can_archive` | UI and automation affordances for the selected run. |
| `actionability` | Versioned repair and evidence contract described below. |

Automation should prefer selected-run detail over visual screenshots. A
screenshot can show what an operator saw; selected-run detail explains why a
workflow can or cannot be repaired, queried, cancelled, archived, or replayed.

## History Export

History exports are replay/debug bundles. They intentionally include stored
workflow, command, activity, update, task, failure, and history payloads so a
developer can reproduce or archive the selected run.

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/waterline/api/instances/{instanceId}/history-export` | Export the current selected run for an instance. |
| `GET` | `/waterline/api/instances/{instanceId}/runs/{runId}/history-export` | Export an explicit run. |
| `GET` | `/waterline/api/flows/{id}/history-export` | Legacy/detail export by Waterline row or run id. |

If an export can leave a protected environment, configure the workflow
history-export redactor first. Downstream tools should preserve the export's
`redaction` metadata so reviewers can tell which policy shaped the artifact.

## Actionability Contract

Waterline annotates list rows, selected-run detail responses, timeline entries,
and history exports with the versioned actionability contract:

```json
{
  "actionability_contract": {
    "schema": "waterline.actionability",
    "version": 1
  }
}
```

The contract identifier is `actionability_contract.schema = waterline.actionability`
with `actionability_contract.version = 1`.

Run-level `actionability` answers whether the selected run can be repaired:

| Field | Meaning |
| --- | --- |
| `repair_state` | One of `repairable`, `blocked`, `not_needed`, or `unknown`. |
| `repairable` | Boolean shorthand for `repair_state = repairable`. |
| `blocked_reason` | Stable reason code when `repair_state = blocked`. |
| `status_bucket` | The Waterline bucket that shaped the run-level decision. |
| `closed_reason` | Durable close reason when the run is closed. |
| `task_problem` | Whether Waterline saw a task-level problem on the run. |
| `diagnostic_only_evidence` | True when at least one row is informative but not a resume source. |

Evidence rows under `activities`, `waits`, `timers`, `exceptions`, `logs`, and
timeline/export entries can include their own `actionability` block:

| Field | Meaning |
| --- | --- |
| `state` | `actionable` when the row is a valid repair source, otherwise `diagnostic_only`. |
| `repair_source` | True only for rows backed by a repairable source authority. |
| `diagnostic_only` | True when the row must not be used as a resume source. |
| `history_authority` | Source authority such as `typed_history`, `mutable_open_fallback`, `failure_row_fallback`, or `unsupported_terminal_without_history`. |
| `history_unsupported_reason` | Stable reason code for unsupported fallback history. |

Agents and scripts must gate repair/resume decisions on
`actionability.repair_state`, `actionability.repairable`, and row-level
`actionability.repair_source`. A row with `diagnostic_only = true` is evidence,
not permission to replay or resume.

## Operator Actions

Waterline actions are durable commands issued through the selected-run contract.
Instance routes target the current run; run routes reject stale or wrong-run
targets explicitly.

| Method | Path | Purpose |
| --- | --- | --- |
| `POST` | `/waterline/api/instances/{instanceId}/queries/{query}` | Execute a declared query against the current run. |
| `POST` | `/waterline/api/instances/{instanceId}/signals/{signal}` | Send a signal to the current run. |
| `POST` | `/waterline/api/instances/{instanceId}/updates/{update}` | Submit an update to the current run. |
| `POST` | `/waterline/api/instances/{instanceId}/repair` | Dispatch a repair pass for the current run when actionability allows it. |
| `POST` | `/waterline/api/instances/{instanceId}/cancel` | Request cancellation for the current run. |
| `POST` | `/waterline/api/instances/{instanceId}/terminate` | Terminate the current run. |
| `POST` | `/waterline/api/instances/{instanceId}/archive` | Archive the selected closed run. |
| `POST` | `/waterline/api/instances/{instanceId}/runs/{runId}/queries/{query}` | Execute a query against an explicit selected run. |
| `POST` | `/waterline/api/instances/{instanceId}/runs/{runId}/signals/{signal}` | Send a signal only if the selected run is current. |
| `POST` | `/waterline/api/instances/{instanceId}/runs/{runId}/updates/{update}` | Submit an update only if the selected run is current. |
| `POST` | `/waterline/api/instances/{instanceId}/runs/{runId}/repair` | Repair an explicit selected run. |
| `POST` | `/waterline/api/instances/{instanceId}/runs/{runId}/cancel` | Cancel only if the selected run is current. |
| `POST` | `/waterline/api/instances/{instanceId}/runs/{runId}/terminate` | Terminate only if the selected run is current. |
| `POST` | `/waterline/api/instances/{instanceId}/runs/{runId}/archive` | Archive the selected closed run. |

Signals, updates, and queries accept JSON `arguments`. Invalid arguments return
field-level validation failures. Replay blocks return `409 Conflict` with a
stable reason such as `workflow_definition_unavailable` instead of attempting
best-effort execution.

## Update Inspection

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/waterline/api/instances/{instanceId}/updates/{updateId}` | Inspect an update lifecycle row for the current run. |
| `GET` | `/waterline/api/instances/{instanceId}/runs/{runId}/updates/{updateId}` | Inspect an update lifecycle row for an explicit run. |
| `GET` | `/waterline/api/flows/{id}/updates/{updateId}` | Legacy/detail update lookup. |

Use update inspection when a UI or agent needs to explain whether an update is
accepted, applied, completed, rejected, timed out while waiting, or blocked by
replay compatibility.

## Schedules

Waterline schedule routes are v2 visibility and operator-action routes:

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/waterline/api/v2/schedules` | List schedules visible in the current scope. |
| `GET` | `/waterline/api/v2/schedules/{scheduleId}` | Describe one schedule. |
| `GET` | `/waterline/api/v2/schedules/{scheduleId}/history` | Read the schedule's lifecycle audit events. |
| `POST` | `/waterline/api/v2/schedules/{scheduleId}/pause` | Pause future fires. |
| `POST` | `/waterline/api/v2/schedules/{scheduleId}/resume` | Resume a paused schedule. |
| `POST` | `/waterline/api/v2/schedules/{scheduleId}/trigger` | Trigger an immediate fire. |
| `POST` | `/waterline/api/v2/schedules/{scheduleId}/backfill` | Backfill a time window. |
| `DELETE` | `/waterline/api/v2/schedules/{scheduleId}` | Delete a schedule. |

A schedule is identified by its namespace and its `schedule_id`. The same
`schedule_id` can exist in more than one namespace. Every per-schedule route
accepts an optional `namespace` parameter in the query string or request body:

- A namespace-scoped Waterline acts only on its own namespace. This includes
  embedded Waterline with `WATERLINE_NAMESPACE` set and every service-mode
  deployment. A request for another namespace returns `404`.
- A cluster-wide embedded Waterline acts on the requested namespace. Without
  `namespace`, a `schedule_id` that exists in more than one namespace returns
  `409` with reason `schedule_namespace_required`, and nothing changes.

```bash
curl -sS -X POST \
  "$APP_URL/waterline/api/v2/schedules/nightly-invoice-sync/pause?namespace=billing" \
  -H "Accept: application/json" \
  -H "Content-Type: application/json" \
  -d '{"note": "Supplier maintenance"}' | jq '.status, .note'
```

### Schedule Rows

The list response is paginated with `data`, `current_page`, `last_page`,
`per_page`, `total` and `operator_scope`. Each row in `data` has the same
fields in embedded and service mode:

| Field | Meaning |
| --- | --- |
| `schedule_id` | The schedule's ID. Send it with `namespace` to every per-schedule route. |
| `id` | The same value as `schedule_id`, kept for older clients. |
| `namespace` | The schedule's namespace. |
| `workflow_type`, `workflow_class` | The workflow the schedule starts. Either can be `null`. |
| `status`, `paused` | `active`, `paused` or `deleted`, and whether the schedule is paused. |
| `note` | The operator note, or `null`. |
| `spec` | The cadence: `cron_expressions` (cron strings), `intervals` (objects with ISO 8601 `every` and optional `offset` durations) and `timezone`. |
| `overlap_policy` | `skip`, `buffer_one`, `buffer_all`, `cancel_other`, `terminate_other` or `allow_all`. |
| `next_fire_at`, `last_fired_at` | ISO 8601 timestamps, or `null`. |
| `last_outcome` | The most recent recorded fire result, or `null` when none was recorded. It has `outcome` (for example `scheduled`, `drained`, `backfilled`, `skipped` or `failed`), `at`, `reason`, `workflow_id` and `run_id`. |
| `fires_count`, `failures_count`, `skipped_trigger_count`, `remaining_actions` | Counters, or `null` when the backend does not report one. |
| `search_attributes` | The schedule's search attributes, or `null`. |
| `created_at`, `updated_at` | ISO 8601 timestamps. |

A row with `last_fired_at` and no `last_outcome` means the backend reported
the fire time without its result.

List rows and the detail, `pause` and `resume` responses also include
`action`, `max_runs`, `jitter_seconds`, `paused_at`, `last_skip_reason`,
`last_skipped_at`, `latest_workflow_instance_id`, `buffered_actions`, `memo`
and `recent_actions`. Each recent action has `outcome`, `fired_at`, `reason`,
`workflow_id`, `run_id` and `occurrence_time`. `latest_instance_id` repeats
`latest_workflow_instance_id` for older clients. A `spec` keeps any other keys
the backend stored after its cadence fields.

### Trigger And Backfill Results

`trigger` accepts an optional `overlap_policy`. It returns `schedule_id`,
`namespace`, `outcome` (`triggered`, `buffered`, `buffer_full` or `skipped`),
`triggered`, `workflow_id`, `run_id` and `reason`. `triggered` is `true` only
when a workflow started. `instance_id` repeats `workflow_id` for older clients.

`backfill` requires `from` and `to` ISO 8601 timestamps and accepts an optional
`overlap_policy`. It returns `schedule_id`, `namespace`, `fires_attempted`,
`started_count`, `skipped_count`, `failed_count` and `results`. Each result has
`fire_time`, `workflow_id`, `outcome` (`started`, `skipped` or `failed`) and
`reason`. For older clients, each result also repeats `schedule_id`, and
`cron_time`, `instance_id` and `error` repeat `fire_time`, `workflow_id` and a
failed result's `reason`. `backfill` refuses a window whose end is not after
its start with `422` and reason `invalid_time_range`.

Both routes refuse an unknown `overlap_policy` with `422` and reason
`invalid_overlap_policy`.

## Saved Views And Preferences

Saved views and preferences are UI-owned operator configuration, not workflow
history. They are still useful for stable runbooks because they let teams share
the exact filters and table layout used during an incident.

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/waterline/api/saved-views` | List saved operator views. |
| `POST` | `/waterline/api/saved-views` | Create a saved view. |
| `GET` | `/waterline/api/saved-views/{view}` | Read one saved view. |
| `PUT` | `/waterline/api/saved-views/{view}` | Update a saved view. |
| `DELETE` | `/waterline/api/saved-views/{view}` | Delete a saved view. |
| `GET` | `/waterline/api/preferences/{surface}` | Read UI preferences for one surface. |
| `PUT` | `/waterline/api/preferences/{surface}` | Update UI preferences for one surface. |

Do not treat saved-view names or preference payloads as durable workflow facts.
For evidence, cite selected-run detail or history export.

## Error Contract

Waterline uses ordinary HTTP status codes and JSON error details:

| Status | Meaning |
| --- | --- |
| `400` | Malformed request or unsupported action payload. |
| `404` | Workflow instance, run, update, schedule, saved view, or preference surface was not found in the current namespace. |
| `409` | The selected run cannot execute the action, often because it is historical, closed, replay-blocked, or definition-unavailable. A cluster-wide schedule request also returns `409` when its `schedule_id` needs a `namespace`. |
| `422` | Validation failed; response includes field-level errors. |
| `500` | Unexpected application failure. |

Automation should branch on status plus stable fields such as `blocked_reason`,
`query_blocked_reason`, `outcome`, `reason`, and `actionability` values. Do
not parse toast text or button labels as the contract.

## See Also

- [Monitoring](./monitoring.md)
- [Failures and Recovery](./failures-and-recovery.md)
- [Queries](./features/queries.md)
- [Cancel and Terminate](./features/cancel-and-terminate.md)
- [Agent Tooling Contract](./agent-tooling-contract.md)

<!-- Source: docs/operator-operating-envelope.md -->

# Operator Operating Envelope

This guide defines the operator-facing contract for Durable Workflow v2.
Use it to decide which diagnostics block rollouts, which ones are advisory,
which queue facts belong to Waterline versus worker telemetry, how to verify
rebuild and export workflows, and which deployment shapes are part of the
documented operating envelope.

The deployment and recovery procedures here apply to embedded Laravel and
self-hosted Server. Durable Workflow Cloud is a separate managed service: Cloud
owns runtime persistence, single-region placement, backup, restore, and
recovery, while
customers operate SDK clients and workers through the namespace runtime URL.
Do not use this guide to attach a Server to Cloud; use the
[Cloud Managed Runtime](/docs/polyglot/cloud-control-plane) guide for that
customer boundary.

## Source-of-truth surfaces

Use these surfaces together:

| Surface | Use it for | Contract class |
| --- | --- | --- |
| `php artisan workflow:v2:doctor --strict` | Backend capability gating before v2 traffic or upgrades | Blocking |
| `GET /waterline/api/v2/health` | Current engine-source readiness plus blocking vs advisory v2 health checks | Blocking when `status = error`, advisory when `status = warning` |
| `GET /waterline/api/stats` | Durable fleet totals, backlog counters, repair-loop facts, projection drift counts, worker compatibility summaries | Advisory and benchmarking |
| `php artisan workflow:v2:rebuild-projections ...` | Previewing and repairing projection drift | Maintenance |
| `php artisan workflow:v2:backfill-command-contracts ...` | Previewing and backfilling legacy command-contract snapshots | Maintenance |
| `php artisan workflow:v2:history-export ...` and Waterline history-export routes | Replay, archive handoff, and incident artifacts | Verification |
| Waterline archive actions and control-plane `archive()` | Lifecycle state transitions for closed runs | Lifecycle |
| Worker SDK metrics, traces, and logs | Schedule-to-start latency, poll success, sticky-cache behavior, and custom application telemetry | Runtime telemetry |

The durable-state operator contract lives in the runtime that owns each run.
Waterline projects that contract either from the embedded workflow package or
from a standalone server through the PHP SDK. Worker telemetry remains the
source of truth for latency and process-level behavior inside your workers.

### Surface mapping by deployment shape

The Waterline routes in the table above are available from both Waterline
deployment shapes. Embedded mode reads the Laravel host's durable state in
process. Service mode runs the published Waterline image and reads one
standalone-server namespace through the PHP SDK. The authenticated server API
and `dw` CLI remain available directly with or without Waterline:

| Operator question | Waterline (embedded or service mode) | Standalone-server native surface |
| --- | --- | --- |
| Engine-source readiness and blocking vs advisory health | `GET /waterline/api/v2/health` | `GET /api/system/health` (admin auth, control-plane v2); `dw server:health` for liveness and `dw server:info` for the topology, protocol, and rollout-safety summary |
| Durable fleet totals, backlog, repair, worker compatibility, projection drift | `GET /waterline/api/stats` | `GET /api/system/operator-metrics` and `dw system:operator-metrics` |
| Selected-run detail and history export | `GET /waterline/api/instances/...` and `/waterline/api/.../history-export` | `GET /api/workflows/{workflowId}`, `/runs/{runId}`, and `/runs/{runId}/history/export` (see the [Server API Reference](./polyglot/server-api-reference.md)) |
| Operator commands (cancel, terminate, repair, archive, signal/update/query) | `POST /waterline/api/instances/.../{cancel\|terminate\|repair\|archive}` and signal/update/query routes | `POST /api/workflows/{workflowId}/{cancel\|terminate\|repair\|archive}` and `POST /api/system/repair/pass` |
| Topology and node-identity discovery | Embedded: `php artisan workflow:v2:doctor --json` (`topology` object); service: use the connected server's native surface | `GET /api/cluster/info`, `GET /api/health`, `GET /api/ready` (or `dw server:info`) |

The field families and contract names below stay the same regardless of
which surface you read them through. A Waterline deployment is scoped to its
configured runtime and namespace; it does not combine embedded and
server-managed runs.

## Dated 2.0 evidence snapshot

This snapshot separates behavior that has been measured on released artifacts
from procedures that the product supports but each operator must rehearse in
their own environment. A passing row applies to the documented 2.0 scenario
and topology; it is not a blanket claim for different substrates or larger
deployments.

The current proof inventory is:

| Proof and date (UTC) | What the proof directly measured | Boundary |
| --- | --- | --- |
| Single-region failover rehearsal, 2026-07-28 | Cross-node terminal completion, loss of one API node, MySQL interruption and recovery, Redis interruption and database-poll fallback, worker lease loss and reclaim, and singleton scheduler restart. The result recorded passing loss and duplicate assertions for every phase. | Two API nodes behind one shared endpoint, one MySQL instance, one Redis acceleration layer, one queue worker, and one scheduler/maintenance runner in one region. The database and Redis events were container interruptions, not managed-provider promotion tests. |
| Timer restart matrix, 2026-07-28 | Sleeping workflows completed after worker restart and server restart, with one timer schedule and fire per run and no duplicate replay command. | Durable timer and replay recovery on the published-server test topology; not arbitrary process state or external side-effect recovery. |
| Activity runtime matrix, 2026-07-29 | Worker-restart result recovery, retry and timeout behavior, heartbeat and cancellation observation, durable terminal result recording, and idempotent completion handling across the required product surfaces. | Activity-level engine behavior. Applications still own idempotency for external side effects. |
| Workflow lifecycle matrix, 2026-07-29 | Terminal completion and failure states, duplicate-start and workflow-id reuse policy, timeout and retry behavior, history continuity, and duplicate side-effect prevention across PHP, Python, Rust, CLI, API, history, and Waterline. | Lifecycle correctness for the exercised scenarios; not an availability or throughput benchmark. |

Use the [Compatibility contract](./compatibility.md) for supported release
lines, and rehearse the documented recovery procedure against your deployment
before making a topology-specific failover claim.

### Measured guarantee vs supported procedure

| Failure or outcome | Directly proven by the dated evidence | Supported operator procedure | Not established by this evidence |
| --- | --- | --- | --- |
| One API-node loss | The failover rehearsal killed one of two API nodes, reached the surviving node through the shared endpoint, preserved acknowledged state, and completed the run. | Remove failed nodes with `/api/ready`, retry interrupted client requests, then verify topology and worker registration before returning traffic. | Whole-fleet loss, availability-zone isolation, or arbitrary network partition behavior. |
| Database interruption | Both API nodes became not ready, durable state survived the interruption, a replacement worker reclaimed the task after lease expiry, the run completed, and a duplicate completion was refused. | Restore the writable database first, wait for readiness, fence the stale task owner, verify one representative run, then resume traffic. Managed-service promotion needs provider-native fencing, RPO, and elapsed-time evidence in the recovery packet. | Managed database promotion, acknowledged-write behavior during split brain, or an RPO/RTO beyond the measured local interruption. |
| Redis interruption | Readiness reported acceleration degradation while database-backed polling preserved durable state; replaying the same poll request did not create a duplicate lease; readiness recovered after Redis returned. | Keep Redis as an acceleration layer, tolerate warning readiness, restore it after durable persistence, and verify poll-discovery latency returns to the deployment baseline. | Redis dual-primary behavior, cache behavior across a network partition, or provider promotion timing. |
| Worker restart or loss | The failover proof reclaimed and completed a leased workflow after worker loss; the newer activity matrix proved durable result recovery after worker restart and idempotent completion handling. | Let leases expire or drain workers, start a compatible replacement, verify registration and queue pickup, and keep external effects idempotent. | Recovery of process-local state that was not stored durably or exactly-once external side effects. |
| Server or scheduler restart | The timer matrix completed sleeping workflows after a server restart without duplicate timer commands. The failover proof also preserved progress through one API-node loss and one singleton scheduler restart. | Restore persistence, bring server roles back to readiness, confirm exactly one scheduler/maintenance runner, then verify worker registration and one representative completion. | Simultaneous loss of every server and persistence node, multi-region failover, or duplicate schedulers as a steady-state topology. |
| Duplicate delivery | Database-recovery and worker-loss phases each recorded one logical completion and rejected the duplicate completion with HTTP `409`; Redis degradation replayed the original lease without issuing a duplicate. The activity matrix separately passed its idempotent-completion cell. | Use stable request and task identities, honor stale-attempt refusals, and make activity side effects idempotent. | A universal exactly-once delivery or side-effect guarantee outside the durable engine boundary. |
| Terminal completion | Cross-node, API-loss, database-recovery, and worker-loss workflows all reached terminal `completed` state; the newer lifecycle matrix passed completion, failure, cancellation, termination, timeout, and retry cells. | Verify the selected run and its history/export evidence before archiving or declaring recovery complete. | Completion of workload shapes, integrations, or failure combinations that were not exercised. |

### Largest current load cell

The largest configured server load cell starts **1,000 workflows** with start
concurrency **8** while also running **8** concurrent polling workers for 120
seconds. It requires at least a 98% workflow-start completion ratio and checks
endpoint availability, cache-key drainage, sample coverage, and bounded
resource ceilings.

Treat this as a short **correctness smoke**, not a sustained production-scale
benchmark. It does not establish a universal throughput, latency, memory, or
maximum-concurrency promise. Establish those values with the benchmark and
long-soak packet for the exact topology, worker mix, database, cache, and
release you intend to operate.

### Explicitly out of scope

The 2.0 evidence claim does not cover:

- network partitions
- deliberate clock drift
- multi-region operation
- split-brain behavior

Those experiments are not release prerequisites for 2.0. They are prerequisites
only for making the corresponding partition, clock-skew, multi-region, or
split-brain claim. The supported 2.0 release envelope remains the published
topology plus its recovery packet and the behavior measured by the documented
scenarios; unsupported chaos scenarios should not be presented as
evidence that the documented single-region procedures are unusable.

## Supported topologies

Durable Workflow v2 supports these operator shapes. The shape names in the
first column match the `topology.current_shape` values published by
`/api/cluster/info` and the [Server Role Topology](./polyglot/server-role-topology.md)
manifest, so the operator contract here lines up with the discovery contract
your automation already reads.

| Operator shape (`topology.current_shape`) | Supported operator contract | Primary failure domains | Recovery and failover expectation |
| --- | --- | --- | --- |
| `embedded`, single node | Waterline, control-plane routes, health, rebuild, export, and archive all run from one app process against one durable database and one cache store. | The Laravel app process, the durable database, and the cache store on one host. | Treat host or database loss as a full service interruption. Restore durable state first, bring one app node back to readiness, then verify worker registration before resuming traffic. |
| `embedded`, small same-region cluster | Use one shared database, one shared cache backend for wake-signal coordination, identical workflow compatibility/config across nodes, and keep active nodes in the same datacenter or region so queue wake-up and timer wake-up latency stay bounded. | Shared database, shared cache/wake coordination, load balancer routing, and the singleton scheduler or maintenance role. | One app-node loss should reduce capacity, not correctness. Database loss blocks durable traffic; Redis-only loss degrades wake acceleration and reports a readiness warning while database polling preserves durable correctness. Scheduler failover and upgrades remain explicit operator procedures rather than automatic HA promises. |
| `standalone_server` distribution | Use the [Self-Hosting Deployments](./deployment.md) guide for the server-specific deployment matrix, then apply the same health, stats, export, archive, and queue-health distinctions through the native `/api/system/...` and `/api/workflows/...` routes or a separately deployed Waterline service (see the surface mapping above). | Shared database, shared Redis, API container set, independently scaled workers, the optional Waterline observer, and the single scheduler or maintenance runner. | API containers are replaceable server process nodes, and the Waterline container is a replaceable observer of server-owned state. The database, Redis, and singleton scheduler path define recovery order. Restore persistence first, then verify `/api/ready`, `/api/cluster/info`, and worker registration before shifting traffic back. |
| `split_control_execution` | Same product contract as `standalone_server`, with each role isolated into its own process class (`ingress_node`, `control_plane_node`, `scheduler_node`, `matching_node`, `execution_node`). The same operator-metrics, health, and command surfaces apply per-node; route admin reads to the node that hosts the role you are interrogating. | Each role runs as its own process class, so the failure-domain checklist in [Server Role Topology](./polyglot/server-role-topology.md) governs which subsystem fails first. The shared database, Redis, and singleton scheduler election remain fleet-wide failure domains. | Recovery follows the same order as `standalone_server`, but verify `topology.current_shape`, `topology.current_process_class`, and `topology.current_roles` per node before declaring the deployment ready. Hosted routes return `503 topology_role_unavailable` when sent to the wrong node class. |

`split_control_execution` is not a separate engine or product. It is the same
operator contract as `standalone_server` with the role-specific process classes
named in `topology.shape_assignments`. Treat the rest of this guide as
shape-agnostic for those two server shapes unless a section calls out a
specific role. [Server Role Topology](./polyglot/server-role-topology.md)
holds the role vocabulary, authority boundaries, and migration path.

Waterline service mode is an observer deployment, not another server topology.
Adding or removing it does not change `topology.current_shape`, server-native
API or CLI availability, or the runtime that owns a run.

Publish the restore order, backup cadence, expected failover lag, and any
region-pinned behavior in the runbook for the topology you operate. The product
contract tells you which facts to measure; your deployment contract records the
recovery timing, manual steps, and failure domains you accept.

### Failure-domain checklist by supported shape

Use the topology table above as the quick summary, then write your runbook
against these more explicit loss models:

- **Embedded Laravel, single node**: One application process owns the control
  plane, matching, projection, scheduler, and execution roles together. Losing
  that process is a full service interruption for durable commands, workflow
  progress, schedule firing, and operator reads until the same app returns to
  readiness against intact durable storage.
- **Embedded Laravel, small same-region cluster**: Losing one ordinary app node
  should remove only a share of HTTP and worker capacity while the remaining
  nodes keep claiming work from the shared durable store. Treat the shared
  database, the shared cache-backed wake path, and whichever node currently
  owns the singleton scheduler or maintenance duty as the main correctness
  boundaries for the fleet.
- **Standalone server distribution (`standalone_server`)**: Losing one
  `server_http_node` should stop ingress and control-plane commands only on
  that node; healthy worker nodes can still finish leased work and other API
  nodes can keep serving traffic. Losing one `worker_node` should raise
  backlog, queue age, or compatibility warnings only for the affected
  `(connection, queue, compatibility)` scopes. Losing the `scheduler_node`
  should pause new schedule fires and maintenance sweeps without invalidating
  already running workflows. Database loss is a fleet-level outage. Redis-only
  loss keeps durable database polling available, reports
  `long_poll_wake_acceleration` as degraded, and increases discovery latency
  until Redis reconnects.
- **Split-role server distribution (`split_control_execution`)**: Each role
  runs as its own process class — `ingress_node`, `control_plane_node`,
  `scheduler_node`, `matching_node`, and `execution_node`. Losing any one
  process class only degrades the role it owns: ingress loss stops external
  HTTP traffic at the edge, control-plane loss makes operator commands fail
  fast while leased work continues, matching loss falls back to direct
  ready-task discovery, scheduler loss pauses schedule fires and records
  missed runs, and execution loss accumulates ready tasks without losing
  durable state. Database loss remains a fleet-wide outage. Redis-only loss is
  an acceleration-layer degradation with the same database-poll fallback and
  warning readiness behavior as the `standalone_server` shape.

If your deployment depends on different assumptions, treat that topology as a
separate runbook with its own validated contract instead of assuming the
self-serve guidance still applies unchanged.

### Published recovery packet by topology

The supported topologies above are only production-ready when the deployment
runbook publishes the matching recovery packet alongside them:

| Topology | Publish these operator-owned facts |
| --- | --- |
| `embedded`, single node | Backup schedule for the database, cache-preservation expectations, the exact app revision and env/config snapshot used for restore, the maximum accepted restore lag, and the latest successful restore rehearsal evidence. |
| `embedded`, small same-region cluster | Everything from the single-node packet, plus which node or process currently owns scheduler or maintenance duty, the expected impact of losing one ordinary node versus losing the shared database or cache backend, and the failover steps required to restore queue wake coordination. |
| `standalone_server` distribution | Database and Redis backup cadence, pinned server image or digest, auth-material location, the expected failover behavior for `server_http_node`, `worker_node`, and `scheduler_node`, the latest `/api/ready` plus `/api/cluster/info` restore verification evidence, and the latest worker re-registration proof after restore. |
| `split_control_execution` distribution | Everything from the `standalone_server` packet, plus the per-process-class scaling and failure expectations for `ingress_node`, `control_plane_node`, `scheduler_node`, `matching_node`, and `execution_node`, and the routing rules clients use when a hosted route returns `503 topology_role_unavailable` from the wrong node class. |

If that packet is missing, stale, or untested, treat the topology as
development-grade regardless of how many nodes are currently running.

### Verify live topology identity before trusting the baseline

For standalone-server and split-role deployments, confirm the node identity
that the product itself reports before you interpret queue, scheduler, or role
failure signals. `GET /api/cluster/info` is the source of truth for that
identity:

| Field | Use it for |
| --- | --- |
| `topology.current_shape` | Confirms whether the node is currently advertising `embedded`, `standalone_server`, or `split_control_execution`. |
| `topology.current_roles` | Confirms the logical roles actually hosted by this node. |
| `topology.supported_shapes` | Confirms which deployment shapes the current server build publicly supports. |
| `topology.shape_assignments` | Maps each supported shape to its documented process-class role bundles so you can compare the current role bundle against the supported topology. |

Use those fields as the first topology-drift check during rollouts:

- In the self-serve standalone-server shape, API nodes should continue to
  report the `api_ingress`, `control_plane`, `matching`, and
  `history_projection` role bundle; scheduler nodes should report `scheduler`;
  worker nodes should report `execution_plane`.
- In the split-role shape, verify that each node's `current_roles` match one of
  the documented role bundles under `shape_assignments` before you interpret
  backlog or scheduler lag as a worker problem.
- If `current_roles` drift from the deployment plan,
  treat queue and failover baselines as suspect until the node identity is
  corrected.

Embedded installs do not publish `/api/cluster/info`. For the package-local
topology view, run `php artisan workflow:v2:doctor --json` and inspect the
`topology` object. It publishes the same role-topology schema and includes the
embedded app's `current_shape`, `current_process_class`, `current_roles`,
`execution_mode`, and nested `matching_role` summary.

## Blocking and advisory diagnostics

Durable Workflow v2 separates blocking diagnostics from advisory diagnostics.

| Severity | Meaning | Typical operator action |
| --- | --- | --- |
| Blocking | The current configuration or readiness state is not safe to trust for v2 traffic | Stop rollout, fix the prerequisite, rerun verification |
| Advisory | The surface remains readable, but some derived facts need rebuild, backfill, or manual review before you rely on them | Keep serving traffic when appropriate, then repair the named surface |
| Healthy | No current issue was found in that surface | Continue normal operation |

Apply that rule to the shipped surfaces:

- `workflow:v2:doctor --strict` blocks when backend capability issues have
  `error` severity. Examples include an unsupported queue driver in queue mode
  or a cache store without locks. Informational queue diagnostics in poll mode
  remain advisory.
- `GET /waterline/api/v2/health` returns:
  - `status = ok` when the v2 operator surface is ready and the current checks
    are aligned.
  - `status = warning` when the surface remains readable but specific facts
    need rebuild, backfill, or repair before you trust them fully.
  - `status = error` with HTTP `503` when the engine-source bridge is not ready
    or a blocking capability problem makes the v2 surface unavailable.
- `GET /waterline/api/stats` publishes durable operator facts. Treat those JSON
  fields as operator diagnostics for dashboards and scripts, not as a metrics
  scrape endpoint.

## Correctness vs acceleration checks

Every v2 health check carries a `category` of either `correctness` or
`acceleration`, and the snapshot publishes a per-category rollup so operators
can answer two separate questions without re-aggregating the check list.

- **Correctness checks** describe whether durable ready-task discovery,
  projection freshness, command-contract backfill, history retention, worker
  compatibility, and backend capabilities are intact. A correctness check in
  `status = error` means safe task pickup or operator-trusted state is at
  risk; rollouts should stop until it clears.
- **Acceleration checks** describe whether optional wake-signal propagation
  is keeping up. The durable pollers are the correctness path, so an
  acceleration check in `status = warning` means cross-node wake-up latency
  may be higher than steady state but no task is stranded.

Each entry under `checks` carries its `category`, and the snapshot adds a
`categories` rollup so dashboards can summarize both questions at a glance:

```json
{
  "status": "warning",
  "categories": {
    "correctness": {"status": "ok", "check_count": 8},
    "acceleration": {"status": "warning", "check_count": 1}
  }
}
```

Treat a degraded `acceleration` rollup as acceleration-only: investigate
cache or wake backend health, but do not block traffic that depends only on
durable ready-task discovery. A degraded `correctness` rollup is the
blocking signal. The `long_poll_wake_acceleration` check is the canonical
acceleration entry and never escalates above `warning`; every other check
is a correctness entry.

## Queue-health semantics

Queue health is split between durable queue state and worker/runtime telemetry.

### Durable queue facts

Use Waterline dashboard stats and queue views for durable task state:

| Fact | Meaning |
| --- | --- |
| `operator_metrics.backlog.runnable_tasks` | Durable tasks that are ready to be claimed now. |
| `operator_metrics.backlog.delayed_tasks` | Durable tasks that exist but are still waiting for `available_at`. |
| `operator_metrics.backlog.leased_tasks` | Durable tasks currently claimed by a worker. |
| `operator_metrics.backlog.tasks_added_last_minute` | Distinct durable task rows created in the trailing 60 seconds. Treat this as durable queue inflow, not as a transport-attempt counter. |
| `operator_metrics.backlog.tasks_dispatched_last_minute` | Distinct durable task rows whose latest successful `last_dispatched_at` landed in the trailing 60 seconds. Compare it with `tasks_added_last_minute` to tell whether durable inflow is outrunning dispatch. |
| `operator_metrics.starts.pending_runs`, `operator_metrics.starts.pending_commands`, `operator_metrics.starts.ready_tasks`, `operator_metrics.starts.oldest_pending_start_at`, `operator_metrics.starts.max_pending_ms` | Durable workflow-start backlog. Use these facts to distinguish starts that have been accepted but have not yet become active workflow-task work from ordinary worker-side queue lag. |
| `operator_metrics.tasks.oldest_ready_due_at`, `operator_metrics.tasks.max_ready_due_age_ms` | The oldest currently actionable task and its ready-to-dispatch age. This is the machine-readable backlog-latency pair behind "oldest ready task". |
| `operator_metrics.tasks.dispatch_overdue`, `operator_metrics.tasks.oldest_dispatch_overdue_since`, `operator_metrics.tasks.max_dispatch_overdue_age_ms` | Ready durable tasks that still have no successful dispatch wake plus the age of the stalest example. Use these facts to spot degraded notifier acceleration without confusing it for ordinary queue growth. |
| `operator_metrics.backlog.unhealthy_tasks` | Durable tasks with dispatch failure, claim failure, overdue dispatch, or expired lease state. |
| `operator_metrics.backlog.repair_needed_runs` | Open runs that do not currently have a trusted durable resume path. |
| `operator_metrics.tasks.oldest_lease_expired_at`, `operator_metrics.tasks.max_lease_expired_age_ms` | The oldest expired lease and its age. Use this pair as the primary stuck-lease and duplicate-risk age indicator. |
| `operator_metrics.backlog.oldest_compatibility_blocked_started_at`, `operator_metrics.backlog.max_compatibility_blocked_age_ms` | The oldest compatibility routing block and its age. Use this when work is preserved but no compatible worker is currently eligible to claim it. |
| Active vs stale pollers | Whether registered workers are still heartbeating for a queue. |
| Current leases | Which workflow or activity tasks are leased right now and whether the lease is expired. |

These facts describe durable workflow-task and activity-task traffic only.

When you need queue-local drill-down instead of fleet totals, use the server
task-queue visibility routes for backlog age, poller state, current leases,
admission budgets, `stats.tasks_added_last_minute`, and
`stats.tasks_dispatched_last_minute`. Compare queue-local inflow with dispatch
to identify the queue building backlog or lacking available worker capacity.
Use the fleet-level `operator_metrics.backlog.*` pair above for the aggregate
comparison.

Waterline's `GET /waterline/api/v2/health` surface publishes the same queue
drill-down under `queue_visibility.*` for the configured namespace. Treat these
field families as the typed queue-health contract:

| Field family | Meaning |
| --- | --- |
| `queue_visibility.available`, `queue_visibility.reason` | Whether Waterline can currently produce queue-local visibility for the configured namespace, and why not when it cannot. |
| `queue_visibility.task_queues[].stats.approximate_backlog_count`, `queue_visibility.task_queues[].stats.approximate_backlog_age` | Queue-local backlog count and oldest durable backlog age. |
| `queue_visibility.task_queues[].stats.tasks_added_last_minute`, `queue_visibility.task_queues[].stats.tasks_dispatched_last_minute` | Per-queue durable inflow versus dispatch over the trailing 60 seconds. Use these when one hot queue is hidden inside healthy fleet totals. |
| `queue_visibility.task_queues[].stats.pollers.active_count`, `queue_visibility.task_queues[].stats.pollers.stale_count`, `queue_visibility.task_queues[].stats.pollers.stale_after_seconds` | Healthy versus stale pollers on that queue and the stale-heartbeat threshold the snapshot used. |
| `queue_visibility.task_queues[].stats.workflow_tasks.*`, `queue_visibility.task_queues[].stats.activity_tasks.*` | Queue-local ready, leased, and expired-lease counts split by workflow-task versus activity-task traffic. |
| `queue_visibility.task_queues[].repair.candidates`, `dispatch_failed`, `expired_leases`, `dispatch_overdue` | Queue-local repair pressure: durable tasks that already need repair, are dispatch-failed, hold expired leases, or are overdue for redispatch. |
| `queue_visibility.task_queues[].repair.oldest_dispatch_failed_at`, `max_dispatch_failed_age_ms`, `oldest_lease_expired_at`, `max_lease_expired_age_ms`, `oldest_dispatch_overdue_since`, `max_dispatch_overdue_age_ms` | Queue-local age signals for the stalest dispatch failure, expired lease, and dispatch-overdue durable task. |

`coordination_alerts[]` on the same `GET /waterline/api/v2/health` payload is
the operator roll-up for those queue-local facts plus the health-check list.
Use it as the page-ready summary for warnings and errors, then drill into the
matching `queue_visibility` or `checks` entries for evidence.

Treat the queue-local admission status as the first-class slot and poller
signal for that queue. `saturated` means live workers are present but every
registered slot is already leased. `throttled` means a server-side lease or
dispatch cap is intentionally holding new work. `no_slots` means workers are
registered but exposed zero capacity for that task kind. `no_active_workers`
means the queue has no healthy poller at all, and `unavailable` means a
configured lock-backed admission guard cannot currently prove safety.

Use `operator_metrics.starts.*` when new workflow starts appear stuck even
though steady-state queue lag looks normal. Those facts separate control-plane
start admission and first-task creation debt from downstream worker pickup.

### Poller pressure and admission budgets

Use task-queue detail routes or `dw task-queue:describe` when queue flow is
degrading and you need to separate "not enough worker capacity" from
"intentional server throttling" or "no live poller at all":

| Queue status | Meaning | Treat it as |
| --- | --- | --- |
| `accepting` | Workers still have available slots and no server cap is full. | Healthy baseline. |
| `saturated` | All registered worker slots are currently leased. | Worker-capacity pressure. |
| `throttled` | A server-side active-lease or dispatch-rate cap is intentionally holding the queue back. | Advisory unless the cap is unexpected or the backlog keeps growing beyond the published baseline. |
| `no_slots` | Active workers are registered, but none advertise slots for that task kind. | Blocking for that queue. |
| `no_active_workers` | No healthy poller is currently serving the queue. | Blocking for that queue. |
| `unavailable` | The queue cannot acquire the lock needed for its configured admission path. | Blocking until the admission dependency recovers. |

Use these statuses with the queue-flow facts together:

- `tasks_added_last_minute > tasks_dispatched_last_minute` plus `saturated`
  means durable inflow is outrunning worker capacity.
- The same rate imbalance plus `throttled` means the queue is being held back
  by an explicit server cap and should be judged against that cap's intended
  contract, not against unrestricted throughput.
- A rising oldest-ready age plus `no_active_workers` or stale pollers means the
  queue has lost healthy claimers and should be treated as a routing outage for
  that scope.

### Matching-role deployment shape

Use `operator_metrics.matching_role.*` when you need to confirm which
matching/dispatch contract the current node is actually serving:

| Fact | Meaning |
| --- | --- |
| `operator_metrics.matching_role.queue_wake_enabled` | Whether this node still runs the in-worker broad-poll wake path on queue-worker loop events. |
| `operator_metrics.matching_role.shape` | `in_worker` when the node still owns that wake path, `dedicated` when the wake/repair sweep is expected to run under a separate `workflow:v2:repair-pass --loop` process. |
| `operator_metrics.matching_role.task_dispatch_mode` | The dispatch mode this node is using for ready tasks: `queue` or `poll`. |
| `operator_metrics.matching_role.partition_primitives` | The frozen routing axes, in order: `connection`, `queue`, `compatibility`, `namespace`. |
| `operator_metrics.matching_role.backpressure_model` | The durable admission boundary the engine enforces. Current v2 reports `lease_ownership`. |

These fields are node-local, not fleet-wide. In a mixed-shape rollout, read
the snapshot from each node or pod you are cutting over so you can confirm the
matching role moved where you intended before you interpret backlog or poller
changes as worker health.

### Worker and SDK telemetry

Use worker metrics, traces, and logs for:

- Workflow and activity `schedule_to_start` latency when the exact Server/SDK pair is qualified
- Poll success rate and sync/eager-dispatch behavior
- Sticky-cache size and eviction behavior
- Worker CPU, memory, thread, and event-loop pressure
- Custom application metrics emitted from activities or worker code

The shared recorder name is `durable_workflow_worker_task_schedule_to_start_seconds`,
a histogram in seconds with exactly `task_kind` and `task_queue` tags. The
[revision-2 contract](https://github.com/durable-workflow/.github/blob/23f9bc3c3798cb6aab6ccf5f9cd210854b820c65/sdk-metrics/README.md#schedule-to-start-observations)
requires Boolean `server_capabilities.claimed_task_ready_age: true` and a valid
integer `task.ready_age_ms`. At actual workflow/activity dispatch, the SDK records
the Server-computed eligibility-to-lease duration, excluding delivery, processing,
scheduled delay and retry backoff. Do not subtract worker and Server clocks.
Missing capability, value or known ready origin means no sample, not zero.
A valid zero remains a sample.

Python's [source hook](https://github.com/durable-workflow/sdk-python/pull/264)
is merged but unreleased. Server publication and installed/exported acceptance
remain pending in [Python #188](https://github.com/durable-workflow/sdk-python/issues/188).
The existing request, poll and processing metrics do not supply this histogram.
Until the exact pair is qualified, alert on
`operator_metrics.tasks.oldest_ready_due_at` and
`operator_metrics.tasks.max_ready_due_age_ms` where available. These aggregate
backlog signals do not measure per-task latency or replace its distribution.

Synchronous queries, live-debug tooling, and other non-durable control-plane
calls should be labeled separately in your dashboards. They do not count as
durable task backlog and they do not change Waterline repair counters.

## Worker compatibility and rollout health

`operator_metrics.workers` publishes the compatibility facts that determine
whether the active worker fleet can safely handle the required workflow
contract:

| Fact | Meaning |
| --- | --- |
| `operator_metrics.workers.required_compatibility` | Compatibility markers a worker must advertise to be eligible for work in the namespace. |
| `operator_metrics.workers.active_workers` | Count of distinct live workers seen through compatibility heartbeat. |
| `operator_metrics.workers.active_worker_scopes` | Count of `(connection, queue)` scopes covered by those workers. |
| `operator_metrics.workers.active_workers_supporting_required` | Workers whose advertised compatibility covers the required markers. |
| `operator_metrics.workers.fleet` | Per-scope list of every active worker with `worker_id`, `connection`, `queue`, advertised `supported` markers, a `supports_required` flag, the heartbeat `source` (`database` or `cache`), and `recorded_at`. |

Use the summary counts to detect rollout states where some workers cannot
safely claim the required work, and drill into `fleet` to identify
exactly which `(connection, queue)` scope is missing coverage. The Waterline
operator dashboard renders the same fleet list under its worker
compatibility panel so operators do not need to query the metric surface by
hand.

When `active_workers_supporting_required` reaches zero for a namespace,
Waterline surfaces a `no_compatible_worker_for_task` run diagnostic on
affected runs so the gap is visible on the run-detail view as well as the
metric surface. The companion `worker_compatibility` health check fires as
`warning` under `correctness` in the same condition, which flips the
`correctness` category rollup to `warning` so the fleet gap is visible at
a glance and not buried inside the check list.

See [Rolling Out Worker Builds With Build IDs](./polyglot/worker-build-id-rollout.md)
for the drain/resume flow that coordinates with these facts during a
build-id rollout, and
[Worker Compatibility and Routing](./polyglot/worker-compatibility-routing.md)
for the pinning contract behind those diagnostics.

## Alert semantics

Alert thresholds are deployment-specific. Publish your own numeric baselines
for queue age, repair lag, worker coverage, and restore timing, then alert when
the contract below stays breached longer than one normal repair or watchdog
window for the topology you operate.

| Alert family | Source | Treat as | Escalate when | Operator response |
| --- | --- | --- | --- | --- |
| Blocking readiness | `workflow:v2:doctor --strict`, `GET /waterline/api/v2/health` | Blocking | `doctor --strict` returns an error or the health endpoint returns `status = error` / HTTP `503` | Stop rollout or traffic shift, fix the blocking prerequisite, then rerun readiness and compatibility checks. |
| Compatible-worker coverage | `operator_metrics.workers.*`, `worker_compatibility` health check, run diagnostic `no_compatible_worker_for_task` | Blocking | `active_workers_supporting_required = 0` for a namespace or required `(connection, queue)` scope | Drain incompatible workers, register compatible workers, and confirm the `correctness` rollup clears before trusting new claims. |
| Durable queue lag | Waterline queue views, `operator_metrics.backlog.*`, qualified worker `schedule_to_start` telemetry | Blocking when sustained; advisory when brief | The oldest ready-task age or schedule-to-start latency stays above the published topology baseline while compatible workers are available | Add worker capacity, inspect task-queue admission limits, and verify the scheduler or matching path is still making forward progress. |
| Poller pressure and admission saturation | Task-queue detail routes, `dw task-queue:describe`, queue `status`, stale pollers, and queue-local add/dispatch rates | Blocking for `no_active_workers`, `no_slots`, or `unavailable`; advisory for intentional `throttled` states | One queue stays `saturated` while its oldest-ready age and add-vs-dispatch gap keep growing, or any queue flips to `no_active_workers`, `no_slots`, or `unavailable` outside a planned maintenance window | Add worker slots, restore the missing poller cohort, or confirm the server-side cap and lock dependency are behaving as designed before you scale blindly. |
| Workflow-start backlog | `operator_metrics.starts.*`, control-plane start telemetry, qualified worker `schedule_to_start` telemetry for first workflow tasks | Blocking when sustained; advisory when brief | `pending_commands`, `ready_tasks`, or `max_pending_ms` stay above the published topology baseline while compatible workers and queue capacity are available | Inspect the start boundary end to end: confirm start commands are turning into durable tasks, verify matching or dispatch is creating the first task promptly, and separate start-path debt from general worker lag before scaling. |
| Projection drift and repair debt | `run_summary_projection` / `selected_run_projections` health checks, `operator_metrics.repair.*` | Advisory | Drift warnings persist past one planned rebuild window or the max candidate age keeps climbing | Run the rebuild or repair previews, execute the repair, then verify the warning clears and stale ages return to baseline. |
| Retry or failure storm | `operator_metrics.backlog.unhealthy_tasks`, durable run diagnostics, worker error telemetry | Advisory, escalating to blocking if it prevents durable progress | Dispatch-failed, claim-failed, expired-lease, or retry-exhaustion facts climb above the topology baseline and stay elevated | Inspect the failing task family, compare worker telemetry with durable error facts, and decide whether to drain traffic or isolate the affected queue. |
| Wake acceleration degradation | `long_poll_wake_acceleration` health check and the `acceleration` category rollup | Advisory | The acceleration warning persists after cache or notifier maintenance windows | Investigate cache or wake propagation health. Do not treat this as a correctness outage unless the `correctness` rollup also degrades. |

The goal is to page on durable contract risk, not on every transient signal.
Queue and worker alerts should only become blocking when they threaten the
operator contract for the topology you actually run.

## Rebuild, repair, and restore expectations

Use these checks in order when the operator surface reports drift:

1. Check `GET /waterline/api/v2/health`.
   - `run_summary_projection` and `selected_run_projections` warnings mean
     Waterline can still answer, but some list or detail facts need rebuild.
   - `command_contract_snapshots` warnings mean some legacy runs still need
     WorkflowStarted contract backfill before operators can trust declared
     signal, update, or query forms.
   - `durable_resume_paths` warnings mean open runs need repair before you rely
     on their projected next resume source.
2. Preview projection work with:

   ```bash
   php artisan workflow:v2:rebuild-projections --needs-rebuild --dry-run
   ```

3. Rebuild the affected projections:

   ```bash
   php artisan workflow:v2:rebuild-projections --needs-rebuild
   ```

4. Preview command-contract backfill work with:

   ```bash
   php artisan workflow:v2:backfill-command-contracts --dry-run
   ```

5. Backfill command contracts when the current workflow class is still
   available:

   ```bash
   php artisan workflow:v2:backfill-command-contracts
   ```

6. Use `--prune-stale` only after your retention workflow has intentionally
   removed durable rows and you want to delete projection rows whose durable run
   or history row no longer exists.

`operator_metrics.repair.*` publishes the repair-loop sweep footprint. Use the
candidate counts, selected counts, maximum candidate age, and scan-limit
pressure to decide whether repair work is comfortably within your baseline or
needs capacity investigation.

### Retention and projection drift

With Workflow 2.6.3 or later, a retained run marked `details_pruned_at` does not
count as a missing run summary, rebuild debt or missing-summary age. The run
still contributes to the drift report's `runs` total. Every projection rebuild mode skips
it, including explicit run or instance selection, so a repair does not restore
its deleted list row or return it to retention selection.

An unmarked run without a summary still reports drift. Empty history alone
does not prove that an older retention job pruned it. Confirm intentional
cleanup from an authoritative retention record before choosing remediation.
Projection rebuild cannot recover deleted history. Use an appropriate backup
when that history is required.

Earlier runtimes can falsely report marked pruned runs as missing summaries.
Upgrade the runtime before using a rebuild to address that retention warning.
After a retention pass, verify that `run_summary_projection` is `ok`,
`projections.run_summaries.needs_rebuild` is zero, and later retention passes
do not select the same pruned runs. Genuine projection loss still needs repair.

## Export and archive verification

History export and archive serve different purposes:

- **History export** creates a replay/debug/archive artifact.
- **Archive** marks a closed run as archived so it leaves active fleet views.
- **Prune** removes projection or durable rows after retention has definitely
  expired.

Use this verification sequence:

1. Export the selected run:

   ```bash
   php artisan workflow:v2:history-export <workflow-instance-id> --run-id=<workflow-run-id> --output=storage/app/workflow-history/run.json --pretty
   ```

2. Verify the bundle includes the expected run id, schema version, and any
   configured redaction metadata.
3. Verify referenced external bytes and retain them for the artifact's required
   lifetime. Follow the [export-before-prune procedure](./configuration/pruning-workflows.md#export-before-pruning)
   to inventory every closed run and detect missing copies.
4. Archive only when you intend to retain the run in the database. Server
   retention skips archived runs and provides no unarchive API. Embedded
   cleanup is application-owned and does not require archival first.
5. Prune eligible details through the owning retention process. Keep the run
   and instance tombstones. A projection rebuild does not restore pruned
   history, and marked pruned runs are excluded from rebuilding.

Checksum and structure checks do not prove that an export contains all original
events or independent external bytes. Reject already-pruned sources; the false
complete-export flag remains an open defect in
[Workflow #856](https://github.com/durable-workflow/workflow/issues/856).
The self-contained verified archive operation requested in
[Server #520](https://github.com/durable-workflow/server/issues/520) is also
not delivered by this manual procedure. Keep those product gaps separate from
the recovery checks your deployment actually passed.

For Waterline users, the matching history-export and archive routes are listed
in the [Waterline Operator API Reference](./waterline-operator-api.md).

## Backup, restore, and disaster-recovery contract

Backup, restore, and disaster recovery are part of the operating envelope, not
an optional private runbook. For every supported topology, publish and rehearse
these facts:

1. The durable backup set: database backup, server or app image reference,
   runtime env file or config set, auth material location, and the exact
   topology or restore notes needed to reattach workers.
2. The recovery targets: maximum accepted restore lag, expected failover lag,
   and who is allowed to declare traffic safe again.
3. The restore order: restore durable persistence first, then cache, then
   bootstrap or migrations, then the singleton scheduler or maintenance role,
   then API readiness, then worker registration.
4. The verification pass: `/api/ready` or `/waterline/api/v2/health`,
   `/api/cluster/info` where applicable, one representative worker
   registration, and one representative history export from restored state.
5. The repair pass after restore: rebuild projections, backfill command
   contracts if needed, and confirm queue, compatibility, and repair metrics
   return to baseline before you call the environment healthy.

Multi-region operation and split-brain behavior remain outside the supported
2.0 operating envelope. The
[self-hosting guide](/docs/deployment#activepassive-multi-region) retains
active/passive architecture and runbook material only for support-led
evaluation; it does not establish self-serve replication, failover, failback,
RPO, or RTO guarantees. The current
[Cloud managed-runtime contract](/docs/polyglot/cloud-control-plane)
operates each namespace in one managed region and likewise does not promise
multi-region replication, regional failover, or failback.

Treat restore rehearsal cadence as part of the public operating contract too.
At minimum, rehearse the documented restore sequence:

- before the first production rollout for a topology
- after any change to the backup mechanism, schema/bootstrap path, auth model,
  or deployment topology
- on a regular recurring cadence that is published in the same runbook as the
  backup schedule

If you cannot produce the latest successful rehearsal date, elapsed restore
time, and verification evidence, then backup and DR remain an unproven claim
for that topology.

## Benchmark envelope

Durable Workflow v2 publishes the dimensions you should benchmark for your own
environment. Record these baselines in staging or canary before production
traffic depends on them:

| Dimension | What to baseline | Source |
| --- | --- | --- |
| Projection health | Steady-state `needs_rebuild = 0`, rebuild duration after intentional drift, and stale/orphan cleanup time | `/waterline/api/v2/health`, `/waterline/api/stats`, `workflow:v2:rebuild-projections` |
| Queue pressure | Backlog age, oldest ready task age, runnable vs delayed task counts, task add vs dispatch rate, dispatch-overdue age, stale poller count, and queue admission status (`accepting`, `saturated`, `throttled`, `no_slots`, `no_active_workers`) | Waterline dashboard stats and queue views plus `operator_metrics.backlog.*` / `operator_metrics.tasks.*` |
| Workflow-start latency | Accepted start commands waiting for first-task creation, oldest pending-start age, and first-task pickup after admission | `operator_metrics.starts.*` plus qualified worker `schedule_to_start` telemetry |
| Schedule-to-start latency | Server-measured eligibility-to-lease duration for workflow/activity tasks, excluding scheduled delay and backoff | Worker SDK histogram only after the exact Server/SDK pair is qualified |
| Timer fan-out wake-up behavior | Wake-signal propagation time and the lag between scheduled fire time and ready-task visibility during burst timers | Worker telemetry plus same-region wake coordination checks |
| Repair-loop sweep cost | Candidate counts, selected counts, max candidate age, max missing-run age, and scan-pressure behavior | `operator_metrics.repair.*` |
| History pressure | Event count, history size, and continue-as-new recommendation thresholds | `operator_metrics.history.*` |

These are benchmark dimensions rather than universal latency promises. Publish
your own acceptable ranges for the topology you operate.

## Long-soak evidence

Benchmark snapshots are not enough on their own. Before you call a topology
trusted for sustained traffic, keep a long-soak evidence packet that shows the
system stayed inside its declared envelope over time.

Include at least:

- workload shape: topology, server image or app revision, worker build ids,
  queue layout, cache backend, database backend, and the representative mix of
  workflow starts, timer load, activities, queries, and exports
- soak window: start and end time, plus enough duration to cover at least one
  normal repair window, one retention or archive pass if applicable, and one
  representative business-cycle traffic swing for that environment
- durable queue stability: backlog age, ready-task age, start backlog age, task
  add versus dispatch rate, and stale-poller counts staying within the
  published baseline for the topology
- correctness stability: no sustained `status = error` from
  `GET /waterline/api/v2/health`, no unexplained growth in
  `operator_metrics.repair.*`, and no persistent compatibility gaps in
  `operator_metrics.workers.*`
- process and cache stability: worker memory, CPU, event-loop or thread
  pressure, and cache/cardinality growth staying bounded rather than climbing
  monotonically under steady load
- recovery evidence: the latest successful backup timestamp, latest restore
  rehearsal timestamp, elapsed restore time, and the verification commands that
  proved the restored environment was ready

Store the packet where the same operators can retrieve the deployment runbook.
If a topology claims published benchmark numbers, alert semantics, or recovery
timing without a matching soak packet, treat those numbers as provisional
rather than trusted operating-envelope evidence.

## End-to-end operator checklist

Use this checklist after upgrades and before trusting a new environment:

1. Run `php artisan workflow:v2:doctor --strict`.
2. Check `GET /waterline/api/v2/health` and confirm whether the state is
   `ok`, `warning`, or `error`.
3. Read `GET /waterline/api/stats` for backlog, repair, history, command
   contract, worker compatibility, and projection drift facts.
4. Run projection rebuild or command-contract backfill previews when health
   reports drift.
5. Export one representative run and verify the archive/replay artifact path.
6. Confirm archived runs leave active fleet views while durable rows remain
   available until retention cleanup.
7. Rehearse the restore or failover sequence recorded in your deployment
   runbook and verify the measured lag matches the published expectation for
   your topology.

## Related Guides

- [Monitoring](./monitoring.md)
- [Waterline Operator API Reference](./waterline-operator-api.md)
- [Pruning Workflows](./configuration/pruning-workflows.md)
- [Self-Hosting Deployments](./deployment.md)

<!-- Source: docs/polyglot/embedded-to-server.md -->

# Embedded to Server Migration

This guide is for teams already running Durable Workflow v2 inside a Laravel
application and moving new workflow traffic to the standalone server. If the
application is still on v1, start with [Migrating to 2.0](/docs/migration)
and keep existing v1 runs on the v1 engine until they finish.

Use [Laravel Adoption and Runtime Transition](/docs/laravel-adoption/) first
when the destination is not yet chosen or when Laravel framework integration is
part of the decision. This page is the self-hosted Server runbook after that
choice; it is not the Cloud migration path.

The migration is an adoption path, not an in-place database move. The
standalone server owns new control-plane starts, durable history, schedules,
worker registrations, and task delivery over HTTP. Your application workers
continue to own workflow and activity code.

<div
  data-public-payload-codec-contract="avro-only"
  data-payload-codec="avro"
  data-authority-manifest="https://durable-workflow.github.io/sdk-neutrality-contract.json"
  data-legacy-v1-import-drain="internal">
  Durable Workflow 2.0 has one public payload codec: <code data-payload-codec-field="codec">avro</code>.
  JSON is the HTTP document transport, not a durable payload codec. PHP-only v1
  readers remain internal to the import/drain path and cannot be selected for
  new v2 runs. See the <a href="/docs/polyglot/avro-value-protocol/">Avro Value protocol</a>.
</div>

This is a self-hosted migration guide. It does not turn the embedded deployment
into a Cloud namespace, and the resulting Server is not attached to Durable
Workflow Cloud. For the separate managed-service choice, see
[Cloud Managed Runtime](/docs/polyglot/cloud-control-plane).

## Current Boundary

Supported today:

- Start new workflows through the server control-plane API or CLI.
- Register PHP, Python, Rust, or custom HTTP workers against the server.
- Poll and complete workflow and activity tasks through the worker protocol.
- Import eligible embedded v2 history-export bundles into the server as
  server-managed workflow state.
- Export closed-run history bundles from embedded v2 or server v2 for audit,
  debugging, and archival handoff.
- Observe server-managed workflows through the server API, CLI, and SDK
  surfaces that read server state, or deploy
  [Waterline service mode](/docs/monitoring#waterline-service) as a
  separate observer connected through the PHP SDK. The Waterline package in
  the original Laravel app continues to read only that embedded runtime's
  state.

Not supported as an automatic operation today:

- Moving v1 runs into the server.
- Replaying v1 history with non-PHP workers.

Plan the cutover so v1 runs drain where they started. Embedded v2 runs may
either drain in place or move through the import workflow below.

## Deployment Mode Contract

For the frozen embedded-vs-service comparison, see
[Deployment Modes](/docs/polyglot/deployment-modes). This migration guide
adds four cutover-specific rules on top of that shared contract:

- Existing embedded runs keep executing where they started.
- New server-managed runs use stable type keys, namespace names, task queues,
  and the Avro payload contract from the first cutover.
- Signals, queries, updates, repair, cancel, terminate, and archive must keep routing
  to the runtime that owns the target run.
- Each Waterline deployment must keep targeting the runtime and namespace it
  is intended to observe; it does not merge embedded and server-managed runs.

## Cutover Invariants

Keep these rules true throughout the migration:

- Configure the server as an explicit remote dependency: set the base URL,
  namespace, task queue, and auth material directly instead of inferring them
  from Laravel app-local settings.
- Keep workflow ids, run-targeting rules, workflow/activity type keys, payload
  codec tag (`avro`), and compatibility markers stable across both runtimes.
- Route signals, queries, updates, repair, cancel, terminate, and archive to the same
  runtime that owns the target run.
- Before importing an embedded run, pause embedded workers long enough to export
  a quiesced bundle with no leased workflow/activity task and no running
  activity attempt.
- Treat language neutrality as part of the migration contract: server-managed
  workflows use stable aliases and the fixed Avro Value schema, not PHP-only
  class names or payload formats.

## Phase A: Prepare Embedded v2

Before deploying the server, make the embedded app use language-neutral
contracts. This reduces the amount of code that changes during cutover.

1. Define stable workflow and activity type keys.

   ```php
   // config/workflows.php
   'v2' => [
       'types' => [
           'workflows' => [
               'orders.process' => App\Workflows\ProcessOrderWorkflow::class,
           ],
           'activities' => [
               'orders.reserve-inventory' => App\Activities\ReserveInventory::class,
               'orders.capture-payment' => App\Activities\CapturePayment::class,
           ],
       ],
   ],
   ```

2. Use those keys at every external boundary.

   Server clients send `workflow_type: "orders.process"`, and workers register
   `supported_workflow_types` / `supported_activity_types` with those same
   strings. Do not expose PHP FQCNs as the durable public contract.

3. Use the v2 Avro payload contract.

   ```php
   // config/workflows.php
   'serializer' => 'avro',
   ```

   Avro is mandatory for new embedded and server-managed v2 workflows. A
   legacy PHP reader may be retained only inside the v1 import/drain path while
   old v1 runs finish.

4. Pick namespace and task queue names.

   Choose names you can keep stable throughout the embedded-to-Server cutover:

   ```bash
   export DURABLE_WORKFLOW_NAMESPACE=production
   export DURABLE_WORKFLOW_TASK_QUEUE=orders
   ```

5. Decide whether compatibility markers are needed.

   A single-fleet deployment can leave compatibility unset. The standard
   v2 rollout mechanics (canary, drain, rollback, replay-debug) all
   operate inside one stable v2 contract — there is no "mixed-fleet v2"
   adoption lane. If a build-skew window does open during the embedded
   to server cutover, set `DW_V2_CURRENT_COMPATIBILITY` and
   `DW_V2_SUPPORTED_COMPATIBILITIES` before the cutover so a new build
   does not claim runs from an incompatible build during that window.

## Phase B: Deploy the Server Beside Embedded

Run the standalone Server beside the Laravel app while the app continues
handling existing embedded runs.

```bash
git clone https://github.com/durable-workflow/server.git
cd server
cp .env.example .env
docker compose up -d
```

For local development you may set `DW_AUTH_DRIVER=none`. For
shared environments, use role-scoped credentials:

```bash
DW_AUTH_DRIVER=token
DW_WORKER_TOKEN=worker-secret
DW_OPERATOR_TOKEN=operator-secret
DW_ADMIN_TOKEN=admin-secret
```

Verify discovery and create the namespace:

```bash
export SERVER=http://localhost:8080
export ADMIN_TOKEN=admin-secret
export OPERATOR_TOKEN=operator-secret

curl "$SERVER/api/health"

curl "$SERVER/api/cluster/info" \
  -H "Authorization: Bearer $OPERATOR_TOKEN"

curl -X POST "$SERVER/api/namespaces" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{"name":"production","description":"Production workflows","retention_days":30}'
```

The server rejects control-plane requests without
`X-Durable-Workflow-Control-Plane-Version: 2`; workers use the separate
`X-Durable-Workflow-Protocol-Version: 1.0` header.

## Phase C: Connect Workers

Workers must register before polling. The registration advertises runtime,
task queue, and the type keys the worker can execute.

```bash
export WORKER_TOKEN=worker-secret

curl -X POST "$SERVER/api/worker/register" \
  -H "Authorization: Bearer $WORKER_TOKEN" \
  -H "X-Namespace: production" \
  -H "X-Durable-Workflow-Protocol-Version: 1.0" \
  -H "Content-Type: application/json" \
  -d '{
    "worker_id": "orders-php-1",
    "task_queue": "orders",
    "runtime": "php",
    "sdk_version": "<installed-sdk-version>",
    "supported_workflow_types": ["orders.process"],
    "supported_activity_types": [
      "orders.reserve-inventory",
      "orders.capture-payment"
    ],
    "max_concurrent_workflow_tasks": 10,
    "max_concurrent_activity_tasks": 50
  }'
```

Then run workers in server mode for the same namespace and task queue. Python
workers follow the same registration and poll contract through the Python SDK.
Custom workers can use the [Worker Protocol](/docs/polyglot/worker-protocol)
directly.

Check visibility before cutover:

```bash
curl "$SERVER/api/workers" \
  -H "Authorization: Bearer $OPERATOR_TOKEN" \
  -H "X-Namespace: production" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2"

curl "$SERVER/api/task-queues/orders" \
  -H "Authorization: Bearer $OPERATOR_TOKEN" \
  -H "X-Namespace: production" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2"
```

## Phase D: Route New Starts to the Server

Cut over one workflow family at a time. Keep the embedded app and queue workers
running for old runs until they drain.

Start a shadow workflow first:

```bash
curl -X POST "$SERVER/api/workflows" \
  -H "Authorization: Bearer $OPERATOR_TOKEN" \
  -H "X-Namespace: production" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{
    "workflow_id": "orders-shadow-1001",
    "workflow_type": "orders.process",
    "task_queue": "orders",
    "input": [{"order_id":"1001","mode":"shadow"}]
  }'
```

Watch the run:

```bash
curl "$SERVER/api/workflows/orders-shadow-1001" \
  -H "Authorization: Bearer $OPERATOR_TOKEN" \
  -H "X-Namespace: production" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2"
```

After the shadow path works, move the production caller from
`WorkflowStub::start()` to the server client, CLI, or direct HTTP API. Leave
signals, queries, updates, cancel, and terminate routed to the same runtime that
started the workflow. A workflow started embedded should receive embedded
commands; a workflow started on the server should receive server control-plane
commands.

## Phase E: Import Eligible Embedded v2 Runs

Use history export as the import format. The embedded runtime remains the source
of truth for the bundle; the server verifies it, writes durable rows inside one
transaction, and rebuilds server projections from those rows.

Eligibility:

- The bundle schema must be `durable-workflow.v2.history-export` with
  `schema_version: 1`.
- The source run must be embedded v2 and the export must carry
  `workflow.source_runtime: "embedded"`. v1 history remains out of scope.
- A non-terminal run must be the current embedded run.
- A terminal run must have `history_complete: true`.
- Redacted bundles are rejected.
- Leased workflow tasks, leased activity tasks, and running activity attempts
  are rejected. Pause workers or let leases expire, then export again.

Import source-of-truth rules:

- `history_events` are copied as the authoritative replay and audit log.
- Workflow identity, payloads, commands, signals, updates, tasks, activity
  executions, timers, failures, and lineage links are reconstructed from the
  bundle.
- Server summary, wait, timer, timeline, and lineage projections are rebuilt
  after import. Projection rows are not the import authority.
- Pending workflow tasks remain claimable by compatible server workers.
- Pending activity executions and ready activity tasks remain claimable by
  compatible server activity workers.
- Pending timers keep their `fire_at` timestamp and are visible to server timer
  repair/recovery.

Failure and rollback:

- Import is all-or-nothing in one database transaction.
- If the process exits or validation fails before commit, no partial run state
  remains on the server.
- Retrying the same bundle is idempotent by `run_id` and `dedupe_key`.
- If a different server run already owns the same `run_id`, import is rejected.

Visibility and audit:

- Imported runs expose `engine_source: embedded_v2_import` in server list/detail
  views.
- The durable run row records `import_source=embedded_v2`, `import_id`,
  `import_dedupe_key`, `import_contract_version`, and `imported_at`.

Operator workflow:

For embedded v2 runs:

```bash
php artisan workflow:v2:history-export order-123 \
  --output=storage/workflow-history/order-123.json \
  --pretty
```

Dry-run the import on the server:

```bash
php artisan workflow:v2:history-import storage/workflow-history/order-123.json \
  --namespace=production \
  --dry-run \
  --json
```

Import the bundle:

```bash
php artisan workflow:v2:history-import storage/workflow-history/order-123.json \
  --namespace=production \
  --import-id=orders-cutover-2026-05-05
```

Or call the HTTP operator endpoint:

```bash
curl -X POST "$SERVER/api/workflows/import/embedded-v2" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "X-Namespace: production" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  -H "Content-Type: application/json" \
  --data-binary @order-123-import-request.json
```

where `order-123-import-request.json` is:

```json
{
  "import_id": "orders-cutover-2026-05-05",
  "bundle": {
    "schema": "durable-workflow.v2.history-export",
    "schema_version": 1,
    "workflow": {
      "source_runtime": "embedded"
    }
  }
}
```

The `bundle` object above is abbreviated; pass the complete
`workflow:v2:history-export` JSON document.

For server-managed runs that you only need to preserve or audit:

```bash
curl "$SERVER/api/workflows/order-123/runs/$RUN_ID/history/export" \
  -H "Authorization: Bearer $OPERATOR_TOKEN" \
  -H "X-Namespace: production" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  > order-123-history.json
```

Archive terminal server runs after export when you want to keep them out of
retention pruning:

```bash
curl -X POST "$SERVER/api/workflows/order-123/archive" \
  -H "Authorization: Bearer $OPERATOR_TOKEN" \
  -H "X-Namespace: production" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{"reason":"history exported to archive storage"}'
```

## Phase F: Add Polyglot Workers

Once a workflow family runs through the server, add Python, Rust, or custom workers by
registering the same namespace, task queue, and type keys. Every public v2
payload uses `avro`; keep activity inputs and outputs language-neutral: arrays,
objects, strings, numbers, booleans, and nulls.

Use [the Python SDK guide](/docs/polyglot/python) or
[the Rust SDK guide](/docs/polyglot/rust). For direct
HTTP implementations, use [the worker protocol reference](/docs/polyglot/worker-protocol).

## Cutover Checklist

- [ ] v1 runs are drained or intentionally left on the v1 engine.
- [ ] Embedded v2 uses stable type keys, not PHP FQCNs, at external boundaries.
- [ ] Every new v2 workflow uses the sole public `avro` codec.
- [ ] Server `/api/health` and `/api/cluster/info` pass from the deployment
  network.
- [ ] Target namespace exists on the server.
- [ ] Workers register with the expected task queue and supported type keys.
- [ ] A shadow workflow starts, is claimed by a worker, and reaches the expected
  state.
- [ ] Operators can list workflows, inspect task queues, and view worker
  registrations.
- [ ] Old embedded starters are paused or moved so duplicate business keys do
  not start in both runtimes.
- [ ] Embedded v2 runs selected for import are exported from a quiesced embedded
  runtime and pass `workflow:v2:history-import --dry-run`.
- [ ] Imported runs show `engine_source: embedded_v2_import` and the expected
  `import_id` in server detail/list surfaces.
- [ ] Closed old runs that only need retention are exported to durable storage.

## Troubleshooting

| Symptom | Likely cause | Fix |
| --- | --- | --- |
| `missing_control_plane_version` | Client called a control-plane route without the v2 header | Send `X-Durable-Workflow-Control-Plane-Version: 2` |
| `missing_protocol_version` | Worker called a worker route without the worker protocol header | Send `X-Durable-Workflow-Protocol-Version: 1.0` |
| `namespace_not_found` | The namespace has not been created on the server | Create it with `POST /api/namespaces` |
| Worker polls return no task | Task queue, type key, namespace, or compatibility marker does not match | Compare workflow start payload, worker registration, and task queue visibility |
| A v1 payload cannot be decoded by a non-PHP worker | The run contains PHP-native v1 history | Keep the run on the internal PHP v1 import/drain path; do not route it to a v2 worker |
| Old workflow does not respond to server signal/query/update | The run was started in embedded mode | Send commands through the embedded app until that run finishes |
| Import rejects `tasks.leased_task_present` | The embedded export captured a leased task | Pause embedded workers, wait for leases to release or complete, then export again |
| Import returns `already_imported` | The same `run_id` and `dedupe_key` are already present on the server | Treat the retry as successful and inspect the server run |
| Imported run is visible but not claimed | Worker type key, task queue, namespace, or compatibility marker does not match the imported row | Compare the import report, run detail, and worker registration |

## Related Guides

- [Server setup](/docs/polyglot/server)
- [Worker protocol](/docs/polyglot/worker-protocol)
- [Python SDK](/docs/polyglot/python)
- [Migrating to 2.0](/docs/migration)

<!-- Source: docs/polyglot/deployment-modes.md -->

# Deployment Modes

Durable Workflow v2 has two deployment modes:

- **Service mode:** applications and workers connect through SDKs to a remote
  runtime. Choose [Durable Workflow Cloud](/docs/polyglot/cloud-control-plane/)
  or a [self-hosted Server](/docs/polyglot/server/).
- **Embedded mode:** a Laravel application installs
  `durable-workflow/workflow` and owns the runtime directly.

Cloud and self-hosted Server are runtime choices inside service mode, not
components to run together. In Cloud, Durable Workflow operates orchestration
and persistence while customers run SDK clients and workers. Cloud includes
Managed Waterline. **Cloud customers do not install, deploy, or attach their
own Server or Waterline service.** A self-hosted Server does not include
Waterline; operators can separately deploy Waterline against a Server-owned
namespace.

Use this page when deciding which shape should own a workflow fleet, planning a
cutover between them, or documenting which parts of the product contract must
stay identical across both.

Laravel teams should use the focused
[Laravel adoption and runtime transition guide](/docs/laravel-adoption/)
for the v1-to-v2 and embedded-to-service paths, including the shipped PHP SDK
bridge and its Laravel test fake.

## Choose a service-mode runtime

| Runtime choice | Who operates durable state | What your team runs | Start here |
| --- | --- | --- | --- |
| Durable Workflow Cloud | Durable Workflow operates the managed namespace runtime, persistence, upgrades, service endpoint, and Managed Waterline. | Application clients plus PHP, Python, or Rust workers using provisioned credentials. Do not run Server or a separate Waterline service. | [Cloud Managed Runtime](/docs/polyglot/cloud-control-plane/) |
| Self-hosted Server | Your team deploys, secures, scales, backs up, and upgrades the Server and its persistence. | Server plus application clients and PHP, Python, or Rust workers. If wanted, deploy Waterline as a separate service against the Server-owned namespace. | [Self-hosted Server](/docs/polyglot/server/) |

Both choices use the same client and worker model. The difference is runtime
operations and credentials, not workflow authoring.

## Same durable model, different boundary

Embedded mode and service mode keep one v2 kernel. What changes is the hosting,
auth, and transport boundary around it.
Service mode keeps the same kernel behind HTTP+JSON control-plane and worker
surfaces; there is no mandatory gRPC and no second engine.

| Surface | Embedded mode | Service mode | Stable in both modes |
| --- | --- | --- | --- |
| Durable workflow model | A Laravel app hosts the package directly and writes workflow state inside the app runtime. | Cloud or a self-hosted Server owns workflow state behind the service API. | Workflow ids, run ids, typed history, command outcomes, retries, repair semantics, and history export remain the same v2 contract. |
| Control plane | Starts and commands come from app code, `WorkflowStub`, or app-local operator tooling. | Starts and commands go through the server API, CLI, or SDKs over HTTP+JSON with explicit auth and protocol headers. Framework-neutral PHP callers use `DurableWorkflow\Client` from `durable-workflow/sdk`. | Duplicate-start policy, run targeting, command ids, and named outcomes stay the same. Route follow-up commands to the runtime that accepted the start. |
| Worker transport | Laravel queue workers execute workflow and activity tasks inside the app deployment. | Workers register, long-poll, heartbeat, and complete work over the HTTP+JSON worker protocol. PHP remote workers use `DurableWorkflow\Worker` from `durable-workflow/sdk`. | Task leases, compatibility markers, replay semantics, and at-least-once activity execution stay the same. |
| Task dispatch default | Tasks are normally dispatched to the Laravel queue in-process with the application. | The service runtime uses poll dispatch so external workers discover work over HTTP. Self-hosted Server operators can explicitly override that default. | The ready/leased/repair lifecycle and durable task model stay the same. |
| Workflow and activity type keys | PHP aliases can resolve to local classes inside the app. | Workers advertise supported type keys during registration. | Public type keys should stay stable and language-neutral. Do not make PHP FQCNs or mirrored PHP placeholder types the public contract. |
| Operator surface | The embedded Waterline package or app-local tooling reads the Laravel app's durable state in process. | Cloud provides Managed Waterline for its namespace. Self-hosted operators can separately deploy Waterline against a Server-owned namespace. Service APIs, CLI, and SDKs also read runtime-owned state. | Visibility facts such as search attributes, memos, run status, queue diagnostics, and history export are durable facts within the runtime that owns the run. Waterline does not combine runtimes or namespaces. |
| Auth and tenancy boundary | App auth is whatever the Laravel host exposes around its own routes and sessions. | Namespace selection and server auth tokens or signatures are mandatory API boundaries. | Namespace names, task queues, compatibility markers, and the fixed Avro payload contract should stay stable across a cutover. |
| Runtime discovery | The app can resolve services in-process or through app-local configuration. | Workers and clients must target an explicit remote base URL. | Do not couple either mode to shared `APP_URL`, `APP_KEY`, localhost assumptions, or same-container discovery. |
| Migration boundary | Existing embedded runs keep executing where they started. | New service-managed runs start in the selected Cloud or self-hosted runtime and stay there. | There is no automatic live migration of in-flight runs between modes. Export is for audit/debugging, not for importing live state. |

## Choose Embedded Mode When

- Your Laravel application owns workflow authoring, worker execution, and
  operator access in one deployment.
- The app's existing queue and auth model is the right boundary for workflow
  operations.
- You want the smallest self-contained runtime and do not need a
  language-neutral worker protocol.
- Your operators can use Waterline or host-app tooling as the primary workflow
  surface.

Start with [Embedded Installation](/docs/installation/) and the
[Embedded documentation](/docs/category/embedded/), including its
Configuration group.

## Choose Service Mode When

- Multiple applications or teams should share one workflow runtime.
- Workers, control-plane callers, or operators are not all Laravel/PHP.
- You need an explicit remote auth and namespace boundary between clients and
  the workflow engine.
- You want to scale API ingress, matching/dispatch, and workers independently
  within the supported
  [server role topology](/docs/polyglot/server-role-topology).
- For self-hosted Server, you want to deploy Waterline as an observer over a
  server-owned namespace. Cloud instead includes Managed Waterline.

For a managed runtime, start with
[Durable Workflow Cloud](/docs/polyglot/cloud-control-plane/). For
self-hosting, start with [Server](/docs/polyglot/server/) and
[Self-Hosting Deployments](/docs/deployment/). Then choose the
[PHP SDK](/docs/polyglot/php/),
[Python SDK](/docs/polyglot/python/), or
[Rust SDK](/docs/polyglot/rust/). Cloud users operate through Managed
Waterline. Self-hosted operators can use the
[Server API Reference](/docs/polyglot/server-api-reference/) and
[Monitoring](/docs/monitoring#waterline-service) when deploying a separate
Waterline service.

## Migration tooling to self-hosted service mode

The supported path from embedded mode to service mode is staged adoption, not a
live handoff of in-flight state:

- Use [Embedded to Server Migration](/docs/polyglot/embedded-to-server) for
  the step-by-step cutover.
- Use `GET /api/cluster/info` to confirm the target server build, topology, and
  capability contract before switching traffic.
- Use `POST /api/worker/register` plus the worker protocol to prove external
  workers can serve the stable type keys you chose.
- Use `GET /api/system/operator-metrics`, `dw worker:list`, or Waterline
  operator views to verify worker registration and compatible fleet coverage
  before shifting production traffic.
- Use
  [Client and Worker Capabilities](/docs/polyglot/cli-python-parity/)
  when replacing app-local control-plane calls with server-backed automation.
- Use Cloud's Managed Waterline, the Waterline deployment attached to a
  self-hosted runtime, or server-native history export for audit/debugging
  evidence; do not treat export bundles as an import path for live
  server-managed runs.

Three migration rules are non-negotiable:

1. Existing runs stay on the runtime where they started.
2. New server-managed runs use stable type keys, namespace names, task queues,
   and the fixed Avro payload contract from the first cutover.
3. Signals, queries, updates, repair, cancel, terminate, and archive must go to the
   runtime that owns the target run.

## Related References

[Moving between runtimes](/docs/polyglot/runtime-migration/) covers self-hosted-to-self-hosted, self-hosted-to-Cloud and Cloud-to-self-hosted drain and cutover.

- [Installation](/docs/installation)
- [Server](/docs/polyglot/server)
- [PHP SDK](/docs/polyglot/php)
- [Cloud Managed Runtime](/docs/polyglot/cloud-control-plane)
- [Embedded to Server Migration](/docs/polyglot/embedded-to-server)
- [Server Role Topology](/docs/polyglot/server-role-topology)
- [Server Config Reference](/docs/polyglot/server-config-reference)

<!-- Source: docs/polyglot/server.md -->

# Server

The published Durable Workflow server is a standalone workflow-orchestration
service. Server 3.x is a native (Rust) implementation that runs as one process
and exposes a language-neutral HTTP+JSON control plane and worker protocol.
The implementation language is not an application-language requirement. PHP
Server 2.x, which ran the embedded PHP engine behind this boundary, is frozen at
2.11.8 and receives no further releases.

:::note Existing PHP Server 2.x installations

The PHP 2.x stack (server, queue worker, scheduler, bootstrap job and Redis) is
[`docker-compose.published.yml` at the 2.11.8 tag](https://github.com/durable-workflow/server/blob/2.11.8/docker-compose.published.yml),
kept on current Server releases as
[`docker-compose.php.yml`](https://github.com/durable-workflow/server/blob/main/docker-compose.php.yml).
Existing PHP 2.x installations can move to native with `upgrade-takeover` by
following the [native takeover runbook](https://github.com/durable-workflow/server/blob/a051f57fcc7e3605b301581ee9105461b3ef5815/docs/rust-server-upgrade-takeover.md).
It is qualified on the published 3.0.4 image (arm64) for databases written by
PHP 2.11.8 on SQLite, MySQL 8.0 and PostgreSQL 17. MariaDB, amd64 and databases
from other PHP releases (layouts 2.8.0–2.11.8) have source-CI coverage only.
Takeover is one-way and backup-first, and PHP must be stopped. It refuses
databases and state it cannot carry over; the runbook's
[refusal tables](https://github.com/durable-workflow/server/blob/a051f57fcc7e3605b301581ee9105461b3ef5815/docs/rust-server-upgrade-takeover.md#refusals)
list each case. Rehearse it on a restored copy of your backup first. There is
no PHP rollback after takeover, and takeover is planned for deprecation in the
3.1.x line.
Configuration below marked as PHP 2.x does not apply to the native Server.

:::

## Retired PHP Server security {#retired-php-server-security}

:::warning Retired and unpatched

PHP Server 2.11.8 is retired and remains unpatched for the November Laravel advisories. Its published image contains Laravel 13.34.0. Updating a source checkout's Composer dependencies does not patch that installed image. Read the [13 November 2026 security notice](https://github.com/durable-workflow/server/blob/3f21b754418ffea5e786f2bd48481a35c92ab1a2/docs/security/retired-php-server-2026-11-13.md) for the reviewed conditions.

Laravel registers signed `GET /storage/{path}` and `PUT /storage/{path}` routes. The shipped application does not issue their temporary signed URLs or call `Concurrency::run()`. The dependency finding does not demonstrate an exploitable default Server journey. Storage-route exposure depends on obtaining or issuing a valid temporary signed URL and the advisory's request-parsing conditions. Review custom routes and extensions separately.

Stop exposing the retired PHP service and plan migration to native. Until retirement:

- Block all `/storage/*` requests at ingress, including GET and PUT, and verify the rule on the actual endpoint. These routes are separate from authenticated `/api/external-payloads/v1` transport.
- Stop issuing and reusing local-disk temporary download or upload URLs. Review extensions that generate signed routes or call Laravel Concurrency.
- Preserve a restorable database and external-payload-storage backup. Stop every PHP role during a migration that writes the database.

These controls reduce the identified storage-route exposure; they do not install the fixes or establish that custom extensions are safe. Native uses a Rust runtime outside this Laravel dependency finding. Follow the [backup-first takeover runbook](https://github.com/durable-workflow/server/blob/a051f57fcc7e3605b301581ee9105461b3ef5815/docs/rust-server-upgrade-takeover.md) only within its exact version/backend/state limits and rehearse on a restored copy. Native startup is not a PHP database conversion, and the latest native release does not newly qualify takeover. Subsequent native upgrades retain their [explicit schema and restore requirements](../deployment.md#native-schema-upgrade-3-10).

:::

## Is this only for PHP teams?

No. PHP, Python, and Rust are first-party SDK surfaces. A Python- or Rust-only
application team can deploy the published standalone server as infrastructure,
write application workflows and activities with its native SDK, and communicate
through the public protocol. Its application does not embed Laravel and does
not become a Laravel application.

The adoption paths are separate:

- **Standalone adoption:** operators deploy the published server image and its
  database. Application teams run PHP,
  Python, or Rust workers against that endpoint. The server owns orchestration
  state and the public control-plane/worker boundary.
- **Embedded adoption:** a Laravel application installs the PHP workflow
  package and owns the engine through its own queue, database, configuration,
  and deployment. This is the Laravel-native advantage, not a prerequisite for
  the standalone path.
- **Managed Cloud adoption:** Durable Workflow Cloud operates the orchestration
  runtime, persistence, placement, and recovery. Application teams run SDK
  clients and workers against a provisioned Cloud namespace. They do not deploy
  this Server distribution for Cloud or attach a self-hosted Server to it. See
  [Cloud Managed Runtime](/docs/polyglot/cloud-control-plane/).

All three choices use the same durable execution concepts. Standalone and
Cloud-managed external workers use stable string workflow/activity type
names and the shared codec envelope, so a cross-language child workflow or
activity preserves payload shape rather than exposing PHP serialization.

If you are deciding between the standalone server and package embedding, start
with [Deployment Modes](/docs/polyglot/deployment-modes). This page covers
the service-mode distribution.

Use the standalone server when you need:
- **Polyglot workflows** — PHP, Python, and Rust workflow and activity workers sharing one durable runtime
- **Microservice orchestration** — orchestrate services written in different languages
- **Centralized workflow runtime** — multiple applications sharing one workflow engine
- **Non-Laravel environments** — use Durable Workflow outside Laravel

If you already run v2 embedded in a Laravel app, use the
[embedded-to-server migration guide](/docs/polyglot/embedded-to-server) to
prepare type keys, deploy the server beside embedded execution, connect workers,
and route only new workflow starts to the server. Keep
[Deployment Modes](/docs/polyglot/deployment-modes) nearby during that
cutover so ids, command outcomes, task semantics, and runtime ownership rules
stay explicit.

Use [Worker Compatibility and Routing](/docs/polyglot/worker-compatibility-routing)
when you roll worker build cohorts, drain old cohorts, or need to keep
long-running runs pinned to compatible executors during rollback.

Use [Server Role Topology](/docs/polyglot/server-role-topology) when you
need the role vocabulary, process classes, authority boundaries, failure
domains, or migration path that PHP Server 2.x publishes from
`GET /api/cluster/info`. The native 3.x Server runs as one process and
publishes no topology manifest.

## Quick Start

:::info Managed alternative

[Run it managed: Cloud Dev, $0.075/hour, capped at $50/month, no per-workflow charges.](https://cloud.durable-workflow.com/pricing)
The provisioned-hour rate is metered by the minute, with a $2 minimum per active
month and 5 GB included storage. Prices are in USD per runtime. Dev has no uptime
SLA. HA plans are planned and not yet available to provision.

:::

### Published Image + SQLite

The fastest source-free way to run the server is the published Docker image.
The native image defaults to SQLite at `/app/database/database.sqlite`; mount
`/app/database` on a volume so the database survives container replacement:

```bash
server_image=durableworkflow/server:3.10.0
export DW_AUTH_TOKEN=dev-token

docker volume create durable-workflow-server-quickstart

docker rm -f durable-workflow-server >/dev/null 2>&1 || true
docker run -d --name durable-workflow-server \
  -p 127.0.0.1:8080:8080 \
  -v durable-workflow-server-quickstart:/app/database \
  -e DW_AUTH_TOKEN="$DW_AUTH_TOKEN" \
  "$server_image"

until curl -sf http://localhost:8080/api/ready >/dev/null; do sleep 1; done
curl http://localhost:8080/api/health
curl -H "Authorization: Bearer $DW_AUTH_TOKEN" \
  http://localhost:8080/api/cluster/info \
  | jq '{version, implementation}'
```

On first start the Server creates its schema and the `default` namespace. The
same process fires durable timers and schedules and runs its maintenance
passes, including activity-timeout enforcement, while SDK workers are stopped.
There is no bootstrap command, queue worker or scheduler. The Server refuses
to start without `DW_AUTH_TOKEN`.

The API is bound to loopback and `dev-token` is for this local exercise. For
remote access, use the Compose deployment with your own credentials, TLS and
network access controls.

Remove this local exercise container and database when finished:

```bash
docker rm -f durable-workflow-server
docker volume rm durable-workflow-server-quickstart
```

### Published Image + Compose

Use the published Compose artifact when you want the source-free stack backed
by MySQL. Fetch it from the selected release and pin the image:

```bash
curl -fsSLO https://raw.githubusercontent.com/durable-workflow/server/3.10.0/docker-compose.published.yml

export DW_SERVER_IMAGE=durableworkflow/server:3.10.0
export DW_AUTH_TOKEN=dev-token

docker compose -f docker-compose.published.yml up -d --wait
curl -H "Authorization: Bearer $DW_AUTH_TOKEN" \
  http://localhost:8080/api/cluster/info \
  | jq '{version, implementation}'
```

The stack is the native Server and MySQL. Compose refuses to start until
`DW_AUTH_TOKEN` is set. The [deployment guide](/docs/deployment#single-node-production)
covers the production env file, backups and upgrades.
Upgrading to native 3.10.0, including from 3.9.x, requires the
[explicit schema upgrade](../deployment.md#native-schema-upgrade-3-10) before restart.
Older versions also need their intervening schema steps. Back up both the
database and external-payload storage; rollback restores that backup.

### Ports

| Service | Port | Purpose |
|---------|------|---------|
| Server API | 8080 | Control-plane and worker-protocol endpoints, published on `127.0.0.1` by default (`SERVER_BIND_ADDRESS`, `SERVER_PORT`) |
| MySQL | 3306 | Database, reachable only on the Compose network |

## Configuration

The server uses environment variables for configuration. Key settings are
summarized below; the full operator-facing `DW_*` contract is documented in
the [server config reference](/docs/polyglot/server-config-reference).

### Database

```bash
DB_CONNECTION=mysql
DB_HOST=mysql
DB_PORT=3306
DB_DATABASE=workflow
DB_USERNAME=workflow
DB_PASSWORD=secret
```

Declared baselines are MySQL 8.0, PostgreSQL 13 and SQLite 3.35. The
[database table](./database-support.md) separates those compatibility declarations
from exact source tests and published-image qualification. The native Server
accepts `DB_CONNECTION=sqlite` (its default), `mysql`, `mariadb` or `pgsql`, and
waits for a MySQL or PostgreSQL database that is not accepting connections yet.

### Cache and Queue (PHP 2.x only)

The native Server has no cache store, Laravel queue or Redis connection; these
settings apply only to PHP Server 2.x.

```bash
CACHE_STORE=redis
QUEUE_CONNECTION=redis

REDIS_HOST=redis
REDIS_PORT=6379
REDIS_PASSWORD=null
REDIS_DB=0
```

Cache must support [atomic locks](https://laravel.com/docs/12.x/cache#atomic-locks). Queue drivers: Redis, Amazon SQS, Beanstalkd, database.

On PHP 2.x, atomic cache locks are required for server-side
[task queue admission caps](/docs/polyglot/task-queue-admission) and
query-task backpressure. Use Redis for PHP multi-node deployments that need
workflow, activity, or query admission to hold across every server process.

### Authentication

The native Server uses token authentication. A shared `DW_AUTH_TOKEN` is
required: the Server refuses to start without it, and it accepts that token for
every role.

```bash
DW_AUTH_DRIVER=token
DW_AUTH_TOKEN=your-secret-token-here
```

All requests must send `Authorization: Bearer your-secret-token-here`.

For least-privilege clients, add role-scoped tokens and give workers and
applications the narrower token. Keep the shared token secret, like the admin
token; native refuses `DW_AUTH_BACKWARD_COMPATIBLE=false`, so it cannot be
disabled.

```bash
DW_AUTH_DRIVER=token
DW_AUTH_TOKEN=shared-secret
DW_WORKER_TOKEN=worker-secret
DW_OPERATOR_TOKEN=operator-secret
DW_ADMIN_TOKEN=admin-secret
```

Worker tokens can register workers, poll tasks, heartbeat, and complete work.
Operator tokens can start, list, signal, query, update, repair, cancel,
terminate, archive, and observe workflows. Admin tokens can use administrative
endpoints such as namespace and retention management.

The native Server refuses the `signature` driver. A release image also refuses
`DW_AUTH_DRIVER=none`, which would admit every request as an administrator.

**PHP 2.x only — HMAC signature:**

```bash
DW_AUTH_DRIVER=signature
DW_SIGNATURE_KEY=your-signature-secret
```

Requests must include `X-Signature`, calculated as
`hash_hmac('sha256', request_body, DW_SIGNATURE_KEY)`. PHP 2.x
also accepts role-scoped signature keys:

```bash
DW_AUTH_DRIVER=signature
DW_WORKER_SIGNATURE_KEY=worker-signature-secret
DW_OPERATOR_SIGNATURE_KEY=operator-signature-secret
DW_ADMIN_SIGNATURE_KEY=admin-signature-secret
```

**PHP 2.x only — no auth** (development only):

```bash
DW_AUTH_DRIVER=none
```

⚠️ **Do not use `none` in production.** All endpoints become publicly accessible.

### Workflow Package (PHP 2.x only)

The PHP 2.x Docker image installs the `durable-workflow/workflow` package. The
native image does not. For PHP 2.x builds, control which version:

```bash
# Build-time arg (set in docker-compose.yml or pass to docker build)
WORKFLOW_PACKAGE_REF=v2        # branch, tag, or commit
WORKFLOW_PACKAGE_SOURCE=       # custom Git remote (optional)
```

### Retention

The PHP Server uses namespace retention to select closed run history for
cleanup. The default bounded duration is configured with:

```bash
DW_HISTORY_RETENTION_DAYS=30
```

Each scan uses the namespace's **current** policy. Changing the duration also
affects history that was already closed before the change; it does not give
each run a new retention period starting on the update date.

| Policy change | Effect on retained closed runs |
| --- | --- |
| Shorten bounded retention | Older runs can become eligible at the next scan. |
| Extend bounded retention | Runs still retained may stop being eligible. Previously pruned history is not restored. |
| Change to `forever` | Retention scans select no runs, and explicit retention passes skip them. |
| Change from `forever` to `bounded` | Supply `retention_days`; existing closed history is assessed against that duration. |

For example, at **19 October, 12:00 UTC**, a run closed on **1 October,
12:00 UTC** is 18 days old. It is outside a 7-day window and inside a 90-day
window. Changing 90 days to 7 makes it eligible immediately; cleanup can
process it on a later pass. A run closed exactly at the cutoff remains outside
that scan's selection, which uses a strict earlier-than comparison.

#### Review a retention change {#review-a-retention-change}

Before shortening retention, review close times and preserve the history you
need. Export important runs through the [history export API](./server-api-reference.md#workflow-control-plane)
and keep recoverable database and external-payload backups. A replay export
must retain its referenced payload objects; it does not replace a database
and storage backup.

Update the namespace with an admin credential:

```bash
curl -sS -X PUT "$DURABLE_WORKFLOW_SERVER_URL/api/namespaces/orders-prod" \
  -H "Authorization: Bearer $DW_ADMIN_TOKEN" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{"retention_mode":"bounded","retention_days":7}'
```

For `forever`, send `{"retention_mode":"forever","retention_days":null}`.
Returning to bounded retention requires an explicit duration from 1 to 365 days.

Inspect the current namespace policy and bounded eligible-run list:

```bash
curl -sS "$DURABLE_WORKFLOW_SERVER_URL/api/system/retention?limit=100" \
  -H "Authorization: Bearer $DW_ADMIN_TOKEN" \
  -H "X-Namespace: orders-prod" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2"
```

`cutoff` describes the current scan. `expired_run_count` counts the returned
IDs, up to `scan_limit`, rather than the entire backlog. `scan_pressure: true`
means that limit was reached. This read does not delete history.

The update is not a cleanup preview or a grace period: ordinary worker
heartbeats can already trigger a small throttled retention pass after the
policy changes. Scheduled `history:prune` and explicit system passes can also
process eligible runs. Back up and review **before** shortening the policy.

#### Cleanup and protections {#retention-cleanup-and-protections}

`POST /api/system/retention/pass` without `run_ids` selects expired runs under
the current policy, up to its limit. Inspect the response's `pruned`, `skipped`,
`failed` and per-run reasons to distinguish selection from completed deletion.

:::warning Explicit run IDs bypass the age scan

In the PHP Server, supplying `run_ids` directly targets those closed runs,
including runs younger than the current bounded retention window. It is not
an intersection with the expired-run list. Review each target as a destructive
operation; omit `run_ids` for ordinary age-based cleanup.

:::

Retention passes skip archived runs and namespaces configured as `forever`.
Archive marks history retained in the same database; it is not an independent
backup. Other retention holds, such as unfinished detached activity work, can
also prevent deletion. Check the reported reason rather than assuming a
selected run was removed.

For a record longer than the bounded 1-to-365-day window, choose between
retaining history in the database and maintaining independent verified exports.
The [long-term history procedure](../configuration/pruning-workflows.md#keeping-history-beyond-database-retention)
covers `CloseTime` selection, pagination, external bytes and export-before-prune
reconciliation. The published API has no unarchive operation. An archived run
therefore cannot later be removed through Server retention passes.

Current history exports preserve external references, not an independent copy
of their objects. The self-contained verified archive workflow is requested in
[Server #520](https://github.com/durable-workflow/server/issues/520), not supplied
by an export plus a checksum. Also reject already-pruned history rather than
trusting its `history_complete` flag; the known false-complete case remains open
in [Workflow #856](https://github.com/durable-workflow/workflow/issues/856).

Pruning removes history, related details and eligible external payloads. A
later policy extension cannot recover them. The [embedded pruning guide](../configuration/pruning-workflows.md)
describes a separate application-owned process; its cleanup API does not apply
the Server's namespace retention policy. For a different runtime,
verify its retention behavior before applying this PHP Server procedure.

### Namespaces

The native Server creates the `default` namespace when it creates its schema at
startup, and a request that omits the namespace header uses `default`. Native
refuses to start if `DW_DEFAULT_NAMESPACE` is set to anything other than
`default`. On PHP 2.x, the `server-bootstrap` command runs migrations and seeds
the namespace, and `DW_DEFAULT_NAMESPACE` changes the namespace used when a
request omits the header:

```bash
DW_DEFAULT_NAMESPACE=default
```

Create namespaces via the API:

```bash
curl -X POST http://localhost:8080/api/namespaces \
  -H "Authorization: Bearer $TOKEN" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "production",
    "description": "Production workflows",
    "retention_days": 90
  }'
```

## Health Checks

The native Server publishes three discovery endpoints. `GET /api/health` and
`GET /api/ready` need no credentials. `GET /api/cluster/info` needs a worker,
operator or admin token. None of the three requires the control-plane version
header.

### API Health

```bash
curl http://localhost:8080/api/health
```

Returns `200 OK` when the Server can run a trivial query against its database:

```json
{"status":"serving","checks":{"database":true}}
```

When that query fails, it returns `503 Service Unavailable` with
`"status":"degraded"` and `"database":false`. Because a database outage fails
this check, do not use it as a restart-triggering liveness probe. The Helm
chart uses a TCP liveness probe and puts readiness on `/api/ready`.

### Readiness

```bash
curl http://localhost:8080/api/ready
```

`/api/ready` is the deployment gate. It returns `200 OK` only when the
in-process timer and maintenance loop is running and healthy and the Server can
read every table its schema needs:

```json
{
  "status": "ready",
  "development": false,
  "checks": {
    "database": {
      "status": "ok",
      "mode": "schema_read",
      "write_capacity_verified": false,
      "tls": {
        "mode": null,
        "encrypted": null,
        "ca_verified": null,
        "hostname_verified": null,
        "evidence": "not_verified",
        "scope": "rust_to_configured_database_endpoint"
      }
    }
  },
  "qualification": "release"
}
```

Otherwise it returns `503 Service Unavailable` with `"status":"not_ready"` and
`checks.database.status: "failed"`, without the `tls` block.

- `checks.database.mode: "schema_read"` and `write_capacity_verified: false`
  mean readiness proves the schema can be read, not that the database has
  write capacity.
- `checks.database.tls` reports what the [database TLS](/docs/polyglot/server-config-reference#database-tls)
  settings guarantee for the connection. The example shows SQLite, where no
  TLS applies.
- `development` is `false` and `qualification` is `"release"` on a published
  image. A source build without a release version reports `true` and
  `"unpublished_execution_slice"`.

Native has no migration checks, rollout-safety verdict or bootstrap gate. It
creates or checks its schema at startup, and it refuses to start against a PHP
or unknown database until the [native takeover](https://github.com/durable-workflow/server/blob/a051f57fcc7e3605b301581ee9105461b3ef5815/docs/rust-server-upgrade-takeover.md)
has run.

### Server Capabilities

```bash
curl http://localhost:8080/api/cluster/info \
  -H "Authorization: Bearer $TOKEN"
```

Returns the build identity, capability flags, protocol manifests, limits and
discovery for the namespace named by `X-Namespace` (or the `namespace` query
parameter, else `default`). Abridged output from the 3.5.0 image:

```json
{
  "version": "3.5.0",
  "implementation": "rust",
  "development": false,
  "server_id": "2145ce2abb12",
  "default_namespace": "default",
  "supported_sdk_versions": {
    "php": ">=3.8.1,<4.0.0",
    "python": ">=2.13.0,<3.0.0",
    "rust": ">=4.10.3,<5.0.0",
    "cli": ">=2.0.0,<3.0.0"
  },
  "client_compatibility": {
    "schema": "durable-workflow.v2.client-compatibility",
    "version": 2,
    "authority": "protocol_manifests",
    "fail_closed": true,
    "...": "..."
  },
  "capabilities": {
    "workflow_tasks": true,
    "activity_tasks": true,
    "signals": true,
    "queries": true,
    "updates": true,
    "schedules": true,
    "child_workflow_start_turn": true,
    "condition_wait_occurrence_deadline": true,
    "timer_runtime_contract": false,
    "payload_codecs": ["avro"],
    "response_compression": []
  },
  "control_plane": {
    "version": "2",
    "request_contract": { "schema": "durable-workflow.v2.control-plane-request.contract", "version": 1, "...": "..." }
  },
  "worker_protocol": {
    "version": "1.20",
    "server_capabilities": {
      "supported_workflow_task_commands": ["schedule_activity", "start_timer", "complete_workflow", "..."],
      "query_tasks": true,
      "workflow_updates": true,
      "sticky_execution": { "supported": true, "...": "..." },
      "sticky_history_suffix": { "supported": true, "version": 1 },
      "activity_workflow_continuation": { "version": 1, "...": "..." },
      "...": "..."
    }
  },
  "namespace": { "name": "default", "exists": true, "status": "active", "...": "..." },
  "limits": { "max_payload_bytes": 2097152, "...": "..." },
  "structural_limits": { "...": "..." },
  "discovery": {
    "mode": "compatibility",
    "operator_metrics_path": "/api/system/operator-metrics",
    "task_repair_path": "/api/system/repair"
  },
  "native_unqualified": {
    "topology": "Native runs as one process role and publishes no topology manifest.",
    "...": "..."
  },
  "payload_codec": "avro",
  "qualified_for_php_takeover": false
}
```

Key field notes for client code:

- The build version is `version`, not `server_version`. `implementation` is
  `"rust"` on the native Server.
- Check `control_plane.version` and `worker_protocol.version` before calling
  the API. Native 3.5.0 serves control plane `2` and worker protocol `1.20`,
  and accepts worker requests that send any protocol version from `1.0` to
  `1.20`.
- Workflow-task commands live under
  `worker_protocol.server_capabilities.supported_workflow_task_commands`.
- Since 3.5.0, `worker_protocol.server_capabilities` advertises
  `sticky_execution` and `sticky_history_suffix`. Since 3.4.0 it advertises
  `activity_workflow_continuation`, which a worker uses only after its own
  opt-in. Enabling it currently costs throughput: in our benchmark, runs
  were 24–49% slower under load ([measurement](https://github.com/durable-workflow/server/issues/1078)).
- A capability flag set to `false` names a feature or published contract that
  native does not serve. `response_compression` is empty because native sends
  uncompressed responses.
- `native_unqualified` lists each member of the PHP 2.x response that native
  leaves out, with the reason. It includes `topology`.
- Since 3.1.0, native also serves `supported_sdk_versions` and
  `client_compatibility`, which the CLI's `server:info` and `doctor` read.
  `client_compatibility.authority` is `"protocol_manifests"`: check the
  protocol manifests, not the SDK ranges, before calling the API.
- The sole payload codec is Avro: `capabilities.payload_codecs` is exactly
  `["avro"]`.
- `qualified_for_php_takeover` is always `false`. It describes the running
  server, which never takes over a database at startup, not whether
  `upgrade-takeover` will accept yours.
- The response also carries `auth_composition_contract`,
  `principal_attribution_contract`, `service_catalog_read_contract` and
  `message_streams_contract` manifests.

## PHP Server 2.x health, readiness and topology

PHP Server 2.x, frozen at 2.11.8, publishes a larger discovery surface than
native. Everything in this section applies to PHP 2.x only; the native Server
publishes none of it.

### PHP 2.x API health

`GET /api/health` returns `200 OK` with the node's topology summary:

```json
{
  "status": "serving",
  "timestamp": "2026-04-15T12:00:00Z",
  "checks": {
    "database": "ok"
  },
  "topology": {
    "schema": "durable-workflow.v2.role-topology",
    "version": 4,
    "current_shape": "standalone_server",
    "current_process_class": "server_http_node",
    "current_roles": ["api_ingress", "control_plane", "matching", "history_projection"],
    "execution_mode": "remote_worker_protocol",
    "matching_role": {
      "queue_wake_enabled": true,
      "shape": "in_worker",
      "wake_owner": "worker_loop",
      "task_dispatch_mode": "poll",
      "partition_primitives": ["connection", "queue", "compatibility", "namespace"],
      "backpressure_model": "lease_ownership",
      "discovery_limits": {
        "poll_batch_cap": 100,
        "availability_ceiling_seconds": 1,
        "wake_signal_ttl_seconds": 60,
        "workflow_task_lease_seconds": 300,
        "activity_task_lease_seconds": 300
      }
    }
  }
}
```

### Public Topology Summary

Unauthenticated `GET /api/health` and `GET /api/ready` both publish the
responding node's `topology` summary. That public block is intentionally
smaller than `/api/cluster/info`, but it still exposes the fields needed to
identify split-role nodes before control-plane auth or namespace resolution:

- `topology.schema`
- `topology.version`
- `topology.current_shape`
- `topology.current_process_class`
- `topology.current_roles`
- `topology.execution_mode`
- `topology.matching_role.queue_wake_enabled`
- `topology.matching_role.shape`
- `topology.matching_role.wake_owner`
- `topology.matching_role.task_dispatch_mode`
- `topology.matching_role.partition_primitives`
- `topology.matching_role.backpressure_model`
- `topology.matching_role.discovery_limits.poll_batch_cap`
- `topology.matching_role.discovery_limits.availability_ceiling_seconds`
- `topology.matching_role.discovery_limits.wake_signal_ttl_seconds`
- `topology.matching_role.discovery_limits.workflow_task_lease_seconds`
- `topology.matching_role.discovery_limits.activity_task_lease_seconds`

`topology.matching_role.discovery_limits` is the frozen numeric matching-role
contract that compiles into the workflow package: `poll_batch_cap` is the
maximum batch of ready-task rows returned per poll,
`availability_ceiling_seconds` is the cross-backend tolerance applied to
`available_at` so freshly-available tasks survive sub-second timestamp drift,
`wake_signal_ttl_seconds` is the default `CacheLongPollWakeStore` signal TTL,
and `workflow_task_lease_seconds` / `activity_task_lease_seconds` are the
default workflow and activity task lease durations. Tightening any of these
values is a protocol-level change because workers and downstream tooling read
them as the authoritative matching-role contract.

The same summary appears on `/api/ready` even when the deployment is not ready,
so probes can still distinguish `server_http_node`, `scheduler_node`,
`matching_node`, and `execution_node` responses while bootstrap blockers are
active.

### PHP 2.x readiness

```bash
curl http://localhost:8080/api/ready
```

`/api/ready` is the deployment gate. It returns `200 OK` only when bootstrap
prerequisites and rollout-safety health are in a ready or warning state.

Treat the machine-readable fields as follows:

- `checks.migrations.repository_exists` and `checks.migrations.pending_migrations`
  tell you whether the migration repository exists and which migration records
  are still pending.
- `checks.migrations.adoptable_migrations` lists create-table migrations that
  only need adoption into migration history. This is a `warning`, not a
  fail-closed outage, so the server can stay ready while operators schedule the
  adoption.
- `checks.migrations.blocking_migrations` lists rollout-safety migration records
  that must land before the server should admit traffic. When this array is not
  empty, readiness fails closed with `checks.migrations.status = "pending"`.
- `checks.migrations.missing_tables` reports durable tables that are still
  absent, and `checks.migrations.operator_surface` tells you whether the v2
  operator surface is available enough to explain rollout safety once the server
  boots.
- `checks.migrations.readiness_contract.version` pins the boot and migration
  adoption contract revision that scripts should parse.
- `checks.workflow_v2` mirrors the all-namespaces rollout-safety verdict. When
  rollout-safety cannot be evaluated yet it returns `status: "blocked"` plus
  `blocked_by`, `message`, and `remediation` so operators can fix the upstream
  readiness gate instead of chasing queue symptoms.

### Workflow Bootstrap Gate

`checks.workflow_v2.status: "blocked"` is also a route-level gate, not just a
readiness signal. While workflow v2 bootstrap is blocked, the server fails
closed on workflow start/mutation, schedule mutation, bridge-adapter, and
worker-protocol routes with HTTP `503` and a machine-readable
`reason: "workflow_v2_blocked"` payload. The gate runs after role and
protocol-version validation but before namespace resolution, so blocked
requests never observe namespace existence.

The bootstrap-gate response always carries:

- `reason: "workflow_v2_blocked"` so callers branch on a machine-readable name
  instead of a prose message.
- `blocked_by`: the ordered list of upstream readiness blockers (for example
  `migrations`).
- `remediation`: the operator-facing instruction for clearing the listed
  blockers, mirrored from `/api/ready` `checks.workflow_v2.remediation`.

Bootstrap-gated route families:

- **Workflow start and mutation** — `/api/workflows` start, command, and
  run-targeted command routes.
- **Schedule mutation** — `POST /api/schedules`,
  `PUT /api/schedules/{scheduleId}`, `DELETE /api/schedules/{scheduleId}`,
  `POST /api/schedules/{scheduleId}/pause`,
  `POST /api/schedules/{scheduleId}/resume`,
  `POST /api/schedules/{scheduleId}/trigger`, and
  `POST /api/schedules/{scheduleId}/backfill`.
- **Bridge adapters** — `POST /api/bridge-adapters/webhook/{adapter}`.
- **Worker protocol** — every `/api/worker` and `/api/worker/*` route, including
  registration, heartbeat, workflow-task, query-task, and activity-task verbs.
  Worker-protocol routes return the bootstrap-gate payload in the
  worker-protocol envelope and keep the
  `X-Durable-Workflow-Protocol-Version` header so worker SDKs can branch on the
  same `reason: "workflow_v2_blocked"` field they parse from the control plane.

Schedule **reads** are intentionally exempted so operators can inspect
schedule state during recovery: `GET /api/schedules`,
`GET /api/schedules/{scheduleId}`, and
`GET /api/schedules/{scheduleId}/history` continue to serve while the
bootstrap gate is blocking other routes.

### PHP 2.x server capabilities

```bash
curl http://localhost:8080/api/cluster/info \
  -H "Authorization: Bearer $TOKEN"
```

Returns the server build version, supported SDK versions, engine capabilities,
the client compatibility policy, and the independently-versioned control-plane
and worker-protocol manifests:

```json
{
  "server_id": "server-1",
  "version": "2.0.0",
  "default_namespace": "default",
  "supported_sdk_versions": {
    "php": ">=1.0",
    "python": ">=0.2,<1.0",
    "cli": ">=0.1,<1.0"
  },
  "client_compatibility": {
    "schema": "durable-workflow.v2.client-compatibility",
    "version": 1,
    "authority": "protocol_manifests",
    "top_level_version_role": "informational",
    "fail_closed": true
  },
  "capabilities": {
    "workflow_tasks": true,
    "activity_tasks": true,
    "signals": true,
    "queries": true,
    "updates": true,
    "schedules": true,
    "child_workflow_retry_policy": true,
    "child_workflow_timeouts": true,
    "payload_codecs": ["avro"],
    "response_compression": ["gzip", "deflate"]
  },
  "control_plane": {
    "version": "2",
    "header": "X-Durable-Workflow-Control-Plane-Version",
    "request_contract": { "schema": "durable-workflow.v2.control-plane-request.contract", "version": 1, "...": "..." },
    "response_contract": { "schema": "durable-workflow.v2.control-plane-response.contract", "version": 1, "...": "..." }
  },
  "worker_protocol": {
    "version": "1.0",
    "server_capabilities": {
      "long_poll_timeout": 30,
      "supported_workflow_task_commands": [
        "complete_workflow",
        "fail_workflow",
        "continue_as_new",
        "schedule_activity",
        "start_timer",
        "start_child_workflow"
      ],
      "workflow_task_poll_request_idempotency": true,
      "poll_status": true,
      "history_page_size_default": 500,
      "history_page_size_max": 1000,
      "activity_retry_policy": true,
      "activity_timeouts": true,
      "child_workflow_retry_policy": true,
      "child_workflow_timeouts": true,
      "parent_close_policy": true,
      "non_retryable_failures": true,
      "response_compression": ["gzip", "deflate"],
      "history_compression": {
        "supported_encodings": ["gzip"],
        "compression_threshold": 8192
      }
    }
  }
}
```

Treat `client_compatibility.authority: "protocol_manifests"` as the rule for
client checks. The top-level `version` is build identity; CLI and SDK clients
should fail closed when `control_plane.version`,
`control_plane.request_contract`, or `worker_protocol.version` is missing or
unsupported.

### Role topology and deployment shape

The field-by-field reference for this manifest lives on
[Server Role Topology](/docs/polyglot/server-role-topology). Keep this
section for the inline `cluster/info` example and use the dedicated page when
you need the supported shapes, authority boundaries, failure domains, scaling
boundaries, or migration-path contract in one place.

`GET /api/cluster/info` also publishes a `topology` manifest. It is the
machine-readable role map for the node that answered the request, so operators
and automation can read one contract instead of inferring node duties from
container names or rollout runbooks.

```json
{
  "topology": {
    "schema": "durable-workflow.v2.role-topology",
    "version": 2,
    "supported_shapes": [
      "embedded",
      "standalone_server",
      "split_control_execution"
    ],
    "role_vocabulary": [
      "api_ingress",
      "control_plane",
      "matching",
      "history_projection",
      "scheduler",
      "execution_plane"
    ],
    "current_shape": "standalone_server",
    "current_process_class": "server_http_node",
    "current_roles": [
      "api_ingress",
      "control_plane",
      "matching",
      "history_projection"
    ],
    "execution_mode": "remote_worker_protocol",
    "matching_role": {
      "queue_wake_enabled": true,
      "shape": "in_worker",
      "wake_owner": "worker_loop",
      "task_dispatch_mode": "poll",
      "partition_primitives": [
        "connection",
        "queue",
        "compatibility",
        "namespace"
      ],
      "backpressure_model": "lease_ownership",
      "discovery_limits": {
        "poll_batch_cap": 100,
        "availability_ceiling_seconds": 1,
        "wake_signal_ttl_seconds": 60,
        "workflow_task_lease_seconds": 300,
        "activity_task_lease_seconds": 300
      }
    },
    "shape_assignments": {
      "embedded": {
        "process_classes": [
          {
            "name": "application_process",
            "roles": [
              "control_plane",
              "matching",
              "history_projection",
              "scheduler",
              "execution_plane"
            ]
          }
        ]
      },
      "standalone_server": {
        "process_classes": [
          {
            "name": "server_http_node",
            "roles": [
              "api_ingress",
              "control_plane",
              "matching",
              "history_projection"
            ]
          },
          {
            "name": "scheduler_node",
            "roles": ["scheduler"]
          },
          {
            "name": "worker_node",
            "roles": ["execution_plane"]
          }
        ]
      },
      "split_control_execution": {
        "process_classes": [
          {
            "name": "ingress_node",
            "roles": ["api_ingress"]
          },
          {
            "name": "control_plane_node",
            "roles": ["control_plane", "history_projection"]
          },
          {
            "name": "scheduler_node",
            "roles": ["scheduler"]
          },
          {
            "name": "matching_node",
            "roles": ["matching"]
          },
          {
            "name": "execution_node",
            "roles": ["execution_plane"]
          }
        ]
      }
    },
    "authority_boundaries": {
      "control_plane": {
        "writes": [
          "workflow_instances",
          "workflow_runs.status",
          "workflow_tasks.lifecycle"
        ]
      },
      "execution_plane": {
        "writes": [
          "workflow_tasks.outcomes",
          "activity_attempts",
          "worker_compatibility_heartbeats"
        ]
      },
      "matching": {
        "writes": [
          "workflow_tasks.leases",
          "activity_tasks.leases"
        ]
      },
      "history_projection": {
        "writes": [
          "history_events",
          "workflow_run_summaries",
          "workflow_history_exports"
        ]
      },
      "scheduler": {
        "writes": [
          "workflow_schedules.fire_state",
          "workflow_starts.scheduled"
        ]
      },
      "api_ingress": {
        "writes": ["worker_registrations"]
      }
    },
   "failure_domains": {
      "control_plane_down": {
        "effect": "workers_continue_claimed_tasks_only_until_lease_expiry",
        "operator_signal": "operator_commands_fail_fast"
      },
      "execution_plane_down": {
        "effect": "ready_tasks_accumulate_without_loss",
        "operator_signal": "operators_see_ready_depth_growth"
      },
      "matching_down": {
        "effect": "claim_falls_back_to_direct_ready_task_discovery",
        "operator_signal": "ready_depth_rises_while_claim_rate_falls"
      },
      "history_projection_down": {
        "effect": "projection_reads_may_stale_while_durable_writes_continue",
        "operator_signal": "projection_lag_seconds_may_increase"
      },
      "scheduler_down": {
        "effect": "scheduled_workflows_stop_firing_and_record_missed_runs",
        "operator_signal": "operators_see_missed_schedule_state"
      },
      "api_ingress_down": {
        "effect": "external_http_traffic_stops_at_the_edge",
        "operator_signal": "embedded_in_process_calls_may_continue"
      }
    },
    "scaling_boundaries": {
      "api_ingress": "incoming_http_request_rate",
      "control_plane": "operator_commands_and_run_lifecycle_transitions",
      "matching": "ready_task_rate_and_poller_count",
      "history_projection": "durable_event_rate",
      "scheduler": "active_schedule_count",
      "execution_plane": "workflow_and_activity_task_rate"
    },
    "migration_path": [
      {
        "step": "audit_role_boundaries",
        "result": "tooling flags cross-role writes before runtime shape changes",
        "reversible": true
      },
      {
        "step": "expose_role_bindings",
        "result": "container seams allow out-of-process adapters without patching the package",
        "reversible": true
      },
      {
        "step": "introduce_dedicated_matching_shape",
        "result": "matching can run as its own process class without changing the claim contract",
        "reversible": true
      },
      {
        "step": "split_history_projection",
        "result": "history and projections can move out of process without introducing a second writer",
        "reversible": true
      },
      {
        "step": "split_scheduler",
        "result": "schedule firing can move behind leader election while single-replica deployments stay legal",
        "reversible": true
      },
      {
        "step": "optional_execution_partitioning",
        "result": "workers can partition by namespace, connection, queue, and compatibility",
        "reversible": true
      }
    ],
    "kernel_invariants": [
      {
        "id": "single_persistence_engine",
        "summary": "one workflow database backs every topology shape; role split does not introduce a second persistence engine",
        "applies_to": ["embedded", "standalone_server", "split_control_execution"]
      },
      {
        "id": "single_worker_protocol",
        "summary": "one HTTP worker protocol carries claim, complete, fail, and heartbeat traffic across every topology; role split does not fork the worker contract",
        "applies_to": ["embedded", "standalone_server", "split_control_execution"]
      },
      {
        "id": "single_history_writer",
        "summary": "history_events has exactly one durable writer per logical event regardless of where the history/projection role runs",
        "applies_to": ["embedded", "standalone_server", "split_control_execution"]
      },
      {
        "id": "single_control_authority_per_run",
        "summary": "every mutation of a given workflow run routes through one control-plane authority; per-run row locks serialise transitions across replicas",
        "applies_to": ["embedded", "standalone_server", "split_control_execution"]
      },
      {
        "id": "embedded_topology_remains_supported",
        "summary": "the embedded shape where one process fills every role MUST stay legal; existing embedded hosts are never forced to migrate",
        "applies_to": ["embedded", "standalone_server", "split_control_execution"]
      },
      {
        "id": "role_split_is_topology_only",
        "summary": "splitting roles is a topology change, not a product fork; collapsing the roles back onto a single process is always a legal topology",
        "applies_to": ["embedded", "standalone_server", "split_control_execution"]
      }
    ]
  },
  "coordination_health": {
    "schema": "durable-workflow.v2.coordination-health.contract",
    "version": 2,
    "namespace_scope": "all_namespaces",
    "status": "ok",
    "http_status": 200,
    "warning_checks": [],
    "error_checks": [],
    "categories": {
      "correctness": "ok"
    },
    "checks": [
      {
        "name": "worker_compatibility",
        "status": "ok",
        "category": "correctness",
        "message": null
      },
      {
        "name": "activity_path",
        "status": "ok",
        "category": "correctness",
        "message": null
      }
    ],
    "routing_drains": {
      "queues_with_drains": 0,
      "draining_build_id_count": 0,
      "active_worker_count": 0,
      "draining_worker_count": 0,
      "stale_worker_count": 0,
      "queues": []
    }
  }
}
```

Treat `topology.version` as the role-manifest schema version, not as a synonym
for the top-level server build version. Automation should check that field
before assuming fields added by a newer topology manifest revision. The
current public contract includes `supported_shapes`, `role_vocabulary`,
`current_shape`, `current_process_class`, `current_roles`, `execution_mode`,
`matching_role`, `role_catalog`, `shape_assignments`,
`authority_boundaries`, `authority_surfaces`, `failure_domains`,
`supported_topologies`, `scaling_boundaries`, `migration_path`, and
`kernel_invariants`.

Read the fields as follows:

- `supported_shapes` names the legal product topologies.
- `role_vocabulary` is the fixed list of v2 role names. Treat it as the
  canonical vocabulary for automation and diagnostics.
- `current_shape`, `current_process_class`, and `current_roles` describe the
  node you queried right now. Use `current_process_class` as the node's
  declared identity, then compare the current role bundle against
  `shape_assignments` for the current shape when you need to validate that
  declaration.
- `execution_mode` distinguishes embedded local queue execution
  (`local_queue_worker`) from standalone server worker-protocol execution
  (`remote_worker_protocol`).
- `matching_role.queue_wake_enabled`, `matching_role.shape`,
  `matching_role.wake_owner`, `matching_role.task_dispatch_mode`,
  `matching_role.partition_primitives`, and
  `matching_role.backpressure_model` tell you whether the node still runs the
  in-worker wake path or expects a dedicated repair or matching loop to own
  that sweep, which routing axes remain stable, and which durable admission
  boundary the matching layer currently enforces.
- `matching_role.discovery_limits` freezes the numeric matching-role contract
  values the workflow package compiles in: `poll_batch_cap` (the maximum batch
  of ready-task rows returned per poll), `availability_ceiling_seconds` (the
  cross-backend tolerance applied to `available_at` so freshly-available tasks
  survive sub-second timestamp drift), `wake_signal_ttl_seconds` (the default
  long-poll wake-signal TTL), `workflow_task_lease_seconds` (the default
  workflow task lease), and `activity_task_lease_seconds` (the default
  activity task lease). Operators read these to verify the deployment matches
  the documented matching-role contract without grepping the package source;
  tightening any value is a protocol-level change.
- `role_catalog` and `authority_surfaces` tell you which interfaces and
  durable mutation paths each role owns on the current manifest revision.
- `shape_assignments` maps each supported shape to the process classes and role
  bundles that shape is allowed to run.
- `supported_topologies` summarizes the deployment families the product
  supports and the node classes each family expects.
- `authority_boundaries` names which durable write surfaces each role is
  expected to mutate, so operators can catch cross-role drift before they split
  a deployment.
- `failure_domains` describes the first operator-visible degradation signal when
  a role goes down, instead of leaving that expectation implicit in a runbook.
- `scaling_boundaries` names the main load dimension for each role when the
  topology is split.
- authenticated hosted routes fail closed when the responding node does not
  host the HTTP control surface. In that case the server returns `503` with
  `reason: "topology_role_unavailable"` plus `current_shape`,
  `current_process_class`, `current_roles`, `required_roles`, and
  `missing_roles` so callers can reroute to a node that actually exposes the
  requested surface.
- `coordination_health` is the fleet-wide rollout-safety summary published from
  the same discovery call. It uses `all_namespaces` scope, summarizes the
  current status and HTTP posture, lists the normalized warning/error check
  names that also feed readiness health, and adds `blocked_by`, `message`, plus
  `remediation` when rollout-safety evaluation is blocked by upstream readiness
  problems.
- `coordination_health.checks[]` always includes the frozen check `activity_path`
  next to `worker_compatibility`, `task_transport`, `routing_health`,
  `durable_resume_paths`, and the projection/scheduler checks. `activity_path`
  is the activity-side counterpart of `task_transport`: it surfaces activity
  executions whose schedule-to-start, start-to-close, schedule-to-close, or
  heartbeat deadline has passed without enforcement (`timeout_overdue`,
  `oldest_timeout_overdue_at`, `max_timeout_overdue_age_ms`) and the sustained
  retry backlog (`retrying`, `oldest_retrying_started_at`,
  `max_retrying_age_ms`). Renaming the check is a protocol-level change.
- `coordination_health.routing_drains` summarizes draining build-id cohorts
  across queues and namespaces. `queues_with_drains` greater than zero means the
  fleet is intentionally holding traffic away from at least one draining cohort.
- `migration_path` lists the ordered rollout steps from today's standalone
  distribution toward more isolated role boundaries without introducing a
  second engine. Each entry's `reversible: true` flag declares that
  collapsing back to a less-isolated shape stays a legal topology.
- `kernel_invariants` enumerates the durable-kernel guarantees the role
  split must preserve regardless of which supported shape is running:
  `single_persistence_engine`, `single_worker_protocol`,
  `single_history_writer`, `single_control_authority_per_run`,
  `embedded_topology_remains_supported`, and `role_split_is_topology_only`.
  Each entry's `applies_to` lists the supported shapes the invariant
  covers; rollout automation MAY use the field to assert that a candidate
  topology change preserves the kernel before applying the shape change.

This keeps the role split as a topology change, not a second engine or a
separate control-plane API. When a deployment evolves from a narrow
`standalone_server` fleet toward a more explicit `split_control_execution`
shape, operators still read the same discovery surface. The values under
`current_shape`, `current_roles`, `execution_mode`, `matching_role`,
`shape_assignments`, `authority_boundaries`, `failure_domains`,
`scaling_boundaries`, and `migration_path` are
versioned as one manifest so rollout tooling can reason about the same
topology surface the server ships.

The same constraint also surfaces machine-readably through
`topology.kernel_invariants` so rollout automation can verify that no
candidate topology change introduces a second persistence engine, a
forked worker protocol, a second history writer, or a non-reversible
migration before applying the change.

The hosted-route gate applies only to authenticated API and worker endpoints.
`GET /api/health`, `GET /api/ready`, and authenticated `GET /api/cluster/info`
stay available for discovery, liveness, and topology inspection even on
`scheduler_node`, `matching_node`, or `execution_node` processes that do not
host the current HTTP control surface.

For carrier-neutral external handlers, the same endpoint publishes
`worker_protocol.external_execution_surface_contract`. That manifest names the
[activity-grade external execution surface](/docs/polyglot/external-execution),
links the external task input/result envelope contracts, and keeps workflow
replay, `ContinueAsNew`, signal/update/query ordering, and event-history
interpretation inside real runtimes.

Key field notes for client code:

- The app version is `version`, not `server_version`.
- Workflow-task command capabilities live under `worker_protocol.server_capabilities.supported_workflow_task_commands`, not at the top of `worker_protocol`. The same nested object is echoed on every worker-plane response via the `server_capabilities` field.
- `worker_protocol.server_capabilities.poll_status` means poll responses keep a
  machine-readable `poll_status` field even when no task is leased, so workers
  can distinguish `empty`, `throttled`, `unavailable`, and `draining` outcomes
  without scraping prose error messages.
- Worker command-option capabilities, including retry policies, timeout fields, parent-close policy, and non-retryable failures, are also echoed in `server_capabilities` so workers can negotiate behavior without a separate cluster-info request.
- The sole v2 payload codec lives under `capabilities.payload_codecs`; the list is exactly `['avro']`. PHP-only v1 import/drain serializers are internal migration mechanics and never appear in runtime capabilities.

## Connecting Workers

Workers poll the server for tasks and execute workflow code or activities. See the [Worker Protocol](/docs/polyglot/worker-protocol) reference for the full API contract.
For the route role matrix, namespace lookup rules, and exact worker
registration payload, see
[Namespace, Auth, And Worker Registration](/docs/polyglot/namespace-auth-workers).

### PHP Workers

Framework-neutral PHP applications and remote workers use the standalone-server
SDK. They do not install Laravel or the embedded Workflow package:

<!-- docs-example id="server.php-sdk.install" -->
```bash
composer require durable-workflow/sdk:3.18.2
```

Create a client and register workflow or activity callables with a worker:

```php
use DurableWorkflow\Client;
use DurableWorkflow\Worker;
use DurableWorkflow\Worker\WorkflowContext;

$client = new Client(
    'http://localhost:8080',
    token: getenv('DURABLE_WORKFLOW_AUTH_TOKEN') ?: null,
    namespace: 'default',
);

$worker = new Worker($client, taskQueue: 'polyglot-php');
$worker->registerWorkflow(
    'invoice',
    static function (WorkflowContext $context, string $invoiceId): array {
        $context->activity('charge-card', [$invoiceId]);

        return ['invoice_id' => $invoiceId, 'status' => 'paid'];
    },
);
$worker->run();
```

See the [PHP SDK guide](/docs/polyglot/php/) for the client, authentication,
payload, and worker lifecycle. The separate `durable-workflow/workflow` package
is the embedded Laravel runtime and the engine hosted inside the published
server image. Embedded Laravel workflows run package-local tasks through the
application's queue worker and do not require this standalone server.

### Python Workers

Python workers use the `durable-workflow` SDK:

```bash
pip install durable-workflow==2.17.2
```

See the [Python SDK](/docs/polyglot/python) guide for worker setup.

### Custom Language Workers

Any language can implement a worker by:
1. Registering with `POST /api/worker/register`
2. Long-polling for tasks with `POST /api/worker/workflow-tasks/poll`, `POST /api/worker/activity-tasks/poll`, or `POST /api/worker/query-tasks/poll`
3. Completing tasks with `POST /api/worker/workflow-tasks/{id}/complete`, `POST /api/worker/activity-tasks/{id}/complete`, or `POST /api/worker/query-tasks/{id}/complete`

All requests require:
- `Authorization: Bearer $TOKEN`
- `X-Namespace: your-namespace`
- `X-Durable-Workflow-Protocol-Version: 1.0`

The server validates that the namespace exists. Register it via
`POST /api/namespaces` before directing workers or clients at it, or the
server returns `404` with `reason: "namespace_not_found"`.

See the [server README](https://github.com/durable-workflow/server#getting-started-end-to-end-workflow) for a curl-based walkthrough.

See [Task Queue Admission](/docs/polyglot/task-queue-admission) to tune
worker registration slots, server-side active lease caps, per-minute dispatch
budgets, and query-task backpressure.

## CLI

The [Durable Workflow CLI](/docs/polyglot/cli) provides a shell interface to the server:

```bash
# Install — Linux and macOS
curl -fsSL https://durable-workflow.com/install.sh | sh

# Install — macOS (Homebrew alternative)
brew install durable-workflow/tap/dw

# Install — Windows (PowerShell)
# irm https://durable-workflow.com/install.ps1 | iex

# Configure
export DURABLE_WORKFLOW_SERVER_URL=http://localhost:8080
export DURABLE_WORKFLOW_AUTH_TOKEN=your-token
export DURABLE_WORKFLOW_NAMESPACE=default

# Use
dw server:health
dw workflow:list
dw workflow:start --type=my-workflow --input='["value"]'
dw workflow:start --type=my-workflow --input-file=input.json
```

See the [CLI install page](/docs/polyglot/cli#install) for a platform-detecting installer and direct binary downloads.

Task queue commands include admission status for workflow tasks, activity
tasks, and query tasks. Use them to distinguish missing workers, saturated
worker slots, server-side active lease or dispatch-rate throttling, and
query-task overflow.

## Deployment

Use the [self-hosting deployment guide](/docs/deployment) to choose a
supported topology before deploying production traffic. It separates local
development, single-node production, small clustered deployments, raw
Kubernetes manifests, and support-led topologies.

The self-serve small-cluster contract is a PHP Server 2.x contract and is
deliberately narrow: 2-3 stateless API nodes behind a load balancer, one shared external MySQL or PostgreSQL database,
shared Redis, independently scaled workers, and exactly one scheduler or
maintenance runner. Choose stop-the-world upgrades or
[rolling upgrades](/docs/rolling-upgrades) per release; the rolling-upgrade
contract names the version-skew, schema, drain, readiness, and rollback
guarantees that must hold. SQLite clustering, Redis-less multi-node mode,
duplicate schedulers, active/active multi-region, and provider-specific
failover are outside that contract until separately validated. No native
Compose small-cluster recipe has been published; for more than one native
Server replica, use the separately versioned Helm chart's native mode,
documented in the
[deployment guide](/docs/deployment#helm-chart-for-kubernetes). Active/passive
multi-region material in the
[self-hosting guide](/docs/deployment#activepassive-multi-region) is
support-led evaluation guidance, not a proven self-serve 2.0 contract; each
candidate region must still start from the documented single-region or
small-cluster shape.

For self-hosted server deployments, start from published images rather than
source-tree builds:

- Docker Hub: `durableworkflow/server:3.10.0`
- GitHub Container Registry: `ghcr.io/durable-workflow/server:3.10.0`
- Published-image Compose:
  [`docker-compose.published.yml`](https://github.com/durable-workflow/server/blob/3.10.0/docker-compose.published.yml)
- Helm chart: `oci://ghcr.io/durable-workflow/charts/durable-workflow --version 2.0.0`
- Raw Kubernetes manifests (PHP 2.x shape):
  [`k8s/`](https://github.com/durable-workflow/server/tree/main/k8s)

Production deployments should pin a version tag or image digest, use
role-scoped credentials, let the native Server check its schema at startup
(PHP 2.x: run bootstrap/migrations) before serving traffic, and prove readiness with `/api/ready`, `/api/cluster/info`, and worker
registration. Do not shift production traffic based on `/api/health` alone.

## API Reference

For a complete endpoint-by-endpoint reference, including required headers,
roles, worker-protocol routes, external payload storage routes, and named error
reasons, see the [Server API Reference](/docs/polyglot/server-api-reference).

The server exposes three API surfaces:

### Control Plane

Start, describe, signal, query, update, cancel, and terminate workflows; manage namespaces, task queues, schedules, search attributes, and workers. Every control-plane request requires `X-Durable-Workflow-Control-Plane-Version: 2`. Requests without it are rejected with `missing_control_plane_version`.

Key endpoints:
- `POST /api/workflows` — Start a workflow
- `GET /api/workflows/{id}` — Describe a workflow
- `POST /api/workflows/{id}/signal/{name}` — Send a signal
- `POST /api/workflows/{id}/query/{name}` — Execute a query
- `POST /api/workflows/{id}/update/{name}` — Execute an update
- `POST /api/workflows/{id}/cancel` — close the run as cancelled immediately
- `POST /api/workflows/{id}/terminate` — Terminate immediately
- `GET /api/workflows/{id}/runs/{runId}/history` — List run history events
- `GET /api/workflows/{id}/runs/{runId}/history/export` — Export a replay bundle
- `GET /api/namespaces`, `POST /api/namespaces`, `GET|PUT /api/namespaces/{namespace}` — Namespace management
- `GET /api/workers`, `GET|DELETE /api/workers/{id}` — Worker fleet management
- `GET /api/task-queues`, `GET /api/task-queues/{taskQueue}` — Task queue backlog, pollers, leases, and admission visibility
- `GET|POST /api/schedules`, `GET|PUT|DELETE /api/schedules/{id}`, `POST /api/schedules/{id}/{pause|resume|trigger|backfill}` — Schedule management
- `GET|POST|DELETE /api/search-attributes` — Search attribute management
- `GET|POST|PUT|DELETE /api/service-endpoints...` — Admin-only service catalog endpoints, nested services, operation bindings, and durable service-call snapshots
- `POST /api/system/repair/pass`, `POST /api/system/activity-timeouts/pass`, `POST /api/system/retention/pass` — Operator passes

Workflow control-plane responses, including run-history listing responses,
include the nested `control_plane` contract metadata that identifies the
operation and response contract version. History export is intentionally not
wrapped in that envelope; it returns the replay bundle unchanged so the bundle
integrity checksum and optional signature cover the exact artifact received by
the client.

Validation failures return HTTP 422 with `reason: validation_failed` plus
`errors` and `validation_errors`. Workflow operation routes also project that
reason and validation detail into `control_plane.reason` and
`control_plane.validation_errors`. Current run-targeted command routes project
the URL `run_id` in the response and `control_plane.run_id`, so clients can
distinguish instance-level commands from explicit selected-run commands.

Task queue visibility is the operator surface for deciding whether a queue is
falling behind because durable backlog is growing, workers have no available
slots, or the server is enforcing admission limits. `GET /api/task-queues`
returns one summary entry per queue; `GET /api/task-queues/{taskQueue}`
expands one queue with `pollers` and `current_leases`. Both routes expose
`stats.approximate_backlog_count`, `stats.approximate_backlog_age_seconds`,
and the per-kind `stats.workflow_tasks.*` / `stats.activity_tasks.*` readiness
and lease counters. The detailed route also includes the `admission` object so
automation can separate worker-capacity pressure from server-side queue or
query-task throttling. Fleet-level durable inflow versus dispatch rates live on
the operator-metrics surfaces (`operator_metrics.backlog.tasks_added_last_minute`
and `operator_metrics.backlog.tasks_dispatched_last_minute`). The task-queue
routes also expose `stats.tasks_added_last_minute` and
`stats.tasks_dispatched_last_minute` for the same comparison within one queue.

### Worker Protocol

Workers register, poll for tasks, heartbeat, and complete tasks. Requires `X-Durable-Workflow-Protocol-Version: 1.0`.

Key endpoints:
- `POST /api/worker/register` — Register a worker
- `POST /api/worker/workflow-tasks/poll` — Long-poll for workflow tasks
- `POST /api/worker/workflow-tasks/{id}/complete` — Complete workflow task
- `POST /api/worker/query-tasks/poll` — Long-poll for server-routed workflow query tasks
- `POST /api/worker/query-tasks/{id}/complete` — Complete workflow query task
- `POST /api/worker/query-tasks/{id}/fail` — Fail or reject workflow query task
- `POST /api/worker/activity-tasks/poll` — Long-poll for activity tasks
- `POST /api/worker/activity-tasks/{id}/complete` — Complete activity task

See the [Worker Protocol](/docs/polyglot/worker-protocol) reference for details.

### Discovery (unversioned)

The only endpoints that do **not** require `X-Durable-Workflow-Control-Plane-Version` are discovery and health probes:

- `GET /api/health` — Database liveness check (no auth required)
- `GET /api/ready` — Readiness gate (no auth required)
- `GET /api/cluster/info` — Server capabilities, protocol versions, and the sole Avro payload codec. Clients should hit this first to discover which control-plane and worker-protocol versions the server supports.

## Troubleshooting

### Workers not receiving tasks

**Check:**
1. Workers registered? `curl http://localhost:8080/api/workers -H "Authorization: Bearer $TOKEN" -H "X-Durable-Workflow-Control-Plane-Version: 2" -H "X-Namespace: default"`
2. Workers polling correct task queue?
3. Workflow started with matching task queue?
4. PHP 2.x only: cache backend shared across server instances?

### Long-poll connections timing out immediately

**Check:**
1. Load balancer timeout set higher than long-poll timeout (default: 60s)?
2. PHP 2.x only: cache driver supports atomic locks? Test with `php artisan workflow:v2:doctor --strict`
3. PHP 2.x only: Redis reachable from server?

### Database connection errors

**Check:**
1. Database host and port correct?
2. Credentials valid?
3. Database exists?
4. Native: does the Server log a schema refusal at startup? It refuses a PHP
   or unknown database until the [native takeover](https://github.com/durable-workflow/server/blob/a051f57fcc7e3605b301581ee9105461b3ef5815/docs/rust-server-upgrade-takeover.md)
   has run. PHP 2.x: migrations run? `php artisan migrate:status`

### Auth failures

**Check:**
1. `DW_AUTH_DRIVER` matches client auth method?
2. Token (or, on PHP 2.x, HMAC secret) matches between server and client?
3. Auth headers present? `Authorization: Bearer $TOKEN` or HMAC signature headers?

## Learn More

- [Worker Protocol Reference](/docs/polyglot/worker-protocol) — Full API contract for workers
- [Embedded to Server Migration](/docs/polyglot/embedded-to-server) — Adopt the server from a Laravel embedded v2 app
- [Python SDK](/docs/polyglot/python) — Build Python workers
- [CLI](/docs/polyglot/cli) — Command-line interface
- [Server Repository](https://github.com/durable-workflow/server) — Source code, issues, releases

<!-- Source: docs/polyglot/cli.mdx -->

# CLI

The Durable Workflow CLI (`dw`) is a shell interface to the
[standalone server](/docs/polyglot/server). It lets operators start, list,
signal, query, update, repair, cancel, terminate, and archive workflows, manage
schedules, inspect task queues, and check server health — all from the command
line.

The same CLI works against any Durable Workflow server, regardless of which
language your workflows are written in. See
[Client and Worker Capabilities](/docs/polyglot/cli-python-parity/)
for the supported and intentionally different CLI, PHP, Python, and Rust
surfaces.

## Install

### Verify

```bash
dw --version
```

### Supported channel install

The default installer follows the CLI release from the last passing qualified
artifact tuple without requiring an RC sequence number. A newer published CLI
prerelease is not selected until the public compatibility authority qualifies
it. For reproducible CI, record the resolved `dw --version` value and pass that
exact tag through `VERSION` on later runs.

Both installer scripts download `SHA256SUMS` from the release and verify the
asset checksum before replacing `dw`, so a tampered mirror fails the install.

## Linux and macOS (shell installer)

```bash
curl -fsSL https://durable-workflow.com/install.sh | sh
dw --version
```

`VERSION` accepts any published release tag, `supported`, `prerelease`, or
`stable`. Leave it unset to follow the qualified `supported` channel. The
explicit `prerelease` channel requires the qualified release to be a
prerelease. Additional environment variables:

- `DURABLE_WORKFLOW_INSTALL_DIR` — install location (default `~/.local/bin`).
- `DURABLE_WORKFLOW_BIN_NAME` — installed executable name (default `dw`).

A GitHub Actions example:

```yaml
- name: Install Durable Workflow CLI
  run: |
    curl -fsSL https://durable-workflow.com/install.sh | sh
    echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- name: Verify
  run: dw --version
```

## Windows (PowerShell installer)

```powershell
irm https://durable-workflow.com/install.ps1 | iex
dw --version
```

The installer writes `dw.exe` to `%USERPROFILE%\.durable-workflow\bin` and adds
that directory to the user `PATH`. Set
`$env:DURABLE_WORKFLOW_INSTALL_NO_PATH_UPDATE = '1'` first to leave the user
`PATH` unchanged, for example in a temporary CI install.

### Moving from `dw.phar`

CLI 2.5.0 and later ship no PHAR. If you run `php dw.phar`, switch to the
standalone binary, which needs no PHP, with the shell or PowerShell installer
above. Then delete `dw.phar`. The installers check the binary against the
release's `SHA256SUMS` before installing it.

The 2.3.x PHAR receives security fixes only until 2027-02-28. It is
unsupported after that date.

If you don't pipe scripts into a shell, download the release and verify it
first. Pick your platform's asset: `dw-linux-x86_64`, `dw-linux-aarch64`,
`dw-macos-aarch64`, `dw-macos-x86_64` or `dw-windows-x86_64.exe`.

```bash
mkdir release
gh release download 2.8.0 --repo durable-workflow/cli --dir release
sh release/verify-release.sh --attest release/
install -m 0755 release/dw-linux-x86_64 ~/.local/bin/dw
```

On Windows, download, verify and copy without running a script:

```powershell
New-Item -ItemType Directory -Force release | Out-Null
gh release download 2.8.0 --repo durable-workflow/cli --dir release
if ($LASTEXITCODE -ne 0) { throw 'gh release download failed' }
$expected = ((Select-String -Path release\SHA256SUMS -Pattern ' dw-windows-x86_64\.exe$').Line -split '\s+')[0]
$actual = (Get-FileHash release\dw-windows-x86_64.exe -Algorithm SHA256).Hash.ToLower()
if ($actual -ne $expected) { throw 'dw-windows-x86_64.exe does not match SHA256SUMS' }
gh attestation verify release\dw-windows-x86_64.exe --repo durable-workflow/cli --signer-workflow durable-workflow/cli/.github/workflows/release-rust.yml --deny-self-hosted-runners
if ($LASTEXITCODE -ne 0) { throw 'dw-windows-x86_64.exe failed attestation verification' }
New-Item -ItemType Directory -Force "$env:LOCALAPPDATA\Programs\dw" | Out-Null
Copy-Item release\dw-windows-x86_64.exe "$env:LOCALAPPDATA\Programs\dw\dw.exe"
```

Add `%LOCALAPPDATA%\Programs\dw` to your `PATH` if it isn't there already.

In a 2.x PHAR install, `dw upgrade` still answers "Download the matching PHAR
manually". That message comes from the old release and can't be changed, so
use the commands above instead. Standalone-binary installs upgrade as usual
with `dw upgrade`.

On an Intel Mac, move from 2.x with the shell installer. Published 2.x
`dw upgrade` has no Intel macOS asset to move to.

### Update An Installed Binary

For standalone release installs (the shell and PowerShell installers above),
`dw upgrade` replaces the running binary with the latest published release
after verifying the asset against the release's `SHA256SUMS`.

```bash
dw upgrade                 # upgrade through the stable release channel
dw upgrade --tag=<release-tag>     # pin to a specific release tag
dw upgrade --dry-run       # resolve the target release without downloading
```

`dw upgrade` refuses to rewrite Composer vendor and Homebrew cellar installs
because those paths are owned by another tool. Reinstall a pinned
public release with the installer, update the owning package manager, or use
`brew upgrade durable-workflow/tap/dw` for tap-managed Homebrew installs. See the
[CLI reference](./cli-reference.md#self-upgrade) for the full stable option
and status-field contract.

## Configure

Point the CLI at your server:

```bash
export DURABLE_WORKFLOW_SERVER_URL=http://localhost:8080
export DURABLE_WORKFLOW_AUTH_TOKEN=your-token
export DURABLE_WORKFLOW_NAMESPACE=default
```

Or pass them per-command:

```bash
dw --server=http://localhost:8080 --token=your-token workflow:list
```

## Five-Minute Operator Quickstart

This path is for checking the CLI against a real standalone server without
writing application code. It starts the published local server stack, installs
a pinned `dw`, creates a reusable profile, starts one workflow, and watches the
run reach the worker queue.

```bash
export DW_SERVER_IMAGE=durableworkflow/server:3.10.0
export DW_AUTH_TOKEN=dev-token

docker volume create durable-workflow-cli-quickstart

docker rm -f durable-workflow-server >/dev/null 2>&1 || true
docker run -d --name durable-workflow-server \
  -p 127.0.0.1:8080:8080 \
  -v durable-workflow-cli-quickstart:/app/database \
  -e DW_AUTH_TOKEN="$DW_AUTH_TOKEN" \
  "$DW_SERVER_IMAGE"

until curl -sf http://localhost:8080/api/ready >/dev/null; do sleep 1; done
```

Install the CLI in another terminal:

```bash
curl -fsSL https://durable-workflow.com/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"
dw --version
```

Save the server connection once:

```bash
dw env:set local \
  --server=http://localhost:8080 \
  --token=dev-token \
  --namespace=default \
  --make-default

dw doctor
dw server:health
```

Start a workflow and inspect the run:

```bash
dw workflow:start \
  --type=quickstart.order \
  --workflow-id=quickstart-order-001 \
  --task-queue=quickstart \
  --input='{"order_id":"order-001","total":42.50}' \
  --json

dw workflow:describe quickstart-order-001 --output=json
dw workflow:history quickstart-order-001 <run-id-from-start-or-describe>
dw task-queue:describe quickstart
```

The run is now durable server state. Until a worker for `quickstart.order`
polls the `quickstart` queue, `dw task-queue:describe quickstart` is the fastest
way to see why the run is waiting. When a worker is attached, use
`dw watch workflow quickstart-order-001` to follow the run to a terminal state.
For an end-to-end run that attaches a published Python SDK worker and reaches
`status=completed`, follow the
[Durable Workflow 2.0 quickstart](/docs/quickstart).

## Commands

### Server

```bash
dw server:health          # Check server health
dw server:info            # Show server version, role topology, protocols, and worker fleet
```

`dw server:info` mirrors `GET /api/cluster/info`. In addition to build,
protocol, and worker-fleet facts, it prints the server role-topology manifest:
supported shapes, the current shape/process class/roles, matching-role wake
and partition details, current write boundaries, scaling boundaries, and
failure domains. Use `--output=json` when automation needs the raw
`topology.*` fields, or read
[Server Role Topology](./server-role-topology.md) for the field-by-field
contract behind that output.

### Workflows

```bash
dw workflow:list                                          # List workflows
dw workflow:start --type=MyWorkflow --input='["arg1"]'    # Start a workflow
dw workflow:start --type=MyWorkflow --input-file=input.json
dw workflow:describe <workflow-id>                         # Describe a workflow
dw workflow:signal <workflow-id> <signal-name> --input='["ok"]'
dw workflow:query <workflow-id> <query-name>               # Run a query
dw workflow:cancel <workflow-id>                           # Close as cancelled immediately
dw workflow:terminate <workflow-id>                        # Force terminate
dw workflow:history <workflow-id> <run-id>                 # Show run history
```

Every command that accepts caller payloads uses the same input shape:
`--input` for inline values, `--input-file` for a file path or `-` for stdin,
and `--input-encoding=json|raw|base64`. JSON is the default; raw and base64
inputs are passed as one positional workflow argument.

Use `dw workflow:update-outcome` to read an accepted update later without
submitting it again. Pending is distinct from a completed null result. See
[read an update outcome](./cli-reference.md#read-update-outcome) for the Server
capability requirement and wait options.

### Bridge Adapters

```bash
dw bridge:webhook stripe \
  --action=start_workflow \
  --idempotency-key=stripe-event-1001 \
  --target='{"workflow_type":"orders.fulfillment","task_queue":"external-workflows"}' \
  --input='{"order_id":"order-1001"}'

dw bridge:webhook pagerduty \
  --action=signal_workflow \
  --idempotency-key=pd-event-3003 \
  --target='{"workflow_id":"wf-remediation-42","signal_name":"incident_escalated"}' \
  --input='{"severity":"critical"}' \
  --json
```

Bridge adapters are bounded ingress tools for integration events. They return
named bridge outcomes for accepted, duplicate, and rejected deliveries; they do
not become workflow runtimes.

### Schedules

```bash
dw schedule:list                          # List schedules
dw schedule:create --workflow-type=MyWorkflow \
  --cron='0 * * * *'                      # Create hourly schedule
dw schedule:update <schedule-id> --input-file=input.json
dw schedule:describe <schedule-id>        # Describe a schedule
dw schedule:pause <schedule-id>           # Pause a schedule
dw schedule:resume <schedule-id>          # Resume a schedule
dw schedule:trigger <schedule-id>         # Trigger immediately
dw schedule:delete <schedule-id>          # Delete a schedule
```

### Activities

```bash
dw activity:complete <task-id> <attempt-id> --input='{"ok":true}'
dw activity:complete <task-id> <attempt-id> --input-file=result.json
dw activity:fail <task-id> <attempt-id> --message='upstream failed'
```

### Workers and Task Queues

```bash
dw worker:list                    # List registered workers
dw worker:describe <worker-id>    # Describe a worker
dw task-queue:list                # List active task queues
dw task-queue:describe <queue>    # Describe a task queue
```

`dw task-queue:list` is the compact fleet view. It shows admission status
columns for workflow, activity, and query tasks. `dw task-queue:describe`
expands the same server payload with pollers, current leases, queue,
namespace, and downstream budget-group dispatch capacity, remaining capacity,
budget source, and query-task pending capacity.

For scripts, `dw task-queue:describe --json` exposes queue-local backlog,
lease, poller, and admission state. Use it to distinguish missing workers from
admission throttling or a single queue that is building backlog:

```bash
dw task-queue:describe orders --json | jq '.stats | {
  approximate_backlog_count,
  approximate_backlog_age_seconds,
  workflow_tasks,
  activity_tasks
}'
```

Fleet-level durable inflow versus dispatch rates do not come from the
task-queue JSON contract. Read them from `dw system:operator-metrics --json`
instead:

```bash
dw system:operator-metrics --json | jq '.operator_metrics.backlog | {
  tasks_added_last_minute,
  tasks_dispatched_last_minute
}'
```

See [Task Queue Admission](/docs/polyglot/task-queue-admission) for the
operator tuning contract behind those fields.

To inspect the matching-role routing contract on the node you queried, read the
same payload's `matching_role` block:

```bash
dw system:operator-metrics --json | jq '.operator_metrics.matching_role | {
  queue_wake_enabled,
  shape,
  task_dispatch_mode,
  partition_primitives,
  backpressure_model
}'
```

`partition_primitives` freezes the routing axes (`connection`, `queue`,
`compatibility`, `namespace`) and `backpressure_model` tells you whether the
engine is relying on lease occupancy or some other admission boundary. Current
v2 reports `lease_ownership`.

### Namespaces

```bash
dw namespace:list                             # List namespaces
dw namespace:create --name=production         # Create a namespace
dw namespace:describe <namespace>             # Describe a namespace
```

### Search Attributes

```bash
dw search-attribute:list                                  # List search attributes
dw search-attribute:create --name=env --type=keyword      # Register an attribute
```

## Exit Codes

The CLI uses a stable exit-code policy so scripts and CI pipelines can react
to specific failure modes without parsing stderr. Values follow Symfony
Console's canonical `0`/`1`/`2` for success / failure / usage, and extend
from there.

| Code | Name | Meaning |
|------|------|---------|
| `0` | `SUCCESS` | Operation completed successfully. |
| `1` | `FAILURE` | Generic failure — command ran but did not succeed. |
| `2` | `INVALID` | Invalid usage — bad arguments, unknown options, or local validation. Also returned for HTTP 4xx responses that are not covered below (e.g. 400, 422). |
| `3` | `NETWORK` | Could not reach the server (connection refused, DNS failure, TLS handshake failure, transport error). |
| `4` | `AUTH` | Authentication or authorization failure. Returned for HTTP `401` and `403`. |
| `5` | `NOT_FOUND` | Resource not found. Returned for HTTP `404`. |
| `6` | `SERVER` | Server error. Returned for HTTP `5xx`. |
| `7` | `TIMEOUT` | Request timed out before the server responded. Also returned for HTTP `408`. |
| `8` | `COMPATIBILITY` | Compatibility failure. The CLI and Server cannot safely run the command, for example because the Server does not advertise a capability it needs, so the command refused before sending the request. |

Example:

```bash
dw workflow:describe chk-does-not-exist
echo $?  # 5 (NOT_FOUND)

dw server:health --server=http://unreachable:9999
echo $?  # 3 (NETWORK)
```

The canonical source is [`DurableWorkflow\Cli\Support\ExitCode`](https://github.com/durable-workflow/cli/blob/main/src/Support/ExitCode.php) in the CLI repository.

## Reference

See the [CLI command reference](/docs/polyglot/cli-reference) for command
shapes, options, output modes, and automation failure behavior.

<!-- Source: docs/polyglot/cloud-control-plane.md -->

# Cloud Managed Runtime

Durable Workflow Cloud is a managed orchestration service. Cloud operates both
the hosted control plane and the orchestration runtime, including workflow
state, history, schedules, task queues, leases, and durable visibility.
Customers run SDK clients and workers against a provisioned Cloud namespace.

This is separate from self-hosting. A self-hosted Durable Workflow Server runs
independently and is never attached to Cloud.

## Plans And Pricing

Pay for provisioned capacity, not workflow semantics. Each namespace receives
an isolated managed runtime; customer PHP, Python, and Rust workers run in your
own environment. Check [Cloud pricing](https://cloud.durable-workflow.com/pricing)
and the dashboard for current provisioning availability.

Cloud Dev is available now. The four HA plans are in qualification and cannot
yet be purchased or provisioned. Their prices, capacities, availability and SLA
terms below are planned terms, not current purchase offers.

| Plan | Runtime capacity | Included durable storage | Availability | Price (USD) |
| --- | --- | --- | --- | --- |
| Cloud Dev | 1 vCPU, 1 GB RAM | 5 GB | Single host, no SLA | $0.075/hour, $2 active-month minimum, $50/month cap |
| Cloud Standard | 2 vCPU, 4 GB RAM | 25 GB | Planned single-region HA, 99.99% SLA | $0.45/hour, $10 active-month minimum, $300/month cap |
| Cloud Multi-Region | 2 vCPU, 4 GB RAM | 25 GB | Planned multi-region HA, 99.99% SLA | $0.60/hour, $15 active-month minimum, $400/month cap |
| Cloud Business | 4 vCPU, 8 GB RAM | 100 GB | Planned single-region HA, 99.99% SLA | $2.25/hour, $50 active-month minimum, $1,500/month cap |
| Cloud Business Multi-Region | 4 vCPU, 8 GB RAM | 100 GB | Planned multi-region HA, 99.99% SLA | $3.00/hour, $75 active-month minimum, $2,000/month cap |

Capacity covers the managed runtime components, not customer worker compute.
For planned HA plans, replication and standby capacity are included in the price; the table
does not add replicas together as extra workflow-execution capacity. Every plan
includes Managed Waterline, basic encrypted backups, upgrades, and a stable
runtime URL.

Provisioned runtimes are billed for runtime time, metered by the minute from
activation to deprovisioning. Minimums and caps apply per runtime purchase in a
UTC calendar month. Idle provisioned time is billable; a month with no
provisioned time has no runtime charge. A fresh purchase has its own minimum
and cap; service-managed repair or recovery does not reset them. Stripe invoices
accrued usage monthly. Prices exclude applicable taxes.

Storage is fixed at the included allocation. There are no separately purchased
storage add-ons, automatic storage increases, or storage overage charges. For
more space, [request a larger runtime plan](https://cloud.durable-workflow.com/contact)
before reaching the limit. Moving from single-region to multi-region alone
does not increase storage. A capacity upgrade requires an actual runtime change,
not just a billing change; it is currently arranged with support. Network use
is measured, not currently billed, and is not unlimited.

See [Cloud pricing](https://cloud.durable-workflow.com/pricing) to choose a plan.
For larger capacity, different connectivity, SSO, enterprise support, or a
custom availability requirement, [contact us](https://cloud.durable-workflow.com/contact).

## Managed Service Boundary

The customer-visible boundary is one Cloud namespace:

```text
Cloud organization
  project
    environment
      namespace
        stable runtime URL
        client runtime credential  ---> workflow starts and commands
        worker runtime credential  ---> registration, polling, and completion

Cloud-operated runtime
  workflow state and history
  schedules and task queues
  leases, matching, and visibility
  Managed Waterline
```

Cloud owns namespace provisioning, runtime operation, persistence, placement,
runtime health, recovery, and the Managed Waterline surface for the namespace.
Customers own application code, workflow and activity implementations, and the
processes that run their workers. Cloud customers do not deploy a separate
Waterline service.

The namespace's HTTPS endpoint connects directly to its managed runtime.
SDK traffic is not proxied through the Cloud website/control-plane application.
Use the returned runtime URL unchanged, without appending `/api`; the SDK and
CLI construct their API paths. Administrative API calls still use
`https://cloud.durable-workflow.com/api/v1`.

Cloud administration and runtime traffic use different credentials:

- A Cloud API key (`dwc_...`) manages projects, environments, namespaces,
  billing, and runtime-credential lifecycle.
- A client runtime credential (`dwr_...`) starts and controls workflows in one
  managed namespace.
- A worker runtime credential (`dwr_...`) registers workers, long-polls for
  tasks, sends heartbeats, and settles work in that namespace.

A Cloud API key is not accepted by the namespace runtime URL. Runtime
credentials are scoped to one namespace and role, returned only when created,
and omitted from later list and audit responses.

If Cloud onboarding uses the CLI, install it from the [CLI guide](./cli.mdx)
and update an existing standalone installation explicitly with `dw upgrade`.
The CLI never updates in the background. After installing or upgrading, run
`command -v dw` and `dw --version`; resolve any installer `PATH` remediation
before using Cloud credentials so the selected release is the active binary.

## Provision And Connect A Namespace

### 1. Create and provision the namespace

In the [Cloud dashboard](https://cloud.durable-workflow.com/), create an
organization, project, environment, and namespace, then select its capacity
plan. Creating an account or namespace definition does not provision paid
capacity. Complete payment setup through Stripe, then select **Provision** on
the namespace page. Adding a card alone does not start provisioning.

Provisioning is asynchronous. The namespace page updates its progress while
Cloud prepares the runtime. Wait for it to become active before starting SDK
work. The page also provides runtime-credential creation and Managed Waterline
access for inspecting workflows and their history.

For API-driven setup, complete payment setup first. The following example uses
the organization's default plan; `capacity_plan_version` can select another
available plan. Do not supply your own Server URL, deployment identifier, or
placement record:

```bash
curl -X POST \
  https://cloud.durable-workflow.com/api/v1/projects/PROJECT/environments/ENVIRONMENT/namespaces \
  -H "Authorization: Bearer dwc_..." \
  -H "Content-Type: application/json" \
  -d '{"name":"orders","retention_days":30}'

curl -X POST \
  https://cloud.durable-workflow.com/api/v1/projects/PROJECT/environments/ENVIRONMENT/namespaces/orders/provision \
  -H "Authorization: Bearer dwc_..."
```

After provisioning completes, the namespace response provides its stable
`runtime_url`, its `runtime_namespace`, managed status, and customer-visible
region information. Treat the returned URL and namespace value as configuration
owned by Cloud; do not derive an endpoint or replace it with a self-hosted
Server address.

### 2. Issue separate client and worker credentials

Issue the two runtime roles independently:

```bash
curl -X POST \
  https://cloud.durable-workflow.com/api/v1/projects/PROJECT/environments/ENVIRONMENT/namespaces/orders/runtime-credentials \
  -H "Authorization: Bearer dwc_..." \
  -H "Content-Type: application/json" \
  -d '{"name":"orders-client","role":"client"}'

curl -X POST \
  https://cloud.durable-workflow.com/api/v1/projects/PROJECT/environments/ENVIRONMENT/namespaces/orders/runtime-credentials \
  -H "Authorization: Bearer dwc_..." \
  -H "Content-Type: application/json" \
  -d '{"name":"orders-worker","role":"worker"}'
```

Each token is displayed once in its create response. Store it in the secret
store used by only the corresponding role. A deliberately combined client and
worker process receives both values as two distinct secrets. Rotate and revoke
the roles independently, and never substitute a Cloud API key for either
runtime credential.

### 3. Complete a first Cloud workflow {#cloud-first-workflow}

The Sample App's shared external-runtime playground runs the same authored
workflow-and-activity journey with PHP, Python, or Rust. Open a [Sample App
Codespace](https://codespaces.new/durable-workflow/sample-app?quickstart=1&ref=main),
export the provisioned namespace values and two runtime credentials, and choose
an application task queue:

```bash
export DURABLE_WORKFLOW_RUNTIME_URL='<provisioned-runtime-url>'
export DURABLE_WORKFLOW_NAMESPACE='<provisioned-runtime-namespace>'
export DURABLE_WORKFLOW_CLIENT_TOKEN='<client-runtime-credential>'
export DURABLE_WORKFLOW_WORKER_TOKEN='<worker-runtime-credential>'
export DURABLE_WORKFLOW_TASK_QUEUE='<language-task-queue>'
```

Then choose a first-party SDK and run the same managed-runtime contract:

```bash
language=php # Choose php, python, or rust.
scripts/playground "$language" --runtime managed \
  --runtime-url "$DURABLE_WORKFLOW_RUNTIME_URL" \
  --namespace "$DURABLE_WORKFLOW_NAMESPACE" \
  --task-queue "$DURABLE_WORKFLOW_TASK_QUEUE"
```

The runner resolves the current stable artifact versions, gives each
credential only to its matching child process, waits up to 60 seconds for a
worker registration advertising the exact queue and generated workflow and
activity types, then starts one client request. It succeeds only after the SDK
returns the expected result, `dw` reports `completed`, and the required
workflow and activity history is present. The command does not run a local
Server or Waterline in managed mode.

For each language, rerun the complete managed command above, changing only
`language`. Continue with its guide for SDK-specific authoring:

- [PHP SDK](/docs/polyglot/php/): set `language=php`.
- [Python SDK](/docs/polyglot/python/): set `language=python`.
- [Rust managed-runtime quickstart](/docs/polyglot/rust-cloud-quickstart/):
  set `language=rust` and use its worker-ready, completed-result, and
  mismatch diagnostics.

A combined client-and-worker journey holds both runtime credentials as distinct
secrets. A Cloud administration key is not a runtime credential and must not be
exported under either runtime-token variable.

### 4. Continue with managed operation

The SDK client starts workflows and sends follow-up commands through the
namespace runtime URL. Customer-run workers register and long-poll through the
same URL using the worker role. Cloud authenticates and scopes each request,
executes the orchestration protocol in the managed runtime, and persists the
workflow state and history.

The customer application does not select a runtime deployment for an
operation. Workflow IDs, run IDs, task queues, compatibility markers, and Avro
payload encoding remain durable application contracts within the Cloud namespace.

## Customer-Run Worker Connectivity

Workers can run in your network, VM fleet, container platform, or application
environment. They need outbound HTTPS reachability to the namespace runtime URL
and must allow the worker protocol's long-lived poll requests.

- No inbound connection from Cloud to a worker is required.
- Proxies and egress gateways must not shorten long polls into a busy retry
  loop.
- Workers should retry transient connection failures and service-unavailable
  responses with bounded backoff.
- Moving a worker process does not move workflow state; Cloud retains the
  namespace's durable state and history.
- Credential rotation does not require changing the runtime URL, namespace, or
  task queue.

## Region Placement And Recovery Boundary

Cloud Dev is available with the recovery boundary below. The HA boundaries
describe the planned plans that remain unavailable pending qualification:

- **Cloud Dev:** one isolated host with persistent state and backups. Maintenance
  and recovery may interrupt service. There is no uptime SLA or automatic
  regional failover.
- **Planned single-region HA:** three replicated hosts in one region, with automatic
  primary failover. One host may fail without losing the remaining quorum.
  A whole-region outage is outside this plan's SLA coverage.
- **Planned multi-region HA:** three replicated hosts across three regions, with
  automatic primary failover. The SLA includes loss or isolation of one
  configured region, provided the remaining members can form a quorum.

For the planned HA profiles, the stable runtime URL is intended to follow the
elected primary; you do not change SDK
configuration during a supported failover. An isolated former primary is
prevented from continuing to serve authoritative work. If a safe primary
cannot be established, the runtime stops serving rather than accepting
conflicting writes. HA is not a promise of uninterrupted requests: clients and
workers still need bounded retries for transient failures.

The namespace page exposes its topology, region information, and service
status. Backups support recovery but are not a substitute for live replication.
Short failover tests demonstrate the exercised failure cases, not a universal
recovery-time guarantee or a month's achieved availability.

### SLA Measurement And Credits

The four planned SLA plans specify a 99.99% uptime SLA over a UTC calendar month,
measured at the customer runtime endpoint in one-minute windows. Missing
measurements count as unavailable, and planned maintenance is not excluded.
Customer-hosted worker availability is separate from managed runtime availability.

| Monthly availability | Automatic account credit |
| --- | ---: |
| At least 99.99% | None |
| At least 99.9%, below 99.99% | 10% |
| At least 99.0%, below 99.9% | 25% |
| Below 99.0% | 100% |

SLA plans receive automatic account service credits. Credits never exceed what
the affected runtime was charged that month, and there are no cash refunds for
outages. Cloud Dev has no SLA credits. See the [Cloud Terms](https://cloud.durable-workflow.com/terms)
and [Cloud pricing](https://cloud.durable-workflow.com/pricing) for the plan's
availability scope and billing terms.

## Private Connectivity And Support Boundary

The 2.0 self-serve Cloud contract assumes outbound access from clients and
workers to the public namespace runtime URL. Private-only ingress, bespoke VPN
or peering arrangements, and provider-specific private routing are support-led
connectivity designs, not hidden defaults. Customers are never given internal
runtime addresses or asked to route around the namespace URL.

## Cloud Or Self-Hosted Server

Choose Cloud when Durable Workflow should operate the orchestration runtime,
persistence, the selected plan's availability, recovery, and Managed Waterline while
your team operates the SDK clients and workers.

Choose [self-hosted Server](/docs/polyglot/server) when your team needs to
operate the Server image, database, cache, networking, authentication, backups,
and failover independently. A self-hosted Server cannot be registered with,
attached to, or used as the backing runtime for a Cloud namespace. Embedded
Laravel, self-hosted Server, and Cloud are separate deployment choices.

## Standard Workflow Benchmarks

[DW Standard Workflow v1](https://github.com/durable-workflow/server/tree/main/benchmarks/capacity)
defines a small, repeatable workload: one workflow start, one external activity,
and one workflow completion, with defined 1 KiB Avro inputs and results.
The customer worker runs outside the managed runtime allocation.

Cloud Dev was measured on September 21, 2026 using the same plan customers
provision. The other four baselines are historical: they include their replicated
HA topology, but predate the current runtime release. They are not extrapolated
from Dev and do not qualify the current SLA plans' capacity.

| Plan | Standard workflows/second | 30-day workflow actions | Measurement |
| --- | ---: | ---: | --- |
| Cloud Dev | 0.25 | 1,296,000 | September 21, 2026; details below |
| Cloud Standard | 0.25 | 1,296,000 | Historical; remeasurement pending |
| Cloud Multi-Region | 0.10 | 518,400 | Historical; remeasurement pending |
| Cloud Business | 0.50 | 2,592,000 | Historical; remeasurement pending |
| Cloud Business Multi-Region | 0.20 | 1,036,800 | Historical; remeasurement pending |

The 30-day estimate is `workflows/second x 2,592,000 seconds x 2 workflow actions`:
one start and one activity for this comparison. It assumes that rate runs
continuously for 30 days. It is not an included-action quota, a billing unit,
or an exact mapping to another engine's action definitions.

These are measured workload baselines, not maximum throughput or guaranteed
capacity for every application. Larger payloads, more activities, timers,
signals, queries, replay-heavy histories, cross-region communication, and
customer worker latency change the result. The table does not claim separate
timer, signal, replay, or saturation benchmarks. Plan for your actual workflow
mix rather than multiplying these numbers by an arbitrary workflow size.

### Cloud Dev Measurement {#cloud-dev-capacity}

Cloud Dev is an isolated, single-host managed runtime for development and
evaluation. This measurement used an ordinarily provisioned Dev runtime, without
changing its CPU, memory, storage, ingress or backup settings:

| Resource | Cloud Dev |
| --- | --- |
| Runtime compute | 1 shared vCPU, 1 GB RAM |
| Durable storage included | 5 GB |
| Runtime services | Server, queue worker, scheduler, MySQL, and Redis |
| Managed Waterline | Included through the authenticated Cloud application; not a process on the Dev host |
| Network path | Direct, space-specific HTTPS ingress |
| Customer workers | Run in the customer's environment |
| Availability | No SLA; maintenance interruptions are allowed |
| Current runtime price | See [Plans And Pricing](#plans-and-pricing) |
| More storage | Request a larger runtime plan; no storage add-ons |

The run used the public [DW Standard Workflow v1 PHP
definition](https://github.com/durable-workflow/server/blob/68168f34b637a19c054211f4db2df9c616799a86/benchmarks/capacity/v1/bindings/php/capacity_adapter.php):
one workflow start, one external echo activity, and one completion, carrying a
1,024-byte string in Avro at each workflow/activity input and result boundary.
Server 2.3.12 contained Workflow 2.0.14; the external worker and clients used
PHP SDK 2.0.11 on PHP 8.4.25. The runtime used PHP 8.3.33 and MySQL 8.0.46.

One PHP worker used a one-second task-poll timeout. Two independent client slots
offered one workflow every four seconds after a separate 30-second warmup.
The fixed measurement window was **18:50:30-18:55:30 UTC on September 21, 2026**.
Only completions observed within those five minutes count toward throughput;
draining afterward cannot increase the reported rate. Every result, completed
status and ordered workflow/activity history was checked.

| Measured result | Value |
| --- | ---: |
| Offered and completed rate | 0.25 standard workflows/second |
| Completed workflows | 75 of 75 |
| Errors / missed start slots | 0 / 0 |
| Start-to-result latency, p50 | 2.76 seconds |
| Start-to-result latency, p95 | 4.18 seconds |
| Start-to-result latency, p99 | 4.33 seconds |
| Unfinished workflows in the measured cohort | 0 |
| 30-day workflow actions | 1,296,000 |

Latency includes client networking and result polling every 100 ms. It is not
internal task scheduling latency, and should not be compared with an engine's
internal scheduling metric.

Host CPU averaged **63.5% busy**, peaking at **89%** across five-second samples.
There were **no container OOM kills or restarts**. Peak observed container memory
was 95.21 MiB for Server, 197.1 MiB for MySQL, 45.84 MiB for the queue worker,
43.75 MiB for the scheduler, 9.49 MiB for Redis and 34.59 MiB for the ingress
connector. These individual peaks are not simultaneous whole-host RSS; the OS
and other host processes also consume memory. The host had its normal 1 GiB
swap configuration, with up to about 90 MiB occupied during measurement.
An initial encrypted backup had been verified before the window; no backup
capture overlapped it.

This is a measured development baseline, not a universal conversion for every
workflow, a maximum-throughput test, a long-duration soak or an SLA. Larger payloads, additional activities, timers,
signals, queries, replay-heavy histories, and customer worker latency change
capacity. Cloud billing is based on provisioned runtime time with fixed included
storage, not workflow operations.

## Billing Usage API

Cloud Dev uses one isolated, single-host runtime cell for each provisioned Dev
space. Its billing terms are:

| Billing term | Cloud Dev |
| --- | ---: |
| Provisioned runtime | $0.075 per hour, metered by minute |
| Calendar-month maximum | $50 per runtime purchase |
| Active-month minimum | $2 per runtime purchase in an active UTC month |
| Managed capacity | 1 vCPU, 1 GB memory, 5 GB durable storage |
| Additional durable storage | Not sold separately; request a larger runtime plan |
| Basic encrypted backups | Included |
| Availability | Single host, no SLA |

The allocation covers Cloud-operated Server, MySQL, Redis, scheduling,
Waterline access, backups, upgrades, and infrastructure. Customer PHP, Python,
and Rust workers run outside that allocation. Workflow starts, activity
attempts, retries, timers, signals, queries, updates, and child workflows are
operational telemetry, not separate billing units. Prices exclude applicable
taxes. Included storage does not create a separate charge.

Cloud exposes organization-scoped billing usage for finance, operations, and
chargeback automation. The endpoint is authenticated by a Cloud API key and
does not accept a customer or organization id in the request; the caller's
organization is resolved from the `dwc_` bearer token so one customer cannot
query another customer's usage.

```http
GET /api/v1/billing/usage?period_start=2026-05-01&period_end=2026-05-31
Authorization: Bearer dwc_...
Accept: application/json
```

`period_start` and `period_end` are optional ISO-8601 dates. When omitted,
Cloud returns the current calendar month. Billing usage reads and exports stay
available even if billing restrictions pause namespace provisioning or workflow
operations, so finance teams can still recover account standing.

The response schema is
`durable_workflow.cloud.namespace_capacity_usage.v1`. It separates allocated
capacity time from semantic event counters. Storage-related fields remain in
the schema for historical billing; they do not authorize an add-on purchase.
The abbreviated Cloud Dev response below shows the current plan terms.

```json
{
  "schema": "durable_workflow.cloud.namespace_capacity_usage.v1",
  "access_control": {
    "scope": "organization_billing_usage",
    "read_allowed": true,
    "export_allowed": true
  },
  "current_period": {
    "starts_at": "2026-05-01T00:00:00+00:00",
    "ends_at": "2026-05-31T23:59:59+00:00"
  },
  "metering_policy": {
    "invoice_drivers": [
      "namespace_plan_capacity",
      "additional_durable_storage_gb_month"
    ],
    "semantic_events_are_invoice_units": false,
    "network": "measured_not_billable",
    "basic_backups": "included",
    "customer_worker_compute": "excluded"
  },
  "by_namespace": [
    {
      "namespace": "development",
      "project": "sample-app",
      "environment": "development",
      "plan": {
        "version": "cloud-dev.single-host-v2",
        "name": "Cloud Dev",
        "availability_class": "development_single_host",
        "sla_status": "none",
        "billing_terms": {
          "currency": "usd",
          "unit": "provisioned_runtime_hour",
          "hourly_rate_cents": "7.5",
          "monthly_cap_cents": 5000,
          "active_month_minimum_cents": 200,
          "scope": "runtime_purchase",
          "billing_period": "calendar_month_utc",
          "metering_resolution_seconds": 60,
          "additional_storage_unit": "gb_month",
          "additional_storage_rate_cents": 0,
          "additional_storage_available": false,
          "additional_storage_metering_unit": "gb_hour",
          "additional_storage_in_monthly_cap": false
        }
      },
      "allocation": {
        "managed_cpu_vcpu": 1,
        "managed_memory_gb": 1,
        "included_durable_storage_gb": 5
      },
      "operational_telemetry": {
        "billing_status": "not_billable",
        "counters": {
          "workflow_execution_count": 20,
          "activity_execution_count": 40,
          "timer_fire_count": 5,
          "signal_delivery_count": 3,
          "update_delivery_count": 0,
          "query_task_count": 2
        }
      }
    }
  ]
}
```

Cloud Dev's time meter starts when its isolated runtime is activated and
stops when that runtime is deprovisioned. The monthly cap and minimum apply per
runtime purchase in UTC calendar months. Storage does not grow or start billing
automatically. Cloud preserves an operating and recovery reserve on the runtime
disk; request a larger runtime plan before reaching the included allocation.

Idle runtimes still incur capacity charges. Deprovisioning stops runtime
capacity billing and removes active runtime data and credentials; it is not a
pause/resume operation. Planned SLA terms use the same provisioned-time model at their
own rates, minimums and caps. Use the plan's returned `billing_terms` when
interpreting usage; historical invoices keep their accepted terms.

Before deprovisioning, follow [moving between runtimes](/docs/polyglot/runtime-migration/#preserve-evidence): retain verified history and payload evidence, and confirm that all source-owned runs are terminal. Capacity billing continues while the source remains provisioned.

Export the same evidence as CSV or a JSON report when a downstream finance
system needs a file handoff:

```bash
curl -OJ "https://cloud.durable-workflow.com/api/v1/billing/usage/export?period_start=2026-05-01&period_end=2026-05-31" \
  -H "Authorization: Bearer dwc_..."

curl -OJ "https://cloud.durable-workflow.com/api/v1/billing/usage/report?period_start=2026-05-01&period_end=2026-05-31" \
  -H "Authorization: Bearer dwc_..."
```

For a JSON-backed dashboard panel, request the same API with the panel's time
range:

```text
GET https://cloud.durable-workflow.com/api/v1/billing/usage?period_start=${__from:date:YYYY-MM-DD}&period_end=${__to:date:YYYY-MM-DD}
Authorization: Bearer dwc_...
```

Then flatten namespace capacity rows with:

```jq
.by_namespace[]
| {
    project,
    environment,
    namespace,
    plan: .plan.name,
    cpu_vcpu: .allocation.managed_cpu_vcpu,
    memory_gb: .allocation.managed_memory_gb,
    included_storage_gb: .allocation.included_durable_storage_gb,
    capacity_status
  }
```

Use `invoice_units` for capacity and storage reconciliation. Use
`operational_telemetry` to understand workload shape and benchmark behavior;
do not convert those event counters into charges.

## Related References

- [Deployment Modes](/docs/polyglot/deployment-modes)
- [PHP SDK](/docs/polyglot/php)
- [Python SDK](/docs/polyglot/python)
- [Rust SDK](/docs/polyglot/rust)
- [Server](/docs/polyglot/server)
- [Self-Hosting Deployments](/docs/deployment)
- [Support](/docs/support)

<!-- Source: docs/polyglot/external-execution.md -->

# External Execution Surface

Durable Workflow v2 treats external execution as a contract-first product
surface, not as a shell convention or a single worker transport. The standalone
server publishes the machine-readable umbrella at `GET /api/cluster/info` under
`worker_protocol.external_execution_surface_contract`.

That contract is named `activity_grade_external_execution`. It is for durable,
bounded work that can run outside a full workflow runtime:

- operator maintenance activities
- platform automation
- integration handoffs
- bridge adapters that start, signal, update, or hand off work
- script, daemon, HTTP, queue-backed, serverless, or agent-driven handlers

The primary wedge is operator, platform, and integration automation. AI agents
and scripts are important consumers because they need stable handles, but they
do not redefine the runtime boundary.

## Boundary

External handlers may:

- execute one leased workflow or activity task
- heartbeat lease progress through the worker protocol
- return a structured success or failure envelope
- use a bridge adapter to start, signal, update, or hand off bounded work

External handlers must not:

- interpret workflow replay semantics
- own `ContinueAsNew` behavior
- apply signal, update, or query ordering rules outside the runtime contract
- mutate event history directly
- act as an unbounded workflow runtime

Those rules stay inside the server and real SDK runtimes. A carrier should only
move declared input and result envelopes across a transport boundary.

## Published Contract Seams

Read `GET /api/cluster/info` before wiring a carrier. The relevant v2 paths are:

| Path | Purpose |
| --- | --- |
| `worker_protocol.external_execution_surface_contract` | Product boundary, runtime boundary, valid carrier classes, and seam status. |
| `worker_protocol.external_task_input_contract` | Carrier-neutral input envelope for one leased workflow or activity task. |
| `worker_protocol.external_task_result_contract` | Carrier-neutral success, failure, and malformed-output envelope. |
| `worker_protocol.server_capabilities.external_execution_surface` | Compact capability pointer for worker-plane negotiation. |

The external execution surface also names planned seams for deterministic
auth/profile/TLS composition, config-first handler mappings, bounded bridge
adapters, payload external storage, and admission/rollout safety. Treat those
as contract seams, not private implementation details.

## Carrier Requirements

A valid carrier can be a poll-based CLI or daemon, an HTTP handler invocation,
a queue-backed worker, or a serverless invocation. The transport can differ,
but these requirements do not:

- emit the declared input schema
- accept the declared result schema
- preserve `task.id`, `task.attempt`, and `task.idempotency_key`
- map transport failures to a structured failure or malformed-output outcome
- resolve auth, TLS, profile, and environment inputs deterministically

Exit codes, stderr, process crashes, HTTP failures, queue visibility timeouts,
and serverless platform errors are transport facts. They only become workflow
facts after the carrier maps them to the declared result envelope or to
`malformed_output`.

The first concrete carrier under this contract is the
[invocable HTTP carrier](./invocable-carrier.md), published at
`worker_protocol.invocable_carrier_contract`. It is activity-task only,
requires HTTPS (with loopback HTTP allowed only for development), and resolves
auth through the external executor configuration. Its transport-level
`retry_policy` is distinct from the durable activity retry policy, which
remains the server/runtime authority once a result is reported.

## Bridge Adapters

Bridge adapters are bounded ingress or handoff surfaces. They can start,
signal, update, or hand off work, but they are not workflow runtimes and should
not hide replay or event-history behavior.

Every bridge adapter needs explicit machine outcomes for:

- unknown target
- auth failure
- malformed payload
- duplicate start
- unsupported routing
- accepted handoff
- rejected handoff

Use the same operator clarity as the rest of the control plane: stable status
codes, named `reason` values, and enough context for `dw`, Waterline, SDKs, and
agents to explain what happened without scraping prose.

The CLI exposes the webhook bridge surface directly:

```bash
dw bridge:webhook stripe \
  --action=start_workflow \
  --idempotency-key=stripe-event-1001 \
  --target='{"workflow_type":"orders.fulfillment","task_queue":"external-workflows","business_key":"order-1001"}' \
  --input='{"order_id":"order-1001"}' \
  --json
```

The JSON response is the same bridge-adapter outcome contract published by
`/api/cluster/info`, including `outcome`, `reason`,
`control_plane_outcome`, `idempotency_key`, and the redacted `target` summary.

## Agent-Operable Shape

The external execution surface is part of the v2 AI-assisted development
posture. Humans learn the workflow/activity/replay invariant; tools operate
through stable contracts:

1. discover the external execution surface from `/api/cluster/info`
2. choose a carrier that satisfies the published requirements
3. emit an external task input envelope
4. preserve task identity and idempotency
5. return a success, failure, or malformed-output envelope
6. report bridge and transport failures with named outcomes

This keeps AI-assisted scaffolding and operator automation boring: the tool can
cite the protocol manifest, the task envelope, the result envelope, and the
bridge outcome instead of inferring behavior from one demo script.

<!-- Source: docs/polyglot/server-api-reference.md -->

# Server API Reference

The standalone server exposes a versioned HTTP+JSON API. Use this page when
building SDKs, scripts, bridge adapters, or operator runbooks that call the
server directly. Use the [server guide](/docs/polyglot/server) for
deployment and configuration, and the [CLI command reference](/docs/polyglot/cli-reference)
when shelling out to `dw`.

## Headers And Versioning

All authenticated requests use bearer tokens unless the server is configured
for another auth driver:

```http
Authorization: Bearer <token>
X-Namespace: default
Content-Type: application/json
Accept: application/json
```

Control-plane routes require:

```http
X-Durable-Workflow-Control-Plane-Version: 2
```

Worker-plane routes require:

```http
X-Durable-Workflow-Protocol-Version: 1.0
```

The server publishes supported versions and machine-readable contracts from
`GET /api/cluster/info`. Clients should discover versions there before
starting long-lived automation. Missing or unsupported control-plane versions
fail closed with a named reason such as `missing_control_plane_version` or
`unsupported_control_plane_version`.

For validation, code generation, and drift checks, use the normative
[Platform Protocol Specs](/docs/platform-protocol-specs) catalog instead
of this prose reference. It links the control-plane OpenAPI document, worker
protocol OpenAPI and AsyncAPI documents, the `cluster_info` JSON Schema, and
the adjacent MCP, history, Waterline, and repair/actionability schemas.

## Discovery And Health

These routes are used by load balancers, SDK bootstraps, and compatibility
checks.

| Method | Path | Auth | Purpose |
| --- | --- | --- | --- |
| `GET` | `/api/health` | no | Liveness probe plus a machine-readable topology summary for the responding node. |
| `GET` | `/api/ready` | no | Readiness probe plus the same topology summary and rollout-safety bootstrap checks. |
| `GET` | `/api/cluster/info` | yes | Server identity, supported SDK ranges, role topology, coordination-health summary, control-plane contract, worker-protocol contract, payload codecs, and feature capabilities. |

Example:

<!-- docs-example id="server.cluster-info.curl" -->
```bash
curl -sS "$DURABLE_WORKFLOW_SERVER_URL/api/cluster/info" \
  -H "Authorization: Bearer $DURABLE_WORKFLOW_AUTH_TOKEN" \
  -H "X-Namespace: default" | jq '.control_plane.version, .worker_protocol.version'
```

`/api/cluster/info` intentionally does not require the control-plane version
header because it is the endpoint that advertises the supported versions. The
same response also includes `coordination_health`, the all-namespaces rollout
and readiness summary that mirrors the checks already feeding the server's
readiness posture.

### Public Topology Summary

`GET /api/health` and `GET /api/ready` both return a top-level `topology`
object from the landed health-summary contract. Use it when you need to
identify which node answered a probe before control-plane auth, namespace
resolution, or broader `/api/cluster/info` discovery succeeds.

The public summary always includes:

- `topology.schema`
- `topology.version`
- `topology.current_shape`
- `topology.current_process_class`
- `topology.current_roles`
- `topology.execution_mode`
- `topology.matching_role.queue_wake_enabled`
- `topology.matching_role.shape`
- `topology.matching_role.wake_owner`
- `topology.matching_role.task_dispatch_mode`
- `topology.matching_role.partition_primitives`
- `topology.matching_role.backpressure_model`
- `topology.matching_role.discovery_limits.poll_batch_cap`
- `topology.matching_role.discovery_limits.availability_ceiling_seconds`
- `topology.matching_role.discovery_limits.wake_signal_ttl_seconds`
- `topology.matching_role.discovery_limits.workflow_task_lease_seconds`
- `topology.matching_role.discovery_limits.activity_task_lease_seconds`

`topology.matching_role.discovery_limits` is the frozen numeric matching-role
contract: `poll_batch_cap` is the maximum batch of ready-task rows returned per
poll, `availability_ceiling_seconds` is the cross-backend tolerance applied to
`available_at` so freshly-available tasks survive sub-second timestamp drift,
`wake_signal_ttl_seconds` is the default `CacheLongPollWakeStore` signal TTL,
and `workflow_task_lease_seconds` / `activity_task_lease_seconds` are the
default workflow and activity task lease durations. Tightening any of these
values is a protocol-level change because workers and downstream tooling read
them as the authoritative matching-role contract; renaming a field is also a
protocol-level break.

`/api/ready` returns the same `topology` block even when the top-level
readiness `status` is `not_ready`, so probes can still distinguish
`server_http_node`, `scheduler_node`, `matching_node`, and `execution_node`
responses while bootstrap blockers are active.

Example:

<!-- docs-example id="server.health.topology.curl" -->
```bash
curl -sS "$DURABLE_WORKFLOW_SERVER_URL/api/health" | jq '{
  status,
  topology: {
    schema: .topology.schema,
    version: .topology.version,
    current_shape: .topology.current_shape,
    current_process_class: .topology.current_process_class,
    current_roles: .topology.current_roles,
    execution_mode: .topology.execution_mode,
    matching_role: .topology.matching_role
  }
}'
```

### Readiness Blockers

`GET /api/ready` returns a top-level `status` plus machine-readable `checks`.
Two checks define whether the server can safely evaluate rollout-safety health:

- `checks.migrations` is the bootstrap and migration gate. It publishes
  `repository_exists`, `pending_migrations`, `adoptable_migrations`,
  `blocking_migrations`, `missing_tables`, `operator_surface`, and
  `readiness_contract`.
- `adoptable_migrations` means existing workflow tables only need migration
  history adoption. The server stays ready and reports `status: "warning"` so
  operators can schedule the adoption before the next migrate pass.
- `blocking_migrations` means rollout-safety migration records are still
  required. The server fails closed with `status: "pending"` and a
  `remediation` string instead of serving as if the fleet were current.
- `operator_surface.available` and `operator_surface.required_tables` tell you
  whether the v2 operator surface has the durable tables it needs to explain
  rollout safety after boot.
- `readiness_contract.version` pins the install and adoption contract revision
  that scripts should expect when they parse these readiness fields.
- `checks.workflow_v2` mirrors the all-namespaces rollout-safety verdict. When
  readiness prerequisites are missing it reports `status: "blocked"` and adds
  `blocked_by`, `message`, and `remediation` instead of pretending the fleet is
  healthy.

Example:

<!-- docs-example id="server.ready.blockers.curl" -->
```bash
curl -sS "$DURABLE_WORKFLOW_SERVER_URL/api/ready" | jq '{
  status,
  migrations: {
    status: .checks.migrations.status,
    adoptable_migrations: .checks.migrations.adoptable_migrations,
    blocking_migrations: (.checks.migrations.blocking_migrations | map(.migration)),
    missing_tables: .checks.migrations.missing_tables,
    operator_surface: .checks.migrations.operator_surface,
    readiness_contract: .checks.migrations.readiness_contract
  },
  workflow_v2: {
    status: .checks.workflow_v2.status,
    blocked_by: .checks.workflow_v2.blocked_by,
    remediation: .checks.workflow_v2.remediation
  }
}'
```

### Cluster Topology Manifest

`/api/cluster/info` also returns the node's `topology` manifest under the
schema `durable-workflow.v2.role-topology`. That manifest is the supported way
to discover whether the node is currently acting as `standalone_server`,
`embedded`, or `split_control_execution`, which roles it owns, and what the
server expects from `matching_role`, `shape_assignments`,
`authority_boundaries`, `failure_domains`, `scaling_boundaries`, and
`migration_path`. The same
response also publishes live rollout-safety state for that node.

Read the manifest as follows:

- `topology.current_shape`, `topology.current_process_class`,
  `topology.current_roles`, and `topology.execution_mode` tell you which role
  shape the node is actually serving. These fields describe the responding
  node, not the full fleet.
- `topology.role_vocabulary` is the fixed list of legal v2 role names.
- `topology.matching_role.queue_wake_enabled`,
  `topology.matching_role.shape`, `topology.matching_role.wake_owner`,
  `topology.matching_role.task_dispatch_mode`,
  `topology.matching_role.partition_primitives`, and
  `topology.matching_role.backpressure_model` tell you whether broad ready-task
  discovery is happening in-worker or through a dedicated matching-role sweep,
  which process owns that sweep, which routing axes stay stable, and which
  durable admission boundary v2 enforces today.
- `topology.matching_role.discovery_limits` publishes the frozen numeric
  matching-role contract values: `poll_batch_cap`,
  `availability_ceiling_seconds`, `wake_signal_ttl_seconds`,
  `workflow_task_lease_seconds`, and `activity_task_lease_seconds`. Use these
  to verify the deployment matches the documented matching-role contract; the
  package emits the same identifiers in `dw server:info`, the operator-metrics
  snapshot, and the namespace-scoped health surface.
- `topology.role_catalog` and `topology.authority_surfaces` map those role
  names to the interfaces, durable-write surfaces, and read paths automation
  should expect on the responding node.
- `topology.shape_assignments` is the machine-readable process-class inventory
  for each supported shape. Compare the current role bundle against that table
  when you need to map the responding node onto a documented process class.
- `topology.supported_topologies` summarizes which deployment families the
  product supports and which node classes each family expects.
- `coordination_health` summarizes fleet-wide rollout and compatibility risk in
  one machine-readable block. Besides `status` and `http_status`, it can publish
  `blocked_by`, `message`, and `remediation` when rollout-safety evaluation is
  blocked by upstream readiness issues such as missing migrations or database
  reachability.
- `coordination_health.checks[]` always includes the frozen
  `activity_path` check next to `worker_compatibility`, `task_transport`,
  `routing_health`, `durable_resume_paths`, and the projection/scheduler checks.
  `activity_path` is the activity-side counterpart of `task_transport`: it
  surfaces activity executions whose schedule-to-start, start-to-close,
  schedule-to-close, or heartbeat deadline has passed without enforcement
  (`timeout_overdue`, `oldest_timeout_overdue_at`,
  `max_timeout_overdue_age_ms`) and the sustained activity retry backlog
  (`retrying`, `oldest_retrying_started_at`, `max_retrying_age_ms`). Renaming
  the check is a protocol-level change.
- `coordination_health.routing_drains` summarizes draining build-id cohorts
  across namespaces and queues. Use `queues_with_drains` and the per-queue
  `build_ids` entries to see where traffic is intentionally being held away from
  draining workers.
- `execution_mode` distinguishes `local_queue_worker` embedded execution from
  `remote_worker_protocol` worker-protocol execution.
- `split_control_execution` is a supported product topology, not a second
  server product or a different API.

Example:

<!-- docs-example id="server.cluster-info.topology.curl" -->
```bash
curl -sS "$DURABLE_WORKFLOW_SERVER_URL/api/cluster/info" \
  -H "Authorization: Bearer $DURABLE_WORKFLOW_AUTH_TOKEN" \
  -H "X-Namespace: default" \
  | jq '{
    current_shape: .topology.current_shape,
    current_roles: .topology.current_roles,
    execution_mode: .topology.execution_mode,
    matching_role: .topology.matching_role,
    coordination_health: {
      status: .coordination_health.status,
      http_status: .coordination_health.http_status,
      blocked_by: .coordination_health.blocked_by,
      queues_with_drains: .coordination_health.routing_drains.queues_with_drains
    }
  }'
```

For the conceptual contract behind those fields, including the role vocabulary
and migration path, see
[Server Role Topology](/docs/polyglot/server-role-topology).

### Wrong-Node Topology Rejections

Authenticated hosted routes fail closed when the responding node does not host
the HTTP control surface required for that endpoint. The gate runs after role
and protocol-version validation but before namespace resolution, so wrong-node
requests do not leak namespace existence.

When the gate blocks a request, the server returns `503` with
`reason: "topology_role_unavailable"` plus:

- `current_shape`: the responding node's advertised topology shape.
- `current_process_class`: the responding node's declared process class, such
  as `scheduler_node` or `execution_node`.
- `current_roles`: the roles that node actually hosts.
- `required_roles`: the hosted route roles the endpoint needs.
- `missing_roles`: the subset of `required_roles` missing from the responding
  node.

Control-plane routes return that payload in the control-plane envelope. Worker
protocol routes return the same fields in the worker-protocol envelope and keep
the normal worker-protocol version header.

Example wrong-node response from `GET /api/workflows` when the request lands on
a scheduler-only node:

```json
{
  "reason": "topology_role_unavailable",
  "message": "This node does not host the topology roles required for this endpoint.",
  "current_shape": "standalone_server",
  "current_process_class": "scheduler_node",
  "current_roles": ["scheduler"],
  "required_roles": ["api_ingress", "control_plane"],
  "missing_roles": ["api_ingress", "control_plane"]
}
```

`GET /api/health`, `GET /api/ready`, and authenticated `GET /api/cluster/info`
stay available for liveness and discovery even on nodes that do not host the
current HTTP control surface.

### Workflow Bootstrap Gate

Authenticated routes that mutate or serve workflow v2 traffic also fail closed
when `checks.workflow_v2.status` on the responding node is `blocked`. The gate
runs after role and protocol-version validation but before namespace
resolution, so a request sent during a blocked rollout never observes namespace
existence.

When the gate trips, the server returns `503` with
`reason: "workflow_v2_blocked"` plus:

- `blocked_by`: the ordered list of upstream readiness blockers (for example
  `migrations`) that are keeping workflow v2 from serving safely.
- `remediation`: the short operator-facing instruction for clearing the listed
  blockers, mirrored from the `/api/ready` `checks.workflow_v2.remediation`
  field.

The bootstrap-gated route families are:

- **Workflow start and mutation** — every `/api/workflows` route in the start,
  describe, command, and run-targeted command groups (for example
  `POST /api/workflows`, `POST /api/workflows/{workflowId}/signal/{signalName}`,
  `POST /api/workflows/{workflowId}/runs/{runId}/cancel`).
- **Schedule mutation** — `POST /api/schedules`,
  `PUT /api/schedules/{scheduleId}`, `DELETE /api/schedules/{scheduleId}`,
  `POST /api/schedules/{scheduleId}/pause`,
  `POST /api/schedules/{scheduleId}/resume`,
  `POST /api/schedules/{scheduleId}/trigger`, and
  `POST /api/schedules/{scheduleId}/backfill`.
- **Bridge adapters** — `POST /api/bridge-adapters/webhook/{adapter}`.
- **Worker protocol** — every `/api/worker` and `/api/worker/*` route, including
  registration, heartbeat, and workflow-task, query-task, and activity-task
  poll/complete/fail/heartbeat verbs.

Schedule **reads** are intentionally exempted so operators can inspect schedule
state during recovery: `GET /api/schedules`,
`GET /api/schedules/{scheduleId}`, and
`GET /api/schedules/{scheduleId}/history` continue to serve while the bootstrap
gate is blocking other routes.

Control-plane routes return the bootstrap-gate payload in the control-plane
envelope, including the `X-Durable-Workflow-Control-Plane-Version` header.
Worker-protocol routes return the same `reason`, `blocked_by`, and
`remediation` fields in the worker-protocol envelope and keep the
`X-Durable-Workflow-Protocol-Version` header so workers can branch on the
machine-readable reason instead of inferring queue state from a bare `503`.

Example bootstrap-gate response from `POST /api/workflows` while a rollout-safety
migration is missing:

```json
{
  "reason": "workflow_v2_blocked",
  "message": "This node is not ready to serve workflow v2 traffic until bootstrap blockers are cleared.",
  "blocked_by": ["migrations"],
  "remediation": "Restore database connectivity and migrate the workflow tables before relying on workflow v2 rollout-safety health."
}
```

The same payload is returned in the worker-protocol envelope for
`/api/worker/*` routes, so worker SDKs can keep branching on `reason` and
retrying after the upstream blocker clears.

### Namespace-Scoped System Health

`GET /api/system/health` is the authenticated rollout-safety and coordination
health surface for one namespace. It requires admin auth plus
`X-Durable-Workflow-Control-Plane-Version: 2`, resolves the namespace through
the normal control-plane request rules, and returns the exact namespace the
server evaluated plus the current `health` snapshot:

<!-- docs-example id="server.system-health.curl" -->
```bash
curl -sS "$DURABLE_WORKFLOW_SERVER_URL/api/system/health" \
  -H "Authorization: Bearer $DW_ADMIN_TOKEN" \
  -H "X-Namespace: orders-prod" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  | jq '{namespace, status: .health.status, healthy: .health.healthy}'
```

Treat the payload as:

- `namespace`: the namespace whose rollout/coordination state was evaluated.
- `health.status` and `health.healthy`: the top-level machine-readable health
  verdict for that namespace.
- `health.checks` and `health.categories`: per-surface readiness, compatibility,
  projection, and coordination facts.
- `health.operator_metrics`: the current namespace-scoped queue, worker, and
  repair metrics bundled into the same snapshot.
- `health.structural_limits`: the effective structural limits and any related
  diagnostics the server is enforcing for that namespace.

## Workflow Control Plane

Workflow routes are operator/control-plane routes. They require an operator or
admin role and `X-Durable-Workflow-Control-Plane-Version: 2`.

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/api/workflows` | List workflow runs in the namespace, with status, type, visibility query, `page_size` and `next_page_token`. |
| `POST` | `/api/workflows` | Start a workflow instance. |
| `GET` | `/api/workflows/{workflowId}` | Describe the current run for one workflow id. |
| `GET` | `/api/workflows/{workflowId}/debug` | Return bounded diagnostic facts for stuck-run investigation. |
| `GET` | `/api/workflows/{workflowId}/runs` | List runs for one workflow id. |
| `GET` | `/api/workflows/{workflowId}/runs/{runId}` | Describe a specific run. |
| `GET` | `/api/workflows/{workflowId}/runs/{runId}/debug` | Return bounded diagnostic facts for a selected run. |
| `GET` | `/api/workflows/{workflowId}/runs/{runId}/history` | Page through run history events. |
| `GET` | `/api/workflows/{workflowId}/runs/{runId}/history/export` | Export the archival replay bundle for a run. |

For supported fields, typed comparisons, quoting, precedence, errors and paging examples, see the [Visibility Query Language](./visibility-query-language.md). This listing grammar differs from a workflow's named query handlers and from schedule-list filters.

Start requests use the language-neutral control-plane shape:

<!-- docs-example id="server.workflow-start.curl" -->
```bash
curl -sS -X POST "$DURABLE_WORKFLOW_SERVER_URL/api/workflows" \
  -H "Authorization: Bearer $DURABLE_WORKFLOW_AUTH_TOKEN" \
  -H "X-Namespace: default" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{
    "workflow_type": "orders.fulfillment",
    "workflow_id": "order-1001",
    "task_queue": "orders",
    "input": ["order-1001"],
    "memo": {"source": "api-reference"},
    "search_attributes": {"CustomerId": "cust-42"},
    "duplicate_policy": "fail"
  }'
```

`duplicate_policy` accepts only `fail` and `use-existing`. Omitting it or sending
`null` selects `fail`. Both policies create a run when the workflow id is unused.

| Policy | An active run already exists | The existing run has closed |
| --- | --- | --- |
| `fail` (default) | Reject the duplicate start. | Reject the duplicate start. |
| `use-existing` | Return the existing pending, running or waiting run without starting another. | Reject the duplicate start. |

Neither policy reopens a closed run. Use a new workflow id for a new execution.
`reject` is not an accepted value and returns HTTP 422.

### Workflow Commands

The server also provides cooperative
`request-cancellation` routes. See [Cooperative cancellation](/docs/polyglot/cancellation/)
for the bounded cleanup contract and worker requirements.

Instance-targeted command routes operate on the current run for a workflow id:

| Method | Path | Purpose |
| --- | --- | --- |
| `POST` | `/api/workflows/{workflowId}/signal/{signalName}` | Send a signal. |
| `POST` | `/api/workflows/{workflowId}/query/{queryName}` | Execute a read-only query. |
| `POST` | `/api/workflows/{workflowId}/update/{updateName}` | Submit or execute an update. |
| `POST` | `/api/workflows/{workflowId}/cancel` | Close the run as cancelled immediately; does not resume workflow cleanup. |
| `POST` | `/api/workflows/{workflowId}/terminate` | Force termination. |
| `POST` | `/api/workflows/{workflowId}/repair` | Ask the server to repair retryable stuck state. |
| `POST` | `/api/workflows/{workflowId}/archive` | Archive a closed workflow run. |

Run-targeted command routes reject historical or wrong-run targets explicitly:

| Method | Path | Purpose |
| --- | --- | --- |
| `POST` | `/api/workflows/{workflowId}/runs/{runId}/signal/{signalName}` | Send a signal only if the selected run is current. |
| `POST` | `/api/workflows/{workflowId}/runs/{runId}/query/{queryName}` | Execute a query against the selected run. |
| `POST` | `/api/workflows/{workflowId}/runs/{runId}/update/{updateName}` | Submit or execute an update only if the selected run is current. |
| `POST` | `/api/workflows/{workflowId}/runs/{runId}/cancel` | Cancel only if the selected run is current. |
| `POST` | `/api/workflows/{workflowId}/runs/{runId}/terminate` | Terminate only if the selected run is current. |
| `POST` | `/api/workflows/{workflowId}/runs/{runId}/repair` | Repair only if the selected run is current. |
| `POST` | `/api/workflows/{workflowId}/runs/{runId}/redrive` | Redrive the selected failed run: start a successor that reuses its completed steps. A run that was already redriven returns its existing successor, and a `request_id` already used for another redrive is refused. A refusal returns `reason`. See [Redrive a failed run](../failures-and-recovery.md#redrive). |
| `POST` | `/api/workflows/{workflowId}/runs/{runId}/archive` | Archive only if the selected run is current and closed. |

Commands carry business values in Avro payload envelopes. First-party SDKs
encode them for you. A direct HTTP caller must use the official Avro encoder's
`{codec, blob}` envelope or the runtime external reference below. The surrounding
HTTP document is JSON; raw business JSON is not an alternative payload codec.

## Namespace And Storage

Namespace routes require operator or admin roles. Mutating namespace and
external-storage routes require admin role.

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/api/namespaces` | List namespaces. |
| `POST` | `/api/namespaces` | Create a namespace. |
| `GET` | `/api/namespaces/{namespace}` | Describe namespace retention, metadata, and storage policy. |
| `PUT` | `/api/namespaces/{namespace}` | Update namespace metadata or retention. |
| `PUT` | `/api/namespaces/{namespace}/external-storage` | Configure the namespace external payload storage policy. |
| `POST` | `/api/storage/test` | Round-trip a small and large payload through the configured external storage driver. |

External payload storage policies let large payload envelopes carry stable
references instead of raw bytes. Local policies resolve through the configured
filesystem path. Object-storage policies such as `s3`, `gcs`, and `azure` use
an explicitly configured filesystem disk and bucket/prefix settings on the
server.

### External Payload Reference Envelope

Server accepts **runtime** references, not the provider-URI references used by
embedded storage helpers. A request containing `external_storage` is rejected
with HTTP `422` and `external_payload_unsupported`. Do not copy a `file://`,
`s3://`, `gs://` or `azure://` provider reference into a Server request.

The runtime payload envelope has exactly `codec` and `external_payload`:

```json
{
  "codec": "avro",
  "external_payload": {
    "schema": "durable-workflow.v2.runtime-external-payload-reference.v1",
    "reference_id": "ep_01ARZ3NDEKTSV4RRFFQ69G5FAV",
    "codec": "avro",
    "size_bytes": 1048576,
    "sha256": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
  }
}
```

The reference id and digest above are illustrative. Use the reference returned
by an actual upload, unchanged. The nested reference has exactly five fields:

| Field | Required | Meaning |
| --- | --- | --- |
| `schema` | yes | `durable-workflow.v2.runtime-external-payload-reference.v1`. |
| `reference_id` | yes | Opaque Server id: `ep_` followed by 26 uppercase ULID characters. It is not a provider URI. |
| `codec` | yes | `avro`, matching the outer envelope. |
| `size_bytes` | yes | Non-negative integer size of the uploaded encoded bytes. |
| `sha256` | yes | SHA-256 of those same bytes, returned as lowercase hexadecimal. |

### Runtime External Payload Transport {#runtime-external-payload-transport}

Discover `namespace.external_payload_storage.transport` from
`GET /api/cluster/info`. Its schema is
`durable-workflow.v2.runtime-external-payload-transport.v1`, version `1`.
Use its upload/fetch routes and limits only when the namespace storage status is
`available`. Server owns provider credentials and returns no provider locations.

| Method | Path | Result |
| --- | --- | --- |
| `POST` | `/api/external-payloads/v1` | Upload bounded encoded bytes. `201` returns schema `durable-workflow.v2.runtime-external-payload-upload.v1`, `transport_version: 1` and `reference`. |
| `GET` | `/api/external-payloads/v1/{referenceId}` | Fetch and verify the referenced encoded bytes. `200` returns `application/octet-stream`. |

Both routes use a normal runtime credential with `worker`, `operator` or `admin`
role and `X-Namespace`. Both require `X-Durable-Workflow-Payload-Codec`,
`X-Durable-Workflow-Payload-Size` and `X-Durable-Workflow-Payload-SHA256`.
Upload also requires `Content-Type: application/octet-stream`.
Cross-namespace lookup returns not found without revealing another namespace's
object. Possession of an id does not grant access or workflow-completion authority.

The uploaded bytes are the **UTF-8 bytes of the official Avro envelope's base64
`blob` string**, without an added newline. They are not decoded binary Avro and
are not raw business JSON. Compute length and SHA-256 over precisely those bytes.
For example, after installing the Python SDK:

```bash
python - <<'PY'
from pathlib import Path
from durable_workflow.serializer import encode

Path("payload.avro.txt").write_bytes(
    encode([{"message": "large value", "optional": None}]).encode("utf-8")
)
PY

PAYLOAD_SIZE=$(wc -c < payload.avro.txt | tr -d ' ')
PAYLOAD_SHA256=$(sha256sum payload.avro.txt | cut -d ' ' -f 1)

curl --fail-with-body -sS -X POST "$DURABLE_WORKFLOW_SERVER_URL/api/external-payloads/v1" \
  -H "Authorization: Bearer $DURABLE_WORKFLOW_AUTH_TOKEN" \
  -H "X-Namespace: billing" \
  -H "Content-Type: application/octet-stream" \
  -H "X-Durable-Workflow-Payload-Codec: avro" \
  -H "X-Durable-Workflow-Payload-Size: $PAYLOAD_SIZE" \
  -H "X-Durable-Workflow-Payload-SHA256: $PAYLOAD_SHA256" \
  --data-binary @payload.avro.txt > upload.json

REFERENCE_ID=$(jq -r '.reference.reference_id' upload.json)
curl --fail-with-body -sS "$DURABLE_WORKFLOW_SERVER_URL/api/external-payloads/v1/$REFERENCE_ID" \
  -H "Authorization: Bearer $DURABLE_WORKFLOW_AUTH_TOKEN" \
  -H "X-Namespace: billing" \
  -H "X-Durable-Workflow-Payload-Codec: avro" \
  -H "X-Durable-Workflow-Payload-Size: $PAYLOAD_SIZE" \
  -H "X-Durable-Workflow-Payload-SHA256: $PAYLOAD_SHA256" \
  -o fetched.avro.txt
cmp payload.avro.txt fetched.avro.txt
```

An upload verifies byte integrity before committing a reference. It does not
execute a workflow or prove that the stored bytes form a valid business payload.
The first verified state-bearing request claims the reference; payload validation
and normal request, namespace, task-attempt and lease guards still apply.
Use the official SDK for worker completion uploads and its discovered completion
context rather than inventing a lease allowance in a direct client.

Read `limits.max_payload_bytes`, `request_timeout_seconds` and
`abandoned_upload_expiry_seconds` from the transport manifest. Namespace byte and
object quotas, ordinary request limits and completion upload limits also apply.
Threshold selection leaves values equal to the namespace threshold inline.
SDKs externalize larger values and may externalize additional values to fit an
aggregate request.

| Outcome | HTTP status | Action |
| --- | --- | --- |
| Missing or expired reference | `404` / `410` | Restore the required object or correct the reference. Do not decode a missing value as `null`. |
| Unauthorized | `401` / `403` | Correct credential, role or namespace authorization. |
| Unsupported envelope or integrity mismatch | `415` / `422` | Correct codec, fields, size, hash or encoded bytes. |
| Oversized payload | `413` | Reduce the encoded value or use an allowed larger policy. |
| Namespace quota or upload allowance exhausted | `429` | Inspect capacity and retry only within the original operation's budget. |
| Storage unavailable | `503` | Restore storage availability before retrying within that budget. |
| Stale or conflicting completion | `409` | Do not retry with invented task identity or a new lease. |

Server owns retention. There is no client delete route. A cache eviction must
not delete the backing object. Unclaimed uploads can expire; retained owners,
shared objects, storage failures and backup holds affect cleanup. A history
export containing a reference is not a self-contained backup of its bytes.

### External Payload Support By Mode {#external-payload-support-by-mode}

| Caller or worker | Storage path | Provider credentials |
| --- | --- | --- |
| Embedded Laravel workflow package | Application namespace policy and provider-URI references. | Embedded application. |
| PHP SDK | Automatic discovered runtime upload/fetch with opaque references. | Server. |
| Python SDK | Automatic discovered runtime upload/fetch with opaque references. | Server. |
| Rust SDK | Automatic discovered runtime upload/fetch with opaque references. | Server. |
| Direct HTTP | Explicit runtime upload/fetch plus the returned runtime reference in a supported payload slot. | Server. |

All SDK rows require an available compatible runtime transport. Provider driver
helpers serve application-owned embedded/direct storage integrations. Their
availability does not make provider envelopes a supported Server or Cloud ingress
format.

For the full request-authority contract, including namespace resolution,
role-scoped credentials, and worker registration fields, see
[Namespace, Auth, And Worker Registration](/docs/polyglot/namespace-auth-workers).

## Service Catalog Admin APIs

Service-catalog routes are authenticated admin control-plane routes. They use
the same namespace resolution, topology-role gating, and
`X-Durable-Workflow-Control-Plane-Version: 2` requirement as the rest of the
hosted control plane.

Use this route family to register namespace-scoped endpoint, service, and
operation metadata for the cross-namespace service catalog. Names are
case-insensitive on input and are normalized to lowercase in responses and
lookups.

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/api/service-endpoints` | List service endpoints for the current namespace. |
| `POST` | `/api/service-endpoints` | Create a service endpoint. |
| `GET` | `/api/service-endpoints/{endpointName}` | Describe one endpoint. |
| `PUT` | `/api/service-endpoints/{endpointName}` | Update endpoint description or metadata. |
| `DELETE` | `/api/service-endpoints/{endpointName}` | Delete an unused endpoint. |
| `GET` | `/api/service-endpoints/{endpointName}/services` | List services registered under one endpoint. |
| `POST` | `/api/service-endpoints/{endpointName}/services` | Create a service under one endpoint. |
| `GET` | `/api/service-endpoints/{endpointName}/services/{serviceName}` | Describe one service. |
| `PUT` | `/api/service-endpoints/{endpointName}/services/{serviceName}` | Update service description or metadata. |
| `DELETE` | `/api/service-endpoints/{endpointName}/services/{serviceName}` | Delete an unused service. |
| `GET` | `/api/service-endpoints/{endpointName}/services/{serviceName}/operations` | List operations registered under one service. |
| `POST` | `/api/service-endpoints/{endpointName}/services/{serviceName}/operations` | Create an operation binding. |
| `GET` | `/api/service-endpoints/{endpointName}/services/{serviceName}/operations/{operationName}` | Describe one operation. |
| `GET` | `/api/service-endpoints/{endpointName}/services/{serviceName}/operations/{operationName}/service-calls/{serviceCallId}` | Describe one durable service-call snapshot. |
| `PUT` | `/api/service-endpoints/{endpointName}/services/{serviceName}/operations/{operationName}` | Update an operation binding. |
| `DELETE` | `/api/service-endpoints/{endpointName}/services/{serviceName}/operations/{operationName}` | Delete an unused operation. |

Response collections use `service_endpoints`, `services`, or `operations`
arrays. Individual resources include stable lowercase names plus metadata and
timestamps:

- Endpoints return `id`, `namespace`, `endpoint_name`, `description`,
  `metadata`, `created_at`, and `updated_at`.
- Services add `endpoint_id` and `service_name`.
- Operations add `service_id`, `operation_name`, `operation_mode`,
  `handler_binding_kind`, `handler_target_reference`, `handler_binding`,
  `deadline_policy`, `idempotency_policy`, `cancellation_policy`,
  `retry_policy`, `boundary_policy`, and `metadata`.
- Service-call snapshots add `caller_namespace`, caller and linked workflow
  ids, `status`, `resolved_binding_kind`, `resolved_target_reference`,
  payload references, policy snapshots, and lifecycle timestamps such as
  `accepted_at`, `started_at`, `completed_at`, `failed_at`, and
  `cancelled_at`.

Operation create/update requests use the same JSON field names as the response.
`operation_mode` is `sync` or `async`. `handler_binding_kind` is one of
`start_workflow`, `signal_workflow`, `update_workflow`, `query_workflow`,
`activity_execution`, or `invocable_http`. New operations must provide either
`handler_target_reference` or a non-empty `handler_binding` payload.

The nested service-call detail also exposes `service_call_attempts`,
`retry_attempt_count`, the original `call_deadline_at`, typed failure metadata
and the retained Avro `result_envelope`. Accepted or started calls are
nonterminal. A completion-waiting workflow caller resolves with the linked
workflow's actual result or terminal failure/cancellation, preserving the
original call identity and deadline.

Use [CLI service-call inspection](./cli-reference.md#service-call-inspection)
or Waterline's service catalog to inspect that original call. Inspection does
not request another service execution.

Delete routes fail closed with HTTP `409` and a named reason when dependents
still exist, such as `endpoint_has_services`, `service_has_operations`, or
`operation_has_service_calls`.

## Bridge Adapters

Bridge adapters are bounded ingress endpoints. They do not execute workflow
code; they hand events to the control plane and return a named outcome.

| Method | Path | Purpose |
| --- | --- | --- |
| `POST` | `/api/bridge-adapters/webhook/{adapter}` | Start, signal, or update a workflow from a webhook-shaped event. |

Example:

```bash
curl -sS -X POST "$DURABLE_WORKFLOW_SERVER_URL/api/bridge-adapters/webhook/stripe" \
  -H "Authorization: Bearer $DURABLE_WORKFLOW_AUTH_TOKEN" \
  -H "X-Namespace: default" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "start_workflow",
    "idempotency_key": "evt_1001",
    "target": {
      "workflow_type": "orders.fulfillment",
      "task_queue": "orders",
      "business_key": "order-1001"
    },
    "input": {"order_id": "order-1001"}
  }'
```

Use response fields such as `outcome`, `reason`, `idempotency_key`, and
`control_plane_outcome` instead of inferring behavior from HTTP status alone.

## Worker Protocol

Worker routes require a worker role and
`X-Durable-Workflow-Protocol-Version: 1.0`. SDK workers use these endpoints
internally; custom language workers can implement the same protocol.

| Method | Path | Purpose |
| --- | --- | --- |
| `POST` | `/api/worker/register` | Register worker identity, task queues, supported workflow/activity types, capacity, runtime, and build metadata. |
| `POST` | `/api/worker/heartbeat` | Refresh worker fleet visibility and compatibility facts. |
| `POST` | `/api/worker/workflow-tasks/poll` | Long-poll for workflow tasks. |
| `POST` | `/api/worker/workflow-tasks/{taskId}/history` | Fetch paginated task history for a leased workflow task. |
| `POST` | `/api/worker/workflow-tasks/{taskId}/heartbeat` | Heartbeat a leased workflow task. |
| `POST` | `/api/worker/workflow-tasks/{taskId}/complete` | Complete a workflow task with commands. |
| `POST` | `/api/worker/workflow-tasks/{taskId}/fail` | Fail a workflow task. |
| `POST` | `/api/worker/query-tasks/poll` | Long-poll for server-routed query tasks. |
| `POST` | `/api/worker/query-tasks/{queryTaskId}/complete` | Complete a query task. |
| `POST` | `/api/worker/query-tasks/{queryTaskId}/fail` | Fail or reject a query task. |
| `POST` | `/api/worker/activity-tasks/poll` | Long-poll for activity tasks. |
| `POST` | `/api/worker/activity-tasks/{taskId}/heartbeat` | Heartbeat a leased activity task. |
| `POST` | `/api/worker/activity-tasks/{taskId}/complete` | Complete an activity task. |
| `POST` | `/api/worker/activity-tasks/{taskId}/fail` | Fail an activity task. |

Workers should treat lease ids, attempts, task ids, and heartbeat endpoints as
opaque server-issued values. A stale lease or wrong task id returns a named
worker-protocol error instead of silently completing work.

When `worker_protocol.server_capabilities.poll_status` is `true`, every
workflow-task, activity-task, and query-task poll response carries a
machine-readable `poll_status` field. Use it as the first branch point before
inspecting route-specific payload fields:

| `poll_status` | Typical HTTP status | Meaning |
| --- | --- | --- |
| `leased` | `200` | The server leased a task and `task` contains the task payload. |
| `empty` | `200` | No matching task was ready before the poll returned. |
| `throttled` | `200` | The queue is visible, but lease or dispatch admission limits withheld a new task for this poll. |
| `unavailable` | `503` or `200` | The server could not safely coordinate a poll path for the queue and returned a typed unavailable outcome instead of silently acting empty. |
| `draining` | `409` | The registered worker cohort is draining, so the server refuses to lease new work and returns `reason: "worker_draining"`. |

## Fleet And Task Queue Visibility

These routes expose operator diagnostics for worker fleets and queue
admission. They are control-plane routes.

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/api/workers` | List registered workers. |
| `GET` | `/api/workers/{workerId}` | Describe one worker. |
| `DELETE` | `/api/workers/{workerId}` | Deregister one worker. |
| `GET` | `/api/task-queues` | List task queues and admission status. |
| `GET` | `/api/task-queues/{taskQueue}` | Describe workflow/activity/query capacity for one queue. |
| `GET` | `/api/task-queues/{taskQueue}/build-ids` | List build ids observed for one queue. |
| `POST` | `/api/task-queues/{taskQueue}/build-ids/drain` | Mark a build-id cohort as draining so it stops claiming new tasks. |
| `POST` | `/api/task-queues/{taskQueue}/build-ids/resume` | Clear a previous drain so the cohort can claim new tasks again. |

Use task queue responses to distinguish no-worker conditions from saturated
worker slots, active lease caps, dispatch budgets, and query-task backpressure.
Drain and resume take a JSON body of `{"build_id": "..."}` (or
`{"build_id": null}` for the unversioned cohort), are idempotent, and persist
operator intent on the cohort so rollout state stays honest even after the
workers are removed. Once a worker heartbeat observes `drain_intent:
"draining"`, worker poll routes return HTTP `409` with `poll_status:
"draining"` and `reason: "worker_draining"` instead of leasing new tasks. See
[Worker Build-Id Rollout](/docs/polyglot/worker-build-id-rollout) for the
full unversioned-to-versioned cutover, canary, drain, and rollback lifecycle.

## Schedules And Search Attributes

Schedule routes are control-plane routes.

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/api/schedules` | List schedules. |
| `POST` | `/api/schedules` | Create a schedule. |
| `GET` | `/api/schedules/{scheduleId}` | Describe one schedule. |
| `PUT` | `/api/schedules/{scheduleId}` | Update schedule spec, action, note, memo, or search attributes. |
| `DELETE` | `/api/schedules/{scheduleId}` | Delete a schedule. |
| `POST` | `/api/schedules/{scheduleId}/pause` | Pause future fires. |
| `POST` | `/api/schedules/{scheduleId}/resume` | Resume a paused schedule. |
| `POST` | `/api/schedules/{scheduleId}/trigger` | Trigger a schedule immediately. |
| `POST` | `/api/schedules/{scheduleId}/backfill` | Backfill a time window. |
| `GET` | `/api/search-attributes` | List registered search attributes. |
| `POST` | `/api/search-attributes` | Register a search attribute. |
| `DELETE` | `/api/search-attributes/{name}` | Delete a search attribute. |

Search attribute names and types are part of the namespace search contract.
Avoid using high-cardinality attributes for operator dashboards or metric
labels.

### Schedule dispatch-summary authority {#schedule-dispatch-summary}

Published native Server 3.10.0 adds a complete `dispatch_summary` with
`version: 1`, `state`, `last_outcome` and `buffered_actions_count` to schedule
list items and describe responses. Older producers omit it; an empty list
does not establish support. Inspect the complete field rather than infer
support from a version. The [tagged schedule contract](https://github.com/durable-workflow/server/blob/3.10.0/docs/server-reference.md#schedules)
defines the exact shape.

| State | Last outcome | Current buffer count |
|---|---|---|
| `known` | Complete recorded dispatch | Integer from 0 through 9007199254740991 |
| `never` | `null`, established only by new creation | `0` |
| `unavailable` | `null`, including legacy or invalid stored authority | `null` |

Known outcomes are `started`, `buffered`, `skipped` or `failed`, with the
complete outcome ID, recording time, reason and applicable workflow/run
identities. The current buffer count is separate: a later drain can leave a
recorded buffered outcome with count zero. The latest workflow's execution
status is also separate; a Started dispatch may point to a workflow that later Failed.

Each summary is bounded to 4096 serialized UTF-8 bytes. A transition that
cannot record the complete contract stores unavailable authority, without
truncating its reason or changing the scheduling operation. Absent, null,
partial or unsupported data is not v1 authority. A valid unavailable summary
overrides legacy evidence. A later describe owns the whole summary, including
absence or null; never merge it field by field with an older list result.
Schema upgrades do not reconstruct authority from legacy arrays. The bounded
list-page read needs no per-row describe requests. This is producer behavior;
Waterline's published consumer and visible-refresh acceptance remain separately qualified.

### Schedule list filters and continuation {#schedule-list-filters}

`GET /api/schedules` reads the request namespace. The [normal published PHP Server qualification](https://github.com/durable-workflow/durable-workflow.github.io/issues/402#issuecomment-6370122227) verifies combined filters and continuation. Send the following query parameters, with normal URL encoding:

| Parameter | Accepted value |
| --- | --- |
| `status` | `active` or `paused`. |
| `workflow_type` | Nonempty workflow type, at most 255 characters. Whitespace alone is invalid. |
| `query` | Nonempty visibility expression, at most 2048 characters. |
| `page_size` | Integer from 1 to 200, default 50. |
| `next_page_token` | Nonempty opaque token, at most 8192 characters. |

Status, workflow type and query predicates combine with `AND`. A schedule visibility query accepts equality predicates joined by `AND`, such as `Region = "eu" AND TaskQueue = "orders"`. Built-in fields are `ScheduleId`, `Status`, `WorkflowType`, `TaskQueue` and `Note`. Other fields must be registered search attributes in the request namespace, with literals matching their types. Equality on a `keyword_list` attribute tests membership. Operators such as `OR`, `NOT`, `IN`, `LIKE` and range comparisons are unsupported and return `422`.

The response includes `namespace`, `schedules`, `schedule_count` for this page and `next_page_token`. Continue only when a token is returned. Pass it back unchanged with the same namespace, status, workflow type and query. The token is bound to the parsed filter selection. Pagination reads changing data rather than a snapshot. A deleted cursor, or one that no longer matches the filters, requires a fresh traversal without the token.

| HTTP / reason | Next action |
| --- | --- |
| `422` / `unsupported_schedule_list_filter` | Remove unsupported parameter names. |
| `422` / `validation_failed` | Correct the field validation errors. |
| `422` / `unsupported_schedule_visibility_field` | Use a built-in field or a registered attribute in this namespace. |
| `422` / `unsupported_schedule_visibility_predicate` | Use equality predicates joined by `AND`. |
| `400` / `malformed_schedule_page_token` | Use the returned token unchanged or restart without one. |
| `403` / `schedule_page_token_namespace_mismatch` | Use the original namespace or restart in the intended namespace. |
| `409` / `schedule_page_token_filter_mismatch` | Restart without a token after changing filters. |
| `409` / `stale_schedule_page_token` | Restart without a token against the current filtered set. |

[CLI schedule filtering and pagination examples](/docs/polyglot/cli-reference#schedule-list-filters) use the same contract. A list refusal includes the affected `field` and validation details. Inspect its reason before retrying.

## System Operations

System routes require admin role. They are explicit operator passes; prefer
status routes before pass routes in automation.

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/api/system/health` | Return the namespace-scoped rollout-safety and coordination health snapshot, nested under `health`. |
| `GET` | `/api/system/metrics` | Return bounded JSON metrics. |
| `GET` | `/api/system/operator-metrics` | Return the namespace-scoped operator metrics snapshot for runs, tasks, backlog, repair, workers, and structural limits. |
| `GET` | `/api/system/repair` | Inspect workflow repair backlog. |
| `POST` | `/api/system/repair/pass` | Run one workflow repair pass. |
| `GET` | `/api/system/activity-timeouts` | Inspect activity-timeout backlog. |
| `POST` | `/api/system/activity-timeouts/pass` | Run one activity-timeout enforcement pass. |
| `GET` | `/api/system/retention` | Inspect retention cleanup backlog. |
| `POST` | `/api/system/retention/pass` | Run one retention cleanup pass. |

For PHP Server retention, the status route reads the current namespace policy
and returns a bounded eligible-run list. Policy changes also affect existing
closed history. An unfiltered pass selects expired runs; explicit `run_ids`
directly target closed runs without the bounded age filter. Forever retention
and archived-run protections still apply. Review the [retention procedure and
backup steps](./server.md#retention) before changing policy or running cleanup.

`/api/system/health` is the quickest way to answer whether one namespace is
healthy enough to keep taking traffic. It returns `{namespace, health}`, where
`health` contains the categorized rollout-safety checks, the nested
`health.operator_metrics` snapshot, and the structural-limit summary used by
the health surface.

`/api/system/operator-metrics` is the namespace-scoped companion to
`/api/cluster/info` when you need raw backlog counts, compatibility-blocked
age, worker fleet detail, or other operator metrics behind the summarized
health surface.

`/api/system/metrics` is a JSON operator surface, not a Prometheus scrape
endpoint. Metric names and dimensions are bounded by the server's
bounded-growth policy.

## Error Contract

Error responses use HTTP status codes plus named machine-readable reasons.
Clients should branch on `reason` or nested control-plane/worker-protocol
reason fields, not on prose messages.

Common statuses:

| Status | Meaning |
| --- | --- |
| `400` | Missing or unsupported protocol/version header, malformed query, or unsupported route method. |
| `401` | Missing or invalid authentication. |
| `403` | Authenticated token lacks the required role. |
| `404` | Namespace, workflow, run, schedule, worker, or search attribute was not found. |
| `409` | Duplicate or conflict, such as an already-started workflow or invalid run target. |
| `422` | Validation failed; response includes field-level validation details. |
| `429` | Admission or task queue capacity is full. |
| `503` | The request reached a node that does not host the required topology roles. Hosted routes return `reason: "topology_role_unavailable"` plus `current_shape`, `current_process_class`, `current_roles`, `required_roles`, and `missing_roles`. The same status with `reason: "workflow_v2_blocked"` plus `blocked_by` and `remediation` covers workflow start/mutation, schedule mutation, bridge-adapter, and worker-protocol routes while workflow v2 bootstrap is blocked; schedule read routes (`GET /api/schedules`, `GET /api/schedules/{scheduleId}`, `GET /api/schedules/{scheduleId}/history`) stay available so operators can inspect schedule state during recovery. |
| `500` | Server failure. Treat as retryable only when the operation is idempotent or has an idempotency key. |

Validation responses include `reason: "validation_failed"` plus `errors` or
`validation_errors`. Workflow command responses also project validation and
operation details into the nested `control_plane` object.

## See Also

- [Server guide](/docs/polyglot/server)
- [Namespace, Auth, And Worker Registration](/docs/polyglot/namespace-auth-workers)
- [Worker Protocol](/docs/polyglot/worker-protocol)
- [Task Queue Admission](/docs/polyglot/task-queue-admission)
- [External Execution](/docs/polyglot/external-execution)
- [CLI Command Reference](/docs/polyglot/cli-reference)

<!-- Source: docs/polyglot/server-config-reference.md -->

# Server Config Reference

This page documents the operator-facing `DW_*` environment variable contract
for the Durable Workflow server image. Use it when building deployment
templates, reviewing production configuration, or migrating older
`WORKFLOW_*` / `ACTIVITY_*` names to the v2 server contract. The current Server
is the native 3.x Server; [Native Server 3.x](#native-server-3x) lists which
settings it honors, ignores or refuses. PHP Server 2.x is frozen at 2.11.8.

PHP Server 2.11.8 remains unpatched for the November Laravel advisories. Follow the [conditional exposure and ingress/signed-link actions](./server.md#retired-php-server-security) for an existing pinned deployment; changing these settings does not patch the image.

The PHP 2.x server also consumes ordinary Laravel runtime settings such as
`DB_*`, `REDIS_*`, `QUEUE_CONNECTION`, and `CACHE_STORE`. Those runtime
infrastructure variables are listed separately because they are deployment
plumbing, not Durable Workflow API controls.

## Native Server 3.x {#native-server-3x}

The variable tables on this page record the `DW_*` contract published by PHP
Server 2.11.0, and their purposes describe PHP 2.x. The native 3.x Server reads
the same names, with the same legacy aliases, through a startup audit that
gives every contract setting one disposition:

- **Honored:** native reads the setting and applies the PHP meaning. Settings
  not listed below are honored.
- **Ignored:** native accepts the setting, but it changes nothing clients or
  operators observe.
- **Refused unless default:** native refuses to start unless the setting is
  unset or keeps the PHP default. The startup error names the setting, the
  accepted value and the reason.

Native reads its environment once when the process starts and keeps no
configuration cache. To change a setting, restart or replace the Server
process. Like PHP's `env:audit`, the native audit warns about unknown `DW_*`
names and legacy names, and `DW_ENV_AUDIT_STRICT=1` turns those warnings into a
startup refusal.

Native refuses these settings unless they keep the listed value:

| Reason | Settings and accepted values |
| --- | --- |
| Native always pins runs to their recorded workflow definition | `DW_V2_PIN_TO_RECORDED_FINGERPRINT`: unset or `true` |
| Native cannot apply a changed Nexus operation limit | `DW_MAX_NEXUS_OPERATIONS_PER_CALLER`: unset or `200`<br />`DW_MAX_OPERATION_NAME_LENGTH`: unset or `256` |
| Native cannot apply a changed metrics series limit | `DW_METRICS_PROMETHEUS_ACTIVITY_SERIES_LIMIT`: unset or `100`<br />`DW_METRICS_PROMETHEUS_TASK_QUEUE_SERIES_LIMIT`: unset or `100`<br />`DW_METRICS_PROMETHEUS_WORKFLOW_SERIES_LIMIT`: unset or `100`<br />`DW_METRICS_WORKFLOW_TASK_FAILURE_TYPE_LIMIT`: unset or `20` |
| Native cannot apply a changed pending query task cap | `DW_QUERY_TASK_MAX_PENDING_PER_QUEUE`: unset or `1024` |
| Native cannot apply a changed structural limit | `DW_V2_LIMIT_COMMAND_BATCH_SIZE`: unset or `1000` |
| Native cannot apply changed task repair timing | `DW_V2_TASK_REPAIR_FAILURE_BACKOFF_MAX_SECONDS`: unset or `60`<br />`DW_V2_TASK_REPAIR_REDISPATCH_AFTER_SECONDS`: unset or `3`<br />`DW_V2_TASK_REPAIR_SCAN_LIMIT`: unset or `25` |
| Native does not publish package provenance | `DW_EXPOSE_PACKAGE_PROVENANCE`: unset or `false` |
| Native does not report a multi-node declaration | `DW_V2_MULTI_NODE`: unset or `false` |
| Native does not run Workflow 1 workflows | `DW_V1_ENABLED`: unset or `false` |
| Native does not validate worker fleet compatibility | `DW_V2_FLEET_VALIDATION_MODE`: unset or not `fail` |
| Native hands tasks only to polling workers | `DW_TASK_DISPATCH_MODE`: unset or `poll` |
| Native has no external executor configuration | `DW_EXTERNAL_EXECUTOR_CONFIG_OVERLAY`: unset or empty<br />`DW_EXTERNAL_EXECUTOR_CONFIG_PATH`: unset or empty |
| Native has no memo migration recovery mode | `DW_WORKFLOW_MEMO_MIGRATION_RECOVERY`: unset or empty |
| Native has no namespace durable-state limits | `DW_NAMESPACE_DURABLE_HARD_LIMITS`: unset or `{}`<br />`DW_NAMESPACE_DURABLE_LIMITS`: unset or `{}`<br />`DW_NAMESPACE_DURABLE_OVERRIDES`: unset or `{}` |
| Native has no namespace request admission limits | `DW_NAMESPACE_ADMISSION_OVERRIDES`: unset or `{}`<br />`DW_NAMESPACE_HARD_MAX_CONCURRENT_REQUESTS`: unset or empty<br />`DW_NAMESPACE_HARD_MAX_REQUESTS_PER_MINUTE`: unset or empty<br />`DW_NAMESPACE_MAX_CONCURRENT_REQUESTS`: unset or empty<br />`DW_NAMESPACE_MAX_REQUESTS_PER_MINUTE`: unset or empty |
| Native has no service boundary policy | `DW_SERVICE_BOUNDARY_CROSS_NAMESPACE_DEFAULT`: unset or `allow`<br />`DW_SERVICE_BOUNDARY_HARD_MAX_IN_FLIGHT`: unset or empty<br />`DW_SERVICE_BOUNDARY_HARD_RATE_LIMIT_PER_MINUTE`: unset or empty<br />`DW_SERVICE_BOUNDARY_MAX_IN_FLIGHT`: unset or empty<br />`DW_SERVICE_BOUNDARY_NAMESPACE_MAX_IN_FLIGHT`: unset or empty<br />`DW_SERVICE_BOUNDARY_NAMESPACE_OVERRIDES`: unset or `{}`<br />`DW_SERVICE_BOUNDARY_NAMESPACE_RATE_LIMIT_PER_MINUTE`: unset or empty<br />`DW_SERVICE_BOUNDARY_RATE_LIMIT_PER_MINUTE`: unset or empty |
| Native has no task admission limits | `DW_ACTIVITY_TASK_MAX_ACTIVE_LEASES_PER_NAMESPACE`: unset, empty or not a positive number<br />`DW_ACTIVITY_TASK_MAX_ACTIVE_LEASES_PER_QUEUE`: unset, empty or not a positive number<br />`DW_ACTIVITY_TASK_MAX_DISPATCHES_PER_MINUTE`: unset, empty or not a positive number<br />`DW_ACTIVITY_TASK_MAX_DISPATCHES_PER_MINUTE_PER_NAMESPACE`: unset, empty or not a positive number<br />`DW_TASK_QUEUE_ADMISSION_OVERRIDES`: unset or `{}`<br />`DW_WORKFLOW_TASK_MAX_ACTIVE_LEASES_PER_NAMESPACE`: unset, empty or not a positive number<br />`DW_WORKFLOW_TASK_MAX_ACTIVE_LEASES_PER_QUEUE`: unset, empty or not a positive number<br />`DW_WORKFLOW_TASK_MAX_DISPATCHES_PER_MINUTE`: unset, empty or not a positive number<br />`DW_WORKFLOW_TASK_MAX_DISPATCHES_PER_MINUTE_PER_NAMESPACE`: unset, empty or not a positive number |
| Native implements one worker protocol version | `DW_WORKER_PROTOCOL_VERSION`: unset or `1.20` |
| Native never trusts forwarded attribution headers | `DW_TRUST_FORWARDED_ATTRIBUTION_HEADERS`: unset or `false` |
| Native pins runs to registered worker build ids only; it has no server-wide compatibility markers | `DW_V2_COMPATIBILITY_HEARTBEAT_TTL`: unset or `30`<br />`DW_V2_COMPATIBILITY_NAMESPACE`: unset or empty<br />`DW_V2_CURRENT_COMPATIBILITY`: unset or empty<br />`DW_V2_NAMESPACE`: unset or empty<br />`DW_V2_SUPPORTED_COMPATIBILITIES`: unset or empty |
| Native reports no payload codec issues | `DW_SERIALIZER`: unset, empty or `avro` |
| Native runs only as a standalone HTTP server process | `DW_MODE`: unset or `service`<br />`DW_SERVER_PROCESS_CLASS`: unset or not one of `scheduler_node`, `worker_node`<br />`DW_SERVER_TOPOLOGY_SHAPE`: unset or not one of `embedded`, `split_control_execution` |
| Native serves `default` to requests that name no namespace | `DW_DEFAULT_NAMESPACE`: unset or `default` |
| Native stores external payloads only on its local disk | `DW_EXTERNAL_PAYLOAD_S3_BUCKET`: unset or empty |
| Native supports only its token authentication | `DW_AUTH_BACKWARD_COMPATIBLE`: unset or `true`<br />`DW_AUTH_PROVIDER`: unset or empty<br />`DW_PRINCIPAL_TOKENS`: unset or empty |

Native accepts but ignores: `DW_ADMIN_SIGNATURE_KEY`, `DW_AUTH_METHOD_HEADER`, `DW_AUTH_STATUS_HEADER`, `DW_BOOTSTRAP_DELAY_SECONDS`, `DW_BOOTSTRAP_RETRIES`, `DW_CALLER_LABEL_HEADER`, `DW_CALLER_TYPE_HEADER`, `DW_COMPRESSION_ENABLED`, `DW_DUE_TIMER_RECOVERY_SCAN_LIMIT`, `DW_EXPIRED_WORKFLOW_TASK_RECOVERY_TTL_SECONDS`, `DW_EXTERNAL_PAYLOAD_S3_ACCESS_KEY_ID`, `DW_EXTERNAL_PAYLOAD_S3_ENDPOINT`, `DW_EXTERNAL_PAYLOAD_S3_REGION`, `DW_EXTERNAL_PAYLOAD_S3_SECRET_ACCESS_KEY`, `DW_EXTERNAL_PAYLOAD_S3_SESSION_TOKEN`, `DW_EXTERNAL_PAYLOAD_S3_USE_PATH_STYLE_ENDPOINT`, `DW_MAX_HISTORY_EVENTS`, `DW_MAX_TASKS_PER_POLL`, `DW_OBSERVABILITY_BAGGAGE_ALLOWLIST`, `DW_OBSERVABILITY_CONTEXT_ENABLED`, `DW_OPERATOR_SIGNATURE_KEY`, `DW_PACKAGE_PROVENANCE_PATH`, `DW_POLLING_CACHE_PATH`, `DW_QUERY_TASK_POLL_MAX_CONCURRENT`, `DW_QUERY_TASK_POLL_MAX_CONCURRENT_PER_NAMESPACE`, `DW_SIGNATURE_KEY`, `DW_SQLITE_CLAIM_LOCK_TTL_SECONDS`, `DW_SQLITE_CLAIM_LOCK_WAIT_SECONDS`, `DW_V2_CACHE_VALIDATION_MODE`, `DW_V2_GUARDRAILS_BOOT`, `DW_V2_LIMIT_SEARCH_ATTRIBUTE_SIZE_BYTES`, `DW_V2_LIMIT_WARNING_THRESHOLD_PERCENT`, `DW_V2_MATCHING_ROLE_QUEUE_WAKE`, `DW_V2_TASK_DISPATCH_MODE`, `DW_V2_TASK_REPAIR_LOOP_THROTTLE_SECONDS`, `DW_V2_TENANCY_ENVIRONMENT`, `DW_V2_TENANCY_ORGANIZATION`, `DW_V2_TENANCY_PROJECT`, `DW_V2_UPDATE_WAIT_POLL_INTERVAL_MS`, `DW_V2_VALIDATE_CACHE_BACKEND`, `DW_WAKE_SIGNAL_TTL_SECONDS`, `DW_WORKER_LONG_POLL_MAX_CONCURRENT`, `DW_WORKER_LONG_POLL_MAX_CONCURRENT_PER_NAMESPACE`, `DW_WORKER_LONG_POLL_RESERVED_HTTP_WORKERS`, `DW_WORKER_POLL_INTERVAL_MS`, `DW_WORKER_POLL_SIGNAL_CHECK_INTERVAL_MS`, `DW_WORKER_SIGNATURE_KEY`.

Native 3.9.1 supports opt-in storage admission from an operator-published local
observation. Set `DW_STORAGE_ADMISSION_FILE` to its absolute path and
`DW_STORAGE_ADMISSION_SOURCE` to the expected source identity;
`DW_STORAGE_ADMISSION_MAX_AGE_SECONDS` defaults to 15 (allowed range 1–300).
With both path and source unset it is disabled. Missing, invalid or stale
observations refuse new work with `503 storage_admission_unavailable`;
`draining` or `fenced` observations refuse it with `503 storage_pressure`.
Readiness also returns 503 while database health can remain 200. The
[published 3.9.1 file-consumer controls](https://github.com/durable-workflow/server/issues/951#issuecomment-6853859351)
verify these cases and recovery after atomic file replacement. They do not
qualify physical quota measurement or a production Cloud mapping. This is
admission control, not a write fence or byte reservation.

Native 3.9.2 additionally requires the already-opened observation descriptor to
belong to root or the effective Server UID. A foreign owner fails closed as
`storage_admission_unavailable`; the owner/type/mode/content checks use the same
descriptor. Operators must still provide trusted ancestors and the intended
read-only container mapping: the runtime does not enforce ancestor ownership.
The [published 3.9.2 correction](https://github.com/durable-workflow/server/releases/tag/3.9.2#storage-admission-observation-ownership)
does not establish physical collector or Cloud adoption acceptance and adds no
draining completion-upload allowance.

Native has no server-side task-queue lease or dispatch caps, budget groups,
namespace request admission, namespace durable-state quotas or held-poll caps,
and no native equivalent of them yet. Each native Server process opens at most
four database connections. Query tasks have a fixed bound of 16 pending or
leased tasks per namespace, beyond which new queries are refused with HTTP
`429` and `query_task_queue_full`, as on PHP; it is not configurable. To bound task
concurrency on native, size each SDK worker's slots and the number of workers.
[Task queue admission](/docs/polyglot/task-queue-admission) describes the PHP
2.x controls.

Native authentication differs from PHP 2.x in these ways:

- `DW_AUTH_TOKEN` is required. Native refuses to start without it, and it
  accepts the shared token for every role alongside any role-scoped tokens.
- `DW_AUTH_DRIVER` accepts only `token`. Native refuses `signature`, and a
  release image also refuses `none`. The signature-key settings are ignored.

Native also reads settings outside the PHP contract:

- `DW_BIND_ADDRESS`: the listen address. The binary defaults to
  `127.0.0.1:8080`; the published image sets `0.0.0.0:8080`.
- `DW_RUST_MAINTENANCE_LOOP`: `1` or `true` adds the repair pass (task repair
  and activity timeouts) to the in-process timer and schedule loop. The
  published image sets `1`, because it has no separate scheduler.
- `DW_RUST_EXPERIMENTAL`: required as `1` only by development builds without a
  release version. Release images do not need it.
- `DW_UPGRADE_BACKUP_FINGERPRINT` and `DW_RUST_UPGRADE_QUEUE_*`: inputs to the
  one-way [native takeover](https://github.com/durable-workflow/server/blob/a051f57fcc7e3605b301581ee9105461b3ef5815/docs/rust-server-upgrade-takeover.md)
  of a PHP database, qualified on the published 3.0.4 image within the
  runbook's stated scope.
- `DW_V2_LIMIT_HISTORY_TRANSACTION_ROWS` (legacy
  `WORKFLOW_V2_LIMIT_HISTORY_TRANSACTION_ROWS`): honored ahead of its addition
  to the PHP contract.

## How Changes Take Effect

Native 3.x reads its environment only at process start; restart or replace the
Server process to apply a change. The rest of this section describes PHP 2.x.

The published PHP Server image runs `php artisan config:cache` in its
entrypoint. Environment values are resolved at that startup and written to
Laravel's configuration cache. Editing an environment file, changing an
orchestrator's environment declaration, or replacing a mounted configuration
file does not hot-reload a running deployment.

Restart or replace every affected process with the new configuration: HTTP
API replicas, local Laravel queue workers, and scheduler processes. Restart
long-lived SDK workers separately when changing their own client or worker
configuration. A successful API restart does not update a still-running queue
worker or scheduler. Keep the same intended configuration across the roles
that share a database and cache.

During a rolling change, each node enforces its own loaded limits. Shared
admission counters do not make differently configured nodes agree on a limit,
so a request can be accepted on one API replica and refused on another. For a
limit change that must take effect consistently:

1. Pause new caller submissions and drain affected API replicas from ingress.
   Let in-flight requests finish. Stop affected queue and scheduler roles
   gracefully according to their shutdown budgets.
2. Apply the same configuration to every affected role and restart or replace
   them. Let each container's entrypoint rebuild its configuration cache.
3. Check startup logs for configuration errors, verify readiness and exercise
   the changed limit before restoring ingress and caller submissions. See the
   [deployment and readiness guide](/docs/deployment#readiness-contract).

Database-backed operator settings have a different lifecycle. Successful
mutations below are available to subsequent operations without restarting
Server:

| Setting | Live mutation | Scope and limits |
| --- | --- | --- |
| Namespace description and retention | [`PUT /api/namespaces/{namespace}`](/docs/polyglot/server-api-reference#namespace-and-storage), or [`dw namespace:update`](/docs/polyglot/cli-reference#namespace-and-search-attribute-commands) | Retention policy governs later retention passes. Changing it does not restore already-pruned data. See [Server retention](/docs/polyglot/server#retention). |
| Namespace external payload storage policy | [`PUT /api/namespaces/{namespace}/external-storage`](/docs/features/external-payload-storage#configuring-a-namespace) | Changes the stored namespace policy. Provider credentials, configured filesystem disks and other process configuration still require the restart procedure above. Preserve access to objects referenced by existing runs. |
| Namespace search attribute definitions | [`POST` / `DELETE /api/search-attributes`](/docs/polyglot/server-api-reference#schedules-and-search-attributes) | Updates the registration used by subsequent requests. It does not rewrite workflow history or replace environment-configured size and count limits. |

The `DW_*` defaults and caps in this reference remain process configuration
unless a documented control-plane operation explicitly changes stored state.
Server does not poll configuration files for changes.

## How The Contract Is Enforced

The server repository keeps the canonical contract in `config/dw-contract.php`.
Native 3.x runs the equivalent audit at startup, as described in
[Native Server 3.x](#native-server-3x). On PHP 2.x, at container boot,
`php artisan env:audit` warns when it sees unknown `DW_*`
variables or legacy names. Set `DW_ENV_AUDIT_STRICT=1` when you want the
entrypoint to fail instead of booting with warnings.

```bash
DW_ENV_AUDIT_STRICT=1
DW_AUTH_DRIVER=token
DW_ADMIN_TOKEN="${DW_ADMIN_TOKEN}"
DW_OPERATOR_TOKEN="${DW_OPERATOR_TOKEN}"
DW_WORKER_TOKEN="${DW_WORKER_TOKEN}"
```

Legacy names are fallback-only. Prefer the `DW_*` name in every new
deployment, and treat a legacy warning as migration debt even when the server
still honors the value.

## Server Identity And Mode

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_MODE` | `service` | Server mode: `service` makes external workers poll; `embedded` dispatches locally through the Laravel queue. | `WORKFLOW_SERVER_MODE` |
| `DW_SERVER_ID` | `gethostname()` | Unique server instance identifier used in lease ownership and worker registration. | `WORKFLOW_SERVER_ID` |
| `DW_SERVER_TOPOLOGY_SHAPE` | `standalone_server` | Advertised topology shape for cluster discovery, such as `embedded`, `standalone_server`, or `split_control_execution`. | `WORKFLOW_SERVER_TOPOLOGY_SHAPE` |
| `DW_SERVER_PROCESS_CLASS` | `server_http_node` | Advertised process class for this node within the selected topology shape, such as `server_http_node`, `worker_node`, or `scheduler_node`. | `WORKFLOW_SERVER_PROCESS_CLASS` |
| `DW_SERVER_KEY` | generated at container boot | Optional server-internal runtime key. Docker images generate one automatically when unset. | - |
| `DW_DEFAULT_NAMESPACE` | `default` | Namespace used when a request omits the namespace header. | `WORKFLOW_SERVER_DEFAULT_NAMESPACE` |
| `DW_TASK_DISPATCH_MODE` | unset | Overrides `workflows.v2.task_dispatch_mode`; in service mode the server defaults to `poll` unless you set a different value. | `WORKFLOW_V2_TASK_DISPATCH_MODE` |
| `DW_WORKFLOW_MEMO_MIGRATION_RECOVERY` | unset | One-run MySQL recovery proof for an unrecorded memo rewrite: `raw-json` or `envelope-prefix:<last-converted-id>`. | `WORKFLOW_SERVER_WORKFLOW_MEMO_MIGRATION_RECOVERY` |
| `DW_EXTERNAL_EXECUTOR_CONFIG_PATH` | unset | Path to a `durable-workflow.external-executor.config` JSON file for external executor handler mappings. | `WORKFLOW_SERVER_EXTERNAL_EXECUTOR_CONFIG_PATH` |
| `DW_EXTERNAL_EXECUTOR_CONFIG_OVERLAY` | unset | Overlay name from the external executor config file to apply before server validation and discovery. | `WORKFLOW_SERVER_EXTERNAL_EXECUTOR_CONFIG_OVERLAY` |

## Authentication

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_AUTH_PROVIDER` | unset | PHP 2.x: Laravel-resolvable class implementing `App\Contracts\AuthProvider`. Native 3.x refuses to start when it is set. | `WORKFLOW_SERVER_AUTH_PROVIDER` |
| `DW_AUTH_DRIVER` | `token` | PHP 2.x auth driver: `none`, `token`, or `signature`. Native 3.x accepts only `token`. | `WORKFLOW_SERVER_AUTH_DRIVER` |
| `DW_AUTH_TOKEN` | unset | PHP 2.x: single shared bearer token when no role-scoped token is configured. Native 3.x: required, and accepted for every role. | `WORKFLOW_SERVER_AUTH_TOKEN` |
| `DW_SIGNATURE_KEY` | unset | PHP 2.x: shared HMAC signature key when no role-scoped signature key is configured. Ignored by native 3.x, which has no signature auth. | `WORKFLOW_SERVER_SIGNATURE_KEY` |
| `DW_WORKER_TOKEN` | unset | Bearer token for worker registration, polling, heartbeats, and completions. | `WORKFLOW_SERVER_WORKER_TOKEN` |
| `DW_OPERATOR_TOKEN` | unset | Bearer token for read and operator control-plane actions. | `WORKFLOW_SERVER_OPERATOR_TOKEN` |
| `DW_ADMIN_TOKEN` | unset | Bearer token for namespace, retention, and other administrative mutations. | `WORKFLOW_SERVER_ADMIN_TOKEN` |
| `DW_PRINCIPAL_TOKENS` | unset | PHP 2.x: JSON token map for named bearer-token principals. Each entry supplies `token`, `subject`, `roles`, optional `tenant`, `label`, and non-secret `claims`. Native 3.x refuses to start when it is set. | `WORKFLOW_SERVER_PRINCIPAL_TOKENS` |
| `DW_WORKER_SIGNATURE_KEY` | unset | HMAC key for worker-role requests when using signature auth. | `WORKFLOW_SERVER_WORKER_SIGNATURE_KEY` |
| `DW_OPERATOR_SIGNATURE_KEY` | unset | HMAC key for operator-role requests when using signature auth. | `WORKFLOW_SERVER_OPERATOR_SIGNATURE_KEY` |
| `DW_ADMIN_SIGNATURE_KEY` | unset | HMAC key for admin-role requests when using signature auth. | `WORKFLOW_SERVER_ADMIN_SIGNATURE_KEY` |
| `DW_AUTH_BACKWARD_COMPATIBLE` | `true` | PHP 2.x: honor shared `DW_AUTH_TOKEN` / `DW_SIGNATURE_KEY` as a fallback when a role-scoped credential is missing. Native 3.x always accepts the shared token and refuses to start when this is `false`. | `WORKFLOW_SERVER_AUTH_BACKWARD_COMPATIBLE` |

Use role-scoped credentials for production. On PHP 2.x, `DW_AUTH_DRIVER=none`
is only for local smoke work because every endpoint becomes reachable without a
bearer token or signature; native release images refuse it.

Use `DW_PRINCIPAL_TOKENS` when audit trails need stable actor subjects rather
than role labels. The server derives the recorded principal from the matched
token entry; clients cannot override it with request payloads or headers.

## Command Attribution

These settings let a trusted gateway preserve caller metadata in durable
command history. Leave `DW_TRUST_FORWARDED_ATTRIBUTION_HEADERS=false` unless
the server is behind a gateway that strips untrusted client-supplied headers.

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_TRUST_FORWARDED_ATTRIBUTION_HEADERS` | `false` | Record forwarded caller/auth headers into workflow command history. | `WORKFLOW_SERVER_TRUST_FORWARDED_ATTRIBUTION_HEADERS` |
| `DW_CALLER_TYPE_HEADER` | `X-Workflow-Caller-Type` | Header carrying forwarded caller type. | `WORKFLOW_SERVER_CALLER_TYPE_HEADER` |
| `DW_CALLER_LABEL_HEADER` | `X-Workflow-Caller-Label` | Header carrying forwarded caller label. | `WORKFLOW_SERVER_CALLER_LABEL_HEADER` |
| `DW_AUTH_STATUS_HEADER` | `X-Workflow-Auth-Status` | Header carrying forwarded auth status. | `WORKFLOW_SERVER_AUTH_STATUS_HEADER` |
| `DW_AUTH_METHOD_HEADER` | `X-Workflow-Auth-Method` | Header carrying forwarded auth method. | `WORKFLOW_SERVER_AUTH_METHOD_HEADER` |

## Worker Polling And Admission

These values control long-poll timing, wake coordination, server-side
admission caps, and bounded task dispatch budgets.

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_WORKER_POLL_TIMEOUT` | `30` | Default wait when a poll request omits `timeout_seconds`. SDK-supplied waits override it; this is not their maximum. See [end-to-end timeout budgets](../deployment.md#worker-long-poll-timeout-budgets). | `WORKFLOW_SERVER_WORKER_POLL_TIMEOUT` |
| `DW_WORKER_POLL_INTERVAL_MS` | `1000` | Milliseconds between internal scans while a poll is held open. | `WORKFLOW_SERVER_WORKER_POLL_INTERVAL_MS` |
| `DW_WORKER_POLL_SIGNAL_CHECK_INTERVAL_MS` | `100` | Milliseconds between wake-signal checks while a poll is held open. | `WORKFLOW_SERVER_WORKER_POLL_SIGNAL_CHECK_INTERVAL_MS` |
| `DW_POLLING_CACHE_PATH` | `storage/framework/cache/server-polling/<APP_ENV>` | Directory for worker-poll coordination state when using file-backed polling cache. | `WORKFLOW_SERVER_POLLING_CACHE_PATH` |
| `DW_WAKE_SIGNAL_TTL_SECONDS` | `max(DW_WORKER_POLL_TIMEOUT + 5, 60)` | TTL for per-queue wake signals that short-circuit a pending poll. | `WORKFLOW_SERVER_WAKE_SIGNAL_TTL_SECONDS` |
| `DW_WORKER_LONG_POLL_MAX_CONCURRENT` | unset; derived for `PHP_CLI_SERVER_WORKERS` | Optional cap for concurrent held workflow/activity worker long-poll waits on this server node. Query-task polls use a separate wait budget. Native 3.x ignores it; see [Native Server 3.x](#native-server-3x). | `WORKFLOW_SERVER_WORKER_LONG_POLL_MAX_CONCURRENT` |
| `DW_WORKER_LONG_POLL_RESERVED_HTTP_WORKERS` | `2` | PHP CLI server workers reserved for health and control-plane requests when deriving the workflow/activity long-poll wait cap. Native 3.x ignores it; see [Native Server 3.x](#native-server-3x). | `WORKFLOW_SERVER_WORKER_LONG_POLL_RESERVED_HTTP_WORKERS` |
| `DW_MAX_TASKS_PER_POLL` | `1` | Maximum tasks returned per worker poll. | `WORKFLOW_SERVER_MAX_TASKS_PER_POLL` |
| `DW_SQLITE_CLAIM_LOCK_TTL_SECONDS` | `10` | Seconds the SQLite quickstart backend holds the cache-backed worker poll claim gate before the lock expires. | `WORKFLOW_SERVER_SQLITE_CLAIM_LOCK_TTL_SECONDS` |
| `DW_SQLITE_CLAIM_LOCK_WAIT_SECONDS` | `5` | Seconds SQLite worker poll claims wait for the cache-backed claim gate before returning backend lock pressure. | `WORKFLOW_SERVER_SQLITE_CLAIM_LOCK_WAIT_SECONDS` |
| `DW_WORKFLOW_TASK_MAX_ACTIVE_LEASES_PER_QUEUE` | unset | Active workflow-task lease cap per namespace/task queue. Native 3.x refuses to start when it is a positive number; see [Native Server 3.x](#native-server-3x). | `WORKFLOW_SERVER_WORKFLOW_TASK_MAX_ACTIVE_LEASES_PER_QUEUE` |
| `DW_WORKFLOW_TASK_MAX_ACTIVE_LEASES_PER_NAMESPACE` | unset | Active workflow-task lease cap across all queues in a namespace. Native 3.x refuses to start when it is a positive number; see [Native Server 3.x](#native-server-3x). | `WORKFLOW_SERVER_WORKFLOW_TASK_MAX_ACTIVE_LEASES_PER_NAMESPACE` |
| `DW_WORKFLOW_TASK_MAX_DISPATCHES_PER_MINUTE` | unset | Per-minute workflow-task dispatch cap per namespace/task queue. Native 3.x refuses to start when it is a positive number; see [Native Server 3.x](#native-server-3x). | `WORKFLOW_SERVER_WORKFLOW_TASK_MAX_DISPATCHES_PER_MINUTE` |
| `DW_WORKFLOW_TASK_MAX_DISPATCHES_PER_MINUTE_PER_NAMESPACE` | unset | Per-minute workflow-task dispatch cap across a namespace. Native 3.x refuses to start when it is a positive number; see [Native Server 3.x](#native-server-3x). | `WORKFLOW_SERVER_WORKFLOW_TASK_MAX_DISPATCHES_PER_MINUTE_PER_NAMESPACE` |
| `DW_ACTIVITY_TASK_MAX_ACTIVE_LEASES_PER_QUEUE` | unset | Active activity-task lease cap per namespace/task queue. Native 3.x refuses to start when it is a positive number; see [Native Server 3.x](#native-server-3x). | `WORKFLOW_SERVER_ACTIVITY_TASK_MAX_ACTIVE_LEASES_PER_QUEUE` |
| `DW_ACTIVITY_TASK_MAX_ACTIVE_LEASES_PER_NAMESPACE` | unset | Active activity-task lease cap across all queues in a namespace. Native 3.x refuses to start when it is a positive number; see [Native Server 3.x](#native-server-3x). | `WORKFLOW_SERVER_ACTIVITY_TASK_MAX_ACTIVE_LEASES_PER_NAMESPACE` |
| `DW_ACTIVITY_TASK_MAX_DISPATCHES_PER_MINUTE` | unset | Per-minute activity-task dispatch cap per namespace/task queue. Native 3.x refuses to start when it is a positive number; see [Native Server 3.x](#native-server-3x). | `WORKFLOW_SERVER_ACTIVITY_TASK_MAX_DISPATCHES_PER_MINUTE` |
| `DW_ACTIVITY_TASK_MAX_DISPATCHES_PER_MINUTE_PER_NAMESPACE` | unset | Per-minute activity-task dispatch cap across a namespace. Native 3.x refuses to start when it is a positive number; see [Native Server 3.x](#native-server-3x). | `WORKFLOW_SERVER_ACTIVITY_TASK_MAX_DISPATCHES_PER_MINUTE_PER_NAMESPACE` |
| `DW_TASK_QUEUE_ADMISSION_OVERRIDES` | `{}` | JSON overrides keyed by `namespace:task_queue`, `namespace:*`, `task_queue`, or `*` for active leases, dispatch rate, namespace caps, and downstream budget groups. Native 3.x refuses to start unless it is unset or `{}`; see [Native Server 3.x](#native-server-3x). | `WORKFLOW_SERVER_TASK_QUEUE_ADMISSION_OVERRIDES` |
| `DW_DUE_TIMER_RECOVERY_SCAN_LIMIT` | `5` | Maximum due service-mode timer tasks recovered per worker poll pass. | `WORKFLOW_SERVER_DUE_TIMER_RECOVERY_SCAN_LIMIT` |
| `DW_EXPIRED_WORKFLOW_TASK_RECOVERY_SCAN_LIMIT` | `5` | Maximum expired workflow tasks recovered per pass. | `WORKFLOW_SERVER_EXPIRED_WORKFLOW_TASK_RECOVERY_SCAN_LIMIT` |
| `DW_EXPIRED_WORKFLOW_TASK_RECOVERY_TTL_SECONDS` | `5` | Minimum seconds between expired-task recovery passes per queue. | `WORKFLOW_SERVER_EXPIRED_WORKFLOW_TASK_RECOVERY_TTL_SECONDS` |

Sticky execution has no standalone server-image environment variables. See
[Sticky Execution](/docs/features/sticky-execution) for the sticky-cache
lifecycle, worker protocol fields, replay modes, and diagnostics.

Example admission override:

```json
{
  "production:billing": {
    "workflow_tasks": {
      "max_active_leases": 50,
      "max_dispatches_per_minute": 600
    },
    "activity_tasks": {
      "max_active_leases": 200
    }
  },
  "production:*": {
    "workflow_tasks": {
      "max_active_leases_per_namespace": 500
    }
  }
}
```

## Worker Protocol And Query Transport

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_WORKER_PROTOCOL_VERSION` | `WorkerProtocolVersion::VERSION` | Worker-protocol version advertised on worker-plane responses. | `WORKFLOW_SERVER_WORKER_PROTOCOL_VERSION` |
| `DW_HISTORY_PAGE_SIZE_DEFAULT` | `WorkerProtocolVersion::DEFAULT_HISTORY_PAGE_SIZE` | Default page size for worker history reads. | `WORKFLOW_SERVER_HISTORY_PAGE_SIZE_DEFAULT` |
| `DW_HISTORY_PAGE_SIZE_MAX` | `WorkerProtocolVersion::MAX_HISTORY_PAGE_SIZE` | Maximum page size honored on worker history reads. | `WORKFLOW_SERVER_HISTORY_PAGE_SIZE_MAX` |
| `DW_UPDATE_VALIDATION_TIMEOUT` | `10` | Seconds the control plane waits for a synchronous pre-accept update validator result. | `WORKFLOW_SERVER_UPDATE_VALIDATION_TIMEOUT` |
| `DW_UPDATE_VALIDATION_LEASE_TIMEOUT` | `5` | Seconds an update-validation task lease remains owned before a replacement validator-capable worker may retry it. | `WORKFLOW_SERVER_UPDATE_VALIDATION_LEASE_TIMEOUT` |
| `DW_QUERY_TASK_TIMEOUT` | `DW_WORKER_POLL_TIMEOUT` | Seconds the control plane waits for a query task response from the worker. | `WORKFLOW_SERVER_QUERY_TASK_TIMEOUT` |
| `DW_QUERY_TASK_LEASE_TIMEOUT` | `DW_WORKFLOW_TASK_TIMEOUT` | Lease timeout for ephemeral query tasks handed to workers. | `WORKFLOW_SERVER_QUERY_TASK_LEASE_TIMEOUT` |
| `DW_QUERY_TASK_TTL_SECONDS` | `180` | How long the server retains query-task result rows before reaping them. | `WORKFLOW_SERVER_QUERY_TASK_TTL_SECONDS` |
| `DW_QUERY_TASK_MAX_PENDING_PER_QUEUE` | `1024` | Maximum pending cache-backed query tasks per namespace/task queue before new queries are rejected. Native 3.x refuses any value other than `1024` and applies its own fixed query-task bound instead; see [Native Server 3.x](#native-server-3x). | `WORKFLOW_SERVER_QUERY_TASK_MAX_PENDING_PER_QUEUE` |
| `DW_QUERY_TASK_POLL_TIMEOUT` | `5` | Maximum seconds each idle query-task worker poll waits before rechecking workflow-task pressure and control-plane availability. | `WORKFLOW_SERVER_QUERY_TASK_POLL_TIMEOUT` |
| `DW_QUERY_TASK_POLL_MAX_CONCURRENT` | unset; derived for `PHP_CLI_SERVER_WORKERS` | Optional cap for concurrent held idle query-task worker long-poll waits on this server node. Pending query tasks can still be claimed immediately before an idle poll waits. Native 3.x ignores it; see [Native Server 3.x](#native-server-3x). | `WORKFLOW_SERVER_QUERY_TASK_POLL_MAX_CONCURRENT` |
| `DW_WORKFLOW_TASK_TIMEOUT` | `60` | Default workflow-task lease timeout in seconds. | `WORKFLOW_TASK_TIMEOUT` |
| `DW_ACTIVITY_TASK_TIMEOUT` | `300` | Default activity-task lease timeout in seconds. | `ACTIVITY_TASK_TIMEOUT` |
| `DW_WORKER_STALE_AFTER_SECONDS` | `max(DW_WORKER_HEARTBEAT_INTERVAL_SECONDS * 3, 30)` | Seconds after a worker heartbeat before the worker registration is stale. | `WORKFLOW_SERVER_WORKER_STALE_AFTER_SECONDS` |
| `DW_WORKER_HEARTBEAT_INTERVAL_SECONDS` | `10` | Cadence in seconds advertised to SDKs in worker register and heartbeat acknowledgements. | `WORKFLOW_SERVER_WORKER_HEARTBEAT_INTERVAL_SECONDS` |

## Limits, Retention, And Metrics

These settings are request-boundary and bounded-growth controls. If an
application regularly approaches these values, prefer smaller payloads,
external payload storage, or continue-as-new rather than raising the limits by
default.

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_METRICS_WORKFLOW_TASK_FAILURE_TYPE_LIMIT` | `20` | Maximum `workflow_type` series reported by `dw_workflow_task_consecutive_failures`; excess types are summarized. | `WORKFLOW_SERVER_METRICS_WORKFLOW_TASK_FAILURE_TYPE_LIMIT` |
| `DW_METRICS_PROMETHEUS_WORKFLOW_SERIES_LIMIT` | `100` | Maximum workflow series reported by `/api/system/prometheus-metrics` before excess series are summarized. | `WORKFLOW_SERVER_METRICS_PROMETHEUS_WORKFLOW_SERIES_LIMIT` |
| `DW_METRICS_PROMETHEUS_ACTIVITY_SERIES_LIMIT` | `100` | Maximum activity series reported by `/api/system/prometheus-metrics` before excess series are summarized. | `WORKFLOW_SERVER_METRICS_PROMETHEUS_ACTIVITY_SERIES_LIMIT` |
| `DW_METRICS_PROMETHEUS_TASK_QUEUE_SERIES_LIMIT` | `100` | Maximum task-queue runtime series reported by `/api/system/prometheus-metrics` before excess series are summarized. | `WORKFLOW_SERVER_METRICS_PROMETHEUS_TASK_QUEUE_SERIES_LIMIT` |
| `DW_MAX_HISTORY_EVENTS` | `50000` | Has no effect. No Server release reads it and history per run is not capped. Bound long histories with continue-as-new. | `WORKFLOW_MAX_HISTORY_EVENTS` |
| `DW_HISTORY_RETENTION_DAYS` | `30` | Default number of days closed-run history is retained when a namespace does not override it. | `WORKFLOW_HISTORY_RETENTION_DAYS` |
| `DW_MAX_PAYLOAD_BYTES` | `2097152` | Maximum serialized bytes for one payload. | `WORKFLOW_MAX_PAYLOAD_BYTES` |
| `DW_EXTERNAL_PAYLOAD_MAX_BYTES` | `67108864` | Maximum encoded size in bytes (64 MiB) accepted by the authenticated external-payload upload and fetch transport. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_MAX_BYTES` |
| `DW_EXTERNAL_PAYLOAD_REQUEST_TIMEOUT` | `30` | Client-facing upload and fetch request-timeout budget advertised by the server, in seconds. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_REQUEST_TIMEOUT` |
| `DW_EXTERNAL_PAYLOAD_UPLOAD_EXPIRY` | `3600` | Seconds an uploaded external payload may remain unclaimed before its opaque reference expires. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_UPLOAD_EXPIRY` |
| `DW_MAX_MEMO_BYTES` | `262144` | Maximum serialized bytes for a workflow memo. | `WORKFLOW_MAX_MEMO_BYTES` |
| `DW_MAX_SEARCH_ATTRIBUTES` | `100` | Maximum number of search attributes on one workflow. | `WORKFLOW_MAX_SEARCH_ATTRIBUTES` |
| `DW_MAX_SEARCH_ATTRIBUTE_KEY_LENGTH` | `128` | Maximum byte length for one search-attribute key. | `WORKFLOW_MAX_SEARCH_ATTRIBUTE_KEY_LENGTH` |
| `DW_MAX_SEARCH_ATTRIBUTE_VALUE_BYTES` | `2048` | Maximum byte size for one search-attribute string value. | `WORKFLOW_MAX_SEARCH_ATTRIBUTE_VALUE_BYTES` |
| `DW_MAX_OPERATION_NAME_LENGTH` | `256` | Maximum byte length for a signal, update, or query name. | `WORKFLOW_MAX_OPERATION_NAME_LENGTH` |
| `DW_MAX_PENDING_ACTIVITIES` | `2000` | Maximum pending activities per workflow run before rejecting a command batch. | `WORKFLOW_MAX_PENDING_ACTIVITIES` |
| `DW_MAX_PENDING_CHILDREN` | `2000` | Maximum pending child workflows per run before rejecting a command batch. | `WORKFLOW_MAX_PENDING_CHILDREN` |
| `DW_MAX_NEXUS_OPERATIONS_PER_CALLER` | `200` | Maximum Nexus operations returned per caller from the operations history surface before clients must paginate. | `WORKFLOW_MAX_NEXUS_OPERATIONS_PER_CALLER` |
| `DW_COMPRESSION_ENABLED` | `true` | Enable JSON response compression above the minimum size threshold. | `WORKFLOW_SERVER_COMPRESSION_ENABLED` |

## Server request-boundary limits {#server-request-boundary-limits}

When using the standalone server, a separate set of caps is enforced at the HTTP request boundary before a workflow task, signal, update, or query ever reaches the control plane. These limits fail fast with a `422 validation_failed` (or `413 payload_too_large` for whole-body checks) response so clients learn the rejection reason without the server writing any row to the database.

| Limit | Default | Config key | What it caps |
|---|---|---|---|
| Body size | 2 MiB | `server.limits.max_payload_bytes` | Total HTTP request body bytes, except a workflow-task completion on native Server 3.6.0 or later (next row) |
| Workflow-task completion size | 32 MiB (16 × body size) | derived from `server.limits.max_payload_bytes` | Native Server 3.6.0 or later: a request that completes a workflow task with commands. Each command's arguments still meet the per-payload limit |
| Memo size | 256 KiB | `server.limits.max_memo_bytes` | Serialized memo on `POST /workflows` and `POST /schedules` |
| Search-attribute count | 100 | `server.limits.max_search_attributes` | Registered custom search attributes per namespace |
| Search-attribute key length | 128 bytes | `server.limits.max_search_attribute_key_length` | Length of a single SA key on start |
| Search-attribute value size | 2 KiB | `server.limits.max_search_attribute_value_bytes` | Each string value (and each element of an array value) on start |
| Signal / update / query name | 256 bytes | `server.limits.max_operation_name_length` | URL path segment for signal/update/query names |
| `workflow_id` length | 128 chars | controller validator | Workflow ID on `POST /workflows` |
| `workflow_type` / `task_queue` / `business_key` | 255 chars | controller validator | String fields on `POST /workflows` |
| `request_id` | 255 chars | controller validator | Deduplication token on signals/updates/cancel/terminate |
| `reason` | 1,000 chars | controller validator | Reason text on cancel/terminate/archive |

Use the documented `DW_*` settings for configurable PHP Server 2.x limits.
Controller-validator bounds are fixed. Native Server 3.x supports the settings
listed in its [implementation notes](#native-server-3x). On native 3.8.0,
`DW_MAX_PAYLOAD_BYTES=0` sets a zero-byte ceiling for ordinary requests and
workflow-task completions, rejecting nonempty bodies. Its operation-name
ceiling is fixed at 256 bytes.

Native 3.9.2 applies the existing 255 Unicode code-point limit to signal and
update command keys throughout admission. A 255-character multibyte
`request_id` is accepted; 256 characters are refused without admission. This
corrects the [3.9.1 UTF-8 byte-count refusal](https://github.com/durable-workflow/server/issues/934#issuecomment-6853943056).
Use the body `request_id` as the retry key; the HTTP `X-Request-Id` header is
not a command key.

Fresh keyed signal and accepted update receipts in 3.9.2 echo the literal
`request_id` with `retained:false` and `intent_verified:true`. Retained receipts
echo that key with `retained:true` and preserve the original command, run and
update identities. Modern intent proof reports `intent_verified:true`;
compatible pre-intent legacy readback reports `intent_verified:false`.
Unkeyed and bridge receipts keep their existing behavior. The
[published correction](https://github.com/durable-workflow/server/releases/tag/3.9.2#command-keys-and-receipts)
does not establish every installed client cohort under
[shared #216](https://github.com/durable-workflow/.github/issues/216); earlier
3.9.1 failures must be rerun on the exact corrected tuple.

The currently-configured values are published under `limits` on the `GET /api/cluster/info` response, so clients can discover them at runtime:

```json
{
  "limits": {
    "max_payload_bytes": 2097152,
    "max_memo_bytes": 262144,
    "max_search_attributes": 100,
    "max_search_attribute_key_length": 128,
    "max_search_attribute_value_bytes": 2048,
    "max_operation_name_length": 256,
    "max_pending_activities": 2000,
    "max_pending_children": 2000
  }
}
```

Validation errors are returned in the standard control-plane error envelope with `reason: "validation_failed"` and a `validation_errors` map keyed by the offending field (`signal_name`, `update_name`, `query_name`, `search_attributes`, `memo`, and so on). Payload-size rejections use `reason: "payload_too_large"` and the `413` status code.

### Native worker-command names and IDs {#native-worker-command-name-limits}

The table lists storage-character bounds. Existing runtime byte guards also
apply, as explained below. The workflow-ID row is the separate start API
limit; child-call fields have stricter bounds than other commands.

| Field | Maximum characters | Scope |
|---|---|---|
| `workflow_id` | 128 | `POST /api/workflows` |
| `activity_type` | 191 | Activity commands |
| `signal_name` | 191 | `open_signal_wait` |
| `condition_key` / `condition_definition_fingerprint` | 191 | `open_condition_wait` |
| `connection` | 255 | Worker commands |
| `queue` | 255 | Worker commands except `start_child_workflow` |
| `workflow_type` | 255 | `continue_as_new` |
| `workflow_type` / `queue` | 191 | `start_child_workflow` |

Native 3.8.0 and 3.9.1 also retain UTF-8 byte checks.
`activity_type`, `open_signal_wait.signal_name`, an explicit
`schedule_activity.queue`, and `continue_as_new.workflow_type` / `queue`
use a 255-byte parser ceiling (`422 invalid_request`). Child `workflow_type`
and `queue` have an additional 191-byte guard that can return
`422 unsupported_child_options`. For example, an activity type containing
128 copies of `é` is 256 bytes and is refused despite having only 128
characters. The new character checks do not remove these byte guards.

Native Server 3.9.1 includes the [published correction](https://github.com/durable-workflow/server/releases/tag/3.9.1#other-changes):
field-specific `422 validation_failed` refusals, such as
`commands.0.activity_type`, before writes. Native 3.8.0 did not consistently
validate these character bounds; SQLite could silently store over-long names
while other backends could return database errors. Review existing activity
types and signal waits longer than 191 characters before upgrading: new
commands using those names are now refused. The remaining identifier sweep is
tracked in [Server #607](https://github.com/durable-workflow/server/issues/607).
The 256-byte HTTP operation-name ceiling above is a separate boundary.

### Workflow-task completion size {#workflow-task-completion-size}

On native Server 3.6.0 and later, a request that completes a workflow task
with commands (`POST /api/worker/workflow-tasks/{id}/complete` and
`POST /api/worker/activity-tasks/{id}/continuation/complete`) may be up to 16
times `max_payload_bytes`: 32 MiB with the default 2 MiB. It follows
`DW_MAX_PAYLOAD_BYTES` and has no setting of its own. Each command's arguments
still meet the per-payload limit inside the completion. PHP Server 2.x and
earlier native releases cap a completion at one payload limit, like any other
request.

Native 3.9.1 advertises the actual budget as
`limits.max_workflow_task_completion_bytes` in cluster info. It measures the
complete encoded JSON request body, including envelopes, metadata and commands,
for both completion routes. Use that advertised value rather than inferring a
multiplier from the Server version or summing decoded arguments. Per-value and
ordinary-route limits remain unchanged. The field grants no additional external
payload upload or reference-placement authority.

Rust SDK 4.16 completion planning honors a positive advertised cap even below
the ordinary limit. An absent field preserves the conservative fallback; an explicitly
malformed, null, zero, negative or non-integer value refuses completion planning
before upload. Ordinary routes ignore this unused completion field.

A larger completion is refused with `413 payload_too_large` and
`limit_kind: workflow_task_completion_bytes` before anything is recorded.
Replaying the task produces the same commands, so sending the completion again
gets the same refusal. When PHP SDK 3.17.0 or later receives this typed refusal, it does not resend
the completion: the worker fails the run once with a `fail_workflow` command whose message starts
`Structural limit exceeded: workflow_task_completion_bytes:` and names both
sizes, and the Server records the failure with
`failure_category = structural_limit`.

Python SDK 2.17 and Rust SDK 4.15 include handling for the typed refusal.
On Server 3.8.0, a published Python control hit `httpx.ReadError` before receiving
HTTP 413 and left the workflow pending. Native 3.9.1 corrects that response
path with bounded request discard: at most 64 MiB of remaining upload data for
at most two seconds before returning the original refusal and closing the
connection. Uploads beyond either bound may still encounter transport errors.
Published Python SDK 2.17.1 with qualified native Server 3.9.1
(linux/amd64, SQLite) now passes the
[actual installed-worker controls](https://github.com/durable-workflow/.github/issues/228#issuecomment-6853834512):
300 × 10 KB within-cap fan-out, one typed 413 and one structural failure with
zero partial activities for an oversized completion, then a successful task on
the same worker. This qualifies that exact Python/Server tuple.

[Published PHP SDK 3.18.1 / native Server 3.9.1 controls](https://github.com/durable-workflow/.github/issues/228#issuecomment-6855610551)
(PHP 8.3.36, linux/amd64, SQLite) also pass 300 × 10 KB without external storage,
one typed 413 and one structural failure with zero partial activities for a
60 × 450 KB completion, then fresh work on the same Worker. This qualifies that
exact PHP/Server tuple; Rust 4.16 is qualified below. Cross-product closure under
shared #228 and request-key acceptance under shared #216 remain separate.

To keep a completion within the limit:

- Split a large fan-out into smaller batches across workflow tasks.
- Move work into child workflows, or continue as new.
- Use external payload storage only for supported payload placements.
  Activity-continuation command arguments remain inline; an activity/result
  upload allowance does not authorize references in those command arguments.

#### Rust command planning {#rust-completion-planning}

Rust SDK 4.16 uses the positive explicit completion budget advertised by the
producer. The [published 4.16/native 3.9.1 controls](https://github.com/durable-workflow/sdk-rust/pull/258#issuecomment-6854288671)
(linux/amd64, SQLite) pass 300 × 10 KB inline fan-out without external storage,
an oversized 60 × 450 KB batch failing structurally with zero partial activities,
and a successful follow-up on the same worker. Without the explicit field,
older producers keep the conservative ordinary cap. The larger budget grants
no new upload or reference-placement authority; activity-continuation command
arguments remain inline. These controls do not qualify request-key retries
under shared #216 or native timeout cleanup.

## Docker Bootstrap And Provenance

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_EXPOSE_PACKAGE_PROVENANCE` | `false` | Include `package_provenance` in `/api/cluster/info` for admin requests. | `WORKFLOW_SERVER_EXPOSE_PACKAGE_PROVENANCE` |
| `DW_PACKAGE_PROVENANCE_PATH` | `<base_path>/.package-provenance` | Absolute path to the Docker build provenance file. | `WORKFLOW_SERVER_PACKAGE_PROVENANCE_PATH` |
| `DW_SERVICE_BOUNDARY_CROSS_NAMESPACE_DEFAULT` | `allow` | Default service-call boundary action for cross-namespace calls when no more-specific rule matches. | `WORKFLOW_SERVER_SERVICE_BOUNDARY_CROSS_NAMESPACE_DEFAULT` |
| `DW_SERVICE_BOUNDARY_RATE_LIMIT_PER_MINUTE` | unset | Optional per-minute service-call boundary rate limit. | `WORKFLOW_SERVER_SERVICE_BOUNDARY_RATE_LIMIT_PER_MINUTE` |
| `DW_SERVICE_BOUNDARY_MAX_IN_FLIGHT` | unset | Optional service-call boundary concurrency limit. | `WORKFLOW_SERVER_SERVICE_BOUNDARY_MAX_IN_FLIGHT` |

## Docker Bootstrap

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_ENV_AUDIT_STRICT` | `0` | Fail container boot when `env:audit` finds unknown or legacy `DW_*` variables. | - |
| `DW_BOOTSTRAP_RETRIES` | `30` | Bootstrap attempts before entrypoint gives up on migrations and default namespace seed. | `WORKFLOW_SERVER_BOOTSTRAP_RETRIES` |
| `DW_BOOTSTRAP_DELAY_SECONDS` | `2` | Seconds between bootstrap attempts. | `WORKFLOW_SERVER_BOOTSTRAP_DELAY_SECONDS` |

## Workflow Package Controls

The server image bundles `durable-workflow/workflow`, so it also exposes the
package-level `DW_V2_*` controls. These should match worker deployments that
execute the same workflows.

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_V2_NAMESPACE` | unset | Scope workflow instances to a namespace. | `WORKFLOW_V2_NAMESPACE` |
| `DW_V2_TENANCY_ORGANIZATION` | unset | Optional organization segment for the workflow package tenancy hierarchy exposed to readiness, discovery, and operator surfaces. | `WORKFLOW_V2_TENANCY_ORGANIZATION` |
| `DW_V2_TENANCY_PROJECT` | unset | Optional project segment for the workflow package tenancy hierarchy exposed to readiness, discovery, and operator surfaces. | `WORKFLOW_V2_TENANCY_PROJECT` |
| `DW_V2_TENANCY_ENVIRONMENT` | unset | Optional environment segment for the workflow package tenancy hierarchy exposed to readiness, discovery, and operator surfaces. | `WORKFLOW_V2_TENANCY_ENVIRONMENT` |
| `DW_V2_CURRENT_COMPATIBILITY` | unset | Worker-compatibility marker this worker advertises. | `WORKFLOW_V2_CURRENT_COMPATIBILITY` |
| `DW_V2_SUPPORTED_COMPATIBILITIES` | unset | Comma-separated compatibility markers this worker accepts, or `*`. | `WORKFLOW_V2_SUPPORTED_COMPATIBILITIES` |
| `DW_V2_COMPATIBILITY_NAMESPACE` | unset | Compatibility namespace for shared workflow databases with independent fleets. | `WORKFLOW_V2_COMPATIBILITY_NAMESPACE` |
| `DW_V2_COMPATIBILITY_HEARTBEAT_TTL` | `30` | Seconds a worker-compatibility heartbeat remains valid. | `WORKFLOW_V2_COMPATIBILITY_HEARTBEAT_TTL` |
| `DW_V2_PIN_TO_RECORDED_FINGERPRINT` | `true` | Prefer the recorded workflow definition; see the runtime-specific fallback boundary below. | `WORKFLOW_V2_PIN_TO_RECORDED_FINGERPRINT` |
| `DW_V2_CONTINUE_AS_NEW_EVENT_THRESHOLD` | `10000` | History event count at which the package asks the workflow author to continue-as-new. | `WORKFLOW_V2_CONTINUE_AS_NEW_EVENT_THRESHOLD` |
| `DW_V2_CONTINUE_AS_NEW_SIZE_BYTES_THRESHOLD` | `5242880` | Serialized-history byte size at which the package asks for continue-as-new. | `WORKFLOW_V2_CONTINUE_AS_NEW_SIZE_BYTES_THRESHOLD` |
| `DW_V2_HISTORY_EXPORT_SIGNING_KEY` | unset | Optional HMAC key for authenticating history export archives. | `WORKFLOW_V2_HISTORY_EXPORT_SIGNING_KEY` |
| `DW_V2_HISTORY_EXPORT_SIGNING_KEY_ID` | unset | Key identifier recorded alongside signed history exports. | `WORKFLOW_V2_HISTORY_EXPORT_SIGNING_KEY_ID` |
| `DW_V2_UPDATE_WAIT_COMPLETION_TIMEOUT_SECONDS` | `10` | Seconds the server waits for an update to reach a terminal stage. | `WORKFLOW_V2_UPDATE_WAIT_COMPLETION_TIMEOUT_SECONDS` |
| `DW_V2_UPDATE_WAIT_POLL_INTERVAL_MS` | `50` | Milliseconds between update-stage polls. | `WORKFLOW_V2_UPDATE_WAIT_POLL_INTERVAL_MS` |
| `DW_V2_GUARDRAILS_BOOT` | `warn` | Boot-time structural guardrail mode: `warn`, `fail`, or `silent`. | `WORKFLOW_V2_GUARDRAILS_BOOT` |
| `DW_V2_LIMIT_PENDING_ACTIVITIES` | `2000` | Package-level pending-activity ceiling. | `WORKFLOW_V2_LIMIT_PENDING_ACTIVITIES` |
| `DW_V2_LIMIT_PENDING_CHILDREN` | `1000` | Package-level pending-child-workflow ceiling. | `WORKFLOW_V2_LIMIT_PENDING_CHILDREN` |
| `DW_V2_LIMIT_PENDING_TIMERS` | `2000` | Package-level pending-timer ceiling. | `WORKFLOW_V2_LIMIT_PENDING_TIMERS` |
| `DW_V2_LIMIT_PENDING_SIGNALS` | `5000` | Package-level pending-signal ceiling. | `WORKFLOW_V2_LIMIT_PENDING_SIGNALS` |
| `DW_V2_LIMIT_PENDING_UPDATES` | `500` | Package-level pending-update ceiling. | `WORKFLOW_V2_LIMIT_PENDING_UPDATES` |
| `DW_V2_LIMIT_COMMAND_BATCH_SIZE` | `1000` | Maximum commands accepted in one workflow-task completion. | `WORKFLOW_V2_LIMIT_COMMAND_BATCH_SIZE` |
| `DW_V2_LIMIT_PAYLOAD_SIZE_BYTES` | `2097152` | Package-level single-payload byte ceiling. | `WORKFLOW_V2_LIMIT_PAYLOAD_SIZE_BYTES` |
| `DW_V2_LIMIT_MEMO_SIZE_BYTES` | `262144` | Package-level workflow-memo byte ceiling. | `WORKFLOW_V2_LIMIT_MEMO_SIZE_BYTES` |
| `DW_V2_LIMIT_SEARCH_ATTRIBUTE_SIZE_BYTES` | `40960` | Package-level search-attribute byte ceiling. | `WORKFLOW_V2_LIMIT_SEARCH_ATTRIBUTE_SIZE_BYTES` |
| `DW_V2_LIMIT_HISTORY_TRANSACTION_SIZE` | `5000` | Package-level history-transaction event ceiling. | `WORKFLOW_V2_LIMIT_HISTORY_TRANSACTION_SIZE` |
| `DW_V2_LIMIT_WARNING_THRESHOLD_PERCENT` | `80` | Percent of a structural limit at which the package emits a warning. | `WORKFLOW_V2_LIMIT_WARNING_THRESHOLD_PERCENT` |
| `DW_V2_TASK_DISPATCH_MODE` | `queue` | Package-level task dispatch mode, usually overridden by `DW_TASK_DISPATCH_MODE` in server mode. | - |
| `DW_V2_MATCHING_ROLE_QUEUE_WAKE` | `true` | Whether queue workers run the in-worker matching-role wake on every Looping event. Set to `false` to opt execution-only nodes out of the broad-poll wake when a dedicated `php artisan workflow:v2:repair-pass --loop` daemon owns the sweep instead. | `WORKFLOW_V2_MATCHING_ROLE_QUEUE_WAKE` |
| `DW_V2_TASK_REPAIR_REDISPATCH_AFTER_SECONDS` | `3` | Seconds before an orphaned workflow task is redispatched by repair. | `WORKFLOW_V2_TASK_REPAIR_REDISPATCH_AFTER_SECONDS` |
| `DW_V2_TASK_REPAIR_LOOP_THROTTLE_SECONDS` | `5` | Minimum seconds between task-repair passes per queue. | `WORKFLOW_V2_TASK_REPAIR_LOOP_THROTTLE_SECONDS` |
| `DW_V2_TASK_REPAIR_SCAN_LIMIT` | `25` | Maximum tasks considered per task-repair pass. | `WORKFLOW_V2_TASK_REPAIR_SCAN_LIMIT` |
| `DW_V2_TASK_REPAIR_FAILURE_BACKOFF_MAX_SECONDS` | `60` | Maximum task-repair failure backoff in seconds. | `WORKFLOW_V2_TASK_REPAIR_FAILURE_BACKOFF_MAX_SECONDS` |
| `DW_V2_MULTI_NODE` | `false` | Declare a multi-node deployment so cache backends are validated for cross-node coordination. | `WORKFLOW_V2_MULTI_NODE` |
| `DW_V2_VALIDATE_CACHE_BACKEND` | `true` | Validate the long-poll cache backend at boot. | `WORKFLOW_V2_VALIDATE_CACHE_BACKEND` |
| `DW_V2_CACHE_VALIDATION_MODE` | `warn` | Cache-backend validation failure mode: `fail`, `warn`, or `silent`. | `WORKFLOW_V2_CACHE_VALIDATION_MODE` |
| `DW_V2_FLEET_VALIDATION_MODE` | `warn` | Fleet-compatibility validation mode: `warn` logs, `fail` blocks dispatch and fails closed when no compatible worker is available. | `WORKFLOW_V2_FLEET_VALIDATION_MODE` |
| `DW_SERIALIZER` | `avro` | Payload codec diagnostic input; final v2 resolves new-run payloads to Avro. | `WORKFLOW_SERIALIZER` |

**Released fingerprint preference.** Server 2.9.9 makes the fingerprint
preference effective for service workers. With pinning enabled, a changed
definition may fall back only when no eligible recorded-definition worker
remains and the replacement advertises `workflow_entry_arguments_validation`.
With pinning disabled, the source preference is removed; build pins,
namespace/queue/type isolation, registration identity and strict replay remain.
The accompanying entry safeguards are available from PHP SDK 2.5.4, Python SDK
2.6.2 and Rust SDK 3.5.6. These are minimum feature versions; install current
stable releases for security fixes. Do not assume this behavior on an
older runtime merely because its configuration contains the same key. See
[definition routing](./worker-compatibility-routing.md#definition-fingerprint-evolution).

Use `DW_V2_GUARDRAILS_BOOT` in CI and deployment manifests. The older `WORKFLOW_V2_GUARDRAILS_BOOT` name is retained only so `env:audit` can point alpha-era operators at the rename; the workflow package no longer reads it as a runtime fallback.

Use [Task Matching and Dispatch](/docs/polyglot/task-matching-dispatch)
when you configure `DW_V2_MATCHING_ROLE_QUEUE_WAKE` or a dedicated
`workflow:v2:repair-pass --loop` daemon. Those settings change where
ready-task discovery runs, not the worker-protocol contract itself.

## Additional published PHP controls {#additional-published-php-controls}

This reference covers the published **PHP Server 2.11.0** contract, including the controls below. Check the image you actually run: the independently selected installation version can be older. `unset` means no environment value is supplied; its effect depends on the control. Defaults and aliases come from the [published contract](https://github.com/durable-workflow/server/blob/36d16273604755f3dd0743cab331cc7c62e51d8e/config/dw-contract.php). The versioned contract snapshot is checked against every locale during the documentation build. This reference does not change the selected installation tuple.

### Namespace HTTP admission {#namespace-request-controls}

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_NAMESPACE_MAX_REQUESTS_PER_MINUTE` | `unset` | Optional per-minute control-plane request cap for each namespace. Unset means unlimited. Native 3.x refuses to start when it is non-empty; see [Native Server 3.x](#native-server-3x). | `WORKFLOW_SERVER_NAMESPACE_MAX_REQUESTS_PER_MINUTE` |
| `DW_NAMESPACE_MAX_CONCURRENT_REQUESTS` | `unset` | Optional concurrent control-plane request cap for each namespace. Unset means unlimited. Native 3.x refuses to start when it is non-empty; see [Native Server 3.x](#native-server-3x). | `WORKFLOW_SERVER_NAMESPACE_MAX_CONCURRENT_REQUESTS` |
| `DW_NAMESPACE_HARD_MAX_REQUESTS_PER_MINUTE` | `unset` | Non-bypassable ceiling for namespace control-plane requests per minute, including namespace overrides. Native 3.x refuses to start when it is non-empty; see [Native Server 3.x](#native-server-3x). | `WORKFLOW_SERVER_NAMESPACE_HARD_MAX_REQUESTS_PER_MINUTE` |
| `DW_NAMESPACE_HARD_MAX_CONCURRENT_REQUESTS` | `unset` | Non-bypassable ceiling for concurrent namespace control-plane requests, including namespace overrides. Native 3.x refuses to start when it is non-empty; see [Native Server 3.x](#native-server-3x). | `WORKFLOW_SERVER_NAMESPACE_HARD_MAX_CONCURRENT_REQUESTS` |
| `DW_NAMESPACE_ADMISSION_OVERRIDES` | `{}` | JSON object keyed by namespace with optional request-rate and concurrent-request overrides. Native 3.x refuses to start unless it is unset or `{}`; see [Native Server 3.x](#native-server-3x). | `WORKFLOW_SERVER_NAMESPACE_ADMISSION_OVERRIDES` |

[Namespace request admission](./task-queue-admission.md#namespace-request-admission).

### Namespace durable-state quotas {#namespace-durable-controls}

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_NAMESPACE_DURABLE_LIMITS` | `{}` | JSON object of default per-namespace durable row limits across workflow instances, runs, history, tasks, timers, waits, commands, schedules, workers, and streams. Lifetime and pending/open limits are independent; pending workflow tasks count queued ready rows because active leases have separate admission limits. An empty object is unlimited. Native 3.x refuses to start unless it is unset or `{}`; see [Native Server 3.x](#native-server-3x). | `WORKFLOW_SERVER_NAMESPACE_DURABLE_LIMITS` |
| `DW_NAMESPACE_DURABLE_HARD_LIMITS` | `{}` | JSON object of non-bypassable ceilings for the namespace durable row limit fields, including per-namespace overrides. An empty object adds no operator hard ceilings. Native 3.x refuses to start unless it is unset or `{}`; see [Native Server 3.x](#native-server-3x). | `WORKFLOW_SERVER_NAMESPACE_DURABLE_HARD_LIMITS` |
| `DW_NAMESPACE_DURABLE_OVERRIDES` | `{}` | JSON object keyed by namespace with optional namespace durable row limit overrides. Hard limits remain authoritative. Native 3.x refuses to start unless it is unset or `{}`; see [Native Server 3.x](#native-server-3x). | `WORKFLOW_SERVER_NAMESPACE_DURABLE_OVERRIDES` |

[Namespace durable-state quotas](./task-queue-admission.md#namespace-durable-state).

### External payload quotas and S3 transport {#external-payload-quotas-and-s3}

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_EXTERNAL_PAYLOAD_COMPLETION_MAX_BYTES` | `unset` | Distinct bytes allowed across immutable upload slots for one leased completion; unset uses DW_EXTERNAL_PAYLOAD_MAX_BYTES. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_COMPLETION_MAX_BYTES` |
| `DW_EXTERNAL_PAYLOAD_MAX_BYTES_PER_NAMESPACE` | `unset` | Optional cumulative external-payload byte cap for each namespace. Ready and in-progress objects count toward the cap; unset means unlimited. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_MAX_BYTES_PER_NAMESPACE` |
| `DW_EXTERNAL_PAYLOAD_MAX_OBJECTS_PER_NAMESPACE` | `unset` | Optional external-payload object-count cap for each namespace. Ready and in-progress objects count toward the cap; unset means unlimited. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_MAX_OBJECTS_PER_NAMESPACE` |
| `DW_EXTERNAL_PAYLOAD_HARD_MAX_BYTES_PER_NAMESPACE` | `unset` | Non-bypassable cumulative external-payload byte ceiling for each namespace, including namespace overrides. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_HARD_MAX_BYTES_PER_NAMESPACE` |
| `DW_EXTERNAL_PAYLOAD_HARD_MAX_OBJECTS_PER_NAMESPACE` | `unset` | Non-bypassable external-payload object-count ceiling for each namespace, including namespace overrides. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_HARD_MAX_OBJECTS_PER_NAMESPACE` |
| `DW_EXTERNAL_PAYLOAD_NAMESPACE_OVERRIDES` | `{}` | JSON object keyed by namespace with optional max_bytes and max_objects external-payload quota overrides. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_NAMESPACE_OVERRIDES` |
| `DW_EXTERNAL_PAYLOAD_S3_ACCESS_KEY_ID` | `unset` | Optional access-key ID for the runtime-managed S3-compatible external-payload disk. May be omitted when workload identity supplies credentials. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_S3_ACCESS_KEY_ID` |
| `DW_EXTERNAL_PAYLOAD_S3_SECRET_ACCESS_KEY` | `unset` | Optional secret access key for the runtime-managed S3-compatible external-payload disk. Must be set with the access-key ID. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_S3_SECRET_ACCESS_KEY` |
| `DW_EXTERNAL_PAYLOAD_S3_SESSION_TOKEN` | `unset` | Optional session token for temporary credentials on the runtime-managed S3-compatible external-payload disk. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_S3_SESSION_TOKEN` |
| `DW_EXTERNAL_PAYLOAD_S3_REGION` | `us-east-1` | Region used by the runtime-managed S3-compatible external-payload disk. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_S3_REGION` |
| `DW_EXTERNAL_PAYLOAD_S3_BUCKET` | `unset` | Bucket used by the runtime-managed S3-compatible external-payload disk. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_S3_BUCKET` |
| `DW_EXTERNAL_PAYLOAD_S3_ENDPOINT` | `unset` | Optional HTTPS endpoint for an S3-compatible external-payload service. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_S3_ENDPOINT` |
| `DW_EXTERNAL_PAYLOAD_S3_USE_PATH_STYLE_ENDPOINT` | `false` | Use path-style bucket addressing for the runtime-managed S3-compatible external-payload service. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_S3_USE_PATH_STYLE_ENDPOINT` |

### Caller-namespace service boundaries {#service-caller-namespace-controls}

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_SERVICE_BOUNDARY_NAMESPACE_RATE_LIMIT_PER_MINUTE` | `unset` | Optional shared per-minute service-call limit aggregated across each caller namespace. | `WORKFLOW_SERVER_SERVICE_BOUNDARY_NAMESPACE_RATE_LIMIT_PER_MINUTE` |
| `DW_SERVICE_BOUNDARY_NAMESPACE_MAX_IN_FLIGHT` | `unset` | Optional shared in-flight service-call limit aggregated across each caller namespace. | `WORKFLOW_SERVER_SERVICE_BOUNDARY_NAMESPACE_MAX_IN_FLIGHT` |
| `DW_SERVICE_BOUNDARY_HARD_RATE_LIMIT_PER_MINUTE` | `unset` | Non-bypassable per-minute ceiling for service-call boundary and caller-namespace policies. | `WORKFLOW_SERVER_SERVICE_BOUNDARY_HARD_RATE_LIMIT_PER_MINUTE` |
| `DW_SERVICE_BOUNDARY_HARD_MAX_IN_FLIGHT` | `unset` | Non-bypassable in-flight ceiling for service-call boundary and caller-namespace policies. | `WORKFLOW_SERVER_SERVICE_BOUNDARY_HARD_MAX_IN_FLIGHT` |
| `DW_SERVICE_BOUNDARY_NAMESPACE_OVERRIDES` | `{}` | JSON object of caller-namespace service-call rate and in-flight overrides; hard ceilings still apply. | `WORKFLOW_SERVER_SERVICE_BOUNDARY_NAMESPACE_OVERRIDES` |

### Coordination, storage and runtime {#coordination-storage-and-runtime-controls}

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_OBSERVABILITY_CONTEXT_ENABLED` | `false` | Enable optional bounded trace context persistence, delivery and capability discovery when its optional API and schema are installed. | - |
| `DW_OBSERVABILITY_BAGGAGE_ALLOWLIST` | `[]` | JSON list of baggage keys permitted on run origins; default empty. Links never carry baggage. | - |
| `DW_STORAGE_ADMISSION_FILE` | `unset` | Optional read-only local storage-pressure observation file; invalid or stale observations fence writes. | - |
| `DW_STORAGE_ADMISSION_SOURCE` | `unset` | Expected storage resource identity in the admission observation. | - |
| `DW_STORAGE_ADMISSION_MAX_AGE_SECONDS` | `15` | Maximum age of a storage observation in seconds (1-300). | - |
| `DW_RUNTIME_CREDENTIALS_ENABLED` | `false` | Enable durable, database-backed, namespace-bound runtime bearer credentials managed through the admin API. | `WORKFLOW_SERVER_RUNTIME_CREDENTIALS_ENABLED` |
| `DW_WORKER_LONG_POLL_MAX_CONCURRENT_PER_NAMESPACE` | `unset` | Optional per-node cap for concurrent held workflow/activity long-poll waits from one namespace. The effective cap cannot exceed the global worker wait cap. Configured namespace isolation fails closed when shared cache authority is unavailable. Native 3.x ignores it; see [Native Server 3.x](#native-server-3x). | `WORKFLOW_SERVER_WORKER_LONG_POLL_MAX_CONCURRENT_PER_NAMESPACE` |
| `DW_QUERY_TASK_POLL_MAX_CONCURRENT_PER_NAMESPACE` | `unset` | Optional per-node cap for concurrent held query-task long-poll waits from one namespace. The effective cap cannot exceed the global query-task wait cap. Configured namespace isolation fails closed when shared cache authority is unavailable. Native 3.x ignores it; see [Native Server 3.x](#native-server-3x). | `WORKFLOW_SERVER_QUERY_TASK_POLL_MAX_CONCURRENT_PER_NAMESPACE` |
| `DW_V1_ENABLED` | `false` | Enable legacy embedded v1 migration and queue-drain behavior. Standalone Server defaults to v2-only; enable only when deliberately preserving v1 state. | `WORKFLOW_V1_ENABLED` |

## Runtime Infrastructure Variables

Runtime infrastructure variables are framework and dependency controls. The
server audit recognizes them so it does not warn, but they are not stable
Durable Workflow API knobs. Native 3.x reads the Database group, including the
TLS settings below. It has no Redis, cache or Laravel queue, writes its logs to
standard error and does not read `LOG_CHANNEL` or `LOG_LEVEL`.

| Group | Variables | Use |
| --- | --- | --- |
| Application | `APP_NAME`, `APP_ENV`, `APP_KEY`, `APP_DEBUG`, `APP_URL`, `APP_VERSION`, locale, timezone, maintenance, and cipher settings | Laravel application runtime identity and boot behavior. |
| Logging | `LOG_CHANNEL`, `LOG_LEVEL`, `LOG_STDERR_FORMATTER`, stack, daily, Slack, and Papertrail settings | Laravel logging destinations, formatting and retention. |
| Database | `DB_CONNECTION`, `DB_HOST`, `DB_PORT`, `DB_DATABASE`, `DB_USERNAME`, `DB_PASSWORD`, `DB_SOCKET`, `DB_URL`, `DB_SSLMODE` (`DB_SSL_MODE`), `DB_SSLROOTCERT` (`DB_SSL_ROOT_CERT`), `MYSQL_ATTR_SSL_CA`, charset, collation, and foreign-key settings | SQL state store for workflow, namespace, worker, and projection tables. See [database TLS](#database-tls). |
| Redis/cache/queue | `REDIS_*`, `QUEUE_CONNECTION`, `QUEUE_FAILED_DRIVER`, `CACHE_STORE`, `CACHE_PREFIX`, `SESSION_*` | Queue workers, cache locks, long-poll signaling, and web/session runtime support. |
| Filesystems/mail/broadcasting | `FILESYSTEM_DISK`, `MAIL_*`, `BROADCAST_*`, `PUSHER_*`, `AWS_*` | Framework integrations used by deployment-specific features. |
| Build/runtime | `MYSQL_VERSION`, `REDIS_VERSION`, `PHP_CLI_SERVER_WORKERS`, `VITE_APP_NAME`, `BCRYPT_ROUNDS` | Docker Compose images and framework runtime behavior. |

## Database TLS {#database-tls}

Native 3.x resolves these settings as PHP 2.x does: the same inputs, aliases,
mode names and refusals. It has one Server process to configure, plus any
`schema-upgrade` or takeover command run against the same database. On PHP
2.x, set the same TLS settings on the API server, queue worker,
bootstrap/migration job and scheduler. PostgreSQL defaults to `prefer`; MySQL/MariaDB keep their
connection defaults when no mode is set. Neither default guarantees an
authenticated encrypted connection.

| `DB_SSLMODE` | PostgreSQL | MySQL / MariaDB |
| --- | --- | --- |
| `disable` | No TLS. | No TLS; conflicting TLS options are refused. |
| `allow`, `prefer` | Opportunistic TLS without an encryption or identity guarantee. | Refused. |
| `require` | Encryption required. | Encryption required with mysqlnd and OpenSSL. |
| `verify-ca` | Encryption and a trusted certificate chain required. | Refused: PDO mysqlnd cannot verify only the chain independently of the hostname. |
| `verify-full` | Encryption, trusted chain and matching hostname required. | Encryption, trusted chain and matching hostname required. |

`DB_SSL_MODE` aliases `DB_SSLMODE`; `DB_SSL_ROOT_CERT` aliases
`DB_SSLROOTCERT`. For certificate verification, mount a readable CA bundle at
the configured path in **every role**, including migrations. `MYSQL_ATTR_SSL_CA`
can supply the CA for explicit MySQL/MariaDB `verify-full`. Conflicting
canonical, alias or URL settings refuse startup. Required TLS needs an explicit
TCP hostname; Unix sockets are refused. Explicit MySQL/MariaDB TLS modes need
PDO MySQL with mysqlnd and OpenSSL.

```dotenv
DB_CONNECTION=pgsql
DB_HOST=database.example.com
DB_SSLMODE=verify-full
DB_SSLROOTCERT=/run/database-ca/ca.pem
```

Use the certificate's DNS hostname. With Helm, configure
`externalDatabase.sslMode` and `externalDatabase.tls.existingSecret` / `caKey`.
Create the Secret before installation or upgrade; the chart mounts its CA in
all four roles, including the bootstrap hook. Restart persistent roles after
changing environment settings because configuration is cached at startup.

## Container logs

Native 3.x always writes its logs to standard error. On PHP 2.x, set
`LOG_CHANNEL=stderr` on API, worker and scheduler containers so `docker logs`
or `kubectl logs` can collect application logs. Set it explicitly on existing
releases. Use `LOG_LEVEL` to select the minimum severity. For newline-delimited
JSON, also set:

```dotenv
LOG_STDERR_FORMATTER="Monolog\Formatter\JsonFormatter"
```

Restart persistent roles after changing these settings because configuration is
cached at startup. An explicit file channel needs its own storage and collection;
Kubernetes does not collect `storage/logs/laravel.log` automatically.
`DW_LOG_FORMAT` is not a supported setting.

## Migration Notes

When migrating an older deployment, change the public name first and leave the
legacy name unset. If both are present, the `DW_*` value is the value operators
should reason about, and the legacy name should be removed.

```bash
# Before
WORKFLOW_SERVER_AUTH_DRIVER=token
WORKFLOW_SERVER_OPERATOR_TOKEN=operator-secret

# After
DW_AUTH_DRIVER=token
DW_OPERATOR_TOKEN=operator-secret
```

Run the server image with `DW_ENV_AUDIT_STRICT=1` after migration to catch
misspelled variables, stale aliases, and settings copied from older runbooks.

<!-- Source: docs/polyglot/cli-reference.md -->

# CLI Command Reference

This page documents the v2 `dw` command surface as an operator and automation
contract. Use the [CLI guide](/docs/polyglot/cli) for installation and
profile setup; use this page when wiring scripts, CI jobs, runbooks, or AI
agents to exact command shapes.

All server-backed commands target the standalone server control-plane protocol
version `2`. The CLI validates the server-published protocol manifests before
trusting canonical request and response fields.

## Global Options

Server-backed commands accept these options unless noted otherwise.

| Option | Meaning |
| --- | --- |
| `--server`, `-s` | Server base URL. Overrides profiles and environment variables for this invocation. |
| `--namespace` | Target namespace. Overrides the profile namespace. |
| `--token` | Bearer token for this invocation. Prefer profiles with `--token-env` for stored automation. |
| `--env` | Named CLI environment profile. Unknown profile names fail instead of falling back. |
| `--output=table\|json\|jsonl` | Output contract. `table` is human-readable, `json` is one JSON document, and `jsonl` is one JSON object per line for list commands. |
| `--json` | Command-local alias for JSON output on commands that expose it. |

Commands that accept caller payloads use one shared input contract:

| Option | Meaning |
| --- | --- |
| `--input`, `-i` | Inline input document. |
| `--input-file` | Read input from a file, or `-` for stdin. Mutually exclusive with `--input`. |
| `--input-encoding=json\|raw\|base64` | Decode input as JSON, pass raw text as one argument, or decode base64 as one argument. Defaults to `json`. |

JSON input for workflow starts, signals, queries, updates, schedules, and
activity completions represents the v2 positional argument array. Raw and
base64 input become one positional argument so the server still receives the
canonical `input` array.

## Connection And Diagnostics

| Command | Purpose | Important options |
| --- | --- | --- |
| `dw --version` | Print CLI build identity. When `DW_ENV` or `DURABLE_WORKFLOW_SERVER_URL` selects a target, also performs a short compatibility probe. | `-V`, `--version` |
| `dw server:health` | Check server health and auth reachability. | global options, `--json` |
| `dw server:info` | Show server version, the role-topology manifest, protocol manifests, request contract, worker protocol, worker-fleet facts, and compatibility metadata. | global options, `--json` |
| `dw doctor` | Explain the resolved profile/server/token/TLS state, remote compatibility warnings, and next steps. | global options, `--json` |
| `dw debug workflow <workflow-id>` | Capture stuck-run diagnostics for one workflow: state, pending tasks, queue facts, failures, and compatibility metadata. | `--run-id`, global options, `--json` |
| `dw server:start-dev` | Serve an existing, configured Laravel project's HTTP application. Requires external PHP. | `--port`, `--host` (default `127.0.0.1`), `--db=sqlite\|mysql\|pgsql` |
| `dw watch workflow <workflow-id>` | Poll a workflow until terminal or until a configured polling limit. | `--run-id`, `--interval`, `--max-polls`, global options |
| `dw upgrade` | Replace the running standalone `dw` binary with a newer (or pinned) release. Refuses to rewrite Composer vendor and Homebrew cellar installs. | `--tag`, `--dry-run`, `--force`, `--output=table\|json` |

Use `server:info` when validating contract shape, `doctor` when explaining why
a CLI cannot talk to a server, and `debug workflow` when support needs one
machine-readable run capture.

### Local Project HTTP Server

Run `dw server:start-dev` from an installed Laravel project's directory containing
`artisan`. The native CLI requires a separate PHP executable on `PATH` for this
command. Configure the database and run migrations first. The selected `--db`
value is passed as `DB_CONNECTION`, so clear Laravel's cached configuration when
changing drivers. Run queue workers and the scheduler separately when required.
Restart the command after editing `.env`. The selected driver stays applied to
HTTP requests through Artisan's `--no-reload` mode.

The command defaults to loopback. Use `--host=0.0.0.0` explicitly to bind to all
interfaces. MySQL and PostgreSQL modes require project Compose services named
`mysql` or `pgsql`, respectively, and `redis`. Failed dependency or HTTP processes
return nonzero. Compose services remain running when the HTTP process ends.

POSIX native binaries forward SIGINT and SIGTERM to the active child and wait
up to five seconds for its process group to stop. Windows uses its console
interruption behavior.

For a standalone Durable Workflow Server using published images, follow the
[service quickstart](/docs/quickstart).

### Server Info And Role Topology

`dw server:info` is the CLI surface for `GET /api/cluster/info`. In table mode
it includes a `Topology:` section that summarizes the role-topology contract
the server published for the responding node:

- `Supported Shapes` and `Current Shape` identify which deployment shapes are
  legal and which one this node is serving right now.
- `Current Process Class` and `Current Roles` name the process class and role
  bundle on the responding node.
- `Matching Role`, `Matching Partitions`, and `Matching Backpressure` expose
  the matching-role shape, wake ownership, task-dispatch mode, and partition
  primitives that determine how ready work is claimed.
- `Matching Discovery Limits` summarizes the frozen numeric matching-role
  contract — `poll_batch_cap`, `availability_ceiling_seconds`,
  `wake_signal_ttl_seconds`, `workflow_task_lease_seconds`, and
  `activity_task_lease_seconds` — so operators can verify the deployment
  matches the documented matching-role contract without grepping the package
  source.
- `Current Write Boundaries` lists the durable write surfaces currently owned
  by the roles on this node.
- `Scaling Boundaries` and `Failure Domains` tell operators what load driver
  or first failure signal to expect for each role.

Use `--output=json` when scripts need the raw manifest. `topology.schema` and
`topology.version` pin the manifest contract revision so scripts can detect
shape drift; `topology.execution_mode` reports the dispatch mode the
responding node serves. The stable machine fields live under `topology`,
including `supported_shapes`, `current_shape`, `current_process_class`,
`current_roles`, `matching_role`, `role_catalog`, `authority_boundaries`,
`authority_surfaces`, `supported_topologies`, `scaling_boundaries`, and
`failure_domains`. The `topology.matching_role` block also publishes
`partition_primitives` and `backpressure_model` so scripts can check which
routing axes the responding node uses for ready-task discovery without
parsing prose. `topology.matching_role.discovery_limits` exposes the frozen
numeric matching-role contract — `poll_batch_cap`,
`availability_ceiling_seconds`, `wake_signal_ttl_seconds`,
`workflow_task_lease_seconds`, and `activity_task_lease_seconds` — so scripts
can pin the workflow package's matching-role numbers without scraping a
human-readable section.

```bash
dw server:info --output=json \
  | jq '.topology | {current_shape, current_process_class, current_roles, matching_role, scaling_boundaries, failure_domains}'
```

For the meaning of those fields, see
[Server Role Topology](/docs/polyglot/server-role-topology).

### Server Info And Coordination Health

`dw server:info` also publishes the server's all-namespaces rollout-safety
verdict so operators can read coordination health without standing up a
separate health surface. In table mode the CLI renders a `Coordination
Health:` section under `Topology:`. In `--output=json` the same data lives
under `coordination_health`, with these stable machine fields:

- `coordination_health.schema` and `coordination_health.version` pin the
  manifest contract revision.
- `coordination_health.namespace_scope` reports whether the verdict covers
  one namespace or the whole fleet.
- `coordination_health.status` and `coordination_health.http_status` report
  the top-level verdict and HTTP gate the server applies to readiness.
- `coordination_health.generated_at` records when the snapshot was taken.
- `coordination_health.categories` summarizes per-category counts (such as
  `correctness`, `safety`, `routing`).
- `coordination_health.warning_checks` and `coordination_health.error_checks`
  list the check names that pushed the verdict to warning or error so
  scripts can branch on the failing surfaces.
- `coordination_health.checks[]` is the per-check detail array, where each
  entry pins `name`, `status`, `category`, and `message` automation can use
  to explain a degraded verdict. The frozen check inventory always includes
  `worker_compatibility`, `task_transport`, `routing_health`,
  `durable_resume_paths`, the projection/scheduler checks, and `activity_path`.
  `activity_path` is the activity-side counterpart of `task_transport`: it
  surfaces activity executions whose schedule-to-start, start-to-close,
  schedule-to-close, or heartbeat deadline has passed without enforcement, plus
  the sustained activity retry backlog. Renaming `activity_path` is a
  protocol-level change.

```bash
dw server:info --output=json \
  | jq '.coordination_health | {status, http_status, namespace_scope, warning_checks, error_checks}'
```

For the underlying readiness contract behind these fields, see the
[Server API Reference](/docs/polyglot/server-api-reference).

### Self-Upgrade

`dw upgrade` downloads the matching platform asset from the
`durable-workflow/cli` GitHub release, verifies it against the release's
`SHA256SUMS`, and replaces the running binary only on a successful checksum
match. Use `--tag` to pin to a specific release tag, `--dry-run` to resolve
the target release and print the asset URLs without downloading, and `--force`
to re-download and replace even when the current and target versions match.

The command refuses to rewrite installations managed by another tool. JSON
output uses a stable `status` field so automation can branch without parsing
prose:

| `status` | Meaning |
| --- | --- |
| `upgraded` | Binary replaced with `target_version`. |
| `noop` | `current_version` already matches `target_version`; `--force` bypasses this. |
| `dry-run` | `--dry-run` resolved `target_version`, `asset_url`, and `checksum_url` without downloading. |
| `refused` | Install kind is not a standalone release binary (Composer vendor or Homebrew cellar), or the platform has no published asset. |
| `permission-denied` | The install directory is not writable; the payload includes a `hint` with the recommended next step. |
| `error` | Release catalog fetch, checksum mismatch, or filesystem error. |

For refusals, the JSON payload's `installation.kind` identifies the managing
tool (`composer-vendor`, `homebrew`, `binary`, or `unknown`) and `reason` names
the right owner to use instead. Only a 2.x PHAR, running its own 2.x code,
reports `phar`. For public, source-free automation, reinstall
a pinned release through `https://durable-workflow.com/install.sh`; for
tap-managed Homebrew installs, use `brew upgrade durable-workflow/tap/dw`.

## Environment Profiles

Profiles live under `~/.config/dw/config.json`, or
`$XDG_CONFIG_HOME/dw/config.json` when set. Set `DW_CONFIG_HOME` to isolate a
test or CI profile directory.

| Command | Purpose | Important options |
| --- | --- | --- |
| `dw env:set <name>` | Create or update a named profile. | `--server`, `--namespace`, `--token-env`, `--token`, `--tls-verify=true\|false`, `--profile-output=table\|json\|jsonl`, `--make-default`, `--json` |
| `dw env:list` | List profiles with literal tokens redacted by default. | `--show-token`, `--json`, `--output=jsonl` |
| `dw env:show [name]` | Show one profile, defaulting to the current profile. | `--show-token`, `--json` |
| `dw env:use <name>` | Set the default profile. Unknown names fail. | `--json` |
| `dw env:delete <name>` | Delete a profile. | `--json` |

Prefer `--token-env=NAME` for production profiles so secrets stay in the
runtime environment and not in the profile file.

## Workflow Commands

| Command | Purpose | Important options |
| --- | --- | --- |
| `dw workflow:start` | Start a workflow through the control plane. | `--type`, `--workflow-id`, `--business-key`, `--task-queue`, `--duplicate-policy`, `--memo`, `--search-attr key=value`, `--execution-timeout`, `--run-timeout`, `--wait`, input options, `--json` |
| `dw workflow:list` | List workflow instances. | `--type`, `--status`, `--query`, `--limit`, global output options |
| `dw workflow:describe <workflow-id>` | Describe the current or selected run. | `--run-id`, `--follow`, `--json` |
| `dw workflow:list-runs <workflow-id>` | List runs for a workflow instance. | `--json` |
| `dw workflow:show-run <workflow-id> <run-id>` | Show one run. | `--follow`, `--json` |
| `dw workflow:history <workflow-id> <run-id>` | Read run history events. | `--follow`, `--page-size`, `--json` |
| `dw workflow:history-export <workflow-id> <run-id>` | Export the archival run-history payload. | `--output-file`, global options |
| `dw workflow:signal <workflow-id> <signal-name>` | Send a signal. | `--run-id`, input options, `--json` |
| `dw workflow:query <workflow-id> <query-name>` | Execute a read-only workflow query. | `--run-id`, input options, `--json` |
| `dw workflow:update <workflow-id> <update-name>` | Submit or execute a workflow update. | `--wait=accepted\|completed`, `--run-id`, input options, `--json` |
| `dw workflow:update-outcome <workflow-id> <update-id>` | Read a retained update outcome without submitting it again. | `--wait=accepted\|completed`, `--wait-timeout`, `--json` |
| `dw workflow:cancel [workflow-id]` | Close one workflow or a selected batch as cancelled; no workflow-code cleanup runs. | `--reason`, `--run-id`, `--all-matching`, `--type`, `--status`, `--limit`, `--yes`, `--json` |
| `dw workflow:terminate <workflow-id>` | Force terminate a workflow. | `--reason`, `--run-id`, `--json` |
| `dw workflow:repair <workflow-id>` | Ask the server to repair a stuck or retryable run. | `--json` |
| `dw workflow:redrive <workflow-id> <run-id>` | Redrive a failed run in a new run that reuses its completed steps. See [Redrive a failed run](../failures-and-recovery.md#redrive). | `--request-id`, `--json` |
| `dw workflow:archive <workflow-id>` | Archive a closed run. | `--reason`, `--json` |

Examples:

```bash
dw workflow:start \
  --type=App\\Workflows\\ProcessOrder \
  --workflow-id=order-123 \
  --task-queue=payments \
  --input='["order-123"]' \
  --json

dw workflow:update order-123 approve --wait=completed --input='["manager"]'
dw workflow:cancel --all-matching='WorkflowType = "ImportJob"' --limit=25 --yes
```

`workflow:start`, `workflow:signal`, `workflow:query`, and `workflow:update`
validate canonical request fields against the server-published request
contract. Non-canonical legacy aliases are rejected before the request is sent.

### Read an update outcome {#read-update-outcome}

CLI 2.8 and later can read an accepted update by its `update_id` on a Server
that advertises `capabilities.update_outcome_reads` (native 3.6.0 or later).
Use the ID from the accepted receipt:

```bash
dw workflow:update-outcome order-123 "$UPDATE_ID"
dw workflow:update-outcome order-123 "$UPDATE_ID" --wait=completed --wait-timeout=30
```

The default `--wait=accepted` answers immediately. An accepted update that has
not finished stays pending, including when a completion wait expires. A
completed update can have a null result. `--json` returns the checked receipt,
so use its `update_status` to distinguish these cases. Each invocation makes
one outcome read and never resubmits the update.

`--wait-timeout` requires `--wait=completed` and an integer of at least one
second. A failed update exits 2. An unknown or pruned update ID exits 5, which
does not prove the update was never admitted. A Server without the capability,
including PHP Server 2.x, exits 8 before an outcome request is sent.

## Service Call Inspection

Use `dw service-call:describe <endpoint> <service> <operation> <call-id>`
to inspect one durable Nexus call in the endpoint's namespace. For example,
`dw service-call:describe billing invoices create call-42 --namespace=shared --json`
returns the complete Server response.

The response preserves the original caller and linked target, call deadline,
status, outcome, attempt records, typed failure and retained payload references.
Successful results remain lossless Avro `result_envelope` values. An accepted
or started call is still pending, rather than a successful completed result.

The command requests a single GET. It does not request a retry, cancellation
or external-object download. The Server can reconcile a call with its linked
workflow's terminal outcome while serving the read. Namespace authorization and
the endpoint, service, operation and call selectors all apply.

## Bridge Adapter Commands

Bridge commands are bounded ingress and handoff tools for integration events.
They call the server bridge-adapter surface and return the
`durable-workflow.v2.bridge-adapter-outcome.contract` shape in JSON mode.
They do not execute workflow code or own workflow state transitions.

| Command | Purpose | Important options |
| --- | --- | --- |
| `dw bridge:webhook <adapter>` | Send one webhook bridge event that starts, signals, or updates a workflow through the control plane. | `--action=start_workflow\|signal_workflow\|update_workflow`, `--idempotency-key`, `--target`, input options, `--correlation`, `--json` |

Examples:

```bash
dw bridge:webhook stripe \
  --action=start_workflow \
  --idempotency-key=stripe-event-1001 \
  --target='{"workflow_type":"orders.fulfillment","task_queue":"external-workflows","business_key":"order-1001"}' \
  --input='{"order_id":"order-1001"}' \
  --json

dw bridge:webhook pagerduty \
  --action=signal_workflow \
  --idempotency-key=pd-event-3003 \
  --target='{"workflow_id":"wf-remediation-42","signal_name":"incident_escalated"}' \
  --input='{"severity":"critical"}'
```

Use the bridge outcome fields instead of inferring behavior from HTTP status
alone. `outcome`, `reason`, `control_plane_outcome`, `idempotency_key`, and
the redacted `target` summary are the automation contract for duplicates,
routing misses, malformed payloads, and accepted handoffs.

## Schedule Commands

| Command | Purpose | Important options |
| --- | --- | --- |
| `dw schedule:create` | Create a schedule. | `--schedule-id`, `--workflow-type`, `--cron`, `--interval`, `--task-queue`, `--timezone`, `--execution-timeout`, `--run-timeout`, `--overlap-policy`, `--jitter`, `--max-runs`, `--paused`, `--note`, input options, `--json` |
| `dw schedule:list` | List schedules. | `--status`, `--type`, `--query`, `--limit`, `--next-page-token`, `--json` |
| `dw schedule:describe <schedule-id>` | Describe one schedule. | `--json` |
| `dw schedule:update <schedule-id>` | Update schedule spec or workflow input. | schedule create options, input options, `--json` |
| `dw schedule:pause <schedule-id>` | Pause a schedule. | `--note`, `--json` |
| `dw schedule:resume <schedule-id>` | Resume a schedule. | `--note`, `--json` |
| `dw schedule:trigger <schedule-id>` | Trigger a schedule immediately. | `--overlap-policy`, `--json` |
| `dw schedule:backfill <schedule-id>` | Backfill a time window. | `--start-time`, `--end-time`, `--overlap-policy`, `--json` |
| `dw schedule:delete <schedule-id>` | Delete a schedule. | `--json` |

Use either `--cron` or `--interval` when creating interval-based schedules.
Use `--paused` for deploy-time registration that should not start work yet.

### Filter and page schedules {#schedule-list-filters}

`dw schedule:list` returns one filtered page in the resolved namespace. Omitting `--namespace` selects the resolved default namespace, not every namespace. Status, workflow type and visibility predicates combine with `AND`. The [normal published CLI and PHP Server qualification](https://github.com/durable-workflow/durable-workflow.github.io/issues/402#issuecomment-6370122227) verifies these options.

| CLI option | Server query parameter | Meaning |
| --- | --- | --- |
| `--status` | `status` | `active` or `paused`. |
| `--type` | `workflow_type` | Exact workflow type. |
| `--query` | `query` | Schedule visibility expression. |
| `--limit` | `page_size` | Page size from 1 to 200, default 50. |
| `--next-page-token` | `next_page_token` | Opaque continuation from the previous response. |

For an existing `production` namespace with a registered keyword attribute `Region`, list paused `orders.rollup` schedules in region `eu` on task queue `orders`:

```bash
dw schedule:list --namespace=production --status=paused --type=orders.rollup \
  --query='Region = "eu" AND TaskQueue = "orders"' --limit=25 --json
```

Read `next_page_token` from the JSON response and set `NEXT_PAGE_TOKEN` to that value. Repeat the same namespace and filters for the next page:

```bash
dw schedule:list --namespace=production --status=paused --type=orders.rollup \
  --query='Region = "eu" AND TaskQueue = "orders"' --limit=25 --json \
  --next-page-token="$NEXT_PAGE_TOKEN"
```

Tokens belong to the namespace and filter selection. Start again without a token after changing filters or receiving a stale-cursor refusal. Each call reads a live page, not a frozen snapshot. See [schedule-list API constraints and refusal reasons](/docs/polyglot/server-api-reference#schedule-list-filters) for the supported query grammar and response fields.

## Worker And Task Queue Commands

| Command | Purpose | Important options |
| --- | --- | --- |
| `dw worker:register [worker-id]` | Register a worker with capacity and compatibility metadata. | `--task-queue`, `--runtime`, `--sdk-version`, `--build-id`, `--workflow-type`, `--activity-type`, `--max-workflow-tasks`, `--max-activity-tasks`, `--json` |
| `dw worker:list` | List workers. | `--task-queue`, `--status`, global output options |
| `dw worker:describe <worker-id>` | Describe one worker. | `--json` |
| `dw worker:deregister <worker-id>` | Deregister one worker. | `--json` |
| `dw task-queue:list` | List active task queues and admission status. | global output options |
| `dw task-queue:describe <queue>` | Describe worker capacity, leases, dispatch budgets, and pending query-task capacity. | `--json` |
| `dw task-queue:build-ids <queue>` | Inspect per-build-id cohort state and rollout status for one queue. | `--json` |
| `dw task-queue:drain <queue>` | Mark a build-id cohort as draining so it stops claiming new tasks. | `--build-id <value>`, `--unversioned`, `--json` |
| `dw task-queue:resume <queue>` | Clear a previous drain so the cohort can claim new tasks again. | `--build-id <value>`, `--unversioned`, `--json` |
| `dw task-queue:pause-activities <queue>` | Hold new activity dispatch on one task queue until it is resumed. | `--reason <text>`, `--json` |
| `dw task-queue:resume-activities <queue>` | Resume activity dispatch on a paused task queue. | `--json` |
| `dw task-queue:activity-dispatch <queue>` | Show whether activity dispatch is paused, and who paused or resumed it, when and why. | `--json` |

The task queue commands are the preferred operator view for throttling,
capacity, and no-worker diagnoses. See
[Task Queue Admission](/docs/polyglot/task-queue-admission) for the
server-side policy behind those fields and
[Worker Build-Id Rollout](/docs/polyglot/worker-build-id-rollout) for the
full unversioned-to-versioned cutover, canary, drain, and rollback lifecycle.

`dw task-queue:drain` and `dw task-queue:resume` both require either
`--build-id <value>` to target a specific build cohort or `--unversioned` to
target the cohort of workers registered without a `build_id`. Combining the
two fails fast with an invalid-option error. Both commands are idempotent:
repeated drains do not shift the recorded `drained_at` timestamp, and
resuming an already-active cohort is a no-op.

`dw task-queue:pause-activities`, `dw task-queue:resume-activities` and
`dw task-queue:activity-dispatch` (CLI 2.7.0 and later) pause, resume and show
activity dispatch on one task queue of the selected namespace, for example
during a downstream outage:

```bash
dw task-queue:pause-activities orders --reason='payments vendor outage'
dw task-queue:activity-dispatch orders
dw task-queue:resume-activities orders
```

While a queue is paused, no activity task on it is leased. Ready tasks stay
ready and keep their attempt count, activities already running finish, and
activity deadlines keep running, so a schedule-to-start or schedule-to-close
timeout still fires. Workflow tasks, signals, queries and updates are
delivered as before. `--reason` takes up to 1000 characters. Pause and resume
are idempotent: `--json` returns the Server's `outcome` (`paused`,
`already_paused`, `resumed` or `not_paused`) with the `activity_dispatch`
state. A pause is not a build-id drain: to stop one build's workers, use
`dw task-queue:drain`.

The three commands need a Server that advertises
`capabilities.task_queue_activity_dispatch_pause` (native Server 3.8.0 or
later). On any other Server, including published PHP Server 2.x, `dw` exits
with code `8` before sending the request. `dw task-queue:describe` and
`dw task-queue:list` show the activity dispatch state when the Server reports
it.

## Worker Protocol Commands

These commands are low-level protocol tools for diagnostics, smoke tests, and
non-SDK worker experiments. Normal PHP and Python workers should use their SDK
worker loops.

| Command | Purpose | Important options |
| --- | --- | --- |
| `dw workflow-task:poll <worker-id>` | Poll one workflow task. | `--task-queue`, `--build-id`, `--poll-request-id`, `--history-page-size`, `--accept-history-encoding`, `--json` |
| `dw workflow-task:history <task-id> <page-token>` | Fetch the next history page for a leased workflow task. | `--lease-owner`, `--attempt`, `--json` |
| `dw workflow-task:complete <task-id> <attempt>` | Complete one workflow task with command payloads. | `--lease-owner`, `--complete-result`, `--command`, `--json` |
| `dw workflow-task:fail <task-id> <attempt>` | Report workflow-task execution failure for retry or diagnosis. Distinct from completing a task with a `fail_workflow` command. | `--lease-owner`, `--message`, `--type`, `--stack-trace`, `--json` |
| `dw query-task:poll <worker-id>` | Poll and lease one routed workflow query task. | `--task-queue`, `--json` |
| `dw query-task:complete <query-task-id> <attempt>` | Complete a leased query task with a JSON result and matching envelope. | `--lease-owner`, `--result`, `--json` |
| `dw query-task:fail <query-task-id> <attempt>` | Report a leased query task failure with a machine-readable reason. | `--lease-owner`, `--message`, `--reason`, `--type`, `--stack-trace`, `--json` |
| `dw activity:complete <task-id> <attempt-id>` | Complete one leased activity attempt. | `--lease-owner`, input options, `--json` |
| `dw activity:fail <task-id> <attempt-id>` | Fail one leased activity attempt. | `--lease-owner`, `--message`, `--type`, `--non-retryable`, `--json` |

Use `workflow-task:fail` for worker-side execution failures such as replay
mismatches or deserialization errors; it targets
`POST /worker/workflow-tasks/{taskId}/fail`. Completing a task with a workflow
command that fails the workflow is a separate concern and goes through
`workflow-task:complete` with the appropriate command payload.

`query-task:poll`, `query-task:complete`, and `query-task:fail` drive the
routed-query worker surface at `/worker/query-tasks/…`. Normal workers pull
query tasks through their SDK query handler; the CLI surface is for
diagnostics, CLI/SDK parity checks, and non-SDK worker experiments. Query-task
failures default to `--reason=query_rejected`; use a stable reason string such
as `unknown_query`, `decode_failure`, or a runtime-specific identifier so
callers can distinguish expected rejections from runtime errors.

### Workflow Task History Pages

`workflow-task:history` is the CLI diagnostic wrapper around the worker-plane
history-page endpoint. Use it only after `workflow-task:poll` returns a leased
workflow task with `next_history_page_token`; normal workers should let their
SDK fetch extra history pages.

<!-- docs-example id="cli.workflow-task-history.command" -->
```bash
dw workflow-task:history workflow-task-01 history-page-2 \
  --lease-owner=python-worker-1 \
  --attempt=2 \
  --json
```

JSON output is the server response without field renaming:

<!-- docs-example id="cli.workflow-task-history.response" -->
```json
{
  "history_events": [
    {"event_id": 2, "event_type": "ActivityScheduled", "payload": {}}
  ],
  "total_history_events": 4,
  "next_history_page_token": "history-page-3"
}
```

Automation should read `history_events`, `total_history_events`, and
`next_history_page_token`. The worker-history endpoint does not use the
control-plane run-history fields `events` or `next_page_token`.

## Namespace And Search Attribute Commands

| Command | Purpose | Important options |
| --- | --- | --- |
| `dw namespace:list` | List namespaces. | global output options |
| `dw namespace:create <name>` | Create a namespace. | `--description`, `--retention`, `--json` |
| `dw namespace:describe <name>` | Describe one namespace. | `--json` |
| `dw namespace:update <name>` | Update namespace metadata. | `--description`, `--retention`, `--json` |
| `dw namespace:set-storage-driver <name> <driver>` | Configure the namespace external payload storage policy used when encoded payloads exceed the offload threshold. | `--threshold-bytes`, `--uri`, `--disk`, `--bucket`, `--prefix`, `--region`, `--endpoint`, `--auth-profile`, `--disable`, `--json` |
| `dw storage:test` | Round-trip a small inline payload and a large offloaded payload through the selected namespace storage policy or driver override. | `--driver=local\|s3\|gcs\|azure`, `--small-bytes`, `--large-bytes`, global options, `--json` |
| `dw search-attribute:list` | List search attributes. | global output options |
| `dw search-attribute:create <name> <type>` | Register a search attribute. | `--json` |
| `dw search-attribute:delete <name>` | Delete a search attribute. | `--json` |

Search attribute types are server-compatible values such as `keyword`, `text`,
`int`, `double`, `bool`, `datetime`, and `keyword_list`.

External payload storage commands call the server's namespace storage API. The
driver argument is one of `local`, `s3`, `gcs`, or `azure`; object-store drivers
use server-side filesystem configuration, so CLI flags describe the namespace
policy rather than carrying provider credentials. Use `--disk` to bind the
`s3`, `gcs`, or `azure` driver to a named server-side filesystem disk that
holds the actual provider credentials. Use `--disable` to keep the policy
record while preventing new offloads.

Examples:

```bash
dw namespace:set-storage-driver billing s3 \
  --disk=external-payload-objects \
  --bucket=dw-payloads \
  --prefix=billing/ \
  --threshold-bytes=2097152 \
  --json

dw namespace:set-storage-driver dev local \
  --uri=file:///var/lib/durable-workflow/payloads

dw storage:test --namespace=billing --large-bytes=2097152 --json
dw storage:test --driver=s3 --small-bytes=128 --large-bytes=3145728 --json
```

In JSON mode, `namespace:set-storage-driver` returns the namespace payload with
its `external_payload_storage` policy. `storage:test` returns the diagnostic
status plus `small_payload` and `large_payload` result objects; automation
should branch on those fields instead of parsing the human table.

The diagnostic is also the fastest operator check for the reference-envelope
contract. A passing large-payload result proves the selected policy can write
encoded bytes, return a `durable-workflow.v2.external-payload-reference.v1`
reference, read the object back, and verify `size_bytes` plus `sha256`. A failed
diagnostic should be treated as a storage-policy problem before workflows are
allowed to offload payloads through that namespace.

## System Commands

System commands expose server maintenance passes as explicit, scriptable
operations.

| Command | Purpose | Important options |
| --- | --- | --- |
| `dw system:repair-status` | Show workflow repair backlog/status. | `--json` |
| `dw system:repair-pass` | Run one repair pass. | `--run-id`, `--limit`, `--json` |
| `dw system:activity-timeout-status` | Show activity timeout backlog/status. | `--json` |
| `dw system:activity-timeout-pass` | Run one activity timeout pass. | `--task-id`, `--limit`, `--json` |
| `dw system:retention-status` | Show retention backlog/status. | `--json` |
| `dw system:retention-pass` | Run one retention cleanup pass. | `--run-id`, `--limit`, `--json` |

Prefer status commands before pass commands in runbooks so operators can see
the pending scope before mutating server state.

## Schema Commands

| Command | Purpose | Important options |
| --- | --- | --- |
| `dw schema:list` | List published machine-readable schemas. | no server connection required |
| `dw schema:show <command-name>` | Show the bundled JSON Schema for one command output. | `--output=json\|jsonl`; no server connection required |
| `dw schema:manifest` | Show the schema manifest. | no server connection required |

Schema commands are useful when an AI client or CI job needs the current
control-plane, response, or output contract without scraping prose docs.
The [current v4 manifest](https://durable-workflow.github.io/cli-json-envelopes/v4/manifest.json)
binds every JSON envelope and record-level JSONL schema by public resolver and
digest. Its workflow start/run and query/update payload fields accept only the
`avro` codec. The retained
[v3 manifest](https://durable-workflow.github.io/cli-json-envelopes/v3/manifest.json)
and [v2 manifest](https://durable-workflow.github.io/cli-json-envelopes/v2/manifest.json)
remain available with their original bytes for revision-pinned consumers.

## Output And Exit Contract

Use `--output=json` when a command returns one object and `--output=jsonl` when
a list command feeds a stream processor. Human tables are allowed to improve
over time; JSON field names are the automation contract.

All commands use the stable exit-code policy documented in the
[CLI guide](/docs/polyglot/cli#exit-codes):

| Code | Meaning |
| --- | --- |
| `0` | Success. |
| `1` | Command ran but failed generically. |
| `2` | Invalid local usage or validation error. |
| `3` | Network or transport failure. |
| `4` | Authentication or authorization failure. |
| `5` | Resource not found. |
| `6` | Server-side `5xx` failure. |
| `7` | Timeout. |
| `8` | Compatibility failure: the command refused before sending the request. |

For support bundles, collect `dw doctor --output=json`, `dw server:info
--output=json`, and `dw debug workflow <workflow-id> --output=json`.

## Related Guides

- [CLI](./cli.mdx) covers installation, profile setup, and exit-code behavior.
- [Server](./server.md) documents the HTTP control plane that server-backed
  commands call.
- [Client and Worker Capabilities](./cli-python-parity.md) compares
  CLI, PHP, Python, and Rust client and worker surfaces.

<!-- Source: docs/polyglot/invocable-carrier.md -->

# Invocable HTTP Carrier

The invocable HTTP carrier is the first concrete carrier defined under the
[external execution surface](./external-execution.md). It is published from
`GET /api/cluster/info` at
`worker_protocol.invocable_carrier_contract` with
`schema: durable-workflow.v2.invocable-carrier.contract`, `version: 1`, and
`carrier_type: invocable_http`.

When a configured handler mapping resolves to an `invocable_http` carrier, the
configured endpoint is invoked with the carrier-neutral
[external task input envelope](./external-execution.md#published-contract-seams)
and reconciles the response against the
[external task result envelope](./external-execution.md#published-contract-seams).
Workflow tasks, signal/update ordering, replay, and history mutation stay
inside the server. The carrier only moves declared input and result envelopes
across an HTTPS boundary.

:::warning

**Current limitation:** the Server does not send the HTTP request itself. It validates the mapping and returns it on the activity poll response. A process you run must poll the queue, call the endpoint and report the result. If nothing polls the queue, the activity task waits. A Server-side dispatcher will be built when users need it ([#527](https://github.com/durable-workflow/server/issues/527)).

:::

## What It Is

The invocable HTTP carrier exists for activity-grade work that an operator
team would rather host as an HTTPS handler than as a long-poll worker:

- operator maintenance activities and platform automation hosted in an internal
  HTTP service
- bounded integration handoffs that already terminate at an HTTPS endpoint
- serverless or container-based activity handlers that expose a single POST
  route per activity type

Activity execution is delegated to the configured endpoint. Durable workflow
state, history, and the activity-completion contract remain server-owned.

## Activity-Only Scope

The published manifest fixes the carrier scope to `task_kinds: [activity_task]`
and names explicit non-goals so the boundary cannot drift through configuration:

- `workflow_task_execution` — workflow tasks remain on real workflow runtimes
- `workflow_replay` — the carrier never replays workflow history
- `history_mutation` — handlers may not edit event history directly
- `generic_webhook_ingress` — generic ingress goes through a bridge adapter,
  not through this carrier

A handler mapping with a non-activity `kind`, or a carrier mapping with a
non-`activity_task` capability, fails configuration validation with
`invalid_invocable_carrier_scope` before it can appear on the activity poll
response.

## HTTPS Target And Method

Carrier `target_fields` are validated by the server before a mapping is
exposed:

| Field | Required | Allowed | Notes |
| --- | --- | --- | --- |
| `url` | yes | absolute HTTPS URL, or HTTP for loopback dev (`localhost`, `127.0.0.0/8`, `::1`) | URL credentials (`scheme://user:pass@host`) are forbidden. |
| `method` | no | `POST` | Defaults to `POST`. No other HTTP method is accepted. |
| `timeout_seconds` | no | integer 1–900 | Transport deadline for one handler attempt. The task deadline is enforced separately by the server/runtime. |
| `retry_policy` | no | object (see below) | Carrier-owned transport retry budget. |

Invalid targets fail closed with `invalid_carrier_target` and never appear in
discovery output.

## Auth Model

Non-loopback `invocable_http` mappings must resolve an `auth_ref` from the
external executor configuration. Unauthenticated invocable HTTP is allowed
only against loopback HTTP targets so a developer can iterate locally.

- `auth_refs` are declared once at the top of the config and referenced from
  defaults or per-mapping. Supported types include `profile`, `env`,
  `token_file`, `mtls`, and `signed_headers`.
- A mapping that resolves to a non-loopback target without an effective
  `auth_ref` fails configuration validation with `missing_invocable_auth_ref`.
- Tokens, secrets, signatures, and authorization headers are never echoed in
  cluster diagnostics, in `dw server:info` output, or on the activity poll
  response. The mapping diagnostics report which `auth_ref` resolved and the
  redacted summary, never the credential value.

The intent is that an operator can read the redacted runtime diagnostics and
confirm which auth secret will be used, without the server ever exposing the
secret itself.

## Transport Retry Versus Durable Activity Retry

The optional `retry_policy` on an invocable carrier is **transport-only**. It
governs the carrier's HTTP delivery before the handler reports a result:

| `retry_policy` field | Allowed | Default | Meaning |
| --- | --- | --- | --- |
| `max_attempts` | integer 1–5 | `1` | Maximum HTTP delivery attempts the carrier will make for one task lease. |
| `backoff_seconds` | array of integers, each 0–300, up to 5 entries | `[]` | Per-attempt backoff before the next HTTP try. |
| `retryable_status_codes` | subset of `[408, 425, 429, "5xx"]` | `[408, 429, "5xx"]` | Response codes that count as transport-retryable. |

Transport retries never become history events. The server only learns about
the carrier's final attempt — either the structured success/failure envelope
the handler returned, or a transport timeout that maps to
`failure.kind=timeout, classification=deadline_exceeded`, or one of the
malformed-output paths.

Once the handler reports a result, the **durable activity retry policy
remains the server/runtime authority**. Task-level retry, scheduling, and
backoff continue to follow the activity retry policy declared on the
workflow side. The carrier's `retry_policy` does not extend, override, or
substitute for it.

## Request And Response Envelope

The carrier process sends the carrier-neutral input envelope and expects
the carrier-neutral result envelope back:

| Direction | Content type | Schema |
| --- | --- | --- |
| Request | `application/vnd.durable-workflow.external-task-input+json` | `external_task_input_contract` |
| Response | `application/vnd.durable-workflow.external-task-result+json` | `external_task_result_contract` |

The handler must preserve `task.id`, `task.attempt`, and
`task.idempotency_key` from the input envelope, and must respond with the
declared success or failure envelope (or fail-closed by emitting a
malformed-output mapping).

The carrier maps transport facts to result facts deterministically:

- transport timeout → `failure.kind=timeout`,
  `classification=deadline_exceeded`
- non-2xx response without a valid result envelope → `malformed_output`
- invalid JSON or schema mismatch on the response → `malformed_output`
- result envelope referencing an unsupported payload reference →
  `unsupported_payload`

Handlers must be idempotent. The same `task.id` and
`task.idempotency_key` may arrive more than once if the carrier retries
transport delivery or the runtime redelivers an unfinished lease.

## Configuration Example

The server reads handler mappings from
`DW_EXTERNAL_EXECUTOR_CONFIG_PATH`, optionally selecting an overlay with
`DW_EXTERNAL_EXECUTOR_CONFIG_OVERLAY`. See the
[Server Config Reference](./server-config-reference.md) for the full list of
environment variables.

Below is a minimal `durable-workflow.external-executor.config` document that
registers one `invocable_http` carrier and one activity mapping:

```json
{
  "schema": "durable-workflow.external-executor.config",
  "version": 1,
  "defaults": {
    "profile": "prod",
    "namespace": "operations",
    "task_queue": "operator-tasks",
    "auth_ref": "handler-token"
  },
  "auth_refs": {
    "handler-token": {
      "type": "env",
      "env": "DURABLE_WORKFLOW_HANDLER_TOKEN"
    }
  },
  "carriers": {
    "ops-invocable": {
      "type": "invocable_http",
      "url": "https://handlers.example.com/durable/activity",
      "method": "POST",
      "timeout_seconds": 60,
      "capabilities": ["activity_task"],
      "retry_policy": {
        "max_attempts": 3,
        "backoff_seconds": [2, 5],
        "retryable_status_codes": [408, 429, "5xx"]
      }
    }
  },
  "mappings": [
    {
      "name": "billing.reconcile-ledger",
      "kind": "activity",
      "task_queue": "operator-tasks",
      "activity_type": "billing.reconcile-ledger",
      "carrier": "ops-invocable",
      "handler": "billing.reconcile-ledger",
      "timeout_seconds": 60
    }
  ]
}
```

A loopback dev variant uses `"url": "http://127.0.0.1:8080/durable/activity"`
and may omit `auth_ref` because loopback HTTP is the one allowed
unauthenticated path.

## Inspection And Diagnostics

Two CLI commands surface the published invocable carrier contract from a
running server:

- `dw server:info` renders the schema name, contract version, carrier type,
  task kinds, allowed request/response content types, and the redacted
  external executor mapping diagnostics. Use it to confirm the server has
  loaded the expected mappings.
- `dw doctor` runs the cluster-info diagnostic, prints the
  `invocable_carrier_contract` block, and surfaces any
  `invalid_carrier_target`, `missing_invocable_auth_ref`,
  `invalid_invocable_carrier_scope`, `unknown_carrier`, `unknown_auth_ref`,
  or `unknown_handler` errors that blocked a mapping from being advertised.

The activity poll response itself reports the resolved mapping, the carrier
target (with auth redacted), and the effective `retry_policy`. Workers and
operator tooling read these fields as the source of truth instead of caching
their own copy of the configuration file.

## Coexistence And Rollout

The invocable carrier rollout boundary is published in the same manifest:

- A poll-based carrier and an `invocable_http` carrier may share a queue only
  when the mappings are activity-type specific. Two mappings cannot claim the
  same `(task_queue, activity_type)` pair.
- Operators must remove or overlay-disable invocable mappings before deleting
  the credentials they reference. Pulling credentials before the mapping is
  drained will move every new attempt to `failure.kind=auth`, not silently
  succeed.
- Carrier `retry_policy` is purely transport. The durable activity retry
  policy remains the only authority over how many task attempts the workflow
  observes.

## Related Surfaces

- [External Execution Surface](./external-execution.md) — the carrier-neutral
  product boundary that this carrier implements.
- [PHP Invocable Activity Handler](./invocable-php-handler.md) — the PHP
  helper an external process uses to parse input envelopes, dispatch
  activity callables, and emit the structured result envelopes this carrier
  expects.
- [Server Config Reference](./server-config-reference.md) — environment
  variables for `DW_EXTERNAL_EXECUTOR_CONFIG_PATH` and
  `DW_EXTERNAL_EXECUTOR_CONFIG_OVERLAY`.
- [External Payload Storage](../features/external-payload-storage.md) — how
  oversized request or result payloads are offloaded to a configured driver
  and represented as a verifiable reference envelope inside the same input
  and result schemas the invocable carrier uses.
- [Worker Protocol](./worker-protocol.md) — the broader worker-plane contract
  that publishes this carrier alongside poll-based handler shapes.

<!-- Source: docs/polyglot/namespace-auth-workers.md -->

# Namespace, Auth, And Worker Registration

Use this reference when provisioning a standalone server, issuing automation
credentials, or implementing a worker runtime. The server has three separate
contracts that must line up before work can flow:

- the request names a namespace through `X-Namespace`, `?namespace=`, or the
  server default namespace
- the credential has the role required by the route family
- workers register the same namespace, task queue, runtime, type keys, and
  capacity that workflow starts and task polls use

## Request Authority

Durable Workflow treats namespace, auth role, and protocol version as request
authority. Do not infer them from workflow ids, task ids, or display labels.

| Request family | Required credential role | Required version header | Namespace source |
| --- | --- | --- | --- |
| Discovery `GET /api/cluster/info` | `worker`, `operator`, or `admin` | none | optional `X-Namespace` for context |
| Namespace list/describe | `operator` or `admin` | `X-Durable-Workflow-Control-Plane-Version: 2` | route target or request context |
| Namespace create/update/storage policy | `admin` | `X-Durable-Workflow-Control-Plane-Version: 2` | route target or request body |
| Workflow, schedule, task queue, bridge adapter, worker visibility | `operator` or `admin` | `X-Durable-Workflow-Control-Plane-Version: 2` | `X-Namespace`, `?namespace=`, then default |
| System health, metrics, passes, and storage tests | `admin` | `X-Durable-Workflow-Control-Plane-Version: 2` | `X-Namespace`, `?namespace=`, then default |
| Worker registration, polling, heartbeats, completion | `worker` | `X-Durable-Workflow-Protocol-Version: 1.0` | `X-Namespace`, `?namespace=`, then default |

The server checks the route role before namespace existence on role-gated
endpoints. A wrong-role token receives an auth failure instead of a namespace
existence signal. After the role and version checks pass, namespace-scoped
routes reject unknown namespaces with `reason: "namespace_not_found"`.

## Auth Roles

Token auth is the default production path:

```bash
DW_AUTH_DRIVER=token
DW_WORKER_TOKEN=worker-secret
DW_OPERATOR_TOKEN=operator-secret
DW_ADMIN_TOKEN=admin-secret
```

If a deployment uses one shared `DW_AUTH_TOKEN`, that token effectively has all
route permissions. Prefer role-scoped tokens for production so a worker process
cannot mutate namespaces or run system maintenance passes.

The same role split exists for signature auth:

```bash
DW_AUTH_DRIVER=signature
DW_WORKER_SIGNATURE_KEY=worker-signature-secret
DW_OPERATOR_SIGNATURE_KEY=operator-signature-secret
DW_ADMIN_SIGNATURE_KEY=admin-signature-secret
```

`DW_AUTH_DRIVER=none` is for local development only. It removes the auth
boundary from every route and should never be exposed outside a trusted local
network.

### Token rejected because of whitespace {#token-whitespace}

Native Server 3.5.0 and later refuses to start if `DW_AUTH_TOKEN`,
`DW_WORKER_TOKEN`, `DW_OPERATOR_TOKEN`, `DW_ADMIN_TOKEN` or a legacy token
setting contains leading or trailing whitespace or a control character. The
diagnostic names the setting and says `has leading or trailing whitespace or a
control character`. It does not include the token value.

A common cause is a Kubernetes Secret created from a file with a trailing
newline, or with `echo` without `-n`. Create a new Secret from the intended
whitespace-free token value without adding a newline:

```bash
printf '%s' "$DW_WORKER_TOKEN" | kubectl create secret generic durable-workflow-worker \
  --from-file=token=/dev/stdin
```

`printf` does not remove whitespace already present in the variable. Correct
the source value first if it is malformed. For an existing Secret, update it
through your deployment process, then restart affected Server and worker
processes so they load the corrected value. Both sides must use the same token
bytes.

## Signal authorization and attribution

Before admitting a signal, Server authenticates the request and enforces the
configured role and namespace policy. The workflow signal routes require an
`operator` or `admin` credential. These route permissions do not implement your
application's order-approval policy. Enforce that business decision in a
trusted application gateway before forwarding the signal, or in your supported
custom auth provider. Keep privileged Server credentials in that trusted
boundary when end users must not send signals directly.

The Server derives a command's `principal` from the authenticated credential.
A signal argument such as `approvedBy` remains application data even when it
has the same spelling as an actor subject. A gateway using one service token
records that service's principal, not necessarily the human behind its request.
Derive any human approval field from the gateway's authenticated user and
authorization decision rather than accepting it from the client.

For audit inspection, command responses and linked command/history metadata
expose the recorded principal. The persisted command stores it under
`context.principal`; this is separate from the signal's arguments and from the
value a workflow's named signal wait returns. Missing principal metadata is
not proof of a particular sender. See
[the embedded signal example](/docs/features/signals#sender-identity-and-authorization)
for a Laravel policy check and host-side command context.

Use [named principal tokens](/docs/polyglot/server-config-reference#authentication)
when audit trails need stable actor subjects. A shared role token does not
identify an individual application user, and auth-disabled development mode
provides no authenticated human identity. Forwarded caller/auth attribution
headers may add configured audit labels; they do not supply the
Server-authenticated principal. Keep forwarded attribution disabled unless
the [trusted-gateway requirements](/docs/polyglot/server-config-reference#command-attribution)
are met.

## Namespace Contract

The bootstrap process seeds the default namespace. Set
`DW_DEFAULT_NAMESPACE` when omitted namespace headers should resolve somewhere
other than `default`:

```bash
DW_DEFAULT_NAMESPACE=default
```

Create every tenant or environment namespace before directing clients or
workers at it:

```bash
curl -sS -X POST "$DURABLE_WORKFLOW_SERVER_URL/api/namespaces" \
  -H "Authorization: Bearer $DW_ADMIN_TOKEN" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "orders-prod",
    "description": "production order workflows",
    "retention_days": 90
  }'
```

Namespace names are normalized to lowercase and may contain letters, numbers,
dot, underscore, and dash. They are unique after normalization. For example,
creating `Production` and then `production` is a conflict with
`reason: "namespace_already_exists"`.

A normal workflow start names the namespace with `X-Namespace`:

```bash
curl -sS -X POST "$DURABLE_WORKFLOW_SERVER_URL/api/workflows" \
  -H "Authorization: Bearer $DW_OPERATOR_TOKEN" \
  -H "X-Namespace: orders-prod" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{
    "workflow_type": "orders.fulfillment",
    "workflow_id": "order-1001",
    "task_queue": "orders",
    "input": ["order-1001"]
  }'
```

Control-plane reads, workflow commands, schedule operations, search-attribute
operations, task-queue visibility, and worker visibility use that same
namespace context. Namespace administration routes themselves are the
exception: they target the namespace in the route or request body and do not
require that namespace to already exist before create or describe can run.

## Worker Registration Contract

Every worker process must register before polling. Registration is namespaced,
so the tuple `(namespace, worker_id)` identifies the worker record. The
registered `task_queue` must match future poll requests for that worker id.

```bash
curl -sS -X POST "$DURABLE_WORKFLOW_SERVER_URL/api/worker/register" \
  -H "Authorization: Bearer $DW_WORKER_TOKEN" \
  -H "X-Namespace: orders-prod" \
  -H "X-Durable-Workflow-Protocol-Version: 1.0" \
  -H "Content-Type: application/json" \
  -d '{
    "worker_id": "py-orders-1",
    "task_queue": "orders",
    "runtime": "python",
    "sdk_version": "<installed-sdk-version>",
    "build_id": "orders-worker-2026-04-22",
    "supported_workflow_types": ["orders.fulfillment"],
    "workflow_definition_fingerprints": {
      "orders.fulfillment": "sha256:definition-fingerprint"
    },
    "supported_activity_types": ["payments.capture"],
    "max_concurrent_workflow_tasks": 10,
    "max_concurrent_activity_tasks": 50
  }'
```

| Field | Required | Contract |
| --- | --- | --- |
| `worker_id` | no | Stable process identity. The server generates one when omitted, but long-running runtimes should set it for logs and task queue diagnostics. |
| `task_queue` | yes | Queue this worker polls. Poll requests for a different queue fail with `reason: "task_queue_mismatch"`. |
| `runtime` | yes | One of `php`, `python`, `rust`, `typescript`, `go`, `java`, or `external`. An accepted runtime identifier does not imply a first-party SDK is available. |
| `sdk_version` | no | Runtime SDK version shown in worker visibility and diagnostics. |
| `build_id` | no | Deploy/build identity used by task queue build-id visibility and rollout cohorts. It should stay stable for one replay-compatible worker family. |
| `supported_workflow_types` | no | Workflow type keys this worker can replay. Empty means no workflow-type filter. |
| `workflow_definition_fingerprints` | no | Per-workflow deterministic definition fingerprints. Active re-registration with a changed fingerprint is rejected. |
| `supported_activity_types` | no | Activity type keys this worker can execute. Empty means no activity-type filter. |
| `max_concurrent_workflow_tasks` | no | Advertised local workflow-task slots. Defaults to `100`; minimum `1`. |
| `max_concurrent_activity_tasks` | no | Advertised local activity-task slots. Defaults to `100`; minimum `1`. |

Active re-registration with the same worker id is allowed when the advertised
definition fingerprints are unchanged. If a running worker changes workflow
code for an already advertised type, it must restart with a new `worker_id`;
otherwise registration fails with `reason: "workflow_definition_changed"`.

`build_id` is not just decorative metadata. It is the operator-facing cohort
identity used by task-queue rollout APIs. Keep one `build_id` for workers that
can safely replay the same in-flight work, and change it when a rollout creates
a new compatibility family. See
[Worker Compatibility and Routing](/docs/polyglot/worker-compatibility-routing)
for the pinning and rollback contract, and
[Worker Build-Id Rollout](/docs/polyglot/worker-build-id-rollout) for the
drain/resume workflow.

## Polling And Visibility

Worker poll requests must use the same namespace, worker id, and task queue
that registration used:

```bash
curl -sS -X POST "$DURABLE_WORKFLOW_SERVER_URL/api/worker/workflow-tasks/poll" \
  -H "Authorization: Bearer $DW_WORKER_TOKEN" \
  -H "X-Namespace: orders-prod" \
  -H "X-Durable-Workflow-Protocol-Version: 1.0" \
  -H "Content-Type: application/json" \
  -d '{
    "worker_id": "py-orders-1",
    "task_queue": "orders",
    "timeout_seconds": 30
  }'
```

Operators can inspect the same registration state through control-plane
visibility:

```bash
curl -sS "$DURABLE_WORKFLOW_SERVER_URL/api/task-queues/orders" \
  -H "Authorization: Bearer $DW_OPERATOR_TOKEN" \
  -H "X-Namespace: orders-prod" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" | jq '.pollers, .admission'
```

Task queue visibility is the first place to check when workers receive no
tasks. It distinguishes missing workers from queue mismatches, unsupported
workflow/activity type filters, saturated worker slots, server active-lease
caps, dispatch-rate caps, and query-task backpressure.

Poll responses themselves expose the same machine-readable outcome through
`poll_status` when the server advertises
`worker_protocol.server_capabilities.poll_status = true` in
`GET /api/cluster/info`. Workflow-task, activity-task, and query-task poll
routes keep that field even when `task` is `null`, so worker runtimes can
branch on one stable surface:

- `leased`: a task was leased successfully.
- `empty`: no task was ready before the poll returned.
- `throttled`: queue admission limits withheld a new task for this poll.
- `unavailable`: the server could not safely coordinate the queue and returned
  a typed unavailable outcome instead of pretending the queue was empty.
- `draining`: the worker's build-id cohort is draining, so the poll fails with
  HTTP `409` and `reason: "worker_draining"` until the cohort resumes.

## Error Surface

Automation should branch on named reasons rather than prose messages.

| Reason | Where | Meaning | Operator action |
| --- | --- | --- | --- |
| `missing_control_plane_version` | control-plane route | The request omitted `X-Durable-Workflow-Control-Plane-Version: 2`. | Add the version header or upgrade the client profile. |
| `missing_protocol_version` | worker route | The request omitted `X-Durable-Workflow-Protocol-Version: 1.0`. | Fix the worker client or SDK version negotiation. |
| `namespace_not_found` | namespace-scoped route | The namespace from `X-Namespace`, query string, or server default does not exist. | Create the namespace or correct the client namespace. |
| `namespace_already_exists` | `POST /api/namespaces` | A normalized namespace name already exists. | Reuse it or choose a distinct name. |
| `task_queue_mismatch` | worker poll route | A worker id registered for one queue attempted to poll another. | Restart with a new worker id or poll the registered queue. |
| `worker_draining` | worker poll route | The worker's build-id cohort is marked draining, so it may finish in-flight work but cannot claim new tasks. | Resume the cohort for rollback, or stop the worker after its current leases drain. |
| `workflow_definition_changed` | `POST /api/worker/register` | Active worker id tried to advertise changed workflow fingerprints. | Restart the changed process with a new worker id. |
| `validation_failed` | any JSON route | A field is missing, malformed, too large, or outside allowed bounds. | Read `errors` or `validation_errors` and correct the payload. |

## See Also

- [Server API Reference](/docs/polyglot/server-api-reference)
- [Worker Protocol](/docs/polyglot/worker-protocol)
- [Task Queue Admission](/docs/polyglot/task-queue-admission)
- [CLI Command Reference](/docs/polyglot/cli-reference)

<!-- Source: docs/polyglot/python.md -->

# Python SDK

The Python SDK is a thin, async-first client for a self-hosted Durable Workflow
Server or Durable Workflow Cloud namespace runtime. It lets Python processes
start, observe, signal, and cancel workflows through the runtime's
control-plane API, and register as workers that execute workflow tasks and
activities.

## Try the local Sample App playground

For the shortest no-Cloud authoring journey, open the current Sample App
[`main` branch in GitHub Codespaces](https://codespaces.new/durable-workflow/sample-app?quickstart=1&ref=main)
and run:

<!-- docs-example id="sdk.python.sample-app-playground" -->
```bash
scripts/playground python
```

The local playground generates caller-owned workflow and activity source,
selects the current stable artifacts, and starts the published
Server and Waterline. It waits for a worker registration whose identity,
workflow type, activity type, and task queue match the generated contract before
starting the workflow. Success requires the expected completed result and
history; the terminal then prints the exact local Waterline run link and the
path to structured JSON evidence.

The package installation, inline quickstart, API reference, and repository
example below remain the direct paths for users who do not want Sample App.

The SDK targets the same durable model as the PHP package — instance IDs, run IDs, history events, task queues, and type keys are shared across languages. A Python worker can serve activities for a PHP-authored workflow, and vice versa.

For a capability comparison across the Python SDK, PHP SDK, Rust SDK, and
`dw`, see
[Client and Worker Capabilities](/docs/polyglot/cli-python-parity/).

For constructor signatures, return types, exception classes, and metric names,
see the generated [Python SDK API reference](https://python.durable-workflow.com/).

Python SDK 2.5.0 adds an optional bounded history cache for sticky execution.
Use Server 2.5.10 or newer and enable it on each worker. See
[Portable Worker Affinity](/docs/polyglot/portable-worker-affinity) and the
[Python guide and runnable example](https://python.durable-workflow.com/guides/sticky-execution/).

Cloud customers use the provisioned namespace's runtime URL and namespace with
separate client and worker credentials. See
[Cloud Managed Runtime](/docs/polyglot/cloud-control-plane) for that
connection boundary; the quickstart below uses local self-hosted values.

## Requirements

- Python 3.10 or later
- Docker (for the local Server used in this quickstart), an existing
  [self-hosted Server](/docs/polyglot/server), or a provisioned
  [Cloud namespace runtime](/docs/polyglot/cloud-control-plane)

## Installation

Install the <PythonPackageReleaseLink authority="qualified">stable Python SDK
release</PythonPackageReleaseLink>. The exact requirement is generated
from the same tuple as the Server quickstart, and a lock or constraints file can
retain it for reproducibility.

<!-- docs-example id="python.sdk.install" -->
```bash
pip install durable-workflow==2.17.2
```

The SDK depends on [httpx](https://www.python-httpx.org/) for HTTP and on the
`fastavro` Python package to encode the Apache Avro wire format used by the
only public v2 payload codec. Prometheus metrics support is optional.

## Upgrading workers {#upgrading-workers}

Python SDK 2.14.1 fixes `wait_condition()` fingerprints that changed across
Python versions or worker restarts. Upgrade every worker sharing a task queue
together. Once a run records a v2 wait fingerprint, keep all workers that can
replay it on 2.14.1 or later. Do not roll the workers on that queue back to an
SDK version below 2.14.1.

Legacy v1 fingerprints that no longer match are accepted as unverifiable, so
that compatibility path cannot detect predicate changes. `wait_condition()`
lambdas on the same line that contain `list`, `dict` or `set` comprehensions
share one combined fingerprint. Put those predicates on separate lines in new
workflow definitions to detect switching between them. See the
[tagged 2.14.1 compatibility notes](https://github.com/durable-workflow/sdk-python/blob/2.14.1/CHANGELOG.md#2141---2026-11-04).

Python SDK 2.15 and 2.16 add:

- **Sticky history suffixes (2.16).** Workers with a sticky cache
  (`sticky_cache_capacity > 0`) advertise `sticky_history_suffix`. On Server
  3.5.0 or later, a sticky hit receives only the history after the sequence the
  worker cached. A suffix that doesn't continue the cached history exactly is
  dropped and the full history is fetched.
- **Typed workflow values (2.15).** `@workflow.defn(..., typed=True)` decodes
  `run` and handler arguments to their annotations, and `result_type=` decodes
  child and client results. Untyped workflows receive plain values as before,
  so in-flight histories replay unchanged.
- **Lazy pages (2.15).** `Client.iter_workflows`, `Client.iter_history` and
  `WorkflowHandle.iter_history` follow page tokens while holding one page.
  See [workflow list and history pages](#workflow-list-and-history-pages)
  for traversal, resume and error handling.
- **Run activities (2.15).** `Client.get_run_activities` and
  `WorkflowHandle.get_activities` return typed activity and attempt metadata,
  and `ActivityInfo.activity_execution_id` exposes the logical activity
  identity.
- **Local activity backoff (2.15.1).** Retries past the end of the backoff list
  repeat the final entry instead of retrying immediately.

See the [2.16.0 changelog](https://github.com/durable-workflow/sdk-python/blob/2.16.0/CHANGELOG.md)
for the complete list.

Python SDK 2.17 adds `Client.update_outcome` and
`Client.update_outcome_receipt` to read an accepted update later without
submitting it again. These methods require
`capabilities.update_outcome_reads` (native Server 3.6.0 or later).
`update_outcome` waits for completion by default and raises `UpdateStillPending`
if the update is unfinished. A returned `None` is a completed null result.
`update_outcome_receipt` defaults to the accepted stage and preserves the
explicit `update_status`. See the
[Python SDK update-outcome guide](https://python.durable-workflow.com/reference/client/#read-an-update-outcome)
for language-specific examples.

## Quickstart

Here is a complete Python program that defines a workflow with one activity, starts it against a local Durable Workflow server, and waits for the result:

<!-- docs-example id="python.quickstart.worker" -->
```python
import asyncio
import uuid

from durable_workflow import Client, Worker, workflow, activity

@activity.defn(name="greet")
async def greet(name: str) -> dict:
    return {"greeting": f"Hello, {name}!", "length": len(name)}

@workflow.defn(name="greeter")
class GreeterWorkflow:
    def run(self, ctx, *args):
        result = yield ctx.schedule_activity("greet", list(args))
        return result

async def main():
    workflow_id = f"greeting-{uuid.uuid4().hex}"

    async with Client("http://localhost:8080", token="dev-token", namespace="default") as client:
        handle = await client.start_workflow(
            workflow_type="greeter",
            task_queue="default",
            workflow_id=workflow_id,
            input=["world"],
        )

        worker = Worker(
            client,
            task_queue="default",
            workflows=[GreeterWorkflow],
            activities=[greet],
        )

        await worker.run_until(workflow_id=workflow_id, timeout=30.0)
        result = await handle.result(timeout=10.0)

    print(result)  # {"greeting": "Hello, world!", "length": 5}

asyncio.run(main())
```

### Running against a local server

The program above assumes a Durable Workflow server reachable at
`http://localhost:8080` with the local `dev-token`. If you don't have one yet,
the fastest source-free path is the published server image:

```bash
export DW_SERVER_IMAGE=durableworkflow/server:3.10.0
export DW_AUTH_TOKEN=dev-token

docker volume create durable-workflow-python-quickstart

docker rm -f durable-workflow-server >/dev/null 2>&1 || true
docker run -d --name durable-workflow-server \
  -p 127.0.0.1:8080:8080 \
  -v durable-workflow-python-quickstart:/app/database \
  -e DW_AUTH_TOKEN="$DW_AUTH_TOKEN" \
  "$DW_SERVER_IMAGE"

until curl -sf http://localhost:8080/api/ready > /dev/null; do sleep 1; done
```

The native Server creates its SQLite schema and the `default` namespace on
first start, so there is no separate bootstrap step. For production
deployment — authentication, database config, TLS — see the [server setup guide](/docs/polyglot/server).

For a larger example, the SDK repository includes [`examples/order_processing`](https://github.com/durable-workflow/sdk-python/tree/main/examples/order_processing), a Docker Compose stack that runs a Python worker through an order workflow end to end.

## Client API Reference

The async `durable_workflow.Client` is the public entry point for control-plane
and worker-plane HTTP calls. Use it as an async context manager so the
underlying `httpx.AsyncClient` connection pool closes cleanly.

```python
from durable_workflow import Client

async with Client(
    "https://workflow.example.com",
    token="shared-token",
    namespace="default",
    timeout=60.0,
) as client:
    info = await client.get_cluster_info()
```

### Constructor

| Argument | Type | Default | Use when |
| --- | --- | --- | --- |
| `base_url` | `str` | required | Server origin, without `/api`. |
| `token` | `str \| None` | `None` | One bearer token should authorize both control-plane and worker-plane calls. |
| `control_token` | `str \| None` | `None` | Control-plane calls need a different bearer token than worker polling. |
| `worker_token` | `str \| None` | `None` | Worker-plane calls need a different bearer token than operator calls. |
| `namespace` | `str` | `"default"` | Target a server namespace through `X-Namespace`. |
| `timeout` | `float` | `60.0` | Override the default HTTP timeout. |
| `retry_policy` | `TransportRetryPolicy \| None` | default policy | Tune transport retries for transient HTTP failures. |
| `metrics` | `MetricsRecorder \| None` | no-op | Emit client and worker metrics to a custom recorder. |
| `payload_size_limit_bytes` | `int` | SDK default | Match the server's max payload-byte contract. |
| `payload_size_warning_threshold_percent` | `int` | SDK default | Warn before a payload reaches the configured limit. |
| `payload_size_warnings` | `bool` | `True` | Disable local payload-size warnings in tests or controlled scripts. |
| `failure_report_limit_bytes` | `int` | `1048576` | Lower the complete encoded failure-report budget. See [bounded failure reports](#bounded-failure-reports). |

`token` is the simplest option. If both `control_token` and `worker_token` are
set, control-plane methods use `control_token` and worker-plane methods use
`worker_token`.

### Bounded failure reports {#bounded-failure-reports}

Python SDK **2.11.0** introduced bounded outgoing failure reports. Defaults are **62 KiB UTF-8 for the message**, **128 KiB for the stack**, **64 KiB for combined serialized optional details and diagnostics**, and **1 MiB for the complete HTTPX-encoded body**. Markers, JSON escaping, claim identity and other commands in the batch count toward that body limit. A lower configured or discovered transport limit takes precedence. Failure type, class, code, retry policy and claim identity are preserved.

Set a smaller body budget when needed. Payload-size warning settings remain advisory.

```python
from durable_workflow import Client

async with Client(
    "https://workflow.example.com",
    worker_token="worker-token",
    control_token="operator-token",
    failure_report_limit_bytes=512 * 1024,
) as client:
    info = await client.get_cluster_info()
```

A visible truncation marker and original byte counts identify shortened fields. Avro details remain intact or are omitted whole. An unrepresentable required envelope raises `FailureReportTooLarge` before HTTP submission. Local failures are normalized before workflow code observes them, so first execution and cold replay see the persisted message. An already observed local message remains immutable during final batch fitting.

### External Payload Storage

Normal `Client` and `Worker` operations use Server's authenticated runtime
transport automatically. The SDK discovers namespace policy and transport
limits through `GET /api/cluster/info`, uploads encoded values above the
threshold or needed to fit the request, and fetches referenced values before
decode. Application workflow and activity code receives the decoded value.
Provider credentials stay on Server. Remote workers do not need boto3, provider
clients or access to Server's local storage directory.

The runtime reference has schema
`durable-workflow.v2.runtime-external-payload-reference.v1` and an opaque
`reference_id`. It has no provider `uri`. Upload/fetch uses the normal role
credential and namespace, verifies encoded-byte size and SHA-256, and preserves
task-attempt and lease fencing for worker completion. See the
[runtime HTTP contract](./server-api-reference.md#runtime-external-payload-transport)
for routes, limits and typed refusals. SDK support depends on an available
compatible transport, not merely on a configured storage driver.

The following **direct provider helpers** serve application-managed storage
and test fixtures that deliberately access their own provider. They use a
different provider-URI schema. They are not required for normal Server or Cloud
clients, and a provider `external_storage` envelope is rejected by Server with
HTTP `422` and `external_payload_unsupported`.

| API | Role | Failure surface |
| --- | --- | --- |
| `ExternalStorageDriver` | Protocol for `put(data, sha256=..., codec=...)`, `get(uri)`, and `delete(uri)`. | Driver-raised storage errors. |
| `LocalFilesystemExternalStorage` | Dependency-free `file://` driver for local development and tests. | `ValueError` when a referenced URI escapes the configured root. |
| `S3ExternalStorage` | Adapter for a boto3-compatible client. | `ValueError` for foreign bucket/prefix references or non-byte responses. |
| `GCSExternalStorage` | Adapter for a google-cloud-storage-style client. | `ValueError` for foreign bucket/prefix references or non-byte responses. |
| `AzureBlobExternalStorage` | Adapter for an Azure container client. | `ValueError` for foreign container/prefix references or non-byte responses. |
| `ExternalPayloadReference` | Immutable wire reference with `uri`, `sha256`, `size_bytes`, `codec`, and schema. | `ValueError` when `from_dict()` receives an unsupported schema or malformed fields. |
| `ExternalPayloadCache` | Bounded replay cache for already verified external payload bytes. | Constructor rejects non-positive entry or byte limits. |
| `store_external_payload()` | Stores encoded bytes through a driver and returns `ExternalPayloadReference`. | Driver errors. |
| `fetch_external_payload()` | Fetches referenced bytes, then verifies size and SHA-256 before decode. | `ExternalPayloadIntegrityError` on size/hash mismatch. |
| `delete_external_payload()` | Deletes the referenced object and evicts any cache entry. | Driver-raised storage errors. |
| `external_storage_envelope()` | Encodes a value inline until the threshold is crossed, then writes bytes through a driver. | `ValueError` when the threshold is invalid or no driver can resolve a reference. |
| `external_storage_driver_from_policy()` | Builds a direct driver from an explicitly supplied policy and application-owned provider clients. It does not authorize direct provider access on Server or Cloud. | `ValueError` when the policy is disabled, unsupported, or missing the required client/bucket/container. |

```python
from durable_workflow import (
    ExternalPayloadCache,
    LocalFilesystemExternalStorage,
    external_storage_envelope,
    to_avro_payload_value,
)
from durable_workflow.serializer import encode
from durable_workflow.external_storage import fetch_external_payload, store_external_payload

storage = LocalFilesystemExternalStorage("/var/lib/durable-workflow/payloads")
cache = ExternalPayloadCache(max_entries=256, max_bytes=32 * 1024 * 1024)

payload = to_avro_payload_value({"invoice_pdf": "x" * 1_000_000})
envelope = external_storage_envelope(
    payload,
    external_storage=storage,
    threshold_bytes=64 * 1024,
)

reference = store_external_payload(
    storage,
    encode({"archived": True}).encode("utf-8"),
    codec="avro",
)
payload_bytes = fetch_external_payload(storage, reference, cache=cache)
```

For these direct helpers, the reference schema is `EXTERNAL_PAYLOAD_REFERENCE_SCHEMA`
(`durable-workflow.v2.external-payload-reference.v1`). Object-store adapters do
not add cloud SDK dependencies to `durable-workflow`; applications pass their
already configured S3, GCS, or Azure clients.

Their provider reference has the following fields. This is not the runtime
reference accepted by Server HTTP routes:

```json
{
  "schema": "durable-workflow.v2.external-payload-reference.v1",
  "uri": "s3://dw-payloads/billing/run-001/input.avro",
  "sha256": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
  "size_bytes": 1048576,
  "codec": "avro",
  "expires_at": "2026-05-22T00:00:00Z"
}
```

`ExternalPayloadReference.from_dict()` rejects unknown schemas and malformed
fields. `fetch_external_payload()` verifies `size_bytes` and `sha256` before
returning bytes, and raises `ExternalPayloadIntegrityError` on size or hash
mismatch. Missing objects can also raise the provider driver's error. Use
`ExternalPayloadCache` for verified bytes only. Cache eviction does not grant
permission to delete a Server-managed object. `delete_external_payload()` is a
direct provider operation, not a runtime-reference cleanup API.

Use `external_storage_driver_from_policy()` in code that deliberately accesses
application-owned provider storage and owns the required client. Reading a
namespace policy does not make provider envelopes valid Server requests.
Provider clients remain application-owned; these optional adapters do not add
boto3, google-cloud-storage or azure-storage-blob as runtime dependencies:

```python
from durable_workflow.external_storage import external_storage_driver_from_policy

namespace = await client.describe_namespace("billing")
driver = external_storage_driver_from_policy(
    namespace.external_payload_storage,
    s3_client=application_owned_s3_client,
)
```

The factory raises `ValueError` when the policy is disabled, when the named
driver is unsupported, or when the matching provider client is missing.
Disabled policies do not return a no-op driver; callers should branch on
`policy.enabled` before asking for a driver.

### Namespaces

Namespaces are the tenancy boundary for workflows, schedules, search
attributes, and external payload storage. The Client targets one namespace at a
time through the `namespace=` constructor argument, but the operator surface
below applies to any namespace the bearer token is authorized for.

| Method | Returns | Failure surface |
| --- | --- | --- |
| `await client.list_namespaces()` | `NamespaceList` | Auth/server errors. |
| `await client.describe_namespace(name)` | `NamespaceDescription` | `NamespaceNotFound`, auth/server errors. |
| `await client.create_namespace(name, description=None, retention_days=30)` | `NamespaceDescription` | `InvalidArgument` for duplicate names or invalid retention, auth/server errors. |
| `await client.update_namespace(name, description=None, retention_days=None)` | `NamespaceDescription` | `NamespaceNotFound`, `InvalidArgument`, auth/server errors. Only provided fields are sent. |
| `await client.set_namespace_external_storage(name, driver=..., enabled=True, threshold_bytes=None, config=None)` | `NamespaceDescription` | `InvalidArgument` when the policy fails server validation, auth/server errors. |
| `await client.test_external_storage(driver=None, small_payload_bytes=None, large_payload_bytes=None)` | `StorageTestResult` | `InvalidArgument` when the bound policy is missing required fields, auth/server errors. |

`set_namespace_external_storage` mirrors `dw namespace:set-storage-driver`. The
first positional argument is the namespace `name`, matching `describe_namespace`,
`create_namespace`, and `update_namespace`. The deprecated `namespace=` alias
emits a `DeprecationWarning`; new code should use the positional `name`. The
`config` dict carries driver-specific keys including the optional `disk` field
on `s3`, `gcs`, and `azure` so credentials stay on the server. The returned
`NamespaceDescription` reflects the policy the server actually persisted —
including the threshold the server fell back to when no caller value was
provided.

`test_external_storage` mirrors `dw storage:test`. The server round-trips a
small and large payload through the bound external storage driver and returns
`StorageTestResult`, which exposes per-payload `StoragePayloadTestResult`
records with `wrote`, `read`, `verified`, `latency_ms`, and `bytes` fields.

`NamespaceDescription` carries the namespace's `external_payload_storage`
policy as documented in [External Payload Storage](#external-payload-storage).
Normal clients use the discovered runtime transport automatically. Direct
provider helpers require application-owned provider access; a policy description
does not make provider envelopes valid Server requests.

### Cluster and Task Queues

| Method | Returns | Notes |
| --- | --- | --- |
| `await client.health()` | `dict[str, Any]` | Calls the health endpoint for liveness checks. |
| `await client.get_cluster_info()` | `dict[str, Any]` | Reads server version, protocol, capability, and compatibility metadata. |
| `await client.list_task_queues()` | `TaskQueueList` | Lists task queues visible in the namespace. |
| `await client.describe_task_queue(name)` | `TaskQueueDescription` | Returns worker capacity, current leases, query admission, and dispatch-budget facts. |
| `await client.list_task_queue_build_ids(task_queue)` | `TaskQueueBuildIdRollout` | Snapshots the per-build-id cohort state for a queue, including unversioned workers under a cohort whose `build_id` is `None`. |
| `await client.drain_task_queue_build_id(task_queue, build_id)` | `TaskQueueBuildIdRolloutState` | Marks a build-id cohort as draining so it stops claiming new tasks. Pass `build_id=None` to drain unversioned workers. Idempotent. |
| `await client.resume_task_queue_build_id(task_queue, build_id)` | `TaskQueueBuildIdRolloutState` | Reverts a previous drain so the cohort can claim work again. Pass `build_id=None` to resume unversioned workers. Idempotent. |

Task queue return types expose nested `TaskQueueAdmission`,
`TaskQueueTaskAdmission`, `TaskQueueQueryAdmission`, `TaskQueueBuildIdCohort`,
`TaskQueueBuildIdRollout`, and `TaskQueueBuildIdRolloutState` dataclasses so
scripts can check server-side capacity and build-id rollout without parsing
prose output. See
[Worker Build-Id Rollout](/docs/polyglot/worker-build-id-rollout) for the
end-to-end rollout walkthrough; the CLI mirrors of these methods are
`dw task-queue:build-ids`, `dw task-queue:drain`, and `dw task-queue:resume`.

### Workflow Operations

| Method | Returns | Failure surface |
| --- | --- | --- |
| `await client.start_workflow(...)` | `WorkflowHandle` | `WorkflowAlreadyStarted`, `InvalidArgument`, `Unauthorized`, `ServerError` |
| `await client.describe_workflow(workflow_id)` | `WorkflowExecution` | `WorkflowNotFound`, auth/server errors |
| `await client.list_workflows(...)` | `WorkflowList` (one page) | Auth/server errors |
| `client.iter_workflows(...)` | `AsyncIterator[WorkflowExecution]` | Page errors and cancellation propagate during iteration. See [pagination](#workflow-list-and-history-pages). |
| `await client.list_workflow_runs(workflow_id)` | `WorkflowRunList` | `WorkflowNotFound`, auth/server errors |
| `await client.describe_workflow_run(workflow_id, run_id)` | `WorkflowRun` | `WorkflowNotFound`, auth/server errors |
| `await client.get_history(workflow_id, run_id, ...)` | one page of decoded history | `WorkflowNotFound`, auth/server errors |
| `client.iter_history(workflow_id, run_id, ...)` | `AsyncIterator[dict[str, Any]]` | Page errors and cancellation propagate during iteration. |
| `handle.iter_history(...)` | `AsyncIterator[dict[str, Any]]` | Same history traversal. Requires a handle with an explicit `run_id`. |
| `await client.export_history(workflow_id, run_id)` | decoded archival history payload | `WorkflowNotFound`, auth/server errors |
| `await client.signal_workflow(workflow_id, signal_name, args=None)` | admission receipt (`dict`) | `WorkflowNotFound`, `InvalidArgument`, auth/server errors |
| `await client.query_workflow(workflow_id, query_name, args=None)` | query receipt (`dict`) | `QueryFailed`, `WorkflowNotFound`, auth/server errors |
| `await client.update_workflow(workflow_id, update_name, args=None, ...)` | update receipt (`dict`) | `UpdateRejected`, `UpdateValidationFailed`, `UpdateFailed`, `WorkflowNotFound`, auth/server errors |
| `await client.cancel_workflow(workflow_id, reason=None)` | `None` | `WorkflowNotFound`, auth/server errors |
| `await client.terminate_workflow(workflow_id, reason=None)` | `None` | `WorkflowNotFound`, auth/server errors |
| `await client.repair_workflow(workflow_id)` | `WorkflowCommandResult` | `WorkflowNotFound`, auth/server errors |
| `await client.archive_workflow(workflow_id, reason=None)` | `WorkflowCommandResult` | `WorkflowNotFound`, auth/server errors |
| `await client.get_result(handle, poll_interval=0.5, timeout=30.0)` | decoded workflow output | `WorkflowFailed`, `WorkflowCancelled`, `WorkflowTerminated`, `TimeoutError` |

`signal_workflow` returns the admission receipt, including its `request_id`.
`query_workflow` and `WorkflowHandle.query` return a query receipt; read
`receipt["result"]` for the decoded handler value, including when the handler successfully
returns `None`. See the [Python query receipt guide](https://python.durable-workflow.com/reference/client/#read-a-query-result).

Python SDK 2.17.2 fixes valid Avro query results containing bytes. The
`result_envelope` stays authoritative; only an optional plain JSON projection
that cannot be encoded is omitted. [Published 2.17.2 / native Server 3.9.1
installed controls](https://github.com/durable-workflow/sdk-python/pull/318#issuecomment-6857958320)
(linux/amd64, SQLite) verify nested-byte and `None` query receipts, cold Worker
replacement and pending-to-completed-null update reads. Query return shapes
remain as described above. This patch does not advertise timeout-cleanup or
parallel-policy capabilities. The earlier 2.17.1 completion-size evidence keeps
its exact version and experiment scope.

`update_workflow` defaults to `wait_for="accepted"`. Its receipt can have
`update_status="accepted"` and `result=None`. With `wait_for="completed"`,
`wait_timeout_seconds` bounds the wait. On expiry, the receipt stays accepted
and sets `wait_timed_out=True`. Only `update_status="completed"` identifies a
finished result, including `None`. Keep `update_id` for
[later outcome reads](https://python.durable-workflow.com/reference/client/#read-an-update-outcome).

`start_workflow` accepts `workflow_type`, `task_queue`, optional
`workflow_id`, optional `input`, `duplicate_policy`, `memo`,
`search_attributes`, `business_key`, `execution_timeout_seconds`, and
`run_timeout_seconds`. All caller payloads are Avro-enveloped before they cross
the HTTP boundary.

Use `client.get_workflow_handle(workflow_id, run_id=None, workflow_type="")`
when a script already knows the workflow id and wants handle-style methods.

| `WorkflowHandle` method | Equivalent client method |
| --- | --- |
| `await handle.result(...)` | `client.get_result(handle, ...)` |
| `await handle.describe()` | `client.describe_workflow(handle.workflow_id)` |
| `await handle.signal(name, args=None)` | `client.signal_workflow(...)` |
| `await handle.query(name, args=None)` | `client.query_workflow(...)` |
| `await handle.update(name, args=None, ...)` | `client.update_workflow(...)` |
| `await handle.cancel(reason=None)` | `client.cancel_workflow(...)` |
| `await handle.terminate(reason=None)` | `client.terminate_workflow(...)` |

Both operations close the Server run immediately with distinct terminal
outcomes. `cancel_workflow` does not deliver a cooperative request to workflow
code or run saga/finally cleanup. Use `request_cancellation()` with cooperating
workers for bounded workflow cleanup. See the
[cooperative cancellation guide](/docs/polyglot/cancellation) for operation
policies, supervision and recovery.

### Current workflow identity {#workflow-identity}

Python SDK 2.9.0 and later exposes read-only `ctx.workflow_id` and `ctx.run_id`
inside workflow code. Both are strings from the current task and stay identical
during replay. After continue-as-new, the workflow ID stays unchanged and the
successor gets its own run ID. Do not replace that ID with a predecessor ID
passed as an argument.

```python
from durable_workflow import workflow

@workflow.defn(name="identity")
class IdentityWorkflow:
    def run(self, ctx):
        identity = {"workflow_id": ctx.workflow_id, "run_id": ctx.run_id}
        yield ctx.schedule_activity("record_identity", [identity])
        return identity
```

These properties read local task identity without an HTTP request. For status
polling outside workflow code, use `await client.describe_workflow(workflow_id)`.
Use `await client.query_workflow(workflow_id, query_name)` for state exposed by a
registered query handler. Identity properties do not replace either operation.

### Schedules

Schedules use `ScheduleSpec` for calendar/interval rules and `ScheduleAction`
for the workflow start request issued when a schedule fires.

| Method | Returns | Notes |
| --- | --- | --- |
| `await client.create_schedule(...)` | `ScheduleHandle` | Creates a schedule and returns a handle. |
| `await client.list_schedules()` | `ScheduleList` | Lists visible schedules. |
| `await client.describe_schedule(schedule_id)` | `ScheduleDescription` | Reads schedule status, action, next fire, and counters. |
| `await client.update_schedule(schedule_id, ...)` | `None` | Updates spec, action, overlap policy, jitter, memo, search attributes, or note. |
| `await client.pause_schedule(schedule_id, note=None)` | `None` | Pauses future fires. |
| `await client.resume_schedule(schedule_id, note=None)` | `None` | Resumes a paused schedule. |
| `await client.trigger_schedule(schedule_id, overlap_policy=None)` | `ScheduleTriggerResult` | Requests an immediate fire. |
| `await client.backfill_schedule(schedule_id, start_time=..., end_time=..., overlap_policy=None)` | `ScheduleBackfillResult` | Replays missed fire windows. |
| `await client.get_schedule_history(schedule_id, *, limit=None, after_sequence=None)` | `ScheduleHistoryPage` | Returns one page of the schedule's audit history stream, ordered by `sequence` ascending. `limit` is clamped server-side between 1 and 500 (default 100); `after_sequence` is a non-negative cursor obtained from the previous page's `next_cursor`. Raises `ScheduleNotFound` if the schedule id is unknown. History survives a `delete_schedule` call so operators can audit why a schedule was removed. |
| `client.iter_schedule_history(schedule_id, *, limit=None, after_sequence=None)` | `AsyncIterator[ScheduleHistoryEvent]` | Yields every audit event for the schedule, paging under the hood until the server reports `has_more=False`. Same failure surface as `get_schedule_history`. |
| `await client.delete_schedule(schedule_id)` | `None` | Deletes the schedule. |

`client.get_schedule_handle(schedule_id)` returns a `ScheduleHandle` with
`describe`, `update`, `pause`, `resume`, `trigger`, `backfill`, `history`,
`iter_history`, and `delete` methods that forward to the corresponding client
methods. `ScheduleHandle.history(...)` returns a `ScheduleHistoryPage` and
`ScheduleHandle.iter_history(...)` returns an `AsyncIterator[ScheduleHistoryEvent]`.

A `ScheduleHistoryPage` carries the ordered `events` list, a `has_more` flag,
a `next_cursor` integer (or `None` on the final page), the `schedule_id`, and
the owning `namespace`. Each `ScheduleHistoryEvent` carries `sequence`,
`event_type` (`ScheduleCreated`, `SchedulePaused`, `ScheduleResumed`,
`ScheduleUpdated`, `ScheduleTriggered`, `ScheduleTriggerSkipped`, or
`ScheduleDeleted`), `recorded_at`, optional `workflow_instance_id` and
`workflow_run_id` for fired workflows, and the raw `payload` dictionary the
control plane recorded for the transition.

### Bridge Events and Worker-Plane Methods

`await client.send_webhook_bridge_event(adapter, action=..., target=..., input=..., idempotency_key=..., correlation=None)`
returns `BridgeAdapterOutcome`, the same machine-readable outcome contract used
by `dw bridge:webhook`. It is the Python entry point for bounded ingress from
webhook-shaped systems.

The low-level worker-plane methods are public for custom workers and protocol
tests, but normal applications should use `Worker`:

| Method group | Methods |
| --- | --- |
| Worker registration | `register_worker` |
| Workflow tasks | `poll_workflow_task`, `complete_workflow_task`, `fail_workflow_task`, `workflow_task_history` |
| Query tasks | `poll_query_task`, `complete_query_task`, `fail_query_task` |
| Activity tasks | `poll_activity_task`, `complete_activity_task`, `fail_activity_task`, `heartbeat_activity_task` |

These methods send `X-Durable-Workflow-Protocol-Version` and use `worker_token`
when one is configured. Prefer the higher-level `Worker` unless you are writing
an SDK adapter or protocol conformance test.

`workflow_task_history(...)` pages replay history for one already leased
workflow task. Call it only after `poll_workflow_task(...)` returns
`next_history_page_token`:

```python
page = await client.workflow_task_history(
    task_id="workflow-task-01",
    next_history_page_token="history-page-2",
    lease_owner="python-worker-1",
    workflow_task_attempt=2,
)
```

The request body uses `next_history_page_token`, `lease_owner`, and
`workflow_task_attempt`. The decoded response uses the worker-protocol field
names `history_events`, `total_history_events`, and
`next_history_page_token`; it does not use the control-plane run-history names
`events` or `next_page_token`.

### Workers

The worker registry exposes which workers the server has seen recently and what
each worker can run. Use these methods to drive build-id rollouts, find stale
workers to deregister, and reconcile fleet capacity from operator scripts.

| Method | Returns | Failure surface |
| --- | --- | --- |
| `await client.list_workers(task_queue=None, status=None)` | `WorkerList` | Auth/server errors. |
| `await client.describe_worker(worker_id)` | `WorkerDescription` | `WorkerNotFound`, auth/server errors. |
| `await client.deregister_worker(worker_id)` | `dict[str, Any]` | `WorkerNotFound`, auth/server errors. |

`list_workers` filters server-side: pass `task_queue` to scope to one queue and
`status` to a single status string the server recognizes. The default returns
every registered worker for the namespace.

`WorkerDescription` carries the worker's runtime, SDK version, build id,
declared workflow and activity types, last heartbeat, and current task
admission so a script can decide which cohort to drain or which workers to
remove from the roster.

`deregister_worker` is idempotent on the server side; it removes a worker that
no longer heartbeats so capacity accounting and `list_workers` stay clean.
It does not interrupt in-flight leases.

### Search Attributes

Search attributes are typed namespace metadata that appear on workflow
executions for filtering and indexing. The Python client mirrors the same
control-plane surface as the CLI.

| Method | Returns | Failure surface |
| --- | --- | --- |
| `await client.list_search_attributes()` | `SearchAttributeList` | Auth/server errors. |
| `await client.create_search_attribute(name, attribute_type)` | `dict[str, Any]` | `InvalidArgument` for duplicate names or unsupported types, auth/server errors. |
| `await client.delete_search_attribute(name)` | `dict[str, Any]` | `SearchAttributeNotFound`, `InvalidArgument` when the attribute is system-defined, auth/server errors. |

`SearchAttributeList` separates `system` and `custom` attribute definitions so
operator scripts can verify that the engine-defined keys (workflow id, status,
type, start time, etc.) are present before refusing to create a clashing custom
key. Supported `attribute_type` values are `keyword`, `text`, `int`, `double`,
`bool`, `datetime`, and `keyword_list`.

### System Maintenance

Operator scripts and on-call automation drive the same maintenance loops as the
`dw system:*` commands through the Client. Each method requires the bearer
token to carry admin scope; without it the server returns `Unauthorized`.

| Method | Returns | CLI mirror |
| --- | --- | --- |
| `await client.repair_status()` | `dict[str, Any]` | `dw system:repair-status` |
| `await client.repair_pass(run_ids=None, instance_id=None)` | `dict[str, Any]` | `dw system:repair-pass` |
| `await client.retention_status()` | `dict[str, Any]` | `dw system:retention-status` |
| `await client.retention_pass(run_ids=None, limit=None)` | `dict[str, Any]` | `dw system:retention-pass` |
| `await client.activity_timeout_status()` | `dict[str, Any]` | `dw system:activity-timeout-status` |
| `await client.activity_timeout_pass(execution_ids=None, limit=None)` | `dict[str, Any]` | `dw system:activity-timeout-pass` |

`repair_pass` runs one task-repair sweep. With no filters the server runs a
full-scope pass over the namespace; pass `run_ids` to narrow the sweep to a
specific list of workflow runs, or `instance_id` to bound it to a single
running instance.

With no `run_ids`, `retention_pass` asks the PHP Server to prune expired
terminal runs under the current namespace policy, up to its scan limit.
Explicit `run_ids` directly target closed runs and bypass that age scan;
they can prune history younger than the bounded retention window. Forever
retention and archived-run protections still apply. The companion
`retention_status()` reports the current policy, cutoff and bounded eligible
list without deleting history. Review the [retention procedure](./server.md#retention)
before shortening a policy or submitting explicit targets.

`activity_timeout_pass` enforces start-to-close and schedule-to-close deadlines
on activity executions that have already passed their deadline. With no
filters the server processes any expired activity executions up to its scan
limit; pass `execution_ids` to target a specific list, or `limit` to bound a
single pass.

These methods do not raise on empty work — repeated calls during a quiet
period return the same `passes` / `repaired` / `pruned` counters with zero
deltas, so they are safe to call from cron-driven operator scripts.

## Defining Workflows

Workflows are Python classes decorated with `@workflow.defn`. The `run` method is an ordinary synchronous function or a synchronous generator that yields durable commands to the server. Use the generator form when the workflow needs durable work; `async def run` is not supported.

```python
from durable_workflow import workflow

@workflow.defn(name="order-processing")
class OrderWorkflow:
    def run(self, ctx, *args):
        order = args[0] if args else {}

        # Schedule an activity and wait for the result
        validated = yield ctx.schedule_activity(
            "validate_order", [order]
        )

        # Start a timer (durable sleep)
        yield ctx.start_timer(seconds=60)

        # Schedule another activity
        receipt = yield ctx.schedule_activity(
            "process_payment", [validated]
        )

        return receipt
```

The `name` in `@workflow.defn(name="...")` is the type key used across all languages. It must be a plain string — not a Python module path or class reference.

### Signals, Queries, and Updates

Signals are recorded in durable history and dispatched to Python handlers
during workflow replay:

```python
@workflow.defn(name="approval")
class ApprovalWorkflow:
    def __init__(self) -> None:
        self.approved = False
        self.approved_by = None

    @workflow.signal("approve")
    def approve(self, by: str) -> None:
        self.approved = True
        self.approved_by = by

    def run(self, ctx, *args):
        ready = yield ctx.wait_condition(
            lambda: self.approved, key="approval.approved", timeout=60,
        )
        return {
            "status": "approved" if ready else "timed_out",
            "approved": self.approved,
            "approved_by": self.approved_by,
        }
```

The wait belongs to the workflow, not an activity. Its predicate reads replayed
signal-handler state; the worker does not hold an activity slot for the approval.
The condition's timeout is separate from the workflow's finite run and execution
deadlines. For external-job completion, use a short submit-and-return activity
and a [correlated callback](./polling-patterns.md#callback-and-signal-completion).

Query and update receiver decorators are available so workflow classes can
publish stable handler names. Python workers execute server-routed query tasks
by replaying the committed workflow history and invoking the registered query
handler against that replayed state. They also apply accepted updates delivered
through workflow tasks:

```python
@workflow.defn(name="approval")
class ApprovalWorkflow:
    def __init__(self) -> None:
        self.approved = False

    @workflow.query("status")
    def status(self) -> dict:
        return {"approved": self.approved}

    @workflow.update("set_approval")
    def set_approval(self, approved: bool) -> dict:
        self.approved = approved
        return {"approved": self.approved}

    @set_approval.validator
    def validate_set_approval(self, approved: bool) -> None:
        if not isinstance(approved, bool):
            raise ValueError("approved must be boolean")
```

Python workers complete server-routed queries by returning a query-task result
to the server. For a declared update validator, the worker instead receives a
dedicated validation task, replays authoritative workflow state, and invokes
only the validator. The server records no accepted update and dispatches no
update handler until that validation succeeds. Validator-bearing workers refuse
to register unless discovery advertises the synchronous pre-accept contract;
rejection, missing or incompatible workers, worker loss, timeout, and fenced
duplicate or stale completion remain explicit typed outcomes. PHP and Rust do
not currently expose update-validator authoring APIs and advertise that absence
in their workflow contracts rather than claiming validator parity.

### Handler lifecycle {#handler-lifecycle}

**Released validation.** Python SDK 2.6.2 introduced the handler guards below. Server-routed asynchronous queries and update validators remain supported. See [SDK #177](https://github.com/durable-workflow/sdk-python/issues/177#issuecomment-6296781770) for the corrected scope.

Every replay constructs a fresh workflow instance before executing `run()`. Signals already recorded at workflow start are applied after `run()` reaches its first `yield`, so constructor and pre-yield initialization happen first. Signal and update handlers mutate that replayed state. Queries inspect it without changing durable state; update validators inspect a disposable replayed state before acceptance. Put durable activities, timers and child workflow commands in `run()`, not in a handler.

| Handler | Supported form and failure outcome |
| --- | --- |
| `@workflow.signal` | Synchronous function. An exception while applying the signal during replay fails the workflow run. |
| `@workflow.update` | Synchronous function. A handler exception fails that update. |
| `@workflow.update_validator` or `@update_method.validator` | Synchronous function or `async def` under the Worker. An exception rejects the update before acceptance. |
| `@workflow.query` | Synchronous function or `async def` under the Worker. A handler exception fails the query, not the workflow run. |

The validation rejects coroutine functions for signals and updates, and rejects generator or async-generator functions for **all** decorated handler kinds, including queries and validators. It raises `InvalidWorkflowDefinition` at definition, registration or replay, including inherited handlers and offline tools. Synchronous signal and update wrappers cannot bypass the guards by returning asynchronous values. Returning generator or async-generator values is refused for every handler kind. These restrictions do not forbid the synchronous generator used by `run()`.

For a server-routed query, the Worker explicitly awaits an asynchronous result under the original query deadline and claim/cancellation authority. Awaiting does not grant a fresh budget or permit a result after authority is lost. For pre-accept validation, the Worker also awaits an asynchronous validator result against disposable replayed state.

The local synchronous helpers cannot await. `workflow.query_state()` refuses an asynchronous result with `QueryFailed`, and `workflow.validate_update()` refuses it with `UpdateRejected`. They close returned coroutines instead of leaking them as operation results.

### Workflow Context

The `WorkflowContext` passed to `run` provides deterministic operations:

| Method | Description |
|--------|-------------|
| `ctx.schedule_activity(type, args)` | Schedule an activity task, optionally with per-call retry and timeout options |
| `ctx.start_timer(seconds)` | Durable sleep |
| `ctx.start_child_workflow(type, args)` | Start a child workflow, optionally with per-call retry and workflow timeout options |
| `yield [command, [...]]` | Join a nested deterministic activity, child-workflow, timer, or mixed group in input order |
| `yield ctx.select({key: command, ...})` | Resume with the first durably committed member while retaining handles for every non-winner |
| `ctx.saga()` | Register and run reverse-order durable activity compensations |
| `ctx.side_effect(fn)` | Capture a non-deterministic value |
| `ctx.get_version(change_id, min, max)` | Safe workflow code versioning |
| `ctx.upsert_search_attributes(attrs)` | Update search attributes |
| `ctx.continue_as_new(*args)` | Restart the workflow with new input |
| `ctx.now()` | Persisted workflow-start time, fixed for this run |
| `ctx.random()` | Seeded random generator |
| `ctx.uuid4()` | Deterministic UUID |
| `ctx.logger` | Logger that is silent during replay |

### Entry compatibility and recovery {#entry-compatibility-and-recovery}

**Released support.** Python SDK 2.6.2 introduced the entry safeguard below, with routing and recovery support in Server 2.9.9. These are minimum feature versions; install current stable releases for security fixes.

Before calling `run(ctx, *start_input)`, the worker binds the recorded arguments against the entry signature. A missing required argument or incompatible arity raises `WorkflowEntryArgumentsMismatchError` with reason `workflow_entry_arguments_mismatch` before the body runs. Defaults fill omitted arguments only. Python annotations remain unenforced: changing an annotation does not validate or convert recorded data. A `TypeError` raised by a successfully entered ordinary or generator body remains an application failure.

The worker automatically advertises `workflow_entry_arguments_validation`. With a qualified Server, the failed task becomes replay-blocked while the original run remains nonterminal. Older runtimes may retry that task instead; SDK-side refusal alone does not establish the complete recovery contract.

Inspect the exact run with `await client.describe_workflow_run(workflow_id, run_id)` and the selected-run debug endpoint. After restoring a compatible entry method, use the selected-run HTTP repair route in the [recovery procedure](../failures-and-recovery.md#service-entry-recovery). `await client.repair_workflow(workflow_id)` and `handle.repair()` instead resolve the instance's current run; neither accepts a run-id argument. Preserve the original input, history, namespace, queue, build and deadlines. For command changes, put `ctx.get_version` inside that compatible method and qualify cold replay through [definition routing](./worker-compatibility-routing.md#definition-fingerprint-evolution).

### Workflow time and durable waits

During worker replay, `ctx.now()` returns the run's **persisted start time**.
It does not advance after an activity, timer, signal, or other durable step.
Repeated cold replays of the same history return that same start value. A
continue-as-new successor has its own start time; it does not keep the
predecessor's clock anchor.

The [published Python implementation](https://github.com/durable-workflow/sdk-python/blob/150c6f6bc14e3516f75cc615b2f75f093ed84ea1/src/durable_workflow/workflow.py#L2594-L2601)
uses `WorkflowStarted.payload.timestamp` when present, otherwise the canonical
`WorkflowStarted.recorded_at`. Missing or invalid persisted replay time is an
explicit clock error, not permission to read the worker's wall clock. This is
the fixed-start contract settled in [SDK #146](https://github.com/durable-workflow/sdk-python/issues/146).

Wait for a durable operation instead of looping until `ctx.now()` changes:

```python
from durable_workflow import workflow

@workflow.defn(name="wait_without_clock_loop")
class WaitWithoutClockLoop:
    def run(self, ctx):
        started_at = ctx.now()
        yield ctx.start_timer(30)
        return {
            "run_started_at": started_at.isoformat(),
            "after_wait": ctx.now().isoformat(),
        }
```

The timer's durable completion resumes the workflow. Both returned timestamps
are the same start time. A loop such as `while ctx.now() < deadline` will not
become false merely because timers fire.

For a relative delay, yield `ctx.start_timer(seconds)` or `ctx.sleep(seconds)`.
Use [first-completion selection](#first-completion-selection) when work should
race a durable timer. Set `execution_timeout_seconds` and `run_timeout_seconds`
on `client.start_workflow(...)` when Server must enforce a workflow deadline,
including while workers are stopped. An HTTP or `get_result()` waiting timeout
only limits that local request; it does not replace the workflow's persisted
deadline. Keep an application deadline across continue-as-new by explicitly
passing it as input, without replacing the Server's original execution deadline.

For measured elapsed time within one activity invocation, read a monotonic
clock there and return the duration as the activity result. For an external
UTC deadline, let an activity check the authoritative external clock and
return its decision; workflow code replays that recorded result. Never compare
UTC timestamps with monotonic values, or subtract monotonic readings from
different processes or hosts. `ctx.now()` subtraction measures neither active
processing time nor elapsed wall time in Python.

#### Clock differences across SDKs

These APIs are deterministic where provided, but they do not share one clock:

| Authoring surface | Repeated calls and durable steps | Continue-as-new successor |
| --- | --- | --- |
| Python `ctx.now()` | Fixed to this run's persisted start time, including after a timer or activity. | Uses the successor run's start anchor. |
| Embedded PHP `Workflow::now()` | Starts at the run's start time and advances to the recorded event time consumed by replay. See [workflow time](../features/timers.md#reading-deterministic-time). | Starts with the new run, then advances through that run's consumed events. |
| PHP service-mode SDK | No general public workflow `now()` helper. Use durable operations or recorded activity/side-effect values. | Pass application timing state explicitly when it must survive the handoff. |
| Rust service-mode SDK | No general public `WorkflowContext::now()` helper. A `side_effect` can record a small wall-clock snapshot; it is not an advancing clock. | A side effect in the new run is a new operation; pass earlier timing state explicitly if needed. |

The embedded PHP behavior is present in
[published Core](https://github.com/durable-workflow/workflow/blob/64efa176e4c37e5d6ae80e767ba3b41464024e53/src/V2/Support/WorkflowFiberContext.php);
the separate [PHP SDK context](https://github.com/durable-workflow/sdk-php/blob/2.5.2/src/Worker/WorkflowContext.php)
and [Rust SDK context](https://github.com/durable-workflow/sdk-rust/blob/3.5.5/src/lib.rs)
have their own APIs. Port the durable operation and deadline policy rather than
assuming that a clock helper has identical semantics in every SDK.

#### Order emitted events with sequence values

A workflow timestamp is not an event ordering key. Python events stamped with
`ctx.now()` can all have the same timestamp. Use
[Workflow Stream offsets](./workflow-streams.md), the applicable
[embedded outbox sequence](../features/message-streams.md), or a deterministic
workflow-local counter. Pair a run-local counter with `run_id`; carry it through
continue-as-new explicitly if the application needs one logical sequence.
Independent streams or runs do not gain a global order from their timestamps.

### Activity Retries and Timeouts

Use `ActivityRetryPolicy` and timeout keyword arguments on `ctx.schedule_activity(...)`
when one activity call needs a different retry budget or deadline than the
default single-attempt behavior:

```python
from durable_workflow import ActivityRetryPolicy

receipt = yield ctx.schedule_activity(
    "process_payment",
    [validated],
    retry_policy=ActivityRetryPolicy(
        max_attempts=4,
        initial_interval_seconds=1,
        backoff_coefficient=2,
        maximum_interval_seconds=30,
        non_retryable_error_types=["ValidationError"],
    ),
    start_to_close_timeout=120,
    schedule_to_close_timeout=300,
    heartbeat_timeout=15,
)
```

`start_to_close_timeout` caps one attempt after a worker starts it.
`schedule_to_close_timeout` caps the whole activity execution across all
attempts. `heartbeat_timeout` requires long-running activities to call
`activity.context().heartbeat(...)` before the interval expires. The retry
policy is snapped onto the durable activity execution when it is scheduled, so
later deploys do not change already-running attempts.

`ctx.schedule_activity(..., queue="payments-v2")` selects the queue for a newly scheduled activity. An already scheduled execution, its retries and repaired tasks keep their recorded queue. Changing the worker's `task_queue` does not reroute them or change an existing run's workflow queue. Keep compatible old-queue workers until the remaining dependencies are accounted for. Follow [recorded activity queues and recovery](./task-matching-dispatch.md#recorded-activity-queues).

### Determinism Rules

Workflow code is replayed from history. It must not perform any non-deterministic operations directly:

- No I/O (HTTP calls, file reads, database queries)
- No `datetime.now()` or `time.time()` — use `ctx.now()` only for the fixed run-start anchor; use durable timers for waits
- No `random.random()` — use `ctx.random()`
- No `uuid.uuid4()` — use `ctx.uuid4()`

All non-determinism must flow through the context or be captured with `ctx.side_effect()`.

#### Finally blocks and context managers {#workflow-cleanup}

**SDK 2.12.2 and later.** A suspension is not a workflow exit. Python may execute
`finally` blocks and `with` exits while disposing of a suspended replay. The SDK
discards durable commands and `ctx.logger` messages produced only by disposal;
queries see the state at the pending command. Actual completion, failure or
cooperative cancellation runs cleanup through the normal history, and cold
replay uses its recorded results.

Keep direct I/O out of these hooks. Schedule external cleanup as an Activity
with `yield ctx.schedule_activity(...)`, and wrap that Activity in
`with ctx.cancellation_shield():` if it must finish after cooperative cancellation.

### Fan-Out

Yield a list of commands to run them concurrently:

```python
@workflow.defn(name="fan-out-example")
class FanOutWorkflow:
    def run(self, ctx, *args):
        items = args[0]

        # Nested lists may mix activities, children, and timers.
        results = yield [
            ctx.schedule_activity("process_item", [items[0]]),
            [
                ctx.start_child_workflow("process-batch", [items[1:]]),
                ctx.start_timer(1),
            ],
        ]

        return results  # same nested shape and input order
```

The worker emits one ordinary command per durable leaf and attaches the shared
stable `parallel_group_*` fields plus the full outer-to-inner path. History can
close members in any order; replay still binds results and failures by durable
input position. Exact duplicate terminal delivery is ignored. Pending history
after a worker restart and fully completed history reconstruct the same group
without rescheduling completed work.

### First-completion selection

Use `ctx.select()` when independent work should start together but the workflow
can make progress after one member completes:

```python
selected = yield ctx.select({
    "resolver": ctx.schedule_activity("resolve-request", [request_id]),
    "input": ctx.wait_condition(lambda: self.resolution is not None, key="resolution-ready"),
    "deadline": ctx.start_timer(2),
})

if selected.key == "deadline":
    yield selected.handles["resolver"].cancel()
    return {"status": "timed_out"}

yield selected.handles["deadline"].cancel()
return {"status": "resolved", "value": selected.result()}
```

The `SelectionResult` includes the winner's stable key, input index, operation
kind and durable identity, typed outcome, and all `DurableOperationHandle`
instances. `yield handle.await_result()` waits for a non-winner later; `yield
handle.cancel()` records an explicit void request. Only committed
`SelectionOperationCancelled` history proves cancellation won; replay advances
past an unmarked request, and `await_result()` still returns a completion that
committed first. Replay consumes the persisted winner even if later history
contains another completion first. Duplicate and out-of-order external
delivery cannot replace an already committed winner.

### Saga Compensation

```python
def forward(saga):
    flight = yield ctx.schedule_activity("trip.reserve-flight", [])
    saga.add_compensation("trip.cancel-flight", [flight])

    hotel = yield ctx.schedule_activity("trip.reserve-hotel", [])
    saga.add_compensation("trip.cancel-hotel", [hotel])

    yield ctx.schedule_activity("trip.charge", [])
    return {"status": "booked"}

return (yield from ctx.saga().run(forward))
```

The saga executes ordinary activity commands sequentially in reverse
registration order after a failure. Terminal `cancel_workflow` does not
resume the workflow to run compensations. The helper stops on the
first compensation failure. `SagaCompensationFailed` preserves the initiating
failure, compensation failure, compensation activity type, and deterministic
registration order as structured diagnostics.

### Child Workflows

```python
from durable_workflow import ChildWorkflowRetryPolicy

result = yield ctx.start_child_workflow(
    "child-workflow-type",
    [{"input": "data"}],
    task_queue="child-queue",
    parent_close_policy="terminate",
    retry_policy=ChildWorkflowRetryPolicy(
        max_attempts=3,
        initial_interval_seconds=2,
        backoff_coefficient=2,
        non_retryable_error_types=["ValidationError"],
    ),
    execution_timeout_seconds=600,
    run_timeout_seconds=120,
)
```

`execution_timeout_seconds` caps the logical child workflow execution across
retries and continue-as-new runs. `run_timeout_seconds` caps each child run
attempt. Retry backoff is applied after a child run fails; invalid child start
commands are protocol errors and are not retried as child attempts.

### Continue-as-New

For long-running workflows, use continue-as-new to reset the history:

```python
from durable_workflow import ContinueAsNew

@workflow.defn(name="polling-workflow")
class PollingWorkflow:
    def run(self, ctx, *args):
        iteration = args[0] if args else 0

        result = yield ctx.schedule_activity("poll_source", [])

        if result.get("done"):
            return result

        # Continue with incremented iteration
        return ContinueAsNew(arguments=[iteration + 1])
```

## Defining Activities

Activities are async Python functions decorated with `@activity.defn`. Unlike workflows, activities can perform I/O freely.

```python
from durable_workflow import activity

@activity.defn(name="send_email")
async def send_email(to: str, subject: str, body: str) -> dict:
    # Activities can do I/O: HTTP calls, database queries, etc.
    response = await some_email_client.send(to=to, subject=subject, body=body)
    return {"message_id": response.id, "sent": True}
```

The `name` is the type key shared across languages. A PHP workflow can schedule an activity named `"send_email"` and a Python worker will pick it up, and vice versa.

### Activity Context

Inside an activity, access execution metadata and heartbeat via `activity.context()`:

```python
@activity.defn(name="long_running_task")
async def long_running_task(items: list) -> dict:
    ctx = activity.context()

    print(f"Attempt #{ctx.info.attempt_number}")
    print(f"Task queue: {ctx.info.task_queue}")

    for i, item in enumerate(items):
        # Check for cancellation
        if ctx.is_cancelled:
            return {"partial": True, "processed": i}

        await process(item)

        # Heartbeat to keep the task alive
        await ctx.heartbeat({"progress": i + 1, "total": len(items)})

    return {"processed": len(items)}
```

#### Logical activity execution identity {#activity-identity}

Python SDK 2.15.0 and later expose the read-only optional
`activity.context().info.activity_execution_id`. It identifies one logical
activity execution across retries. `info.workflow_id` and `info.run_id`
identify its owning workflow instance and run. Use the logical execution ID
for an external service's idempotency key. Task IDs and attempt IDs change
on retry and must not replace it.

Missing, invalid or blank execution IDs remain `None`, including on older
claims. Nonblank received strings retain their exact spelling. If an operation
requires this key, refuse it before performing the external effect when the
ID is unavailable. Session-routed activities use their ordinary claim identity.
Prepared local activities use their validated admission receipt's execution
ID. Inline local activities have no admitted execution ID and expose `None`,
while retaining parent workflow/run IDs. Server-free mocks do not create
Server activity identity.

Use the SDK's [idempotency-key helper and activity identity guide](https://python.durable-workflow.com/sdk-reference/#activity-execution-identity)
for the service-mode example. Give different external operations stable
operation suffixes, and ensure the downstream service honors the key. An
execution ID is public metadata, not proof of a live claim or authorization
for an external effect. Keep credentials, current-attempt fencing and outcome
reconciliation. The [published-package ordinary retry receipt](https://github.com/durable-workflow/.github/issues/211#issuecomment-6869562527)
binds Python SDK 2.17.2 to native Server 3.9.2. It proves a stable logical
execution/workflow/run identity and distinct task/attempt IDs across attempts
1 and 2. Local and session behavior retain their separately defined scope.
See [execution guarantees](../constraints/execution-guarantees.md#service-activity-identity)
for the shared idempotency contract.

#### Received retry policy and original attempt {#activity-retry-metadata}

Requires Python SDK **2.12.0 or later**. `info = activity.context().info`
exposes the original `attempt_number` and an immutable `retry_info` snapshot.
`info.retry_info.policy()` returns a fresh copy, including unknown fields.

A missing policy and root `retry_policy: null` both have an unknown retry
limit. `policy_provided` distinguishes them (`False` and `True`).
A received policy object with `max_attempts: null` explicitly means unlimited:
`is_unlimited()` is `True`, while `max_attempts()` remains `None`.
For a positive integer limit, `max_attempts()` returns that limit and
`is_last_configured_attempt()` compares it with the original attempt.
An unknown limit returns `None` for that last-attempt check.

This check describes the received budget. Cancellation, deadlines, workflow
closure and non-retryable failures can end execution earlier. Ordinary and
supervised remote callbacks receive the original snapshot. Inline and prepared
local callbacks do not inherit the parent's remote retry policy.

```python
from durable_workflow import activity

@activity.defn(name="inspect_retry_budget")
async def inspect_retry_budget() -> dict:
    retry = activity.context().info.retry_info
    return {
        "attempt": retry.attempt_number,
        "max_attempts": retry.max_attempts(),
        "last_configured_attempt": retry.is_last_configured_attempt(),
    }
```

#### Signal the originating run {#signal-originating-run}

`info.workflow_id` and `info.run_id` preserve the original received identities
and can be `None`. Local callbacks inherit only known parent identities.
The SDK never substitutes a task ID. Pass an authorized application client
to this helper; the handle pins the original run:

```python
from durable_workflow import Client, activity

async def signal_originating_run(client: Client, business_event_id: str) -> dict:
    info = activity.context().info
    if info.workflow_id is None or info.run_id is None:
        raise RuntimeError("This activity has no known originating workflow run")
    handle = client.get_workflow_handle(info.workflow_id, run_id=info.run_id)
    return await handle.signal(
        "payment_recorded",
        [business_event_id],
        request_id=f"payment-recorded:{business_event_id}",
    )
```

Keep `business_event_id` stable across retries and within the Server's
command-key length limit. The currently published Server does not retain
these keys for every signal path. Make business-event processing idempotent
and reconcile uncertain delivery. See the
[canonical activity reference](https://python.durable-workflow.com/reference/activity/)
and [signal receipt-retention requirement](https://github.com/durable-workflow/.github/issues/216).

### Heartbeat requests {#activity-heartbeat-requests}

For remote activities, call `await ctx.heartbeat(...)` during bounded work, well before `heartbeat_timeout`. Its first window starts at runtime claim, before Python receives the task. Each recorded heartbeat uses the runtime's recording time, not the time its response reaches Python. Pauses, request latency and transport retries consume the budget. Follow the [cadence guidance](../features/heartbeats.md#heartbeat-cadence-and-receipts).

The direct `Client.heartbeat_activity_task()` call uses the configured `TransportRetryPolicy`. After that policy, transport errors can propagate as `httpx.TransportError`, and HTTP refusals become mapped SDK exceptions such as `ServerError`. This is HTTP retry behavior, not another activity attempt or more heartbeat time. `ActivityContext.heartbeat()` can raise `ActivityCancelled` when the worker tells the activity to stop. When cooperative process supervision is enabled, failed heartbeat or ownership checks can stop or abandon the callback. Do not assume every failure returns to a catch block or that cleanup code always runs.

Do not suppress heartbeat/ownership failures and continue external work or report success. Progress is not an external commit or a resumable-checkpoint receipt. Keep external effects idempotent.

### Heartbeat refusals and callback exit {#heartbeat-refusals-and-callback-exit}

In **Python SDK 2.9.0**, `ctx.heartbeat()` and `ctx.heartbeat_sync()` raise `ActivityLeaseAuthorityLost` when a heartbeat response says `can_continue=false`. The exception retains the original Server reason, such as `activity_deadline_elapsed`, `attempt_closed` or `run_closed`. A cancellation request instead raises `ActivityCancelled`. See the [published SDK qualification](https://github.com/durable-workflow/sdk-python/issues/186#issuecomment-6361330674).

The heartbeat that observes the refusal is a cooperative exit point. Stop external work and propagate the exception; do not catch it and return success. Code that never heartbeats can keep running until it returns. A helper heartbeat cannot forcibly interrupt a blocked library call: use finite I/O deadlines and the call's own cancellation mechanism.

The physical activity slot stays occupied until the callback/thread exits. Losing attempt authority does not make that thread available for replacement work, and the worker does not publish a late result for the closed attempt. Runtime timeout enforcement and stopping application code are separate boundaries. See [activity timeouts](../features/timeouts.md#activity-timeouts) and the [helper-thread guidance](#heartbeat-helper-thread).

### Non-Retryable Errors

Raise `NonRetryableError` to fail the activity without retries:

```python
from durable_workflow import NonRetryableError

@activity.defn(name="validate")
async def validate(data: dict) -> dict:
    if "required_field" not in data:
        raise NonRetryableError("Missing required_field")
    return data
```

### Terminal business errors {#terminal-business-errors}

`non_retryable_error_types` matches the **outermost** exception's short name (`exception_type`) or fully qualified class name (`exception_class`) exactly. For example, `ValidationError` does not match `SpecificValidationError`, even if it is a subclass. A qualified entry such as `app.errors.ValidationError` must match that recorded class name. The policy does not walk subclasses or exception causes. `raise RuntimeError(...) from NonRetryableError(...)` reports the outer `RuntimeError` and can still be retried.

When converting a known business exception, make `NonRetryableError` the outer exception and keep the original Python cause with `cause=error`. This marks the activity failure non-retryable without relying on a name list. It does not make the entire workflow fail automatically: workflow code receives `ActivityFailed` when the activity has no retries left, either because it is non-retryable or because its retry budget is exhausted.

```python
from durable_workflow import ActivityFailed, NonRetryableError, activity, workflow

@activity.defn(name="validate_order")
async def validate_order(order: dict) -> dict:
    try:
        if order.get("amount", 0) <= 0:
            raise ValueError("amount must be positive")
    except ValueError as error:
        raise NonRetryableError("Order validation failed", cause=error) from error
    return order

@workflow.defn(name="place_order")
class PlaceOrderWorkflow:
    def run(self, ctx, order: dict):
        try:
            yield ctx.schedule_activity("validate_order", [order])
        except ActivityFailed as failure:
            if (
                failure.activity_type != "validate_order"
                or failure.exception_type != "NonRetryableError"
                or not failure.non_retryable
            ):
                raise
            return {"order_id": order["id"], "status": "rejected"}
        return {"order_id": order["id"], "status": "accepted"}
```

For input `{"id": "order-123", "amount": 0}`, the activity fails once and the workflow completes normally with `status: rejected`. Handle only the business failure you intend to handle; the example rethrows other failures. Continue with further durable steps if your business flow requires them. An uncaught `ActivityFailed` fails the workflow. Preserving a Python cause does not mean retry policy examines that chain or that every SDK exposes an identical cause object.

### Synchronous and asynchronous activities {#activity-execution}

Ordinary `def` remote activities run in a dedicated thread pool bounded by
`max_concurrent_activity_tasks`. Each callback keeps its activity context.
Polling, workflow tasks and worker heartbeats can progress while a callback
blocks. `async def` activities run on the event loop and must use nonblocking
I/O; a blocking call inside an async function still stalls that loop.

From a synchronous callback, call
`activity.context().heartbeat_sync(details)` to send progress through the owner
event loop with the original task, attempt and worker identity. Async callbacks
use `await activity.context().heartbeat(details)`.

Cancelling an await does not stop a running Python thread. Its physical slot
stays occupied until the callback returns, and a late return does not submit a
result after cancellation. If a callback is still running after
`shutdown_timeout`, shutdown raises `RuntimeError` and retains the worker
registration. Set timeouts on downstream I/O and use separately isolated worker
processes for CPU-heavy work. See the SDK's
[execution and shutdown reference](https://python.durable-workflow.com/reference/worker/).

### Heartbeating from a helper thread {#heartbeat-helper-thread}

For one long blocking call, capture `ctx = activity.context()` inside the
synchronous activity and pass it to the helper. A new thread does not
automatically inherit the activity context. Use `ctx.heartbeat_sync(details)`
from the helper, never from the worker event loop. See the SDK's
[complete helper-thread recipe](https://python.durable-workflow.com/reference/activity/#heartbeats-during-one-blocking-call).

Stop and join the helper when the call returns or raises. If a heartbeat raises
`ActivityCancelled` or `ActivityLeaseAuthorityLost`, signal cooperative
cancellation to the call and re-raise the original exception before returning
a result. A helper exception cannot forcibly interrupt a blocked library call.
Give that call its own finite deadline and cancellation mechanism, and bound
heartbeat request timeouts and retries. Helper heartbeats can otherwise keep
reporting liveness while the work is hung.

## Worker

The `Worker` registers with the server, polls for tasks, and dispatches them to your workflow and activity implementations.

For independent workflow and activity processes, follow the Python SDK's
[runnable process-split guide](https://python.durable-workflow.com/guides/separate-workers/).
It covers separate registrations, shared or distinct task queues, retries,
role-specific slots and graceful shutdown.

```python
from durable_workflow import Client, Worker

async with Client("http://localhost:8080", token="secret") as client:
    worker = Worker(
        client,
        task_queue="default",
        workflows=[GreeterWorkflow, OrderWorkflow],
        activities=[greet, send_email, validate],
        max_concurrent_workflow_tasks=10,
        max_concurrent_activity_tasks=10,
    )

    await worker.run()  # blocks until worker.stop() is called
```

For smoke tests and one-workflow examples,
`await worker.run_until(workflow_id="...", timeout=60.0)` registers the same
worker and drives one workflow to a terminal state with sequential polling. Use
`run()` for deployed workers that should keep polling.

| Parameter | Default | Description |
|-----------|---------|-------------|
| `task_queue` | required | The task queue to poll |
| `workflows` | `()` | Workflow classes to register |
| `activities` | `()` | Activity functions to register |
| `worker_id` | auto-generated | Unique worker identifier |
| `poll_timeout` | `35.0` | Long-poll timeout in seconds |
| `max_concurrent_workflow_tasks` | `10` | Max parallel workflow tasks |
| `max_concurrent_activity_tasks` | `10` | Max parallel activity tasks |
| `shutdown_timeout` | `30.0` | Seconds to drain in-flight tasks on stop |
| `metrics` | client's recorder | Optional metrics recorder for poll and task counters/histograms |
| `interceptors` | `()` | Ordered worker task wrappers for instrumentation, tracing, and policy hooks |

The two `max_concurrent_*` values are advertised to the server during worker
registration and appear in task queue admission diagnostics. Treat them as the
worker's local capacity. Use server-side
[task queue admission](/docs/polyglot/task-queue-admission) caps when a
namespace, queue, or downstream budget group needs a hard shared budget across
multiple workers.

### Worker API Reference

| Method | Returns | Use when |
| --- | --- | --- |
| `await worker.run()` | `None` | Long-running process supervised by systemd, Docker, Kubernetes, or a local dev shell. Registers once, then polls workflow, activity, and query tasks until stopped or cancelled. |
| `await worker.run_until(workflow_id=..., timeout=60.0, poll_interval=0.5)` | `WorkflowExecution` | Smoke tests and examples that start one workflow and want the same process to drive it until a terminal status. |
| `await worker.stop()` | `None` | Cooperative shutdown. Stops new polls and drains in-flight tasks up to `shutdown_timeout`. |

`run()` validates server compatibility before it starts polling. The worker
requires the server's published `control_plane.version`,
`control_plane.request_contract`, `worker_protocol.version`, and
`auth_composition_contract` to match the SDK's supported contract versions. A
missing or incompatible manifest raises `RuntimeError` during registration, so
supervisors fail fast instead of running a worker that cannot safely complete
tasks.

`run_until()` uses the same registration and dispatch path as `run()`, but
polls sequentially and returns the final `WorkflowExecution` for the named
workflow. It raises `TimeoutError` when the workflow is still non-terminal
after the timeout.

During task execution:

- unknown workflow or activity types are reported back to the server as task
  failures, not hidden in local logs
- `NonRetryableError` marks activity failures as non-retryable
- `ActivityCancelled` propagates as a cancellation outcome
- unhandled activity exceptions are reported as retryable failures unless the
  activity retry policy or server deadline says otherwise
- query handler exceptions are reported as `QueryFailed`

### Worker Interceptors

Pass `interceptors=[...]` when worker execution needs tracing, metrics, audit
logging, or local policy checks around tasks. Interceptors run in the order
provided; the first interceptor is the outer wrapper and should call `next` to
continue the chain.

```python
from durable_workflow import (
    ActivityInterceptorContext,
    PassthroughWorkerInterceptor,
)

class AuditInterceptor(PassthroughWorkerInterceptor):
    async def execute_activity(self, context: ActivityInterceptorContext, next):
        print("activity started", context.activity_type, context.worker_id)
        return await next(context)

worker = Worker(
    client,
    task_queue="orders",
    workflows=[OrderWorkflow],
    activities=[charge_card],
    interceptors=[AuditInterceptor()],
)
```

| Hook | Context fields | `next` returns |
| --- | --- | --- |
| `execute_workflow_task(context, next)` | `worker_id`, `task_queue`, `task` | workflow commands, or `None` |
| `execute_activity(context, next)` | `worker_id`, `task_queue`, `task`, `activity_type`, `args` | decoded activity result |
| `execute_query_task(context, next)` | `worker_id`, `task_queue`, `task` | encoded query result string |

Use `PassthroughWorkerInterceptor` as a base class when you only need one
hook. Implement `WorkerInterceptor` directly when you want type checkers to
force all hooks to be present.

### Worker sessions {#worker-sessions}

Use `WorkerSessionOptions` when sequential remote activities need one worker's local resource. This example is checked with Python SDK **2.9.0** and Server **2.11.0**; see the [portable affinity version requirements](./portable-worker-affinity.md). Use a unique `job_id` for each workflow run so a new run does not reuse a terminal session id.

The cache below is disposable. Each activity can rebuild it from the durable business input, so a replacement holder does not depend on a previous process's memory:

```python
from durable_workflow import ActivityFailed, WorkerSessionOptions, activity, workflow

# Disposable process-local state. Rebuild it from durable business input.
documents = {}

def ensure_document(job_id):
    rebuilt = job_id not in documents
    if rebuilt:
        documents[job_id] = {"text": "document:" + job_id}
    return documents[job_id], rebuilt

@activity.defn(name="prepare_document")
async def prepare_document(job_id):
    document, rebuilt = ensure_document(job_id)
    return {"holder": activity.context().info.worker_id, "rebuilt": rebuilt}

@activity.defn(name="inspect_document")
async def inspect_document(job_id):
    document, rebuilt = ensure_document(job_id)
    return {"holder": activity.context().info.worker_id,
            "rebuilt": rebuilt, "text": document["text"]}

@workflow.defn(name="inspect_document_workflow")
class InspectDocumentWorkflow:
    def run(self, ctx, job_id):
        session = WorkerSessionOptions(
            session_id="documents:" + job_id,
            requirements=("fs:documents",),
            lease_seconds=5,
            ttl_seconds=60,
            max_concurrent_activities=1,
            allow_reacquire_after_failure=True,
        )
        options = dict(worker_session=session,
                       start_to_close_timeout=5,
                       schedule_to_start_timeout=12,
                       schedule_to_close_timeout=20,
                       retry_policy={"max_attempts": 1})
        try:
            prepared = yield ctx.schedule_activity("prepare_document", [job_id], **options)
            inspected = yield ctx.schedule_activity("inspect_document", [job_id], **options)
        except ActivityFailed as error:
            if error.activity_type not in {"prepare_document", "inspect_document"}:
                raise
            return {"status": "resource_unavailable", "job_id": job_id}
        return {"status": "inspected", "prepared": prepared, "inspected": inspected}
```

Register both activities and declare the required capability on the worker:

```python
from durable_workflow import Worker

worker = Worker(
    client,
    task_queue="documents",
    workflows=[InspectDocumentWorkflow],
    activities=[prepare_document, inspect_document],
    capabilities=["fs:documents"],
)
await worker.run()
```

Start `inspect_document_workflow` on the `documents` task queue with `[job_id]` as input. The first activity lazily acquires the session; the second uses the same holder while its lease remains valid. `max_concurrent_activities=1` bounds simultaneous activities in that session. Queue routing and capability checks still apply. These short leases/timeouts demonstrate bounded behavior; size them for your real work and heartbeat well before expiry.

After holder loss or lease expiry, `allow_reacquire_after_failure=True` permits another eligible worker to reacquire the session. Its local resource must be rebuilt. Already committed activity results replay, while uncommitted effects can repeat: use durable resource identifiers and idempotent external operations. A session does not migrate memory, files or open handles.

Explicit close and absolute TTL expiry are terminal for that session id. A later activity cannot reopen it and can remain unadmitted until its configured timeout is enforced. The native Server enforces activity timeouts in its own maintenance loop, which the published image enables. On PHP Server 2.x, run the scheduler/maintenance role from the [PHP deployment recipe](https://github.com/durable-workflow/server/blob/2.11.8/docker-compose.published.yml); an API-only PHP process is insufficient for timeout enforcement. The workflow handles exhausted activity failure with `ActivityFailed` and returns `resource_unavailable`; the application can then choose a fresh session and rebuild from the business input. This example does not promise an immediate session-closed exception or automatically reopen a terminal session.

For explicit holder lifecycle operations, use `worker.worker_session(options)` and its `create()`, `renew()` and `close(reason)` methods. Graceful worker shutdown closes its held sessions. See the [shared lifecycle contract](../features/worker-sessions.md#lifecycle).

## Logging

The SDK uses Python's standard `logging` module with structured logger names:

| Logger Name | What It Logs |
|-------------|--------------|
| `durable_workflow.worker` | Worker registration, task polls, task completion, errors |
| `durable_workflow.workflow.replay` | Workflow replay events (silent during replay) |

### Configuring Logging

Set the logging level in your application's entry point:

```python
import logging

# Show INFO-level worker events (registration, task completion)
logging.basicConfig(level=logging.INFO)

# Or configure specific loggers
logging.getLogger("durable_workflow.worker").setLevel(logging.DEBUG)
logging.getLogger("durable_workflow.workflow.replay").setLevel(logging.INFO)
```

### Log Levels

- **INFO**: Worker registration, task starts/completions, workflow completion
- **DEBUG**: Detailed task payloads (truncated), poll cycles
- **WARNING**: Retryable errors (failed API calls, unknown workflow types)
- **ERROR**: Non-retryable failures, replay crashes

### Replay-Aware Logging

Inside workflows, use `ctx.logger` for replay-aware logging:

```python
@workflow.defn(name="order_processor")
class OrderProcessor:
    def run(self, ctx, order_id: str):
        ctx.logger.info("Processing order %s", order_id)  # Only logs during execution, not replay
        result = yield ctx.schedule_activity("process_order", [order_id])
        ctx.logger.info("Order processed: %s", result)
        return result
```

Log statements are **silent during replay** to avoid duplicate log spam when workflows recover or continue execution.

### Structured Logging

For JSON-structured logs, configure your application's root logger with a JSON formatter:

```python
import logging
import json

class JSONFormatter(logging.Formatter):
    def format(self, record):
        return json.dumps({
            "timestamp": self.formatTime(record),
            "level": record.levelname,
            "logger": record.name,
            "message": record.getMessage(),
        })

handler = logging.StreamHandler()
handler.setFormatter(JSONFormatter())
logging.getLogger("durable_workflow").addHandler(handler)
logging.getLogger("durable_workflow").setLevel(logging.INFO)
```

## Metrics

`Client(metrics=...)` and `Worker(metrics=...)` accept any recorder with two methods:

```python
def increment(name: str, value: float = 1.0, tags: dict[str, str] | None = None) -> None: ...
def record(name: str, value: float, tags: dict[str, str] | None = None) -> None: ...
```

The default recorder is no-op, so metrics collection has no setup cost. Use `InMemoryMetrics` for tests or custom exporter loops:

```python
from durable_workflow import Client, InMemoryMetrics, Worker

metrics = InMemoryMetrics()

async with Client("http://localhost:8080", token="secret", metrics=metrics) as client:
    worker = Worker(client, task_queue="default", workflows=[GreeterWorkflow], activities=[greet])
```

For Prometheus, install the optional extra and pass `PrometheusMetrics`:

```bash
pip install 'durable-workflow[prometheus]'
```

```python
from durable_workflow import Client, PrometheusMetrics

metrics = PrometheusMetrics()
client = Client("http://localhost:8080", token="secret", metrics=metrics)
```

The SDK records:

| Metric | Type | Tags |
|--------|------|------|
| `durable_workflow_client_requests` | Counter | `method`, `route`, `plane`, `status_code`, `outcome` |
| `durable_workflow_client_request_duration_seconds` | Histogram | `method`, `route`, `plane`, `status_code`, `outcome` |
| `durable_workflow_worker_polls` | Counter | `task_kind`, `task_queue`, `outcome` |
| `durable_workflow_worker_poll_duration_seconds` | Histogram | `task_kind`, `task_queue`, `outcome` |
| `durable_workflow_worker_tasks` | Counter | `task_kind`, `task_queue`, `outcome` |
| `durable_workflow_worker_task_duration_seconds` | Histogram | `task_kind`, `task_queue`, `outcome` |

The separate `durable_workflow_worker_task_schedule_to_start_seconds` histogram
is not in the selected stable package. Its [SDK source hook](https://github.com/durable-workflow/sdk-python/pull/264)
is merged but unreleased, and Server publication plus installed/exported
acceptance remain pending in [#188](https://github.com/durable-workflow/sdk-python/issues/188).
The six metrics above do not provide that measurement. Use the
[operator guide's interim queue-age signals](/docs/operator-operating-envelope#worker-and-sdk-telemetry)
while that delivery is pending.

## Schedules

Create and manage scheduled workflows through the client:

```python
from durable_workflow import ScheduleSpec, ScheduleAction

# Create a schedule
handle = await client.create_schedule(
    schedule_id="hourly-report",
    spec=ScheduleSpec(cron_expressions=["0 * * * *"]),
    action=ScheduleAction(
        workflow_type="generate-report",
        task_queue="default",
        input=[{"format": "pdf"}],
    ),
    overlap_policy="skip",
    jitter_seconds=30,
)

# List all schedules
schedule_list = await client.list_schedules()

# Describe a schedule
desc = await handle.describe()
print(f"Next fire: {desc.next_fire_at}")
print(f"Total fires: {desc.fires_count}")

# Pause and resume
await handle.pause(note="maintenance window")
await handle.resume(note="maintenance complete")

# Trigger immediately
result = await handle.trigger()

# Backfill missed runs
backfill = await handle.backfill(
    start_time="2024-01-01T00:00:00Z",
    end_time="2024-01-02T00:00:00Z",
)

# Update the schedule
await handle.update(
    spec=ScheduleSpec(cron_expressions=["*/30 * * * *"]),
    note="Changed to every 30 minutes",
)

# Read the audit history stream (paused/resumed/triggered/updated events)
page = await handle.history(limit=100)
for event in page.events:
    print(event.sequence, event.event_type, event.recorded_at)
while page.has_more and page.next_cursor is not None:
    page = await handle.history(limit=100, after_sequence=page.next_cursor)
    for event in page.events:
        print(event.sequence, event.event_type, event.recorded_at)

# Or iterate every event without managing cursors
async for event in handle.iter_history():
    print(event.sequence, event.event_type)

# Delete the schedule (history survives the delete for audit)
await handle.delete()
```

## Synchronous Client

For scripts, notebooks, and non-async contexts, use the synchronous wrapper:

<!-- docs-example id="python.sync-client" -->
```python
import uuid

from durable_workflow.sync import Client as SyncClient

client = SyncClient("http://localhost:8080", token="secret")
workflow_id = f"sync-greeting-{uuid.uuid4().hex}"

handle = client.start_workflow(
    workflow_type="greeter",
    task_queue="default",
    workflow_id=workflow_id,
    input=["world"],
)

execution = client.describe_workflow(workflow_id)
print(execution.status)
```

The synchronous client mirrors the async API but wraps each call with `asyncio.run`.

## Error Handling

The SDK maps server error codes to typed Python exceptions:

| Exception | When |
|-----------|------|
| `WorkflowNotFound` | Workflow ID does not exist |
| `WorkflowAlreadyStarted` | Duplicate workflow ID with conflicting policy |
| `WorkflowFailed` | Workflow execution failed |
| `WorkflowCancelled` | Workflow was cancelled (inherits from `BaseException`, not `Exception`) |
| `WorkflowTerminated` | Workflow was terminated |
| `ActivityCancelled` | Activity was cancelled during execution (inherits from `BaseException`, not `Exception`) |
| `ChildWorkflowFailed` | A child workflow failed |
| `QueryFailed` | Query handler returned an error |
| `UpdateRejected` | Update was rejected by the workflow |
| `ScheduleNotFound` | Schedule ID does not exist |
| `ScheduleAlreadyExists` | Duplicate schedule ID |
| `NamespaceNotFound` | Namespace does not exist |
| `InvalidArgument` | Invalid request parameters |
| `Unauthorized` | Authentication failed |
| `ServerError` | Server returned an unexpected error |

<!-- docs-example id="python.duplicate-start-recovery" -->
```python
from durable_workflow import WorkflowNotFound, WorkflowAlreadyStarted

try:
    handle = await client.start_workflow(
        workflow_type="greeter",
        task_queue="default",
        workflow_id="existing-id",
        input=["world"],
    )
except WorkflowAlreadyStarted:
    handle = client.get_workflow_handle("existing-id")
except WorkflowNotFound:
    print("Workflow type not registered on any worker")
```

### Cancellation is intentionally uncatchable by `except Exception`

`WorkflowCancelled` and `ActivityCancelled` inherit from `BaseException`, not `Exception`. A generic `except Exception:` block in an activity body — or in code that awaits `client.get_result()` — will **not** catch them. This is deliberate: cancellation is a control-plane outcome, and silently swallowing it in a catch-all would let an activity report success after its workflow asked it to stop.

If an activity needs local cleanup when its task is cancelled, catch the class
by name and re-raise. This does not resume the closed workflow for compensation:

```python
from durable_workflow import ActivityCancelled, activity

@activity.defn(name="long_task")
async def long_task(items: list) -> dict:
    ctx = activity.context()
    try:
        for i, item in enumerate(items):
            await process(item)
            await ctx.heartbeat({"progress": i + 1})
        return {"done": True}
    except ActivityCancelled:
        await cleanup_partial_state()
        raise
```

This mirrors the standard-library precedent set by `asyncio.CancelledError` and `KeyboardInterrupt`.

## Testing

Workflow authors should be able to test workflow code without a running server
or worker. The `durable_workflow.testing` module ships two entry points:

- `WorkflowEnvironment` drives a workflow to completion in a single Python
  process against user-registered activity mocks.
- `replay_history` and `replay_history_file` replay a captured production
  history against current workflow code and raise on any non-determinism.

Both entry points reuse the same `durable_workflow.workflow.replay` machinery
the worker uses at runtime, so a workflow that passes its test harness behaves
the same way under a real worker.

### WorkflowEnvironment

`WorkflowEnvironment` dispatches yielded workflow commands against registered
mocks and auto-fires timers, side effects, and search-attribute upserts. Tests
do not need a real clock, Redis, or server.

```python
from durable_workflow import workflow
from durable_workflow.testing import WorkflowEnvironment

@workflow.defn(name="greeter")
class Greeter:
    def run(self, ctx, name: str):
        greeting = yield ctx.schedule_activity("greet", [name])
        return greeting

def test_greeter_returns_activity_result() -> None:
    env = WorkflowEnvironment()
    env.register_activity_result("greet", "hello, world")

    result = env.execute_workflow(Greeter, "world")

    assert result == "hello, world"
```

| Method | Purpose |
| --- | --- |
| `register_activity_result(name, result)` | Return `result` for every call to activity `name`. Use this when the test does not care about arguments. |
| `register_activity(name, fn)` | Call `fn(*arguments)` for each scheduled invocation of activity `name`. Use this when the mock must vary with arguments or capture invocations. |
| `register_child_workflow_result(workflow_type, result)` | Return `result` when the workflow starts a child of type `workflow_type`. |
| `signal(name, args=None, run=None)` | Queue a signal to be delivered before the next replay iteration. The harness injects a `SignalReceived` event and dispatches it to the registered `@workflow.signal` handler. Pass `run=N` to target link `N` of a continue-as-new chain. |
| `register_workflow(workflow_cls)` | Make an additional workflow class resolvable by name. Required when a chain calls `continue_as_new(workflow_type=...)` with a type other than the starting workflow. |
| `execute_workflow(workflow_cls, *args, run_id="test-run")` | Drive the workflow to a terminal state and return its result. Follows `continue_as_new` links to the final run and returns that run's result. Raises `WorkflowFailed` when the workflow ends in the failed state. |
| `runs` / `run_count` | After `execute_workflow` returns, expose one `WorkflowRunRecord` per link in the chain (input, workflow type, history events, terminal command) so tests can assert on the full continuation chain. |

The harness fails loudly on missing fixtures:

- scheduling an activity that has no registered mock raises `KeyError`
- starting a child workflow that has no registered mock raises `KeyError`
- a workflow that never reaches a terminal state within the iteration limit
  (default `1000`) raises `RuntimeError`

Pass `iteration_limit=...` to `WorkflowEnvironment(...)` to tune the cap for
workflows that legitimately iterate more than the default.

#### Callable activity mocks

Use `register_activity` when the mock needs to respond based on arguments or
record calls:

```python
def test_callable_mock_captures_arguments() -> None:
    captured: list[str] = []

    def record_greet(name: str) -> str:
        captured.append(name)
        return f"greeted:{name}"

    env = WorkflowEnvironment()
    env.register_activity("greet", record_greet)

    assert env.execute_workflow(Greeter, "alice") == "greeted:alice"
    assert captured == ["alice"]
```

#### Signals

Signals queued with `env.signal(...)` are drained before the next replay
iteration. The signal payload is wrapped in the same `{codec, blob}` envelope
the worker sees at runtime and dispatched to the workflow's registered
`@workflow.signal` handler:

```python
@workflow.defn(name="approval")
class Approval:
    def __init__(self) -> None:
        self.approved_by: str | None = None

    @workflow.signal("approve")
    def on_approve(self, by: str) -> None:
        self.approved_by = by

    def run(self, ctx):
        yield ctx.schedule_activity("wait", [])
        return {"approved_by": self.approved_by}

def test_signal_is_delivered_before_run_returns() -> None:
    env = WorkflowEnvironment()
    env.register_activity_result("wait", None)
    env.signal("approve", ["alice"])

    result = env.execute_workflow(Approval)

    assert result == {"approved_by": "alice"}
```

#### Timers, side effects, and search attributes

The harness auto-fires the corresponding history event for each of these
commands, so workflows do not block on wall-clock time inside tests:

- `ctx.sleep(seconds)` → `TimerFired`
- `ctx.side_effect(...)` → `SideEffectRecorded`
- `ctx.upsert_search_attributes(...)` → `SearchAttributesUpserted`
- `workflow.version(...)` markers → `VersionMarkerRecorded`

#### Continue-as-new chains

When a workflow returns `ctx.continue_as_new(...)`, the harness appends a
`WorkflowContinuedAsNew` event to the completing run, resets history, and
starts a new run with the command's arguments. The return value of
`execute_workflow` is the terminal result of the final link.

```python
@workflow.defn(name="countdown")
class Countdown:
    def run(self, ctx, counter: int):
        yield ctx.schedule_activity("emit", [counter])
        if counter > 0:
            return ctx.continue_as_new(counter - 1)
        return {"final_counter": counter}

def test_chain_returns_final_run_result() -> None:
    env = WorkflowEnvironment()
    env.register_activity_result("emit", None)

    result = env.execute_workflow(Countdown, 3)

    assert result == {"final_counter": 0}
    assert env.run_count == 4
    assert [r.input for r in env.runs] == [[3], [2], [1], [0]]
```

When the chain switches workflow types, register the follow-on class first:

```python
@workflow.defn(name="stage-one")
class StageOne:
    def run(self, ctx):
        yield ctx.schedule_activity("stage_one", [])
        return ctx.continue_as_new(workflow_type="stage-two")

@workflow.defn(name="stage-two")
class StageTwo:
    def run(self, ctx):
        return (yield ctx.schedule_activity("stage_two", []))

def test_chain_can_switch_workflow_type() -> None:
    env = WorkflowEnvironment()
    env.register_workflow(StageTwo)
    env.register_activity_result("stage_one", None)
    env.register_activity_result("stage_two", "done")

    assert env.execute_workflow(StageOne) == "done"
    assert [r.workflow_type for r in env.runs] == ["stage-one", "stage-two"]
```

Target a signal at a specific link in the chain with `run=N`:

```python
env.signal("approve", ["alice"], run=2)  # delivered to the second run
```

The chain length is capped by `continue_as_new_limit` (default `50`).
Exceeding the limit raises `RuntimeError` so tests catch runaway
continuations instead of spinning forever; tune the limit with
`WorkflowEnvironment(continue_as_new_limit=...)` when a chain legitimately
runs longer.

#### Failure assertions

Workflows that raise a Python exception surface as `WorkflowFailed`:

```python
import pytest
from durable_workflow.errors import WorkflowFailed

@workflow.defn(name="failing")
class Failing:
    def run(self, ctx):
        yield ctx.schedule_activity("step", [])
        raise RuntimeError("boom")

def test_workflow_failure_surfaces_as_workflow_failed() -> None:
    env = WorkflowEnvironment()
    env.register_activity_result("step", None)

    with pytest.raises(WorkflowFailed) as exc_info:
        env.execute_workflow(Failing)

    assert "boom" in str(exc_info.value)
```

### Replay testing against production history

Use `replay_history` to regression-test a workflow change against a real
history captured from the server. The replayer runs the current workflow code
against the recorded event sequence and raises if it yields a different
command than the one history recorded — the definition of a non-determinism
bug.

```python
from durable_workflow import Client
from durable_workflow.testing import replay_history

async with Client("http://localhost:8080") as client:
    history = await client.get_history("order-42", run_id="...")

replay_history(OrderWorkflow, history["events"], start_input=["order-42"])
```

A workflow that previously completed must still complete when replayed
against the same history. If the workflow code changed in a way that diverges
from the recorded sequence (reordered activity calls, removed branches,
changed activity types), `replay_history` raises so the regression is caught
in CI rather than in production.

`replay_history_file` is a convenience wrapper that reads a JSON file in
either of two shapes: a top-level list of events, or a dict with an `events`
key matching the `get_history` response shape:

```python
from durable_workflow.testing import replay_history_file

replay_history_file(
    OrderWorkflow,
    "tests/histories/order-42.json",
    start_input=["order-42"],
)
```

Both functions accept an optional `payload_codec` validation hint. Leave it
unset to use the history's recorded tag. An explicit or recorded tag must be
`avro`; `json`, unknown, or untagged durable payloads fail before decode.

#### Class-based replayer

When one test pass replays histories for several different workflow types, or
when the test should let the captured `WorkflowStarted` event decide which
workflow to replay, use the `Replayer` class. Register every workflow class
up front and call `replay(...)` once per captured history:

```python
from durable_workflow import Replayer, ReplayOutcome

replayer = Replayer(workflows=[OrderWorkflow, RefundWorkflow])

# Explicit type and input — equivalent to replay_history(OrderWorkflow, ...).
outcome: ReplayOutcome = replayer.replay(
    history["events"],
    start_input=["order-42"],
    workflow_type="order",
)

# Type and start input inferred from a WorkflowStarted event in the history.
outcome = replayer.replay(history)  # history may be an events list or a
                                    # dict with an "events" key

for command in outcome.commands:
    # Inspect commands the replayed workflow would have emitted next.
    ...
```

`Replayer(workflows=[...])` rejects an empty workflow set or a duplicate
registration with `ValueError`. `replay(...)` raises `ValueError` when a
history asks for a workflow type that was not registered, or when multiple
workflows are registered and the caller does not provide `workflow_type` and
the history does not include a `WorkflowStarted` event.

`ReplayOutcome.commands` is the same command list the functional
`replay_history` helpers return; the class-based entry point exists so a test
suite can share one registration across many histories.

### Test Harness Reference

| Symbol | Purpose |
| --- | --- |
| `durable_workflow.testing.WorkflowEnvironment` | In-process test harness. Drive a workflow to completion against registered activity and child-workflow mocks. |
| `durable_workflow.testing.replay_history(workflow_cls, events, start_input=None, *, run_id="", payload_codec=None)` | Replay a history event sequence against current workflow code. Raises on non-determinism. |
| `durable_workflow.testing.replay_history_file(workflow_cls, path, start_input=None, *, run_id="", payload_codec=None)` | Load a JSON history from disk and replay it. Accepts either a top-level list of events or a dict with an `events` key. |
| `durable_workflow.Replayer(*, workflows=[...])` | Class-based replayer. Register one or more workflow classes, then call `replay(history, start_input=None, *, workflow_type=None, workflow_id=None, run_id="", payload_codec=None)` to replay each captured history. Infers `workflow_type` and `start_input` from a `WorkflowStarted` event when the history contains one. |
| `durable_workflow.ReplayOutcome` | Dataclass returned by `Replayer.replay`. Carries `commands: list[Command]` — the commands the replayed workflow would have emitted next. |
| `durable_workflow.errors.WorkflowFailed` | Raised by `execute_workflow` when the workflow terminates in the failed state. |
| `durable_workflow.errors.WorkflowCancelled` | Terminal state when the workflow was cancelled. Inherits from `BaseException`. |
| `durable_workflow.errors.WorkflowTerminated` | Terminal state when the workflow was terminated by the server. Inherits from `BaseException`. |

## Payload Codec

<div
  data-public-payload-codec-contract="avro-only"
  data-payload-codec="avro"
  data-authority-manifest="https://durable-workflow.github.io/sdk-neutrality-contract.json"
  data-legacy-v1-import-drain="none">
  Every payload that crosses the worker-protocol boundary is codec-tagged.
  Durable Workflow 2.0 has one public payload codec: <code data-payload-codec-field="codec">avro</code>.
  The Python SDK rejects JSON-tagged, unknown, and untagged durable payloads
  instead of selecting another decoder. See the <a href="/docs/polyglot/avro-value-protocol/">Avro Value protocol</a>.
</div>

### Avro support is built in

The pinned 2.0 SDK artifact pulls in `fastavro` as a runtime dependency, so
every outgoing surface (`start_workflow`, `signal_workflow`, `query_workflow`,
`update_workflow`, activity result encoding, schedule actions) emits
Avro-tagged payloads through the optimized production path. There is no
optional codec extra to install.

### Fixed typed values

The SDK explicitly selects named branches in the shared
`durable_workflow.protocol.Value` schema. It does not JSON-encode the value
inside Avro. Integers and doubles, strings and bytes, booleans and integers,
and lists and maps therefore remain distinct across SDKs. The wire uses Avro
single-object framing and the schema fingerprint; unknown fingerprints fail
with `unsupported_payload_schema`.

Every client and worker surface works end-to-end on the Avro default:

- **Client starts, signals, queries, updates** — `start_workflow`, `signal_workflow`, `query_workflow`, and `update_workflow` emit `payload_codec = "avro"` payloads through the fixed Value schema. A Python client can therefore drive workflows that PHP and other polyglot SDKs will replay, and vice-versa.
- **Activity worker** — Avro-tagged activity arguments decode transparently. The worker encodes activity results as Avro so PHP, Python, and future SDK workers share one payload boundary.
- **Activity failures** — `fail_activity_task(..., details=...)` sends `failure.details` as a `{codec, blob}` envelope. The server records the blob plus `details_payload_codec`, so diagnostic failure data from Python workers remains language-neutral in history exports and observability views.
- **Workflow worker history replay** — Avro-tagged start input and activity result events are decoded during replay, so a Python workflow can participate in an Avro-coded run.
- **Workflow query tasks** — Server-routed query tasks carry Avro-tagged workflow arguments, query arguments, and replay history. The worker returns the query result as an Avro envelope.

### Running a Python activity worker against a v2 run

No codec configuration is needed. The SDK validates `payload_codec` before
decoding every claim, accepts only `avro`, runs the activity, and returns an
Avro-tagged result. Any other tag fails closed with
`unsupported_payload_codec`; the worker never sniffs or guesses the format.

### Types that round-trip cleanly across Python and PHP

| Python type | Avro Value branch | PHP type |
|-------------|------|----------|
| `str` | `StringValue` | `string` |
| `bytes` | `BytesValue` | `AvroBinaryValue` |
| `int` | `LongValue` | `int` |
| `float` | `DoubleValue` | `float` |
| `bool` | `BooleanValue` | `bool` |
| `None` | `null` | `null` |
| `list` | `ArrayValue` | `array` (list) |
| `dict[str, ...]` | `MapValue` | `array` (string-keyed map) |

### Typed activity values {#typed-activity-values}

The selected stable SDK supports dataclasses and enums at activity boundaries.
It sends a dataclass as a string-keyed map of its fields and an `Enum` as its
`.value`, preserving the same Avro bytes as explicit neutral values.

| Boundary | Published opt-in behavior |
| --- | --- |
| Activity arguments | `@activity.defn(..., typed=True)` decodes positional arguments using parameter annotations. |
| Activity results | `ctx.schedule_activity(..., result_type=T)` decodes the completed result at the workflow's yield, including parallel groups and `ctx.select(...)`. |
| Workflow, child, client, signal, query, update and validator values | The broader adapters in [SDK #255](https://github.com/durable-workflow/sdk-python/pull/255) are merged but unreleased. Keep explicit neutral-value conversion for these boundaries. |

Supported receiving annotations are dataclasses, `Enum` (including `IntEnum`
and `StrEnum`), `Optional[T]` / `T | None`, `list[T]`, `dict[str, T]`, `str`,
`int`, `float`, `bool`, `bytes` and `Any`. Other annotations, including a union
of two non-`None` types, raise `TypeError` at declaration or call time.
Decoding remains opt-in. `result_type` is local and is never sent, compared or
written to history. See the [published SDK reference](https://github.com/durable-workflow/sdk-python/blob/2.14.1/docs/sdk-reference.md#typed-activity-values)
for defaults, unknown fields and `TypedValueError` handling.

Sets, tuples, datetime, UUID, Decimal, pydantic and attrs objects still require
an explicit adapter to canonical strings, numbers, maps or lists. Changing that
durable value shape is a workflow compatibility change.

## Running Against a Shared Server

The [Quickstart](#quickstart) above shows how to bring up a local server with
the published Docker image. In a team environment you usually point the Python
worker at an existing server (staging, production, or a shared dev instance):

```python
from durable_workflow import Client

client = Client(
    "https://workflow.example.internal",
    control_token="team-orders-operator-token",
    worker_token="team-orders-worker-token",
    namespace="team-orders",
)
```

Set the `namespace` argument to whichever tenant namespace the shared server has
provisioned for your team, and use the credentials issued for that namespace.
The Server operator manages namespace creation; see the
[Server guide](/docs/polyglot/server) for details.

The same constructor supports a Cloud managed runtime. Use the namespace's
Cloud-provided runtime URL and namespace value, set `control_token` to the
client runtime credential, and set `worker_token` to the worker runtime
credential. Cloud provisions the namespace; do not substitute a self-hosted
Server address. See
[Cloud Managed Runtime](/docs/polyglot/cloud-control-plane).

## Workflow list and history pages {#workflow-list-and-history-pages}

The selected stable Python SDK provides lazy `Client.iter_workflows`,
`Client.iter_history` and `WorkflowHandle.iter_history`. Use `async for` to
consume workflow executions or history event dictionaries in Server order.
See the [SDK-owned examples and pagination reference](https://python.durable-workflow.com/sdk-reference/#whole-run-histories-and-workflow-lists)
and [published client source](https://github.com/durable-workflow/sdk-python/blob/2.17.2/src/durable_workflow/client.py).

Each iterator fetches one page at a time without retaining earlier result pages.
It follows empty pages that have a continuation and stops when the continuation
is missing, `None` or empty. Stopping iteration fetches no subsequent page.
HTTP errors, cancellation, history pages whose `events` field is not a list of
dictionaries, and malformed or cyclic continuations propagate. Earlier yielded items are not rolled back, so
handle an interrupted traversal as incomplete.

Keep namespace, filters and page size fixed throughout a traversal. History
also requires the same explicit `workflow_id` and `run_id`; the handle helper
refuses a handle without `run_id`. A namespace change is refused before the
next request. Supply `next_page_token` to resume at an opaque continuation
from the same selection. Do not parse, alter or reuse it for different filters,
namespaces or runs. Traversing a changing workflow list does not create a
snapshot.

The existing `await client.list_workflows(...)` and
`await client.get_history(workflow_id, run_id, ...)` still return **one page**.
For manual paging, pass the returned `next_page_token` to the next call and
stop at the final missing/null/empty token. A partial page or a sequence of
event dictionaries is not a replay bundle. Use
`await client.export_history(workflow_id, run_id)` or
`await handle.export_history()` for offline replay.

The [installed SDK 2.17.2 / native Server 3.9.2 pagination check](https://github.com/durable-workflow/sdk-python/pull/332#issuecomment-6877331165)
covered filtered lists, selected-run history, opaque resume, early close and
namespace refusal, with [independent acceptance](https://github.com/durable-workflow/sdk-python/pull/332#issuecomment-6877373099).

<!-- Source: docs/polyglot/rust-cloud-quickstart.md -->

# Rust Cloud Quickstart

:::caution Controlled early access

Durable Workflow Cloud is available through controlled early access. Use this
guide only after Cloud has provisioned a namespace and two role-scoped runtime
credentials. The generally available Rust journey does not require Cloud; start
with the [Rust SDK guide](./rust.md) or run `scripts/playground rust` against
the playground's default local runtime.

:::

The Sample App exposes one symmetric playground for PHP, Python, and Rust. This
page selects Rust and changes only the runtime target. The corresponding
[PHP](./php.md) and [Python](./python.md) SDK paths use the same command with
`php` or `python`; the [Sample App playground
contract](https://github.com/durable-workflow/sample-app/blob/main/README.md#symmetric-sdk-playground)
documents all three choices.

The playground resolves the current stable artifact versions from the Sample
App's machine-owned metadata. Use those generated versions instead of copying
version numbers into these commands or adding a separate SDK installation step.

## 1. Open the prepared Sample App

[Create a Codespace from the Sample App `main`
branch](https://codespaces.new/durable-workflow/sample-app?quickstart=1&ref=main),
wait for setup to finish, and open a terminal at the repository root. The
prepared image contains the SDK toolchains and `dw` required by the shared
playground.

Cloud provides the runtime URL, runtime namespace, and two role credentials.
Choose an application task queue, then export these placeholders only after
replacing them with the corresponding values:

```bash
export DURABLE_WORKFLOW_RUNTIME_URL='<provisioned-runtime-url>'
export DURABLE_WORKFLOW_RUNTIME_NAMESPACE='<provisioned-runtime-namespace>'
export DURABLE_WORKFLOW_CLIENT_TOKEN='<client-runtime-credential>'
export DURABLE_WORKFLOW_WORKER_TOKEN='<worker-runtime-credential>'
export DURABLE_WORKFLOW_TASK_QUEUE='<rust-task-queue>'
```

The runtime URL is the namespace runtime root returned by Cloud. It must be an
absolute HTTPS URL without a query, fragment, or terminal `/api`; the SDK and
CLI append their own API routes. The runtime namespace is the separate value
returned with that URL, not a name inferred from its path.

The client credential starts, describes, and reads workflows. The worker
credential registers, polls, heartbeats, and completes tasks. They must be
different role-scoped secrets even though this one command launches both child
processes. A Cloud administration key manages namespaces and credentials; do
not export or pass it as either runtime credential.

## 2. Run the Rust journey

Run the shared external-runtime contract:

```bash
scripts/playground rust --runtime managed \
  --runtime-url "$DURABLE_WORKFLOW_RUNTIME_URL" \
  --namespace "$DURABLE_WORKFLOW_RUNTIME_NAMESPACE" \
  --task-queue "$DURABLE_WORKFLOW_TASK_QUEUE"
```

The command scaffolds missing files under `.playground/rust` without replacing
caller-owned source. It prints the effective workflow type, activity type,
queue, worker command, client command, input, and expected result before it
starts anything. The worker receives only the worker credential; after its
exact registration becomes visible, the client receives only the client
credential.

The registration wait is bounded to 60 seconds. Do not treat process startup
alone as readiness. Continue only after the command prints a checkpoint shaped
like this (identities are illustrative):

```text
Worker ready: target=managed runtime_url=<provisioned-runtime-url> namespace=<provisioned-runtime-namespace> id=<worker-id> queue=<rust-task-queue> workflow_type=sample-app.playground.rust.authored-workflow activity_type=sample-app.playground.rust.authored-activity
```

The same command starts one workflow, waits up to 120 seconds for the SDK
client result, confirms `status=completed` with `dw`, and checks the required
workflow and activity history. One successful result looks like:

```text
Completed rust workflow <workflow-id>: {"greeting":"Hello, Durable Workflow, from the Sample App Rust playground","input":{"name":"Durable Workflow"},"activity_runtime":"rust","workflow_runtime":"rust"}
```

The final `Playground success` line repeats the runtime target, namespace,
queue, registered types, and expected result shape without credential values.
The command also writes `storage/app/playground-rust-evidence.json` with the
selected artifact versions, exact registration, workflow/run identity,
`completed` status, result, and history event types. Managed Waterline remains
the operator surface for the provisioned namespace; the managed journey does
not start a local Server or Waterline.

## Bounded diagnosis

If the `Worker ready` checkpoint does not appear within 60 seconds, start with
the effective contract printed above the error:

- **Queue mismatch:** the runner queries the exact value passed to
  `--task-queue`. Confirm that Cloud admits that queue and that the printed
  registration uses the same value; then rerun the same command.
- **Type mismatch:** the error names the workflow and activity types the worker
  must advertise. Compare them with the effective contract. If caller-owned
  files contain older hard-coded registrations, update them or prove the
  current scaffold in a new directory with
  `--source "$HOME/durable-rust-worker"`.
- **Credential-role mismatch:** authorization before registration points to
  the worker credential; authorization while describing or starting the run
  points to the client credential. Do not swap the values or replace either
  with a Cloud administration key.
- **Runtime mismatch:** both roles must use the exact provisioned runtime URL
  and runtime namespace. Remove a terminal `/api`; do not substitute the Cloud
  administration URL or a self-hosted Server URL.

If registration succeeds but completion fails, keep the printed workflow/run
identity. Inspect that selected run in Managed Waterline and compare its
pending workflow or activity type and task queue with the effective contract.
Fix that mismatch before starting another run. The retained evidence path and
bounded worker output identify whether the client result, durable status,
expected result, or required history check failed.

Return to the broader [Rust SDK guide](./rust.md), or use the generated [Rust
API reference](https://rust.durable-workflow.com/durable_workflow/) for
individual types and methods.

<!-- Source: docs/polyglot/invocable-php-handler.md -->

# PHP Invocable Activity Handler

The [invocable HTTP carrier](./invocable-carrier.md) lets the server POST a
leased activity task to an HTTPS endpoint instead of waiting for a long-poll
worker. The PHP helper `Workflow\V2\Support\InvocableActivityHandler` is the
reference implementation that an external PHP process uses to turn that
request into a result envelope the server can reconcile.

Use it to wire an activity handler into:

- an AWS Lambda or Google Cloud Function invoked over HTTPS
- a Laravel controller sitting behind a thin HTTP service
- a container exposing one POST endpoint per activity queue

The helper parses the carrier-neutral external task input envelope, looks up
the registered callable, enforces the lease deadline, and emits the
carrier-neutral external task result envelope — including the failure shapes
the server expects.

## Quick Start

Install the Durable Workflow package in the external process, register one
callable per activity handler name, and hand the request body to
`handle()`:

```php
use Workflow\V2\Support\InvocableActivityHandler;

$handler = new InvocableActivityHandler([
    'billing.charge-card' => static function (int $amount, string $currency): array {
        // Real charge-card logic. Must be idempotent per task id.
        return [
            'approved' => true,
            'amount' => $amount,
            'currency' => $currency,
        ];
    },
]);

$envelope = json_decode(file_get_contents('php://input'), associative: true);
$result = $handler->handle($envelope);

header('Content-Type: application/vnd.durable-workflow.external-task-result+json');
echo json_encode($result, JSON_UNESCAPED_SLASHES);
```

The handler receives the input envelope and returns the result envelope.
The carrier contract (HTTPS, POST, auth, timeouts, retry budget) is owned by
the invocable HTTP carrier. The PHP helper owns argument decoding, handler
dispatch, deadline enforcement, result encoding, and the failure taxonomy.

## Registering Handlers

The first constructor argument is a map keyed by the `task.handler` value the
server sends on the input envelope. That value is the `handler` field on the
matching entry in the external executor config:

```php
new InvocableActivityHandler(
    handlers: [
        'billing.charge-card' => [$billingService, 'chargeCard'],
        'billing.refund' => [$billingService, 'refund'],
        'ops.rotate-key' => static fn (string $keyId): array => $keys->rotate($keyId),
    ],
    carrier: 'billing-lambda',
    resultCodec: 'avro',
);
```

An input with a `task.handler` that is not registered produces a
`failed` result with `failure.kind = application`,
`classification = application_error`, and
`type = UnknownActivityHandler`. The configured activity retry policy still
applies on the server side.

The optional `carrier` name is echoed in `metadata.carrier` on every result
envelope so operators can tell which external runtime produced the response.
Use a stable, redaction-safe identifier (`billing-lambda`,
`ops-cloud-run`, `laravel-admin-api`).

The optional `resultCodec` controls how the helper serializes the return
value into `result.payload.blob`. It defaults to `avro`, which matches the
codec that PHP workers already use for durable payloads. The result codec
must be one the server's `CodecRegistry` knows about; `protobuf` is not
accepted and fails fast in the constructor.

## Result Envelope

On success, `handle()` returns the carrier-neutral success envelope:

```json
{
  "schema": "durable-workflow.v2.external-task-result",
  "version": 1,
  "outcome": {
    "status": "succeeded",
    "recorded": true
  },
  "task": {
    "id": "acttask_01HV7D3G3G61TAH2YB5RK45XJS",
    "kind": "activity_task",
    "attempt": 1,
    "idempotency_key": "attempt_01HV7D3KJ1C8WQNNY8MVM8J40X"
  },
  "result": {
    "payload": {
      "codec": "avro",
      "blob": "<base64-encoded payload>"
    },
    "metadata": {
      "content_type": "application/vnd.durable-workflow.result+json"
    }
  },
  "metadata": {
    "handler": "billing.charge-card",
    "carrier": "billing-lambda",
    "duration_ms": 42
  }
}
```

The `task.id`, `task.attempt`, and `task.idempotency_key` fields are copied
from the input envelope so the server can reconcile the result with the
original lease.

On failure, `handle()` returns the failure envelope. The `failure` block
carries the kind, classification, message, originating PHP type, stack
trace, and whether the failure is retryable. A deadline failure also
includes the `deadline` name and `expires_at` value:

```json
{
  "schema": "durable-workflow.v2.external-task-result",
  "version": 1,
  "outcome": {
    "status": "failed",
    "retryable": true,
    "recorded": true
  },
  "task": {"id": "...", "kind": "activity_task", "attempt": 1, "idempotency_key": "..."},
  "failure": {
    "kind": "timeout",
    "classification": "deadline_exceeded",
    "message": "Invocable activity task received after lease.expires_at.",
    "type": "ExternalTaskDeadlineExceeded",
    "stack_trace": null,
    "timeout_type": "deadline_exceeded",
    "cancelled": false,
    "details": {
      "deadline": "lease.expires_at",
      "expires_at": "2026-04-22T15:14:02.000000Z"
    }
  },
  "metadata": {"handler": "billing.charge-card", "carrier": "billing-lambda", "duration_ms": 3}
}
```

## Failure Taxonomy

| `failure.kind` | `classification` | Retryable | When it fires |
| --- | --- | --- | --- |
| `timeout` | `deadline_exceeded` | yes | Lease or input deadline already expired when the envelope arrived, or the handler returned after one expired during execution. |
| `decode_failure` | `decode_failure` | no | Arguments could not be decoded with the declared codec, a deadline string was unparseable, the handler raised `TypeError` / `ValueError` on its parameters, or the success payload could not be re-encoded with the configured result codec. |
| `application` | `application_error` | depends on thrown exception | The handler threw. `retryable` is `false` when the thrown exception implements `Workflow\Exceptions\NonRetryableExceptionContract`; otherwise `true`. The message is the exception message and `type` is the exception class. |
| `application` | `application_error` | no | `task.kind` is not `activity_task`, or `task.handler` is not registered in the map. |

The helper never returns a bare exception. Every code path produces a
structured result envelope so the invocable carrier can reconcile the
response deterministically.

## Deadlines And Idempotency

The input envelope carries the active `lease.expires_at` plus the declared
activity deadlines (`schedule_to_start`, `start_to_close`,
`schedule_to_close`, `heartbeat`). The helper checks all of them:

- Before dispatching the handler, any expired deadline short-circuits into a
  `timeout` failure with `details.deadline` naming which field expired. The
  registered callable is not invoked.
- After the handler returns, the helper re-checks the deadlines. A handler
  that ran for longer than its lease produces the same `timeout` failure
  and the return value is dropped from the envelope.

Because the carrier retries transport delivery and the runtime redelivers
leases that were not reported, the same `task.id` and
`task.idempotency_key` can arrive more than once. Handler code must be
idempotent. The idempotency key on every envelope is stable across retries
for the same attempt.

## Payload Codecs And External Storage

Argument payloads arrive on `payloads.arguments` with a `codec` field. The
helper uses the server's `CodecRegistry` to decode them. The only public v2
codec is `avro`; `json`, unknown codecs, and malformed or untagged blobs fail
closed. JSON remains the HTTP carrier document, not a durable payload codec.

When workflow inputs exceed the configured
[external payload storage](../features/external-payload-storage.md)
threshold, the server stores the bytes in the configured driver and sends
a reference envelope instead of an inline blob. Pass an
`ExternalPayloadStorageDriver` into the constructor so the helper can
resolve references before calling the handler:

```php
use Workflow\V2\Contracts\ExternalPayloadStorageDriver;
use Workflow\V2\Support\InvocableActivityHandler;

$handler = new InvocableActivityHandler(
    handlers: $handlers,
    carrier: 'billing-lambda',
    resultCodec: 'avro',
    externalStorage: $driver,
);
```

The driver must satisfy the same contract the server uses to write the
payload so the reference, hash, and codec all match. A reference that the
external process cannot resolve produces a `decode_failure` instead of a
silent empty payload.

## Production Wiring

A Laravel controller that hosts the handler behind a single POST route
looks like this:

```php
use Illuminate\Http\JsonResponse;
use Illuminate\Http\Request;
use Workflow\V2\Support\InvocableActivityHandler;

final class BillingActivityController
{
    public function __construct(private readonly InvocableActivityHandler $handler) {}

    public function __invoke(Request $request): JsonResponse
    {
        $result = $this->handler->handle($request->all());

        return new JsonResponse(
            data: $result,
            headers: ['Content-Type' => 'application/vnd.durable-workflow.external-task-result+json'],
        );
    }
}
```

Route the external executor config at the HTTPS URL that fronts this
controller and declare an `auth_ref` so the route is authenticated. The
loopback HTTP exemption only applies to developer iteration.

## Related Surfaces

- [Invocable HTTP Carrier](./invocable-carrier.md) — the server-side carrier
  contract that delivers input envelopes and reconciles result envelopes.
- [External Execution Surface](./external-execution.md) — the carrier-neutral
  product boundary, including the input and result envelope schemas the
  helper implements.
- [External Payload Storage](../features/external-payload-storage.md) — how
  oversized arguments or results are offloaded to a configured driver and
  represented as a verifiable reference envelope.
- [Worker Protocol](./worker-protocol.md) — the broader worker-plane contract
  that publishes the invocable carrier contract alongside poll-based handler
  shapes.

<!-- Source: docs/polyglot/rust.md -->

# Rust SDK

Use this guide for the general-access SDK surface, then continue to the
generated API reference for individual types and methods. No Cloud account is
required: the primary path uses the published Rust crate with a self-hosted
Server. Developers already enrolled in Durable Workflow Cloud controlled early
access can instead choose the clearly separate [managed-runtime
quickstart](./rust-cloud-quickstart.md).

The first-party Rust SDK is a workflow-authoring surface, not only a protocol
compatibility client. Rust authors deterministic workflows, activities, and
long-running worker services against the same durable execution model used by
PHP and Python. The async control-plane client starts, signals, queries,
updates, cancels, terminates, and awaits executions; the worker runtime replays
workflow history, runs workflow/activity/update handlers, reports worker and
activity heartbeats, and exchanges language-neutral payloads with a self-hosted
Server or Durable Workflow Cloud namespace runtime.

The [Rust documentation landing page](https://rust.durable-workflow.com/)
provides the SDK index and general entry points. For crate modules, structs,
traits, and methods, continue to the generated [Rust SDK API
reference](https://rust.durable-workflow.com/durable_workflow/).

Rust SDK 4.12.0 also provides [`RawRequest`](https://rust.durable-workflow.com/durable_workflow/struct.RawRequest.html)
for tools that need to call Server routes and inspect exact response bytes,
headers and status codes. Raw requests make one attempt. Use typed client
methods for application code.

The stable Rust SDK supports durable
timers, child workflows, activity retries and timeouts, local activities, worker sessions, sticky execution, signals, replayed query
handlers, cancellation and termination, server-enforced workflow deadlines,
typed side effects, version markers, updates, and typed terminal/replay
failures. It does not yet claim schedule management. Use the
[2.0 Capability Index](/docs/capabilities/) instead of assuming every SDK
has identical feature breadth.

## Try the local Sample App playground

For the shortest no-Cloud authoring journey, open the current Sample App
[`main` branch in GitHub Codespaces](https://codespaces.new/durable-workflow/sample-app?quickstart=1&ref=main)
and run:

<!-- docs-example id="sdk.rust.sample-app-playground" -->
```bash
scripts/playground rust
```

The local playground generates caller-owned workflow and activity source,
selects the current stable artifacts, and starts the published
Server and Waterline. It waits for a worker registration whose identity,
workflow type, activity type, and task queue match the generated contract before
starting the workflow. Success requires the expected completed result and
history; the terminal then prints the exact local Waterline run link and the
path to structured JSON evidence.

This intentionally small, transport-first scaffold uses `serde_json::Value`;
it is not the only recommended Rust application contract. Continue to the
crate's existing [typed input/output
example](https://github.com/durable-workflow/sdk-rust/blob/4b2440a5618ccc1c98960ea2e1e62127042c3590/examples/hello_world.rs),
[retry, timeout, heartbeat, and terminal-failure activity policy
example](https://github.com/durable-workflow/sdk-rust/blob/4b2440a5618ccc1c98960ea2e1e62127042c3590/examples/activity_options.rs)
and [local activity example](https://github.com/durable-workflow/sdk-rust/blob/3.1.0/examples/local_activities.rs).
The package,
repository example, and generated API reference below remain the direct paths
for users who do not want Sample App.

## Package and source

- [Crate on crates.io](https://crates.io/crates/durable-workflow)
- [Source repository](https://github.com/durable-workflow/sdk-rust)

Install the Rust SDK from the last passing qualified tuple. The exact requirement
is generated from the same machine-readable authority as the Server quickstart:

<!-- docs-example id="rust.sdk.install" -->
```bash
cargo add durable-workflow@=4.17.0
```

Or declare the same qualified requirement directly in `Cargo.toml`:

```toml
[dependencies]
durable-workflow = "=4.17.0"
```

The crate requires Rust 1.86 or newer. Its package metadata records the Server
version used to qualify that SDK release, plus the worker protocol default and
minimum and control plane 2. Compatibility with a newer Server is verified
separately. During deployment, use the protocol manifests and capabilities
advertised by `GET /api/cluster/info`.

Server negotiates worker-protocol headers within major `1`: a server
advertising `1.N` accepts a worker header `1.M` only when `M <= N`. Rust SDK
3.5.0 defaults to `X-Durable-Workflow-Protocol-Version: 1.20` and requires a
server advertising `1.20` or newer. For an older compatible Server, select
`WorkerProtocolVersion::V119` through the client builder's
`worker_protocol_version` setting. SDK 3.4.x uses `1.19` by default.
The opt-in cooperative cancellation profile requires `1.20` and refuses an
explicit `1.19` selection.

Negotiation fails closed. A missing or malformed header, a different major,
or a worker minor newer than the server's advertised minor is rejected. The
SDK and Server release versions are tracked independently. Their runtime
protocol manifests determine compatibility.

## Upgrade to Rust SDK 4.13 {#upgrade-to-4-13}

- **Static tokens.** Client construction now rejects a malformed shared,
  control or worker static token, including unused or masked options. The
  diagnostic names only the option. Check configured token values before
  upgrading. Empty tokens keep their existing precedence.
- **Local activity backoff.** Retries past the end of the backoff list repeat
  the final entry instead of retrying immediately, as Server-scheduled
  activities do.
- **Run activities.** `Client::get_run_activities` returns typed activity and
  attempt metadata, including heartbeat progress.

See the [4.13.0 changelog](https://github.com/durable-workflow/sdk-rust/blob/4.13.0/CHANGELOG.md).

## Read retained update outcomes {#read-update-outcome}

Rust SDK 4.15 adds `Client::update_outcome` and
`Client::update_outcome_receipt` to read an accepted update by ID without
submitting it again. These methods require `capabilities.update_outcome_reads`
(native Server 3.6.0 or later). `update_outcome` returns
`Error::UpdateStillPending` for an unfinished update. A completed null result
is `Ok(Value::Null)`. Receipt reads preserve the explicit `update_status`.
See the [Client API](https://rust.durable-workflow.com/durable_workflow/struct.Client.html#method.update_outcome)
for the Rust interface.

Rust SDK 4.16 uses the Server's explicit
`limits.max_workflow_task_completion_bytes` budget for completion planning.
Older producers without that field keep the conservative ordinary cap. See
[completion planning](./server-config-reference.md#rust-completion-planning)
for the qualified native 3.9.1 tuple and payload-placement limits.

## Condition wait deadlines and worker upgrades {#condition-wait-deadlines}

Rust SDK 4.11.2 measures an ungrouped `wait_condition` timeout from its first
open. When a wake finds the predicate still false, the wait reopens with its
remaining time, so unrelated signals do not extend its deadline. Waits in
`parallel` or `select` groups still reopen with the authored timeout for
compatibility with PHP Server 2.11.8.

Workers on 4.11.1 or earlier refuse histories that record a shorter reopen
timeout. Keep those runs on workers using 4.11.2 or later during rolling
deployments and rollback. See the
[tagged 4.11.2 compatibility notes](https://github.com/durable-workflow/sdk-rust/blob/4.11.2/CHANGELOG.md#4112).

Rust SDK 4.12.0 also replays repeated opens recorded while a condition wait
was still open, including `parallel` or `select` condition groups woken by one
signal on PHP Server 2.x. Each member consumes its own reopen, even with a
timer between members. A timeout recorded before the next open still ends
that logical wait. Earlier workers can refuse these histories. Keep affected
runs on 4.12.0 or later, and test replay with the intended older worker before rollback. See
the [tagged 4.12.0 compatibility notes](https://github.com/durable-workflow/sdk-rust/blob/4.12.0/CHANGELOG.md#4120).

## Workflow lists and complete run history {#workflow-lists-and-history}

These APIs require Rust SDK **4.6.0 or later**; check `Cargo.lock` because the separate installation tuple above may select an older crate. The [normal installed-package acceptance](https://github.com/durable-workflow/sdk-rust/issues/122#issuecomment-6363352910) verifies three list pages, six history pages, a 128 KiB payload, export/replay and the read-only example against published Server 2.11.0. Offline testing helpers require 4.5.0; these listing/history methods and iterators require 4.6.0. This does not change the selected Server/SDK tuple.

`Client::list_workflows(WorkflowListOptions)` and `Client::get_workflow_history(workflow_id, run_id, WorkflowHistoryOptions)` each return **one page**. Resume manually with its opaque `next_page_token`, or use lazy iterators for complete reads. The methods preserve Server validation, filters and namespace scope; they add no query fields or grammar. For example, find an open run using a registered keyword search attribute:

```rust
use durable_workflow::WorkflowListOptions;
async fn find(client: durable_workflow::Client) -> durable_workflow::Result<()> {
let mut matches = client.iter_workflows(
    WorkflowListOptions::new()
        .workflow_type("payments")
        .status("running")
        .query("OrderRef = 'order-42'")
        .page_size(2),
);
match (matches.next().await?, matches.next().await?) {
    (Some(run), None) => println!("open run: {} / {}", run.workflow_id, run.run_id),
    (None, _) => println!("no open run"),
    _ => println!("multiple open runs; refine the query"),
}
Ok(())
}
```

The Server's `status=running` filter selects the active **status bucket**, including raw `pending` and `waiting` statuses. `WorkflowSummary` preserves `status`, `status_bucket`, `is_terminal`, search attributes and unknown metadata in `raw`. Search attributes are not unique: a filter may match both a parent and children that inherited its value. Handle zero, one and multiple matches as above. Create the client in the intended namespace before constructing the iterator; namespace and filters stay fixed across every token.

Read an entire selected run without loading every page at once:

```rust
use durable_workflow::WorkflowHistoryOptions;
async fn audit(client: durable_workflow::Client) -> durable_workflow::Result<()> {
let mut events = client.iter_workflow_history(
    "payment-42", "selected-run-id", WorkflowHistoryOptions::new().page_size(100),
);
while let Some(event) = events.next().await? {
    println!("{}", event.event_type);
}
Ok(())
}
```

`iter_workflows(options)` yields each `WorkflowSummary` through `next().await`. Both iterators support `into_stream()` for `futures_util::TryStreamExt`; pin the returned stream before polling. Construction performs no I/O; the next page is fetched only after the current page is exhausted. A null/empty token ends iteration; an empty page with a token is followed. Repeated or cyclic tokens return `Error::Pagination` with bounded token state. A page failure preserves the original typed error, then that iterator ends. Handle the error: a partial read is not a complete result. Collecting a stream still stores every collected item.

History iteration keeps the original workflow/run identity and enumerates available events; it neither follows continue-as-new nor subscribes indefinitely. Event metadata and payload wire envelopes are retained. For offline replay use `Client::export_workflow_history` with `testing::WorkflowHistory::from_export`, rather than passing a partial history page. See [offline replay](#rust-replay-export).

Published Server 2.11 uses 50 list rows by default (maximum 200) and 100 history events (maximum 1,000) per page. Page sizes pass unchanged to Server validation. Stop early when enough results are available, choose a suitable page size and apply an application timeout. Pagination adds no snapshot isolation while runs change concurrently. See the [released read-only example](https://github.com/durable-workflow/sdk-rust/blob/4.6.0/examples/workflow_pagination.rs), [Client API](https://rust.durable-workflow.com/durable_workflow/struct.Client.html) and [pagination types](https://rust.durable-workflow.com/durable_workflow/pagination/index.html).

## Metrics and Prometheus export {#rust-sdk-metrics}

These APIs require Rust SDK **4.7.0 or later**. Check `Cargo.lock`: the separate installation tuple above may select an older crate. The [normal installed acceptance](https://github.com/durable-workflow/sdk-rust/pull/131#issuecomment-6364178456) verifies four HTTP controls, all six metrics from a real published Server 2.11.0 worker, and the released exporter example. This feature minimum does not promote the selected tuple.

The default recorder is a **no-op**. Supply an `Arc<dyn metrics::MetricsRecorder>` implementing `increment(name, count, tags)` and `record(name, seconds, tags)`. The client, its clones and newly constructed workers inherit that recorder. `Worker::metrics_recorder` overrides worker poll/task observations; client request observations keep the client recorder. Callbacks must be fast and handle exporter failures locally. Unwinding recorder panics are isolated from SDK results; blocking exporters or process aborts cannot be repaired by this hook.

Each observed client request, worker poll or claimed-task dispatch increments one counter by one and records one nonnegative monotonic duration in seconds. The six stable recorder names are:

| Counter | Duration histogram |
| --- | --- |
| `durable_workflow_client_requests` | `durable_workflow_client_request_duration_seconds` |
| `durable_workflow_worker_polls` | `durable_workflow_worker_poll_duration_seconds` |
| `durable_workflow_worker_tasks` | `durable_workflow_worker_task_duration_seconds` |

For the counter/duration pairs above, client tag keys are exactly `method`, `route`, `plane`, `status_code`, `outcome`; worker keys are `task_kind`, `task_queue`, `outcome`. Reviewed route placeholders omit IDs, query strings, payloads and credentials; unknown paths use `unknown`. Queue names are application-controlled: use a bounded set, not one queue per run.

Client outcomes include `ok`, `http_error`, `bridge_rejected`, `TransportError`, `TimeoutError`, `SdkError` and `pending` for a cancelled future. `status_code` keeps an observed decimal HTTP status, otherwise `none`. Rust exception outcome names need not match another SDK’s names.

One client observation includes internal transport/storage-admission retries and response/payload handling; application/worker-managed calls are separate observations. A lazy iterator observes each fetched page. Worker polls include managed retry/backoff and distinguish `task`, `empty`, `backpressure`, `error`. Task outcomes `completed`, `failed`, `expired` describe dispatch processing, not the workflow’s terminal state. These are process observations, not durable counts, billing units or schedule-to-start latency; abrupt process loss can omit a final sample.

Rust SDK 4.16 adds the seventh instrument,
`durable_workflow_worker_task_schedule_to_start_seconds`, with only
`task_kind` and `task_queue` tags. It records Server-measured eligibility-to-claim
delay in seconds when the producer advertises `claimed_task_ready_age` and
supplies valid `ready_age_ms`. Missing, invalid or unsupported data produces no
sample; the SDK never invents a zero. The
[published 4.16/native 3.9.1 controls](https://github.com/durable-workflow/sdk-rust/pull/258#issuecomment-6854288671)
(linux/amd64, SQLite) qualify all seven metrics, workflow/activity ready-age
observations and the Prometheus adapter.

For the official Prometheus adapter, enable the optional `metrics-prometheus` Cargo feature. This example uses the exact released dependency recipe:

```toml
[dependencies]
durable-workflow = { version = "=4.7.0", features = ["metrics-prometheus"] }
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
```

Save the [released example](https://github.com/durable-workflow/sdk-rust/blob/4.7.0/examples/metrics_prometheus.rs) as `examples/metrics_prometheus.rs`. These exact source bytes were compiled and run in the normal installed-package acceptance; the example prints health-request metrics:

```rust
//! Preview application-owned Prometheus export; serve `encode()` from your HTTP metrics route.
use durable_workflow::{metrics::PrometheusMetrics, Client};
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let recorder = Arc::new(PrometheusMetrics::new()?);
    let client = Client::builder(std::env::var("DURABLE_WORKFLOW_SERVER_URL")?)
        .token(std::env::var("DURABLE_WORKFLOW_AUTH_TOKEN").ok())
        .metrics_recorder(recorder.clone())
        .build()?;
    client.health().await?;
    // Workers inherit this client recorder; Worker::metrics_recorder overrides
    // only worker poll/task observations. Applications own the scrape endpoint.
    print!("{}", String::from_utf8(recorder.encode()?)?);
    Ok(())
}
```

Run it against your self-hosted Server:

```bash
DURABLE_WORKFLOW_SERVER_URL=http://localhost:8080 \
DURABLE_WORKFLOW_AUTH_TOKEN=your-token \
cargo run --features metrics-prometheus --example metrics_prometheus
```

`PrometheusMetrics::new()` owns a dedicated registry backed by the official Prometheus Rust client 0.14. Serve `encode()` bytes from your application’s metrics route with `Content-Type: text/plain; version=0.0.4`. The adapter creates no listener or global registry. Your application owns scrape access/authentication, network exposure, resources and shutdown. Counter exports append `_total`; histograms use standard bucket/sum/count series. `registry()` allows application-owned collection and additional metrics.

For OpenTelemetry metrics, implement the recorder using your existing meter: count-unit (`1`) counters and second-unit (`s`) histograms with the exact names/tags above. Provider, aggregation, resource labels and exporter lifecycle remain application-owned. This is separate from execution tracing. See the [recorder API](https://rust.durable-workflow.com/durable_workflow/metrics/trait.MetricsRecorder.html), [Prometheus adapter API](https://rust.durable-workflow.com/durable_workflow/metrics/struct.PrometheusMetrics.html) and [released metrics contract](https://github.com/durable-workflow/sdk-rust/blob/4.7.0/README.md#client-and-worker-metrics).

## Worker task concurrency {#worker-task-concurrency}

The bounded worker lanes below require Rust SDK **4.3.1 or newer**. Check the crate in `Cargo.lock` before using them: the installation pin above belongs to a separate qualified artifact tuple and may select an older SDK. The [published-package acceptance](https://github.com/durable-workflow/sdk-rust/issues/107#issuecomment-6360719863) verifies three configured slots, a fourth callback admitted after a slot is released, and seven completed workflows including a handled activity failure. This guide does not change the Server protocol requirements above.

`Worker::max_concurrent_workflow_tasks(...)` and `Worker::max_concurrent_activity_tasks(...)` each default to **10**. `run()` and `run_until()` use independent bounded workflow and remote-activity lanes. Set a limit to **1** for serial execution within that task family. For example, configure an existing client before registering your application handlers:

```rust
use durable_workflow::Worker;

let mut worker = Worker::new(client.clone(), "payments")
    .max_concurrent_workflow_tasks(1)
    .max_concurrent_activity_tasks(1);
```

Register the workflow and activity handlers, then start `run()` or `run_until()` as usual. Setting both limits to one still permits a workflow task and a remote activity task to run at the same time. Queries keep a separate polling lane. Worker clones share the configured capacity, including simultaneous `run_once()` calls; cloning does not create another independent allowance.

Heartbeat availability follows actual claimed tasks, including input/history loading, callback execution and settlement. An empty long poll leaves capacity available. Success, failure, refusal and cancellation release the local slot. Inline local activities occupy their **workflow-task** slot and bypass the remote-activity pool, so the activity limit does not separately cap inline local execution across workflows.

Handlers are async futures on Tokio and must yield for other work to progress. Use `tokio::task::spawn_blocking` or a separate process for blocking work, with an explicit cancellation and shutdown boundary. Raising concurrency does not provide CPU isolation or make a detached thread cancellable. See the [Worker API](https://rust.durable-workflow.com/durable_workflow/struct.Worker.html) for the installed SDK's controls.

## Received activity retry budgets {#activity-retry-info}

Rust SDK **4.8.0 or newer** exposes `ActivityContext::retry_info()` in service activity handlers. Check the installed crate in `Cargo.lock`: the separately selected installation pin above can be older. The returned `ActivityRetryInfo` preserves the original `attempt_number()` and received `policy()` JSON, including unknown fields. It is an immutable snapshot of this invocation, independent of mutable compatibility fields on the context.

Use `attempt_limit()` and `is_last_configured_attempt()` to interpret the received budget:

| Received retry policy | `ActivityAttemptLimit` | Final configured attempt |
| --- | --- | --- |
| Object with positive integer `max_attempts: n` | `Bounded(n)`, including the first attempt | `Some(true)` once the original attempt reaches or exceeds `n`, otherwise `Some(false)` |
| Object with `max_attempts: null` | `Unlimited` | `Some(false)` |
| Omitted policy, root null, missing field or invalid limit | `Unknown` | `None` |

The helper also returns `None` for an invalid zero attempt number. A runtime's scheduling default does not establish received metadata: the published Server 2.11.0 no-authored-policy control omits the snapshot and therefore reports `Unknown`, not an inferred `Bounded(1)`. `policy()` distinguishes an omitted policy from a received JSON null without treating either as unlimited. The [normal installed acceptance](https://github.com/durable-workflow/sdk-rust/issues/135#issuecomment-6366200791) includes this control and the recipe below.

For a final-configured-attempt fallback or application alert, register a handler on your existing Worker:

```rust
use durable_workflow::{json, Worker};

pub fn register_charge(worker: &mut Worker) {
    worker.register_activity("charge", |ctx, _| async move {
        let retry = ctx.retry_info();
        if retry.is_last_configured_attempt() == Some(true) {
            // Select a final-attempt fallback or record an application alert.
        }
        Ok(json!({"attempt": retry.attempt_number()}))
    });
}
```

`Some(false)` is not a promise of another execution. Non-retryable failure, cancellation, workflow closure and elapsed deadlines can end work before its configured budget is used. The helper does not change retry admission or grant extra time.

Local activities and direct `testing::WorkflowEnvironment::run_activity` calls have no received Server snapshot and report unknown policy. Fresh `run_workflow` tests expose the modeled snapshot recorded in their generated history. These test semantics do not prove a deployed Server sends a policy. See [ActivityRetryInfo](https://rust.durable-workflow.com/durable_workflow/struct.ActivityRetryInfo.html) and [ActivityAttemptLimit](https://rust.durable-workflow.com/durable_workflow/enum.ActivityAttemptLimit.html).

## Stable activity identity and downstream idempotency {#activity-identity}

Rust SDK **4.9.0 or newer** exposes the original received identity through `ActivityContext`. Check the installed crate in `Cargo.lock`; the separately selected installation pin above can be older. These accessors return immutable `Option<&str>` values captured from the original claim, including worker-session activities.

| Accessor | Identity and use |
| --- | --- |
| `activity_execution_id()` | One logical activity execution across retries; use it to deduplicate an external operation. |
| `workflow_id()` | The owning workflow **instance** ID; use it for business correlation. |
| `run_id()` | The owning workflow **run** ID; use it to correlate this particular run. |

Derive a key for a particular operation and pass it to an idempotent downstream API or durable effect ledger. This handler only constructs the key: it does not charge a payment or implement downstream deduplication. It refuses missing activity identity before any effect; an application can instead supply its own explicitly stable business key.

```rust
use durable_workflow::{json, ActivityError, Worker};

pub fn register_charge(worker: &mut Worker) {
    worker.register_activity("charge", |ctx, _| async move {
        let execution_id = ctx.activity_execution_id().ok_or_else(|| {
            ActivityError::new("MissingActivityIdentity", "Server activity identity is required")
                .non_retryable(true)
        })?;
        // Pass this key to an idempotent downstream API or durable effect ledger.
        let key = format!("payments:charge:{execution_id}");
        Ok(json!({
            "idempotency_key": key,
            "workflow_id": ctx.workflow_id(),
            "run_id": ctx.run_id(),
        }))
    });
}
```

When an older Server omits these fields, they remain `None`. Never replace a missing logical activity ID with a per-attempt task ID, attempt ID or attempt number: that would produce a different key on retry. Inline local activities can receive real parent workflow/run IDs, but have no Server activity execution ID before the Server records their command. Server-free activity tests have no received claim identity.

A stable key makes duplicate effects preventable only when the downstream system honors that key. Activity delivery remains at-least-once; this API does not make external work exactly-once, restore heartbeat progress or implement resumable activity checkpoints.

See [ActivityContext identity accessors](https://rust.durable-workflow.com/durable_workflow/struct.ActivityContext.html#method.activity_execution_id) and the [normal installed 4.9.0 receipt](https://github.com/durable-workflow/sdk-rust/issues/139#issuecomment-6367230500) for ordinary/cooperative, local and worker-session controls. See [execution guarantees](../constraints/execution-guarantees.md#service-activity-identity) for the shared idempotency contract.

## Bounded failure reports {#bounded-failure-reports}

These reporting limits require an actually installed **Rust SDK 4.10.1 or newer**. Check the worker’s `Cargo.lock`; a newer client or source checkout does not update an older selected worker. The limits apply to outgoing failure reporting, not incoming history or unrelated business payloads.

| Field or envelope | Maximum |
| --- | --- |
| UTF-8 message, including its marker | 62 KiB |
| Stack where supported | 128 KiB |
| Combined serialized optional details, diagnostics and metadata | 64 KiB |
| Complete encoded failure request body | 1 MiB |

The effective message cap reserves 2 KiB below the original 64 KiB budget for published Server timeline prefixes. JSON escaping, truncation markers, claim identity and all other fields count toward the encoded body. A lower known transport cap can reduce these maxima further. Small reports retain their original content.

Oversized text carries a visible marker. Existing `diagnostics.sdk_failure_truncation`, where the schema supports it, records original byte sizes and omitted fields. Workflow/query-task schemas without diagnostics and plain command messages carry those statistics in the message marker. Avro details remain a complete valid envelope/runtime reference or are omitted whole and marked; base 64 is never cut into a corrupt value.

The original exception type, retry decision and required claim identity are preserved. Low-level consumers can use `Client::fail_activity_task_with_failure` and `Client::fail_workflow_task_with_failure` with the published failure schema. `Error::FailureReporting` explicitly refuses a report whose mandatory data cannot fit; it does not decide the application’s retry outcome or fabricate a smaller identity.

Local failures are bounded before workflow code observes them, including the published Server’s whitespace/NUL normalization. The final request check does not rewrite an already observed local message or another business payload; reporting refuses if those fixed inputs cannot fit.

These SDK reporting bounds do not establish a Core derived-summary storage fix, another SDK’s delivery, embedded behavior or Server image adoption. Older workers can still fail to report an oversized failure. See the [published-source policy](https://github.com/durable-workflow/sdk-rust/blob/dd58dd146f1337c3be24ad18fe65e3bb31d78b58/README.md#failure-report-limits) and [shared failure guidance](../failures-and-recovery.md#rust-failure-report-limits).

## Activity retry backoff {#activity-retry-backoff}

`ActivityRetryPolicy::new(max_attempts)` counts total attempts, including the
first. With `exponential_backoff(initial, coefficient, maximum_interval)`,
passing `None` selects no interval cap. Pass `Some(maximum)` to cap the
intervals explicitly. The default policy remains one attempt with an empty
backoff list.

For nine total attempts, an initial one-second delay, coefficient two and
`None` encode `1, 2, 4, 8, 16, 32, 64, 128` seconds for the eight retries.
The finite attempt budget and original schedule-to-close deadline, when
configured, still bound retries. Arithmetic or protocol overflow returns
`ActivityOptionsErrorKind::BackoffOverflow` during validation.

Replay preserves the complete recorded delay list and compares it and the
other known policy fields exactly. Changing only the maximum is compatible
when the encoded list stays identical. A maximum that changes the list is a
policy change, even if a changed entry was unused. Adding a 60-second cap to
the example changes its list and can refuse replay with
`activity_retry_policy_mismatch`.

See the [Rust retry policy recipe](https://github.com/durable-workflow/sdk-rust/blob/7325699d7f09e57293d66169541d26efdd484f82/README.md#activity-retry-policies)
and the published [ActivityRetryPolicy API](https://rust.durable-workflow.com/durable_workflow/struct.ActivityRetryPolicy.html#method.exponential_backoff).

## Local activities

Rust SDK 3.1.0 adds inline local execution on workers configured with
`Worker::local_activities(true)`. Register callbacks with the ordinary activity
registration methods, then use the workflow context's local activity methods.
See the [runnable example](https://github.com/durable-workflow/sdk-rust/blob/3.1.0/examples/local_activities.rs)
and [LocalActivityOptions reference](https://rust.durable-workflow.com/durable_workflow/struct.LocalActivityOptions.html)
for retry and timeout configuration.

Server records the terminal result for cold replay. Uncommitted local effects
can execute again after worker loss and must be idempotent. Async callbacks
must yield to Tokio. Blocking work needs separate process supervision.
Inline local execution and prepared cooperative local supervision use separate
worker profiles.

## Worker sessions

Enable `Worker::worker_sessions(true)` and declare the requirements your worker
satisfies with `capabilities(...)`. Route remote activities with
`in_worker_session(WorkerSessionOptions::new(session_id))`. The activity context
provides the shared handle through `worker_session()`. Explicit handles support
`create()`, `renew()` and `close(reason)` after worker registration.

Bound worker session count with `max_concurrent_worker_sessions(...)` and session
activity concurrency with `max_concurrent_activities(...)`. Renew idle handles
explicitly. Activity heartbeats also renew the holder lease, preserving the
original absolute TTL. An async callback that yields to Tokio is dropped when
its locally observed lease or TTL expires without requiring application heartbeats.

Session resources live in the holder process. A replacement must rebuild them,
and interrupted external effects need idempotency and attempt fencing. Committed
results replay without rerunning the callback. Graceful shutdown closes held
sessions before deregistration. Local activities cannot use session routing.

Session support starts with Rust SDK 3.2.0. Use Server 2.5.1 or newer. The
[runnable example](https://github.com/durable-workflow/sdk-rust/blob/3.2.0/examples/worker_sessions.rs)
reuses a process-local cache across two typed activities and prints its generation.
See [WorkerSessionOptions](https://rust.durable-workflow.com/durable_workflow/struct.WorkerSessionOptions.html)
and [WorkerSession](https://rust.durable-workflow.com/durable_workflow/struct.WorkerSession.html)
for configuration and lifecycle methods.

## Bounded sticky execution

Enable `Worker::sticky_cache(StickyCacheOptions::new(capacity))` to retain
immutable encoded history for subsequent replay. The cache is disabled by
default. Set `max_history_bytes(...)` and `ttl(...)` to bound retained history
and its lifetime. Allow additional memory for decoding and replay.

The cache uses workflow, run and build identity. Worker replacement, expiry,
eviction or an invalid history cursor falls back to complete durable history.
Terminal workflows and stopped workers discard their entries. Workflow code
must replay correctly with an empty cache.

Support starts with Rust SDK 3.4.0 and the qualified Server baseline. See the
[configuration reference](https://rust.durable-workflow.com/durable_workflow/struct.StickyCacheOptions.html),
[runnable example](https://github.com/durable-workflow/sdk-rust/blob/3.4.0/examples/sticky_execution.rs)
and [portable worker-affinity guide](./portable-worker-affinity.md).

## Prepare the released repository example

The repository's
[`hello_world` example](https://github.com/durable-workflow/sdk-rust/blob/4b2440a5618ccc1c98960ea2e1e62127042c3590/examples/hello_world.rs)
registers a Rust worker, starts a workflow, sends a signal, runs an activity,
reports an activity heartbeat, and waits for the completed result. Because that
example runs the application client and worker in one process, the no-account
path below connects it to a self-hosted Server. A provisioned Cloud namespace
is an optional secondary runtime with separate role-scoped credentials.

For a reproducible source exercise, obtain the exact crate source recorded by
the qualified tuple. The value below is generated from that machine-readable
authority. The example directory is absolute so either connection path can
enter it directly:

<!-- docs-example id="rust.sdk.repository-source" -->
```bash
export DURABLE_WORKFLOW_RUST_VERSION=4.17.0
export DURABLE_WORKFLOW_RUST_EXAMPLE_DIR="$PWD/durable-workflow-rust-${DURABLE_WORKFLOW_RUST_VERSION}"
git clone --depth 1 --single-branch --branch "$DURABLE_WORKFLOW_RUST_VERSION" \
  https://github.com/durable-workflow/sdk-rust.git "$DURABLE_WORKFLOW_RUST_EXAMPLE_DIR"
```

## Run the combined example with self-hosted Server

The unmodified `hello_world` example accepts one token for a self-hosted
Durable Workflow Server whose authentication policy allows the same credential
to make workflow commands and poll for work:

<!-- docs-example id="rust.sdk.self-hosted" -->
```bash
cd "$DURABLE_WORKFLOW_RUST_EXAMPLE_DIR"
DURABLE_WORKFLOW_SERVER_URL=http://localhost:8080 \
DURABLE_WORKFLOW_TOKEN=dev-token \
cargo run --example hello_world
```

The example uses the `default` namespace provisioned by the local Server
quickstart. Use `TASK_QUEUE` to override its default `rust-workers` task queue.
No Cloud enrollment or Cloud credential is involved in this path.

## Optional secondary path: connect to Durable Workflow Cloud

For the shortest managed-runtime proof, use the [Rust Cloud
quickstart](./rust-cloud-quickstart.md). It runs the same symmetric Sample App
playground available to PHP and Python, selects current stable artifacts
dynamically, verifies exact worker readiness, and waits for a completed result.

The lower-level combined SDK example below is useful when adapting an existing
Rust process. Export the values returned when Cloud provisions the namespace
and creates its two runtime credentials:

See [Cloud Managed Runtime](/docs/polyglot/cloud-control-plane) for the
managed connection boundary.

<!-- docs-example id="rust.sdk.cloud.environment" -->
```bash
export DURABLE_WORKFLOW_RUNTIME_URL='https://your-runtime-url'
export DURABLE_WORKFLOW_RUNTIME_NAMESPACE='orders'
export DURABLE_WORKFLOW_CLIENT_TOKEN='dwr_client_credential'
export DURABLE_WORKFLOW_WORKER_TOKEN='dwr_worker_credential'
```

In `$DURABLE_WORKFLOW_RUST_EXAMPLE_DIR/examples/hello_world.rs`, replace the
existing `server_url`, `token`, and `Client::builder(...)` setup with this
split-token builder configuration:

<!-- docs-example id="rust.sdk.cloud.client" -->
```rust
let runtime_url = std::env::var("DURABLE_WORKFLOW_RUNTIME_URL")
    .expect("DURABLE_WORKFLOW_RUNTIME_URL must be set");
let runtime_namespace = std::env::var("DURABLE_WORKFLOW_RUNTIME_NAMESPACE")
    .expect("DURABLE_WORKFLOW_RUNTIME_NAMESPACE must be set");
let client_token = std::env::var("DURABLE_WORKFLOW_CLIENT_TOKEN")
    .expect("DURABLE_WORKFLOW_CLIENT_TOKEN must be set");
let worker_token = std::env::var("DURABLE_WORKFLOW_WORKER_TOKEN")
    .expect("DURABLE_WORKFLOW_WORKER_TOKEN must be set");

let client = Client::builder(runtime_url)
    .namespace(runtime_namespace)
    .control_token(Some(client_token))
    .worker_token(Some(worker_token))
    .build()?;
```

Then run the example normally:

<!-- docs-example id="rust.sdk.cloud.run" -->
```bash
cd "$DURABLE_WORKFLOW_RUST_EXAMPLE_DIR"
cargo run --example hello_world
```

The example's workflow start, signal, describe, and result calls use
`control_token`; its `Worker` registration, polling, heartbeat, and completion
calls use `worker_token`. Both roles use the same Cloud-provided runtime URL,
runtime namespace, and task queue, but they do not reuse a credential. Do not
replace either Cloud credential with `.token(...)`: that method is the generic
single-token fallback used by the self-hosted configuration above.

## Start with server-enforced workflow timeouts

The Rust SDK provides `WorkflowStartOptions` and
`Client::start_workflow_with_options` for workflow deadlines that the server
enforces even after the starting process exits. Execution timeout covers the
whole workflow instance, including continue-as-new runs; run timeout covers one
run and is recomputed when a new run begins.

<!-- docs-example id="rust.workflow-start-timeouts" -->
```rust
use durable_workflow::{json, Client, Result, WorkflowStartOptions};

async fn start(client: &Client) -> Result<()> {
    let handle = client.start_workflow_with_options(
        "orders.await-payment",
        "orders",
        "order-42",
        WorkflowStartOptions::new()
            .execution_timeout_seconds(300)
            .run_timeout_seconds(30),
        json!([{"order_id": "order-42"}]),
    ).await?;

    println!("workflow={} run={:?}", handle.workflow_id, handle.run_id);
    Ok(())
}
```

Both values are seconds, must be positive, and the run timeout cannot exceed
the execution timeout. The existing `Client::start_workflow` convenience
method uses `WorkflowStartOptions::default()`: a 3600-second execution timeout
and a 600-second run timeout. See [Timeouts](/docs/features/timeouts) for
the server's deadline and continue-as-new semantics.

These are workflow policy, not HTTP or result-polling timeouts. In particular,
`WorkflowResultOptions::timeout` only stops the local `result()` call from
waiting. It does not close, cancel, or otherwise change the workflow run. The
caller can inspect the returned identity and wait again. An execution or run
deadline configured with `WorkflowStartOptions` is durable server state; when
it expires, the server closes the run with a terminal `timed_out` outcome.

## Deterministic parallel groups

`WorkflowContext::parallel` and its `join` alias compose activities, child
workflows, timers, mixed groups, and nested groups without adding a new Server
command. Constructors on `ParallelOperation` defer every leaf until the entire
tree is known:

```rust
use durable_workflow::{json, ChildWorkflowOptions, ParallelOperation};
use std::time::Duration;

let results = ctx.parallel(vec![
    ParallelOperation::activity("load-profile", json!(["customer-42"])),
    ParallelOperation::group(vec![
        ParallelOperation::child_workflow(
            "quote-shipping",
            ChildWorkflowOptions::new("shipping-workers"),
            json!(["customer-42"]),
        ),
        ParallelOperation::timer(Duration::from_secs(1)),
    ]),
]).await?;
```

Every leaf carries the shared stable group identity plus its complete
outer-to-inner path. Successful results retain nested input shape and order.
`Error::ParallelFailed` preserves the typed leaf cause, deterministic member
path, group path, and completed siblings. Pending and completed histories replay
without rescheduling; exact duplicates and late completions do not change the
chosen positional outcome.

## Durable first-completion selection

`WorkflowContext::select` and `select_keyed` start activities, child workflows,
timers, signals, condition waits, and nested ordinary groups together, then
resume from the winner recorded by the runtime:

```rust
use durable_workflow::{json, ParallelOperation, SelectionKey};
use std::time::Duration;

let selected = ctx.select_keyed(vec![
    ("resolver", ParallelOperation::activity("resolve-request", json!([request_id]))),
    ("input", ParallelOperation::signal("resolution.received")),
    ("deadline", ParallelOperation::timer(Duration::from_secs(2))),
]).await?;

if selected.key == SelectionKey::Name("deadline".to_string()) {
    if let Some(resolver) = selected.handle(&SelectionKey::Name("resolver".to_string())) {
        resolver.cancel().await?;
    }
}
```

`SelectionResult` preserves the stable key/index, operation kind and identity,
typed value or failure, and a `DurableOperationHandle` for every member. A loser
continues unless workflow code later awaits `handle.await_result()` or awaits
`handle.cancel()`. The cancellation future resolves to unit; only committed
`SelectionOperationCancelled` history proves cancellation won. Replay advances
past an unmarked request, and `await_result()` still returns a completion that
committed first. Restart and replay consume the persisted winner; later or
duplicate terminal events cannot change it.

## Saga compensation

Register a compensation only after the corresponding forward activity has
completed, then give the forward result to `Saga::finish`:

```rust
let mut saga = ctx.saga();
let outcome = async {
    let flight = ctx.activity("trip.reserve-flight", json!([])).await?;
    saga.add_compensation("trip.cancel-flight", json!([flight]))?;

    let hotel = ctx.activity("trip.reserve-hotel", json!([])).await?;
    saga.add_compensation("trip.cancel-hotel", json!([hotel]))?;

    ctx.activity("trip.charge", json!([])).await?;
    Ok(json!({"status": "booked"}))
}.await;

saga.finish(outcome).await
```

Handled failure runs the existing activity command in
reverse registration order, one compensation at a time. Compensation stops at
its first failure. `Error::SagaCompensationFailed` preserves both typed errors,
the compensation activity type, and its registration order through restart and
replay.
Terminal `cancel_workflow` does not resume workflow code to run compensations.

## Deterministic side effects and version markers

The Rust SDK records small non-deterministic values
with `WorkflowContext::side_effect` and derives deterministic UUIDv4 values
with `WorkflowContext::uuid_v4`. A cold replay decodes the recorded value
instead of invoking the callback again; reordered, missing, duplicate, or
codec-incompatible markers return a typed `Error::NonDeterministicReplay`.

Use `WorkflowContext::get_version(change_id, min_supported, max_supported)` to
keep old and new workflow branches replay-compatible during a rollout. New
runs record `max_supported`; existing runs reuse the durable marker.
`patched(change_id)` provides the boolean rollout form, and
`deprecate_patch(change_id)` preserves the marker after the legacy branch has
drained. See [Side Effects](/docs/features/side-effects/) and
[Versioning](/docs/features/versioning/) for the shared durable semantics.

## Replay-aware workflow logging {#workflow-logging}

Rust SDK 4.17.0 adds `ctx.logger().info("message")` for observations from workflow code. It routes to the application's existing `tracing` subscriber using target `durable_workflow::workflow`, with SDK-bound immutable `namespace`, `workflow_instance_id` and `workflow_run_id` fields. Configure the host subscriber; the SDK installs no global subscriber.

Ordinary log calls execute again as a worker reconstructs workflow history. The workflow logger stays quiet for committed-prefix, query/offline and completed-history replay, including cold worker replacement, and outside an active invocation. It can emit only in new execution beyond that prefix with a complete bound run identity. `ctx.replay_state()` observes that live consumed-history boundary rather than a Boolean fixed when the task starts:

| `WorkflowReplayState` | Observation |
| --- | --- |
| `Replaying` | Reconstructing committed history or a query; workflow logs are suppressed. |
| `Executing` | New execution beyond the committed prefix; an uncommitted task retry can repeat logs. |
| `Unavailable` | Outside this workflow invocation, or the state cannot be read safely; workflow logs are suppressed. |

Keep durable command order independent of this observation state and log delivery. A log from an uncommitted attempt can repeat after a crash or retry, so the logger is not an exactly-once outbox. Keep application handlers bounded and nonblocking, for example with an application-owned logging queue. Unwinding subscriber panics are isolated from the workflow result; process aborts and blocking handlers remain application responsibilities.

See the [published logging guide](https://github.com/durable-workflow/sdk-rust/blob/4.17.0/README.md#workflow-logging), [WorkflowLogger API](https://rust.durable-workflow.com/durable_workflow/struct.WorkflowLogger.html) and [WorkflowReplayState API](https://rust.durable-workflow.com/durable_workflow/enum.WorkflowReplayState.html). [Registry lifecycle/deadline controls](https://github.com/durable-workflow/sdk-rust/pull/269#issuecomment-6868388853) and [four-process logging acceptance](https://github.com/durable-workflow/sdk-rust/pull/269#issuecomment-6868494262) were recorded on the named native Server 3.10.0 linux/amd64 tuple. Other bindings and broader replay/producer qualification retain their own scope.

## Rust determinism rules {#rust-determinism-rules}

A replay must produce the same durable commands in the same order for the recorded input and history. Rust's ordinary collections, clocks and concurrency primitives do not automatically preserve that contract:

| Pattern in workflow code | Durable replacement |
| --- | --- |
| Scheduling while iterating a `HashMap` | Sort keys first or use a `BTreeMap`. |
| System-clock reads, `thread_rng` or a new UUID | Record a snapshot with `ctx.side_effect(...)`, or use an activity. |
| `tokio::time::sleep` or `std::thread::sleep` | `ctx.sleep(...)` or `ctx.start_timer(...)`. |
| `tokio::select!` or racing ordinary futures | `ctx.select(...)` or `ctx.select_keyed(...)`. |
| Untracked concurrent durable calls | `ctx.parallel(...)` with stable input ordering. |
| Network, filesystem or mutable external I/O | A registered activity. |
| Changing existing command order on deployment | A stable `ctx.get_version(...)` branch, tested against recorded history. |

Activities still need idempotent external effects; determinism does not make a repeated payment or file write safe. See [idempotency versus determinism](../constraints/idempotent-vs-deterministic.md).

## Offline tests and replay {#offline-testing-and-replay}

These APIs require Rust SDK **4.5.0 or newer**. Check `Cargo.lock`: the separate installation tuple above may select an older crate without them. The [normal installed 4.5.0 receipt](https://github.com/durable-workflow/sdk-rust/issues/105#issuecomment-6362112580) verifies twelve Server-free controls, this compiled example and a real Server 2.11.0 export/replay. It does not promote a new Server/SDK tuple.

### Activity and workflow unit tests {#rust-activity-workflow-tests}

`testing::WorkflowEnvironment` runs registered workflows and activities in the current process. Use `from_worker(worker)` to reuse the application's registrations, or register callbacks directly; `worker_mut()` exposes the normal typed registration APIs. Mocks are explicit: real activity callbacks execute their own code, so replace external effects deliberately.

Use `durable-workflow = "=4.5.0"` for the qualified recipe, Tokio with `macros`, `rt-multi-thread` and `time`, and `serde_json = "1"` for the export helper below. Save the released example as `examples/workflow_testing.rs` and run `cargo run --example workflow_testing`:

```rust
//! Run with `cargo run --example workflow_testing`; no Server is required.
use durable_workflow::{
    json,
    testing::{WorkflowEnvironment, WorkflowHistory},
    Result,
};

#[tokio::main]
async fn main() -> Result<()> {
    let mut env = WorkflowEnvironment::new()?;
    env.register_activity("greet", |_, args| async move {
        Ok(json!(format!(
            "Hello, {}!",
            args[0].as_str().unwrap_or("friend")
        )))
    });
    env.register_workflow("hello", |ctx, args| async move {
        ctx.activity("greet", args).await
    });

    // An activity unit test executes the real callback and argument adapters.
    assert_eq!(
        env.run_activity("greet", "Ada").await?,
        json!("Hello, Ada!")
    );
    // A workflow unit test may replace activity callbacks explicitly.
    env.mock_activity("greet", |_, args| async move {
        Ok(json!({"greeting_for":args[0]}))
    });
    assert_eq!(
        env.run_workflow("hello", "Ada").await?,
        json!({"greeting_for":"Ada"})
    );

    // Replay executes the same registered workflow without running activities.
    // In a deployment test, read the JSON produced by dw workflow:history-export
    // or Client::export_workflow_history instead of this captured test history.
    let history = WorkflowHistory::from_export(env.last_history().unwrap().export())?;
    assert!(env.replay(&history)?.completed);
    println!("activity, workflow and offline replay passed");
    Ok(())
}
```

Direct `run_activity` executes one attempt. Within `run_workflow`, recorded retry policy applies before a terminal failure reaches workflow code. Real or mocked remote/local activity failures use typed `Error::ActivityFailed`, with exception type, attempt number and non-retryable metadata; an uncaught workflow error keeps its original Rust variant. Heartbeats are captured locally. Use the `*_avro_value` variants when JSON cannot preserve the original Avro value.

### Replay an exported run {#rust-replay-export}

Register the candidate workflow, then load a self-contained JSON bundle from `dw workflow:history-export` or `Client::export_workflow_history(workflow_id, run_id).await?`. Add this helper to the example above; it uses those imports and expects a completed fixture:

```rust
fn replay(env: WorkflowEnvironment) -> std::result::Result<(), Box<dyn std::error::Error>> {
let export = serde_json::from_slice(&std::fs::read("workflow-history.json")?)?;
let history = WorkflowHistory::from_export(export)?;
let report = env.replay(&history)?;
assert!(report.completed); // A running snapshot can remain durably pending.
Ok(())
}
```

For raw history, combine **all pages** and use `WorkflowHistory::from_history("hello", original_input, response)`, supplying the original task queue when absent. Keep original input and recorded results; pruned/redacted payloads, unresolved external references and remaining page tokens are refused. The SDK export helper resolves payload references while downloading. Replay itself contacts no Server and runs no activity callbacks.

Replay uses the worker's decoding, side-effect, version-marker and recorded select-resolution paths. Changed or reordered commands return `Error::NonDeterministicReplay(ReplayFailure)` with `sequence`, `expected` and `actual`; the installed real-export control finds a changed activity at sequence 3. New commands beyond the snapshot also fail. An open snapshot may report a durably pending prefix rather than completion: assert the expected outcome, and use fresh in-process execution plus integration tests for new branches. See the [released replay controls](https://github.com/durable-workflow/sdk-rust/blob/4.5.0/tests/workflow_testing.rs).

### Execution limits {#rust-test-execution-limits}

Timer and retry delays are skipped. Parallel callbacks run in command order, so this environment does not test scheduling races or performance. It does not simulate queues, leases or deployment behavior. Activity callbacks must yield; `max_steps(n)` bounds durable turns, commands and attempts (default 10,000), while `tokio::time::timeout` can bound asynchronous callback time.

Activity-timeout simulation, **new select races**, signals, children, services, sessions, continue-as-new and cancellation need integration tests; unsupported commands return `Error::Testing`. A recorded select winner can replay even though a new select race is not simulated. Keep live-runtime tests for those boundaries. See the [testing API](https://rust.durable-workflow.com/durable_workflow/testing/index.html) and [released execution contract](https://github.com/durable-workflow/sdk-rust/blob/4.5.0/README.md#testing-without-a-server).

## Entry compatibility and recovery {#entry-compatibility-and-recovery}

**Released support.** Rust SDK 3.5.6 introduced the entry safeguard below, with routing and recovery support in Server 2.9.9. These are minimum feature versions; install current stable releases for security fixes.

`Worker::register_typed_workflow` and the typed stateful adapters deserialize recorded input before invoking the application handler or its state factory. An incompatible input shape reports task failure type `WorkflowEntryArgumentsMismatch` and reason `workflow_entry_arguments_mismatch`, without submitting `fail_workflow`. The SDK's local error is `Error::HandlerType` for workflow input, with the workflow and expected Rust type rather than the recorded value. Raw workflow handlers are not given new Rust signature checks. An error returned by a successfully entered application handler remains an ordinary workflow failure, even if its text resembles the entry diagnostic.

The managed worker automatically advertises `workflow_entry_arguments_validation`. A qualified Server retains the failed task as replay-blocked and leaves the original run nonterminal. Older runtimes may retry the task instead, so the SDK alone does not establish the complete recovery contract.

Use `Client::describe_workflow_run(workflow_id, run_id)` to inspect the original and the selected-run debug endpoint for the task reason. Restore compatible input decoding and code before using the selected-run HTTP repair route in the [recovery procedure](../failures-and-recovery.md#service-entry-recovery). This crate has no `repair_workflow` convenience method. Keep original input, history, namespace, queue, build and deadlines; version durable-command changes inside a compatible handler and qualify cold replay. See [definition routing](./worker-compatibility-routing.md#definition-fingerprint-evolution).

## Cancel, terminate, and handle terminal outcomes

Service-mode `cancel_workflow` and `terminate_workflow` both close the run
immediately. They record distinct terminal outcomes, but neither asks workflow
code to run cleanup. Use `request_cancellation()` with cooperating workers for
bounded workflow cleanup. See the
[cooperative cancellation guide](/docs/polyglot/cancellation) for operation
policies, supervision and recovery.

```rust
use durable_workflow::{Client, WorkflowCommandOptions};

# async fn cancel(client: &Client) -> durable_workflow::Result<()> {
client.cancel_workflow(
    "order-42",
    WorkflowCommandOptions::new()
        .reason("customer withdrew the order")
        .request_id("cancel-order-42"),
).await?;
# Ok(())
# }
```

Instance-targeted `cancel_workflow` and `terminate_workflow` resolve the
current run on the server. For selected-run safety, call
`cancel_workflow_run` or `terminate_workflow_run`, or use a handle's
`cancel_selected_run` and `terminate_selected_run` methods. If a selected run
is stale, `Error::WorkflowCommandRejected` exposes the stable
`historical_run_command_rejected` reason together with workflow ID, run ID,
target scope, HTTP status, and the response body.

Successful `WorkflowHandle::result` calls continue to return the decoded JSON
value. Match the typed terminal variants for every other outcome. Branch on
the stable reason and category fields instead of display text:

<!-- docs-example id="rust.typed-workflow-timeouts" -->
```rust
use durable_workflow::{Error, WorkflowHandle, WorkflowResultOptions};

# async fn wait(handle: WorkflowHandle) -> durable_workflow::Result<()> {
match handle.result(WorkflowResultOptions::default()).await {
    Ok(value) => println!("completed: {value}"),
    Err(Error::WorkflowCancelled(outcome)) => {
        println!("cancelled {:?}: {}", outcome.run_id, outcome.reason);
    }
    Err(Error::WorkflowTerminated(outcome)) => {
        println!("terminated: {}", outcome.reason);
    }
    Err(Error::WorkflowFailed(outcome)) => {
        println!("failure {:?}: {:?}", outcome.failure_id, outcome.exception_class);
    }
    Err(Error::WorkflowTimedOut(outcome)) => match (
        outcome.reason.as_str(),
        outcome.failure_category.as_deref(),
    ) {
        ("result_wait_timeout", Some("client_timeout")) => {
            println!(
                "caller deadline for {} / {:?}; the run may still be open",
                outcome.workflow_id, outcome.run_id,
            );
        }
        ("execution_timeout" | "run_timeout", category) => {
            println!(
                "server timeout for {} / {:?}: reason={} category={:?}",
                outcome.workflow_id, outcome.run_id, outcome.reason, category,
            );
        }
        (reason, category) => {
            println!("other typed timeout: reason={reason} category={category:?}");
        }
    }
    Err(error) => return Err(error),
}
# Ok(())
# }
```

Each terminal outcome carries workflow and run identity. It also retains the
public reason, failure category and identity, exception type and class,
non-retryable state, message, and exception payload when the server supplies
them. A local wait deadline has reason `result_wait_timeout` and category
`client_timeout`; a server timeout is a terminal `timed_out` run whose stable
reason is `execution_timeout` or `run_timeout`.

Handles returned by either start method retain the selected `run_id`.
`WorkflowHandle::result` describes that run-specific route, so reusing the same
workflow ID for a newer run cannot make a wait silently report the newer run's
outcome. Preserve both `outcome.workflow_id` and `outcome.run_id` in logs,
metrics, and retry records; use instance-level lookups only when following the
current run is intentional.

## Workflow updates

Rust supports durable updates across application-client, selected-handle, and
worker-authoring roles. Use `Client::update_workflow` or
`WorkflowHandle::update` for JSON-compatible values, and the matching
`update_workflow_avro_value` or `update_avro_value` methods for explicitly
typed Avro values.

Workers register named handlers with `Worker::register_update`; use
`register_update_avro_value` when the handler consumes and returns Avro values
directly. An update is a durable, result-bearing workflow mutation. It is
separate from fire-and-forget signals and read-only replayed queries.

## Payload envelope

Workflow input, signals, activities, queries, and results use the published
`PayloadEnvelope` contract: a `codec` plus encoded `blob`. The SDK's default
`avro` path uses its declared `apache-avro` dependency and the platform's
fixed versioned Value schema; do not hand-roll the blob or replace the envelope
with an implementation-specific record.

<!-- docs-example id="rust.avro-payload-envelope" -->
```rust
use durable_workflow::{decode_payload, json, PayloadEnvelope, Result, Value};

fn round_trip() -> Result<()> {
    let envelope = PayloadEnvelope::avro(&json!({"order_id": "order-42"}))?;
    assert_eq!(envelope.codec, "avro");

    let decoded: Value = decode_payload(&envelope)?;
    assert_eq!(decoded["order_id"], "order-42");
    Ok(())
}
```

`start_workflow` and `start_workflow_with_options` apply this envelope
automatically to their serializable input. Use the public helpers only when a
program needs to exchange an envelope directly.

## Activity heartbeats {#activity-heartbeats}

For remote activities, `ActivityContext::heartbeat(...).await` returns a `Result<ActivityHeartbeatResponse>`. Propagate errors and inspect `should_stop()` on a successful response. Direct request failures include `Error::Transport` and typed `Error::ActivityTaskRejected` HTTP refusals. A cooperatively supervised claim can instead return `Error::ActivityExecutionAbandoned` when heartbeat or ownership checks fail. Stop external work after losing authority. Do not assume a callback always reaches application cleanup code.

The first heartbeat window starts at runtime claim, before task delivery. A recorded heartbeat uses its runtime timestamp, not the caller's receipt time. Choose a cadence well inside the configured timeout, bound each work/I/O step and allow for scheduling, request latency and any transport retries. Follow the [cadence guidance](../features/heartbeats.md#heartbeat-cadence-and-receipts). No automatic progress-heartbeat or batching schedule is promised.

On cancellation, release process-local resources when execution still permits it and return promptly. A late completion is rejected by durable state and cannot turn a cancelled or terminated run into success. Managed workers continue polling after that definitive rejection and restart. A successful heartbeat is not an external-effect commit or a replacement attempt's resumable checkpoint.

For server images, authentication, and production topology, continue with the
[server setup guide](/docs/polyglot/server).

<!-- Source: docs/polyglot/worker-compatibility-routing.md -->

# Worker Compatibility and Routing

Compatibility is a routing contract. Keep workers that can replay existing
histories available while changing which build receives new workflow starts.
Draining a build does not transfer its runs to a different build.

## Two related identities {#two-related-identities}

- **Build id** is the standalone Server cohort's registered `build_id`, scoped
  to a namespace and task queue. Promotion, drain and resume operate on this cohort.
- **Compatibility marker** is recorded on durable work. Embedded PHP uses
  `DW_V2_CURRENT_COMPATIBILITY` and `DW_V2_SUPPORTED_COMPATIBILITIES`; standalone
  worker build ids provide the corresponding routing identity.

Treat identities as opaque strings. A newer-looking name does not establish
replay compatibility or authorize an existing run to change its pin.

## What gets pinned {#what-gets-pinned}

The standalone Server selects the pin for new starts from explicit promotion,
then its single-active-cohort fallbacks; ambiguous starts can be unpinned.
See [the selection order](/docs/polyglot/worker-build-id-rollout#canary-a-new-build).

Workflow tasks and activities inherit the run's compatibility. Task retries,
lease expiry and redispatch preserve it. Retry runs, children and
continue-as-new also inherit the original pin. Promotion affects subsequent
new instances; it does not move a continuing chain to the new build.

## How routing works {#how-routing-works}

### Poll-time narrowing {#poll-time-narrowing}

Workers request a namespace, task queue and compatible build cohort. Poll-time
selection avoids offering incompatible work, but is not the final authority.

### Claim-time enforcement {#claim-time-enforcement}

The claim checks the task's original compatibility and worker identity. An
incompatible worker cannot gain its lease merely because another build is
absent. Lease expiry does not widen that compatibility.

## Definition fingerprint evolution {#definition-fingerprint-evolution}

**Released support.** Server 2.9.9 and PHP SDK 2.5.4, Python SDK 2.6.2 and Rust SDK 3.5.6 introduced the entry safeguards and routing/recovery behavior below. These are minimum feature versions; install current stable releases for security fixes.

The original build pin and the definition fingerprint answer different questions. A replay-compatible rebuild can keep the same build family while changing its source fingerprint. By default, the Server prefers an eligible worker with the fingerprint recorded at `WorkflowStarted`. If none remains, a changed definition can claim the original only when its worker advertises `workflow_entry_arguments_validation`. That capability promises pre-entry refusal without failing the business run. It is not inferred from the runtime language or protocol version, and manually adding the string does not make an unsafe worker implementation safe.

Fallback still requires the original namespace, task queue, workflow type, build and replay capabilities. The run keeps its recorded fingerprint and admission snapshots. The SDK must validate original entry input and replay committed commands; a source change, version marker or successful registration is not proof that this replay will succeed. Restart changed code with a new worker id, because an active registration cannot change its advertised fingerprint.

Run detail, selected-run debug, repair responses and task-queue detail expose `definition_routing`:

| Status | Meaning and action |
| --- | --- |
| `matching_definition_available` / `matching_definition_preferred` | Eligible recorded-definition capacity is available or preferred over the changed worker. |
| `replay_required` | A capable changed definition may claim. Entry validation and strict replay still have to succeed. This is not a blocked verdict. |
| `entry_validation_unavailable` | The replacement lacks the entry-validation capability. Upgrade the SDK or restore the recorded definition. |
| `no_eligible_worker` / `worker_definition_missing` / `recorded_definition_missing` | Inspect missing capacity or fingerprint information while preserving original routing guards. |
| `pinning_disabled` | The matching-source preference is disabled; other claim and replay guards still apply. |

`blocked = true` describes a due ready workflow task without an eligible definition route. Queue detail samples at most ten runs and reports whether more ready runs exist; it is not a complete count of all passive or blocked runs. Repair cannot manufacture a matching worker or rewrite history. After a genuine entry refusal, follow the [same-original recovery procedure](../failures-and-recovery.md#service-entry-recovery).

`DW_V2_PIN_TO_RECORDED_FINGERPRINT=false` removes the matching-source preference. It does not clear build pins, namespace/queue/type boundaries, registration identity, lease fencing or strict replay checks. It is not a substitute for the safe SDK and default-path upgrade. Older runtimes can silently strand a changed definition or retry an unclassified entry refusal; use the exact qualified runtime/SDK combination.

## What operators should configure {#what-operators-should-configure}

For embedded PHP, configure the current and explicitly supported families:

```bash
DW_V2_CURRENT_COMPATIBILITY=orders-2026-04-28
DW_V2_SUPPORTED_COMPATIBILITIES=orders-2026-04-28,orders-2026-04-21
```

The first value pins new embedded runs; the second permits claims for those
families. Use `*` only with a deliberately verified compatibility policy.
For standalone workers, register a stable `build_id` and use the Server's
promotion and lifecycle controls. Keep a family stable across replay-compatible
rebuilds; changing a label does not make incompatible workflow code replay safely.

## Safe rollout pattern {#safe-rollout-pattern}

1. Deploy and inspect the new cohort's namespace, queue, supported workflow
   types and fresh registrations. Keep the old cohort available.
2. Explicitly promote the desired build for subsequent new starts. Worker
   counts do not define a percentage traffic split.
3. Keep compatible capacity for existing pinned runs, including passive runs
   waiting for a timer, signal, activity or child.
4. Drain only when intentionally stopping that cohort's new claims. If an
   early drain strands work, resume and restore the original compatible build.
5. To roll back new starts, restore/resume and promote the known-good cohort,
   then roll back the bad deployment. Existing pins remain unchanged.

Zero pending workflow tasks does not establish safe retirement. Retain the
artifact and recovery path while any pinned run or inherited chain can need
it. [Server #519](https://github.com/durable-workflow/server/issues/519) tracks
the separate guarded retirement capability; this checklist does not deliver it.

## Explicit missing-worker state {#explicit-missing-worker-state}

Inspect [per-cohort pending workflow tasks](/docs/polyglot/worker-build-id-rollout#pending-workflow-tasks-by-cohort),
fresh/draining/stale registrations and the original run pin together.
`no_compatible_worker` identifies pending workflow work for a non-null build
with no active compatible worker. It does not count activity queues or all
passive pinned runs. A missing field, failed request or stale heartbeat is not
an empty fleet or a zero backlog. Restore the matching build or investigate
its queue, namespace and workflow-type declarations before adding unrelated workers.

## How this relates to build-id rollouts {#how-this-relates-to-build-id-rollouts}

Rollout state describes registered capacity and new-start/lifecycle intent;
compatibility determines whether a specific original task can be claimed.
Use both. Neither a drain nor a rollback rewrites durable compatibility or
proves that a build can be removed.

## Related references {#related-references}

- [Namespace, Auth, And Worker Registration](/docs/polyglot/namespace-auth-workers)
- [Worker Build-Id Rollout](/docs/polyglot/worker-build-id-rollout)
- [Server Config Reference](/docs/polyglot/server-config-reference)
- [Operator Operating Envelope](/docs/operator-operating-envelope)

<!-- Source: docs/polyglot/worker-protocol.md -->

# Worker Protocol

Durable Workflow exposes a versioned worker protocol through two bridge contracts. These contracts define the complete set of verbs that external workers — including the standalone Durable Workflow server — use to poll, claim, execute, and complete workflow and activity tasks.

## Protocol Version

The current server-advertised protocol version is **1.20**. The protocol
follows semver-style numbering:

- **Major** bumps when a change is backwards-incompatible (new required fields, removed verbs, changed pagination semantics).
- **Minor** bumps for additive changes (new optional fields, new non-terminal command types).

Workers may use an older minor within the same major. A server advertising
`1.N` accepts `X-Durable-Workflow-Protocol-Version: 1.M` when `M <= N` and
returns its advertised `1.N` on the response. The default `1.20` server
therefore accepts request versions `1.0` through `1.20`. Discover each SDK's
required minor and optional capability profile before deployment.
Missing or malformed headers, different majors, and worker
minors newer than the server fail closed.

PHP SDK 2.3.2, Python SDK 2.5.7 and Rust SDK 3.5.0 use `1.20` by default.
These releases also support explicit `1.19` selection for an older compatible
Server. Optional worker capabilities, including cooperative cancellation,
remain explicit opt-ins. Selecting a protocol version does not enable them.
See each [SDK's documentation](/docs/polyglot/deployment-modes/) for its
configuration and published compatibility metadata.

You can retrieve the full protocol description programmatically:

```php
use Workflow\V2\Support\WorkerProtocolVersion;

$summary = WorkerProtocolVersion::describe();
// Returns version, verb lists, command types, and pagination defaults.
```

## Capability Discovery

The standalone server publishes worker-protocol capabilities under
`worker_protocol.server_capabilities` in `GET /api/cluster/info`. The same
object is echoed as `server_capabilities` on worker-plane responses, including
poll, heartbeat, complete, and fail responses.

Read these fields before sending optional command fields:

- `supported_workflow_task_commands`: command types accepted by workflow-task completion.
- `poll_status`: poll responses carry a machine-readable status even when no
  task is leased, distinguishing `leased`, `empty`, `throttled`, and
  `unavailable` without forcing clients to infer queue state from `task: null`.
- `activity_retry_policy` and `activity_timeouts`: activity command retry and timeout options.
- `worker_session_verbs` and `worker_sessions`: worker-session lifecycle verbs,
  activity command fields, renewal behavior, failure detection, and terminal
  statuses. The worker-session runtime shape is specified by
  [`worker-sessions-runtime.schema.json`](/platform-protocol-specs/worker-sessions-runtime.schema.json).
- `child_workflow_retry_policy` and `child_workflow_timeouts`: child workflow retry and timeout options.
- `parent_close_policy`: child workflow parent-close policy support.
- `query_tasks`: server-routed workflow query tasks for external runtimes.
- `non_retryable_failures`: workflow and activity failure metadata support.

Before polling, each worker must register its namespace, task queue, runtime,
supported type keys, and local capacity through `POST /api/worker/register`.
The [Namespace, Auth, And Worker Registration](/docs/polyglot/namespace-auth-workers)
reference freezes that registration payload and the role-scoped auth contract.
For the broader ready-task discovery and lease-assignment contract behind these
verbs, see [Task Matching and Dispatch](/docs/polyglot/task-matching-dispatch).
For activity affinity across multiple durable steps, see
[Worker Sessions](/docs/features/worker-sessions).

## Execution Semantics

Worker transport is at-least-once distributed coordination, not exactly-once
delivery:

- workflow-task replay rebuilds state from committed history; it is not
  workflow-level retry
- activity-task lease expiry can trigger redelivery to another worker
- late completion or failure reports can be rejected as stale because another
  attempt already won the durable race

Use `activity_execution_id` as the default remote idempotency key when a
worker or carrier talks to another system. Reach for `activity_attempt_id`
only when the downstream system must distinguish separate tries of the same
logical activity execution.

Read [Execution Guarantees and Idempotency](/docs/constraints/execution-guarantees)
for the authoritative replay, retry, lease-expiry, redelivery, and
exactly-once durable-history contract behind these verbs.

## Workflow Task Bridge

The `WorkflowTaskBridge` contract defines how an external worker interacts with durable workflow tasks:

| Verb | Description |
|------|-------------|
| `poll` | Find ready workflow tasks matching queue and compatibility criteria |
| `claim` / `claimStatus` | Claim a specific task, acquiring a 5-minute lease |
| `historyPayload` | Retrieve the full replay history for a claimed task |
| `historyPayloadPaginated` | Retrieve history in pages for large workflows |
| `execute` | Claim and execute a task in-process using the package executor |
| `complete` | Submit commands from an external worker to complete a task |
| `fail` | Record a task failure from an external worker |
| `heartbeat` | Extend the lease on a claimed task |

### Paginated History

For workflows with large histories, use `historyPayloadPaginated` to retrieve events in pages:

```php
use Workflow\V2\Contracts\WorkflowTaskBridge;

$bridge = app(WorkflowTaskBridge::class);

$afterSequence = 0;
$allEvents = [];

do {
    $page = $bridge->historyPayloadPaginated($taskId, $afterSequence, 500);
    $allEvents = array_merge($allEvents, $page['history_events']);
    $afterSequence = $page['next_after_sequence'] ?? $afterSequence;
} while ($page['has_more']);
```

The default page size is 500 events (matching `WorkerProtocolVersion::DEFAULT_HISTORY_PAGE_SIZE` and the `default_history_page_size` value the server publishes in its worker-protocol capabilities); the maximum is 1000. Servers can advertise a different effective default through the worker-protocol manifest, so prefer reading the published capability over hard-coding either number. The response includes `has_more` and `next_after_sequence` for cursor-based pagination.

### History Compression

For workflows with very large histories, the bridge or server can compress the history events payload to reduce transfer size. Compression is opt-in: the caller must request it via an `Accept-Encoding`-style parameter.

When the event count in a response exceeds the compression threshold (50 events), the bridge may return:

- `history_events`: `[]` (empty array, signalling events are in the compressed key)
- `history_events_compressed`: base64-encoded compressed payload
- `history_events_encoding`: the algorithm used (`gzip` or `deflate`)

The caller decompresses by decoding base64, inflating with the indicated algorithm, and JSON-decoding the result to recover the original `history_events` array.

```php
use Workflow\V2\Support\HistoryPayloadCompression;

// Compress a history payload for transfer (bridge/server side).
$compressed = HistoryPayloadCompression::compress($payload, 'gzip');

// Decompress on the worker side.
$original = HistoryPayloadCompression::decompress($compressed);
```

If the caller does not request compression, or the event count is below the threshold, the response contains the standard uncompressed `history_events` array.

### Long-Poll Semantics

Both `poll` verbs support an optional long-poll mode. When the caller includes a `timeout_seconds` parameter, the bridge or server holds the connection open for up to that duration waiting for a matching task to become ready, instead of returning an empty result immediately.

| Parameter | Default | Min | Max |
|-----------|---------|-----|-----|
| `timeout_seconds` | 30 | 1 | 60 |

Behavior:

- If a task becomes ready during the wait, it is returned immediately.
- If the timeout expires with no task, the response keeps the normal poll
  envelope and returns `task: null` with `poll_status: "empty"`.
- The client should retry immediately on an empty long-poll response unless shutting down.
- HTTP-level timeouts on the transport should be set above 60 seconds to avoid premature disconnects.

Every worker poll response includes `poll_status`:

| Value | Meaning |
|------|---------|
| `leased` | The server leased a task to this worker. |
| `empty` | No matching task was ready before the short poll or long-poll timeout ended. |
| `throttled` | Admission limits prevented the server from leasing work even though the queue may still have ready tasks. |
| `unavailable` | The queue or matching path was temporarily unavailable, so the worker should treat the poll as a transient infrastructure miss. |

Branch on `poll_status` before assuming an empty poll means idle capacity.
`task: null` plus `poll_status: "throttled"` is backpressure, not absence of
work.

```php
use Workflow\V2\Support\WorkerProtocolVersion;

$semantics = WorkerProtocolVersion::longPollSemantics();
// ['default_timeout_seconds' => 30, 'min_timeout_seconds' => 1, 'max_timeout_seconds' => 60]

// Clamp a caller-supplied timeout to the valid range.
$clamped = WorkerProtocolVersion::clampLongPollTimeout($userTimeout);
```

### Poll Response Status

The server keeps one poll-response contract across workflow-task,
activity-task, and query-task polling:

- `task`: the leased task payload, or `null` when the poll did not lease work.
- `poll_status`: the machine-readable outcome for the poll attempt.
- `protocol_version` and `server_capabilities`: the echoed worker-protocol
  manifest fields.

Workers should branch on `poll_status` before making route-specific
assumptions about `task`:

| `poll_status` | Typical HTTP status | Meaning |
| --- | --- | --- |
| `leased` | `200` | The server leased work and `task` contains the payload. |
| `empty` | `200` | No matching task was ready before the poll returned. |
| `throttled` | `200` | Queue admission limits withheld a new lease for this poll attempt. |
| `unavailable` | `503` or `200` | The server could not safely coordinate the queue and returned a typed unavailable outcome. |
| `draining` | `409` | The worker's build-id cohort is draining, so the server refuses to lease new work and returns `reason: "worker_draining"`. |

### Completion, Heartbeat, and Fail Requests

Workflow-task `complete`, `heartbeat`, and `fail` endpoints all require the
worker to echo two lease-identity fields from the poll or claim response:

| Field | Type | Description |
|------|------|-------------|
| `lease_owner` | string | Worker identity that holds the task lease. Must match the `lease_owner` returned from `poll` or `claim`. |
| `workflow_task_attempt` | integer ≥ 1 | Attempt number of the leased task. Must match the `workflow_task_attempt` returned from `poll` or `claim`. |

Stale attempts or wrong lease owners are rejected before any command is
applied, so an expired worker cannot commit replay commands against a
re-claimed task.

Request endpoints and bodies:

- `POST /api/worker/workflow-tasks/{task_id}/complete` — requires `lease_owner`, `workflow_task_attempt`, and a non-empty `commands` array. See [Command Types](#command-types) for the command shapes.
- `POST /api/worker/workflow-tasks/{task_id}/heartbeat` — requires `lease_owner` and `workflow_task_attempt`. Returns the renewed lease expiry and current run status.
- `POST /api/worker/workflow-tasks/{task_id}/fail` — requires `lease_owner`, `workflow_task_attempt`, and a `failure` object containing `message` (required) plus optional `type` and `stack_trace`.

Example complete request:

```json
{
  "lease_owner": "py-worker-1",
  "workflow_task_attempt": 1,
  "commands": [
    { "type": "complete_workflow" }
  ]
}
```

Example heartbeat request:

```json
{
  "lease_owner": "py-worker-1",
  "workflow_task_attempt": 1
}
```

Example fail request:

```json
{
  "lease_owner": "py-worker-1",
  "workflow_task_attempt": 1,
  "failure": {
    "message": "Replay mismatch at event 7",
    "type": "DeterminismFailed"
  }
}
```

### Command Types

When completing a workflow task, the external worker submits a list of typed commands. At most one terminal command is allowed per completion.

**Non-terminal commands** (zero or more, processed in order):

| Type | Required Fields | Description |
|------|----------------|-------------|
| `schedule_activity` | `activity_type` | Schedule an activity task for execution |
| `start_timer` | `delay_seconds` | Schedule a durable timer |
| `start_child_workflow` | `workflow_type` | Start a child workflow instance |
| `complete_update` | `update_id` | Mark an accepted update as applied and completed |
| `fail_update` | `update_id`, `message` | Mark an accepted update as failed |
| `record_side_effect` | `result` | Record a deterministic side-effect result |
| `record_version_marker` | `change_id`, `version`, `min_supported`, `max_supported` | Record a versioning decision |
| `upsert_search_attributes` | `attributes` | Upsert indexed metadata on the workflow run |

`schedule_activity` accepts optional `retry_policy`, `start_to_close_timeout`,
`schedule_to_start_timeout`, `schedule_to_close_timeout`, and
`heartbeat_timeout` fields. `retry_policy` uses `max_attempts`,
`backoff_seconds`, and `non_retryable_error_types`.

`max_attempts` counts total attempts, including the first one. When
`backoff_seconds` runs out, repeat its final entry. For `max_attempts: 5`
and `backoff_seconds: [1, 2]`, the four retry delays are `[1, 2, 2, 2]`
seconds. An empty list, `backoff_seconds: []`, requests immediate retries
(zero backoff), subject to the attempt and timeout limits.

The policy is recorded when the activity is scheduled. Keep its recorded
delays for that activity, including replay under a later SDK. Do not
recompute them from newer defaults.

`start_child_workflow` accepts optional `parent_close_policy`, `retry_policy`,
`execution_timeout_seconds`, and `run_timeout_seconds` fields.
`parent_close_policy` is one of `abandon`, `request_cancel`,
`request_cancellation`, or `terminate`. `request_cancel` closes the child
immediately; `request_cancellation` requests cooperative cleanup with the
original cancellation identity and deadline. See
[cooperative cancellation](/docs/polyglot/cancellation).
Child retry policy uses the same `max_attempts`, `backoff_seconds`, and
`non_retryable_error_types` object shape as activities. Retry backoff applies
after a child run fails; invalid child start commands are protocol errors and
do not consume child retry attempts.

`complete_update` closes the accepted update named by `update_id` after the
worker applies the update handler. It accepts an optional `result` payload
using the same `{codec, blob}` envelope as workflow completion results.
`fail_update` closes the accepted update as failed and accepts optional
`exception_class`, `exception_type`, and `non_retryable` fields in addition to
the required `message`.

**Terminal commands** (at most one):

| Type | Required Fields | Description |
|------|----------------|-------------|
| `complete_workflow` | — | Mark the run as completed (optional `result`) |
| `fail_workflow` | `message` | Mark the run as failed |
| `continue_as_new` | — | Close the run and start a new one (optional `arguments`, `workflow_type`) |

If a cancel or terminate command closes the run while a workflow task is
leased, workflow-task `history`, `heartbeat`, `complete`, and `fail` calls keep
the worker-protocol envelope but reject with `reason: "run_closed"`. The
response also includes `can_continue: false`, `cancel_requested: true`, and a
concrete `stop_reason` such as `run_cancelled` or `run_terminated`, so workers
can distinguish cancellation observation from a generic lease error. The same
response includes `run_closed_reason` and `run_closed_at` from the durable run
record so workers can log the exact closure state that stopped the leased task.

Workflow-task poll responses include stable resume context copied from the
durable task payload:

| Field | Meaning |
|------|---------|
| `workflow_wait_kind` | The wait being applied by this task: `update`, `signal`, `child`, `condition`, `timer`, or `null` for ordinary replay/start tasks |
| `open_wait_id` | Stable wait identity such as `update:{id}` or `signal-application:{id}` |
| `resume_source_kind` / `resume_source_id` | Durable source that woke the task, such as `workflow_update`, `workflow_signal`, `timer`, or `child_workflow_run` |
| `workflow_update_id` | Accepted update id when the task applies an update |
| `workflow_signal_id` | Accepted signal id when the task applies a signal |
| `signal_name` / `signal_wait_id` | Signal target and stable wait identity when the task applies a signal or a timer-backed signal wait |
| `workflow_command_id` | Control-plane command id that produced the task, when available |
| `activity_execution_id` / `activity_attempt_id` / `activity_type` | Activity identifiers when the task resumes after a completed or failed activity |
| `child_call_id` / `child_workflow_run_id` | Child wait identifiers when the task resolves a child workflow |
| `timer_id` / `condition_wait_id` | Pure timer and timer-backed condition identifiers when the task resumes after a timer |
| `condition_key` / `condition_definition_fingerprint` | Stable condition label and predicate fingerprint when a timer-backed condition wait recorded them |
| `workflow_sequence` / `workflow_event_type` | History sequence and event type for event-backed activity, child, and timer resolution tasks |

Fields that do not apply are `null`. SDK workers should prefer these fields
over scanning history when they need to correlate a leased task with an
accepted update, signal, activity result, child resolution, or timer-backed
wait. Pure timer resumes set `workflow_wait_kind: "timer"`,
`open_wait_id: "timer:{timer_id}"`, `resume_source_kind: "timer"`, and
`timer_id`. Signal-backed resumes set `workflow_wait_kind: "signal"` plus
`signal_name`; accepted-signal application tasks also set `workflow_signal_id`
and timer-backed signal waits set `signal_wait_id` with the firing `timer_id`.
Condition-timeout resumes set `workflow_wait_kind: "condition"`,
`condition_wait_id`, and, for keyed waits, `condition_key` plus
`condition_definition_fingerprint`.

## Query Tasks

When a control-plane query targets a workflow whose code is owned by an
external runtime, the standalone server cannot replay that workflow in the PHP
process. Instead, it creates an ephemeral query task and waits for an active
query-capable worker on the workflow's task queue to execute it. This includes
remote PHP SDK workers whose workflow code is outside the Server process.

Query tasks are read-only. Workers replay the supplied history, invoke the
registered query handler, and then complete or fail the query task. They do not
write durable history events and they are not retried after the caller's
control-plane query times out.

| Endpoint | Description |
|----------|-------------|
| `POST /api/worker/query-tasks/poll` | Long-poll for a query task on a worker's registered task queue |
| `POST /api/worker/query-tasks/{query_task_id}/complete` | Submit the query result |
| `POST /api/worker/query-tasks/{query_task_id}/fail` | Reject or fail the query |

Poll request:

```json
{
  "worker_id": "py-worker-1",
  "task_queue": "orders"
}
```

Poll response:

```json
{
  "poll_status": "leased",
  "task": {
    "query_task_id": "01J...",
    "query_task_attempt": 1,
    "workflow_id": "order-123",
    "run_id": "01J...",
    "workflow_type": "order-processing",
    "query_name": "status",
    "payload_codec": "avro",
    "workflow_arguments": { "codec": "avro", "blob": "<base64-avro-bytes>" },
    "query_arguments": { "codec": "avro", "blob": "<base64-avro-bytes>" },
    "history_events": [],
    "task_queue": "orders",
    "lease_owner": "py-worker-1",
    "lease_expires_at": "2026-04-18T12:00:00.000000Z"
  },
  "protocol_version": "1.13",
  "server_capabilities": { "query_tasks": true }
}
```

`task` is `null` when the poll returns no lease. Use `poll_status` to
distinguish an ordinary empty wait from throttling or temporary queue
unavailability. The worker must echo `lease_owner` and `query_task_attempt` on
completion or failure; stale attempts and wrong lease owners are rejected.

Complete request:

```json
{
  "lease_owner": "py-worker-1",
  "query_task_attempt": 1,
  "result": { "status": "ready" },
  "result_envelope": { "codec": "avro", "blob": "<base64-avro-bytes>" }
}
```

Fail request:

```json
{
  "lease_owner": "py-worker-1",
  "query_task_attempt": 1,
  "failure": {
    "reason": "rejected_unknown_query",
    "message": "unknown query 'status'",
    "type": "QueryFailed"
  }
}
```

Use `reason: "rejected_unknown_query"` when the workflow type has no matching
query handler; the control-plane caller receives `404`. Other worker-side
query failures should use `reason: "query_rejected"` and return `409`.
If no active worker can accept the query, the control plane returns
`query_worker_unavailable`; if no result arrives before the configured timeout,
it returns `query_worker_timeout`.

## Activity Task Bridge

The `ActivityTaskBridge` contract defines how an external worker interacts with activity tasks:

| Verb | Description |
|------|-------------|
| `poll` | Find ready activity tasks matching queue and compatibility criteria |
| `claim` / `claimStatus` | Claim a specific activity task with lease |
| `complete` | Record activity completion with a result |
| `fail` | Record activity failure, with optional codec-tagged `failure.details` |
| `status` | Check liveness and cancellation state without renewing the lease |
| `heartbeat` | Extend the lease and report optional progress |

Activity heartbeat responses include `can_continue` and `cancel_requested`
fields, allowing long-running activities to respond to cancellation requests.
When a run-level cancel or terminate command stops a leased activity, heartbeat,
complete, and fail responses also include `run_closed_reason` and
`run_closed_at`.

## Payload Codecs

Every payload byte string that crosses the worker-protocol boundary is tagged with a **`payload_codec`** naming the format of the accompanying blob. v2 uses one language-neutral codec: **`avro`** — so any SDK (PHP, Python, Go, TypeScript, Rust) can encode and decode payloads without sharing a runtime or an app key. The running server advertises its codec support on `GET /api/cluster/info` under **`capabilities.payload_codecs`**.

### The `avro` codec

`avro` is the v2 typed-value codec. Every payload uses the fixed recursive
`durable_workflow.protocol.Value` schema and standard Avro single-object
framing. Its named branches preserve booleans, signed 64-bit integers, finite
doubles, bytes, UTF-8 strings, lists, and string-keyed maps without
workflow-specific schemas or a registry. See the
[Avro Value protocol](/docs/polyglot/avro-value-protocol/).

### Wire Format: Payload Envelope

On fields that carry payload bytes (`arguments`, `result`, `payload`, etc.), the worker protocol surfaces the codec alongside the opaque string. Poll responses look like:

```json
{
  "task_id": "...",
  "payload_codec": "avro",
  "arguments": {
    "codec": "avro",
    "blob": "<base64-avro-bytes>"
  },
  "history_events": [ ... ]
}
```

The worker reads `payload_codec` and confirms it is `avro` before decoding. An unrecognised codec value is an error — the worker should not attempt to sniff or guess.

Activity completions send `result` as the same `{codec, blob}` envelope.
Activity failures may send structured diagnostic payloads under
`failure.details`; when present, `failure.details` is also a `{codec, blob}`
envelope. The server stores the details blob verbatim and records
`details_payload_codec` with the durable failure payload so non-PHP workers can
round-trip diagnostic data without PHP serialization.

The stable cross-language failure surface is `activity_type`,
`failure_category`, `exception_type`, `message`, `code`, `non_retryable`, and
codec-tagged `details`. Runtime fields such as exception class names, source
file paths, line numbers, and stack traces are diagnostics only. SDKs should
not expose those runtime fields in their default `exception_payload`; they may
surface them only when a worker or server explicitly records a `diagnostics` or
`runtime_diagnostics` envelope.

### Starting a Workflow

`POST /api/workflows` accepts `input` in two shapes:

1. **Plain JSON array** — the server adapts each JSON value to the fixed Avro
   Value schema. JSON input cannot express the bytes branch; clients that need
   bytes send an explicit Avro envelope.

   ```json
   { "workflow_type": "MyWorkflow", "input": ["hello", 42] }
   ```

2. **Explicit envelope** — for clients that already hold pre-encoded bytes:

   ```json
   {
     "workflow_type": "MyWorkflow",
     "input": { "codec": "avro", "blob": "<base64-avro-bytes>" }
   }
   ```

The server stores the blob verbatim and tags the run with the `avro` codec.

The codec is stored on the `WorkflowRun` and **propagates for the life of the run**: activity arguments, results, signal/update arguments, and child-workflow inputs are all Avro-encoded.

Embedded/package starts (workflows kicked off from PHP via `WorkflowStub::make(...)->start(...)` rather than the HTTP API) also resolve the new-run default through final v2's Avro-only codec contract.

## Resolving the Bridges

Both bridges are registered in the Laravel container and can be resolved directly:

```php
use Workflow\V2\Contracts\WorkflowTaskBridge;
use Workflow\V2\Contracts\ActivityTaskBridge;

$workflowBridge = app(WorkflowTaskBridge::class);
$activityBridge = app(ActivityTaskBridge::class);
```

## Related Guides

- [Server](./server.md) documents the control-plane endpoints and deployment
  shape that host this protocol.
- [External Execution Surface](./external-execution.md) explains the
  activity-grade worker, bridge, and handler contracts that build on the worker
  protocol.

<!-- Source: docs/polyglot/task-matching-dispatch.md -->

# Task Matching and Dispatch

Use this guide when you need to reason about how Durable Workflow finds ready
work, assigns it to compatible workers, and scales that assignment path beyond
"every node polls everything." This is the contract behind workflow-task polls,
activity-task polls, queue wake behavior, and dedicated matching-role
deployments.

The core idea is that Durable Workflow separates two concerns:

- durable history and task state stay in the database
- task matching decides which live worker gets the next eligible task

That separation matters even before you introduce a separate matching service.
It lets operators talk about ready-task discovery, queue ownership, lease
churn, and backpressure as one explicit role instead of as incidental side
effects of every worker process.

## What the matching role does

The matching role is responsible for:

- discovering ready workflow and activity tasks
- narrowing the eligible set by namespace, connection, queue, and
  compatibility family
- surfacing tasks to one worker at a time
- converting a successful claim into a lease with expiry
- preserving the same task when a claim fails, a lease expires, or a worker
  disappears
- making wake and backlog state visible to operators

Matching does not replace durable history, workflow replay, or task execution.
It decides who gets a chance to execute next.

## Deployment shapes

Durable Workflow supports three practical shapes today:

### In-worker matching

This is the default shape. Worker processes long-poll for ready work and use
claim-time fencing to ensure only one worker owns a task lease at a time.

Use this when:

- you run a single node or a small fleet
- queue pressure is moderate
- you do not need a dedicated matching daemon yet

### In-server HTTP matching

In standalone-server deployments, external workers reach the same matching
contract through the worker protocol:

- `POST /api/worker/workflow-tasks/poll`
- `POST /api/worker/activity-tasks/poll`

The server becomes the network entrypoint for the same ready-task discovery,
claim, heartbeat, and completion flow.

### Dedicated matching-role deployment

Larger fleets can concentrate broad ready-task discovery into a dedicated
matching-role process. The documented operator shape today is:

```bash
php artisan workflow:v2:repair-pass --loop
```

Run that daemon in a dedicated process and set
`DW_V2_MATCHING_ROLE_QUEUE_WAKE=0` on execution-only nodes so they stop doing
the broad queue-worker wake on every loop tick.

Use this shape when:

- execution nodes should focus on replay and activity execution
- broad independent polling is creating unnecessary contention
- operators want an explicit place to own ready-task discovery and sweep
  cadence

The dedicated role changes where matching happens, not the worker-protocol
contract. Poll, claim, lease, and redelivery semantics stay the same.

## Ready-task discovery

The matching role looks for durable tasks that are actually ready to run now:

- the task kind matches the poller (`workflow` or `activity`)
- the task is in `ready` state
- the task's `available_at` has arrived
- namespace, queue, connection, and compatibility filters still match
- activity polls also require a matching advertised activity type

Long-poll wakeups are an acceleration path, not the correctness path. If the
wake signal is delayed or missing, the task still becomes visible through
durable polling. A wake failure should raise latency, not strand work.

## Claim, lease, and backpressure

Matching only offers an opportunity. Ownership starts when one worker
successfully claims the task and receives a lease.

That lease gives Durable Workflow its backpressure behavior:

- one worker owns the task until the lease is renewed, completed, released, or
  expired
- a failed or stale worker does not permanently trap the task
- redelivery is normal and must be handled as part of at-least-once execution
- incompatible workers do not steal the lease; they get an explicit rejection

Because work is lease-based, queue pressure shows up as backlog, stale leases,
or repeated redelivery instead of as hidden in-memory loss.

## Queue partitioning

The main partitioning primitives are:

- **namespace** for tenant or environment boundaries
- **connection** and **queue** for work-routing boundaries
- **compatibility family** for compatibility-marker routing safety
- **activity type filters** for workers that only execute selected activities

Use separate queues when you want stronger isolation, independent worker pools,
or different downstream budgets. Use compatibility families when the same
queue must stay live across a rollout or rollback without letting in-flight
work drift to incompatible executors.

## Recorded activity queues {#recorded-activity-queues}

An activity's resolved connection and queue are stored when it is scheduled. A change to an activity class, `ActivityOptions`, PHP SDK activity options or Python `schedule_activity(..., queue=...)` affects newly scheduled activities. Existing executions keep their recorded queue through retries and repair. A worker registering on a new queue does not move old work. Existing workflow runs keep their own workflow queue too.

### Change routing with an overlap window {#change-recorded-routing}

1. Start workers for the new queue in the correct namespace. Verify their activity types, positive activity slots and build compatibility. Keep the old workflow and activity pools available.
2. Deploy the new routing for new work. Preserve compatible code for existing runs and respect their build pins. A new queue is not a migration of recorded executions.
3. Inspect affected runs and both queues. Account for pending and leased activities, future retry availability, passive open runs, children and continuation chains that still need old workers. A zero ready-task count does not cover these dependencies.
4. Retire old workers only after you have accounted for those dependencies and stopped new work from creating them. Current snapshots are not an atomic retirement guard. If you cannot establish this, keep the old pool available. [Server #519](https://github.com/durable-workflow/server/issues/519) owns the guarded inventory and retirement capability.

### Inspect and recover stranded work {#recorded-queue-diagnostics}

Use an operator credential and the same namespace as the affected run. Set the origin, namespace, old queue and selected workflow/run IDs explicitly:

```bash
curl -fsS "$DW_SERVER_URL/api/task-queues/$OLD_QUEUE" \
  -H "Authorization: Bearer $DW_OPERATOR_TOKEN" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  -H "X-Namespace: $DW_NAMESPACE"
curl -fsS "$DW_SERVER_URL/api/workflows/$WORKFLOW_ID/runs/$RUN_ID/debug" \
  -H "Authorization: Bearer $DW_OPERATOR_TOKEN" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  -H "X-Namespace: $DW_NAMESPACE"
```

If server admission is available and not throttled, queue visibility reports `admission.activity_tasks.status: no_active_workers` when no active, non-stale worker is registered on that queue. The run debug response includes `pending_activities[].queue`, `activity_task_queues` and findings such as `pending_activity_queue_without_active_pollers`. If workers are present but do not advertise the required activity type, look for `pending_activity_type_unsupported`. Registration alone does not prove a worker is making progress.

The debug response is a bounded run snapshot, not a complete queue inventory. It includes at most 25 pending activities. Queue counters and empty snapshots cannot prove that concurrent or future work has no dependency on the old pool.

Restore a compatible worker on the **recorded** queue, with the required activity type and available slots, before its deadlines expire. Repair or restarting a worker does not change that queue or extend the original total deadline. If the work is no longer wanted, deliberately cancel the affected run and reconcile any external effects. Do not edit stored queue fields as a migration procedure.

A finite schedule-to-start timeout can expire while an activity waits for a worker. Schedule-to-close bounds the whole activity execution, including retry delays. These bounds apply only when configured. Deadline enforcement does not move the activity to a live queue, and an already terminal timeout is not revived by restoring workers. See [activity and workflow timeouts](../features/timeouts.md).

## Wake signals and dedicated sweeps

Wake signals shorten the time between "task became ready" and "a poller noticed
it." They are not the durable source of truth.

In the default shape, queue workers can emit wake signals as part of their
normal loop. In a dedicated matching-role deployment, execution-only nodes
disable that broad wake path and the matching daemon owns the sweep cadence
instead.

Treat these as tuning levers:

- `DW_V2_MATCHING_ROLE_QUEUE_WAKE` controls whether execution nodes perform the
  in-worker wake
- `DW_WAKE_SIGNAL_TTL_SECONDS` controls how long wake markers remain visible
- long-poll timeout settings control how long pollers stay parked waiting for
  work

## What operators should watch

Use these surfaces together:

- `GET /api/cluster/info` for the per-node role topology:
  `current_shape`, `current_process_class`, current roles, and the full
  `matching_role` contract: `queue_wake_enabled`, deployment `shape`,
  `wake_owner`, `task_dispatch_mode`, frozen `partition_primitives`, and the
  current `backpressure_model`
- task-queue visibility for ready depth, active slots, and throttling
- worker fleet visibility for active vs stale pollers on each queue
- `dw system:operator-metrics --json` or `/api/system/operator-metrics` for
  the same node-local `matching_role` contract beside live backlog, repair,
  worker, and health counters from the responding process
- health and Waterline diagnostics for unhealthy tasks, stale leases, and
  compatible-worker gaps
- rolling-upgrade checks when an overlap window is live and matching must
  block unsafe claims

If the oldest ready-task age grows while compatible workers are available, the
matching path is not making forward progress quickly enough. If ready tasks are
preserved but no compatible worker can claim them, that is a compatibility
problem, not a matching-loss problem.

## When to introduce a dedicated matching role

Move from default in-worker matching to a dedicated matching-role shape when:

- broad polling is creating visible database or cache contention
- execution nodes should stop paying the overhead of ready-task sweeps
- queue fairness and dispatch ownership need to be reasoned about as one
  explicit subsystem
- operators want a distinct process to scale, supervise, and debug for
  ready-task discovery

Stay on the default shape when the current fleet is small and the main need is
clear rollout and compatibility behavior rather than a new topology.

## Related references

- [Worker Protocol](/docs/polyglot/worker-protocol) for poll, heartbeat,
  complete, and fail verbs
- [Task Queue Admission](/docs/polyglot/task-queue-admission) for slot,
  lease, and dispatch budgets on top of matching
- [Rolling Upgrades](/docs/rolling-upgrades) for rollout procedure when
  matching and compatibility are both in play
- [Server Config Reference](/docs/polyglot/server-config-reference) for
  `DW_V2_MATCHING_ROLE_QUEUE_WAKE`, poll timing, and wake-signal settings

<!-- Source: docs/polyglot/task-queue-admission.md -->

# Task Queue Admission

Task queue admission keeps one queue, tenant, or downstream dependency from consuming the whole worker fleet. Durable Workflow exposes admission in three layers:

- worker registrations advertise local workflow and activity slots
- the server can cap active workflow and activity leases per namespace and queue
- the server can cap workflow and activity dispatches per minute per namespace and queue
- the server can cap dispatches per minute for named downstream budget groups shared by several queues
- query tasks have a bounded pending queue so synchronous reads fail fast instead of growing without limit

Use admission controls when a queue is tied to a rate-limited dependency, tenants share the same server, or operators need to prove why a workflow is waiting.

:::note Native Server 3.x

The native 3.x Server enforces none of the server-side admission controls on this page, and there is no native equivalent of them yet. It refuses to start when any of these sets a limit:

- a workflow-task or activity-task lease or dispatch cap, or `DW_TASK_QUEUE_ADMISSION_OVERRIDES`
- a namespace request-admission setting
- a namespace durable-state limit

It ignores `DW_WORKER_LONG_POLL_MAX_CONCURRENT`, `DW_QUERY_TASK_POLL_MAX_CONCURRENT` and their per-namespace variants. `DW_QUERY_TASK_MAX_PENDING_PER_QUEUE` must stay unset or `1024`. Native instead applies a fixed bound of 16 pending or leased query tasks per namespace and refuses further queries with HTTP `429` and `query_task_queue_full`, as PHP does.

On native, bound task concurrency with each SDK worker's configured slots and the number of workers. `GET /api/task-queues/{name}` reports the registered slots and the leased and ready counts under `admission`, with every server-side limit and remaining-capacity field `null`. See the [native configuration reference](./server-config-reference.md#native-server-3x). The rest of this page describes PHP Server 2.x.

:::

Admission sits on top of the matching contract. Read
[Task Matching and Dispatch](/docs/polyglot/task-matching-dispatch) for
how ready work is discovered and leased before these budgets decide whether the
next task is allowed through.

## How The Budget Is Applied

Workflow and activity polling starts with the workers that are currently registered for a namespace and task queue. Each worker advertises `max_concurrent_workflow_tasks` and `max_concurrent_activity_tasks`; the server sums active, non-stale workers to calculate the queue's registered slot capacity.

Server-side active lease and dispatch-rate caps are optional. When configured, the server checks a short-lived cache lock before leasing the next workflow or activity task. If the active lease cap is full, polling returns no task for that poll instead of exceeding the in-flight budget. If the per-minute dispatch cap is full, polling returns no task until the next minute bucket has capacity. Downstream budget groups apply the same per-minute dispatch behavior across every queue in the namespace that shares a `dispatch_budget_group` name.

Query tasks are different: the control plane enqueues an ephemeral query task and waits for a worker response. `DW_QUERY_TASK_MAX_PENDING_PER_QUEUE` caps how many pending query tasks can exist for each namespace and task queue. When the queue is full, new queries return `query_task_queue_full` with HTTP `429`. If the cache store cannot provide the lock needed to mutate the query-task queue, queries return `query_task_queue_unavailable` with HTTP `503`.

## Retry load and downstream budgets {#retry-load-and-downstream-budgets}

**First attempts and retries both count** against queue, namespace and budget-group dispatch caps. The cap is a budget for successfully claimed activity-task attempts across workflows, not just retries. A full cap leaves eligible work waiting; it does not itself fail the activity. Schedule-to-start and schedule-to-close clocks keep running, as do workflow deadlines, so waiting can exhaust an authored timeout.

A dispatch cap measures attempts per minute, not success ratio or downstream health. It does not open a circuit breaker. One claim may make several HTTP calls if your activity or HTTP client retries internally; bound those calls separately. A fast-failed attempt also consumes a claim even when it never contacts the downstream.

For example, in namespace `production`, let the ordinary `payments` and `refunds` activity queues share one `payment-api` budget of 20 claims per minute. Set the same group cap on every participating queue; unrelated queues do not join the group. This is a minute-bucket ceiling, not a smooth rate or a reserved share for either queue:

```bash
DW_TASK_QUEUE_ADMISSION_OVERRIDES='{
  "production:payments": {
    "activity_tasks": {
      "dispatch_budget_group": "payment-api",
      "max_dispatches_per_minute_per_budget_group": 20
    }
  },
  "production:refunds": {
    "activity_tasks": {
      "dispatch_budget_group": "payment-api",
      "max_dispatches_per_minute_per_budget_group": 20
    }
  }
}'
```

Combine the Server cap with bounded activity attempts and backoff. This Python SDK 2.9.0 example assumes your application supplies an asynchronous shared breaker and payment client through `myapp.payments`; configure bounded I/O and stable downstream idempotency in that client. Register `charge` on the `payments` worker in `production`, and register `PaymentWorkflow` on its workflow worker:

```python
from durable_workflow import NonRetryableError, activity, workflow
from myapp.payments import breaker, payment_api

@activity.defn(name="payments.charge")
async def charge(order):
    if await breaker.is_open("payment-api"):
        raise NonRetryableError("payment-api breaker is open")
    return await payment_api.charge(order, idempotency_key=order["id"])

@workflow.defn(name="payments.order")
class PaymentWorkflow:
    def run(self, ctx, order):
        return (yield ctx.schedule_activity(
            "payments.charge", [order], queue="payments",
            retry_policy={"max_attempts": 3, "backoff_seconds": [2, 5]},
            start_to_close_timeout=10,
            schedule_to_start_timeout=30,
            schedule_to_close_timeout=90,
        ))
```

An open application breaker raises `NonRetryableError` before the payment call; ordinary retryable payment failures may use at most three attempts with the authored backoff. The original timeout budgets can expire before all three attempts occur. Workflow code can catch the final activity failure when business logic requires it; an uncaught final failure can close the workflow. Choose an overall workflow deadline separately.

Keep protected downstream calls in **ordinary queued activities**. Inline local activities create no ordinary activity task, so queue, namespace and budget-group activity dispatch caps do not meter them. A prepared local-activity claim validates execution authority, not this dispatch budget. Embedded Laravel mode has no standalone Server admission; use its own deployment controls. See [local activities](../features/local-activities.md).

## Namespace request admission {#namespace-request-admission}

The following contract is for published **PHP Server 2.11.0**. Check your actual image and the [versioned configuration reference](./server-config-reference.md#additional-published-php-controls). Namespace HTTP budgets are separate from worker slots, task dispatch, held worker polls and durable-state quotas. The HTTP budget belongs to the request namespace resolved from `Durable-Workflow-Namespace` or `DW_DEFAULT_NAMESPACE`, including namespace-management requests whose path names another namespace.

Authenticated requests that reach namespace admission count on these control and management route families, including their descendants: `/api/namespaces`, `/api/storage`, `/api/workflows`, `/api/activities`, `/api/bridge-adapters`, `/api/workers`, `/api/worker-sessions`, `/api/task-queues`, `/api/deployments`, `/api/schedules`, `/api/search-attributes`, `/api/service-endpoints`, `/api/service-catalog`. Plural `/api/workers` management requests count.

Singular `/api/worker/*` registration, polling and outcome routes do not use this HTTP budget. Health/readiness, `/api/cluster/info`, admin `/api/system/*`, runtime credentials and external-payload transport also stay outside this gate; their own authentication, capacity and storage checks still apply. Requests refused earlier by authentication, protocol or namespace resolution never reach this counter.

Rate admission uses fixed server-clock minute buckets, not a smooth or rolling rate. A budget of N can admit N requests just before a boundary and another N just after it. Requests that fail later still count; rate admission happens before the concurrency check, so a concurrency refusal can consume minute budget. A concurrency slot remains held until the request finishes and cleanup releases it; it does not count running workflows.

Across API replicas, use the same shared Redis store/key prefix, synchronized clocks and consistent loaded configuration. The shared namespace budget is not multiplied by adding replicas. A local file cache only coordinates processes with compatible writable access to its files; it is not the multi-node baseline. Per-node held worker/query-poll caps remain separate. [Restart affected roles after environment changes](./server-config-reference.md#how-changes-take-effect).

```dotenv
DW_NAMESPACE_MAX_REQUESTS_PER_MINUTE=100
DW_NAMESPACE_MAX_CONCURRENT_REQUESTS=8
DW_NAMESPACE_HARD_MAX_REQUESTS_PER_MINUTE=150
DW_NAMESPACE_HARD_MAX_CONCURRENT_REQUESTS=10
DW_NAMESPACE_ADMISSION_OVERRIDES='{"payments":{"max_requests_per_minute":200,"max_concurrent_requests":12},"reports":{"max_requests_per_minute":40,"max_concurrent_requests":3}}'
```

For existing namespaces, this example resolves `payments` to 150 requests/minute and 10 concurrent requests, `reports` to 40/3, and other namespaces to 100/8. Overrides can raise or lower defaults, but the 150/10 hard caps still apply. This configuration does not create namespaces or reserve throughput.

For these HTTP controls, unset means no corresponding configured limit; a hard cap alone can still impose a limit. A null override falls back to the default. Values must be positive integers: zero is invalid. PHP implementation ceilings are 10000000 requests/minute and 1024 concurrent requests, independent of the actual HTTP pool capacity. Published PHP 2.11.0 silently treats malformed `DW_NAMESPACE_ADMISSION_OVERRIDES` JSON as an empty override map, falling back to global defaults. Validate the JSON before deployment; if the defaults and hard caps are unset, the intended namespace cap is lost. This [confirmed product gap](https://github.com/durable-workflow/durable-workflow.github.io/issues/335#issuecomment-6369150521) remains separate from the documentation update.

| Reason | HTTP | Retry hint in seconds |
| --- | --- | --- |
| `namespace_request_rate_exhausted` | `429` | Until the next minute, 1–60 |
| `namespace_request_concurrency_exhausted` | `429` | 1 |
| `namespace_request_admission_unavailable` | `503` | 1 |

The JSON `retry_after_seconds` and `Retry-After` header agree. Unavailable configured coordination or invalid numeric limits fails closed with 503. Inspect namespace-scoped admin metrics even when ordinary requests are exhausted; `dw_namespace_request_admission_rejections` reports the finite reasons. Apply bounded caller retry/backoff policy and preserve idempotency; an uncertain transport outcome is not permission to resubmit a write automatically. See [timeout budgets](../deployment.md#worker-long-poll-timeout-budgets).

## Namespace durable-state quotas {#namespace-durable-state}

These quotas are a PHP Server 2.x control; native 3.x refuses to start when they set a limit.

`DW_NAMESPACE_DURABLE_LIMITS` sets default retained-row limits, `DW_NAMESPACE_DURABLE_OVERRIDES` changes them per namespace, and `DW_NAMESPACE_DURABLE_HARD_LIMITS` adds non-bypassable ceilings. Empty objects add no limits. Overrides can raise or lower defaults; null falls back to the default. Unlike HTTP admission, a durable limit of zero is valid and prevents new allocations. In this example, existing `payments` can hold 250 open runs, other namespaces 100, under a 500 hard ceiling.

```dotenv
DW_NAMESPACE_DURABLE_LIMITS='{"max_open_workflow_runs":100,"max_open_workflow_run_waits":200,"max_pending_workflow_timers":200}'
DW_NAMESPACE_DURABLE_HARD_LIMITS='{"max_open_workflow_runs":500}'
DW_NAMESPACE_DURABLE_OVERRIDES='{"payments":{"max_open_workflow_runs":250}}'
```

Use `max_<resource>` fields for the resources below. Counts belong to the namespace owning each row, or its linked workflow/run. Total retained rows and open/pending subsets are independent limits. Enforcement uses database-authoritative counts while serializing mutations on the namespace row.

| Resource | What counts |
| --- | --- |
| `workflow_instances`, `workflow_runs` | All retained workflow instance/run rows, including terminal runs. |
| `open_workflow_runs` | Runs with `pending`, `running` or `waiting` status. |
| `workflow_history_events`, `workflow_commands`, `workflow_tasks` | Retained history, command and workflow-task rows. |
| `pending_workflow_tasks` | Queued workflow-task rows with `ready` status; active leases have separate admission controls. |
| `workflow_timers`, `pending_workflow_timers` | All retained timers, and the `pending` subset respectively. |
| `workflow_run_waits`, `open_workflow_run_waits` | All retained run waits, and the `open` subset respectively. |
| `schedules`, `schedule_history_events`, `worker_registrations` | Retained schedules, schedule history and worker-registration rows; stale registrations count until removed. |
| `workflow_streams`, `workflow_stream_items` | Durable and inbound stream rows, and their item rows. |

A parent waiting for a child or service call still consumes an open run in its own namespace. A child adds its own run; in the same namespace, plan for both open runs and their waits/timers. A service operation that creates a workflow consumes rows in that workflow’s namespace; signal/update operations on an existing run do not automatically create a second run. Service-call boundary and external-payload quotas are separate controls.

Exhaustion returns 429 `namespace_<resource>_exhausted`. Read `retryable`: some quotas are nonretryable and provide no retry hint; retryable durable refusals advertise 60 seconds, without promising capacity then. Failure to evaluate a configured quota returns 503 `namespace_durable_state_quota_unavailable` with a one-second hint. Retained-row pressure may require retention or a capacity/configuration change, not a tight retry loop.

Select the namespace with the request header when reading admin metrics. `dw_namespace_durable_state_usage` and the JSON `limits`, `usage`, `remaining`, `configuration_status` and `measurement_status` expose that scope; namespace is request scope rather than an unbounded metric label. Unavailable measurements are not zero usage or unlimited capacity. See the [complete configuration reference](./server-config-reference.md#namespace-durable-controls).

## Server Configuration

These settings apply to PHP Server 2.x. Native 3.x refuses to start when a lease or dispatch cap or `DW_TASK_QUEUE_ADMISSION_OVERRIDES` sets a limit.

Set global caps when every queue should share the same ceiling:

```bash
DW_WORKFLOW_TASK_MAX_ACTIVE_LEASES_PER_QUEUE=25
DW_ACTIVITY_TASK_MAX_ACTIVE_LEASES_PER_QUEUE=100
DW_WORKFLOW_TASK_MAX_ACTIVE_LEASES_PER_NAMESPACE=500
DW_ACTIVITY_TASK_MAX_ACTIVE_LEASES_PER_NAMESPACE=2000
DW_WORKFLOW_TASK_MAX_DISPATCHES_PER_MINUTE=600
DW_ACTIVITY_TASK_MAX_DISPATCHES_PER_MINUTE=1200
DW_WORKFLOW_TASK_MAX_DISPATCHES_PER_MINUTE_PER_NAMESPACE=12000
DW_ACTIVITY_TASK_MAX_DISPATCHES_PER_MINUTE_PER_NAMESPACE=24000
DW_QUERY_TASK_MAX_PENDING_PER_QUEUE=1024
```

Queue caps protect one task queue. Namespace caps protect the tenant-wide total across every task queue in the namespace, which is useful when a tenant can shard work across many queues but still shares one downstream quota. Budget-group caps protect a named downstream dependency across selected queues without throttling every queue in the namespace.

Use `DW_TASK_QUEUE_ADMISSION_OVERRIDES` when specific queues or namespaces need different budgets. Keys are checked in this order: `namespace:task_queue`, `namespace:*`, `task_queue`, then `*`.

```bash
DW_TASK_QUEUE_ADMISSION_OVERRIDES='{
  "production:payments": {
    "workflow_tasks": {
      "max_active_leases_per_queue": 8,
      "max_dispatches_per_minute": 120,
      "dispatch_budget_group": "downstream-openai",
      "max_dispatches_per_minute_per_budget_group": 600
    },
    "activity_tasks": {
      "max_active_leases_per_queue": 12,
      "max_dispatches_per_minute": 240
    }
  },
  "production:*": {
    "workflow_tasks": {
      "max_active_leases_per_namespace": 300,
      "max_dispatches_per_minute_per_namespace": 6000
    },
    "activity_tasks": {
      "max_active_leases_per_namespace": 1200,
      "max_dispatches_per_minute_per_namespace": 12000
    }
  },
  "email": {
    "activity_tasks": {
      "max_active_leases_per_queue": 4,
      "max_dispatches_per_minute": 60,
      "dispatch_budget_group": "downstream-sendgrid",
      "max_dispatches_per_minute_per_budget_group": 300
    }
  },
  "*": {
    "workflow_tasks": { "max_active_leases_per_queue": 50 }
  }
}'
```

The override value also accepts `max_active_leases` as an alias for `max_active_leases_per_queue` and `budget_group` as an alias for `dispatch_budget_group`.

Cache must support atomic locks for server-side active lease caps, dispatch-rate caps, and query-task admission. Dispatch-rate counters are short-lived minute buckets created only for capped queues that actually lease tasks. Redis is the recommended cache store for multi-node deployments.

## Worker Slot Registration

To give Python workflow and activity handlers independent process resources,
use the [separate worker processes example](https://python.durable-workflow.com/guides/separate-workers/).
Choose the registered handler lists to separate roles. Keep both concurrency
values positive, and apply server caps when the total across processes must be
bounded.

Python workers expose local semaphores through `Worker(...)` and send the same values during registration:

```python
worker = Worker(
    client,
    task_queue="payments",
    workflows=[PaymentWorkflow],
    activities=[charge_card, send_receipt],
    max_concurrent_workflow_tasks=8,
    max_concurrent_activity_tasks=12,
)
```

For custom HTTP workers, send the slot fields to `POST /api/worker/register`:

```json
{
  "worker_id": "payments-python-1",
  "task_queue": "payments",
  "runtime": "python",
  "supported_workflow_types": ["payments.PaymentWorkflow"],
  "supported_activity_types": ["payments.charge_card", "payments.send_receipt"],
  "max_concurrent_workflow_tasks": 8,
  "max_concurrent_activity_tasks": 12
}
```

Worker slots are not a hard tenant budget by themselves. They describe what active workers can currently process. Add server caps when you need a queue-wide ceiling that still holds if more workers are deployed.

## Inspect Admission

Use the CLI when debugging:

```bash
dw task-queue:list
dw task-queue:describe payments
dw task-queue:describe payments --json | jq '.admission'
```

The server exposes the same data through:

- `GET /api/task-queues`
- `GET /api/task-queues/{name}`

An admission payload has three sections:

```json
{
  "workflow_tasks": {
    "status": "throttled",
    "active_worker_count": 3,
    "configured_slot_count": 24,
    "leased_count": 8,
    "ready_count": 5,
    "available_slot_count": 16,
    "server_max_active_leases_per_queue": 8,
    "server_active_lease_count": 8,
    "server_remaining_active_lease_capacity": 0,
    "server_max_active_leases_per_namespace": 300,
    "server_namespace_active_lease_count": 149,
    "server_remaining_namespace_active_lease_capacity": 151,
    "server_max_dispatches_per_minute": 120,
    "server_dispatch_count_this_minute": 120,
    "server_remaining_dispatch_capacity": 0,
    "server_max_dispatches_per_minute_per_namespace": 6000,
    "server_namespace_dispatch_count_this_minute": 3520,
    "server_remaining_namespace_dispatch_capacity": 2480,
    "server_dispatch_budget_group": "downstream-openai",
    "server_max_dispatches_per_minute_per_budget_group": 600,
    "server_budget_group_dispatch_count_this_minute": 600,
    "server_remaining_budget_group_dispatch_capacity": 0,
    "server_lock_required": true,
    "server_lock_supported": true,
    "budget_source": "worker_registration.max_concurrent_workflow_tasks",
    "server_budget_source": "server.admission.queue_overrides"
  },
  "activity_tasks": {
    "status": "accepting",
    "configured_slot_count": 36,
    "server_max_active_leases_per_queue": 12,
    "server_remaining_active_lease_capacity": 4,
    "server_max_dispatches_per_minute": 240,
    "server_remaining_dispatch_capacity": 197
  },
  "query_tasks": {
    "status": "accepting",
    "max_pending_per_queue": 1024,
    "approximate_pending_count": 7,
    "remaining_pending_capacity": 1017,
    "lock_supported": true,
    "budget_source": "server.query_tasks.max_pending_per_queue"
  }
}
```

## Status Reference

| Section | Status | Meaning |
|---------|--------|---------|
| Workflow/activity | `accepting` | Active workers have available slots and no server cap is full. |
| Workflow/activity | `throttled` | The optional server-side active lease cap or dispatch-per-minute cap is full. |
| Workflow/activity | `saturated` | Registered worker slots are all leased, even if no server cap is configured. |
| Workflow/activity | `no_slots` | Active workers registered zero slots for that task kind. |
| Workflow/activity | `no_active_workers` | No active, non-stale worker is polling that queue. |
| Workflow/activity | `unavailable` | A configured server cap needs a cache lock, but the lock is unavailable. |
| Query | `accepting` | The pending query-task queue has remaining capacity. |
| Query | `full` | Pending query tasks reached `DW_QUERY_TASK_MAX_PENDING_PER_QUEUE`; new queries return HTTP `429`. |
| Query | `unavailable` | The query-task queue cannot acquire its cache lock; new queries return HTTP `503`. |

## Tuning Pattern

1. Start with worker slots sized to the process: CPU-bound workflow tasks are usually lower than I/O-heavy activity tasks.
2. Add active lease caps for queues that need an in-flight ceiling across all workers.
3. Add namespace-wide active lease caps when one tenant can create many queues but still needs a total in-flight ceiling.
4. Add dispatch-per-minute caps for queues that protect a rate-limited external API, database pool, tenant, or legacy service from bursts even when workers have free slots.
5. Add budget-group dispatch caps when several queues share one downstream dependency but unrelated queues in the namespace should keep flowing.
6. Add namespace-wide dispatch caps when the downstream quota is tenant-wide rather than queue-specific.
7. Inspect `dw task-queue:describe <queue>` during load. `saturated` means add worker capacity or lower workflow fan-out. `throttled` means an active lease or dispatch-rate cap is doing its job. `no_active_workers` means the queue has no healthy poller.
8. Keep query-task capacity large enough for normal operator reads, but low enough to fail fast during incidents. Query-task overflow is backpressure, not data loss.

## Related Guides

- [Server](/docs/polyglot/server)
- [CLI](/docs/polyglot/cli)
- [Python SDK](/docs/polyglot/python)
- [Worker Protocol](/docs/polyglot/worker-protocol)

<!-- Source: docs/polyglot/worker-build-id-rollout.md -->

# Worker Build-Id Rollout

## Replay before changing builds {#replay-before-changing-builds}

Before promoting a candidate, replay representative open and recently closed histories against its exact application code. Before rollback, replay histories written by the newer build against the older artifact. Follow the [replay testing gate](../testing.md#replay-before-deployment); a compatible routing cohort or successful integrity check alone does not prove replay compatibility.

Use this guide to inspect worker cohorts, choose the build for new workflow
starts, and keep compatible workers available for existing runs. Promotion
changes new-start selection. Draining stops new claims by the selected cohort.
Neither operation migrates existing runs or proves that a build can be deleted.

Read [Worker Compatibility and Routing](/docs/polyglot/worker-compatibility-routing)
for the identity carried by workflow tasks, activities, retries, children and
continue-as-new. A cohort is scoped to `(namespace, task_queue, build_id)`.
Registrations without a `build_id` belong to the unversioned, `null` cohort.

## Rollout State The Server Records {#rollout-state-the-server-records}

| Field | Meaning |
| --- | --- |
| `build_id` | Registered build identity; `null` is the unversioned cohort. |
| `rollout_status` | `active`, `active_with_draining`, `draining`, `stale_only`, or `no_workers`. |
| `drain_intent`, `drained_at` | Persistent operator drain intent and its first timestamp. Repeating a drain does not move the timestamp. |
| `promoted_at`, `rolled_back_at` | Persisted promotion and rollback timestamps. |
| `new_start_selected` | This cohort is the selected explicit promotion. A false value does not rule out the single-cohort fallback described below. |
| `active_worker_count` | Fresh, non-draining registrations. This is not a count of free task slots or proof that every workflow type is covered. |
| `draining_worker_count` | Fresh registrations marked draining, which cannot claim new tasks. They do not necessarily hold a lease. |
| `stale_worker_count` | Registrations whose heartbeat is older than `stale_after_seconds`. |
| `total_worker_count` | Active, draining and stale registrations combined. |
| `runtimes`, `sdk_versions` | Runtime and SDK strings observed in registrations. |
| `last_heartbeat_at`, `first_seen_at` | Registration freshness and age, not proof that a build has no remaining runs. |
| `pending_workflow_tasks` | Workflow-task counts and their diagnosis, described below. |

Drain intent survives worker restarts, missing workers and stale heartbeats.
Resume explicitly clears it. A missing worker is not implicitly resumed.

## Inspect The Rollout {#inspect-the-rollout}

Read the exact namespace and queue before changing capacity:

```bash
curl -fsS "$DURABLE_WORKFLOW_SERVER_URL/api/task-queues/orders-critical/build-ids" \
  -H "Authorization: Bearer $DW_OPERATOR_TOKEN" \
  -H "X-Namespace: orders-prod" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2"
```

```bash
dw task-queue:build-ids orders-critical --namespace orders-prod --json
```

```python
from durable_workflow import Client

async with Client("https://durable-workflow.example", token=operator_token,
                  namespace="orders-prod") as client:
    rollout = await client.list_task_queue_build_ids("orders-critical")
    for cohort in rollout.build_ids:
        print(cohort.build_id, cohort.rollout_status, cohort.total_worker_count)
```

The HTTP snapshot is the reference for additive fields that an older SDK's
response model may not expose.

## Pending Workflow Tasks By Cohort {#pending-workflow-tasks-by-cohort}

`pending_workflow_tasks` counts only workflow tasks on this namespace and queue,
grouped by effective task compatibility, falling back to the run's compatibility.
It is not an activity backlog, number of open runs, throughput or process load.

| Nested field | Interpretation |
| --- | --- |
| `ready_count` | Workflow-task rows in `ready` state. |
| `leased_count` | Workflow-task rows in `leased` state. |
| `total_count` | `ready_count + leased_count`. |
| `status` | `idle` for zero tasks; otherwise `pending`, or `no_compatible_worker` for a non-null build with no active worker. |
| `operator_visible_signal`, `message` | Diagnosis when compatible workers are missing; otherwise these can be null. |

The unversioned cohort does not receive that build-specific missing-worker
diagnosis. Do not treat a missing field, failed request or stale observation as
zero. Re-read the snapshot and check worker heartbeats and type declarations.

For example, an old build can have two ready tasks and one leased task while a
new build has zero pending tasks. A queue total of three does not tell you
which fleet needs capacity. If the old build has zero active workers, restore
compatible old-build capacity or deliberately resume it when its drain intent
was accidental. Adding new-build workers does not consume work pinned to the old
build. If only stale registrations remain, investigate lost processes or
heartbeats before counting them as available capacity.

Reduce a cohort conservatively only after checking its own task counts, fresh
workers, declared workflow types, available slots and drain intent. Keep enough
compatible capacity for leased work and future work from existing runs. A run
waiting on a signal, timer, activity or child can have no pending workflow task
now and still require the old build later. Activities can use other recorded
queues. This endpoint does not establish safe retirement or authority to scale
the last required cohort to zero.

[Server #519](https://github.com/durable-workflow/server/issues/519) tracks a
guarded retirement decision. [Server #534](https://github.com/durable-workflow/server/issues/534)
tracks the separate scrape/scale-to-zero capability. These are not delivered by
this checklist, and there is no documented built-in KEDA/HPA trigger here.

## First Cutover: Unversioned To Versioned {#first-cutover-unversioned-to-versioned}

1. Start the new workers with a stable build id and verify their namespace,
   queue, workflow types and fresh heartbeats.
2. Inspect both cohorts and the original runs. Do not infer compatibility from
   a newer build name or a worker count.
3. Explicitly promote the new build when new starts should select it:

   ```bash
   dw task-queue:promote orders-critical --namespace orders-prod --build-id orders-v2
   ```

4. Keep compatible workers for existing histories. Promotion does not assign
   the new pin to old runs, and draining is not a migration.
5. Retire old capacity only when its remaining dependencies have been resolved
   independently. Empty task counts and zero worker registrations are not that
   proof. Keep the build artifact and recovery path while any run can need it.

## Canary A New Build {#canary-a-new-build}

The Server does not provide a percentage ramp between build cohorts. For new
starts it selects the most recently promoted cohort that is neither draining
nor rolled back. Without such a promotion, it falls back to the only active
cohort advertising the requested workflow type, then the only active cohort on
the queue. Otherwise the run starts unpinned. Several active cohorts serving
the same type with no promotion therefore do not imply a weighted split.

Adding workers changes available capacity, not the promoted new-start policy.
For an isolated canary, choose a separate queue and explicitly route selected
application starts there. Keep old-build workers for old pinned runs when
promoting the new build on the original queue.

## Drain An Older Build {#drain-an-older-build}

Draining prevents that cohort's workflow, activity and query polls from claiming
new work. It does not move those tasks to another build. A poll from a draining
worker returns HTTP409 with `poll_status: "draining"` and
`reason: "worker_draining"`. Existing leases retain their original authority.

```bash
dw task-queue:drain orders-critical --namespace orders-prod --build-id orders-v1
```

Do not issue this just because the new build can claim its own work. Existing
old-build runs may remain waiting after the drain. To recover an early drain,
resume the cohort and restore workers with the required original build:

```bash
dw task-queue:resume orders-critical --namespace orders-prod --build-id orders-v1
```

Draining or stopping every worker is not proof that the build artifact is safe
to remove. There is no guarded retirement decision in this snapshot.

## Roll Back A Bad Build {#roll-back-a-bad-build}

Restore known-good workers, resume their cohort if necessary, and promote that
build for future starts. Then use the deployment rollback action for the bad
build. Rollback does not rewrite existing run pins or deliver their remaining
work to a different build. Preserve compatible workers for those runs or make
an explicit application recovery decision.

Retries, children and continue-as-new inherit the original compatibility pin.
An indefinitely continued chain does not migrate to the newly promoted build.
Use a stable build identity for replay-compatible deployments, or finish the
old chain and start a genuinely new instance under the new-start policy. Do
not edit stored compatibility fields to force a migration.

## Endpoints And Commands Reference {#endpoints-and-commands-reference}

| Intent | HTTP endpoint | CLI | Python SDK method |
| --- | --- | --- | --- |
| Inspect cohorts | `GET /api/task-queues/{taskQueue}/build-ids` | `dw task-queue:build-ids` | `Client.list_task_queue_build_ids` |
| Promote for new starts | `POST /api/task-queues/{taskQueue}/build-ids/promote` | `dw task-queue:promote` | `Client.promote_task_queue_build_id` |
| Drain | `POST /api/task-queues/{taskQueue}/build-ids/drain` | `dw task-queue:drain` | `Client.drain_task_queue_build_id` |
| Resume | `POST /api/task-queues/{taskQueue}/build-ids/resume` | `dw task-queue:resume` | `Client.resume_task_queue_build_id` |

These mutation routes accept `{"build_id":"orders-v2"}` or `{"build_id":null}`
for the unversioned cohort. CLI mutation commands use `--build-id` or
`--unversioned`, never both.

For the complete lifecycle, use `GET /api/deployments` and
`POST /api/deployments/{name}/{promote,drain,resume,rollback}`. Use the returned
name, such as `orders-prod/orders-critical@orders-v2`, with its original namespace.
On the published Apache path, preserve the name's slash separator: encoding the
entire compound name can produce an Apache HTML404 before the API is reached.
Refused lifecycle actions return HTTP409 with a typed `reason` and `blockages`
array. Inspect `expected_resolution`; a refusal is not a successful promotion.
The separate [Server #521](https://github.com/durable-workflow/server/issues/521)
tracks the incomplete current OpenAPI mirror, not a new runtime implementation
or a change to frozen schemas.

## Related References {#related-references}

- [Worker Compatibility and Routing](/docs/polyglot/worker-compatibility-routing)
- [Namespace, Auth, And Worker Registration](/docs/polyglot/namespace-auth-workers)
- [Task Queue Admission](/docs/polyglot/task-queue-admission)
- [Rolling Upgrades](/docs/rolling-upgrades)
- [Server API Reference](/docs/polyglot/server-api-reference)
- [CLI Command Reference](/docs/polyglot/cli-reference)

<!-- Source: docs/polyglot/php.md -->

# PHP SDK

Use `durable-workflow/sdk` when a PHP application or remote worker connects to
the standalone Durable Workflow Server or a Durable Workflow Cloud namespace
runtime URL. This first-party SDK is framework-neutral: it provides the
control-plane client, authentication, transport, public payload codec, replay
handler, and managed remote-worker lifecycle without requiring Laravel or the
embedded engine package.

## Try the local Sample App playground

For the shortest no-Cloud authoring journey, open the current Sample App
[`main` branch in GitHub Codespaces](https://codespaces.new/durable-workflow/sample-app?quickstart=1&ref=main)
and run:

<!-- docs-example id="sdk.php.sample-app-playground" -->
```bash
scripts/playground php
```

The local playground generates caller-owned workflow and activity source,
selects the current stable artifacts, and starts the published
Server and Waterline. It waits for a worker registration whose identity,
workflow type, activity type, and task queue match the generated contract before
starting the workflow. Success requires the expected completed result and
history; the terminal then prints the exact local Waterline run link and the
path to structured JSON evidence.

The package-owned quickstart and API reference below remain the direct path for
users who want to add the SDK to an existing project without Sample App.

For step-by-step onboarding, framework paths, testing, deployment, and
troubleshooting, use the authored
[PHP developer portal](https://php.durable-workflow.com/). For exact
constructor signatures, return types, and exception classes, use its distinct
[generated API reference](https://php.durable-workflow.com/api/). The portal's
[machine-readable contract](https://php.durable-workflow.com/quickstart-contract.json)
identifies the package, runtime forms, role credentials, shipped source files,
expected result, and published-artifact smoke as one tested path.

Cloud customers use the runtime URL and namespace returned during provisioning,
with separate client and worker credentials. See
[Cloud Managed Runtime](/docs/polyglot/cloud-control-plane) for that
connection boundary; the examples below show the same SDK against local
self-hosted values.

Use `durable-workflow/workflow` for the separate embedded Laravel path, where
the application owns workflow state in its existing database and executes work
through its Laravel queues. See [Deployment Modes](/docs/polyglot/deployment-modes/)
for the complete ownership comparison.

## Requirements

- PHP 8.1 or later for the framework-neutral SDK
- For the Laravel service-mode bridge in PHP SDK 3.10 and later: PHP 8.2 or
  later and Laravel 12.69 or later in 12.x, or Laravel 13.30 or later in 13.x
  with PHP 8.3 or later
- A reachable [self-hosted Server](/docs/polyglot/server/) or provisioned
  [Cloud namespace runtime](/docs/polyglot/cloud-control-plane)

## Install

Install the current published PHP SDK. The exact requirement below is generated
from the registry-refreshed published-artifact authority, and Composer records
the resolved package in `composer.lock`:

<!-- docs-example id="php.sdk.install" -->
```bash
composer require durable-workflow/sdk:3.18.2
```

The SDK uses the official `apache/avro` Composer package for the public payload
envelope. Its production dependency graph excludes Laravel, Illuminate,
`durable-workflow/workflow`, and `durable-workflow/server`.

## Start and inspect a workflow

The SDK-owned quickstart creates a clean Composer project, defines one
attributed workflow and activity, starts the worker, starts a unique workflow,
and waits for the result. The same shipped `bootstrap.php`, `worker.php`, and
`client.php` files are installed from the package and executed by the protected
published-artifact smoke, so this page does not maintain a second code listing.

Choose only the runtime connection value:

| Runtime | Value passed to `Client` | SDK request path behavior |
| --- | --- | --- |
| Self-hosted Server | Bare origin such as `http://localhost:8080` | Adds one `/api` segment; callers do not append `/api`. |
| Durable Workflow Cloud | Complete provisioned URI such as `https://cloud.example/api/runtime/v1/namespaces/<runtime-id>` | Preserves the namespace runtime path and appends endpoint `/api` after it. |

Open the [tested PHP path](https://php.durable-workflow.com/) for the exact
commands and visible source. Client operations read
`DURABLE_WORKFLOW_CLIENT_TOKEN`; worker polling reads
`DURABLE_WORKFLOW_WORKER_TOKEN`. The guide keeps those credentials in separate
processes without echoing or committing either value.

`Worker::register()` discovers `#[Workflow]` and `#[Activity]` handlers in the
same source file. The guide also documents the direct
`registerWorkflow()`/`registerActivity()` alternative for callable-first code.
The bootstrap removes autoloader-path selection from users when those shipped
files run in a standalone project, SDK checkout, installed package, or a
playground/container that places them beside its Composer `vendor/` directory.

`WorkflowHandle` follows the current run after a continue-as-new transition.
Use its selected-run methods when an operation must remain guarded to one
specific run.

## Lifecycle, updates, schedules, and visibility

The current public client is broader than selected-run result handling:

- `WorkflowHandle` exposes `describe`, `result`, `signal`, `query`, `cancel`,
  and `terminate`, with selected-run variants for run-specific safety.
- `Client` exposes `listWorkflows` with server filtering and pagination,
  `workflowHistory`, `updateWorkflow`, `cancelWorkflow`, and
  `terminateWorkflow`.
- `requestCancellation()` requests bounded cleanup from cooperating workers.
  Existing `cancel()` and `cancelWorkflow()` close the run immediately. See
  [cooperative cancellation](/docs/polyglot/cancellation) for the authoring and
  recovery contract.
- Schedule methods cover create, describe, list, update, pause, resume,
  trigger, backfill, and delete.
- Operational visibility includes `listNamespaces`, `listWorkers`, and
  `listTaskQueues`, with matching describe methods.

Remote workers register workflow, activity, query, and update handlers through
`registerWorkflow`, `registerActivity`, `registerQuery`, and `registerUpdate`.
Use the generated [PHP SDK API reference](https://php.durable-workflow.com/api/)
for complete parameters and return types.

## Activity retry context and owning run {#activity-retry-context}

PHP SDK **3.6.1 or later** exposes `$context->retryInfo`, `$context->workflowId` and `$context->runId` in service-mode activities. Check `$context->retryInfo->isLastConfiguredAttempt() === true` for the final configured attempt of a bounded policy. `attemptNumber` preserves the original received attempt number.

The retry snapshot preserves three distinct states: `policyProvided === false` means the claim omitted the policy; a provided root `null` is unknown legacy policy; an object with explicit `max_attempts: null` means unlimited. `maxAttempts()` returning `null` alone does not establish unlimited retries. The SDK does not invent a default budget. Cancellation, non-retryable failure, workflow closure and elapsed deadlines can end execution before the budget is exhausted.

The workflow and run IDs identify the original owning run. Missing legacy identity stays `null`. Local activities retain known parent IDs without inheriting a remote retry policy. To signal the owning workflow, require both IDs, pin the signal to `runId` and reuse a stable business-event request ID across retries.

Keep signal processing idempotent for the business event. A stable request ID alone does not establish once-only signal admission.

See the [complete final-attempt and own-run signal examples](https://php.durable-workflow.com/build/workflows-activities/) in the PHP SDK guide.

## Logical activity execution identity {#activity-identity}

PHP SDK **3.10.0 or later** exposes the read-only nullable `ActivityContext::activityExecutionId` received with an activity claim. It identifies the logical activity across retries. `workflowId` and `runId` identify its owning workflow instance and run; `activityAttemptId` identifies an attempt. Do not substitute a task, attempt or session ID for a missing logical execution ID.

The SDK preserves the received nonempty string exactly. Omitted, null, empty or non-string identity values produce `null`. Use an application-owned business key when it is unavailable. The [SDK-owned service-key example](https://php.durable-workflow.com/build/workflows-activities/#inspect-retry-budget-and-the-owning-workflow) shows that fallback; the [property reference](https://php.durable-workflow.com/api/classes/DurableWorkflow-Worker-ActivityContext.html#property_activityExecutionId) defines the public API. Scope keys to the application and effect, use distinct keys for distinct effects, and reconcile uncertain outcomes with the destination. Identity is public metadata, not claim authority or an exactly-once external-effects guarantee.

Ordinary and cooperatively supervised remote callbacks read the actual claim field. Session callbacks expose it only when their claim includes it. Inline local and Server-free callbacks without Server admission have `null` execution identity, even when their parent workflow/run IDs are known. Historical prepared-local source forwards its validated admission ID; that source behavior does not establish native prepared-local support. These are source/dataflow boundaries, not installed retry observations for every execution mode.

## Activity heartbeats {#activity-heartbeats}

For remote activities, call `ActivityContext::heartbeat($details)` during bounded work, well before the configured heartbeat timeout. The first window starts at runtime claim, not when PHP receives the task. Request latency, pauses and transport retries consume that window. Follow the [cadence guidance](../features/heartbeats.md#heartbeat-cadence-and-receipts).

The method returns `void`. In the direct handler path, a stop reply raises `ActivityCancelled`. HTTP and transport failures from `Client::heartbeatActivityTask()` surface as `ServerException`. A cooperatively supervised worker can stop or abandon its callback when a heartbeat or ownership check fails instead of delivering a normal response to activity code. Do not assume cleanup code always runs. Do not suppress a failure after losing the claim or submit success after losing authority.

A returned call does not commit external work or provide a resumable checkpoint. Keep external effects idempotent and do not treat a transport retry as a fresh heartbeat window.

## Run a remote PHP worker

Workflow handlers are ordinary callables that run as straight-line code inside
a managed Fiber. Call operations such as `WorkflowContext::activity()`
directly; the SDK suspends the Fiber at durable decisions and returns recorded
results during replay without repeating external activity. Do not declare a
workflow as a Generator or yield `WorkflowContext` commands: Generator results
are rejected. The managed worker registers its workflow and activity type
names, polls the public worker protocol, heartbeats, completes or fails tasks,
and handles graceful shutdown when `pcntl` is available.

### First-completion selection

Use `WorkflowContext::select()` to start independent deferred activities,
child workflows, timers, condition waits, or nested ordinary barriers and
resume from the first durably committed winner. The returned `SelectionResult`
contains stable member keys and identities plus one handle per member:

```php
$selected = $ctx->select([
    'resolver' => fn () => $ctx->activity('resolve-request', [$requestId]),
    'input' => fn () => $ctx->waitCondition(
        fn (): bool => $this->resolution !== null,
        key: 'resolution-ready',
    ),
    'deadline' => fn () => $ctx->sleep(2),
]);

if ($selected->key === 'deadline') {
    $selected->handles['resolver']->cancel();
}
```

Selection does not cancel non-winners. Call `await()` on a handle to consume
its eventual result or `cancel()` to record explicit cancellation. A cold worker
or completed-history replay consumes the recorded winner even when duplicate or
later input and terminal events appear in another delivery order.
`cancel()` returns void and does not report its terminal outcome. Only
`SelectionOperationCancelled` history proves cancellation won; replay advances
past an unmarked cancel request, and if the operation completed first,
`await()` still returns that result.

### Durations {#durations}

From PHP SDK 3.9.0, workflow timers and timeouts take the same duration forms as
the embedded Laravel package. `sleep()`, `deferTimer()`, the `timeout` of
`waitCondition()` and `deferCondition()`, the activity timeout options, the
child workflow `execution_timeout_seconds` and `run_timeout_seconds` options and
the service operation `wait_timeout_seconds` option accept int or float
seconds, a string such as `'30 minutes'`, or a `DateInterval` or
`CarbonInterval`:

```php
use Carbon\CarbonInterval;

$verified = $ctx->waitCondition(
    static fn (): bool => $ctx->signals('verify') !== [],
    key: 'email-verified',
    timeout: '30 minutes',
);

$ctx->sleep(CarbonInterval::hours(2));

$receipt = $ctx->activity('payments.capture', [$orderId], [
    'start_to_close_timeout' => '30 seconds',
    'schedule_to_close_timeout' => '10 minutes',
]);
```

Each value converts once to whole seconds, rounding up, so `'30 minutes'`,
`1800` and `CarbonInterval::minutes(30)` record the same history. Strings
convert exactly as the embedded `timer()` and `await()` do. The SDK rejects
ISO 8601 strings such as `'PT30M'` (pass `new DateInterval('PT30M')` instead),
negative values, fractions of a month or a longer calendar unit, parts with more
than 15 integer digits, and unparseable text. A numeric string such as `'30'` keeps its
earlier meaning: timers and condition timeouts round it up to whole seconds,
and activity and child workflow options pass it through unchanged. A zero
duration is valid for timers, condition waits and `wait_timeout_seconds`;
activity and child workflow timeouts need at least one second. Earlier SDK
releases accept seconds only. PHP SDK 3.9.0 adds `nesbot/carbon`
(`^2.72.6|^3.8.4`) as a required dependency, so durations convert with the
same parser as the embedded package.
The [PHP SDK duration reference](https://php.durable-workflow.com/build/workflows-activities/#durations)
lists every unit.

## Entry compatibility and recovery {#entry-compatibility-and-recovery}

**Released support.** PHP SDK 2.5.4 introduced the entry safeguard below, with routing and recovery support in Server 2.9.9. These are minimum feature versions; install current stable releases for security fixes.

The managed worker checks the handler's recorded positional input before entering its Fiber. Missing required arguments or incompatible declared PHP types raise `DurableWorkflow\Exception\WorkflowEntryArgumentsMismatch` and report the task reason `workflow_entry_arguments_mismatch`. Optional defaults apply only to omitted input; they never replace an incompatible recorded value. A `TypeError` thrown after successful entry remains an application failure.

The worker advertises `workflow_entry_arguments_validation` automatically. With the matching Server support, a refused task is replay-blocked and the original run stays nonterminal. An older runtime may instead retry the failed task, so an SDK upgrade alone does not establish the complete recovery contract.

Inspect the original with `Client::describeWorkflow($workflowId, $runId)` and the selected-run debug endpoint. Restore compatible code before calling `Client::repairWorkflow($workflowId, $runId)`, then verify that same run resumes without changing input, history, routing identity or deadlines. Do not catch this failure inside the workflow body or repeatedly repair incompatible code. See the [service recovery procedure](../failures-and-recovery.md#service-entry-recovery) and [definition routing](./worker-compatibility-routing.md#definition-fingerprint-evolution).

## Framework service mode and embedded Laravel

The same package ships first-party
[Laravel service-mode](https://github.com/durable-workflow/sdk-php#laravel-service-mode)
and
[Symfony service-mode](https://github.com/durable-workflow/sdk-php#symfony-service-mode)
bridges. They retain framework dependency injection, configuration, logging,
console workers, and test fakes while connecting to Cloud or Server.

Those bridges are distinct from
[embedded Laravel workflows](/docs/installation/), where
`durable-workflow/workflow` makes the Laravel application itself own durable
state and execute through Laravel queues. See
[Laravel Adoption and Runtime Transition](/docs/laravel-adoption/) for the
same representative Laravel use case across v1, v2 embedded, and this shipped
service-mode bridge, including drain and rollback. Use
[Deployment Modes](/docs/polyglot/deployment-modes/) for the wider runtime
boundary comparison.

## Query results with binary values {#query-binary-results}

PHP SDK 3.18.2 fixes query completion for valid Avro results containing nested
`AvroBinaryValue` bytes. The `result_envelope` remains authoritative. The SDK
omits only the optional plain `result` projection when it cannot be JSON
encoded; ordinary JSON-compatible results keep that projection.

[Published PHP 3.18.2 / native Server 3.9.1 installed controls](https://github.com/durable-workflow/sdk-php/pull/415#issuecomment-6858062312)
(PHP 8.3.36, linux/amd64, SQLite) verify an actual Worker round-trip for nested
binary bytes, ordinary JSON, signed zero and `null`, plus workflow completion
and graceful Worker deregistration. This is a compatible query fix; it does
not add a capability or establish full request-key or timeout-cleanup support.
See the [PHP query guide](https://php.durable-workflow.com/build/messages/#register-a-read-only-query).

## Read an accepted update later {#read-an-update-outcome}

The selected PHP SDK 3.18.2 supports reading a retained update by its stable ID
without admitting it again. `Client::updateOutcome($workflowId, $updateId)`
defaults to `waitFor: 'completed'`: pending work throws `UpdateStillPending`;
a completed result may legitimately be `null`. `updateOutcomeReceipt()` defaults
to `waitFor: 'accepted'` and returns the Server receipt so callers can inspect
`update_status` and `wait_timed_out` separately from a completed result.

These GET reads require the `update_outcome_reads` capability (native Server
3.6.0 and later). Without it, the SDK throws `LogicException` and does not
resubmit the update. An initial `updateWorkflow()` pending exception retains
the caller's request ID. A pending GET read retains `updateId`, but its original
request key may be unavailable: `$pending->requestId` is then `''`, meaning
unknown. Continue reading with that update ID; never submit an empty, inferred
or internal hashed key as the original request ID.

The [published PHP guide](https://php.durable-workflow.com/build/messages/#apply-a-tracked-update)
shows the workflow-side handler and client journey. [Installed PHP 3.18.1 /
native Server 3.9.1 controls](https://github.com/durable-workflow/sdk-php/pull/411#issuecomment-6855609359)
verify pending-to-completed-null reads without a second update admission, plus
[completion-size handling](./server-config-reference.md#workflow-task-completion-size).
This evidence does not establish full request-key acceptance under shared #216
or native timeout-cleanup support.

## Upgrade to PHP SDK 3.12 through 3.14 {#upgrade-to-3-12-through-3-14}

PHP `^8.1`, the Laravel bridge floors, worker protocol `1.20` and control plane
`2` are unchanged from 3.11. Check these behavior changes before deploying:

- **Updates that outlast their wait.** From 3.13, when a
  `waitFor: 'completed'` update wait ends before the update finishes,
  `Client::updateWorkflow()` and `WorkflowHandle::update()` throw
  `UpdateStillPending`. Earlier releases returned the Server receipt as if it
  were the update's result. To keep waiting, call the update again with
  `$pending->requestId`. That does not apply the update twice.
- **Stale worker registration.** From 3.13, `Worker::run()` registers again
  after `stale_worker_registration` instead of stopping. Tasks the fenced
  worker held are dropped without completing, and the Server redelivers them.
  Each recovery logs `worker.reregistered` and increments
  `durable_workflow_worker_reregistrations`. Pass
  `stopOnStaleRegistration: true` to keep the old behavior.
- **Local activity backoff.** From 3.13, a local activity with more retries
  than `backoff_seconds` entries repeats the last delay instead of retrying
  immediately, as Server-scheduled activities do.
- **Condition waits in groups on native Server.** Native Server doesn't accept
  a condition or signal wait inside `all()` or `parallel()` yet. From 3.13.1,
  the worker fails that workflow task with `grouped_condition_wait_unsupported`
  and keeps running, instead of exiting. Await the condition outside the group.

New in these releases:

- **Sticky history suffixes (3.14).** Workers advertise
  `sticky_history_suffix`. On Server 3.5.0 or later, a worker that already
  holds a run's history receives only the events after the sequence it cached.
  A suffix that doesn't continue the cached history exactly is dropped and the
  full history is fetched. Older Servers keep sending full history.
- **Schedule origin (3.13).** `WorkflowContext::scheduleOrigin()` returns the
  schedule id, occurrence time and trigger number of the schedule fire that
  started the run, or `null` for an ordinary start. The value is the same on
  replay. From Server 3.5.0, a run that continues as new keeps its schedule
  origin.
- **Activity continuations (3.12).** Pass `enableActivityContinuation: true` to
  the `Worker` constructor to submit the next workflow commands together with a
  successful activity result when the Server advertises
  `activity_workflow_continuation` (native Server 3.4.0 or later). It defaults
  to `false`. Turning it on currently costs throughput: in our benchmark under
  load, workflows ran 24–49% slower with it on and used twice as many database
  transactions ([measurement](https://github.com/durable-workflow/server/issues/1078)). Use 3.12.1 or later if you enable it, and drain or disable
  opted-in workers before downgrading the Server or adding older Server nodes.

See the [3.14.0 changelog](https://github.com/durable-workflow/sdk-php/blob/3.14.0/CHANGELOG.md)
for the complete list.

## Upgrade to PHP SDK 3.10 and 3.11 {#upgrade-to-3-10-and-3-11}

Read the [PHP SDK upgrade guide](https://github.com/durable-workflow/sdk-php/blob/3.11.0/docs/portal/operate/upgrading.md)
before deploying. Upgrade every worker that can receive work from a task queue
together, including standby and autoscaling pools.

- **Laravel bridge.** From 3.10, the Laravel service-mode bridge follows
  Workflow 2.21: PHP `^8.2` and patched Laravel `^12.69.0|^13.30.0`. Laravel
  10 and 11 are no longer supported. The provider refuses unsupported or
  unknown framework and PHP versions before registering anything. The bare SDK
  keeps PHP `^8.1` and Symfony 6.4/7/8 support.
- **Condition fingerprints.** From 3.10, new condition waits record v2
  (`dwphp2:sha256:`) predicate fingerprints. Older workers reject them, so
  rolling workers back below 3.10 after v2 waits exist can fail replay. Stop or
  drain older workers before replacements start, deploy PHP sources atomically
  and restart every long-running worker, including workers using OPcache.
  Workers that replay v2 waits need PHP's tokenizer. Existing v1 waits still
  require an exact source-line match.
- **Static credentials.** From 3.10, static credentials are validated as
  given. Malformed UTF-8, Unicode whitespace at either edge and embedded
  control characters now fail instead of being trimmed or ignored. Check shared,
  client, worker and framework process-handoff values before upgrading.
- **Occurrence deadlines.** In 3.11, a new top-level condition wait keeps its
  first deadline across false wakes when the Server advertises
  `condition_wait_occurrence_deadline` (native Server 3.3.0 or later). Without
  that capability, workers send the same commands as before. Grouped condition
  waits do not opt in yet. On an older Server, upgrade every worker on the queue
  to 3.11 before upgrading the Server. On a Server that already advertises the
  capability, drain or stop every older worker before any 3.11 worker consumes
  the queue. Once an anchored wait exists, do not roll that queue's workers back
  below 3.11.

PHP SDK 3.10 also adds `WorkflowContext::startChildWorkflow()`, which returns a
`ChildWorkflowHandle` once the child start commits and needs a runtime that
advertises `child_workflow_start_turn`. Other additions are
`ActivityContext::activityExecutionId`, optional typed declarations for
`upsertSearchAttributes()`, lazy workflow list and history iterators, and the
`enableCompletionPipelining` Worker option, which defaults to `false`.
Completion-carried polls are used only when the runtime advertises them. See
the [3.11.0 changelog](https://github.com/durable-workflow/sdk-php/blob/3.11.0/CHANGELOG.md)
for the complete list.

## Protocol and release boundary

The SDK declares its supported server range, worker protocol version,
control-plane version, and payload codecs in Composer metadata. The server also
publishes its accepted protocol and codec set from `GET /api/cluster/info`.
Check runtime discovery during deployment instead of inferring compatibility
from a server patch version.

The PHP SDK is versioned independently from the Laravel package. Keep the
exact published pin in runnable examples and evaluate release notes when
upgrading SDK releases. Compatibility follows the SDK's declared contract,
not matching package version numbers.

## Related references

- [Standalone Server](/docs/polyglot/server/)
- [Deployment Modes](/docs/polyglot/deployment-modes/)
- [Worker Protocol](/docs/polyglot/worker-protocol/)
- [Capability Index](/docs/capabilities/)
- [PHP developer portal](https://php.durable-workflow.com/)
- [PHP API reference](https://php.durable-workflow.com/api/)
- [PHP executable quickstart contract](https://php.durable-workflow.com/quickstart-contract.json)
- [PHP SDK source](https://github.com/durable-workflow/sdk-php)

## Workflow list and history pages {#workflow-list-and-history-pages}

These published calls return **one page**, not an automatic complete read: `Client::listWorkflows(...)` / `Client::workflowHistory(...)`. Continue with the returned opaque token while keeping namespace, filters and selected workflow/run fixed; stop at the final null/empty token and handle page errors explicitly. A partial page is not a replay bundle. Use the SDK history-export API for offline replay. The [published PHP 3.2.0 client source](https://github.com/durable-workflow/sdk-php/blob/3.2.0/src/Client.php) defines the page parameters.

PHP SDK 3.10.0 and later also provide `Client::iterateWorkflows()` and `Client::iterateWorkflowHistory()`. They read a complete workflow listing or exact-run history lazily, one page at a time, keeping the opaque cursor, namespace and filters. Errors on later pages propagate, and malformed continuations or token cycles are refused instead of truncating or looping. See the [published PHP 3.11.0 client source](https://github.com/durable-workflow/sdk-php/blob/3.11.0/src/Client.php).

## Bounded failure reports {#bounded-failure-reports}

PHP SDK **3.5.0** introduced bounded outgoing failure reports. Defaults are **62 KiB UTF-8 for the message**, **128 KiB for the stack**, **64 KiB for combined serialized optional details and diagnostics**, and **1 MiB for the complete JSON-encoded body**. Markers, JSON escaping, claim identity and other commands in the batch count toward that body limit. A lower configured or transport-declared limit takes precedence. Failure type, class, code, retry policy and claim identity are preserved.

`withFailureReportLimit()` returns a new client with a smaller whole-request budget, from 1 to 1,048,576 bytes. Use the returned instance. A lower transport-declared cap also applies.

```php
<?php

use DurableWorkflow\Client;

$client = (new Client(
    baseUri: 'https://workflow.example.com',
    controlToken: 'operator-token',
    workerToken: 'worker-token',
))->withFailureReportLimit(512 * 1024);
```

A visible truncation marker and original byte counts identify shortened fields. Avro details remain intact or are omitted whole. An unrepresentable required envelope raises `DurableWorkflow\Exception\FailureReportRefused` before HTTP submission. Local failures are normalized before workflow code observes them, so first execution and cold replay see the persisted message. An already observed local message remains immutable during final batch fitting.

<!-- Source: docs/polyglot/portable-worker-affinity.md -->

# Portable Worker Affinity

Service workers implement the features listed below through Server's worker
protocol. SDK versions and worker configuration determine what is available.

Local activities, worker sessions, and sticky execution share one portability
rule: a service worker must declare each feature as supported or explicitly
refused. Protocol version `1.18` is the floor for these declarations. The
server rejects a flat routing capability unless the worker's structured
manifest marks the same feature as supported.

## SDK support

| SDK worker | Local activities | Worker sessions | Sticky execution |
| --- | --- | --- | --- |
| PHP | Supported | Supported | Supported |
| Python | Supported since 2.2.0 | Supported since 2.3.0 | Supported since 2.5.0 with `sticky_cache_capacity` above zero |
| Rust | Supported since 3.1.0 with `Worker::local_activities(true)` | Supported since 3.2.0 with `Worker::worker_sessions(true)` | Supported since 3.4.0 with `Worker::sticky_cache(...)` |

Workers advertise only the capabilities implemented and enabled for their
profile. Rust local activities, sessions and sticky execution, and Python sticky execution,
require explicit opt-in. Older SDK versions refuse capabilities they do not
implement. Ordinary workflows and
queued activities use complete durable-history replay without these optimizations.

## Local activity recording

PHP, Python and opted-in Rust workers run a local activity inside the workflow worker. The
workflow-task completion contains the arguments, attempt outcomes, retry and
timeout settings, heartbeat progress, and terminal result or failure. The
server records that sequence atomically as normal activity history marked
`execution_mode=local`.

Replay consumes the recorded terminal activity event. It does not invoke the
local handler again. A worker lost before Server commits completion can execute
the handler again, so external effects must be idempotent.

Ordinary PHP workers execute the synchronous handler
inline. Cancellation and elapsed heartbeat, per-attempt, and total timeouts
are observed before an attempt, at `ActivityContext::heartbeat()`, or after the
handler returns. These handlers must remain short and divide blocking work
with safe heartbeat boundaries.

Python and Rust async callbacks must yield to their language runtime.
Rust renews the exact workflow-task lease independently of application
heartbeats, drops the callback on timeout, lost authority or worker shutdown,
and records bounded attempt and heartbeat reports. See the
[Rust local activity example](https://github.com/durable-workflow/sdk-rust/blob/3.1.0/examples/local_activities.rs)
and [API reference](https://rust.durable-workflow.com/durable_workflow/struct.LocalActivityOptions.html).

PHP workers that enable [cooperative cancellation](/docs/polyglot/cancellation)
use protocol 1.20 and supervised callback processes for prepared local activities.
Rust inline local execution uses a separate ordinary worker profile and cannot
be combined with prepared cooperative local supervision. Blocking callbacks
need process supervision to guarantee physical stop.

## Worker session lifecycle

The PHP, Python and Rust SDKs expose typed session options and create, use, renew, and close
operations. Options include requirements, queue, lease duration, total TTL,
maximum concurrent activities, and reacquisition policy. A worker closes the
sessions it holds during graceful shutdown.

If a holder disappears, its lease and concurrency reservation expire. A new
holder may reacquire the session when requirements match, but it must rebuild
worker-local resources before the first activity uses them. Session identity
never makes process memory durable.

Reacquisition preserves the session's original absolute TTL. Renewal extends
holder authority without extending that TTL. Use Server 2.5.1 or newer for
original TTL preservation and recorded session routing during cold replay.
See the [Rust session example](https://github.com/durable-workflow/sdk-rust/blob/3.2.0/examples/worker_sessions.rs)
and [WorkerSessionOptions](https://rust.durable-workflow.com/durable_workflow/struct.WorkerSessionOptions.html).

## Sticky execution and cold replay

The PHP, Python and Rust caches are bounded and keyed by the exact workflow ID, run
ID, and worker build ID. They report `hit`, `miss`, `eviction`, and `forced_cold_replay`.
Expiry, eviction, worker replacement, holder loss, or a build mismatch discards
the optimization and replays complete durable history.

Use Python SDK 2.5.0 with Server 2.5.10 or newer. Its cache is disabled by
default. Enable it with `sticky_cache_capacity` and set the retained history
byte limit and TTL for your worker. It stores encoded durable history and can
reuse validated page cursors to save repeated downloads. Allow additional
memory for decoding and deterministic replay. See the
[Python sticky execution guide and runnable example](https://python.durable-workflow.com/guides/sticky-execution/).

Rust SDK 3.4.0 supports the same published Server baseline. Enable its cache
with `Worker::sticky_cache(StickyCacheOptions::new(...))` and configure the
encoded-history byte limit and TTL. Each replay decodes a fresh snapshot.
Reused history cursors are checked against the current task lease and durable
history before a warm hit is accepted. The cache holds history, not live
workflow instances or session resources. See
[StickyCacheOptions](https://rust.durable-workflow.com/durable_workflow/struct.StickyCacheOptions.html)
and the [runnable Rust example](https://github.com/durable-workflow/sdk-rust/blob/3.4.0/examples/sticky_execution.rs).

Server 2.5.10 preserves full replay for PHP SDKs before 2.2.0. Use PHP SDK
2.2.0 or newer to consume replay hints for histories beginning with StartAccepted.

Sticky affinity does not reserve ready tasks for the cache owner or give that
owner matching priority. An eligible worker can claim a task while another
live worker's affinity is active, including when the owner has no free
workflow slots. That claim is labelled `forced_cold_replay`. Increasing the
affinity TTL does not make matching wait for the owner.

A forced cold replay is diagnostic evidence that the cache optimization was
unavailable. It is not a workflow correctness failure. Workflow code must
remain deterministic with an empty cache. See the
[matching and diagnostic rules](/docs/features/sticky-execution#routing-identity).

## Safe defaults and rolling fleets

Ordinary workflows require no session or sticky configuration. Mixed-version
fleets fail closed at the protocol floor: the server checks the negotiated
version, the flat capability, the structured manifest, and exact sticky cache
identity before accepting feature-specific completion data.

The published [cross-SDK scenario manifest](/platform-conformance/portable-worker-affinity-runtime-scenarios.json)
covers manifest truth, local-activity replay, session holder loss and
reacquisition, sticky hits and eviction, worker replacement, forced cold
replay, and zero-configuration workflows.

For embedded Laravel implementations, see
[Local Activities](/docs/features/local-activities),
[Worker Sessions](/docs/features/worker-sessions), and
[Sticky Execution](/docs/features/sticky-execution). Those APIs belong to the
workflow package. Use the corresponding SDK's API for service workers.

<!-- Source: docs/polyglot/avro-value-protocol.md -->

# Avro Value protocol

Durable Workflow 2.0 uses one fixed recursive
[`durable_workflow.protocol.Value`](/schemas/v2/durable_workflow.protocol.Value.v1.avsc)
schema for every Avro payload. Workflow inputs and results, activity values and
failure details, signals, queries, updates, replay history, and externally
stored payloads all use the same schema. Applications do not publish their own
Avro schemas, and the platform does not require a network schema registry.

The value union has distinct named branches for null, boolean, signed 64-bit
integer, finite double, bytes, UTF-8 string, list, and string-keyed map. That
keeps `7` distinct from `7.0`, text distinct from bytes, and lists distinct from
maps in PHP, Python, and Rust.

## Wire frame

The `blob` field is base64 around standard Avro single-object bytes:

```text
C3 01 || 8-byte little-endian CRC-64-AVRO fingerprint || Avro datum
```

Schema v1 has fingerprint `e2a33dff55802237`. SDKs bundle the immutable schema
for each supported fingerprint, select the writer schema from the frame, and
resolve it against the current reader. An unknown fingerprint or incompatible
new branch fails as `unsupported_payload_schema`; decoders do not guess or
fall back to JSON.

Future value kinds are new uniquely named record branches appended to the
union. Released branches are never reordered or reused.

## Value policy

- Map keys must be strings. SDKs reject other keys instead of stringifying
  them.
- Integers must fit the signed 64-bit Avro `long` range.
- Doubles must be finite; NaN and infinities are rejected.
- Python `bytes` and Rust `AvroValue::Bytes` select Avro `bytes`. PHP callers
  use `AvroBinaryValue::fromBytes()` because a PHP string alone cannot declare
  whether it is text or binary.
- Decimal, arbitrary-precision integer, date/time, UUID, enum, dataclass,
  Pydantic, and domain objects require explicit adapters to a canonical value
  kind.

Avro is the only Durable Workflow 2.0 payload codec. HTTP request and response
documents remain JSON transport, but every durable value inside those
documents uses this fixed schema and single-object frame. A `json` codec tag,
an unknown codec, or an untagged raw durable blob fails closed with
`unsupported_payload_codec`; runtimes never transcode or guess.

## JSON inspection projection

Run descriptions retain `input_envelope`, `output_envelope`, and result
envelopes as the lossless payload authority. JSON-facing inspection surfaces
such as the CLI and Waterline render values that JSON cannot represent with a
typed projection:

```json
{"$type":"bytes","base64":"AP8="}
{"$type":"map","entries":[{"key":"0","value":"zero"}]}
```

The map projection is used for empty maps and numeric-looking string keys that
PHP arrays cannot retain without changing their type. Ordinary scalars, lists,
and unambiguous string-keyed maps remain ordinary JSON values. Consumers that
need the original typed value decode the accompanying envelope rather than the
display projection.

## Repeatable benchmark

Each SDK ships the same representative-value benchmark and enforces a budget
for its selected production path:

```bash
# PHP SDK checkout
composer benchmark-avro-value

# Python SDK checkout
python benchmarks/avro_value.py --enforce

# Rust SDK checkout
cargo run --release --example avro_value_benchmark -- --enforce
```

The JSON output compares compact JSON, the removed JSON-in-Avro wrapper, and
the fixed typed schema. It reports raw datum, framed payload, and actual
`{codec, blob}` HTTP-envelope sizes together with end-to-end adapter,
encode, and decode latency. `AVRO_VALUE_ENCODE_BUDGET_US` and
`AVRO_VALUE_DECODE_BUDGET_US` can tighten the defaults on a qualification
runner. Release CI executes these commands with budget enforcement; a
production-path regression must be explained or corrected before release.
The old wrapper implementation exists only inside the benchmark, not as a
runtime compatibility path.

## Deployment preflight

Server bootstrap inventories every persisted `payload_codec` and verifies the
single-object magic and fixed-schema fingerprint of inline frames,
nested-history envelopes, and external payload references. Deployment stops
before the new runtime starts if any active or replay-relevant non-Avro payload,
untagged payload, corrupt reference, or obsolete frame exists. Active runs may
drain on the current prerelease; retained terminal and replay-relevant state
must follow the backup-first
[prerelease history migration](/docs/polyglot/prerelease-history-migration).
Exporting a run does not alter the rejected database state. Never delete
history to bypass the preflight.

<!-- Source: docs/polyglot/cancellation.md -->

# Cooperative cancellation

Server 2.5.0 adds cooperative cancellation through protocol 1.20. It requires
workers that explicitly enable cooperation. Check runtime
discovery and the SDK's capability support before using it.

## Request cleanup

Use `requestCancellation()` in the PHP SDK, `request_cancellation()` in Python
or Rust, or the Server's `request-cancellation` route. The run stays active while
workflow code handles the request and performs cleanup. Successful cleanup ends
the run as `Cancelled`.

```http
POST /api/workflows/{workflowId}/runs/{runId}/request-cancellation
X-Durable-Workflow-Control-Plane-Version: 2
Authorization: Bearer <operator-token>
X-Namespace: <namespace>
Content-Type: application/json

{"reason":"Order withdrawn","cleanup_timeout_seconds":30}
```

The selected run must still be current. A first accepted request returns HTTP
202. A duplicate returns HTTP 200 with the original request identity and deadline.
Changing the timeout on a duplicate does not extend the cleanup budget.

The immutable cancellation context exposes the request and root IDs, reason,
requester, source, original request time, deadline and ordered lineage. Workflow
code receives it at the recorded delivery boundary. Earlier completed calls
replay normally. The deterministic `deadline()` and `remaining()` helpers let
cleanup make the same decisions after a restart.

Existing `cancel()` closes a run immediately and does not resume workflow
cleanup. `terminate()` also closes it immediately, with a `Terminated` outcome.

## Choose operation policies

| Policy | Child workflow | Remote activity | Local activity |
| --- | --- | --- | --- |
| TryCancel | Request child cleanup and continue parent cleanup | Request callback stop and continue workflow cleanup | Stop the supervised callback and continue cleanup |
| WaitCancellationCompleted | Wait for the child's recorded terminal outcome | Wait for the original callback's durable stop receipt | Join the original callback's durable stop receipt |
| Abandon | Leave the child independent | Leave the activity independent under its original finite total timeout | Rejected before admission |

Parent closure has a separate policy. `RequestCancellation` sends a cooperative
request with the original cancellation lineage and bounded deadline.
`RequestCancel` retains its legacy immediate cancellation behavior.

Each cooperating child inherits the root request identity and cleanup budget.
A child can receive a narrower deadline, but cannot enlarge that budget. An
abandoned operation has an independent lifetime and is outside that cleanup
join.

## Supervision and recovery

Cooperative workers observe cancellation independently of application activity
heartbeats. PHP uses supervised callback processes. Python supervises async and
synchronous callbacks. Rust supervises async callback futures. Detached work and
downstream services need their own cancellation and idempotency behavior.

Cleanup can use a cancellation shield to perform durable activities, timers and
child calls. The shield preserves the original deadline. If a worker dies,
a replacement replays the same delivery boundary and completed cleanup calls
before resuming. Expiry or termination revokes remaining execution authority.

Embedded Laravel workflows use `cancellationShield()`. See [Protected cleanup](../features/cancel-and-terminate.md#protected-cleanup). Saga compensation in embedded mode and in the PHP and Python SDKs runs under a shield automatically.

An old activity attempt cannot publish a result after it is fenced. Already
accepted external side effects still need application idempotency or
reconciliation.

## Inspect the cascade

Run diagnostics expose `cancellation_cascade`, including the root request,
participating runs, activity attempts, policies, stop receipts, deadlines and
cleanup outcomes. The CLI and Waterline present the same view.

The lifecycle distinguishes `requested`, `delivered`, `cleaning_up`, `cancelled`,
`deadline_expired` and `terminated`. An incomplete view identifies what cannot
be verified rather than treating a missing receipt as proof of success.

A request against an incompatible active worker claim returns an explicit
capability refusal without recording a partial cancellation. Upgrading a
registration does not upgrade a claim that was already issued.

## Release boundary

Whole-run requests, child and activity policies, bounded shielded cleanup,
recovery and cascade inspection form the supported release. Independently
cancellable operation scopes remain a disabled source preview and are marked
experimental in the protocol reference.

See the [worker protocol specification](/platform-protocol-specs/worker-protocol-api.openapi.yaml)
for exact request, receipt and capability shapes. Language-specific authoring
guides live on the [PHP](https://php.durable-workflow.com/),
[Python](https://python.durable-workflow.com/) and
[Rust](https://rust.durable-workflow.com/) SDK sites.

<!-- Source: docs/polyglot/cli-python-parity.md -->

# Client and Worker Capabilities

Choose a surface for what the process needs to do. The `dw` CLI and the
first-party PHP, Python, and Rust SDKs share the v2 control-plane and worker
contracts, but their operator, client, and worker roles are not identical.

This all-client guide remains at the original
`/docs/polyglot/cli-python-parity/` route so existing links keep working.

## Capability comparison

| Capability | `dw` CLI | PHP SDK | Python SDK | Rust SDK |
| --- | --- | --- | --- | --- |
| Workflow lifecycle | **Supported:** start, list, inspect, wait, cancel, terminate, and archive operator commands. [Commands](./cli-reference.md#workflow-commands) | **Supported:** start, describe, list, await results, cancel, and terminate through `Client` and `WorkflowHandle`. [Lifecycle evidence](./php.md#lifecycle-updates-schedules-and-visibility) | **Supported:** start, list/describe, await results, cancel, and terminate through the async client. [Workflow operations](./python.md#workflow-operations) | **Supported:** start, describe, await results, cancel, and terminate through `Client` and `WorkflowHandle`. [Terminal operations](./rust.md#cancel-terminate-and-handle-terminal-outcomes) |
| Signals | **Supported:** send by workflow ID or selected run. [Commands](./cli-reference.md#workflow-commands) | **Supported:** client and handle send methods plus worker signal history. [PHP API](./php.md#lifecycle-updates-schedules-and-visibility) | **Supported:** async client sends and workflow workers handle signals. [Messages](./python.md#signals-queries-and-updates) | **Supported:** client and handle sends plus worker signal handling. [Rust API](https://rust.durable-workflow.com/) |
| Queries | **Supported:** execute a named read-only query with structured output. [Commands](./cli-reference.md#workflow-commands) | **Supported:** client and handle queries plus registered worker query handlers. [PHP API](./php.md#lifecycle-updates-schedules-and-visibility) | **Supported:** async client queries plus replayed worker query handlers. [Messages](./python.md#signals-queries-and-updates) | **Supported:** client queries and replayed worker query handlers when runtime discovery advertises query-task support. [Rust API](https://rust.durable-workflow.com/) |
| Updates | **Supported:** submit and wait for accepted or completed outcomes. [Commands](./cli-reference.md#workflow-commands) | **Supported:** `updateWorkflow` and `registerUpdate`; the PHP SDK does not expose validator authoring and declares no validators. [PHP API](./php.md#lifecycle-updates-schedules-and-visibility) | **Supported:** clients, workflow handlers, and synchronous declared validators when discovery advertises the pre-accept contract. Validator-bearing workers refuse unsupported runtimes. [Messages](./python.md#signals-queries-and-updates) | **Supported:** client, handle, Avro payloads, and registered worker update surfaces; the Rust SDK does not expose validator authoring and declares no validators. [Updates](./rust.md#workflow-updates) |
| Schedules | **Supported:** complete schedule lifecycle, backfill, and audit history. [Commands](./cli-reference.md#schedule-commands) | **Supported:** complete schedule lifecycle and listing through `Client`. [PHP API](./php.md#lifecycle-updates-schedules-and-visibility) | **Supported:** complete async schedule lifecycle and audit-history paging. [Schedules](./python.md#schedules) | **Not supported:** the current Rust SDK does not claim a schedule-management API. Use CLI, PHP, Python, or the server API. |
| Visibility | **Supported:** workflow/run search, workers, task queues, history, and diagnostic JSON. [Commands](./cli-reference.md#workflow-commands) | **Supported:** workflow filtering/pagination, history, namespaces, workers, and task queues. [Visibility evidence](./php.md#lifecycle-updates-schedules-and-visibility) | **Supported:** workflow, schedule, namespace, worker, queue, history, and search-attribute client surfaces, including lazy workflow-list and explicit-run history iterators. [Pagination](./python.md#workflow-list-and-history-pages) | **Different:** selected-run describe/result is supported; fleet-wide list/search and namespace administration are not claimed. [Rust client](./rust.md#package-and-source) |
| Worker execution | **Intentionally different:** low-level worker-protocol commands support diagnostics and conformance; `dw` is not an application worker runtime. [Worker commands](./cli-reference.md#worker-protocol-commands) | **Supported:** remote workflow, activity, query, and update handlers through `durable-workflow/sdk`. [PHP worker](./php.md#run-a-remote-php-worker) | **Supported:** deterministic workflow and activity workers. [Python worker](./python.md#worker) | **Supported:** native workflow, activity, query, and update handlers. [Rust worker API](https://rust.durable-workflow.com/) |

**Supported** means the named current release surface exposes the capability.
**Different** identifies an intentional scope boundary. **Not supported** is
an explicit current gap, not a hidden promise. Runtime protocol discovery
remains authoritative when a capability depends on a negotiated worker
protocol.

## Evidence by product surface

### CLI

The [CLI overview](./cli.mdx) defines installation, profiles, structured
output, and exit behavior. The
[command reference](./cli-reference.md) is the complete operator surface,
including lifecycle, messages, schedules, visibility, and the low-level
worker-protocol commands that are intentionally not an SDK worker loop.

### PHP SDK

The pinned PHP SDK exposes framework-neutral client and remote-worker APIs.
Its current public surface includes workflow lifecycle and result handles,
signals, queries, updates, schedules, workflow filtering/history, namespace,
worker, and task-queue visibility, plus registered workflow, activity, query,
and update handlers. See the [PHP SDK guide](./php.md) and generated
[PHP API reference](https://php.durable-workflow.com/api/).

### Python SDK

The Python SDK combines an async control-plane client with deterministic
workflow and activity workers. Its guide publishes
[client operations](./python.md#client-api-reference),
[message handlers](./python.md#signals-queries-and-updates),
[worker execution](./python.md#worker), and
[schedule management](./python.md#schedules). The generated
[Python API reference](https://python.durable-workflow.com/) carries exact
signatures and result types.

### Rust SDK

The published Rust SDK exposes control-plane and selected-run lifecycle,
signals, replayed queries, updates, and native workflow/activity workers.
Update support includes `Client::update_workflow`,
`WorkflowHandle::update`, `Worker::register_update`, and their Avro-value
variants. Schedule management and fleet-wide list/search remain explicit gaps.
See the [Rust SDK guide](./rust.md) and generated
[Rust API reference](https://rust.durable-workflow.com/).

## Shared contract evidence

All four products target the same versioned HTTP+JSON control plane and public
payload envelope. Evidence is split by what it proves:

- the [Capability Index](/docs/capabilities/) records exact artifact floors
  and current breadth;
- the [Platform Conformance Suite](/docs/platform-conformance/) records
  cross-client and cross-worker runtime scenarios;
- CLI and Python repositories retain shared request fixtures for operations
  whose semantic request bodies are byte-for-byte compared today;
- PHP and Rust public API references and release tests establish the additional
  supported methods listed above.

The existence of a shared fixture in two repositories does not imply that
other SDKs lack the operation. Conversely, a common endpoint does not imply
that every product exposes the same operator or worker role.

## Adding or extending a client surface

When adding a new CLI or SDK operation:

1. Keep paths, methods, semantic fields, payload envelopes, and error outcomes
   language-neutral.
2. Add request and runtime evidence for every participating client or worker.
3. Document deliberately different syntax or role boundaries.
4. Mark unsupported products explicitly.
5. Treat language-specific serialization, file paths, class names, or error
   shapes as bugs in the public contract.

<!-- Source: docs/polyglot/database-support.md -->

# Database compatibility and qualification

Use this table for embedded Laravel and self-hosted Server database choices.
Cloud owns its runtime persistence. Cloud customers configure SDK clients and
workers rather than a runtime database.

A declared compatibility baseline is not proof that every later version is
qualified. Source tests and installed, published artifacts are different
evidence. This snapshot binds Workflow 2.20.3 source and Server 3.3.2 release
receipts. It does not change runtime compatibility or establish a lowest/newest
tested range.

| Engine | Declared compatibility | Observed source qualification | Published-artifact qualification |
| --- | --- | --- | --- |
| MySQL | Embedded: 8.0 baseline. Server: 8.0 baseline. | Core: 26.10.0 in the original floating-image feature jobs. Server profiles: 8.0.46 and 8.4.5. | Core observation is release-source-bound. Server 3.3.2's release smoke and exact-index Compose smoke use digest-pinned MySQL 8.0 for start/health checks, not a full execution/capacity/TLS qualification. |
| PostgreSQL | Embedded: 13 baseline. Server: 13 baseline. | Core: 18.6. Server profiles: 16.15 and 17.11. | Core observation is release-source-bound. Server 3.3.2's release smoke cold-starts each architecture's image on digest-pinned PostgreSQL 17 for readiness and cluster info only, not a full execution/capacity/TLS qualification. |
| MariaDB | Embedded: 10.5 baseline. Server accepts `mariadb`; no version baseline is established here. | Core: 10.5.29. Server profile: 10.11.19. | Core observation is release-source-bound. No Server 3.3.2-index-bound MariaDB endpoint receipt was found in the inspected release evidence. |
| SQLite | Embedded capability/tests include SQLite. Server declares a 3.35 baseline. | The original Core source jobs did not emit the SQLite library version. | Installed Core 2.20.3 acceptance exercised SQLite, without establishing a library-version range. Server 3.3.2's release smoke cold-starts on SQLite, also without a library-version range. |
| SQL Server | Published embedded Core advertises `sqlsrv`, but it remains unqualified. Server qualification is not established here. | No SQL Server job was recovered from the published Core matrix. The proposed backend guard is separate source work. | No corresponding release qualification was recovered. Do not infer support from the embedded capability advertisement. |

## Reading the evidence {#reading-the-evidence}

The [Core inventory](https://github.com/durable-workflow/durable-workflow.github.io/issues/347#issuecomment-6642279325)
binds original image digests and four MySQL, four PostgreSQL and one MariaDB
feature jobs at the released source. Its twelve published upgrade cells cover
Laravel/PHP combinations, not twelve database endpoints. The separate
[normally installed Core receipt](https://github.com/durable-workflow/workflow/pull/1025#issuecomment-6632522927)
covers 67 SQLite-backed stream/replay/fake cases. Neither a PHP version nor a
floating image selector supplies an unreported database version.

The [Server inventory](https://github.com/durable-workflow/server/issues/563#issuecomment-6642659277)
separates native and legacy PHP source-job scopes. For Server 3.3.2, the
[release run](https://github.com/durable-workflow/server/actions/runs/42591612884)
cold-starts the amd64 and arm64 images on SQLite, PostgreSQL 17 and MySQL 8.0,
checks `/api/ready` and authenticated cluster info, and stops each with SIGTERM.
The [exact-index Compose smoke](https://github.com/durable-workflow/server/actions/runs/42602031026)
has four architecture/profile cells against index `sha256:4aeceb42…`. Both
`local` and `production` use MySQL, with `mysql:8.0` pinned by digest. The
[3.3.2 delivery record](https://github.com/durable-workflow/server/issues/325#issuecomment-6739628227)
accepts both. These runs start empty databases. They do not upgrade
an existing one, qualify write capacity or verify TLS, and their logs do not
report an exact engine patch version. The earlier
[stable 3.0.5 receipt](https://github.com/durable-workflow/server/issues/563#issuecomment-6643169603)
had the same Compose shape with a floating `mysql:8.0`. Source profiles do not
qualify a different published image.

## Vendor lifecycle and distributed engines {#vendor-lifecycle-and-distributed-engines}

For new deployments, prefer maintained vendor lines after the exact runtime and
database pair is qualified. [PostgreSQL's policy](https://www.postgresql.org/support/versioning/)
marks 13 unsupported and ends 14 on November 12, 2026.
[Oracle's notice](https://www.mysql.com/support/eol-notice.html) places MySQL 8.0
in Sustaining Support from April 21, 2026. Vendor lifecycle does not establish or
remove our artifact qualification.

TiDB, Vitess, PlanetScale and other distributed MySQL-compatible deployments are
untested here. Protocol compatibility does not prove transaction isolation,
row-lock behavior, commit/after-commit visibility or retry behavior. Validate
those semantics with the exact runtime before adoption. This table does not
extend the [single-region operating envelope](../operator-operating-envelope.md)
to multi-region operation.

<!-- Source: docs/polyglot/polling-patterns.md -->

# Polling external state {#polling-external-state}

Poll an external job in an activity, never by making network requests in workflow code. Choose where the wait belongs before choosing its interval.

## Choose the waiting boundary {#choose-the-waiting-boundary}

| Pattern | Worker use | Durable history and control |
| --- | --- | --- |
| Retry one activity until ready | An activity slot during each attempt; released during retry backoff | Each attempt and retry adds history. Workflow code does not resume between attempts, so it cannot inspect a history budget or continue as new then. |
| One long activity with heartbeats | Holds its activity slot throughout the wait | Each accepted heartbeat adds a history event. Use finite activity deadlines and a bounded downstream request timeout; heartbeats do not extend the whole execution deadline. |
| Activity check followed by a durable timer | Releases activity slots between checks | Each check adds activity events and each interval adds timer events. Workflow code can decide whether to stop, compensate or continue as new between checks. |
| A dedicated polling child workflow | The child uses the slots required by its chosen pattern | Moves polling history into the child; adds child start/resolution events to the parent. It does not make the child history or execution free. Choose the parent-close policy explicitly. |
| Submit and return, then receive a callback signal | Releases the activity slot after submission; the workflow waits durably | Submission, signal admission/application and any deadline-backed wait add history. Requires authenticated correlation and application duplicate handling. |

Use short, retry-safe checks. A callback blocked in a downstream request cannot become responsive merely because its workflow has a timer or cancellation policy.

## Callback and signal completion {#callback-and-signal-completion}

When the external system can notify you, submit its job in a short activity and return the acknowledgement. Let the **workflow** wait for a declared signal or replayed condition with a finite deadline. Do not keep an activity, HTTP response or SSE connection open until the job finishes.

Choose the workflow instance id and store the correlation between the authenticated external operation, namespace, **public workflow instance id** and declared signal name before starting the workflow. Its activity may submit the job as soon as start commits. A run id identifies only one run and is not a stable callback address across continue-as-new. Authenticate the callback and check that stored mapping before forwarding it; do not trust routing or identity fields supplied by an arbitrary request.

### Embedded PHP callback {#embedded-php-callback}

Configure `services.jobs.url` for your provider and `services.jobs.callback_url` for your authenticated application adapter. Adapt the illustrated `/jobs` request to the provider's API: it must acknowledge submission quickly and return `job_id`. Implement provider-specific authentication and a bounded request. The operation key is useful only if that provider actually honors it for deduplication.

```php
use Illuminate\Support\Facades\Http;
use Workflow\V2\Activity;
use Workflow\V2\Attributes\Signal;
use Workflow\V2\Attributes\Type;
use Workflow\V2\Support\ActivityOptions;
use Workflow\V2\Workflow;
use function Workflow\V2\activity;
use function Workflow\V2\await;

#[Type('external.submit')]
final class SubmitExternalJob extends Activity
{
    public function handle(string $workflowId, string $operationId): array
    {
        return Http::connectTimeout(2)->timeout(5)
            ->withHeaders(['Idempotency-Key' => $operationId])
            ->post(config('services.jobs.url').'/jobs', [
                'workflow_id' => $workflowId,
                'operation_id' => $operationId,
                'signal' => 'job-completed',
                'callback_url' => config('services.jobs.callback_url'),
            ])->throw()->json();
    }
}

#[Type('external-completion')]
#[Signal('job-completed', [
    ['name' => 'operationId', 'type' => 'string'],
    ['name' => 'result', 'type' => 'array'],
])]
final class ExternalCompletion extends Workflow
{
    public function handle(string $operationId): array
    {
        $job = activity(SubmitExternalJob::class, new ActivityOptions(
            maxAttempts: 1, startToCloseTimeout: 10, scheduleToCloseTimeout: 15,
        ), $this->workflowId(), $operationId);

        $received = await('job-completed', timeout: 10);
        if ($received === null) {
            return ['status' => 'timed_out', 'job_id' => $job['job_id']];
        }
        if ($received[0] !== $operationId) {
            return ['status' => 'unexpected_operation'];
        }
        return ['status' => 'completed', 'result' => $received[1]];
    }
}
```

Start with a caller-selected instance id and finite run/execution timeouts. Register `ExternalCompletion` with [authenticated webhooks](../features/webhooks.md); the application's callback adapter can forward validated input to its instance-targeted signal route:

```bash
curl -fsS -X POST "$APP_URL/webhooks/instances/external-job-42/signals/job-completed" \
  -H "Authorization: $WEBHOOK_TOKEN" -H 'Content-Type: application/json' \
  -d '{"arguments":["op-42",{"value":42}]}'
```

This is the embedded webhook's `arguments` format, not the Server's payload envelope. The provider should call your authenticated application adapter, not receive your Server operator or worker credentials.

### Python callback and condition {#python-callback-and-condition}

The activity below performs only the bounded submission. Configure `JOB_API_URL`, `JOB_CALLBACK_URL` and provider authentication in the activity environment. Register both definitions in the [Python worker](./python.md). Pass the caller's known workflow instance id explicitly; this example does not invent a workflow-context id accessor.

```python
import os
import httpx
from durable_workflow import activity, workflow
from durable_workflow.workflow import ActivityRetryPolicy

@activity.defn(name="external.submit")
async def submit_external_job(workflow_id, operation_id):
    async with httpx.AsyncClient(timeout=5.0) as provider:
        response = await provider.post(
            os.environ["JOB_API_URL"] + "/jobs",
            headers={"Idempotency-Key": operation_id},
            json={"workflow_id": workflow_id, "operation_id": operation_id,
                  "signal": "job-completed",
                  "callback_url": os.environ["JOB_CALLBACK_URL"]},
        )
        response.raise_for_status()
        return response.json()

@workflow.defn(name="external-completion")
class ExternalCompletion:
    def __init__(self):
        self.results = {}

    @workflow.signal("job-completed")
    def completed(self, operation_id: str, result: dict):
        self.results.setdefault(operation_id, result)

    def run(self, ctx, workflow_id, operation_id):
        job = yield ctx.schedule_activity(
            "external.submit", [workflow_id, operation_id],
            retry_policy=ActivityRetryPolicy(max_attempts=1, backoff_seconds=[1]),
            start_to_close_timeout=10, schedule_to_close_timeout=15,
        )
        ready = yield ctx.wait_condition(
            lambda: operation_id in self.results,
            key="external.completed", timeout=10,
        )
        if not ready:
            return {"status": "timed_out", "job_id": job["job_id"]}
        return {"status": "completed", "result": self.results[operation_id]}
```

Choose the instance id and persist the operation's namespace/instance/signal mapping **before calling start**. The workflow's activity can begin as soon as the start commits. The application then starts that instance with finite budgets:

```python
handle = await client.start_workflow(
    workflow_type="external-completion", task_queue="external-jobs",
    workflow_id="external-job-42", input=["external-job-42", "op-42"],
    execution_timeout_seconds=60, run_timeout_seconds=30,
)
```

After authenticating the provider and resolving its stored operation mapping, the callback adapter uses a Client in that **same namespace**:

```python
await client.signal_workflow(
    "external-job-42", "job-completed", args=["op-42", {"value": 42}],
)
```

This calls the Server's `POST /api/workflows/{workflowId}/signal/{signalName}` and uses the SDK's normal Avro envelope. A successful signal call confirms admission, not that workflow code processed the callback; inspect the workflow result or retained signal status when acknowledgement matters.

### Early, continued and late callbacks {#early-continued-and-late-callbacks}

- Declare the signal before starting the run. A callback arriving before its wait is retained and consumed in durable order; Python reconstructs handler state during replay. A later registration cannot rewrite an old run's declarations.
- Instance-targeted callbacks follow continue-as-new; explicitly run-targeted calls remain fenced to that run. On compatible Core 2.7.0+ runtimes, eligible received instance signals retain original identity and order across handoff. State from **already applied** callbacks must be carried in continuation input when the application still needs it.
- A callback after terminal closure is explicitly refused. On the published Server, Python exposes `SignalFailed.reason == "run_not_active"` with status409. An accepted but unapplied signal can instead retain a rejected closure disposition; the old acceptance receipt is not erased. See [signal acceptance and closure](../features/signals.md#acceptance-and-closure).
- A wait deadline does not cancel the external job. Reconcile its outcome or request provider cancellation through an activity. A late result may need an application recovery decision; do not silently start replacement work.

### Duplicate delivery and streaming limits {#duplicate-delivery-and-streaming-limits}

Providers retry callbacks, and a network error can occur after signal admission. Persist your application operation/event key and exact intent, deduplicate at the callback boundary and in business processing, and retain unresolved deliveries for reconciliation. The Python handler keeps the first result per operation in this run; it is not a cross-run or once-only admission guarantee. Do not mark an uncertain forward as processed merely because the request was sent. [Shared #216](https://github.com/durable-workflow/.github/issues/216) still owns safe signal retries and retained admission identity. A provider idempotency header or one activity attempt does not by itself guarantee once-only external effects.

Start the workflow before this recipe submits work. Service `start(use-existing)` followed by `signal` is **two operations**, with a closure race; atomic service intake remains [shared #222](https://github.com/durable-workflow/.github/issues/222). Embedded `signalWithStart` is a separate atomic API. The debounce request [Website #280](https://github.com/durable-workflow/durable-workflow.github.io/issues/280) was superseded by that product requirement, not completed by this callback recipe.

An HTTP/SSE socket held inside an activity is not durable: worker/process loss ends it, it occupies an activity slot, and reconnecting can replay or miss external events. Submission and callbacks still add activity, signal and wait history; this is not a free or unlimited execution. For durable **workflow output**, use [Workflow Streams](./workflow-streams.md) and their persisted sequence/cursor contract rather than treating a transport connection as storage.

The PHP and Python code above was exercised on published **Server 2.9.7 / Core 2.7.2**, with normally installed **Python SDK 2.6.1**. Separate PostgreSQL 17.11 controls covered early and waiting callbacks, late refusal, and continue-as-new with a callback while the Python worker was absent followed by a cold reader. Embedded controls used the real webhook route. This is recipe evidence, not all-database, provider-authentication, graceful-shutdown or native-runtime qualification. [The qualification record](https://github.com/durable-workflow/durable-workflow.github.io/issues/321#issuecomment-6292485315) preserves the unsuccessful originals separately.

## Bound retries and history {#bound-retries-and-history}

Retries are opt-in: PHP activities default to one attempt. For example, configure `maxAttempts: 3`, `backoff: [5, 30]`, `startToCloseTimeout: 10` and `scheduleToCloseTimeout: 90` for three remote attempts with finite delays and original total time. Python's matching retry policy is `ActivityRetryPolicy(max_attempts=3, backoff_seconds=[5, 30])`, with separate attempt/total timeout options. A successful “not ready” return is not a failure and does not trigger a retry. See [failures and recovery](../failures-and-recovery.md).

Retries are suitable when not-ready is an explicit retryable outcome and workflow code need not run between checks. Do not use unlimited retries as an invisible polling loop. Local-activity retry-delay parity is tracked in [the retry contract issue](https://github.com/durable-workflow/.github/issues/219); the examples here use remote activities.

## Roll over deliberately {#roll-over-deliberately}

This runnable demonstration checks once per run, waits, and carries a finite remaining-check count into the next run. Replace only the activity's synthetic status lookup with a real bounded request. The example returns the final status even when the check limit is reached; callers must inspect `ready`.

### Embedded PHP {#embedded-php}

```php
use Workflow\V2\Activity;
use Workflow\V2\Support\ActivityOptions;
use Workflow\V2\Workflow;
use function Workflow\V2\activity;
use function Workflow\V2\timer;
use function Workflow\V2\continueAsNew;

final class CheckDemoStatus extends Activity
{
    public function handle(int $remaining): array
    {
        return ['ready' => $remaining <= 1];
    }
}

final class PollDemo extends Workflow
{
    public function handle(int $remaining = 2, int $intervalSeconds = 30): array
    {
        $status = activity(CheckDemoStatus::class, new ActivityOptions(
            maxAttempts: 1, startToCloseTimeout: 10, scheduleToCloseTimeout: 15,
        ), $remaining);
        if ($status['ready'] || $remaining <= 1) {
            return $status;
        }
        timer($intervalSeconds);
        return continueAsNew($remaining - 1, $intervalSeconds);
    }
}
```

Start it through the normal [embedded workflow setup](../installation.md), with finite run and execution timeouts.

### Python service worker {#python-service-worker}

```python
from durable_workflow import activity, workflow
from durable_workflow.workflow import ActivityRetryPolicy

@activity.defn(name="poll-demo.status")
async def check_status(remaining):
    return {"ready": remaining <= 1}

@workflow.defn(name="poll-demo")
class PollDemo:
    def run(self, ctx, remaining=2, interval_seconds=30):
        result = yield ctx.schedule_activity(
            "poll-demo.status", [remaining],
            retry_policy=ActivityRetryPolicy(max_attempts=1, backoff_seconds=[1]),
            start_to_close_timeout=10, schedule_to_close_timeout=15,
        )
        if result["ready"] or remaining <= 1:
            return result
        yield ctx.sleep(interval_seconds)
        return ctx.continue_as_new(remaining - 1, interval_seconds)
```

Register `PollDemo` and `check_status` in the same [Python worker](./python.md), then start with arguments `[2, 30]` and finite execution/run timeouts. Observe the current workflow instance to follow its successor; a run-specific handle still selects its original run.

The two examples were exercised on published Server **2.9.7 / Core 2.7.2**, with normally installed Python **2.6.1** and test input `[2, 1]`. Both completed two real checks across two runs. Continue-as-new resets run history, adds run/link overhead and preserves the original execution deadline. It does not provide an unlimited lifetime.

Use relative durable waits here. Python's advancing clock and replay-stable history-budget reads remain separate [clock](https://github.com/durable-workflow/.github/issues/215) and [budget](https://github.com/durable-workflow/workflow/issues/859) work. This recipe does not branch on `ctx.now()` or recomputed history length. Do not treat `DW_MAX_HISTORY_EVENTS` as an enforced automatic rollover: [Server #504](https://github.com/durable-workflow/server/issues/504) tracks that gap.

## Interrupt and reconcile {#interrupt-and-reconcile}

For a signal-driven stop, [select a signal against the pending wait](../features/concurrency.md) and explicitly cancel the losing operation according to its supported cancellation policy. Selection alone does not cancel losers. A request to stop an activity does not prove that an external side effect never happened: use idempotency keys and reconcile before compensating or restarting work. Immediate cancellation, termination and hard workflow timeout do not execute workflow compensation; use the supported [cooperative cancellation](./cancellation.md) path when cleanup must run.

## Inspect the actual history {#inspect-the-actual-history}

The published Python service-mode measurements above recorded:

| Completed case | Canonical history events |
| --- | --- |
| One remote activity succeeding on attempt three | 10 total: 1 scheduled, 3 started, 2 retry-scheduled, 1 completed, plus 3 run lifecycle events |
| One activity with two accepted heartbeats | 8 total: 3 activity events, 2 heartbeat events, plus 3 lifecycle events |
| Two successful checks separated by one timer | 11 total: 6 activity events, 2 timer events, plus 3 lifecycle events |

The rollover example recorded **8 + 5** events across its Python runs and **8 + 6** across PHP runs; PHP also records `StartAccepted` for the successor. These are measured operation/run costs, not universal event totals, billing units or performance results. Other commands, failures, cancellation and runtime admission metadata can add events. Inspect each run's history, retained attempt policy and original deadlines rather than estimating growth from loop iterations alone. Exact observations and unsuccessful fixture records are retained in [the qualification receipt](https://github.com/durable-workflow/durable-workflow.github.io/issues/283#issuecomment-6288504564).

<!-- Source: docs/polyglot/prerelease-history-migration.md -->

# Migrate retained prerelease history

Use this offline procedure when an Avro-only Durable Workflow 2.0 deployment
reports `unsupported_payload_codec` for state retained from an earlier public
2.0 prerelease. It converts the known JSON-tagged, untagged, and obsolete Avro
Value representations without making any legacy codec available to a running
v2 application.

Do not use history export as a substitute for this procedure. An export is a
portable copy of a run; it does not change the database rows that deployment
startup inspects. Never delete history to make the preflight pass.

## Decide what must be converted

Runs that are still active can finish on the currently deployed prerelease.
Drain them before the maintenance window when that is operationally safe.
Terminal runs, closed executions retained for replay, and state that cannot
drain must be converted. The migration inventories both groups, and does not
delete either one.

The dry run covers the protocol-owned surfaces checked by Server startup:

- workflow inputs and outputs, activity payloads, commands, signals, updates,
  service calls, update-validation tasks, and durable-stream references;
- inline single-object frames and the explicit payload envelopes nested in
  history events; and
- externally stored payload references, including reference shape, codec,
  object availability, byte length, and SHA-256 integrity.

Customer-owned memo, search-attribute, context, and diagnostic maps are not
codec declarations and are left unchanged. Unknown codecs, unknown schema
fingerprints, corrupt references, and values that the fixed Avro Value schema
cannot represent are reported as unsafe. An unsafe finding blocks apply.

## Ownership boundary

The command belongs to the Workflow PHP package, but it must run in the
process that owns the database connection and external-payload configuration:

- **Embedded Laravel:** install the target Workflow package in the application
  checkout, then run the application's `php artisan` command while its web,
  queue, scheduler, and maintenance processes are stopped.
- **Standalone Server:** use a one-shot shell or job from the target Server
  image, with the same database, namespace configuration, credentials, and
  external-payload mounts as the deployment. Keep the normal bootstrap hook
  and all Server processes stopped until conversion succeeds.

Do not point an unrelated application checkout at the database. Namespace
storage policy is part of the integrity check; a process that cannot read an
external object reports it as unsafe instead of guessing.

## Deployment order

1. Keep the old prerelease available long enough to drain any active runs you
   choose to finish. Then stop every database writer: API nodes, workers,
   schedulers, bootstrap jobs, and maintenance jobs.
2. Take the database and external-object-store snapshots required by your
   normal recovery policy. Retain the old application or Server artifact.
3. Install or pull the target artifact that contains
   `workflow:v2:migrate-prerelease-history`, but do not start it. Run the dry
   inventory from that artifact:

   ```bash
   php artisan workflow:v2:migrate-prerelease-history --dry-run --json
   ```

   A zero exit code means every affected value is safely convertible. Review
   the paths and affected run IDs. A nonzero exit code with `unsafe_fields`
   greater than zero blocks the upgrade; correct storage access or remain on
   the old prerelease.
4. Choose a new private backup path and a new private evidence directory on
   persistent storage. Neither may already exist. Apply once:

   ```bash
   php artisan workflow:v2:migrate-prerelease-history \
     --backup=/secure/dw-prerelease-history-backup.json \
     --evidence-dir=/secure/dw-prerelease-history-evidence
   ```

   The command writes the backup before beginning updates and refuses to
   overwrite evidence. It locks and updates database rows in one transaction,
   preserves workflow and run identities, verifies external-object hashes and
   sizes, and retains every original external object. Replacement external
   objects are content-addressed Avro copies. No history is deleted.
5. Wait for the command to finish. It exports and strict-replays every affected
   retained run, rebuilds its summary, wait, timeline, timer, and lineage
   projections, checks those projections for drift, and repeats the payload
   inventory. Success requires all three checks. Keep the generated replay and
   projection reports with the backup.
6. Run the normal target bootstrap only after migration succeeds:

   ```bash
   php artisan server:bootstrap --force
   ```

   For embedded Laravel, run the application's normal migration/bootstrap
   procedure instead. The Avro-only payload preflight must be clean before any
   new worker, API node, scheduler, or maintenance process starts.
7. Start the target deployment and verify readiness and a retained-run replay
   or query through the normal operator surface. Retain the migration packet
   for the duration of the rollback window.

## Rollback

If apply, replay, projection verification, or postflight fails, the upgrade is
blocked. Leave all writers stopped and run the rollback command printed by the
migration. With the paths above it is:

```bash
php artisan workflow:v2:migrate-prerelease-history \
  --rollback-from=/secure/dw-prerelease-history-backup.json \
  --evidence-dir=/secure/dw-prerelease-history-evidence
```

Rollback verifies the backup digest and migration state, refuses to overwrite
rollback evidence, and refuses to restore a row changed after conversion. It
restores the original database values atomically and retains replacement
external objects as evidence. Then restore the previous application or Server
artifact. If rollback refuses because another writer changed state, keep the
upgrade blocked and restore the coordinated database and object-store snapshot
from step 2.

After rollback, the Avro-only deployment is expected to fail preflight again.
Resolve the reported unsafe state, create new backup and evidence paths, and
repeat the complete procedure before retrying the upgrade.

<!-- Source: docs/polyglot/runtime-migration.md -->

# Moving between runtimes

Move new starts to the destination while existing runs finish on their source. This procedure covers independent service-mode runtimes. A history export preserves evidence; it does not transfer a live Server-sourced run. The supported [embedded-v2 import](./embedded-to-server.md#phase-e-import-eligible-embedded-v2-runs) accepts eligible embedded sources only.

## Choose the destination {#paths}

| Move | Destination preparation |
| --- | --- |
| Self-hosted Server to another self-hosted Server | Deploy the destination with its own authoritative database, payload storage and credentials. Follow the [deployment guide](../deployment.md). |
| Self-hosted Server to Cloud | Provision the destination namespace and verify its returned runtime URL, credentials and capabilities. Follow [Cloud setup](./cloud-control-plane.md). |
| Cloud to self-hosted Server | Prepare the self-hosted destination and its SDK workers before changing application traffic. Keep the Cloud source provisioned while its runs drain. |

## Prepare both runtimes {#prepare}

Record each source endpoint, namespace, workflow instance and run ID, task queue, workflow/activity type and payload-store dependency. Preserve a tested recovery path for self-hosted storage. A Cloud move does not grant access to a database backup or copy Cloud credentials to the destination. Verify the actual destination release and capabilities, role credentials, worker registrations and advertised types.

## Switch new starts in small groups {#cutover}

1. Run the destination and its workers beside the source. Complete a uniquely identified canary with representative activities and results. Keep its side effects isolated from real business work.
2. Inventory every start producer: application calls, schedules, webhook/bridge adapters and service calls. Choose a cutover point for one workflow family. Coordinate schedule pauses and destination registration so both runtimes do not start the same occurrence.
3. Route that family’s new starts to the destination. Keep an application-owned record of which endpoint and namespace accepted each workflow. Reusing a business key on another runtime does not provide cross-runtime deduplication.
4. Keep signals, approvals, updates, queries, cancellation and result/history reads pointed at the runtime owning the run. Keep source workers, queues, timers, credentials and payload access available. Changing a client URL does not move accepted work or its activity leases.
5. Check outcomes and failures before moving the next family. If a start has an uncertain transport outcome, resolve it at the original runtime before submitting it elsewhere. A long-lived run that never reaches terminal state has not drained.

## Preserve evidence before retirement {#preserve-evidence}

Export the history you must retain, including every required continuation run, original payload bytes and codecs. Check completeness and integrity with a compatible consumer, and retain the exact producer identity. [History export and replay guidance](../testing.md#service-sdk-replay) explains the separate checks. Audit exports do not make Server-sourced runs eligible for live import.

Use the producer actually serving the export. Core 2.11.4 alone does not qualify Server, bundled CLI, SDK HTTP or Cloud exports. Follow the separate [Server HTTP producer/image qualification](https://github.com/durable-workflow/server/issues/692). Old lossy bundles remain invalid. Do not change numbers or checksums to force acceptance, and do not destroy the authoritative source while required evidence is unavailable or unverifiable.

## Retire the source last {#retire-source}

Confirm that no source-owned nonterminal runs remain across every relevant namespace and all list pages. Include waiting parents/children and runs created by remaining start producers. A healthy endpoint, an empty first page or unavailable metrics does not prove zero open work. Confirm that no callers, callbacks or workers still depend on the source and that required archives are readable.

Only then retire self-hosted resources or deprovision the Cloud runtime. Cloud capacity charges continue while it remains provisioned, including while idle or draining. Deprovisioning removes active runtime data and credentials and is not pause/resume. Use the existing plan’s returned billing terms; this procedure creates no automatic refund or billing exception.

## Roll back routing without moving runs {#rollback}

Route future starts back only to a healthy, compatible source. Runs already accepted by the destination remain destination-owned and need its workers and command routing until terminal. Preserve both ownership records and prevent duplicate starts. Do not restore old source storage over newer accepted state or resume a second live copy of a run.

<!-- Source: docs/polyglot/server-role-topology.md -->

# Server Role Topology

## Why This Manifest Exists

`GET /api/cluster/info` publishes the server's `topology` manifest under the
schema `durable-workflow.v2.role-topology`. Treat that manifest as the public
contract for role names, supported deployment shapes, durable-write authority,
failure-domain expectations, scaling boundaries, and the ordered migration path
from today's standalone distribution toward a split control/execution topology.

Use this page when you need to reason about server shape from scripts,
dashboards, runbooks, or rollout automation. Use the
[Server API Reference](/docs/polyglot/server-api-reference) for the raw
HTTP surface and the [Server Guide](/docs/polyglot/server) for deployment
setup.

## Reading The Topology Manifest

The `topology` object answers these contract questions:

| Field family | Question it answers |
| --- | --- |
| `schema`, `version` | Which topology contract revision are you reading? |
| `supported_shapes` | Which product deployment shapes are legal? |
| `role_vocabulary` | Which role names are valid on this contract? |
| `current_shape`, `current_process_class`, `current_roles`, `execution_mode` | What is the responding node doing right now? |
| `matching_role.*` | Who owns broad ready-task wake, which routing axes are frozen, and which dispatch/backpressure posture is active? |
| `role_catalog`, `authority_surfaces` | Which interfaces and durable mutation surfaces belong to each role? |
| `shape_assignments` | Which process classes are allowed for each supported shape? |
| `authority_boundaries`, `failure_domains`, `scaling_boundaries` | Which role is allowed to write what, how each role fails, and what load axis each role scales on? |
| `supported_topologies`, `migration_path` | What deployment families are product-supported, and what is the ordered path from the standalone shape toward more isolated roles? |
| `kernel_invariants` | Which durable-kernel guarantees the role split must preserve regardless of which shape is currently running? |

`current_shape`, `current_process_class`, and `current_roles` describe the node
that answered the HTTP request, not the full fleet. Use
`current_process_class` as the node's declared identity, then compare
`current_roles` against the process-class bundles in `shape_assignments` when
you need to verify that declaration.

```bash
curl -sS "$DURABLE_WORKFLOW_SERVER_URL/api/cluster/info" \
  -H "Authorization: Bearer $DURABLE_WORKFLOW_AUTH_TOKEN" \
  -H "X-Namespace: default" \
  | jq '{
    current_shape: .topology.current_shape,
    current_process_class: .topology.current_process_class,
    current_roles: .topology.current_roles,
    execution_mode: .topology.execution_mode,
    matching_role: .topology.matching_role,
    scaling_boundaries: .topology.scaling_boundaries,
    migration_path: .topology.migration_path
  }'
```

## Role Vocabulary

The public role names are fixed by `topology.role_vocabulary`:

| Role | Responsibility |
| --- | --- |
| `api_ingress` | Accept external HTTP traffic, including discovery and control-plane entrypoints. |
| `control_plane` | Start, signal, update, repair, cancel, terminate, archive, and otherwise mutate workflow lifecycle state. |
| `matching` | Discover ready work, own task leases, and coordinate dispatch pressure. |
| `history_projection` | Persist durable history and maintain derived run summaries and exports. |
| `scheduler` | Fire schedules and persist schedule-run state. |
| `execution_plane` | Run workflow and activity task work. |

Automation should treat these exact identifiers as the stable vocabulary.

## Supported Deployment Shapes

`topology.supported_shapes` names the legal product deployment shapes:

| Shape | Process classes published in `shape_assignments` | Contract meaning |
| --- | --- | --- |
| `embedded` | `application_process` | One application process owns control-plane, matching, history projection, scheduler, and execution. |
| `standalone_server` | `server_http_node`, `scheduler_node`, `worker_node` | The current standalone server distribution: HTTP ingress/control-plane on the server node, scheduler isolated as its own process class, and execution on worker nodes. |
| `split_control_execution` | `ingress_node`, `control_plane_node`, `scheduler_node`, `matching_node`, `execution_node` | The same product contract split into narrower role-specific process classes so scaling and failure boundaries can move by subsystem. |

`split_control_execution` is a supported topology, not a second engine or a
different API. The same discovery surface describes both the standalone and the
split-role shapes.

## Current Node Identity

Use `current_shape`, `current_process_class`, `current_roles`, and
`execution_mode` together:

- `current_shape` identifies the responding node's shape contract.
- `current_process_class` identifies the declared process class for that node.
- `current_roles` identifies the active role bundle on that node.
- `execution_mode` distinguishes `remote_worker_protocol` from
  `local_queue_worker`.

For the standalone server distribution, `current_shape` remains
`standalone_server` even when `DW_MODE=embedded` switches execution to local
queue workers. In that case, `execution_mode` changes to `local_queue_worker`
while the HTTP node keeps the standalone-server role contract.

### Hosted Route Gating

The same topology contract also tells callers when a node should reject hosted
traffic outright. Authenticated hosted routes fail closed unless the responding
node advertises the HTTP control bundle those routes require today:
`api_ingress` plus `control_plane`.

Wrong-node responses return `503` with
`reason: "topology_role_unavailable"` and the topology evidence needed to
reroute:

- `current_shape`
- `current_process_class`
- `current_roles`
- `required_roles`
- `missing_roles`

That gate runs before namespace resolution on hosted routes, so a request sent
to a `scheduler_node`, `matching_node`, or `execution_node` does not learn
whether the named namespace exists before it is redirected to the correct node
class. `GET /api/health`, `GET /api/ready`, and authenticated
`GET /api/cluster/info` remain available so automation can discover the node's
shape before retrying elsewhere.

### Workflow Bootstrap Gate

A second route-level gate fails closed when the responding node has unresolved
workflow v2 bootstrap blockers. While `checks.workflow_v2.status` is `blocked`,
authenticated workflow start/mutation, schedule mutation, bridge-adapter, and
worker-protocol routes return `503` with `reason: "workflow_v2_blocked"` plus:

- `blocked_by`: the ordered list of upstream readiness blockers, for example
  `migrations`.
- `remediation`: the operator-facing instruction for clearing the listed
  blockers, mirrored from `/api/ready` `checks.workflow_v2.remediation`.

The bootstrap gate sits in the same slot as the hosted-route topology gate:
after role and protocol-version validation, before namespace resolution. A
blocked request therefore never observes namespace existence.

The gated route families are workflow routes such as `/api/workflows`, schedule
mutations such as `POST /api/schedules`, `PUT /api/schedules/{scheduleId}`,
`DELETE /api/schedules/{scheduleId}`, `/pause`, `/resume`, `/trigger`, and
`/backfill`, bridge-adapter routes such as
`/api/bridge-adapters/webhook/{adapter}`, and worker-protocol routes under
`/api/worker`.

Schedule **reads** (`GET /api/schedules`, `GET /api/schedules/{scheduleId}`,
`GET /api/schedules/{scheduleId}/history`) are intentionally exempted so
operators can inspect schedule state during recovery. Worker-protocol routes
return the same bootstrap-gate payload in the worker-protocol envelope, so
worker SDKs branch on the same machine-readable `reason: "workflow_v2_blocked"`
the control plane returns.

## Matching Role Contract

`topology.matching_role` freezes the live matching and wake posture for the
responding node:

| Field | Meaning |
| --- | --- |
| `queue_wake_enabled` | Whether short-lived queue wake signals are currently enabled. |
| `shape` | Which matching deployment shape this node advertises: `in_worker` or `dedicated`. |
| `wake_owner` | Which implementation currently owns the broad wake sweep: `worker_loop` or `dedicated_repair_pass`. |
| `task_dispatch_mode` | Whether dispatch is happening through `poll`-driven remote workers or `queue`-driven local execution. |
| `partition_primitives` | The frozen routing axes the matching role reasons about, in order: `connection`, `queue`, `compatibility`, `namespace`. |
| `backpressure_model` | The durable admission boundary the matching role enforces. Current v2 reports `lease_ownership`. |
| `discovery_limits` | The frozen numeric matching-role contract the workflow package compiles in: `poll_batch_cap`, `availability_ceiling_seconds`, `wake_signal_ttl_seconds`, `workflow_task_lease_seconds`, and `activity_task_lease_seconds`. |

This lets operators and automation distinguish "matching exists but wake is
degraded" from "this node is intentionally running a different dispatch mode,"
and it gives the same routing and backpressure vocabulary the server itself
publishes in operator metrics.

`discovery_limits` is the matching-role numeric contract: `poll_batch_cap`
freezes the maximum batch of ready-task rows returned per poll,
`availability_ceiling_seconds` freezes the cross-backend tolerance applied to
`available_at` so freshly-available tasks survive sub-second timestamp drift,
`wake_signal_ttl_seconds` freezes the default `CacheLongPollWakeStore` signal
TTL, and `workflow_task_lease_seconds` / `activity_task_lease_seconds` freeze
the default workflow and activity task lease durations. Operators read these
values to verify the deployment matches the documented matching-role contract
without grepping the package source. Tightening any of these is a
protocol-level change because dispatch, worker, and acceleration timing
elsewhere in the contract depend on them; renaming a field is also a
protocol-level break.

## Authority Boundaries

`topology.authority_boundaries` names the durable write surfaces each role is
supposed to mutate:

| Role | Published writes |
| --- | --- |
| `api_ingress` | `worker_registrations` |
| `control_plane` | `workflow_instances`, `workflow_runs.status`, `workflow_tasks.lifecycle` |
| `matching` | `workflow_tasks.leases`, `activity_tasks.leases` |
| `history_projection` | `history_events`, `workflow_run_summaries`, `workflow_history_exports` |
| `scheduler` | `workflow_schedules.fire_state`, `workflow_starts.scheduled` |
| `execution_plane` | `workflow_tasks.outcomes`, `activity_attempts`, `worker_compatibility_heartbeats` |

Use this contract to catch cross-role drift before you split processes or add
new topology-specific automation.

## Failure And Scaling Boundaries

### Failure Domains

`topology.failure_domains` names the first degraded behavior and the first
operator-visible signal for each role outage:

| Failure domain | `effect` | `operator_signal` |
| --- | --- | --- |
| `control_plane_down` | `workers_continue_claimed_tasks_only_until_lease_expiry` | `operator_commands_fail_fast` |
| `execution_plane_down` | `ready_tasks_accumulate_without_loss` | `operators_see_ready_depth_growth` |
| `matching_down` | `claim_falls_back_to_direct_ready_task_discovery` | `ready_depth_rises_while_claim_rate_falls` |
| `history_projection_down` | `projection_reads_may_stale_while_durable_writes_continue` | `projection_lag_seconds_may_increase` |
| `scheduler_down` | `scheduled_workflows_stop_firing_and_record_missed_runs` | `operators_see_missed_schedule_state` |
| `api_ingress_down` | `external_http_traffic_stops_at_the_edge` | `embedded_in_process_calls_may_continue` |

These are product-facing expectations, not internal implementation trivia. Use
them to describe what should happen when a role is degraded before reading logs.

### Scaling Boundaries

`topology.scaling_boundaries` tells you which load axis each role primarily
scales on in the split-role model:

| Role | Scaling boundary |
| --- | --- |
| `api_ingress` | `incoming_http_request_rate` |
| `control_plane` | `operator_commands_and_run_lifecycle_transitions` |
| `matching` | `ready_task_rate_and_poller_count` |
| `history_projection` | `durable_event_rate` |
| `scheduler` | `active_schedule_count` |
| `execution_plane` | `workflow_and_activity_task_rate` |

This is the explicit answer to "what do we scale independently?" for the
split-role topology.

## Migration Path

`topology.migration_path` is ordered. Each step preserves one durable kernel
while isolating responsibilities more clearly:

1. `audit_role_boundaries`
   Result: tooling flags cross-role writes before runtime shape changes.
2. `expose_role_bindings`
   Result: container seams allow out-of-process adapters without patching the package.
3. `introduce_dedicated_matching_shape`
   Result: matching can run as its own process class without changing the claim contract.
4. `split_history_projection`
   Result: history and projections can move out of process without introducing a second writer.
5. `split_scheduler`
   Result: schedule firing can move behind leader election while single-replica deployments stay legal.
6. `optional_execution_partitioning`
   Result: workers can partition by namespace, connection, queue, and compatibility.

Read this list as the supported topology transition order, not as a separate
product roadmap detached from the current engine.

Each `topology.migration_path[]` entry carries an explicit
`reversible: true` flag. Treat the migration path as bidirectional: a
deployment that has reached `split_history_projection` MAY collapse the
history role back into the control-plane process and remain a legal
topology shape. Rollback is part of the contract, not an unmodelled
edge case.

## Durable Kernel Invariants

`topology.kernel_invariants` enumerates the guarantees the role split
preserves regardless of which supported shape is running. Use this list
when validating that a candidate topology change is product-supported
rather than a fork of the engine:

| Invariant | What it guarantees |
| --- | --- |
| `single_persistence_engine` | One workflow database backs every topology shape; role split does not introduce a second persistence engine. |
| `single_worker_protocol` | One HTTP worker protocol carries claim, complete, fail, and heartbeat traffic across every topology; role split does not fork the worker contract. |
| `single_history_writer` | `history_events` has exactly one durable writer per logical event regardless of where the history/projection role runs. |
| `single_control_authority_per_run` | Every mutation of a given workflow run routes through one control-plane authority; per-run row locks serialise transitions across replicas. |
| `embedded_topology_remains_supported` | The embedded shape where one process fills every role MUST stay legal; existing embedded hosts are never forced to migrate. |
| `role_split_is_topology_only` | Splitting roles is a topology change, not a product fork; collapsing the roles back onto a single process is always a legal topology. |

Each entry's `applies_to` field lists the shapes the invariant covers.
For the supported topology family, every invariant currently applies to
`embedded`, `standalone_server`, and `split_control_execution`. If an
upgrade adds a new shape, the invariants whose `applies_to` does not
include it MUST be reviewed before that shape is treated as
product-supported.

```bash
curl -sS "$DURABLE_WORKFLOW_SERVER_URL/api/cluster/info" \
  -H "Authorization: Bearer $DURABLE_WORKFLOW_AUTH_TOKEN" \
  -H "X-Namespace: default" \
  | jq '.topology.kernel_invariants[] | {id, applies_to}'
```

## Coordination Health

`/api/cluster/info` also publishes `coordination_health` beside `topology`.
Keep the distinction clear:

- `topology` tells you what the node is allowed to do and how the product shape
  is supposed to behave.
- `coordination_health` tells you whether rollout-safety and coordination checks
  are currently healthy across namespaces.
- `coordination_health.blocked_by`, `coordination_health.message`, and
  `coordination_health.remediation` appear when the server cannot evaluate
  rollout-safety health because readiness prerequisites such as migrations or
  database connectivity are missing.
- `coordination_health.routing_drains` summarizes draining build-id cohorts
  across queues and namespaces. `queues_with_drains` tells you whether rollout
  automation is intentionally holding traffic away from any cohort right now.
- `coordination_health.warning_checks`, `coordination_health.error_checks`, and
  `coordination_health.checks` remain the normalized check inventory once
  rollout-safety evaluation is running.
- `coordination_health.checks[]` always includes the frozen `activity_path`
  check next to `worker_compatibility`, `task_transport`, `routing_health`,
  `durable_resume_paths`, and the projection/scheduler checks. `activity_path`
  is the activity-side counterpart of `task_transport`: it surfaces activity
  executions whose schedule-to-start, start-to-close, schedule-to-close, or
  heartbeat deadline has passed without enforcement, plus the sustained activity
  retry backlog. Renaming the check is a protocol-level change.

Use both surfaces together when deciding whether a topology change is both
supported and currently safe.

## Related References

- [Server API Reference](/docs/polyglot/server-api-reference) for the
  authenticated `/api/cluster/info` HTTP contract.
- [Server Guide](/docs/polyglot/server) for deployment setup and the
  broader standalone server operating model.
- [Deployment Modes](/docs/polyglot/deployment-modes) for when to choose
  embedded, standalone server, or broader support-led topologies.

<!-- Source: docs/polyglot/service-operations.md -->

# Cross-namespace service calls {#cross-namespace-service-calls}

A service operation, also called a Nexus operation, is the supported durable
call between namespaces in one Durable Workflow cluster. It is addressed by
the target namespace and an endpoint/service/operation triple. A child workflow
or ordinary signal stays within its namespace. Neither a queue name nor a
different language changes that rule.

## Choose a supported authoring surface {#choose-a-supported-authoring-surface}

| Caller | Published surface | Target namespace |
| --- | --- | --- |
| Embedded PHP | `Workflow::serviceOperation()` or the `serviceOperation()` helper | `ServiceOperationOptions(targetNamespace: 'billing')` |
| PHP SDK | `WorkflowContext::serviceOperation()` | Array option `['namespace' => 'billing']` |
| Python SDK | `yield context.call_nexus_service(...)` | Keyword `target_namespace='billing'` |
| Rust SDK | Authored Nexus waits are not published | Tracked in [Rust SDK #86](https://github.com/durable-workflow/sdk-rust/issues/86). A held draft is not installed support. |

Use the [embedded example](../features/service-operations.md) for a Laravel
workflow, the [PHP SDK service-call example](https://php.durable-workflow.com/build/workflows-activities/)
for its straight-line worker API, or the [Python workflow reference](https://python.durable-workflow.com/reference/workflow/)
for generator-based authoring. Discover the connected runtime's advertised
service-operation contract before deploying a caller. A client management API
alone does not prove workflow-worker support.

The PHP SDK's array option is currently the cross-namespace authoring path.
An equivalent typed option is tracked in [PHP SDK #193](https://github.com/durable-workflow/sdk-php/issues/193).

## Publish the target operation {#publish-the-target-operation}

The target namespace owns the endpoint, service, operation and handler worker.
Create the namespace first and run a worker that registers the target workflow
type on its selected queue. The following catalog uses target namespace
`billing`, workflow type `billing.invoice` and queue `billing`.

Catalog mutations require an admin credential. The runtime enforces caller
authority separately when a leased workflow emits its service command.

```bash
curl --fail-with-body -X POST "$DURABLE_WORKFLOW_SERVER_URL/api/service-endpoints" \
  -H "Authorization: Bearer $DW_ADMIN_TOKEN" \
  -H "X-Namespace: billing" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{"endpoint_name":"billing"}'

curl --fail-with-body -X POST "$DURABLE_WORKFLOW_SERVER_URL/api/service-endpoints/billing/services" \
  -H "Authorization: Bearer $DW_ADMIN_TOKEN" \
  -H "X-Namespace: billing" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{"service_name":"invoices"}'

curl --fail-with-body -X POST "$DURABLE_WORKFLOW_SERVER_URL/api/service-endpoints/billing/services/invoices/operations" \
  -H "Authorization: Bearer $DW_ADMIN_TOKEN" \
  -H "X-Namespace: billing" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{"operation_name":"create","operation_mode":"sync","handler_binding_kind":"start_workflow","handler_target_reference":"billing.invoice","handler_binding":{"workflow_type":"billing.invoice","queue":"billing"},"deadline_policy":{"timeout_seconds":30}}'
```

The catalog names are normalized to lowercase. Endpoint and service records
organize the operation; they do not replace its real handler. Keep a compatible
worker available in the target namespace. Registration does not prove that work
has executed.

## Waiting and recorded results {#waiting-and-recorded-results}

The PHP SDK defaults to `mode_override=sync` and `wait_for=completed`. Python
uses `mode='sync'` and `wait_for='completed'`. Both suspend until the service
reaches a durable terminal outcome and replay that recorded result or typed
failure at the original call. A pending or started call is not a successful
business result.

For admission-only behavior, PHP uses `wait_for='accepted'` or
`mode_override='async'`; Python supports `wait_for='accepted'` or
`mode='async'` and the `start_nexus_operation()` helper. Preserve the returned
call id for inspection. A later completion does not replace the caller's
original recorded admission. Choose completion waiting when workflow decisions
need the target result.

Bound service execution with the catalog's `deadline_policy`, as above. It
accepts exactly one positive integer `timeout_seconds` or an offset-bearing
RFC3339 `deadline_at`. The runtime records one `call_deadline_at` from original
acceptance. Replay, retries and later catalog edits cannot reset it. A null
policy does not create a new service deadline.

Do not use the caller's `wait_timeout_seconds` as an execution timeout. A
one-second waiting option with no catalog deadline can still deliver a target
that completes after three seconds. Inspect the original call and linked
target after a timeout; expiration does not prove that external effects did
not happen, and target cancellation is a separate policy.

## Failures, retries and idempotency {#failures-retries-and-idempotency}

PHP exposes terminal service outcomes through `ServiceOperationFailed`;
Python raises `NexusOperationFailed` at the yield point. Keep typed failure
details and the call id when handling an error. Expected business failures may
be handled by workflow code; unexpected ones can remain workflow failures.

The runtime owns the recorded service retry policy. Do not add a second HTTP
retry loop inside workflow code. PHP and Python derive stable keys for normal
workflow calls. If the application supplies a key, keep the same target,
request and intended operation for retries and replay. A new key is a new
business request, not recovery of a lost acknowledgment.

Keep target namespace, catalog triple, payload and waiting options stable for
open histories. Use a workflow version branch when that durable intent changes.

## Enforce the namespace boundary {#enforce-the-namespace-boundary}

The cross-namespace default is `allow`. Authentication, role checks, other
boundary policies and namespace authorization still
apply. Caller identity comes from the leased workflow; application input cannot
impersonate a different caller namespace.

For a self-hosted PHP Server that should refuse cross-namespace service calls,
configure and restart its API/worker roles consistently:

```bash
DW_SERVICE_BOUNDARY_CROSS_NAMESPACE_DEFAULT=deny
```

The policy refuses the cross-namespace call before handler dispatch. Same-
namespace calls and other boundary policies still need their own review.
Do not treat this variable as a replacement for credentials or tenant ACLs.

## Inspect the original call {#inspect-the-original-call}

Use the [service-call detail and catalog API](./server-api-reference.md#service-catalog-admin-apis)
or [CLI service-call inspection](./cli-reference.md#service-call-inspection).
Keep the caller workflow/run and target service-call ids together. Read-only
inspection does not request another execution. Reuse the original id when
checking a lost response or worker replacement.

<!-- Source: docs/polyglot/visibility-query-language.md -->

# Visibility Query Language

Use `GET /api/workflows?query=...` to find runs in one Server namespace. This reference describes the published native Server selected in the [compatibility manifest](../compatibility.md#last-qualified-reproducible-tuple), currently `3.10.0`. Listing reads stored visibility facts. It does not execute a workflow's named query handler.

The request needs an operator or admin bearer token, `X-Namespace` and `X-Durable-Workflow-Control-Plane-Version: 2`. See [namespace authorization](./namespace-auth-workers.md). Results include matching runs, including earlier runs of an instance and children whose attributes match. There is no built-in roots-only predicate.

## Make a request {#make-a-request}

Set `DURABLE_WORKFLOW_SERVER_URL` to your Server base URL and `DURABLE_WORKFLOW_AUTH_TOKEN` to an operator or admin token. This example needs Bash, curl and jq. Replace the fixed UTC interval with the interval you want to inspect.

```bash
query='Status = failed AND CloseTime >= "2026-11-14T00:00:00Z" AND CloseTime < "2026-11-15T00:00:00Z"'
page=$(curl --fail-with-body --silent --show-error --get \
  "$DURABLE_WORKFLOW_SERVER_URL/api/workflows" \
  -H "Authorization: Bearer $DURABLE_WORKFLOW_AUTH_TOKEN" \
  -H 'X-Namespace: default' \
  -H 'X-Durable-Workflow-Control-Plane-Version: 2' \
  --data-urlencode "query=$query" \
  --data-urlencode 'page_size=50')
printf '%s\n' "$page" | jq '{workflow_count, workflows, next_page_token}'
```

`Status = failed` selects a bucket that also includes cancelled and terminated runs. Use `Status = cancelled` or `Status = terminated` for those exact statuses. `running` includes pending, running and waiting. `completed` selects completed runs. Status values are case-insensitive.

Use `--data-urlencode` for the query and page token. Shell quoting, query-string escaping and URL encoding are separate steps. A successful response with `workflows: []` means no runs matched this request.

## Fields {#fields}

Field names are case-sensitive. Built-in names take precedence over custom attributes.

| Field | Value | Operators |
| --- | --- | --- |
| `WorkflowType`, `WorkflowId`, `RunId`, `TaskQueue` | Run type, instance ID, run ID or task queue | `=`, `!=`, `IN`, negated `IN` |
| `Status`, `ExecutionStatus` | Status or status bucket, as above | `=`, `!=` |
| `StartTime`, `ExecutionTime` | The run's start instant. These names are aliases. | Scalar comparisons, `IN`, negated `IN` |
| `CloseTime` | The run's close instant | Scalar comparisons, `IN`, negated `IN` |
| `BuildId`, `BuildIds` | Aliases for the run's compatibility value, not a worker-fleet lookup | Keyword operators, subject to the native refusal below |

Scalar comparisons are `=`, `!=`, `<`, `<=`, `>` and `>=`. A missing start or close instant never matches a comparison, including a negated one. Native listing refuses a returned run with non-null compatibility as HTTP `409` / `unqualified_build_visibility`. A parsed build predicate does not establish qualified build-based listing.

Other fields must be registered search attributes in this namespace. Use `GET /api/search-attributes` to inspect definitions and `POST /api/search-attributes` with a JSON body such as `{"name":"OrderTotal","type":"double"}` to register one. Registration needs an admin token. An inferred stored type alone does not make an unregistered key queryable. Deleting a definition leaves stored values but makes queries on that key invalid.

Identifiers start with an ASCII letter and continue with ASCII letters, digits or `_`. Use the registered name exactly. Memos, visibility labels, business keys and parent/root relationship names are not additional built-in structured fields. Plain search can still find a business key as described below.

## Types and comparisons {#types-and-comparisons}

| Registered type | Literal and comparison behavior |
| --- | --- |
| `keyword` | Whole-value text equality or inequality, `IN` and negated `IN`. No ordering. |
| `string`, `text` | Whole-value text comparisons, with the same operators as `keyword`. No substring, full-text or `LIKE` operator. |
| `int` | Unquoted integer, such as `42` or `-3`. Scalar comparisons, `IN` and negated `IN`. A quoted number or a decimal is not an integer literal. |
| `double` | Unquoted integer or decimal, including exponent notation such as `1.5e3`. Scalar comparisons, `IN` and negated `IN`. |
| `bool` | Unquoted `true` or `false`, case-insensitive. Scalar comparisons, `IN` and negated `IN`; `false` sorts before `true`. `1` and `"true"` are not boolean literals. |
| `datetime` | Quoted datetime. Scalar comparisons, `IN` and negated `IN` compare instants. Use explicit UTC or offset timestamps, such as `"2026-11-14T12:34:56.123456Z"`. |
| `keyword_list` | `Tags = "urgent"` tests membership. `Tags IN ("urgent", "review")` matches any listed member. Inequality, ordering and negated membership are refused. |

A run without a queried attribute never matches that predicate, including `!=` and negated `IN`. There is no existence or `IS NULL` predicate. A value retained under an earlier, different registered type is not automatically converted to the new type.

Keyword, text and built-in text comparisons follow the database's equality rules. The published MySQL/MariaDB schema uses `utf8mb4_unicode_ci`, which ignores case, accents and trailing spaces. SQLite and the native PostgreSQL comparison are exact. Keyword-list membership is exact. Do not assume identical text results across these backends.

Datetimes accept ISO dates and timestamps, with fractions retained to microseconds. A date alone means midnight UTC; an omitted timezone is UTC. Use explicit timestamps for portable queries. Relative dates such as `tomorrow` and timezone names such as `Europe/Paris` are not accepted datetime values. Calculate a rolling interval in the caller and send fixed boundaries.

## Syntax, precedence and escaping {#syntax-and-escaping}

- Write comparisons as `Field operator literal`. Separate predicates with `AND` or `OR`. Keywords are case-insensitive.
- `NOT` applies to the next predicate, before `AND`; `AND` binds more tightly than `OR`. `A = 1 OR B = 2 AND C = 3` means `A = 1 OR (B = 2 AND C = 3)`.
- Parentheses delimit a nonempty, comma-separated `IN` list. They do not group boolean expressions. Repeat predicates across `OR` branches when necessary.
- Negated membership is **`NOT Region IN ("eu", "us")`**. The SQL spelling `Region NOT IN (...)` is not supported. `Status IN (...)` is also unsupported; use `Status = running OR Status = completed`.
- Use single or double quotes for strings. Escape the chosen quote and a literal backslash with `\`. For example, `Customer = 'O\'Brien'` and `Path = "C:\\orders"`. The parser also decodes C-style escapes such as `\n`, `\t` and `\r`; JSON's `\uXXXX` syntax is not a Unicode escape here. Write Unicode characters directly.
- Bare text literals allow ASCII letters, digits, `_`, `.`, `:`, and `-`. Prefer quoted text so booleans and numbers are not inferred accidentally. There is no SQL `SELECT`, `ORDER BY` or `LIMIT` clause.

In an expression, a bare registered boolean field can mean `Flag = true`, and `NOT Flag` means `Flag = false`. A lone `Flag` is instead plain search text. Explicit comparisons avoid this ambiguity.

The Server recognizes an expression from comparison syntax, `IN (...)` or a leading `NOT` predicate. Other text is a substring search over workflow ID and business key. For example, `query=order-1001` is plain search. Unsupported SQL-like text can therefore fall back to plain search rather than produce a syntax error. Use the documented grammar and inspect `query` errors when a recognized expression is invalid.

## Query examples {#query-examples}

Pass any expression below as the `query` value in the curl request above. Custom examples assume `OrderTotal: double`, `Attempt: int`, `Region: keyword`, `Tags: keyword_list`, `Approved: bool`, and `DueAt: datetime` are registered and populated in the selected namespace.

| Task | Query |
| --- | --- |
| Orders over a numeric threshold | `WorkflowType = "orders.fulfillment" AND OrderTotal > 1000` |
| Attempts within an inclusive range | `Attempt >= 2 AND Attempt <= 5` |
| Either region | `Region IN ("eu", "us")` |
| Other recorded regions | `NOT Region IN ("eu", "us")` |
| A tag from either set | `Tags IN ("urgent", "review")` |
| Explicit false | `Approved = false` |
| A time window | `DueAt >= "2026-11-14T00:00:00Z" AND DueAt < "2026-11-15T00:00:00Z"` |
| Two queues under the same type | `WorkflowType = "orders.fulfillment" AND TaskQueue = "orders" OR WorkflowType = "orders.fulfillment" AND TaskQueue = "priority"` |

## Pagination and limits {#pagination-and-limits}

| Parameter | Behavior |
| --- | --- |
| `workflow_type` | Optional exact type filter, combined with `query` and `status` using `AND`. |
| `status` | Optional filter. Prefer canonical buckets `running`, `completed` or `failed`. The current list endpoint also accepts exact `cancelled` and `terminated`; `pending` and `waiting` are not accepted here. |
| `page_size` | Integer from `1` to `200`, default `50`. Use this parameter, not `limit`. |
| `next_page_token` | Copy the previous response's token without decoding or constructing it. Stop when it is null. |

`workflow_count` is the number in this page, not the total number of matching runs. Results are ordered by descending start/sort time, then descending run ID. Paging reads live data using an offset. Keep namespace, query, filters and page size unchanged. Inserts, closures or attribute updates between requests can move matches and cause repeats or omissions; this is not a snapshot export. An invalid token currently restarts at the first page rather than returning an error.

Fetch the next page from the first example, if present:

```bash
next=$(printf '%s\n' "$page" | jq -r '.next_page_token // empty')
if [ -n "$next" ]; then
  curl --fail-with-body --silent --show-error --get \
    "$DURABLE_WORKFLOW_SERVER_URL/api/workflows" \
    -H "Authorization: Bearer $DURABLE_WORKFLOW_AUTH_TOKEN" \
    -H 'X-Namespace: default' \
    -H 'X-Durable-Workflow-Control-Plane-Version: 2' \
    --data-urlencode "query=$query" \
    --data-urlencode 'page_size=50' \
    --data-urlencode "next_page_token=$next" | jq .
fi
```

Page size limits returned rows, not query evaluation work. Keep expressions and `IN` lists small and use selective type, status or attribute predicates. The workflow parser does not define the schedule API's 2048-character query limit. HTTP servers and proxies can impose their own request-URI limits. [Schedule listing](./server-api-reference.md#schedule-list-filters) has a different, equality-and-`AND` grammar.

## Invalid queries {#invalid-queries}

A recognized expression that fails parsing, field lookup or type checking returns HTTP `422` with `reason: "validation_failed"`, a `message`, and arrays in `errors.query` and `validation_errors.query`. An invalid page size or status uses that parameter's key instead.

| Invalid expression | Correction |
| --- | --- |
| `Unknown = 1` | Register the attribute in this namespace or correct the field name. |
| `OrderTotal = "1000"` | Use the numeric literal `1000`. |
| `Approved = 1` | Use `true` or `false`. |
| `Region > "eu"` | Keywords support equality and membership, not ordering. |
| `Tags != "urgent"` or `NOT Tags IN ("urgent")` | Negative membership is not supported for keyword lists. |
| `Attempt IN ()`, `Attempt = 1 AND` | Supply a nonempty list or complete predicate. |
| `DueAt > "tomorrow"` | Send a concrete timestamp. |

## Embedded mode and Waterline {#mode-support}

This language is available through the Server workflow-list API. Current embedded listing and Waterline saved-view filters do not implement this full grammar. Their existing exact `key=value` metadata filters and substring fields do not establish range, `IN`, `!=` or boolean-expression support. Service-mode Waterline also needs explicit UI/API support before a saved view can be treated as this query language. Shared parity work remains open in [the visibility-query feature](https://github.com/durable-workflow/.github/issues/224).

The reference follows the [published native parser and its unit tests](https://github.com/durable-workflow/server/blob/b6e04ce7e679464e91f6ceae83310d3bb3204092/rust/src/runtime/visibility_query.rs), [HTTP pagination implementation](https://github.com/durable-workflow/server/blob/b6e04ce7e679464e91f6ceae83310d3bb3204092/rust/src/runtime/visibility.rs), and [listing API tests](https://github.com/durable-workflow/server/blob/b6e04ce7e679464e91f6ceae83310d3bb3204092/rust/tests/execution/visibility.rs). These describe the published Server behavior, not completion of embedded or Waterline parity.

<!-- Source: docs/polyglot/workflow-streams.md -->

# Workflow Streams

Workflow Streams are named, run-scoped output logs owned by Durable Workflow
Server. PHP, Python, and Rust expose typed list, describe, subscribe, append,
close, and error operations, so applications do not need to assemble HTTP
requests. The runtime advertises
`durable-workflow.v2.workflow-streams.contract@1`; the
[machine-readable SDK matrix](/workflow-stream-capabilities.json) records the
exact support in each SDK.

Each stream assigns monotonically increasing offsets beginning at 0. A page's
`next_offset` is the next inclusive offset to request. Delivery to consumers is
at least once: finish the page's idempotent effects, then durably checkpoint
`next_offset`. A crash before that checkpoint may redeliver items.

## Emit from workflow code

The three SDKs emit through the same deterministic command boundary:
`record_side_effect.workflow_stream`. Each SDK derives the item idempotency key
from the task's durable workflow command identity, the stream-command ordinal,
and the batch index. Server commits the stream mutation and recorded side
effect together. Replay consumes that side effect and cannot create another
durable item for the same logical append.

PHP:

```php
use DurableWorkflow\Model\WorkflowStreamAppendItem;

$context->appendWorkflowStream('progress', [
    new WorkflowStreamAppendItem(['percent' => 50]),
]);
$context->closeWorkflowStream('progress');
```

Python:

```python
from durable_workflow import WorkflowStreamAppendItem

yield ctx.append_workflow_stream("progress", [
    WorkflowStreamAppendItem(payload={"percent": 50}),
])
yield ctx.close_workflow_stream("progress")
```

Rust:

```rust
let item = WorkflowStreamAppendItem::new(serde_json::json!({"percent": 50}))?;
ctx.append_workflow_stream("progress", &[item], None)?;
ctx.close_workflow_stream("progress", None)?;
```

The workflow helpers are for deterministic authoring. Code outside a workflow
can use each client's typed `append` operation and supply its own stable
idempotency key.

## Subscribe and resume

Subscription reads a bounded page, optionally waiting for new items for up to
60 seconds. PHP accepts a cancellation callback between bounded polls. Python
accepts normal task cancellation or an `asyncio.Event` that cancels the
in-flight poll. Dropping Rust's subscription future cancels its request.

| SDK | Page operation | Resume value | Typed iteration |
| --- | --- | --- | --- |
| PHP | `subscribeWorkflowStream(...)` | `WorkflowStreamPage::$nextOffset` | `iterateWorkflowStream(...)` |
| Python | `await subscribe_workflow_stream(...)` | `WorkflowStreamPage.next_offset` | `iter_workflow_stream(...)` |
| Rust | `subscribe_workflow_stream(...).await` | `WorkflowStreamPage::next_offset` | Repeat the bounded page future until `terminal`. |

`open`, `closed`, and `errored` are the lifecycle states. The typed description
also exposes `last_offset`, `total_items`, `pending_items`, and `error_reason`.
An append to a terminal stream is rejected. When the configured pending bound
is reached, append returns `stream_full`; producers must slow down or wait for
consumers to drain the stream.

## External payload references

Inline values use the shared Avro Value envelope. External references follow
each SDK's existing storage contract rather than inventing another transport:

| SDK | External reference behavior |
| --- | --- |
| PHP | Appends and returns an opaque `payload_reference`; storage upload and fetch remain application-owned. |
| Python | Uses the configured external-storage driver to upload, integrity-check, cache, fetch, and decode an external Avro envelope. Without a driver, callers can append or inspect an opaque `payload_reference`. |
| Rust | Appends and returns an opaque reference and its metadata. The current Rust SDK does not claim an external-storage driver. |

## Workflow Streams and embedded MessageStream

The names align where behavior aligns, but the models are not interchangeable.

| Concept | Service-mode Workflow Stream | Embedded Laravel MessageStream |
| --- | --- | --- |
| Address | Workflow run + stream name | Workflow instance/run + stream key |
| Direction | Workflow output only | Workflow inbox and outbox |
| First offset | 0 | 1 |
| Delivery | At least once; consumer-owned checkpoint | At least once; engine-owned message cursor |
| Continue as new | No stream cursor transfer | Inbox cursor transfers to the continued run |
| Inbound workflow messaging | Not provided; use signals or updates | Provided by `inbox()->receive()` |

Waterline uses one normalized table for both modes and shows the mode,
lifecycle, head/cursor offsets, pending count, direction, and error. It does not
present service output streams as an inbound workflow inbox.

## Qualification

The public
[Workflow Stream scenario manifest](/platform-conformance/workflow-stream-runtime-scenarios.json)
requires published-artifact runs for PHP producer to Python and Rust consumers,
producer worker restart, consumer reconnect, lifecycle/backpressure/cancellation,
and external payload references. A release result must record every required
scenario; source-unit tests alone are not conformance evidence.

<!-- Source: docs/rolling-upgrades.md -->

# Rolling Upgrades

## Replay before changing builds {#replay-before-changing-builds}

Before promoting a candidate, replay representative open and recently closed histories against its exact application code. Before rollback, replay histories written by the newer build against the older artifact. Follow the [replay testing gate](./testing.md#replay-before-deployment); a compatible routing cohort or successful integrity check alone does not prove replay compatibility.

Run a rolling upgrade when you want to replace API nodes, workers, or the
scheduler without taking the deployment offline. This contract covers the
small clustered shape from the [self-hosting deployments](/docs/deployment)
guide: two or three API nodes behind a load balancer, shared external MySQL
or PostgreSQL, shared Redis, independently scaled workers, and exactly one
scheduler or maintenance runner.

A rolling upgrade is supported when every guarantee on this page holds.
Outside that envelope, use the documented stop-the-world flow in the
deployment guide instead.

The role vocabulary and shape manifest behind this contract are documented in
[Server Role Topology](/docs/polyglot/server-role-topology). This guide
focuses on how the current `standalone_server` process classes roll in place.

## What rolling upgrade means here

A **rolling upgrade** replaces processes one at a time, draining each one
before stopping it, while the rest of the fleet keeps serving traffic. The
result is zero downtime for the deployment as a whole and a bounded overlap
window where old and new processes coexist.

The contract distinguishes four process classes in the current
`standalone_server` shape, each with its own rollout posture:

- **HTTP/API nodes**: stateless server processes serving HTTP traffic and
  currently hosting the `api_ingress`, `control_plane`, `matching`, and
  `history_projection` roles.
- **Workers**: SDK processes that poll the worker plane and execute
  activity and workflow tasks as the `execution_plane`.
- **Scheduler / maintenance runner**: the singleton process that fires
  schedules and drives activity-timeout and history retention as the
  `scheduler` role.
- **Bootstrap**: the one-shot process that runs database migrations and
  default-namespace seeding.

API nodes and workers roll independently. The scheduler is a singleton —
you stop the old one before starting the new one, but the rest of the
deployment keeps serving traffic across that gap.

## Compatible version-skew rules

The rolling-upgrade overlap window is the time during which more than one server
image, workflow package version, or worker SDK version is live. The
window must satisfy every rule in this section.

### Server image and workflow package

- **Adjacent versions only.** During a rolling upgrade, every API node
  and other Server process must run a server image whose workflow package
  version is the same major version as the cluster's previous package and within
  one minor version of every other live process. Skipping a major
  version requires a stop-the-world upgrade.
- **Forward-additive migrations.** Every Durable Workflow v2 schema
  change is additive within a major version. New nodes must not require
  a column or table that has not been migrated in. Old nodes must not
  break when a new column they do not read is present. The
  [migration order rules](#schemabootstrap-ordering) below enforce this.
- **Adjacent control-plane and worker-protocol versions.** Every node
  publishes its supported `control_plane.version` and
  `worker_protocol.version` ranges from `GET /api/cluster/info`. During
  a rolling upgrade, the new image's supported range must overlap with
  every live old node's supported range. Discover the range before the
  rollout, and abort if it does not overlap.

### Worker SDK and build identity

- **Workers tag every build.** Every worker that may participate in a
  rolling upgrade must register through `POST /api/worker/register`
  with a stable `build_id`. The unversioned cohort
  (`build_id: null`) is the pre-rollout default; the
  [worker build-id rollout guide](/docs/polyglot/worker-build-id-rollout)
  explains the first cutover.
- **Preserve original routing and qualify changed definitions.** A run keeps
  its build pin and the fingerprint recorded at `WorkflowStarted`. With
  `DW_V2_PIN_TO_RECORDED_FINGERPRINT=true` (the default), Server 2.9.9 and later
  prefer eligible recorded-definition workers. If none
  remains, a replacement with `workflow_entry_arguments_validation` may claim
  under the original namespace, queue, type, build and replay guards. The SDK
  must still validate original entry input and replay committed commands.
- **Overlapping-build admission posture.** Choose the
  [`DW_V2_FLEET_VALIDATION_MODE`](/docs/polyglot/server-config-reference)
  posture before you start. `warn` lets the rollout proceed even when
  the required compatibility marker has no live worker;  `fail` blocks
  dispatch and fails the readiness contract closed during that window.
  Production rollouts that require a clean cutover should be on `fail`.

**Released support.** That replacement path is available from Server 2.9.9 with
PHP SDK 2.5.4, Python SDK 2.6.2 or Rust SDK 3.5.6. These are minimum feature
versions; install current stable releases for security fixes.
Older runtimes can keep refusing a changed fingerprint even when application
code adds a valid version branch. Disabling fingerprint pinning is not the
default safe-upgrade procedure. Follow [definition routing](./polyglot/worker-compatibility-routing.md#definition-fingerprint-evolution),
inspect the actual worker capability and retain rollback artifacts until the
new path is qualified. A version branch cannot fix incompatible entry arguments
before the workflow body starts; see [entry compatibility](./features/versioning.md#service-sdk-entry-safeguard).

## Schema/bootstrap ordering

Schema changes ride a single bootstrap pass. Order matters.

1. **Run bootstrap first, exactly once.** Run `php artisan
   server:bootstrap --force` (or the published-image equivalent) from
   one container before starting any new API node, worker, or
   scheduler. Bootstrap runs `migrate` plus default-namespace seeding;
   it adopts any workflow package migrations whose tables already
   exist on the connection.
2. **New schema must be backwards-compatible with old code.** Every v2
   migration in this release path is additive (new tables, new
   columns, new indexes). Old API nodes and workers continue running
   against the new schema.
3. **Do not start new code before bootstrap completes.** Roll the new
   server image only after bootstrap exits successfully. New API nodes
   and workers may rely on the freshly migrated tables, and starting
   them before bootstrap finishes is the most common cause of a 5xx
   surge during the cutover.
4. **Bootstrap is idempotent.** Re-running it is safe. If bootstrap
   fails partway, fix the underlying error and re-run; the migration
   ledger picks up where it left off, and the namespace seed is a
   no-op when the row exists.

A migration that lands on one server before another MUST NOT corrupt
the readiness surface. If you discover a non-additive change during
planning, take a stop-the-world upgrade window for that release and
return to rolling upgrades on the next one.

## Drain and admission during the overlap window

Three admission surfaces enforce overlap-window safety automatically:

- **Boot-time admission.** Every server process loads
  `BackendCapabilities`, `LongPollCacheValidator`, `WorkflowModeGuard`,
  and the readiness contract at boot. A process whose backend or
  cache cannot satisfy the v2 contract refuses to mark itself ready.
- **Worker compatibility.** When
  `DW_V2_FLEET_VALIDATION_MODE=fail` and no live worker advertises the
  required compatibility marker for a task's connection and queue
  scope, the matching role blocks dispatch and the
  `worker_compatibility` health check escalates from `warning` to
  `error`. Tasks stay ready and visible — they are never dropped — and
  the readiness contract returns 503 on that node so the load balancer
  takes it out of rotation.
- **Routing safety.** A ready task whose required compatibility has no
  live worker is preserved and counted under the
  `compatibility_blocked_runs` backlog metric. Routing safety never
  silently escalates to task loss; the at-least-once execution
  guarantee still applies after a routing drain.

To keep the worker overlap window short, drain old
worker cohorts as new workers come online. Use the
[worker build-id rollout](/docs/polyglot/worker-build-id-rollout)
flow:

```bash
dw task-queue:drain orders-critical --build-id orders-worker-2026-04-21-z9
```

The cohort's `drain_intent` flips to `draining`. Workers under that
build keep finishing in-flight work but stop claiming new tasks. Wait
for the cohort's `active_worker_count` and `draining_worker_count` to
reach zero before stopping the old worker processes.

The scheduler does not run a long-lived task queue, so it does not need
a worker drain. Stop the old scheduler container, run bootstrap if it
has not run yet, and start the new one. The window between the two is
bounded by how long it takes the new container to come up; schedule
firing resumes from the persisted state on next tick.

If the rollout also introduces a dedicated matching-role deployment, make that
topology change explicit instead of assuming every execution node will keep
doing broad ready-task sweeps. See
[Task Matching and Dispatch](/docs/polyglot/task-matching-dispatch) for
the documented `workflow:v2:repair-pass --loop` plus
`DW_V2_MATCHING_ROLE_QUEUE_WAKE=0` shape. Verify the live node contract from
`GET /api/cluster/info`: `topology.current_shape` should still match the
deployment you are cutting over, `topology.current_roles` should still match
the documented role bundle for that node, and
`topology.matching_role.queue_wake_enabled`,
`topology.matching_role.shape`, and `topology.matching_role.wake_owner`
should show the expected broad-ready-task owner. The default shape reports
`queue_wake_enabled: true`, `shape: "in_worker"`, and
`wake_owner: "worker_loop"`; dedicated matching rollouts flip execution nodes
to `queue_wake_enabled: false`, `shape: "dedicated"`, and
`wake_owner: "dedicated_repair_pass"`.

## Readiness and cutover

Use the readiness contract — not just the liveness probe — to decide
when traffic flows to a node.

- `GET /api/health` proves the process is serving HTTP.
- `GET /api/ready` proves the process can use its configured runtime
  dependencies, including migrations, default namespace, and (under
  `DW_V2_FLEET_VALIDATION_MODE=fail`) the worker-compatibility
  admission check.
- `GET /api/cluster/info` proves an authenticated client can discover
  build identity, control-plane protocol, worker protocol, payload
  codecs, and server capabilities.
- `POST /api/worker/register` proves workers can authenticate into the
  expected namespace and task queue.

Cutover sequence for one API node:

1. Take the node out of the load balancer rotation. The simplest path
   is to fail the load balancer's readiness probe by stopping the new
   image's pre-start hook before bringing the new container up.
2. Drain in-flight HTTP requests. Most clients retry on connection
   reset; long-running connections (worker long-polls) reconnect
   against the rest of the fleet.
3. Stop the old container, start the new one.
4. Wait for `GET /api/ready` to return 200 and for
   `GET /api/cluster/info` to advertise the new build identity. When the
   rollout changes the matching topology, also confirm
   `topology.matching_role.task_dispatch_mode`,
   `topology.matching_role.queue_wake_enabled`,
   `topology.matching_role.shape`, `topology.matching_role.wake_owner`,
   `topology.matching_role.partition_primitives`, and
   `topology.matching_role.backpressure_model` match the intended deployment
   before returning the node to traffic. Use `/api/system/operator-metrics`
   when you want the same node-local matching-role contract alongside live
   backlog, repair, and worker counters from the responding process.
5. Return the node to rotation.

Repeat one node at a time. Do not roll the next node until the
previous one is back in rotation and serving traffic cleanly.

For workers, the cutover is per-cohort:

1. Bring the new worker cohort online with a new `build_id`.
2. Confirm both cohorts report `rollout_status: "active"` and non-zero
   `active_worker_count` from `dw task-queue:build-ids <queue> --json`.
3. Drain the old cohort with `dw task-queue:drain`.
4. Wait until the old cohort's `active_worker_count` and
   `draining_worker_count` reach zero.
5. Stop the old worker processes.

## Rollback

Every step is reversible. Plan for rollback before you start.

- **Bootstrap rollback.** v2 migrations are reversible by the standard
  Laravel `down()` path. A rollback that reverts a migration the new
  image relies on requires stopping every new node first; otherwise
  the new code observes a missing column and the readiness contract
  returns 503 on those nodes. Most rollbacks do not need to reverse
  migrations because schema changes are additive.
- **API node rollback.** Stop the new container, restart the old one
  on the same node. Take the node out of rotation while it boots and
  return it once `GET /api/ready` is green. Repeat for any other
  upgraded API nodes. Old code keeps reading the new schema cleanly
  because the schema change was additive.
- **Worker rollback.** Resume the previously drained cohort, drain
  the bad cohort, and scale the known-good build back up:

  ```bash
  dw task-queue:resume orders-critical --build-id orders-worker-2026-04-21-z9
  dw task-queue:drain  orders-critical --build-id orders-worker-2026-04-22
  ```

  Resume clears `drain_intent` and `drained_at`, and any worker
  heartbeating under the resumed `build_id` flips back to `active` on
  the next poll. Both calls are idempotent.

If the rollout exposed a non-additive schema problem, take a
stop-the-world upgrade window to roll back, run the corrective
migrations, and re-plan.

## Operator verification

Verify each phase of the rollout from operator surfaces, not from logs.

| Question | Surface |
| --- | --- |
| Is bootstrap finished? | `php artisan server:bootstrap --force` exit code 0; `migrate:status` shows every migration ran. |
| Is the new node ready? | `GET /api/ready` returns 200; `GET /api/cluster/info` reports the new build. |
| Is compatibility admission healthy? | `GET /api/system/operator-metrics` `workers.fleet`, `workers.active_workers`, and `workers.active_workers_supporting_required` agree on a non-zero supporter count for every required compatibility marker. |
| Is the worker drain progressing? | `dw task-queue:build-ids <queue> --json` shows `active_worker_count` and `draining_worker_count` falling for the draining cohort. |
| Is routing safe? | `GET /api/system/operator-metrics` `backlog.compatibility_blocked_runs` and `backlog.max_compatibility_blocked_age_ms` stay near zero; the `worker_compatibility` health check is not in `error`. |
| Is the scheduler caught up? | `GET /api/system/operator-metrics` `schedules.missed` is zero and `schedules.oldest_overdue_at` is null. |
| Are stuck runs piling up? | `GET /api/system/operator-metrics` `runs.repair_needed` and `runs.max_repair_needed_age_ms` stay near their pre-rollout baseline. |

`dw system:operator-metrics --json` exposes the same operator-metrics snapshot
on the console for the standalone-server fleet, so operators may pick whichever
surface matches their existing workflow. A separately deployed Waterline
service can read the same server-owned rollout state through the PHP SDK under
`/waterline/api/v2/health` and `/waterline/api/stats`; keep its endpoint,
namespace, and server token aligned with the fleet being rolled. Embedded
Laravel deployments expose those Waterline routes from their in-process
package instead. The server API and CLI remain available in either case, as
documented in the
[Operator Operating Envelope](/docs/operator-operating-envelope).

## Failure modes and what to do

| Symptom | Likely cause | Action |
| --- | --- | --- |
| New API node fails `GET /api/ready` after start | Bootstrap did not finish, or `DW_V2_FLEET_VALIDATION_MODE=fail` and no compatible worker is live yet | Rerun bootstrap; bring a compatible worker cohort online before adding the API node back to rotation. |
| `worker_compatibility` health check escalates to `error` mid-rollout | The required compatibility marker has no live supporting worker | Bring more workers under a supporting `build_id` online; resume a previously drained cohort if rollback is the right call. |
| `backlog.compatibility_blocked_runs` climbs and `max_compatibility_blocked_age_ms` grows | Tasks are queued for a marker no live worker supports | Same as above; tasks are preserved and will redispatch automatically once a compatible worker heartbeats. |
| `dw task-queue:drain` returns success but workers keep claiming tasks | The worker process did not heartbeat after the drain | Wait one heartbeat cycle; if the cohort stays active, restart the worker process so it picks up the drain intent. |
| Schedule fires stop after a scheduler restart | Old and new scheduler are both stopped | Start the new scheduler container; verify `schedules.missed` returns to zero on next operator-metrics scrape. |

If a symptom is not on this list, treat the rollout as failed: stop
adding new processes, drain whatever new cohorts you started, and
restore the previous build before debugging further.

## Related references

- [Self-Hosting Deployments](/docs/deployment) for the deployment
  shapes this contract assumes.
- [Worker Build-Id Rollout](/docs/polyglot/worker-build-id-rollout)
  for the per-cohort drain and resume calls.
- [Operator Operating Envelope](/docs/operator-operating-envelope)
  for the diagnostic, queue, and rebuild contract that operators read
  alongside the rollout signals.
- [Server Config Reference](/docs/polyglot/server-config-reference)
  for the rollout-safety environment variables (`DW_V2_FLEET_VALIDATION_MODE`,
  `DW_V2_PIN_TO_RECORDED_FINGERPRINT`, `DW_V2_GUARDRAILS_BOOT`,
  `DW_V2_CACHE_VALIDATION_MODE`, `DW_V2_MULTI_NODE`,
  `DW_V2_VALIDATE_CACHE_BACKEND`, `DW_V2_TASK_REPAIR_*`).
- [Server API Reference](/docs/polyglot/server-api-reference) for
  the readiness, cluster info, and operator-metrics endpoints used to
  verify each phase of the rollout.

<!-- Source: docs/compatibility.md -->

# Version Compatibility

This page is the **canonical compatibility and release-authority contract** for
the Durable Workflow public platform. It is the single source of truth for:

- which surfaces are public,
- the stability level of each public surface,
- which changes may ship in a patch, minor, or major release,
- whether a given field is part of the contract or is diagnostic-only,
- and the runtime version-negotiation protocol clients use to fail closed when
  the server advertises a surface they cannot speak.

Per-package stability documents (for example `docs/api-stability.md` in the
`durable-workflow/workflow` repository, the `dw` CLI reference, the Waterline
operator API page) are **downstream** of this page. They add per-package
detail under these rules; when a per-package document and this page disagree,
this page wins, and the disagreement is a bug in the per-package document.

The same contract is published in machine-readable form so SDKs, server
manifests, and CI gates can validate themselves against one source of truth:

- `surface_stability_contract` in the response body of
  `GET /api/cluster/info` on the standalone Durable Workflow server, schema
  `durable-workflow.v2.surface-stability.contract`, version `5`.
- A frozen mirror of the same manifest in this repository at
  `static/compatibility-contract.json`.
- The PHP class `Workflow\V2\Support\SurfaceStabilityContract`, which is
  the in-process source the server re-exports.

A release that changes any surface listed below — its stability level, its
field set, its breaking-change rules — must update this page, the JSON
mirror, the PHP manifest, and any per-package stability document in the same
change. Docs CI validates the machine-readable contract, released Rust metadata
when available, and worker-protocol specs.
Editorial alignment between this page and the manifest remains an explicit
release-review responsibility.

## Companion: Platform Protocol Spec Catalog

This page says *which* surfaces are public and *how* they may change.
The companion [Platform Protocol Specs](/docs/platform-protocol-specs)
catalog says *where* the normative machine-readable specification for
each surface lives, *which format* the spec uses (OpenAPI for HTTP APIs,
JSON Schema for object families, AsyncAPI for event-stream semantics),
*which repository* owns the spec, which object families it governs, and
which public URL resolves the artifact. SDK authors, agents, and operators
should validate against the spec catalog rather than re-reading prose or
depending on repository-local implementation details.

The catalog is advertised as `platform_protocol_specs` in
`GET /api/cluster/info`.

Every catalog entry's `surface_family` must exist in the contract above;
docs-site CI validates the catalog, resolves each public spec URL, and rejects
repository-local authority fields.

Every required platform protocol catalog entry is marked `published`; the
invocable carrier entry remains `in_progress`. Every available entry links
directly to a public OpenAPI, AsyncAPI, or JSON Schema document. The
[`cluster_info_envelope`](/docs/platform-protocol-specs#cluster-info-envelope-notes)
schema pins the discovery surface every other catalog entry can be
reached from.

## Published Laravel support {#laravel-support}

Select the package release before comparing PHP and Laravel requirements.
The embedded package and the service-mode SDK have separate support contracts.

| Package line | PHP | Laravel |
| --- | --- | --- |
| Current embedded Workflow 2.28.1 | PHP 8.2+ in 8.x; PHP 8.3+ for Laravel 13 | `^12.72.0` or `^13.41.0` |
| Embedded Workflow 2.20.5 | PHP `^8.1`, subject to Laravel's own minimum | `^10.50.4`, `^11.56.1`, `^12.69.0` or `^13.30.0` |
| Published PHP SDK 3.14 | Bare SDK PHP `^8.1`; the Laravel bridge needs PHP `^8.2`, and PHP 8.3+ for Laravel 13 | `^12.69.0` or `^13.30.0` |
| PHP SDK 3.9.5 | Bare SDK PHP `^8.1`; the Laravel bridge also follows Laravel's PHP minimum | `^10.50.4`, `^11.56.1`, `^12.69.0` or `^13.30.0` |

The older rows record the requirements of their named releases.

The [published embedded upgrade contract](https://github.com/durable-workflow/workflow/blob/2.28.1/resources/laravel-embedded-upgrade-contract.json)
and each release's `composer.json` are the package authorities. The current
embedded release was verified on Laravel 12 with PHP 8.2–8.4 and Laravel 13 with
PHP 8.3–8.4. Its PHP 8.1 and Laravel 10/11 support has been removed. This does
not change the [historical stable-v1 installation guide](/docs/1.x/installation/)
or the independently published [PHP SDK requirements](https://php.durable-workflow.com/).

Workflow 2.28.1 is a security-only patch that raises the Laravel minimums shown
above. Runtime and schema files are identical to 2.28.0, so upgrading from
2.28.0 adds no database migration. The [published package verification and five upgrade/rollback journeys](https://github.com/durable-workflow/workflow/pull/1141#issuecomment-6871390667)
qualify this package.

Before upgrading an older embedded application, upgrade PHP and Laravel to a
supported combination. If that migration is not ready, `~2.20.5` keeps Composer
on the 2.20 patch line; it does not promise indefinite maintenance. A broad
`^2.0` constraint can select a later minor once the platform permits it.
Preserve the original runtime for open v1 runs, and follow the
[Laravel adoption guide](/docs/laravel-adoption) for drain or coexistence.
Changing a Composer requirement does not migrate durable history.

PHP SDK 3.10 moved the Laravel service-mode bridge to the same PHP and Laravel
floors as Workflow 2.21 and removed Laravel 10/11 support; 3.11 through 3.14 keep them.
Its provider refuses unsupported or unknown framework and PHP versions before
registering configuration, clients or workers. Existing SDK 3.9.x applications
keep their own contract while the platform is upgraded. See the
[PHP SDK upgrade notes](./polyglot/php.md#upgrade-to-3-10-and-3-11).

Workflow 2.22 adds no database migration and keeps the 2.21 support floors.
Its embedded worker protocol advertises `1.22` and still accepts `1.20`
requests. `1.22` adds an opt-in occurrence deadline that external callers can
request for condition waits; the embedded runner keeps its existing waiting
behavior. After any workflow has used that policy, roll back to an earlier
Workflow release only by restoring a backup taken before the upgrade. Older
releases can discard the policy and mix the wait chain, and upgrading again
does not repair that history. See the
[2.22.0 changelog](https://github.com/durable-workflow/workflow/blob/2.22.0/CHANGELOG.md).

Workflow 2.23 and 2.24 keep the 2.21 support floors and the embedded worker
protocol `1.22`. 2.23 adds no database migration. 2.24 adds one, so run
`php artisan migrate` when you upgrade. It adds nullable
`last_heartbeat_progress`, `last_heartbeat_progress_at` and
`last_heartbeat_recorded_at` columns to `activity_attempts`, with no backfill.
Behavior changes to check before upgrading:

- From 2.24, a heartbeat writes an `ActivityHeartbeatRecorded` event only when
  its progress changed, at most once per max(heartbeat timeout, 60 seconds).
  See [Heartbeats](./features/heartbeats.md).
- From 2.24, creating a schedule with an id that is already taken, including a
  deleted or exhausted schedule, raises `ScheduleAlreadyExistsException`, and
  re-applying an unchanged schedule records nothing.
- From 2.23.1, a new activity with `maxAttempts` below 1 fails the run instead
  of running once. For unlimited attempts, omit `maxAttempts` and set the
  activity class `$tries` to 0. Histories recorded earlier keep replaying.
- From 2.23.1, embedded saga compensation runs under a cancellation shield,
  and an update accepted while the run is closing still gets its result.

See the release notes for
[2.23.0](https://github.com/durable-workflow/workflow/releases/tag/2.23.0),
[2.23.1](https://github.com/durable-workflow/workflow/releases/tag/2.23.1),
[2.23.2](https://github.com/durable-workflow/workflow/releases/tag/2.23.2) and
[2.24.0](https://github.com/durable-workflow/workflow/releases/tag/2.24.0).

Workflow 2.25 and 2.26 keep the 2.21 support floors and the embedded worker
protocol `1.22`. 2.25 adds no database migration. 2.26 adds one, so run
`php artisan migrate` when you upgrade. It creates the
`workflow_child_supervisions` table. Changes to check before upgrading:

- From 2.26, a parent that continues as new while children it started are
  still open hands them to the continued run. The chain's final run applies
  their parent-close policy, and a child that fails is still retried under its
  recorded retry policy. See
  [Continue-as-new and open children](./features/child-workflows.md#continue-as-new-and-open-children).
- From 2.26, the migrations from earlier releases that fill in existing rows
  (`2026_07_13`, `2026_09_24`, `2026_09_29` and `2026_10_11`) no longer lock the
  table for the whole backfill. They add the column first, fill rows in small
  batches and then build their indexes without blocking writes. If one of them
  is interrupted, run `php artisan migrate` again. Installs that already ran
  them are not affected.
- From 2.26, redrive reuses a child workflow that completed before the failure.
  Releases before 2.26 replay such a successor with a different workflow time
  after the reused child, so keep workers on 2.26 or later once one exists. See
  [Redrive a Failed Run](./failures-and-recovery.md#redrive).
- From 2.26, `historyBudgetAtStep()` and `shouldContinueAsNewAtStep()` read
  the history budget the same way on every replay. Use them in loops. See
  [History Budget](./features/continue-as-new.md#history-budget).
- From 2.25, going over a fixed memo or search attribute cap raises
  `StructuralLimitExceededException` instead of `InvalidArgumentException`,
  and the `memo_size_bytes` default is 65,536. See
  [Structural Limits](./constraints/structural-limits.md).

See the release notes for
[2.25.0](https://github.com/durable-workflow/workflow/releases/tag/2.25.0) and
[2.26.0](https://github.com/durable-workflow/workflow/releases/tag/2.26.0).

Workflow 2.27 keeps the 2.21 support floors and the embedded worker protocol `1.22`, and adds no database migration. Changes to check before upgrading:

- From 2.27, an embedded app repairs and runs work only in its own namespaces. If the app starts, schedules or signals workflows in namespaces other than `DW_V2_NAMESPACE`, list them all in `DW_V2_NAMESPACES`. A process that serves every namespace sets `DW_V2_ALL_NAMESPACES=true`. See [Which namespaces an app runs](./configuration/options.md#namespace-allowlist).
- From 2.27, closing a child costs the parent the same work however many siblings it has. See [Cost of wide fan-out](./features/child-workflows.md#fan-out-cost).

See the [2.27.0 release notes](https://github.com/durable-workflow/workflow/releases/tag/2.27.0).

Workflow 2.28.0 keeps the 2.21 support floors and the embedded worker protocol `1.22`, and adds no database migration. Behavior change to check before upgrading:

- From 2.28, a run that completes, fails or continues as new cancels the activities and timers it never awaited, such as `select()` losers, in the same transaction, instead of leaving them running. A late result from such an activity is refused with `run_closed`. If a workflow needs a loser's effect, await it before returning. See [Losers when the run closes](./features/concurrency.md#selection-losers-at-close).

See the [2.28.0 release notes](https://github.com/durable-workflow/workflow/releases/tag/2.28.0).

## Stability Levels

Every public surface in Durable Workflow carries exactly one of these
stability levels. Levels are explicit; a surface that is not classified is
not public.

| Level | Meaning | When breaking changes are allowed |
|-------|---------|-----------------------------------|
| `frozen` | Wire-format or persisted shape that must decode the same way for the workflow lifetime. Renaming, removing, or repurposing a field is a protocol break, never a minor change. | Only by introducing a **parallel primitive** with a new type name. The original shape stays decodable indefinitely. |
| `stable` | Public surface covered by the platform semver guarantee. Additive changes ship in minor releases. | Major release only. |
| `prerelease` | Public surface that is feature-complete but still allowed to change before the matching `1.0.0` / `2.0.0` cut. | In clearly labelled prerelease versions; called out in release notes. |
| `experimental` | Public-but-unstable surface. May change in any release, including patch releases. Callers must opt in by reading the experimental flag on the surface. | Any release; release notes call out the change. |

## Public Surface Families

This is the complete list of public surface families. Adding, removing, or
re-classifying a family requires a contract change (`SurfaceStabilityContract`
version bump, this page, and the JSON mirror in the same commit).

| Family | Stability | Authority manifest in `/api/cluster/info` | What it covers |
|--------|-----------|-------------------------------------------|----------------|
| `server_api` | `stable` | `control_plane` | Standalone server HTTP API: control-plane routes, namespace routes, schedule routes, system routes, plus `/api/health`, `/api/ready`, `/api/cluster/info`. Per-route version is governed by `control_plane.request_contract` and `control_plane.response.contract`. The top-level server `version` is build identity, not the client compatibility authority. |
| `worker_protocol` | `stable` | `worker_protocol` | Worker-plane HTTP API used by external SDK workers to register, poll, heartbeat, manage worker-session leases, complete, and fail workflow, activity, and query tasks. Includes the `worker_sessions` and `local_activities` runtime contracts, `external_execution_surface_contract`, `external_executor_config_contract`, `invocable_carrier_contract`, `external_task_input_contract`, and `external_task_result_contract`. |
| `cli_json` | `stable` | n/a (see CLI reference) | The `--output=json` and `--output=jsonl` shapes emitted by `dw`. JSON exit codes and JSON field names are the durable surface; the human-readable `--output=table` form is documentation, not contract. |
| `waterline_api` | `stable` | n/a (see Waterline operator API) | Waterline observability HTTP API at `/waterline/api/v2/*`, the engine-source contract, and the dashboard JSON shapes. Waterline must match the workflow package major version. |
| `mcp_discovery_results` | `stable` | n/a (see MCP workflows page) | The `/mcp/*` Model Context Protocol surfaces and the `llms.txt` / `llms-2.0.txt` discovery files. MCP tool names, parameter schemas, and `payload_preview_limit_bytes` semantics are part of the contract; tool descriptions and discovery hints are diagnostic. |
| `official_sdks` | `stable` | `client_compatibility` | The first-party SDKs: PHP `durable-workflow/sdk`, the `durable_workflow` Python SDK, and the `durable-workflow` Rust SDK. The `dw` CLI is the official command client. Each SDK's public surface is governed by its own per-package stability document, which must defer to this page. |
| `history_event_wire_formats` | `frozen` | n/a (frozen shapes; see workflow `docs/api-stability.md`) | The persisted shape of every row in `workflow_history_events` and `workflow_schedule_history_events`. Once a workflow writes an event, every future SDK that replays it must decode the same field set. |
| `cluster_info_manifests` | `stable` | `surface_stability_contract`, `client_compatibility`, `control_plane`, `worker_protocol`, `auth_composition_contract`, `coordination_health` | The protocol manifests published by `GET /api/cluster/info` itself. Each nested manifest carries its own `schema` and `version` and evolves under its own contract rules. The envelope keys are stable. |

### Per-package stability documents

These documents add per-package detail under the rules on this page:

- `durable-workflow/workflow` (PHP) — [`docs/api-stability.md`](https://github.com/durable-workflow/workflow/blob/main/docs/api-stability.md). Authoritative for the PHP authoring API, the `Support\*` server-facing classes, and the frozen history-event wire-format tables.
- `durable-workflow/sdk` (PHP) — [`README.md`](https://github.com/durable-workflow/sdk-php/blob/main/README.md). Authoritative for the framework-neutral remote client and worker API distributed from Packagist.
- `durable-workflow/server` — [`README.md`](https://github.com/durable-workflow/server/blob/main/README.md) and `docs/contracts/*`. Authoritative for the standalone server's request/response contracts.
- `dw` CLI — [`/docs/polyglot/cli-reference`](/docs/polyglot/cli-reference). Authoritative for the JSON output shapes and exit codes.
- Python SDK — `README.md` in `durable-workflow/sdk-python`. Authoritative for the `durable_workflow` package public API.
- Rust SDK — `README.md` and `[package.metadata.durable-workflow]` in
  `durable-workflow/sdk-rust`. Authoritative for the `durable-workflow` crate
  public API and its package compatibility declaration.

## Release Rules

These rules apply to every public surface family above. They are reproduced in
the JSON mirror under `release_rules`.

### Patch releases

Allowed:
- bug fixes that preserve the documented contract
- documentation fixes
- dependency bumps that do not change the public surface
- changes to surfaces marked `experimental`

Forbidden:
- removing or renaming any `stable` or `frozen` field, route, command, or class
- narrowing accepted input on any `stable` route or command
- changing the meaning of an existing `stable` field

### Minor releases

Allowed:
- adding new fields, routes, commands, or classes to a `stable` surface
- adding new optional parameters with safe defaults
- adding new capability flags to discovery responses
- promoting a `prerelease` or `experimental` surface to `stable`

Forbidden:
- removing or renaming any `stable` or `frozen` field, route, command, or class
- changing the meaning of an existing `stable` or `frozen` field

### Major releases

Allowed:
- removing, renaming, or narrowing a `stable` surface
- increasing the required `control_plane.version` or `worker_protocol.version`
- dropping a previously supported SDK or CLI version range

Required:
- announce in release notes at least
  one minor release before cutting the major
- where feasible, ship the new surface alongside the old surface in a
  previous minor release so callers can migrate before the major
- document the migration path on the [migration guide](/docs/migration)
  before publish

## Diagnostic-Only Versus Guaranteed Fields

Every field in every `stable` or `frozen` surface is either **guaranteed** or
**diagnostic-only**. The two have different change rules:

- **Guaranteed fields** are part of the documented contract. Producers must
  keep emitting them in the documented shape; consumers may rely on their
  presence and meaning. Removing or renaming a guaranteed field on a `stable`
  surface is a major change.
- **Diagnostic-only fields** are emitted for human triage, debugging, and
  observability. They may be added, renamed, or removed in any minor
  release. They must be marked `diagnostic_only: true` (or the doc-page
  equivalent) wherever they are documented. **Consumers must not parse,
  persist, or branch on diagnostic fields in production decision logic.**

Unknown additive fields on a `stable` or `frozen` shape must be ignored by
older consumers (forward compatibility). Unknown required fields must fail
closed. SDKs and CLIs publish their own forward-compatibility behavior in
their per-package stability documents.

## Compatibility Matrix

This is the operational compatibility matrix. It records which client
versions are validated against which server protocol manifests. Components
validate the matrix at runtime via `GET /api/cluster/info` and fail closed
when the manifests do not agree.

### Last qualified reproducible tuple

The table shows the reviewed stable combination selected for these docs from
[`stable-releases.json`](/stable-releases.json). Each component is versioned
independently. A compatible combination can therefore contain different
package versions. Release and qualification reports retain the exact versions
and image digests used for each completed check.

| Component | Supported version | Install identity |
|-----------|-------------------|------------------|
| Server | `3.10.0` | `durableworkflow/server:3.10.0` |
| Helm chart | `2.0.0` | `oci://ghcr.io/durable-workflow/charts/durable-workflow --version 2.0.0` |
| CLI | `2.8.0` | `VERSION=2.8.0` |
| Workflow engine | `2.28.1` | `durable-workflow/workflow:2.28.1` |
| Waterline operator | `2.18.0` | `durable-workflow/waterline:2.18.0` |
| PHP SDK | `3.18.2` | `durable-workflow/sdk:3.18.2` |
| Python SDK | `2.17.2` | `durable-workflow==2.17.2` |
| Rust SDK | `4.17.0` | `durable-workflow = "=4.17.0"` |

The selected Server is the native 3.x Server. PHP Server 2.x is frozen at 2.11.8 and receives no further releases; existing PHP installations can move to native with `upgrade-takeover` by following the [native takeover runbook](https://github.com/durable-workflow/server/blob/a051f57fcc7e3605b301581ee9105461b3ef5815/docs/rust-server-upgrade-takeover.md). Takeover is qualified on the published 3.0.4 image within the runbook's stated scope; it is one-way and backup-first.

PHP Server 2.11.8 remains unpatched for the November Laravel advisories. Existing pinned deployments must follow the [conditional exposure and retirement actions](./polyglot/server.md#retired-php-server-security); the historical takeover evidence retains its original limits.

The Helm chart's recorded `appVersion` is `3.10.0`, matching the selected Server, and the chart runs the native Server by default. Its [published release receipt](https://github.com/durable-workflow/server/actions/runs/43487220214) records the chart release run, OCI/package digests, source identity, and a kind install in that run; it is not a production cluster qualification. See [chart selection and upgrades](./deployment.md#helm-chart-selection) before installing or overriding the image.

PyPI renders the Python distribution version as
`2.17.2`; the documented PEP 440 install spelling
`2.17.2` resolves to that same release. This
normalization does not create a second supported version.

The selected combination advances after affected consumers are qualified.
The published Server registries and generated quickstarts use the selected
artifacts. Packages can release independently, with compatibility verified
through their protocol manifests, capabilities and consumer checks.

Earlier alpha and beta artifacts are historical. They are not alternative
onboarding choices, and release history remains intact.

Each stable component follows semantic versioning. Additive capabilities can
ship in compatible releases; breaking public changes require a major release.
Use runtime discovery to check the capabilities available in your installed
Server and workers before enabling a feature.

### Runtime protocol compatibility

Top-level package versions select the supported train. Runtime protocol
manifests provide a second, fail-closed check:

| Client | Product train | Control plane | Default worker registration protocol |
|--------|---------------|---------------|--------------------------------------|
| CLI | `2.8.0` | `2` | n/a |
| PHP SDK | `3.18.2` | `2` | `1.20` |
| Python SDK | `2.17.2` | `2` | `1.20` |
| Rust SDK | `4.17.0` | `2` | `1.20` |

The SDKs also support explicitly selecting worker protocol `1.19` for a
compatible older Server. Cooperative cancellation requires `1.20`; selecting
`1.19` does not provide that capability. Individual operations can use their
documented minimum protocol version, so the registration default is not a
claim that every worker request sends `1.20`.

The current published Server advertises the worker protocol version shown in
the authority-role table above. It accepts request headers from the same major
with a minor less than or equal to the advertised minor, then returns the
advertised version. Missing or malformed headers, different majors, and worker
minors ahead of the server fail closed. The CLI validates
`control_plane.version: "2"`.

The server's top-level `version` is build identity. Clients must use the
`control_plane`, `worker_protocol`, `client_compatibility`, and
`surface_stability_contract` manifests returned by
`GET /api/cluster/info` for protocol negotiation.

Workflow and Waterline must use compatible 2.x releases. Runtime discovery and
package constraints provide the compatibility boundary after the stable cut.

### Runtime validation examples

The SDKs validate discovery before registering a worker. An incompatible
server produces an explicit compatibility error rather than attempting a
legacy prerelease path. All worker requests send
`X-Durable-Workflow-Protocol-Version`; control-plane requests send
`X-Durable-Workflow-Control-Plane-Version: 2`.

Before a product train is promoted, release qualification must start from a
clean machine, install only the published artifacts named above, and complete
the PHP, Python, and Rust conformance paths. Source checkouts and unpublished
substitutions do not count as public-artifact evidence.

Stable package metadata and runtime discovery now define the supported 2.x
compatibility boundary. Release qualification still installs public artifacts
together before publication, while patch and minor releases follow the semantic
versioning rules below.

### Release progression

Patch releases preserve documented stable contracts. Minor releases may add
fields, routes, commands, classes, or optional parameters with safe defaults.
Breaking stable changes require a major release and a documented migration
path. Frozen history-event shapes remain decodable indefinitely; a new shape
uses a parallel primitive rather than mutating an existing event.

Every release must keep package metadata, release notes, installation commands,
and cross-language examples synchronized.

### Release review checklist

- Confirm the machine-readable compatibility contract matches the Workflow
  surface-stability manifest.
- Confirm every selected SDK version passes against the selected Server.
- Confirm installation examples use the stable release channel.
- Confirm package metadata identifies the exact SDK release and protocol
  versions; record later Server qualification in the compatibility evidence.
- Confirm clean-machine published-artifact conformance passes for PHP, Python,
  and Rust.
- Confirm release notes describe post-baseline additions and do not present
  older prereleases as supported choices.

## See Also

- [Server Setup](/docs/polyglot/server) — Deploying the standalone server
- [Server API Reference](/docs/polyglot/server-api-reference) — `GET /api/cluster/info` and the protocol manifests
- [PHP SDK](/docs/polyglot/php) — PHP client and worker
- [Python SDK](/docs/polyglot/python) — Python client and worker
- [Rust SDK](/docs/polyglot/rust) — Rust client and worker
- [CLI](/docs/polyglot/cli) — Command-line interface
- [Migration Guide](/docs/migration) — Migrating from v1 to v2
- [PHP workflow `docs/api-stability.md`](https://github.com/durable-workflow/workflow/blob/main/docs/api-stability.md) — per-package stability for the PHP workflow package

<!-- Source: docs/platform-protocol-specs.md -->

# Platform Protocol Specs

The platform protocol catalog tells SDK authors, agents, operators, and
third-party tooling exactly which machine-readable specification governs each
public Durable Workflow surface.

Use the
[machine-readable catalog](https://durable-workflow.github.io/platform-protocol-specs.json)
for automation. Every available entry provides both a stable specification
identifier and a public HTTPS URL that resolves directly to an OpenAPI,
AsyncAPI, or JSON Schema artifact.

Repository paths, implementation symbols, and test fixtures are intentionally
not consumer authority. They are excluded from the published catalog and kept
only as validation diagnostics.

## Catalog identity and discovery

The catalog is also available through:

- the public JSON catalog at
  https://durable-workflow.github.io/platform-protocol-specs.json;
- platform_protocol_specs in GET /api/cluster/info on the standalone server;
- this 2.0 guide for human-readable context.

The JSON URL is the machine-consumable catalog authority. The server re-exports
the same catalog so clients can discover the active protocol surface without
assuming a repository layout.

## Consumer fields

Each entry exposes the following contract:

| Field | Meaning |
|---|---|
| spec_id | Stable Durable Workflow protocol identifier. Published documents carry the matching identity. |
| spec_url | Direct public HTTPS URL for the machine-readable specification. |
| format | openapi, json_schema, or asyncapi. |
| status | Whether the referenced artifact is published, in progress, or planned. |
| surface_family | Compatibility-policy family that governs the surface. |
| authority_manifest | Discovery manifest through which a runtime advertises the surface. |
| owner_repo | Project responsible for the specification. |
| object_families | Public object-family names and their owning projects. |
| evolution_rule | Compatibility rule for additive and breaking changes. |
| breaking_change_release | Release boundary required by that evolution rule. |

Consumers should resolve spec_url and validate the document identity against
spec_id. They do not need source checkout access.

## Available specifications

This inventory is rendered from the public JSON catalog. Specification
identity, availability, public links, and object-family ownership therefore
remain catalog data rather than a second prose authority.

<!-- durable-workflow-platform-protocol-catalog:start -->
```json
{
  "catalog_schema": "durable-workflow.v2.platform-protocol-specs.catalog",
  "catalog_version": 16,
  "entries": [
    {
      "catalog_entry": "control_plane_api",
      "availability": "available",
      "spec_id": "durable-workflow.v2.control-plane-api",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/control-plane-api.openapi.yaml",
      "owner_repo": "durable-workflow/server",
      "format": "openapi",
      "status": "published",
      "object_families": [
        {
          "name": "control_plane_request_contract",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "control_plane_response_envelope",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "control_plane_operation_contract",
          "owner_repo": "durable-workflow/server"
        }
      ]
    },
    {
      "catalog_entry": "worker_protocol_api",
      "availability": "available",
      "spec_id": "durable-workflow.v2.worker-protocol-api",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/worker-protocol-api.openapi.yaml",
      "owner_repo": "durable-workflow/server",
      "format": "openapi",
      "status": "published",
      "object_families": [
        {
          "name": "worker_registration_request",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "worker_deregistration_result",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "worker_task_poll_request",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "worker_task_result",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "worker_query_task_poll_request",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "worker_query_task_result",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "external_task_input_contract",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "external_task_result_contract",
          "owner_repo": "durable-workflow/server"
        }
      ]
    },
    {
      "catalog_entry": "worker_protocol_stream",
      "availability": "available",
      "spec_id": "durable-workflow.v2.worker-protocol-stream",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/worker-protocol-stream.asyncapi.yaml",
      "owner_repo": "durable-workflow/server",
      "format": "asyncapi",
      "status": "published",
      "object_families": [
        {
          "name": "worker_poll_stream",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "worker_task_lease",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "worker_task_heartbeat",
          "owner_repo": "durable-workflow/server"
        }
      ]
    },
    {
      "catalog_entry": "worker_sessions_runtime",
      "availability": "available",
      "spec_id": "durable-workflow.v2.worker-sessions-runtime",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/worker-sessions-runtime.schema.json",
      "owner_repo": "durable-workflow/server",
      "format": "json_schema",
      "status": "published",
      "object_families": [
        {
          "name": "worker_session_runtime_contract",
          "owner_repo": "durable-workflow/workflow"
        },
        {
          "name": "worker_session_options",
          "owner_repo": "durable-workflow/workflow"
        },
        {
          "name": "worker_session_lifecycle",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "worker_session_visibility",
          "owner_repo": "durable-workflow/server"
        }
      ]
    },
    {
      "catalog_entry": "local_activity_runtime",
      "availability": "available",
      "spec_id": "durable-workflow.v2.local-activity-runtime",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/local-activity-runtime.schema.json",
      "owner_repo": "durable-workflow/workflow",
      "format": "json_schema",
      "status": "published",
      "object_families": [
        {
          "name": "local_activity_runtime_contract",
          "owner_repo": "durable-workflow/workflow"
        },
        {
          "name": "local_activity_options",
          "owner_repo": "durable-workflow/workflow"
        },
        {
          "name": "local_activity_history_markers",
          "owner_repo": "durable-workflow/workflow"
        },
        {
          "name": "local_activity_visibility",
          "owner_repo": "durable-workflow/workflow"
        }
      ]
    },
    {
      "catalog_entry": "history_event_payloads",
      "availability": "available",
      "spec_id": "durable-workflow.v2.history-event-payloads",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/history-event-payloads.schema.json",
      "owner_repo": "durable-workflow/workflow",
      "format": "json_schema",
      "status": "published",
      "object_families": [
        {
          "name": "workflow_history_events",
          "owner_repo": "durable-workflow/workflow"
        },
        {
          "name": "workflow_schedule_history_events",
          "owner_repo": "durable-workflow/workflow"
        }
      ]
    },
    {
      "catalog_entry": "history_export_bundle",
      "availability": "available",
      "spec_id": "durable-workflow.v2.history-export-bundle",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/history-export-bundle.schema.json",
      "owner_repo": "durable-workflow/workflow",
      "format": "json_schema",
      "status": "published",
      "object_families": [
        {
          "name": "history_export_bundle",
          "owner_repo": "durable-workflow/workflow"
        }
      ]
    },
    {
      "catalog_entry": "replay_bundle",
      "availability": "available",
      "spec_id": "durable-workflow.v2.replay-bundle",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/replay-bundle.schema.json",
      "owner_repo": "durable-workflow/workflow",
      "format": "json_schema",
      "status": "published",
      "object_families": [
        {
          "name": "replay_bundle",
          "owner_repo": "durable-workflow/workflow"
        }
      ]
    },
    {
      "catalog_entry": "waterline_read_api",
      "availability": "available",
      "spec_id": "durable-workflow.v2.waterline-read-api",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/waterline-read-api.openapi.yaml",
      "owner_repo": "durable-workflow/waterline",
      "format": "openapi",
      "status": "published",
      "object_families": [
        {
          "name": "waterline_read_envelope",
          "owner_repo": "durable-workflow/waterline"
        },
        {
          "name": "waterline_operator_action_envelope",
          "owner_repo": "durable-workflow/waterline"
        }
      ]
    },
    {
      "catalog_entry": "waterline_diagnostic_objects",
      "availability": "available",
      "spec_id": "durable-workflow.v2.waterline-diagnostic-objects",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/waterline-diagnostic-objects.schema.json",
      "owner_repo": "durable-workflow/waterline",
      "format": "json_schema",
      "status": "published",
      "object_families": [
        {
          "name": "waterline_run_detail",
          "owner_repo": "durable-workflow/waterline"
        },
        {
          "name": "waterline_health_diagnostics",
          "owner_repo": "durable-workflow/waterline"
        },
        {
          "name": "waterline_timeline_rows",
          "owner_repo": "durable-workflow/waterline"
        },
        {
          "name": "waterline_lineage_edges",
          "owner_repo": "durable-workflow/waterline"
        }
      ]
    },
    {
      "catalog_entry": "repair_actionability_objects",
      "availability": "available",
      "spec_id": "durable-workflow.v2.repair-actionability-objects",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/repair-actionability-objects.schema.json",
      "owner_repo": "durable-workflow/workflow",
      "format": "json_schema",
      "status": "published",
      "object_families": [
        {
          "name": "task_repair_policy",
          "owner_repo": "durable-workflow/workflow"
        },
        {
          "name": "task_repair_candidates",
          "owner_repo": "durable-workflow/workflow"
        },
        {
          "name": "operator_queue_visibility",
          "owner_repo": "durable-workflow/workflow"
        },
        {
          "name": "actionability",
          "owner_repo": "durable-workflow/waterline"
        },
        {
          "name": "agent_root_cause",
          "owner_repo": "durable-workflow/durable-workflow.github.io"
        },
        {
          "name": "agent_remediation",
          "owner_repo": "durable-workflow/durable-workflow.github.io"
        },
        {
          "name": "safe_mutation",
          "owner_repo": "durable-workflow/durable-workflow.github.io"
        }
      ]
    },
    {
      "catalog_entry": "cli_json_envelopes",
      "availability": "available",
      "spec_id": "durable-workflow.v2.cli-json-envelopes",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/cli-json-envelopes.schema.json",
      "owner_repo": "durable-workflow/cli",
      "format": "json_schema",
      "status": "published",
      "object_families": [
        {
          "name": "cli_output_schema_manifest",
          "owner_repo": "durable-workflow/cli"
        },
        {
          "name": "cli_command_output_schema",
          "owner_repo": "durable-workflow/cli"
        }
      ]
    },
    {
      "catalog_entry": "mcp_discovery",
      "availability": "available",
      "spec_id": "durable-workflow.v2.mcp-discovery",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/mcp-discovery.schema.json",
      "owner_repo": "durable-workflow/durable-workflow.github.io",
      "format": "json_schema",
      "status": "published",
      "object_families": [
        {
          "name": "mcp_tool_discovery",
          "owner_repo": "durable-workflow/durable-workflow.github.io"
        },
        {
          "name": "llms_txt_discovery",
          "owner_repo": "durable-workflow/durable-workflow.github.io"
        }
      ]
    },
    {
      "catalog_entry": "mcp_tool_results",
      "availability": "available",
      "spec_id": "durable-workflow.v2.mcp-tool-results",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/mcp-tool-results.schema.json",
      "owner_repo": "durable-workflow/durable-workflow.github.io",
      "format": "json_schema",
      "status": "published",
      "object_families": [
        {
          "name": "mcp_tool_result_envelope",
          "owner_repo": "durable-workflow/durable-workflow.github.io"
        },
        {
          "name": "agent_root_cause",
          "owner_repo": "durable-workflow/durable-workflow.github.io"
        },
        {
          "name": "agent_remediation",
          "owner_repo": "durable-workflow/durable-workflow.github.io"
        },
        {
          "name": "safe_mutation",
          "owner_repo": "durable-workflow/durable-workflow.github.io"
        }
      ]
    },
    {
      "catalog_entry": "cluster_info_envelope",
      "availability": "available",
      "spec_id": "durable-workflow.v2.cluster-info-envelope",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/cluster-info-envelope.schema.json",
      "owner_repo": "durable-workflow/server",
      "format": "json_schema",
      "status": "published",
      "object_families": [
        {
          "name": "cluster_info_envelope",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "client_compatibility_manifest",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "surface_stability_contract",
          "owner_repo": "durable-workflow/workflow"
        },
        {
          "name": "platform_protocol_specs_catalog",
          "owner_repo": "durable-workflow/workflow"
        },
        {
          "name": "platform_conformance_suite_manifest",
          "owner_repo": "durable-workflow/workflow"
        },
        {
          "name": "sdk_neutrality_contract",
          "owner_repo": "durable-workflow/workflow"
        }
      ]
    },
    {
      "catalog_entry": "invocable_carrier_execution",
      "availability": "available",
      "spec_id": "durable-workflow.v2.invocable-carrier-execution",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/invocable-carrier-execution.schema.json",
      "owner_repo": "durable-workflow/server",
      "format": "json_schema",
      "status": "in_progress",
      "object_families": [
        {
          "name": "invocable_carrier_contract",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "external_execution_surface_contract",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "external_executor_config_contract",
          "owner_repo": "durable-workflow/server"
        }
      ]
    }
  ]
}
```
<!-- durable-workflow-platform-protocol-catalog:end -->

### Worker session runtime notes

This schema covers worker-session capability discovery, lifecycle envelopes,
task-affinity snapshots, and operator visibility. Its stable identifier is
durable-workflow.v2.worker-sessions-runtime.

### Local activity runtime notes

This schema covers local-activity capability discovery, option snapshots,
history markers, retry semantics, and operator visibility. Its stable
identifier is durable-workflow.v2.local-activity-runtime.

### Cluster-info envelope notes

This schema covers GET /api/cluster/info and the nested discovery manifests
available from that endpoint. Its stable identifier is
durable-workflow.v2.cluster-info-envelope.

## Formats

| Format | Use |
|---|---|
| OpenAPI 3.1 | HTTP and JSON request-response surfaces whose routes, methods, status codes, and envelopes are part of the contract. |
| JSON Schema 2020-12 | Persisted records, event payloads, result envelopes, runtime options, and related object families. |
| AsyncAPI 2.6 or newer | Poll, stream, lease-renewal, ordering, and delivery semantics. |

## Status levels

| Status | Meaning |
|---|---|
| published | The public machine-readable specification is available at spec_url and can be consumed directly. |
| in_progress | A public specification is available, but its coverage is partial. Listed fields and routes are normative. |
| planned | The entry has a stable catalog identity but no consumable specification yet. Planned entries do not advertise spec_url. |

## Evolution rules

additive_minor_breaking_major allows additive changes in minor releases.
Removals, renames, type narrowings, and semantic changes require a major
release and should use a parallel route or field where practical.

parallel_primitive_only governs frozen wire formats. A breaking shape must be
introduced under a new event type, command type, or schema identifier while
the original remains decodable.

experimental_any_release applies only to explicitly experimental
specifications and allows change in any release.

## Release validation

Docs-site CI enforces the following machine checks:

| Gate | What CI checks |
|---|---|
| catalog_aligned_with_surface_families | Every entry references a declared public compatibility family. |
| owner_repo_known | Entry and object-family owners use the catalog vocabulary. |
| format_known | Every available artifact parses as its declared format. |
| public_spec_references_resolve | Every available spec_url is an HTTPS URL in the public protocol-spec namespace and resolves to a shipped artifact whose identity matches spec_id. |
| repository_local_authority_fields_rejected | Published entries contain no repository-local paths, implementation symbols, test references, or legacy authority fields. |
| workflow_package_mirror_aligned | The public catalog matches the packaged Workflow catalog when that release input is available. |
| server_owned_spec_mirrors_aligned | Server-owned published artifacts match owner-repository mirrors when those inputs are available. |
| diagnostic_provenance_complete | Validation-only provenance covers every catalog entry and object family. |
| object_family_metadata_declared | Catalog entries and published documents agree on object-family names and owners. |
| rendered_retrieval_surfaces_aligned | The built 2.0 page and full 2.0 model-retrieval bundles expose every available entry's catalog identity, public URL, and object-family ownership. |
| breaking_change_release_consistent_with_evolution_rule | Every breaking-change release value matches its evolution rule. |
| deliverable_specs_published | Every required platform surface has a published, parseable specification. |

The machine check loads the public JSON catalog, validates its vocabulary and
consumer-safe references, parses shipped specifications, checks object-family
and embedded-schema metadata, and compares package mirrors when available.
After the site and model bundles are generated, a semantic check compares their
rendered catalog values to that same JSON authority. Documentation prose and
heading text are not part of the comparison.

When the contract changes, update the packaged Workflow catalog, this public
JSON mirror, and affected published specification artifacts together. Human
review confirms that explanatory prose remains useful without treating it as
machine authority.

Continue to the [Platform Conformance Suite](/docs/platform-conformance)
to resolve the active, byte-bound fixtures that exercise these protocol
specifications. Historical fixture evidence is identified separately in that
suite and does not replace a current catalog authority.

<!-- Source: docs/platform-conformance.md -->

# Platform Conformance Suite

This page is the public authority for the Durable Workflow **platform
conformance suite**. It defines the conformance target matrix, reusable
fixture catalog, harness contract, pass / fail rules, and release gates
for implementations that claim Durable Workflow v2 compatibility.

The same manifest is advertised by the standalone server from
`GET /api/cluster/info` under `platform_conformance_suite`. The
[Platform Protocol Specs](/docs/platform-protocol-specs) catalog names
that nested manifest as the `platform_conformance_suite_manifest`
object family in the `cluster_info_envelope` spec.

The suite is downstream of the
[Version Compatibility](/docs/compatibility) authority. Where the suite
enumerates a surface family or stability rule, it must match the
surface-stability contract. The compatibility authority defines what
the contract is; this page defines how an implementation proves it
follows that contract.

## Public Conformance Authorities

The suite registers stable machine-readable authorities by schema identity
and public URL. The framework-neutral PHP SDK authority is published as
[`durable-workflow.v2.php-sdk-conformance-contract`](pathname:///platform-conformance/php-sdk-conformance.json).
It declares the released-package topology, scenario and evidence requirements,
and public runner and result-schema identifiers used to evaluate
`durable-workflow/sdk` against the released standalone server.

## Target Matrix

A conformance **target** is a kind of implementation that can claim
Durable compatibility. An implementation may claim more than one target.
For example, the standalone server claims `standalone_server`,
`worker_protocol_implementation`, and `repair_actionability_surface`.

| Target | Required surface families | Required fixture categories |
| --- | --- | --- |
| `standalone_server` | `server_api`, `worker_protocol`, `cluster_info_manifests` | `control_plane_request_response`, `signal_query_runtime_contract`, `workflow_update_runtime_contract`, `search_attribute_runtime_contract`, `schedules_runtime_contract`, `namespace_runtime_contract`, `child_workflow_runtime_contract`, `saga_runtime_contract`, `worker_versioning_runtime_contract`, `migration_runtime_contract`, `skew_refusal_matrix_contract`, `principal_attribution_contract`, `worker_task_lifecycle`, `failure_repair_actionability` |
| `embedded_engine` | `history_event_wire_formats` | `history_replay_bundles` |
| `official_sdk` | `official_sdks`, `worker_protocol`, `history_event_wire_formats` | `control_plane_request_response`, `signal_query_runtime_contract`, `workflow_update_runtime_contract`, `search_attribute_runtime_contract`, `schedules_runtime_contract`, `namespace_runtime_contract`, `child_workflow_runtime_contract`, `saga_runtime_contract`, `worker_versioning_runtime_contract`, `migration_runtime_contract`, `skew_refusal_matrix_contract`, `principal_attribution_contract`, `worker_task_lifecycle`, `history_replay_bundles` |
| `worker_protocol_implementation` | `worker_protocol`, `history_event_wire_formats` | `worker_task_lifecycle`, `signal_query_runtime_contract`, `workflow_update_runtime_contract`, `search_attribute_runtime_contract`, `schedules_runtime_contract`, `namespace_runtime_contract`, `child_workflow_runtime_contract`, `saga_runtime_contract`, `worker_versioning_runtime_contract`, `migration_runtime_contract`, `skew_refusal_matrix_contract`, `history_replay_bundles` |
| `cli_json_client` | `cli_json` | `control_plane_request_response`, `signal_query_runtime_contract`, `workflow_update_runtime_contract`, `search_attribute_runtime_contract`, `schedules_runtime_contract`, `namespace_runtime_contract`, `child_workflow_runtime_contract`, `saga_runtime_contract`, `worker_versioning_runtime_contract`, `migration_runtime_contract`, `skew_refusal_matrix_contract`, `principal_attribution_contract`, `cli_json_envelopes` |
| `waterline_contract_surface` | `waterline_api` | `signal_query_runtime_contract`, `workflow_update_runtime_contract`, `search_attribute_runtime_contract`, `namespace_runtime_contract`, `saga_runtime_contract`, `worker_versioning_runtime_contract`, `migration_runtime_contract`, `skew_refusal_matrix_contract`, `principal_attribution_contract`, `waterline_observer_envelopes` |
| `repair_actionability_surface` | `worker_protocol`, `server_api` | `failure_repair_actionability` |
| `mcp_discovery_surface` | `mcp_discovery_results` | `mcp_discovery_envelopes` |
| `prerelease_release_candidate` | `server_api`, `official_sdks`, `cli_json`, `waterline_api`, `cluster_info_manifests` | `skew_refusal_matrix_contract`, `workflow_update_runtime_contract`, `principal_attribution_contract`, `prerelease_readiness_contract` |

Targets are stable. Adding a target, adding a required surface to an
existing target, adding a required fixture category, promoting a
provisional category to required, changing stable runtime scenario
`operations` or `pass_criteria`, changing a stable runtime scenario public
requirement field (`artifact_policy`, `common_result_evidence`,
`required_matrix`, `scenario_requirements`, or `host_runner_contract`), or
changing a pass / fail rule is a suite contract change and must advance the
manifest version.

## Fixture Catalog

Each stable source is a public artifact with three machine-readable bindings in
the suite catalog: `artifact_id`, `resolver_url`, and `sha256`. Runtime artifact
ids end in the suite version. Protocol artifact ids end in the platform
protocol catalog version. A consumer resolves an artifact by fetching its
`resolver_url` and accepting the bytes only when their SHA-256 digest matches
the catalog binding.

| Category | Status | Consumer-resolvable authority |
| --- | --- | --- |
| `control_plane_request_response` | `stable` | [`durable-workflow.v2.control-plane-api@catalog-16`](https://raw.githubusercontent.com/durable-workflow/durable-workflow.github.io/f781ced1ae33c8697835bd527a125bdf3eaf4321/static/platform-protocol-specs/control-plane-api.openapi.yaml) |
| `worker_task_lifecycle` | `stable` | [`durable-workflow.v2.worker-protocol-api@catalog-16`](https://durable-workflow.github.io/platform-protocol-specs/v1.19/worker-protocol-api.openapi.yaml) |
| `worker_task_lifecycle` | `stable` | [`durable-workflow.v2.worker-protocol-stream@catalog-16`](https://durable-workflow.github.io/platform-protocol-specs/v1.19/worker-protocol-stream.asyncapi.yaml) |
| `signal_query_runtime_contract` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/signal_query_runtime_contract@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/signal-query-runtime-scenarios.json) |
| `workflow_update_runtime_contract` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/workflow_update_runtime_contract@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/workflow-update-runtime-scenarios.json) |
| `search_attribute_runtime_contract` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/search_attribute_runtime_contract@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/search-attribute-runtime-scenarios.json) |
| `schedules_runtime_contract` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/schedules_runtime_contract@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/schedules-runtime-scenarios.json) |
| `history_replay_bundles` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/history_replay_bundles@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/replay-runtime-scenarios.json) |
| `history_replay_bundles` | `stable` | [`durable-workflow.v2.history-event-payloads@catalog-16`](https://raw.githubusercontent.com/durable-workflow/durable-workflow.github.io/f781ced1ae33c8697835bd527a125bdf3eaf4321/static/platform-protocol-specs/history-event-payloads.schema.json) |
| `history_replay_bundles` | `stable` | [`durable-workflow.v2.replay-bundle@catalog-16`](https://raw.githubusercontent.com/durable-workflow/durable-workflow.github.io/f781ced1ae33c8697835bd527a125bdf3eaf4321/static/platform-protocol-specs/replay-bundle.schema.json) |
| `namespace_runtime_contract` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/namespace_runtime_contract@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/namespace-runtime-scenarios.json) |
| `child_workflow_runtime_contract` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/child_workflow_runtime_contract@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/child-workflow-runtime-scenarios.json) |
| `worker_versioning_runtime_contract` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/worker_versioning_runtime_contract@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/worker-versioning-runtime-scenarios.json) |
| `saga_runtime_contract` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/saga_runtime_contract@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/saga-runtime-scenarios.json) |
| `migration_runtime_contract` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/migration_runtime_contract@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/migration-runtime-scenarios.json) |
| `skew_refusal_matrix_contract` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/skew_refusal_matrix_contract@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/skew-refusal-matrix-scenarios.json) |
| `prerelease_readiness_contract` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/prerelease_readiness_contract@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/prerelease-readiness-scenarios.json) |
| `failure_repair_actionability` | `stable` | [`durable-workflow.v2.repair-actionability-objects@catalog-16`](https://raw.githubusercontent.com/durable-workflow/durable-workflow.github.io/f781ced1ae33c8697835bd527a125bdf3eaf4321/static/platform-protocol-specs/repair-actionability-objects.schema.json) |
| `cli_json_envelopes` | `stable` | [`durable-workflow.cli.output-schema-manifest@3`](https://durable-workflow.github.io/cli-json-envelopes/v3/manifest.json) |
| `principal_attribution_contract` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/principal_attribution_contract@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/principal-attribution-scenarios.json) |

The suite row retains CLI schema revision v3 as immutable historical evidence.
The current CLI contract is the
[`durable-workflow.cli.output-schema-manifest@4`](https://durable-workflow.github.io/cli-json-envelopes/v4/manifest.json)
closure, which publishes the Avro-only payload fields without rewriting v2 or
v3 bytes.

The current conformance-suite worker-protocol target uses the lifecycle-neutral
protocol 1.19 bytes recorded by
`durable-workflow.v2.worker-protocol-api@catalog-16` and
`durable-workflow.v2.worker-protocol-stream@catalog-16`. Protocol revisions
1.15 through 1.18 remain available at the versioned resolver URLs retained by
the suite history. These versioned fixtures do not replace the unversioned
current published Server protocol authority. The former beta-worded bytes
remain available only as the
explicitly historical
[`durable-workflow.v2.worker-protocol-api@catalog-16-beta-history`](https://raw.githubusercontent.com/durable-workflow/durable-workflow.github.io/e990bc36731463cc5b2cb2a9175dbccfdea61704/static/platform-protocol-specs/worker-protocol-api.openapi.yaml)
binding recorded in the suite manifest.

The planned `waterline_observer_envelopes` and `mcp_discovery_envelopes`
categories remain `provisional`. Their planned source-tree locations are
non-normative placeholders: harnesses must not resolve them as fixtures or use
them for a stable conformance claim.

## Workflow Lifecycle Release Authority

The exact released workflow-lifecycle scenario authority is
[published as JSON](https://durable-workflow.github.io/platform-conformance/workflow-lifecycle-scenarios.json).
It records the lifecycle requirements exercised at the current published
server, PHP SDK, and Rust SDK release boundary.

The PHP shard installs the exact `durable-workflow/sdk` package named by the
current artifact tuple from Packagist into a disposable Composer project. It
runs separate PHP client and worker processes against the matching public
server image, records the Packagist distribution and official `apache/avro`
provenance, and must report
`local_product_source_checkouts_used=false`.

The Rust shard installs the exact crate named by the current artifact tuple
from crates.io and records the registry source and checksum for that crate and
the official `apache-avro` crate. The payload proof uses the SDK's published
Avro envelope backed by `apache-avro`; a custom codec implementation or local
product checkout is not acceptable provenance.

The Rust shard must execute and report all of these cells:

- `instance_cancel` and `instance_terminate` through the public SDK commands.
- `selected_run_guard` and `stale_run_rejection` so a selected run cannot be
  confused with the instance's current run.
- `typed_failed`, `typed_cancelled`, `typed_terminated`, and `typed_timed_out`
  as typed terminal outcomes carrying workflow and run identity.
- `cancellation_heartbeat` and `late_activity_completion_refused` so activity
  code observes cancellation and a late completion cannot overwrite the
  terminal result.
- `worker_restart_during_cancellation` while cancellation settlement is still
  pending, proving that a replacement worker does not reclaim the closed
  activity.

The manifest also requires the exact artifact and server versions, cluster
identity, install provenance, workflow identities, per-cell outcomes, stable
reasons, payload contract, executor topology, Rust shard contract version,
runner identity, and shard exit status. Missing, unsupported, runner-blocked,
or unsuccessful Rust evidence cannot satisfy the lifecycle category.

A fixture category is required for a target only when the target lists
it and the category status is not `provisional`. Provisional categories
emit advisory warnings and become load-bearing only when promoted to
`stable` in a later suite version.

The `signal_query_runtime_contract` category is stable. A result for it
must record concrete pinned published artifact versions and must name
every required scenario as `pass`, `fail`, `unsupported`, `not_covered`,
or `runner_blocked` with linked findings. Placeholder or unresolved
version tokens such as `latest`, `current`, `head`, `<latest>`,
`${VERSION}`, or `{{ version }}` fail the result gate. Only `pass` cells
count toward a passing category:
`published_artifact_install_only`,
`python_worker_cli_and_sdk_baseline`, `php_worker_cli_and_sdk_baseline`,
`python_worker_php_facing_and_cli_clients`,
`php_worker_python_and_cli_clients`,
`rust_worker_rust_php_python_clients`, `python_worker_rust_client`,
`php_worker_rust_client`, `rust_query_error_and_immutability`,
`ordered_signal_delivery`,
`dedup_contract_observation`, `signal_during_replay`,
`query_during_replay`,
`rust_replayed_instance_state_query_after_cold_restart`,
`completed_run_signal_and_query`,
`unknown_signal_and_query_errors`,
`malformed_signal_and_query_payloads`, and
`waterline_operator_visibility`.

Those scenario ids and their pass criteria are published as the
machine-readable runtime scenario manifest at
[`static/platform-conformance/signal-query-runtime-scenarios.json`](pathname:///platform-conformance/signal-query-runtime-scenarios.json).
Implementation tests may exercise the scenarios, but they are not stable
fixture sources for external harnesses.

The Rust cells install the exact `durable-workflow` crate version declared by
the runtime scenario manifest from crates.io and record the Cargo registry
source and checksum for both the SDK and its resolved `apache-avro` dependency.
Snapshot-derived query transport is graded by
`rust_worker_rust_php_python_clients` and
`rust_query_error_and_immutability`. It is not replayed workflow-instance
state. The separate
`rust_replayed_instance_state_query_after_cold_restart` cell uses
`register_replayed_workflow` and `register_replayed_query`, starts a fresh Rust
worker process after a cold stop, restores durable history, and compares
running, restored, and completed state through Rust, PHP, and Python callers.
Both successful and failed query sequences capture history and workflow-command
counts before the first successful measured query and must leave those counts
unchanged.

For `completed_run_signal_and_query`, a completed cleanly run with a
replayable declared query handler must return its final query state
through every claimed public query surface. Stable terminal-state errors
are valid only for explicitly unsupported terminal states or unavailable
handlers; a generic completed-run terminal error is not a passing result
for a replayable completed run.

The `workflow_update_runtime_contract` category is a stable runtime
scenario category. A result for it must use published artifacts, pin the
server, CLI, Python SDK, PHP SDK, and Waterline versions,
state whether any local product source checkouts were used, and name
every required update scenario as `pass`, `fail`, `unsupported`,
`not_covered`, or `runner_blocked` with linked findings. Passing update
evidence must exercise public control-plane, history, CLI/SDK, and
operator-readable surfaces; local source execution does not count.

Required workflow update scenarios cover published-artifact install,
declared update contract visibility, accepted update control-plane and
history evidence, running or waiting operator visibility, completed
result round trip, failed or refused outcomes, duplicate request or
idempotency behavior, unknown update refusal, invalid input refusal,
payload envelope round trip, terminal-workflow behavior, authenticated
principal attribution, PHP client/worker parity, Python client/worker
parity, and operator diagnostics. If an SDK lacks first-class update
support, the result must report a typed unsupported cell with a focused
SDK finding instead of silently omitting the language.

Those scenario ids and their pass criteria are published as the
machine-readable runtime scenario manifest at
[`static/platform-conformance/workflow-update-runtime-scenarios.json`](pathname:///platform-conformance/workflow-update-runtime-scenarios.json).
The current host-runner handoff is intentionally `runner_blocked` until a
registered host runner can install the pinned published artifacts and
drive the full workflow-update matrix.

The `search_attribute_runtime_contract` category is a stable runtime
scenario category. A result for it must use published artifacts, cover
PHP and Python workflow start/upsert behavior, CLI query and error
surfaces, Waterline operator visibility, cross-language codec round
trips, equality/range/bool queries, OR/NOT grammar, keyword-list
membership, type safety, indexing latency distribution, load latency,
namespace isolation, reserved-name refusal, and query injection
hardening. A Python/server smoke subset is nonconforming until every
required cell is recorded as `pass`, `fail`, `unsupported`,
`not_covered`, or `runner_blocked` with linked findings. Only `pass`
cells count toward a passing category.

Those search-attribute scenario ids and their pass criteria are
published at
[`static/platform-conformance/search-attribute-runtime-scenarios.json`](pathname:///platform-conformance/search-attribute-runtime-scenarios.json).

The `schedules_runtime_contract` category is a stable runtime scenario
category. A result for it must use published artifacts and cover cron
cadence, fixed-rate cadence, list and describe visibility, pause/resume
windows with no fires, delete stopping future fires, missed-fire policy,
restart survival, CLI schedule operations, Python SDK schedule
operations, PHP-facing schedule operations, Python-created PHP workflow
fires, PHP-created Python workflow fires, invalid cron refusal, and
nonexistent workflow type outcomes. A schedule lifecycle smoke subset is
nonconforming until every required cell is recorded as `pass`, `fail`,
`unsupported`, `not_covered`, or `runner_blocked` with linked findings.
Only `pass` cells count toward a passing category.

Those schedules scenario ids and their pass criteria are published at
[`static/platform-conformance/schedules-runtime-scenarios.json`](pathname:///platform-conformance/schedules-runtime-scenarios.json).

The `history_replay_bundles` category is also a stable runtime scenario
category. A result for it must use published artifacts, cover PHP and
Python replay for completed histories, worker restart, activity,
signal/update, wait-condition, version-marker, saga-compensation,
explicit code-divergence refusal, server-side history mutation refusal,
malformed history refusal, and in-flight signal restart timing. A
golden-history smoke subset is nonconforming until every required cell is
recorded as `pass`, `fail`, `unsupported`, `not_covered`, or
`runner_blocked` with linked findings. Only `pass` cells count toward a
passing category.

Those replay scenario ids and their pass criteria are published at
[`static/platform-conformance/replay-runtime-scenarios.json`](pathname:///platform-conformance/replay-runtime-scenarios.json).

The `namespace_runtime_contract` category is a stable runtime scenario
category. A result for it must use published artifacts, cover namespace
create/update/describe/list, lifecycle cleanup and recreate, workflow
visibility and mutation isolation, PHP worker task-queue isolation, CLI
namespace context and default-scope behavior, SDK namespace selection
parity, search-attribute schema and value query isolation, schedule
isolation, Waterline/operator scoped visibility, explicit Nexus
cross-namespace calls, reserved-name refusal, and result-record routing
for product findings. A namespace smoke subset is nonconforming until
every required cell is recorded as `pass`, `fail`, `unsupported`,
`not_covered`, or `runner_blocked` with linked findings. Only `pass`
cells count toward a passing category.

Non-normative implementation notes for Waterline/operator scoped
visibility can come from the published Waterline artifact or a focused
Waterline shard. Useful reviewer captures include scoped workflow list
and detail views, schedule views when the product exposes them,
search-attribute values, dashboard scope, operator API stats, and the
documented verdict for any default or unscoped view advertised by the
product; the normative pass criteria remain the
`waterline_operator_namespace_visibility` scenario below.

Those namespace scenario ids and their pass criteria are published at
[`static/platform-conformance/namespace-runtime-scenarios.json`](pathname:///platform-conformance/namespace-runtime-scenarios.json).

The `child_workflow_runtime_contract` category is a stable runtime
scenario category. A result for it must use published artifacts and cover
same-language PHP and Python parent/child runs, PHP-to-Python and
Python-to-PHP parent/child runs, child failure round-trip typing, parent
cancellation propagation to a child, direct child cancellation observed
by a parent, replay across parent worker restart while waiting on a
child, concurrent fan-out to five children, and namespace behavior. A
single parent/child smoke is nonconforming until every required cell is
recorded as `pass`, `fail`, `unsupported`, `not_covered`, or
`runner_blocked` with linked findings. Only `pass` cells count toward a
passing category.

Those child-workflow scenario ids and their pass criteria are published
at
[`static/platform-conformance/child-workflow-runtime-scenarios.json`](pathname:///platform-conformance/child-workflow-runtime-scenarios.json).

The `saga_runtime_contract` category is a stable runtime scenario
category. A result for it must use published artifacts and cover forward
success, failure after a later step with reverse-order compensation,
early-step failure with no extra compensation, compensation retry
idempotence, compensation-failure terminal visibility, mid-compensation
worker restart, PHP workflow to Python compensation, Python workflow to
PHP compensation, typed compensation error round trips, and
operator-visible in-progress compensation status. A saga smoke that only
proves one happy path or one SDK is nonconforming until every required
cell is recorded as `pass`, `fail`, `unsupported`, `not_covered`, or
`runner_blocked` with linked findings. Only `pass` cells count toward a
passing category. The PHP package is identified as `workflow-php` in the
runtime matrix and may also be recorded as `workflow` when comparing
against the platform release artifact set.

Those saga scenario ids and their pass criteria are published at
[`static/platform-conformance/saga-runtime-scenarios.json`](pathname:///platform-conformance/saga-runtime-scenarios.json).

The `worker_versioning_runtime_contract` category is a stable runtime
scenario category. A result for it must use published artifacts and
cover worker build-ID registration, operator rollout visibility,
drain/resume controls, per-run compatibility pins, replay only by
compatible workers, promoted-version routing for new starts, replay
after cache eviction, no-compatible-worker diagnostics, CLI and
Waterline visibility surfaces, PHP/Python cross-language pinning,
adversarial no-version-bump behavior, and history API version pins. A
worker-versioning smoke subset is nonconforming until every required
cell is recorded as `pass`, `fail`, `unsupported`, `not_covered`, or
`runner_blocked` with linked findings. Only `pass` cells count toward a
passing category.

Cross-language PHP/Python pinning evidence must record worker runtime
identities, workflow and run IDs, task-queue rollout state, the public
poll and rollout outcomes used to verify pinning, published worker
artifact install source and version, and confirmation that no local
product source checkout was used.

Those worker-versioning scenario ids and their pass criteria are
published at
[`static/platform-conformance/worker-versioning-runtime-scenarios.json`](pathname:///platform-conformance/worker-versioning-runtime-scenarios.json).

The `migration_runtime_contract` category is a stable runtime scenario
category. A result for it must use published artifacts and cover the
latest supported v1 state setup, the documented migration steps,
completed-history preservation and replay, in-flight workflow progress,
mid-activity retry state, signal-or-timer wait state, schedule cadence, worker registration
projection, Waterline operator visibility, CLI access to preupgrade
state, new v2 workflow starts, queue-aware rollback semantics, and loud
refusal for unsupported version skew. Rollback evidence must inventory ready,
delayed, and reserved v1 queue work; record whether the recovery boundary was
drained, captured as an application-consistent SQL-plus-queue cut, or accepted
as unrecoverable; and prove that each restored nonterminal v1 row has a
runnable queue or signal wake path. An eligible stale `pending` row may instead
use the supported v1.0.77 Watchdog only when evidence observes the worker loop
dispatching the enabled-by-default Watchdog, the Watchdog redispatching that
workflow after its five-minute stale bound, and the row advancing on its
recorded queue. That pending-only path does not replace preserved queue state
for retries, timers, or `waiting`/`running` work. SQL-only restore is passing
rollback evidence only when no other nonterminal queue-dependent v1 execution
exists at the recovery cut. The recovery manifest must record only the
`APP_KEY` secret-manager reference and version, with the key and recovery
credentials kept separately access-controlled from SQL and queue backups. A
fresh-install smoke or a migration run that
does not start from realistic v1 state is nonconforming until every
required cell is recorded as `pass`, `fail`, `unsupported`,
`not_covered`, or `runner_blocked` with linked findings. Only `pass`
cells count toward a passing category.

The documented-steps cell must record the live guide command list, the
commands actually executed in guide order, exit codes, and per-command
timings. Before and after state snapshots must include observed
completed-history, in-flight workflow, retrying-activity, signal-or-timer
wait, schedule, and worker-registration cells.

Those migration scenario ids and their pass criteria are published at
[`static/platform-conformance/migration-runtime-scenarios.json`](pathname:///platform-conformance/migration-runtime-scenarios.json).

The `skew_refusal_matrix_contract` category is a stable runtime scenario
category. A result for it must use published artifacts and cover
compatible, backward-skewed, forward-skewed, and outside-window pairings
for CLI, Python SDK, the standalone PHP SDK worker, and Waterline surfaces.
Workflow is present only as the embedded Laravel and Waterline engine. It
must also probe future-version boundaries, capture requests and
responses for every skewed operation, classify worker skew as
`register_refused`, `register_and_serve`, or `register_and_drop`, and
classify Waterline skew as `banner`, `render_refused`, or
`stale_render`. A protocol-manifest smoke subset is nonconforming until
every required cell is recorded as `pass`, `fail`, `unsupported`,
`not_covered`, or `runner_blocked` with linked findings. Only `pass`
cells count toward a passing category; `register_and_drop` and
`stale_render` without a loud warning are blocking product findings.

Those skew-refusal scenario ids and their pass criteria are published at
[`static/platform-conformance/skew-refusal-matrix-scenarios.json`](pathname:///platform-conformance/skew-refusal-matrix-scenarios.json).

The `principal_attribution_contract` category is a stable runtime
scenario category. A result for it must use published artifacts and
prove that workflow history records server-derived principals for start,
signal, query, cancellation, completion, failure, anonymous, and
server-originated event surfaces. It must exercise adversarial
payload/header spoofing, alice/bob named identities, credential
rotation, CLI operator visibility, Waterline operator visibility, and
authenticated start or signal operations through both the Python SDK and
the PHP `DurableWorkflow\Client`. SDK cells must record the
package version, operation outputs, history/API principal samples, and
raw HTTP reference principals so the harness can compare principal shape
and expected principal ids. The PHP cell must resolve `durable-workflow/sdk`
as an exact Packagist distribution and record its dist type, URL, and
reference. `durable-workflow/workflow` is resolved separately only for the
embedded Laravel and Waterline engine. A role-token smoke subset is nonconforming
until every required scenario is recorded as `pass`, `fail`,
`unsupported`, `not_covered`, or `runner_blocked` with linked findings.

Those principal-attribution scenario ids and their pass criteria are
published at
[`static/platform-conformance/principal-attribution-scenarios.json`](pathname:///platform-conformance/principal-attribution-scenarios.json).

The `prerelease_readiness_contract` category is a stable runtime
scenario category for the coordinated 2.0 release candidate. A result
for it must use only published artifacts and public user-facing docs,
record separate Workflow and Waterline GO / NO-GO verdicts, cover core
feature completeness, migration readiness, public API stability,
documentation accuracy, configuration understandability, and
cross-component compatibility, and evaluate server, CLI, PHP, Python, and
Rust SDKs, Workflow, Waterline, sample app, and public docs as one ecosystem
tuple. It must also execute each versioned 2.0 standalone quickstart path and
the separate embedded Laravel path from live public docs through observable
completed workflows within 10 minutes, recording exact commands, outputs,
artifact versions, package provenance, and wall-clock timings. A
discovery-only quickstart check is nonconforming until it records the PHP,
Python, Rust, and Laravel quickstart scenarios as `pass`, `fail`,
`unsupported`, `not_covered`, or `runner_blocked`.
Any missing artifact, stale docs page, undocumented migration step,
installability gap, API instability, cross-component breaking-change
risk, or release-channel mismatch must be recorded as a non-pass cell
with a linked finding. Runner-only evidence is nonconforming and cannot
make prerelease readiness green.

Those prerelease readiness scenario ids and their pass criteria are
published at
[`static/platform-conformance/prerelease-readiness-scenarios.json`](pathname:///platform-conformance/prerelease-readiness-scenarios.json).
The concise [2.0 quickstart](/docs/quickstart/) is the public onboarding path.

## Pass / Fail Rules

1. **`guaranteed_field_equality`.** Every field marked guaranteed in the
   fixture schema must be present, type-correct, and value-equal in the
   implementation response. Diagnostic-only fields are ignored.
2. **`unknown_additive_fields_tolerated`.** Extra fields pass only if
   they are documented diagnostic-only fields or the fixture is on a
   stability level that allows additive evolution.
3. **`frozen_shape_exact_match`.** Fixtures backed by a `frozen` surface
   family must match exactly. A frozen-shape mismatch is always a
   failure.
4. **`required_fixtures_must_pass`.** A release that claims a target must
   pass every required fixture category for that target. One failed
   required fixture means the release does not conform for that target.
5. **`stable_runtime_scenario_coverage`.** A stable runtime category
   must report every required scenario it declares with one of the
   statuses published by its runtime scenario manifest: `pass`, `fail`,
   `unsupported`, `not_covered`, or `runner_blocked`. Full conformance
   requires every required scenario to pass. A smoke-only subset,
   omitted scenario, unsupported public surface, uncovered cell, or
   runner-blocked cell is nonconforming and must link the owning
   finding. This status set and pass-only runtime rule are suite version
   5+ semantics.
6. **`provisional_categories_warn_only`.** A failed fixture in a
   provisional category emits a warning and does not block the release.
7. **`diagnostic_only_mismatches_pass`.** If only diagnostic-only fields
   differ, the harness records the difference in `diagnostic_diff` and
   the fixture passes.

The harness result declares one of four conformance levels:

| Level | Meaning |
| --- | --- |
| `full` | Every required fixture passes for every claimed target. |
| `partial` | Every required fixture passes for at least one claimed target, but another claimed target is failing. |
| `provisional` | Only provisional categories failed; required categories all passed. |
| `nonconforming` | At least one required fixture failed for every claimed target. |

## Harness Contract

A conforming harness:

- loads the suite manifest from `platform_conformance_suite` in
  `GET /api/cluster/info`, or from the static mirror for offline runs;
- loads each declared fixture from its source-of-truth path;
- drives the implementation through the fixture's documented operation;
- compares the response under the pass / fail rules above;
- emits one result document per run with schema
  `durable-workflow.v2.platform-conformance.result`, suite version,
  implementation identity, per-fixture results, diagnostic diff, and
  overall conformance level;
- exits non-zero if and only if the conformance level is
  `nonconforming`.

The result document is an artifact. A compatibility claim is valid only
when the result was produced against the implementation build and the
suite version named by that build.

## Release Gates

| Release | Required claimed target(s) | Required artifact |
| --- | --- | --- |
| `durable-workflow/server` | `standalone_server`, `worker_protocol_implementation`, `repair_actionability_surface` | Harness result document attached to the release. |
| `durable-workflow/workflow` | `embedded_engine` | Harness result document attached to the release. |
| `durable-workflow/sdk` | `official_sdk`, `worker_protocol_implementation` | Harness result document attached to the release. |
| `durable_workflow` | `official_sdk`, `worker_protocol_implementation` | Harness result document attached to the release. |
| `dw` | `cli_json_client` | Harness result document attached to the release. |
| `waterline` | `waterline_contract_surface` | Harness result document attached to the release. |
| `durable-workflow/2.0-release-candidate` | `prerelease_release_candidate` | Conformance record stores the published-artifact prerelease readiness result. |

Release reviewers confirm that the harness result is attached, the
conformance level is `full` or `provisional`, and the suite version in
the result matches the version exposed by the build under test. A
`nonconforming` result blocks the release.

## Release Check

The docs-site release check in
`scripts/check-platform-conformance-authority.js` fails the build if
the static manifest points at a missing, repo-local, version-alias-only,
or non-docs-site authority. It also validates every stable fixture-level
`authority_doc` value as a canonical current-version docs-site URL and
rejects repository-relative source trees, test fixtures, documentation,
schema directories, unversioned artifact ids, non-public resolvers, and
incorrect byte digests in every stable fixture category. Runtime categories
must additionally expose a suite-bound public scenario manifest.
The same check requires this page to list the manifest schema, target
names, fixture category names, pass / fail rules, and release gates from
the machine-readable mirror.

When the suite changes, update this page and
`static/platform-conformance-contract.json` together. If the change
adds a target, adds a required fixture category, promotes a provisional
category to stable, changes stable runtime scenario `operations` or
`pass_criteria`, changes a stable runtime scenario public requirement
field (`artifact_policy`, `common_result_evidence`, `required_matrix`,
`scenario_requirements`, or `host_runner_contract`), or changes a pass /
fail rule, advance the suite version in the same release change. The
release check pins stable runtime scenario criteria and public runtime
requirement snapshots by suite-versioned digests so external harnesses
cannot observe new criteria or evidence-policy requirements under an old
suite version.

Published runtime scenario criteria and public requirement digest entries
are append-only. To advance stable runtime scenario `operations` or
`pass_criteria`, leave every existing
`VERSIONED_RUNTIME_SCENARIO_CRITERIA_DIGESTS` entry unchanged, increase the
suite version, update the scenario manifest to that version, and add a new
criteria digest entry for the new current suite. To change public runtime
manifest evidence requirements, artifact policy, required matrix, scenario
requirement fields, or host-runner contract fields, leave every existing
`VERSIONED_RUNTIME_SCENARIO_PUBLIC_REQUIREMENT_DIGESTS` entry unchanged,
advance the suite version, update the scenario manifest to that version,
and add the corresponding public requirement digest entry for the new
current suite. The release check compares published digest entries from the
target branch against the current change, so editing or deleting an older
suite entry fails even when the current suite version adds a new digest.

<!-- Source: docs/sdk-neutrality.md -->

# SDK Neutrality

This page is the human-readable guide to the platform-wide SDK neutrality
contract. It enumerates the minimum neutrality rules that
every public Durable Workflow surface must satisfy and the standing
language-agnosticism audit that release reviewers apply to new server,
workflow, CLI, Waterline, and MCP surfaces.

The upstream architecture guide is published at
[`sdk-neutrality.md`](https://github.com/durable-workflow/workflow/blob/main/docs/architecture/sdk-neutrality.md).
The consumable machine-readable authority is published from this site at
[`/sdk-neutrality-contract.json`](/sdk-neutrality-contract.json) under
the schema id `durable-workflow.v2.sdk-neutrality.contract`. Its protocol and
SDK-breadth authority comes from the JSON shipped in the Workflow Composer
package at `resources/sdk-neutrality-contract.json`. The public mirror adds
current package-distribution metadata from the centralized docs artifact
tuple. The standalone Durable Workflow server re-exports the packaged base
manifest from `GET /api/cluster/info` under `sdk_neutrality_contract`. When
this page disagrees with the published JSON contract, the JSON contract wins
and the disagreement is a documentation bug.

This contract is downstream of the
[Version Compatibility](/docs/compatibility) authority (which says
*which* surfaces are public and *how* they may change) and the
[Platform Protocol Specs](/docs/platform-protocol-specs) catalog
(which says *where* the normative spec for each surface lives). It says
*what shape* those specs are allowed to take so that a future SDK outside the
current PHP, Python, and Rust roster can target them without requiring a
protocol redesign.

## Why this exists

Durable ships three first-party standalone SDKs: the PHP `durable-workflow/sdk`
package, Python `durable_workflow` package, and Rust `durable-workflow` crate.
The separate PHP `durable-workflow/workflow` package remains the embedded
Laravel engine and replay owner; it is not the framework-neutral PHP SDK.
Building or maintaining a wide first-party SDK roster is **not** a release goal.
Demand for SDKs in TypeScript, Go, Java, and .NET ecosystems has not
yet been demonstrated, and the maintenance cost of a broad official
roster is high.

What this contract protects against is a different failure mode: the
public contracts under those SDKs quietly hard-coding language-specific
assumptions. If a future TypeScript or Go SDK becomes worth
building, the work should be "write a new client against the published
wire protocol", not "redesign the protocol so another language can speak it
at all".

## Scope

| Scope key | Meaning |
| --- | --- |
| `goal` | Preserve protocol and contract neutrality so a future TypeScript, Go, Java, or .NET SDK does not require a protocol redesign to exist. |
| `non_goal` | Ship a broad official SDK portfolio. First-party SDK breadth is intentionally narrow and grows only when adoption demand justifies it. |
| `present_priority` | Python is the current highest-value non-PHP path for existing users and is treated as a priority surface for parity coverage. |
| `future_posture` | TypeScript, Go, Java, .NET and other languages are demand-driven. They have no reserved release slot, but every public contract must be shaped so a future SDK in those languages can be written without breaking the wire protocol. |

## Official SDK breadth policy

The official-SDK roster is intentionally narrow:

| Language | Posture | Published package | Conformance authority |
| --- | --- | --- | --- |
| PHP | `priority` | [`durable-workflow/sdk`](https://packagist.org/packages/durable-workflow/sdk) | [`signal_query_runtime_contract`](https://durable-workflow.github.io/platform-conformance/signal-query-runtime-scenarios.json), actors `sdk_php`, `php_sdk_client`, and `php_worker` |
| Python | `priority` | <PythonPackageReleaseLink>`durable_workflow` 2.17.2</PythonPackageReleaseLink> | [`history_replay_bundles`](https://durable-workflow.github.io/platform-conformance/replay-runtime-scenarios.json), actor `python_sdk_runtime` |
| Rust | `priority` | [`durable-workflow`](https://crates.io/crates/durable-workflow) | [`signal_query_runtime_contract`](https://durable-workflow.github.io/platform-conformance/signal-query-runtime-scenarios.json), actors `rust_sdk`, `rust_worker`, and `rust_sdk_client` |
| TypeScript | `demand_driven` | None | Public contracts must remain implementable in TypeScript without protocol redesign. |
| Go | `demand_driven` | None | Public contracts must remain implementable in Go without protocol redesign. |
| Java | `demand_driven` | None | Public contracts must remain implementable in Java without protocol redesign. |
| .NET | `demand_driven` | None | Public contracts must remain implementable in .NET without protocol redesign. |

The JSON contract lists the exact `scenario_ids` that constitute coverage for
`first_party.php_sdk`, `first_party.python_sdk`, and `first_party.rust_sdk`.
It tracks the Laravel package separately as
`embedded_engines.php_workflow_engine`, with replay coverage under actor
`workflow_php_runtime`. Consumers can resolve those identifiers directly in
the linked catalogs without cloning an SDK repository or reading its test
suite.

A new first-party SDK is added only when:

1. There is documented user demand the existing SDKs cannot serve.
2. A candidate maintainer team commits to keeping the SDK on the
   conformance harness, the protocol-spec catalog, and the release
   authority manifest.
3. Adding the SDK does not require breaking changes to the worker
   protocol, control plane, history-event wire formats, or replay
   fixtures. If it would, the protocol is the bug, not the SDK.

## Neutrality rules

Every public Durable contract must satisfy each of the seven neutrality
rules below. The JSON manifest is the authority for the exact field
shapes; the summaries are for reviewers. Each rule on the JSON manifest
carries a `requirement`, `rationale`, `how_to_apply` line, and an `authority`
array. Every authority entry has a stable catalog or schema ID and an absolute
URL to its published artifact.

| Rule | Requirement |
| --- | --- |
| `protocol_neutrality` | Public RPC and event surfaces use HTTP+JSON or AsyncAPI shapes that any HTTP-capable runtime can produce and consume. |
| `codec_neutrality` | Every durable payload that crosses a public boundary advertises `avro`. No other public or engine-specific v2 codec is offered. |
| `error_shape_neutrality` | Public failure objects use a structured envelope of (`code`, `message`, optional `details`). PHP and Python exception class names are diagnostic only. |
| `type_identity_neutrality` | Workflow, activity, child workflow, and exception types are identified by stable string names. Class FQCNs and module paths are SDK-input convenience, not contract. |
| `replay_fixture_neutrality` | Replay fixtures and golden history bundles are JSON conforming to the published `history_event_payloads` and `replay_bundle` schemas. |
| `discovery_neutrality` | Every public surface is reachable from `GET /api/cluster/info` and the `platform_protocol_specs` catalog. |
| `documentation_neutrality` | Public-contract docs describe shapes in schema, route, and field semantics. PHP and Python class behaviour appears as SDK examples, not as the normative contract. |

## Standing language-agnosticism audit

Every new server, workflow, CLI, Waterline, or MCP surface must clear
the neutrality audit before promotion to `stable`. The audit is a
standing review item on the release PR for every change that touches an
audit-scoped surface family. The audit-scoped families are:

- `server_api`
- `worker_protocol`
- `cli_json`
- `waterline_api`
- `mcp_discovery_results`
- `cluster_info_manifests`

The checklist has eight steps. Seven correspond to the neutrality rules
above. The eighth — the **future-SDK thought experiment** — asks the
reviewer to describe in two sentences how a TypeScript or Go SDK would
consume the new surface using only the published spec catalog and a
standard HTTP+JSON toolchain. If the answer requires a first-party SDK,
the surface is not neutral and either the surface is reshaped or the
neutrality gap is recorded as a known limitation before promotion.

## What a future SDK relies on

The contract identifies the surfaces a future SDK must be able to read
without inspecting any first-party SDK source:

- **Protocol** — `durable-workflow.v2.control-plane-api`,
  `durable-workflow.v2.worker-protocol-api`, and
  `durable-workflow.v2.worker-protocol-stream` in the published
  [protocol catalog](https://durable-workflow.github.io/platform-protocol-specs.json).
- **Codecs** — the universal codec set documented by
  `durable-workflow.v2.worker-protocol-api` and advertised through the
  `durable-workflow.v2.cluster-info-envelope` discovery schema.
- **Error shape** — the worker-protocol failure envelope and
  `durable-workflow.v2.repair-actionability-objects` schema.
- **Replay inputs** — the `durable-workflow.v2.history-event-payloads` and
  `durable-workflow.v2.replay-bundle` JSON Schemas plus scenario IDs in the
  public [`history_replay_bundles`](https://durable-workflow.github.io/platform-conformance/replay-runtime-scenarios.json)
  catalog.
- **Discovery** — `durable-workflow.v2.cluster-info-envelope` and the
  `durable-workflow.v2.platform-protocol-specs.catalog` itself.

If any of those surfaces is not reachable for a candidate SDK in a
given language, building the SDK requires protocol changes and the
language-agnosticism guarantee is not being honored.

## Release gates

A release that introduces a new public surface family or promotes an
existing surface from `prerelease` or `experimental` to `stable` must
record the audit outcome on the release PR. The release-gates section
of the manifest enumerates the specific checks:

- `audit_recorded` — the release PR description states which audit
  steps were applied and links the protocol-spec catalog entry, the
  conformance fixture, and the discovery entry for the new surface.
- `no_php_or_python_only_required_fields` — no guaranteed field on a
  `stable` surface requires a PHP-only or Python-only codec.
- `universal_codec_advertised` — worker protocol negotiation advertises
  exactly one universal codec, `avro`.
- `fixture_schema_validated` — new replay fixtures or golden history
  bundles validate against the published JSON Schemas.
- `discovery_entry_present` — new public surfaces have a
  `platform_protocol_specs` catalog entry with a non-empty
  `surface_family`, `owner_repo`, and `format`.

Enforcement is split between machine and human gates. Release CI resolves every
authority URL, protocol/schema ID, and conformance scenario ID in the public
contract; cross-references the audit scope against the surface stability
families; and rejects repository-local authority paths or implementation
symbols. Release reviewers tick the SDK-neutrality audit on every release PR
that adds or promotes a public surface and remain responsible for the
future-SDK thought experiment.

## Changing this contract

Adding a neutrality rule, tightening an existing rule, adding a
required audit step, adding a surface family to the audit scope, or
changing the official-SDK breadth policy is a contract change. Bump the
manifest version, update the architecture guide and this page in the same
change, and align the consumable contract at
[`/sdk-neutrality-contract.json`](/sdk-neutrality-contract.json).
Removing a neutrality rule or audit step is a major change.