<!-- Source: docs/introduction.md -->

# Introduction

Durable Workflow 2.0 keeps workflow state and history outside short-lived
application processes so PHP, Python, and Rust workers can resume safely after
a restart. If this is your first visit, take the
[Durable Workflow 2.0 Quickstart](/docs/quickstart/) first: it gets one workflow
to `completed` before you enter the reference-heavy
[Capability Index](/docs/capabilities/).

## Choose a deployment model

### Service mode

Applications use a remote durable runtime through first-party SDKs. Choose who
operates that runtime:

- **[Durable Workflow Cloud](/docs/polyglot/cloud-control-plane/)** is the
  managed choice. Durable Workflow operates orchestration, persistence, and
  Managed Waterline; your team runs SDK clients and workers against its
  provisioned namespace. **Cloud users do not install or run Durable Workflow
  Server or a separate Waterline service.**
- **[Self-hosted Server](/docs/polyglot/server/)** gives your team the same
  service boundary while you deploy, secure, scale, back up, and upgrade the
  runtime. Waterline observation is a separate service you may deploy against
  the Server-owned namespace.

Both choices expose the same versioned HTTP+JSON control plane, worker
protocol, namespace model, and language-neutral payload envelope. See
[Deployment Modes](/docs/polyglot/deployment-modes/) for their ownership
boundaries.

### Embedded Laravel

Embedded mode is a separate deployment model for a Laravel application that
wants workflow state, queues, configuration, and operator tooling inside its
own infrastructure. It installs `durable-workflow/workflow` and does not
connect to Cloud or require a separate Server. The embedded Waterline package
reads that application-owned state in process.

Start with [Embedded Installation](/docs/installation/) only when that
in-application ownership model is intentional.

Laravel teams coming from stable v1—or reconsidering an existing 2.0 embedded
deployment—should use the
[Laravel adoption and runtime transition guide](/docs/laravel-adoption/)
to compare the executable embedded and PHP SDK paths before changing traffic.

## Choose a service-mode SDK

- **[PHP SDK](/docs/polyglot/php/):** install `durable-workflow/sdk` in a
  framework-neutral PHP application or remote worker.
- **[Python SDK](/docs/polyglot/python/):** author deterministic workflows
  and activities and use the async control-plane client. The
  stable release manifest as the Server quickstart.
- **[Rust SDK](/docs/polyglot/rust/):** author deterministic workflows and
  activities and run native worker services.

All three are first-party implementations of the same public service boundary.
The [Client and Worker Capabilities](/docs/polyglot/cli-python-parity/)
guide makes supported and intentionally different client and worker surfaces
explicit.

## Your first completed workflow

The [Quickstart](/docs/quickstart/) states the goal, runtime choice,
prerequisites, time, and expected outcome up front. It keeps PHP, Python, and
Rust equally available while showing one runnable language path at a time.
Use the local self-hosted path for a source-free published-artifact exercise,
or use the Cloud onboarding values for a managed namespace without running
Server.

## How service mode fits together

A service-mode deployment has three parts:

- **The runtime** owns durable state, command and history recording, task
  matching, timers, schedules, namespaces, and authenticated protocols. Cloud
  operates it for managed namespaces; your team operates it when self-hosting.
- **Application workers** run workflow and activity code through the PHP,
  Python, or Rust SDK. They can deploy with an application or as independent
  services and scale separately from the runtime.
- **Clients and operational tools** start, inspect, and command the same
  runtime-owned state through SDK clients, the `dw` CLI, HTTP APIs,
  machine-readable schemas, Waterline, and agent interfaces.

## One public durable-execution contract

The first-party SDKs share registered string workflow and activity type names
and a public payload envelope. The envelope identifies its codec and carries
portable values instead of PHP serialization, Python pickles, or Rust
implementation types.

Workflow workers reconstruct decisions from durable command and history
records. Activity and child-workflow input and results can cross language
boundaries when workers advertise the same public codec and register matching
type names. Consult the [Capability Index](/docs/capabilities/) and runtime
discovery before depending on a specific SDK surface.

## Learn from the matching examples

- **Service mode and polyglot:** use the
  [Quickstart](/docs/quickstart/) and the PHP, Python, or Rust SDK guide.
- **Embedded Laravel:** use the [Sample App](/docs/sample-app/) gallery to
  explore Laravel-native workflow patterns and Waterline evidence.

The embedded gallery is not a universal starting point for Cloud or
self-hosted service-mode readers.

## Agent-operable by contract

Human operators and autonomous agents use the same machine-readable contract.
The testable loop is **Discover -> Change -> Run -> Diagnose -> Repair**:
version and capability manifests, explicit workflow commands, structured
results, typed history and worker/queue diagnostics, safe mutations, and
post-change verification. See the
[Agent Operating Loop](/docs/agent-operating-loop/) and the direct
[AI-agent evaluator](/docs/ai-agent-workflow-engine/).

## Do you need a workflow?

You probably need a workflow if:

- The process spans minutes, hours, or days
- You need to wait for a human approval step
- You need to wait for a webhook or other external event
- You need to pause and continue later without keeping a process running
- You need to be able to restart after a crash without causing bugs or duplicating work

If your task is "run five queued jobs in order and bail on the first failure,"
a job chain is usually a better fit. Durable Workflow is for cases where the
next step depends on an external event, a wait, or a decision that cannot be
known up front.

<!-- Source: docs/capabilities.md -->

# Durable Workflow 2.0 Capability Index

This is the capability index for the stable Durable Workflow 2.0 release line.

## Supported installable channel

Exact artifact identities belong to package registries and release metadata,
not this capability summary.

| Surface | Supported channel | Role in this index |
| --- | --- | --- |
| CLI | 2.0 stable | Machine-readable operator and diagnostic client. |
| PHP SDK | 2.0 stable | First-party service-mode `durable-workflow/sdk` client and remote workflow/activity worker for a Cloud namespace runtime or self-hosted Server. |
| Embedded Laravel engine | 2.0 stable | Separate in-application `durable-workflow/workflow` deployment mode with its own authoring runtime, persistence engine, queues, and replay. |
| Python SDK | 2.0 stable | First-party service-mode deterministic workflow/activity SDK plus operational and control-plane client for a Cloud namespace runtime or self-hosted Server. Requires Python 3.10 or newer. |
| Rust SDK | 2.0 stable | First-party service-mode deterministic workflow/activity SDK, worker service, and control-plane client for a Cloud namespace runtime or self-hosted Server. Requires Rust 1.86 or newer. |
| Server | 2.0 stable | Self-hosted, PHP-implemented language-neutral runtime for the v2 control-plane and worker protocols. |
| Waterline | 2.0 stable | Cloud includes Managed Waterline; self-hosted operators separately deploy the service image against a Server namespace; embedded Laravel installs the Composer package and reads application-owned state in process. |

PHP, Python, and Rust are the three first-party service-mode SDK languages.
Each can target a provisioned Durable Workflow Cloud namespace runtime or a
self-hosted Server where runtime discovery reports the published capability.
They share stable string type names, one durable command/history model, control
plane version `2`, worker protocol major `1`, and the public payload envelope.
Rust is a workflow SDK: it runs deterministic workflow code, activities, and
worker services, not only raw protocol calls. Python combines workflow
authoring with namespace, schedule, worker, queue, history, repair, and other
operational client surfaces.

Embedded Laravel remains a separate in-application PHP deployment mode; it is
not required for PHP service-mode clients or workers. The framework-neutral PHP
SDK is versioned independently from the embedded Laravel engine.

Use [Deployment Modes](/docs/polyglot/deployment-modes/) to choose the
ownership boundary, then continue with
[Cloud Managed Runtime](/docs/polyglot/cloud-control-plane/) or the
[self-hosted Server](/docs/polyglot/server/). The capability evidence below
applies only where the selected runtime advertises the required contract.

The current self-hosted Server floor advertises worker protocol `1.13`. Python
uses its declared `1.1` baseline, while Rust uses `1.2` and requires the
additive `1.8` query-task surface for replayed queries. Runtime discovery, not a
Server patch-number guess, decides whether a client may connect. See
[Version Compatibility](/docs/compatibility/).

## Capability and SDK floors

“Service mode” means either Cloud or a self-hosted Server owns the remote
runtime. “Embedded” means a Laravel application owns the runtime in process.
The SDK column deliberately records differences; a blank or limited SDK is not
implied to have parity.

| Capability | Current 2.0 contract and first-party floor | Explicit 2.0 evidence |
| --- | --- | --- |
| Workflows | Remote PHP, Python, and Rust workers author deterministic workflows; embedded Laravel authoring uses the Workflow package. | [Workflow authoring](/docs/defining-workflows/workflows/), [PHP SDK](/docs/polyglot/php/), [Python SDK](/docs/polyglot/python/), [Rust SDK](/docs/polyglot/rust/) |
| Activities and services | The embedded engine and all three service-mode SDKs author activities. Rust workflows and activities run in first-party worker services; Python and PHP also expose first-party control/service clients. | [Activities](/docs/defining-workflows/activities/), [PHP SDK](/docs/polyglot/php/), [Activity execution](/docs/features/activity-execution-model/), [External execution](/docs/polyglot/external-execution/) |
| Signals | Durable, asynchronous mutation is available to PHP, Python, and Rust workflow surfaces at their current floors. | [Signals](/docs/features/signals/), [Python SDK](/docs/polyglot/python/), [Rust SDK](/docs/polyglot/rust/) |
| Queries | Read-only replay queries are supported. PHP, Python, and Rust expose query handlers/clients; Rust replayed query tasks require worker protocol `1.8` or newer. | [Queries](/docs/features/queries/), [Python SDK](/docs/polyglot/python/), [Rust SDK](/docs/polyglot/rust/) |
| Updates | PHP, Python, and Rust expose client and worker-handler update surfaces. Python declared validators use synchronous pre-accept validation when Server discovery advertises that exact contract: acceptance follows approval, while rejection, worker loss, timeout, and fenced completion fail explicitly. PHP and Rust do not expose validator authoring and declare no validators. | [Updates](/docs/features/updates/), [Python SDK](/docs/polyglot/python/), [Rust SDK](/docs/polyglot/rust/#workflow-updates) |
| Timers | Service-runtime-backed durable timers replay in PHP, Python, and Rust. | [Timers](/docs/features/timers/), [Rust SDK](/docs/polyglot/rust/) |
| Retries | Durable activity retry policy is recorded with the command. PHP, Python, and Rust expose activity retry options at the current floors. | [Failures and recovery](/docs/failures-and-recovery/), [Python SDK](/docs/polyglot/python/), [Rust SDK](/docs/polyglot/rust/) |
| Timeouts | Service-mode runtimes in Cloud and self-hosted Server enforce activity and workflow timeout families. Rust includes activity options and workflow execution/run start deadlines. | [Timeouts](/docs/features/timeouts/), [Rust SDK](/docs/polyglot/rust/) |
| Child workflows | PHP, Python, and Rust start and await durable children. Cross-language type identity is a stable string and child payloads use the shared envelope. | [Child workflows](/docs/features/child-workflows/), [Python SDK](/docs/polyglot/python/) |
| Deterministic parallel composition | PHP `all()`/`parallel()`, Python nested list-yield, and Rust `WorkflowContext::parallel()`/`join()` schedule activity, child-workflow, timer, mixed, and nested groups through ordinary commands. All three publish the shared group identity/path metadata and restore results in nested input order. | [Concurrency](/docs/features/concurrency/), [Python SDK](/docs/polyglot/python/#fan-out), [Rust SDK](/docs/polyglot/rust/#deterministic-parallel-groups) |
| <span id="durable-first-completion-selection-capability">Durable first-completion selection</span> | Embedded PHP and the PHP, Python, and Rust service SDKs start independent activity, child-workflow, timer, external-input wait, and nested-group members. One winner is persisted with stable member identity; non-winners continue and retain await/cancel handles. | [Concurrency](/docs/features/concurrency/#first-completion-selection), [Python SDK](/docs/polyglot/python/#first-completion-selection), [Rust SDK](/docs/polyglot/rust/#durable-first-completion-selection) |
| Saga compensation | PHP's Saga helpers, Python `WorkflowContext.saga()`, and Rust `WorkflowContext::saga()` register durable compensations after forward success and unwind sequentially in reverse order by default. Python and Rust stop at the first compensation failure and preserve the initiating plus compensation failures; PHP additionally exposes opt-in parallel and continue-on-error policies. | [Sagas](/docs/features/sagas/), [Python SDK](/docs/polyglot/python/#saga-compensation), [Rust SDK](/docs/polyglot/rust/#saga-compensation) |
| Cancellation | Cooperative cancellation is a durable lifecycle command with selected-run safety and typed outcomes. PHP, Python, and Rust clients expose it. | [Cancel and terminate](/docs/features/cancel-and-terminate/), [Rust SDK](/docs/polyglot/rust/) |
| Termination | Forced termination is separate from cancellation and is exposed through Cloud and self-hosted Server service runtimes, CLI, PHP, Python, and Rust control surfaces. | [Cancel and terminate](/docs/features/cancel-and-terminate/), [Server API](/docs/polyglot/server-api-reference/) |
| Side effects | PHP, Python, and Rust record non-deterministic values exactly once and decode the recorded value on replay. | [Side effects](/docs/features/side-effects/), [Python SDK](/docs/polyglot/python/), [Rust SDK](/docs/polyglot/rust/) |
| Workflow Streams | PHP, Python, and Rust expose typed list, describe, offset subscription/resume, append, close, and error operations for Server-owned run-scoped output streams. All three author replay-safe emits from a durable command identity. Python resolves external payloads with its configured storage driver; PHP and Rust preserve opaque references. | [Workflow Streams](/docs/polyglot/workflow-streams/), [exact SDK matrix](/workflow-stream-capabilities.json), [runtime scenarios](/platform-conformance/workflow-stream-runtime-scenarios.json) |
| Version markers | PHP, Python, and Rust expose durable version markers for compatible code evolution. | [Versioning](/docs/features/versioning/), [Python SDK](/docs/polyglot/python/), [Rust SDK](/docs/polyglot/rust/) |
| Deterministic replay | PHP, Python, and Rust workflow workers reconstruct decisions from durable history and report typed non-determinism rather than re-running external work. | [Execution guarantees](/docs/constraints/execution-guarantees/), [Platform conformance](/docs/platform-conformance/), [Rust SDK](/docs/polyglot/rust/) |
| Schedules | Cloud and self-hosted Server service runtimes own durable schedules. PHP, Python, and CLI expose schedule operations. Rust does not claim a schedule-management API. | [Schedules](/docs/features/schedules/), [CLI reference](/docs/polyglot/cli-reference/), [Python SDK](/docs/polyglot/python/) |
| Namespaces | Service-mode runtimes are namespace-scoped. PHP/Python clients and CLI manage namespaces where the selected runtime exposes that operation; Rust workers/clients target a namespace but do not claim namespace administration at the current floor. | [Namespace, auth, and workers](/docs/polyglot/namespace-auth-workers/), [Server API](/docs/polyglot/server-api-reference/) |
| Search attributes | Cloud and self-hosted Server service runtimes index typed search attributes. PHP and Python expose authoring/control surfaces; CLI and operator APIs expose structured discovery and filtering. | [Search attributes](/docs/features/search-attributes/), [Python SDK](/docs/polyglot/python/) |
| Worker compatibility | SDKs register runtime, SDK version, build ID, supported types, protocol version, and capacity; the server publishes accepted versions and routing facts. | [Compatibility](/docs/compatibility/), [Worker compatibility and routing](/docs/polyglot/worker-compatibility-routing/) |
| Local activities, worker sessions, and sticky execution | Available in embedded Laravel and the PHP service SDK. Python and Rust service workers do not yet implement these features; explicit refusal prevents incompatible routing and is not feature parity. | [Embedded activity execution](/docs/features/activity-execution-model/), [Service-mode support matrix](/docs/polyglot/portable-worker-affinity/) |
| Codec interoperability | PHP, Python, and Rust use the public `codec` + `blob` envelope and one fixed recursive Avro Value schema. Named branches preserve integers versus doubles, text versus bytes, booleans versus integers, and lists versus maps. | [Avro Value protocol](/docs/polyglot/avro-value-protocol/), [Worker protocol](/docs/polyglot/worker-protocol/), [Rust SDK](/docs/polyglot/rust/) |
| Diagnostics | Service runtimes and CLI publish version, protocol, worker, task-queue, replay, history, typed failure, and repair facts as JSON. Managed Waterline, a separately deployed self-hosted service, or the embedded package presents evidence from its owning runtime. | [Monitoring](/docs/monitoring/), [CLI reference](/docs/polyglot/cli-reference/), [Waterline operator API](/docs/waterline-operator-api/) |
| Agent tooling | Discover -> Change -> Run -> Diagnose -> Repair is available through public manifests, schemas, HTTP operations, CLI JSON, SDK clients, typed history/diagnostics, and safe mutations. MCP is one optional interface, not the definition. | [Agent tooling contract](/docs/agent-tooling-contract/), [Agent operating loop](/docs/agent-operating-loop/) |

## Payload interoperability boundary

Cross-language workflow and activity calls do not pass serialized PHP objects,
Python pickles, or Rust implementation types. They pass registered string type
names and a public payload envelope. With `avro`, each first-party SDK uses the
official language implementation and the same fixed Value schema; decoded
maps, lists, strings, bytes, integers, doubles, booleans, and nulls preserve
their primitive type. A
language-specific class is reconstructed only inside its owning SDK.

That contract applies to workflow input and result, child workflows, activity
input and result, signals, queries, updates, and external execution. Deployments
must still verify the codecs advertised by `GET /api/cluster/info` before
admitting a worker.

## Release and maturity boundary

The platform contracts indexed here are the stable 2.0 product surface.
Use the [AI-agent evaluator](/docs/ai-agent-workflow-engine/) for a concise
fit decision.

<!-- Source: docs/quickstart.md -->

# Durable Workflow 2.0 Quickstart

## Before you begin

**Goal:** run one service-mode workflow and read its completed durable result
from PHP, Python, or Rust.

**Expected time:** about 15 minutes after your runtime is available.

**Completed outcome:** the selected SDK starts a worker and workflow, then
prints a workflow ID, `status=completed`, and `Hello, <language>!`.

**Prerequisites:**

- `curl` and a terminal
- Docker for the self-hosted local path, or a provisioned Durable Workflow
  Cloud namespace
- one language toolchain: PHP 8.1+ with Composer, Python 3.10+, or Rust 1.86+

You do not need Laravel for service mode. The embedded Laravel path is separate
at the end of this guide.

## 1. Choose your service-mode runtime

| Runtime | Choose it when | Next action |
| --- | --- | --- |
| Durable Workflow Cloud | You want Durable Workflow to operate the runtime, persistence, and Managed Waterline. | Follow the executable [Cloud first workflow](/docs/polyglot/cloud-control-plane/#cloud-first-workflow), which maps PHP, Python, and Rust complete sources to provisioned credentials and a `completed` result. **Do not run Server or a separate Waterline service.** |
| Self-hosted Server | You want to operate the runtime yourself or run this exact local published-artifact exercise. | Continue below with Docker and `curl`; deploy Waterline separately only when you want its operator UI. |

The runnable source below uses a local self-hosted Server so it can be exercised
without an account or source checkout. Cloud uses the same SDK and worker model;
replace the local development connection with the provisioned values shown in
[Cloud Managed Runtime](/docs/polyglot/cloud-control-plane/).

## 2. Start the local Server

Skip this action when you chose Cloud. For the self-hosted path, expand and run
the exact pinned setup. It starts a source-free Server with SQLite and a
development token.

## Start the pinned Server image

<!-- docs-example id="quickstart.server.setup" -->
```bash
export DW_SERVER_IMAGE=durableworkflow/server:2.0.0
export DW_AUTH_TOKEN=dev-token

docker volume create durable-workflow-quickstart

docker run --rm \
  -v durable-workflow-quickstart:/app/database \
  -e DW_AUTH_DRIVER=token \
  -e DW_AUTH_TOKEN="$DW_AUTH_TOKEN" \
  "$DW_SERVER_IMAGE" server-bootstrap

docker rm -f durable-workflow-server >/dev/null 2>&1 || true
docker run -d --name durable-workflow-server \
  -p 8080:8080 \
  -v durable-workflow-quickstart:/app/database \
  -e DW_AUTH_DRIVER=token \
  -e DW_AUTH_TOKEN="$DW_AUTH_TOKEN" \
  "$DW_SERVER_IMAGE"

until curl -sf http://localhost:8080/api/ready >/dev/null; do sleep 1; done
curl -H "Authorization: Bearer $DW_AUTH_TOKEN" \
  http://localhost:8080/api/cluster/info
```

**Expected result:** the readiness request succeeds and cluster info identifies
the local standalone Server. Keep it running while you complete one language
route.

## 3. Choose one language {#choose-one-language}

All three first-party SDKs are available at the same level. Only the selected
tab is shown, so you can follow one path without scrolling past two other
programs.

## 4. Clean up the local Server

Cloud users have no local Server to remove. For the self-hosted exercise:

```bash
docker rm -f durable-workflow-server
docker volume rm durable-workflow-quickstart
```

## Separate path: embedded Laravel

Embedded Laravel is a separate first-party PHP deployment mode for
applications that want workflow state, queue execution, configuration, and
operator tooling inside their existing Laravel infrastructure. It installs
`durable-workflow/workflow`; it does not use the standalone server or
`durable-workflow/sdk`.

Start a fresh embedded application with the published package:

```bash
composer create-project laravel/laravel durable-workflow-laravel-quickstart
cd durable-workflow-laravel-quickstart
composer require durable-workflow/workflow:2.0.1
php artisan migrate
php artisan queue:work
```

When you want the operator UI inside that same Laravel application, add the
qualified embedded Waterline Composer package:

```bash
composer require durable-workflow/waterline:2.0.0
php artisan waterline:install
```

This Composer package is not the install identity for the separately deployed
self-hosted Waterline service. Embedded Laravel does not run Server or install
one of the service-mode SDKs.

Continue with [Embedded Installation](/docs/installation/) to configure a
non-`sync` Laravel queue, then [define](/docs/defining-workflows/workflows/)
and [start](/docs/defining-workflows/starting-workflows/) an embedded
workflow. [Deployment Modes](/docs/polyglot/deployment-modes/) compares
this specialized route with the service-mode platform.

## Next steps

- Use the [Capability Index](/docs/capabilities/) to check the supported
  surface for your selected runtime and SDK.
- Continue with the service-mode [PHP SDK](/docs/polyglot/php/),
  [Python SDK](/docs/polyglot/python/), or
  [Rust SDK](/docs/polyglot/rust/) guide.
- Compare lifecycle, messages, schedules, visibility, and worker execution in
  [Client and Worker Capabilities](/docs/polyglot/cli-python-parity/).
- Operate the matching runtime through
  [Cloud Managed Runtime](/docs/polyglot/cloud-control-plane/) or the
  [self-hosted Server](/docs/polyglot/server/), then add the
  [CLI](/docs/polyglot/cli/) when shell automation is useful.
- Plan safe service-worker deployment with
  [worker compatibility and routing](/docs/polyglot/worker-compatibility-routing/)
  and [build-ID rollout](/docs/polyglot/worker-build-id-rollout/).

For embedded Laravel authoring features such as timers, signals, queries,
activities, and child workflows, use the separate
[Embedded documentation](/docs/category/embedded/).

Release qualification is intentionally separate from this first-success
guide. The [Platform Conformance Suite](/docs/platform-conformance/)
contains the exact artifact matrix, public-source checks, full execution
transcripts, wall-clock criteria, teardown, and machine-readable quickstart
contract used for certification.

<!-- Source: docs/installation.md -->

# Installation

This guide covers installing the Durable Workflow PHP package for Laravel applications.

If you are deciding between package embedding and the standalone server, start
with [Deployment Modes](/docs/polyglot/deployment-modes). This page covers
the embedded Laravel path.

> **Prefer to start from a working app?** The
> [Sample App](/docs/sample-app) is a runnable Laravel 13 project on
> Durable Workflow 2.0 with one workflow per pattern surface, a Codespaces
> flow, and a `docker compose` flow. Clone it, run `php artisan app:init`,
> and you have the same install applied for you. Come back here when you
> are ready to add Durable Workflow to your own Laravel application.

## Requirements

- PHP 8.1 or later
- Laravel 9 or later

Durable Workflow can be used with any queue driver that Laravel supports (except the `sync` driver), including:

- Amazon SQS
- Beanstalkd
- Database
- Redis

Each queue driver has its own [prerequisites](https://laravel.com/docs/12.x/queues#driver-prerequisites).

Durable Workflow also requires a cache driver that supports [locks](https://laravel.com/docs/12.x/cache#atomic-locks).

## Installing Durable Workflow

Durable Workflow is installable via Composer:

```bash
composer require durable-workflow/workflow:2.0.1
```

Use `durable-workflow/workflow:^2.0` when you want Composer to accept compatible
2.x updates automatically. See [Version Compatibility](/docs/compatibility)
for runtime and package compatibility rules.

The package auto-loads its migrations, so a normal migrate run is enough after install:

```bash
php artisan migrate
```

## Running Workers

Durable Workflow uses queues to run workflows and activities in the background. You will need to either run the `queue:work` [command](https://laravel.com/docs/12.x/queues#the-queue-work-command) or use [Horizon](https://laravel.com/docs/12.x/horizon) to run your queue workers. Without a queue worker, workflows and activities will not be processed. To run workflows and activities in parallel, you will need more than one queue worker.

<!-- Source: docs/laravel-adoption.md -->

# Laravel Adoption and Runtime Transition

Laravel applications have two first-party Durable Workflow 2.0 destinations.
Embedded mode keeps durable execution inside the application. Service mode
keeps Laravel as the application and worker framework while Cloud or a
self-hosted Server owns durable state.

This is the decision and transition page for three journeys:

1. Stable v1 Laravel to 2.0 embedded.
2. Stable v1 Laravel to the 2.0 PHP SDK in service mode.
3. 2.0 embedded Laravel to the 2.0 PHP SDK in service mode.

If your immediate goal is a package-level v1 upgrade, keep the detailed
[2.0 migration guide](/docs/migration/) open beside this page. If you have
already selected a self-hosted service runtime, continue with the
[embedded-to-Server runbook](/docs/polyglot/embedded-to-server/) after
choosing the transition policy here.

## Pick the destination

| Starting point | Destination | Choose it when | First transition boundary |
| --- | --- | --- | --- |
| Stable v1 Laravel | 2.0 embedded | The Laravel app should continue to own workflow persistence, queues, execution, and embedded Waterline. | Upgrade the maintained `durable-workflow/workflow` package to the 2.0 release line. Existing v1 runs finish through the package's v1 compatibility path; new starts use v2. |
| Stable v1 Laravel | PHP SDK service mode | Laravel should keep dependency injection, configuration, Artisan, logging, events, and test fakes, but Cloud or Server should own durable state. | Install `durable-workflow/sdk` as a separate service client/worker boundary. v1 history and durable Laravel queue jobs stay on the v1 runtime until terminal. |
| 2.0 embedded Laravel | PHP SDK service mode | The application should stop owning the orchestration runtime or needs a shared/polyglot runtime boundary. | Route new starts to the service runtime after its workers are ready. Drain embedded runs where they started, or use the explicit eligible embedded-v2 import procedure for self-hosted Server. |

The PHP SDK bridge and the embedded package support Laravel 9 through 13 on
their compatible PHP versions. A supported v1 application can therefore choose
either destination without an unrelated Laravel upgrade.

## Compare the ownership boundary

| Concern | Stable v1 Laravel | 2.0 embedded Laravel | 2.0 PHP SDK service mode |
| --- | --- | --- | --- |
| Runtime owner | The Laravel application owns orchestration and replay. | The Laravel application owns the v2 engine, history, matching, timers, and replay. | Cloud operates the namespace runtime, or your team operates a self-hosted Server. Laravel runs clients and remote workers, not the runtime. |
| Composer package | `durable-workflow/workflow` on the stable 1.x line. | `durable-workflow/workflow` from the 2.0 artifact authority. | `durable-workflow/sdk` from the independently published PHP SDK authority. It does not depend on the embedded engine. |
| Required processes | Laravel web/CLI processes plus Laravel queue workers or Horizon. | Laravel web/CLI processes plus queue workers or Horizon; scheduler and repair roles follow the embedded deployment configuration. | Laravel application processes plus one or more `php artisan durable-workflow:worker` processes. Cloud supplies the runtime. Self-hosting additionally requires Server and its persistence. |
| Queues and storage | Workflow rows and PHP-oriented history live in configured Laravel databases; workflow, activity, retry, and timer jobs live in Laravel queue backends. | Event history and projections live in the application's configured database; embedded tasks normally use Laravel queues. | Durable history and task state live in Cloud or Server persistence. `DURABLE_WORKFLOW_TASK_QUEUE` is a remote task-queue identity polled by the SDK worker, not a Laravel queue connection. |
| Workflow/activity identity | PHP workflow and activity classes are durable identities. | Configure stable aliases under `workflows.v2.types`; classes remain local implementations. | `#[Workflow('orders.fulfill')]` and `#[Activity('orders.reserve')]` publish stable string type keys during worker registration. Do not use PHP FQCNs as the cross-runtime contract. |
| Credentials | Laravel application, database, cache, and queue credentials; the original `APP_KEY` remains part of v1 recovery. | Laravel application credentials plus the database, cache, queue, and any embedded Waterline authentication owned by the host app. | Cloud application processes receive only a control credential and worker processes only a worker credential. A self-hosted deployment may use one shared token or equivalent scoped credentials. Credentials are process inputs and are deliberately absent from cached Laravel configuration. |
| Operational tooling | Laravel logs/events, queue tooling, Horizon, v1 Waterline, and `workflow:v1:list` where available. | Laravel logs/events, queue tooling, `workflow:v2:doctor`, history/repair commands, and embedded Waterline. | SDK diagnostics flow through Laravel's PSR logger and `WorkerDiagnosticEvent`. Cloud includes Managed Waterline. Self-hosted operators use Server APIs/CLI and may deploy Waterline separately against the Server namespace. |
| Cutover and rollback owner | Your team preserves the v1 database, queue state, configuration, and secrets as one recovery set. | Your team separates v1 and v2 starts, retains v1 wake paths, and decides when old tables/queues can be retired. | Your team routes each command to the runtime that owns its run. Cloud owns runtime rollback inside the managed service; self-hosted operators own Server persistence and rollback. Application rollback must keep both old and new runtime routes until their runs are terminal. |

Cloud and self-hosted Server are alternatives inside service mode. Cloud users
do not install Server or a separate Waterline service. Self-hosted operators
deploy, secure, back up, scale, and upgrade Server; Waterline is a separate
optional deployment against the Server-owned namespace.

## One Laravel use case in all three modes

The examples reserve inventory for `orders.fulfill`. Map the stable v1 class to
the v2 type keys deliberately, then keep those v2 type keys and the configured
task queue stable. Inventory reservation and the business event consumer must
be idempotent because an activity can execute more than once after a retry or
lost completion acknowledgement.

The application action remains the same in production and tests:

| Mode | Production entry point | Entry point after enabling its fake |
| --- | --- | --- |
| Stable v1 Laravel | `app(StartOrderFulfillment::class)->start('1001')` | `app(StartOrderFulfillment::class)->start('1001')` |
| 2.0 embedded Laravel | `app(StartOrderFulfillment::class)->start('1001')` | `app(StartOrderFulfillment::class)->start('1001')` |
| PHP SDK service mode | `app(StartOrderFulfillment::class)->start('1001')` | `app(StartOrderFulfillment::class)->start('1001')` |

Laravel constructor injection remains available at the application boundary in
every mode. Stable v1 injects activity dependencies into `execute()`. Embedded
v2 and service mode both ask Laravel's container to construct workflow and
activity objects, so their application dependencies use ordinary constructors.
Keep anything read by a replayed workflow deterministic and config-backed;
database, network, clock, and other side effects belong in activities.

### Stable v1 Laravel

Keep the [stable installation](/docs/installation/) and its real queue-driver
requirements. This representative implementation uses a constructor-injected
application starter and method-injected activity services:

<!-- docs-example id="laravel.adoption.v1.handlers" -->
```php
<?php

use App\Contracts\InventoryGateway;
use App\Contracts\OrderPolicy;
use App\Events\InventoryReserved;
use Illuminate\Contracts\Events\Dispatcher;
use Psr\Log\LoggerInterface;
use Workflow\Activity;
use Workflow\Workflow;
use Workflow\WorkflowOptions;
use Workflow\WorkflowStub;
use function Workflow\activity;

final class FulfillOrderWorkflow extends Workflow
{
    public function execute(string $orderId): \Generator
    {
        return yield activity(ReserveInventoryActivity::class, $orderId);
    }
}

final class ReserveInventoryActivity extends Activity
{
    public function execute(
        OrderPolicy $orders,
        InventoryGateway $inventory,
        LoggerInterface $logger,
        Dispatcher $events,
        string $orderId,
    ): array {
        $orders->assertFulfillable($orderId);
        $reservation = $inventory->reserve($orderId);
        $logger->info('Inventory reserved', ['order_id' => $orderId]);
        $events->dispatch(new InventoryReserved($orderId));

        return $reservation;
    }
}

final class StartOrderFulfillment
{
    public function __construct(private readonly LoggerInterface $logger)
    {
    }

    public function start(string $orderId): void
    {
        $workflow = WorkflowStub::make(FulfillOrderWorkflow::class);
        $workflow->start($orderId, new WorkflowOptions('redis', 'orders'));
        $this->logger->info('Order workflow started', ['order_id' => $orderId]);
    }
}
```

`config/queue.php` and the deployment environment still select the real
Laravel queue connection. Run a worker for the configured queue:

<!-- docs-example id="laravel.adoption.v1.worker" -->
```bash
php artisan queue:work redis --queue=orders
```

The stable fake runs the same injected application action used in production:

<!-- docs-example id="laravel.adoption.v1.fake" -->
```php
<?php

use Workflow\WorkflowStub;

WorkflowStub::fake();
WorkflowStub::mock(ReserveInventoryActivity::class, ['status' => 'reserved']);

app(StartOrderFulfillment::class)->start('1001');

WorkflowStub::assertDispatched(
    ReserveInventoryActivity::class,
    fn (string $orderId): bool => $orderId === '1001',
);
```

Laravel event assertions remain available when testing the real activity. Do
not replace v1 workers merely because the v2 SDK has been added elsewhere in
the application.

### 2.0 embedded Laravel

The stable install uses the same Workflow package as the embedded quickstart:

<!-- docs-example id="laravel.adoption.v2-embedded.install" -->
```bash
composer require durable-workflow/workflow:2.0.1
```

Publish or update `config/workflows.php`, then give the same business contract
stable aliases. The published file contains additional required v1 and v2
settings; merge these values into its existing arrays instead of replacing the
complete file with this focused excerpt:

<!-- docs-example id="laravel.adoption.v2-embedded.config" -->
```php
<?php

return [
    'v2' => [
        'namespace' => env('DW_V2_NAMESPACE', 'production'),
        'types' => [
            'workflows' => [
                'orders.fulfill' => App\Workflows\FulfillOrderWorkflow::class,
            ],
            'activities' => [
                'orders.reserve' => App\Activities\ReserveInventoryActivity::class,
            ],
        ],
    ],
];
```

The authoring API changes to `handle()` and no longer yields activity results.
Laravel constructs both objects, while the activity remains the right place
for logging and business events that represent side effects:

<!-- docs-example id="laravel.adoption.v2-embedded.handlers" -->
```php
<?php

use App\Contracts\InventoryGateway;
use App\Contracts\OrderPolicy;
use App\Events\InventoryReserved;
use Illuminate\Contracts\Events\Dispatcher;
use Psr\Log\LoggerInterface;
use Workflow\V2\Activity;
use Workflow\V2\Workflow;
use Workflow\V2\WorkflowStub;
use function Workflow\V2\activity;

final class FulfillOrderWorkflow extends Workflow
{
    public ?string $connection = 'redis';
    public ?string $queue = 'orders';

    public function __construct(private readonly OrderPolicy $orders)
    {
    }

    public function handle(string $orderId): array
    {
        $this->orders->assertFulfillable($orderId);

        return activity(ReserveInventoryActivity::class, $orderId);
    }
}

final class ReserveInventoryActivity extends Activity
{
    public ?string $connection = 'redis';
    public ?string $queue = 'orders';

    public function __construct(
        private readonly OrderPolicy $orders,
        private readonly InventoryGateway $inventory,
        private readonly LoggerInterface $logger,
        private readonly Dispatcher $events,
    ) {
    }

    public function handle(string $orderId): array
    {
        $this->orders->assertFulfillable($orderId);
        $reservation = $this->inventory->reserve($orderId);
        $this->logger->info('Inventory reserved', ['order_id' => $orderId]);
        $this->events->dispatch(new InventoryReserved($orderId));

        return $reservation;
    }
}

final class StartOrderFulfillment
{
    public function __construct(private readonly LoggerInterface $logger)
    {
    }

    public function start(string $orderId): void
    {
        WorkflowStub::make(FulfillOrderWorkflow::class, "order-{$orderId}")
            ->start($orderId);
        $this->logger->info('Order workflow started', ['order_id' => $orderId]);
    }
}
```

Run the Laravel queue worker exactly as an embedded deployment requires:

<!-- docs-example id="laravel.adoption.v2-embedded.worker" -->
```bash
php artisan queue:work redis --queue=orders
```

Use the embedded fake through the same application action. It executes ready
workflow tasks inline and records activity dispatches:

<!-- docs-example id="laravel.adoption.v2-embedded.fake" -->
```php
<?php

use Workflow\V2\WorkflowStub;

WorkflowStub::fake();
WorkflowStub::mock(ReserveInventoryActivity::class, ['status' => 'reserved']);

app(StartOrderFulfillment::class)->start('1001');
while (WorkflowStub::runReadyTasks() > 0) {
}

WorkflowStub::assertDispatched(
    ReserveInventoryActivity::class,
    fn (string $orderId): bool => $orderId === '1001',
);
```

This fake exercises embedded durable rows and history. It is not the SDK client
fake and does not stand in for a service runtime.

### PHP SDK service mode in Laravel

Install the independently published SDK package. This command is rendered from
the published PHP SDK artifact authority, not copied from a point release:

<!-- docs-example id="laravel.adoption.service.install" -->
```bash
composer require durable-workflow/sdk:2.0.0
```

Laravel 9 through 13 auto-discover the provider. Publish the package
configuration and list the attributed handler classes:

<!-- docs-example id="laravel.adoption.service.config" -->
```php
<?php

return [
    'runtime_url' => env('DURABLE_WORKFLOW_RUNTIME_URL', 'http://localhost:8080'),
    'namespace' => env('DURABLE_WORKFLOW_NAMESPACE', 'production'),
    'task_queue' => env('DURABLE_WORKFLOW_TASK_QUEUE', 'orders'),
    'handlers' => [
        App\Workflows\FulfillOrderWorkflow::class,
        App\Activities\ReserveInventoryActivity::class,
    ],
    'poll_timeout_seconds' => 5,
];
```

The service workflow uses the same public strings as embedded v2. Laravel
container-constructs both attributed service classes:

<!-- docs-example id="laravel.adoption.service.handlers" -->
```php
<?php

use App\Contracts\InventoryGateway;
use App\Contracts\OrderPolicy;
use App\Events\InventoryReserved;
use DurableWorkflow\Attribute\Activity;
use DurableWorkflow\Attribute\Workflow;
use DurableWorkflow\Worker\ActivityContext;
use DurableWorkflow\Worker\WorkflowContext;
use Illuminate\Contracts\Events\Dispatcher;
use Psr\Log\LoggerInterface;

final class FulfillOrderWorkflow
{
    public function __construct(private readonly OrderPolicy $orders)
    {
    }

    #[Workflow('orders.fulfill')]
    public function run(WorkflowContext $context, string $orderId): array
    {
        $this->orders->assertFulfillable($orderId);

        return $context->activity('orders.reserve', [$orderId]);
    }
}

final class ReserveInventoryActivity
{
    public function __construct(
        private readonly OrderPolicy $orders,
        private readonly InventoryGateway $inventory,
        private readonly LoggerInterface $logger,
        private readonly Dispatcher $events,
    ) {
    }

    #[Activity('orders.reserve')]
    public function run(ActivityContext $context, string $orderId): array
    {
        $this->orders->assertFulfillable($orderId);
        $reservation = $this->inventory->reserve($orderId);
        $this->logger->info('Inventory reserved', ['order_id' => $orderId]);
        $this->events->dispatch(new InventoryReserved($orderId));

        return $reservation;
    }
}
```

Application code injects the Laravel-native SDK interface instead of reaching
through a facade or assembling a transport client. The attributed class
supplies the workflow type, and published configuration supplies the default
task queue:

<!-- docs-example id="laravel.adoption.service.client" -->
```php
<?php

use DurableWorkflow\Bridge\Laravel\LaravelWorkflowClientInterface;
use DurableWorkflow\WorkflowHandleInterface;

final class StartOrderFulfillment
{
    public function __construct(
        private readonly LaravelWorkflowClientInterface $workflows,
    ) {
    }

    public function start(string $orderId): WorkflowHandleInterface
    {
        return $this->workflows->start(
            FulfillOrderWorkflow::class,
            [$orderId],
            workflowId: "order-{$orderId}",
        );
    }

    public function handle(string $orderId): WorkflowHandleInterface
    {
        return $this->workflows->handle(
            FulfillOrderWorkflow::class,
            "order-{$orderId}",
        );
    }
}
```

After publishing the configuration, cache only non-secret settings. Set
`RUNTIME_URL` to the complete provisioned Cloud namespace runtime URI or to the
self-hosted Server origin; do not append `/api`.

Start the worker with only its worker-role credential:

<!-- docs-example id="laravel.adoption.service.worker" -->
```bash
php artisan vendor:publish --tag=durable-workflow-config
env -u DURABLE_WORKFLOW_TOKEN \
  -u DURABLE_WORKFLOW_CLIENT_TOKEN \
  -u DURABLE_WORKFLOW_WORKER_TOKEN \
  DURABLE_WORKFLOW_RUNTIME_URL="$RUNTIME_URL" \
  DURABLE_WORKFLOW_NAMESPACE=production \
  DURABLE_WORKFLOW_TASK_QUEUE=orders \
  php artisan config:cache
env -u DURABLE_WORKFLOW_CLIENT_TOKEN \
  DURABLE_WORKFLOW_WORKER_TOKEN="$WORKER_SECRET" \
  php artisan durable-workflow:worker
```

With no `--queue` override, the Artisan worker polls the configured `orders`
task queue. Its registered-and-polling diagnostic names the runtime, namespace,
queue, attributed workflow/activity types, and worker credential role before
normal polling begins.

Application/web/queue processes instead receive
`DURABLE_WORKFLOW_CLIENT_TOKEN` and not the worker token. Self-hosted operators
may use `DURABLE_WORKFLOW_TOKEN` when one credential is intentionally authorized
for both roles.

The SDK worker writes lifecycle, retry, handler-failure, and shutdown
diagnostics through Laravel's PSR logger and dispatches
`DurableWorkflow\Bridge\Event\WorkerDiagnosticEvent` through Laravel events.
Business activities can keep using the injected logger and event dispatcher as
shown above.

The shipped facade fake replaces the same injected interface used by
application code:

<!-- docs-example id="laravel.adoption.service.fake" -->
```php
<?php

use DurableWorkflow\Bridge\Laravel\Facades\DurableWorkflow;

$workflows = DurableWorkflow::fake()
    ->setWorkflowResult('order-1001', ['status' => 'fulfilled']);

app(StartOrderFulfillment::class)->start('1001');
$result = app(StartOrderFulfillment::class)->handle('1001')->result();

$this->assertSame(['status' => 'fulfilled'], $result);
$workflows->assertWorkflowStarted(
    FulfillOrderWorkflow::class,
    ['1001'],
    workflowId: 'order-1001',
);
$workflows->assertResultRequested('order-1001');
```

This fake proves the Laravel application interaction without requiring Cloud or
Server. Use the SDK's worker test harness when testing workflow/activity handler
commands; the client fake does not execute remote worker code.

## Drain, cut over, and roll back

Changing a Composer requirement does not move an in-flight workflow, its
history, a delayed timer, an activity retry, or a ready/reserved queue job.
Treat the old and new runtimes as separate durable systems throughout the
transition.

### 1. Establish the recovery cut

Before changing traffic:

- Inventory the v1/embedded database connection, every configured workflow and
  activity queue, queue backend account/region/prefix, cache locks, and the
  secret-manager reference for the original `APP_KEY`.
- Block new starts and message ingress long enough to quiesce the source. Stop
  every worker after its current job boundary; `queue:restart` alone is not a
  quiesce when a supervisor immediately replaces the process.
- Either drain all source runs, or capture an application-consistent recovery
  cut containing SQL plus every ready, delayed, and reserved queue job. A
  database-only backup does not recreate queue-backed timers, retries, or
  activities.
- Record which runtime owns each nonterminal workflow ID. Signals, queries,
  updates, cancellation, termination, repair, and result reads must continue to
  follow that ownership record.

### 2. Apply the path-specific rule

| Transition | Existing nonterminal runs | New runs after cutover | Rollback boundary |
| --- | --- | --- | --- |
| v1 to 2.0 embedded | Finish through the v1 compatibility path. Retain their v1 tables, PHP decoding requirements, Laravel queue consumers, and queue state until terminal. | Start through `Workflow\V2\WorkflowStub` only after the v2 canary passes. | Restore the matched v1 SQL/queue recovery cut and exact app configuration, or fix forward. Do not restore SQL over newer queue state. |
| v1 to Cloud or self-hosted service mode | Stay on v1. v1 history cannot be imported into the service runtime or replayed by remote workers. | Start with `LaravelWorkflowClientInterface` after a registered SDK worker advertises the expected type keys. | Route new starts back to v1 only if the v1 runtime remains healthy. Runs already accepted by service mode stay there. |
| Embedded v2 to Cloud | Drain embedded runs in place; Cloud onboarding does not turn a package change into history import. | Start in the provisioned Cloud namespace after client/worker credential and task-queue checks pass. | Route new starts back to embedded if needed; keep commands for Cloud-accepted runs pointed at Cloud. |
| Embedded v2 to self-hosted Server | Drain in place, or separately perform the supported export, dry-run, and atomic import for an eligible quiesced embedded-v2 run. Never infer eligibility from package installation. | Start through the SDK after Server discovery, namespace, worker registration, and queue checks pass. | A failed import writes no partial run. After a committed import, Server owns that run; do not resume the embedded copy. |

For the self-hosted import option, use the complete
[embedded-to-Server migration procedure](/docs/polyglot/embedded-to-server/#phase-e-import-eligible-embedded-v2-runs).
It rejects v1, redacted history, leased tasks, running activity attempts, and
other unsafe snapshots. History export by itself is audit/debugging data; only
that explicit validated import operation creates Server-owned state.

### 3. Prove the destination before switching traffic

1. Configure one namespace, task queue, and stable workflow/activity type set.
2. Start the destination runtime and its SDK workers beside the old runtime.
3. Verify Server discovery or Cloud provisioning, role credentials, worker
   registration, advertised type keys, task-queue visibility, and Laravel
   diagnostics.
4. Send a uniquely identified shadow/canary order and wait for a terminal
   result in the destination's operator surface.
5. Switch new starts one workflow family at a time. Prevent the same business
   key from starting in both runtimes.
6. Keep source workers, queue state, credentials, logs/events, and the old
   operator surface until every source-owned run is terminal or has completed a
   supported self-hosted import.

Retiring an old package, queue, table, secret, or Waterline deployment is the
last step, never the cutover mechanism.

## Continue with the chosen path

- [Stable v1 installation](/docs/installation/)
- [Stable v1 migration planning](/docs/migration/)
- [2.0 embedded installation](/docs/installation/)
- [Detailed v1-to-v2 package migration](/docs/migration/)
- [PHP SDK service mode](/docs/polyglot/php/)
- [Cloud managed runtime](/docs/polyglot/cloud-control-plane/)
- [Self-hosted Server](/docs/polyglot/server/)
- [Embedded v2 to self-hosted Server](/docs/polyglot/embedded-to-server/)

<!-- Source: docs/defining-workflows/workflows.md -->

# Workflows

Workflows and activities are defined as classes that extend the base `Workflow` and `Activity` classes provided by the framework. A workflow is a class that defines a sequence of activities that run in parallel, series or a mixture of both.

You may use the `make:workflow` artisan command to create a new workflow:

```php
php artisan make:workflow MyWorkflow
```

It is defined by extending the `Workflow` class and implementing the `handle()` method.

```php
use function Workflow\V2\activity;
use Workflow\V2\Workflow;

class MyWorkflow extends Workflow
{
    public function handle()
    {
        return activity(MyActivity::class);
    }
}
```

<!-- Source: docs/defining-workflows/activities.md -->

# Activities

An activity is a unit of work that performs a specific task or operation (e.g. making an API request, processing data, sending an email) and can be executed by a workflow.

:::note Durable Execution Contract
Ordinary v2 activities run as durable queued tasks. Local activities are an
explicit same-process primitive for short activity work that still records
durable activity history. Worker sessions are available when multiple durable
activity steps need one worker lease, and sticky execution is a supported
replay optimization, not a correctness contract. See [Activity Execution Model](/docs/features/activity-execution-model),
[Local Activities](/docs/features/local-activities), [Worker Sessions](/docs/features/worker-sessions),
and [Sticky Execution](/docs/features/sticky-execution) for the exact
contracts.
:::

You may use the `make:activity` artisan command to create a new activity:

```php
php artisan make:activity MyActivity
```

It is defined by extending the `Activity` class and implementing the `handle()` method.

```php
use Workflow\V2\Activity;

class MyActivity extends Activity
{
    public function handle()
    {
        // Perform some work...
        return $result;
    }
}
```

## Execution Contract

Every `activity(...)` call records an activity command on workflow history and
creates a durable queued task for a worker to claim. Ordinary activities may
be claimed by any compatible worker, while [worker sessions](/docs/features/worker-sessions)
can pin a sequence of activity attempts to one worker-session lease when the
workflow explicitly opts into that contract.

If you need a replay-safe value without scheduling queued work, use
[`sideEffect(...)`](/docs/features/side-effects) instead. If you need a
short retryable same-process activity with timeout, heartbeat, and activity
history semantics, use [Local Activities](/docs/features/local-activities).
For the full placement model, see [Activity Execution Model](/docs/features/activity-execution-model).
For sticky replay-cache behavior, see
[Sticky Execution](/docs/features/sticky-execution).

## Idempotency and Durable Identity

Activity execution is at-least-once. Retries, lease expiry, and redelivery can
cause the same logical activity to be observed more than once, so the side
effect or remote target must be safe to repeat.

Inside the activity, use the runtime's durable identifiers when you need
correlation or remote dedupe:

- `activityId()` identifies one logical activity execution across retries and
  is the default remote idempotency key.
- `attemptId()` identifies one concrete try of that execution.
- `attemptCount()` tells you which try is currently running.

Prefer `activityId()` when the remote system should treat retries as the same
logical request. Reach for `attemptId()` only when the remote system truly
needs to distinguish separate tries. If a worker finishes remote work, loses
its lease, and reports late, the engine may reject that late completion because
another worker already won the durable race, but the remote side effect may
still have happened.

See [Execution Guarantees and Idempotency](/docs/constraints/execution-guarantees),
[Heartbeats](/docs/features/heartbeats), and
[Failures and Recovery](/docs/failures-and-recovery) for the operational
model behind those identifiers.

## Per-Call Overrides

Routing and retries default to the activity class's own `$connection`, `$queue`, `$tries`, and `backoff()` properties. When a single call needs to override those — for example, routing one call to a higher-priority queue or giving it more retry attempts — pass an `ActivityOptions` instance:

```php
use function Workflow\V2\activity;
use Workflow\V2\Support\ActivityOptions;

$result = activity(
    MyActivity::class,
    new ActivityOptions(queue: 'high-priority', maxAttempts: 5),
    'Taylor',
);
```

See [Activity options](/docs/configuration/options#activityoptions) for the full list of fields, including timeouts and heartbeats.

## Local Activity Calls

Use `localActivity(...)` when a short idempotent side effect should run inside
the current workflow worker process but still needs activity retry, timeout,
heartbeat, cancellation, and history semantics:

```php
use Workflow\V2\Support\LocalActivityOptions;
use function Workflow\V2\localActivity;

$receipt = localActivity(
    SendReceiptActivity::class,
    new LocalActivityOptions(maxAttempts: 3, startToCloseTimeout: 10),
    $orderId,
);
```

Local activities reject `connection`, `queue`, worker-session routing, and
schedule-to-start options because they do not create ordinary activity tasks.
See [Local Activities](/docs/features/local-activities) for the complete
contract.

<!-- Source: docs/defining-workflows/starting-workflows.md -->

# Starting Workflows

To start a workflow, create a workflow instance and then call the `start()` method on it.

```php
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::make(MyWorkflow::class);
$workflow->start();
```

Once a workflow has been started, it will be executed asynchronously by a queue worker. The `start()` method returns immediately and does not block the current request.

Pass arguments to the workflow's `handle()` method through `start()`:

```php
$workflow->start($orderId);
```

You can attach visibility labels, a business key, memo, or timeouts through `StartOptions`. See [Start Options](/docs/configuration/options#startoptions) when you need them.

<!-- Source: docs/defining-workflows/passing-data.md -->

# Passing Data

You can pass data into a workflow via the `start()` method.

```php
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::make(MyWorkflow::class);
$workflow->start('world');
```

Arguments are passed to the workflow's `handle()` method.

Similarly, you can pass data into an activity via the `activity()` helper function.

```php
use function Workflow\V2\activity;
use Workflow\V2\Workflow;

class MyWorkflow extends Workflow
{
    public function handle($name)
    {
        return activity(MyActivity::class, $name);
    }
}
```

Arguments are passed to the activity's `handle()` method.

```php
use Workflow\V2\Activity;

class MyActivity extends Activity
{
    public function handle($name)
    {
        return "Hello, {$name}!";
    }
}
```

In general, you should only pass small amounts of data in this manner. Rather than passing large amounts of data, you should write the data to the database, cache or file system. Then pass the key or file path to the workflow and activities. The activities can then use the key or file path to read the data.

When the application genuinely needs to carry large bytes through history — documents, media blobs, serialized exports — enable [External Payload Storage](../features/external-payload-storage.md) on the namespace. The runtime offloads over-threshold payloads to a configured object store and records a verifiable reference envelope in history, keeping replay integrity while staying under the [`payload_size_bytes` structural limit](../constraints/structural-limits.md#payload-size).

## Output

Once the workflow has completed, you can retrieve the output using the `output()` method.

```php
$workflow->output();
=> 'Hello, world!'
```

## Models

Passing in models works similarly to `SerializesModels`.

```php
use App\Models\User;
use function Workflow\V2\activity;
use Workflow\V2\Workflow;

class MyWorkflow extends Workflow
{
    public function handle(User $user)
    {
        return activity(MyActivity::class, $user->name);
    }
}
```

When an Eloquent model is passed to a workflow or activity, only its `ModelIdentifier` is serialized. This reduces the size of the payload, ensuring that your workflows remain efficient and performant.

```
object(ModelIdentifier) {
    id: 42,
    class: "App\Models\User",
    relations: [],
    connection: "mysql"
}
```

When the workflow or activity runs, it will retrieve the complete model instance, including any loaded relationships, from the database. If you wish to prevent extra database calls during the execution of a workflow or activity, consider converting the model to an array before passing it.

## Dependency Injection

In addition to passing data, you are able to type-hint dependencies on the workflow or activity `handle()` methods. The Laravel service container will automatically inject those dependencies.

```php
use Illuminate\Contracts\Foundation\Application;
use Workflow\V2\Workflow;

class MyWorkflow extends Workflow
{
    public function handle(Application $app)
    {
        if ($app->runningInConsole()) {
            // ...
        }
    }
}
```

Dependency injection does not relax workflow determinism. A workflow may
type-hint stable services such as the Laravel application container or
configuration readers, but injected dependencies that do work still need the
right durable boundary. Put external I/O, database access, service calls, or
other work that can fail or should retry in `activity(...)` or
`localActivity(...)`. Use `sideEffect()` only for a one-time replay-safe value
snapshot that should be frozen in history, such as a wall-clock read, generated
randomness, or a config value captured at decision time.

<!-- Source: docs/defining-workflows/workflow-status.md -->

# Workflow Status

You can monitor the status of the workflow by calling the `running()` method, which returns `true` if the workflow is still running and `false` if it has completed or failed.

```php
while ($workflow->running());
```

## Status Values

The `status()` method returns string statuses:

```text
reserved
pending
running
waiting
cancelled
terminated
completed
failed
```

- `reserved` means an instance id has been created but the first start command has not been accepted yet
- `pending` means the run exists and has work ready to be claimed
- `running` means a workflow task is actively leased to a worker
- `waiting` means the run is blocked on a durable resume source such as an activity, timer, or named signal
- `cancelled` means an accepted engine-level cancel command closed the current run
- `terminated` means an accepted engine-level terminate command force-closed the current run
- `completed`, `failed`, `cancelled`, and `terminated` are terminal run states

When a run uses `continueAsNew()`, the old run ends with `status = completed` and `closed_reason = continued`. Waterline keeps that run in the completed bucket while still surfacing the exact `closed_reason` so operators can see that the instance rolled forward into a newer run.

`running()` returns `true` for `pending`, `running`, and `waiting`.

## State Machine

This is the state machine for a workflow status.

<!-- Source: docs/defining-workflows/workflow-api.md -->

# Workflow API

This page is the complete reference. For an easier to read version, please look at the individual feature page.

- [Workflows](./workflows.md) for the workflow class shape and deterministic
  orchestration model.
- [Activities](./activities.md) for side effects, retries, and routing.
- [Local Activities](../features/local-activities.md) for short same-process
  activity work with activity history and retry semantics.
- [Signals](../features/signals.md), [Updates](../features/updates.md), and
  [Queries](../features/queries.md) for workflow-facing contracts.
- [Timers](../features/timers.md),
  [Condition Waits](../features/condition-waits.md), and
  [Continue As New](../features/continue-as-new.md) for long-running control
  flow.
- [Message Streams](../features/message-streams.md) for repeated ordered
  messages with cursor semantics.

## More Info for AI

Most readers can skip the disclosure below. Open it when you need exact
signatures, return contracts, machine-operable notes, or the full API
surface in one place.

## More Info for AI

**Base workflow object**

```php
use Workflow\V2\Workflow;

abstract class Workflow
{
    public ?string $connection = null;
    public ?string $queue = null;

    public function workflowId(): string;
    public function runId(): string;
    public function lastChild(): ?ChildWorkflowHandle;
    public function children(): array;
    public function historyLength(): int;
    public function historySize(): int;
    public function shouldContinueAsNew(): bool;
}
```

| Member | Use when | Return contract |
| --- | --- | --- |
| `workflowId()` | The workflow needs its stable public instance id. | Instance id string, unchanged across continue-as-new. |
| `runId()` | The workflow needs the currently executing run id. | Run id string for the selected execution. |
| `lastChild()` | The workflow needs to signal the most recently spawned child. | `ChildWorkflowHandle` or `null`. |
| `children()` | The workflow needs handles for children visible to the current replay sequence. | List of `ChildWorkflowHandle`. |
| `historyLength()` | The workflow needs a count-based history budget signal. | Current history event count. |
| `historySize()` | The workflow needs a byte-based history budget signal. | Approximate persisted history size in bytes. |
| `shouldContinueAsNew()` | The workflow should rotate before history becomes expensive. | `true` when configured history budgets recommend rotation. |

## Durable commands

The static facade delegates to namespaced helpers in `Workflow\V2`. The two
forms are equivalent:

```php
use Workflow\V2\Workflow;
use function Workflow\V2\activity;

$resultFromFacade = Workflow::activity(SendReceipt::class, $orderId);
$resultFromHelper = activity(SendReceipt::class, $orderId);
```

<span id="durable-command-surface"></span>

| Facade | Helper | Signature | Durable effect |
| --- | --- | --- | --- |
| `Workflow::activity()` | `activity()` | `activity(string $activity, mixed ...$arguments): mixed` | Schedules an activity and waits for its result. |
| `Workflow::executeActivity()` | `activity()` | `executeActivity(string $activity, mixed ...$arguments): mixed` | Alias for `activity()`. |
| `Workflow::localActivity()` | `localActivity()` | `localActivity(string $activity, mixed ...$arguments): mixed` | Runs a short activity in the current workflow worker process and records activity history with `execution_mode=local`. |
| `Workflow::executeLocalActivity()` | `localActivity()` | `executeLocalActivity(string $activity, mixed ...$arguments): mixed` | Alias for `localActivity()`. |
| `Workflow::child()` | `child()` | `child(string $workflow, ChildWorkflowOptions? $options = null, mixed ...$arguments): mixed` | Starts a child workflow and waits for its result. Pass a `ChildWorkflowOptions` as the first argument to set the parent-close policy (default `ParentClosePolicy::Abandon`) or override child routing. |
| `Workflow::executeChildWorkflow()` | `child()` | `executeChildWorkflow(string $workflow, ChildWorkflowOptions? $options = null, mixed ...$arguments): mixed` | Alias for `child()`. |
| `Workflow::async()` | `async()` | `async(callable $callback): mixed` | Runs a callable as an auto-generated child workflow. |
| `Workflow::all()` | `all()` | `all(iterable $calls): mixed` | Waits for concurrent calls and returns results in iteration order. |
| `Workflow::parallel()` | `all()` | `parallel(iterable $calls): mixed` | Alias for `all()`. |
| `Workflow::select()` | `select()` | `select(iterable $calls): SelectionResult` | Starts independent durable calls and returns the first committed winner plus stable handles for every member. |
| `Workflow::await()` | `await()` | `await(callable\|string $condition, int\|string\|CarbonInterval\|null $timeout = null, ?string $conditionKey = null): mixed` | Waits for a named signal or replay-safe condition. |
| `Workflow::awaitWithTimeout()` | `await()` | `awaitWithTimeout(int\|string\|CarbonInterval $timeout, callable\|string $condition, ?string $conditionKey = null): mixed` | Waits for a signal or condition with an explicit timeout. |
| `Workflow::awaitSignal()` | `await()` | `awaitSignal(string $name): mixed` | Waits for a named signal. |
| `Workflow::timer()` | `timer()` | `timer(int\|string\|CarbonInterval $duration): mixed` | Suspends until durable time advances. |
| `Workflow::sideEffect()` | `sideEffect()` | `sideEffect(callable $callback): mixed` | Records a non-deterministic result in history and replays it. |
| `Workflow::uuid4()` | `uuid4()` | `uuid4(): mixed` | Generates a replay-stable UUIDv4. |
| `Workflow::uuid7()` | `uuid7()` | `uuid7(): mixed` | Generates a replay-stable UUIDv7. |
| `Workflow::continueAsNew()` | `continueAsNew()` | `continueAsNew(mixed ...$arguments): mixed` | Ends the current run and starts a new run for the same instance. |
| `Workflow::getVersion()` | `getVersion()` | `getVersion(string $changeId, int $minSupported = WorkflowStub::DEFAULT_VERSION, int $maxSupported = 1): mixed` | Negotiates a replay-safe workflow-code version. |
| `Workflow::patched()` | `patched()` | `patched(string $changeId): mixed` | Returns whether the run crossed a named patch marker. |
| `Workflow::deprecatePatch()` | `deprecatePatch()` | `deprecatePatch(string $changeId): mixed` | Keeps a patch marker alive after old code is removed. |
| `Workflow::upsertMemo()` | `upsertMemo()` | `upsertMemo(array $entries): void` | Updates non-indexed run metadata. |
| `Workflow::upsertSearchAttributes()` | `upsertSearchAttributes()` | `upsertSearchAttributes(array $attributes): void` | Updates indexed operator-visible metadata. |
| `Workflow::now()` | `now()` | `now(): CarbonInterface` | Reads deterministic workflow time. |

`activity()` and `executeActivity()` schedule durable queued activity tasks.
`localActivity()` and `executeLocalActivity()` run in the current workflow
worker process and record normal activity history with the local marker. Use
`sideEffect()` for replay-safe snapshots that should not use activity retry,
timeout, heartbeat, or cancellation semantics. Use `Workflow::workerSession()`
or `Workflow\V2\workerSession()` when multiple ordinary activity steps need
the supported worker-session affinity contract. See
[Activity Execution Model](/docs/features/activity-execution-model),
[Local Activities](/docs/features/local-activities), and
[Worker Sessions](/docs/features/worker-sessions) for the full execution
contract.

**Timer helpers**

Timer helpers are shorthand for `timer()` and `Workflow::timer()`:

```php
use Workflow\V2\Workflow;

Workflow::seconds(30);
Workflow::minutes(5);
Workflow::hours(2);
Workflow::days(1);
Workflow::weeks(1);
Workflow::months(1);
Workflow::years(1);
```

| Helper | Equivalent |
| --- | --- |
| `seconds(int $seconds)` | `timer($seconds)` |
| `minutes(int $minutes)` | `timer($minutes * 60)` |
| `hours(int $hours)` | `timer($hours * 3600)` |
| `days(int $days)` | `timer($days * 86400)` |
| `weeks(int $weeks)` | `timer($weeks * 604800)` |
| `months(int $months)` | `timer("{$months} months")` |
| `years(int $years)` | `timer("{$years} years")` |

Use explicit `timer()` calls when the duration comes from configuration or
workflow input. Use timer helpers when the source code should read as a fixed
business wait.

**Message streams**

Open durable message streams from the workflow instance:

```php
use Workflow\V2\MessageStream;
use Workflow\V2\Workflow;

final class AssistantWorkflow extends Workflow
{
    public function handle(string $targetWorkflowId): array
    {
        $message = $this->inbox('ai.user')->receiveOne();

        if ($message === null) {
            return ['status' => 'waiting'];
        }

        $reply = $this->outbox('ai.assistant')->sendReference(
            targetInstanceId: $targetWorkflowId,
            payloadReference: 'app://payloads/reply-123',
            correlationId: $this->workflowId(),
            idempotencyKey: 'reply-123',
            metadata: ['kind' => 'assistant_reply'],
        );

        return [
            'status' => 'sent',
            'stream' => $reply->stream_key,
            'sequence' => $reply->sequence,
        ];
    }
}
```

| Method | Signature | Contract |
| --- | --- | --- |
| `$this->messages()` | `messages(?string $streamKey = null, ?MessageService $messages = null): MessageStream` | Opens the stream for reading or sending. |
| `$this->inbox()` | `inbox(?string $streamKey = null, ?MessageService $messages = null): MessageStream` | Alias for inbound authoring code. |
| `$this->outbox()` | `outbox(?string $streamKey = null, ?MessageService $messages = null): MessageStream` | Alias for outbound authoring code. |
| `MessageStream::key()` | `key(): string` | Returns the stream key. |
| `MessageStream::cursor()` | `cursor(): int` | Returns the durable cursor position for this run. |
| `MessageStream::hasPending()` | `hasPending(): bool` | Returns whether unconsumed messages exist on the stream. |
| `MessageStream::pendingCount()` | `pendingCount(): int` | Returns the number of unconsumed messages on the stream. |
| `MessageStream::peek()` | `peek(int $limit = 100): Collection` | Reads pending messages without consuming them. |
| `MessageStream::receive()` | `receive(int $limit = 1, ?int $consumedBySequence = null): Collection` | Reads and consumes messages, recording cursor advancement. |
| `MessageStream::receiveOne()` | `receiveOne(?int $consumedBySequence = null): ?WorkflowMessage` | Reads and consumes one message. |
| `MessageStream::sendReference()` | `sendReference(string $targetInstanceId, ?string $payloadReference = null, MessageChannel\|string $channel = MessageChannel::WorkflowMessage, ?string $correlationId = null, ?string $idempotencyKey = null, array $metadata = [], ?DateTimeInterface $expiresAt = null): WorkflowMessage` | Sends an ordered payload-reference message to another workflow instance. |

Use message streams for repeated ordered messages with cursor semantics. Use
[Signals](../features/signals.md) for one-shot external events and
[Updates](../features/updates.md) for request/return mutations.

**Attributes and public contracts**

```php
use Workflow\QueryMethod;
use Workflow\UpdateMethod;
use Workflow\V2\Attributes\Signal;
use Workflow\V2\Attributes\Type;
use Workflow\V2\Workflow;

#[Type('order-approval')]
#[Signal('approved-by', [
    ['name' => 'approvedBy', 'type' => 'string', 'allows_null' => false],
])]
final class OrderApprovalWorkflow extends Workflow
{
    private string $stage = 'waiting';

    public function handle(): void
    {
        $this->stage = Workflow::awaitSignal('approved-by');
    }

    #[QueryMethod('current-stage')]
    public function currentStage(): string
    {
        return $this->stage;
    }

    #[UpdateMethod('mark-ready')]
    public function markReady(): string
    {
        return $this->stage = 'ready';
    }
}
```

| Attribute | Target | Stable contract |
| --- | --- | --- |
| `#[Type('type-key')]` | Workflow or activity class | Declares the language-neutral durable type key. |
| `#[Signal('signal-name', [...])]` | Workflow class, repeatable | Declares accepted signal names and optional ordered parameter contracts. |
| `#[QueryMethod('query-name')]` | Workflow method | Declares a replay-safe query name. Omit the name to use the PHP method name. |
| `#[UpdateMethod('update-name')]` | Workflow method | Declares a replay-safe update name. Omit the name to use the PHP method name. |

Signals, queries, and updates are public workflow contracts. Prefer explicit
names so PHP method renames do not become API breaks.

**Failure surface**

Authoring API failures are durable workflow failures unless the command is
rejected before execution:

| Surface | Typical failure | Operator meaning |
| --- | --- | --- |
| `activity()` | Activity throws, times out, or exhausts retry policy. | The run records activity failure history and follows workflow error handling. |
| `child()` | Child workflow fails, cancels, terminates, or times out. | The parent observes a child failure outcome at the waiting command. |
| `await()` | Timeout elapses before the condition or signal is satisfied. | The wait returns or fails according to the selected await form. |
| `timer()` | Invalid duration input after normalization. | Authoring code should pass positive durations or explicit zero-duration waits. |
| `continueAsNew()` | New run cannot be created. | Current run remains the evidence point for the failed transition. |
| `upsertSearchAttributes()` | Attribute key, count, or total size exceeds limits. | The run fails before invalid indexed metadata is persisted. |
| `MessageStream::receive()` | No positive workflow history sequence is available. | Receive must occur from workflow execution, not direct service code. |
| `MessageStream::sendReference()` | Payload reference, route, or storage contract is invalid downstream. | Message ordering remains separate from payload-store integrity. |

For payload and history limits, see [Structural Limits](../constraints/structural-limits.md).
For command rejection responses outside PHP workflow code, see
[Server API Reference](../polyglot/server-api-reference.md).

**Determinism rules**

Workflow code must be replay-safe. Keep irreversible or non-deterministic work
behind durable commands:

```php
use Workflow\V2\Workflow;

final class DeterministicWorkflow extends Workflow
{
    public function handle(): array
    {
        $workflowTime = Workflow::now();
        $stableId = Workflow::uuid7();
        $remoteQuote = Workflow::activity(FetchQuote::class);

        return [
            'time' => $workflowTime->toIso8601String(),
            'id' => $stableId,
            'quote' => $remoteQuote,
        ];
    }
}
```

- Use `Workflow::now()` instead of wall-clock time in workflow branches.
- Use `Workflow::uuid4()` or `Workflow::uuid7()` instead of direct randomness.
- Put network calls, filesystem writes, email sends, and external side effects
  in activities.
- Use `Workflow::sideEffect()` only when the value must be captured in history
  and the side effect itself is not the business action.
- Use `Workflow::getVersion()`, `Workflow::patched()`, and
  `Workflow::deprecatePatch()` to evolve workflow code without breaking replay.

<!-- Source: docs/defining-workflows/workflow-id.md -->

# Workflow ID

## Instance IDs and Run IDs

A durable workflow has two identifiers with different lifecycles:

- The **instance id** (`id()`, `workflowId()`) is the stable public handle for the workflow. It is generated when the workflow is first started — or supplied by the caller — and never changes for the lifetime of that workflow, including across continue-as-new chains.
- The **run id** (`runId()`) identifies one concrete execution generation of that workflow. It is newly allocated every time a new run is created, which happens on the first start and when the workflow continues as new.

Treat the instance id as the public, callable handle. Treat the run id as a pointer to a specific generation that is useful for inspection, debugging, or audit — not as something long-lived control-plane callers should track.

```php
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::make(MyWorkflow::class);

$id = $workflow->id();

$workflow->start();

$runId = $workflow->runId();
```

Use the id when you want the stable public handle for the workflow. Use the run id when you need to inspect one concrete execution.

## Run ID Lifecycle

A run id is an execution-generation identifier. Every v2 workflow generation has its own run id, and a single instance id can span many generations over time:

- A fresh `start()` creates the first run and its first run id.
- `continueAsNew(...)` closes the current run as `completed` / `continued` and starts a new run with a fresh run id under the same instance id. See [Continue As New](/docs/features/continue-as-new).

Because a run id belongs to one generation, callers that want durable behavior across those transitions should resolve by instance id:

- Signals, queries, updates, and operator commands always resolve the newest durable run for an instance. Callers do not need to track run ids across continue-as-new boundaries.
- Drill-down tools that pin a specific generation — for example Waterline's run-detail view, a history export, or a replay debugger — accept an explicit run id so they can inspect one execution in isolation.

The run id surfaces as `runId()` inside workflow code and activities, and as the `workflow_run_id` / `run_id` field on history, projections, and exports. It is always the run id of the generation that owns the event or row. Do not expect a run id recorded on older history events to equal the current run id of the same instance.

## User-defined Instance IDs

When starting a workflow you may optionally specify the id. User-defined ids must be URL-safe strings up to 191 characters using only letters, numbers, `.`, `_`, `-`, and `:`.

```php
$workflow = WorkflowStub::make(MyWorkflow::class, 'order-123');
```

Later when you want to reference the workflow, you can load it with your custom id.

```php
$workflow = WorkflowStub::load('order-123');
```

## Accessing IDs Inside Activities and Workflows

Inside an activity:

```php
use Workflow\V2\Activity;

class MyActivity extends Activity
{
    public function handle(): void
    {
        $id = $this->workflowId();
        $runId = $this->runId();
    }
}
```

Inside a workflow, `$this->workflowId()` returns the instance id and `$this->runId()` returns the selected run id for the currently executing run.

<!-- Source: docs/constraints/overview.md -->

# Overview

Workflows and activities live on opposite sides of the durable boundary and carry different constraints. Workflow code is **replayed**; activity code is **retried**. Those are not the same operation, and the rules flow from that split.

Start with [Idempotent vs. Deterministic Workflows](/docs/constraints/idempotent-vs-deterministic/) for a direct comparison and concrete examples showing why neither property implies the other.

- **Workflow authoring code must be deterministic.** The engine replays history to rebuild a workflow's state whenever it resumes — on another worker, after a restart, after a deployment, or during a long-running execution. Replay re-invokes the workflow body, not activities, so the workflow must produce the same decisions in the same order every time it sees the same history. Wall-clock reads, live cache reads, random numbers, network calls, and other sources of change are prohibited inside the workflow body; use [`Workflow::now()`](../defining-workflows/workflow-api.md), [`sideEffect(...)`](../features/side-effects.md), activities, and similar helpers to cross the durable boundary.

- **Activity code must be idempotent.** Activity attempts are **at-least-once**. Retries, lease expiry, and redelivery can all cause the same logical work to be observed more than once, and that is first-class behavior — not a bug condition. The framework records at most one terminal outcome per attempt at the durable state layer, but an activity body may start executing more than once before the engine sees the winning report. Treat repeat observation as normal and use an idempotency key, a deterministic target resource, or a naturally idempotent operation when the external side effect must not duplicate.

- **Event sourcing persists history, not rerun side effects.** The engine writes each durable step — activity completion, timer fired, signal received, side effect recorded — as a typed history event. Replay reads that history and hands cached results back to the workflow body; it does not re-dispatch activities, timers, or signals. Durable state events for a given identifier are exactly-once at the history layer even when the transport behind them delivered work more than once.

Together, determinism and idempotency let the engine resume workflows that span deployments, worker restarts, and distributed retries without losing place and without duplicating external side effects the application has already made safe to repeat.

See [Execution Guarantees and Idempotency](./execution-guarantees.md) for the
public v2 contract around replay, redelivery, lease expiry, and exactly-once
durable history. Then use [Workflow Constraints](./workflow-constraints.md)
for the specific authoring rules that keep replay deterministic, and
[Activity Constraints](./activity-constraints.md) for the idempotency guidance
that keeps at-least-once activity execution safe.

<!-- Source: docs/constraints/idempotent-vs-deterministic.md -->

# Idempotent vs. Deterministic Workflows: What's the Difference?

**Deterministic behavior makes the same decisions when it is given the same inputs and history. Idempotent behavior has the same intended effect whether an operation runs once or several times.** Determinism makes one execution predictable; idempotency makes repeated executions safe. Neither property implies the other.

| Question | Deterministic | Idempotent |
| --- | --- | --- |
| What stays the same? | The decisions made from the same inputs and history | The intended external effect after one or many calls |
| Why does Durable Workflow need it? | Replay must reconstruct the same workflow path | Retries and redelivery must not duplicate side effects |
| Where does it belong? | Workflow and orchestration code | Activities and the systems they call |
| Does it prohibit IO or randomness? | Yes, inside replayed workflow code unless the result is recorded | No; an activity may use IO, time, or randomness and still make its effect safe to repeat |

## The Properties Are Independent

### Deterministic but not idempotent

Consider an operation that increments a counter:

```text
increment(counter):
    counter.value = counter.value + 1
```

Given the same starting counter value, this operation always makes the same calculation, so it is deterministic. It is not idempotent: calling it twice increments the counter twice, while calling it once increments it once.

### Idempotent but not deterministic

Now consider an activity that sets an order to a known state but uses randomized backoff while contacting the service:

```text
archive(order_id):
    wait(random_backoff())
    PUT /orders/{order_id}/status {"status": "archived"}
```

The timing and number of internal attempts can differ, so this implementation is not deterministic. The operation can still be idempotent: after one successful call or several, the order is archived. The response, latency, and internal path do not have to be identical; the intended external effect does.

## Why Workflow Code Must Be Deterministic

Durable Workflow rebuilds orchestration state by replaying committed history. With the same workflow input and the same history, workflow code must schedule the same activities, timers, and waits in the same order. A wall-clock read, random value, live database query, or network response inside the workflow body can choose a different branch during replay and make the new decisions disagree with history.

Move those operations into activities, or use a workflow helper that records the value in history. Replay then returns the recorded activity or side-effect result instead of performing the external operation again. Determinism does not require every activity invocation to produce a universal, timeless answer; it requires replayed orchestration to make decisions that agree with its recorded history.

See [Workflow Constraints](./workflow-constraints.md) for the authoring rules and [How It Works](../how-it-works.md) for the replay model.

## Why Activities and Side Effects Must Be Idempotent

Activities cross the durable boundary into databases, payment APIs, email providers, object storage, and other systems. They can be delivered again after a timeout, retry, worker failure, or expired lease. A worker may even complete the external change and then fail before Durable Workflow receives its report.

Design the activity's intended effect so another attempt is safe:

- send a stable idempotency key to the remote API
- upsert by a durable business or execution identifier
- write to a deterministic object or resource name
- use a naturally idempotent operation such as setting a value or deleting a known resource

An idempotency key does not make activity code deterministic, and it does not prevent another attempt from starting. It lets the external system recognize that repeated attempts represent the same logical operation.

See [Activity Constraints](./activity-constraints.md), [Activity Execution Model](../features/activity-execution-model.md), and [Execution Guarantees and Idempotency](./execution-guarantees.md) for the retry, redelivery, and durable-history contracts.

## One Boundary, Across Every SDK

The same design applies whether the worker is written in PHP, Python, or Rust:

1. Keep orchestration replay-safe: given the same input and history, issue the same durable commands.
2. Put network calls, clocks, randomness, and mutable external state behind an activity boundary.
3. Give each side-effecting activity a stable identity that the target system can deduplicate.

The language syntax changes. The determinism-versus-idempotency boundary does not.

<!-- Source: docs/constraints/workflow-constraints.md -->

# Workflow Constraints

The determinism constraints for workflow classes dictate that a workflow class must not depend on external state or services that may change over time. This means that a workflow class should not perform any operations that rely on the current date and time, the current user, external network resources, or any other source of potentially changing state.

Here are some examples of things you shouldn't do inside of a workflow class:

- Don't use the `Carbon::now()` method to get the current date and time, as this will produce different results each time it is called. Instead, use `Workflow\V2\Workflow::now()` (or the `Workflow\V2\now()` helper), which returns replay-safe workflow time.
- Don't use the `Auth::user()` method to get the current user, as this will produce different results depending on who is currently logged in. Instead, pass the user as an input to the workflow when it is started.
- Don't make network requests to external resources, as these may be slow or unavailable at different times. Instead, pass the necessary data as inputs to the workflow when it is started or use an activity to retrieve the data.
- Don't use random number generators (unless using a side effect) or other sources of randomness, as these will produce different results each time they are called. Instead, pass any necessary randomness as an input to the workflow when it is started.

## Boot-Time Guardrails

When workflow classes are registered under `workflows.v2.types.workflows`, the package scans them at boot time for obvious replay-unsafe calls (`Carbon::now()`, `Auth::user()`, `DB::`, `Http::`, `random_int()`, and similar) and surfaces findings according to `workflows.v2.guardrails.boot`:

| Mode | Behavior |
|------|----------|
| `warn` (default) | Logs a warning for each detected finding. Does not block application boot. |
| `silent` | Skips boot-time scanning entirely. |
| `throw` | Throws a `LogicException` on the first finding. Useful for CI pipelines. |

```php
// config/workflows.php
'v2' => [
    'guardrails' => [
        'boot' => env('DW_V2_GUARDRAILS_BOOT', 'warn'),
    ],
],
```

Set `DW_V2_GUARDRAILS_BOOT=throw` in CI to fail builds that introduce new replay-unsafe calls; keep `warn` in production so rollouts are not blocked by a latent finding.

First-release scope: boot-time scanning is the only blocking workflow-mode guardrail. The runtime does not rerun determinism diagnostics again at workflow-task claim time. That deferral is intentional for 2.0: boot scanning catches locally registered PHP workflows before rollout, and Waterline surfaces definition-fingerprint drift for long-lived runs without turning cross-build claims into a new source of deploy failures.

Pre-fingerprint runs also follow a conservative first-release policy. If a run reaches a new `getVersion()` branch and its `WorkflowStarted` history predates the fingerprint snapshot, the runtime keeps that run on `WorkflowStub::DEFAULT_VERSION` instead of assuming the current definition is safe. See [Versioning](../features/versioning.md).

<!-- Source: docs/constraints/activity-constraints.md -->

# Activity Constraints

Activities are where workflow code crosses the durable boundary into IO and
side effects. They are not replayed like workflow code. They are **executed
at-least-once**, which means retries, lease expiry, and redelivery can all
cause the same logical work to be observed more than once.

That behavior is first-class, not an error path. If your activity creates a
charge, sends an email, writes to another system, or mutates an external
resource, the operation must be safe to repeat.

## What This Means In Practice

- The default idempotency surface for one logical activity execution is
  `activity_execution_id`.
- Each retry attempt also gets its own `activity_attempt_id`.
- A worker can finish external work, lose its lease, and then report late.
  The engine may reject that late report because another worker already won
  the durable race, but the remote side effect may still have happened.
- Activity code is allowed to use IO, wall-clock time, and mutable process
  state. Workflow code is not. Keep that boundary clear.

## Preferred Idempotency Patterns

Many external APIs support passing an `Idempotency-Key`. Use the workflow
runtime's logical activity identity when the remote service supports it.

- Prefer `activity_execution_id` when the remote system should treat retries
  as the same logical request.
- Use `activity_attempt_id` only when the remote system must distinguish
  separate tries of the same logical work.

Other good patterns:

- Write to a deterministic external resource name or natural key.
- Use upserts or dedupe tables keyed by a durable identifier.
- Make the operation naturally idempotent so the second call becomes a no-op.

Many operations are naturally idempotent. If you encode a video twice, you
still end up with the same video. If you delete the same file twice, the
second deletion does nothing.

Some operations are not inherently idempotent, but duplication may still be
the safer failure mode. If you are unsure whether an email actually left the
provider, a duplicate email may be preferable to silently dropping the
notification. Make that trade-off deliberately.

## What Not To Assume

- Do not assume one activity attempt only ever runs on one worker.
- Do not assume a retry means the previous external side effect failed.
- Do not assume late completion implies the activity never ran.
- Do not move side effects into workflow code to avoid retries; that only
  turns an idempotency problem into a determinism bug.

See [Execution Guarantees and Idempotency](./execution-guarantees.md) for the
full contract and [Failures and Recovery](../failures-and-recovery.md) for the
operator-facing recovery model.

<!-- Source: docs/constraints/execution-guarantees.md -->

# Execution Guarantees and Idempotency

Durable Workflow v2 draws a hard line between **workflow replay** and
**activity execution**:

- Workflow code is replayed from committed history and must be deterministic.
- Activity code performs side effects and is **at-least-once**.
- The durable history layer records committed workflow and activity outcomes
  exactly once for a given durable identifier, even when transport delivered
  work more than once.

Those guarantees are what let the engine survive worker restarts, lease
expiry, queue redelivery, and rolling deploys without losing the run's place.

## Replay Is Not Retry

Workflow tasks rebuild state by replaying committed history and then deciding
what to do next. Replay re-invokes the workflow body, but it does **not**
re-run activities, re-send signals, or repeat side effects that were already
recorded on history.

That is why workflow authoring code must stay deterministic. Use workflow-safe
helpers such as [`Workflow::now()`](../defining-workflows/workflow-api.md),
[`sideEffect(...)`](../features/side-effects.md), queries, updates, activity
results, memos, and search attributes when you need to cross the durable
boundary.

## Activity Execution Is At-Least-Once

Activities are the side-effecting part of the system, so the contract is
different:

- An activity attempt can be claimed more than once.
- Lease expiry can cause redelivery to another worker.
- A worker can finish external work, lose its lease, and still report late.
- A retry schedules a new durable attempt for the same logical activity
  execution.

Duplicate observation is therefore **first-class behavior, not a bug
condition**. The application author must make the activity body or the remote
system it calls safe to repeat.

See [Activity Constraints](./activity-constraints.md) for authoring guidance
and [Failures and Recovery](../failures-and-recovery.md) for operator-facing
recovery behavior.

## What Is Exactly Once

Durable Workflow does **not** promise that a worker process sees side-effecting
work only once. It does promise that committed durable facts are authoritative
and do not duplicate for the same durable identifier.

In practice that means:

- A committed workflow decision is persisted once in typed history for the
  durable command or step id it represents.
- A committed terminal outcome for one activity attempt is persisted once for
  that `activity_attempt_id`.
- Replay reads those committed facts back and rebuilds workflow state from
  them instead of re-running external work.

That split is the core mental model:

- **Transport and workers are at-least-once.**
- **Committed durable history is exactly-once per durable identifier.**

## Lease Expiry and Redelivery

Lease expiry is a normal distributed-systems recovery path:

- A claimed task carries a lease owner and expiry time.
- If the lease expires before the worker reports progress or completion, the
  task becomes eligible for redelivery.
- A different worker may then claim the same logical work.

Redelivery does not mean the engine forgot what already committed. It means
the engine is recovering from uncertainty at the worker or transport layer.

When you see duplicate execution symptoms, ask two questions separately:

1. Did the side effect happen more than once?
2. Did the durable state record more than one committed outcome for the same
   durable identifier?

The first question is solved with idempotent activity design. The second is
the engine contract.

## Default Idempotency Surfaces

These identifiers are the stable places to dedupe work:

| Surface | What it identifies | Typical use |
| --- | --- | --- |
| `workflow_instance_id` | One public workflow instance | Duplicate-start handling and business-level run identity |
| `workflow_run_id` | One specific durable run | Pinning one selected run for queries, export, or diagnostics |
| `workflow_command_id` | One mutating external command | Client-side request retry dedupe |
| `activity_execution_id` | One logical activity execution across retries | Default remote idempotency key for external side effects |
| `activity_attempt_id` | One concrete attempt of that activity | Correlation when a remote system must distinguish separate tries |
| `schedule_id` | One schedule definition | Dedupe around schedule ownership and trigger identity |
| message-stream `idempotencyKey` | One retried logical message send | Prevent duplicate message ingestion when a sender retries |

When in doubt, use `activity_execution_id` as the default idempotency key for
an external operation. Reach for `activity_attempt_id` only when the external
target truly needs every retry attempt to be distinguishable.

```php
use Workflow\V2\Activity;

final class ChargeCard extends Activity
{
    public function handle(array $payload): string
    {
        return app(PaymentGateway::class)->charge(
            $payload,
            idempotencyKey: $this->activityId(),
            attemptCorrelation: $this->attemptId(),
        );
    }
}
```

## What Developers Must Make Idempotent

You do **not** need to make workflow replay itself idempotent. The framework
handles replay by rebuilding state from committed history.

You **do** need to make external effects safe to repeat, including:

- payment or billing calls
- emails, texts, and webhooks
- writes to another database or service
- file creation or upload
- any command that creates or mutates state outside the workflow history

Common approaches:

- Pass the remote API an idempotency key.
- Write to a deterministic target resource such as a known object key.
- Use an upsert or transaction keyed by a durable identifier.
- Make the action naturally repeatable so the second call is a no-op.

## Operator Guidance

When diagnosing a run in Waterline, the CLI, or server logs:

- Treat duplicate activity observation after lease expiry as expected until the
  durable attempt outcome shows otherwise.
- Treat late completion or failure reports as a race the engine resolves, not
  as proof that the remote side effect never happened.
- Treat workflow-task replay as recovery, not as workflow-level retry.
- Treat missing compatible workers, stuck leases, or repeated repair as
  operational signals that need investigation, not as a reason to assume the
  workflow body should re-run side effects.

The most important operator distinction is between **transport uncertainty**
and **durable outcome**. Durable Workflow surfaces both so you can tell the
difference.

## Related Guides

- [Overview](./overview.md) introduces the workflow/activity split.
- [Workflow Constraints](./workflow-constraints.md) covers deterministic
  authoring rules.
- [Activity Constraints](./activity-constraints.md) covers side-effect safety
  and idempotency techniques.
- [Failures and Recovery](../failures-and-recovery.md) covers retries,
  timeout enforcement, and repair.
- [Activity Execution Model](../features/activity-execution-model.md) explains
  how queued activities, local activities, worker sessions, and sticky
  execution fit together.
- [Local Activities](../features/local-activities.md) explains same-process
  activity attempts, workflow-task heartbeats, retries, and cold replay.
- [Sticky Execution](../features/sticky-execution.md) explains sticky replay
  caches and why cold replay remains the correctness fallback.

<!-- Source: docs/constraints/constraints-summary.md -->

# Constraints Summary

| Constraint | Workflow code | Activity code |
| --- | --- | --- |
| IO | Not allowed | Allowed |
| Mutable global variables | Not allowed | Allowed |
| Non-deterministic functions | Not allowed | Allowed |
| `Carbon::now()` | Not allowed | Allowed |
| `sleep()` | Not allowed | Allowed |
| External side effects | Move them into activities | Allowed only when safe to repeat |

Workflow code must be deterministic because the engine rebuilds a workflow's state by replaying its history of durable steps — it re-invokes the workflow body, not activities. If the body produces different decisions on replay, the engine cannot reconstruct the workflow's state and continue execution correctly. The workflow body does not need to be idempotent; determinism is the replay requirement.

Activities must be idempotent because activity execution is at-least-once. Retries, lease expiry, and redelivery can cause the same attempt to be observed more than once, and that is first-class behavior rather than an error. Making the activity body or its external target safe to repeat — with an idempotency key, a deterministic target resource, or a naturally idempotent operation — is what keeps duplicate execution from producing duplicate side effects.

See [Idempotent vs. Deterministic Workflows](/docs/constraints/idempotent-vs-deterministic/)
for a direct comparison and examples that separate the two properties.

See [Execution Guarantees and Idempotency](./execution-guarantees.md) for the
public v2 contract behind those terms.

<!-- Source: docs/constraints/structural-limits.md -->

# Structural Limits

Structural limits cap the resource consumption of a single workflow run. When an operation would exceed a configured limit, the engine records a typed failure with a machine-readable `structural_limit` failure category and the specific limit kind, then fails the run. This protects the system from unbounded fan-out, oversized payloads, and metadata bloat.

## Limit kinds

| Limit kind | Default | What it caps |
|---|---|---|
| `pending_activity_count` | 2,000 | Non-terminal activity executions open simultaneously |
| `pending_child_count` | 1,000 | Non-terminal child workflows open simultaneously |
| `pending_timer_count` | 2,000 | Pending timers open simultaneously |
| `pending_signal_count` | 5,000 | Unprocessed signals pending simultaneously |
| `pending_update_count` | 500 | Unresolved updates pending simultaneously |
| `command_batch_size` | 1,000 | Items in a single parallel fan-out (`all()`) |
| `payload_size_bytes` | 2 MiB | Serialized size of a single argument payload |
| `memo_size_bytes` | 256 KiB | Serialized size of non-indexed memo metadata |
| `search_attribute_size_bytes` | 40 KiB | Serialized size of indexed search-attribute metadata |
| `history_transaction_size` | 5,000 | History events produced by a single workflow task execution |

All limits are enforced at the point of scheduling, recording, or command intake. A value of `0` disables the check for that limit kind.

## Soft-limit warnings

Before a hard limit terminates a run or rejects a command, the engine can warn you that a resource is approaching its ceiling. When a count-based resource (pending activities, children, timers, signals, updates, command batch size, or history transaction events) crosses a configurable percentage of the hard limit, the engine logs a structured warning.

The default warning threshold is **80%**. For example, with the default `pending_activity_count` limit of 2,000, a warning is logged when a run reaches 1,600 pending activities. The run continues executing normally — the warning gives operators time to react (scale workers, trigger continue-as-new, raise the limit) before the hard guard fails the run.

Configure the threshold via `workflows.v2.structural_limits.warning_threshold_percent`:

```env
DW_V2_LIMIT_WARNING_THRESHOLD_PERCENT=80
```

Set to `0` to disable soft-limit warnings entirely.

Warning log entries include structured context:

```
[Durable Workflow] Run 42 approaching structural limit [pending_activity_count]: 1620 / 2000 (81% utilization, warning at 80%).
```

The structured log context includes `workflow_run_id`, `workflow_type`, `limit_kind`, `current`, `limit`, and `utilization_percent` for integration with log aggregation and alerting tools.

## Configuration

Override any limit through `workflows.v2.structural_limits` in your config or via environment variables:

```php
// config/workflows.php
'v2' => [
    'structural_limits' => [
        'pending_activity_count' => (int) env('DW_V2_LIMIT_PENDING_ACTIVITIES', 2000),
        'pending_child_count' => (int) env('DW_V2_LIMIT_PENDING_CHILDREN', 1000),
        'pending_timer_count' => (int) env('DW_V2_LIMIT_PENDING_TIMERS', 2000),
        'pending_signal_count' => (int) env('DW_V2_LIMIT_PENDING_SIGNALS', 5000),
        'pending_update_count' => (int) env('DW_V2_LIMIT_PENDING_UPDATES', 500),
        'command_batch_size' => (int) env('DW_V2_LIMIT_COMMAND_BATCH_SIZE', 1000),
        'payload_size_bytes' => (int) env('DW_V2_LIMIT_PAYLOAD_SIZE_BYTES', 2097152),
        'memo_size_bytes' => (int) env('DW_V2_LIMIT_MEMO_SIZE_BYTES', 262144),
        'search_attribute_size_bytes' => (int) env('DW_V2_LIMIT_SEARCH_ATTRIBUTE_SIZE_BYTES', 40960),
        'history_transaction_size' => (int) env('DW_V2_LIMIT_HISTORY_TRANSACTION_SIZE', 5000),
        'warning_threshold_percent' => (int) env('DW_V2_LIMIT_WARNING_THRESHOLD_PERCENT', 80),
    ],
],
```

## Enforcement points

### Pending count limits (executor-side)

Before the executor schedules an activity, child workflow, or timer, it counts the currently non-terminal items of that type on the run. If the count is already at or above the configured limit, the run fails immediately with a `StructuralLimitExceededException`.

This protects against patterns like unbounded parallel fan-out loops that accumulate thousands of pending operations:

```php
// This will fail if $items exceeds the pending_activity_count limit
$calls = [];
foreach ($items as $item) {
    $calls[] = fn () => activity(ProcessItemActivity::class, $item);
}
return all($calls); // Also checked against command_batch_size
```

To handle large batches within the limits, process items in bounded chunks:

```php
foreach (array_chunk($items, 500) as $chunk) {
    $calls = [];
    foreach ($chunk as $item) {
        $calls[] = fn () => activity(ProcessItemActivity::class, $item);
    }
    all($calls);
}
```

### Pending count limits (intake-side)

When a signal or update command arrives via the control plane, webhook, or `WorkflowStub`, the engine checks the count of unprocessed signals (`received` status) or unresolved updates (`accepted` status) on the target run before accepting the command.

If the count is at or above the configured limit, the command is **rejected** with reason `structural_limit_exceeded`. The rejection response includes machine-readable metadata (`structural_limit_kind`, `structural_limit_value`, `structural_limit_configured`) so callers can identify the root cause.

Unlike executor-side limits (which fail the run), intake-side limits reject the individual command without terminating the workflow. The run remains active, and the caller can retry once pending items have been processed.

```php
// If 5,000 signals are already pending, this will be rejected:
$result = $workflow->attemptSignal('process-item', $data);
if ($result->rejected()) {
    // $result->rejectionReason() === 'structural_limit_exceeded'
    // back off and retry later
}
```

### Command batch size

The `all()` function checks the total number of leaf operations in a single fan-out group against `command_batch_size`. This is checked before any individual activities or children are scheduled, so the run fails cleanly rather than partially scheduling a batch.

### Payload size

When the executor schedules an activity or child workflow, it serializes the argument payload and checks the byte length against `payload_size_bytes`. If the serialized payload exceeds the limit, the run fails before any database rows are created for the operation.

This applies to:

- **Activity arguments** — checked at the point `scheduleActivity` serializes the `ActivityCall` arguments.
- **Child workflow arguments** — checked at the point `scheduleChildWorkflow` serializes the child's start arguments, before creating the child instance or run rows.

```php
// A 3 MiB payload will fail with the default 2 MiB limit
activity(ProcessDocumentActivity::class, $threeMegabyteBlob);
```

To work within the limit, either enable
[External Payload Storage](../features/external-payload-storage.md) on the
namespace so the runtime transparently offloads over-threshold payloads to a
configured object store, or store the bytes yourself and pass an
application-level reference:

```php
$ref = Storage::put('docs/incoming.pdf', $blob);
activity(ProcessDocumentActivity::class, $ref);
```

External payload storage preserves replay integrity by recording a hashed
`durable-workflow.v2.external-payload-reference.v1` envelope in history, so
the reference envelope becomes the payload the limit sees — not the bytes.

### Memo size

When a workflow upserts memo entries via `upsertMemo()`, the executor merges the new entries into the existing memo map, then JSON-encodes the merged result and checks the byte length against `memo_size_bytes`. If the merged memo exceeds the limit, the run fails before the memo is persisted.

### History transaction size

Each workflow task execution (a single "turn" of replay and forward progress) may produce new history events — activity scheduling, timer creation, side-effect recording, search-attribute upserts, and so on. The `history_transaction_size` limit caps the total number of new events a single task can produce.

This catches runaway loops that create unbounded events in a single task without yielding control:

```php
// If a workflow schedules thousands of operations in one task,
// the history transaction limit prevents the task from growing
// without bound. Process large batches in bounded chunks instead.
foreach (array_chunk($items, 500) as $chunk) {
    $calls = [];
    foreach ($chunk as $item) {
        $calls[] = fn () => activity(ProcessItemActivity::class, $item);
    }
    all($calls);  // Each chunk is a separate task execution
}
```

The check runs at the top of each iteration of the executor's main loop. Events created during replay (reading existing history) do not count toward the limit — only new events written during the current task contribute.

### Search attribute size

When a workflow upserts search attributes via `upsertSearchAttributes()`, the executor merges the new attributes into the existing set, then JSON-encodes the merged result and checks the byte length against `search_attribute_size_bytes`. If the merged attributes exceed the limit, the run fails before the attributes are persisted.

## Failure taxonomy

When a structural limit is exceeded, the engine records:

- A `WorkflowFailure` row with `failure_category = structural_limit`
- A `WorkflowFailed` history event with:
  - `failure_category = structural_limit`
  - `structural_limit_kind` — the specific limit that was exceeded (e.g. `pending_activity_count`, `command_batch_size`)
  - `structural_limit_value` — the current count or size that triggered the limit
  - `structural_limit_configured` — the configured ceiling

This metadata is machine-readable, so operators, Waterline, and external tooling can identify the root cause without parsing free-text messages.

## Health check

The current structural limits configuration is included in the v2 health check snapshot under `structural_limits`, making the active ceilings visible to operators:

```json
{
  "structural_limits": {
    "pending_activity_count": 2000,
    "pending_child_count": 1000,
    "pending_timer_count": 2000,
    "pending_signal_count": 5000,
    "pending_update_count": 500,
    "command_batch_size": 1000,
    "payload_size_bytes": 2097152,
    "memo_size_bytes": 262144,
    "search_attribute_size_bytes": 40960,
    "history_transaction_size": 5000,
    "warning_threshold_percent": 80
  }
}
```

## Backend-dependent limits

The backend capabilities snapshot publishes the full structural-limit contract adjusted for the current infrastructure. Most limits are backend-independent configuration values, but certain backends impose additional constraints:

- **SQS queue** — Amazon SQS caps delayed message delivery at 900 seconds, so the capability snapshot includes `max_single_timer_delay_seconds: 900`. Timers exceeding this are chunked by the transport layer.
- **SQLite database** — SQLite serializes writes, so the snapshot notes `concurrent_write_safety: limited`. High pending-count limits may cause lock contention under concurrent worker load.

The full contract is available in the `structural_limits` section of the backend capabilities response:

```json
{
  "structural_limits": {
    "configured": { "pending_activity_count": 2000, "..." : "..." },
    "backend_adjustments": { "max_single_timer_delay_seconds": 900 },
    "effective": { "pending_activity_count": 2000, "max_single_timer_delay_seconds": 900, "..." : "..." },
    "issues": [
      {
        "component": "structural_limits",
        "severity": "info",
        "code": "queue_max_delay_constraint",
        "message": "The [sqs] queue driver limits delayed dispatch to 900 seconds; timers exceeding this are chunked by the transport layer."
      }
    ]
  }
}
```

## Waterline

Waterline surfaces structural-limit failures in the exceptions table with the `structural_limit` failure category. The timeline failure details include the limit kind, current value, and configured ceiling.

## Server request-boundary limits

When using the standalone server, a separate set of caps is enforced at the HTTP request boundary before a workflow task, signal, update, or query ever reaches the control plane. These limits fail fast with a `422 validation_failed` (or `413 payload_too_large` for whole-body checks) response so clients learn the rejection reason without the server writing any row to the database.

| Limit | Default | Config key | What it caps |
|---|---|---|---|
| Body size | 2 MiB | `server.limits.max_payload_bytes` | Total HTTP request body bytes |
| Memo size | 256 KiB | `server.limits.max_memo_bytes` | Serialized memo on `POST /workflows` and `POST /schedules` |
| Search-attribute count | 100 | `server.limits.max_search_attributes` | Registered custom search attributes per namespace |
| Search-attribute key length | 128 bytes | `server.limits.max_search_attribute_key_length` | Length of a single SA key on start |
| Search-attribute value size | 2 KiB | `server.limits.max_search_attribute_value_bytes` | Each string value (and each element of an array value) on start |
| Signal / update / query name | 256 bytes | `server.limits.max_operation_name_length` | URL path segment for signal/update/query names |
| `workflow_id` length | 128 chars | controller validator | Workflow ID on `POST /workflows` |
| `workflow_type` / `task_queue` / `business_key` | 255 chars | controller validator | String fields on `POST /workflows` |
| `request_id` | 255 chars | controller validator | Deduplication token on signals/updates/cancel/terminate |
| `reason` | 1,000 chars | controller validator | Reason text on cancel/terminate/archive |

Each limit is individually configurable via `DW_*` environment variables (see `config/dw-contract.php` for the full contract). Setting a value of `0` disables the check for that specific limit, but leaves the others in force.

The currently-configured values are published under `limits` on the `GET /api/cluster/info` response, so clients can discover them at runtime:

```json
{
  "limits": {
    "max_payload_bytes": 2097152,
    "max_memo_bytes": 262144,
    "max_search_attributes": 100,
    "max_search_attribute_key_length": 128,
    "max_search_attribute_value_bytes": 2048,
    "max_operation_name_length": 256,
    "max_pending_activities": 2000,
    "max_pending_children": 2000
  }
}
```

Validation errors are returned in the standard control-plane error envelope with `reason: "validation_failed"` and a `validation_errors` map keyed by the offending field (`signal_name`, `update_name`, `query_name`, `search_attributes`, `memo`, and so on). Payload-size rejections use `reason: "payload_too_large"` and the `413` status code.

<!-- Source: docs/features/signals.md -->

# Signals

Signals allow you to trigger events in a workflow from outside the workflow. This can be useful for reacting to external events, enabling *human-in-the-loop* interventions, or for signaling the completion of an external task. For repeated ordered inputs with durable cursor advancement, use [Message Streams](./message-streams.md).

## Named Signal Waits

A workflow calls `await('signal-name')` directly. The next accepted signal command with that name resumes the run and returns a deterministic value to the suspended workflow.

```php
use Workflow\V2\Attributes\Signal;
use Workflow\V2\Attributes\Type;
use Workflow\V2\Workflow;
use function Workflow\V2\await;

#[Type('order-approval')]
#[Signal('approved-by', [
    ['name' => 'approvedBy', 'type' => 'string'],
])]
final class OrderApprovalWorkflow extends Workflow
{
    public function handle(): array
    {
        $approvedBy = await('approved-by');

        return [
            'approved_by' => $approvedBy,
            'workflow_id' => $this->workflowId(),
            'run_id' => $this->runId(),
        ];
    }
}
```

Trigger the signal from PHP by addressing the public instance id:

```php
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::load('order-123');

$result = $workflow->attemptSignalWithArguments('approved-by', [
    'approvedBy' => 'Taylor',
]);

$result->accepted();    // true
$result->outcome();     // "signal_received"
$result->commandId();   // Durable signal-command id
$result->instanceId();  // "order-123"
```

Signal behavior:

- Declare each external signal name up front with a repeatable `#[Signal('signal-name')]` class attribute. Optionally include an ordered parameter contract.
- Signal commands target the public workflow instance id — not a run id — so continue-as-new chains keep the same public signal route.
- With a parameter contract, intake rejects invalid payloads as `rejected_invalid_arguments` with machine-readable `validation_errors`.
- Without a contract, `await('name')` returns `true` when no arguments were sent, the single argument when one was sent, or the full argument array when several were sent.
- Unknown signal names reject as `rejected_unknown_signal`; signals against a closed or unstarted instance reject as `rejected_not_active` or `rejected_not_started`.

**Important:** The `await()` function should only be used in a workflow, not an activity.

For condition waits — waiting until a predicate over durable state becomes true — see [Condition Waits](./condition-waits.md). For a timeout-backed `await`, see [Signal + Timer](./signal+timer.md). For repeated inbox/outbox flows, see [Message Streams](./message-streams.md).

## Run this pattern

The webhook-started workflow in the
[Sample App](/docs/sample-app) is the runnable reference for the
named-signal-wait pattern on this page:

```bash
php artisan app:webhook
```

`App\Workflows\Webhooks\WebhookWorkflow` starts from an HTTP webhook
ingress and parks on `await('ready')` until the matching signal lands.
Open Waterline while it is parked and you will see the
`WorkflowExecutionSignaled` event materialize the moment the signal
is accepted.

<!-- Source: docs/features/queries.md -->

# Queries

Queries allow you to retrieve information about the current state of a workflow without affecting its execution. This is useful for monitoring and debugging purposes.

## Replay-Safe Query Methods

Queries replay committed history for the current selected run and then invoke the annotated method on the hydrated workflow object. They do not apply accepted-but-not-yet-applied signal or update commands implicitly.

```php
use Workflow\QueryMethod;
use Workflow\V2\Workflow;
use function Workflow\V2\await;

final class ApprovalWorkflow extends Workflow
{
    private string $stage = 'booting';

    public function handle(): void
    {
        $this->stage = 'waiting-for-approval';

        await('approved-by');

        $this->stage = 'approved';
    }

    #[QueryMethod('current-stage')]
    public function currentStage(): string
    {
        return $this->stage;
    }

    #[QueryMethod('starts-with')]
    public function startsWith(string $prefix): bool
    {
        return str_starts_with($this->stage, $prefix);
    }
}
```

Address the current run through the public instance id:

```php
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::load('order-123');

$workflow->currentStage(); // "waiting-for-approval"
$workflow->query('starts-with', 'waiting'); // true
$workflow->queryWithArguments('starts-with', ['prefix' => 'waiting']); // true
```

When you want to pin one historical or selected run explicitly, query through `loadRun($runId)` instead:

```php
$selectedRun = WorkflowStub::loadRun($runId);

$selectedRun->currentStage();
```

Query behavior:

- Queries are replay-safe: they observe committed history only and do not mutate workflow state.
- Arguments are forwarded to the annotated method. Declare a stable public name with `#[QueryMethod('public-name')]` so the callable survives PHP method renames.
- `load($instanceId)` queries the newest durable run for that instance; `loadRun($runId)` targets one specific run (useful for pre–continue-as-new queries).
- Accepted-but-not-yet-applied signals and updates are visible in command history but do not count as applied state until the worker records `SignalApplied` / `UpdateApplied`.

To define a query method on a workflow, use the `QueryMethod` annotation. The optional string argument lets you freeze a public durable query name that survives PHP method renames:

```php
use Workflow\QueryMethod;
use Workflow\V2\Workflow;

final class MyWorkflow extends Workflow
{
    private bool $ready = false;

    #[QueryMethod('is-ready')]
    public function getReady(): bool
    {
        return $this->ready;
    }
}
```

To query a workflow, call the method on the workflow instance. The query method will return the data from the workflow.

```php
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::load('order-123');

$ready = $workflow->getReady();
$sameReady = $workflow->query('is-ready');
```

Use `queryWithArguments()` when your caller already has one positional list or a named parameter map:

```php
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::load('order-123');

$workflow->queryWithArguments('starts-with', [
    'prefix' => 'wait',
]);
```

Waterline uses the selected-run query contract surface. The dashboard exposes `declared_query_targets[*]` alongside signals and updates, but only shows query execution when `can_query = true`. The selected-run or current-run query operator posts JSON `arguments` to `/waterline/api/instances/{instanceId}/queries/{query}` or `/waterline/api/instances/{instanceId}/runs/{runId}/queries/{query}`. Query execution still requires a loadable workflow definition because the selected run must be replayed before the query method can run; when durable query targets exist but that definition is unavailable, selected-run detail reports `can_query = false` with `query_blocked_reason = workflow_definition_unavailable`, and query POSTs return HTTP `409 Conflict` with `blocked_reason = workflow_definition_unavailable`. If the run only has an incomplete snapshot and the current build can no longer finish backfilling it, detail reports `declared_contract_source = unavailable`; surviving query targets remain visible as diagnostic metadata, but named query arguments still reject with `422` until a compatible build persists the missing contract.

The public webhook bridge exposes that same replay-safe query surface outside Waterline:

```text
POST /webhooks/instances/{instanceId}/queries/{query}
POST /webhooks/instances/{instanceId}/runs/{runId}/queries/{query}
```

Those webhook routes accept the same JSON `arguments` field as Waterline, return a typed JSON `result` on success, and use the same error shape for invalid arguments (`422` with `validation_errors`) and replay blocks (`409` with `blocked_reason`). When the current workflow definition is still loadable, callers may address the query by either its durable `#[QueryMethod('public-name')]` target or the underlying PHP method name; successful HTTP responses normalize `query_name` back to the durable public target so external callers do not harden on method-renaming details.

**Important:** Querying a workflow does not advance its execution, unlike signals.

<!-- Source: docs/features/updates.md -->

# Updates

Updates allow you to retrieve information about the current state of a workflow and mutate the workflow state at the same time. They are essentially both a query and a signal combined into one.

## Explicit Update Commands

Each accepted update:

- Is a request/response call against a running workflow. `attemptUpdate*()` waits for the handler to finish; `submitUpdate*()` returns as soon as the command is durably accepted and exposes `inspectUpdate($updateId)` to poll later.
- Uses the declared durable update name in command history and webhook routing — `#[UpdateMethod('mark-approved')]` keeps the public callable name stable across PHP method renames.
- Rejects against historical or already-closed runs rather than silently mutating the current run.
- Rejects undeclared method names as `rejected_unknown_update` and contract-invalid arguments as `rejected_invalid_arguments` (with machine-readable `validation_errors`) before the handler runs.

Like queries, updates replay committed history first. Unlike queries, updates are allowed to mutate replay-safe workflow state and return a value.

To define an update method on a workflow, use the `UpdateMethod` annotation. The optional string argument lets you freeze a public durable name that survives PHP method renames:

```php
use Workflow\UpdateMethod;
use Workflow\V2\Workflow;

final class MyWorkflow extends Workflow
{
    private bool $ready = false;

    #[UpdateMethod('mark-ready')]
    public function updateReady(bool $ready): bool
    {
        $this->ready = $ready;

        return $this->ready;
    }
}
```

Call the update method directly when you want the raw return value:

```php
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::load('order-123');

$ready = $workflow->updateReady(true);
```

That direct PHP call still uses the method name. The durable command target remains `mark-ready`.

Use `attemptUpdate()` when you want the durable command outcome as well. Pass the durable update name there:

```php
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::load('order-123');

$result = $workflow->attemptUpdate('mark-ready', true);

$result->accepted();        // true
$result->completed();       // true when the update body ran successfully
$result->updateStatus();    // "accepted", "completed", "failed", or "rejected"
$result->updateId();        // Durable update lifecycle id
$result->result();          // Raw update return value when completed
$result->failureMessage();  // Failure message when the update body threw
```

`attemptUpdate()` records the accepted update first, then waits for the workflow worker to apply it and close the update lifecycle before returning. The wait is time-bounded by `workflows.v2.update_wait.completion_timeout_seconds`; if the worker has not closed the update when the budget expires, `attemptUpdate()` returns the accepted lifecycle with `waitTimedOut() === true` and `updateStatus() === 'accepted'`.

Use `withUpdateWaitTimeout()` when one call needs a different completion budget:

```php
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::load('order-123')
    ->withUpdateWaitTimeout(5);

$result = $workflow->attemptUpdate('mark-ready', true);
```

Use `attemptUpdateWithArguments()` when the caller already has a positional list or named parameter map:

```php
$result = $workflow->attemptUpdateWithArguments('mark-ready', [
    'ready' => true,
]);
```

Named maps are validated against the durably snapped update contract and normalized into declaration order before acceptance.

Use `submitUpdate()` or `submitUpdateWithArguments()` when the caller only needs durable acceptance and is fine with the workflow worker applying the update later:

```php
$accepted = $workflow->submitUpdate('mark-ready', true);

$accepted->accepted();     // true
$accepted->completed();    // false
$accepted->updateStatus(); // "accepted"
$accepted->result();       // null until the worker records UpdateCompleted
```

Use `inspectUpdate()` when you already have an `update_id` and want to read the stored lifecycle later without waiting again:

```php
$latest = $workflow->inspectUpdate($accepted->updateId());

$latest->updateStatus(); // "accepted"
$latest->closedAt();     // null until the lifecycle closes
```

`inspectUpdate()` does not wait for workflow execution. It reloads the stored durable lifecycle and returns the current `UpdateResult`.

Update rules:

- `load($instanceId)` updates the newest durable run for that instance (including after continue-as-new).
- `loadRun($runId)` rejects with `rejected_not_current` once that selected run is historical.
- Closed runs reject with `rejected_not_active`.
- Failed update bodies do not close the workflow run; they are recorded as update-scoped failures and leave the run open for the next replay task.
- Queries replay completed updates, but rejected or failed updates remain non-replayable command and history facts only.

<!-- Source: docs/features/timers.md -->

# Timers

The framework provides the ability to suspend the execution of a workflow and resume at a later time. These are durable timers, meaning they survive restarts and failures while remaining consistent with workflow replay semantics. This can be useful for implementing delays, retry logic, or timeouts.

To use timers, call `timer($duration)` within your workflow:

```php
use function Workflow\V2\timer;
use Workflow\V2\Workflow;

class MyWorkflow extends Workflow
{
    public function handle(): string
    {
        timer(30);

        return 'The workflow waited 30 seconds.';
    }
}
```

Timer behavior:

- each `timer()` call creates a durable timer row plus typed `TimerScheduled`, `TimerFired`, and, when superseded, `TimerCancelled` history events
- delayed timers run through a dedicated timer task before the workflow task is resumed
- `timer(0)` fires inline during the workflow task and does not create a timer task
- replay and query paths treat typed `TimerScheduled`, `TimerFired`, and `TimerCancelled` history as authoritative for timer lifecycle, so pure timers stay blocked until the committed fire event arrives and selected-run detail can rebuild open, fired, or cancelled timer waits from history
- Waterline surfaces timer waits in run detail and dashboard payloads
- engine-level `cancel()` and `terminate()` commands supersede open timer waits durably, and late timer jobs no-op instead of reopening the run

Timers of any duration work with every Laravel queue driver, including Amazon SQS — the engine transparently chunks long delays over the driver's per-message limit.

`sideEffect()` is available for replay-safe snapshots such as randomness or one-time branch inputs.

## Reading deterministic time

Inside a workflow body, use `Workflow::now()` (or `Workflow\V2\now()`) instead of Laravel's `now()` helper or `Carbon::now()` when you need the current time:

```php
use function Workflow\V2\activity;
use function Workflow\V2\now;
use Workflow\V2\Workflow;

class DurationAwareWorkflow extends Workflow
{
    public function handle(string $name): array
    {
        $startedAt = Workflow::now();
        $greeting = activity(GreetingActivity::class, $name);
        $finishedAt = Workflow::now();

        return [
            'greeting' => $greeting,
            'took_ms' => $finishedAt->getTimestampMs() - $startedAt->getTimestampMs(),
        ];
    }
}
```

`Workflow::now()` advances as the executor replays history events — it returns `recorded_at` of the last activity completion, timer fire, signal receipt, condition resolution, or child workflow completion the replay has consumed. Before any event is consumed, it returns the run's `started_at`. Outside a workflow fiber (for example in an activity or a query method that forwards to a non-workflow helper), it falls back to wall-clock `now()`.

Using `Workflow::now()` keeps your workflow deterministic: two replays of the same history produce the same time values, even if wall-clock time has advanced between them.

<!-- Source: docs/features/signal+timer.md -->

# Signal + Timer

`Workflow\V2` supports both `await($condition, timeout: $seconds, conditionKey: $key)` for timeout-backed condition waits and `await('signal-name', timeout: $seconds)` for timeout-backed named signal waits.

Use it when the workflow should continue as soon as some durable replayed state becomes true, but should also unblock after a deadline if that state never changes.

```php
use Workflow\UpdateMethod;
use function Workflow\V2\await;
use Workflow\V2\Attributes\Type;
use Workflow\V2\Workflow;

use function Workflow\V2\minutes;

#[Type('approval-with-timeout')]
class MyWorkflow extends Workflow
{
    private bool $ready = false;

    public function handle(): string
    {
        $approved = await(fn () => $this->ready, timeout: minutes(5), conditionKey: 'approval.ready');

        return $approved ? 'approved' : 'timed out';
    }

    #[UpdateMethod]
    public function markReady(bool $ready = true): array
    {
        $this->ready = $ready;

        return ['ready' => $this->ready];
    }
}
```

`await()` with a `timeout:` parameter works like this:

- The predicate must depend only on replayed durable inputs (activity results, update mutations, child-workflow results). Non-durable state will not wake the wait.
- The optional condition key is a stable operator label for the wait. Replay validates the key against prior history so a redeployment cannot silently reuse the same workflow step for a different predicate.
- If the predicate becomes true before the timer fires, `await()` returns `true`.
- If the deadline wins first, `await()` returns `false`.

Timer sugar helpers are available for readable timeout values: `seconds()`, `minutes()`, `hours()`, `days()`, `weeks()`, `months()`, `years()`.

To change the predicate durably from application code, call the declared update:

```php
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::load('approval-with-timeout');

$workflow->markReady();
```

The return value is `true` when the condition becomes true before the timeout task fires and `false` when the timeout wins.

For named signals, `await('name', timeout: minutes(5))` returns the signal payload when the signal arrives and `null` when the timeout wins. `null` is reserved for timeout: no-argument signals resolve to `true`, one argument resolves to that value, and multiple arguments resolve to an array. `Workflow\V2` does not use legacy `#[SignalMethod]` mutator methods to flip workflow state.

## `awaitWithTimeout()` Helper

`Workflow::awaitWithTimeout($timeout, $condition, $conditionKey = null)` is a sugar wrapper for `await($condition, timeout: $timeout, conditionKey: $conditionKey)`. The argument order puts the deadline first to make the timeout intent obvious at the call site.

```php
use function Workflow\V2\{await, minutes};
use Workflow\V2\Workflow;

#[Type('approval-with-timeout')]
class MyWorkflow extends Workflow
{
    private bool $ready = false;

    public function handle(): string
    {
        $approved = Workflow::awaitWithTimeout(
            minutes(5),
            fn () => $this->ready,
            'approval.ready',
        );

        return $approved ? 'approved' : 'timed out';
    }

    #[UpdateMethod]
    public function markReady(bool $ready = true): array
    {
        $this->ready = $ready;

        return ['ready' => $this->ready];
    }
}
```

Return semantics are the same as `await()` with a `timeout:` parameter: the call returns `true` when the condition wins, `false` when the deadline wins for a callable predicate, the signal payload when a named signal wins, and `null` when the deadline wins for a named signal.

`awaitWithTimeout()` reads "wait at most this long for a condition" rather than "wait for a condition, with a timeout option". Use whichever spelling reads more clearly at the call site — the runtime treats them identically and they record the same workflow history.

<!-- Source: docs/features/condition-waits.md -->

# Condition Waits

`await($condition, $conditionKey = null)` provides replay-safe condition waits. Use it when the predicate depends only on workflow state that was already derived from durable inputs such as updates, activity results, or child results. If you want one named external signal value directly, call `await('name')` — see [Signals](./signals.md).

Condition waits are driven by an update method instead of signal mutators:

```php
use Workflow\UpdateMethod;
use function Workflow\V2\await;
use Workflow\V2\Attributes\Type;
use Workflow\V2\Workflow;

#[Type('approval-workflow')]
class MyWorkflow extends Workflow
{
    private bool $ready = false;

    public function handle(): void
    {
        await(fn () => $this->ready, 'approval.ready');
    }

    #[UpdateMethod]
    public function setReady(bool $ready = true): array
    {
        $this->ready = $ready;

        return ['ready' => $this->ready];
    }
}
```

The optional condition key is a stable, URL-safe operator label for the wait. Replay validates the previously recorded key so a later deployment cannot accidentally reuse the same workflow step for a different predicate — see [Versioning](./versioning.md) for how to evolve a workflow safely across deployments.

For a condition wait with a timeout, see [Signal + Timer](./signal+timer.md).

**Important:** `await()` should only be used in a workflow, not an activity.

<!-- Source: docs/features/side-effects.md -->

# Side Effects

A side effect is a closure containing non-deterministic code. The closure is only executed once and the result is saved. It will not execute again if the workflow is retried. Instead, it will return the saved result. This makes the workflow deterministic because replaying the workflow will always return the same stored value rather than re-running the non-deterministic code.

```php
use function Workflow\V2\await;
use function Workflow\V2\sideEffect;
use Workflow\V2\Attributes\Signal;
use Workflow\V2\Workflow;

#[Signal('finish')]
class MyWorkflow extends Workflow
{
    public function handle(): array
    {
        $token = sideEffect(fn () => random_int(1000, 9999));
        $finish = await('finish');

        return compact('token', 'finish');
    }
}
```

The workflow will only call `random_int()` once and save the result, even if the workflow later fails and is retried.

## When to use side effects

Use `sideEffect()` when you need a non-deterministic value that:

- is computed locally without external I/O (random numbers, UUIDs, timestamps)
- should never change once recorded, even across replays
- does not need retry semantics — the closure runs exactly once

```php
// Generate a correlation token for downstream systems.
$correlationId = sideEffect(fn () => (string) Str::uuid());

// Snapshot the current time for a business rule.
$decidedAt = sideEffect(fn () => now()->toIso8601String());
```

## When to use an activity instead

If the code can fail, talks to an external service, or needs retry/timeout semantics, use an [activity](../defining-workflows/activities.md) instead of a side effect:

| Scenario | Use |
|---|---|
| Generate a random token | `sideEffect()` |
| Read a config value at decision time | `sideEffect()` |
| Call an external API | `activity()` |
| Write to a database | `activity()` |
| Send an email or notification | `activity()` |
| Compute an expensive value that can throw | `activity()` |

The rule of thumb: if the closure can throw an exception that you would want to retry, it belongs in an activity.

## How it works

- each `sideEffect()` call appends a typed `SideEffectRecorded` history event with the workflow step sequence
- workflow replay and query replay both reuse that committed value instead of re-running the closure
- Waterline surfaces the side-effect snapshot as a typed history entry in the selected run timeline
- side effects are still for replay-safe snapshots only, not for work that can fail or that needs retry semantics

## Anti-patterns

**Do not call external services inside a side effect.** If the service call fails, the side effect will not be retried and the workflow will fail permanently:

```php
// BAD: HTTP calls can fail and side effects do not retry.
$price = sideEffect(fn () => Http::get('/api/price')->json('amount'));

// GOOD: Use an activity for external calls.
$price = activity(FetchPriceActivity::class);
```

**Do not put slow or blocking operations inside a side effect.** The closure runs on the workflow task thread. Long-running work delays the entire workflow task:

```php
// BAD: Expensive computation blocks the workflow task.
$hash = sideEffect(fn () => bcrypt($largePayload));

// GOOD: Offload heavy work to an activity.
$hash = activity(ComputeHashActivity::class, $largePayload);
```

**Do not rely on mutable external state.** The closure is executed exactly once. If you read a value that changes over time, the snapshot is frozen at the moment of first execution — not at replay time:

```php
// The cached value is whatever it was during the first execution.
// If the cache changes later, this workflow still sees the old value.
$setting = sideEffect(fn () => cache('feature.flag'));
```

This is by design — the snapshot is intentionally frozen for determinism. If you need a value that updates over the lifetime of the workflow, use a signal or an activity.

## Run this pattern

The elapsed-time workflow in the
[Sample App](/docs/sample-app) is the runnable reference for
keeping clock reads behind `sideEffect()`:

```bash
php artisan app:elapsed
```

`App\Workflows\Elapsed\ElapsedTimeWorkflow` records start and end
timestamps as integer values inside `sideEffect()` callbacks so the
recorded value survives Avro payload decoding on replay. The
Waterline run detail shows two `MarkerRecorded` events bracketing the
timer fire — that pair of markers is the on-disk evidence that the
clock reads stayed deterministic.

<!-- Source: docs/features/heartbeats.md -->

# Heartbeats

Heartbeats let a long-running activity report that its current attempt is still alive.

`heartbeat()` updates the currently claimed `activity_attempts` row, mirrors the latest heartbeat onto the live activity execution, renews that activity task's lease, and appends a typed `ActivityHeartbeatRecorded` history event for the current attempt. Waterline selected-run detail and history exports rebuild attempt status, task id, worker, heartbeat, lease, cancellation, and close timestamps from typed `ActivityStarted`, `ActivityHeartbeatRecorded`, `ActivityRetryScheduled`, `ActivityCompleted`, `ActivityFailed`, and `ActivityCancelled` history first, with mutable attempt rows kept as fallback or enrichment for older data.

```php
use Workflow\V2\Activity;

final class PollRemoteJob extends Activity
{
    public function handle(string $jobId): array
    {
        do {
            sleep(1);

            $status = RemoteService::status($jobId);

            $this->heartbeat([
                'message' => 'Polling remote job',
                'current' => $status['completed'] ?? null,
                'total' => $status['total'] ?? null,
                'unit' => 'steps',
                'details' => [
                    'remote_state' => $status['state'] ?? 'running',
                ],
            ]);
        } while ($status['state'] === 'running');

        return $status;
    }
}
```

Inside the activity, `activityId()`, `attemptId()`, and `attemptCount()` expose the durable execution and current-attempt identity if you need correlation keys for external work. Use `activityId()` as the default remote idempotency key; use `attemptId()` only for systems that need to distinguish separate tries of the same durable activity execution.

Polyglot workers can heartbeat via the HTTP worker bridge without constructing the PHP activity class — see the [worker protocol](/docs/polyglot/worker-protocol) for the endpoints.

## Progress Snapshots

Both `Activity::heartbeat()` and `ActivityTaskBridge::heartbeat()` accept an optional bounded progress snapshot:

- `message`
  - non-empty string up to 280 characters
- `current`
  - non-negative integer or float
- `total`
  - non-negative integer or float
- `unit`
  - non-empty string up to 64 characters
- `details`
  - flat map of up to 20 `key => scalar|null` entries, where keys match `[A-Za-z0-9_.:-]{1,64}`

Use that payload for operator-facing progress like `"Downloading chunk"` or `2 / 5 chunks`, not for large logs or arbitrary nested state. Waterline selected-run detail, the timeline heartbeat entry, and history export surface it back as `last_heartbeat_progress` on the activity and current attempt, while the raw typed `ActivityHeartbeatRecorded` history event keeps the same normalized payload under `progress`.

## Durable Attempt Tracking

The runtime records one first-class durable row per activity attempt, including attempt number, status, started time, latest heartbeat, lease expiry, and close time. It also records typed attempt history as activities start, heartbeat, retry, complete, fail, or cancel, so historical attempt detail survives if mutable attempt or task rows drift later. Each successful heartbeat from the currently claimed attempt records a compact `ActivityHeartbeatRecorded` timeline point with the activity execution id, activity attempt id, heartbeat timestamp, lease expiry, and event-era activity snapshot. If a cancel or terminate command closes the run, the bridge records `ActivityCancelled` when the worker observes the stop and rejects later completion or failure as stale instead of turning cancellation into a result.

Heartbeats are operational liveness signals. Because every accepted heartbeat is durable history, keep the interval meaningful for your timeout and recovery needs rather than using it as a high-volume progress log, and keep the optional progress snapshot compact enough for operator diagnostics instead of treating it like a streaming event feed.

<!-- Source: docs/features/child-workflows.md -->

# Child Workflows

The current `Workflow\V2` slice supports durable child workflows through straight-line `child()` calls and through `all([...])` fan-in barriers that can be child-only, mixed with activities, or nested inside larger `all([...])` groups. Together, these give a parent workflow a durable way to schedule one or more sub-workflows and wait for their outcomes without keeping the parent process alive.

A child run is still a normal workflow run with its own workflow instance id and run id. What makes it a child is the durable linkage back to the parent run plus one stable parent-issued `child_call_id` for that invocation.

```php
use function Workflow\V2\child;
use Workflow\V2\Workflow;

final class ParentWorkflow extends Workflow
{
    public function handle(string $name): array
    {
        $child = child(ChildWorkflow::class, $name);

        return [
            'parent_workflow_id' => $this->workflowId(),
            'parent_run_id' => $this->runId(),
            'child' => $child,
        ];
    }
}
```

## Current Behavior

- Import the helper with `use function Workflow\V2\child;` and call `child(ChildWorkflow::class, ...)`. The call suspends the parent, creates a durable child run, and returns the child's result when the child closes.
- A failed child throws an exception in the parent; a successful child returns its output. Cancellation and termination surface as distinct exception types.
- `$this->child()` returns the most recent child handle; `$this->children()` returns every child handle in workflow-step order. Handles expose `id()`, `runId()`, `callId()`, and signal helpers such as `signal()` and `signalWithArguments()`.
- If a child uses `continueAsNew()`, the parent transparently follows the newest run — `runId()` moves forward while `id()` stays stable.
- `all()` can combine `fn () => child(...)` and `fn () => activity(...)` closures into one barrier; the parent wakes on the first child failure but waits for every successful branch to close before resuming.

## Parallel Child Barrier

Import the helpers with `use function Workflow\V2\all;` and `use function Workflow\V2\child;` when you want one parent step to wait on several child workflows together.

```php
use function Workflow\V2\all;
use function Workflow\V2\child;
use Workflow\V2\Workflow;

final class ParentWorkflow extends Workflow
{
    public function handle(): array
    {
        $children = all([
            fn () => child(FirstChildWorkflow::class),
            fn () => child(SecondChildWorkflow::class),
        ]);

        return $children;
    }
}
```

That child-only form is still useful when every parallel member is a child workflow:

- it waits for the whole child group as one parent step
- it preserves the original array order in the returned results
- it throws the selected non-successful child closure back into the parent workflow body using the same earliest-close-time, lowest-index tie break

If you need to combine child workflows with activities in the same fan-in step, use one mixed `all([...])` barrier instead. The child waits and activity waits share the same `parallel_group_id`, and Waterline labels that shared barrier as `parallel_group_kind = mixed`. If one mixed or child-only subgroup sits inside a larger `all([...])`, the innermost group still drives `parallel_group_id` while `parallel_group_path` preserves the full outer-to-inner path.

## Identity Contract

- `child()->id()` and Waterline `target_name` refer to the child workflow instance id, which stays stable across that child instance's run chain.
- `child_call_id` refers to the parent-issued invocation itself, which stays stable even if the child later uses `continueAsNew()`.
- `resume_source_id`, `child_workflow_run_id`, and selected-run routes refer to one concrete child run in that invocation chain.

## Child Handles

Once the parent has durably reached a child step, the workflow can inspect that invocation through a `ChildWorkflowHandle`.

```php
use Workflow\UpdateMethod;
use function Workflow\V2\child;
use Workflow\V2\Workflow;

final class ParentWorkflow extends Workflow
{
    public function handle()
    {
        return child(ApprovalWorkflow::class);
    }

    #[UpdateMethod('approve-child')]
    public function approveChild(string $approvedBy): void
    {
        $this->child()?->signal('approved-by', $approvedBy);
    }
}
```

- `$this->child()` returns the latest visible child handle or `null`.
- `$this->children()` returns every visible handle in workflow-step order.
- During `all([fn () => child(...)])` barriers, those handles appear in the same step order as the child calls inside the barrier, and `$this->child()` returns the last visible one.
- A handle's `runId()` follows the newest parent-recorded `ChildRunStarted` in that child invocation chain, so a parent waiting on a child that used `continueAsNew()` still sees the latest child run id without losing the original `callId()`.

## Handle Availability

Child handles are history-backed, not speculative:

- Before the parent has durably reached a child step, `$this->child()` returns `null` and `$this->children()` returns an empty list.
- Once the parent has recorded child scheduling or child-start history for that step, the handle becomes visible to queries, updates, and later workflow replay.
- Query replay never dispatches child signals from recorded updates; it only restores the in-memory state implied by committed history.

## Waterline Visibility

When the parent is blocked on a child, Waterline shows it as a child wait rather than as a separate task type. That means:

- the parent run summary reports `wait_kind = child` and `liveness_state = waiting_for_child`
- `waits` includes `kind = child`, the stable `child_call_id`, the child workflow instance id in `target_name`, and the child run id in `resume_source_id`
- when several child waits are open at once, detail also exposes `open_wait_count`
- child waits created by one child-only or mixed `all([...])` barrier share `parallel_group_id` and expose `parallel_group_kind`, `parallel_group_base_sequence`, `parallel_group_size`, and `parallel_group_index`
- nested child waits also expose `parallel_group_path`, ordered from the outermost enclosing barrier to the innermost one
- `timeline` includes typed child events such as `ChildWorkflowScheduled`, `ChildRunStarted`, `ChildRunCompleted`, and `ChildRunFailed`, with the same stable `child_call_id`
- open child waits and lineage now prefer the parent's typed `ChildWorkflowScheduled` / `ChildRunStarted` history, keep one logical child entry per stable `child_call_id`, and follow the newest parent-recorded child run even if copied `workflow_links` rows disappear or the child instance's mutable `current_run_id` drifts after history has already been committed
- once the parent has recorded a typed child-resolution event, selected-run detail keeps that child wait resolved from parent history even if the mutable child run row later drifts; child terminal history or legacy link data can enrich lineage and diagnostics, but cannot replace missing parent typed child step history for replay
- once that typed child-resolution event exists, selected-run lineage and history export also keep the child workflow type, class, run number, status bucket, and closed reason from parent history instead of re-reading the mutable child run row for those resolved-child fields
- if a terminal child row or link exists but the parent has no typed `ChildWorkflowScheduled`, `ChildRunStarted`, `ChildRunCompleted`, `ChildRunFailed`, `ChildRunCancelled`, or `ChildRunTerminated` history for that child step, worker and query replay block with `history_shape_mismatch` and recorded events `no typed history`; Waterline marks the selected child wait `status = unsupported`, exposes `history_authority = unsupported_terminal_without_history` and `history_unsupported_reason = terminal_child_link_without_typed_parent_history`, and surfaces child identity from the blocked task or lineage fallback when available
- if that parent resume workflow task row is lost after the child-resolution event, selected-run detail stays `repair_needed`, exposes a synthetic missing workflow task with `workflow_wait_kind = child`, `child_call_id`, and `child_workflow_run_id`, and manual `repair()`, `workflow:v2:repair-pass`, or worker-loop repair recreates the same child-resolution task from typed parent history
- lineage arrays expose the durable parent/child relationship alongside continue-as-new links, and child-workflow entries now carry the same `child_call_id`

## Parent-Close Policy

When a parent workflow closes — whether by completing, failing, timing out, being cancelled, or being terminated — each open child workflow is affected according to its **parent-close policy**. The policy is set per child call and controls what happens to that child when the parent run exits.

| Policy | Value | Behavior |
|---|---|---|
| Abandon | `abandon` | The child continues running independently. This is the default. |
| Request Cancel | `request_cancel` | A cancel command is sent to the child when the parent closes. |
| Terminate | `terminate` | A terminate command is sent to the child when the parent closes. |

### Default policy

A `child()` call that does not pass a `ChildWorkflowOptions` always runs under `ParentClosePolicy::Abandon`. The source-compatible helpers — `Workflow\V2\child()`, `Workflow\V2\Workflow::child()`, and `Workflow\V2\Workflow::executeChildWorkflow()` — all construct the default options when the first argument is not a `ChildWorkflowOptions`, which sets `parentClosePolicy` to `ParentClosePolicy::Abandon`. The same default applies to every `child()` closure inside an `all([...])` barrier that omits an options argument. Override the default by passing an explicit `ChildWorkflowOptions` as shown below.

### Setting the policy

Pass a `ChildWorkflowOptions` as the first argument to `child()`:

```php
use function Workflow\V2\child;
use Workflow\V2\Enums\ParentClosePolicy;
use Workflow\V2\Support\ChildWorkflowOptions;
use Workflow\V2\Workflow;

final class ParentWorkflow extends Workflow
{
    public function handle(): array
    {
        $options = new ChildWorkflowOptions(
            parentClosePolicy: ParentClosePolicy::RequestCancel,
        );

        return child(ChildWorkflow::class, $options, 'argument1');
    }
}
```

The same pattern works with closures inside `all()` for parallel barriers:

```php
use function Workflow\V2\all;
use function Workflow\V2\child;
use Workflow\V2\Enums\ParentClosePolicy;
use Workflow\V2\Support\ChildWorkflowOptions;

$cancelOptions = new ChildWorkflowOptions(
    parentClosePolicy: ParentClosePolicy::RequestCancel,
);

$results = all([
    fn () => child(FirstChild::class, $cancelOptions),
    fn () => child(SecondChild::class, $cancelOptions, 'arg'),
]);
```

### How it works

- The policy is recorded on the `workflow_links` row (`parent_close_policy`) and in the `ChildWorkflowScheduled` history event payload.
- When the parent run closes for any reason, the engine queries open child links with a non-abandon policy and sends the appropriate command (cancel or terminate) to each open child.
- If the child has already closed by the time the policy is enforced, no action is taken — the command is silently skipped.
- Policy enforcement is best-effort: if a child command is rejected (e.g. the child is already terminal), the parent's closure is not affected. When enforcement succeeds, a `ParentClosePolicyApplied` history event is recorded on the parent run. When enforcement fails, a `ParentClosePolicyFailed` history event is recorded instead, so operators can distinguish successful enforcement from silent failures.
- Continue-as-new does **not** trigger parent-close policy, because the workflow instance remains active under a new run.

### Parent disposition matrix

Parent-close policy fires on every terminal parent disposition, and stays inert for runs that are still live or handed off to a continued run. The engine's behavior is the same for every disposition that marks the parent terminal — the policy is applied to any non-abandon, still-open child link.

| Parent disposition | Policy enforced? | Notes |
|---|---|---|
| Completed | Yes | Fires after `WorkflowCompleted` is recorded. A straight-line `child()` call always waits for the child, so a natural completion normally has no open children; the enforcer is still called as a safety pass. |
| Failed | Yes | Fires after `WorkflowFailed` is recorded for any terminal workflow-task failure. |
| Timed out | Yes | Fires after `WorkflowTimedOut` is recorded for run-timeout and execution-timeout closures. |
| Cancelled | Yes | Fires after the accepted `CancelRequested` command closes the parent run. |
| Terminated | Yes | Fires after the accepted `TerminateRequested` command closes the parent run. |
| Continue-as-new | No | The workflow instance stays active under the new run, so existing child links are re-pointed at the continued run with their original policy preserved. |
| Reset | Not currently applicable | The v2 runtime does not expose a standalone reset command or reset terminal disposition. A future reset or repair operation that closes or replaces a parent run must define whether child ownership transfers; if the old run becomes terminal, it must apply parent-close policy before it stops owning open children. |

### When to use each policy

**Abandon** (default) is correct when children represent independent work that should complete regardless of the parent's fate — for example, a notification workflow or a cleanup task that must finish.

**Request Cancel** is correct when children should receive a graceful shutdown signal. The child can handle the cancellation and run compensation logic before closing.

**Terminate** is correct when children must stop immediately. Use this for children that are purely auxiliary to the parent and have no independent value after the parent closes.

### Waterline visibility

Waterline shows the `parent_close_policy` in:

- the child link entry on the parent run's lineage view
- the `ChildWorkflowScheduled` timeline entry payload
- the child's `CancelRequested` or `TerminateRequested` history event when policy enforcement fires, with a reason that names the parent closure

## Current Limitations

The current surface does not include:

- built-in bounded-concurrency helpers beyond the current `all([fn () => child(...)])`, `all([fn () => activity(...)])`, and mixed `all([fn () => child(...), fn () => activity(...)])` barriers

In other words, durable child handles are supported for already-reached child steps, but it does not yet include higher-level launch-handle or bounded-concurrency APIs beyond today's `child(...)` and `all([...])` surface.

## Run this pattern

The microservice coordination workflow in the
[Sample App](/docs/sample-app) is the runnable reference for
parent–child orchestration across application boundaries:

```bash
php artisan app:microservice
```

`App\Workflows\Microservice\MicroserviceWorkflow` runs the parent on
the main Laravel app and dispatches child work to the bundled
microservice worker. Open Waterline while the run is in flight to see
the child links populated under the parent run, with the close policy
visible on the lineage view exactly as this page describes.

<!-- Source: docs/features/concurrency.md -->

# Concurrency

This guide covers concurrency in embedded Laravel workflows using
`Workflow\V2\Workflow`. For service-mode workers, use the
[PHP SDK guide](https://php.durable-workflow.com/build/workflows-activities/),
[Python SDK guide](https://python.durable-workflow.com/sdk-reference/#deterministic-parallel-groups), or
[Rust SDK reference](https://rust.durable-workflow.com/durable_workflow/struct.WorkflowContext.html#method.parallel).

`all([...])` describes the complete durable group before suspension. The
embedded runtime schedules its activity and child-workflow commands and returns
results in the original nested input shape.

Use a selection group when progress depends on the first completed member
instead of the whole barrier. Selection is also durable: it starts every member,
records one winner, and leaves every non-winner running until workflow code
awaits or explicitly cancels it.

## Series

This example will execute 3 activities in series, waiting for the completion of each activity before continuing to the next one.

```php
use function Workflow\V2\activity;
use Workflow\V2\Workflow;

class MyWorkflow extends Workflow
{
    public function handle()
    {
        return [
            activity(MyActivity1::class),
            activity(MyActivity2::class),
            activity(MyActivity3::class),
        ];
    }
}
```

## Parallel

This example will execute 3 activities in parallel, waiting for the completion of all activities and collecting the results.

```php
use function Workflow\V2\{all, activity};
use Workflow\V2\Workflow;

class MyWorkflow extends Workflow
{
    public function handle()
    {
        return all([
            fn () => activity(MyActivity1::class),
            fn () => activity(MyActivity2::class),
            fn () => activity(MyActivity3::class),
        ]);
    }
}
```

The main difference between the serial example and the parallel execution example is where suspension happens. In the serial example, each `activity()` call suspends and resumes the workflow directly. In the parallel example, the closures describe the whole barrier first and `all()` suspends once for the whole group, so every member can run in parallel before the workflow resumes.

## First-completion selection

`select([...])` starts independent activities, child workflows, timers, signal
waits, condition waits, or nested ordinary barriers and resumes when one member
commits an eligible result or typed failure. Give members stable application
keys when later code needs to distinguish or revisit them.
Member keys must be a non-empty string or a non-negative integer.

The following coordinator starts its deadline at the same durable step as the
resolver. Input processing and resolver progress cannot reset or postpone that
deadline:

```php
use function Workflow\V2\{activity, await, select, timer};
use Workflow\V2\Workflow;

final class ResolveInRealTime extends Workflow
{
    public function handle(string $requestId): array
    {
        $selected = select([
            'resolved' => fn () => activity(ResolveRequest::class, $requestId),
            'manual' => fn () => await('resolution.received'),
            'deadline' => fn () => timer('2 seconds'),
        ]);

        if ($selected->key === 'deadline') {
            $selected->handles['resolved']->cancel();

            return ['status' => 'timed_out'];
        }

        // The deadline is not cancelled just because another member won.
        $selected->handles['deadline']->cancel();

        return [
            'status' => 'resolved',
            'source' => $selected->key,
            'value' => $selected->result(),
        ];
    }
}
```

The result contains the stable member key and index, operation kind and durable
identity, result or typed failure, the winner handle, and a handle for every
member. A non-winning handle can be awaited later with `await()` or cancelled
with `cancel()`. Selection never implicitly discards or cancels a sibling.
`cancel()` is a void/unit request and never reports whether cancellation won.
`SelectionOperationCancelled` history is the outcome authority. If completion
commits first, the runtime writes no cancellation marker, replay advances past
the cancel call only after a committed workflow-task boundary, successor
command, or workflow terminal event proves the no-op committed. Query replay
stops at an in-flight cancel instead of exposing speculative state after it.
Awaiting the handle after the committed no-op returns the earlier completion.

The runtime records the first eligible resolution while holding the parent-run
commit lock. Cold restart, persisted-history reload, query replay, and later
loser completions therefore reuse the recorded winner instead of racing local
language futures or reinterpreting history order. Exact duplicate delivery is
idempotent. Concurrent inputs are ordered by their committed durable history;
an input that arrives while an activity runs is visible on the next workflow
task, and a late or duplicate input cannot replace an already recorded winner.

## Nested Barriers

Nested `all([...])` groups let one workflow step express a tree of durable fan-out and fan-in work. The runtime schedules every activity or child workflow as a durable leaf sequence, records the leaf's full `parallel_group_path`, waits until every enclosing barrier can make progress, and then rebuilds the original nested result shape before resuming the workflow body. During replay, an activity or child leaf from an `all([...])` step must still match that recorded group path; typed leaf history that has no group metadata is treated as incompatible older preview history instead of being guessed into the current barrier.

```php
use function Workflow\V2\{all, activity};
use Workflow\V2\Workflow;

final class NestedWorkflow extends Workflow
{
    public function handle(): array
    {
        return all([
            fn () => activity(BuildSummary::class),
            fn () => all([
                fn () => activity(BuildInvoice::class),
                fn () => activity(BuildShipment::class),
            ]),
        ]);
    }
}
```

In that example, Waterline exposes three open leaf waits, not one synthetic "nested" wait. The first leaf belongs only to the outer barrier, while the second and third leaves expose a two-entry `parallel_group_path` so operators can see both the outer group and the inner subgroup that is still open.

The outer group size counts durable leaves, not nested arrays.
Each nested leaf carries an outer-to-inner `parallel_group_path`; every path
entry preserves the same durable workflow position. The group schedules all
leaves before it suspends, then assembles successful values by input position.
Worker restart and completed-history replay rebuild the same group identity.
Exact duplicate terminal delivery is ignored, and a late sibling completion
can enrich partial diagnostics without changing the failed member already
selected by the embedded runtime's deterministic policy.

The embedded runtime raises the typed leaf failure from `all()` and retains
the barrier's durable group metadata in history and operator views.

## Async Callback

`async(...)` runs a serializable callback as a durable child workflow with the system type `durable-workflow.async`. Async callbacks use the same straight-line-only helper contract as named v2 workflows, so `activity()`, `await()`, `timer()`, `sideEffect()`, and the other single-step helpers suspend directly inside the callback body without forcing `yield`.

```php
use function Workflow\V2\{activity, async};
use Workflow\V2\Workflow;

final class CustomerWorkflow extends Workflow
{
    public function handle(string $customerId): array
    {
        $profile = async(static function () use ($customerId): array {
            $customer = activity(LoadCustomer::class, $customerId);

            return [
                'customer' => $customer,
                'score' => activity(ScoreCustomer::class, $customer['id']),
            ];
        });

        return ['profile' => $profile];
    }
}
```

The parent run sees the callback as a child wait, so command history, lineage, and Waterline detail use the same `child_call_id`, child run id, and child outcome history as an explicit `child(...)` call. The callback is serialized with Laravel's serializable-closure support, so keep it app-local and deployment-local. Use a named `child(SomeWorkflow::class, ...)` call when the work needs a stable public workflow type for cross-service routing or long-lived code evolution. `async(...)` callbacks are now straight-line only in v2, so call helpers like `activity()`, `child()`, `await()`, `timer()`, and `all([...])` directly without `yield`.

## Mixed Activity + Child Barriers

The same `all()` helper can also fan in a mixed group of activities and child workflows. Results still come back in the original array order, successful members still wait for the rest of the group, and the first failed member still wakes the parent immediately.

When more than one barrier member has already closed unsuccessfully by the time the parent replays, the parent receives the failure with the earliest recorded close time. If two failures have the same recorded time, the lower barrier leaf index wins, so workflow resume and query replay select the same exception. Later sibling failures do not replace the exception that has already been thrown into the parent step.

```php
use function Workflow\V2\{all, activity, child};
use Workflow\V2\Workflow;

final class OrderWorkflow extends Workflow
{
    public function handle(): array
    {
        [$charge, $shipment] = all([
            fn () => activity(ChargeCustomer::class),
            fn () => child(ShipOrderWorkflow::class),
        ]);

        return compact('charge', 'shipment');
    }
}
```

## Current Limits

The current concurrency surface does not yet include:

- built-in bounded-concurrency helpers beyond explicit nested `all([...])` groups

## Child Workflows in Parallel

Child workflows can also run in their own `all([...])` barrier. It works the same way as parallel activity execution, but for child workflows: the parent fans out several child runs durably and resumes only when the whole child barrier can make progress.

```php
use function Workflow\V2\{all, child};
use Workflow\V2\Workflow;

final class ParentWorkflow extends Workflow
{
    public function handle(): array
    {
        return all([
            fn () => child(MyChild1::class),
            fn () => child(MyChild2::class),
            fn () => child(MyChild3::class),
        ]);
    }
}
```

This makes it easy to build hierarchical parallelism into your workflows, including nested child-only or mixed child-plus-activity groups when one parent step needs more than one fan-in layer.

<!-- Source: docs/features/sagas.md -->

# Sagas

This guide covers compensation in embedded Laravel workflows using
`Workflow\V2\Workflow`. For service-mode workers, use the
[PHP SDK guide](https://php.durable-workflow.com/build/workflows-activities/),
[Python SDK guide](https://python.durable-workflow.com/sdk-reference/#saga-compensation), or
[Rust SDK reference](https://rust.durable-workflow.com/durable_workflow/struct.Saga.html).

Sagas are an established design pattern for managing complex, long-running operations:

- A saga manages distributed transactions using a sequence of local transactions.
- A local transaction is a work unit performed by a saga participant (an activity).
- Each operation in the saga can be reversed by a compensatory activity.
- The saga pattern assures that all operations are either completed successfully or the corresponding compensation activities are run to undo any completed work.

```php
use function Workflow\V2\activity;
use Workflow\V2\Attributes\Type;
use Workflow\V2\Workflow;

#[Type('booking-saga')]
class BookingSagaWorkflow extends Workflow
{
    public function handle(): array
    {
        try {
            $flightId = activity(BookFlightActivity::class);
            $this->addCompensation(fn () => activity(CancelFlightActivity::class, $flightId));

            $hotelId = activity(BookHotelActivity::class);
            $this->addCompensation(fn () => activity(CancelHotelActivity::class, $hotelId));

            $carId = activity(BookRentalCarActivity::class);
            $this->addCompensation(fn () => activity(CancelRentalCarActivity::class, $carId));

            return compact('flightId', 'hotelId', 'carId');
        } catch (\Throwable $e) {
            $this->compensate();

            throw $e;
        }
    }
}
```

When the workflow catches an exception, `$this->compensate()` runs every registered compensation in **reverse order**. In the example above, if `BookRentalCarActivity` fails, the engine cancels the hotel first and then the flight — unwinding the saga from the most recent step backward.

Register each compensation after its forward activity succeeds. Replay
reconstructs that registration order and reuses recorded activity results,
including completed compensations.

## Compensation ordering

By default, compensations execute **sequentially in reverse registration order**. This is the safest default because later steps may depend on earlier ones.

By default, `compensate()` stops at the first compensation failure and
propagates it to the caller.

## Parallel compensation

To run compensations in parallel, use `setParallelCompensation(true)`. When parallel compensation is enabled, each compensation closure should return a started (but not awaited) activity call so the engine can execute them concurrently:

```php
use function Workflow\V2\activity;

use Workflow\V2\Attributes\Type;
use Workflow\V2\Workflow;

#[Type('parallel-saga')]
class ParallelSagaWorkflow extends Workflow
{
    public function handle(): void
    {
        $this->setParallelCompensation(true);

        try {
            $flightId = activity(BookFlightActivity::class);
            $this->addCompensation(fn () => activity(CancelFlightActivity::class, $flightId));

            $hotelId = activity(BookHotelActivity::class);
            $this->addCompensation(fn () => activity(CancelHotelActivity::class, $hotelId));

            activity(ChargePaymentActivity::class);
        } catch (\Throwable $e) {
            $this->compensate();

            throw $e;
        }
    }
}
```

When parallel compensation is enabled, compensation closures return activity calls that the engine collects and runs through `all()`.

## Continue with error

By default, if a compensation activity throws an exception, the remaining compensations are skipped and the error propagates. To run all compensations regardless of individual failures, use `setContinueWithError(true)`:

```php
$this->setContinueWithError(true);
```

When enabled, the engine catches and discards exceptions from each compensation closure and continues to the next one. This is useful when compensations are independent and you want a best-effort cleanup even if some steps fail.

### Combining both flags

`setParallelCompensation(true)` and `setContinueWithError(true)` can be used together. When both are enabled, all compensations run concurrently through `all()`, and if any compensation throws, the error is caught so the remaining compensations still complete. Without `setContinueWithError(true)`, a parallel compensation failure propagates immediately and the workflow fails.

```php
$this->setParallelCompensation(true);
$this->setContinueWithError(true);
```

## How it works

- `addCompensation()` registers a callable that will be invoked during `compensate()`
- `compensate()` iterates the registered compensations in reverse order
- each compensation closure is a normal V2 workflow step — the activities it calls produce durable history events just like any other activity
- compensation activities are visible in Waterline's timeline and history export
- if the workflow succeeds, compensation closures are never invoked and produce no history

## Run this pattern

The signal-driven travel-agent saga in the
[Sample App](/docs/sample-app) is the runnable reference for this
page. Clone the sample app, set `OPENAI_API_KEY`, and run:

```bash
php artisan app:ai
```

`App\Workflows\Ai\AiWorkflow` registers compensations on every booking
activity, so a flight failure after a successful hotel booking unwinds
the hotel through the compensation list — the same pattern this page
describes, with the events visible in Waterline's run timeline.

<!-- Source: docs/features/events.md -->

# Events

Lifecycle events are dispatched at key stages of workflow and activity execution to notify your application of progress, completion, or failures. These are standard Laravel events — register listeners in your `EventServiceProvider` or with `Event::listen()`.

All V2 lifecycle events are dispatched **after the durable state is committed** to the database. This means listeners only observe events backed by committed truth — if a transaction rolls back, no event is dispatched.

## Event Identity

Every V2 event carries durable identity fields:

| Field | Description |
|---|---|
| `instanceId` | The workflow instance ID (stable across continue-as-new). |
| `runId` | The specific execution run ID. |
| `workflowType` | The durable type key registered via `#[Type('...')]`. |
| `workflowClass` | The PHP class name of the workflow. |
| `committedAt` | ISO 8601 timestamp of when the durable record was committed (wall-clock commit time). |

Activity events additionally include:

| Field | Description |
|---|---|
| `activityExecutionId` | The durable activity execution ID. |
| `activityType` | The durable type key of the activity. |
| `activityClass` | The PHP class name of the activity. |
| `sequence` | The position of the activity within the workflow execution. |
| `attemptNumber` | The attempt number (starts at 1). |

## Workflow Events

### WorkflowStarted

Dispatched when a workflow start is durably committed — meaning the first run has been created and the `WorkflowStarted` history event has been recorded.

```php
use Workflow\V2\Events\WorkflowStarted;

Event::listen(WorkflowStarted::class, function (WorkflowStarted $event) {
    Log::info('Workflow started', [
        'instance_id' => $event->instanceId,
        'run_id' => $event->runId,
        'type' => $event->workflowType,
    ]);
});
```

This event also fires when a new run begins via continue-as-new.

### WorkflowCompleted

Dispatched when a workflow run completes successfully.

```php
use Workflow\V2\Events\WorkflowCompleted;

Event::listen(WorkflowCompleted::class, function (WorkflowCompleted $event) {
    Log::info('Workflow completed', [
        'instance_id' => $event->instanceId,
        'run_id' => $event->runId,
    ]);
});
```

### WorkflowFailed

Dispatched when a workflow run fails terminally.

Additional fields:
- `exceptionClass`: The PHP exception class name.
- `message`: The exception message.

```php
use Workflow\V2\Events\WorkflowFailed;

Event::listen(WorkflowFailed::class, function (WorkflowFailed $event) {
    Log::error('Workflow failed', [
        'instance_id' => $event->instanceId,
        'exception' => $event->exceptionClass,
        'message' => $event->message,
    ]);
});
```

## Activity Events

### ActivityStarted

Dispatched when an activity task is claimed and execution begins.

```php
use Workflow\V2\Events\ActivityStarted;

Event::listen(ActivityStarted::class, function (ActivityStarted $event) {
    Log::info('Activity started', [
        'activity' => $event->activityType,
        'sequence' => $event->sequence,
        'attempt' => $event->attemptNumber,
    ]);
});
```

### ActivityCompleted

Dispatched when an activity completes successfully.

```php
use Workflow\V2\Events\ActivityCompleted;

Event::listen(ActivityCompleted::class, function (ActivityCompleted $event) {
    Log::info('Activity completed', [
        'activity' => $event->activityType,
        'execution_id' => $event->activityExecutionId,
    ]);
});
```

### ActivityFailed

Dispatched when an activity fails terminally (all retries exhausted or non-retryable exception). Retryable failures that will be retried do **not** trigger this event.

Additional fields:
- `exceptionClass`: The PHP exception class name.
- `message`: The exception message.

```php
use Workflow\V2\Events\ActivityFailed;

Event::listen(ActivityFailed::class, function (ActivityFailed $event) {
    Log::error('Activity failed', [
        'activity' => $event->activityType,
        'exception' => $event->exceptionClass,
        'message' => $event->message,
    ]);
});
```

## Failure Events

### FailureRecorded

Dispatched whenever a durable failure record is committed — for both workflow and activity terminal failures. This is the single hook for error-reporting integrations like Sentry or Bugsnag.

| Field | Description |
|---|---|
| `failureId` | The durable failure record ID. |
| `sourceKind` | `"workflow_run"` or `"activity_execution"`. |
| `sourceId` | The ID of the source (run ID or activity execution ID). |
| `exceptionClass` | The PHP exception class name. |
| `message` | The exception message. |

```php
use Workflow\V2\Events\FailureRecorded;

Event::listen(FailureRecorded::class, function (FailureRecorded $event) {
    // Report to Sentry, Bugsnag, etc.
    report(new \RuntimeException(
        "[{$event->sourceKind}] {$event->exceptionClass}: {$event->message}"
    ));
});
```

## Timestamp Semantics

The `committedAt` field on all events represents **commit time** — the wall-clock time at which the durable record (history event or failure record) was written to the database. This is distinct from:

- **Workflow virtual time**: The logical time inside the workflow execution (used by timers).
- **Attempt time**: When a specific activity attempt started or finished.
- **Resume latency**: How long after a timer was due the workflow actually resumed.

Commit time is the most useful timestamp for external integrations because it reflects when the state became durable and observable.

## Lifecycle

A typical successful workflow lifecycle:

```
Workflow\V2\Events\WorkflowStarted
Workflow\V2\Events\ActivityStarted
Workflow\V2\Events\ActivityCompleted
Workflow\V2\Events\WorkflowCompleted
```

A workflow lifecycle with a terminal activity failure:

```
Workflow\V2\Events\WorkflowStarted
Workflow\V2\Events\ActivityStarted
Workflow\V2\Events\ActivityFailed
Workflow\V2\Events\FailureRecorded    (source: activity_execution)
Workflow\V2\Events\WorkflowFailed
Workflow\V2\Events\FailureRecorded    (source: workflow_run)
```

## Event Namespace

V2 events live in the `Workflow\V2\Events` namespace. If you are upgrading an existing app, see [Migration](../migration.md) for the V1 compatibility event mapping.

<!-- Source: docs/features/webhooks.md -->

# Webhooks

The framework provides webhooks that allow external systems to start workflows and send signals dynamically. This feature enables seamless integration with external services, APIs, and automation tools.

## Enabling Webhooks
To enable webhooks, register the webhook routes in your application’s routes file (`routes/web.php` or `routes/api.php`):

```php
use Workflow\V2\Webhooks;

Webhooks::routes([
    App\Workflows\OrderWorkflow::class,
    'manual-invoice' => App\Workflows\InvoiceWorkflow::class,
]);
```

Pass the explicit map of workflow classes you want to expose. Each alias becomes the public route segment, and each workflow class must carry a stable durable type key via `#[Type(...)]` or a registered entry in `workflows.v2.types.workflows`. See the [Explicit Command And Query Webhooks](#explicit-command-and-query-webhooks) section for the full route matrix.

## Visibility Metadata
When you register routes through `Workflow\V2\Webhooks::routes(...)`, the start route also accepts a reserved `visibility` object. Those fields are stored as workflow visibility metadata and are not passed to the workflow `handle()` method.

```bash
curl -X POST "https://example.com/webhooks/start/order-workflow" \
     -H "Content-Type: application/json" \
     -d '{
           "workflow_id": "order-123",
           "orderId": 123,
           "visibility": {
             "business_key": "order-123",
             "labels": {
               "tenant": "acme",
               "region": "us-east"
             },
             "memo": {
               "customer": {
                 "id": 42,
                 "name": "Taylor"
               },
               "source": "checkout"
             }
           }
         }'
```

`business_key` and `visibility.labels` are copied onto the instance, run, run summary, typed start history, selected-run detail, and history export. Waterline can filter list screens by those fields with exact-match query parameters.

`visibility.memo` is copied onto the instance, run, typed start history, selected-run detail, and history export, and later `continueAsNew()` runs inherit the same memo by default. It must be a JSON object at the top level, with nested scalars, `null`, arrays, or objects. `memo` is returned-only metadata, not a list-filter or run-summary search field.

## Webhook Authentication
By default, webhooks don't require authentication, but you can configure one of several strategies in `config/workflows.php`.

**Important:** If webhook URLs are shared with external parties or exposed publicly, enable authentication (token or HMAC signature) to prevent unauthorized access.

### Authentication Methods
It supports:
1. No Authentication (none)
2. Token-based Authentication (token)
3. HMAC Signature Verification (signature)
4. Custom Authentication (custom)

### Token Authentication
For token authentication, webhooks require a valid API token in the request headers. The default header is `Authorization` but you can change this in the configuration settings.

#### Example Request
```bash
curl -X POST "https://example.com/webhooks/start/order-workflow" \
     -H "Content-Type: application/json" \
     -H "Authorization: your-api-token" \
     -d '{"orderId": 123}'
```

### HMAC Signature Authentication
For HMAC authentication, it verifies requests using a secret key. The default header is `X-Signature` but this can also be changed.

#### Example Request
```bash
BODY='{"orderId": 123}'
SIGNATURE=$(echo -n "$BODY" | openssl dgst -sha256 -hmac "your-secret-key" | awk '{print $2}')

curl -X POST "https://example.com/webhooks/start/order-workflow" \
     -H "Content-Type: application/json" \
     -H "X-Signature: $SIGNATURE" \
     -d "$BODY"
```

### Custom Authentication
To use a custom authenticator, create a class that implements the `WebhookAuthenticator` interface:

```php
use Illuminate\Http\Request;
use Workflow\Auth\WebhookAuthenticator;

class CustomAuthenticator implements WebhookAuthenticator
{
    public function validate(Request $request): Request
    {
        $allow = true;

        if ($allow) {
            return $request;
        } else {
            abort(401, 'Unauthorized');
        }
    }
}
```

Then configure it in `config/workflows.php`:

```php
'webhook_auth' => [
    'method' => 'custom',
    'custom' => [
        'class' => App\Your\CustomAuthenticator::class,
    ],
],
```

The `validate()` method should return the `Request` if valid, or call `abort(401)` if unauthorized.

## Configuring Webhook Routes
By default, webhooks are accessible under `/webhooks`. You can customize the route path in `config/workflows.php`:

```php
'webhooks_route' => 'workflows',
```

After this change, webhooks will be accessible under:
```
POST /workflows/start/order-workflow
POST /workflows/signal/order-workflow/{workflowId}/mark-as-shipped
POST /workflows/instances/{workflowId}/queries/{query}
POST /workflows/instances/{workflowId}/runs/{runId}/queries/{query}
POST /workflows/instances/{workflowId}/signals/{signal}
POST /workflows/instances/{workflowId}/runs/{runId}/signals/{signal}
POST /workflows/instances/{workflowId}/repair
POST /workflows/instances/{workflowId}/cancel
POST /workflows/instances/{workflowId}/terminate
GET  /workflows/instances/{workflowId}/describe
POST /workflows/instances/{workflowId}/runs/{runId}/repair
POST /workflows/instances/{workflowId}/runs/{runId}/cancel
POST /workflows/instances/{workflowId}/runs/{runId}/terminate
GET  /workflows/instances/{workflowId}/runs/{runId}/describe
GET  /workflows/workflow-tasks/poll
GET  /workflows/activity-tasks/poll
POST /workflows/control-plane/start
```

## Explicit Command And Query Webhooks

Durable start, signal, update, repair, cancel, and terminate commands are exposed over HTTP, plus replay-safe query routes for current-run and selected-run reads.

You register an explicit alias map and route only the workflows you want to expose.

```php
use Workflow\V2\Webhooks;

Webhooks::routes([
    App\Workflows\OrderWorkflow::class,
    'manual-invoice' => App\Workflows\InvoiceWorkflow::class,
]);
```

When you register a workflow class directly, it must define a stable type key with `#[Type(...)]` or be registered under `workflows.v2.types.workflows`:

```php
use Workflow\V2\Attributes\Type;
use Workflow\V2\Workflow;

#[Type('order-workflow')]
class OrderWorkflow extends Workflow
{
    public function handle(int $orderId)
    {
        // ...
    }
}
```

Equivalent config registration:

```php
// config/workflows.php
'v2' => [
    'types' => [
        'workflows' => [
            'order-workflow' => App\Workflows\OrderWorkflow::class,
        ],
    ],
],
```

The resulting start route is:

```
POST /webhooks/start/order-workflow
```

Example request:

```bash
curl -X POST "https://example.com/webhooks/start/order-workflow" \
     -H "Content-Type: application/json" \
     -d '{"workflow_id":"order-123","orderId":123}'
```

The `workflow_id` is the opaque public workflow instance id. It is not a run id and should be treated as an opaque string.

Caller-supplied `workflow_id` values must be non-empty URL-safe strings up to 191 characters using only letters, numbers, `.`, `_`, `-`, and `:`. Blank, overlong, or unsupported-character ids are rejected at webhook validation time with HTTP `422`.

The same durable type map is also the current worker fallback when a stored workflow class name drifts after a refactor. If you keep `order-workflow` stable and repoint the config registration to the new class, queued v2 work can still resolve the durable type key even when the original stored PHP class name is no longer loadable.

You can also request explicit duplicate-start behavior:

```bash
curl -X POST "https://example.com/webhooks/start/order-workflow" \
     -H "Content-Type: application/json" \
     -d '{"workflow_id":"order-123","orderId":123,"on_duplicate":"return_existing_active"}'
```

Supported `on_duplicate` values are:

- `reject_duplicate`
- `return_existing_active`

All command webhooks return the same JSON envelope:

```json
{
  "outcome": "started_new",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "command_source": "webhook",
  "rejection_reason": null
}
```

Accepted response when a new run is created:

```json
{
  "outcome": "started_new",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "command_source": "webhook",
  "rejection_reason": null
}
```

Accepted response when the caller requested reuse of the existing active run:

```json
{
  "outcome": "returned_existing_active",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "command_source": "webhook",
  "rejection_reason": null
}
```

Rejected duplicate-start response:

```json
{
  "outcome": "rejected_duplicate",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "requested_run_id": null,
  "resolved_run_id": "01J...",
  "command_id": "01J...",
  "workflow_type": "order-workflow",
  "command_status": "rejected",
  "command_source": "webhook",
  "rejection_reason": "instance_already_started"
}
```

Response fields:

- `workflow_id` is the public workflow instance id.
- `run_id` is the first active run created by the accepted start command.
- `requested_run_id` is only set when the caller explicitly addressed one selected run.
- `resolved_run_id` is the run the engine actually resolved for this command; for instance-targeted starts it matches `run_id`.
- `command_id` is the durable start-command id.
- `command_source` is `webhook` for the webhook routes.
- instance-targeted signal, update, repair, cancel, and terminate routes resolve the newest durable run for that instance instead of trusting only the mutable current-run pointer, so continue-as-new chains stay addressable through the same public id even if that column drifts.
- run-targeted command routes pin one selected run under the same public instance id and reject with `target_scope = run`, `outcome = rejected_not_current`, and `rejection_reason = selected_run_not_current` if that run is no longer current; in that case `run_id` and `requested_run_id` stay on the historical selection while `resolved_run_id` points at the current run that callers should address next.
- payload keys are matched to the workflow `handle()` parameter names in declaration order.
- Workflows define their start arguments on `handle()`.
- Workflow classes that do not declare `handle()` are rejected with HTTP `422` validation errors before a run is created.
- missing required payload keys are rejected with HTTP `422` validation errors instead of being silently dropped.
- blank, overlong, or non-route-safe `workflow_id` values are rejected with HTTP `422` validation errors instead of falling through duplicate-start handling.
- invalid `on_duplicate` values are rejected with HTTP `422` validation errors.
- `on_duplicate = return_existing_active` returns HTTP `200` when the current active run is reused.
- duplicate starts that are not reused return HTTP `409` with `outcome = rejected_duplicate`, `command_status = rejected`, and `rejection_reason = instance_already_started`.
- the durable `workflow_commands` row also stores webhook ingress context separately from business arguments, including the caller label, auth method or outcome, request route name, request path, and a request fingerprint derived from the normalized payload plus selected request headers such as `X-Request-Id` and `X-Correlation-Id`

Current HTTP response matrix for the start webhook:

- `202` with `outcome = started_new` when a new run is created
- `200` with `outcome = returned_existing_active` when the existing active run is reused
- `409` with `outcome = rejected_duplicate` when the duplicate start is rejected
- `401` for webhook auth failure
- `404` for an unknown alias
- `422` for payload validation failure

### Signal Command Webhooks

Both instance-targeted and run-targeted signal commands are available:

```text
POST /webhooks/instances/{workflowId}/signals/{signal}
POST /webhooks/instances/{workflowId}/runs/{runId}/signals/{signal}
```

`workflowId` is the public workflow instance id. The instance-targeted route resolves the current active run at apply time. The run-targeted route also takes one selected `runId` and rejects if that selected run is historical. The signal name comes from the route parameter. The request body currently accepts one optional top-level field:

```json
{
  "arguments": ["Taylor"]
}
```

The `arguments` field must be an array. When it is omitted, a workflow waiting with `await('signal-name')` resumes with `true`.
The targeted workflow class must also declare the signal name with `#[Workflow\V2\Attributes\Signal('...')]`; undeclared names are rejected durably instead of being buffered blindly.

Signal example:

```bash
curl -X POST "https://example.com/webhooks/instances/order-123/signals/approved-by" \
     -H "Content-Type: application/json" \
     -d '{"arguments":["Taylor"]}'
```

Accepted signal response:

```json
{
  "outcome": "signal_received",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "requested_run_id": null,
  "resolved_run_id": "01J...",
  "command_id": "01J...",
  "command_sequence": 2,
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "command_source": "webhook",
  "rejection_reason": null
}
```

Accepted and rejected signal commands also get one first-class durable `workflow_signal_records` lifecycle row linked back to the originating command. Signal command responses still return the command id as the HTTP correlation id; Waterline selected-run detail and history exports expose the lifecycle row as `signals[*].id`, together with the signal name, `signal_wait_id`, command sequence, workflow sequence once applied, status, outcome, validation errors, and stored arguments.

Rejected response when the instance exists but has not started a run yet:

```json
{
  "outcome": "rejected_not_started",
  "workflow_id": "order-123",
  "run_id": null,
  "command_id": "01J...",
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "rejected",
  "rejection_reason": "instance_not_started"
}
```

Rejected response when the current run is already closed:

```json
{
  "outcome": "rejected_not_active",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "rejected",
  "rejection_reason": "run_not_active"
}
```

Rejected response when the route targets an undeclared signal name:

```json
{
  "outcome": "rejected_unknown_signal",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "rejected",
  "rejection_reason": "unknown_signal"
}
```

Rejected response when a run-targeted signal route addresses a historical run:

```json
{
  "outcome": "rejected_not_current",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "requested_run_id": "01J...",
  "resolved_run_id": "01J...",
  "command_id": "01J...",
  "target_scope": "run",
  "workflow_type": "order-workflow",
  "command_status": "rejected",
  "rejection_reason": "selected_run_not_current"
}
```

Current HTTP response matrix for the signal webhook:

- `202` with `outcome = signal_received` when the signal command is accepted
- `401` for webhook auth failure
- `404` for an unknown workflow instance id
- `404` with `outcome = rejected_unknown_signal` when the workflow does not declare that signal name; when the run already carries a typed `WorkflowStarted` contract snapshot, or when an older `WorkflowStarted` event can be backfilled on first compatible intake, that rejection no longer depends on reflecting the live workflow class first
- `409` with `outcome = rejected_not_started` when the instance has no current run yet
- `409` with `outcome = rejected_not_active` when the targeted current run is already closed
- `409` with `outcome = rejected_not_current` when the run-targeted route addresses a historical selected run
- `422` when `arguments` is present but not an array

### Signal-With-Start Webhooks

One instance-targeted linked-intake route starts a new run or reuses the current active run before recording a signal:

```text
POST /webhooks/start/{alias}/signals/{signal}
```

`alias` is the webhook workflow alias you registered in `Workflow\V2\Webhooks::routes([...])`. `signal` is the durable signal name declared by `#[Signal(...)]`.

The request body combines the normal start fields plus one reserved `signal_arguments` field for the attached signal payload:

```json
{
  "workflow_id": "order-123",
  "signal_arguments": ["Taylor"],
  "visibility": {
    "business_key": "order-123"
  }
}
```

`signal_arguments` must be an array. The route defaults `on_duplicate` to `return_existing_active`, and rejects any other duplicate policy value for this linked-intake route.

Accepted response when the route starts a new run:

```json
{
  "outcome": "signal_received",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "requested_run_id": null,
  "resolved_run_id": "01J...",
  "command_id": "01J...",
  "command_sequence": 2,
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "command_source": "webhook",
  "start_command_id": "01J...",
  "start_command_sequence": 1,
  "start_outcome": "started_new",
  "start_command_status": "accepted",
  "intake_group_id": "01J...",
  "rejection_reason": null
}
```

When the workflow instance already has an active current run, the same route returns `start_outcome = returned_existing_active` and keeps the signal command as the top-level `command_*` response object. The linked start and signal commands share the same `intake_group_id`, request route metadata, and request correlation headers, and the runtime records both commands in one transaction before the worker can run the first user workflow step.

Selected-run detail and history export freeze that same compound intake under `linked_intakes[*]`. The durable authority is the shared `workflow_commands.context.intake.group_id` plus `mode` recorded on each accepted command row. For the `signal_with_start` mode, the grouped row exposes `start_command_*`, the non-start `primary_command_*`, `complete`, `missing_expected_command_types`, and the ordered nested `commands[*]` snapshots.

If the signal name is unknown or the signal payload fails durable contract validation, the runtime rejects the signal command before creating a new run. In that case the response leaves `run_id`, `start_command_id`, and `start_outcome` as `null`.

Current HTTP response matrix for the signal-with-start webhook:

- `202` with `outcome = signal_received` and `start_outcome = started_new` when the request creates a run and records the signal
- `202` with `outcome = signal_received` and `start_outcome = returned_existing_active` when the request reuses the current active run and records the signal
- `401` for webhook auth failure
- `404` for an unknown webhook workflow alias
- `404` with `outcome = rejected_unknown_signal` when the workflow does not declare that signal name
- `422` with `outcome = rejected_invalid_arguments` when `signal_arguments` is an array but does not satisfy the durable signal contract
- `422` when `signal_arguments` is present but not an array, or when `on_duplicate` is set to any value other than `return_existing_active`

### Query Webhooks

Both instance-targeted and run-targeted query routes are available:

```text
POST /webhooks/instances/{workflowId}/queries/{query}
POST /webhooks/instances/{workflowId}/runs/{runId}/queries/{query}
```

`workflowId` is the public workflow instance id. The instance-targeted route resolves the newest durable run for that instance after refresh. The run-targeted route pins one selected `runId`, including historical or already-closed runs, because queries are read-only replay operations.

`query` is the public query target declared by `#[QueryMethod]`. When the workflow definition is still loadable, the route also accepts the underlying PHP method name, but successful responses normalize `query_name` back to the durable public target.

The request body accepts either a positional `arguments` list or a named `arguments` map keyed by the declared query parameters:

```json
{
  "arguments": ["start"]
}
```

```json
{
  "arguments": {
    "prefix": "start"
  }
}
```

Unlike the mutating webhook routes, query webhooks do not append a durable command row. They replay the selected run and return a serialized result immediately.

Query example:

```bash
curl -X POST "https://example.com/webhooks/instances/order-123/queries/events-starting-with" \
     -H "Content-Type: application/json" \
     -d '{"arguments":{"prefix":"start"}}'
```

Accepted query response:

```json
{
  "query_name": "events-starting-with",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "target_scope": "instance",
  "result": "i:1;"
}
```

Rejected response when the query arguments do not match the declared contract:

```json
{
  "query_name": "events-starting-with",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "target_scope": "instance",
  "message": "Workflow query [events-starting-with] received invalid arguments.",
  "validation_errors": {
    "prefix": [
      "The prefix argument is required."
    ]
  }
}
```

Rejected response when the selected run's workflow definition can no longer be replayed:

```json
{
  "query_name": "events-starting-with",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "target_scope": "instance",
  "blocked_reason": "workflow_definition_unavailable",
  "message": "Workflow 01J... [order-123] cannot execute query [events-starting-with] because the workflow definition is unavailable for durable type [order-workflow]."
}
```

Current HTTP response matrix for the query webhook:

- `200` with a serialized `result` when the query replay succeeds
- `401` for webhook auth failure
- `404` for an unknown workflow instance id or selected run id
- `409` when the selected instance has not started yet, when the requested query is not declared on that selected run, or when replay is blocked by `blocked_reason = workflow_definition_unavailable`
- `422` when the declared query contract rejects the provided `arguments`

### Update Command Webhooks

Instance-targeted and run-targeted update POST routes plus durable lifecycle lookup routes are available:

```text
POST /webhooks/instances/{workflowId}/updates/{update}
POST /webhooks/instances/{workflowId}/runs/{runId}/updates/{update}
GET /webhooks/instances/{workflowId}/updates/{updateId}
GET /webhooks/instances/{workflowId}/runs/{runId}/updates/{updateId}
```

`workflowId` is the public workflow instance id. `update` is the durable update name declared by `#[UpdateMethod]`. `updateId` is the durable lifecycle id returned by update responses. The instance-targeted POST route resolves the current active run at apply time. The run-targeted POST route also takes one selected `runId` and rejects if that selected run is historical. The instance-targeted GET route reads the stored lifecycle for that workflow instance, while the run-targeted GET route narrows lookup to the selected run.

If you use `#[UpdateMethod('mark-approved')]`, the public webhook target is `mark-approved` even if the underlying PHP method is named differently.

The request body accepts either a positional `arguments` list or a named `arguments` map keyed by the declared update parameters:

```json
{
  "arguments": [true, "api"]
}
```

```json
{
  "arguments": {
    "approved": true
  }
}
```

Set `wait_for` to `accepted` when the caller only needs the durable update command to be accepted and wants the workflow worker to apply the update later:

```json
{
  "arguments": {
    "approved": true
  },
  "wait_for": "accepted"
}
```

When `wait_for` is omitted or set to `completed`, the webhook records the accepted update first, then waits up to the configured completion budget for the workflow worker to apply it. Override that budget per request with `wait_timeout_seconds`:

```json
{
  "arguments": {
    "approved": true
  },
  "wait_timeout_seconds": 5
}
```

If the worker completes within that budget, the webhook returns the normal completed or failed update response. If the wait budget expires first, the webhook returns HTTP `202` with the still-open accepted lifecycle instead of blocking indefinitely. When `wait_for` is `accepted`, the webhook records the command row, the first-class update lifecycle row, and typed `UpdateAccepted` history, then schedules or re-dispatches a workflow task and returns HTTP `202` without waiting for application.

Every update POST and GET response includes the normal command fields plus `update_id`, `update_name`, `update_status`, `workflow_sequence`, `accepted_at`, `applied_at`, `rejected_at`, and `closed_at`. Accepted-but-open lifecycles return `workflow_sequence = null`, `result = null`, `applied_at = null`, `rejected_at = null`, and `closed_at = null` until the worker closes the lifecycle.

When a named map is accepted, the engine normalizes it into the declared parameter order before it appends typed `UpdateAccepted` or `UpdateApplied` history. Optional parameters also fill from their PHP defaults during that normalization step.

Every accepted or rejected update is backed by one durable `workflow_updates` row keyed by `update_id`, so callers can POST once and later GET the stored lifecycle without inferring state from generic command rows.

Update example:

```bash
curl -X POST "https://example.com/webhooks/instances/order-123/updates/mark-ready" \
     -H "Content-Type: application/json" \
     -d '{"arguments":{"approved":true}}'
```

Completed update response:

```json
{
  "outcome": "update_completed",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "update_id": "01J...",
  "command_sequence": 2,
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "command_source": "webhook",
  "update_name": "mark-ready",
  "update_status": "completed",
  "workflow_sequence": 1,
  "accepted_at": "2026-04-10T12:00:00.000000Z",
  "applied_at": "2026-04-10T12:00:01.000000Z",
  "rejected_at": null,
  "closed_at": "2026-04-10T12:00:01.000000Z",
  "wait_for": "completed",
  "wait_timed_out": false,
  "wait_timeout_seconds": 10,
  "rejection_reason": null,
  "validation_errors": [],
  "result": {
    "approved": true
  },
  "failure_id": null,
  "failure_message": null
}
```

Accepted-only update response:

```json
{
  "outcome": null,
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "update_id": "01J...",
  "command_sequence": 2,
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "command_source": "webhook",
  "update_name": "mark-ready",
  "update_status": "accepted",
  "workflow_sequence": null,
  "accepted_at": "2026-04-10T12:00:00.000000Z",
  "applied_at": null,
  "rejected_at": null,
  "closed_at": null,
  "wait_for": "accepted",
  "wait_timed_out": false,
  "wait_timeout_seconds": null,
  "rejection_reason": null,
  "validation_errors": [],
  "result": null,
  "failure_id": null,
  "failure_message": null
}
```

Lifecycle lookup example:

```bash
curl "https://example.com/webhooks/instances/order-123/updates/01J..." \
     -H "Content-Type: application/json"
```

Lookup response while the update is still open:

```json
{
  "outcome": null,
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "update_id": "01J...",
  "command_sequence": 2,
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "command_source": "webhook",
  "update_name": "mark-ready",
  "update_status": "accepted",
  "workflow_sequence": null,
  "accepted_at": "2026-04-10T12:00:00.000000Z",
  "applied_at": null,
  "rejected_at": null,
  "closed_at": null,
  "wait_for": "status",
  "wait_timed_out": false,
  "wait_timeout_seconds": null,
  "rejection_reason": null,
  "validation_errors": [],
  "result": null,
  "failure_id": null,
  "failure_message": null
}
```

Lookup response after the lifecycle closes:

```json
{
  "outcome": "update_completed",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "update_id": "01J...",
  "command_sequence": 2,
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "command_source": "webhook",
  "update_name": "mark-ready",
  "update_status": "completed",
  "workflow_sequence": 1,
  "accepted_at": "2026-04-10T12:00:00.000000Z",
  "applied_at": "2026-04-10T12:00:01.000000Z",
  "rejected_at": null,
  "closed_at": "2026-04-10T12:00:01.000000Z",
  "wait_for": "status",
  "wait_timed_out": false,
  "wait_timeout_seconds": null,
  "rejection_reason": null,
  "validation_errors": [],
  "result": {
    "approved": true
  },
  "failure_id": null,
  "failure_message": null
}
```

Lookup routes do not wait for execution. They only read the stored durable lifecycle, so they always return `wait_for = status`, `wait_timed_out = false`, and `wait_timeout_seconds = null`.

When the selected run already has an earlier accepted signal waiting to be applied, the later POST rejects as `rejected_pending_signal` instead of running that workflow task inline on the caller path. Invalid named or positional payloads reject as `rejected_invalid_arguments` with `validation_errors`. Historical run-targeted POST routes reject as `rejected_not_current`.

Current HTTP response matrix for the update POST routes:

- `200` with `outcome = update_completed` when the update command is accepted and completed
- `202` with `update_status = accepted` when `wait_for = accepted` durably accepts the update command and leaves application to the workflow worker
- `202` with `update_status = accepted`, `wait_for = completed`, and `wait_timed_out = true` when the webhook waited up to `wait_timeout_seconds` or the configured default and the worker still had not closed the update lifecycle
- `401` for webhook auth failure
- `404` for an unknown workflow instance id
- `404` with `outcome = rejected_unknown_update` for an unknown durable update name; when the run already carries a typed `WorkflowStarted` contract snapshot, or when an older `WorkflowStarted` event can be backfilled on first compatible intake, that rejection can come from durable run metadata before any live-definition fallback
- `409` with `outcome = rejected_not_started` when the instance has no current run yet
- `409` with `outcome = rejected_not_active` when the current run is already closed
- `409` with `outcome = rejected_not_current` when the run-targeted route addresses a historical selected run
- `409` with `outcome = rejected_pending_signal` when an earlier accepted signal still has to be applied before the update can run
- `422` with `outcome = rejected_invalid_arguments` when the update payload is an array but does not satisfy the declared update contract, including missing arguments, unknown arguments, type mismatches, or nullability violations
- `422` with `outcome = update_failed` when the update body throws
- `422` when `arguments` is present but not an array; that request-shape failure is rejected before a durable command row is created

Current HTTP response matrix for the update GET lookup routes:

- `200` once the lifecycle is closed and `update_status` is `completed`, `failed`, or `rejected`
- `202` while `update_status = accepted`
- `401` for webhook auth failure
- `404` for an unknown workflow instance id, selected run id, or `update_id` within the requested scope

### Repair And Terminal Command Webhooks

Both instance-targeted and run-targeted repair and terminal command routes are available:

```text
POST /webhooks/instances/{workflowId}/repair
POST /webhooks/instances/{workflowId}/cancel
POST /webhooks/instances/{workflowId}/terminate
POST /webhooks/instances/{workflowId}/runs/{runId}/repair
POST /webhooks/instances/{workflowId}/runs/{runId}/cancel
POST /webhooks/instances/{workflowId}/runs/{runId}/terminate
```

`workflowId` here is the public workflow instance id. The instance routes always resolve the current active run. The run routes also take one selected `runId` and reject historical selections with the durable `rejected_not_current` outcome. The same `workflows.webhook_auth` configuration used by the start route also applies to these terminal command routes.

Repair example:

```bash
curl -X POST "https://example.com/webhooks/instances/order-123/repair" \
     -H "Content-Type: application/json"
```

Cancel example:

```bash
curl -X POST "https://example.com/webhooks/instances/order-123/cancel" \
     -H "Content-Type: application/json"
```

Terminate example:

```bash
curl -X POST "https://example.com/webhooks/instances/order-123/terminate" \
     -H "Content-Type: application/json"
```

Accepted repair response when the runtime restores durable progress for the current run:

```json
{
  "outcome": "repair_dispatched",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "rejection_reason": null
}
```

Accepted repair response when the current run already has a healthy durable resume path:

```json
{
  "outcome": "repair_not_needed",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "rejection_reason": null
}
```

Accepted cancel response:

```json
{
  "outcome": "cancelled",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "rejection_reason": null
}
```

Accepted terminate response:

```json
{
  "outcome": "terminated",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "accepted",
  "rejection_reason": null
}
```

Rejected response when the instance exists but has not started a run yet:

```json
{
  "outcome": "rejected_not_started",
  "workflow_id": "order-123",
  "run_id": null,
  "command_id": "01J...",
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "rejected",
  "rejection_reason": "instance_not_started"
}
```

Rejected response when the current run is already closed:

```json
{
  "outcome": "rejected_not_active",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "command_id": "01J...",
  "target_scope": "instance",
  "workflow_type": "order-workflow",
  "command_status": "rejected",
  "rejection_reason": "run_not_active"
}
```

Rejected response when a run-targeted repair, cancel, or terminate route addresses a historical run:

```json
{
  "outcome": "rejected_not_current",
  "workflow_id": "order-123",
  "run_id": "01J...",
  "requested_run_id": "01J...",
  "resolved_run_id": "01J...",
  "command_id": "01J...",
  "target_scope": "run",
  "workflow_type": "order-workflow",
  "command_status": "rejected",
  "rejection_reason": "selected_run_not_current"
}
```

Current HTTP response matrix for the repair, cancel, and terminate webhooks:

- `200` with `outcome = repair_dispatched` when repair re-dispatches an overdue ready task, reclaims an expired lease, or recreates a missing durable task that has not started running yet, including accepted-update and accepted-signal workflow application tasks whose rows were lost before the command could be applied
- `200` with `outcome = repair_not_needed` when the run already has a healthy durable resume path or when the selected run is already inside an in-flight activity with no task row
- `200` with `outcome = cancelled` or `outcome = terminated` when the current run is closed successfully
- `401` for webhook auth failure
- `404` for an unknown workflow instance id
- `409` with `outcome = rejected_not_started` when the instance has no current run yet
- `409` with `outcome = rejected_not_active` when the current run is already closed
- `409` with `outcome = rejected_not_current` when the run-targeted route addresses a historical selected run

### Describe Webhooks

Both instance-targeted and run-targeted describe routes are available for inspecting workflow state over HTTP:

```text
GET /webhooks/instances/{workflowId}/describe
GET /webhooks/instances/{workflowId}/runs/{runId}/describe
```

`workflowId` is the public workflow instance id. The instance-targeted route returns the current run's state. The run-targeted route describes a specific run by id, including historical or already-closed runs.

Describe is a read-only inspection operation. It does not create a durable command row and does not replay the workflow. It returns instance metadata, current or selected run state, summary fields, and action availability from committed projection data.

Describe example:

```bash
curl "https://example.com/webhooks/instances/order-123/describe"
```

Response for an active workflow:

```json
{
  "found": true,
  "workflow_instance_id": "order-123",
  "workflow_type": "order-workflow",
  "workflow_class": "App\\Workflows\\OrderWorkflow",
  "business_key": "order-123",
  "run": {
    "workflow_run_id": "01J...",
    "run_number": 1,
    "is_current_run": true,
    "status": "waiting",
    "status_bucket": "running",
    "closed_reason": null,
    "compatibility": "build-a",
    "connection": "redis",
    "queue": "default",
    "started_at": "2026-04-12T12:00:00+00:00",
    "closed_at": null,
    "last_progress_at": "2026-04-12T12:00:01+00:00",
    "wait_kind": "signal",
    "wait_reason": "Waiting for signal [approved-by]"
  },
  "run_count": 1,
  "actions": {
    "can_signal": true,
    "can_query": true,
    "can_update": true,
    "can_cancel": true,
    "can_terminate": true
  },
  "reason": null
}
```

Response for a terminated workflow:

```json
{
  "found": true,
  "workflow_instance_id": "order-123",
  "workflow_type": "order-workflow",
  "workflow_class": "App\\Workflows\\OrderWorkflow",
  "business_key": "order-123",
  "run": {
    "workflow_run_id": "01J...",
    "run_number": 1,
    "is_current_run": true,
    "status": "terminated",
    "status_bucket": "failed",
    "closed_reason": "terminated",
    "compatibility": "build-a",
    "connection": "redis",
    "queue": "default",
    "started_at": "2026-04-12T12:00:00+00:00",
    "closed_at": "2026-04-12T12:01:00+00:00",
    "last_progress_at": "2026-04-12T12:01:00+00:00",
    "wait_kind": null,
    "wait_reason": null
  },
  "run_count": 1,
  "actions": {
    "can_signal": false,
    "can_query": false,
    "can_update": false,
    "can_cancel": false,
    "can_terminate": false
  },
  "reason": null
}
```

Action availability reflects whether the operation can succeed right now: closed runs cannot accept commands, remote-only workflows cannot serve queries or updates locally, and non-current runs cannot receive signals, updates, cancellations, or terminations.

Run-targeted describe example:

```bash
curl "https://example.com/webhooks/instances/order-123/runs/01J.../describe"
```

The run-targeted response includes the same payload but `is_current_run` will be `false` when the selected run is historical, and all mutating actions will show `false`.

Response when the instance is not found:

```json
{
  "found": false,
  "workflow_instance_id": "order-123",
  "workflow_type": null,
  "workflow_class": null,
  "business_key": null,
  "run": null,
  "run_count": 0,
  "actions": {
    "can_signal": false,
    "can_query": false,
    "can_update": false,
    "can_cancel": false,
    "can_terminate": false
  },
  "reason": "instance_not_found"
}
```

Response when the instance exists but the selected run is not found:

```json
{
  "found": true,
  "workflow_instance_id": "order-123",
  "workflow_type": "order-workflow",
  "workflow_class": "App\\Workflows\\OrderWorkflow",
  "business_key": "order-123",
  "run": null,
  "run_count": 1,
  "actions": {
    "can_signal": false,
    "can_query": false,
    "can_update": false,
    "can_cancel": false,
    "can_terminate": false
  },
  "reason": "run_not_found"
}
```

Current HTTP response matrix for the describe webhooks:

- `200` when the instance is found, whether the run is active, closed, or historical
- `200` when the instance is found but the selected run does not exist (`reason = run_not_found`)
- `401` for webhook auth failure
- `404` when the workflow instance id does not exist (`reason = instance_not_found`)

### Control-Plane Start Webhook

A control-plane start route accepts a durable workflow type key directly, without requiring the workflow class to be locally resolvable. This is the preferred start path for an external consumer that drives workflows through type keys rather than PHP class aliases.

```text
POST /webhooks/control-plane/start
```

The request body accepts:

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| `workflow_type` | string | yes | Durable workflow type key |
| `instance_id` | string | no | Caller-supplied public workflow instance id |
| `arguments` | string | no | Codec-tagged serialized arguments |
| `connection` | string | no | Queue connection override |
| `queue` | string | no | Queue name override |
| `business_key` | string | no | Caller-supplied business key |
| `labels` | object | no | Visibility labels |
| `memo` | object | no | Non-indexed metadata |
| `duplicate_start_policy` | string | no | `reject_duplicate` or `return_existing_active` |

Example request:

```bash
curl -X POST "https://example.com/webhooks/control-plane/start" \
     -H "Content-Type: application/json" \
     -d '{
           "workflow_type": "order-workflow",
           "instance_id": "order-123",
           "arguments": "{\"orderId\":123}",
           "connection": "redis",
           "queue": "default",
           "business_key": "order-123",
           "labels": {"tenant": "acme"}
         }'
```

Response for a newly started workflow:

```json
{
  "started": true,
  "workflow_instance_id": "order-123",
  "workflow_run_id": "01J...",
  "workflow_type": "order-workflow",
  "outcome": "started_new",
  "task_id": "01J...",
  "reason": null
}
```

When `instance_id` is omitted, the engine generates a ULID-based instance id automatically.

When the workflow type key maps to a locally resolvable class through `workflows.v2.types.workflows` config or a `#[Type(...)]` attribute, the full command-contract snapshot and routing are applied at start time. When the class is not locally available, the instance is created with the type key and explicit routing from options; the command-contract snapshot and definition fingerprint are deferred to the worker that claims the first task. This means an external consumer can start workflows for type keys that only the worker fleet can resolve.

Current HTTP response matrix for the control-plane start webhook:

- `202` with `outcome = started_new` when a new run is created
- `200` with `outcome = returned_existing_active` when the existing active run is reused
- `409` with `outcome = rejected_duplicate` when the duplicate start is rejected
- `401` for webhook auth failure
- `422` when `workflow_type` is missing or `instance_id` is invalid

The `WorkflowControlPlane` contract is also available as a container-resolvable singleton for programmatic use. Resolve `Workflow\V2\Contracts\WorkflowControlPlane` from the Laravel container to call `start()`, `signal()`, `query()`, `update()`, `cancel()`, `terminate()`, and `describe()` directly from PHP code, tests, or Artisan commands.

The webhook system covers explicit alias-based start intake plus both instance-targeted and run-targeted signal, update, query, describe, repair, cancel, and terminate command webhooks, task poll routes for workflow and activity tasks, and a control-plane start route for type-key-based workflow creation.

<!-- Source: docs/features/continue-as-new.md -->

# Continue As New

The **Continue As New** pattern allows a running workflow to restart itself with new arguments.
This is useful when you need to:

* Prevent unbounded workflow history growth.
* Model iterative loops or recursive workflows.
* Split long-running workflows into smaller, manageable executions while preserving continuity.

## Using `continueAsNew`

To restart a workflow as new, call the helper function `continueAsNew(...)` from within the workflow’s `handle()` method. `continueAsNew()` is a first-class run transition.

```php
use function Workflow\V2\activity;
use function Workflow\V2\continueAsNew;
use Workflow\V2\Workflow;

class CounterWorkflow extends Workflow
{
    public function handle(int $count = 0, int $max = 3)
    {
        $result = activity(CountActivity::class, $count);

        if ($count >= $max) {
            return [
                ‘count’ => $result,
                ‘workflow_id’ => $this->workflowId(),
                ‘run_id’ => $this->runId(),
            ];
        }

        return continueAsNew($count + 1, $max);
    }
}
```

In this example:

* The workflow executes an activity each iteration.
* If the maximum count has not been reached, it continues as new with incremented arguments.
* The final result is returned only when the loop completes.

When a workflow continues as new:

- The instance id stays stable across the chain; `WorkflowStub::runId()` moves forward to the newest run.
- The continued run closes as `completed` / `continued` and the next run starts immediately with a fresh run id.
- Signals, queries, updates, and operator commands always resolve the newest durable run for that instance — callers do not need to track run ids across continue-as-new boundaries.

Long-lived loops can shed history without inventing a new public workflow id every time the run rolls forward.

## History Budget

Workflow code can observe its current history length and a continue-as-new suggestion flag through explicit runtime context. This lets long-lived loops hand off deliberately before they hit history-budget cliffs.

```php
use function Workflow\V2\activity;
use function Workflow\V2\continueAsNew;
use Workflow\V2\Workflow;

class PollingWorkflow extends Workflow
{
    public function handle(int $iteration = 0)
    {
        $result = activity(PollActivity::class);

        if ($this->shouldContinueAsNew()) {
            return continueAsNew($iteration + 1);
        }

        return ['result' => $result, 'iteration' => $iteration];
    }
}
```

### Available methods

| Method | Return type | Description |
| --- | --- | --- |
| `$this->historyLength()` | `int` | Number of history events in the current run |
| `$this->historySize()` | `int` | Total size of history events in bytes |
| `$this->shouldContinueAsNew()` | `bool` | `true` when the run exceeds the configured event count or byte size budget |

### Configuration

The continue-as-new recommendation thresholds are configurable:

```php
// config/workflows.php
'v2' => [
    'history_budget' => [
        'continue_as_new_event_threshold' => 10000,
        'continue_as_new_size_bytes_threshold' => 5242880, // 5 MB
    ],
],
```

`shouldContinueAsNew()` returns `true` when either threshold is crossed. Run summaries also track `history_event_count`, `history_size_bytes`, and `continue_as_new_recommended` so Waterline can flag long-lived runs in fleet views.

## Metadata Carry-Forward

When a workflow continues as new, the new run inherits the previous run's metadata automatically:

| Metadata | Carry-forward behavior |
| --- | --- |
| Search attributes | Full merged map carries forward, including mid-run upserts |
| Memo | Full merged map carries forward, including mid-run upserts |
| Visibility labels | Carried forward as-is (set once at start time) |
| Business key | Carried forward as-is |
| Compatibility marker | Carried forward so the new run targets the same worker build |
| Message cursor position | Carried forward so the new run knows which inbound messages were already consumed through [Message Streams](./message-streams.md) |

The new run starts with the inherited metadata and can continue upserting search attributes and memo from there. This means a long-lived polling workflow can accumulate metadata across generations without losing state when it sheds history.

```php
use function Workflow\V2\{activity, continueAsNew, upsertMemo, upsertSearchAttributes};
use Workflow\V2\Workflow;

class PollingWorkflow extends Workflow
{
    public function handle(int $iteration = 0)
    {
        $result = activity(PollActivity::class);

        upsertSearchAttributes(['iteration' => (string) $iteration, 'status' => 'polling']);
        upsertMemo(['last_result' => $result, 'iteration' => $iteration]);

        if ($this->shouldContinueAsNew()) {
            // The new run inherits all search attributes and memo
            return continueAsNew($iteration + 1);
        }

        return ['result' => $result, 'iteration' => $iteration];
    }
}
```

For long-lived human-input or assistant loops, prefer the first-class
`$this->inbox()` / `$this->outbox()` [Message Streams](./message-streams.md)
facade. Pending stream messages and cursor position move to the continued run,
while consumed messages remain on the original run as history evidence.

<!-- Source: docs/features/versioning.md -->

# Versioning

Since workflows can run for long periods, sometimes months or even years, it's common to need to make changes to a workflow definition while executions are still in progress. Without versioning, modifying workflow code that affects the execution path would cause non-determinism errors during replay.

The `getVersion()` helper function allows you to safely introduce changes to running workflows by creating versioned branch points.

## Using `getVersion`

`getVersion()` is a replay-safe straight-line helper. Each change point records a durable version marker for the run on first execution, and every later replay reuses that committed value instead of recalculating the branch from today's code. Old runs that predate a newly introduced branch point conservatively receive `DEFAULT_VERSION`.

```php
use Workflow\V2\Workflow;
use Workflow\V2\WorkflowStub;
use function Workflow\V2\{activity, getVersion};

final class MyWorkflow extends Workflow
{
    public function handle(): void
    {
        $version = getVersion(
            'my-change-id',
            WorkflowStub::DEFAULT_VERSION,
            1
        );

        if ($version === WorkflowStub::DEFAULT_VERSION) {
            activity(OldActivity::class);
        } else {
            activity(NewActivity::class);
        }
    }
}
```

## How It Works

The `getVersion()` method takes three parameters:

- **changeId** - A unique identifier for this change point
- **minSupported** - The minimum version this code still supports
- **maxSupported** - The maximum (current) version for new executions

When a workflow encounters `getVersion()`:

- **New executions** append a typed `VersionMarkerRecorded` history event with the `change_id`, chosen `version`, and supported range, then return `maxSupported`
- **Replaying executions** return the previously recorded version from that history marker
- **Runs without a marker yet** fall back to `WorkflowStub::DEFAULT_VERSION` when the runtime determines the branch predates the current workflow definition. It first checks the run's compatibility marker against the current worker, then compares the run's durably snapped `workflow_definition_fingerprint` to the current loadable workflow class. If the fingerprints match, the run is on the same definition and the marker is recorded fresh; if they differ, the run predates the change and the fallback fires. Runs whose `WorkflowStarted` history predates the fingerprint snapshot (no recorded fingerprint) conservatively receive `DEFAULT_VERSION` because the runtime cannot prove the definition has not changed. This conservative fallback is the frozen 2.0 policy for pre-fingerprint runs: the runtime does not block claim solely because an older run lacks a recorded fingerprint. The fallback does not append a synthetic marker or consume a new workflow step
- **Query methods** replay the same committed version marker before invoking the annotated method, so queries see the same branch the workflow task saw
- **Waterline** exposes recorded markers in the selected-run timeline with `version_change_id`, `version`, `version_min_supported`, and `version_max_supported`, and selected-run detail now also surfaces `workflow_definition_fingerprint`, `workflow_definition_current_fingerprint`, and `workflow_definition_matches_current` so operators can see when a long-lived run started on an older definition even before a version marker was committed

This allows new workflows to use the latest code path while existing workflows continue using their original path.

## Adding a New Version

Suppose you have an existing workflow that calls `prePatchActivity`:

```php
use Workflow\V2\Workflow;
use function Workflow\V2\activity;

final class MyWorkflow extends Workflow
{
    public function handle()
    {
        $result = activity(PrePatchActivity::class);

        return $result;
    }
}
```

To replace it with `postPatchActivity` without breaking running workflows:

```php
use Workflow\V2\Workflow;
use Workflow\V2\WorkflowStub;
use function Workflow\V2\{activity, getVersion};

final class MyWorkflow extends Workflow
{
    public function handle()
    {
        $version = getVersion(
            'activity-change',
            WorkflowStub::DEFAULT_VERSION,
            1
        );

        $result = $version === WorkflowStub::DEFAULT_VERSION
            ? activity(PrePatchActivity::class)
            : activity(PostPatchActivity::class);

        return $result;
    }
}
```

When you roll out a deployment that introduces a new `getVersion()` branch point, keep rotating `DW_V2_CURRENT_COMPATIBILITY` for that build wave. The runtime uses the run's start-time `workflow_definition_fingerprint` as its primary authority when deciding whether a missing version marker belongs to a fresh execution or to an older run that should stay on `DEFAULT_VERSION`. The compatibility marker still matters for compatibility-aware worker routing and fires before the fingerprint check. Runs that predate the fingerprint snapshot (no recorded fingerprint) conservatively stay on `DEFAULT_VERSION` since the runtime cannot verify the definition has not changed.

## Adding More Versions

When you need to make additional changes, increment `maxSupported`:

```php
$version = getVersion(
    'activity-change',
    WorkflowStub::DEFAULT_VERSION,
    2
);

$result = match($version) {
    WorkflowStub::DEFAULT_VERSION => activity(PrePatchActivity::class),
    1 => activity(PostPatchActivity::class),
    2 => activity(AnotherPatchActivity::class),
};
```

## Deprecating Old Versions

After all workflows using an old version have completed, you can drop support by increasing `minSupported`. This removes the need to maintain old code paths.

```php
// After all DEFAULT_VERSION workflows have completed:
$version = getVersion(
    'activity-change',
    1,  // No longer supporting DEFAULT_VERSION
    2
);

$result = match($version) {
    1 => activity(PostPatchActivity::class),
    2 => activity(AnotherPatchActivity::class),
};
```

If a workflow with a version older than `minSupported` tries to replay, it will throw a `VersionNotSupportedException`. That includes older-compatibility runs whose safe fallback is still `WorkflowStub::DEFAULT_VERSION`.

If you accidentally reuse the same `changeId` for a different branch point later, the runtime treats that as a determinism error instead of silently accepting the mismatch. Keep one stable `changeId` per logical code change.

## Multiple Change Points

You can use multiple `getVersion()` calls in the same workflow for independent changes:

```php
use Workflow\V2\Workflow;
use Workflow\V2\WorkflowStub;
use function Workflow\V2\getVersion;

final class MyWorkflow extends Workflow
{
    public function handle(): void
    {
        $version1 = getVersion('change-1', WorkflowStub::DEFAULT_VERSION, 1);
        $version2 = getVersion('change-2', WorkflowStub::DEFAULT_VERSION, 1);

        // Each change point is tracked independently
    }
}
```

**Important:** Each `changeId` should be unique within a workflow. The chosen version is recorded as typed workflow history and replayed deterministically on later workflow tasks, queries, and Waterline detail views. When Waterline shows no `VersionMarkerRecorded` entry for a change point on an older run, that means replay stayed on the legacy `DEFAULT_VERSION` path without backfilling a new marker into existing history. The selected-run detail fields `workflow_definition_fingerprint`, `workflow_definition_current_fingerprint`, and `workflow_definition_matches_current` tell you whether that run started on a different workflow definition than the one your current build can load today.

## `patched()` Shorthand

`patched($changeId)` is a two-state shorthand for the common "did this run cross a one-time code change?" question. It records the same durable `VersionMarkerRecorded` history event as `getVersion()` and resolves to a boolean instead of an integer.

```php
use Workflow\V2\Workflow;
use function Workflow\V2\{activity, patched};

final class MyWorkflow extends Workflow
{
    public function handle(): string
    {
        if (patched('use-new-payment-activity')) {
            return activity(NewPaymentActivity::class);
        }

        return activity(LegacyPaymentActivity::class);
    }
}
```

Behavior:

- New runs commit the marker, take the new branch, and `patched()` returns `true`.
- Runs that started before this `changeId` was added stay on `DEFAULT_VERSION` under the same fingerprint fallback `getVersion()` uses, and `patched()` returns `false`.
- Replays of either kind read the previously committed marker and return the same value, so the branch decision is durable.

`patched()` is exactly equivalent to `getVersion($changeId, DEFAULT_VERSION, 1) === 1`. Use it when you only have two branches and you do not need to keep the legacy branch around forever — the boolean spelling reads cleaner at the call site than a `match` over `DEFAULT_VERSION` and `1`.

If you need more than two branches, or you plan to add another version of the same logical change later, use `getVersion()` with an explicit `maxSupported`. Switching from `patched()` to `getVersion()` for the same `changeId` is a determinism error because the existing marker was recorded with `maxSupported = 1`.

## Removing the Legacy Branch with `deprecatePatch()`

When every legacy run for a `patched()` change point has finished, you can delete the legacy branch from your workflow code. The compatible way to do that is to replace the `patched()` call with `deprecatePatch()` rather than removing the call entirely.

```php
use Workflow\V2\Workflow;
use function Workflow\V2\{activity, deprecatePatch};

final class MyWorkflow extends Workflow
{
    public function handle(): string
    {
        deprecatePatch('use-new-payment-activity');

        return activity(NewPaymentActivity::class);
    }
}
```

`deprecatePatch()` keeps the change point on the workflow timeline so already-committed `VersionMarkerRecorded` history events still match a known call, but it returns `null` and unconditionally takes the new path. Once you ship this version:

- New runs commit a `deprecate_patch` marker, take the new branch, and the `deprecatePatch()` call returns `null`.
- Existing runs that already committed a `patched` marker for this `changeId` keep replaying that marker and resolve through the deprecated branch — which is now the only branch — without erroring.
- Existing runs that committed `false` for `patched()` (the legacy path) are no longer in flight by assumption. If one is still running and tries to replay, the workflow will still read the legacy marker but execute the new branch — which is why the safe lifecycle is "wait for legacy runs to drain before deploying `deprecatePatch()`".

Two-phase patch lifecycle and placement rules:

1. **Introduce `patched()`** at the change point. Both branches stay in the workflow code. New runs take the new branch; older runs keep taking the legacy branch.
2. **Wait for every legacy run to finish.** Use Waterline run search or `dw workflow:list` to confirm there are no open runs that could replay through the legacy branch.
3. **Replace `patched()` with `deprecatePatch()`** at the same call site, with the same `changeId`, and delete the legacy branch from the function body. Keep the call there — do not remove it — so durable history still matches a known change point.
4. **Optional: remove `deprecatePatch()` entirely** once you are also confident no historical replay (queries, exports, audits) will ever read history that committed this marker. That is an irreversible step; most workflows can leave `deprecatePatch()` in place indefinitely without paying any runtime cost.

Placement rules:

- `patched()` and `deprecatePatch()` must be called from the workflow function body, not from activities, signal handlers, or query methods. They are workflow steps and must replay deterministically.
- Each `changeId` is one logical change point. Reusing the same `changeId` for a different branch is a determinism error.
- A `changeId` can appear in `patched()` or `deprecatePatch()` form during its lifetime, but never both at the same time. The deploy that swaps the call must replace, not coexist.
- `getVersion()` and `patched()` cannot coexist for the same `changeId`. Pick one model when you introduce the change point.

<!-- Source: docs/features/cancel-and-terminate.md -->

# Cancel and Terminate

This guide covers the embedded Laravel `Workflow\V2\WorkflowStub` API. For
service-mode clients and workers, use the
[PHP SDK guide](https://php.durable-workflow.com/),
[Python cancellation reference](https://python.durable-workflow.com/reference/errors/#durable_workflow.errors.ActivityCancelled), or
[Rust lifecycle API](https://rust.durable-workflow.com/durable_workflow/struct.Client.html#method.cancel_workflow).

Cancel and terminate are first-class durable commands that close a running workflow. Both are recorded in command history, appear in typed history events, and surface in Waterline.

In the current embedded API, both commands close the run immediately. **Cancel**
records a `cancelled` outcome; **terminate** records a `terminated` outcome.
Neither command schedules cleanup inside the closed workflow. If your
application needs durable compensation first, signal the workflow to run that
cleanup before closing it.

## Cancel

Cancel immediately transitions the run to `cancelled` and records durable history.

```php
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::load('order-123');

$result = $workflow->cancel();

$result->accepted();   // true
$result->outcome();    // "cancelled"
$result->commandId();  // Durable command id
$result->reason();     // null (no reason provided)
```

### Cancel with reason

You can provide a structured reason to distinguish user-driven, policy-driven, and operator-driven cancellation in audit trails, command history, and Waterline.

```php
$result = $workflow->cancel('Customer requested cancellation');

$result->reason(); // "Customer requested cancellation"
```

The reason is persisted on the durable command record and in both the `CancelRequested` and `WorkflowCancelled` typed history events, so it survives replay, export, and offline analysis.

### What cancel does

When a cancel command is accepted, the engine closes any open activity executions and pending timers, transitions the run to `cancelled`, and resumes a parent workflow waiting on the cancelled child so it can observe the outcome.

### Cancel is rejected when

- The instance has not started yet: `rejection_reason = instance_not_started`
- The current run is already closed: `rejection_reason = run_not_active`
- A run-targeted cancel addresses a historical (non-current) run: `rejection_reason = selected_run_not_current`

## Terminate

Terminate immediately closes a running workflow without scheduling further workflow code. Use terminate when graceful cleanup is not needed or when a workflow is stuck.

```php
$result = $workflow->terminate();

$result->accepted();   // true
$result->outcome();    // "terminated"
```

### Terminate with reason

```php
$result = $workflow->terminate('Operator emergency shutdown');

$result->reason(); // "Operator emergency shutdown"
```

### What terminate does

Terminate follows the same transactional steps as cancel, but:

- Records `TerminateRequested` and `WorkflowTerminated` history events instead.
- Creates the `WorkflowFailure` row with `failure_category = terminated` and `propagation_kind = terminated`.
- Sets `closed_reason = terminated` on the run.
- Does not schedule any further workflow-code execution.

### Terminate is rejected when

Same rejection conditions as cancel.

## Run-targeted commands

Both cancel and terminate can target a specific run instead of the instance's current run.

```php
$selectedRun = WorkflowStub::loadRun($runId);
$result = $selectedRun->attemptCancel('Draining old run');

$result->targetScope(); // "run"
```

Run-targeted commands reject with `selected_run_not_current` when the addressed run is no longer the instance's current run. The response includes `requested_run_id` (the run the caller addressed) and `resolved_run_id` (the current run that should be used next).

## Non-throwing API

Use `attemptCancel()` and `attemptTerminate()` when you want to handle rejection without exceptions:

```php
$result = $workflow->attemptCancel('Duplicate order');

if ($result->rejected()) {
    $result->rejectionReason(); // e.g. "run_not_active"
}
```

`cancel()` and `terminate()` throw `LogicException` on rejection.

## Webhooks

Cancel and terminate are available through webhook routes:

```text
POST /webhooks/instances/{workflowId}/cancel
POST /webhooks/instances/{workflowId}/terminate
POST /webhooks/instances/{workflowId}/runs/{runId}/cancel
POST /webhooks/instances/{workflowId}/runs/{runId}/terminate
```

Pass the reason in the request body:

```json
{
  "reason": "Operator: duplicate order"
}
```

The response includes the command outcome, the public instance id, and the reason:

```json
{
  "outcome": "cancelled",
  "workflow_id": "order-123",
  "run_id": "01J10000000000000000000021",
  "reason": "Operator: duplicate order",
  "command_id": "01J40000000000000000000021",
  "command_status": "accepted"
}
```

## Cancellation is not an error you catch by accident

Cancellation is an explicit lifecycle outcome, not an unexpected application
error. The embedded package's
`Workflow\V2\Exceptions\WorkflowCancelledException` extends `\Error`, not
`\Exception`. A `catch (\Exception $e)` block will not catch it; a
`catch (\Throwable $t)` block will.

When reading a cancelled workflow's result, catch the exception by name if you
need to distinguish cancellation from other failures. Catching a result-side
exception does not reopen the cancelled run or schedule cleanup inside it.

## Waterline

Waterline exposes cancel and terminate as operator actions on the selected-run detail view. The detail payload includes `can_cancel` and `can_terminate` flags driven from durable state.

The command history view shows each cancel or terminate command with its reason, caller identity, and outcome. The `commands[*].reason` field carries the operator- or caller-provided reason when one was supplied.

## Status mapping

Both `cancelled` and `terminated` runs land in the `failed` status bucket for list routing. The raw `status` and `closed_reason` fields distinguish them:

| Status | Status bucket | Closed reason |
| --- | --- | --- |
| `cancelled` | `failed` | `cancelled` |
| `terminated` | `failed` | `terminated` |

## Typed history

A cancelled run produces this history sequence:

```
StartAccepted
WorkflowStarted
... (workflow progress) ...
CancelRequested       <- reason field present when supplied
TimerCancelled        <- for each open timer
ActivityCancelled     <- for each open activity
WorkflowCancelled     <- failure_id, failure_category, reason when supplied
```

A terminated run produces:

```
StartAccepted
WorkflowStarted
... (workflow progress) ...
TerminateRequested    <- reason field present when supplied
TimerCancelled        <- for each open timer
ActivityCancelled     <- for each open activity
WorkflowTerminated    <- failure_id, failure_category, reason when supplied
```

## Cancel vs. terminate

| | Cancel | Terminate |
| --- | --- | --- |
| Further workflow code is scheduled | No | No |
| Open activities cancelled | Yes | Yes |
| Open timers cancelled | Yes | Yes |
| Reason metadata | Yes | Yes |
| Durable command history | Yes | Yes |
| Parent receives child outcome | Yes | Yes |
| Failure row recorded | Yes (`cancelled`) | Yes (`terminated`) |
| Status bucket | `failed` | `failed` |
| Closed reason | `cancelled` | `terminated` |

<!-- Source: docs/features/search-attributes.md -->

# Search Attributes

Search attributes are typed, indexed key-value pairs that a workflow can upsert at any point during execution. Unlike visibility labels (which are set once at start time), search attributes can be updated as the workflow progresses, making them ideal for tracking workflow status, customer identifiers, or any operator-visible metadata that changes over the lifetime of a run.

## Upserting Search Attributes

`upsertSearchAttributes()` is a durable straight-line helper. Each call records a typed `SearchAttributesUpserted` history event and merges the new attributes into the run's persisted search attributes.

```php
use Workflow\V2\Workflow;
use function Workflow\V2\{activity, upsertSearchAttributes};

final class OrderWorkflow extends Workflow
{
    public function handle(string $orderId, string $customer): array
    {
        upsertSearchAttributes([
            'status' => 'processing',
            'customer' => $customer,
            'order_id' => $orderId,
        ]);

        $result = activity(ProcessOrderActivity::class, $orderId);

        upsertSearchAttributes([
            'status' => 'completed',
            'result' => $result->outcome,
        ]);

        return $result->toArray();
    }
}
```

## Setting Search Attributes at Start Time

Search attributes can also be set when starting a workflow via `StartOptions::withSearchAttributes()`. Start-time attributes are recorded on the run immediately and appear in the `StartAccepted` and `WorkflowStarted` history events. Any subsequent `upsertSearchAttributes()` calls merge on top of them.

```php
use Workflow\V2\StartOptions;
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::make(OrderWorkflow::class, 'order-42');
$workflow->start(
    $orderId,
    $customer,
    StartOptions::rejectDuplicate()->withSearchAttributes([
        'env' => 'production',
        'region' => 'us-east',
        'priority' => 'high',
    ]),
);
```

You can chain `withSearchAttributes()` with other `StartOptions` builders:

```php
$options = StartOptions::rejectDuplicate()
    ->withBusinessKey('order-42')
    ->withLabels(['tenant' => 'acme'])
    ->withMemo(['note' => 'VIP order'])
    ->withSearchAttributes(['priority' => 'high']);

$workflow->start($orderId, $customer, $options);
```

Start-time search attributes follow the same validation rules as `upsertSearchAttributes()`: keys must be 1-64 URL-safe characters, values must be scalar or null, and null values are dropped (not persisted).

### Control plane

The control plane `start()` method also accepts search attributes:

```php
$controlPlane->start('orders.process-order', 'order-42', [
    'arguments' => $serializedArgs,
    'search_attributes' => ['env' => 'production', 'priority' => 'high'],
]);
```

## How It Works

The `upsertSearchAttributes()` function accepts an associative array of key-value pairs:

- **Keys** must be 1-64 characters, URL-safe (letters, digits, hyphens, underscores)
- **Values** must be scalar (string, int, float, bool) or null; string values are capped at 191 characters
- Passing `null` as a value removes that key from the search attributes

Each call:

1. Suspends the workflow fiber and yields an `UpsertSearchAttributesCall` command
2. The executor validates and normalizes the attributes (keys are sorted alphabetically)
3. A `SearchAttributesUpserted` history event is appended with the upserted `attributes` and the full `merged` result
4. The run's `search_attributes` column is updated with the merged map
5. The run summary projection is refreshed so Waterline reflects the change immediately

On replay, recorded search attribute events are reused without re-executing the upsert, preserving determinism.

## Merging Behavior

Search attributes merge across multiple upserts within the same run. Each upsert overlays new keys on top of existing ones:

```php
// First upsert
upsertSearchAttributes(['status' => 'processing', 'customer' => 'Taylor']);
// search_attributes = { customer: Taylor, status: processing }

// Second upsert
upsertSearchAttributes(['status' => 'completed', 'result' => 'success']);
// search_attributes = { customer: Taylor, result: success, status: completed }
```

To remove a key, set it to `null`:

```php
upsertSearchAttributes(['temporary_flag' => null]);
```

## Visibility and Filtering

Search attributes appear in:

- **Run detail view** — the merged search attributes map is displayed alongside labels and memo
- **Run summary projection** — search attributes are included in the denormalized summary for fast reads
- **Visibility filters** — operators can filter workflow runs by search attribute values in Waterline saved views
- **History timeline** — each `SearchAttributesUpserted` event appears as a typed entry in the run timeline
- **History export** — search attributes are included in JSON history exports

## Continue-as-New

When a workflow continues as new, the current search attributes are carried forward to the new run automatically. The new run starts with the full merged attribute map from the previous run and can continue upserting from there.

## Privacy and Limits

Search attributes are **plain-text operator metadata** — their values are visible to anyone with Waterline access and are stored unencrypted. Never store secrets, passwords, tokens, or personally identifiable information (PII) in search attributes.

Constraints:

- Key names: 1-64 characters, URL-safe (`[a-zA-Z0-9_-]`)
- Values: scalar types only (string, int, float, bool), with string values capped at 191 characters
- Setting a value to `null` removes that key

## Search Attributes vs. Memo vs. Visibility Labels

| | Search Attributes | Memo | Visibility Labels |
|---|---|---|---|
| **Set when** | Start time or any time during execution | Start time or any time during execution | Start time only |
| **Mutable** | Yes, via `upsertSearchAttributes()` | Yes, via `upsertMemo()` | No |
| **Value types** | Scalar only | Any JSON-serializable | Scalar only |
| **Indexed** | Yes | No | Yes |
| **Filterable** | Yes | No | Yes |
| **Use case** | Dynamic status, progress tracking | Rich metadata, notes, context | Static classification |
| **History events** | `SearchAttributesUpserted` per upsert | `MemoUpserted` per upsert | None (set on start) |

For rich, structured metadata that does not need to be filtered or sorted, use [memo](./memo.md) instead.

<!-- Source: docs/features/memo.md -->

# Memo

Memos are non-indexed key-value metadata that a workflow can read and update at any point during execution. Unlike [search attributes](./search-attributes.md) (which are indexed and filterable), memos are designed for richer, structured metadata that appears in detail views and history exports but is excluded from fleet-wide filtering and sorting by contract.

## Upserting Memos

`upsertMemo()` is a durable straight-line helper. Each call records a typed `MemoUpserted` history event and merges the new entries into the run's persisted memo.

```php
use Workflow\V2\Workflow;
use function Workflow\V2\{activity, upsertMemo};

final class OrderWorkflow extends Workflow
{
    public function handle(string $orderId, string $customer): array
    {
        upsertMemo([
            'customer_name' => $customer,
            'order_id' => $orderId,
            'status' => 'processing',
            'line_items' => [
                ['sku' => 'WIDGET-1', 'qty' => 2],
                ['sku' => 'GADGET-3', 'qty' => 1],
            ],
        ]);

        $result = activity(ProcessOrderActivity::class, $orderId);

        upsertMemo([
            'status' => 'completed',
            'result_summary' => $result->outcome,
        ]);

        return $result->toArray();
    }
}
```

Memos can also be set at start time via `StartOptions`:

```php
use Workflow\V2\StartOptions;
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::make(OrderWorkflow::class, 'order-123');
$workflow->start(
    'ORD-456',
    'Taylor',
    StartOptions::rejectDuplicate()->withMemo([
        'source' => 'api',
        'priority' => 'high',
    ]),
);
```

## How It Works

The `upsertMemo()` function accepts an associative array of key-value pairs:

- **Keys** must be non-empty strings up to 64 characters
- **Values** can be any JSON-serializable data: scalars, null, arrays, or nested objects
- Passing `null` as a value removes that key from the memo

Each call:

1. Suspends the workflow fiber and yields an `UpsertMemoCall` command
2. The executor validates and normalizes the entries (keys are sorted alphabetically)
3. A `MemoUpserted` history event is appended with the upserted `entries` and the full `merged` result
4. The run's `memo` column is updated with the merged map

On replay, recorded memo events are reused without re-executing the upsert, preserving determinism.

## Merging Behavior

Memos merge across multiple upserts within the same run. Each upsert overlays new keys on top of existing ones:

```php
// First upsert
upsertMemo(['status' => 'processing', 'customer' => 'Taylor']);
// memo = { customer: Taylor, status: processing }

// Second upsert
upsertMemo(['status' => 'completed', 'result' => 'success']);
// memo = { customer: Taylor, result: success, status: completed }
```

To remove a key, set it to `null`:

```php
upsertMemo(['temporary_note' => null]);
```

## Visibility

Memos appear in:

- **Run detail view** — the full merged memo is displayed in Waterline's workflow detail page
- **History timeline** — each `MemoUpserted` event appears as a typed entry in the run timeline
- **History export** — memos are included in JSON history exports
- **Describe** — the control plane `describe` response includes the current memo

Memos are **not** included in run summary projections and are **not** available for filtering, sorting, or saved views. Use [search attributes](./search-attributes.md) for indexed, filterable metadata.

## Continue-as-New

When a workflow continues as new, the current memo is carried forward to the new run automatically. The new run starts with the full merged memo from the previous run and can continue upserting from there.

## Nested Structures

Unlike search attributes (which are limited to scalar values), memos support nested JSON structures:

```php
upsertMemo([
    'order' => [
        'id' => 'ORD-456',
        'items' => [
            ['sku' => 'WIDGET-1', 'qty' => 2],
            ['sku' => 'GADGET-3', 'qty' => 1],
        ],
    ],
    'metadata' => [
        'source' => 'api',
        'version' => 2,
    ],
]);
```

Nested objects have the same key constraints (non-empty strings up to 64 characters per key at each level).

## Constraints

- Key names: non-empty strings up to 64 characters per nesting level
- Values: any JSON-serializable type (scalars, null, arrays, nested objects)
- Setting a value to `null` removes that key
- Memos are eventually consistent metadata, not replay authority — do not branch workflow logic based on memo values

## Memo vs. Search Attributes vs. Visibility Labels

| | Memo | Search Attributes | Visibility Labels |
|---|---|---|---|
| **Set when** | Start time or any time during execution | Any time during execution | Start time only |
| **Mutable** | Yes, via `upsertMemo()` | Yes, via `upsertSearchAttributes()` | No |
| **Value types** | Any JSON-serializable | Scalar only | Scalar only |
| **Indexed** | No | Yes | Yes |
| **Filterable** | No | Yes | Yes |
| **Use case** | Rich metadata, notes, context | Dynamic status, progress tracking | Static classification |
| **History events** | `MemoUpserted` per upsert | `SearchAttributesUpserted` per upsert | None (set on start) |

<!-- Source: docs/features/message-streams.md -->

# Message Streams

Message streams are the v2 authoring surface for repeated workflow messages:
human input loops, assistant replies, workflow-to-workflow notifications, and
other ordered messages that must survive replay and continue-as-new.

This page describes the embedded Laravel inbox/outbox API. Server-owned
[Workflow Streams](/docs/polyglot/workflow-streams/) reuse names such as
stream, offset, pending count, lifecycle, and error where their semantics
match, but they are output-only. They do not add inbound workflow messaging or
continue-as-new cursor transfer to service mode.

Workflow authors should use the first-class facade:

- `$this->inbox('stream-key')`
- `$this->outbox('stream-key')`
- `$this->messages('stream-key')`
- `Workflow\V2\MessageStream`

Do not write `workflow_messages` rows or cursor rows directly from application
workflow code. The facade is the stable contract; lower-level message services
exist for package/runtime integration.

## Receiving Messages

Use `peek()` when a workflow needs to inspect pending inbound messages without
advancing its durable cursor. Use `receive()` or `receiveOne()` when the
workflow is ready to consume messages and record cursor advancement in history.

```php
use Workflow\V2\Workflow;

final class ApprovalInboxWorkflow extends Workflow
{
    public function handle(): array
    {
        $pending = $this->inbox('approval.requests')->peek(limit: 10);

        if ($pending->isEmpty()) {
            return ['status' => 'waiting'];
        }

        $message = $this->inbox('approval.requests')->receiveOne();

        return [
            'status' => 'received',
            'payload_reference' => $message?->payload_reference,
            'correlation_id' => $message?->correlation_id,
        ];
    }
}
```

`peek()` is read-only. `receive()` and `receiveOne()` mark messages consumed and
advance the durable message cursor for the current run. Cursor advancement is a
history fact, so replay, Waterline exports, and continue-as-new handoff can all
reason about which inbound messages were already consumed.

## Sending References

Use `outbox(...)->sendReference(...)` when a workflow needs to publish an
ordered outbound message. The message table stores routing metadata and a
payload reference; large or application-owned content stays in the app's
payload store.

```php
use Workflow\V2\Workflow;

final class AssistantReplyWorkflow extends Workflow
{
    public function handle(string $targetWorkflowId, string $replyReference): array
    {
        $message = $this->outbox('ai.assistant')->sendReference(
            targetInstanceId: $targetWorkflowId,
            payloadReference: $replyReference,
            correlationId: $this->workflowId(),
            metadata: ['kind' => 'assistant_reply'],
        );

        return [
            'status' => 'sent',
            'stream' => $message->stream_key,
            'sequence' => $message->sequence,
        ];
    }
}
```

This keeps the workflow history small and lets consumers validate the payload
store separately from durable message ordering.

## Continue-As-New

Message streams are instance-first. When a workflow continues as new, pending
inbound messages and the cursor position are transferred to the continued run.
Consumed messages remain attached to the original run as historical evidence.

That means a long-lived workflow can shed history and keep the same public
message route:

```php
use function Workflow\V2\continueAsNew;
use Workflow\V2\Workflow;

final class HumanInputLoopWorkflow extends Workflow
{
    public function handle(int $iteration = 0): array
    {
        $message = $this->inbox('human.replies')->receiveOne();

        if ($message === null) {
            return ['status' => 'waiting', 'iteration' => $iteration];
        }

        if ($this->shouldContinueAsNew()) {
            return continueAsNew($iteration + 1);
        }

        return [
            'status' => 'processed',
            'iteration' => $iteration,
            'payload_reference' => $message->payload_reference,
        ];
    }
}
```

The caller keeps addressing the workflow instance id. It does not need to know
which run currently owns the stream cursor.

## Authoring Rules

- Use semantic stream keys such as `ai.assistant`, `human.replies`, or
  `approval.requests`; treat them as part of the workflow's public contract.
- Use `peek()` for inspection and `receive()` / `receiveOne()` for durable
  consumption.
- Use `sendReference()` for outbound messages; store large payloads in an
  application payload store and pass a reference.
- Keep application code on `Workflow::inbox()`, `Workflow::outbox()`, and
  `MessageStream`; do not depend on `MessageService`, `WorkflowMessage`, or
  cursor-row writes as the authoring pattern.
- For one-shot external events, use [Signals](./signals.md). For request/return
  state changes, use [Updates](./updates.md). Use message streams when repeated
  ordered messages need cursor semantics.

## Run this pattern

The travel-agent workflow in the
[Sample App](/docs/sample-app) is the runnable reference for the
inbox/outbox shape this page describes:

```bash
php artisan app:ai
```

`App\Workflows\Ai\AiWorkflow` stores large assistant payloads in the
app-owned `ai_workflow_messages` table and publishes only a durable
reference on the `ai.assistant` outbox. Replies arrive through the
matching inbox stream via the `receive` update. Read the workflow class
next to this page to see the same `inbox()` / `outbox()` /
`sendReference()` calls the snippets above use, end to end.

<!-- Source: docs/features/timeouts.md -->

# Timeouts

Workflow-level timeouts let you bound how long a workflow is allowed to run. There are two timeout scopes:

- **Execution timeout** — caps the total wall-clock time across all runs of a workflow instance, including continue-as-new transitions. The deadline is computed once at start and carried forward unchanged.
- **Run timeout** — caps a single run. The deadline is recomputed each time a new run begins (including continue-as-new).

Both are optional and can be combined.

## Configuration

Timeouts are configured through `StartOptions` when starting a workflow:

```php
use Workflow\V2\StartOptions;
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::make(MyWorkflow::class, 'order-123');

$workflow->start(
    $orderId,
    StartOptions::rejectDuplicate()
        ->withExecutionTimeout(7200)   // 2 hours across all runs
        ->withRunTimeout(3600),        // 1 hour per run
);
```

### Execution timeout

The execution timeout spans the entire workflow instance. If the workflow uses continue-as-new, the deadline stays the same across every run in the chain.

```php
StartOptions::rejectDuplicate()->withExecutionTimeout(86400); // 24 hours
```

The timeout value is stored on the `WorkflowInstance` model and the computed deadline is snapped on every `WorkflowRun`.

### Run timeout

The run timeout applies to a single run. When a workflow continues as new, the new run gets a fresh deadline computed from the current time plus the configured run timeout.

```php
StartOptions::rejectDuplicate()->withRunTimeout(1800); // 30 minutes per run
```

### Validation

Timeout values must be at least 1 second. Passing zero or a negative value throws a `LogicException`:

```php
// Throws LogicException: "Workflow v2 execution timeout must be at least 1 second."
StartOptions::rejectDuplicate()->withExecutionTimeout(0);
```

### Control plane

Timeouts can also be set when starting a workflow through the control plane:

```php
$controlPlane->start('my-app.order-workflow', 'order-123', [
    'execution_timeout_seconds' => 7200,
    'run_timeout_seconds' => 3600,
]);
```

The `describe` response includes timeout and deadline fields:

```php
$description = $controlPlane->describe('order-123');

$description['execution_timeout_seconds']; // 7200
$description['run']['run_timeout_seconds']; // 3600
$description['run']['execution_deadline_at']; // ISO 8601 timestamp
$description['run']['run_deadline_at'];       // ISO 8601 timestamp
```

### Waterline

When timeouts are configured, the run detail view in Waterline displays the timeout durations and their computed deadlines.

### History

The `WorkflowStarted` history event payload includes timeout and deadline fields when configured:

```json
{
    "execution_timeout_seconds": 7200,
    "run_timeout_seconds": 3600,
    "execution_deadline_at": "2026-04-12T14:00:00+00:00",
    "run_deadline_at": "2026-04-12T13:00:00+00:00"
}
```

### Continue-as-new

When a workflow continues as new:

- The **execution deadline** is carried forward unchanged from the previous run.
- The **run timeout** value is carried forward, but the **run deadline** is recomputed from the current time.

This means the execution timeout always measures from the original start, while each new run gets its own fresh run-timeout window.

## Enforcement

Workflow-level timeouts are enforced at two points:

1. **At workflow task start** — every workflow task checks `deadlineExpired()` before executing the workflow. If the deadline has passed, the run is immediately timed out.
2. **By the TaskWatchdog** — the watchdog scans for non-terminal runs whose execution or run deadline has passed but that have no open workflow task. When found, it creates a deadline-expired workflow task and dispatches it, which triggers the timeout on the next task execution.

When a timeout fires, the engine:

- Cancels all open tasks (activity, timer, workflow) except the current one
- Cancels all open activity executions with `ActivityCancelled` history events
- Cancels all pending timers with `TimerCancelled` history events
- Records a `WorkflowFailure` with `failure_category = timeout` and a `WorkflowTimeoutException`
- Records a terminal `WorkflowTimedOut` history event with `timeout_kind` set to `execution_timeout` or `run_timeout`
- Applies parent-close policy to any open child workflows
- Notifies parent workflows if this was a child run

The failure row stores `Workflow\V2\Exceptions\WorkflowTimeoutException` as the exception class, carrying the `timeout_kind` and the deadline timestamp for programmatic inspection.

## Activity timeouts

Activity timeouts let you bound how long individual activity executions are allowed to take. There are four activity timeout scopes:

- **Schedule-to-start** — caps the time from scheduling to the first worker claim. Enforced while the activity is `Pending`.
- **Start-to-close** — caps the time from when a worker claims the activity to when it must complete. Resets on each retry attempt.
- **Schedule-to-close** — caps the total wall-clock time from scheduling to completion across all retry attempts. This is always terminal — retrying would not help because the overall deadline has passed.
- **Heartbeat** — caps the time between heartbeats. For long-running activities that call `$this->heartbeat()`, the engine requires a heartbeat within the configured interval or the activity is considered unresponsive.

All are optional and can be combined. Configure them through `ActivityOptions` when calling an activity:

```php
use function Workflow\V2\activity;
use Workflow\V2\Support\ActivityOptions;

$result = activity(
    LongRunningActivity::class,
    new ActivityOptions(
        scheduleToStartTimeout: 30,     // must be claimed within 30s
        startToCloseTimeout: 300,       // each attempt has 5 minutes
        scheduleToCloseTimeout: 600,    // total 10 minutes across all retries
        heartbeatTimeout: 15,           // must heartbeat every 15 seconds
        maxAttempts: 3,
    ),
    $input,
);
```

### Schedule-to-start timeout

The deadline is computed when the activity is scheduled. If no worker claims the activity before the deadline, the `TaskWatchdog` enforces the timeout. If retry attempts remain, a new activity task is scheduled with the snapped backoff and the schedule-to-start deadline is recomputed relative to the retry's available-at time — each retry gets a fresh window. If no `scheduleToStartTimeout` was configured, the deadline is cleared on retry. If all attempts are exhausted, a terminal `ActivityTimedOut` history event is recorded and the workflow is woken.

### Start-to-close timeout

The deadline is computed when a worker claims the activity task. Each retry gets a fresh start-to-close deadline. If the activity does not complete before the deadline, the current attempt is closed. If retry attempts remain, the execution returns to `Pending` with a reset schedule-to-start deadline (if configured) so the retried task is not immediately re-timed-out. If all attempts are exhausted, a terminal timeout is recorded.

### Schedule-to-close timeout

The deadline is computed once at scheduling time and never resets. When it expires, the activity is immediately failed as terminal — even if retry attempts remain, because the total allowed wall-clock time has passed. This is useful for bounding the overall cost of a flaky activity that might otherwise retry indefinitely within its per-attempt limits.

### Heartbeat timeout

The initial deadline is computed when a worker claims the activity task. Each successful `$this->heartbeat()` call extends the deadline by the configured interval. If the activity does not call `heartbeat()` before the deadline expires, the engine assumes the worker is unresponsive. If retry attempts remain, a new attempt is scheduled with a reset schedule-to-start deadline (if configured); otherwise a terminal timeout is recorded.

```php
use Workflow\V2\Activity;

class LongRunningActivity extends Activity
{
    public function handle($input)
    {
        foreach ($items as $item) {
            $this->heartbeat(['processed' => $count]);
            // ... process item ...
        }
    }
}
```

### Enforcement

Activity timeouts are enforced by the `TaskWatchdog` on each worker-loop pass. The watchdog scans for executions whose deadline columns have passed and delegates to `ActivityTimeoutEnforcer`. Each enforcement records:

- A terminal `ActivityTimedOut` history event with `timeout_kind` set to `schedule_to_start`, `start_to_close`, `schedule_to_close`, or `heartbeat`
- A `WorkflowFailure` row with `failure_category = timeout` and `propagation_kind = timeout`
- A workflow resume task to wake the parent workflow so it can observe the failure

Waterline displays the retry policy including all configured timeout types in the activity detail view. The timeline shows the timeout kind in the activity timed-out event message.

### What is not yet covered

The following are planned but not yet implemented:

- Retry policies at the workflow level

### Structural limits

Typed structural-limit failures for payload size, pending fan-out counts, and metadata size ceilings are enforced by the engine. See [Structural Limits](../constraints/structural-limits.md) for the full limit contract, configuration, and failure taxonomy.

<!-- Source: docs/features/schedules.md -->

# Schedules

Schedules let you start workflow runs on a recurring basis using cron expressions. Each schedule is a named, durable entity that the engine evaluates on every tick to determine whether a new run should be triggered.

## Creating a schedule

Use `ScheduleManager::create()` to define a named schedule:

```php
use Workflow\V2\Enums\ScheduleOverlapPolicy;
use Workflow\V2\Support\ScheduleManager;

$schedule = ScheduleManager::create(
    scheduleId: 'daily-invoice-sync',
    workflowClass: InvoiceSyncWorkflow::class,
    cronExpression: '0 2 * * *',
    arguments: ['nightly'],
    timezone: 'America/New_York',
    overlapPolicy: ScheduleOverlapPolicy::Skip,
    labels: ['team' => 'billing'],
    memo: ['origin' => 'scheduled'],
    searchAttributes: ['tenant_id' => '42'],
    notes: 'Runs every night at 2 AM ET.',
);
```

The `scheduleId` is a unique, user-chosen identifier for the schedule. Each triggered run gets a deterministic workflow instance ID derived from the schedule ID and trigger timestamp.

### Parameters

| Parameter | Type | Default | Description |
|---|---|---|---|
| `scheduleId` | `string` | required | Unique identifier for the schedule |
| `workflowClass` | `string` | required | The workflow class to start |
| `cronExpression` | `string` | required | Standard cron expression (5 fields) |
| `arguments` | `array` | `[]` | Arguments passed to the workflow's `handle()` method |
| `timezone` | `string` | `'UTC'` | Timezone for evaluating the cron expression |
| `overlapPolicy` | `ScheduleOverlapPolicy` | `Skip` | What to do when the previous run is still active |
| `labels` | `array` | `[]` | Visibility labels applied to each triggered run |
| `memo` | `array` | `[]` | Memo fields applied to each triggered run |
| `searchAttributes` | `array` | `[]` | Search attributes applied to each triggered run |
| `jitterSeconds` | `int` | `0` | Maximum random delay in seconds added to each fire time (thundering-herd mitigation) |
| `maxRuns` | `int\|null` | `null` | Maximum number of runs before auto-deleting the schedule |
| `connection` | `string\|null` | `null` | Queue connection for triggered runs (overrides the workflow class default) |
| `queue` | `string\|null` | `null` | Queue name for triggered runs (overrides the workflow class default) |
| `notes` | `string\|null` | `null` | Free-form operator notes |
| `namespace` | `string\|null` | `null` | Namespace for the schedule (defaults to the configured `workflows.v2.namespace` or `'default'`) |

## Advanced scheduling

The sections below cover scheduling features you will reach for when the basic cron pattern is not enough: fixed-interval firing, mixing cron and interval specs, and overlap policies. Skip ahead unless you need one.

### Interval-based schedules

In addition to cron expressions, schedules support interval-based firing using ISO 8601 duration syntax. Use `ScheduleManager::createFromSpec()` for full control over the schedule spec:

```php
use Workflow\V2\Support\ScheduleManager;

$schedule = ScheduleManager::createFromSpec(
    scheduleId: 'health-check-30m',
    spec: [
        'intervals' => [
            ['every' => 'PT30M'],
        ],
    ],
    action: [
        'workflow_type' => 'health-check',
        'workflow_class' => HealthCheckWorkflow::class,
        'input' => ['region' => 'us-east-1'],
    ],
);
```

### Interval spec fields

| Field | Type | Description |
|---|---|---|
| `every` | `string` | ISO 8601 duration (e.g., `PT30M` for 30 minutes, `PT1H` for 1 hour, `P1D` for 1 day) |
| `offset` | `string\|null` | Phase offset as ISO 8601 duration — shifts the alignment point of the interval |

The offset parameter controls where in the interval cycle the schedule fires. For example, an hourly interval with a 5-minute offset fires at `:05`, `:05+1h`, etc.:

```php
$schedule = ScheduleManager::createFromSpec(
    scheduleId: 'offset-hourly',
    spec: [
        'intervals' => [
            ['every' => 'PT1H', 'offset' => 'PT5M'],
        ],
    ],
    action: [
        'workflow_type' => 'sync-workflow',
        'workflow_class' => SyncWorkflow::class,
        'input' => [],
    ],
);
```

### Mixed cron and interval specs

A single schedule can combine cron expressions and intervals. The engine evaluates all specs and uses the earliest upcoming fire time:

```php
$schedule = ScheduleManager::createFromSpec(
    scheduleId: 'mixed-schedule',
    spec: [
        'cron_expressions' => ['0 12 * * *'],  // noon daily
        'intervals' => [['every' => 'PT6H']],  // every 6 hours
        'timezone' => 'America/Chicago',
    ],
    action: [
        'workflow_type' => 'report-workflow',
        'workflow_class' => ReportWorkflow::class,
        'input' => [],
    ],
);
```

The `createFromSpec` method accepts all the same lifecycle parameters as `create()` (`overlapPolicy`, `jitterSeconds`, `maxRuns`, `connection`, `queue`, `namespace`, etc.) as separate named arguments.

## Overlap policies

When a schedule fires and the previous run is still active, the overlap policy controls behavior:

| Policy | Behavior |
|---|---|
| `Skip` | Do not start a new run (default) |
| `BufferOne` | Buffer one pending trigger; skip further triggers until the buffer is drained. On the next `tick()` after the active run completes, the buffered trigger fires automatically. |
| `BufferAll` | Buffer all pending triggers with no cap; each buffered trigger drains sequentially as previous runs complete. |
| `AllowAll` | Start the new run regardless of the previous run's state |
| `CancelOther` | Cancel the previous run, then start the new run |
| `TerminateOther` | Terminate the previous run, then start the new run |

## Managing schedules

### Pause and resume

```php
ScheduleManager::pause($schedule);

// The schedule will not trigger while paused.

ScheduleManager::resume($schedule);
// next_fire_at is recalculated from now.
```

### Update

```php
ScheduleManager::update(
    $schedule,
    cronExpression: '30 3 * * *',
    timezone: 'America/Chicago',
    overlapPolicy: ScheduleOverlapPolicy::AllowAll,
    notes: 'Moved to 3:30 AM CT.',
);
```

Updating the cron expression or timezone recalculates `next_fire_at`.

### Delete

```php
ScheduleManager::delete($schedule);
```

Deleting is soft — the row remains with status `deleted` and a `deleted_at` timestamp. A deleted schedule cannot be paused, resumed, updated, or triggered.

### Describe

```php
$description = ScheduleManager::describe($schedule);

$description->scheduleId;      // 'daily-invoice-sync'
$description->namespace;       // 'default'
$description->status;          // ScheduleStatus::Active
$description->spec;            // ['cron_expressions' => ['0 2 * * *'], 'timezone' => 'America/New_York']
$description->overlapPolicy;   // ScheduleOverlapPolicy::Skip
$description->firesCount;      // 47
$description->nextFireAt;      // DateTimeInterface|null
$description->lastFiredAt;     // DateTimeInterface|null
$description->latestInstanceId; // 'schedule:daily-invoice-sync:...'
$description->jitterSeconds;   // 0
$description->note;            // 'Runs every night at 2 AM ET.'
$description->toArray();       // full array representation
```

### Find by schedule ID

```php
$schedule = ScheduleManager::findByScheduleId('daily-invoice-sync');
```

## Triggering schedules

### Manual trigger

```php
$instanceId = ScheduleManager::trigger($schedule);
```

This immediately evaluates the overlap policy and, if allowed, starts a new workflow run. Returns the instance ID of the started workflow, or `null` if the trigger was skipped.

### Tick (evaluate all due schedules)

```php
$results = ScheduleManager::tick();

// Returns rows with schedule_id, instance_id, outcome, occurrence_time,
// last_fired_at, and next_fire_at when those fields apply.
```

`tick()` finds all active schedules whose `next_fire_at` is in the past and triggers them in order. The `occurrence_time` field is the due fire time the scheduler observed. After each trigger, `next_fire_at` advances from the current clock to the next cron or interval occurrence.

### Missed-fire policy

Live schedule evaluation uses **fire once, then resume** semantics. If the scheduler is down across one or more nominal fire times, the first tick after recovery starts one workflow for the overdue `next_fire_at`, records that due time as `occurrence_time`, and then advances `next_fire_at` from the current clock. Additional nominal occurrences that passed while the scheduler was down are skipped by live evaluation.

Use `backfill()` when every missed occurrence must run. Backfill enumerates the requested window explicitly and records each occurrence separately.

### Artisan command

Run a single tick from the command line:

```bash
php artisan workflow:v2:schedule-tick
php artisan workflow:v2:schedule-tick --json
```

The standalone server exposes the same evaluation pass with its own
command:

```bash
php artisan schedule:evaluate --limit=100
```

To evaluate schedules continuously, call this command from Laravel's task scheduler:

```php
// app/Console/Kernel.php
$schedule->command('workflow:v2:schedule-tick')->everyMinute();
```

## Max runs

When `maxRuns` is set, the schedule tracks `remaining_actions`. After the last allowed trigger, the schedule is automatically soft-deleted.

```php
$schedule = ScheduleManager::create(
    scheduleId: 'one-shot-retry',
    workflowClass: RetryWorkflow::class,
    cronExpression: '*/5 * * * *',
    maxRuns: 3,
);

// After 3 triggers, the schedule status becomes 'deleted'.
```

## Backfill

Backfill triggers workflows for past cron occurrences that were missed (e.g., after a schedule was paused, a deployment outage, or late creation):

```php
$results = ScheduleManager::backfill(
    $schedule,
    from: new DateTimeImmutable('2026-04-10 00:00:00'),
    to: new DateTimeImmutable('2026-04-14 00:00:00'),
);

// Returns: [['schedule_id' => '...', 'instance_id' => '...|null', 'cron_time' => '...'], ...]
```

Each missed cron occurrence is triggered sequentially. The schedule's overlap policy applies to each occurrence, with one exception: **buffer policies (`BufferOne`, `BufferAll`) are treated as `AllowAll` during backfill**. Buffering is a real-time flow-control mechanism that has no meaning for catch-up operations — backfill should start every missed occurrence, not queue them into a buffer that will never drain. You can also override the policy explicitly:

```php
$results = ScheduleManager::backfill(
    $schedule,
    from: new DateTimeImmutable('2026-04-10 00:00:00'),
    to: new DateTimeImmutable('2026-04-14 00:00:00'),
    overlapPolicyOverride: ScheduleOverlapPolicy::AllowAll,
);
```

Backfill respects `maxRuns` — if the schedule's remaining actions are exhausted mid-backfill, the operation stops and the schedule is auto-deleted.

Backfill instance IDs are deterministic: `schedule:{scheduleId}:backfill:{timestamp}`.

## Queue routing

When `connection` or `queue` is set on a schedule, triggered workflows dispatch to that connection and queue instead of the workflow class default:

```php
$schedule = ScheduleManager::create(
    scheduleId: 'priority-sync',
    workflowClass: InvoiceSyncWorkflow::class,
    cronExpression: '0 * * * *',
    connection: 'redis',
    queue: 'high-priority',
);

// Every triggered run dispatches to redis/high-priority,
// regardless of InvoiceSyncWorkflow's default routing.
```

The routing precedence is: schedule fields → workflow class defaults → global queue config.

## Jitter

When multiple schedules share the same cron expression, they all fire at the exact same instant, creating a thundering-herd spike. The `jitterSeconds` parameter spreads triggers across a random window to smooth the load.

```php
$schedule = ScheduleManager::create(
    scheduleId: 'hourly-report',
    workflowClass: ReportWorkflow::class,
    cronExpression: '0 * * * *',
    jitterSeconds: 300, // fire within 0–300 seconds after the top of the hour
);
```

When `jitterSeconds` is set, each computed `next_fire_at` is offset by a random value between 0 and `jitterSeconds` (inclusive). The jitter is re-rolled every time the next fire time is calculated — after a trigger, after a resume, or after an update.

Jitter applies only to the tick-evaluation fire time stored in the database. Backfill enumeration always uses canonical (unjittered) cron times so that backfilled occurrences land on exact cron boundaries.

Setting `jitterSeconds` to `0` (the default) disables jitter entirely — fire times are exact cron matches.

## History event types

Schedule lifecycle events are recorded on two separate event streams:

- **Workflow-run lineage.** When a schedule triggers a workflow, a
  `ScheduleTriggered` event is appended to the started run's history
  (`workflow_history_events`). This gives the run a verifiable link back
  to the schedule that started it.
- **Schedule audit stream.** Every schedule lifecycle transition is
  recorded in the per-schedule audit log
  (`workflow_schedule_history_events`) under a monotonically increasing
  `sequence` field. The audit stream is authoritative for "what happened
  to this schedule"; the run history is authoritative for "why this run
  was started".

Both streams use the same `HistoryEventType` enum and the same
`HistoryEventPayloadContract` payload-key registry, so event names and
payload shapes stay in sync across streams.

### Run-lineage event (on the started workflow run)

`ScheduleTriggered` is appended to the triggered workflow run's history
with the following payload keys:

- `schedule_id` — schedule's user-facing identifier.
- `schedule_ulid` — schedule's internal ULID primary key.
- `cron_expression`, `timezone`, `overlap_policy` — the schedule's
  primary cron expression, IANA timezone, and active overlap policy at
  trigger time.
- `trigger_number` — which trigger this was (1-indexed).
- `occurrence_time` — the schedule fire time the scheduler evaluated.
  Tick-driven triggers record the due `next_fire_at`; backfill triggers
  record the enumerated backfill occurrence. With jitter enabled, the
  tick-driven value includes the jittered fire time stored on the
  schedule.

### Schedule audit stream (on the schedule itself)

Every schedule lifecycle transition is recorded on the schedule's own
audit stream. Sequences start at `1` for `ScheduleCreated` and increment
monotonically per schedule.

| Event | When recorded | Payload keys |
| --- | --- | --- |
| `ScheduleCreated` | Schedule was created | `spec`, `action`, `overlap_policy`, `next_fire_at`, `command_context` |
| `SchedulePaused` | Schedule was paused | `reason`, `paused_at`, `command_context` |
| `ScheduleResumed` | Schedule was resumed | `next_fire_at`, `command_context` |
| `ScheduleUpdated` | Schedule cron, timezone, spec, action, or policy was changed | `changed_fields`, `spec`, `action`, `overlap_policy`, `next_fire_at`, `command_context` |
| `ScheduleTriggered` | A workflow run was started from the schedule | `workflow_instance_id`, `workflow_run_id`, `outcome`, `effective_overlap_policy`, `trigger_number`, `occurrence_time`, `command_context` |
| `ScheduleTriggerSkipped` | A trigger was skipped due to overlap policy, non-triggerable status, or exhausted actions | `reason`, `skipped_trigger_count`, `last_skipped_at`, `command_context` |
| `ScheduleDeleted` | Schedule was soft-deleted — either by an explicit delete call or by exhausting `max_runs` (`reason: max_runs_exhausted`) | `reason`, `deleted_at`, `command_context` |

`command_context` carries the principal, request id, source, and reason
attributes recorded by `ScheduleManager` when the caller passes a
`CommandContext`. It is optional — events recorded without a context
omit the key rather than writing a blank value.

### Payload contract stability

The payload keys in both tables above are declared in
`Workflow\V2\Support\HistoryEventPayloadContract`, asserted on every
write, and pinned by test coverage in the workflow package. Adding a
new key to any schedule event is a wire-format change and follows the
history-event change rules in the
[version compatibility contract](../compatibility.md).

### Retention

The audit stream is retained for the life of the schedule row. Schedule
deletion is a soft delete that writes a final `ScheduleDeleted` event;
the audit rows themselves are not cascaded when a schedule row is
removed. Operators that purge historical schedules must choose an
explicit retention strategy for their deployment — the package ships no
built-in TTL for audit events.

### Visibility

The audit stream is exposed through every operator surface:

- **In-process (Eloquent).** `WorkflowSchedule::historyEvents()`
  (resolved through `ConfiguredV2Models::query(
  'schedule_history_event_model', ...)`) returns the stream for a
  schedule from within the host application.
- **Waterline HTTP.**
  `GET /waterline/api/v2/schedules/{scheduleId}/history` returns the
  stream with `limit` (1–500, default 100) and `after_sequence`
  cursor pagination. The response is scoped to the Waterline
  namespace so multi-tenant deployments only see their tenant's
  audit rows.
- **Waterline UI.** The **History** action on each row in the
  Waterline schedule registry opens a modal that renders the stream
  for that schedule. Events are shown with their sequence, recorded
  timestamp, event type, linked workflow instance and run IDs, and
  formatted payload, with a **Load more** control that advances the
  cursor in batches of 100.
- **Standalone server HTTP.**
  `GET /api/schedules/{scheduleId}/history` on the standalone
  server returns the same stream with the same pagination contract
  and the same `X-Namespace` scoping rules. History remains
  available after a schedule is soft-deleted, since the audit trail
  is what operators reach for to reconstruct a removed schedule.
- **CLI.** `dw schedule:history <schedule-id>` prints the stream
  as a table (`Seq`, `Event`, `Recorded At`, `Workflow Refs`) with
  a **More events available** hint when `has_more` is true.
  `--limit` and `--after-sequence` forward to the server endpoint,
  `--all` pages through every remaining event, and
  `--output=json` / `--output=jsonl` emit structured output
  (`jsonl` drops the cursor envelope so each line is a
  self-contained event).
- **Python SDK.** `Client.get_schedule_history(schedule_id, *, limit=None,
  after_sequence=None)` returns a single `ScheduleHistoryPage`, and
  `Client.iter_schedule_history(schedule_id, *, limit=None, after_sequence=None)`
  is an `AsyncIterator[ScheduleHistoryEvent]` that pages through the full
  stream with the same keyword arguments. `ScheduleHandle` exposes
  matching `.history(...)` and `.iter_history(...)` convenience methods.
  `limit` is clamped server-side between 1 and 500 (default 100) and
  `after_sequence` is a non-negative cursor obtained from the previous
  page's `next_cursor`.

All of these surfaces read from the same `workflow_schedule_history_events`
table; the payload-key contract in
[Payload contract stability](#payload-contract-stability) applies
regardless of which surface the operator uses.

### Migration behavior for pre-audit schedules

The schedule audit stream was introduced together with the
`workflow_schedule_history_events` table. Schedules that were
created before that migration ran have no retroactive
`ScheduleCreated` event — `ScheduleManager` records audit events
only as lifecycle transitions happen, and the migration does not
synthesize a backfilled event for existing schedules.

What this means for operators:

- A pre-existing schedule's stream is empty until its next
  lifecycle transition. Pausing, resuming, updating, triggering,
  skipping a trigger, or deleting the schedule appends events as
  normal from that point on.
- Sequence numbers still start at `1` and increment monotonically
  per schedule. A pre-existing schedule whose first recorded event
  is a `SchedulePaused` will have `sequence = 1` on that row; the
  absence of a preceding `ScheduleCreated` is expected.
- `ScheduleTriggered` and `ScheduleTriggerSkipped` events are
  written every time a tick evaluates the schedule, so any schedule
  that fires on a cron cadence after the migration will accumulate
  audit rows without operator intervention.
- No operator action is required to opt a schedule into the audit
  stream. The stream is always on; it simply has no rows until the
  first post-migration event is written.

## Skip tracking

When a trigger is skipped (due to overlap policy, non-triggerable status, or exhausted actions), the schedule tracks the skip:

- `last_skip_reason` — why the most recent trigger was skipped (e.g., `overlap_policy_skip`, `status_not_triggerable`, `remaining_actions_exhausted`)
- `last_skipped_at` — when the skip occurred
- `skipped_trigger_count` — cumulative number of skipped triggers

These fields are included in `ScheduleManager::describe()` and the Waterline schedule detail API.

## Namespace scoping

Schedules belong to a namespace. When `namespace` is passed to `create()` or `createFromSpec()`, the schedule is scoped to that namespace. When omitted, the schedule inherits the configured `workflows.v2.namespace` (defaulting to `'default'`).

Schedule IDs are unique within a namespace — the same `scheduleId` can exist in different namespaces without conflict.

```php
$schedule = ScheduleManager::create(
    scheduleId: 'daily-sync',
    workflowClass: SyncWorkflow::class,
    cronExpression: '0 2 * * *',
    namespace: 'billing',
);

// Find by schedule ID within a namespace:
$found = ScheduleManager::findByScheduleId('daily-sync', namespace: 'billing');
```

When Waterline is configured with a namespace (`waterline.namespace`), the schedule list and detail endpoints automatically scope to that namespace. This ensures multi-tenant deployments show only the schedules belonging to the operator's namespace.

## Database

The schedule table (`workflow_schedules`) is created by migration `2026_04_14_000157`. The model class is configurable via `workflows.v2.schedule_model`.

If your deployment runs package migrations alongside application migrations, migration 157 detects a pre-existing `workflow_schedules` table and handles it gracefully: if the table already matches the package schema it is left as-is; if it was created by an earlier shim migration with a different schema, it is replaced.

<!-- Source: docs/features/external-payload-storage.md -->

# External Payload Storage

External payload storage offloads large workflow payloads to a pluggable
object store (S3, GCS, Azure Blob, or a local filesystem) and replaces the
inline bytes in workflow history with a small, verifiable reference envelope.
Use it when activity or child-workflow arguments, results, signals, or update
payloads are too large to live inline in the database row that backs workflow
history.

The runtime still carries inline payloads as long as the encoded size stays
under the namespace threshold. Only payloads that cross the threshold are
written to the configured driver and recorded in history as a
`durable-workflow.v2.external-payload-reference.v1` envelope. Replay and
history export fail closed when a reference is missing, mutated, or outside
the configured prefix — the system never silently substitutes an empty value
for a missing blob.

## When To Use It

Prefer external payload storage whenever the application legitimately needs
to pass bytes larger than a few hundred kilobytes through a workflow:

- Document and media processing pipelines that hand PDFs, images, or audio
  blobs from one activity to the next.
- Reports, exports, or archives whose final output is a large serialized
  artifact.
- Message stream payloads produced by external systems that do not expose a
  stable object URL the workflow can reference directly.
- Any payload that would otherwise trip the
  [`payload_size_bytes` structural limit](../constraints/structural-limits.md#payload-size).

Small payloads — control-plane fields, ids, status flags, typical JSON —
stay inline and pay nothing extra. The policy is threshold-gated, so enabling
external storage on a namespace does not move small payloads.

## How Offload Works

Each namespace carries an independent external payload storage policy. When
the runtime encodes a payload for durable storage, it checks the encoded byte
length against the configured `threshold_bytes`:

- Encoded size is under the threshold. The payload is stored inline, as
  today. Nothing in history changes.
- Encoded size is at or over the threshold. The runtime hands the encoded
  bytes to the configured driver, receives back a driver-owned URI, and
  records an external payload reference in history. The reference carries the
  URI, a SHA-256 hash, the exact byte length, the payload codec, and an
  optional `expires_at` hint.

On replay, workers fetch the referenced bytes through the same driver,
verify that the returned object has the expected size and SHA-256, and only
then hand the payload to the decoder. A size or hash mismatch raises
`ExternalPayloadIntegrityException` (PHP) or `ExternalPayloadIntegrityError`
(Python) and surfaces as a replay failure — never as a silent empty payload.

The reference envelope is a stable wire format. It is identical whether the
producer is a PHP workflow, a Python SDK worker, or a direct HTTP API caller.
For the full field contract see
[External Payload Reference Envelope](../polyglot/server-api-reference.md#external-payload-reference-envelope).

## Decode Trust Boundary

Payload storage and payload decode are separate trust boundaries. An object
store can hold encoded bytes or references, while a codec server, custom
decoder, worker process, or history-export tool that decodes those bytes can
see plaintext application payloads.

Treat any codec server as a customer-managed trust boundary: decide where it
runs, which network can reach it, which keys it can access, what audit logs it
emits, and how decoded previews are redacted before they reach operator
surfaces. Durable Workflow records codec names, reference URIs, hashes, sizes,
schema fingerprints, and bounded previews, but those facts are not equivalent
to end-to-end encryption.

## Driver Choices

| Driver | URI scheme | Typical use |
| --- | --- | --- |
| `local` | `file://` | Local development, CI, and single-node deployments where the server and workers share a filesystem. Not suitable when workers run on different hosts than the server. |
| `s3` | `s3://` | Amazon S3 and S3-compatible object stores (MinIO, Cloudflare R2, etc.) through a server-side filesystem disk. |
| `gcs` | `gs://` | Google Cloud Storage through a server-side filesystem disk. |
| `azure` | `azure://` | Azure Blob Storage through a server-side filesystem disk. |

Object-store drivers configure the actual bucket/container credentials
through a named server-side filesystem disk, so secrets live in the server's
configuration rather than in the namespace policy record.

## Configuring A Namespace

Configure the policy with the [CLI](../polyglot/cli-reference.md#namespace-and-search-attribute-commands)
or the [server HTTP API](../polyglot/server-api-reference.md#namespace-and-storage).
Both write the same `external_payload_storage` envelope on the namespace
record.

### With The CLI

```bash
# Production namespace using Amazon S3 through the 'external-payload-objects' disk.
dw namespace:set-storage-driver billing s3 \
  --disk=external-payload-objects \
  --bucket=dw-payloads \
  --prefix=billing/ \
  --threshold-bytes=2097152

# Development namespace using the local filesystem.
dw namespace:set-storage-driver dev local \
  --uri=file:///var/lib/durable-workflow/payloads

# Disable offload while keeping the policy record (all payloads stay inline).
dw namespace:set-storage-driver billing s3 \
  --disk=external-payload-objects \
  --bucket=dw-payloads \
  --disable
```

### With The Server API

```bash
curl -sS -X PUT "$DURABLE_WORKFLOW_SERVER_URL/api/namespaces/billing/external-storage" \
  -H "Authorization: Bearer $DURABLE_WORKFLOW_AUTH_TOKEN" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": true,
    "driver": "s3",
    "threshold_bytes": 2097152,
    "config": {
      "disk": "external-payload-objects",
      "bucket": "dw-payloads",
      "prefix": "billing/"
    }
  }'
```

The namespace description returned by `GET /api/namespaces/{name}` or
`dw namespace:describe` carries the resolved `external_payload_storage`
envelope so operators and automation can verify the active policy without
re-issuing a write.

## Verifying The Policy

Use the round-trip diagnostic to prove a configured policy can actually
write and read bytes under the namespace's credentials before opening it to
workflow traffic:

```bash
dw storage:test --namespace=billing --large-bytes=2097152 --json
```

The diagnostic writes a small inline payload plus one payload that crosses
the threshold, fetches both back, verifies size and SHA-256, and returns
machine-readable `small_payload` and `large_payload` result objects.
A passing large-payload result proves the driver can produce a valid
`durable-workflow.v2.external-payload-reference.v1` envelope end to end. A
failing diagnostic should be treated as a storage-policy problem — do not
enable workflow traffic through a namespace whose policy cannot pass the
round trip.

## Picking A Threshold

The default behavior is to leave inline payloads alone unless they cross
`threshold_bytes`. Good starting points:

- Match the threshold to the point at which inline payloads start creating
  operational pressure — usually somewhere between 256 KiB and 2 MiB of
  encoded bytes.
- Leave comfortable headroom under the namespace
  [`payload_size_bytes`](../constraints/structural-limits.md#payload-size)
  structural limit so that the reference envelope is the cap, not the bytes
  themselves.
- Set a single threshold per namespace. Choose it from the payload-producing
  activity or workflow that drives the highest bytes-per-run, rather than
  tuning it for the median payload.

There is no benefit to setting a very low threshold: small payloads round
trip through the database faster than they round trip through external
storage, and the reference envelope itself consumes a (small) amount of
history space.

## Replay, Retention, And Cleanup

- **Replay integrity.** Every fetch verifies the stored object against the
  reference's `size_bytes` and `sha256`. A mutated or missing blob raises an
  integrity exception rather than silently substituting a different value.
- **Verified-fetch cache.** Workers cache verified bytes by
  `(uri, sha256, size, codec)` with a bounded entry count and byte ceiling.
  Repeated history reads on the same run avoid refetching the same object
  without weakening the integrity check on first load.
- **Retention.** When the server's retention pass removes a workflow run,
  it also deletes the external payload objects referenced by that run's
  history. Orphan objects do not accumulate as long as retention is running.
- **History export.** Exported history preserves the reference envelope.
  Downstream consumers that need the referenced bytes should fetch through
  the same driver and verify against the envelope before decode — the export
  format does not inline external bytes.

## Using It From Code

Most applications never call the storage API directly: the runtime offloads
transparently based on the namespace policy, and the SDK decodes references
on replay. Applications that need to build or consume envelopes outside the
runtime — for example, a language-neutral bridge handler or a test that
synthesizes a large payload — use the SDK helpers.

- **PHP (workflow package).** The
  `Workflow\V2\Support\ExternalPayloadStorage` helper stores and fetches
  bytes through any driver implementing
  `Workflow\V2\Contracts\ExternalPayloadStorageDriver`.
  `LocalFilesystemExternalPayloadStorage` handles `file://` URIs, and the
  standalone server ships a filesystem-disk driver that backs the `s3`,
  `gcs`, and `azure` policy drivers through a named Laravel disk.
- **Python SDK.** See
  [External Payload Storage](../polyglot/python.md#external-payload-storage)
  for `ExternalPayloadReference`, `ExternalPayloadCache`,
  `store_external_payload()`, `fetch_external_payload()`, and the
  `LocalFilesystemExternalStorage`, `S3ExternalStorage`,
  `GCSExternalStorage`, and `AzureBlobExternalStorage` adapters. Cloud SDK
  clients remain application-owned; the SDK does not add boto3,
  google-cloud-storage, or azure-storage-blob as runtime dependencies.
- **Direct HTTP.** HTTP callers that encode payloads manually can store
  bytes through the driver, then submit the reference envelope as the
  payload field on the request. The worker-protocol payload envelope
  (`{codec, blob}`) still carries references for activity arguments,
  results, signal payloads, and update payloads.

## See Also

- [Passing Data](../defining-workflows/passing-data.md) for the default
  inline payload contract.
- [Structural Limits: Payload Size](../constraints/structural-limits.md#payload-size)
  for the engine-enforced ceiling that external storage lets you work under.
- [Server API Reference: Namespace And Storage](../polyglot/server-api-reference.md#namespace-and-storage)
  for the full HTTP contract, including the reference envelope fields.
- [CLI Reference: Namespace And Search Attribute Commands](../polyglot/cli-reference.md#namespace-and-search-attribute-commands)
  for `dw namespace:set-storage-driver` and `dw storage:test` usage.
- [Python SDK: External Payload Storage](../polyglot/python.md#external-payload-storage)
  for Python-side drivers, helpers, and replay-cache guidance.

<!-- Source: docs/features/activity-execution-model.md -->

# Activity Execution Model

For service-worker support and fail-closed capability negotiation across PHP,
Python, and Rust, see the service-mode
[Portable Worker Affinity](/docs/polyglot/portable-worker-affinity) support matrix.

Durable Workflow v2 now has explicit primitives for the common activity
placement choices:

- ordinary queued activities for durable, independently leased work;
- [local activities](/docs/features/local-activities) for short
  same-process activity work that still records activity history;
- [worker sessions](/docs/features/worker-sessions) for durable activity
  affinity across multiple queued activity steps;
- [sticky execution](/docs/features/sticky-execution) for workflow replay
  cache affinity, with ordinary replay as the correctness fallback.

## Ordinary Queued Activities

`activity(...)` and `Workflow::activity(...)` are ordinary queued activities.
They remain the default activity primitive.

1. Workflow code calls `activity(MyActivity::class, ...)`.
2. The workflow task records `ActivityScheduled` and creates a durable activity
   task on the configured connection and queue.
3. A compatible worker claims that activity task under a lease.
4. The worker runs the activity class and reports completion, failure,
   cancellation, heartbeat, or timeout.
5. The engine records the activity outcome on workflow history and resumes the
   workflow from durable state.

Ordinary activity attempts may run on any compatible worker. A retry may land
on a different process, host, or build. Activity code must be idempotent across
duplicate delivery, retries after lease expiry, and late completion races. Use
`activity_execution_id` as the default remote idempotency key and
`activity_attempt_id` only when a downstream system needs per-attempt
correlation.

```php
use function Workflow\V2\activity;

$quote = activity(FetchQuoteActivity::class, $customerId);
$invoice = activity(CreateInvoiceActivity::class, $quote['id']);
```

## Local Activities

`localActivity(...)`, `Workflow::localActivity(...)`, and
`Workflow::executeLocalActivity(...)` run an activity class inside the workflow
worker process that is currently executing the workflow task. They do not
create an ordinary activity task.

Local activities still record normal activity history:

- `ActivityScheduled`
- `ActivityStarted`
- `ActivityHeartbeatRecorded`
- `ActivityRetryScheduled`
- `ActivityCompleted`
- `ActivityFailed`
- `ActivityCancelled`
- `ActivityTimedOut`

Each local activity event carries `execution_mode=local` and
`local_activity=true`, and the activity execution snapshot stores
`activity_options.execution_mode=local`. Activity heartbeats renew the owning
workflow task lease while the local attempt runs.

Use local activities for short, idempotent side effects that need activity
retry, timeout, heartbeat, cancellation, and visibility semantics but do not
need queue routing or an independent activity worker fleet. See
[Local Activities](/docs/features/local-activities) for the API,
timeouts, retry, shutdown, cold-replay, routing, and metrics contract.

```php
use Workflow\V2\Support\LocalActivityOptions;
use function Workflow\V2\localActivity;

$receipt = localActivity(
    SendReceiptActivity::class,
    new LocalActivityOptions(maxAttempts: 3, startToCloseTimeout: 10),
    $orderId,
);
```

## Worker Sessions

Worker sessions pin a sequence of ordinary queued activity attempts to one
worker-session lease. Use them when multiple durable steps must reuse
worker-local resources such as GPU memory, a mounted filesystem, or a loaded
model.

Worker sessions do not make an activity local. Each in-session activity is
still a durable activity task with its own lease, heartbeat, timeout, retry,
and terminal history. The session adds explicit affinity and admission rules on
top of normal queue routing.

See [Worker Sessions](/docs/features/worker-sessions) for the session API,
lease lifecycle, routing, failure handling, shutdown behavior, and operator
diagnostics.

## Sticky Execution

Sticky execution is a workflow-task replay optimization. A worker may keep a
warm process-local replay cache after completing a workflow task, and matching
may prefer that worker for the next workflow task for the same run.

Sticky execution does not make workflow progress process-local. Correctness
always falls back to ordinary cold replay from durable history after cache
misses, worker restart, drain, rollout, or eviction. Workflow code must not
rely on in-memory state outside history.

See [Sticky Execution](/docs/features/sticky-execution) for the sticky
cache lifecycle, routing identity, fallback rules, deployment controls, and
metrics.

## Choosing The Right Primitive

Use workflow code directly for deterministic branching and calculations.

Use [`sideEffect(...)`](/docs/features/side-effects) for a replay-safe
snapshot of a non-deterministic value that does not need activity retry,
timeout, heartbeat, or cancellation semantics.

Use [local activities](/docs/features/local-activities) for short,
retryable, same-process side effects that should be visible as activity
attempts but should not enter normal task matching.

Use ordinary [activities](/docs/defining-workflows/activities) for remote
calls, slow I/O, CPU-heavy work, queue backpressure, dedicated worker fleets,
or work that should continue through a separately leased activity task.

Use [worker sessions](/docs/features/worker-sessions) when multiple
ordinary activity steps need explicit worker-local affinity.

<!-- Source: docs/features/local-activities.md -->

# Local Activities

Local activities are the v2 primitive for short activity work that should run
inside the workflow worker process currently executing the workflow task. They
preserve activity retry, timeout, heartbeat, cancellation, history, and
operator visibility semantics, but they bypass ordinary activity-task queueing.

Use local activities when the work is low-latency, idempotent, and appropriate
for the workflow worker itself. Use ordinary queued activities when the work
needs independent worker scaling, queue routing, long execution, or a separate
activity-task lease.

## Contract Summary

- `localActivity(...)` and `Workflow::localActivity(...)` execute the activity
  class in the same process as the current workflow task.
- The runtime creates an `activity_executions` row and normal activity history
  events marked with `execution_mode=local` and `local_activity=true`.
- No ordinary `TaskType::Activity` task is created, so `connection`, `queue`,
  worker-session, and schedule-to-start routing options are rejected.
- The local attempt owns the workflow task lease. Activity `heartbeat()` renews
  that workflow task lease and records `ActivityHeartbeatRecorded`.
- Retries are durable workflow tasks with backoff, not hidden loop retries.
- Cold replay reads committed activity history. If a started local attempt has
  no terminal event after worker loss, the next attempt records
  `retry_reason=cold_replay`.
- Run detail, history export, timelines, and operator metrics expose local
  attempts separately from ordinary queued attempts.

## Authoring API

Use the namespaced helper:

```php
use Workflow\V2\Support\LocalActivityOptions;
use function Workflow\V2\localActivity;

$receipt = localActivity(
    SendReceiptActivity::class,
    new LocalActivityOptions(
        maxAttempts: 3,
        startToCloseTimeout: 10,
        scheduleToCloseTimeout: 30,
        heartbeatTimeout: 5,
    ),
    $orderId,
);
```

Or use the static workflow facade:

```php
use Workflow\V2\Workflow;

$receipt = Workflow::localActivity(SendReceiptActivity::class, $orderId);
$receipt = Workflow::executeLocalActivity(SendReceiptActivity::class, $orderId);
```

`LocalActivityOptions` accepts retry and timeout fields:

- `maxAttempts`
- `backoff`
- `startToCloseTimeout`
- `scheduleToCloseTimeout`
- `heartbeatTimeout`
- `nonRetryableErrorTypes`

It rejects `connection`, `queue`, worker-session routing, and
`scheduleToStartTimeout` because a local activity does not enter normal task
matching.

## Execution And History

When workflow replay reaches a local activity, the current workflow task:

1. creates the activity execution with `activity_options.execution_mode=local`;
2. records `ActivityScheduled` and `ActivityStarted` with the local marker;
3. instantiates and runs the activity class in the workflow worker process;
4. records `ActivityCompleted`, `ActivityFailed`, `ActivityTimedOut`, or
   `ActivityCancelled`;
5. resumes workflow code from the recorded activity event.

Replay does not rerun a completed local activity. Query replay and cold replay
read the same activity history events that ordinary activities use.

## Heartbeats

A local activity does not own an activity task lease. It owns the workflow task
lease that is currently executing the workflow. At attempt start, the runtime
renews the workflow task lease. When activity code calls `$this->heartbeat()`,
the runtime records progress, updates the activity attempt, and renews the
workflow task lease.

Long-running local activities must heartbeat often enough to keep both the
local activity heartbeat timeout and the workflow task lease healthy:

```php
use Workflow\V2\Activity;

final class PollShortJobActivity extends Activity
{
    public function handle(string $jobId): array
    {
        $state = $this->fetch($jobId);

        $this->heartbeat([
            'message' => 'Polling remote job',
            'job_id' => $jobId,
            'state' => $state['status'],
        ]);

        return $state;
    }
}
```

## Timeouts And Retries

`startToCloseTimeout` limits one attempt. `scheduleToCloseTimeout` limits the
whole local execution across retries. `heartbeatTimeout` limits the gap between
recorded local activity heartbeats.

When a retryable failure or timeout occurs, the runtime records
`ActivityRetryScheduled` and creates a workflow task that becomes available
after the retry backoff. That retry task replays workflow history, reaches the
same local activity sequence, and starts the next local attempt.

Each local attempt is a new `activity_attempts` row. A retry records
`retry_reason` as `failure`, `timeout`, or `cold_replay`.

## Cancellation And Worker Loss

Cancellation is cooperative. A local activity observes cancellation at
heartbeat, timeout enforcement, and attempt-completion boundaries. A cancelled
local attempt records `ActivityCancelled` with the local marker.

If a worker exits before committing a terminal local activity event, the
workflow task lease expires and normal task repair reclaims the workflow task.
Cold replay then reads committed history. If history contains a started local
attempt without a terminal event, the runtime schedules a retry with
`retry_reason=cold_replay`.

## Visibility

Operators can distinguish local activities everywhere activity state is
reported:

- history payloads include `execution_mode=local` and `local_activity=true`;
- `activity_executions.activity_options.execution_mode` is `local`;
- run detail and history export include `execution_mode` and `local_activity`;
- operator metrics expose `activities.local`, `activities.local_open`,
  `activities.local_attempts`, and queued-vs-local activity counters.

The runtime manifest is published at
`worker_protocol.server_capabilities.local_activities` in
`GET /api/cluster/info`. The machine-readable contract is
[`local-activity-runtime.schema.json`](/platform-protocol-specs/local-activity-runtime.schema.json)
and is indexed by the
[Platform Protocol Specs](/docs/platform-protocol-specs#local-activity-runtime-notes)
catalog.

The event names remain normal activity event names so timelines and replay
tools preserve ordering without a parallel event family.

## Choosing The Right Primitive

Use a local activity for short, retryable, idempotent side effects that are
best executed by the workflow worker process and do not need queue routing.

Use an ordinary [activity](/docs/defining-workflows/activities) for remote
calls, slow I/O, CPU-heavy work, dedicated worker fleets, backpressure, or
work that should keep making progress through a separately leased activity
task after workflow worker loss.

Use [worker sessions](/docs/features/worker-sessions) when multiple
ordinary activity steps must reuse worker-local resources such as GPU memory
or a mounted filesystem.

Use [`sideEffect(...)`](/docs/features/side-effects) only for replay-safe
snapshots that do not need activity retry, timeout, heartbeat, or cancellation
semantics.

<!-- Source: docs/features/worker-sessions.md -->

# Worker Sessions

Worker sessions are the v2 activity-affinity primitive. Use them when several
durable activity steps must reuse process-local state, GPU memory, a mounted
filesystem, or another worker-local resource.

## Contract Summary

- A session has one lease owner at a time.
- The first admitted in-session activity creates or reacquires the lease.
- Activity heartbeats renew both the activity attempt lease and the session
  lease.
- `POST /api/worker/sessions`, `POST /api/worker/sessions/{sessionId}/heartbeat`,
  and `DELETE /api/worker/sessions/{sessionId}` expose explicit lifecycle
  verbs for external workers.
- Queue routing still runs first; worker-session admission runs after the
  worker is registered for the task queue.
- Worker registration `capabilities` must satisfy every session requirement.
- Expired and orphaned sessions are visible to operators and may be reacquired
  when the session allows failure recovery.

## Authoring API

PHP workflow authors create a session handle with `Workflow::workerSession()`
or `Workflow\V2\workerSession()` and schedule activities through that handle:

```php
use Workflow\V2\Support\WorkerSessionOptions;
use Workflow\V2\Workflow;

$session = Workflow::workerSession(
    'gpu-render',
    new WorkerSessionOptions(
        queue: 'gpu-activities',
        requirements: ['gpu:nvidia-l4'],
        leaseSeconds: 120,
        ttlSeconds: 1800,
        maxConcurrentActivities: 1,
    ),
);

$frames = $session->activity(RenderFramesActivity::class, $videoId);
$manifest = $session->activity(AssembleManifestActivity::class, $frames['path']);
```

The activity option snapshot stores the same contract under
`activity_options.worker_session`. External runtimes use the same JSON shape
on a `schedule_activity` workflow-task command under `worker_session`.

```json
{
  "type": "schedule_activity",
  "activity_type": "media.render-frames",
  "worker_session": {
    "session_id": "gpu-render",
    "queue": "gpu-activities",
    "requirements": ["gpu:nvidia-l4"],
    "lease_seconds": 120,
    "ttl_seconds": 1800,
    "max_concurrent_activities": 1,
    "create_if_missing": true,
    "allow_reacquire_after_failure": true
  }
}
```

## Lifecycle

Session creation is lazy. The matching layer creates or reacquires a session
when the first in-session activity task is admitted to a capable worker. A
worker may also create the session explicitly before polling by calling
`POST /api/worker/sessions`.

An active session ends when the holder closes it, the session lease expires,
the absolute TTL expires, or the holding worker is detected as failed. TTL
expiry and explicit close are terminal for that session id. Lease expiry and
orphan detection may be reacquired when `allow_reacquire_after_failure` is
true.

## Lease And Ownership

At most one worker owns a session lease at a time. The server admits
in-session activity tasks only when one of these is true:

- no session exists and `create_if_missing` is true;
- the active session is already owned by the polling worker;
- the session is expired, failed, or orphaned and reacquisition is allowed.

Session ownership does not replace the per-attempt activity lease. Each
activity attempt still has its own `activity_attempt_id`, lease owner,
heartbeat, completion, failure, timeout, and cancellation path.

## Admission And Routing

Worker sessions participate in normal queue routing. `WorkerSessionOptions`
may set `connection` and `queue`; per-call `ActivityOptions` can still
override them. The server enforces session-specific admission after normal
task-queue admission:

- worker registration `capabilities` must cover every session requirement;
- `max_concurrent_activities` caps leased activity attempts inside the
  session;
- `max_concurrent_worker_sessions` caps active session leases held by one
  worker registration.

Fleet-specific routing uses plain capability strings such as `gpu:nvidia-l4`,
`gpu:a100`, `fs:/mnt/models`, or `zone:us-east-1a`.

## Failure, Cancellation, And Shutdown

Activity heartbeats renew the activity attempt lease and the session lease.
Workers may also renew the session directly through the worker protocol. If
the session lease expires, the session becomes `expired`; if the holding
worker registration heartbeat is stale or missing, the session becomes
`orphaned`.

When the holding worker dies mid-sequence, in-flight activities keep ordinary
at-least-once behavior: their attempt leases expire, repair makes them
claimable again, and stale completion may be rejected. A capable worker may
reacquire the session when the contract allows it. Workflow authors must
expect process-local state to be rebuilt after reacquisition.

Workflow cancellation still propagates through activity heartbeat responses.
A session lease never authorizes an in-session activity to ignore
`cancel_requested`. Planned worker shutdown should close held sessions through
`DELETE /api/worker/sessions/{sessionId}` before stopping the process;
in-flight activities still finish, fail, cancel, or expire under their own
attempt leases.

## Visibility

Worker-protocol capabilities advertise `worker_session_verbs` and the
`worker_sessions` runtime contract through `GET /api/cluster/info` and every
worker-plane response. The machine-readable contract is published in
[`worker-sessions-runtime.schema.json`](/platform-protocol-specs/worker-sessions-runtime.schema.json)
and is indexed by the
[Platform Protocol Specs](/docs/platform-protocol-specs#worker-session-runtime-notes)
catalog.

Operators can list active, closed, expired, failed, and orphaned sessions at
`GET /api/worker-sessions`. The detail surface includes session id, holder,
queue, requirements, lease expiry, TTL expiry, active activity count, and
failure reason. System operator metrics include counts for each session
status.

## Choosing Worker Sessions

Prefer ordinary queued activities when each step is independent. Prefer one
larger activity when the whole operation is one atomic side effect. Use a
worker session only when multiple durable activity steps must reuse a
worker-local resource and the workflow can tolerate rebuilding that resource
after worker failure.

See [Activity Execution Model](/docs/features/activity-execution-model)
for how worker sessions relate to ordinary queued activities, local
activities, and sticky execution.

<!-- Source: docs/features/sticky-execution.md -->

# Sticky Execution

Sticky execution is a supported Durable Workflow v2 replay optimization. A
worker can keep a warm, process-local workflow cache after it completes a
workflow task, and matching can prefer that worker for the next workflow task
for the same run.

Sticky execution is not a correctness feature. Workflow progress is still
committed only through durable history, and ordinary cold replay from history
is always valid. Workflow code must not rely on process-local state for
correctness.

## Contract Summary

Sticky execution has four guarantees:

- Sticky caches are owned by worker processes, not by the server or database.
- Sticky routing uses the worker protocol `worker_id` as the routing identity.
- Sticky affinity is advisory and expires; cold replay is the mandatory
  fallback for cache misses, worker restart, drain, rollout, or eviction.
- Operators have named controls and diagnostics for enablement, TTL, capacity,
  hit rate, miss rate, forced cold replay, and capacity pressure.

The durable affinity fields are `sticky_worker_id` and `sticky_until` on runs
and workflow tasks. The workflow-task diagnostic fields are
`sticky_replay_mode` and `sticky_claimed_at`.

## Sticky-Cache Lifecycle

A sticky cache is a local worker data structure containing replayed workflow
state for one or more runs. When a sticky-capable worker completes a workflow
task, the server may record that worker as the run's sticky owner until
`sticky_until`. Follow-up workflow tasks inherit that affinity when they are
created.

The worker owns cache contents and eviction policy. A worker may evict cached
runs when it reaches capacity, begins draining, restarts, changes build,
detects unsafe cached state, or chooses to free memory. The server never treats
cache contents as durable state.

If a worker receives a sticky-routed task but no longer has a valid cache entry,
the worker must perform cold replay from durable history. The task lease remains
the only authority for committing workflow progress.

## Routing Identity

The routing identity is the worker protocol `worker_id`. Workers opt into
sticky routing by registering with sticky cache enabled and by continuing to
heartbeat as active workers on the task queue.

Matching follows these rules:

- A task with active affinity for the polling worker is preferred.
- A task with no active affinity can be claimed by any compatible worker.
- A task with active affinity for another live sticky worker is held for that
  owner until `sticky_until` expires or the owner becomes unavailable.
- A task with expired affinity, stale-owner affinity, or disabled sticky
  execution can be claimed normally and cold replayed.

Sticky routing does not bypass compatibility, queue, namespace, or lease
checks. It only changes ready-task preference while ordinary replay remains the
fallback.

## Fallback Semantics

Cold replay is mandatory fallback. It happens when:

- sticky execution is disabled.
- the worker did not register sticky-cache support.
- the task has no active `sticky_worker_id` and `sticky_until`.
- the sticky owner is stale, missing, draining, restarted, or rolled out.
- the sticky owner evicted the run.
- the polling worker is not the sticky owner after the affinity expired.

The replay-mode diagnostics are:

- `sticky_hit_expected` - the sticky owner claimed the task before expiry.
- `cold_replay` - no sticky affinity applied.
- `forced_cold_replay` - affinity existed, but the task must be replayed cold.

`forced_cold_replay` is not a correctness failure. It means sticky execution
did not deliver its intended replay-speed benefit for that task.

## Deployment, Drain, and Rollout

Sticky execution follows the worker lifecycle. During drain, a worker should
stop claiming new workflow tasks while completing, heartbeating, failing, or
letting existing leases expire under the normal lease contract. Once the worker
is stale or no longer active, other compatible workers can claim its sticky
tasks after the affinity expires and cold replay them.

Replacement workers do not inherit process-local caches. A rollout can
therefore increase `forced_cold_replay` until new workers warm their own
caches. Use unique `worker_id` values per worker process or restart so
operator diagnostics can distinguish an old cache owner from a new process.

Build-id compatibility and workflow-definition fingerprinting still decide
whether a worker may execute a workflow task. Sticky execution is never a way to
route incompatible code to a run.

## Operator Controls

Sticky execution is controlled by workflow/runtime configuration and worker
capabilities, not by standalone server-image environment variables. The
workflow package configuration exposes the enablement flag and affinity TTL:

```php
'workflows' => [
    'v2' => [
        'sticky_execution' => [
            'enabled' => true,
            'ttl_seconds' => 300,
        ],
    ],
],
```

Worker cache capacity is advertised by each worker at registration and on
heartbeat. Disable sticky routing in runtime configuration, or run workers
without sticky-cache support, without changing workflow semantics. Existing
runs continue by ordinary cold replay.

## Worker Protocol Fields

Sticky-capable workers advertise support at registration:

```json
{
  "worker_id": "orders-worker-01",
  "task_queue": "orders",
  "runtime": "python",
  "sticky_cache_enabled": true,
  "sticky_cache_capacity": 100
}
```

Workers report cache diagnostics on heartbeat:

```json
{
  "worker_id": "orders-worker-01",
  "sticky_cache": {
    "enabled": true,
    "capacity": 100,
    "size": 72,
    "hit_count": 940,
    "miss_count": 31,
    "forced_cold_replay_count": 8,
    "eviction_count": 15
  }
}
```

Workflow-task poll responses include `task.sticky_execution` with the
`sticky_worker_id`, `sticky_until`, `replay_mode`, and `cache_directive`.
Workers should treat `resume_if_present` as permission to use a valid warm
cache and should cold replay if the cache entry is absent or invalid.

## Metrics and Diagnostics

Operator metrics include:

- `sticky_execution.active_sticky_runs`
- `sticky_execution.ready_sticky_tasks`
- `sticky_execution.leased_sticky_tasks`
- `sticky_execution.hit_expected_last_minute`
- `sticky_execution.miss_last_minute`
- `sticky_execution.forced_cold_replay_last_minute`
- `sticky_execution.cold_replay_last_minute`
- `sticky_execution.hit_rate_last_minute`
- `sticky_execution.miss_rate_last_minute`
- `sticky_execution.capacity_pressure_tasks`

The standalone server also reports worker cache capacity, cache size, cache hit
count, cache miss count, forced cold replay count, cache eviction count, and
capacity-pressure worker count under `sticky_execution_workers`.

Use these diagnostics this way:

- Low hit rate with healthy workers usually means the TTL is too short, workers
  are being replaced often, or cache capacity is too small.
- High miss rate or high forced cold replay means correctness is protected, but
  sticky execution is not improving replay cost.
- Capacity pressure means workers are near or above their reported sticky-cache
  capacity and may evict warm runs.

## Replay-Safe Code

Workflow code must behave identically under sticky and cold-replay execution.
Only durable history is safe for workflow decisions: workflow inputs, activity
results, timers, signals, updates, side effects, version markers, memo, and
search attributes.

Do not rely on mutable globals, local files, open sockets, object identity,
random values, wall-clock reads, or any other process-local state for
correctness. Sticky execution may preserve those values by accident on one
task, then lose them on a cache miss, worker restart, rollout, or eviction.

Use [`sideEffect(...)`](/docs/features/side-effects) for non-deterministic
values that must be recorded once, and use ordinary activities for external
side effects. See
[Execution Guarantees and Idempotency](/docs/constraints/execution-guarantees)
for the replay and durable-history contract.

<!-- Source: docs/configuration/publishing-config.md -->

# Publishing Config

This will create a `workflows.php` configuration file in your `config` folder.

```bash
php artisan vendor:publish --provider="Workflow\Providers\WorkflowServiceProvider" --tag="config"
```

## Changing Workflows Folder

By default, the `make` commands will write to the `app/Workflows` folder.

```php
php artisan make:workflow MyWorkflow
php artisan make:activity MyActivity
```

This can be changed by updating the `workflows_folder` setting.

```php
'workflows_folder' => 'Workflows',
```

## Using Custom Models (Legacy v1)

:::note Legacy

These `stored_workflow_*` keys configure the v1 `Workflow\Models\StoredWorkflow*` classes. Workflow v2 uses the durable model overrides below — `instance_model`, `run_model`, `task_model`, and so on. Keep the v1 keys only if you are still running v1 workflows during migration.

:::

In the published `workflows.php` config file you can update the v1 model classes to use your own subclasses.

```php
'stored_workflow_model' => App\Models\StoredWorkflow::class,

'stored_workflow_exception_model' => App\Models\StoredWorkflowException::class,

'stored_workflow_log_model' => App\Models\StoredWorkflowLog::class,

'stored_workflow_signal_model' => App\Models\StoredWorkflowSignal::class,

'stored_workflow_timer_model' => App\Models\StoredWorkflowTimer::class,
```

## Using Model Overrides (v2)

The runtime also exposes model overrides for the durable instance, run, task, history, and projection tables:

```php
'v2' => [
    'instance_model' => App\Models\WorkflowInstance::class,
    'run_model' => App\Models\WorkflowRun::class,
    'task_model' => App\Models\WorkflowTask::class,
    'history_event_model' => App\Models\WorkflowHistoryEvent::class,
    'run_summary_model' => App\Models\WorkflowRunSummary::class,
    'run_wait_model' => App\Models\WorkflowRunWait::class,
    'run_timeline_entry_model' => App\Models\WorkflowTimelineEntry::class,
    'run_timer_entry_model' => App\Models\WorkflowRunTimerEntry::class,
    'run_lineage_entry_model' => App\Models\WorkflowRunLineageEntry::class,
],
```

Those overrides are not limited to reads or projection rebuilds. `WorkflowStub::make()`, `load()`, `loadSelection()`, instance reservation, run selection, and workflow-task execution all use the configured `instance_model`, `run_model`, and `task_model`, and Waterline detail plus history export read through the same configured classes. One app-level override therefore governs both the core runtime path and operator-facing reads.

Keep custom subclasses schema-compatible with the built-in models. If a subclass also changes table names or other Eloquent conventions that the package models normally infer, override the affected relations on that subclass as well so `currentRun()`, `runs()`, and similar lookups stay aligned with your custom schema.

See the [Customization Matrix](./customization-matrix.md) for the frozen v2
support contract, including which overrides are safe as inherited subclasses,
which ones need explicit relation overrides, and how serializer, repository,
health, import, migration, and write-side behavior fit together.

## Payload Codec

v2 uses `avro` for new workflow payloads:

```php
'serializer' => 'avro',
```

- **`avro`** (default, required for new v2 workflows) — Apache Avro binary encoding. Compact on the wire and in storage, and faster to encode/decode for large payloads than the legacy PHP serializer.

If a published v1 config still sets `serializer`, final v2 keeps reading the value for `workflow:v2:doctor` diagnostics, but new v2 payloads still resolve to Avro.

### Legacy codecs (v1 migration only)

Two PHP-only codecs remain available for reading v1 history during migration:

- `workflow-serializer-y` — PHP `SerializableClosure` with byte-escape encoding (the v1 default).
- `workflow-serializer-base64` — PHP `SerializableClosure` with base64 encoding.

Setting `serializer` to a legacy codec will be flagged by `php artisan workflow:v2:doctor`. New v2 workflows still resolve to Avro; keep a legacy codec setting only while you are finishing or importing v1 runs that need PHP-native payload decoding.

The internal v1 import/drain reader recognizes legacy fully-qualified class names (for example, `Workflow\Serializers\Y::class`). They are not public v2 codec aliases and cannot be selected by a new v2 run or SDK.

## Compatibility Markers

The runtime can stamp each new run with a compatibility marker and let workers advertise which markers they can execute safely. This is the runtime fence that keeps long-lived runs on compatible builds during deliberate worker build rollouts.

Set the marker for new runs on the current build:

```env
DW_V2_CURRENT_COMPATIBILITY=build-2026-04
```

Optionally tell a worker to accept more than one marker during a rollout:

```env
DW_V2_SUPPORTED_COMPATIBILITIES=build-2026-04,build-2026-03
```

Tune how long one worker heartbeat snapshot stays visible in the database-backed fleet view:

```env
DW_V2_COMPATIBILITY_HEARTBEAT_TTL=30
```

Optionally scope that fleet view to one app or deployment namespace when several apps share the same workflow database:

```env
DW_V2_COMPATIBILITY_NAMESPACE=sample-app
```

The published `workflows.php` config maps those values here:

```php
'v2' => [
    'compatibility' => [
        'current' => env('DW_V2_CURRENT_COMPATIBILITY'),
        'supported' => env('DW_V2_SUPPORTED_COMPATIBILITIES'),
        'namespace' => env('DW_V2_COMPATIBILITY_NAMESPACE'),
        'heartbeat_ttl_seconds' => (int) env('DW_V2_COMPATIBILITY_HEARTBEAT_TTL', 30),
    ],
],
```

If `supported` is omitted, workers default to the single `current` marker. Older task and run-summary rows that were created before task-level compatibility markers existed are backfilled from their run marker during migration and again on the runtime claim or recovery path if needed. Tasks and runs that truly have no marker anywhere still remain claimable by any worker.

The `getVersion()` fallback prefers the run's start-time `workflow_definition_fingerprint` when a replay reaches a newly introduced branch point that does not have a typed `VersionMarkerRecorded` event yet. That lets a same-compatibility run keep the `DEFAULT_VERSION` branch when it clearly started on an older workflow definition. Older runs whose `WorkflowStarted` history predates the fingerprint snapshot still fall back to the start-time compatibility marker and occupied-sequence checks, so you should keep rotating `DW_V2_CURRENT_COMPATIBILITY` for deployment waves that introduce new versioned workflow code and temporarily list both the old and new markers in `DW_V2_SUPPORTED_COMPATIBILITIES` while the older worker cohort is draining.

Each queue worker also records a database-backed compatibility heartbeat snapshot during `Looping` and task handling. Waterline and the detail helpers expose both the local-build view (`compatibility_supported`, `compatibility_reason`) and the fleet view (`compatibility_namespace`, `compatibility_supported_in_fleet`, `compatibility_fleet_reason`, `compatibility_fleet`). When `DW_V2_COMPATIBILITY_NAMESPACE` is set, database-backed heartbeat rows must match that namespace, and each database snapshot reports its own `namespace` alongside `worker_id`, queue scope, supported markers, and `source = database`. During a rolling upgrade, that fleet view also reads the older cache heartbeat format from workers that have not restarted onto the new snapshot table yet; those legacy cache rows remain visible as rollout fallback even under a configured namespace, but they surface with `namespace = null` until the older workers restart onto the namespaced snapshot path. In other words, strict namespace isolation only becomes complete after every worker has restarted onto the database-backed heartbeat path. Transport-level recovery such as re-dispatching an overdue task or recreating a missing task no longer depends on the scanning worker being able to execute that task; only the eventual claim step stays compatibility-fenced.

When an open task already exists but neither the current build nor any active worker heartbeat snapshot advertises its marker, the run stays visible as waiting for a compatible worker instead of surfacing a false `repair_needed` state on that build.

## History Budgets

Waterline uses the run-summary projection to report how large a selected run's typed history has become. These thresholds control when the projection flips `continue_as_new_recommended`:

```env
DW_V2_CONTINUE_AS_NEW_EVENT_THRESHOLD=10000
DW_V2_CONTINUE_AS_NEW_SIZE_BYTES_THRESHOLD=5242880
```

The published `workflows.php` config maps those values here:

```php
'v2' => [
    'history_budget' => [
        'continue_as_new_event_threshold' => (int) env('DW_V2_CONTINUE_AS_NEW_EVENT_THRESHOLD', 10000),
        'continue_as_new_size_bytes_threshold' => (int) env('DW_V2_CONTINUE_AS_NEW_SIZE_BYTES_THRESHOLD', 5242880),
    ],
],
```

Set either threshold to `0` to disable that side of the recommendation. The flag is advisory; use it to plan `continueAsNew()` boundaries before replay cost grows without changing the selected run's runtime behavior.

## Update Wait Policy

Completion-waiting update APIs such as `attemptUpdate()`, the webhook update routes, and Waterline's update controls wait only up to a bounded budget before they fall back to the still-accepted update lifecycle. Configure that default budget here:

```env
DW_V2_UPDATE_WAIT_COMPLETION_TIMEOUT_SECONDS=10
DW_V2_UPDATE_WAIT_POLL_INTERVAL_MS=50
```

The published `workflows.php` config maps those values here:

```php
'v2' => [
    'update_wait' => [
        'completion_timeout_seconds' => (int) env('DW_V2_UPDATE_WAIT_COMPLETION_TIMEOUT_SECONDS', 10),
        'poll_interval_milliseconds' => (int) env('DW_V2_UPDATE_WAIT_POLL_INTERVAL_MS', 50),
    ],
],
```

`completion_timeout_seconds` controls how long `attemptUpdate*` and HTTP completion waits try to get a worker-applied result before they return an accepted lifecycle instead of blocking indefinitely. `poll_interval_milliseconds` only tunes how often the caller checks that durable update row while waiting; it does not change worker execution order or replay behavior. Waterline's operator metrics expose the active `update_wait` values next to the repair policy so operators can see the effective default without opening config files.

## History Export Redaction

History exports include stored workflow, command, activity, update, task, and failure data by design, because replay-debug and archive handoff need durable facts. If those artifacts can leave a protected environment, configure a redactor before exposing the export endpoint or CLI output broadly.

Create a redactor that implements `Workflow\V2\Contracts\HistoryExportRedactor`:

```php
namespace App\Support;

use Workflow\V2\Contracts\HistoryExportRedactor;

final class WorkflowHistoryExportRedactor implements HistoryExportRedactor
{
    public function redact(mixed $value, array $context): mixed
    {
        return [
            'redacted' => true,
            'path' => $context['path'],
        ];
    }
}
```

Then register it in `config/workflows.php`:

```php
'v2' => [
    'history_export' => [
        'redactor' => App\Support\WorkflowHistoryExportRedactor::class,
        'signing_key' => env('DW_V2_HISTORY_EXPORT_SIGNING_KEY'),
        'signing_key_id' => env('DW_V2_HISTORY_EXPORT_SIGNING_KEY_ID'),
    ],
],
```

The redactor receives the current value plus context such as `path`, `category`, `workflow_instance_id`, `workflow_run_id`, and `workflow_type`. The export calls it for workflow argument/output payloads, history-event payloads, command payload/context, update payloads, task payloads, activity payloads, and failure message/file/trace diagnostics. The resulting bundle includes `redaction.applied`, `redaction.policy`, and `redaction.paths` so downstream tooling can tell which policy shaped the artifact.

Every export also includes an `integrity` block computed after redaction. The checksum uses `canonicalization = json-recursive-ksort-v1` and `checksum_algorithm = sha256`; when `signing_key` is configured, the same canonical payload is signed with `signature_algorithm = hmac-sha256` and the optional `signing_key_id` is reported as `key_id`. Keep the signing key outside the exported artifact and rotate the key id when downstream verifiers need to distinguish keys.

<!-- Source: docs/configuration/options.md -->

# Options

There are various options available when defining your workflows and activities. These options include the number of times a workflow or activity may be attempted before it fails, the connection and queue, and the maximum number of seconds it is allowed to run.

```php
use Workflow\V2\Activity;

class MyActivity extends Activity
{
    public ?string $connection = 'default';
    public ?string $queue = 'default';

    public int $tries = 3;

    public function backoff(): array
    {
        return [1, 2, 5, 10, 15, 30, 60, 120];
    }
}
```

The `$connection` and `$queue` properties on `Workflow\V2\Workflow` and `Workflow\V2\Activity` are declared as `public ?string` and default to `null`. Subclass overrides must keep the nullable type so PHP's invariant public-property typing rules accept the redeclaration. Use `null` when you want to inherit the application's default connection or queue instead of hard-coding a value.

Activity timeouts are not configured through a class property. Use [`ActivityOptions`](#activityoptions) per-call (for example `startToCloseTimeout`) or the activity retry policy snapshot taken at schedule time. See also [Task Repair Policy](/docs/polyglot/server-config-reference#workflow-package-controls) for worker-loop timing settings.

## StartOptions

`Workflow\V2\StartOptions` carries visibility, deduplication, and execution timeout configuration at workflow start time. It does not select a queue — queue routing is driven by the workflow and activity class `$connection` and `$queue` properties plus per-call `ActivityOptions`.

```php
use Workflow\V2\StartOptions;
use Workflow\V2\WorkflowStub;
use Workflow\V2\Enums\DuplicateStartPolicy;

$workflow = WorkflowStub::make(MyWorkflow::class);

$workflow->start(
    'arg1',
    new StartOptions(
        duplicateStartPolicy: DuplicateStartPolicy::ReturnExistingActive,
        businessKey: 'order-12345',
        labels: ['tenant' => 'acme'],
        executionTimeoutSeconds: 3600,
    ),
);
```

`StartOptions` are consumed by the workflow engine and are not passed as arguments to your workflow `handle()` method. They are persisted with the workflow and used for subsequent workflow/activity dispatching (including replay and continue-as-new behavior).

## ActivityOptions

`Workflow\V2\Support\ActivityOptions` provides per-call overrides for routing, retries, and timeouts when invoking an activity, without requiring changes to the activity class itself:

```php
use function Workflow\V2\activity;
use Workflow\V2\Support\ActivityOptions;

$result = activity(
    ChargeCard::class,
    new ActivityOptions(
        connection: 'redis',
        queue: 'critical',
        maxAttempts: 5,
        startToCloseTimeout: 30,
    ),
    $orderId,
);
```

## Connection

The `$connection` setting is used to specify which queue connection the workflow or activity should be sent to. By default, the `$connection` value is not set which will use the default connection. This can be overridden by setting the `$connection` property on the workflow or activity class.

## Queue

The `$queue` setting is used to specify which queue the workflow or activity should be added to. By default, the `$queue` value is not set which uses the default queue for the specified connection. This can be overridden by setting the `$queue` property on the workflow or activity class.

## Retries

The `$tries` setting is used to control the total number of attempts for an activity before it is considered failed. By default, `$tries` is `1` (a single attempt, no automatic retries). Set `$tries` to a value greater than `1` to allow retries, or set it to `0` to retry forever. This can be overridden per call through `ActivityOptions::$maxAttempts`.

## Timeout

The v2 `Activity` base class has no `$timeout` class property. Configure activity timeouts per call through [`ActivityOptions`](#activityoptions) using `startToCloseTimeout`, `scheduleToStartTimeout`, `scheduleToCloseTimeout`, or `heartbeatTimeout`. The runtime snapshots the resulting retry policy onto the activity execution when it is scheduled, so the timeout is stable for an already scheduled attempt even if a later deploy changes the activity class or options. Worker-loop level dispatch timing is controlled through [Task Repair Policy](/docs/polyglot/server-config-reference#workflow-package-controls).

## Backoff

The `backoff` method returns an array of integers corresponding to the current attempt. The default `backoff` method decays exponentially to 2 minutes. This can be overridden by implementing the `backoff` method on the activity class.

## Namespace

Workflows can be scoped to a namespace for multi-namespace isolation. When a namespace is configured, it is persisted on every workflow instance, run, task, and run-summary projection created through the control plane. Task bridge polling and Waterline visibility filters can then restrict results to a single namespace.

Namespace names must contain only lowercase alphanumeric characters, dots, underscores, and hyphens (matching `[a-z0-9._-]+`, max 128 characters). Mixed-case input is normalized to lowercase automatically.

Set the default namespace via environment variable:

```env
DW_V2_NAMESPACE=production
```

Or in `config/workflows.php`:

```php
'v2' => [
    'namespace' => env('DW_V2_NAMESPACE'),
    // ...
],
```

The control plane also accepts a per-call namespace override in the `start()` options:

```php
$controlPlane->start('order-processing', 'order-12345', [
    'namespace' => 'staging',
    // ...
]);
```

When no namespace is configured and none is passed explicitly, instances have a `null` namespace and are visible to all consumers.

### Waterline namespace scoping

When Waterline is deployed against a shared database with multiple namespaces, set `WATERLINE_NAMESPACE` to restrict all list views to one namespace:

```env
WATERLINE_NAMESPACE=production
```

This injects a namespace filter into every visibility query so Waterline only shows workflows belonging to the configured namespace. When set, Waterline also scopes all command operations (cancel, signal, terminate, update, repair, archive, and queries) to the configured namespace — a command targeting an instance or run that belongs to a different namespace will return a 404 instead of executing.

### Command namespace scoping

`WorkflowStub::load()`, `loadSelection()`, and `loadRun()` accept an optional `namespace` parameter:

```php
use Workflow\V2\WorkflowStub;

// Load only if the instance belongs to the given namespace
$stub = WorkflowStub::load('order-12345', namespace: 'production');

// Load a specific run, scoped to namespace
$stub = WorkflowStub::loadRun($runId, namespace: 'production');

// Load a specific selection, scoped to namespace
$stub = WorkflowStub::loadSelection('order-12345', $runId, namespace: 'production');
```

When `namespace` is `null` (the default), loading is unscoped and works against all namespaces — this preserves backward compatibility. When a namespace is provided, the query filters by namespace at the database level and throws `ModelNotFoundException` if the workflow does not exist in that namespace.

The control plane command methods (`signal`, `cancel`, `terminate`, `update`, `repair`, `archive`) also accept `namespace` in their options array:

```php
$controlPlane->cancel('order-12345', [
    'namespace' => 'production',
]);
```

### Task bridge namespace filtering

Both the workflow and activity task bridges accept an optional `namespace` parameter on `poll()`:

```php
$tasks = $bridge->poll('redis', 'default', limit: 10, namespace: 'production');
```

When omitted, `poll()` returns tasks from all namespaces (backward-compatible with pre-namespace installations).

## Durable Type Aliases

Durable type keys for workflows and activities are stored when you register them under `workflows.v2.types`. Failure payloads can use the same pattern for exception classes:

```php
'v2' => [
    'types' => [
        'workflows' => [
            'billing.invoice-sync' => App\Workflows\InvoiceSyncWorkflow::class,
        ],
        'activities' => [
            'payments.capture' => App\Activities\CapturePaymentActivity::class,
        ],
        'exceptions' => [
            'billing.invoice-declined' => App\Exceptions\InvoiceDeclined::class,
        ],
        'exception_class_aliases' => [
            App\Exceptions\LegacyInvoiceDeclined::class => App\Exceptions\InvoiceDeclined::class,
        ],
    ],
],
```

When an activity, update, child, or workflow failure is recorded with an exception alias, the engine stores that alias in typed history as `exception_type` and inside the failure payload as `type`. Replay resolves the alias before falling back to the recorded PHP class, so a later class move can keep workflow `catch` semantics stable as long as the alias still points at the current throwable class.

For imported v1 failures that were recorded before an exception alias existed, `workflows.v2.types.exception_class_aliases` can map the recorded legacy exception FQCN to the current throwable class. Durable `exceptions` type aliases still win first. The class-alias map is only a refactor bridge for already-recorded payloads with no durable `type`; new workflows should use durable exception type aliases so history is independent from PHP class names.

Final v2 writes durable exception aliases when the failure is recorded, so configure stable aliases before recording failures whose throwable classes may move later.

If a replayed failure cannot be resolved through the durable `exceptions` map, the class-alias map, or the recorded class, the engine does not fall back to a generic runtime exception inside workflow code. Query replay raises `UnresolvedWorkflowFailureException`, Waterline marks the failure with `exception_replay_blocked = true`, and a worker task that hits the same gap is left failed while the run stays open. Fix the mapping and repair the run rather than relying on broad `catch (RuntimeException)` blocks to handle renamed historical failures.

<!-- Source: docs/configuration/ensuring-same-server.md -->

# Worker Placement and Affinity

A queue name is a routing boundary, not a physical-host affinity guarantee. Setting
the `$queue` property on a workflow or activity restricts which workers may claim
the task, but any compatible worker polling that queue can receive it. Do not pass
data between ordinary activities through a worker's local filesystem unless the
data is also stored in a durable shared location.

## Route Work to a Dedicated Worker Pool

Use a dedicated queue when a class of work needs a particular worker pool, such
as workers with a mounted shared volume or a GPU capability:

```php
use Workflow\V2\Activity;

final class RenderVideo extends Activity
{
    public ?string $queue = 'gpu-activities';
}
```

Start only the intended Laravel workers on that queue:

```bash
php artisan queue:work --queue=gpu-activities
```

Laravel Horizon can supervise the same queue. In either case, the guarantee is
"one of the workers assigned to this queue," not "the same process or host that
ran the previous step." Put handoff data in durable workflow payloads, external
payload storage, or another shared store available to every eligible worker.

## Choose the Affinity Contract You Need

- Use an [ordinary queued activity](/docs/defining-workflows/activities)
  when the work can run on any compatible worker in its queue.
- Use a [local activity](/docs/features/local-activities) when one short,
  retryable activity must execute inside the workflow worker process that is
  currently handling the workflow task.
- Use a [worker session](/docs/features/worker-sessions) when several durable
  activity steps must reuse one worker-session lease for process-local state,
  GPU memory, or a worker-local mounted filesystem.

Worker sessions still apply queue routing first. They add an explicit lease and
capability contract after routing, including expiry and optional reacquisition;
they do not make an individual machine immortal. If state must survive worker
loss or session reacquisition, keep it in durable shared storage.

See [Activity Execution Model](/docs/features/activity-execution-model) for
the placement comparison and [Options](/docs/configuration/options#queue)
for class-level and per-call queue configuration.

<!-- Source: docs/configuration/customization-matrix.md -->

# Customization Matrix

This page freezes the supported Durable Workflow v2 customization contract.
Use it when you need to move v2 durable models onto a different connection,
swap model subclasses, replace the PHP operator-observability repository, or
decide whether an older serializer setting is still valid during migration.

## Supported override matrix

| Surface | Supported contract | Notes |
| --- | --- | --- |
| `workflows.v2.instance_model` with a schema-compatible subclass that keeps the basename `WorkflowInstance` | Supported | Use this shape when you only need a different connection, casts, or app namespace. The inherited relations keep the package's `workflow_instance_id` foreign-key contract. |
| `workflows.v2.instance_model` with a table-swapped or basename-changing subclass | Supported with explicit relation overrides | Override `runs()`, `commands()`, and `updates()` so they keep `workflow_instance_id`. The package now validates this at boot and rejects inherited relation inference that would change those keys. `currentRun()` already pins `current_run_id` explicitly and does not need an override. |
| Other `workflows.v2.*_model` overrides (`run_model`, `task_model`, `history_event_model`, projections, schedules, activity rows, failures, messages, memos, search attributes, child calls) | Supported for schema-compatible subclasses | Keep the package column names, primary keys, and foreign keys. These relations already pin the keys they use, so table-swapped subclasses can stay on the inherited relation methods when the schema stays compatible. |
| Custom table names plus custom foreign-key column names | Unsupported | The v2 runtime and operator surfaces assume the package column and key names. If you change them, you are outside the supported contract. |
| `OperatorObservabilityRepository` replacement | Supported by container binding | Bind your implementation to `Workflow\V2\Contracts\OperatorObservabilityRepository`. This is a PHP runtime integration contract for Waterline and `WorkflowStub::historyExport()`, not a stable cross-language API. The repository receives package model objects. |
| `serializer` | Supported only for `avro` and legacy drain/import codecs | New v2 runs always use Avro semantics. `workflow-serializer-y` and `workflow-serializer-base64` remain valid only for finishing or importing v1 history that still needs PHP-native decoding. Custom serializer classes from v1 are unsupported in v2. |
| Health and doctor diagnostics | Supported | `php artisan workflow:v2:doctor` and Waterline v2 health/stats reflect the configured model classes and flag stale serializer settings as migration debt. |
| Package migrations on a non-default connection | Supported with published migrations | Auto-loaded package migrations run on Laravel's default connection. If your durable models use another connection, publish the migrations and set the migration `$connection` explicitly. |
| Core write-side runtime (`WorkflowStub::make()`, `load()`, task creation, current-run resolution) | Supported through the configured durable models | The runtime honors the configured `instance_model`, `run_model`, and `task_model`. Keep write-side tables schema-compatible with the package columns and keys. |

## Exact v2 model keys

The published `workflows.php` config exposes these durable model keys under
`workflows.v2`:

- `instance_model`
- `run_model`
- `history_event_model`
- `task_model`
- `command_model`
- `link_model`
- `activity_execution_model`
- `activity_attempt_model`
- `timer_model`
- `failure_model`
- `run_summary_model`
- `run_wait_model`
- `run_timeline_entry_model`
- `run_timer_entry_model`
- `run_lineage_entry_model`
- `schedule_model`
- `schedule_history_event_model`

Use subclasses of the package models for those keys. Keep the package's column
names and foreign keys unless a page in this docs set says otherwise.

## Instance-model override rule

The only current v2 override that needs extra relation work is
`workflows.v2.instance_model` when your subclass changes the inferred foreign
key away from `workflow_instance_id`.

Typical safe example:

```php
namespace App\Models\V2;

use Workflow\V2\Models\WorkflowInstance as BaseWorkflowInstance;

final class WorkflowInstance extends BaseWorkflowInstance
{
    protected $connection = 'workflow';
}
```

Because the subclass basename is still `WorkflowInstance`, Eloquent keeps the
same inferred foreign key and the inherited relations remain aligned.

If you use a different basename or a table-swapped storage model, override the
affected relations explicitly:

```php
namespace App\Workflow\Storage;

use Illuminate\Database\Eloquent\Relations\HasMany;
use Workflow\V2\Models\WorkflowCommand;
use Workflow\V2\Models\WorkflowInstance as BaseWorkflowInstance;
use Workflow\V2\Models\WorkflowRun;
use Workflow\V2\Models\WorkflowUpdate;
use Workflow\V2\Support\ConfiguredV2Models;

final class TenantWorkflowInstance extends BaseWorkflowInstance
{
    protected $table = 'tenant_workflow_instances';

    public function runs(): HasMany
    {
        return $this->hasMany(
            ConfiguredV2Models::resolve('run_model', WorkflowRun::class),
            'workflow_instance_id',
        );
    }

    public function commands(): HasMany
    {
        return $this->hasMany(
            ConfiguredV2Models::resolve('command_model', WorkflowCommand::class),
            'workflow_instance_id',
        )->oldest('created_at');
    }

    public function updates(): HasMany
    {
        return $this->hasMany(
            ConfiguredV2Models::resolve('update_model', WorkflowUpdate::class),
            'workflow_instance_id',
        )
            ->orderBy('command_sequence')
            ->oldest('accepted_at')
            ->oldest('created_at')
            ->oldest('id');
    }
}
```

If those overrides are missing, package boot now fails fast instead of letting
the app drift onto inferred keys such as
`tenant_workflow_instance_id` or `custom_workflow_instance_id`.

## Serializer, import, and health rules

- Keep `serializer = 'avro'` for new v2 work.
- Use legacy codecs only while draining or importing v1 history that still
  needs PHP-native payload decoding.
- Do not point v2 at removed custom serializer classes from v1.
- Run `php artisan workflow:v2:doctor` after upgrades and config changes. It is
  the diagnostic surface that flags legacy serializer settings and other boot
  readiness problems.
- Treat Waterline v2 health and stats as the operator-facing view of the same
  durable model configuration.

## Migration and write-side rules

- The normal path is auto-loaded package migrations on the default connection.
- If your durable models use another connection, publish the migrations and set
  the migration `$connection` so `php artisan migrate` targets the same
  database as your configured model subclasses.
- Keep the package table names, primary keys, and foreign keys on every model
  that participates in the runtime write path.
- The write path is not read-only customization: starts, loads, current-run
  resolution, and task execution all use the configured durable models.

## Related Guides

- [Publishing Config](./publishing-config.md)
- [Database Connection](./database-connection.md)
- [Migration Guide](../migration.md)
- [Monitoring](../monitoring.md)

<!-- Source: docs/configuration/database-connection.md -->

# Database Connection

Here is an overview of the steps needed to customize the database connection used for the workflow v2 durable models. This is *only* required if you want to use a different database connection than the default connection you specified for your Laravel application.

For the full v2 override contract, including table-swapped subclasses,
serializer rules, repository replacement, health diagnostics, and migration
expectations, see the [Customization Matrix](./customization-matrix.md).

1. Create classes in your app models directory that extend the base v2 model classes
2. Set the desired `$connection` option in each class
3. Publish the workflow config file
4. Update the `workflows.v2` model bindings to point at your custom classes

## Extending V2 Workflow Models

In `app\Models\V2\WorkflowInstance.php` put this.

```php
namespace App\Models\V2;

use Workflow\V2\Models\WorkflowInstance as BaseWorkflowInstance;

class WorkflowInstance extends BaseWorkflowInstance
{
    protected $connection = 'mysql';
}
```

Repeat the pattern for each durable v2 model you need to re-route — at minimum:

- `Workflow\V2\Models\WorkflowRun`
- `Workflow\V2\Models\WorkflowHistoryEvent`
- `Workflow\V2\Models\WorkflowTask`
- `Workflow\V2\Models\WorkflowCommand`
- `Workflow\V2\Models\WorkflowLink`
- `Workflow\V2\Models\ActivityExecution`
- `Workflow\V2\Models\ActivityAttempt`
- `Workflow\V2\Models\WorkflowTimer`
- `Workflow\V2\Models\WorkflowFailure`
- `Workflow\V2\Models\WorkflowRunSummary`
- `Workflow\V2\Models\WorkflowSchedule`
- `Workflow\V2\Models\WorkflowScheduleHistoryEvent`

Each subclass should declare `protected $connection = 'mysql';` (or whichever connection name you defined in `config/database.php`).

## Registering Custom Models

Publish the workflow config file and update the `workflows.v2.*_model` bindings to point at your custom classes:

```php
// config/workflows.php

'v2' => [
    'instance_model' => App\Models\V2\WorkflowInstance::class,
    'run_model' => App\Models\V2\WorkflowRun::class,
    'history_event_model' => App\Models\V2\WorkflowHistoryEvent::class,
    'task_model' => App\Models\V2\WorkflowTask::class,
    'command_model' => App\Models\V2\WorkflowCommand::class,
    // ...
],
```

The package resolves models through `Workflow\V2\Support\ConfiguredV2Models`, so any relation or query that hits a v2 model will transparently pick up the configured connection.

If you move beyond schema-compatible subclasses and change the inferred foreign
key of your custom `instance_model`, the package now validates that the
`runs()`, `commands()`, and `updates()` relations were overridden explicitly at
boot.

## Migrations

Workflow migrations are auto-loaded from the package (`WorkflowServiceProvider::loadMigrationsFrom`) and run against the default database connection. When you route the v2 tables at a non-default connection, publish the migrations with `php artisan vendor:publish --tag=migrations` and set `protected $connection = 'mysql';` on each published migration class so `php artisan migrate` runs them against the correct database.

<!-- Source: docs/configuration/microservices.md -->

# Microservices

Workflows can span multiple Laravel applications. One app defines workflows, another defines activities. Both share a database and queue, each running its own `queue:work` process.

## Shared Database and Queue

Point both apps at the same database and Redis queue:

```php
// config/database.php — add to both apps

'connections' => [
    'shared' => [
        'driver' => 'mysql',
        'host' => env('SHARED_DB_HOST', '127.0.0.1'),
        'port' => env('SHARED_DB_PORT', '3306'),
        'database' => env('SHARED_DB_DATABASE', 'workflows'),
        'username' => env('SHARED_DB_USERNAME', 'root'),
        'password' => env('SHARED_DB_PASSWORD', ''),
        'charset' => 'utf8mb4',
        'collation' => 'utf8mb4_unicode_ci',
        'prefix' => '',
    ],
],
```

```php
// config/queue.php — add to both apps

'connections' => [
    'shared' => [
        'driver' => 'redis',
        'connection' => env('SHARED_REDIS_QUEUE_CONNECTION', 'default'),
        'queue' => env('SHARED_REDIS_QUEUE', 'default'),
        'retry_after' => 90,
        'block_for' => null,
        'after_commit' => false,
    ],
],
```

Run migrations from one app only:

```bash
php artisan migrate
```

If the apps use a different database connection than `default`, see [Database Connection](./database-connection.md) for how to point the models at the shared connection.

## Defining Workflows and Activities

Register type keys so the engine can route tasks by name:

```php
// App A (workflow service) — config/workflows.php

'v2' => [
    'types' => [
        'workflows' => [
            'order-processing' => App\Workflows\OrderWorkflow::class,
        ],
        'activities' => [],
    ],
],
```

```php
// App B (activity service) — config/workflows.php

'v2' => [
    'types' => [
        'workflows' => [],
        'activities' => [
            'charge-payment' => App\Activities\ChargePaymentActivity::class,
        ],
    ],
],
```

The workflow schedules the activity by type key:

```php
// App A — app/Workflows/OrderWorkflow.php

use function Workflow\V2\activity;
use Workflow\V2\Workflow;

class OrderWorkflow extends Workflow
{
    public ?string $connection = 'shared';
    public ?string $queue = 'workflows';

    public function handle(int $orderId): array
    {
        $charge = activity('charge-payment', $orderId);
        return ['order' => $orderId, 'charge' => $charge];
    }
}
```

```php
// App B — app/Activities/ChargePaymentActivity.php

use Workflow\V2\Activity;

class ChargePaymentActivity extends Activity
{
    public ?string $connection = 'shared';
    public ?string $queue = 'activities';

    public function handle(int $orderId): string
    {
        return "charged-{$orderId}";
    }
}
```

## Running Workers

Each app runs a queue worker on its own queue:

```bash
# App A
php artisan queue:work shared --queue=workflows

# App B
php artisan queue:work shared --queue=activities
```

App A's worker replays workflows and schedules activity tasks. App B's worker picks up those activity tasks, executes them, and returns results. The engine handles the handoff through the shared database.

<!-- Source: docs/configuration/pruning-workflows.md -->

# Pruning Workflows

Workflow v2 separates two distinct lifecycle operations:

1. **Archive** — marks a terminal run as archived so it is excluded from active fleet views. Archiving does **not** delete any rows.
2. **Prune** — removes stale projection or durable rows from the database. Pruning should only run against archived runs.

## Archiving terminal runs

Archive a completed, failed, cancelled, or terminated run through the control plane:

```php
use Workflow\V2\Contracts\WorkflowControlPlane;

$controlPlane = app(WorkflowControlPlane::class);

$result = $controlPlane->archive('order-12345', [
    'reason' => 'Retention period expired',
]);

if ($result['accepted']) {
    // Run's archived_at is now set; it disappears from active fleet views.
}
```

The same operation is available through the control-plane HTTP route. Only closed runs may be archived; archiving an already-archived run returns `accepted` with an `archive_not_needed` outcome.

### Automating archival

Archive terminal runs on a schedule by querying the durable run table for finished runs past your retention window and calling `archive()` for each. For a simple time-based rule, add something like this to `routes/console.php`:

```php
use Illuminate\Support\Facades\Schedule;
use Workflow\V2\Models\WorkflowRun;
use Workflow\V2\Contracts\WorkflowControlPlane;

Schedule::call(function (WorkflowControlPlane $controlPlane): void {
    WorkflowRun::query()
        ->whereIn('status', ['completed', 'failed', 'cancelled', 'terminated'])
        ->whereNull('archived_at')
        ->where('closed_at', '<=', now()->subMonth())
        ->with('instance:id')
        ->chunkById(100, function ($runs) use ($controlPlane): void {
            foreach ($runs as $run) {
                $controlPlane->archive($run->instance->id, [
                    'reason' => 'retention_policy',
                ]);
            }
        });
})->daily();
```

## Pruning projection rows

Run summary, wait, timeline, timer, and lineage projection rows can be rebuilt from durable history and commands. Use `workflow:v2:rebuild-projections --prune-stale` to delete projection rows whose durable run no longer exists:

```bash
# Preview what would be pruned
php artisan workflow:v2:rebuild-projections --prune-stale --dry-run

# Actually prune stale projection rows
php artisan workflow:v2:rebuild-projections --prune-stale
```

## Pruning durable rows

Durable rows (`workflow_instances`, `workflow_runs`, `workflow_history_events`, `workflow_tasks`, `activity_executions`, `activity_attempts`, `workflow_failures`, etc.) should only be deleted once the run has been archived and retention is definitely over. Because these rows are referenced by typed history and lineage, pruning must be done in dependency order. The package does not yet ship an end-to-end durable prune command — if you need to reclaim disk space, do it with a scoped cleanup job that:

1. Targets runs where `archived_at IS NOT NULL AND archived_at <= now()->subMonths(3)`.
2. Deletes dependent rows first (history events, tasks, activity attempts, activity executions, failures, timers, links, commands).
3. Deletes the `workflow_runs` row, then the `workflow_instances` row once all runs are removed.
4. Rebuilds projections with `workflow:v2:rebuild-projections --prune-stale` to clean up any orphaned projection rows.

Track the archival retention window and durable retention window separately. Archived-but-not-pruned runs are still available for history export and incident review.

<!-- Source: docs/sample-app.md -->

# Sample App

https://github.com/durable-workflow/sample-app

This is the embedded Laravel gallery: a sample Laravel 13 application built on
the Durable Workflow 2.0 release-candidate line, with workflows you can run in
a GitHub Codespace. It is the right starting point when your deployment model
is **embedded Laravel** and you want to inspect Laravel queue execution and
Waterline together.

Choose the examples that match your runtime:

| Deployment model | Example path |
| --- | --- |
| Durable Workflow Cloud | Start with [Cloud Managed Runtime](/docs/polyglot/cloud-control-plane/), then choose the PHP, Python, or Rust SDK guide using the provisioned namespace credentials. Cloud users do not run Server. |
| Self-hosted service mode | Use the [Durable Workflow 2.0 Quickstart](/docs/quickstart/) for complete PHP, Python, and Rust published-artifact examples against a local Server. |
| Embedded Laravel | Continue with this gallery and [Embedded Installation](/docs/installation/). |

The gallery is end-to-end evidence for the Laravel-hosted engine, not a
universal starter application for service-mode or polyglot deployments. It is
the first place embedded features get realistic Laravel coverage and the source
this site mirrors when it shows a Laravel-native pattern. If you have a
durable-workflow pattern to share, the
[Contribute a Sample](/docs/contribute-a-sample) guide walks
through the submission flow.

> Looking for the Laravel 12 / Durable Workflow 1.x version? It's preserved on the [`Laravel-12` branch](https://github.com/durable-workflow/sample-app/tree/Laravel-12). Older blog posts and tutorials that reference v1 patterns (e.g. `Workflow\Workflow`, `yield activity(...)`, `Workflow\Activity`) target that branch.

## Sample gallery

Each entry in the gallery names the pattern surface the sample teaches,
the workflow class that exercises it, the artisan command that runs it,
and the Waterline screen that proves the run committed. The gallery
mirrors the README "Sample Index" in the sample-app repository — when a
sample lands or moves, the README and this gallery move together.

| Pattern | Workflow class | Command | Waterline screen |
|---------|----------------|---------|------------------|
| Smallest deterministic v2 workflow | `App\Workflows\Simple\SimpleWorkflow` | `php artisan app:workflow` | Run list → run detail with two activity events and a `WorkflowExecutionCompleted` event |
| Durable elapsed-time measurement without replay drift | `App\Workflows\Elapsed\ElapsedTimeWorkflow` | `php artisan app:elapsed` | Run detail showing two `MarkerRecorded` events for `sideEffect` clock reads bracketing a `TimerFired` event |
| Coordination across Laravel app boundaries | `App\Workflows\Microservice\MicroserviceWorkflow` | `php artisan app:microservice` | Run detail showing activity events with task-queue routing across the app and microservice workers |
| Browser automation with captured artifacts | `App\Workflows\Playwright\CheckConsoleErrorsWorkflow` | `php artisan app:playwright` | Run detail showing the Playwright activity, the FFmpeg activity, and the cleanup activity in order |
| Webhook-started workflow with a signal wait | `App\Workflows\Webhooks\WebhookWorkflow` | `php artisan app:webhook` | Run detail showing `WorkflowExecutionStarted` from the webhook ingress and a `WorkflowExecutionSignaled` event for `ready` |
| AI activity loop with durable retry/validation | `App\Workflows\Prism\PrismWorkflow` | `php artisan app:prism` | Run detail showing repeated activity attempts and the `ActivityTaskCompleted` that satisfies the validator |
| Signal-driven AI agent with saga compensation | `App\Workflows\Ai\AiWorkflow` | `php artisan app:ai` | Run detail showing a message-stream `MarkerRecorded` reference, an `Update` event, and the compensation activities recorded after a saga failure |

The Waterline screen column names the events you should expect to see
in a healthy run; if your local run is missing one, that gap is the
fastest way to localize a worker-side or environment problem.

## Pattern-page cross-links

The pattern pages on this site link directly into the sample workflow
that exercises each surface. Use this table to jump from a pattern
page to the matching runnable workflow.

| Pattern page | Sample workflow |
|--------------|-----------------|
| [Sagas](/docs/features/sagas) | `App\Workflows\Ai\AiWorkflow` (`php artisan app:ai`) |
| [Signals](/docs/features/signals) | `App\Workflows\Webhooks\WebhookWorkflow` (`php artisan app:webhook`) |
| [Message Streams](/docs/features/message-streams) | `App\Workflows\Ai\AiWorkflow` (`php artisan app:ai`) |
| [Child Workflows](/docs/features/child-workflows) | `App\Workflows\Microservice\MicroserviceWorkflow` (`php artisan app:microservice`) |
| [Side Effects](/docs/features/side-effects) | `App\Workflows\Elapsed\ElapsedTimeWorkflow` (`php artisan app:elapsed`) |
| [Webhooks](/docs/features/webhooks) | `App\Workflows\Webhooks\WebhookWorkflow` (`php artisan app:webhook`) |
| [MCP Workflows](/docs/mcp-workflows) | every gallery entry, exposed through `config/workflow_mcp.php` |

**Step 1**

Create a codespace from the main branch of this repo.

**Step 2**

Once the codespace has been created, wait for the codespace to build. This should take between 5 to 10 minutes.

**Step 3**

Once it is done. You will see the editor and the terminal at the bottom.

**Step 4**

Run composer install.

```bash
composer install
```

**Step 5**

Run the init command to setup the app, install extra dependencies and run the migrations.

```bash
php artisan app:init
```

The sample app's `php artisan migrate` path picks up the workflow and Waterline package migrations directly, so all tables and Waterline saved views are ready after the normal install.

**Step 6**

Start the queue worker. This will enable the processing of workflows and activities.

```bash
php artisan queue:work
```

When you want to prove the repair loop itself without waiting for a busy queue worker to hit another `Looping` cycle, run one explicit recovery sweep from a second terminal:

```bash
php artisan workflow:v2:repair-pass
```

That command uses the same scan and backoff policy as the worker loop. It is useful after fixing a local queue/backend issue, while validating a lost-task scenario in the sample app, or when a low-traffic codespace would otherwise take a while to hit another loop pass. Add `--run-id=...` to limit the sweep to one or more selected runs during an experiment, or `--instance-id=...` when the whole instance should stay in scope.

**Step 7**

Create a new terminal window.

**Step 8**

Start the example workflow inside the new terminal window.

```bash
php artisan app:workflow
```

**Step 9**

You can view the waterline dashboard at https://[your-codespace-name]-80.preview.app.github.dev/waterline/dashboard.

Waterline is the durable-state view. It answers whether a workflow started,
which run is current, which typed history events were committed, which waits
are open, and which operator actions are available. Worker-side telemetry is
separate: poll latency, task duration, exporter setup, custom application
metrics, and worker process errors come from the PHP worker logs or from the
SDK metrics endpoint of an external worker.

| Surface | Answers | Sample check |
| --- | --- | --- |
| Waterline and history export | Durable workflow status, history, retries, waits, signals, updates, failures, and operator actions | Open `/waterline/dashboard`, then export the selected run history |
| Worker logs | PHP queue worker process errors and application log lines | Tail `storage/logs/laravel.log` while `php artisan queue:work` is running |
| SDK metrics | External worker/client request counts, poll latency, and task duration | Scrape the SDK worker's Prometheus/OpenMetrics endpoint |

For a Python worker, install the Prometheus extra and expose the worker metrics
from that worker process:

```bash
pip install 'durable-workflow[prometheus]'
```

```python
from prometheus_client import start_http_server

from durable_workflow import Client, PrometheusMetrics, Worker

metrics = PrometheusMetrics()
start_http_server(9102)

async with Client("http://localhost:8080", token="secret", metrics=metrics) as client:
    worker = Worker(
        client,
        task_queue="default",
        workflows=[GreeterWorkflow],
        activities=[greet],
        metrics=metrics,
    )
    await worker.run()
```

Replace `GreeterWorkflow` and `greet` with the workflow and activity handlers
registered by that worker.

Scrape `:9102/metrics` for `durable_workflow_worker_*` and
`durable_workflow_client_*` series. Those metrics explain worker runtime
performance; Waterline remains the source of truth for the committed workflow
history.

Waterline's detail screen includes an "Export History" action for the selected run. When the detail screen is showing the instance's current run, that button uses the instance-scoped current-run export route; historical run detail keeps using the explicit `/runs/{runId}/history-export` path. You can also export the same replay/debug bundle from the sample app terminal:

```bash
php artisan workflow:v2:history-export {workflow-instance-id} --run-id={workflow-run-id} --output=storage/app/workflow-history/example.json --pretty
```

The export includes a SHA-256 integrity checksum. Set `DW_V2_HISTORY_EXPORT_SIGNING_KEY` and `DW_V2_HISTORY_EXPORT_SIGNING_KEY_ID` in the app environment when another system needs to verify the exported bundle with an HMAC signature.

The exported `selected_run` block includes `waits_projection_source`, `timeline_projection_source`, `timers_projection_source`, and `lineage_projection_source`. The exported `links` block also includes `projection_source`, and the `links.parents` / `links.children` sections come from the selected run's typed lineage history first, so child-workflow and continue-as-new relationships remain visible in the bundle even if mutable link rows have drifted during a local experiment. When a lineage row is surviving only through older mutable compatibility data, the bundle now marks that entry with `history_authority = mutable_open_fallback` and `diagnostic_only = true` instead of silently rehydrating extra link metadata during export.

## AI Workflow Message Streams

Repeated AI or human-input workflows should use the first-class v2 message
stream facade as their authoring pattern:

```php
$reply = $this->inbox('ai.assistant')->receiveOne();

$this->outbox('ai.assistant')->sendReference(
    targetInstanceId: $this->workflowId(),
    payloadReference: $storedReplyReference,
    correlationId: $requestId,
);
```

Keep app-owned payload storage for large request/response bodies, then pass
the stored reference through the stream. Do not teach new sample workflows to
write `workflow_messages`, `MessageStreamCursor`, or `MessageService` calls
directly. See [Message Streams](/docs/features/message-streams) for the
stable v2 inbox/outbox contract.

**Step 10**

Run the workflow and activity tests.

```bash
vendor/bin/phpunit
```

That's it! You can now create and test workflows.

## AI Client MCP Server

The sample app also exposes a Laravel MCP server at `/mcp/workflows`. This is the reference AI-client surface for Durable Workflow v2: it gives agents structured workflow discovery, start, status, output, recent typed history, and failure facts without requiring them to scrape Waterline.

For the detailed MCP endpoint and tool contract, see
[MCP Workflow Surface](/docs/mcp-workflows). For the broader AI-assisted
development contract, including v2 LLM manifests, CLI exit codes, Waterline
exports, and SDK references, see
[AI-assisted development](/docs/ai-assisted-development).

The MCP server is registered from `routes/ai.php` by the Laravel MCP package. The exposed workflow keys live in `config/workflow_mcp.php`; each entry can include the workflow class plus discovery metadata such as a description, credential requirements, and expected arguments.

Default tools:

| Tool | Purpose |
| --- | --- |
| `list_workflows` | Lists configured workflow keys, credential requirements, v2 status values, and optionally recent runs. |
| `start_workflow` | Starts a configured v2 workflow and returns `workflow_id`, `run_id`, status, business key, and command outcome. |
| `get_workflow_result` | Polls the current or selected run and returns status, output, visibility metadata, and latest failure summary. |
| `get_workflow_history` | Returns a bounded tail of typed v2 history events and latest durable failures for debugging. |

A typical agent loop is:

```json
{"tool": "list_workflows", "arguments": {"show_recent": true, "limit": 5}}
{"tool": "start_workflow", "arguments": {"workflow": "simple", "business_key": "demo-001"}}
{"tool": "get_workflow_result", "arguments": {"workflow_id": "<workflow_id>"}}
{"tool": "get_workflow_history", "arguments": {"run_id": "<run_id>", "limit": 25}}
```

Use `simple` or `elapsed` for no-credential smoke tests. The `prism` workflow is intentionally exposed as an AI example, but it requires `OPENAI_API_KEY` before a worker can complete it.

<!-- Source: docs/agent-operating-loop.md -->

# Agent Operating Loop

Durable Workflow v2 is easiest for agents to use when every step has a stable
handle. The loop below starts with the same invariant a human learns, then
switches to machine-readable contracts for discovery, execution, diagnosis, and
repair.

## 1. Use The Right Docs Line

For stable 2.0 work, use the canonical bundle. It tracks the same unversioned
default Docs path human readers reach from the public site:

```text
https://durable-workflow.com/llms-full.txt
```

Use the version-specific 2.0 bundle when the URL itself must name the current
major line:

```text
https://durable-workflow.com/llms-full-2.0.txt
```

Use the explicit 1.x pin when the URL itself must name the stable major line:

```text
https://durable-workflow.com/llms-full-1.x.txt
```

## 2. Discover The Local Workflow Surface

Start in the sample app's MCP endpoint when it is available:

```text
/mcp/workflows
```

See [MCP Workflow Surface](./mcp-workflows.md) for the reference tool contract,
safe smoke workflow keys, and agent report shape.

Call `list_workflows` first. The response tells the agent which workflows are
exposed, what credentials they need, which arguments they accept, and which
recent runs already exist. Use `simple` or `elapsed` for no-credential smoke
tests before touching examples that require external API keys.

If MCP is not available, use the CLI and server contracts instead:

```bash
dw server:info --output=json
dw workflow:list --output=json
dw task-queue:list --output=json
```

Those commands provide protocol versions, namespace context, workflow
visibility, and task-queue health without requiring UI scraping.

## 3. Make The Smallest Change

When editing workflow code, preserve the durable boundary:

- keep orchestration decisions in workflow methods
- put I/O, randomness, network calls, and external credentials in activities
- use signals, updates, queries, timers, and message streams instead of ad hoc
  tables or queue jobs
- keep repeated human or AI input on the
  [Message Streams](./features/message-streams.md) contract

That boundary matters more than the language surface. PHP workflow classes,
`dw`, the Python SDK, and external workers should all describe the same control
plane operations.

When no source edit is required, the "change" step can be an explicit operating
choice: select the exposed workflow key, provide a stable `business_key`, choose
`duplicate_start_policy=return_existing_active` for idempotent smoke runs, or
send the documented signal/update input that `diagnose_workflow` recommends.

## 4. Run Through Structured Handles

Use the most specific handle available for the task:

| Task | Preferred handle |
| --- | --- |
| Start a local sample workflow | `start_workflow` through `/mcp/workflows` |
| Repair a local sample workflow | `diagnose_workflow`, then `repair_workflow` only when remediation allows it |
| Start or command a server workflow | `dw workflow:start`, `dw workflow:signal`, `dw workflow:update`, or SDK equivalents |
| Check compatibility | `dw server:info --output=json` and `/api/cluster/info` |
| Inspect queue health | `dw task-queue:describe <queue> --output=json` |
| Compare client and worker surfaces | [Client and Worker Capabilities](./polyglot/cli-python-parity.md) |
| Implement a non-PHP worker | [Worker Protocol](./polyglot/worker-protocol.md) |
| Implement an external handler | [External Execution Surface](./polyglot/external-execution.md) |

Prefer JSON or JSONL outputs for agent loops. Terminal tables are for humans.

## 5. Diagnose Before Repairing

Collect facts before changing code or replaying commands:

```bash
dw doctor --output=json
dw server:info --output=json
dw debug workflow <workflow-id> --output=json
dw workflow:history <workflow-id> <run-id> --output=json
```

For the MCP sample app surface, call:

```json
{"tool": "diagnose_workflow", "arguments": {"workflow_id": "<workflow_id>"}}
```

Read `root_cause.category`, `remediation.classification`, and
`remediation.automatic_repair.allowed`. Call `repair_workflow` only when that
last field is true:

```json
{"tool": "repair_workflow", "arguments": {"workflow_id": "<workflow_id>"}}
```

If Waterline is available, export the selected run history. The export includes
typed history events, selected-run context, waits, timers, lineage, projection
source metadata, integrity checks, and durable failures. Those facts let an
agent distinguish a workflow bug from an unavailable worker, missing
credential, task-queue outage, incompatible client, or pending operator action.

## 6. Report With Contracts

Agent reports should cite stable facts, not screenshots:

- docs version and LLM bundle used
- workflow id, run id, namespace, and task queue
- command or MCP tool called
- JSON status, exit code, or named failure reason
- `root_cause.category`, `remediation.classification`, and whether repair was
  allowed or refused
- recent typed history events and latest durable failure
- compatibility or protocol version from `server:info` or `/api/cluster/info`

That report shape is portable across local sample apps, standalone server
deployments, Python workers, and future client SDKs.

## 7. Published Proof

The sample app conformance harness proves the agent loop from published
artifacts. Its MCP shard performs:

1. discover: JSON-RPC `tools/list` and `list_workflows`
2. change: choose the no-credential `simple` workflow with an explicit
   `business_key`
3. run: `start_workflow`
4. diagnose: `diagnose_workflow` with root-cause and remediation objects
5. repair: `repair_workflow` as a safe structured mutation or refusal
6. verify: `get_workflow_result`, `diagnose_workflow`, and
   `get_workflow_history`

The proof records tool status codes, workflow id, completion status, root-cause
schema id, remediation schema id, safe-mutation schema id, and bounded history
evidence in the conformance metadata.

## Related Pages

- [AI-Assisted Development](./ai-assisted-development.md)
- [Agent Tooling Contract](./agent-tooling-contract.md)
- [MCP Workflow Surface](./mcp-workflows.md)
- [Sample App](./sample-app.md)
- [CLI Command Reference](./polyglot/cli-reference.md)
- [Worker Protocol](./polyglot/worker-protocol.md)
- [External Execution Surface](./polyglot/external-execution.md)

<!-- Source: docs/agent-tooling-contract.md -->

# Agent Tooling Contract

Durable Workflow v2 keeps AI-assisted development boring by exposing product
facts through stable contracts. A tool should not infer workflow state from
HTML, parse logs as the source of truth, or guess which SDK behavior matches a
CLI command. It should read the docs version, discover the local surface, call
documented operations, and report named facts.

This page defines the contract shape that future MCP tools, local agents,
scripts, and SDKs should preserve.

## Contract Layers

| Layer | Stable handle | Contract expectation |
| --- | --- | --- |
| Docs retrieval | Canonical `llms.txt` and `llms-full.txt` track the stable 2.0 documentation. `llms-2.0.txt` and `llms-full-2.0.txt` are pinned aliases for the same release line. | Use canonical URLs for 2.0 product work. Pin `-1.x.txt` when a URL must name the archived 1.x line, or `-2.0.txt` when a consumer requires an explicit major-version URL. |
| Local discovery | `/mcp/workflows` `list_workflows` | The app-owned MCP allow-list names exposed workflow keys, required credentials, expected arguments, and smoke-test suitability. |
| Workflow operations | MCP `start_workflow`, `get_workflow_result`, `get_workflow_history`, `diagnose_workflow`, `repair_workflow`; `dw` JSON commands; SDK clients | Every client reports workflow id, run id, namespace, task queue, command status, root-cause classification, remediation, and named failure fields without scraping a UI. |
| Server diagnostics | `/api/cluster/info`, `dw server:info --output=json`, `dw doctor --output=json`, `dw debug workflow --output=json` | Compatibility, protocol, task-queue, worker, and stuck-run facts are machine-readable and bounded. |
| Durable evidence | Waterline selected-run detail and history export | Replay, waits, timers, lineage, projection source, integrity checks, durable failures, and operator actionability come from typed state. |
| Cross-language parity | CLI/Python request fixtures and SDK tests | Shared control-plane operations keep their request shape aligned across languages. |

Each layer should be usable on its own. Together, they give an agent enough
context to make a small change, prove it, and explain the result.

## MCP Tool Design

New MCP tools should expose Durable Workflow concepts directly:

- Use product nouns such as workflow, run, task queue, schedule, history,
  failure, worker, namespace, and compatibility.
- Return stable identifiers and named status fields instead of prose-only
  summaries.
- Return `root_cause`, `remediation`, and `next_actions` objects for diagnostic
  tools so agents can choose a next command without parsing natural language.
- Include bounded arrays and previews for history, failures, and payloads so a
  client can inspect them without downloading an unbounded trace.
- Separate discovery from mutation. A client should be able to ask what exists
  and what credentials are required before starting or commanding a workflow.
- Keep mutations explicit and structured. For local sample workflows,
  `repair_workflow` is the first-class repair mutation; it returns a
  `durable-workflow.v2.safe-mutation` envelope even when repair is refused or
  not needed.
- Mark no-credential smoke workflows explicitly so agents can test local wiring
  without touching external services.
- Never include secret values, customer-specific hostnames, or account-specific
  credentials in tool descriptions or result metadata.

The sample app's `/mcp/workflows` endpoint is the reference local workflow
surface. Future project-specific MCP servers should keep the same posture:
configuration owns the allow-list, tools operate only the listed workflows, and
results cite durable workflow facts that a human can reproduce through `dw`,
Waterline, or an SDK.

## Command And SDK Parity

Automation should be able to move between clients without semantic drift:

| Operation family | Required parity signal |
| --- | --- |
| start, signal, update, query, repair, cancel, terminate, archive | CLI and SDK request bodies match the documented control-plane shape. |
| history, describe, list, result | Responses preserve stable identifiers, status names, timestamps, and failure fields. |
| task queues and workers | Capacity, leases, slots, compatibility, and worker ids remain structured facts. |
| external execution | Input and result envelopes stay language-neutral and carry named bridge outcomes. |

When adding a CLI command, SDK method, or MCP tool for a control-plane action,
prefer a shared fixture or documented JSON example that another client can
assert against. Human-friendly tables can exist, but JSON or JSONL is the
automation contract.

## Diagnosis Report Shape

When a tool explains a failed or stuck run, the report should include:

- docs version and source page used
- command, SDK method, or MCP tool called
- workflow id, run id, namespace, and task queue
- current status and latest durable failure summary
- recent typed history event names
- pending waits, timers, tasks, or external activity leases
- worker and task-queue compatibility facts when available
- named exit code, HTTP status, validation error, or blocked reason
- machine-readable `root_cause.category` and `remediation.classification`
- whether a safe repair mutation was allowed, applied, refused, or not needed

That shape keeps reports portable across local Laravel apps, standalone server
deployments, Python workers, and future SDKs.

## Root Cause And Remediation

The supported root-cause schema id is
`durable-workflow.v2.agent-root-cause`. Diagnostic tools should include:

- `category`, such as `activity_failure`, `workflow_failure`,
  `task_repair_attention`, `waiting_for_signal`, `history_growth_attention`,
  `in_progress`, or `none`
- `source.kind` and `source.id`
- `retryable`, `severity`, and `actionable`

The supported remediation schema id is
`durable-workflow.v2.agent-remediation`. It includes
`classification`, `summary`, `automatic_repair.tool`,
`automatic_repair.allowed`, and `next_actions`. `repair_workflow` is the
supported local MCP repair surface. When MCP is unavailable, the replacement
surface is the documented `dw` CLI JSON contract plus server HTTP control-plane
routes:

```bash
dw debug workflow <workflow-id> --output=json
dw workflow:history <workflow-id> <run-id> --output=json
dw workflow:repair <workflow-id> --output=json
dw system:repair-status --output=json
dw system:repair-pass --output=json
```

Those commands are the supported non-MCP contract for inspection, safe
mutation, diagnostics, and repair automation.

## Guardrails For Agents

Give agents permission to automate ceremony, not to bypass the durable model:

- Keep workflow orchestration deterministic.
- Put I/O, randomness, external API calls, and credentials in activities or
  external handlers.
- Use signals, updates, queries, schedules, timers, and message streams instead
  of ad hoc state tables for workflow control.
- Treat Waterline history export as evidence, not as a mutation API.
- Treat route lists, database internals, and framework-specific model rows as
  implementation details unless the public docs explicitly name them.

The invariant remains human-readable: workflows record durable decisions,
activities perform fallible work, and replay must be able to explain what
happened. The tooling contract gives agents stable handles for the rest.

## Related Pages

- [AI-Assisted Development](./ai-assisted-development.md)
- [Agent Operating Loop](./agent-operating-loop.md)
- [MCP Workflow Surface](./mcp-workflows.md)
- [Client and Worker Capabilities](./polyglot/cli-python-parity.md)
- [Server HTTP API Reference](./polyglot/server-api-reference.md)
- [Monitoring](./monitoring.md)

<!-- Source: docs/ai-agent-workflow-engine.md -->

# Is Durable Workflow a good workflow engine for AI agents?

**Yes.** Durable Workflow 2.0 is a credible choice when an agent must operate
replay-based durable workflows through
machine-readable contracts, especially when self-hosting, a broadly
approachable public protocol, deployment flexibility, or a legible core matters.

It is not “agent-ready” merely because an MCP server exists. The agent contract
is the complete **Discover -> Change -> Run -> Diagnose -> Repair** loop across
version/capability discovery, schemas, control-plane operations, results,
history, typed failures, worker and queue compatibility, safe mutations, and
post-change verification. MCP is one interface alongside HTTP+JSON, CLI JSON,
SDK clients, Waterline exports, and published protocol schemas.

## Machine-operable loop

| Stage | Public machine-readable surface | Agent decision or proof |
| --- | --- | --- |
| **Discover** | `GET /api/cluster/info`; [2.0 Capability Index](/docs/capabilities/); [protocol-spec catalog](/docs/platform-protocol-specs/); `dw schema:list --output=json`; version-pinned `llms-2.0.txt` and `llms-full-2.0.txt` | Select the 2.0 docs line, verify the exact artifact/protocol/codec/capability tuple, and discover supported workflow, worker, namespace, queue, and schema surfaces before acting. |
| **Change** | Server start/signal/update routes; PHP, Python, and Rust SDK clients; `dw workflow:start`, `workflow:signal`, and `workflow:update` JSON commands; MCP `list_workflows` and `start_workflow` where an app exposes them | Make a bounded code or operating change with a stable type, workflow/run identity, namespace, task queue, input envelope, and idempotency choice. |
| **Run** | Workflow describe/result/history endpoints; SDK handles; CLI `workflow:describe`, `workflow:result`, and `workflow:history` JSON; MCP result/history tools | Observe a named status and typed result for the selected run instead of inferring success from process exit or log text. |
| **Diagnose** | `dw doctor`, `server:info`, `debug workflow`, task-queue and worker JSON; typed replay/history failures; `/api/cluster/info`; Waterline selected-run export | Distinguish code/replay failure from missing or incompatible workers, queue admission, timeout, auth, codec, or runtime-health causes using named fields. |
| **Repair** | Server/CLI repair, retry, cancel, terminate, archive, build-ID drain/resume, and compatibility-routing operations; MCP `repair_workflow` safe-mutation envelope | Apply only an allowed, scoped mutation; then re-run Discover and Run surfaces and verify status, history, worker/queue health, compatibility metadata, and the absence or expected transition of the diagnosed failure. |

The detailed [Agent Tooling Contract](/docs/agent-tooling-contract/) freezes
the report shapes and safe-mutation posture. The
[Agent Operating Loop](/docs/agent-operating-loop/) provides a longer
runbook.

## First-party SDKs and application boundary

PHP, Python, and Rust are first-party standalone SDK surfaces on the stable
2.0 release line. Workflow is the
separately versioned embedded Laravel engine and standalone-server core.

- Framework-neutral PHP applications and remote workers use
  [`durable-workflow/sdk`](/docs/polyglot/php/); embedded Laravel
  applications use `durable-workflow/workflow`.
- Python authors deterministic workflows and activities and is also an
  operational/control-plane surface for workflows, schedules, namespaces,
  workers, queues, history, and repair.
- Rust authors deterministic workflows, activities, and worker services. It
  supports durable timers, child workflows, activity retries/timeouts,
  signals, replayed queries, cancellation/termination, and typed outcomes at
  the current floor. It is not merely a protocol adapter.

All three use the same durable execution model and public protocol.
Cross-language child workflows and activities use registered string types and
the shared Avro envelope, preserving fixed typed Value semantics through the
official Avro language packages. The [2.0 Capability Index](/docs/capabilities/)
records exact floors and deliberate gaps such as Rust's current update-authoring
and schedule-management boundary.

## Does a Python or Rust team need Laravel?

No. The published standalone Server is implemented
in PHP and deployed as infrastructure. Python- or Rust-only application teams
run native SDK workers against its versioned HTTP+JSON protocol; their
application code does not embed Laravel and does not become a Laravel
application.

There are three deployment/control-plane choices:

1. **Standalone:** self-host the published server and connect native SDK
   workers. See [Standalone Server](/docs/polyglot/server/).
2. **Embedded:** install the PHP engine into a Laravel application and reuse its
   queues, database, configuration, and deployment. This is a differentiated
   Laravel-native path, not the platform category.
3. **Durable Workflow Cloud:** provision a managed namespace. Cloud operates
   the orchestration runtime, state, history, schedules, placement, and recovery
   while application teams run SDK clients and workers. A self-hosted Server is
   not attached to Cloud. Evaluate only the exact
   [Cloud Managed Runtime](/docs/polyglot/cloud-control-plane/) contract.

## What exists now

The current published 2.0 tuple implements deterministic workflows,
activities, signals, queries, updates where advertised, timers, retries,
timeouts, child workflows, cancellation, termination, side effects and version
markers on the SDKs named in the capability index, schedules, namespaces,
search attributes, codec interoperability, worker compatibility, typed history
and failures, structured diagnostics, and safe operator commands.

Read the runtime's manifests and the exact
[compatibility matrix](/docs/compatibility/) instead of assuming feature parity
across SDKs or inferring capability from a top-level version.

## Maturity and fit

Durable Workflow 2.0 is stable, but its ecosystem and production history are
younger than established durable-execution platforms. Evaluate the exact
capabilities and operating envelope documented for the deployment you intend
to run.

Durable Workflow 2.0 is compelling where agent operability, self-hosting, a
broadly approachable HTTP+JSON public protocol, a choice among standalone,
embedded, and managed Cloud operation, or a legible PHP core are deciding
factors. The comparison is about fit and maturity, not whether both products
implement replay-based durable execution.

## Who should choose it?

Choose Durable Workflow 2.0 for evaluation when:

- autonomous or human operators need structured discovery, typed diagnostics,
  safe mutations, and post-change verification;
- Python, Rust, and PHP workflow/application code should share one durable
  execution model and codec contract;
- self-hosting or choosing among standalone, Laravel-embedded, and the exact
  current Cloud managed-runtime contract is valuable;
- the team can validate its required capability floors against the published
  runtime manifests.

Do not choose it yet when:

- it needs an SDK feature the [capability index](/docs/capabilities/) marks
  unavailable for its chosen language;
- it requires a longer production history, broader ecosystem breadth, or more
  accumulated production evidence than the current release provides;
- it requires managed capabilities, regions, SLAs, certifications, private
  connectivity, worker/runtime ownership, or other guarantees beyond the exact
  current [Durable Workflow Cloud contract](/docs/polyglot/cloud-control-plane/).

<!-- Source: docs/ai-assisted-development.md -->

# AI-Assisted Development

Durable Workflow v2 is designed so a person can learn the durable execution
model, then let tools handle the repeated inspection and operation work. The
important product surface is not a chat prompt; it is the set of stable handles
an agent can read, call, and cite without guessing.

## Retrieval Surfaces

Use these LLM artifacts to ground an AI assistant in Durable Workflow's
machine-readable docs surface:

| Surface | URL | Use |
| --- | --- | --- |
| canonical manifest | `https://durable-workflow.com/llms.txt` | Stable 2.0 docs index. Tracks the site's unversioned default Docs path. |
| canonical full bundle | `https://durable-workflow.com/llms-full.txt` | Single-file docs bundle matching the canonical manifest. |
| v1 manifest | `https://durable-workflow.com/llms-1.x.txt` | Version-pinned 1.x archive. |
| v1 full bundle | `https://durable-workflow.com/llms-full-1.x.txt` | Version-pinned 1.x stable bundle. |
| v2 manifest | `https://durable-workflow.com/llms-2.0.txt` | Version-pinned 2.0 index, equivalent to the canonical manifest. |
| v2 full bundle | `https://durable-workflow.com/llms-full-2.0.txt` | Version-pinned 2.0 bundle. |

Default agent prompts should fetch `/llms.txt` or `/llms-full.txt` for stable
2.0 work. Pin the `-1.x.txt` URLs when maintaining a 1.x application.

## Local MCP Surface

The [sample app](/docs/sample-app) exposes a Laravel MCP server at
`/mcp/workflows`. It is the reference AI-client integration for local v2
workflow development. The dedicated [MCP Workflow Surface](/docs/mcp-workflows)
page defines the endpoint, tool set, safe smoke workflows, and report shape.

The MCP server gives an agent structured workflow operations instead of UI
scraping:

| Tool | Stable handle |
| --- | --- |
| `list_workflows` | Discover configured workflow keys, credential requirements, status values, and recent runs. |
| `start_workflow` | Start a configured v2 workflow and receive `workflow_id`, `run_id`, status, business key, and command outcome. |
| `get_workflow_result` | Poll the current or selected run for status, output, visibility metadata, and latest failure summary. |
| `get_workflow_history` | Fetch a bounded tail of typed v2 history events and recent durable failures. |

Use `simple` or `elapsed` as no-credential smoke tests. Use workflows with
external credentials only after the local environment contains the required
keys.

When an agent edits repeated AI or human-input workflows, point it at the v2
[Message Streams](/docs/features/message-streams) contract. The stable
authoring pattern is `Workflow::inbox()` / `Workflow::outbox()` /
`MessageStream`; direct `MessageService`, `WorkflowMessage`, or cursor-row
writes are runtime internals, not sample code patterns.

## Command And Diagnostic Contracts

Agents should prefer machine-readable surfaces over screenshots or prose:

- The [Agent Operating Loop](/docs/agent-operating-loop) turns the v2
  docs, MCP endpoint, `dw` JSON output, Waterline history export, and SDK
  references into one repeatable discover-change-run-diagnose workflow.
- The [Agent Tooling Contract](/docs/agent-tooling-contract) defines how
  MCP tools, CLI JSON, server diagnostics, Waterline exports, and SDK fixtures
  line up as one machine-operable surface.
- `dw` commands expose stable exit codes for automation. See the
  [CLI reference](/docs/polyglot/cli#exit-codes).
- [Client and Worker Capabilities](/docs/polyglot/cli-python-parity/)
  compares lifecycle, messages, schedules, visibility, and worker execution
  across `dw` and all three first-party SDKs, with shared request evidence
  where it exists.
- The [external execution surface](/docs/polyglot/external-execution)
  publishes the activity-grade task boundary, carrier requirements, bridge
  outcomes, and input/result envelope paths through `/api/cluster/info`.
- Server health, info, namespace, workflow, schedule, worker, and task-queue
  commands are the right handles for shell-based checks.
- Waterline history export is the source for replay and failure diagnosis. It
  includes typed history events, projection source metadata, integrity checks,
  selected-run context, waits, timers, lineage, and latest durable failures.
- Python SDK types and method signatures live in the generated
  [Python API reference](https://python.durable-workflow.com/).

When an agent needs to explain a stuck workflow, collect these facts first:

1. `workflow_id` and `run_id`
2. current run status and latest failure summary
3. recent typed history events
4. open waits, timers, and pending tasks
5. worker and task-queue health
6. the relevant docs version, using `2.0` for current product work and `1.x`
   only for legacy maintenance

That set is enough to distinguish a durable workflow failure from a worker
runtime failure, a missing external credential, a queue outage, or an operator
action waiting for approval.

## Prompt Shape

Give coding agents explicit constraints and handles:

```text
Use Durable Workflow docs from https://durable-workflow.com/llms-full-2.0.txt.
Use the sample app MCP endpoint at /mcp/workflows for workflow discovery,
start, result, and history. Prefer dw JSON/exit-code contracts and Waterline
history exports over screenshots. For external handlers or bridge adapters,
read worker_protocol.external_execution_surface_contract from /api/cluster/info
and preserve the external task input/result envelopes. Use Durable Workflow
2.0 APIs unless I explicitly ask for legacy 1.x maintenance.
```

The goal is simple: humans learn the workflow/activity/replay invariant, and
tools operate through documented contracts instead of inferred product behavior.

<!-- Source: docs/contribute-a-sample.md -->

# Contribute a Sample

The [Sample App](/docs/sample-app) is the canonical place to show
what a Durable Workflow pattern looks like in a real Laravel project.
If you have a pattern that would help other engineers — a saga shape we
do not yet show, a different way of using message streams, an
integration with a tool the existing samples do not exercise — this
guide is the path that gets it merged.

The contract here is short on purpose. A sample that follows it lands
on a predictable cadence. A sample that skips parts of it sits in
review until the gaps are filled.

## Before you write code

1. **Open a `sample-request` issue first.** The
   [`sample-request` template](https://github.com/durable-workflow/sample-app/issues/new/choose)
   names the pattern surface, the public docs page that defines it,
   and the minimum Durable Workflow package version it needs. The
   issue is the place where maintainers and the contributor agree the
   sample is worth merging *before* the contributor invests in a PR.
2. **Pick a pattern surface that does not already exist in the sample
   index.** The
   [README sample index](https://github.com/durable-workflow/sample-app#sample-index)
   names every covered pattern. If your idea collapses to "a different
   spelling of the simple workflow," that is not a new sample — that
   is a documentation tweak on `App\Workflows\Simple\SimpleWorkflow`.
3. **Find the closest existing workflow class.** A new sample should
   read like the one next to it: same directory layout under
   `app/Workflows/<Pattern>/`, same shape of artisan command, same
   testing posture. New samples that invent novel scaffolding are
   rejected even if the pattern they show is good.

## What a merged sample looks like

A merged sample is *runnable*, *teachable*, *covered*, and *visible*.
That maps to four concrete artifacts in the sample-app repository:

1. **A workflow class** under `app/Workflows/<Pattern>/` that
   demonstrates the pattern end to end. Workflow code stays
   deterministic — clock reads behind `sideEffect()`, external work
   inside activities, waits behind signals, updates, timers, or
   message streams. The workflow class compiles without `OPENAI_API_KEY`
   or other external credentials unless the pattern explicitly
   requires them.
2. **An artisan command** that starts the workflow with realistic input
   so a reader can run one command and watch the run land in
   Waterline. The command lives next to the matching artisan commands
   in `routes/console.php` (or the equivalent registration entry) and
   uses the same naming convention (`app:<short-pattern>`).
3. **A `config/workflow_mcp.php` entry** that names the workflow class,
   pattern, command, required credentials, and arguments. This entry
   is what makes the sample discoverable through the MCP server, and
   the upstream-coverage lint refuses to mark a coverage row `covered`
   until the entry is present.
4. **A test under `tests/`** that exercises the workflow against the
   in-memory v2 worker. The test does not have to assert on every
   typed history event, but it must prove that the workflow completes
   for the input the artisan command passes.

If the sample is meant to demonstrate a *bug fix* rather than a
feature, it lands under `app/Workflows/Bug/<short-id>/` instead, with
the same four artifacts.

## What goes in the docs

Three docs surfaces move when a sample lands:

- **README "Sample Index"** in the sample-app repository — one row
  per sample, with the goal, the workflow class, the artisan command,
  and the MCP key. This is the source of truth.
- **Sample gallery on the docs site
  ([`docs/sample-app.md`](/docs/sample-app))** — a mirrored row
  with the Waterline screen the reader should expect to see. The
  gallery exists so a reader can decide whether the sample is the one
  they want before cloning the repo.
- **Pattern-page cross-link** on the docs-site pattern page that
  defines the surface (sagas, signals, message streams,
  child-workflows, …). The cross-link table at the bottom of the
  sample-app docs page is the canonical list; an entry that names a
  surface but does not link to a sample is treated as a gap.

These three changes ship in the same PR as the sample workflow itself.
A sample that lands without a docs-site mirror is in `gap` state for
the upstream-coverage tracker until the docs PR catches up, so
contributors are encouraged to ship them together.

## Naming, layout, and style

- **Class layout.** `app/Workflows/<Pattern>/<Pattern>Workflow.php`,
  with the activity classes alongside it under
  `app/Workflows/<Pattern>/Activities/`. Bug reproducers go under
  `app/Workflows/Bug/<short-id>/` with the same shape.
- **Artisan command.** `app:<short-pattern>` (lower-case, hyphen-free).
  Long-form names (`app:run-<pattern>`) are not used.
- **MCP key.** Match the artisan command's short pattern so the MCP
  client sees the same name a human types in the terminal.
- **Comments.** Comment the *why*, not the *what*. Replay-safety
  invariants (why a value is wrapped in `sideEffect()`, why a wait
  uses a timer instead of `usleep()`) are worth a comment; the same
  comment on every activity invocation is not.
- **Waterline screenshots.** The docs-site sample-app gallery names
  the Waterline screen the sample is expected to produce; add or
  refresh that screen as part of the sample's PR.

## What gets a sample rejected

A sample is sent back when:

- the workflow code uses `now()`, `random_*()`, or other
  non-deterministic calls outside `sideEffect()` or an activity;
- the activities do durable bookkeeping the engine already owns —
  writing `workflow_messages` rows directly, manually advancing stream
  cursors, or persisting Waterline state from inside workflow code;
- the artisan command requires external credentials without saying so
  in the MCP entry's `requires` field;
- the README sample-index, docs-site gallery row, and pattern-page
  cross-link do not all land in the same change;
- the sample reproduces an existing pattern surface without showing
  something new (a different shape, a different failure mode, a
  different integration). "Same pattern, different prose" is a docs
  change, not a sample.

These are the same things the upstream-coverage lint and the
sample-app review checklist look for, so catching them yourself before
opening the PR is the fastest path to merge.

## Quick checklist

Use this list when you open the PR:

- [ ] `sample-request` issue exists and links to the public pattern
      docs page.
- [ ] Workflow class under `app/Workflows/<Pattern>/`.
- [ ] Artisan command registered with the `app:<short-pattern>` name.
- [ ] `config/workflow_mcp.php` entry with class, pattern, command,
      requires, and arguments.
- [ ] Test that exercises the workflow end to end.
- [ ] README "Sample Index" row.
- [ ] Docs-site gallery row in
      [`docs/sample-app.md`](/docs/sample-app).
- [ ] Cross-link from the matching pattern page on the docs site.
- [ ] Public-boundary scan clean (`scripts/check-public-boundary.sh`).

Maintainers run the same list during review, so a PR that ticks every
box should land within one release cycle.

<!-- Source: docs/mcp-workflows.md -->

# MCP Workflow Surface

The sample app exposes the reference Durable Workflow v2 MCP server at:

```text
/mcp/workflows
```

Use it when an AI client needs to inspect or operate a local workflow app
without scraping Waterline or guessing Laravel internals. The endpoint is a
structured development surface: it names the workflows the app chooses to
expose, describes credential requirements, starts runs, polls results, and
returns bounded history facts.

## Server Contract

The sample app registers the server from `routes/ai.php` and configures exposed
workflow keys in `config/workflow_mcp.php`. Treat that configuration as the
public allow-list for AI clients. A workflow is MCP-operable only when it is
listed there with enough metadata for a client to decide whether it can run the
workflow safely.

Each configured workflow should describe:

- a stable workflow key
- the workflow class behind that key
- required arguments and optional arguments
- credential requirements
- whether it is safe for no-credential local smoke tests
- output and history expectations that an agent can cite

Keep secrets out of tool descriptions. Say that a workflow requires a
credential, but do not include the credential value or account-specific details.

## Tools

The reference server exposes six workflow tools:

| Tool | Use |
| --- | --- |
| `list_workflows` | Discover configured workflow keys, descriptions, credential requirements, v2 statuses, and recent runs. |
| `start_workflow` | Start a configured workflow and return `workflow_id`, `run_id`, status, business key, and command outcome. |
| `get_workflow_result` | Poll the current or selected run for status, output, visibility metadata, and latest failure summary. |
| `get_workflow_history` | Fetch a bounded tail of typed v2 history events and recent durable failures. |
| `diagnose_workflow` | Classify a selected run with structured facts, root cause, remediation, and next actions. |
| `repair_workflow` | Request the built-in v2 repair command and return a structured accepted, refused, or not-needed mutation result. |

Call `list_workflows` before any start. It is the agent's compatibility check:
the response tells the client which workflow keys exist and whether a workflow
can run in the current environment.

Discovery envelopes use the normative schema id
`durable-workflow.v2.mcp-discovery`. Agents can use it to recognize the
published tool list shape, parameter schemas, discovery hints, and
`payload_preview_limit_bytes` semantics before calling a tool.

## Tool Input Contract

The MCP tool schemas intentionally use Durable Workflow terms instead of
Laravel internals. Keep these fields stable when extending the sample app
server:

| Tool | Stable inputs |
| --- | --- |
| `list_workflows` | `show_recent`, `limit`, and optional `status` for recent-run discovery. |
| `start_workflow` | `workflow`, ordered `arguments`, optional `instance_id`, `business_key`, `visibility_labels`, `memo`, `search_attributes`, and `duplicate_start_policy`. |
| `get_workflow_result` | `workflow_id`, optional `run_id`, `include_recent_history`, and `history_limit`. |
| `get_workflow_history` | `workflow_id` or `run_id`, `limit`, and `include_payloads`. |
| `diagnose_workflow` | `workflow_id` or `run_id`, plus optional `history_limit`. |
| `repair_workflow` | `workflow_id` or `run_id`. |

Prefer `arguments` over the legacy `args` input for new agents. `arguments`
maps directly to the workflow `handle()` argument order, while `args` exists
only for older object-shaped callers.

Use caller-supplied `instance_id` only when an agent needs idempotency. Pair it
with `duplicate_start_policy=return_existing_active` when a repeated smoke run
should attach to the active workflow instead of failing as a duplicate.

## Tool Result Contract

Tool-result envelopes use the normative schema id
`durable-workflow.v2.mcp-tool-results`. Agents can use it to parse result
status, payload preview truncation, error fields, root-cause and remediation
objects, safe-mutation envelopes, and schema/version markers.

Agents should treat the following result fields as the durable handles for
cross-tool correlation:

| Tool | Stable result fields |
| --- | --- |
| `list_workflows` | `available_workflows`, `allow_fqcn`, `workflow_id_kind`, `run_id_kind`, `status_values`, and optional `recent_workflows`. |
| `start_workflow` | `workflow_id`, `run_id`, `workflow`, `workflow_class`, `workflow_type`, `status`, `running`, `business_key`, `duplicate_start_policy`, and `command`. |
| `get_workflow_result` | `found`, `workflow_id`, `run_id`, `current_run_id`, `current_run_is_selected`, `status`, `running`, `output`, `error`, visibility metadata, and timestamps. |
| `get_workflow_history` | `found`, `workflow_id`, `run_id`, `current_run_id`, `status`, `history_event_count`, `returned_event_count`, `events_are_most_recent`, `payloads_included`, `events`, and `failures`. |
| `diagnose_workflow` | `found`, `workflow_id`, `run_id`, `diagnosis`, `facts`, `latest_failure`, `recent_history`, `root_cause`, `remediation`, and `next_actions`. |
| `repair_workflow` | `found`, `workflow_id`, `run_id`, `accepted`, `status`, `mutation`, `command`, `remediation`, and `next_actions`. |

History payload previews are intentionally bounded. When `include_payloads` is
true, each event preview reports `payload_preview_limit_bytes`, `size_bytes`,
`preview_bytes`, and `truncated` so an agent can cite whether it saw the full
payload or only a preview.

## Failure And Remediation Taxonomy

`diagnose_workflow` is the machine-readable root-cause surface. Its
`root_cause` object uses the schema id
`durable-workflow.v2.agent-root-cause` and includes:

- `category`, such as `activity_failure`, `workflow_failure`,
  `task_repair_attention`, `waiting_for_signal`, `history_growth_attention`,
  `in_progress`, or `none`
- `source.kind` and `source.id` for the workflow, activity, wait, task queue,
  or history family that produced the classification
- `retryable`, `severity`, and `actionable`
- failure details such as `failure_category`, `exception_class`, and `handled`
  when a durable failure row exists

The companion `remediation` object uses the schema id
`durable-workflow.v2.agent-remediation`. It includes
`classification`, a short `summary`, `automatic_repair.tool`, and
`automatic_repair.allowed`, plus `next_actions` entries for the supported
follow-up commands. Agents should call `repair_workflow` only when
`automatic_repair.allowed` is true. Other classifications tell the agent to
wait, send expected input through the documented workflow command surface,
inspect history, change workflow or activity code, or plan Continue-As-New.

`repair_workflow` returns a `durable-workflow.v2.safe-mutation` envelope. The
`mutation.applied` field says whether a repair command was accepted, while
`command.outcome` distinguishes `repair_dispatched`, `repair_not_needed`, and
structured refusals such as a terminal or non-current run.

## Safe Agent Loop

Use no-credential workflows first:

```json
{"tool": "list_workflows", "arguments": {"show_recent": true, "limit": 5}}
{"tool": "start_workflow", "arguments": {"workflow": "simple", "business_key": "demo-001"}}
{"tool": "diagnose_workflow", "arguments": {"workflow_id": "<workflow_id>"}}
{"tool": "repair_workflow", "arguments": {"workflow_id": "<workflow_id>"}}
{"tool": "get_workflow_result", "arguments": {"workflow_id": "<workflow_id>"}}
{"tool": "get_workflow_history", "arguments": {"run_id": "<run_id>", "limit": 25}}
```

The `simple` and `elapsed` workflow keys are the preferred smoke surfaces. Use
credentialed examples only after `list_workflows` reports the requirement and
the local environment has the needed keys.

## Report Shape

An AI client should report MCP results with stable facts:

- docs version used, normally `2.0`
- MCP endpoint and tool name
- workflow key, `workflow_id`, and `run_id`
- status and latest failure summary
- `root_cause.category` and `remediation.classification`
- whether `remediation.automatic_repair.allowed` was true before a repair
  mutation was attempted
- bounded history event names and timestamps
- whether the run used a no-credential smoke workflow or a credentialed example

Those facts line up with the CLI, Python SDK, and Waterline history-export
surfaces, so a human can reproduce the same run from another client.

## Related Pages

- [AI-Assisted Development](./ai-assisted-development.md)
- [Agent Operating Loop](./agent-operating-loop.md)
- [Agent Tooling Contract](./agent-tooling-contract.md)
- [Sample App](./sample-app.md)
- [Client and Worker Capabilities](./polyglot/cli-python-parity.md)
- [Message Streams](./features/message-streams.md)

<!-- Source: docs/testing.md -->

# Testing

## `Workflow\V2`

`Workflow\V2\WorkflowStub::fake()` provides a deterministic inline test path for the supported `V2` fake surface.

- ready workflow and activity tasks execute inline instead of waiting for a queue worker
- nested child workflows execute as real nested `V2` runs under that same fake mode
- activity mocks still write durable `activity_executions`, task, and history rows
- backend capability checks are bypassed for the fake path, so `sync` remains usable in tests
- `WorkflowStub::assertDispatched()`, `assertDispatchedTimes()`, `assertNotDispatched()`, and `assertNothingDispatched()` cover `V2` activity dispatches
- `WorkflowStub::assertSignalSent()`, `assertSignalSentTimes()`, and `assertSignalNotSent()` cover `V2` signals sent through `WorkflowStub::signal()` or `signalWithStart()`
- `WorkflowStub::assertUpdateSent()`, `assertUpdateSentTimes()`, and `assertUpdateNotSent()` cover `V2` updates sent through `WorkflowStub::update()`, `attemptUpdate()`, or `submitUpdate()`
- delayed timer tasks stay durable queued work until they become due; after you advance time, `WorkflowStub::runReadyTasks()` drains already-due tasks inline

### First-release testing scope

The following are **not** part of the first-release `V2` fake surface and may be added as future additive contracts:

- there is no `V2` equivalent of the legacy `resume()` bridge
- there is no dedicated child-workflow mock or dispatch-assert helper — child workflows execute as real nested `V2` runs under fake mode, so test them through their observable output rather than mocking
- there is no delayed-callback hook for injecting signals or updates at virtual time offsets during a single fake execution — advance time and call `WorkflowStub::runReadyTasks()` between explicit signal/update calls instead

`WorkflowStub::mock()` enforces this scope at runtime: passing a `Workflow` subclass throws a `LogicException` with a clear message directing you to test child workflows through their observable output instead. Only `Activity` classes (and unresolved string keys) are accepted as mock targets.

```php
use function Workflow\V2\activity;
use Workflow\V2\Workflow;

final class MyWorkflow extends Workflow
{
    public function handle(): array
    {
        return [
            'result' => activity(MyActivity::class, 'Taylor'),
            'workflow_id' => $this->workflowId(),
            'run_id' => $this->runId(),
        ];
    }
}
```

```php
use Workflow\V2\Testing\ActivityFakeContext;
use Workflow\V2\WorkflowStub;

public function testWorkflow(): void
{
    WorkflowStub::fake();

    WorkflowStub::mock(MyActivity::class, function (ActivityFakeContext $context, string $name): string {
        $this->assertSame('Taylor', $name);
        $this->assertSame('my-workflow-id', $context->workflowId());

        return "Hello, {$name}!";
    });

    $workflow = WorkflowStub::make(MyWorkflow::class, 'my-workflow-id');
    $workflow->start();

    $this->assertTrue($workflow->refresh()->completed());
    $this->assertSame('Hello, Taylor!', $workflow->output()['result']);

    WorkflowStub::assertDispatched(MyActivity::class, function (string $name): bool {
        return $name === 'Taylor';
    });
}
```

Use `WorkflowStub::assertDispatched()`, `assertDispatchedTimes()`, `assertNotDispatched()`, and `assertNothingDispatched()` to assert the recorded `V2` activity dispatches.

For timer-backed workflows, advance time with Laravel's travel helpers and then drain the due task queue explicitly:

```php
use function Workflow\V2\timer;
use Workflow\V2\Workflow;
use Workflow\V2\WorkflowStub;

final class MyTimerWorkflow extends Workflow
{
    public function handle(): array
    {
        timer(60);

        return ['done' => true];
    }
}

public function testTimerWorkflow(): void
{
    WorkflowStub::fake();

    $workflow = WorkflowStub::make(MyTimerWorkflow::class, 'timer-workflow');
    $workflow->start();

    $this->travel(60)->seconds();

    WorkflowStub::runReadyTasks();

    $this->assertTrue($workflow->refresh()->completed());
    $this->assertSame(['done' => true], $workflow->output());
}
```

`WorkflowStub::runReadyTasks()` only drains tasks that are already due. It does not fast-forward wall clock time or force future-delayed timers to fire early.

### Sending Signals in Fake Mode

Signal-waiting workflows can receive signals in fake mode. When you call `$workflow->signal(...)`, the engine records the durable signal command and creates a workflow task. In fake mode that task executes inline, so the workflow resumes synchronously:

```php
use function Workflow\V2\activity;
use Workflow\V2\Attributes\Signal;
use function Workflow\V2\await;
use Workflow\V2\Workflow;
use Workflow\V2\WorkflowStub;

#[Signal('name-provided')]
final class ApprovalWorkflow extends Workflow
{
    public function handle(): array
    {
        $name = await('name-provided');
        $greeting = activity(GreetingActivity::class, $name);

        return ['name' => $name, 'greeting' => $greeting];
    }
}

public function testSignalWorkflow(): void
{
    WorkflowStub::fake();
    WorkflowStub::mock(GreetingActivity::class, 'Hello, Taylor!');

    $workflow = WorkflowStub::make(ApprovalWorkflow::class, 'approval-1');
    $workflow->start();

    // Workflow suspends at await('name-provided')
    $this->assertSame('waiting', $workflow->refresh()->status());

    // Send the signal — resumes the workflow inline
    $workflow->signal('name-provided', 'Taylor');

    $this->assertTrue($workflow->refresh()->completed());
    $this->assertSame('Taylor', $workflow->output()['name']);

    WorkflowStub::assertSignalSent('name-provided');
    WorkflowStub::assertSignalSentTimes('name-provided', 1);
    WorkflowStub::assertSignalNotSent('other-signal');
}
```

Use `WorkflowStub::assertSignalSent()`, `assertSignalSentTimes()`, and `assertSignalNotSent()` to verify which signals were sent during the test. The callback form receives the instance id and signal arguments:

```php
WorkflowStub::assertSignalSent(
    'name-provided',
    fn (string $instanceId, string $name): bool =>
        $instanceId === 'approval-1' && $name === 'Taylor'
);
```

### Sending Updates in Fake Mode

Updates also apply inline in fake mode. When you call `$workflow->attemptUpdate(...)` or `$workflow->update(...)`, the engine records the durable update command, creates a workflow task that applies the update, and executes it inline:

```php
use Workflow\UpdateMethod;
use Workflow\V2\Attributes\Signal;
use function Workflow\V2\await;
use Workflow\V2\Workflow;
use Workflow\V2\WorkflowStub;

#[Signal('done')]
final class SettingsWorkflow extends Workflow
{
    private bool $enabled = false;

    public function handle(): array
    {
        await('done');

        return ['enabled' => $this->enabled];
    }

    #[UpdateMethod]
    public function toggle(bool $enabled): array
    {
        $this->enabled = $enabled;

        return ['enabled' => $this->enabled];
    }
}

public function testUpdateWorkflow(): void
{
    WorkflowStub::fake();

    $workflow = WorkflowStub::make(SettingsWorkflow::class, 'settings-1');
    $workflow->start();

    $this->assertSame('waiting', $workflow->refresh()->status());

    $result = $workflow->attemptUpdate('toggle', true);

    $this->assertTrue($result->accepted());

    WorkflowStub::assertUpdateSent('toggle');
    WorkflowStub::assertUpdateSentTimes('toggle', 1);
    WorkflowStub::assertUpdateNotSent('other-update');

    // Complete the workflow
    $workflow->signal('done');
    $this->assertTrue($workflow->refresh()->completed());
    $this->assertTrue($workflow->output()['enabled']);
}
```

Use `WorkflowStub::assertUpdateSent()`, `assertUpdateSentTimes()`, and `assertUpdateNotSent()` to verify which updates were sent during the test. The callback form receives the instance id and update arguments:

```php
WorkflowStub::assertUpdateSent(
    'toggle',
    fn (string $instanceId, bool $enabled): bool =>
        $instanceId === 'settings-1' && $enabled === true
);
```

### Testing Start Outcomes and Duplicate-Start Policy

Use `attemptStart()` to verify duplicate-start behavior without throwing exceptions. The `StartResult` exposes typed outcome helpers:

```php
use Workflow\V2\StartOptions;
use Workflow\V2\WorkflowStub;

public function testRejectDuplicateStart(): void
{
    WorkflowStub::fake();
    WorkflowStub::mock(MyActivity::class, 'result');

    $workflow = WorkflowStub::make(MyWorkflow::class, 'order-123');

    $first = $workflow->start('Taylor');
    $this->assertTrue($first->startedNew());

    $second = WorkflowStub::load('order-123');
    $duplicate = $second->attemptStart('Taylor');

    $this->assertTrue($duplicate->rejected());
    $this->assertTrue($duplicate->rejectedDuplicate());
    $this->assertSame('instance_already_started', $duplicate->rejectionReason());
}
```

To test the return-existing-active policy, pass `StartOptions::returnExistingActive()`:

```php
public function testReturnExistingActiveStart(): void
{
    WorkflowStub::fake();

    $workflow = WorkflowStub::make(MySignalWorkflow::class, 'order-123');
    $workflow->start();

    $this->assertSame('waiting', $workflow->refresh()->status());

    $second = WorkflowStub::load('order-123');
    $result = $second->attemptStart(StartOptions::returnExistingActive());

    $this->assertTrue($result->accepted());
    $this->assertTrue($result->returnedExistingActive());
    $this->assertSame($workflow->runId(), $result->runId());
}
```

Both policies record durable command and history events, so you can also assert on `WorkflowCommand` and `WorkflowHistoryEvent` rows for deeper verification.

### Testing History Budget

The `HistoryBudget` fields (`history_event_count`, `history_size_bytes`, `continue_as_new_recommended`) are surfaced on the run summary projection and are available through the `RunDetailView`. In your workflow code, the `Workflow` base class exposes these as `$this->historyLength()`, `$this->historySize()`, and `$this->shouldContinueAsNew()`:

```php
use Workflow\V2\Workflow;

final class LongRunningWorkflow extends Workflow
{
    public function handle(): void
    {
        while (true) {
            // ... process work ...

            if ($this->shouldContinueAsNew()) {
                Workflow::continueAsNew($this->carryForwardState());
            }
        }
    }
}
```

To test history budget thresholds, configure the thresholds low and verify the recommendation:

```php
public function testHistoryBudgetRecommendsContinueAsNew(): void
{
    config()->set('workflows.v2.history_budget.continue_as_new_event_threshold', 5);

    WorkflowStub::fake();

    // Run a workflow that produces enough history events to trip the threshold
    $workflow = WorkflowStub::make(ManyActivitiesWorkflow::class, 'budget-test');
    $workflow->start();

    $summary = $workflow->summary();
    $this->assertTrue($summary->continue_as_new_recommended);
}
```

## Legacy `Workflow\WorkflowStub`

:::warning Legacy v1

The examples in this section use the v1 runtime (`Workflow\Workflow`, `use function Workflow\activity`, generator-style `execute()` with `yield`, `Workflow\Models\StoredWorkflow`). Keep these patterns for tests that still exercise v1 workflows during migration. New tests should use `Workflow\V2\WorkflowStub::fake()` with `Workflow\V2\Workflow` and straight-line `handle()` methods as shown above.

:::

### Workflows

You can execute workflows synchronously in your test environment and mock activities and child workflows to define expected behaviors and outputs without running the actual implementations.

```
use function Workflow\activity;
use Workflow\Workflow;

class MyWorkflow extends Workflow
{
    public function execute()
    {
        $result = yield activity(MyActivity::class);

        return $result;
    }
}
```

The above workflow can be tested by first calling `WorkflowStub::fake()` and then mocking the activity.

```
public function testWorkflow()
{
    WorkflowStub::fake();

    WorkflowStub::mock(MyActivity::class, 'result');

    $workflow = WorkflowStub::make(MyWorkflow::class);
    $workflow->start();

    $this->assertSame($workflow->output(), 'result');
}
```

You can also provide a callback instead of a result value to ` WorkflowStub::mock()`.

The workflow `$context` along with any arguments for the current activity will also be passed to the callback.

```
public function testWorkflow()
{
    WorkflowStub::fake();

    WorkflowStub::mock(MyActivity::class, function ($context) {
        return 'result';
    });

    $workflow = WorkflowStub::make(MyWorkflow::class);
    $workflow->start();

    $this->assertSame($workflow->output(), 'result');
}
```

You can assert which activities or child workflows were dispatched by using the `assertDispatched`, `assertNotDispatched`, and `assertNothingDispatched` methods:

```
WorkflowStub::assertDispatched(MyActivity::class);

// Assert the activity was dispatched twice...
WorkflowStub::assertDispatched(MyActivity::class, 2);

WorkflowStub::assertNotDispatched(MyActivity::class);

WorkflowStub::assertNothingDispatched();
```

You may pass a closure to the `assertDispatched` or `assertNotDispatched` methods in order to assert that an activity or child workflow was dispatched that passes a given "truth test". The arguments for the activity or child workflow will be passed to the callback.

```
WorkflowStub::assertDispatched(TestOtherActivity::class, function ($string) {
    return $string === 'other';
});
```

### Skipping Time

By manipulating the system time with `$this->travel()` or `$this->travelTo()`, you can simulate time-dependent workflows. This strategy allows you to test timeouts, delays, and other time-sensitive logic within your workflows.

```
use function Workflow\{activity, timer};
use Workflow\Workflow;

class MyTimerWorkflow extends Workflow
{
    public function execute()
    {
        yield timer(60);

        $result = yield activity(MyActivity::class);

        return $result;
    }
}
```

The above workflow waits 60 seconds before executing the activity. Using `$this->travel()` and `$workflow->resume()` allows us to skip this waiting period in the legacy runtime.

```
public function testTimeTravelWorkflow()
{
    WorkflowStub::fake();

    WorkflowStub::mock(MyActivity::class, 'result');

    $workflow = WorkflowStub::make(MyTimerWorkflow::class);
    $workflow->start();

    $this->travel(120)->seconds();

    $workflow->resume();

    $this->assertSame($workflow->output(), 'result');
}
```

The helpers `$this->travel()` and `$this->travelTo()` methods use `Carbon:setTestNow()` under the hood.

### Activities

Testing activities is similar to testing Laravel jobs. You manually create the activity and then call the `handle()` method.

```
$workflow = WorkflowStub::make(MyWorkflow::class);

$activity = new MyActivity(0, now()->toDateTimeString(), StoredWorkflow::findOrFail($workflow->id()));

$result = $activity->handle();
```

Notice that legacy activities still execute through the runtime `handle()` job method. In `Workflow\V2`, user-defined workflow and activity code also lives in `handle()`.

<!-- Source: docs/how-it-works.md -->

# How It Works

Durable Workflow uses Laravel's queued jobs and event-sourced persistence to create durable coroutines. Workflows suspend through Fiber-backed helper calls for a durable replay contract.

## Runtime

A workflow is a class whose `handle()` method calls straight-line helpers such as `activity()`, `await()`, `timer()`, `sideEffect()`, `child()`, and `all([...])`. Each helper call suspends the workflow until the corresponding durable step completes, then resumes from where it left off with the recorded result.

Every step produces a durable history event. The engine replays that history whenever the workflow wakes up, rebuilding state from the event stream before running the next unexecuted step. That replay is what lets a workflow survive worker restarts, deployments, and machine failures without losing its place.

`WorkflowStub::make()` reserves a public workflow instance id. Starting the workflow creates the first run and the first workflow task. Each run has its own run id; operations such as `signal()`, `cancel()`, and `terminate()` target the current instance run.

## Event Sourcing

Event sourcing builds up the current state from a sequence of saved events rather than saving the state directly. This has several benefits: it provides a complete history of the execution events, and it can be used to resume a workflow if the worker crashes.

## Coroutines

Coroutines are functions whose execution can be suspended and resumed. Durable suspension points are expressed as straight-line Fiber-backed helper calls such as `activity()`, `await()`, `timer()`, and `sideEffect()`.

User workflow code lives in `handle()`, which is an ordinary method that calls those helpers directly. The runtime first checks whether the step already completed durably. If so, the cached result is replayed from history instead of running the step a second time. Otherwise, the runtime queues the next activity, timer, or child work and suspends until that durable step completes or fails.

## Activities

By calling multiple activities, a workflow can orchestrate the results between each of them. The execution of the workflow and the durable steps it schedules are interleaved: the workflow reaches an activity call, suspends until that activity completes, and then continues execution from where it left off.

If a workflow fails, the events leading up to the failure are replayed to rebuild the current state. This allows the workflow to pick up where it left off, with the same inputs and outputs as before, ensuring determinism.

Activities are always durable queued work in v2. There is no in-process local
activity fast path. Ordinary activities can run on any compatible worker, and
[worker sessions](./features/worker-sessions.md) add an explicit lease when a
sequence of activity steps needs the same worker-local resource. If you need a
replay-safe one-shot value without queueing an activity, use
[`sideEffect(...)`](./features/side-effects.md). For the full contract, see
[Activity Execution Model](./features/activity-execution-model.md).

## Execution Guarantees

Workflow code and activity code have different repeat-execution semantics:

- **Workflow code is replayed.** Re-delivering a workflow task rebuilds state from durable history and re-runs deterministic authoring code. Replay does not re-run external side effects.
- **Activities are at-least-once queued work.** A logical activity may be retried, redelivered after lease expiry, or re-observed after worker loss. Duplicate delivery is a normal distributed-systems condition, not a bug by itself.
- **Activity identity is durable.** `activity_execution_id` identifies one logical activity execution across retries and redelivery, while `activity_attempt_id` identifies an individual try. Use `activity_execution_id` as the default remote idempotency key; use `activity_attempt_id` only when a downstream system needs per-attempt correlation.

See [Execution Guarantees and Idempotency](./constraints/execution-guarantees.md),
[Activity Execution Model](./features/activity-execution-model.md), and
[Failures and Recovery](./failures-and-recovery.md) for the full v2 contract.

## Queues

Queued jobs are background processes that run at a later time. Laravel supports queues via Amazon SQS, Redis, or a relational database. Workflows and activities are both queued jobs, but they behave a little differently. A workflow is dispatched multiple times during normal operation: it runs, dispatches one or more activities, and then exits until the activities complete. An activity is an at-least-once queued task: the common case is one successful attempt, but retry, lease expiry, or worker loss can cause the same logical activity execution to be delivered more than once.

## Example

```php
use Workflow\V2\Workflow;
use function Workflow\V2\{activity, all};

class MyWorkflow extends Workflow
{
    public function handle(): array
    {
        return [
            activity(TestActivity::class),
            activity(TestOtherActivity::class),
            fn () => all([
                fn () => activity(TestParallelActivity::class),
                fn () => activity(TestParallelOtherActivity::class),
            ]),
        ];
    }
}
```

## Sequence Diagram

This sequence diagram shows how a workflow progresses through a series of activities, both serial and parallel.

1. The workflow starts by getting dispatched as a queued job.
2. The first activity, `TestActivity`, is then dispatched as a queued job. The workflow job then exits. Once `TestActivity` has completed, it saves the result to the database and returns control to the workflow by dispatching it again.
3. At this point, the workflow enters the event sourcing replay loop. This is where it goes back to the database and looks at the event stream to rebuild the current state. This is necessary because the workflow is not a long running process. The workflow exits while any activities are running and then is dispatched again after completion.
4. Once the event stream has been replayed, the workflow continues to the next activity, `TestOtherActivity`, and starts it by dispatching it as a queued job. Again, once `TestOtherActivity` has completed, it saves the result to the database and returns control to the workflow by dispatching it as a queued job.
5. The workflow then enters the event sourcing replay loop again, rebuilding the current state from the event stream.
6. Next, the workflow starts two parallel activities, `TestParallelActivity` and `TestOtherParallelActivity`. Both activities are dispatched. Once they have completed, they save the results to the database and return control to the workflow.
7. Finally, the workflow enters the event sourcing replay loop one last time to rebuild the current state from the event stream. This completes the execution of the workflow.

## Determinism

Because history is replayed on every wake-up, workflow code must produce the same commands given the same history. Read [Constraints](./constraints/overview.md) for the authoring rules and the helpers Durable Workflow exposes (`Workflow\now()`, `sideEffect()`, `getVersion()`, and similar) for situations where code would otherwise be non-deterministic.

<!-- Source: docs/failures-and-recovery.md -->

# Failures and Recovery

Before you triage a failure, keep the core execution contract in mind:

- workflow tasks recover by replaying committed history
- activity execution is at-least-once and may be observed more than once
- lease expiry and redelivery are normal recovery paths, not proof that the
  previous worker never executed the side effect

Read [Execution Guarantees and Idempotency](./constraints/execution-guarantees.md)
first when you need the precise semantics behind retries, redelivery, and
durable outcomes.

## Handling Exceptions

When an activity throws an exception, the workflow won't immediately be informed. Instead, it waits until the number of `$tries` has been exhausted. The system will keep retrying the activity based on its retry policy. If you want the exception to be immediately sent to the workflow upon a failure, you can set the number of `$tries` to 1.

```php
use Exception;
use Workflow\V2\Activity;

class MyActivity extends Activity
{
    public int $tries = 1;

    public function handle(): void
    {
        throw new Exception();
    }
}
```

```php
use Exception;
use function Workflow\V2\activity;
use Workflow\V2\Workflow;

class MyWorkflow extends Workflow
{
    public function handle(): void
    {
        try {
            $result = activity(MyActivity::class);
        } catch (Exception) {
            // handle the exception here
        }
    }
}
```

## Non-retryable Exceptions

In certain cases, you may encounter exceptions that should not be retried. These are referred to as non-retryable exceptions. When an activity throws a non-retryable exception, the workflow will immediately mark the activity as failed and stop retrying.

```php
use Workflow\V2\Activity;
use Workflow\Exceptions\NonRetryableException;

class MyNonRetryableActivity extends Activity
{
    public function handle(): void
    {
        throw new NonRetryableException('This is a non-retryable error');
    }
}
```

## Recovery Process

The general process to fix a failing activity is:

1. Check the logs for the activity that is failing and look for any errors or exceptions that are being thrown.
2. Identify the source of the error and fix it in the code.
3. Deploy the fix to the server where the queue is running.
4. Restart or roll the relevant workers so they pick up the new code and can safely reclaim work.
5. Wait for the activity to retry or for repair/redelivery to hand the durable task to a healthy worker.
6. Verify the durable outcome in Waterline, history export, or the server API instead of assuming that one worker log line is the authority.
7. If the activity continues to fail, repeat the process until the issue is resolved.

This allows you to keep the workflow in a running status even while an activity is failing. After you fix the failing activity, the workflow will finish in a completed status. A workflow with a failed status means that all activity `$tries` have been exhausted and the exception wasn't handled.

## Workflow Timeout Enforcement

When `StartOptions::withExecutionTimeout()` or `StartOptions::withRunTimeout()` is set, the engine records a deadline on the workflow run. The execution deadline spans the entire logical workflow (including continue-as-new runs), while the run deadline resets with each new run.

If a deadline has passed when the engine starts a workflow task, the run is closed immediately:

- All open activity executions, timers, and outstanding tasks are cancelled with typed history events (`ActivityCancelled`, `TimerCancelled`).
- A `WorkflowFailure` row is recorded with `failure_category = timeout` and `propagation_kind = timeout`.
- A `WorkflowTimedOut` history event is recorded with `timeout_kind` set to `execution_timeout` or `run_timeout`.
- The run status becomes `failed` with `closed_reason = timed_out`.
- Parent workflows waiting on the timed-out child are notified.

The background task watchdog also scans for non-terminal runs with expired deadlines that have no open workflow task (for example, a run waiting on an activity or timer when the deadline passes). When it finds one, it creates a workflow task so the executor can detect and enforce the timeout on the next pass.

Waterline surfaces `failure_category` in the exceptions table as a dedicated **Category** column and in timeline failure detail entries. History exports include `failure_category` in the `failures[*]` array. Final v2 writes this classification when the failure is recorded; imported v1 rows that cannot be classified remain visible as unclassified diagnostics.

## Activity Retries

`Workflow\V2\Activity` defaults to `$tries = 1`, so an activity failure is sent back to the workflow immediately unless the activity opts into retry attempts.

```php
use RuntimeException;
use Workflow\V2\Activity;

class ChargeCard extends Activity
{
    public int $tries = 3;

    public function backoff(): array
    {
        return [5, 30];
    }

    public function handle(): string
    {
        throw new RuntimeException('temporary gateway failure');
    }
}
```

When a retryable activity throws before `$tries` is exhausted, the engine closes the current `activity_attempts` row as runtime state, returns the `activity_executions` row to `pending`, records a typed `ActivityRetryScheduled` history event for the failed try, and creates a new durable activity task with `available_at` set from the `backoff()` policy. The workflow stays waiting on that same activity execution and is not resumed with the exception until the final retryable attempt fails.

The retry task records `retry_of_task_id`, `retry_after_attempt_id`, `retry_after_attempt`, and `retry_backoff_seconds` in its payload so Waterline can explain why the task is scheduled. Selected-run detail rebuilds the failed attempt in `activities[*].attempts` from typed activity history first, shows `ActivityRetryScheduled` in the timeline, and reports retrying activity counts through `operator_metrics.activities.retrying`, `operator_metrics.activities.failed_attempts`, and `operator_metrics.backlog.retrying_activities`.

`Workflow\Exceptions\NonRetryableExceptionContract` still short-circuits the retry policy: throwing a non-retryable exception fails the activity execution immediately and resumes the workflow with the exception.

### Activity execution identity and idempotency

Retry is not the only reason the same logical activity can be observed more than once. Lease expiry, worker loss, delayed completion reporting, and redelivery can all produce another attempt or a stale completion report for the same durable activity execution.

- `activity_execution_id` identifies the logical activity across retries and redelivery. Use it as the default idempotency key for remote side effects.
- `activity_attempt_id` identifies one specific try of that logical activity. Use it only when a downstream system must distinguish separate attempts.
- A late completion or failure report from a superseded attempt is normal stale-attempt behavior, not proof that the engine committed the same attempt twice.

When operators investigate a late completion after lease expiry:

- trust Waterline, history export, or the server API for which attempt won the
  durable race
- do not assume a rejected late completion means the remote side effect did not
  happen
- check the external system by its idempotency key before forcing manual retry
  or repair

The safest default is to make the remote side effect idempotent under
`activity_execution_id`, then let the durable outcome tell you whether the
engine accepted the report for that specific attempt.

### Non-retryable failure markers

When an activity or workflow throws an exception that implements `Workflow\Exceptions\NonRetryableExceptionContract`, the engine records a `non_retryable = true` flag on the `WorkflowFailure` row and in the typed history event payload (`ActivityFailed`, `WorkflowFailed`, `UpdateCompleted`). This durable marker communicates to operators, external workers, and tooling that the failure is permanent — retrying the same operation will not succeed.

The flag flows through the full visibility stack:

- **Failure rows**: `workflow_failures.non_retryable` boolean column.
- **History events**: `non_retryable` field in the typed event payload.
- **Failure snapshots**: `non_retryable` included in `FailureSnapshots::forRun()`.
- **Run detail view**: `non_retryable` in the exceptions array.
- **Timeline entries**: `non_retryable` in failure detail metadata.
- **History exports**: `non_retryable` in the `failures[*]` array.
- **Waterline**: a "non-retryable" badge next to the failure category in the exceptions table and timeline.
- **External worker bridge**: the `complete()` command payload accepts `non_retryable` so external workflow workers can report non-retryable failures without requiring the host process to resolve the throwable class.

For failures that do not implement the contract, `non_retryable` is `false` by default. Final v2 records that durable marker at failure time, so declare the contract before the failure is written when operators or SDKs need to distinguish permanent failures from retryable ones.

```php
use Workflow\Exceptions\NonRetryableExceptionContract;

class PaymentDeclinedException extends \RuntimeException implements NonRetryableExceptionContract
{
    // This failure will be marked as non-retryable in the durable record.
}
```

## Workflow-Level Retry

Durable Workflow v2 does **not** support automatic workflow-level retry. When a workflow run fails — whether from an unhandled exception, a structural limit, or a timeout — the run is terminal. The engine does not automatically start a new run of the same workflow instance.

This is an intentional design choice:

- **Activities already have retry.** Activity retry policies with configurable `$tries`, `backoff()`, and non-retryable exceptions handle transient failures at the right granularity.
- **Workflow replay is the recovery primitive.** If a workflow task encounters a transient infrastructure failure (database error, worker crash), the durable task system re-dispatches the task, and replay resumes from committed history — no new run needed.
- **Continue-as-new handles long-lived workflows.** Workflows that need fresh state or history compaction use `continueAsNew()` as an explicit workflow-level restart.
- **Repair handles stuck runs.** The `repair()` command and automatic worker-loop repair recover runs where durable task transport was lost.

If your application needs workflow-level retry semantics, model them explicitly:

```php
use function Workflow\V2\activity;
use Throwable;
use Workflow\V2\Workflow;

class RetryableWorkflow extends Workflow
{
    public function handle(string $orderId): void
    {
        try {
            activity(ProcessOrderActivity::class, $orderId);
        } catch (Throwable $e) {
            // Record the failure, then start a new workflow
            // for retry-at-workflow-level scenarios.
            activity(NotifyFailureActivity::class, $orderId, $e->getMessage());
        }
    }
}
```

## Related Guides

- [Execution Guarantees and Idempotency](./constraints/execution-guarantees.md)
  explains the replay, retry, lease-expiry, and redelivery contract that shapes
  every recovery path on this page.
- [Monitoring](./monitoring.md) explains where Waterline, history export,
  worker logs, and runtime telemetry surface the failure facts described here.

<!-- Source: docs/monitoring.md -->

# Monitoring

[Waterline](https://github.com/durable-workflow/waterline) is the operator UI
and API for workflow state. Its delivery follows the selected runtime boundary:
Cloud includes Managed Waterline, while a self-hosted operator deploys
Waterline separately when that surface is wanted.

Waterline is one operator product with three consumption surfaces:

- **Embedded mode** installs the Composer package in the Laravel application
  that owns the workflows and reads that application's durable state in
  process.
- **Self-hosted service mode** runs the published
  `durableworkflow/waterline` image and reads state owned by the
  [standalone server](./polyglot/server.md) through the PHP SDK and public
  server API. The Server distribution does not bundle or operate Waterline.
- **Cloud Managed Waterline** is the namespace-scoped operator experience
  included with [Durable Workflow Cloud](./polyglot/cloud-control-plane.md).
  Cloud operates Managed Waterline and the namespace runtime behind it.

The embedded and self-hosted service surfaces expose the same core Waterline UI
and `/waterline/api/...` operator route families. Cloud provides the operator
capabilities through its managed surface instead of a customer-deployed
Waterline route host. None of the surfaces merge runtime state: each view is
limited to runs owned by its runtime and namespace. For self-hosted Server
operations, the native API, CLI, and SDK operator surfaces remain available
whether or not you deploy Waterline. Use the
[Server API Reference](./polyglot/server-api-reference.md) for those native
routes and the
[Waterline Operator API Reference](./waterline-operator-api.md) for
Waterline's routes and response contracts.

Durable Workflow has two observability planes:

| Plane | Source of truth | Typical questions |
| --- | --- | --- |
| Durable state | The owning runtime's workflow database and API, Waterline projections, and history export | Did the workflow start? Which run is current? Which signal, update, timer, activity, retry, or failure was committed? Which operator action is safe now? |
| Worker/runtime telemetry | Queue worker logs, SDK metrics recorders, Prometheus/OpenMetrics endpoints, and application traces | Are workers polling? How long do tasks take? Is an exporter configured? Did custom application metrics leave the worker process? |

Waterline intentionally does not replace worker metrics. If a custom metric
was recorded in activity or worker code, scrape the worker's telemetry
endpoint. Use Waterline to correlate that runtime signal with the durable
workflow history and current run state.

Worker and client setup also remains separate from the operator surface. Use
the generated [PHP SDK API reference](https://php.durable-workflow.com/api/),
[Python SDK API reference](https://python.durable-workflow.com/), or
[Rust SDK API reference](https://rust.durable-workflow.com/) alongside the
language guides when connecting application clients and workers to the runtime
that owns the namespace.

When worker telemetry shows repeated claims, late completion races, or stuck
leases, read [Execution Guarantees and Idempotency](./constraints/execution-guarantees.md)
alongside this guide. That contract separates at-least-once transport
uncertainty from duplicate durable outcomes so duplicate-looking evidence does
not turn into the wrong operational conclusion.

### Dashboard View

The dashboard shows running totals, recent-run counters, and fleet-wide
metrics so you can tell at a glance whether work is flowing, stalling, or
failing.

Use the [Operator Operating Envelope](./operator-operating-envelope.md) when
you need the rollout and runbook contract for those facts: which diagnostics
block traffic, which are advisory, how queue-health facts split between
Waterline and worker telemetry, and how to verify rebuild, export, and archive
paths.

### Workflow View

The workflow detail view shows the durable timeline for a single run: the
activities, signals, timers, and child workflows that happened in order,
each with its inputs, outputs, and timing.

## Waterline deployment and access

### Embedded Laravel

Install Waterline into your Laravel application alongside the workflow
package and run its migrations. See
[durable-workflow/waterline](https://github.com/durable-workflow/waterline)
for the full installation and configuration guide.

Embedded mode uses the host application's database connection, route
middleware, authentication gate, and workflow package. Continue to operate the
Laravel application, its queue workers, scheduler, migrations, and Waterline
assets as one deployment boundary. The
[Waterline Operator API Reference](./waterline-operator-api.md#installation)
includes the current Composer command and asset-publishing step.

### Waterline service

The published image contains its own PHP and Laravel runtime. It does not need
PHP, Composer, or the workflow package on the container host, and it never
connects to the standalone server's database.

This example binds Waterline to loopback on host port `8080`, persists its own
UI state in a named volume, and connects it to one server namespace:

```bash
export WATERLINE_SERVER_ENDPOINT=https://workflow.example.com
export WATERLINE_SERVER_TOKEN=replace-with-a-server-token

docker run --detach \
  --name waterline \
  --restart unless-stopped \
  --publish 127.0.0.1:8080:8080 \
  --volume waterline-data:/data \
  --env WATERLINE_SERVER_ENDPOINT \
  --env WATERLINE_SERVER_TOKEN \
  --env WATERLINE_NAMESPACE=orders \
  --env WATERLINE_ACCESS_MODE=read_only \
  --env WATERLINE_ALLOW_UNAUTHENTICATED=true \
  --env APP_URL=https://waterline.example.com \
  durableworkflow/waterline:2.0.0
```

Open `/waterline` through the URL represented by `APP_URL`. The image listens
on container port `8080`; set `PORT` only when you intentionally change that
internal port. `WATERLINE_PATH` changes the default `waterline` URL prefix.
The repository also publishes a
[Docker Compose service definition](https://github.com/durable-workflow/waterline/blob/main/deploy/docker-compose.service.yml)
with the same connection boundary.

Service mode has two independent authentication layers:

1. `WATERLINE_SERVER_TOKEN` is the bearer credential Waterline uses when the
   PHP SDK calls `WATERLINE_SERVER_ENDPOINT`. Workflow, worker, queue, and
   schedule observation needs a server operator role; server health and
   operator metrics may need an admin role.
2. Browser and Waterline API access is the deployment's front-door boundary.
   The self-contained image has no host Laravel user directory.
   `WATERLINE_ALLOW_UNAUTHENTICATED=true` therefore belongs only behind an
   authenticating reverse proxy or on a private interface such as the loopback
   binding above. Keep it `false` unless that surrounding authentication
   boundary is in place.

Set `WATERLINE_NAMESPACE` to the same namespace operators intend to inspect.
Waterline sends it on every SDK request. `WATERLINE_ACCESS_MODE=read_only` is
the default and blocks mutating actions in Waterline; use `operator` only with
a server token authorized for the required commands.

The `/data` volume holds Waterline-owned saved views, display preferences, and
Laravel runtime state. With the default settings it contains a file-backed
SQLite database at `/data/waterline.sqlite`. It never contains the server's
workflow history. Use `DATABASE_URL` or the ordinary `DB_*` settings when
Waterline's own state must live in MySQL or PostgreSQL.

#### Health and metrics

Use these probes for different questions:

| Surface | What it proves |
| --- | --- |
| `GET /up` | The Waterline HTTP process started and can answer requests. The image's Docker health check uses this route. |
| `GET /waterline/api/v2/health` | Waterline can assemble namespace-scoped server health, worker registration, and task-queue evidence through the PHP SDK. |
| `GET /waterline/api/stats` | Dashboard totals and the server-backed operator summary used by Waterline. |
| `GET /api/system/health` and `GET /api/system/operator-metrics` on the server | The native server health and operator contracts, independent of Waterline. |

Treat `/waterline/api/stats` as operator JSON, not as a Prometheus scrape
endpoint. Worker SDK metrics, logs, traces, and custom application telemetry
still come from worker processes. A healthy `/up` with an unavailable
Waterline health or stats response points to the server connection,
authorization, namespace, or SDK capability boundary rather than a failed
Waterline process.

#### Workflow visibility and operator actions

In service mode, list views, selected-run detail, history export, schedules,
worker status, task-queue evidence, signals, updates, queries, repair, cancel,
terminate, and archive are projected from the configured standalone server
through the PHP SDK. The Waterline route shapes remain the ones documented in
the [Waterline Operator API Reference](./waterline-operator-api.md); the
underlying server contracts remain documented in the
[Server API Reference](./polyglot/server-api-reference.md).

Waterline only shows the configured namespace and the runs owned by the
connected server. Embedded runs remain visible through their embedded
Waterline deployment, while server-managed runs remain visible through the
service deployment or the server's native surfaces. Changing the Waterline
backend does not migrate or combine runs.

#### Troubleshooting boundaries

| Symptom | Boundary to check |
| --- | --- |
| Container exits before serving `/up` | Check the required `WATERLINE_SERVER_ENDPOINT`, writable `/data`, valid `PORT`, database settings, and bounded startup migration logs. |
| Waterline returns `401` or `403` from a server-backed view | Check `WATERLINE_SERVER_TOKEN` and its server role. A Waterline front-door denial is a separate proxy or `WATERLINE_ALLOW_UNAUTHENTICATED` issue. |
| A mutating route returns `waterline_read_only` | Keep the deployment read-only or explicitly set `WATERLINE_ACCESS_MODE=operator`; the server token must still authorize the command. |
| Expected workflows are absent | Confirm `WATERLINE_NAMESPACE`, the server endpoint, and which runtime accepted the workflow start. Waterline does not search other namespaces or embedded runtimes. |
| `/up` passes but health, stats, or a view reports an unavailable capability | Verify the server endpoint directly with its API or CLI, then check token roles and that the published Waterline/PHP SDK tuple exposes the required method. |
| A service-catalog route reports `backend_capability_unavailable` | Service mode does not mirror Waterline's embedded cross-namespace service catalog. Use the connected server's native service endpoints and API for that capability. |
| A custom metric is missing from Waterline | Inspect the worker's metrics exporter. Waterline reports durable operator facts, not arbitrary process metrics. |

### Cloud Managed Waterline

Cloud customers open Cloud Managed Waterline from the namespace-scoped managed
operator surface. They do not deploy or configure Waterline, Server, PHP, or
internal endpoints, and they do not maintain a second Waterline login. The
self-hosted image, `WATERLINE_*` settings, Server token, database, and
front-door authentication instructions above do not apply to this path.

The customer's Cloud authentication establishes the operator identity.
Authorization across the Cloud organization, project, environment, and
namespace determines which managed Waterline scope that identity can open.
Within that boundary, operators can use workflow lists and search, run detail
and durable history, namespace health, and the operator actions supported for
their role. The managed surface remains limited to the selected namespace; it
does not combine data from other environments or namespaces.

Mutations are role-gated. Cloud attributes each supported operator mutation to
the authenticated Cloud actor in its audit data, while the resulting durable
workflow transition remains visible in workflow history. For example, a
successful archive is attributed in Cloud audit data and records the durable
`WorkflowArchived` history event. Operators never need a private runtime
credential or knowledge of the managed runtime's internal deployment to use
this surface.

## List and detail API

Waterline's list views (`/waterline/api/flows/{bucket}`) and selected-run
detail endpoint (`/waterline/api/flows/{id}`) return typed JSON contracts
that you can consume directly from your own dashboards or scripts. The
[Waterline Operator API Reference](./waterline-operator-api.md) documents the
endpoint list, selected-run field families, history export, actionability,
schedules, saved views, preferences, and operator-action contract.

### Actionability Contract

Waterline annotates list rows, selected-run detail responses, and history
exports with a versioned actionability contract. Consumers should treat
`actionability_contract.schema = waterline.actionability` and
`actionability_contract.version = 1` as the contract identifier for the fields
below.

Run-level `actionability` answers whether the selected run can be repaired:

| Field | Meaning |
| --- | --- |
| `repair_state` | One of `repairable`, `blocked`, `not_needed`, or `unknown`. |
| `repairable` | Boolean shorthand for `repair_state = repairable`. |
| `blocked_reason` | Stable reason code when `repair_state = blocked`. |
| `status_bucket` | The Waterline bucket that shaped the run-level decision. |
| `closed_reason` | Durable close reason when the run is closed. |
| `task_problem` | Whether Waterline saw a task-level problem on the run. |
| `diagnostic_only_evidence` | True when at least one child evidence row is informative but not a resume source. |

Evidence rows under `activities`, `waits`, `timers`, `exceptions`, `logs`, and
timeline/export entries can also include their own `actionability` block:

| Field | Meaning |
| --- | --- |
| `state` | `actionable` when the row is a valid repair source, otherwise `diagnostic_only`. |
| `repair_source` | True only for rows backed by a repairable source authority. |
| `diagnostic_only` | True when the row must not be used as a resume source. |
| `history_authority` | Source authority, such as `typed_history`, `mutable_open_fallback`, `failure_row_fallback`, or `unsupported_terminal_without_history`. |
| `history_unsupported_reason` | Stable reason code for unsupported fallback history. |

Automation should gate repair, resume, and replay affordances from
`actionability.repair_state`, `actionability.repairable`, and row-level
`actionability.repair_source`. A row with `diagnostic_only = true` is never a
durable resume source, even when it contains useful failure or fallback
metadata. Rows with `history_authority = unsupported_terminal_without_history`
are diagnostic evidence only; they explain why a run is blocked, but they do
not prove enough typed history to rebuild progress safely.

## Control-plane actions from Waterline

Operators can cancel, terminate, repair, and archive workflows directly
from the detail view. Each action maps to a `POST` on the same run id and
returns either `200` with the resulting state or `409` when the action is
not valid for the run's current state.

In service mode, Waterline forwards supported commands through the PHP SDK.
Both `WATERLINE_ACCESS_MODE=operator` and a server credential authorized for
the command are required for mutations. Operators can always use the server
API or CLI directly when Waterline is not deployed.

In Cloud Managed Waterline, Cloud role and namespace authorization gate each
supported mutation, and the authenticated Cloud identity supplies its audit
attribution. Customers do not configure a Waterline-to-Server credential for
the managed surface.

## Related Guides

- [Execution Guarantees and Idempotency](./constraints/execution-guarantees.md)
  explains the replay, retry, lease-expiry, and durable-outcome contract that
  shapes operator evidence.
- [Operator Operating Envelope](./operator-operating-envelope.md) ties health,
  queue state, rebuild, export, archive, and topology expectations into one
  operator contract.
- [Failures and Recovery](./failures-and-recovery.md) explains retry exhaustion,
  non-retryable failures, timeouts, and repair behavior behind the dashboard
  facts.
- [AI-Assisted Development](./ai-assisted-development.md) names the Waterline,
  CLI, MCP, and LLM-readable contracts that agents should use when diagnosing
  workflow state.

<!-- Source: docs/deployment.md -->

# Self-Hosting Deployments

Durable Workflow v2 supports several self-hosted shapes. Pick the smallest path
that matches the environment you operate, then keep the image, database, cache,
auth, readiness, and upgrade contract explicit.

This guide covers the standalone server distribution. If you run the Laravel
package embedded in your own app, use the package installation and configuration
pages instead. Durable Workflow Cloud is a separate managed-service choice in
which Cloud operates the runtime, persistence, placement, and recovery; a
self-hosted Server is never attached to Cloud. See
[Cloud Managed Runtime](/docs/polyglot/cloud-control-plane). The current
Cloud contract is a single-region managed runtime and recovery boundary; it
does not promise multi-region replication, automatic regional failover, or
failback.

The self-hosted Server distribution does not bundle or operate Waterline.
Deploy Waterline separately only when you want its operator UI, and connect it
to a Server-owned namespace using the
[Waterline deployment and monitoring guide](/docs/monitoring/#waterline-service).
Cloud instead includes Managed Waterline for its managed namespace.

## Deployment support matrix

| Path | Start from | Supported for | Not promised by this path | Commercial support starts when |
| --- | --- | --- | --- | --- |
| Local development and internal non-production | [`docker-compose.published.yml`](https://github.com/durable-workflow/server/blob/main/docker-compose.published.yml) with `DW_SERVER_TAG=2.0.0` or `DW_SERVER_IMAGE=durableworkflow/server:2.0.0` | One developer machine, LAN demos, shared staging, SDK and worker integration tests | Internet-facing production, durable backup guarantees, strict secret rotation, multi-node failover | You want help turning a working dev stack into a production runbook |
| Single-node production | [`docker-compose.published.yml`](https://github.com/durable-workflow/server/blob/main/docker-compose.published.yml) with a production env file, MySQL and Redis volumes, role-scoped tokens, backups, TLS through a reverse proxy, and pinned image tags or digests | One VM, VPS, or internal Docker host with persistent workflow state and a simple operational model | Host-level HA, automatic database failover, multi-region recovery, zero-downtime major topology changes | The deployment carries production traffic and you want review of backup, restore, auth, TLS, upgrade, or rollback procedures |
| Small clustered deployment | Published `durableworkflow/server` or `ghcr.io/durable-workflow/server` images using the [Compose recipe](https://github.com/durable-workflow/server/blob/main/docker-compose.published.yml) as the container/process template, with 2-3 API nodes, shared external MySQL/PostgreSQL, shared Redis, independently scaled workers, and exactly one scheduler/maintenance runner | Horizontal API and worker capacity when one node is no longer enough; rolling upgrades when every guarantee in the [rolling-upgrade contract](/docs/rolling-upgrades) holds | SQLite clustering, Redis-less multi-node mode, duplicate schedulers as a steady-state topology, active/active multi-writer databases, self-hosted hands-free regional failover, broad "five-nines" or "zero-downtime" SLA promises, and self-serve single-region HA failover until its [release-evidence gate](#release-evidence-status) passes | You need sizing, failure-domain, rollout, or recovery planning across more than one host, or you intend to claim the gated single-region HA behavior |
| Helm chart for Kubernetes | The [Server-owned Helm chart](https://github.com/durable-workflow/server/tree/main/k8s/helm/durable-workflow) from `oci://ghcr.io/durable-workflow/charts/durable-workflow`, with your external database, Redis, ingress, and secret management | A repeatable production install and upgrade path for the chart's server, worker, singleton scheduler, bootstrap, service, probes, and policy resources | Bundled persistence, provider-managed infrastructure, active/active multi-region, custom operators, and self-serve single-region HA failover until its [release-evidence gate](#release-evidence-status) passes | You need provider-specific architecture, capacity, recovery, or changes outside the chart's published values contract |
| Raw Kubernetes manifests | The server repository [`k8s/`](https://github.com/durable-workflow/server/tree/main/k8s) manifests, using published server images and your existing database, Redis, ingress, and secret management | Teams that already operate Kubernetes and want inspectable manifests for API, worker, scheduler, bootstrap, service, probes, config, and secrets | The separate Helm lifecycle, managed-Kubernetes provider validation, active/active multi-region, custom operators, environment-specific storage/networking/security decisions, and self-serve single-region HA failover until its [release-evidence gate](#release-evidence-status) passes | You need overlays, managed-cluster validation, provider-specific production planning, or intend to claim the gated single-region HA behavior |
| Active/passive multi-region evaluation | A validated single-node or small-cluster deployment per region, plus asynchronous database replication from active to standby and a reviewed failover/failback runbook | Support-led architecture evaluation and environment-specific rehearsal for regional disaster recovery | A proven self-serve 2.0 contract, active/active multi-region, automatic or hands-free regional failover, synchronous cross-region replication (RPO=0), cross-region active visibility or federated search, or region-pinned task queues as an engine-enforced routing axis | Before relying on cross-region replication, failover, failback, RPO, or RTO in production |
| Support-led topologies | A reviewed design based on your environment | Self-hosted active/passive or active/active multi-region, hands-free regional failover, RPO=0 cross-writer replication, duplicate scheduler runners as a steady-state topology, bespoke security/networking, private SLOs, custom overlays, migration planning | Self-serve copy/paste operation | The topology itself is part of the product risk |

The public distribution is intentionally optimized for local development,
single-node production, and small clustered deployments. Kubernetes manifests
are provided for teams that already operate Kubernetes. Active/passive
multi-region material is available as a support-led evaluation guide (see
[Active/passive multi-region](#activepassive-multi-region) below), not as a
proven self-serve 2.0 contract.
Single-region HA failover — managed-database failover, managed-Redis failover,
API-node loss, worker loss, and scheduler-runner restart inside one region —
is currently support-led while its exact-release evidence gate remains closed
(see
[Single-region high availability and failover](#single-region-high-availability-and-failover)
below). For self-hosted server deployments, active/active multi-region,
automatic regional failover, duplicate scheduler runners as a steady-state
topology, and provider-specific managed-Kubernetes validation remain
support-led because they depend on your database, cache, networking, security,
runner, and upgrade choices. The published Helm chart is a self-serve packaging
path within those same operator-owned boundaries. Cloud currently provides
single-region managed operation and recovery, not multi-region replication or
regional failover; see
[Cloud Managed Runtime](/docs/polyglot/cloud-control-plane). See the
[support boundary](/docs/support) for the commercial support model.

## Production recovery deliverables

Calling a topology "production-ready" means publishing the recovery packet for
that topology, not just starting the containers. Keep the latest evidence in
the same runbook as the deployment commands:

| Path | Minimum published recovery packet |
| --- | --- |
| Single-node production | Backup schedule, pinned image or digest, env/config snapshot location, maximum accepted restore lag, and the latest successful restore rehearsal timestamp plus verification evidence. |
| Small clustered deployment | The single-node packet plus the expected impact of losing one API node, one worker node, the scheduler/maintenance runner, Redis, or the shared database; the worker re-registration steps after restore; and the documented rolling-upgrade or stop-the-world posture for the current release. |
| Helm chart for Kubernetes | The clustered packet plus chart version, values revision, image digest, cluster-specific ingress and secret owners, and the latest chart upgrade/rollback rehearsal. |
| Raw Kubernetes manifests | The clustered packet plus the cluster-specific storage, secret, ingress, and rollout owners that must be restored or re-applied before traffic is declared healthy again. |
| Single-region HA evaluation (support-led while the release-evidence gate is closed) | The clustered or raw-manifest packet plus rehearsal evidence for managed-database failover, managed-Redis failover, API-node loss, worker loss, and scheduler-runner restart, each completing within the recovery target published in the [Single-region HA contract](#single-region-high-availability-and-failover) without acknowledged-write loss. |
| Active/passive multi-region evaluation (support-led) | The per-region packet plus the database replication and fencing design, measured replication lag, operator-owned RPO/RTO, promotion and failback runbooks, and environment-specific rehearsal evidence. |

If you cannot produce that packet on demand, treat the environment as staging
until the recovery contract is written down and rehearsed. The
[Operator Operating Envelope](/docs/operator-operating-envelope) defines
the restore order, verification pass, and rehearsal cadence those packets must
follow.

## Security, Data, And Audit Posture

Self-hosted Durable Workflow deployments inherit most security controls from
the environment you operate. Publish these facts in the same release or runbook
packet as the image tag, migration plan, and recovery evidence:

| Posture area | Honest release statement |
| --- | --- |
| Data handling | Workflow arguments, results, history, memos, search attributes, visibility labels, command context, audit rows, exception messages, and operator notes can contain customer application data. Treat search attributes and labels as operator-visible metadata, not secret storage. |
| Encryption | Use TLS for every production HTTP surface. At-rest encryption comes from your database, object storage, filesystem, queue, cache, and secret manager. The workflow package and server do not automatically encrypt each payload field. |
| Compliance | The open-source package and self-hosted server provide controls and audit evidence, not a compliance certification by themselves. Claims such as SOC 2, HIPAA, PCI, ISO, or FedRAMP belong to your own program unless a hosted offering documents otherwise. |
| Audit logs | Workflow commands, schedule audit events, history export metadata, and service-call records provide durable operational evidence. They are not a complete SIEM, DLP system, immutable external ledger, or legal-hold system unless you add those components. |
| Support | Role-scoped credentials, TLS termination, backups, restore rehearsal, and narrow self-serve topologies are documented here. Advanced identity, mTLS rollout, private networking, custom policy engines, provider compliance, and bespoke topology review are support-led unless public docs say otherwise. |

Network posture must be explicit:

- **Webhook ingress:** document the public endpoint, auth method, replay or
  idempotency strategy, timeout, payload limit, trusted proxy-header
  configuration, and secret rotation plan.
- **Worker-to-backend traffic:** use TLS verification, role-scoped worker
  credentials, namespace headers, private networking or mTLS when workers cross
  an untrusted network, and rotation that does not grant operator capabilities
  to worker tokens.
- **Operator surfaces:** place Waterline, standalone-server operator APIs, CLI
  automation endpoints, and custom admin panels behind authenticated sessions
  or role-scoped service credentials, with CSRF protection for browser
  sessions and documented proxy/TLS boundaries.

## Published images

Use published images for self-hosted server deployments:

- Docker Hub: `durableworkflow/server:2.0.0`
- GitHub Container Registry: `ghcr.io/durable-workflow/server:2.0.0`
- Digest pinning: `durableworkflow/server@sha256:...` or
  `ghcr.io/durable-workflow/server@sha256:...`

Use mutable tags only for local experiments. Production env files should pin a
specific version tag or digest so upgrade and rollback steps are auditable.

## Local development and internal non-production

Use the published-image Compose recipe when you want a source-free stack backed
by MySQL and Redis:

```bash
curl -fsSLO https://raw.githubusercontent.com/durable-workflow/server/main/docker-compose.published.yml

export DW_SERVER_TAG=2.0.0
export DW_AUTH_TOKEN=dev-token

docker compose -f docker-compose.published.yml up -d --wait
```

Verify the API, readiness, cluster discovery, and worker registration:

```bash
curl http://localhost:8080/api/health
curl http://localhost:8080/api/ready
curl -H "Authorization: Bearer $DW_AUTH_TOKEN" \
  http://localhost:8080/api/cluster/info

curl -X POST http://localhost:8080/api/worker/register \
  -H "Authorization: Bearer $DW_AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -H "X-Namespace: default" \
  -H "X-Durable-Workflow-Protocol-Version: 1.0" \
  -d '{"worker_id":"compose-worker","task_queue":"compose","runtime":"python"}'
```

This path is safe for development and internal staging. It is not a production
security boundary: the example uses one compatibility token, default service
passwords, local named volumes, and no TLS.

## Single-node production

Use the same Compose artifact with production configuration outside source
control:

```env
DW_SERVER_IMAGE=durableworkflow/server:2.0.0
SERVER_PORT=8080
APP_ENV=production
APP_DEBUG=false

DB_DATABASE=durable_workflow
DB_USERNAME=workflow
DB_PASSWORD=replace-with-random-password
DB_ROOT_PASSWORD=replace-with-random-root-password

DW_AUTH_DRIVER=token
DW_AUTH_BACKWARD_COMPATIBLE=false
DW_WORKER_TOKEN=replace-with-worker-token
DW_OPERATOR_TOKEN=replace-with-operator-token
DW_ADMIN_TOKEN=replace-with-admin-token
```

Start with that env file:

```bash
docker compose --env-file durable-workflow.prod.env \
  -f docker-compose.published.yml up -d --wait
```

Operate the host as a production service:

- Put TLS, public routing, request logging, and IP allow lists in a reverse
  proxy in front of the API container.
- Do not expose MySQL or Redis publicly.
- Use `DW_WORKER_TOKEN` for workers, `DW_OPERATOR_TOKEN` for application and
  operator traffic, and `DW_ADMIN_TOKEN` for namespace and administrative work.
- Back up the MySQL volume before every image upgrade and on a regular
  schedule. Redis should be preserved for graceful restarts, but MySQL remains
  the durable workflow-history source of truth.
- Keep the exact env file, image tag or digest, and database backup together for
  restores.

Upgrade order:

1. Back up MySQL and record the current image reference.
2. Change only `DW_SERVER_IMAGE` or `DW_SERVER_TAG`.
3. Pull the new image.
4. Run `docker compose --env-file durable-workflow.prod.env -f docker-compose.published.yml up -d --wait`.
5. Confirm `/api/ready`, `/api/cluster/info`, and worker registration before
   shifting external traffic.

The server README keeps the latest command-level Compose examples in the
[Official Image + Compose](https://github.com/durable-workflow/server#official-image--compose)
section.

## Small clustered deployments

A small cluster is a modest extension of the single-node model. The validated
self-serve contract is intentionally narrow:

- Run 2-3 stateless API containers behind a load balancer. Health,
  readiness, cluster discovery, worker registration, workflow-task polling, and
  workflow-task completion must work without sticky sessions.
- Use one shared external MySQL or PostgreSQL database for durable history.
  SQLite is single-node only and is not a clustered persistence backend.
- Use shared Redis for cache, long-poll wake signals, query-task queue locks,
  task-queue admission locks, and queue state. Redis-less multi-node mode is
  not a supported clustered contract.
- Check `GET /api/cluster/info` on each API node during rollout.
  `topology.current_shape` should remain `standalone_server`,
  `topology.current_roles` should include `api_ingress`, `control_plane`,
  `matching`, and `history_projection`, and `topology.execution_mode` should
  remain `remote_worker_protocol` for standalone server nodes. Use
  `topology.matching_role` to confirm the matching path you actually deployed:
  default nodes report `queue_wake_enabled: true`, `shape: "in_worker"`, and
  `wake_owner: "worker_loop"`, while dedicated matching rollouts flip nodes
  with `DW_V2_MATCHING_ROLE_QUEUE_WAKE=0` to `queue_wake_enabled: false`,
  `shape: "dedicated"`, and `wake_owner: "dedicated_repair_pass"`. The same
  block should continue to advertise the intended `task_dispatch_mode`, the
  frozen routing axes in `partition_primitives`, and the current
  `backpressure_model`. See
  [Server Role Topology](/docs/polyglot/server-role-topology) for the
  field-by-field meaning of the topology manifest.
- Scale external SDK workers independently from API nodes. Workers can run on
  separate hosts or processes, but they should talk to the load-balanced API
  endpoint rather than to one sticky node.
- Configure [task queue admission](/docs/polyglot/task-queue-admission)
  for queues that protect a tenant, external API, database pool, or other
  shared downstream dependency.
- Run exactly one scheduler or maintenance process for schedule evaluation,
  activity-timeout enforcement, and history pruning.
- Run bootstrap/migrations once per rollout before new API and worker
  containers accept traffic.
- Choose a rollout posture for this release: stop-the-world (drain workers,
  stop scheduler/maintenance, replace API nodes, run bootstrap/migrations,
  then restart workers and the scheduler) or
  [rolling upgrades](/docs/rolling-upgrades) when every guarantee on
  that contract holds.
- Treat the database and Redis as the primary failure domains. The server
  containers are replaceable; the persistence and coordination layers are not.

The published self-serve recipes start in the `standalone_server` shape even
though cluster discovery also names `split_control_execution` as a supported
product topology. Treat that as one contract with different role assignments,
not as a second server product. If you pilot a more explicit role split later,
keep reading `topology.current_shape`, `topology.current_roles`, and
`topology.matching_role` from `/api/cluster/info` instead of inferring duties
from hostnames or container names. The
[Server Role Topology](/docs/polyglot/server-role-topology) page explains
those role assignments and the migration path in one place.

Every API node should use the same auth tokens or signature keys, app version,
workflow package version, payload-codec configuration, database connection, and
Redis connection. Give each API node a unique `DW_SERVER_ID` so cluster
discovery and logs can distinguish the nodes.

The unsupported boundaries are explicit: SQLite clustering, Redis-less
multi-node mode, duplicate schedulers as a steady-state topology, active/active
multi-writer databases, self-hosted hands-free regional failover,
and broad "five-nines" or "zero-downtime" SLA promises need separate
validation or support-led design before you rely on them. Active/passive
multi-region guidance is also support-led evaluation material; see
[Active/passive multi-region](#activepassive-multi-region) below.
Single-region HA failover — managed-database failover, managed-Redis failover,
API-node loss, worker loss, and scheduler-runner restart inside one region —
is a support-led evaluation contract until the release-evidence gate passes;
see
[Single-region high availability and failover](#single-region-high-availability-and-failover)
below for the engine recovery targets, the readiness rules during a failover,
and the evidence required to open that gate.

This path is self-serve when your team already has a clear VM, network,
database, cache, backup, and load-balancer model. It becomes support-led when
you need help deciding those boundaries, capacity, rollout order, or recovery
procedures.

## Helm chart for Kubernetes

The Server repository owns the
[chart source and release guidance](https://github.com/durable-workflow/server/tree/main/k8s/helm/durable-workflow).
Install the published chart from the OCI registry:

```bash
helm install durable-workflow \
  oci://ghcr.io/durable-workflow/charts/durable-workflow \
  --namespace durable-workflow --create-namespace \
  -f my-values.yaml
```

Your values file must point to an external MySQL/PostgreSQL database and
external Redis, provide production credentials through existing Kubernetes
Secrets, and configure ingress/TLS for your environment. The chart does not
create persistence or provider-managed infrastructure. For reproducible
production deployments, add `--version <validated-chart-version>` and pin the
Server image by digest. The Server repository publishes chart versions and
documents chart upgrades alongside the chart source.

## Kubernetes manifests

The server repository includes raw manifests under
[`k8s/`](https://github.com/durable-workflow/server/tree/main/k8s) for teams
that already operate Kubernetes:

- Namespace and shared labels
- ConfigMap and Secret split
- Bootstrap/migration Job
- API Deployment and Service
- Worker Deployment
- Scheduler CronJob
- PodDisruptionBudget
- `/api/health` liveness and `/api/ready` readiness probes
- Conservative resource requests and limits

Before applying the manifests, replace the image tag with a specific published
version or digest, provide real database and Redis credentials, and wire the
ConfigMap values to the services your cluster already operates. The manifests
are intentionally raw and inspectable; they are not a Helm chart and do not
promise generic managed-Kubernetes behavior.

For a Kubernetes production rollout, prove at minimum:

```bash
kubectl -n durable-workflow wait --for=condition=complete job/durable-workflow-migrate --timeout=180s
kubectl -n durable-workflow rollout status deploy/durable-workflow-server
kubectl -n durable-workflow rollout status deploy/durable-workflow-worker
kubectl -n durable-workflow port-forward svc/durable-workflow-server 8080:8080
curl http://localhost:8080/api/ready
curl -H "Authorization: Bearer $DW_ADMIN_TOKEN" http://localhost:8080/api/cluster/info
```

Single-region HA failover — managed-database failover, managed-Redis failover,
API-node loss, worker loss, and scheduler-runner restart inside one region —
remains support-led on the raw-manifest path just as it does on the small-cluster
path while the release-evidence gate is closed. The intended contract requires
the readiness, singleton-scheduler, and shared-substrate rules in
[Single-region high availability and failover](#single-region-high-availability-and-failover)
below. Provider-specific load balancers, storage classes, network policies,
active/active multi-region, and self-hosted hands-free regional failover remain
support-led or tracked separately from the raw-manifest contract. Use the
separately versioned Helm path above when you want Helm-managed installation
and upgrades.

## Single-region high availability and failover

Single-region HA failover is currently an **unverified, support-led** contract
candidate layered on the small-cluster shape and the raw Kubernetes shape. It
targets the failure modes that the engine is designed to survive **inside one
region**: managed-database failover (RDS
Multi-AZ, Aurora cluster failover, Cloud SQL HA, Patroni promotion, etc.),
managed-Redis failover (Sentinel, Elasticache replication-group failover,
Memorystore HA, etc.), API-node loss, worker loss, and
scheduler/maintenance runner restart. Cross-region active/passive recovery is
a different, support-led evaluation; see
[Active/passive multi-region](#activepassive-multi-region) below.

The intended contract — engine behavior, readiness rules, the per-event
recovery targets, the split-brain prevention rules, and the rehearsal
acceptance test — lives in the workflow library at
[`docs/deployment/ha-failover.md`](https://github.com/durable-workflow/workflow/blob/main/docs/deployment/ha-failover.md)
and the standalone server at
[`docs/ha-failover-validation.md`](https://github.com/durable-workflow/server/blob/main/docs/ha-failover-validation.md).
This section is the public surface of those documents.

### Release-evidence status

No passing exact-release result is linked for the documented public server
release. The recorded released-image attempt stopped during `topology_start`
while waiting for API readiness, before any failure-matrix or recovery-bound
claim could be established. Do not treat the runner's presence, its scenario
manifest, or a deployment-specific rehearsal as evidence that the released
image has passed the full matrix. Single-region HA therefore remains
support-led.

Self-serve wording is authorized only after a source-free run uses the exact
documented server release for the image, runner, and Compose topology and
publishes a `single-region-failover-result.json` with `outcome: "pass"`,
`runner_blocked: false`, every entry in `phase_outcomes` at `status: "pass"`,
every recovery-bound verdict passing, and evidence for every required scenario
in the public
[single-region failover scenario manifest](/platform-conformance/single-region-failover-scenarios.json).
That public passing result must be linked from this status section and must name
the same server image as `durableworkflow/server:2.0.0`. Until all of those
conditions hold, the recovery bounds below are evaluation targets rather than
released-image validation claims.

### Candidate topology

The intended HA contract applies when the deployment matches the small-cluster
shape or the raw-manifest shape and the operator preserves three rules:

- **One writable workflow database endpoint, always.** Managed failover
  (RDS Multi-AZ, Aurora cluster failover, Cloud SQL HA, Patroni, etc.) is
  permitted on the rule that the previous primary is fenced — revoke the
  write user, demote with `read_only=on`, sever replication, or restore from
  a known-good snapshot — before it can re-attach. A connection proxy (RDS
  Proxy, ProxySQL, PgBouncer) between the API/scheduler containers and the
  database is permitted; it does not change any guarantee, because the
  engine's contract is on the connection it sees.
- **One Redis endpoint at a time, with a documented promotion path.**
  Managed-Redis failover (Sentinel, Elasticache replication-group failover,
  Memorystore HA, etc.) is permitted. Redis is the acceleration layer, not
  the correctness substrate, so a Redis failover is a latency event, never a
  correctness event.
- **One scheduler/maintenance runner, always.** The orchestrator (Compose
  service with `deploy.replicas: 1`, systemd unit guarded by a host-level
  lease, or Kubernetes `Deployment` with `replicas: 1` and
  `RollingUpdate.maxSurge: 0`) is responsible for keeping the singleton
  invariant during restart. Duplicate scheduler runners as a steady-state
  topology are not in this contract.

### Per-event behavior and recovery targets

The intended engine contract defines bounded recovery targets for each event
class. These are not released-image guarantees while the release-evidence gate
is closed. The wall-clock recovery time the operator observes is the engine
target plus the managed service's own promotion latency.

| Event | Candidate engine behavior | Recovery target (after substrate / runner is back) |
| --- | --- | --- |
| Managed-database failover | Writes return errors and are not silently buffered. Reads return errors. `/api/ready` fails on every API node and on the scheduler. No acknowledged work is lost. | One connection-pool reconnect, plus one `task_repair` cadence (default 3s), plus one long-poll timeout (default 30s, max 60s) for in-flight pollers. |
| Managed-Redis failover | Wake signals dropped → discovery falls back to long-poll timeout. Acceleration-layer health checks go to **warning**, not error. `/api/ready` typically stays green. | Redis client reconnect interval. |
| API node loss (1 of N) | Load balancer removes the failed node within its readiness interval. In-flight requests against it fail at the LB and are retried by the client. | Load-balancer readiness interval (operator-controlled, typically 5–10s). |
| Worker loss | Tasks held by the failed worker pause until lease expiry. Other workers continue claiming their own tasks. | Lease expiry (5 min for activity tasks), plus one `task_repair` cadence. |
| Scheduler/maintenance restart | Schedule fires pause; activity-timeout enforcement pauses; history pruning pauses. All three resume on the next tick after restart. No duplicate fires occur because the runner is a singleton. | Orchestrator restart latency, plus one scheduler tick. |

### Load-balancer, readiness, and traffic-shift rules

The load balancer in front of the API nodes is the single decision point
for traffic admission during a failover:

- Wire the load balancer to **`GET /api/ready`**, not `/api/health` alone.
  `/api/health` only proves the process is serving HTTP; `/api/ready` proves
  the server can use its durable database and reports whether Redis wake
  acceleration is healthy or degraded. During a database outage `/api/ready`
  correctly fails on every node — the load balancer MUST tolerate the all-down
  state rather than fall back to a stale "last known good" roster.
- Use a check interval of 5–10 seconds and a removal threshold of 2–3
  consecutive failures.
- Do not require sticky sessions; the small-cluster smoke proves an
  external worker can poll `server-a` and complete on `server-b`.
- During a Redis-only failover, readiness remains green on every node while
  the database-backed durable paths are available. Acceleration-layer
  degradation surfaces as `checks.cache.status=warning` with
  `checks.cache.degraded_capability=long_poll_wake_acceleration`; do not
  configure the load balancer to remove nodes on that warning.

After substrate recovery, the recommended verification sequence is to wait
for at least one node's `/api/ready` to return 200, curl `/api/cluster/info`
through the load balancer with an admin token to confirm the topology
manifest, issue `POST /api/worker/register` for a probe worker through the
load-balanced endpoint, confirm exactly one scheduler runner is alive in
its orchestrator, and resume external traffic.

### Run the exact-artifact rehearsal

The server release includes a reusable baseline rehearsal intended to exercise
the full failure matrix without building product code from a checkout. The
runner and Compose topology must come from the same release tag as the server
image; do not combine a moving default-branch checkout with an arbitrary image.
On a clean host with Docker Engine, Docker Compose v2, Python 3.11 or newer, and
public registry access, run:

<!-- docs-example id="single-region-failover-exact-release" -->
```bash
export DW_SERVER_RELEASE=2.0.0
git clone --depth 1 --single-branch --branch "$DW_SERVER_RELEASE" \
  https://github.com/durable-workflow/server.git "server-$DW_SERVER_RELEASE"
cd "server-$DW_SERVER_RELEASE"
export DW_SERVER_IMAGE="durableworkflow/server:$DW_SERVER_RELEASE"
scripts/conformance/single-region-failover-published-artifacts.sh \
  --result-dir ./failover-result
```

This path deliberately derives the checkout and `DW_SERVER_IMAGE` from one
release variable. Do not override the image independently; select a different
release by changing `DW_SERVER_RELEASE` before cloning. The runner requires a
concrete public server tag or digest, pulls every
supporting image, resolves all runtime images to repository digests, and
rejects Compose build sections, product-source bind mounts, and local or
rolling server references. It starts exactly two API nodes behind one nginx
endpoint, one MySQL database, one Redis service, and one scheduler/maintenance
runner.

The resulting `single-region-failover-result.json` uses schema
`durable-workflow.v2.single-region-failover.result`. It records exact artifact
and tool versions, normalized topology, readiness transitions, measured
recovery times and bound verdicts, workflow/run/task/schedule identities, and
duplicate/loss assertions for cross-node completion, API-node loss, database
interruption, Redis interruption, worker lease loss, and singleton-scheduler
restart. External runners discover the invocation and public scenario manifest
from `GET /api/cluster/info` at `single_region_failover_contract`.

A result validates engine-visible interruption and recovery against the
released image only when it passes the release-evidence gate above. A failed,
partial, or runner-blocked result is diagnostic evidence, not validation. Even
a passing baseline does not turn a local MySQL or Redis container restart into
evidence for a cloud provider's promotion mechanism. Keep provider-native
promotion, fencing, RPO, and elapsed-time evidence alongside the baseline
result before claiming managed-service HA.

### Recovery packet additions

A small-cluster or raw-manifest deployment that claims this contract MUST
extend its recovery packet (per the
[Operator Operating Envelope](/docs/operator-operating-envelope)) with
rehearsal evidence for each event class:

- a managed-database failover that completes without acknowledged-write
  loss and within the bounded recovery time above;
- a managed-Redis failover that does not flap the load-balancer rotation,
  does not lose any acknowledged work, and surfaces
  `checks.cache.status=warning` with `long_poll_wake_acceleration` as the
  degraded capability;
- an API-node loss event that the load balancer absorbs within the
  configured readiness interval, with no acknowledged-write loss;
- a worker-loss event that preserves the durable run through lease expiry,
  reclaims it after the configured repair bound, and completes it exactly once;
- a scheduler-runner restart on a different host that fires no duplicate
  schedules and leaves no schedule unevaluated past its `next_fire_at`
  plus one tick.

While the release-evidence gate is closed, every deployment remains support-led
even if its own rehearsal passes. After a public all-phase result opens the
release gate, a deployment becomes self-serve under this contract only after
its environment-specific rehearsal evidence is recorded in the operator's
recovery packet and refreshed on the cadence the Operator Operating Envelope
publishes.

### Boundary against unsupported HA claims

The single-region HA contract is intentionally narrow. The following remain
**outside** it and continue to require a support-led design pass; the
topology itself is part of the product risk:

- active/active multi-writer database topologies;
- active/passive multi-region and automatic or hands-free regional failover
  for self-hosted topologies (the
  [next section](#activepassive-multi-region) preserves evaluation guidance);
- synchronous cross-region database replication (RPO=0);
- duplicate scheduler/maintenance runners as a steady-state topology;
- engine-enforced region-pinned task queues as a routing axis;
- provider-specific managed-Kubernetes validation beyond the published
  chart's packaging contract;
- broad "five-nines" or "zero-downtime" SLA promises beyond the bounded
  recovery times above.

The contract is *bounded recovery during named events*, not an uptime
promise that depends on the operator's database, network, and orchestrator
choices. Marketing or SLA language for self-hosted deployments MUST NOT
cross that line without dedicated validation.

## Active/passive multi-region

Active/passive multi-region is **support-led evaluation guidance**, not a
proven self-serve contract in the supported 2.0 operating envelope. Use this
section to review a candidate architecture and build an environment-specific
rehearsal with support before relying on it in production. Each region still
starts from a documented single-node, small-cluster, or raw Kubernetes path,
but those single-region contracts do not establish cross-region replication,
failover, failback, RPO, RTO, or split-brain behavior. The deeper design
material — data authority, replication assumptions,
namespace/task-queue/worker behavior, the failover and failback runbook,
fencing, and the consistency/latency tradeoffs — lives in the workflow library
at
[`docs/deployment/multi-region.md`](https://github.com/durable-workflow/workflow/blob/main/docs/deployment/multi-region.md)
and the standalone server at
[`docs/multi-region-validation.md`](https://github.com/durable-workflow/server/blob/main/docs/multi-region-validation.md).

The candidate shape to evaluate:

- One **active region** running the validated single-node or small-cluster
  contract: API container(s) behind a load balancer, shared external MySQL or
  PostgreSQL as the writable durable database, shared Redis, exactly one
  scheduler/maintenance runner, and external workers.
- One **standby region** holding an asynchronously replicated standby of the
  workflow database, optional standby Redis, no scheduler/maintenance
  process, and zero or more pre-provisioned API/worker containers that are
  idle until promotion.
- A **regional failover** that is explicit operator work: stop write traffic
  to the failed region, confirm replication state against the published
  RPO, promote the standby database, run any release-required migrations on
  the new primary, start the singleton scheduler/maintenance runner in the
  new active region, switch worker endpoints, switch external traffic, and
  rebuild any derived projections. There is no automatic cross-region
  cutover.
- A **failback** that runs the same sequence in reverse once the original
  region returns to service, with the recovered primary fenced (revoke
  write user, demote with `read_only=on`, sever replication, or restore
  from a known-good snapshot) before re-attaching as a standby.

Data authority and replication assumptions:

- The workflow database is the single durable source of truth and is
  region-bound: exactly one region writes to it at any given time. The
  standby region's database is a read replica until promotion. Recovery
  point objective (RPO) is the asynchronous replication lag; recovery time
  objective (RTO) is operator runbook execution time.
- Redis is region-local acceleration. Wake signals, query-task queue
  locks, and admission locks do not propagate across regions and must not
  be expected to. Each region runs its own Redis; the standby's cache is
  cold or warm at the operator's discretion and correctness does not
  depend on it being preserved across the failover.
- Visibility is served by the active region's database. Promote first,
  then read.

Namespace, task-queue, and worker-registration behavior:

- Namespaces and task queues are stored in the workflow database and
  survive promotion exactly as they were at the last replicated commit.
  They are not regionally partitioned by the engine.
- Workers in the new active region register against the local API
  endpoint after promotion. Pre-existing registrations from the failed
  region remain in the database and expire through the normal
  worker-expiry path.
- Build-id rollouts and deployment-lifecycle state survive failover
  because they live in the workflow database.

Consistency and latency tradeoffs in steady state:

- Workflow starts, signals, and updates commit against the active
  region's database; their latency is the active-region commit latency,
  and they are refused while authority is being withdrawn during a
  failover.
- Workflow-task and activity-task delivery follow the single-region
  acceleration contract inside the active region: sub-second when Redis
  is healthy, durable poll cadence otherwise.
- Schedules fire from the singleton scheduler in the active region and
  pause while no scheduler is running; fires resume from durable
  schedule rows after promotion.
- Visibility reads are read-after-write within the active region only;
  the engine does not provide cross-region read-your-writes or RPO=0.

The disaster-recovery boundary is explicit: this candidate design is not a
substitute for backups or evidence. The recovery packet documented in
[Operator Operating Envelope](/docs/operator-operating-envelope)
remains required, with replication-lag SLO, promotion-runbook latency,
last successful failover rehearsal date, and the fencing procedure for
the recovered primary added on top.

For self-hosted deployments, active/passive and active/active multi-region,
automatic regional failover, synchronous cross-region replication (RPO=0),
cross-region active visibility, and region-pinned task queues as an
engine-enforced routing axis remain
[support-led](/docs/support) because the topology itself is part of the
product risk. The current
[Cloud managed-runtime contract](/docs/polyglot/cloud-control-plane) is
single-region and does not supply those multi-region guarantees.

## Readiness contract

Use both health and readiness checks:

- `GET /api/health` proves the process is serving HTTP.
- `GET /api/ready` proves the server can use its configured runtime
  dependencies, including migrations and default namespace readiness.
- `GET /api/cluster/info` proves an authenticated client can discover build
  identity, control-plane protocol, worker protocol, payload codecs, and server
  capabilities.
- `POST /api/worker/register` proves workers can authenticate into the expected
  namespace and task queue.

Do not shift traffic based on `/api/health` alone.

<!-- Source: docs/migration.md -->

# Migrating to 2.0

This guide covers the key changes when upgrading an existing Laravel v1 application to v2.
First choose whether Laravel will keep owning the runtime or connect to Cloud or
a self-hosted Server through the PHP SDK in the
[Laravel adoption and runtime transition guide](/docs/laravel-adoption/).
This page then provides the detailed v1-to-v2 embedded package procedure.

## Upgrade procedure

### Before upgrading

**1. Inventory every v1 execution store**

v1 execution state is not necessarily confined to the workflow database. In
addition to the workflow rows and history, Laravel queue jobs can be ready,
delayed, or reserved in Redis, a queue database, SQS, or another queue
backend. Record all of the following before changing code:

- the workflow storage connection and the actual v1 model/table mappings
- every queue connection and queue name used by v1 workflows and activities,
  including per-workflow or per-activity overrides
- the queue backend's database, key prefix, region, account, and other routing
  settings needed to restore the same queues
- the secret-manager reference and immutable version that retrieve the exact
  `APP_KEY`, plus any cache store used for v1 unique-job locks

Do not copy the `APP_KEY` value, secret-manager recovery credentials, database
credentials, or queue-provider credentials into this inventory. Encrypted jobs
and serialized workflow arguments require the original key, but the recovery
manifest should contain only its secret-manager reference and version. Keep the
credentials that can retrieve that key separately access-controlled from the
SQL and queue backups.

The default v1 tables are `workflows`, `workflow_logs`, `workflow_signals`,
`workflow_timers`, `workflow_exceptions`, and `workflow_relationships`.
Published `config/workflows.php` files may replace the
`stored_workflow_model`, `stored_workflow_log_model`,
`stored_workflow_signal_model`, `stored_workflow_timer_model`, or
`stored_workflow_exception_model`. A replacement model may also override its
Eloquent `$table`; `workflow_relationships_table` separately controls the
relationship table. Back up the tables and storage connection resolved by
those configured models, not only the default names.

**2. Choose a rollback boundary and quiesce v1 work**

Prevent new workflow starts and signals while taking the recovery cut. Pause
schedulers and stop every worker that can consume the inventoried queues after
its current job reaches a boundary. For example:

```bash
# Horizon
php artisan horizon:pause

# Supervisor or systemd (use the names from your deployment)
sudo supervisorctl stop <your-worker-group>:*
sudo systemctl stop laravel-worker
```

`php artisan queue:restart` alone is not a quiesce operation when Supervisor,
systemd, Kubernetes, or another process manager immediately starts a
replacement worker. Confirm that no consumer can reserve another job before
capturing state.

Choose and record one of these policies:

- **Drain v1:** block new v1 work, leave v1 workers running until
  `php artisan workflow:v1:list` (when available) or an equivalent query of the
  configured stored-workflow table reports no nonterminal workflows, then stop
  the workers and confirm the relevant queues have no ready, delayed, or
  reserved v1 jobs. SQL-only rollback can cover these terminal v1 workflows.
- **Preserve in-flight v1:** stop workers at job boundaries and take an
  application-consistent snapshot of both SQL and every durable queue backend.
  The queue snapshot must include ready, delayed, and reserved work and must be
  from the same recovery cut as SQL. Wait for a reserved job to finish before
  the cut unless the backend documents how to restore its message and lease
  safely.
- **Accept in-flight loss:** if the backend cannot provide a restorable queue
  cut, record the affected workflow IDs and explicitly accept that
  queue-dependent nonterminal executions cannot be recovered by this rollback.
  Exclude an eligible `pending` row from that disposition only after proving
  the Watchdog path below, and exclude a genuine signal-only wait only after
  testing its retained ingress. Do not present any other SQL row as recoverable
  work.

A signal-only wait may legitimately have no queued job if its external signal
ingress remains available after rollback. Activity retries and timers are
queue-backed: a `workflow_timers` row or a waiting `workflows` row does not
recreate a missing delayed job.

Supported v1.0.77 also starts its enabled-by-default `Workflow\Watchdog` from
the queue worker loop. The Watchdog can find a `pending` workflow whose
`updated_at` is at least five minutes old and whose serialized `arguments` are
present, then redispatch that workflow to its recorded connection and queue.
This is a bounded, pending-only wake path, not a replacement for queue backup.
You may rely on it only after staging evidence shows the restored v1 worker
loop dispatching the Watchdog, the Watchdog redispatching an eligible stale
`pending` row, and that workflow advancing on the expected queue. It does not
recreate activity retries or timer jobs and does not recover `waiting` or
`running` rows. Those states still need their preserved queue job or a valid
external signal path where the workflow is genuinely waiting for a signal.

**3. Back up the rollback recovery set**

Create a full database backup before upgrading:

```bash
# MySQL/MariaDB
mysqldump -u root -p your_database > backup-v1-$(date +%Y%m%d-%H%M%S).sql

# PostgreSQL
pg_dump -U postgres your_database > backup-v1-$(date +%Y%m%d-%H%M%S).sql

# Laravel backup package (if installed)
php artisan backup:run --only-db
```

Then preserve queue state according to the backend:

- **Database queue:** include the queue tables and their connection in the same
  recovery cut. They may be outside the workflow database.
- **Redis queue:** use a restorable Redis snapshot or backup that includes the
  exact queue database/prefix and its ready, delayed, and reserved keys. If the
  same Redis database holds unique-job locks needed by the cut, preserve those
  keys too. Prefer a dedicated queue database or instance; restoring a shared
  Redis snapshot can rewind unrelated application data.
- **SQS or another managed queue:** use a provider-supported, point-in-time
  message restore only if it preserves available, delayed, and in-flight
  messages consistently. SQS does not provide an arbitrary queue snapshot for
  this procedure, so drain v1 work or classify the remaining queue-dependent
  v1 work as unrecoverable. The only SQL-only exceptions are a proven
  Watchdog-eligible `pending` row and a tested signal-only wait.

Store the SQL backup, queue backup, non-secret inventory, and recovery timestamp
together. That recovery manifest may name the `APP_KEY` secret-manager
reference and version, but must not contain the key or any credentials that can
retrieve it. Keep secret-manager, database, queue-provider, and backup recovery
credentials separately access-controlled. Restoring SQL from one cut and queue
state from another is not a supported in-flight rollback.

**4. Test in staging first**

**Do not upgrade production without testing in staging.** The upgrade includes:

- Database schema changes (the v2 durable kernel adds new tables; the
  exact count is whatever `php artisan migrate` applies, and a future
  squashed migration may consolidate the per-feature files)
- Namespace changes requiring code updates
- Queue worker restart (brief interruption)
- Backend capability validation

**Staging test checklist:**

- [ ] Deploy v2 code to staging environment
- [ ] Run migrations against staging database
- [ ] Restart queue workers
- [ ] Run `php artisan workflow:v2:doctor --strict`
- [ ] Start a new v2 workflow and verify it completes
- [ ] Verify v1 workflows (if any) still complete
- [ ] Check Waterline shows both v1 and v2 workflows
- [ ] Run your application's test suite
- [ ] Verify no errors in logs

Only proceed to production after staging validation passes.

### Upgrade steps

**1. Update composer dependency**

```bash
composer require durable-workflow/workflow:2.0.1
```

The maintained Composer package is `durable-workflow/workflow` for both v1 and
v2; this command changes its version constraint to the current public v2
artifact pin. The old `laravel-workflow/laravel-workflow` name is only a
compatibility alias for dependency graphs created before the package rename.
Use the maintained name for new requirements and rollback. The current public
artifact pin names the stable 2.0 package. Use
`durable-workflow/workflow:^2.0` when you want Composer to accept compatible
2.x updates automatically.

**2. Run database migrations**

```bash
php artisan migrate
```

v2 adds the durable-kernel tables that back the v2 feature contract. The
current per-feature migrations create:

- Core: `workflow_instances`, `workflow_runs`, `workflow_history_events`, `workflow_tasks`, `workflow_commands`
- Activity: `activity_executions`, `activity_attempts`
- Features: `workflow_updates`, `workflow_signal_records`, `workflow_run_waits`, `workflow_run_timeline_entries`, `workflow_run_lineage_entries`, `workflow_schedules`, `workflow_schedule_history_events`
- Observability: `workflow_run_summaries`, `workflow_failures`, `workflow_links`, `worker_compatibility_heartbeats`
- Timers: `workflow_run_timers`, `workflow_run_timer_entries`
- Search / memo / message / child: `workflow_search_attributes`, `workflow_memos`, `workflow_messages`, `workflow_child_calls`
- Service catalog: `workflow_service_endpoints`, `workflow_services`, `workflow_service_operations`, `workflow_service_calls`

A future squashed migration may consolidate these into fewer files
without changing the durable contract. The supported way to know what
your database actually has is to run `php artisan migrate:status` after
the upgrade.

The default v1 tables (`workflows`, `workflow_logs`, `workflow_signals`,
`workflow_timers`, `workflow_exceptions`, and `workflow_relationships`) are
preserved for finish-on-v1 execution. If the configured v1 models override
their tables or storage connection, those configured tables are the v1 source
of truth instead.

**3. Update configuration (if needed)**

v2 configuration is backward compatible. If you published `config/workflow.php` in v1, it will continue to work. New v2 options include:

- `durable_types` — type aliases for language-agnostic workflow references
- `task_repair_policy` — how to handle stuck tasks
- `backend_capability_check` — strict vs. permissive validation
- `projection_rebuild` — history rebuild strategies
- `history_budget` — event count limits for continue-as-new

These have sensible defaults. Only configure them if you need non-default behavior. See [Configuration](/docs/configuration/options/) for details.

**Payload codec default changed to `avro`.** v1 defaulted to the PHP-only `Workflow\Serializers\Y::class`; v2 defaults to the language-neutral `avro` codec so Python, Go, and TypeScript workers can decode payloads without a shared PHP runtime. `avro` is the only supported codec for new v2 workflows. New v2 workflows you start will be tagged with `payload_codec = "avro"`.

If you have a published `config/workflows.php` from v1 with `'serializer' => Workflow\Serializers\Y::class`, v2 still reads that value so migration diagnostics can flag it, but new v2 workflow payloads resolve to Avro. Run `php artisan workflow:v2:doctor` after upgrading — it will flag a legacy codec setting as a v1 drain/import concern.

To accept the v2 default explicitly, leave `serializer` unset, or pin it:

```php
// config/workflows.php — v2 default (language-neutral, compact binary)
'serializer' => 'avro',
```

Keep a legacy codec (`'workflow-serializer-y'` or `'workflow-serializer-base64'`) only if you need to finish draining v1 runs that share PHP-native values between a server and PHP-only workers. Legacy class names (`Workflow\Serializers\Y::class`, etc.) are still accepted as aliases for decoding v1 runs.

**Custom serializer classes from v1 are unsupported in v2.** The public v2 registry resolves only `avro`. The legacy `workflow-serializer-y` and `workflow-serializer-base64` readers are confined to the internal v1 import/drain path and cannot be selected for a new v2 run or SDK payload. If you had a custom serializer, drain v1 runs before upgrading or re-encode historical payloads into `avro` — the custom class is not consulted. `php artisan workflow:v2:doctor` flags any other `workflows.serializer` value as migration debt; new-run codec omission resolves to `avro`.

**Custom model subclasses are supported only when they keep the package's column and key contract.** The frozen support matrix in
[Customization Matrix](/docs/configuration/customization-matrix/) is
authoritative: subclassing the v2 instance, run, task, history-event,
projection, schedule, activity, failure, link, message, memo, search-attribute,
and child-call models is supported when the subclass keeps the package
table names, primary keys, and foreign keys. Custom table names with
custom foreign-key column names are out of contract. Waterline reads the
v2 projections through the
`Workflow\V2\Contracts\OperatorObservabilityRepository` contract, so a
schema-compatible subclass does not require a Waterline change.

**Environment variables:**

v2 does not introduce new required environment variables. Existing `QUEUE_CONNECTION`, `CACHE_DRIVER`, and `DB_CONNECTION` continue to work.

**4. Restart queue workers**

Queue workers must be restarted to load v2 code:

```bash
# If using Laravel queue workers
php artisan queue:restart

# If using Supervisor
sudo supervisorctl restart <your-worker-group>:*

# If using systemd
sudo systemctl restart laravel-worker

# If using Horizon
php artisan horizon:terminate
```

Workers will:
1. Finish their current job
2. Exit gracefully
3. Restart with v2 code loaded

**Workers must restart before processing v2 workflows.** v1 workflows can complete with old or new workers (finish-on-v1 compatibility).

### After upgrading

**1. Verify backend capability**

```bash
php artisan workflow:v2:doctor --strict
```

Expected output:

```
✓ Database driver supports required features
✓ Queue driver supports required features
✓ Cache driver supports locks
✓ All backend capabilities present
```

If any check fails, see [Backend Requirements](/docs/installation/#requirements) for driver prerequisites.

**2. Verify v2 workflows start successfully**

Start a test workflow using v2 API:

```php
use Workflow\V2\WorkflowStub;
use Workflow\V2\StartOptions;

$workflow = WorkflowStub::make(TestWorkflow::class, 'test-upgrade');
$result = $workflow->start(['test' => true], new StartOptions());
$runId = $result->runId();
```

`WorkflowStub::start()` returns a `StartResult` object. Pull the run id off it
with `runId()` before comparing against database rows — treating the return
value as a scalar string will silently compare an object against a column.

Check that:

- Workflow appears in Waterline
- `workflow_instances` table has a row with matching `instance_id` (`$workflow->id()`)
- `workflow_runs` table has a row with matching `run_id` (`$result->runId()`)
- Workflow completes or progresses as expected

**3. Check v1 workflows (if any)**

If you have in-flight v1 workflows:

```bash
php artisan workflow:v1:list
```

Verify they continue to progress. v1 workflows should complete on the v1 engine without errors.

**4. Monitor logs for errors**

Watch application logs for workflow-related errors:

```bash
tail -f storage/logs/laravel.log | grep -i workflow
```

Common issues:

- Namespace errors: code still using `Workflow\Workflow` instead of `Workflow\V2\Workflow`
- Method errors: v2 workflow or activity classes that still need to rename their entry method to `handle()`
- Queue driver errors: using `sync` driver in queue mode (not supported); in poll mode (`workflows.v2.task_dispatch_mode=poll`) the queue is unused for task delivery and `sync` is acceptable

**5. Verify Waterline observability**

Open Waterline (default: `/waterline`) and verify:

- v1 workflows (if any) appear with their original data
- v2 workflows appear with full run/history/activity detail
- No errors in Waterline rendering

### Rollback procedure

Rollback is a recovery-set operation. Restoring the workflow SQL database alone
does not restore in-flight v1 execution when its Laravel queue state lives in
Redis, SQS, another database, or another external backend.

If the upgrade fails in production, roll back only to the policy and recovery
cut selected before the upgrade:

**1. Quiesce application ingress, schedulers, and queue workers**

Use the same stop procedure as the pre-upgrade cut. Confirm that no worker can
reserve a job and no request can start or signal a workflow while state is
being restored.

**2. Validate the recovery set**

- A SQL-only recovery set is supported only when the selected cut has no
  nonterminal v1 execution that depends on preserved queued work. An eligible
  stale `pending` row may instead use the proven v1.0.77 Watchdog path described
  above, and a genuine signal-only wait may use a tested external signal
  ingress. Do not extend either exception to other states.
- An in-flight recovery set must contain SQL plus restorable ready, delayed,
  and reserved queue state from the same cut for retries, timers, activities,
  and `waiting` or `running` work. Signal-only waits must retain a working
  signal ingress; timer waits must retain their delayed queue jobs.
- If neither condition holds, stop here. Fix forward, reconcile the recorded
  workflows manually, or proceed under the previously documented acceptance
  that the affected executions are unrecoverable.

**3. Restore the database backup**

```bash
# MySQL/MariaDB
mysql -u root -p your_database < backup-v1-YYYYMMDD-HHMMSS.sql

# PostgreSQL
psql -U postgres -d your_database < backup-v1-YYYYMMDD-HHMMSS.sql
```

Restore every configured custom v1 model table and the
`workflow_relationships_table`, even when they live on another connection.

**4. Restore the queue recovery cut, when required**

Keep consumers stopped. Follow the queue provider's restore procedure and
restore the exact connections, queues, prefixes, ready jobs, delayed jobs, and
reserved jobs captured with SQL. Restore required unique-job lock state when
it was part of the recovery cut. Do not substitute an empty queue with the
same name: it cannot wake restored retries, timers, or `waiting`/`running`
work. An empty but writable queue is sufficient only for an eligible `pending`
row after you prove that the v1.0.77 Watchdog bootstraps and redispatches it.

**5. Revert the Composer dependency**

```bash
composer require durable-workflow/workflow:^1.0 --with-all-dependencies
```

`durable-workflow/workflow` is the maintained package name for both supported
v1 releases and v2. `laravel-workflow/laravel-workflow` is only a legacy alias
declared by the maintained package for compatibility with older dependency
graphs; do not use it in rollback requirements.

Retrieve the exact `APP_KEY` through the inventoried secret-manager reference
and version, using the separately controlled recovery credentials. Restore it
together with the workflow storage connection, queue connections, queue names,
Redis prefixes, and custom model configuration before starting a consumer.
Encrypted jobs and serialized model references depend on that configuration
matching the recovery cut.

**6. Restart queue workers and reopen ingress**

```bash
# Use the start command for your worker system, for example:
sudo supervisorctl start <your-worker-group>:*
sudo systemctl start laravel-worker
php artisan horizon:continue
```

**7. Verify v1 execution reachability**

Waterline visibility proves that a row was restored; it does not prove that a
worker can resume it. Complete this checklist before declaring rollback
successful:

- [ ] `composer show durable-workflow/workflow` reports the intended v1
  release and workers loaded that code.
- [ ] The actual configured v1 tables and `workflow_relationships_table` are
  present on the expected storage connections.
- [ ] Every nonterminal row in the configured stored-workflow table
  (`workflows` by default) is classified by its next wake path: a
  ready/delayed/reserved queue job, a preserved timer job, or a tested external
  signal ingress. An eligible `pending` row may instead cite the proven
  v1.0.77 Watchdog redispatch evidence below.
- [ ] Queue-provider inspection confirms that each queue-backed retry, timer,
  activity, and `waiting` or `running` workflow wake identified above exists on
  the recorded connection and queue. A SQL timer row by itself does not satisfy
  this check, and the Watchdog is not a substitute for these jobs.
- [ ] If a restored `pending` row has no preserved workflow job, a v1.0.77
  worker loop is observed enqueueing the Watchdog; after the row is stale for
  the five-minute bound, the Watchdog enqueues that workflow on its recorded
  connection and queue and the row advances. Without that end-to-end proof,
  treat the row as stranded even if Waterline displays it.
- [ ] No restored retry, timer, `waiting`, or `running` row lacks its required
  runnable queue job or a usable external signal path for a genuine signal
  wait. Treat such a row as stranded; pending-only Watchdog evidence does not
  make it recoverable.
- [ ] A restored retry or timer advances past its pre-cut marker, and a
  controlled signal-wait can enqueue and consume its signal wake.
- [ ] A new v1 canary workflow completes, and logs contain no missing-job,
  decryption, model-table, connection, or queue-routing errors.

With default tables, this query is a starting inventory; substitute the table
resolved by your configured stored-workflow model when it is customized:

```sql
SELECT id, class, status
FROM workflows
WHERE status NOT IN ('completed', 'failed', 'cancelled');
```

**Important rollback notes:**

- Rollback discards any v2 workflows started after upgrade (they exist only in v2 tables)
- SQL-only rollback restores v1 workflow records, not external queue messages
- In-flight v1 execution is supported only when SQL and durable queue state are
  restored from one application-consistent recovery cut, or when a documented
  signal-only wait retains a usable ingress. The additional SQL-only exception
  is an eligible stale `pending` row with proven v1.0.77 Watchdog redispatch;
  it does not cover retries, timers, or `waiting`/`running` work
- Replaying a pre-upgrade recovery cut can repeat external side effects; review
  application idempotency before restoring queued activity work
- If you must preserve v2 workflows started during the upgrade window, do not restore the database — instead fix the upgrade issue forward

## Code changes

The sections below detail the code-level changes needed when migrating from v1 to v2 APIs.

### Namespace change

All v2 classes live under `Workflow\V2`. Update your imports:

```php
// v1
use Workflow\Workflow;
use Workflow\Activity;
use Workflow\WorkflowStub;

// v2
use Workflow\V2\Workflow;
use Workflow\V2\Activity;
use Workflow\V2\WorkflowStub;
```

### Entry method

v2 workflows and activities use `handle()` as the entry method. Rename v1 `execute()` methods to `handle()` as part of the v2 code migration:

```php
// v1
class MyWorkflow extends Workflow
{
    public function execute($input)
    {
        $result = yield ActivityStub::make(MyActivity::class, $input);
        return $result;
    }
}

// v2
use function Workflow\V2\activity;

class MyWorkflow extends Workflow
{
    public function handle($input)
    {
        return activity(MyActivity::class, $input);
    }
}
```

Do not leave `execute()` as the entry method on a v2 workflow or activity — the runtime rejects it.

### Activity calls

v2 replaces `ActivityStub::make()` and `yield` with direct function helpers:

```php
// v1
$result = yield ActivityStub::make(MyActivity::class, $arg1, $arg2);

// v2
use function Workflow\V2\activity;

$result = activity(MyActivity::class, $arg1, $arg2);
```

Activities now have durable identity. Each scheduled activity gets an `activity_executions` row with a stable execution id, and each concrete attempt gets an `activity_attempts` row with typed history.

### Workflow identity

v2 splits identity into instance id and run id:

- `id()` — the public workflow instance id (same across continue-as-new)
- `runId()` — the id of the current run

In v1, these were the same concept.

### Signals

v2 uses named signal waits instead of `#[SignalMethod]` attribute-based mutators:

```php
// v1
#[SignalMethod]
public function approve()
{
    $this->approved = true;
}

// v2
use function Workflow\V2\await;

$approved = await('approve');
```

Named signals support `await('name')` for blocking workflow-code waits and `signal()` / `attemptSignal()` for external input. Cancellation and termination are not modeled as signals — they remain explicit runtime commands.

### Queries

v2 uses replay-safe query methods instead of reading workflow properties directly:

```php
// v1
#[QueryMethod]
public function getStatus(): string
{
    return $this->status;
}

// v2
use function Workflow\V2\query;

// Queries are defined as named, replay-safe accessors
```

### Timers and side effects

The function-based helpers replace the v1 static methods:

```php
// v1
yield Timer::make(60);
$value = yield SideEffect::make(fn() => random_int(1, 100));

// v2
use function Workflow\V2\timer;
use function Workflow\V2\sideEffect;

timer(60);
$value = sideEffect(fn() => random_int(1, 100));
```

### Timeouts

v2 adds workflow-level timeouts through `StartOptions`:

```php
use Workflow\V2\StartOptions;
use Workflow\V2\WorkflowStub;

$workflow = WorkflowStub::make(MyWorkflow::class, 'order-123');

$workflow->start(
    $orderId,
    StartOptions::rejectDuplicate()
        ->withExecutionTimeout(7200)   // 2 hours across all runs
        ->withRunTimeout(3600),        // 1 hour per run
);
```

- **Execution timeout** spans the entire instance, including continue-as-new transitions.
- **Run timeout** applies to a single run and resets on continue-as-new.

### Database migrations

v2 adds new tables and columns. The package auto-loads its migrations, so after updating:

```bash
composer update durable-workflow/workflow
php artisan migrate
```

The 2.0.0 release includes clean base table migrations. The normal path is to
let Laravel auto-load the package migrations and run `php artisan migrate`.

If you previously published Durable Workflow migrations into your application,
choose one migration source and keep it current:

- **Auto-loaded package migrations**: remove old published Durable Workflow
  migration files from `database/migrations` and run `php artisan migrate`.
- **Published migrations**: publish the current set before migrating:

  ```bash
  php artisan vendor:publish \
    --provider="Workflow\Providers\WorkflowServiceProvider" \
    --tag=migrations \
    --force

  php artisan migrate
  ```

Do not keep stale published files while also relying on newly auto-loaded
package files; that can leave your app missing newer v2 tables or repair
migrations.

If you customized migration files, diff your local copies against the package's
`src/migrations` directory during each upgrade. Keep the table names, columns,
indexes, and nullable/default contracts schema-compatible with the package
models. A customized install that routes workflow tables to a non-default
connection should publish the migrations, set the migration `$connection`, and
then continue carrying forward every new package migration in timestamp order.

For pre-release v2 adopters, `workflow_run_summaries.memo` is repaired
idempotently if an older published summary-table migration created
`workflow_run_summaries` without that column. Fresh installs already create the
column in the base summary-table migration.

### Backend capability check

v2 validates that your queue, database, and cache drivers meet its requirements. Run the doctor command after upgrading:

```bash
php artisan workflow:v2:doctor --strict
```

### Configuration

v2 introduces several new configuration options. See the [Configuration](/docs/configuration/options/) section for details on:

- Durable type aliases
- Task repair policy
- Backend capability checks
- Projection rebuilds
- History budgets and export redaction

### Waterline

Waterline (the monitoring UI) has been updated for v2 with:

- Run detail views showing timeout durations and deadlines
- Activity attempt tracking with durable ids
- Updated workflow status displays

### Continue-as-new

v2 adds history budgets that can automatically trigger continue-as-new when the event count exceeds a threshold. Metadata (memo, search attributes, timeouts) is carried forward across transitions.

## Existing workflows

### Finish-on-v1 strategy

**Workflows started under v1 will continue to execute through v1 compatibility paths.** New workflows started after upgrading will use v2 semantics. You do not need to migrate running workflow instances.

When you upgrade to 2.0:

1. **v1 data is preserved** — The default `workflows`, `workflow_logs`, `workflow_signals`, `workflow_timers`, `workflow_exceptions`, and `workflow_relationships` tables remain intact; configured v1 model/table overrides remain authoritative
2. **v1 workflows complete using v1 engine** — In-flight v1 workflows continue executing using the v1 replay engine until they reach a terminal state
3. **v2 workflows use v2 engine** — All workflows started after upgrade use the v2 schema (`workflow_instances`, `workflow_runs`, `workflow_history_events`, etc.)
4. **Waterline shows both** — The monitoring UI displays v1 and v2 workflows side-by-side

Finish-on-v1 also depends on the Laravel queue backend. Retain every v1 queue
connection and queue until its workflows are terminal. Database rows preserve
history and status; they do not reconstruct lost ready, delayed, or reserved
jobs from an external backend. The v1.0.77 Watchdog can redispatch an eligible
stale `pending` workflow from SQL, but that bounded recovery path does not
recreate retries, timers, or `waiting`/`running` work.

### Tracking v1 workflow completion

To see which v1 workflows are still active after upgrading:

```bash
php artisan workflow:v1:list
```

This command lists all v1 workflows that have not yet reached a terminal state (completed, failed, cancelled). Use it to track v1 workflow completion over time.

Sample output:

```
+--------------------------------------+---------------------+-----------+------------+
| ID                                   | Class               | Status    | Created    |
+--------------------------------------+---------------------+-----------+------------+
| 01J1234567890ABCDEFGHIJK             | App\OrderWorkflow   | running   | 2 days ago |
| 01J9876543210ZYXWVUTSRQP             | App\InvoiceWorkflow | pending   | 1 day ago  |
+--------------------------------------+---------------------+-----------+------------+
```

#### Waterline visibility

After upgrading to 2.0, Waterline automatically shows workflows from both engines:

- **v1 workflows** appear with their original `StoredWorkflow` data (class, status, logs, signals, exceptions)
- **v2 workflows** appear with full v2 detail (runs, history events, timers, activities, search attributes)

No configuration is needed — Waterline reads from both table sets and presents a unified view.

### When to clean up v1 tables

Once all v1 workflows have completed (confirmed by `workflow:v1:list` showing zero active workflows), you may optionally drop the v1 tables:

```sql
DROP TABLE IF EXISTS workflow_relationships;
DROP TABLE IF EXISTS workflow_exceptions;
DROP TABLE IF EXISTS workflow_timers;
DROP TABLE IF EXISTS workflow_signals;
DROP TABLE IF EXISTS workflow_logs;
DROP TABLE IF EXISTS workflows;
```

**Important:** Do not drop these tables while any v1 workflows remain active. Doing so will cause v1 replay to fail and leave workflows stuck.

### Why finish-on-v1?

The finish-on-v1 strategy avoids forcing a data migration at upgrade time. v1 and v2 use fundamentally different storage models:

- **v1** stores workflow state as a denormalized `workflows` row with related logs, signals, and timers
- **v2** stores workflow state as event-sourced history with projections (`workflow_instances`, `workflow_runs`, `workflow_history_events`)

Converting in-flight v1 workflows to v2 history would require reconstructing event sequences from v1 logs, which risks data loss and replay inconsistencies. The finish-on-v1 approach lets v1 workflows complete safely on their original engine while new work moves to v2 immediately.

<!-- Source: docs/waterline-operator-api.md -->

# Waterline Operator API Reference

Waterline is the durable-state operator surface for both embedded Laravel and
standalone-server deployments. The UI uses the same HTTP+JSON API documented
here, so scripts, dashboards, and agents can read durable workflow facts
without scraping HTML.

Use this reference when you need typed operator evidence: selected-run detail,
history export, actionability, command affordances, saved views, preferences,
and schedule visibility. Use [Monitoring](./monitoring.md) for the conceptual
split between Waterline durable state and worker/runtime telemetry.

## Installation

Install Waterline in the same embedded Laravel application that installs the
workflow package:

```bash
composer require \
  durable-workflow/waterline:2.0.0 \
  durable-workflow/workflow:2.0.1 \
  durable-workflow/sdk:2.0.0
php artisan waterline:install
```

The generated pins select stable 2.x releases of Waterline, Workflow, and the
PHP SDK. After upgrading Waterline, publish the current assets:

```bash
composer update durable-workflow/waterline
php artisan waterline:publish
```

For the self-contained service image and its connection, authentication, and
persistence inputs, use [Monitoring](./monitoring.md#waterline-service).

## Deployment Boundary

Waterline has two backend adapters:

- Embedded mode runs inside the Laravel host and reads the workflow package's
  durable state in process.
- Service mode runs the published Waterline image and reads one standalone
  server namespace through `durable-workflow/sdk` and the server's public API.

Both adapters expose the Waterline routes documented on this page. Service
mode translates those routes to the corresponding server contracts:

| Waterline route family | Standalone-server contract used by service mode |
| --- | --- |
| `GET /waterline/api/v2/health` | Server health, worker registration, and task-queue visibility. |
| `GET /waterline/api/stats` | The namespace-scoped operator dashboard and metrics snapshot. |
| Flow lists, selected-run detail, and history export | Workflow list, detail, runs, history, and diagnostics from the [Server API Reference](./polyglot/server-api-reference.md). |
| Signal, update, query, cancel, terminate, repair, and archive actions | The matching server workflow command and repair contracts. |
| Routes under `/waterline/api/v2/schedules` | The server schedule list, detail, history, and mutation contracts. |

The server API, CLI, and SDK surfaces remain available directly when
Waterline is deployed. Do not assume cross-mode or cross-namespace visibility:
runs stay readable and actionable from their owning runtime and namespace.
[Deployment Modes](./polyglot/deployment-modes.md) defines that shared product
boundary.

## Base Path And Scope

Waterline mounts under the Laravel app's configured Waterline base path. The
examples below use `/waterline`; if your app publishes Waterline under a
different prefix, keep the `/api/...` suffixes the same.

When `WATERLINE_NAMESPACE` is configured, every list, detail, schedule, and
operator-action route is namespace scoped. Cross-namespace reads and commands
return not-found semantics instead of leaking another namespace's workflow
state.

```bash
curl -sS "$APP_URL/waterline/api/instances/order-1001" \
  -H "Accept: application/json" | jq '.status, .run_id, .actionability'
```

## Authentication

Every Waterline UI and JSON API controller passes through Waterline's access
check. In embedded mode, the Laravel host defines that check and decides which
guards, route middleware, gates, policies, SSO/OIDC/SAML sessions,
directory-backed groups, service tokens, and rate limits apply.

The current JSON operator API group excludes Laravel's CSRF middleware so a
host can apply a session or service-token boundary consistently to both reads
and commands. The host remains responsible for authenticating those requests
and for documenting any browser-session CSRF protections it adds around the
API.

The self-contained service image has no host-application users. Put it behind
an authenticating reverse proxy or on a private interface before setting
`WATERLINE_ALLOW_UNAUTHENTICATED=true`; that setting delegates the Waterline
front-door check to the surrounding deployment. Separately,
`WATERLINE_SERVER_TOKEN` authenticates Waterline's PHP SDK calls to the
standalone server. Never expose that server credential to the browser.

`WATERLINE_ACCESS_MODE=read_only` blocks mutating Waterline routes locally.
`operator` enables them, but the connected server still applies its own token
role and namespace authorization. See
[Monitoring](./monitoring.md#waterline-service) for the deployment example and
the two authentication boundaries.

## Dashboard And Health

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/waterline/api/stats` | Dashboard totals, backlog counters, repair policy, operator metrics, and fleet trend data. |
| `GET` | `/waterline/api/v2/health` | Waterline package/runtime health and v2 data-source readiness. |

`/waterline/api/stats` is the dashboard's summary surface. Treat
`operator_metrics` as Waterline-owned JSON diagnostics, not as a Prometheus
scrape surface. Use worker SDK metrics for runtime latency and custom
application telemetry.

The [Operator Operating Envelope](./operator-operating-envelope.md) defines how
to interpret those diagnostics during rollouts and incident response. In
particular:

| Field family | Meaning |
| --- | --- |
| `operator_metrics.backlog.*` | Durable runnable, delayed, leased, unhealthy, repair-needed, claim-failed, and compatibility-blocked work counts, plus the fleet-level `tasks_added_last_minute` and `tasks_dispatched_last_minute` queue-flow facts. |
| `operator_metrics.matching_role.*` | The node-local matching/dispatch contract for the process serving the request: `queue_wake_enabled`, deployment `shape`, `task_dispatch_mode`, frozen `partition_primitives`, and current `backpressure_model`. |
| `operator_metrics.repair.*` | Repair-loop sweep footprint, including selected candidates, candidate age, and scan pressure. |
| `operator_metrics.projections.*` | Projection-drift counts for run summaries, waits, timelines, timers, and lineage. |
| `operator_metrics.command_contracts.*` | Legacy WorkflowStarted contract snapshots that still need backfill. |
| `operator_metrics.history.*` | History-size and event-count pressure plus continue-as-new recommendations. |
| `queue_visibility.available`, `queue_visibility.reason` | Whether namespace-scoped queue visibility is available on this host app, and the reason when it is not. |
| `queue_visibility.task_queues[].stats.*` | Queue-local backlog, backlog age, poller counts, workflow/activity ready-versus-leased counts, and per-queue `tasks_added_last_minute` / `tasks_dispatched_last_minute` flow facts. |
| `queue_visibility.task_queues[].repair.*` | Queue-local repair pressure, including candidates, dispatch failures, expired leases, dispatch-overdue counts, and the oldest age for each condition. |
| `coordination_alerts[]` | Roll-up warnings and errors derived from the health checks plus queue-visibility risks such as backlog without pollers, stale pollers, or aged repair candidates. |
| `checks[]`, `categories.*` | Blocking versus advisory v2 health checks, with per-check `category = correctness | acceleration` and the category rollups Waterline uses on the workers surface. |
| `engine_source`, `readiness_contract` | Whether Waterline is actively using the v2 operator bridge and which readiness contract governs that state. |

The queue-flow fields answer a specific operator question: is durable queue
input arriving faster than the system is dispatching it? `tasks_added_last_minute`
counts distinct durable task rows created in the trailing 60 seconds, and
`tasks_dispatched_last_minute` counts distinct durable task rows whose latest
successful dispatch landed in that same window.

Use the queue-flow totals with the server task-queue visibility routes rather
than by themselves. Waterline tells you whether durable inflow is outrunning
dispatch, while `/api/task-queues` and `/api/task-queues/{taskQueue}` tell you
whether the hot queue is `saturated`, intentionally `throttled`,
`no_active_workers`, or otherwise short on healthy pollers or slots.

`GET /waterline/api/v2/health` exposes the same queue-local evidence without
leaving the Waterline surface. Scripts and dashboards that need one namespace's
current queue posture should read `queue_visibility.task_queues[]` for
per-queue stats and `coordination_alerts[]` for the summary of which queue or
health-check condition currently needs operator attention.

The `matching_role` fields answer a different question: which matching shape is
this node currently serving? `shape` distinguishes `in_worker` from
`dedicated`, `partition_primitives` freezes the routing axes as
`connection`/`queue`/`compatibility`/`namespace`, and `backpressure_model`
currently reports `lease_ownership`. Treat that block as process-local. During
mixed-shape rollouts, compare it across nodes before assuming backlog or poll
differences mean worker trouble.

`GET /waterline/api/v2/health` uses the same distinction: `error` is blocking,
`warning` is advisory, and `ok` means the current v2 operator bridge is ready.

## List Views

List views are bucketed by durable status:

| Method | Path | Bucket |
| --- | --- | --- |
| `GET` | `/waterline/api/flows/running` | Open or actively blocked runs. |
| `GET` | `/waterline/api/flows/completed` | Runs closed successfully, including continued runs with `closed_reason = continued`. |
| `GET` | `/waterline/api/flows/failed` | Failed runs. |
| `GET` | `/waterline/api/flows/cancelled` | Cancelled runs. |
| `GET` | `/waterline/api/flows/terminated` | Terminated runs. |

List rows are compact operator summaries. Stable fields include workflow
identity (`id`, `instance_id`, `run_id`, `workflow_type`), state (`status`,
`closed_reason`, `archived_at`), timing, `history_event_count`,
`history_size_bytes`, `continue_as_new_recommended`, and repair/actionability
badges such as `repair_blocked`.

## Selected-Run Detail

Waterline has two addressing modes:

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/waterline/api/instances/{instanceId}` | Current selected run for a workflow instance. |
| `GET` | `/waterline/api/instances/{instanceId}/runs/{runId}` | Explicit selected run. |
| `GET` | `/waterline/api/flows/{id}` | Legacy/detail lookup by Waterline row or run id. |

Selected-run detail is the authoritative JSON contract for operator screens.
It contains durable state and derived diagnostics for:

| Field family | Meaning |
| --- | --- |
| `activities`, `timers`, `waits`, `children` | Current and historical wait state rebuilt from typed history first. |
| `signals`, `updates`, `declared_signals`, `declared_updates`, `declared_queries` | Command lifecycle rows plus loadable workflow contract targets. |
| `timeline` | Ordered durable events and diagnostic entries. |
| `exceptions`, `logs` | Failure facts and replay/debug context. |
| `can_signal`, `can_update`, `can_query`, `can_cancel`, `can_terminate`, `can_archive` | UI and automation affordances for the selected run. |
| `actionability` | Versioned repair and evidence contract described below. |

Automation should prefer selected-run detail over visual screenshots. A
screenshot can show what an operator saw; selected-run detail explains why a
workflow can or cannot be repaired, queried, cancelled, archived, or replayed.

## History Export

History exports are replay/debug bundles. They intentionally include stored
workflow, command, activity, update, task, failure, and history payloads so a
developer can reproduce or archive the selected run.

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/waterline/api/instances/{instanceId}/history-export` | Export the current selected run for an instance. |
| `GET` | `/waterline/api/instances/{instanceId}/runs/{runId}/history-export` | Export an explicit run. |
| `GET` | `/waterline/api/flows/{id}/history-export` | Legacy/detail export by Waterline row or run id. |

If an export can leave a protected environment, configure the workflow
history-export redactor first. Downstream tools should preserve the export's
`redaction` metadata so reviewers can tell which policy shaped the artifact.

## Actionability Contract

Waterline annotates list rows, selected-run detail responses, timeline entries,
and history exports with the versioned actionability contract:

```json
{
  "actionability_contract": {
    "schema": "waterline.actionability",
    "version": 1
  }
}
```

The contract identifier is `actionability_contract.schema = waterline.actionability`
with `actionability_contract.version = 1`.

Run-level `actionability` answers whether the selected run can be repaired:

| Field | Meaning |
| --- | --- |
| `repair_state` | One of `repairable`, `blocked`, `not_needed`, or `unknown`. |
| `repairable` | Boolean shorthand for `repair_state = repairable`. |
| `blocked_reason` | Stable reason code when `repair_state = blocked`. |
| `status_bucket` | The Waterline bucket that shaped the run-level decision. |
| `closed_reason` | Durable close reason when the run is closed. |
| `task_problem` | Whether Waterline saw a task-level problem on the run. |
| `diagnostic_only_evidence` | True when at least one row is informative but not a resume source. |

Evidence rows under `activities`, `waits`, `timers`, `exceptions`, `logs`, and
timeline/export entries can include their own `actionability` block:

| Field | Meaning |
| --- | --- |
| `state` | `actionable` when the row is a valid repair source, otherwise `diagnostic_only`. |
| `repair_source` | True only for rows backed by a repairable source authority. |
| `diagnostic_only` | True when the row must not be used as a resume source. |
| `history_authority` | Source authority such as `typed_history`, `mutable_open_fallback`, `failure_row_fallback`, or `unsupported_terminal_without_history`. |
| `history_unsupported_reason` | Stable reason code for unsupported fallback history. |

Agents and scripts must gate repair/resume decisions on
`actionability.repair_state`, `actionability.repairable`, and row-level
`actionability.repair_source`. A row with `diagnostic_only = true` is evidence,
not permission to replay or resume.

## Operator Actions

Waterline actions are durable commands issued through the selected-run contract.
Instance routes target the current run; run routes reject stale or wrong-run
targets explicitly.

| Method | Path | Purpose |
| --- | --- | --- |
| `POST` | `/waterline/api/instances/{instanceId}/queries/{query}` | Execute a declared query against the current run. |
| `POST` | `/waterline/api/instances/{instanceId}/signals/{signal}` | Send a signal to the current run. |
| `POST` | `/waterline/api/instances/{instanceId}/updates/{update}` | Submit an update to the current run. |
| `POST` | `/waterline/api/instances/{instanceId}/repair` | Dispatch a repair pass for the current run when actionability allows it. |
| `POST` | `/waterline/api/instances/{instanceId}/cancel` | Request cancellation for the current run. |
| `POST` | `/waterline/api/instances/{instanceId}/terminate` | Terminate the current run. |
| `POST` | `/waterline/api/instances/{instanceId}/archive` | Archive the selected closed run. |
| `POST` | `/waterline/api/instances/{instanceId}/runs/{runId}/queries/{query}` | Execute a query against an explicit selected run. |
| `POST` | `/waterline/api/instances/{instanceId}/runs/{runId}/signals/{signal}` | Send a signal only if the selected run is current. |
| `POST` | `/waterline/api/instances/{instanceId}/runs/{runId}/updates/{update}` | Submit an update only if the selected run is current. |
| `POST` | `/waterline/api/instances/{instanceId}/runs/{runId}/repair` | Repair an explicit selected run. |
| `POST` | `/waterline/api/instances/{instanceId}/runs/{runId}/cancel` | Cancel only if the selected run is current. |
| `POST` | `/waterline/api/instances/{instanceId}/runs/{runId}/terminate` | Terminate only if the selected run is current. |
| `POST` | `/waterline/api/instances/{instanceId}/runs/{runId}/archive` | Archive the selected closed run. |

Signals, updates, and queries accept JSON `arguments`. Invalid arguments return
field-level validation failures. Replay blocks return `409 Conflict` with a
stable reason such as `workflow_definition_unavailable` instead of attempting
best-effort execution.

## Update Inspection

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/waterline/api/instances/{instanceId}/updates/{updateId}` | Inspect an update lifecycle row for the current run. |
| `GET` | `/waterline/api/instances/{instanceId}/runs/{runId}/updates/{updateId}` | Inspect an update lifecycle row for an explicit run. |
| `GET` | `/waterline/api/flows/{id}/updates/{updateId}` | Legacy/detail update lookup. |

Use update inspection when a UI or agent needs to explain whether an update is
accepted, applied, completed, rejected, timed out while waiting, or blocked by
replay compatibility.

## Schedules

Waterline schedule routes are v2 visibility and operator-action routes:

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/waterline/api/v2/schedules` | List schedules visible in the current namespace. |
| `GET` | `/waterline/api/v2/schedules/{scheduleId}` | Describe one schedule. |
| `POST` | `/waterline/api/v2/schedules/{scheduleId}/pause` | Pause future fires. |
| `POST` | `/waterline/api/v2/schedules/{scheduleId}/resume` | Resume a paused schedule. |
| `POST` | `/waterline/api/v2/schedules/{scheduleId}/trigger` | Trigger an immediate fire. |
| `POST` | `/waterline/api/v2/schedules/{scheduleId}/backfill` | Backfill a time window. |
| `DELETE` | `/waterline/api/v2/schedules/{scheduleId}` | Delete a schedule. |

Schedule responses expose action, status, next fire, recent fire history,
overlap policy, note, memo, and search attributes for operator review.

## Saved Views And Preferences

Saved views and preferences are UI-owned operator configuration, not workflow
history. They are still useful for stable runbooks because they let teams share
the exact filters and table layout used during an incident.

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/waterline/api/saved-views` | List saved operator views. |
| `POST` | `/waterline/api/saved-views` | Create a saved view. |
| `GET` | `/waterline/api/saved-views/{view}` | Read one saved view. |
| `PUT` | `/waterline/api/saved-views/{view}` | Update a saved view. |
| `DELETE` | `/waterline/api/saved-views/{view}` | Delete a saved view. |
| `GET` | `/waterline/api/preferences/{surface}` | Read UI preferences for one surface. |
| `PUT` | `/waterline/api/preferences/{surface}` | Update UI preferences for one surface. |

Do not treat saved-view names or preference payloads as durable workflow facts.
For evidence, cite selected-run detail or history export.

## Error Contract

Waterline uses ordinary HTTP status codes and JSON error details:

| Status | Meaning |
| --- | --- |
| `400` | Malformed request or unsupported action payload. |
| `404` | Workflow instance, run, update, schedule, saved view, or preference surface was not found in the current namespace. |
| `409` | The selected run cannot execute the action, often because it is historical, closed, replay-blocked, or definition-unavailable. |
| `422` | Validation failed; response includes field-level errors. |
| `500` | Unexpected application failure. |

Automation should branch on status plus stable fields such as `blocked_reason`,
`query_blocked_reason`, `outcome`, `reason`, and `actionability` values. Do
not parse toast text or button labels as the contract.

## See Also

- [Monitoring](./monitoring.md)
- [Failures and Recovery](./failures-and-recovery.md)
- [Queries](./features/queries.md)
- [Cancel and Terminate](./features/cancel-and-terminate.md)
- [Agent Tooling Contract](./agent-tooling-contract.md)

<!-- Source: docs/operator-operating-envelope.md -->

# Operator Operating Envelope

This guide defines the operator-facing contract for Durable Workflow v2.
Use it to decide which diagnostics block rollouts, which ones are advisory,
which queue facts belong to Waterline versus worker telemetry, how to verify
rebuild and export workflows, and which deployment shapes are part of the
documented operating envelope.

The deployment and recovery procedures here apply to embedded Laravel and
self-hosted Server. Durable Workflow Cloud is a separate managed service: Cloud
owns runtime persistence, single-region placement, backup, restore, and
recovery, while
customers operate SDK clients and workers through the namespace runtime URL.
Do not use this guide to attach a Server to Cloud; use the
[Cloud Managed Runtime](/docs/polyglot/cloud-control-plane) guide for that
customer boundary.

## Source-of-truth surfaces

Use these surfaces together:

| Surface | Use it for | Contract class |
| --- | --- | --- |
| `php artisan workflow:v2:doctor --strict` | Backend capability gating before v2 traffic or upgrades | Blocking |
| `GET /waterline/api/v2/health` | Current engine-source readiness plus blocking vs advisory v2 health checks | Blocking when `status = error`, advisory when `status = warning` |
| `GET /waterline/api/stats` | Durable fleet totals, backlog counters, repair-loop facts, projection drift counts, worker compatibility summaries | Advisory and benchmarking |
| `php artisan workflow:v2:rebuild-projections ...` | Previewing and repairing projection drift | Maintenance |
| `php artisan workflow:v2:backfill-command-contracts ...` | Previewing and backfilling legacy command-contract snapshots | Maintenance |
| `php artisan workflow:v2:history-export ...` and Waterline history-export routes | Replay, archive handoff, and incident artifacts | Verification |
| Waterline archive actions and control-plane `archive()` | Lifecycle state transitions for closed runs | Lifecycle |
| Worker SDK metrics, traces, and logs | Schedule-to-start latency, poll success, sticky-cache behavior, and custom application telemetry | Runtime telemetry |

The durable-state operator contract lives in the runtime that owns each run.
Waterline projects that contract either from the embedded workflow package or
from a standalone server through the PHP SDK. Worker telemetry remains the
source of truth for latency and process-level behavior inside your workers.

### Surface mapping by deployment shape

The Waterline routes in the table above are available from both Waterline
deployment shapes. Embedded mode reads the Laravel host's durable state in
process. Service mode runs the published Waterline image and reads one
standalone-server namespace through the PHP SDK. The authenticated server API
and `dw` CLI remain available directly with or without Waterline:

| Operator question | Waterline (embedded or service mode) | Standalone-server native surface |
| --- | --- | --- |
| Engine-source readiness and blocking vs advisory health | `GET /waterline/api/v2/health` | `GET /api/system/health` (admin auth, control-plane v2); `dw server:health` for liveness and `dw server:info` for the topology, protocol, and rollout-safety summary |
| Durable fleet totals, backlog, repair, worker compatibility, projection drift | `GET /waterline/api/stats` | `GET /api/system/operator-metrics` and `dw system:operator-metrics` |
| Selected-run detail and history export | `GET /waterline/api/instances/...` and `/waterline/api/.../history-export` | `GET /api/workflows/{workflowId}`, `/runs/{runId}`, and `/runs/{runId}/history/export` (see the [Server API Reference](./polyglot/server-api-reference.md)) |
| Operator commands (cancel, terminate, repair, archive, signal/update/query) | `POST /waterline/api/instances/.../{cancel\|terminate\|repair\|archive}` and signal/update/query routes | `POST /api/workflows/{workflowId}/{cancel\|terminate\|repair\|archive}` and `POST /api/system/repair/pass` |
| Topology and node-identity discovery | Embedded: `php artisan workflow:v2:doctor --json` (`topology` object); service: use the connected server's native surface | `GET /api/cluster/info`, `GET /api/health`, `GET /api/ready` (or `dw server:info`) |

The field families and contract names below stay the same regardless of
which surface you read them through. A Waterline deployment is scoped to its
configured runtime and namespace; it does not combine embedded and
server-managed runs.

## Dated 2.0 evidence snapshot

This snapshot separates behavior that has been measured on released artifacts
from procedures that the product supports but each operator must rehearse in
their own environment. A passing row applies to the documented 2.0 scenario
and topology; it is not a blanket claim for different substrates or larger
deployments.

The current proof inventory is:

| Proof and date (UTC) | What the proof directly measured | Boundary |
| --- | --- | --- |
| Single-region failover rehearsal, 2026-07-28 | Cross-node terminal completion, loss of one API node, MySQL interruption and recovery, Redis interruption and database-poll fallback, worker lease loss and reclaim, and singleton scheduler restart. The result recorded passing loss and duplicate assertions for every phase. | Two API nodes behind one shared endpoint, one MySQL instance, one Redis acceleration layer, one queue worker, and one scheduler/maintenance runner in one region. The database and Redis events were container interruptions, not managed-provider promotion tests. |
| Timer restart matrix, 2026-07-28 | Sleeping workflows completed after worker restart and server restart, with one timer schedule and fire per run and no duplicate replay command. | Durable timer and replay recovery on the published-server test topology; not arbitrary process state or external side-effect recovery. |
| Activity runtime matrix, 2026-07-29 | Worker-restart result recovery, retry and timeout behavior, heartbeat and cancellation observation, durable terminal result recording, and idempotent completion handling across the required product surfaces. | Activity-level engine behavior. Applications still own idempotency for external side effects. |
| Workflow lifecycle matrix, 2026-07-29 | Terminal completion and failure states, duplicate-start and workflow-id reuse policy, timeout and retry behavior, history continuity, and duplicate side-effect prevention across PHP, Python, Rust, CLI, API, history, and Waterline. | Lifecycle correctness for the exercised scenarios; not an availability or throughput benchmark. |

Use the [Compatibility contract](./compatibility.md) for supported release
lines, and rehearse the documented recovery procedure against your deployment
before making a topology-specific failover claim.

### Measured guarantee vs supported procedure

| Failure or outcome | Directly proven by the dated evidence | Supported operator procedure | Not established by this evidence |
| --- | --- | --- | --- |
| One API-node loss | The failover rehearsal killed one of two API nodes, reached the surviving node through the shared endpoint, preserved acknowledged state, and completed the run. | Remove failed nodes with `/api/ready`, retry interrupted client requests, then verify topology and worker registration before returning traffic. | Whole-fleet loss, availability-zone isolation, or arbitrary network partition behavior. |
| Database interruption | Both API nodes became not ready, durable state survived the interruption, a replacement worker reclaimed the task after lease expiry, the run completed, and a duplicate completion was refused. | Restore the writable database first, wait for readiness, fence the stale task owner, verify one representative run, then resume traffic. Managed-service promotion needs provider-native fencing, RPO, and elapsed-time evidence in the recovery packet. | Managed database promotion, acknowledged-write behavior during split brain, or an RPO/RTO beyond the measured local interruption. |
| Redis interruption | Readiness reported acceleration degradation while database-backed polling preserved durable state; replaying the same poll request did not create a duplicate lease; readiness recovered after Redis returned. | Keep Redis as an acceleration layer, tolerate warning readiness, restore it after durable persistence, and verify poll-discovery latency returns to the deployment baseline. | Redis dual-primary behavior, cache behavior across a network partition, or provider promotion timing. |
| Worker restart or loss | The failover proof reclaimed and completed a leased workflow after worker loss; the newer activity matrix proved durable result recovery after worker restart and idempotent completion handling. | Let leases expire or drain workers, start a compatible replacement, verify registration and queue pickup, and keep external effects idempotent. | Recovery of process-local state that was not stored durably or exactly-once external side effects. |
| Server or scheduler restart | The timer matrix completed sleeping workflows after a server restart without duplicate timer commands. The failover proof also preserved progress through one API-node loss and one singleton scheduler restart. | Restore persistence, bring server roles back to readiness, confirm exactly one scheduler/maintenance runner, then verify worker registration and one representative completion. | Simultaneous loss of every server and persistence node, multi-region failover, or duplicate schedulers as a steady-state topology. |
| Duplicate delivery | Database-recovery and worker-loss phases each recorded one logical completion and rejected the duplicate completion with HTTP `409`; Redis degradation replayed the original lease without issuing a duplicate. The activity matrix separately passed its idempotent-completion cell. | Use stable request and task identities, honor stale-attempt refusals, and make activity side effects idempotent. | A universal exactly-once delivery or side-effect guarantee outside the durable engine boundary. |
| Terminal completion | Cross-node, API-loss, database-recovery, and worker-loss workflows all reached terminal `completed` state; the newer lifecycle matrix passed completion, failure, cancellation, termination, timeout, and retry cells. | Verify the selected run and its history/export evidence before archiving or declaring recovery complete. | Completion of workload shapes, integrations, or failure combinations that were not exercised. |

### Largest current load cell

The largest configured server load cell starts **1,000 workflows** with start
concurrency **8** while also running **8** concurrent polling workers for 120
seconds. It requires at least a 98% workflow-start completion ratio and checks
endpoint availability, cache-key drainage, sample coverage, and bounded
resource ceilings.

Treat this as a short **correctness smoke**, not a sustained production-scale
benchmark. It does not establish a universal throughput, latency, memory, or
maximum-concurrency promise. Establish those values with the benchmark and
long-soak packet for the exact topology, worker mix, database, cache, and
release you intend to operate.

### Explicitly out of scope

The 2.0 evidence claim does not cover:

- network partitions
- deliberate clock drift
- multi-region operation
- split-brain behavior

Those experiments are not release prerequisites for 2.0. They are prerequisites
only for making the corresponding partition, clock-skew, multi-region, or
split-brain claim. The supported 2.0 release envelope remains the published
topology plus its recovery packet and the behavior measured by the documented
scenarios; unsupported chaos scenarios should not be presented as
evidence that the documented single-region procedures are unusable.

## Supported topologies

Durable Workflow v2 supports these operator shapes. The shape names in the
first column match the `topology.current_shape` values published by
`/api/cluster/info` and the [Server Role Topology](./polyglot/server-role-topology.md)
manifest, so the operator contract here lines up with the discovery contract
your automation already reads.

| Operator shape (`topology.current_shape`) | Supported operator contract | Primary failure domains | Recovery and failover expectation |
| --- | --- | --- | --- |
| `embedded`, single node | Waterline, control-plane routes, health, rebuild, export, and archive all run from one app process against one durable database and one cache store. | The Laravel app process, the durable database, and the cache store on one host. | Treat host or database loss as a full service interruption. Restore durable state first, bring one app node back to readiness, then verify worker registration before resuming traffic. |
| `embedded`, small same-region cluster | Use one shared database, one shared cache backend for wake-signal coordination, identical workflow compatibility/config across nodes, and keep active nodes in the same datacenter or region so queue wake-up and timer wake-up latency stay bounded. | Shared database, shared cache/wake coordination, load balancer routing, and the singleton scheduler or maintenance role. | One app-node loss should reduce capacity, not correctness. Database loss blocks durable traffic; Redis-only loss degrades wake acceleration and reports a readiness warning while database polling preserves durable correctness. Scheduler failover and upgrades remain explicit operator procedures rather than automatic HA promises. |
| `standalone_server` distribution | Use the [Self-Hosting Deployments](./deployment.md) guide for the server-specific deployment matrix, then apply the same health, stats, export, archive, and queue-health distinctions through the native `/api/system/...` and `/api/workflows/...` routes or a separately deployed Waterline service (see the surface mapping above). | Shared database, shared Redis, API container set, independently scaled workers, the optional Waterline observer, and the single scheduler or maintenance runner. | API containers are replaceable server process nodes, and the Waterline container is a replaceable observer of server-owned state. The database, Redis, and singleton scheduler path define recovery order. Restore persistence first, then verify `/api/ready`, `/api/cluster/info`, and worker registration before shifting traffic back. |
| `split_control_execution` | Same product contract as `standalone_server`, with each role isolated into its own process class (`ingress_node`, `control_plane_node`, `scheduler_node`, `matching_node`, `execution_node`). The same operator-metrics, health, and command surfaces apply per-node; route admin reads to the node that hosts the role you are interrogating. | Each role runs as its own process class, so the failure-domain checklist in [Server Role Topology](./polyglot/server-role-topology.md) governs which subsystem fails first. The shared database, Redis, and singleton scheduler election remain fleet-wide failure domains. | Recovery follows the same order as `standalone_server`, but verify `topology.current_shape`, `topology.current_process_class`, and `topology.current_roles` per node before declaring the deployment ready. Hosted routes return `503 topology_role_unavailable` when sent to the wrong node class. |

`split_control_execution` is not a separate engine or product. It is the same
operator contract as `standalone_server` with the role-specific process classes
named in `topology.shape_assignments`. Treat the rest of this guide as
shape-agnostic for those two server shapes unless a section calls out a
specific role. [Server Role Topology](./polyglot/server-role-topology.md)
holds the role vocabulary, authority boundaries, and migration path.

Waterline service mode is an observer deployment, not another server topology.
Adding or removing it does not change `topology.current_shape`, server-native
API or CLI availability, or the runtime that owns a run.

Publish the restore order, backup cadence, expected failover lag, and any
region-pinned behavior in the runbook for the topology you operate. The product
contract tells you which facts to measure; your deployment contract records the
recovery timing, manual steps, and failure domains you accept.

### Failure-domain checklist by supported shape

Use the topology table above as the quick summary, then write your runbook
against these more explicit loss models:

- **Embedded Laravel, single node**: One application process owns the control
  plane, matching, projection, scheduler, and execution roles together. Losing
  that process is a full service interruption for durable commands, workflow
  progress, schedule firing, and operator reads until the same app returns to
  readiness against intact durable storage.
- **Embedded Laravel, small same-region cluster**: Losing one ordinary app node
  should remove only a share of HTTP and worker capacity while the remaining
  nodes keep claiming work from the shared durable store. Treat the shared
  database, the shared cache-backed wake path, and whichever node currently
  owns the singleton scheduler or maintenance duty as the main correctness
  boundaries for the fleet.
- **Standalone server distribution (`standalone_server`)**: Losing one
  `server_http_node` should stop ingress and control-plane commands only on
  that node; healthy worker nodes can still finish leased work and other API
  nodes can keep serving traffic. Losing one `worker_node` should raise
  backlog, queue age, or compatibility warnings only for the affected
  `(connection, queue, compatibility)` scopes. Losing the `scheduler_node`
  should pause new schedule fires and maintenance sweeps without invalidating
  already running workflows. Database loss is a fleet-level outage. Redis-only
  loss keeps durable database polling available, reports
  `long_poll_wake_acceleration` as degraded, and increases discovery latency
  until Redis reconnects.
- **Split-role server distribution (`split_control_execution`)**: Each role
  runs as its own process class — `ingress_node`, `control_plane_node`,
  `scheduler_node`, `matching_node`, and `execution_node`. Losing any one
  process class only degrades the role it owns: ingress loss stops external
  HTTP traffic at the edge, control-plane loss makes operator commands fail
  fast while leased work continues, matching loss falls back to direct
  ready-task discovery, scheduler loss pauses schedule fires and records
  missed runs, and execution loss accumulates ready tasks without losing
  durable state. Database loss remains a fleet-wide outage. Redis-only loss is
  an acceleration-layer degradation with the same database-poll fallback and
  warning readiness behavior as the `standalone_server` shape.

If your deployment depends on different assumptions, treat that topology as a
separate runbook with its own validated contract instead of assuming the
self-serve guidance still applies unchanged.

### Published recovery packet by topology

The supported topologies above are only production-ready when the deployment
runbook publishes the matching recovery packet alongside them:

| Topology | Publish these operator-owned facts |
| --- | --- |
| `embedded`, single node | Backup schedule for the database, cache-preservation expectations, the exact app revision and env/config snapshot used for restore, the maximum accepted restore lag, and the latest successful restore rehearsal evidence. |
| `embedded`, small same-region cluster | Everything from the single-node packet, plus which node or process currently owns scheduler or maintenance duty, the expected impact of losing one ordinary node versus losing the shared database or cache backend, and the failover steps required to restore queue wake coordination. |
| `standalone_server` distribution | Database and Redis backup cadence, pinned server image or digest, auth-material location, the expected failover behavior for `server_http_node`, `worker_node`, and `scheduler_node`, the latest `/api/ready` plus `/api/cluster/info` restore verification evidence, and the latest worker re-registration proof after restore. |
| `split_control_execution` distribution | Everything from the `standalone_server` packet, plus the per-process-class scaling and failure expectations for `ingress_node`, `control_plane_node`, `scheduler_node`, `matching_node`, and `execution_node`, and the routing rules clients use when a hosted route returns `503 topology_role_unavailable` from the wrong node class. |

If that packet is missing, stale, or untested, treat the topology as
development-grade regardless of how many nodes are currently running.

### Verify live topology identity before trusting the baseline

For standalone-server and split-role deployments, confirm the node identity
that the product itself reports before you interpret queue, scheduler, or role
failure signals. `GET /api/cluster/info` is the source of truth for that
identity:

| Field | Use it for |
| --- | --- |
| `topology.current_shape` | Confirms whether the node is currently advertising `embedded`, `standalone_server`, or `split_control_execution`. |
| `topology.current_roles` | Confirms the logical roles actually hosted by this node. |
| `topology.supported_shapes` | Confirms which deployment shapes the current server build publicly supports. |
| `topology.shape_assignments` | Maps each supported shape to its documented process-class role bundles so you can compare the current role bundle against the supported topology. |

Use those fields as the first topology-drift check during rollouts:

- In the self-serve standalone-server shape, API nodes should continue to
  report the `api_ingress`, `control_plane`, `matching`, and
  `history_projection` role bundle; scheduler nodes should report `scheduler`;
  worker nodes should report `execution_plane`.
- In the split-role shape, verify that each node's `current_roles` match one of
  the documented role bundles under `shape_assignments` before you interpret
  backlog or scheduler lag as a worker problem.
- If `current_roles` drift from the deployment plan,
  treat queue and failover baselines as suspect until the node identity is
  corrected.

Embedded installs do not publish `/api/cluster/info`. For the package-local
topology view, run `php artisan workflow:v2:doctor --json` and inspect the
`topology` object. It publishes the same role-topology schema and includes the
embedded app's `current_shape`, `current_process_class`, `current_roles`,
`execution_mode`, and nested `matching_role` summary.

## Blocking and advisory diagnostics

Durable Workflow v2 separates blocking diagnostics from advisory diagnostics.

| Severity | Meaning | Typical operator action |
| --- | --- | --- |
| Blocking | The current configuration or readiness state is not safe to trust for v2 traffic | Stop rollout, fix the prerequisite, rerun verification |
| Advisory | The surface remains readable, but some derived facts need rebuild, backfill, or manual review before you rely on them | Keep serving traffic when appropriate, then repair the named surface |
| Healthy | No current issue was found in that surface | Continue normal operation |

Apply that rule to the shipped surfaces:

- `workflow:v2:doctor --strict` blocks when backend capability issues have
  `error` severity. Examples include an unsupported queue driver in queue mode
  or a cache store without locks. Informational queue diagnostics in poll mode
  remain advisory.
- `GET /waterline/api/v2/health` returns:
  - `status = ok` when the v2 operator surface is ready and the current checks
    are aligned.
  - `status = warning` when the surface remains readable but specific facts
    need rebuild, backfill, or repair before you trust them fully.
  - `status = error` with HTTP `503` when the engine-source bridge is not ready
    or a blocking capability problem makes the v2 surface unavailable.
- `GET /waterline/api/stats` publishes durable operator facts. Treat those JSON
  fields as operator diagnostics for dashboards and scripts, not as a metrics
  scrape endpoint.

## Correctness vs acceleration checks

Every v2 health check carries a `category` of either `correctness` or
`acceleration`, and the snapshot publishes a per-category rollup so operators
can answer two separate questions without re-aggregating the check list.

- **Correctness checks** describe whether durable ready-task discovery,
  projection freshness, command-contract backfill, history retention, worker
  compatibility, and backend capabilities are intact. A correctness check in
  `status = error` means safe task pickup or operator-trusted state is at
  risk; rollouts should stop until it clears.
- **Acceleration checks** describe whether optional wake-signal propagation
  is keeping up. The durable pollers are the correctness path, so an
  acceleration check in `status = warning` means cross-node wake-up latency
  may be higher than steady state but no task is stranded.

Each entry under `checks` carries its `category`, and the snapshot adds a
`categories` rollup so dashboards can summarize both questions at a glance:

```json
{
  "status": "warning",
  "categories": {
    "correctness": {"status": "ok", "check_count": 8},
    "acceleration": {"status": "warning", "check_count": 1}
  }
}
```

Treat a degraded `acceleration` rollup as acceleration-only: investigate
cache or wake backend health, but do not block traffic that depends only on
durable ready-task discovery. A degraded `correctness` rollup is the
blocking signal. The `long_poll_wake_acceleration` check is the canonical
acceleration entry and never escalates above `warning`; every other check
is a correctness entry.

## Queue-health semantics

Queue health is split between durable queue state and worker/runtime telemetry.

### Durable queue facts

Use Waterline dashboard stats and queue views for durable task state:

| Fact | Meaning |
| --- | --- |
| `operator_metrics.backlog.runnable_tasks` | Durable tasks that are ready to be claimed now. |
| `operator_metrics.backlog.delayed_tasks` | Durable tasks that exist but are still waiting for `available_at`. |
| `operator_metrics.backlog.leased_tasks` | Durable tasks currently claimed by a worker. |
| `operator_metrics.backlog.tasks_added_last_minute` | Distinct durable task rows created in the trailing 60 seconds. Treat this as durable queue inflow, not as a transport-attempt counter. |
| `operator_metrics.backlog.tasks_dispatched_last_minute` | Distinct durable task rows whose latest successful `last_dispatched_at` landed in the trailing 60 seconds. Compare it with `tasks_added_last_minute` to tell whether durable inflow is outrunning dispatch. |
| `operator_metrics.starts.pending_runs`, `operator_metrics.starts.pending_commands`, `operator_metrics.starts.ready_tasks`, `operator_metrics.starts.oldest_pending_start_at`, `operator_metrics.starts.max_pending_ms` | Durable workflow-start backlog. Use these facts to distinguish starts that have been accepted but have not yet become active workflow-task work from ordinary worker-side queue lag. |
| `operator_metrics.tasks.oldest_ready_due_at`, `operator_metrics.tasks.max_ready_due_age_ms` | The oldest currently actionable task and its ready-to-dispatch age. This is the machine-readable backlog-latency pair behind "oldest ready task". |
| `operator_metrics.tasks.dispatch_overdue`, `operator_metrics.tasks.oldest_dispatch_overdue_since`, `operator_metrics.tasks.max_dispatch_overdue_age_ms` | Ready durable tasks that still have no successful dispatch wake plus the age of the stalest example. Use these facts to spot degraded notifier acceleration without confusing it for ordinary queue growth. |
| `operator_metrics.backlog.unhealthy_tasks` | Durable tasks with dispatch failure, claim failure, overdue dispatch, or expired lease state. |
| `operator_metrics.backlog.repair_needed_runs` | Open runs that do not currently have a trusted durable resume path. |
| `operator_metrics.tasks.oldest_lease_expired_at`, `operator_metrics.tasks.max_lease_expired_age_ms` | The oldest expired lease and its age. Use this pair as the primary stuck-lease and duplicate-risk age indicator. |
| `operator_metrics.backlog.oldest_compatibility_blocked_started_at`, `operator_metrics.backlog.max_compatibility_blocked_age_ms` | The oldest compatibility routing block and its age. Use this when work is preserved but no compatible worker is currently eligible to claim it. |
| Active vs stale pollers | Whether registered workers are still heartbeating for a queue. |
| Current leases | Which workflow or activity tasks are leased right now and whether the lease is expired. |

These facts describe durable workflow-task and activity-task traffic only.

When you need queue-local drill-down instead of fleet totals, use the server
task-queue visibility routes for backlog age, poller state, current leases,
and admission budgets. Those routes do not currently expose per-queue
`stats.tasks_added_last_minute` or `stats.tasks_dispatched_last_minute`; use
the fleet-level `operator_metrics.backlog.*` pair above to compare durable
inflow with dispatch, then use queue-local routes to see which queue is
building backlog or has no available worker capacity.

Waterline's `GET /waterline/api/v2/health` surface publishes the same queue
drill-down under `queue_visibility.*` for the configured namespace. Treat these
field families as the typed queue-health contract:

| Field family | Meaning |
| --- | --- |
| `queue_visibility.available`, `queue_visibility.reason` | Whether Waterline can currently produce queue-local visibility for the configured namespace, and why not when it cannot. |
| `queue_visibility.task_queues[].stats.approximate_backlog_count`, `queue_visibility.task_queues[].stats.approximate_backlog_age` | Queue-local backlog count and oldest durable backlog age. |
| `queue_visibility.task_queues[].stats.tasks_added_last_minute`, `queue_visibility.task_queues[].stats.tasks_dispatched_last_minute` | Per-queue durable inflow versus dispatch over the trailing 60 seconds. Use these when one hot queue is hidden inside healthy fleet totals. |
| `queue_visibility.task_queues[].stats.pollers.active_count`, `queue_visibility.task_queues[].stats.pollers.stale_count`, `queue_visibility.task_queues[].stats.pollers.stale_after_seconds` | Healthy versus stale pollers on that queue and the stale-heartbeat threshold the snapshot used. |
| `queue_visibility.task_queues[].stats.workflow_tasks.*`, `queue_visibility.task_queues[].stats.activity_tasks.*` | Queue-local ready, leased, and expired-lease counts split by workflow-task versus activity-task traffic. |
| `queue_visibility.task_queues[].repair.candidates`, `dispatch_failed`, `expired_leases`, `dispatch_overdue` | Queue-local repair pressure: durable tasks that already need repair, are dispatch-failed, hold expired leases, or are overdue for redispatch. |
| `queue_visibility.task_queues[].repair.oldest_dispatch_failed_at`, `max_dispatch_failed_age_ms`, `oldest_lease_expired_at`, `max_lease_expired_age_ms`, `oldest_dispatch_overdue_since`, `max_dispatch_overdue_age_ms` | Queue-local age signals for the stalest dispatch failure, expired lease, and dispatch-overdue durable task. |

`coordination_alerts[]` on the same `GET /waterline/api/v2/health` payload is
the operator roll-up for those queue-local facts plus the health-check list.
Use it as the page-ready summary for warnings and errors, then drill into the
matching `queue_visibility` or `checks` entries for evidence.

Treat the queue-local admission status as the first-class slot and poller
signal for that queue. `saturated` means live workers are present but every
registered slot is already leased. `throttled` means a server-side lease or
dispatch cap is intentionally holding new work. `no_slots` means workers are
registered but exposed zero capacity for that task kind. `no_active_workers`
means the queue has no healthy poller at all, and `unavailable` means a
configured lock-backed admission guard cannot currently prove safety.

Use `operator_metrics.starts.*` when new workflow starts appear stuck even
though steady-state queue lag looks normal. Those facts separate control-plane
start admission and first-task creation debt from downstream worker pickup.

### Poller pressure and admission budgets

Use task-queue detail routes or `dw task-queue:describe` when queue flow is
degrading and you need to separate "not enough worker capacity" from
"intentional server throttling" or "no live poller at all":

| Queue status | Meaning | Treat it as |
| --- | --- | --- |
| `accepting` | Workers still have available slots and no server cap is full. | Healthy baseline. |
| `saturated` | All registered worker slots are currently leased. | Worker-capacity pressure. |
| `throttled` | A server-side active-lease or dispatch-rate cap is intentionally holding the queue back. | Advisory unless the cap is unexpected or the backlog keeps growing beyond the published baseline. |
| `no_slots` | Active workers are registered, but none advertise slots for that task kind. | Blocking for that queue. |
| `no_active_workers` | No healthy poller is currently serving the queue. | Blocking for that queue. |
| `unavailable` | The queue cannot acquire the lock needed for its configured admission path. | Blocking until the admission dependency recovers. |

Use these statuses with the queue-flow facts together:

- `tasks_added_last_minute > tasks_dispatched_last_minute` plus `saturated`
  means durable inflow is outrunning worker capacity.
- The same rate imbalance plus `throttled` means the queue is being held back
  by an explicit server cap and should be judged against that cap's intended
  contract, not against unrestricted throughput.
- A rising oldest-ready age plus `no_active_workers` or stale pollers means the
  queue has lost healthy claimers and should be treated as a routing outage for
  that scope.

### Matching-role deployment shape

Use `operator_metrics.matching_role.*` when you need to confirm which
matching/dispatch contract the current node is actually serving:

| Fact | Meaning |
| --- | --- |
| `operator_metrics.matching_role.queue_wake_enabled` | Whether this node still runs the in-worker broad-poll wake path on queue-worker loop events. |
| `operator_metrics.matching_role.shape` | `in_worker` when the node still owns that wake path, `dedicated` when the wake/repair sweep is expected to run under a separate `workflow:v2:repair-pass --loop` process. |
| `operator_metrics.matching_role.task_dispatch_mode` | The dispatch mode this node is using for ready tasks: `queue` or `poll`. |
| `operator_metrics.matching_role.partition_primitives` | The frozen routing axes, in order: `connection`, `queue`, `compatibility`, `namespace`. |
| `operator_metrics.matching_role.backpressure_model` | The durable admission boundary the engine enforces. Current v2 reports `lease_ownership`. |

These fields are node-local, not fleet-wide. In a mixed-shape rollout, read
the snapshot from each node or pod you are cutting over so you can confirm the
matching role moved where you intended before you interpret backlog or poller
changes as worker health.

### Worker and SDK telemetry

Use worker metrics, traces, and logs for:

- Workflow and activity `schedule_to_start` latency
- Poll success rate and sync/eager-dispatch behavior
- Sticky-cache size and eviction behavior
- Worker CPU, memory, thread, and event-loop pressure
- Custom application metrics emitted from activities or worker code

Synchronous queries, live-debug tooling, and other non-durable control-plane
calls should be labeled separately in your dashboards. They do not count as
durable task backlog and they do not change Waterline repair counters.

## Worker compatibility and rollout health

`operator_metrics.workers` publishes the compatibility facts that determine
whether the active worker fleet can safely handle the required workflow
contract:

| Fact | Meaning |
| --- | --- |
| `operator_metrics.workers.required_compatibility` | Compatibility markers a worker must advertise to be eligible for work in the namespace. |
| `operator_metrics.workers.active_workers` | Count of distinct live workers seen through compatibility heartbeat. |
| `operator_metrics.workers.active_worker_scopes` | Count of `(connection, queue)` scopes covered by those workers. |
| `operator_metrics.workers.active_workers_supporting_required` | Workers whose advertised compatibility covers the required markers. |
| `operator_metrics.workers.fleet` | Per-scope list of every active worker with `worker_id`, `connection`, `queue`, advertised `supported` markers, a `supports_required` flag, the heartbeat `source` (`database` or `cache`), and `recorded_at`. |

Use the summary counts to detect rollout states where some workers cannot
safely claim the required work, and drill into `fleet` to identify
exactly which `(connection, queue)` scope is missing coverage. The Waterline
operator dashboard renders the same fleet list under its worker
compatibility panel so operators do not need to query the metric surface by
hand.

When `active_workers_supporting_required` reaches zero for a namespace,
Waterline surfaces a `no_compatible_worker_for_task` run diagnostic on
affected runs so the gap is visible on the run-detail view as well as the
metric surface. The companion `worker_compatibility` health check fires as
`warning` under `correctness` in the same condition, which flips the
`correctness` category rollup to `warning` so the fleet gap is visible at
a glance and not buried inside the check list.

See [Rolling Out Worker Builds With Build IDs](./polyglot/worker-build-id-rollout.md)
for the drain/resume flow that coordinates with these facts during a
build-id rollout, and
[Worker Compatibility and Routing](./polyglot/worker-compatibility-routing.md)
for the pinning contract behind those diagnostics.

## Alert semantics

Alert thresholds are deployment-specific. Publish your own numeric baselines
for queue age, repair lag, worker coverage, and restore timing, then alert when
the contract below stays breached longer than one normal repair or watchdog
window for the topology you operate.

| Alert family | Source | Treat as | Escalate when | Operator response |
| --- | --- | --- | --- | --- |
| Blocking readiness | `workflow:v2:doctor --strict`, `GET /waterline/api/v2/health` | Blocking | `doctor --strict` returns an error or the health endpoint returns `status = error` / HTTP `503` | Stop rollout or traffic shift, fix the blocking prerequisite, then rerun readiness and compatibility checks. |
| Compatible-worker coverage | `operator_metrics.workers.*`, `worker_compatibility` health check, run diagnostic `no_compatible_worker_for_task` | Blocking | `active_workers_supporting_required = 0` for a namespace or required `(connection, queue)` scope | Drain incompatible workers, register compatible workers, and confirm the `correctness` rollup clears before trusting new claims. |
| Durable queue lag | Waterline queue views, `operator_metrics.backlog.*`, worker `schedule_to_start` telemetry | Blocking when sustained; advisory when brief | The oldest ready-task age or schedule-to-start latency stays above the published topology baseline while compatible workers are available | Add worker capacity, inspect task-queue admission limits, and verify the scheduler or matching path is still making forward progress. |
| Poller pressure and admission saturation | Task-queue detail routes, `dw task-queue:describe`, queue `status`, stale pollers, and queue-local add/dispatch rates | Blocking for `no_active_workers`, `no_slots`, or `unavailable`; advisory for intentional `throttled` states | One queue stays `saturated` while its oldest-ready age and add-vs-dispatch gap keep growing, or any queue flips to `no_active_workers`, `no_slots`, or `unavailable` outside a planned maintenance window | Add worker slots, restore the missing poller cohort, or confirm the server-side cap and lock dependency are behaving as designed before you scale blindly. |
| Workflow-start backlog | `operator_metrics.starts.*`, control-plane start telemetry, worker `schedule_to_start` telemetry for first workflow tasks | Blocking when sustained; advisory when brief | `pending_commands`, `ready_tasks`, or `max_pending_ms` stay above the published topology baseline while compatible workers and queue capacity are available | Inspect the start boundary end to end: confirm start commands are turning into durable tasks, verify matching or dispatch is creating the first task promptly, and separate start-path debt from general worker lag before scaling. |
| Projection drift and repair debt | `run_summary_projection` / `selected_run_projections` health checks, `operator_metrics.repair.*` | Advisory | Drift warnings persist past one planned rebuild window or the max candidate age keeps climbing | Run the rebuild or repair previews, execute the repair, then verify the warning clears and stale ages return to baseline. |
| Retry or failure storm | `operator_metrics.backlog.unhealthy_tasks`, durable run diagnostics, worker error telemetry | Advisory, escalating to blocking if it prevents durable progress | Dispatch-failed, claim-failed, expired-lease, or retry-exhaustion facts climb above the topology baseline and stay elevated | Inspect the failing task family, compare worker telemetry with durable error facts, and decide whether to drain traffic or isolate the affected queue. |
| Wake acceleration degradation | `long_poll_wake_acceleration` health check and the `acceleration` category rollup | Advisory | The acceleration warning persists after cache or notifier maintenance windows | Investigate cache or wake propagation health. Do not treat this as a correctness outage unless the `correctness` rollup also degrades. |

The goal is to page on durable contract risk, not on every transient signal.
Queue and worker alerts should only become blocking when they threaten the
operator contract for the topology you actually run.

## Rebuild, repair, and restore expectations

Use these checks in order when the operator surface reports drift:

1. Check `GET /waterline/api/v2/health`.
   - `run_summary_projection` and `selected_run_projections` warnings mean
     Waterline can still answer, but some list or detail facts need rebuild.
   - `command_contract_snapshots` warnings mean some legacy runs still need
     WorkflowStarted contract backfill before operators can trust declared
     signal, update, or query forms.
   - `durable_resume_paths` warnings mean open runs need repair before you rely
     on their projected next resume source.
2. Preview projection work with:

   ```bash
   php artisan workflow:v2:rebuild-projections --needs-rebuild --dry-run
   ```

3. Rebuild the affected projections:

   ```bash
   php artisan workflow:v2:rebuild-projections --needs-rebuild
   ```

4. Preview command-contract backfill work with:

   ```bash
   php artisan workflow:v2:backfill-command-contracts --dry-run
   ```

5. Backfill command contracts when the current workflow class is still
   available:

   ```bash
   php artisan workflow:v2:backfill-command-contracts
   ```

6. Use `--prune-stale` only after your retention workflow has intentionally
   removed durable rows and you want to delete projection rows whose durable run
   or history row no longer exists.

`operator_metrics.repair.*` publishes the repair-loop sweep footprint. Use the
candidate counts, selected counts, maximum candidate age, and scan-limit
pressure to decide whether repair work is comfortably within your baseline or
needs capacity investigation.

## Export and archive verification

History export and archive serve different purposes:

- **History export** creates a replay/debug/archive artifact.
- **Archive** marks a closed run as archived so it leaves active fleet views.
- **Prune** removes projection or durable rows after retention has definitely
  expired.

Use this verification sequence:

1. Export the selected run:

   ```bash
   php artisan workflow:v2:history-export <workflow-instance-id> --run-id=<workflow-run-id> --output=storage/app/workflow-history/run.json --pretty
   ```

2. Verify the bundle includes the expected run id, schema version, and any
   configured redaction metadata.
3. Archive the closed run only after the export artifact is stored where your
   runbook expects it.
4. Keep archived-but-not-pruned runs available for incident review.
5. Prune durable rows through your retention job, then rebuild/prune projections
   with `workflow:v2:rebuild-projections --prune-stale`.

For Waterline users, the matching history-export and archive routes are listed
in the [Waterline Operator API Reference](./waterline-operator-api.md).

## Backup, restore, and disaster-recovery contract

Backup, restore, and disaster recovery are part of the operating envelope, not
an optional private runbook. For every supported topology, publish and rehearse
these facts:

1. The durable backup set: database backup, server or app image reference,
   runtime env file or config set, auth material location, and the exact
   topology or restore notes needed to reattach workers.
2. The recovery targets: maximum accepted restore lag, expected failover lag,
   and who is allowed to declare traffic safe again.
3. The restore order: restore durable persistence first, then cache, then
   bootstrap or migrations, then the singleton scheduler or maintenance role,
   then API readiness, then worker registration.
4. The verification pass: `/api/ready` or `/waterline/api/v2/health`,
   `/api/cluster/info` where applicable, one representative worker
   registration, and one representative history export from restored state.
5. The repair pass after restore: rebuild projections, backfill command
   contracts if needed, and confirm queue, compatibility, and repair metrics
   return to baseline before you call the environment healthy.

Multi-region operation and split-brain behavior remain outside the supported
2.0 operating envelope. The
[self-hosting guide](/docs/deployment#activepassive-multi-region) retains
active/passive architecture and runbook material only for support-led
evaluation; it does not establish self-serve replication, failover, failback,
RPO, or RTO guarantees. The current
[Cloud managed-runtime contract](/docs/polyglot/cloud-control-plane)
operates each namespace in one managed region and likewise does not promise
multi-region replication, regional failover, or failback.

Treat restore rehearsal cadence as part of the public operating contract too.
At minimum, rehearse the documented restore sequence:

- before the first production rollout for a topology
- after any change to the backup mechanism, schema/bootstrap path, auth model,
  or deployment topology
- on a regular recurring cadence that is published in the same runbook as the
  backup schedule

If you cannot produce the latest successful rehearsal date, elapsed restore
time, and verification evidence, then backup and DR remain an unproven claim
for that topology.

## Benchmark envelope

Durable Workflow v2 publishes the dimensions you should benchmark for your own
environment. Record these baselines in staging or canary before production
traffic depends on them:

| Dimension | What to baseline | Source |
| --- | --- | --- |
| Projection health | Steady-state `needs_rebuild = 0`, rebuild duration after intentional drift, and stale/orphan cleanup time | `/waterline/api/v2/health`, `/waterline/api/stats`, `workflow:v2:rebuild-projections` |
| Queue pressure | Backlog age, oldest ready task age, runnable vs delayed task counts, task add vs dispatch rate, dispatch-overdue age, stale poller count, and queue admission status (`accepting`, `saturated`, `throttled`, `no_slots`, `no_active_workers`) | Waterline dashboard stats and queue views plus `operator_metrics.backlog.*` / `operator_metrics.tasks.*` |
| Workflow-start latency | Accepted start commands waiting for first-task creation, oldest pending-start age, and first-task pickup after admission | `operator_metrics.starts.*` plus worker `schedule_to_start` telemetry |
| Schedule-to-start latency | Workflow and activity queue wait from enqueue to start | Worker SDK metrics |
| Timer fan-out wake-up behavior | Wake-signal propagation time and the lag between scheduled fire time and ready-task visibility during burst timers | Worker telemetry plus same-region wake coordination checks |
| Repair-loop sweep cost | Candidate counts, selected counts, max candidate age, max missing-run age, and scan-pressure behavior | `operator_metrics.repair.*` |
| History pressure | Event count, history size, and continue-as-new recommendation thresholds | `operator_metrics.history.*` |

These are benchmark dimensions rather than universal latency promises. Publish
your own acceptable ranges for the topology you operate.

## Long-soak evidence

Benchmark snapshots are not enough on their own. Before you call a topology
trusted for sustained traffic, keep a long-soak evidence packet that shows the
system stayed inside its declared envelope over time.

Include at least:

- workload shape: topology, server image or app revision, worker build ids,
  queue layout, cache backend, database backend, and the representative mix of
  workflow starts, timer load, activities, queries, and exports
- soak window: start and end time, plus enough duration to cover at least one
  normal repair window, one retention or archive pass if applicable, and one
  representative business-cycle traffic swing for that environment
- durable queue stability: backlog age, ready-task age, start backlog age, task
  add versus dispatch rate, and stale-poller counts staying within the
  published baseline for the topology
- correctness stability: no sustained `status = error` from
  `GET /waterline/api/v2/health`, no unexplained growth in
  `operator_metrics.repair.*`, and no persistent compatibility gaps in
  `operator_metrics.workers.*`
- process and cache stability: worker memory, CPU, event-loop or thread
  pressure, and cache/cardinality growth staying bounded rather than climbing
  monotonically under steady load
- recovery evidence: the latest successful backup timestamp, latest restore
  rehearsal timestamp, elapsed restore time, and the verification commands that
  proved the restored environment was ready

Store the packet where the same operators can retrieve the deployment runbook.
If a topology claims published benchmark numbers, alert semantics, or recovery
timing without a matching soak packet, treat those numbers as provisional
rather than trusted operating-envelope evidence.

## End-to-end operator checklist

Use this checklist after upgrades and before trusting a new environment:

1. Run `php artisan workflow:v2:doctor --strict`.
2. Check `GET /waterline/api/v2/health` and confirm whether the state is
   `ok`, `warning`, or `error`.
3. Read `GET /waterline/api/stats` for backlog, repair, history, command
   contract, worker compatibility, and projection drift facts.
4. Run projection rebuild or command-contract backfill previews when health
   reports drift.
5. Export one representative run and verify the archive/replay artifact path.
6. Confirm archived runs leave active fleet views while durable rows remain
   available until retention cleanup.
7. Rehearse the restore or failover sequence recorded in your deployment
   runbook and verify the measured lag matches the published expectation for
   your topology.

## Related Guides

- [Monitoring](./monitoring.md)
- [Waterline Operator API Reference](./waterline-operator-api.md)
- [Pruning Workflows](./configuration/pruning-workflows.md)
- [Self-Hosting Deployments](./deployment.md)

<!-- Source: docs/polyglot/embedded-to-server.md -->

# Embedded to Server Migration

This guide is for teams already running Durable Workflow v2 inside a Laravel
application and moving new workflow traffic to the standalone server. If the
application is still on v1, start with [Migrating to 2.0](/docs/migration)
and keep existing v1 runs on the v1 engine until they finish.

Use [Laravel Adoption and Runtime Transition](/docs/laravel-adoption/) first
when the destination is not yet chosen or when Laravel framework integration is
part of the decision. This page is the self-hosted Server runbook after that
choice; it is not the Cloud migration path.

The migration is an adoption path, not an in-place database move. The
standalone server owns new control-plane starts, durable history, schedules,
worker registrations, and task delivery over HTTP. Your application workers
continue to own workflow and activity code.

<div
  data-public-payload-codec-contract="avro-only"
  data-payload-codec="avro"
  data-authority-manifest="https://durable-workflow.github.io/sdk-neutrality-contract.json"
  data-legacy-v1-import-drain="internal">
  Durable Workflow 2.0 has one public payload codec: <code data-payload-codec-field="codec">avro</code>.
  JSON is the HTTP document transport, not a durable payload codec. PHP-only v1
  readers remain internal to the import/drain path and cannot be selected for
  new v2 runs. See the <a href="/docs/polyglot/avro-value-protocol/">Avro Value protocol</a>.
</div>

This is a self-hosted migration guide. It does not turn the embedded deployment
into a Cloud namespace, and the resulting Server is not attached to Durable
Workflow Cloud. For the separate managed-service choice, see
[Cloud Managed Runtime](/docs/polyglot/cloud-control-plane).

## Current Boundary

Supported today:

- Start new workflows through the server control-plane API or CLI.
- Register PHP, Python, or custom HTTP workers against the server.
- Poll and complete workflow and activity tasks through the worker protocol.
- Import eligible embedded v2 history-export bundles into the server as
  server-managed workflow state.
- Export closed-run history bundles from embedded v2 or server v2 for audit,
  debugging, and archival handoff.
- Observe server-managed workflows through the server API, CLI, and SDK
  surfaces that read server state, or deploy
  [Waterline service mode](/docs/monitoring#waterline-service) as a
  separate observer connected through the PHP SDK. The Waterline package in
  the original Laravel app continues to read only that embedded runtime's
  state.

Not supported as an automatic operation today:

- Moving v1 runs into the server.
- Replaying v1 history with non-PHP workers.

Plan the cutover so v1 runs drain where they started. Embedded v2 runs may
either drain in place or move through the import workflow below.

## Deployment Mode Contract

For the frozen embedded-vs-service comparison, see
[Deployment Modes](/docs/polyglot/deployment-modes). This migration guide
adds three cutover-specific rules on top of that shared contract:

- Existing embedded runs keep executing where they started.
- New server-managed runs use stable type keys, namespace names, task queues,
  and the Avro payload contract from the first cutover.
- Signals, queries, updates, repair, cancel, terminate, and archive must keep routing
  to the runtime that owns the target run.
- Each Waterline deployment must keep targeting the runtime and namespace it
  is intended to observe; it does not merge embedded and server-managed runs.

## Cutover Invariants

Keep these rules true throughout the migration:

- Configure the server as an explicit remote dependency: set the base URL,
  namespace, task queue, and auth material directly instead of inferring them
  from Laravel app-local settings.
- Keep workflow ids, run-targeting rules, workflow/activity type keys, payload
  codec tag (`avro`), and compatibility markers stable across both runtimes.
- Route signals, queries, updates, repair, cancel, terminate, and archive to the same
  runtime that owns the target run.
- Before importing an embedded run, pause embedded workers long enough to export
  a quiesced bundle with no leased workflow/activity task and no running
  activity attempt.
- Treat language neutrality as part of the migration contract: server-managed
  workflows use stable aliases and the fixed Avro Value schema, not PHP-only
  class names or payload formats.

## Phase A: Prepare Embedded v2

Before deploying the server, make the embedded app use language-neutral
contracts. This reduces the amount of code that changes during cutover.

1. Define stable workflow and activity type keys.

   ```php
   // config/workflows.php
   'v2' => [
       'types' => [
           'workflows' => [
               'orders.process' => App\Workflows\ProcessOrderWorkflow::class,
           ],
           'activities' => [
               'orders.reserve-inventory' => App\Activities\ReserveInventory::class,
               'orders.capture-payment' => App\Activities\CapturePayment::class,
           ],
       ],
   ],
   ```

2. Use those keys at every external boundary.

   Server clients send `workflow_type: "orders.process"`, and workers register
   `supported_workflow_types` / `supported_activity_types` with those same
   strings. Do not expose PHP FQCNs as the durable public contract.

3. Use the v2 Avro payload contract.

   ```php
   // config/workflows.php
   'serializer' => 'avro',
   ```

   Avro is mandatory for new embedded and server-managed v2 workflows. A
   legacy PHP reader may be retained only inside the v1 import/drain path while
   old v1 runs finish.

4. Pick namespace and task queue names.

   Choose names you can keep stable throughout the embedded-to-Server cutover:

   ```bash
   export DURABLE_WORKFLOW_NAMESPACE=production
   export DURABLE_WORKFLOW_TASK_QUEUE=orders
   ```

5. Decide whether compatibility markers are needed.

   A single-fleet deployment can leave compatibility unset. The standard
   v2 rollout mechanics (canary, drain, rollback, replay-debug) all
   operate inside one stable v2 contract — there is no "mixed-fleet v2"
   adoption lane. If a build-skew window does open during the embedded
   to server cutover, set `DW_V2_CURRENT_COMPATIBILITY` and
   `DW_V2_SUPPORTED_COMPATIBILITIES` before the cutover so a new build
   does not claim runs from an incompatible build during that window.

## Phase B: Deploy the Server Beside Embedded

Run the standalone Server beside the Laravel app while the app continues
handling existing embedded runs.

```bash
git clone https://github.com/durable-workflow/server.git
cd server
cp .env.example .env
docker compose up -d
```

For local development you may set `DW_AUTH_DRIVER=none`. For
shared environments, use role-scoped credentials:

```bash
DW_AUTH_DRIVER=token
DW_WORKER_TOKEN=worker-secret
DW_OPERATOR_TOKEN=operator-secret
DW_ADMIN_TOKEN=admin-secret
```

Verify discovery and create the namespace:

```bash
export SERVER=http://localhost:8080
export ADMIN_TOKEN=admin-secret
export OPERATOR_TOKEN=operator-secret

curl "$SERVER/api/health"

curl "$SERVER/api/cluster/info" \
  -H "Authorization: Bearer $OPERATOR_TOKEN"

curl -X POST "$SERVER/api/namespaces" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{"name":"production","description":"Production workflows","retention_days":30}'
```

The server rejects control-plane requests without
`X-Durable-Workflow-Control-Plane-Version: 2`; workers use the separate
`X-Durable-Workflow-Protocol-Version: 1.0` header.

## Phase C: Connect Workers

Workers must register before polling. The registration advertises runtime,
task queue, and the type keys the worker can execute.

```bash
export WORKER_TOKEN=worker-secret

curl -X POST "$SERVER/api/worker/register" \
  -H "Authorization: Bearer $WORKER_TOKEN" \
  -H "X-Namespace: production" \
  -H "X-Durable-Workflow-Protocol-Version: 1.0" \
  -H "Content-Type: application/json" \
  -d '{
    "worker_id": "orders-php-1",
    "task_queue": "orders",
    "runtime": "php",
    "sdk_version": "<installed-sdk-version>",
    "supported_workflow_types": ["orders.process"],
    "supported_activity_types": [
      "orders.reserve-inventory",
      "orders.capture-payment"
    ],
    "max_concurrent_workflow_tasks": 10,
    "max_concurrent_activity_tasks": 50
  }'
```

Then run workers in server mode for the same namespace and task queue. Python
workers follow the same registration and poll contract through the Python SDK.
Custom workers can use the [Worker Protocol](/docs/polyglot/worker-protocol)
directly.

Check visibility before cutover:

```bash
curl "$SERVER/api/workers" \
  -H "Authorization: Bearer $OPERATOR_TOKEN" \
  -H "X-Namespace: production" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2"

curl "$SERVER/api/task-queues/orders" \
  -H "Authorization: Bearer $OPERATOR_TOKEN" \
  -H "X-Namespace: production" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2"
```

## Phase D: Route New Starts to the Server

Cut over one workflow family at a time. Keep the embedded app and queue workers
running for old runs until they drain.

Start a shadow workflow first:

```bash
curl -X POST "$SERVER/api/workflows" \
  -H "Authorization: Bearer $OPERATOR_TOKEN" \
  -H "X-Namespace: production" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{
    "workflow_id": "orders-shadow-1001",
    "workflow_type": "orders.process",
    "task_queue": "orders",
    "input": [{"order_id":"1001","mode":"shadow"}]
  }'
```

Watch the run:

```bash
curl "$SERVER/api/workflows/orders-shadow-1001" \
  -H "Authorization: Bearer $OPERATOR_TOKEN" \
  -H "X-Namespace: production" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2"
```

After the shadow path works, move the production caller from
`WorkflowStub::start()` to the server client, CLI, or direct HTTP API. Leave
signals, queries, updates, cancel, and terminate routed to the same runtime that
started the workflow. A workflow started embedded should receive embedded
commands; a workflow started on the server should receive server control-plane
commands.

## Phase E: Import Eligible Embedded v2 Runs

Use history export as the import format. The embedded runtime remains the source
of truth for the bundle; the server verifies it, writes durable rows inside one
transaction, and rebuilds server projections from those rows.

Eligibility:

- The bundle schema must be `durable-workflow.v2.history-export` with
  `schema_version: 1`.
- The source run must be embedded v2 and the export must carry
  `workflow.source_runtime: "embedded"`. v1 history remains out of scope.
- A non-terminal run must be the current embedded run.
- A terminal run must have `history_complete: true`.
- Redacted bundles are rejected.
- Leased workflow tasks, leased activity tasks, and running activity attempts
  are rejected. Pause workers or let leases expire, then export again.

Import source-of-truth rules:

- `history_events` are copied as the authoritative replay and audit log.
- Workflow identity, payloads, commands, signals, updates, tasks, activity
  executions, timers, failures, and lineage links are reconstructed from the
  bundle.
- Server summary, wait, timer, timeline, and lineage projections are rebuilt
  after import. Projection rows are not the import authority.
- Pending workflow tasks remain claimable by compatible server workers.
- Pending activity executions and ready activity tasks remain claimable by
  compatible server activity workers.
- Pending timers keep their `fire_at` timestamp and are visible to server timer
  repair/recovery.

Failure and rollback:

- Import is all-or-nothing in one database transaction.
- If the process exits or validation fails before commit, no partial run state
  remains on the server.
- Retrying the same bundle is idempotent by `run_id` and `dedupe_key`.
- If a different server run already owns the same `run_id`, import is rejected.

Visibility and audit:

- Imported runs expose `engine_source: embedded_v2_import` in server list/detail
  views.
- The durable run row records `import_source=embedded_v2`, `import_id`,
  `import_dedupe_key`, `import_contract_version`, and `imported_at`.

Operator workflow:

For embedded v2 runs:

```bash
php artisan workflow:v2:history-export order-123 \
  --output=storage/workflow-history/order-123.json \
  --pretty
```

Dry-run the import on the server:

```bash
php artisan workflow:v2:history-import storage/workflow-history/order-123.json \
  --namespace=production \
  --dry-run \
  --json
```

Import the bundle:

```bash
php artisan workflow:v2:history-import storage/workflow-history/order-123.json \
  --namespace=production \
  --import-id=orders-cutover-2026-05-05
```

Or call the HTTP operator endpoint:

```bash
curl -X POST "$SERVER/api/workflows/import/embedded-v2" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "X-Namespace: production" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  -H "Content-Type: application/json" \
  --data-binary @order-123-import-request.json
```

where `order-123-import-request.json` is:

```json
{
  "import_id": "orders-cutover-2026-05-05",
  "bundle": {
    "schema": "durable-workflow.v2.history-export",
    "schema_version": 1,
    "workflow": {
      "source_runtime": "embedded"
    }
  }
}
```

The `bundle` object above is abbreviated; pass the complete
`workflow:v2:history-export` JSON document.

For server-managed runs that you only need to preserve or audit:

```bash
curl "$SERVER/api/workflows/order-123/runs/$RUN_ID/history/export" \
  -H "Authorization: Bearer $OPERATOR_TOKEN" \
  -H "X-Namespace: production" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  > order-123-history.json
```

Archive terminal server runs after export when you want to keep them out of
retention pruning:

```bash
curl -X POST "$SERVER/api/workflows/order-123/archive" \
  -H "Authorization: Bearer $OPERATOR_TOKEN" \
  -H "X-Namespace: production" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{"reason":"history exported to archive storage"}'
```

## Phase F: Add Polyglot Workers

Once a workflow family runs through the server, add Python or custom workers by
registering the same namespace, task queue, and type keys. Every public v2
payload uses `avro`; keep activity inputs and outputs language-neutral: arrays,
objects, strings, numbers, booleans, and nulls.

For Python, use [the Python SDK guide](/docs/polyglot/python). For direct
HTTP implementations, use [the worker protocol reference](/docs/polyglot/worker-protocol).

## Cutover Checklist

- [ ] v1 runs are drained or intentionally left on the v1 engine.
- [ ] Embedded v2 uses stable type keys, not PHP FQCNs, at external boundaries.
- [ ] Every new v2 workflow uses the sole public `avro` codec.
- [ ] Server `/api/health` and `/api/cluster/info` pass from the deployment
  network.
- [ ] Target namespace exists on the server.
- [ ] Workers register with the expected task queue and supported type keys.
- [ ] A shadow workflow starts, is claimed by a worker, and reaches the expected
  state.
- [ ] Operators can list workflows, inspect task queues, and view worker
  registrations.
- [ ] Old embedded starters are paused or moved so duplicate business keys do
  not start in both runtimes.
- [ ] Embedded v2 runs selected for import are exported from a quiesced embedded
  runtime and pass `workflow:v2:history-import --dry-run`.
- [ ] Imported runs show `engine_source: embedded_v2_import` and the expected
  `import_id` in server detail/list surfaces.
- [ ] Closed old runs that only need retention are exported to durable storage.

## Troubleshooting

| Symptom | Likely cause | Fix |
| --- | --- | --- |
| `missing_control_plane_version` | Client called a control-plane route without the v2 header | Send `X-Durable-Workflow-Control-Plane-Version: 2` |
| `missing_protocol_version` | Worker called a worker route without the worker protocol header | Send `X-Durable-Workflow-Protocol-Version: 1.0` |
| `namespace_not_found` | The namespace has not been created on the server | Create it with `POST /api/namespaces` |
| Worker polls return no task | Task queue, type key, namespace, or compatibility marker does not match | Compare workflow start payload, worker registration, and task queue visibility |
| A v1 payload cannot be decoded by a non-PHP worker | The run contains PHP-native v1 history | Keep the run on the internal PHP v1 import/drain path; do not route it to a v2 worker |
| Old workflow does not respond to server signal/query/update | The run was started in embedded mode | Send commands through the embedded app until that run finishes |
| Import rejects `tasks.leased_task_present` | The embedded export captured a leased task | Pause embedded workers, wait for leases to release or complete, then export again |
| Import returns `already_imported` | The same `run_id` and `dedupe_key` are already present on the server | Treat the retry as successful and inspect the server run |
| Imported run is visible but not claimed | Worker type key, task queue, namespace, or compatibility marker does not match the imported row | Compare the import report, run detail, and worker registration |

## Related Guides

- [Server setup](/docs/polyglot/server)
- [Worker protocol](/docs/polyglot/worker-protocol)
- [Python SDK](/docs/polyglot/python)
- [Migrating to 2.0](/docs/migration)

<!-- Source: docs/polyglot/deployment-modes.md -->

# Deployment Modes

Durable Workflow v2 has two deployment modes:

- **Service mode:** applications and workers connect through SDKs to a remote
  runtime. Choose [Durable Workflow Cloud](/docs/polyglot/cloud-control-plane/)
  or a [self-hosted Server](/docs/polyglot/server/).
- **Embedded mode:** a Laravel application installs
  `durable-workflow/workflow` and owns the runtime directly.

Cloud and self-hosted Server are runtime choices inside service mode, not
components to run together. In Cloud, Durable Workflow operates orchestration
and persistence while customers run SDK clients and workers. Cloud includes
Managed Waterline. **Cloud customers do not install, deploy, or attach their
own Server or Waterline service.** A self-hosted Server does not include
Waterline; operators can separately deploy Waterline against a Server-owned
namespace.

Use this page when deciding which shape should own a workflow fleet, planning a
cutover between them, or documenting which parts of the product contract must
stay identical across both.

Laravel teams should use the focused
[Laravel adoption and runtime transition guide](/docs/laravel-adoption/)
for the v1-to-v2 and embedded-to-service paths, including the shipped PHP SDK
bridge and its Laravel test fake.

## Choose a service-mode runtime

| Runtime choice | Who operates durable state | What your team runs | Start here |
| --- | --- | --- | --- |
| Durable Workflow Cloud | Durable Workflow operates the managed namespace runtime, persistence, upgrades, service endpoint, and Managed Waterline. | Application clients plus PHP, Python, or Rust workers using provisioned credentials. Do not run Server or a separate Waterline service. | [Cloud Managed Runtime](/docs/polyglot/cloud-control-plane/) |
| Self-hosted Server | Your team deploys, secures, scales, backs up, and upgrades the Server and its persistence. | Server plus application clients and PHP, Python, or Rust workers. If wanted, deploy Waterline as a separate service against the Server-owned namespace. | [Self-hosted Server](/docs/polyglot/server/) |

Both choices use the same client and worker model. The difference is runtime
operations and credentials, not workflow authoring.

## Same durable model, different boundary

Embedded mode and service mode keep one v2 kernel. What changes is the hosting,
auth, and transport boundary around it.
Service mode keeps the same kernel behind HTTP+JSON control-plane and worker
surfaces; there is no mandatory gRPC and no second engine.

| Surface | Embedded mode | Service mode | Stable in both modes |
| --- | --- | --- | --- |
| Durable workflow model | A Laravel app hosts the package directly and writes workflow state inside the app runtime. | Cloud or a self-hosted Server owns workflow state behind the service API. | Workflow ids, run ids, typed history, command outcomes, retries, repair semantics, and history export remain the same v2 contract. |
| Control plane | Starts and commands come from app code, `WorkflowStub`, or app-local operator tooling. | Starts and commands go through the server API, CLI, or SDKs over HTTP+JSON with explicit auth and protocol headers. Framework-neutral PHP callers use `DurableWorkflow\Client` from `durable-workflow/sdk`. | Duplicate-start policy, run targeting, command ids, and named outcomes stay the same. Route follow-up commands to the runtime that accepted the start. |
| Worker transport | Laravel queue workers execute workflow and activity tasks inside the app deployment. | Workers register, long-poll, heartbeat, and complete work over the HTTP+JSON worker protocol. PHP remote workers use `DurableWorkflow\Worker` from `durable-workflow/sdk`. | Task leases, compatibility markers, replay semantics, and at-least-once activity execution stay the same. |
| Task dispatch default | Tasks are normally dispatched to the Laravel queue in-process with the application. | The service runtime uses poll dispatch so external workers discover work over HTTP. Self-hosted Server operators can explicitly override that default. | The ready/leased/repair lifecycle and durable task model stay the same. |
| Workflow and activity type keys | PHP aliases can resolve to local classes inside the app. | Workers advertise supported type keys during registration. | Public type keys should stay stable and language-neutral. Do not make PHP FQCNs or mirrored PHP placeholder types the public contract. |
| Operator surface | The embedded Waterline package or app-local tooling reads the Laravel app's durable state in process. | Cloud provides Managed Waterline for its namespace. Self-hosted operators can separately deploy Waterline against a Server-owned namespace. Service APIs, CLI, and SDKs also read runtime-owned state. | Visibility facts such as search attributes, memos, run status, queue diagnostics, and history export are durable facts within the runtime that owns the run. Waterline does not combine runtimes or namespaces. |
| Auth and tenancy boundary | App auth is whatever the Laravel host exposes around its own routes and sessions. | Namespace selection and server auth tokens or signatures are mandatory API boundaries. | Namespace names, task queues, compatibility markers, and the fixed Avro payload contract should stay stable across a cutover. |
| Runtime discovery | The app can resolve services in-process or through app-local configuration. | Workers and clients must target an explicit remote base URL. | Do not couple either mode to shared `APP_URL`, `APP_KEY`, localhost assumptions, or same-container discovery. |
| Migration boundary | Existing embedded runs keep executing where they started. | New service-managed runs start in the selected Cloud or self-hosted runtime and stay there. | There is no automatic live migration of in-flight runs between modes. Export is for audit/debugging, not for importing live state. |

## Choose Embedded Mode When

- Your Laravel application owns workflow authoring, worker execution, and
  operator access in one deployment.
- The app's existing queue and auth model is the right boundary for workflow
  operations.
- You want the smallest self-contained runtime and do not need a
  language-neutral worker protocol.
- Your operators can use Waterline or host-app tooling as the primary workflow
  surface.

Start with [Embedded Installation](/docs/installation/) and the
[Embedded documentation](/docs/category/embedded/), including its
Configuration group.

## Choose Service Mode When

- Multiple applications or teams should share one workflow runtime.
- Workers, control-plane callers, or operators are not all Laravel/PHP.
- You need an explicit remote auth and namespace boundary between clients and
  the workflow engine.
- You want to scale API ingress, matching/dispatch, and workers independently
  within the supported
  [server role topology](/docs/polyglot/server-role-topology).
- For self-hosted Server, you want to deploy Waterline as an observer over a
  server-owned namespace. Cloud instead includes Managed Waterline.

For a managed runtime, start with
[Durable Workflow Cloud](/docs/polyglot/cloud-control-plane/). For
self-hosting, start with [Server](/docs/polyglot/server/) and
[Self-Hosting Deployments](/docs/deployment/). Then choose the
[PHP SDK](/docs/polyglot/php/),
[Python SDK](/docs/polyglot/python/), or
[Rust SDK](/docs/polyglot/rust/). Cloud users operate through Managed
Waterline. Self-hosted operators can use the
[Server API Reference](/docs/polyglot/server-api-reference/) and
[Monitoring](/docs/monitoring#waterline-service) when deploying a separate
Waterline service.

## Migration tooling to self-hosted service mode

The supported path from embedded mode to service mode is staged adoption, not a
live handoff of in-flight state:

- Use [Embedded to Server Migration](/docs/polyglot/embedded-to-server) for
  the step-by-step cutover.
- Use `GET /api/cluster/info` to confirm the target server build, topology, and
  capability contract before switching traffic.
- Use `POST /api/worker/register` plus the worker protocol to prove external
  workers can serve the stable type keys you chose.
- Use `GET /api/system/operator-metrics`, `dw worker:list`, or Waterline
  operator views to verify worker registration and compatible fleet coverage
  before shifting production traffic.
- Use
  [Client and Worker Capabilities](/docs/polyglot/cli-python-parity/)
  when replacing app-local control-plane calls with server-backed automation.
- Use Cloud's Managed Waterline, the Waterline deployment attached to a
  self-hosted runtime, or server-native history export for audit/debugging
  evidence; do not treat export bundles as an import path for live
  server-managed runs.

Three migration rules are non-negotiable:

1. Existing runs stay on the runtime where they started.
2. New server-managed runs use stable type keys, namespace names, task queues,
   and the fixed Avro payload contract from the first cutover.
3. Signals, queries, updates, repair, cancel, terminate, and archive must go to the
   runtime that owns the target run.

## Related References

- [Installation](/docs/installation)
- [Server](/docs/polyglot/server)
- [PHP SDK](/docs/polyglot/php)
- [Cloud Managed Runtime](/docs/polyglot/cloud-control-plane)
- [Embedded to Server Migration](/docs/polyglot/embedded-to-server)
- [Server Role Topology](/docs/polyglot/server-role-topology)
- [Server Config Reference](/docs/polyglot/server-config-reference)

<!-- Source: docs/polyglot/server.md -->

# Server

The published Durable Workflow server is a standalone workflow-orchestration
service implemented in PHP. It exposes the same durable engine used by the
embedded PHP package through a language-neutral HTTP+JSON control plane and
worker protocol. PHP is an implementation fact at this boundary, not an
application-language requirement.

## Is this only for PHP teams?

No. PHP, Python, and Rust are first-party SDK surfaces. A Python- or Rust-only
application team can deploy the published standalone server as infrastructure,
write application workflows and activities with its native SDK, and communicate
through the public protocol. Its application does not embed Laravel and does
not become a Laravel application.

The adoption paths are separate:

- **Standalone adoption:** operators deploy the published server image and its
  database, queue/cache, scheduler, and API roles. Application teams run PHP,
  Python, or Rust workers against that endpoint. The server owns orchestration
  state and the public control-plane/worker boundary.
- **Embedded adoption:** a Laravel application installs the PHP workflow
  package and owns the engine through its own queue, database, configuration,
  and deployment. This is the Laravel-native advantage, not a prerequisite for
  the standalone path.
- **Managed Cloud adoption:** Durable Workflow Cloud operates the orchestration
  runtime, persistence, placement, and recovery. Application teams run SDK
  clients and workers against a provisioned Cloud namespace. They do not deploy
  this Server distribution for Cloud or attach a self-hosted Server to it. See
  [Cloud Managed Runtime](/docs/polyglot/cloud-control-plane/).

All three choices use the same durable execution concepts. Standalone and
Cloud-managed external workers use stable string workflow/activity type
names and the shared codec envelope, so a cross-language child workflow or
activity preserves payload shape rather than exposing PHP serialization.

If you are deciding between the standalone server and package embedding, start
with [Deployment Modes](/docs/polyglot/deployment-modes). This page covers
the service-mode distribution.

Use the standalone server when you need:
- **Polyglot workflows** — PHP, Python, and Rust workflow and activity workers sharing one durable runtime
- **Microservice orchestration** — orchestrate services written in different languages
- **Centralized workflow runtime** — multiple applications sharing one workflow engine
- **Non-Laravel environments** — use Durable Workflow outside Laravel

If you already run v2 embedded in a Laravel app, use the
[embedded-to-server migration guide](/docs/polyglot/embedded-to-server) to
prepare type keys, deploy the server beside embedded execution, connect workers,
and route only new workflow starts to the server. Keep
[Deployment Modes](/docs/polyglot/deployment-modes) nearby during that
cutover so ids, command outcomes, task semantics, and runtime ownership rules
stay explicit.

Use [Worker Compatibility and Routing](/docs/polyglot/worker-compatibility-routing)
when you roll worker build cohorts, drain old cohorts, or need to keep
long-running runs pinned to compatible executors during rollback.

Use [Server Role Topology](/docs/polyglot/server-role-topology) when you
need the live role vocabulary, process classes, authority boundaries, failure
domains, or migration path that `GET /api/cluster/info` publishes.

## Quick Start

### Published Image + SQLite

The fastest source-free way to run the server is the published Docker image.
This quickstart uses SQLite, database queues, and file cache inside the
container. Mount `/app/database` so bootstrap and the API server share the same
SQLite file:

```bash
server_image=durableworkflow/server:2.0.0
export DW_AUTH_TOKEN=dev-token

docker volume create durable-workflow-server-quickstart

docker run --rm \
  -v durable-workflow-server-quickstart:/app/database \
  -e DW_AUTH_DRIVER=token \
  -e DW_AUTH_TOKEN="$DW_AUTH_TOKEN" \
  "$server_image" server-bootstrap

docker rm -f durable-workflow-server >/dev/null 2>&1 || true
docker run -d --name durable-workflow-server \
  -p 8080:8080 \
  -v durable-workflow-server-quickstart:/app/database \
  -e DW_AUTH_DRIVER=token \
  -e DW_AUTH_TOKEN="$DW_AUTH_TOKEN" \
  "$server_image"

until curl -sf http://localhost:8080/api/ready >/dev/null; do sleep 1; done
curl http://localhost:8080/api/health
curl -H "Authorization: Bearer $DW_AUTH_TOKEN" \
  http://localhost:8080/api/cluster/info \
  | jq '.topology | {current_shape, current_roles, execution_mode}'
```

This starts one API server container and creates the default namespace. It is
enough for local Python SDK workers and CLI checks. Use the published Compose
path below when you want MySQL, Redis, separate worker and scheduler
containers, or a closer production rehearsal.

### Published Image + Compose

Use the published Compose artifact when you want the source-free
multi-container stack backed by MySQL and Redis:

```bash
curl -fsSLO https://raw.githubusercontent.com/durable-workflow/server/main/docker-compose.published.yml

server_image=durableworkflow/server:2.0.0
export DW_AUTH_TOKEN=dev-token

env DW_SERVER_IMAGE="$server_image" docker compose \
  -f docker-compose.published.yml up -d --wait
curl -H "Authorization: Bearer $DW_AUTH_TOKEN" \
  http://localhost:8080/api/cluster/info \
  | jq '.topology | {current_shape, current_roles, execution_mode}'
```

### Ports

| Service | Port | Purpose |
|---------|------|---------|
| Server API | 8080 | Control-plane and worker-protocol endpoints |
| MySQL | 3306 | Database (exposed for development convenience) |
| Redis | 6379 | Cache and queue (exposed for development convenience) |

## Configuration

The server uses environment variables for configuration. Key settings are
summarized below; the full operator-facing `DW_*` contract is documented in
the [server config reference](/docs/polyglot/server-config-reference).

### Database

```bash
DB_CONNECTION=mysql
DB_HOST=mysql
DB_PORT=3306
DB_DATABASE=workflow
DB_USERNAME=workflow
DB_PASSWORD=secret
```

Supported: MySQL 8.0+, PostgreSQL 13+, SQLite 3.35+.

### Cache and Queue

```bash
CACHE_STORE=redis
QUEUE_CONNECTION=redis

REDIS_HOST=redis
REDIS_PORT=6379
REDIS_PASSWORD=null
REDIS_DB=0
```

Cache must support [atomic locks](https://laravel.com/docs/12.x/cache#atomic-locks). Queue drivers: Redis, Amazon SQS, Beanstalkd, database.

Atomic cache locks are required for server-side
[task queue admission caps](/docs/polyglot/task-queue-admission) and
query-task backpressure. Use Redis for multi-node deployments that need
workflow, activity, or query admission to hold across every server process.

### Authentication

The server supports three auth modes:

**Token-based** (default):

```bash
DW_AUTH_DRIVER=token
DW_AUTH_TOKEN=your-secret-token-here
```

All requests must send `Authorization: Bearer your-secret-token-here`.

For least-privilege deployments, configure role-scoped tokens instead of one
shared token:

```bash
DW_AUTH_DRIVER=token
DW_WORKER_TOKEN=worker-secret
DW_OPERATOR_TOKEN=operator-secret
DW_ADMIN_TOKEN=admin-secret
```

Worker tokens can register workers, poll tasks, heartbeat, and complete work.
Operator tokens can start, list, signal, query, update, repair, cancel,
terminate, archive, and observe workflows. Admin tokens can use administrative
endpoints such as namespace and retention management.

**HMAC signature**:

```bash
DW_AUTH_DRIVER=signature
DW_SIGNATURE_KEY=your-signature-secret
```

Requests must include `X-Signature`, calculated as
`hash_hmac('sha256', request_body, DW_SIGNATURE_KEY)`. The server
also accepts role-scoped signature keys:

```bash
DW_AUTH_DRIVER=signature
DW_WORKER_SIGNATURE_KEY=worker-signature-secret
DW_OPERATOR_SIGNATURE_KEY=operator-signature-secret
DW_ADMIN_SIGNATURE_KEY=admin-signature-secret
```

**No auth** (development only):

```bash
DW_AUTH_DRIVER=none
```

⚠️ **Do not use `none` in production.** All endpoints become publicly accessible.

### Workflow Package

The Docker image installs the `durable-workflow/workflow` package. Control which version:

```bash
# Build-time arg (set in docker-compose.yml or pass to docker build)
WORKFLOW_PACKAGE_REF=v2        # branch, tag, or commit
WORKFLOW_PACKAGE_SOURCE=       # custom Git remote (optional)
```

### Retention

Configure how long completed workflows remain queryable:

```bash
DW_HISTORY_RETENTION_DAYS=30
```

After retention expires, workflows are pruned. Configure per-namespace retention via the API.

### Namespaces

The `server-bootstrap` command runs migrations and seeds the `default`
namespace. Use `DW_DEFAULT_NAMESPACE` to change the namespace used when a
request omits the namespace header:

```bash
DW_DEFAULT_NAMESPACE=default
```

Create namespaces via the API:

```bash
curl -X POST http://localhost:8080/api/namespaces \
  -H "Authorization: Bearer $TOKEN" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "production",
    "description": "Production workflows",
    "retention_days": 90
  }'
```

## Health Checks

### API Health

```bash
curl http://localhost:8080/api/health
```

Returns `200 OK` with:

```json
{
  "status": "serving",
  "timestamp": "2026-04-15T12:00:00Z",
  "checks": {
    "database": "ok"
  },
  "topology": {
    "schema": "durable-workflow.v2.role-topology",
    "version": 4,
    "current_shape": "standalone_server",
    "current_process_class": "server_http_node",
    "current_roles": ["api_ingress", "control_plane", "matching", "history_projection"],
    "execution_mode": "remote_worker_protocol",
    "matching_role": {
      "queue_wake_enabled": true,
      "shape": "in_worker",
      "wake_owner": "worker_loop",
      "task_dispatch_mode": "poll",
      "partition_primitives": ["connection", "queue", "compatibility", "namespace"],
      "backpressure_model": "lease_ownership",
      "discovery_limits": {
        "poll_batch_cap": 100,
        "availability_ceiling_seconds": 1,
        "wake_signal_ttl_seconds": 60,
        "workflow_task_lease_seconds": 300,
        "activity_task_lease_seconds": 300
      }
    }
  }
}
```

### Public Topology Summary

Unauthenticated `GET /api/health` and `GET /api/ready` both publish the
responding node's `topology` summary. That public block is intentionally
smaller than `/api/cluster/info`, but it still exposes the fields needed to
identify split-role nodes before control-plane auth or namespace resolution:

- `topology.schema`
- `topology.version`
- `topology.current_shape`
- `topology.current_process_class`
- `topology.current_roles`
- `topology.execution_mode`
- `topology.matching_role.queue_wake_enabled`
- `topology.matching_role.shape`
- `topology.matching_role.wake_owner`
- `topology.matching_role.task_dispatch_mode`
- `topology.matching_role.partition_primitives`
- `topology.matching_role.backpressure_model`
- `topology.matching_role.discovery_limits.poll_batch_cap`
- `topology.matching_role.discovery_limits.availability_ceiling_seconds`
- `topology.matching_role.discovery_limits.wake_signal_ttl_seconds`
- `topology.matching_role.discovery_limits.workflow_task_lease_seconds`
- `topology.matching_role.discovery_limits.activity_task_lease_seconds`

`topology.matching_role.discovery_limits` is the frozen numeric matching-role
contract that compiles into the workflow package: `poll_batch_cap` is the
maximum batch of ready-task rows returned per poll,
`availability_ceiling_seconds` is the cross-backend tolerance applied to
`available_at` so freshly-available tasks survive sub-second timestamp drift,
`wake_signal_ttl_seconds` is the default `CacheLongPollWakeStore` signal TTL,
and `workflow_task_lease_seconds` / `activity_task_lease_seconds` are the
default workflow and activity task lease durations. Tightening any of these
values is a protocol-level change because workers and downstream tooling read
them as the authoritative matching-role contract.

The same summary appears on `/api/ready` even when the deployment is not ready,
so probes can still distinguish `server_http_node`, `scheduler_node`,
`matching_node`, and `execution_node` responses while bootstrap blockers are
active.

### Readiness

```bash
curl http://localhost:8080/api/ready
```

`/api/ready` is the deployment gate. It returns `200 OK` only when bootstrap
prerequisites and rollout-safety health are in a ready or warning state.

Treat the machine-readable fields as follows:

- `checks.migrations.repository_exists` and `checks.migrations.pending_migrations`
  tell you whether the migration repository exists and which migration records
  are still pending.
- `checks.migrations.adoptable_migrations` lists create-table migrations that
  only need adoption into migration history. This is a `warning`, not a
  fail-closed outage, so the server can stay ready while operators schedule the
  adoption.
- `checks.migrations.blocking_migrations` lists rollout-safety migration records
  that must land before the server should admit traffic. When this array is not
  empty, readiness fails closed with `checks.migrations.status = "pending"`.
- `checks.migrations.missing_tables` reports durable tables that are still
  absent, and `checks.migrations.operator_surface` tells you whether the v2
  operator surface is available enough to explain rollout safety once the server
  boots.
- `checks.migrations.readiness_contract.version` pins the boot and migration
  adoption contract revision that scripts should parse.
- `checks.workflow_v2` mirrors the all-namespaces rollout-safety verdict. When
  rollout-safety cannot be evaluated yet it returns `status: "blocked"` plus
  `blocked_by`, `message`, and `remediation` so operators can fix the upstream
  readiness gate instead of chasing queue symptoms.

### Workflow Bootstrap Gate

`checks.workflow_v2.status: "blocked"` is also a route-level gate, not just a
readiness signal. While workflow v2 bootstrap is blocked, the server fails
closed on workflow start/mutation, schedule mutation, bridge-adapter, and
worker-protocol routes with HTTP `503` and a machine-readable
`reason: "workflow_v2_blocked"` payload. The gate runs after role and
protocol-version validation but before namespace resolution, so blocked
requests never observe namespace existence.

The bootstrap-gate response always carries:

- `reason: "workflow_v2_blocked"` so callers branch on a machine-readable name
  instead of a prose message.
- `blocked_by`: the ordered list of upstream readiness blockers (for example
  `migrations`).
- `remediation`: the operator-facing instruction for clearing the listed
  blockers, mirrored from `/api/ready` `checks.workflow_v2.remediation`.

Bootstrap-gated route families:

- **Workflow start and mutation** — `/api/workflows` start, command, and
  run-targeted command routes.
- **Schedule mutation** — `POST /api/schedules`,
  `PUT /api/schedules/{scheduleId}`, `DELETE /api/schedules/{scheduleId}`,
  `POST /api/schedules/{scheduleId}/pause`,
  `POST /api/schedules/{scheduleId}/resume`,
  `POST /api/schedules/{scheduleId}/trigger`, and
  `POST /api/schedules/{scheduleId}/backfill`.
- **Bridge adapters** — `POST /api/bridge-adapters/webhook/{adapter}`.
- **Worker protocol** — every `/api/worker` and `/api/worker/*` route, including
  registration, heartbeat, workflow-task, query-task, and activity-task verbs.
  Worker-protocol routes return the bootstrap-gate payload in the
  worker-protocol envelope and keep the
  `X-Durable-Workflow-Protocol-Version` header so worker SDKs can branch on the
  same `reason: "workflow_v2_blocked"` field they parse from the control plane.

Schedule **reads** are intentionally exempted so operators can inspect
schedule state during recovery: `GET /api/schedules`,
`GET /api/schedules/{scheduleId}`, and
`GET /api/schedules/{scheduleId}/history` continue to serve while the
bootstrap gate is blocking other routes.

### Server Capabilities

```bash
curl http://localhost:8080/api/cluster/info \
  -H "Authorization: Bearer $TOKEN"
```

Returns the server build version, supported SDK versions, engine capabilities,
the client compatibility policy, and the independently-versioned control-plane
and worker-protocol manifests:

```json
{
  "server_id": "server-1",
  "version": "2.0.0",
  "default_namespace": "default",
  "supported_sdk_versions": {
    "php": ">=1.0",
    "python": ">=0.2,<1.0",
    "cli": ">=0.1,<1.0"
  },
  "client_compatibility": {
    "schema": "durable-workflow.v2.client-compatibility",
    "version": 1,
    "authority": "protocol_manifests",
    "top_level_version_role": "informational",
    "fail_closed": true
  },
  "capabilities": {
    "workflow_tasks": true,
    "activity_tasks": true,
    "signals": true,
    "queries": true,
    "updates": true,
    "schedules": true,
    "child_workflow_retry_policy": true,
    "child_workflow_timeouts": true,
    "payload_codecs": ["avro"],
    "response_compression": ["gzip", "deflate"]
  },
  "control_plane": {
    "version": "2",
    "header": "X-Durable-Workflow-Control-Plane-Version",
    "request_contract": { "schema": "durable-workflow.v2.control-plane-request.contract", "version": 1, "...": "..." },
    "response_contract": { "schema": "durable-workflow.v2.control-plane-response.contract", "version": 1, "...": "..." }
  },
  "worker_protocol": {
    "version": "1.0",
    "server_capabilities": {
      "long_poll_timeout": 30,
      "supported_workflow_task_commands": [
        "complete_workflow",
        "fail_workflow",
        "continue_as_new",
        "schedule_activity",
        "start_timer",
        "start_child_workflow"
      ],
      "workflow_task_poll_request_idempotency": true,
      "poll_status": true,
      "history_page_size_default": 500,
      "history_page_size_max": 1000,
      "activity_retry_policy": true,
      "activity_timeouts": true,
      "child_workflow_retry_policy": true,
      "child_workflow_timeouts": true,
      "parent_close_policy": true,
      "non_retryable_failures": true,
      "response_compression": ["gzip", "deflate"],
      "history_compression": {
        "supported_encodings": ["gzip"],
        "compression_threshold": 8192
      }
    }
  }
}
```

Treat `client_compatibility.authority: "protocol_manifests"` as the rule for
client checks. The top-level `version` is build identity; CLI and SDK clients
should fail closed when `control_plane.version`,
`control_plane.request_contract`, or `worker_protocol.version` is missing or
unsupported.

### Role topology and deployment shape

The field-by-field reference for this manifest lives on
[Server Role Topology](/docs/polyglot/server-role-topology). Keep this
section for the inline `cluster/info` example and use the dedicated page when
you need the supported shapes, authority boundaries, failure domains, scaling
boundaries, or migration-path contract in one place.

`GET /api/cluster/info` also publishes a `topology` manifest. It is the
machine-readable role map for the node that answered the request, so operators
and automation can read one contract instead of inferring node duties from
container names or rollout runbooks.

```json
{
  "topology": {
    "schema": "durable-workflow.v2.role-topology",
    "version": 2,
    "supported_shapes": [
      "embedded",
      "standalone_server",
      "split_control_execution"
    ],
    "role_vocabulary": [
      "api_ingress",
      "control_plane",
      "matching",
      "history_projection",
      "scheduler",
      "execution_plane"
    ],
    "current_shape": "standalone_server",
    "current_process_class": "server_http_node",
    "current_roles": [
      "api_ingress",
      "control_plane",
      "matching",
      "history_projection"
    ],
    "execution_mode": "remote_worker_protocol",
    "matching_role": {
      "queue_wake_enabled": true,
      "shape": "in_worker",
      "wake_owner": "worker_loop",
      "task_dispatch_mode": "poll",
      "partition_primitives": [
        "connection",
        "queue",
        "compatibility",
        "namespace"
      ],
      "backpressure_model": "lease_ownership",
      "discovery_limits": {
        "poll_batch_cap": 100,
        "availability_ceiling_seconds": 1,
        "wake_signal_ttl_seconds": 60,
        "workflow_task_lease_seconds": 300,
        "activity_task_lease_seconds": 300
      }
    },
    "shape_assignments": {
      "embedded": {
        "process_classes": [
          {
            "name": "application_process",
            "roles": [
              "control_plane",
              "matching",
              "history_projection",
              "scheduler",
              "execution_plane"
            ]
          }
        ]
      },
      "standalone_server": {
        "process_classes": [
          {
            "name": "server_http_node",
            "roles": [
              "api_ingress",
              "control_plane",
              "matching",
              "history_projection"
            ]
          },
          {
            "name": "scheduler_node",
            "roles": ["scheduler"]
          },
          {
            "name": "worker_node",
            "roles": ["execution_plane"]
          }
        ]
      },
      "split_control_execution": {
        "process_classes": [
          {
            "name": "ingress_node",
            "roles": ["api_ingress"]
          },
          {
            "name": "control_plane_node",
            "roles": ["control_plane", "history_projection"]
          },
          {
            "name": "scheduler_node",
            "roles": ["scheduler"]
          },
          {
            "name": "matching_node",
            "roles": ["matching"]
          },
          {
            "name": "execution_node",
            "roles": ["execution_plane"]
          }
        ]
      }
    },
    "authority_boundaries": {
      "control_plane": {
        "writes": [
          "workflow_instances",
          "workflow_runs.status",
          "workflow_tasks.lifecycle"
        ]
      },
      "execution_plane": {
        "writes": [
          "workflow_tasks.outcomes",
          "activity_attempts",
          "worker_compatibility_heartbeats"
        ]
      },
      "matching": {
        "writes": [
          "workflow_tasks.leases",
          "activity_tasks.leases"
        ]
      },
      "history_projection": {
        "writes": [
          "history_events",
          "workflow_run_summaries",
          "workflow_history_exports"
        ]
      },
      "scheduler": {
        "writes": [
          "workflow_schedules.fire_state",
          "workflow_starts.scheduled"
        ]
      },
      "api_ingress": {
        "writes": ["worker_registrations"]
      }
    },
   "failure_domains": {
      "control_plane_down": {
        "effect": "workers_continue_claimed_tasks_only_until_lease_expiry",
        "operator_signal": "operator_commands_fail_fast"
      },
      "execution_plane_down": {
        "effect": "ready_tasks_accumulate_without_loss",
        "operator_signal": "operators_see_ready_depth_growth"
      },
      "matching_down": {
        "effect": "claim_falls_back_to_direct_ready_task_discovery",
        "operator_signal": "ready_depth_rises_while_claim_rate_falls"
      },
      "history_projection_down": {
        "effect": "projection_reads_may_stale_while_durable_writes_continue",
        "operator_signal": "projection_lag_seconds_may_increase"
      },
      "scheduler_down": {
        "effect": "scheduled_workflows_stop_firing_and_record_missed_runs",
        "operator_signal": "operators_see_missed_schedule_state"
      },
      "api_ingress_down": {
        "effect": "external_http_traffic_stops_at_the_edge",
        "operator_signal": "embedded_in_process_calls_may_continue"
      }
    },
    "scaling_boundaries": {
      "api_ingress": "incoming_http_request_rate",
      "control_plane": "operator_commands_and_run_lifecycle_transitions",
      "matching": "ready_task_rate_and_poller_count",
      "history_projection": "durable_event_rate",
      "scheduler": "active_schedule_count",
      "execution_plane": "workflow_and_activity_task_rate"
    },
    "migration_path": [
      {
        "step": "audit_role_boundaries",
        "result": "tooling flags cross-role writes before runtime shape changes",
        "reversible": true
      },
      {
        "step": "expose_role_bindings",
        "result": "container seams allow out-of-process adapters without patching the package",
        "reversible": true
      },
      {
        "step": "introduce_dedicated_matching_shape",
        "result": "matching can run as its own process class without changing the claim contract",
        "reversible": true
      },
      {
        "step": "split_history_projection",
        "result": "history and projections can move out of process without introducing a second writer",
        "reversible": true
      },
      {
        "step": "split_scheduler",
        "result": "schedule firing can move behind leader election while single-replica deployments stay legal",
        "reversible": true
      },
      {
        "step": "optional_execution_partitioning",
        "result": "workers can partition by namespace, connection, queue, and compatibility",
        "reversible": true
      }
    ],
    "kernel_invariants": [
      {
        "id": "single_persistence_engine",
        "summary": "one workflow database backs every topology shape; role split does not introduce a second persistence engine",
        "applies_to": ["embedded", "standalone_server", "split_control_execution"]
      },
      {
        "id": "single_worker_protocol",
        "summary": "one HTTP worker protocol carries claim, complete, fail, and heartbeat traffic across every topology; role split does not fork the worker contract",
        "applies_to": ["embedded", "standalone_server", "split_control_execution"]
      },
      {
        "id": "single_history_writer",
        "summary": "history_events has exactly one durable writer per logical event regardless of where the history/projection role runs",
        "applies_to": ["embedded", "standalone_server", "split_control_execution"]
      },
      {
        "id": "single_control_authority_per_run",
        "summary": "every mutation of a given workflow run routes through one control-plane authority; per-run row locks serialise transitions across replicas",
        "applies_to": ["embedded", "standalone_server", "split_control_execution"]
      },
      {
        "id": "embedded_topology_remains_supported",
        "summary": "the embedded shape where one process fills every role MUST stay legal; existing embedded hosts are never forced to migrate",
        "applies_to": ["embedded", "standalone_server", "split_control_execution"]
      },
      {
        "id": "role_split_is_topology_only",
        "summary": "splitting roles is a topology change, not a product fork; collapsing the roles back onto a single process is always a legal topology",
        "applies_to": ["embedded", "standalone_server", "split_control_execution"]
      }
    ]
  },
  "coordination_health": {
    "schema": "durable-workflow.v2.coordination-health.contract",
    "version": 2,
    "namespace_scope": "all_namespaces",
    "status": "ok",
    "http_status": 200,
    "warning_checks": [],
    "error_checks": [],
    "categories": {
      "correctness": "ok"
    },
    "checks": [
      {
        "name": "worker_compatibility",
        "status": "ok",
        "category": "correctness",
        "message": null
      },
      {
        "name": "activity_path",
        "status": "ok",
        "category": "correctness",
        "message": null
      }
    ],
    "routing_drains": {
      "queues_with_drains": 0,
      "draining_build_id_count": 0,
      "active_worker_count": 0,
      "draining_worker_count": 0,
      "stale_worker_count": 0,
      "queues": []
    }
  }
}
```

Treat `topology.version` as the role-manifest schema version, not as a synonym
for the top-level server build version. Automation should check that field
before assuming fields added by a newer topology manifest revision. The
current public contract includes `supported_shapes`, `role_vocabulary`,
`current_shape`, `current_process_class`, `current_roles`, `execution_mode`,
`matching_role`, `role_catalog`, `shape_assignments`,
`authority_boundaries`, `authority_surfaces`, `failure_domains`,
`supported_topologies`, `scaling_boundaries`, `migration_path`, and
`kernel_invariants`.

Read the fields as follows:

- `supported_shapes` names the legal product topologies.
- `role_vocabulary` is the fixed list of v2 role names. Treat it as the
  canonical vocabulary for automation and diagnostics.
- `current_shape`, `current_process_class`, and `current_roles` describe the
  node you queried right now. Use `current_process_class` as the node's
  declared identity, then compare the current role bundle against
  `shape_assignments` for the current shape when you need to validate that
  declaration.
- `execution_mode` distinguishes embedded local queue execution
  (`local_queue_worker`) from standalone server worker-protocol execution
  (`remote_worker_protocol`).
- `matching_role.queue_wake_enabled`, `matching_role.shape`,
  `matching_role.wake_owner`, `matching_role.task_dispatch_mode`,
  `matching_role.partition_primitives`, and
  `matching_role.backpressure_model` tell you whether the node still runs the
  in-worker wake path or expects a dedicated repair or matching loop to own
  that sweep, which routing axes remain stable, and which durable admission
  boundary the matching layer currently enforces.
- `matching_role.discovery_limits` freezes the numeric matching-role contract
  values the workflow package compiles in: `poll_batch_cap` (the maximum batch
  of ready-task rows returned per poll), `availability_ceiling_seconds` (the
  cross-backend tolerance applied to `available_at` so freshly-available tasks
  survive sub-second timestamp drift), `wake_signal_ttl_seconds` (the default
  long-poll wake-signal TTL), `workflow_task_lease_seconds` (the default
  workflow task lease), and `activity_task_lease_seconds` (the default
  activity task lease). Operators read these to verify the deployment matches
  the documented matching-role contract without grepping the package source;
  tightening any value is a protocol-level change.
- `role_catalog` and `authority_surfaces` tell you which interfaces and
  durable mutation paths each role owns on the current manifest revision.
- `shape_assignments` maps each supported shape to the process classes and role
  bundles that shape is allowed to run.
- `supported_topologies` summarizes the deployment families the product
  supports and the node classes each family expects.
- `authority_boundaries` names which durable write surfaces each role is
  expected to mutate, so operators can catch cross-role drift before they split
  a deployment.
- `failure_domains` describes the first operator-visible degradation signal when
  a role goes down, instead of leaving that expectation implicit in a runbook.
- `scaling_boundaries` names the main load dimension for each role when the
  topology is split.
- authenticated hosted routes fail closed when the responding node does not
  host the HTTP control surface. In that case the server returns `503` with
  `reason: "topology_role_unavailable"` plus `current_shape`,
  `current_process_class`, `current_roles`, `required_roles`, and
  `missing_roles` so callers can reroute to a node that actually exposes the
  requested surface.
- `coordination_health` is the fleet-wide rollout-safety summary published from
  the same discovery call. It uses `all_namespaces` scope, summarizes the
  current status and HTTP posture, lists the normalized warning/error check
  names that also feed readiness health, and adds `blocked_by`, `message`, plus
  `remediation` when rollout-safety evaluation is blocked by upstream readiness
  problems.
- `coordination_health.checks[]` always includes the frozen check `activity_path`
  next to `worker_compatibility`, `task_transport`, `routing_health`,
  `durable_resume_paths`, and the projection/scheduler checks. `activity_path`
  is the activity-side counterpart of `task_transport`: it surfaces activity
  executions whose schedule-to-start, start-to-close, schedule-to-close, or
  heartbeat deadline has passed without enforcement (`timeout_overdue`,
  `oldest_timeout_overdue_at`, `max_timeout_overdue_age_ms`) and the sustained
  retry backlog (`retrying`, `oldest_retrying_started_at`,
  `max_retrying_age_ms`). Renaming the check is a protocol-level change.
- `coordination_health.routing_drains` summarizes draining build-id cohorts
  across queues and namespaces. `queues_with_drains` greater than zero means the
  fleet is intentionally holding traffic away from at least one draining cohort.
- `migration_path` lists the ordered rollout steps from today's standalone
  distribution toward more isolated role boundaries without introducing a
  second engine. Each entry's `reversible: true` flag declares that
  collapsing back to a less-isolated shape stays a legal topology.
- `kernel_invariants` enumerates the durable-kernel guarantees the role
  split must preserve regardless of which supported shape is running:
  `single_persistence_engine`, `single_worker_protocol`,
  `single_history_writer`, `single_control_authority_per_run`,
  `embedded_topology_remains_supported`, and `role_split_is_topology_only`.
  Each entry's `applies_to` lists the supported shapes the invariant
  covers; rollout automation MAY use the field to assert that a candidate
  topology change preserves the kernel before applying the shape change.

This keeps the role split as a topology change, not a second engine or a
separate control-plane API. When a deployment evolves from a narrow
`standalone_server` fleet toward a more explicit `split_control_execution`
shape, operators still read the same discovery surface. The values under
`current_shape`, `current_roles`, `execution_mode`, `matching_role`,
`shape_assignments`, `authority_boundaries`, `failure_domains`,
`scaling_boundaries`, and `migration_path` are
versioned as one manifest so rollout tooling can reason about the same
topology surface the server ships.

The same constraint also surfaces machine-readably through
`topology.kernel_invariants` so rollout automation can verify that no
candidate topology change introduces a second persistence engine, a
forked worker protocol, a second history writer, or a non-reversible
migration before applying the change.

The hosted-route gate applies only to authenticated API and worker endpoints.
`GET /api/health`, `GET /api/ready`, and authenticated `GET /api/cluster/info`
stay available for discovery, liveness, and topology inspection even on
`scheduler_node`, `matching_node`, or `execution_node` processes that do not
host the current HTTP control surface.

For carrier-neutral external handlers, the same endpoint publishes
`worker_protocol.external_execution_surface_contract`. That manifest names the
[activity-grade external execution surface](/docs/polyglot/external-execution),
links the external task input/result envelope contracts, and keeps workflow
replay, `ContinueAsNew`, signal/update/query ordering, and event-history
interpretation inside real runtimes.

Key field notes for client code:

- The app version is `version`, not `server_version`.
- Workflow-task command capabilities live under `worker_protocol.server_capabilities.supported_workflow_task_commands`, not at the top of `worker_protocol`. The same nested object is echoed on every worker-plane response via the `server_capabilities` field.
- `worker_protocol.server_capabilities.poll_status` means poll responses keep a
  machine-readable `poll_status` field even when no task is leased, so workers
  can distinguish `empty`, `throttled`, `unavailable`, and `draining` outcomes
  without scraping prose error messages.
- Worker command-option capabilities, including retry policies, timeout fields, parent-close policy, and non-retryable failures, are also echoed in `server_capabilities` so workers can negotiate behavior without a separate cluster-info request.
- The sole v2 payload codec lives under `capabilities.payload_codecs`; the list is exactly `['avro']`. PHP-only v1 import/drain serializers are internal migration mechanics and never appear in runtime capabilities.

## Connecting Workers

Workers poll the server for tasks and execute workflow code or activities. See the [Worker Protocol](/docs/polyglot/worker-protocol) reference for the full API contract.
For the route role matrix, namespace lookup rules, and exact worker
registration payload, see
[Namespace, Auth, And Worker Registration](/docs/polyglot/namespace-auth-workers).

### PHP Workers

Framework-neutral PHP applications and remote workers use the standalone-server
SDK. They do not install Laravel or the embedded Workflow package:

<!-- docs-example id="server.php-sdk.install" -->
```bash
composer require durable-workflow/sdk:2.0.0
```

Create a client and register workflow or activity callables with a worker:

```php
use DurableWorkflow\Client;
use DurableWorkflow\Worker;
use DurableWorkflow\Worker\WorkflowContext;

$client = new Client(
    'http://localhost:8080',
    token: getenv('DURABLE_WORKFLOW_AUTH_TOKEN') ?: null,
    namespace: 'default',
);

$worker = new Worker($client, taskQueue: 'polyglot-php');
$worker->registerWorkflow(
    'invoice',
    static function (WorkflowContext $context, string $invoiceId): array {
        $context->activity('charge-card', [$invoiceId]);

        return ['invoice_id' => $invoiceId, 'status' => 'paid'];
    },
);
$worker->run();
```

See the [PHP SDK guide](/docs/polyglot/php/) for the client, authentication,
payload, and worker lifecycle. The separate `durable-workflow/workflow` package
is the embedded Laravel runtime and the engine hosted inside the published
server image. Embedded Laravel workflows run package-local tasks through the
application's queue worker and do not require this standalone server.

### Python Workers

Python workers use the `durable-workflow` SDK:

```bash
pip install durable-workflow==2.0.0
```

See the [Python SDK](/docs/polyglot/python) guide for worker setup.

### Custom Language Workers

Any language can implement a worker by:
1. Registering with `POST /api/worker/register`
2. Long-polling for tasks with `POST /api/worker/workflow-tasks/poll`, `POST /api/worker/activity-tasks/poll`, or `POST /api/worker/query-tasks/poll`
3. Completing tasks with `POST /api/worker/workflow-tasks/{id}/complete`, `POST /api/worker/activity-tasks/{id}/complete`, or `POST /api/worker/query-tasks/{id}/complete`

All requests require:
- `Authorization: Bearer $TOKEN`
- `X-Namespace: your-namespace`
- `X-Durable-Workflow-Protocol-Version: 1.0`

The server validates that the namespace exists. Register it via
`POST /api/namespaces` before directing workers or clients at it, or the
server returns `404` with `reason: "namespace_not_found"`.

See the [server README](https://github.com/durable-workflow/server#getting-started-end-to-end-workflow) for a curl-based walkthrough.

See [Task Queue Admission](/docs/polyglot/task-queue-admission) to tune
worker registration slots, server-side active lease caps, per-minute dispatch
budgets, and query-task backpressure.

## CLI

The [Durable Workflow CLI](/docs/polyglot/cli) provides a shell interface to the server:

```bash
# Install — Linux and macOS
curl -fsSL https://durable-workflow.com/install.sh | sh

# Install — macOS (Homebrew alternative)
brew install durable-workflow/tap/dw

# Install — Windows (PowerShell)
# irm https://durable-workflow.com/install.ps1 | iex

# Configure
export DURABLE_WORKFLOW_SERVER_URL=http://localhost:8080
export DURABLE_WORKFLOW_AUTH_TOKEN=your-token
export DURABLE_WORKFLOW_NAMESPACE=default

# Use
dw server:health
dw workflow:list
dw workflow:start --type=my-workflow --input='["value"]'
dw workflow:start --type=my-workflow --input-file=input.json
```

See the [CLI install page](/docs/polyglot/cli#install) for a platform-detecting installer and direct binary downloads.

Task queue commands include admission status for workflow tasks, activity
tasks, and query tasks. Use them to distinguish missing workers, saturated
worker slots, server-side active lease or dispatch-rate throttling, and
query-task overflow.

## Deployment

Use the [self-hosting deployment guide](/docs/deployment) to choose a
supported topology before deploying production traffic. It separates local
development, single-node production, small clustered deployments, raw
Kubernetes manifests, and support-led topologies.

The self-serve small-cluster contract is deliberately narrow: 2-3 stateless API
nodes behind a load balancer, one shared external MySQL or PostgreSQL database,
shared Redis, independently scaled workers, and exactly one scheduler or
maintenance runner. Choose stop-the-world upgrades or
[rolling upgrades](/docs/rolling-upgrades) per release; the rolling-upgrade
contract names the version-skew, schema, drain, readiness, and rollback
guarantees that must hold. SQLite clustering, Redis-less multi-node mode,
duplicate schedulers, active/active multi-region, Helm, and provider-specific
failover are outside that contract until separately validated. Active/passive
multi-region material in the
[self-hosting guide](/docs/deployment#activepassive-multi-region) is
support-led evaluation guidance, not a proven self-serve 2.0 contract; each
candidate region must still start from the documented single-region or
small-cluster shape.

For self-hosted server deployments, start from published images rather than
source-tree builds:

- Docker Hub: `durableworkflow/server:2.0.0`
- GitHub Container Registry: `ghcr.io/durable-workflow/server:2.0.0`
- Published-image Compose:
  [`docker-compose.published.yml`](https://github.com/durable-workflow/server/blob/main/docker-compose.published.yml)
- Raw Kubernetes manifests:
  [`k8s/`](https://github.com/durable-workflow/server/tree/main/k8s)

Production deployments should pin a version tag or image digest, use
role-scoped credentials, run bootstrap/migrations before serving traffic, and
prove readiness with `/api/ready`, `/api/cluster/info`, and worker
registration. Do not shift production traffic based on `/api/health` alone.

## API Reference

For a complete endpoint-by-endpoint reference, including required headers,
roles, worker-protocol routes, external payload storage routes, and named error
reasons, see the [Server API Reference](/docs/polyglot/server-api-reference).

The server exposes three API surfaces:

### Control Plane

Start, describe, signal, query, update, cancel, and terminate workflows; manage namespaces, task queues, schedules, search attributes, and workers. Every control-plane request requires `X-Durable-Workflow-Control-Plane-Version: 2`. Requests without it are rejected with `missing_control_plane_version`.

Key endpoints:
- `POST /api/workflows` — Start a workflow
- `GET /api/workflows/{id}` — Describe a workflow
- `POST /api/workflows/{id}/signal/{name}` — Send a signal
- `POST /api/workflows/{id}/query/{name}` — Execute a query
- `POST /api/workflows/{id}/update/{name}` — Execute an update
- `POST /api/workflows/{id}/cancel` — Request cancellation
- `POST /api/workflows/{id}/terminate` — Terminate immediately
- `GET /api/workflows/{id}/runs/{runId}/history` — List run history events
- `GET /api/workflows/{id}/runs/{runId}/history/export` — Export a replay bundle
- `GET /api/namespaces`, `POST /api/namespaces`, `GET|PUT /api/namespaces/{namespace}` — Namespace management
- `GET /api/workers`, `GET|DELETE /api/workers/{id}` — Worker fleet management
- `GET /api/task-queues`, `GET /api/task-queues/{taskQueue}` — Task queue backlog, pollers, leases, and admission visibility
- `GET|POST /api/schedules`, `GET|PUT|DELETE /api/schedules/{id}`, `POST /api/schedules/{id}/{pause|resume|trigger|backfill}` — Schedule management
- `GET|POST|DELETE /api/search-attributes` — Search attribute management
- `GET|POST|PUT|DELETE /api/service-endpoints...` — Admin-only service catalog endpoints, nested services, operation bindings, and durable service-call snapshots
- `POST /api/system/repair/pass`, `POST /api/system/activity-timeouts/pass`, `POST /api/system/retention/pass` — Operator passes

Workflow control-plane responses, including run-history listing responses,
include the nested `control_plane` contract metadata that identifies the
operation and response contract version. History export is intentionally not
wrapped in that envelope; it returns the replay bundle unchanged so the bundle
integrity checksum and optional signature cover the exact artifact received by
the client.

Validation failures return HTTP 422 with `reason: validation_failed` plus
`errors` and `validation_errors`. Workflow operation routes also project that
reason and validation detail into `control_plane.reason` and
`control_plane.validation_errors`. Current run-targeted command routes project
the URL `run_id` in the response and `control_plane.run_id`, so clients can
distinguish instance-level commands from explicit selected-run commands.

Task queue visibility is the operator surface for deciding whether a queue is
falling behind because durable backlog is growing, workers have no available
slots, or the server is enforcing admission limits. `GET /api/task-queues`
returns one summary entry per queue; `GET /api/task-queues/{taskQueue}`
expands one queue with `pollers` and `current_leases`. Both routes expose
`stats.approximate_backlog_count`, `stats.approximate_backlog_age_seconds`,
and the per-kind `stats.workflow_tasks.*` / `stats.activity_tasks.*` readiness
and lease counters. The detailed route also includes the `admission` object so
automation can separate worker-capacity pressure from server-side queue or
query-task throttling. Fleet-level durable inflow versus dispatch rates live on
the operator-metrics surfaces (`operator_metrics.backlog.tasks_added_last_minute`
and `operator_metrics.backlog.tasks_dispatched_last_minute`), not on the
per-queue task-queue routes.

### Worker Protocol

Workers register, poll for tasks, heartbeat, and complete tasks. Requires `X-Durable-Workflow-Protocol-Version: 1.0`.

Key endpoints:
- `POST /api/worker/register` — Register a worker
- `POST /api/worker/workflow-tasks/poll` — Long-poll for workflow tasks
- `POST /api/worker/workflow-tasks/{id}/complete` — Complete workflow task
- `POST /api/worker/query-tasks/poll` — Long-poll for server-routed workflow query tasks
- `POST /api/worker/query-tasks/{id}/complete` — Complete workflow query task
- `POST /api/worker/query-tasks/{id}/fail` — Fail or reject workflow query task
- `POST /api/worker/activity-tasks/poll` — Long-poll for activity tasks
- `POST /api/worker/activity-tasks/{id}/complete` — Complete activity task

See the [Worker Protocol](/docs/polyglot/worker-protocol) reference for details.

### Discovery (unversioned)

The only endpoints that do **not** require `X-Durable-Workflow-Control-Plane-Version` are discovery and health probes:

- `GET /api/health` — Liveness probe plus the public `topology` summary (no auth required)
- `GET /api/ready` — Readiness probe plus the same `topology` summary (no auth required)
- `GET /api/cluster/info` — Server capabilities, protocol versions, and the sole Avro payload codec. Clients should hit this first to discover which control-plane and worker-protocol versions the server supports.

## Troubleshooting

### Workers not receiving tasks

**Check:**
1. Workers registered? `curl http://localhost:8080/api/workers -H "Authorization: Bearer $TOKEN" -H "X-Durable-Workflow-Control-Plane-Version: 2" -H "X-Namespace: default"`
2. Workers polling correct task queue?
3. Workflow started with matching task queue?
4. Cache backend shared across server instances?

### Long-poll connections timing out immediately

**Check:**
1. Cache driver supports atomic locks? Test with `php artisan workflow:v2:doctor --strict`
2. Redis reachable from server?
3. Load balancer timeout set higher than long-poll timeout (default: 60s)?

### Database connection errors

**Check:**
1. Database host and port correct?
2. Credentials valid?
3. Database exists?
4. Migrations run? `php artisan migrate:status`

### Auth failures

**Check:**
1. `DW_AUTH_DRIVER` matches client auth method?
2. Token/HMAC secret matches between server and client?
3. Auth headers present? `Authorization: Bearer $TOKEN` or HMAC signature headers?

## Learn More

- [Worker Protocol Reference](/docs/polyglot/worker-protocol) — Full API contract for workers
- [Embedded to Server Migration](/docs/polyglot/embedded-to-server) — Adopt the server from a Laravel embedded v2 app
- [Python SDK](/docs/polyglot/python) — Build Python workers
- [CLI](/docs/polyglot/cli) — Command-line interface
- [Server Repository](https://github.com/durable-workflow/server) — Source code, issues, releases

<!-- Source: docs/polyglot/cli.mdx -->

# CLI

The Durable Workflow CLI (`dw`) is a shell interface to the
[standalone server](/docs/polyglot/server). It lets operators start, list,
signal, query, update, repair, cancel, terminate, and archive workflows, manage
schedules, inspect task queues, and check server health — all from the command
line.

The same CLI works against any Durable Workflow server, regardless of which
language your workflows are written in. See
[Client and Worker Capabilities](/docs/polyglot/cli-python-parity/)
for the supported and intentionally different CLI, PHP, Python, and Rust
surfaces.

## Install

### Verify

```bash
dw --version
```

### Supported channel install

The default installer follows the CLI release from the last passing qualified
artifact tuple without requiring an RC sequence number. A newer published CLI
prerelease is not selected until the public compatibility authority qualifies
it. For reproducible CI, record the resolved `dw --version` value and pass that
exact tag through `VERSION` on later runs.

Both installer scripts download `SHA256SUMS` from the release and verify the
asset checksum before replacing `dw`, so a tampered mirror fails the install.

## Linux and macOS (shell installer)

```bash
curl -fsSL https://durable-workflow.com/install.sh | sh
dw --version
```

`VERSION` accepts any published release tag, `supported`, `prerelease`, or
`stable`. Leave it unset to follow the qualified `supported` channel. The
explicit `prerelease` channel requires the qualified release to be a
prerelease. Additional environment variables:

- `DURABLE_WORKFLOW_INSTALL_DIR` — install location (default `~/.local/bin`).
- `DURABLE_WORKFLOW_BIN_NAME` — installed executable name (default `dw`).

A GitHub Actions example:

```yaml
- name: Install Durable Workflow CLI
  run: |
    curl -fsSL https://durable-workflow.com/install.sh | sh
    echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- name: Verify
  run: dw --version
```

## Windows (PowerShell installer)

```powershell
irm https://durable-workflow.com/install.ps1 | iex
dw --version
```

The installer writes `dw.exe` to `%USERPROFILE%\.durable-workflow\bin` and adds
that directory to the user `PATH`.

## PHAR (portable, requires PHP 8.2+)

```bash
VERSION=<release-tag>
curl -fsSL -o dw.phar \
  "https://github.com/durable-workflow/cli/releases/download/${VERSION}/dw.phar"
curl -fsSL -o SHA256SUMS \
  "https://github.com/durable-workflow/cli/releases/download/${VERSION}/SHA256SUMS"
sha256sum --check --ignore-missing SHA256SUMS
chmod +x dw.phar
./dw.phar --version
```

The PHAR works wherever PHP 8.2 or newer is already available and is the
recommended artifact for shared CI runners that already ship PHP.

### Update An Installed Binary

For standalone release installs (the shell and PowerShell installers above),
`dw upgrade` replaces the running binary with the latest published release
after verifying the asset against the release's `SHA256SUMS`.

```bash
dw upgrade                 # upgrade through the stable release channel
dw upgrade --tag=<release-tag>     # pin to a specific release tag
dw upgrade --dry-run       # resolve the target release without downloading
```

`dw upgrade` refuses to rewrite Composer vendor, Homebrew cellar, and PHAR
installs because those paths are owned by another tool. Reinstall a pinned
public release with the installer, update the owning package manager, or use
`brew upgrade durable-workflow/tap/dw` for tap-managed Homebrew installs. See the
[CLI reference](./cli-reference.md#self-upgrade) for the full stable option
and status-field contract.

## Configure

Point the CLI at your server:

```bash
export DURABLE_WORKFLOW_SERVER_URL=http://localhost:8080
export DURABLE_WORKFLOW_AUTH_TOKEN=your-token
export DURABLE_WORKFLOW_NAMESPACE=default
```

Or pass them per-command:

```bash
dw --server=http://localhost:8080 --token=your-token workflow:list
```

## Five-Minute Operator Quickstart

This path is for checking the CLI against a real standalone server without
writing application code. It starts the published local server stack, installs
a pinned `dw`, creates a reusable profile, starts one workflow, and watches the
run reach the worker queue.

```bash
export DW_SERVER_IMAGE=durableworkflow/server:2.0.0
export DW_AUTH_TOKEN=dev-token

docker volume create durable-workflow-cli-quickstart

docker run --rm \
  -v durable-workflow-cli-quickstart:/app/database \
  -e DW_AUTH_DRIVER=token \
  -e DW_AUTH_TOKEN="$DW_AUTH_TOKEN" \
  "$DW_SERVER_IMAGE" server-bootstrap

docker rm -f durable-workflow-server >/dev/null 2>&1 || true
docker run -d --name durable-workflow-server \
  -p 8080:8080 \
  -v durable-workflow-cli-quickstart:/app/database \
  -e DW_AUTH_DRIVER=token \
  -e DW_AUTH_TOKEN="$DW_AUTH_TOKEN" \
  "$DW_SERVER_IMAGE"

until curl -sf http://localhost:8080/api/ready >/dev/null; do sleep 1; done
```

Install the CLI in another terminal:

```bash
curl -fsSL https://durable-workflow.com/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"
dw --version
```

Save the server connection once:

```bash
dw env:set local \
  --server=http://localhost:8080 \
  --token=dev-token \
  --namespace=default \
  --make-default

dw doctor
dw server:health
```

Start a workflow and inspect the run:

```bash
dw workflow:start \
  --type=quickstart.order \
  --workflow-id=quickstart-order-001 \
  --task-queue=quickstart \
  --input='{"order_id":"order-001","total":42.50}' \
  --json

dw workflow:describe quickstart-order-001 --output=json
dw workflow:history quickstart-order-001 <run-id-from-start-or-describe>
dw task-queue:describe quickstart
```

The run is now durable server state. Until a worker for `quickstart.order`
polls the `quickstart` queue, `dw task-queue:describe quickstart` is the fastest
way to see why the run is waiting. When a worker is attached, use
`dw watch workflow quickstart-order-001` to follow the run to a terminal state.
For an end-to-end run that attaches a published Python SDK worker and reaches
`status=completed`, follow the
[Durable Workflow 2.0 quickstart](/docs/quickstart).

## Commands

### Server

```bash
dw server:health          # Check server health
dw server:info            # Show server version, role topology, protocols, and worker fleet
```

`dw server:info` mirrors `GET /api/cluster/info`. In addition to build,
protocol, and worker-fleet facts, it prints the server role-topology manifest:
supported shapes, the current shape/process class/roles, matching-role wake
and partition details, current write boundaries, scaling boundaries, and
failure domains. Use `--output=json` when automation needs the raw
`topology.*` fields, or read
[Server Role Topology](./server-role-topology.md) for the field-by-field
contract behind that output.

### Workflows

```bash
dw workflow:list                                          # List workflows
dw workflow:start --type=MyWorkflow --input='["arg1"]'    # Start a workflow
dw workflow:start --type=MyWorkflow --input-file=input.json
dw workflow:describe <workflow-id>                         # Describe a workflow
dw workflow:signal <workflow-id> <signal-name> --input='["ok"]'
dw workflow:query <workflow-id> <query-name>               # Run a query
dw workflow:cancel <workflow-id>                           # Request cancellation
dw workflow:terminate <workflow-id>                        # Force terminate
dw workflow:history <workflow-id> <run-id>                 # Show run history
```

Every command that accepts caller payloads uses the same input shape:
`--input` for inline values, `--input-file` for a file path or `-` for stdin,
and `--input-encoding=json|raw|base64`. JSON is the default; raw and base64
inputs are passed as one positional workflow argument.

### Bridge Adapters

```bash
dw bridge:webhook stripe \
  --action=start_workflow \
  --idempotency-key=stripe-event-1001 \
  --target='{"workflow_type":"orders.fulfillment","task_queue":"external-workflows"}' \
  --input='{"order_id":"order-1001"}'

dw bridge:webhook pagerduty \
  --action=signal_workflow \
  --idempotency-key=pd-event-3003 \
  --target='{"workflow_id":"wf-remediation-42","signal_name":"incident_escalated"}' \
  --input='{"severity":"critical"}' \
  --json
```

Bridge adapters are bounded ingress tools for integration events. They return
named bridge outcomes for accepted, duplicate, and rejected deliveries; they do
not become workflow runtimes.

### Schedules

```bash
dw schedule:list                          # List schedules
dw schedule:create --workflow-type=MyWorkflow \
  --cron='0 * * * *'                      # Create hourly schedule
dw schedule:update <schedule-id> --input-file=input.json
dw schedule:describe <schedule-id>        # Describe a schedule
dw schedule:pause <schedule-id>           # Pause a schedule
dw schedule:resume <schedule-id>          # Resume a schedule
dw schedule:trigger <schedule-id>         # Trigger immediately
dw schedule:delete <schedule-id>          # Delete a schedule
```

### Activities

```bash
dw activity:complete <task-id> <attempt-id> --input='{"ok":true}'
dw activity:complete <task-id> <attempt-id> --input-file=result.json
dw activity:fail <task-id> <attempt-id> --message='upstream failed'
```

### Workers and Task Queues

```bash
dw worker:list                    # List registered workers
dw worker:describe <worker-id>    # Describe a worker
dw task-queue:list                # List active task queues
dw task-queue:describe <queue>    # Describe a task queue
```

`dw task-queue:list` is the compact fleet view. It shows admission status
columns for workflow, activity, and query tasks. `dw task-queue:describe`
expands the same server payload with pollers, current leases, queue,
namespace, and downstream budget-group dispatch capacity, remaining capacity,
budget source, and query-task pending capacity.

For scripts, `dw task-queue:describe --json` exposes queue-local backlog,
lease, poller, and admission state. Use it to distinguish missing workers from
admission throttling or a single queue that is building backlog:

```bash
dw task-queue:describe orders --json | jq '.stats | {
  approximate_backlog_count,
  approximate_backlog_age_seconds,
  workflow_tasks,
  activity_tasks
}'
```

Fleet-level durable inflow versus dispatch rates do not come from the
task-queue JSON contract. Read them from `dw system:operator-metrics --json`
instead:

```bash
dw system:operator-metrics --json | jq '.operator_metrics.backlog | {
  tasks_added_last_minute,
  tasks_dispatched_last_minute
}'
```

See [Task Queue Admission](/docs/polyglot/task-queue-admission) for the
operator tuning contract behind those fields.

To inspect the matching-role routing contract on the node you queried, read the
same payload's `matching_role` block:

```bash
dw system:operator-metrics --json | jq '.operator_metrics.matching_role | {
  queue_wake_enabled,
  shape,
  task_dispatch_mode,
  partition_primitives,
  backpressure_model
}'
```

`partition_primitives` freezes the routing axes (`connection`, `queue`,
`compatibility`, `namespace`) and `backpressure_model` tells you whether the
engine is relying on lease occupancy or some other admission boundary. Current
v2 reports `lease_ownership`.

### Namespaces

```bash
dw namespace:list                             # List namespaces
dw namespace:create --name=production         # Create a namespace
dw namespace:describe <namespace>             # Describe a namespace
```

### Search Attributes

```bash
dw search-attribute:list                                  # List search attributes
dw search-attribute:create --name=env --type=keyword      # Register an attribute
```

## Exit Codes

The CLI uses a stable exit-code policy so scripts and CI pipelines can react
to specific failure modes without parsing stderr. Values follow Symfony
Console's canonical `0`/`1`/`2` for success / failure / usage, and extend
from there.

| Code | Name | Meaning |
|------|------|---------|
| `0` | `SUCCESS` | Operation completed successfully. |
| `1` | `FAILURE` | Generic failure — command ran but did not succeed. |
| `2` | `INVALID` | Invalid usage — bad arguments, unknown options, or local validation. Also returned for HTTP 4xx responses that are not covered below (e.g. 400, 422). |
| `3` | `NETWORK` | Could not reach the server (connection refused, DNS failure, TLS handshake failure, transport error). |
| `4` | `AUTH` | Authentication or authorization failure. Returned for HTTP `401` and `403`. |
| `5` | `NOT_FOUND` | Resource not found. Returned for HTTP `404`. |
| `6` | `SERVER` | Server error. Returned for HTTP `5xx`. |
| `7` | `TIMEOUT` | Request timed out before the server responded. Also returned for HTTP `408`. |

Example:

```bash
dw workflow:describe chk-does-not-exist
echo $?  # 5 (NOT_FOUND)

dw server:health --server=http://unreachable:9999
echo $?  # 3 (NETWORK)
```

The canonical source is [`DurableWorkflow\Cli\Support\ExitCode`](https://github.com/durable-workflow/cli/blob/main/src/Support/ExitCode.php) in the CLI repository.

## Reference

See the [CLI command reference](/docs/polyglot/cli-reference) for command
shapes, options, output modes, and automation failure behavior.

<!-- Source: docs/polyglot/cloud-control-plane.md -->

# Cloud Managed Runtime

Durable Workflow Cloud is a managed orchestration service. Cloud operates both
the hosted control plane and the orchestration runtime, including workflow
state, history, schedules, task queues, leases, and durable visibility.
Customers run SDK clients and workers against a provisioned Cloud namespace.

This is separate from self-hosting. A self-hosted Durable Workflow Server runs
independently and is never attached to Cloud.

## Plans And Pricing

Pay for provisioned capacity, not workflow semantics. Each namespace receives
an isolated managed runtime; customer PHP, Python, and Rust workers run in your
own environment. All five plans are available through Cloud.

| Plan | Runtime capacity | Included durable storage | Availability | Price (USD) |
| --- | --- | --- | --- | --- |
| Cloud Dev | 1 vCPU, 1 GB RAM | 5 GB | Single host, no SLA | $0.03/hour, capped at $20/month |
| Cloud Standard | 1 vCPU, 2 GB RAM | 25 GB | Single-region HA, 99.99% SLA | $100/month |
| Cloud Multi-Region | 1 vCPU, 2 GB RAM | 25 GB | Multi-region HA, 99.99% SLA | $150/month |
| Cloud Business | 4 vCPU, 8 GB RAM | 100 GB | Single-region HA, 99.99% SLA | $500/month |
| Cloud Business Multi-Region | 4 vCPU, 8 GB RAM | 100 GB | Multi-region HA, 99.99% SLA | $650/month |

Capacity covers the managed runtime components, not customer worker compute.
HA replication and standby capacity are included in the plan price; the table
does not add replicas together as extra workflow-execution capacity. Every plan
includes Managed Waterline, basic encrypted backups, upgrades, and a stable
runtime URL.

Cloud Dev is metered by minute, with a $1 active-month minimum and a $20 cap
per space per UTC calendar month. SLA plans have a fixed monthly runtime price,
with applicable plan-change prorations handled by Stripe. Additional durable
storage is $2/GB-month, metered by GB-hour and separate from runtime charges or
the Dev cap. Prices exclude applicable taxes. Storage expansion requires
available capacity; neither disk growth nor network use is unlimited.

See [Cloud pricing](https://cloud.durable-workflow.com/pricing) to choose a plan.
For larger capacity, different connectivity, SSO, enterprise support, or a
custom availability requirement, [contact us](https://cloud.durable-workflow.com/contact).

## Managed Service Boundary

The customer-visible boundary is one Cloud namespace:

```text
Cloud organization
  project
    environment
      namespace
        stable runtime URL
        client runtime credential  ---> workflow starts and commands
        worker runtime credential  ---> registration, polling, and completion

Cloud-operated runtime
  workflow state and history
  schedules and task queues
  leases, matching, and visibility
  Managed Waterline
```

Cloud owns namespace provisioning, runtime operation, persistence, placement,
runtime health, recovery, and the Managed Waterline surface for the namespace.
Customers own application code, workflow and activity implementations, and the
processes that run their workers. Cloud customers do not deploy a separate
Waterline service.

The namespace's HTTPS endpoint connects directly to its managed runtime.
SDK traffic is not proxied through the Cloud website/control-plane application.
Use the returned runtime URL unchanged, without appending `/api`; the SDK and
CLI construct their API paths. Administrative API calls still use
`https://cloud.durable-workflow.com/api/v1`.

Cloud administration and runtime traffic use different credentials:

- A Cloud API key (`dwc_...`) manages projects, environments, namespaces,
  billing, and runtime-credential lifecycle.
- A client runtime credential (`dwr_...`) starts and controls workflows in one
  managed namespace.
- A worker runtime credential (`dwr_...`) registers workers, long-polls for
  tasks, sends heartbeats, and settles work in that namespace.

A Cloud API key is not accepted by the namespace runtime URL. Runtime
credentials are scoped to one namespace and role, returned only when created,
and omitted from later list and audit responses.

If Cloud onboarding uses the CLI, install it from the [CLI guide](./cli.mdx)
and update an existing standalone installation explicitly with `dw upgrade`.
The CLI never updates in the background. After installing or upgrading, run
`command -v dw` and `dw --version`; resolve any installer `PATH` remediation
before using Cloud credentials so the selected release is the active binary.

## Provision And Connect A Namespace

### 1. Create and provision the namespace

In the [Cloud dashboard](https://cloud.durable-workflow.com/), create an
organization, project, environment, and namespace, then select its capacity
plan. Creating an account or namespace definition does not provision paid
capacity. Complete payment setup through Stripe, then select **Provision** on
the namespace page. Adding a card alone does not start provisioning.

Provisioning is asynchronous. The namespace page updates its progress while
Cloud prepares the runtime. Wait for it to become active before starting SDK
work. The page also provides runtime-credential creation and Managed Waterline
access for inspecting workflows and their history.

For API-driven setup, complete payment setup first. The following example uses
the organization's default plan; `capacity_plan_version` can select another
available plan. Do not supply your own Server URL, deployment identifier, or
placement record:

```bash
curl -X POST \
  https://cloud.durable-workflow.com/api/v1/projects/PROJECT/environments/ENVIRONMENT/namespaces \
  -H "Authorization: Bearer dwc_..." \
  -H "Content-Type: application/json" \
  -d '{"name":"orders","retention_days":30}'

curl -X POST \
  https://cloud.durable-workflow.com/api/v1/projects/PROJECT/environments/ENVIRONMENT/namespaces/orders/provision \
  -H "Authorization: Bearer dwc_..."
```

After provisioning completes, the namespace response provides its stable
`runtime_url`, its `runtime_namespace`, managed status, and customer-visible
region information. Treat the returned URL and namespace value as configuration
owned by Cloud; do not derive an endpoint or replace it with a self-hosted
Server address.

### 2. Issue separate client and worker credentials

Issue the two runtime roles independently:

```bash
curl -X POST \
  https://cloud.durable-workflow.com/api/v1/projects/PROJECT/environments/ENVIRONMENT/namespaces/orders/runtime-credentials \
  -H "Authorization: Bearer dwc_..." \
  -H "Content-Type: application/json" \
  -d '{"name":"orders-client","role":"client"}'

curl -X POST \
  https://cloud.durable-workflow.com/api/v1/projects/PROJECT/environments/ENVIRONMENT/namespaces/orders/runtime-credentials \
  -H "Authorization: Bearer dwc_..." \
  -H "Content-Type: application/json" \
  -d '{"name":"orders-worker","role":"worker"}'
```

Each token is displayed once in its create response. Store it in the secret
store used by only the corresponding role. A deliberately combined client and
worker process receives both values as two distinct secrets. Rotate and revoke
the roles independently, and never substitute a Cloud API key for either
runtime credential.

### 3. Complete a first Cloud workflow {#cloud-first-workflow}

The Sample App's shared external-runtime playground runs the same authored
workflow-and-activity journey with PHP, Python, or Rust. Open a [Sample App
Codespace](https://codespaces.new/durable-workflow/sample-app?quickstart=1&ref=main),
export the provisioned namespace values and two runtime credentials, and choose
an application task queue:

```bash
export DURABLE_WORKFLOW_RUNTIME_URL='<provisioned-runtime-url>'
export DURABLE_WORKFLOW_NAMESPACE='<provisioned-runtime-namespace>'
export DURABLE_WORKFLOW_CLIENT_TOKEN='<client-runtime-credential>'
export DURABLE_WORKFLOW_WORKER_TOKEN='<worker-runtime-credential>'
export DURABLE_WORKFLOW_TASK_QUEUE='<language-task-queue>'
```

Then choose a first-party SDK and run the same managed-runtime contract:

```bash
language=php # Choose php, python, or rust.
scripts/playground "$language" --runtime managed \
  --runtime-url "$DURABLE_WORKFLOW_RUNTIME_URL" \
  --namespace "$DURABLE_WORKFLOW_NAMESPACE" \
  --task-queue "$DURABLE_WORKFLOW_TASK_QUEUE"
```

The runner resolves the current stable artifact versions, gives each
credential only to its matching child process, waits up to 60 seconds for a
worker registration advertising the exact queue and generated workflow and
activity types, then starts one client request. It succeeds only after the SDK
returns the expected result, `dw` reports `completed`, and the required
workflow and activity history is present. The command does not run a local
Server or Waterline in managed mode.

Continue with the language guide for SDK-specific authoring:

- [PHP SDK](/docs/polyglot/php/): run `scripts/playground php`.
- [Python SDK](/docs/polyglot/python/): run `scripts/playground python`.
- [Rust managed-runtime quickstart](/docs/polyglot/rust-cloud-quickstart/):
  run `scripts/playground rust` and use its worker-ready, completed-result, and
  mismatch diagnostics.

A combined client-and-worker journey holds both runtime credentials as distinct
secrets. A Cloud administration key is not a runtime credential and must not be
exported under either runtime-token variable.

### 4. Continue with managed operation

The SDK client starts workflows and sends follow-up commands through the
namespace runtime URL. Customer-run workers register and long-poll through the
same URL using the worker role. Cloud authenticates and scopes each request,
executes the orchestration protocol in the managed runtime, and persists the
workflow state and history.

The customer application does not select a runtime deployment for an
operation. Workflow IDs, run IDs, task queues, compatibility markers, and Avro
payload encoding remain durable application contracts within the Cloud namespace.

## Customer-Run Worker Connectivity

Workers can run in your network, VM fleet, container platform, or application
environment. They need outbound HTTPS reachability to the namespace runtime URL
and must allow the worker protocol's long-lived poll requests.

- No inbound connection from Cloud to a worker is required.
- Proxies and egress gateways must not shorten long polls into a busy retry
  loop.
- Workers should retry transient connection failures and service-unavailable
  responses with bounded backoff.
- Moving a worker process does not move workflow state; Cloud retains the
  namespace's durable state and history.
- Credential rotation does not require changing the runtime URL, namespace, or
  task queue.

## Region Placement And Recovery Boundary

Choose the recovery boundary with the plan:

- **Cloud Dev:** one isolated host with persistent state and backups. Maintenance
  and recovery may interrupt service. There is no uptime SLA or automatic
  regional failover.
- **Single-region HA:** three replicated hosts in one region, with automatic
  primary failover. One host may fail without losing the remaining quorum.
  A whole-region outage is outside this plan's SLA coverage.
- **Multi-region HA:** three replicated hosts across three regions, with
  automatic primary failover. The SLA includes loss or isolation of one
  configured region, provided the remaining members can form a quorum.

The stable runtime URL follows the elected primary; you do not change SDK
configuration during a supported failover. An isolated former primary is
prevented from continuing to serve authoritative work. If a safe primary
cannot be established, the runtime stops serving rather than accepting
conflicting writes. HA is not a promise of uninterrupted requests: clients and
workers still need bounded retries for transient failures.

The namespace page exposes its topology, region information, and service
status. Backups support recovery but are not a substitute for live replication.
Short failover tests demonstrate the exercised failure cases, not a universal
recovery-time guarantee or a month's achieved availability.

### SLA Measurement And Credits

The four SLA plans provide a 99.99% uptime SLA over a UTC calendar month,
measured at the customer runtime endpoint in one-minute windows. Missing
measurements count as unavailable, and planned maintenance is not excluded.
Customer-hosted worker availability is separate from managed runtime availability.

| Monthly availability | Automatic account credit |
| --- | ---: |
| At least 99.99% | None |
| At least 99.9%, below 99.99% | 10% |
| At least 99.0%, below 99.9% | 25% |
| Below 99.0% | 100% |

Account credits apply automatically; cash refunds require review and approval.
Cloud Dev has no SLA credits. See [Cloud pricing](https://cloud.durable-workflow.com/pricing)
for the plan's availability scope and billing terms.

## Private Connectivity And Support Boundary

The 2.0 self-serve Cloud contract assumes outbound access from clients and
workers to the public namespace runtime URL. Private-only ingress, bespoke VPN
or peering arrangements, and provider-specific private routing are support-led
connectivity designs, not hidden defaults. Customers are never given internal
runtime addresses or asked to route around the namespace URL.

## Cloud Or Self-Hosted Server

Choose Cloud when Durable Workflow should operate the orchestration runtime,
persistence, the selected plan's availability, recovery, and Managed Waterline while
your team operates the SDK clients and workers.

Choose [self-hosted Server](/docs/polyglot/server) when your team needs to
operate the Server image, database, cache, networking, authentication, backups,
and failover independently. A self-hosted Server cannot be registered with,
attached to, or used as the backing runtime for a Cloud namespace. Embedded
Laravel, self-hosted Server, and Cloud are separate deployment choices.

## Standard Workflow Benchmarks

[DW Standard Workflow v1](https://github.com/durable-workflow/server/tree/main/benchmarks/capacity)
defines a small, repeatable workload: one workflow start, one external activity,
and one workflow completion, with defined 1 KiB Avro inputs and results.
The customer worker runs outside the managed runtime allocation.

The recorded plan baselines below use that workload. The SLA-plan measurements
include their replicated HA topology; they are not extrapolated from a Dev host.

| Plan | Standard workflows/second | 30-day workflow actions |
| --- | ---: | ---: |
| Cloud Dev | 0.25 | 1,296,000 |
| Cloud Standard | 0.25 | 1,296,000 |
| Cloud Multi-Region | 0.10 | 518,400 |
| Cloud Business | 0.50 | 2,592,000 |
| Cloud Business Multi-Region | 0.20 | 1,036,800 |

The 30-day estimate is `workflows/second x 2,592,000 seconds x 2 workflow actions`:
one start and one activity for this comparison. It assumes that rate runs
continuously for 30 days. It is not an included-action quota, a billing unit,
or an exact mapping to another engine's action definitions.

These are measured workload baselines, not maximum throughput or guaranteed
capacity for every application. Larger payloads, more activities, timers,
signals, queries, replay-heavy histories, cross-region communication, and
customer worker latency change the result. The table does not claim separate
timer, signal, replay, or saturation benchmarks. Plan for your actual workflow
mix rather than multiplying these numbers by an arbitrary workflow size.

### Cloud Dev Measurement {#cloud-dev-capacity}

Cloud Dev is an isolated, single-host managed runtime for development and
evaluation. Each provisioned space receives the same runtime shape used for
the measurement below:

| Resource | Cloud Dev |
| --- | --- |
| Runtime compute | 1 shared vCPU, 1 GB RAM |
| Durable storage included | 5 GB |
| Runtime services | Server with Managed Waterline, queue worker, scheduler, MySQL, and Redis |
| Network path | Direct, space-specific HTTPS ingress |
| Customer workers | Run in the customer's environment |
| Availability | No SLA; maintenance interruptions are allowed |
| Runtime price | $0.03/hour, measured by minute, capped at $20/month |
| Storage above 5 GB | $2/GB-month |

Cloud Dev was measured with [DW Standard Workflow
v1](https://github.com/durable-workflow/server/tree/main/benchmarks/capacity),
a fixed comparison workload consisting of one workflow start, one external
activity, and one workflow completion with defined 1 KiB Avro inputs and
results. The test used the provisioned 1-vCPU/1-GB runtime topology, published
Server and PHP SDK artifacts, one PHP worker process, two client slots,
a 30-second warmup, and a five-minute measurement window.

| Measured result | Value |
| --- | ---: |
| Offered and completed rate | 0.25 standard workflows/second |
| Completed workflows | 75 of 75 |
| Errors / throttled starts | 0 / 0 |
| Scheduling latency, p50 | 28.0 ms |
| Scheduling latency, p95 | 95.1 ms |
| Scheduling latency, p99 | 134.5 ms |
| Final workflow backlog | 0 |
| 30-day workflow actions | 1,296,000 |

This is a measured development baseline, not a universal conversion for every
workflow and not an SLA. Larger payloads, additional activities, timers,
signals, queries, replay-heavy histories, and customer worker latency change
capacity. Cloud billing remains based on provisioned runtime time and durable
storage, not workflow operations.

## Billing Usage API

Cloud Dev uses one isolated, single-host runtime cell for each provisioned Dev
space. Its billing terms are:

| Billing term | Cloud Dev |
| --- | ---: |
| Provisioned runtime | $0.03 per hour, metered by minute |
| Calendar-month maximum | $20 per Dev space |
| Active-month minimum | $1 when a Dev space is provisioned during the month |
| Managed capacity | 1 vCPU, 1 GB memory, 5 GB durable storage |
| Additional durable storage | $2 per GB-month, metered by GB-hour |
| Basic encrypted backups | Included |
| Availability | Single host, no SLA |

The allocation covers Cloud-operated Server, MySQL, Redis, scheduling,
Waterline access, backups, upgrades, and infrastructure. Customer PHP, Python,
and Rust workers run outside that allocation. Workflow starts, activity
attempts, retries, timers, signals, queries, updates, and child workflows are
operational telemetry, not separate billing units. Prices exclude applicable
taxes. Additional durable storage is outside the $20 runtime-capacity maximum.

Cloud exposes organization-scoped billing usage for finance, operations, and
chargeback automation. The endpoint is authenticated by a Cloud API key and
does not accept a customer or organization id in the request; the caller's
organization is resolved from the `dwc_` bearer token so one customer cannot
query another customer's usage.

```http
GET /api/v1/billing/usage?period_start=2026-05-01&period_end=2026-05-31
Authorization: Bearer dwc_...
Accept: application/json
```

`period_start` and `period_end` are optional ISO-8601 dates. When omitted,
Cloud returns the current calendar month. Billing usage reads and exports stay
available even if billing restrictions pause namespace provisioning or workflow
operations, so finance teams can still recover account standing.

The response schema is
`durable_workflow.cloud.namespace_capacity_usage.v1`. It separates allocated
capacity time and additional durable storage from semantic event counters. The
abbreviated Cloud Dev response below shows that distinction.

```json
{
  "schema": "durable_workflow.cloud.namespace_capacity_usage.v1",
  "access_control": {
    "scope": "organization_billing_usage",
    "read_allowed": true,
    "export_allowed": true
  },
  "current_period": {
    "starts_at": "2026-05-01T00:00:00+00:00",
    "ends_at": "2026-05-31T23:59:59+00:00"
  },
  "metering_policy": {
    "invoice_drivers": [
      "namespace_plan_capacity",
      "additional_durable_storage_gb_month"
    ],
    "semantic_events_are_invoice_units": false,
    "network": "measured_not_billable",
    "basic_backups": "included",
    "customer_worker_compute": "excluded"
  },
  "by_namespace": [
    {
      "namespace": "development",
      "project": "sample-app",
      "environment": "development",
      "plan": {
        "version": "cloud-dev.single-host-v1",
        "name": "Cloud Dev",
        "availability_class": "development_single_host",
        "sla_status": "none",
        "billing_terms": {
          "currency": "usd",
          "unit": "provisioned_runtime_hour",
          "hourly_rate_cents": 3,
          "monthly_cap_cents": 2000,
          "active_month_minimum_cents": 100,
          "billing_period": "calendar_month_utc",
          "metering_resolution_seconds": 60,
          "additional_storage_unit": "gb_month",
          "additional_storage_rate_cents": 200,
          "additional_storage_metering_unit": "gb_hour",
          "additional_storage_in_monthly_cap": false
        }
      },
      "allocation": {
        "managed_cpu_vcpu": 1,
        "managed_memory_gb": 1,
        "included_durable_storage_gb": 5
      },
      "operational_telemetry": {
        "billing_status": "not_billable",
        "counters": {
          "workflow_execution_count": 20,
          "activity_execution_count": 40,
          "timer_fire_count": 5,
          "signal_delivery_count": 3,
          "update_delivery_count": 0,
          "query_task_count": 2
        }
      }
    }
  ]
}
```

Cloud Dev's time meter starts when its isolated runtime is activated and
stops when that runtime is deprovisioned. The monthly cap and minimum apply per
Dev space in UTC calendar months. Durable storage above the included amount is
metered separately. Cloud preserves an operating and recovery reserve on the
runtime disk and requires a capacity change before storage can consume it.

Idle runtimes still incur capacity charges. Deprovisioning stops runtime
capacity billing and removes active runtime data and credentials; it is not a
pause/resume operation. SLA plans use their selected monthly subscription price,
not the Dev hourly rate. Use the plan's returned `billing_terms` when interpreting
usage, and keep any separately retained billable storage distinct from runtime
capacity.

Export the same evidence as CSV or a JSON report when a downstream finance
system needs a file handoff:

```bash
curl -OJ "https://cloud.durable-workflow.com/api/v1/billing/usage/export?period_start=2026-05-01&period_end=2026-05-31" \
  -H "Authorization: Bearer dwc_..."

curl -OJ "https://cloud.durable-workflow.com/api/v1/billing/usage/report?period_start=2026-05-01&period_end=2026-05-31" \
  -H "Authorization: Bearer dwc_..."
```

For a JSON-backed dashboard panel, request the same API with the panel's time
range:

```text
GET https://cloud.durable-workflow.com/api/v1/billing/usage?period_start=${__from:date:YYYY-MM-DD}&period_end=${__to:date:YYYY-MM-DD}
Authorization: Bearer dwc_...
```

Then flatten namespace capacity rows with:

```jq
.by_namespace[]
| {
    project,
    environment,
    namespace,
    plan: .plan.name,
    cpu_vcpu: .allocation.managed_cpu_vcpu,
    memory_gb: .allocation.managed_memory_gb,
    included_storage_gb: .allocation.included_durable_storage_gb,
    capacity_status
  }
```

Use `invoice_units` for capacity and storage reconciliation. Use
`operational_telemetry` to understand workload shape and benchmark behavior;
do not convert those event counters into charges.

## Related References

- [Deployment Modes](/docs/polyglot/deployment-modes)
- [PHP SDK](/docs/polyglot/php)
- [Python SDK](/docs/polyglot/python)
- [Rust SDK](/docs/polyglot/rust)
- [Server](/docs/polyglot/server)
- [Self-Hosting Deployments](/docs/deployment)
- [Support](/docs/support)

<!-- Source: docs/polyglot/external-execution.md -->

# External Execution Surface

Durable Workflow v2 treats external execution as a contract-first product
surface, not as a shell convention or a single worker transport. The standalone
server publishes the machine-readable umbrella at `GET /api/cluster/info` under
`worker_protocol.external_execution_surface_contract`.

That contract is named `activity_grade_external_execution`. It is for durable,
bounded work that can run outside a full workflow runtime:

- operator maintenance activities
- platform automation
- integration handoffs
- bridge adapters that start, signal, update, or hand off work
- script, daemon, HTTP, queue-backed, serverless, or agent-driven handlers

The primary wedge is operator, platform, and integration automation. AI agents
and scripts are important consumers because they need stable handles, but they
do not redefine the runtime boundary.

## Boundary

External handlers may:

- execute one leased workflow or activity task
- heartbeat lease progress through the worker protocol
- return a structured success or failure envelope
- use a bridge adapter to start, signal, update, or hand off bounded work

External handlers must not:

- interpret workflow replay semantics
- own `ContinueAsNew` behavior
- apply signal, update, or query ordering rules outside the runtime contract
- mutate event history directly
- act as an unbounded workflow runtime

Those rules stay inside the server and real SDK runtimes. A carrier should only
move declared input and result envelopes across a transport boundary.

## Published Contract Seams

Read `GET /api/cluster/info` before wiring a carrier. The relevant v2 paths are:

| Path | Purpose |
| --- | --- |
| `worker_protocol.external_execution_surface_contract` | Product boundary, runtime boundary, valid carrier classes, and seam status. |
| `worker_protocol.external_task_input_contract` | Carrier-neutral input envelope for one leased workflow or activity task. |
| `worker_protocol.external_task_result_contract` | Carrier-neutral success, failure, and malformed-output envelope. |
| `worker_protocol.server_capabilities.external_execution_surface` | Compact capability pointer for worker-plane negotiation. |

The external execution surface also names planned seams for deterministic
auth/profile/TLS composition, config-first handler mappings, bounded bridge
adapters, payload external storage, and admission/rollout safety. Treat those
as contract seams, not private implementation details.

## Carrier Requirements

A valid carrier can be a poll-based CLI or daemon, an HTTP handler invocation,
a queue-backed worker, or a serverless invocation. The transport can differ,
but these requirements do not:

- emit the declared input schema
- accept the declared result schema
- preserve `task.id`, `task.attempt`, and `task.idempotency_key`
- map transport failures to a structured failure or malformed-output outcome
- resolve auth, TLS, profile, and environment inputs deterministically

Exit codes, stderr, process crashes, HTTP failures, queue visibility timeouts,
and serverless platform errors are transport facts. They only become workflow
facts after the carrier maps them to the declared result envelope or to
`malformed_output`.

The first concrete carrier under this contract is the
[invocable HTTP carrier](./invocable-carrier.md), published at
`worker_protocol.invocable_carrier_contract`. It is activity-task only,
requires HTTPS (with loopback HTTP allowed only for development), and resolves
auth through the external executor configuration. Its transport-level
`retry_policy` is distinct from the durable activity retry policy, which
remains the server/runtime authority once a result is reported.

## Bridge Adapters

Bridge adapters are bounded ingress or handoff surfaces. They can start,
signal, update, or hand off work, but they are not workflow runtimes and should
not hide replay or event-history behavior.

Every bridge adapter needs explicit machine outcomes for:

- unknown target
- auth failure
- malformed payload
- duplicate start
- unsupported routing
- accepted handoff
- rejected handoff

Use the same operator clarity as the rest of the control plane: stable status
codes, named `reason` values, and enough context for `dw`, Waterline, SDKs, and
agents to explain what happened without scraping prose.

The CLI exposes the webhook bridge surface directly:

```bash
dw bridge:webhook stripe \
  --action=start_workflow \
  --idempotency-key=stripe-event-1001 \
  --target='{"workflow_type":"orders.fulfillment","task_queue":"external-workflows","business_key":"order-1001"}' \
  --input='{"order_id":"order-1001"}' \
  --json
```

The JSON response is the same bridge-adapter outcome contract published by
`/api/cluster/info`, including `outcome`, `reason`,
`control_plane_outcome`, `idempotency_key`, and the redacted `target` summary.

## Agent-Operable Shape

The external execution surface is part of the v2 AI-assisted development
posture. Humans learn the workflow/activity/replay invariant; tools operate
through stable contracts:

1. discover the external execution surface from `/api/cluster/info`
2. choose a carrier that satisfies the published requirements
3. emit an external task input envelope
4. preserve task identity and idempotency
5. return a success, failure, or malformed-output envelope
6. report bridge and transport failures with named outcomes

This keeps AI-assisted scaffolding and operator automation boring: the tool can
cite the protocol manifest, the task envelope, the result envelope, and the
bridge outcome instead of inferring behavior from one demo script.

<!-- Source: docs/polyglot/server-api-reference.md -->

# Server API Reference

The standalone server exposes a versioned HTTP+JSON API. Use this page when
building SDKs, scripts, bridge adapters, or operator runbooks that call the
server directly. Use the [server guide](/docs/polyglot/server) for
deployment and configuration, and the [CLI command reference](/docs/polyglot/cli-reference)
when shelling out to `dw`.

## Headers And Versioning

All authenticated requests use bearer tokens unless the server is configured
for another auth driver:

```http
Authorization: Bearer <token>
X-Namespace: default
Content-Type: application/json
Accept: application/json
```

Control-plane routes require:

```http
X-Durable-Workflow-Control-Plane-Version: 2
```

Worker-plane routes require:

```http
X-Durable-Workflow-Protocol-Version: 1.0
```

The server publishes supported versions and machine-readable contracts from
`GET /api/cluster/info`. Clients should discover versions there before
starting long-lived automation. Missing or unsupported control-plane versions
fail closed with a named reason such as `missing_control_plane_version` or
`unsupported_control_plane_version`.

For validation, code generation, and drift checks, use the normative
[Platform Protocol Specs](/docs/platform-protocol-specs) catalog instead
of this prose reference. It links the control-plane OpenAPI document, worker
protocol OpenAPI and AsyncAPI documents, the `cluster_info` JSON Schema, and
the adjacent MCP, history, Waterline, and repair/actionability schemas.

## Discovery And Health

These routes are used by load balancers, SDK bootstraps, and compatibility
checks.

| Method | Path | Auth | Purpose |
| --- | --- | --- | --- |
| `GET` | `/api/health` | no | Liveness probe plus a machine-readable topology summary for the responding node. |
| `GET` | `/api/ready` | no | Readiness probe plus the same topology summary and rollout-safety bootstrap checks. |
| `GET` | `/api/cluster/info` | yes | Server identity, supported SDK ranges, role topology, coordination-health summary, control-plane contract, worker-protocol contract, payload codecs, and feature capabilities. |

Example:

<!-- docs-example id="server.cluster-info.curl" -->
```bash
curl -sS "$DURABLE_WORKFLOW_SERVER_URL/api/cluster/info" \
  -H "Authorization: Bearer $DURABLE_WORKFLOW_AUTH_TOKEN" \
  -H "X-Namespace: default" | jq '.control_plane.version, .worker_protocol.version'
```

`/api/cluster/info` intentionally does not require the control-plane version
header because it is the endpoint that advertises the supported versions. The
same response also includes `coordination_health`, the all-namespaces rollout
and readiness summary that mirrors the checks already feeding the server's
readiness posture.

### Public Topology Summary

`GET /api/health` and `GET /api/ready` both return a top-level `topology`
object from the landed health-summary contract. Use it when you need to
identify which node answered a probe before control-plane auth, namespace
resolution, or broader `/api/cluster/info` discovery succeeds.

The public summary always includes:

- `topology.schema`
- `topology.version`
- `topology.current_shape`
- `topology.current_process_class`
- `topology.current_roles`
- `topology.execution_mode`
- `topology.matching_role.queue_wake_enabled`
- `topology.matching_role.shape`
- `topology.matching_role.wake_owner`
- `topology.matching_role.task_dispatch_mode`
- `topology.matching_role.partition_primitives`
- `topology.matching_role.backpressure_model`
- `topology.matching_role.discovery_limits.poll_batch_cap`
- `topology.matching_role.discovery_limits.availability_ceiling_seconds`
- `topology.matching_role.discovery_limits.wake_signal_ttl_seconds`
- `topology.matching_role.discovery_limits.workflow_task_lease_seconds`
- `topology.matching_role.discovery_limits.activity_task_lease_seconds`

`topology.matching_role.discovery_limits` is the frozen numeric matching-role
contract: `poll_batch_cap` is the maximum batch of ready-task rows returned per
poll, `availability_ceiling_seconds` is the cross-backend tolerance applied to
`available_at` so freshly-available tasks survive sub-second timestamp drift,
`wake_signal_ttl_seconds` is the default `CacheLongPollWakeStore` signal TTL,
and `workflow_task_lease_seconds` / `activity_task_lease_seconds` are the
default workflow and activity task lease durations. Tightening any of these
values is a protocol-level change because workers and downstream tooling read
them as the authoritative matching-role contract; renaming a field is also a
protocol-level break.

`/api/ready` returns the same `topology` block even when the top-level
readiness `status` is `not_ready`, so probes can still distinguish
`server_http_node`, `scheduler_node`, `matching_node`, and `execution_node`
responses while bootstrap blockers are active.

Example:

<!-- docs-example id="server.health.topology.curl" -->
```bash
curl -sS "$DURABLE_WORKFLOW_SERVER_URL/api/health" | jq '{
  status,
  topology: {
    schema: .topology.schema,
    version: .topology.version,
    current_shape: .topology.current_shape,
    current_process_class: .topology.current_process_class,
    current_roles: .topology.current_roles,
    execution_mode: .topology.execution_mode,
    matching_role: .topology.matching_role
  }
}'
```

### Readiness Blockers

`GET /api/ready` returns a top-level `status` plus machine-readable `checks`.
Two checks define whether the server can safely evaluate rollout-safety health:

- `checks.migrations` is the bootstrap and migration gate. It publishes
  `repository_exists`, `pending_migrations`, `adoptable_migrations`,
  `blocking_migrations`, `missing_tables`, `operator_surface`, and
  `readiness_contract`.
- `adoptable_migrations` means existing workflow tables only need migration
  history adoption. The server stays ready and reports `status: "warning"` so
  operators can schedule the adoption before the next migrate pass.
- `blocking_migrations` means rollout-safety migration records are still
  required. The server fails closed with `status: "pending"` and a
  `remediation` string instead of serving as if the fleet were current.
- `operator_surface.available` and `operator_surface.required_tables` tell you
  whether the v2 operator surface has the durable tables it needs to explain
  rollout safety after boot.
- `readiness_contract.version` pins the install and adoption contract revision
  that scripts should expect when they parse these readiness fields.
- `checks.workflow_v2` mirrors the all-namespaces rollout-safety verdict. When
  readiness prerequisites are missing it reports `status: "blocked"` and adds
  `blocked_by`, `message`, and `remediation` instead of pretending the fleet is
  healthy.

Example:

<!-- docs-example id="server.ready.blockers.curl" -->
```bash
curl -sS "$DURABLE_WORKFLOW_SERVER_URL/api/ready" | jq '{
  status,
  migrations: {
    status: .checks.migrations.status,
    adoptable_migrations: .checks.migrations.adoptable_migrations,
    blocking_migrations: (.checks.migrations.blocking_migrations | map(.migration)),
    missing_tables: .checks.migrations.missing_tables,
    operator_surface: .checks.migrations.operator_surface,
    readiness_contract: .checks.migrations.readiness_contract
  },
  workflow_v2: {
    status: .checks.workflow_v2.status,
    blocked_by: .checks.workflow_v2.blocked_by,
    remediation: .checks.workflow_v2.remediation
  }
}'
```

### Cluster Topology Manifest

`/api/cluster/info` also returns the node's `topology` manifest under the
schema `durable-workflow.v2.role-topology`. That manifest is the supported way
to discover whether the node is currently acting as `standalone_server`,
`embedded`, or `split_control_execution`, which roles it owns, and what the
server expects from `matching_role`, `shape_assignments`,
`authority_boundaries`, `failure_domains`, `scaling_boundaries`, and
`migration_path`. The same
response also publishes live rollout-safety state for that node.

Read the manifest as follows:

- `topology.current_shape`, `topology.current_process_class`,
  `topology.current_roles`, and `topology.execution_mode` tell you which role
  shape the node is actually serving. These fields describe the responding
  node, not the full fleet.
- `topology.role_vocabulary` is the fixed list of legal v2 role names.
- `topology.matching_role.queue_wake_enabled`,
  `topology.matching_role.shape`, `topology.matching_role.wake_owner`,
  `topology.matching_role.task_dispatch_mode`,
  `topology.matching_role.partition_primitives`, and
  `topology.matching_role.backpressure_model` tell you whether broad ready-task
  discovery is happening in-worker or through a dedicated matching-role sweep,
  which process owns that sweep, which routing axes stay stable, and which
  durable admission boundary v2 enforces today.
- `topology.matching_role.discovery_limits` publishes the frozen numeric
  matching-role contract values: `poll_batch_cap`,
  `availability_ceiling_seconds`, `wake_signal_ttl_seconds`,
  `workflow_task_lease_seconds`, and `activity_task_lease_seconds`. Use these
  to verify the deployment matches the documented matching-role contract; the
  package emits the same identifiers in `dw server:info`, the operator-metrics
  snapshot, and the namespace-scoped health surface.
- `topology.role_catalog` and `topology.authority_surfaces` map those role
  names to the interfaces, durable-write surfaces, and read paths automation
  should expect on the responding node.
- `topology.shape_assignments` is the machine-readable process-class inventory
  for each supported shape. Compare the current role bundle against that table
  when you need to map the responding node onto a documented process class.
- `topology.supported_topologies` summarizes which deployment families the
  product supports and which node classes each family expects.
- `coordination_health` summarizes fleet-wide rollout and compatibility risk in
  one machine-readable block. Besides `status` and `http_status`, it can publish
  `blocked_by`, `message`, and `remediation` when rollout-safety evaluation is
  blocked by upstream readiness issues such as missing migrations or database
  reachability.
- `coordination_health.checks[]` always includes the frozen
  `activity_path` check next to `worker_compatibility`, `task_transport`,
  `routing_health`, `durable_resume_paths`, and the projection/scheduler checks.
  `activity_path` is the activity-side counterpart of `task_transport`: it
  surfaces activity executions whose schedule-to-start, start-to-close,
  schedule-to-close, or heartbeat deadline has passed without enforcement
  (`timeout_overdue`, `oldest_timeout_overdue_at`,
  `max_timeout_overdue_age_ms`) and the sustained activity retry backlog
  (`retrying`, `oldest_retrying_started_at`, `max_retrying_age_ms`). Renaming
  the check is a protocol-level change.
- `coordination_health.routing_drains` summarizes draining build-id cohorts
  across namespaces and queues. Use `queues_with_drains` and the per-queue
  `build_ids` entries to see where traffic is intentionally being held away from
  draining workers.
- `execution_mode` distinguishes `local_queue_worker` embedded execution from
  `remote_worker_protocol` worker-protocol execution.
- `split_control_execution` is a supported product topology, not a second
  server product or a different API.

Example:

<!-- docs-example id="server.cluster-info.topology.curl" -->
```bash
curl -sS "$DURABLE_WORKFLOW_SERVER_URL/api/cluster/info" \
  -H "Authorization: Bearer $DURABLE_WORKFLOW_AUTH_TOKEN" \
  -H "X-Namespace: default" \
  | jq '{
    current_shape: .topology.current_shape,
    current_roles: .topology.current_roles,
    execution_mode: .topology.execution_mode,
    matching_role: .topology.matching_role,
    coordination_health: {
      status: .coordination_health.status,
      http_status: .coordination_health.http_status,
      blocked_by: .coordination_health.blocked_by,
      queues_with_drains: .coordination_health.routing_drains.queues_with_drains
    }
  }'
```

For the conceptual contract behind those fields, including the role vocabulary
and migration path, see
[Server Role Topology](/docs/polyglot/server-role-topology).

### Wrong-Node Topology Rejections

Authenticated hosted routes fail closed when the responding node does not host
the HTTP control surface required for that endpoint. The gate runs after role
and protocol-version validation but before namespace resolution, so wrong-node
requests do not leak namespace existence.

When the gate blocks a request, the server returns `503` with
`reason: "topology_role_unavailable"` plus:

- `current_shape`: the responding node's advertised topology shape.
- `current_process_class`: the responding node's declared process class, such
  as `scheduler_node` or `execution_node`.
- `current_roles`: the roles that node actually hosts.
- `required_roles`: the hosted route roles the endpoint needs.
- `missing_roles`: the subset of `required_roles` missing from the responding
  node.

Control-plane routes return that payload in the control-plane envelope. Worker
protocol routes return the same fields in the worker-protocol envelope and keep
the normal worker-protocol version header.

Example wrong-node response from `GET /api/workflows` when the request lands on
a scheduler-only node:

```json
{
  "reason": "topology_role_unavailable",
  "message": "This node does not host the topology roles required for this endpoint.",
  "current_shape": "standalone_server",
  "current_process_class": "scheduler_node",
  "current_roles": ["scheduler"],
  "required_roles": ["api_ingress", "control_plane"],
  "missing_roles": ["api_ingress", "control_plane"]
}
```

`GET /api/health`, `GET /api/ready`, and authenticated `GET /api/cluster/info`
stay available for liveness and discovery even on nodes that do not host the
current HTTP control surface.

### Workflow Bootstrap Gate

Authenticated routes that mutate or serve workflow v2 traffic also fail closed
when `checks.workflow_v2.status` on the responding node is `blocked`. The gate
runs after role and protocol-version validation but before namespace
resolution, so a request sent during a blocked rollout never observes namespace
existence.

When the gate trips, the server returns `503` with
`reason: "workflow_v2_blocked"` plus:

- `blocked_by`: the ordered list of upstream readiness blockers (for example
  `migrations`) that are keeping workflow v2 from serving safely.
- `remediation`: the short operator-facing instruction for clearing the listed
  blockers, mirrored from the `/api/ready` `checks.workflow_v2.remediation`
  field.

The bootstrap-gated route families are:

- **Workflow start and mutation** — every `/api/workflows` route in the start,
  describe, command, and run-targeted command groups (for example
  `POST /api/workflows`, `POST /api/workflows/{workflowId}/signal/{signalName}`,
  `POST /api/workflows/{workflowId}/runs/{runId}/cancel`).
- **Schedule mutation** — `POST /api/schedules`,
  `PUT /api/schedules/{scheduleId}`, `DELETE /api/schedules/{scheduleId}`,
  `POST /api/schedules/{scheduleId}/pause`,
  `POST /api/schedules/{scheduleId}/resume`,
  `POST /api/schedules/{scheduleId}/trigger`, and
  `POST /api/schedules/{scheduleId}/backfill`.
- **Bridge adapters** — `POST /api/bridge-adapters/webhook/{adapter}`.
- **Worker protocol** — every `/api/worker` and `/api/worker/*` route, including
  registration, heartbeat, and workflow-task, query-task, and activity-task
  poll/complete/fail/heartbeat verbs.

Schedule **reads** are intentionally exempted so operators can inspect schedule
state during recovery: `GET /api/schedules`,
`GET /api/schedules/{scheduleId}`, and
`GET /api/schedules/{scheduleId}/history` continue to serve while the bootstrap
gate is blocking other routes.

Control-plane routes return the bootstrap-gate payload in the control-plane
envelope, including the `X-Durable-Workflow-Control-Plane-Version` header.
Worker-protocol routes return the same `reason`, `blocked_by`, and
`remediation` fields in the worker-protocol envelope and keep the
`X-Durable-Workflow-Protocol-Version` header so workers can branch on the
machine-readable reason instead of inferring queue state from a bare `503`.

Example bootstrap-gate response from `POST /api/workflows` while a rollout-safety
migration is missing:

```json
{
  "reason": "workflow_v2_blocked",
  "message": "This node is not ready to serve workflow v2 traffic until bootstrap blockers are cleared.",
  "blocked_by": ["migrations"],
  "remediation": "Restore database connectivity and migrate the workflow tables before relying on workflow v2 rollout-safety health."
}
```

The same payload is returned in the worker-protocol envelope for
`/api/worker/*` routes, so worker SDKs can keep branching on `reason` and
retrying after the upstream blocker clears.

### Namespace-Scoped System Health

`GET /api/system/health` is the authenticated rollout-safety and coordination
health surface for one namespace. It requires admin auth plus
`X-Durable-Workflow-Control-Plane-Version: 2`, resolves the namespace through
the normal control-plane request rules, and returns the exact namespace the
server evaluated plus the current `health` snapshot:

<!-- docs-example id="server.system-health.curl" -->
```bash
curl -sS "$DURABLE_WORKFLOW_SERVER_URL/api/system/health" \
  -H "Authorization: Bearer $DW_ADMIN_TOKEN" \
  -H "X-Namespace: orders-prod" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  | jq '{namespace, status: .health.status, healthy: .health.healthy}'
```

Treat the payload as:

- `namespace`: the namespace whose rollout/coordination state was evaluated.
- `health.status` and `health.healthy`: the top-level machine-readable health
  verdict for that namespace.
- `health.checks` and `health.categories`: per-surface readiness, compatibility,
  projection, and coordination facts.
- `health.operator_metrics`: the current namespace-scoped queue, worker, and
  repair metrics bundled into the same snapshot.
- `health.structural_limits`: the effective structural limits and any related
  diagnostics the server is enforcing for that namespace.

## Workflow Control Plane

Workflow routes are operator/control-plane routes. They require an operator or
admin role and `X-Durable-Workflow-Control-Plane-Version: 2`.

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/api/workflows` | List workflow instances. Supports filters such as status, type, query text, and limit. |
| `POST` | `/api/workflows` | Start a workflow instance. |
| `GET` | `/api/workflows/{workflowId}` | Describe the current run for one workflow id. |
| `GET` | `/api/workflows/{workflowId}/debug` | Return bounded diagnostic facts for stuck-run investigation. |
| `GET` | `/api/workflows/{workflowId}/runs` | List runs for one workflow id. |
| `GET` | `/api/workflows/{workflowId}/runs/{runId}` | Describe a specific run. |
| `GET` | `/api/workflows/{workflowId}/runs/{runId}/debug` | Return bounded diagnostic facts for a selected run. |
| `GET` | `/api/workflows/{workflowId}/runs/{runId}/history` | Page through run history events. |
| `GET` | `/api/workflows/{workflowId}/runs/{runId}/history/export` | Export the archival replay bundle for a run. |

Start requests use the language-neutral control-plane shape:

<!-- docs-example id="server.workflow-start.curl" -->
```bash
curl -sS -X POST "$DURABLE_WORKFLOW_SERVER_URL/api/workflows" \
  -H "Authorization: Bearer $DURABLE_WORKFLOW_AUTH_TOKEN" \
  -H "X-Namespace: default" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{
    "workflow_type": "orders.fulfillment",
    "workflow_id": "order-1001",
    "task_queue": "orders",
    "input": ["order-1001"],
    "memo": {"source": "api-reference"},
    "search_attributes": {"CustomerId": "cust-42"},
    "duplicate_policy": "reject"
  }'
```

### Workflow Commands

Instance-targeted command routes operate on the current run for a workflow id:

| Method | Path | Purpose |
| --- | --- | --- |
| `POST` | `/api/workflows/{workflowId}/signal/{signalName}` | Send a signal. |
| `POST` | `/api/workflows/{workflowId}/query/{queryName}` | Execute a read-only query. |
| `POST` | `/api/workflows/{workflowId}/update/{updateName}` | Submit or execute an update. |
| `POST` | `/api/workflows/{workflowId}/cancel` | Request cancellation. |
| `POST` | `/api/workflows/{workflowId}/terminate` | Force termination. |
| `POST` | `/api/workflows/{workflowId}/repair` | Ask the server to repair retryable stuck state. |
| `POST` | `/api/workflows/{workflowId}/archive` | Archive a closed workflow run. |

Run-targeted command routes reject historical or wrong-run targets explicitly:

| Method | Path | Purpose |
| --- | --- | --- |
| `POST` | `/api/workflows/{workflowId}/runs/{runId}/signal/{signalName}` | Send a signal only if the selected run is current. |
| `POST` | `/api/workflows/{workflowId}/runs/{runId}/query/{queryName}` | Execute a query against the selected run. |
| `POST` | `/api/workflows/{workflowId}/runs/{runId}/update/{updateName}` | Submit or execute an update only if the selected run is current. |
| `POST` | `/api/workflows/{workflowId}/runs/{runId}/cancel` | Cancel only if the selected run is current. |
| `POST` | `/api/workflows/{workflowId}/runs/{runId}/terminate` | Terminate only if the selected run is current. |
| `POST` | `/api/workflows/{workflowId}/runs/{runId}/repair` | Repair only if the selected run is current. |
| `POST` | `/api/workflows/{workflowId}/runs/{runId}/archive` | Archive only if the selected run is current and closed. |

Commands with caller payloads use an `input` array. The Python and PHP SDKs
encode language-neutral payload envelopes for you; direct HTTP callers must
send JSON values that the target workflow or activity can decode.

## Namespace And Storage

Namespace routes require operator or admin roles. Mutating namespace and
external-storage routes require admin role.

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/api/namespaces` | List namespaces. |
| `POST` | `/api/namespaces` | Create a namespace. |
| `GET` | `/api/namespaces/{namespace}` | Describe namespace retention, metadata, and storage policy. |
| `PUT` | `/api/namespaces/{namespace}` | Update namespace metadata or retention. |
| `PUT` | `/api/namespaces/{namespace}/external-storage` | Configure the namespace external payload storage policy. |
| `POST` | `/api/storage/test` | Round-trip a small and large payload through the configured external storage driver. |

External payload storage policies let large payload envelopes carry stable
references instead of raw bytes. Local policies resolve through the configured
filesystem path. Object-storage policies such as `s3`, `gcs`, and `azure` use
an explicitly configured filesystem disk and bucket/prefix settings on the
server.

### External Payload Reference Envelope

The external payload reference is a stable wire envelope. SDKs may decode it
into native helper types, but HTTP clients should treat these field names as
the contract:

| Field | Required | Meaning |
| --- | --- | --- |
| `schema` | yes | Must be `durable-workflow.v2.external-payload-reference.v1`. Unknown schemas fail closed. |
| `uri` | yes | Driver-owned object location, such as `file:///...`, `s3://bucket/prefix/object`, `gs://bucket/prefix/object`, or `azure://container/prefix/object`. |
| `sha256` | yes | Lowercase hex SHA-256 of the stored encoded bytes. SDKs and the server verify it before decode. |
| `size_bytes` | yes | Byte length of the stored encoded payload. Mismatch is an integrity failure. |
| `codec` | yes | Always `avro`: fixed typed Value schema with Avro single-object framing. The surrounding HTTP document remains JSON. |
| `expires_at` | no | ISO-8601 expiry hint for retention/GC. Missing means the namespace retention policy owns cleanup. |

Payload offload is threshold-gated by the namespace storage policy. Inline
payloads continue to use the normal payload envelope until encoded bytes exceed
`threshold_bytes`; then the driver writes bytes and history stores the reference
envelope. Replay and history export must fail loudly when a referenced blob is
missing, mutated, outside the configured prefix, or owned by an unavailable
provider. They must not silently replace a missing object with `null`, `{}`, or
an empty byte string.

For the full request-authority contract, including namespace resolution,
role-scoped credentials, and worker registration fields, see
[Namespace, Auth, And Worker Registration](/docs/polyglot/namespace-auth-workers).

## Service Catalog Admin APIs

Service-catalog routes are authenticated admin control-plane routes. They use
the same namespace resolution, topology-role gating, and
`X-Durable-Workflow-Control-Plane-Version: 2` requirement as the rest of the
hosted control plane.

Use this route family to register namespace-scoped endpoint, service, and
operation metadata for the cross-namespace service catalog. Names are
case-insensitive on input and are normalized to lowercase in responses and
lookups.

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/api/service-endpoints` | List service endpoints for the current namespace. |
| `POST` | `/api/service-endpoints` | Create a service endpoint. |
| `GET` | `/api/service-endpoints/{endpointName}` | Describe one endpoint. |
| `PUT` | `/api/service-endpoints/{endpointName}` | Update endpoint description or metadata. |
| `DELETE` | `/api/service-endpoints/{endpointName}` | Delete an unused endpoint. |
| `GET` | `/api/service-endpoints/{endpointName}/services` | List services registered under one endpoint. |
| `POST` | `/api/service-endpoints/{endpointName}/services` | Create a service under one endpoint. |
| `GET` | `/api/service-endpoints/{endpointName}/services/{serviceName}` | Describe one service. |
| `PUT` | `/api/service-endpoints/{endpointName}/services/{serviceName}` | Update service description or metadata. |
| `DELETE` | `/api/service-endpoints/{endpointName}/services/{serviceName}` | Delete an unused service. |
| `GET` | `/api/service-endpoints/{endpointName}/services/{serviceName}/operations` | List operations registered under one service. |
| `POST` | `/api/service-endpoints/{endpointName}/services/{serviceName}/operations` | Create an operation binding. |
| `GET` | `/api/service-endpoints/{endpointName}/services/{serviceName}/operations/{operationName}` | Describe one operation. |
| `GET` | `/api/service-endpoints/{endpointName}/services/{serviceName}/operations/{operationName}/service-calls/{serviceCallId}` | Describe one durable service-call snapshot. |
| `PUT` | `/api/service-endpoints/{endpointName}/services/{serviceName}/operations/{operationName}` | Update an operation binding. |
| `DELETE` | `/api/service-endpoints/{endpointName}/services/{serviceName}/operations/{operationName}` | Delete an unused operation. |

Response collections use `service_endpoints`, `services`, or `operations`
arrays. Individual resources include stable lowercase names plus metadata and
timestamps:

- Endpoints return `id`, `namespace`, `endpoint_name`, `description`,
  `metadata`, `created_at`, and `updated_at`.
- Services add `endpoint_id` and `service_name`.
- Operations add `service_id`, `operation_name`, `operation_mode`,
  `handler_binding_kind`, `handler_target_reference`, `handler_binding`,
  `deadline_policy`, `idempotency_policy`, `cancellation_policy`,
  `retry_policy`, `boundary_policy`, and `metadata`.
- Service-call snapshots add `caller_namespace`, caller and linked workflow
  ids, `status`, `resolved_binding_kind`, `resolved_target_reference`,
  payload references, policy snapshots, and lifecycle timestamps such as
  `accepted_at`, `started_at`, `completed_at`, `failed_at`, and
  `cancelled_at`.

Operation create/update requests use the same JSON field names as the response.
`operation_mode` is `sync` or `async`. `handler_binding_kind` is one of
`start_workflow`, `signal_workflow`, `update_workflow`, `query_workflow`,
`activity_execution`, or `invocable_http`. New operations must provide either
`handler_target_reference` or a non-empty `handler_binding` payload.

Delete routes fail closed with HTTP `409` and a named reason when dependents
still exist, such as `endpoint_has_services`, `service_has_operations`, or
`operation_has_service_calls`.

## Bridge Adapters

Bridge adapters are bounded ingress endpoints. They do not execute workflow
code; they hand events to the control plane and return a named outcome.

| Method | Path | Purpose |
| --- | --- | --- |
| `POST` | `/api/bridge-adapters/webhook/{adapter}` | Start, signal, or update a workflow from a webhook-shaped event. |

Example:

```bash
curl -sS -X POST "$DURABLE_WORKFLOW_SERVER_URL/api/bridge-adapters/webhook/stripe" \
  -H "Authorization: Bearer $DURABLE_WORKFLOW_AUTH_TOKEN" \
  -H "X-Namespace: default" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "start_workflow",
    "idempotency_key": "evt_1001",
    "target": {
      "workflow_type": "orders.fulfillment",
      "task_queue": "orders",
      "business_key": "order-1001"
    },
    "input": {"order_id": "order-1001"}
  }'
```

Use response fields such as `outcome`, `reason`, `idempotency_key`, and
`control_plane_outcome` instead of inferring behavior from HTTP status alone.

## Worker Protocol

Worker routes require a worker role and
`X-Durable-Workflow-Protocol-Version: 1.0`. SDK workers use these endpoints
internally; custom language workers can implement the same protocol.

| Method | Path | Purpose |
| --- | --- | --- |
| `POST` | `/api/worker/register` | Register worker identity, task queues, supported workflow/activity types, capacity, runtime, and build metadata. |
| `POST` | `/api/worker/heartbeat` | Refresh worker fleet visibility and compatibility facts. |
| `POST` | `/api/worker/workflow-tasks/poll` | Long-poll for workflow tasks. |
| `POST` | `/api/worker/workflow-tasks/{taskId}/history` | Fetch paginated task history for a leased workflow task. |
| `POST` | `/api/worker/workflow-tasks/{taskId}/heartbeat` | Heartbeat a leased workflow task. |
| `POST` | `/api/worker/workflow-tasks/{taskId}/complete` | Complete a workflow task with commands. |
| `POST` | `/api/worker/workflow-tasks/{taskId}/fail` | Fail a workflow task. |
| `POST` | `/api/worker/query-tasks/poll` | Long-poll for server-routed query tasks. |
| `POST` | `/api/worker/query-tasks/{queryTaskId}/complete` | Complete a query task. |
| `POST` | `/api/worker/query-tasks/{queryTaskId}/fail` | Fail or reject a query task. |
| `POST` | `/api/worker/activity-tasks/poll` | Long-poll for activity tasks. |
| `POST` | `/api/worker/activity-tasks/{taskId}/heartbeat` | Heartbeat a leased activity task. |
| `POST` | `/api/worker/activity-tasks/{taskId}/complete` | Complete an activity task. |
| `POST` | `/api/worker/activity-tasks/{taskId}/fail` | Fail an activity task. |

Workers should treat lease ids, attempts, task ids, and heartbeat endpoints as
opaque server-issued values. A stale lease or wrong task id returns a named
worker-protocol error instead of silently completing work.

When `worker_protocol.server_capabilities.poll_status` is `true`, every
workflow-task, activity-task, and query-task poll response carries a
machine-readable `poll_status` field. Use it as the first branch point before
inspecting route-specific payload fields:

| `poll_status` | Typical HTTP status | Meaning |
| --- | --- | --- |
| `leased` | `200` | The server leased a task and `task` contains the task payload. |
| `empty` | `200` | No matching task was ready before the poll returned. |
| `throttled` | `200` | The queue is visible, but lease or dispatch admission limits withheld a new task for this poll. |
| `unavailable` | `503` or `200` | The server could not safely coordinate a poll path for the queue and returned a typed unavailable outcome instead of silently acting empty. |
| `draining` | `409` | The registered worker cohort is draining, so the server refuses to lease new work and returns `reason: "worker_draining"`. |

## Fleet And Task Queue Visibility

These routes expose operator diagnostics for worker fleets and queue
admission. They are control-plane routes.

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/api/workers` | List registered workers. |
| `GET` | `/api/workers/{workerId}` | Describe one worker. |
| `DELETE` | `/api/workers/{workerId}` | Deregister one worker. |
| `GET` | `/api/task-queues` | List task queues and admission status. |
| `GET` | `/api/task-queues/{taskQueue}` | Describe workflow/activity/query capacity for one queue. |
| `GET` | `/api/task-queues/{taskQueue}/build-ids` | List build ids observed for one queue. |
| `POST` | `/api/task-queues/{taskQueue}/build-ids/drain` | Mark a build-id cohort as draining so it stops claiming new tasks. |
| `POST` | `/api/task-queues/{taskQueue}/build-ids/resume` | Clear a previous drain so the cohort can claim new tasks again. |

Use task queue responses to distinguish no-worker conditions from saturated
worker slots, active lease caps, dispatch budgets, and query-task backpressure.
Drain and resume take a JSON body of `{"build_id": "..."}` (or
`{"build_id": null}` for the unversioned cohort), are idempotent, and persist
operator intent on the cohort so rollout state stays honest even after the
workers are removed. Once a worker heartbeat observes `drain_intent:
"draining"`, worker poll routes return HTTP `409` with `poll_status:
"draining"` and `reason: "worker_draining"` instead of leasing new tasks. See
[Worker Build-Id Rollout](/docs/polyglot/worker-build-id-rollout) for the
full unversioned-to-versioned cutover, canary, drain, and rollback lifecycle.

## Schedules And Search Attributes

Schedule routes are control-plane routes.

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/api/schedules` | List schedules. |
| `POST` | `/api/schedules` | Create a schedule. |
| `GET` | `/api/schedules/{scheduleId}` | Describe one schedule. |
| `PUT` | `/api/schedules/{scheduleId}` | Update schedule spec, action, note, memo, or search attributes. |
| `DELETE` | `/api/schedules/{scheduleId}` | Delete a schedule. |
| `POST` | `/api/schedules/{scheduleId}/pause` | Pause future fires. |
| `POST` | `/api/schedules/{scheduleId}/resume` | Resume a paused schedule. |
| `POST` | `/api/schedules/{scheduleId}/trigger` | Trigger a schedule immediately. |
| `POST` | `/api/schedules/{scheduleId}/backfill` | Backfill a time window. |
| `GET` | `/api/search-attributes` | List registered search attributes. |
| `POST` | `/api/search-attributes` | Register a search attribute. |
| `DELETE` | `/api/search-attributes/{name}` | Delete a search attribute. |

Search attribute names and types are part of the namespace search contract.
Avoid using high-cardinality attributes for operator dashboards or metric
labels.

## System Operations

System routes require admin role. They are explicit operator passes; prefer
status routes before pass routes in automation.

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/api/system/health` | Return the namespace-scoped rollout-safety and coordination health snapshot, nested under `health`. |
| `GET` | `/api/system/metrics` | Return bounded JSON metrics. |
| `GET` | `/api/system/operator-metrics` | Return the namespace-scoped operator metrics snapshot for runs, tasks, backlog, repair, workers, and structural limits. |
| `GET` | `/api/system/repair` | Inspect workflow repair backlog. |
| `POST` | `/api/system/repair/pass` | Run one workflow repair pass. |
| `GET` | `/api/system/activity-timeouts` | Inspect activity-timeout backlog. |
| `POST` | `/api/system/activity-timeouts/pass` | Run one activity-timeout enforcement pass. |
| `GET` | `/api/system/retention` | Inspect retention cleanup backlog. |
| `POST` | `/api/system/retention/pass` | Run one retention cleanup pass. |

`/api/system/health` is the quickest way to answer whether one namespace is
healthy enough to keep taking traffic. It returns `{namespace, health}`, where
`health` contains the categorized rollout-safety checks, the nested
`health.operator_metrics` snapshot, and the structural-limit summary used by
the health surface.

`/api/system/operator-metrics` is the namespace-scoped companion to
`/api/cluster/info` when you need raw backlog counts, compatibility-blocked
age, worker fleet detail, or other operator metrics behind the summarized
health surface.

`/api/system/metrics` is a JSON operator surface, not a Prometheus scrape
endpoint. Metric names and dimensions are bounded by the server's
bounded-growth policy.

## Error Contract

Error responses use HTTP status codes plus named machine-readable reasons.
Clients should branch on `reason` or nested control-plane/worker-protocol
reason fields, not on prose messages.

Common statuses:

| Status | Meaning |
| --- | --- |
| `400` | Missing or unsupported protocol/version header, malformed query, or unsupported route method. |
| `401` | Missing or invalid authentication. |
| `403` | Authenticated token lacks the required role. |
| `404` | Namespace, workflow, run, schedule, worker, or search attribute was not found. |
| `409` | Duplicate or conflict, such as an already-started workflow or invalid run target. |
| `422` | Validation failed; response includes field-level validation details. |
| `429` | Admission or task queue capacity is full. |
| `503` | The request reached a node that does not host the required topology roles. Hosted routes return `reason: "topology_role_unavailable"` plus `current_shape`, `current_process_class`, `current_roles`, `required_roles`, and `missing_roles`. The same status with `reason: "workflow_v2_blocked"` plus `blocked_by` and `remediation` covers workflow start/mutation, schedule mutation, bridge-adapter, and worker-protocol routes while workflow v2 bootstrap is blocked; schedule read routes (`GET /api/schedules`, `GET /api/schedules/{scheduleId}`, `GET /api/schedules/{scheduleId}/history`) stay available so operators can inspect schedule state during recovery. |
| `500` | Server failure. Treat as retryable only when the operation is idempotent or has an idempotency key. |

Validation responses include `reason: "validation_failed"` plus `errors` or
`validation_errors`. Workflow command responses also project validation and
operation details into the nested `control_plane` object.

## See Also

- [Server guide](/docs/polyglot/server)
- [Namespace, Auth, And Worker Registration](/docs/polyglot/namespace-auth-workers)
- [Worker Protocol](/docs/polyglot/worker-protocol)
- [Task Queue Admission](/docs/polyglot/task-queue-admission)
- [External Execution](/docs/polyglot/external-execution)
- [CLI Command Reference](/docs/polyglot/cli-reference)

<!-- Source: docs/polyglot/server-config-reference.md -->

# Server Config Reference

This page documents the operator-facing `DW_*` environment variable contract
for the Durable Workflow server image. Use it when building deployment
templates, reviewing production configuration, or migrating older
`WORKFLOW_*` / `ACTIVITY_*` names to the v2 server contract.

The server still consumes ordinary Laravel runtime settings such as `DB_*`,
`REDIS_*`, `QUEUE_CONNECTION`, and `CACHE_STORE`. Those runtime infrastructure
variables are listed separately because they are deployment plumbing, not
Durable Workflow API controls.

## How The Contract Is Enforced

The server repository keeps the canonical contract in `config/dw-contract.php`.
At container boot, `php artisan env:audit` warns when it sees unknown `DW_*`
variables or legacy names. Set `DW_ENV_AUDIT_STRICT=1` when you want the
entrypoint to fail instead of booting with warnings.

```bash
DW_ENV_AUDIT_STRICT=1
DW_AUTH_DRIVER=token
DW_ADMIN_TOKEN="${DW_ADMIN_TOKEN}"
DW_OPERATOR_TOKEN="${DW_OPERATOR_TOKEN}"
DW_WORKER_TOKEN="${DW_WORKER_TOKEN}"
```

Legacy names are fallback-only. Prefer the `DW_*` name in every new
deployment, and treat a legacy warning as migration debt even when the server
still honors the value.

## Server Identity And Mode

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_MODE` | `service` | Server mode: `service` makes external workers poll; `embedded` dispatches locally through the Laravel queue. | `WORKFLOW_SERVER_MODE` |
| `DW_SERVER_ID` | `gethostname()` | Unique server instance identifier used in lease ownership and worker registration. | `WORKFLOW_SERVER_ID` |
| `DW_SERVER_TOPOLOGY_SHAPE` | `standalone_server` | Advertised topology shape for cluster discovery, such as `embedded`, `standalone_server`, or `split_control_execution`. | `WORKFLOW_SERVER_TOPOLOGY_SHAPE` |
| `DW_SERVER_PROCESS_CLASS` | `server_http_node` | Advertised process class for this node within the selected topology shape, such as `server_http_node`, `worker_node`, or `scheduler_node`. | `WORKFLOW_SERVER_PROCESS_CLASS` |
| `DW_SERVER_KEY` | generated at container boot | Optional server-internal runtime key. Docker images generate one automatically when unset. | - |
| `DW_DEFAULT_NAMESPACE` | `default` | Namespace used when a request omits the namespace header. | `WORKFLOW_SERVER_DEFAULT_NAMESPACE` |
| `DW_TASK_DISPATCH_MODE` | unset | Overrides `workflows.v2.task_dispatch_mode`; in service mode the server defaults to `poll` unless you set a different value. | `WORKFLOW_V2_TASK_DISPATCH_MODE` |
| `DW_WORKFLOW_MEMO_MIGRATION_RECOVERY` | unset | One-run MySQL recovery proof for an unrecorded memo rewrite: `raw-json` or `envelope-prefix:<last-converted-id>`. | `WORKFLOW_SERVER_WORKFLOW_MEMO_MIGRATION_RECOVERY` |
| `DW_EXTERNAL_EXECUTOR_CONFIG_PATH` | unset | Path to a `durable-workflow.external-executor.config` JSON file for external executor handler mappings. | `WORKFLOW_SERVER_EXTERNAL_EXECUTOR_CONFIG_PATH` |
| `DW_EXTERNAL_EXECUTOR_CONFIG_OVERLAY` | unset | Overlay name from the external executor config file to apply before server validation and discovery. | `WORKFLOW_SERVER_EXTERNAL_EXECUTOR_CONFIG_OVERLAY` |

## Authentication

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_AUTH_PROVIDER` | unset | Laravel-resolvable class implementing `App\Contracts\AuthProvider`. | `WORKFLOW_SERVER_AUTH_PROVIDER` |
| `DW_AUTH_DRIVER` | `token` | Auth driver: `none`, `token`, or `signature`. | `WORKFLOW_SERVER_AUTH_DRIVER` |
| `DW_AUTH_TOKEN` | unset | Single shared bearer token when no role-scoped token is configured. | `WORKFLOW_SERVER_AUTH_TOKEN` |
| `DW_SIGNATURE_KEY` | unset | Shared HMAC signature key when no role-scoped signature key is configured. | `WORKFLOW_SERVER_SIGNATURE_KEY` |
| `DW_WORKER_TOKEN` | unset | Bearer token for worker registration, polling, heartbeats, and completions. | `WORKFLOW_SERVER_WORKER_TOKEN` |
| `DW_OPERATOR_TOKEN` | unset | Bearer token for read and operator control-plane actions. | `WORKFLOW_SERVER_OPERATOR_TOKEN` |
| `DW_ADMIN_TOKEN` | unset | Bearer token for namespace, retention, and other administrative mutations. | `WORKFLOW_SERVER_ADMIN_TOKEN` |
| `DW_PRINCIPAL_TOKENS` | unset | JSON token map for named bearer-token principals. Each entry supplies `token`, `subject`, `roles`, optional `tenant`, `label`, and non-secret `claims`. | `WORKFLOW_SERVER_PRINCIPAL_TOKENS` |
| `DW_WORKER_SIGNATURE_KEY` | unset | HMAC key for worker-role requests when using signature auth. | `WORKFLOW_SERVER_WORKER_SIGNATURE_KEY` |
| `DW_OPERATOR_SIGNATURE_KEY` | unset | HMAC key for operator-role requests when using signature auth. | `WORKFLOW_SERVER_OPERATOR_SIGNATURE_KEY` |
| `DW_ADMIN_SIGNATURE_KEY` | unset | HMAC key for admin-role requests when using signature auth. | `WORKFLOW_SERVER_ADMIN_SIGNATURE_KEY` |
| `DW_AUTH_BACKWARD_COMPATIBLE` | `true` | Honor shared `DW_AUTH_TOKEN` / `DW_SIGNATURE_KEY` as a fallback when a role-scoped credential is missing. | `WORKFLOW_SERVER_AUTH_BACKWARD_COMPATIBLE` |

Use role-scoped credentials for production. `DW_AUTH_DRIVER=none` is only for
local smoke work because every endpoint becomes reachable without a bearer
token or signature.

Use `DW_PRINCIPAL_TOKENS` when audit trails need stable actor subjects rather
than role labels. The server derives the recorded principal from the matched
token entry; clients cannot override it with request payloads or headers.

## Command Attribution

These settings let a trusted gateway preserve caller metadata in durable
command history. Leave `DW_TRUST_FORWARDED_ATTRIBUTION_HEADERS=false` unless
the server is behind a gateway that strips untrusted client-supplied headers.

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_TRUST_FORWARDED_ATTRIBUTION_HEADERS` | `false` | Record forwarded caller/auth headers into workflow command history. | `WORKFLOW_SERVER_TRUST_FORWARDED_ATTRIBUTION_HEADERS` |
| `DW_CALLER_TYPE_HEADER` | `X-Workflow-Caller-Type` | Header carrying forwarded caller type. | `WORKFLOW_SERVER_CALLER_TYPE_HEADER` |
| `DW_CALLER_LABEL_HEADER` | `X-Workflow-Caller-Label` | Header carrying forwarded caller label. | `WORKFLOW_SERVER_CALLER_LABEL_HEADER` |
| `DW_AUTH_STATUS_HEADER` | `X-Workflow-Auth-Status` | Header carrying forwarded auth status. | `WORKFLOW_SERVER_AUTH_STATUS_HEADER` |
| `DW_AUTH_METHOD_HEADER` | `X-Workflow-Auth-Method` | Header carrying forwarded auth method. | `WORKFLOW_SERVER_AUTH_METHOD_HEADER` |

## Worker Polling And Admission

These values control long-poll timing, wake coordination, server-side
admission caps, and bounded task dispatch budgets.

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_WORKER_POLL_TIMEOUT` | `30` | Seconds the server holds a poll open waiting for a task. | `WORKFLOW_SERVER_WORKER_POLL_TIMEOUT` |
| `DW_WORKER_POLL_INTERVAL_MS` | `1000` | Milliseconds between internal scans while a poll is held open. | `WORKFLOW_SERVER_WORKER_POLL_INTERVAL_MS` |
| `DW_WORKER_POLL_SIGNAL_CHECK_INTERVAL_MS` | `100` | Milliseconds between wake-signal checks while a poll is held open. | `WORKFLOW_SERVER_WORKER_POLL_SIGNAL_CHECK_INTERVAL_MS` |
| `DW_POLLING_CACHE_PATH` | `storage/framework/cache/server-polling/<APP_ENV>` | Directory for worker-poll coordination state when using file-backed polling cache. | `WORKFLOW_SERVER_POLLING_CACHE_PATH` |
| `DW_WAKE_SIGNAL_TTL_SECONDS` | `max(DW_WORKER_POLL_TIMEOUT + 5, 60)` | TTL for per-queue wake signals that short-circuit a pending poll. | `WORKFLOW_SERVER_WAKE_SIGNAL_TTL_SECONDS` |
| `DW_WORKER_LONG_POLL_MAX_CONCURRENT` | unset; derived for `PHP_CLI_SERVER_WORKERS` | Optional cap for concurrent held workflow/activity worker long-poll waits on this server node. Query-task polls use a separate wait budget. | `WORKFLOW_SERVER_WORKER_LONG_POLL_MAX_CONCURRENT` |
| `DW_WORKER_LONG_POLL_RESERVED_HTTP_WORKERS` | `2` | PHP CLI server workers reserved for health and control-plane requests when deriving the workflow/activity long-poll wait cap. | `WORKFLOW_SERVER_WORKER_LONG_POLL_RESERVED_HTTP_WORKERS` |
| `DW_MAX_TASKS_PER_POLL` | `1` | Maximum tasks returned per worker poll. | `WORKFLOW_SERVER_MAX_TASKS_PER_POLL` |
| `DW_SQLITE_CLAIM_LOCK_TTL_SECONDS` | `10` | Seconds the SQLite quickstart backend holds the cache-backed worker poll claim gate before the lock expires. | `WORKFLOW_SERVER_SQLITE_CLAIM_LOCK_TTL_SECONDS` |
| `DW_SQLITE_CLAIM_LOCK_WAIT_SECONDS` | `5` | Seconds SQLite worker poll claims wait for the cache-backed claim gate before returning backend lock pressure. | `WORKFLOW_SERVER_SQLITE_CLAIM_LOCK_WAIT_SECONDS` |
| `DW_WORKFLOW_TASK_MAX_ACTIVE_LEASES_PER_QUEUE` | unset | Active workflow-task lease cap per namespace/task queue. | `WORKFLOW_SERVER_WORKFLOW_TASK_MAX_ACTIVE_LEASES_PER_QUEUE` |
| `DW_WORKFLOW_TASK_MAX_ACTIVE_LEASES_PER_NAMESPACE` | unset | Active workflow-task lease cap across all queues in a namespace. | `WORKFLOW_SERVER_WORKFLOW_TASK_MAX_ACTIVE_LEASES_PER_NAMESPACE` |
| `DW_WORKFLOW_TASK_MAX_DISPATCHES_PER_MINUTE` | unset | Per-minute workflow-task dispatch cap per namespace/task queue. | `WORKFLOW_SERVER_WORKFLOW_TASK_MAX_DISPATCHES_PER_MINUTE` |
| `DW_WORKFLOW_TASK_MAX_DISPATCHES_PER_MINUTE_PER_NAMESPACE` | unset | Per-minute workflow-task dispatch cap across a namespace. | `WORKFLOW_SERVER_WORKFLOW_TASK_MAX_DISPATCHES_PER_MINUTE_PER_NAMESPACE` |
| `DW_ACTIVITY_TASK_MAX_ACTIVE_LEASES_PER_QUEUE` | unset | Active activity-task lease cap per namespace/task queue. | `WORKFLOW_SERVER_ACTIVITY_TASK_MAX_ACTIVE_LEASES_PER_QUEUE` |
| `DW_ACTIVITY_TASK_MAX_ACTIVE_LEASES_PER_NAMESPACE` | unset | Active activity-task lease cap across all queues in a namespace. | `WORKFLOW_SERVER_ACTIVITY_TASK_MAX_ACTIVE_LEASES_PER_NAMESPACE` |
| `DW_ACTIVITY_TASK_MAX_DISPATCHES_PER_MINUTE` | unset | Per-minute activity-task dispatch cap per namespace/task queue. | `WORKFLOW_SERVER_ACTIVITY_TASK_MAX_DISPATCHES_PER_MINUTE` |
| `DW_ACTIVITY_TASK_MAX_DISPATCHES_PER_MINUTE_PER_NAMESPACE` | unset | Per-minute activity-task dispatch cap across a namespace. | `WORKFLOW_SERVER_ACTIVITY_TASK_MAX_DISPATCHES_PER_MINUTE_PER_NAMESPACE` |
| `DW_TASK_QUEUE_ADMISSION_OVERRIDES` | `{}` | JSON overrides keyed by `namespace:task_queue`, `namespace:*`, `task_queue`, or `*` for active leases, dispatch rate, namespace caps, and downstream budget groups. | `WORKFLOW_SERVER_TASK_QUEUE_ADMISSION_OVERRIDES` |
| `DW_DUE_TIMER_RECOVERY_SCAN_LIMIT` | `5` | Maximum due service-mode timer tasks recovered per worker poll pass. | `WORKFLOW_SERVER_DUE_TIMER_RECOVERY_SCAN_LIMIT` |
| `DW_EXPIRED_WORKFLOW_TASK_RECOVERY_SCAN_LIMIT` | `5` | Maximum expired workflow tasks recovered per pass. | `WORKFLOW_SERVER_EXPIRED_WORKFLOW_TASK_RECOVERY_SCAN_LIMIT` |
| `DW_EXPIRED_WORKFLOW_TASK_RECOVERY_TTL_SECONDS` | `5` | Minimum seconds between expired-task recovery passes per queue. | `WORKFLOW_SERVER_EXPIRED_WORKFLOW_TASK_RECOVERY_TTL_SECONDS` |

Sticky execution has no standalone server-image environment variables. See
[Sticky Execution](/docs/features/sticky-execution) for the sticky-cache
lifecycle, worker protocol fields, replay modes, and diagnostics.

Example admission override:

```json
{
  "production:billing": {
    "workflow_tasks": {
      "max_active_leases": 50,
      "max_dispatches_per_minute": 600
    },
    "activity_tasks": {
      "max_active_leases": 200
    }
  },
  "production:*": {
    "workflow_tasks": {
      "max_active_leases_per_namespace": 500
    }
  }
}
```

## Worker Protocol And Query Transport

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_WORKER_PROTOCOL_VERSION` | `WorkerProtocolVersion::VERSION` | Worker-protocol version advertised on worker-plane responses. | `WORKFLOW_SERVER_WORKER_PROTOCOL_VERSION` |
| `DW_HISTORY_PAGE_SIZE_DEFAULT` | `WorkerProtocolVersion::DEFAULT_HISTORY_PAGE_SIZE` | Default page size for worker history reads. | `WORKFLOW_SERVER_HISTORY_PAGE_SIZE_DEFAULT` |
| `DW_HISTORY_PAGE_SIZE_MAX` | `WorkerProtocolVersion::MAX_HISTORY_PAGE_SIZE` | Maximum page size honored on worker history reads. | `WORKFLOW_SERVER_HISTORY_PAGE_SIZE_MAX` |
| `DW_UPDATE_VALIDATION_TIMEOUT` | `10` | Seconds the control plane waits for a synchronous pre-accept update validator result. | `WORKFLOW_SERVER_UPDATE_VALIDATION_TIMEOUT` |
| `DW_UPDATE_VALIDATION_LEASE_TIMEOUT` | `5` | Seconds an update-validation task lease remains owned before a replacement validator-capable worker may retry it. | `WORKFLOW_SERVER_UPDATE_VALIDATION_LEASE_TIMEOUT` |
| `DW_QUERY_TASK_TIMEOUT` | `DW_WORKER_POLL_TIMEOUT` | Seconds the control plane waits for a query task response from the worker. | `WORKFLOW_SERVER_QUERY_TASK_TIMEOUT` |
| `DW_QUERY_TASK_LEASE_TIMEOUT` | `DW_WORKFLOW_TASK_TIMEOUT` | Lease timeout for ephemeral query tasks handed to workers. | `WORKFLOW_SERVER_QUERY_TASK_LEASE_TIMEOUT` |
| `DW_QUERY_TASK_TTL_SECONDS` | `180` | How long the server retains query-task result rows before reaping them. | `WORKFLOW_SERVER_QUERY_TASK_TTL_SECONDS` |
| `DW_QUERY_TASK_MAX_PENDING_PER_QUEUE` | `1024` | Maximum pending cache-backed query tasks per namespace/task queue before new queries are rejected. | `WORKFLOW_SERVER_QUERY_TASK_MAX_PENDING_PER_QUEUE` |
| `DW_QUERY_TASK_POLL_TIMEOUT` | `5` | Maximum seconds each idle query-task worker poll waits before rechecking workflow-task pressure and control-plane availability. | `WORKFLOW_SERVER_QUERY_TASK_POLL_TIMEOUT` |
| `DW_QUERY_TASK_POLL_MAX_CONCURRENT` | unset; derived for `PHP_CLI_SERVER_WORKERS` | Optional cap for concurrent held idle query-task worker long-poll waits on this server node. Pending query tasks can still be claimed immediately before an idle poll waits. | `WORKFLOW_SERVER_QUERY_TASK_POLL_MAX_CONCURRENT` |
| `DW_WORKFLOW_TASK_TIMEOUT` | `60` | Default workflow-task lease timeout in seconds. | `WORKFLOW_TASK_TIMEOUT` |
| `DW_ACTIVITY_TASK_TIMEOUT` | `300` | Default activity-task lease timeout in seconds. | `ACTIVITY_TASK_TIMEOUT` |
| `DW_WORKER_STALE_AFTER_SECONDS` | `max(DW_WORKER_POLL_TIMEOUT * 2, 60)` | Seconds after a worker heartbeat before the worker registration is stale. | `WORKFLOW_SERVER_WORKER_STALE_AFTER_SECONDS` |
| `DW_WORKER_HEARTBEAT_INTERVAL_SECONDS` | `60` | Cadence in seconds advertised to SDKs in worker register and heartbeat acknowledgements. | `WORKFLOW_SERVER_WORKER_HEARTBEAT_INTERVAL_SECONDS` |

## Limits, Retention, And Metrics

These settings are request-boundary and bounded-growth controls. If an
application regularly approaches these values, prefer smaller payloads,
external payload storage, or continue-as-new rather than raising the limits by
default.

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_METRICS_WORKFLOW_TASK_FAILURE_TYPE_LIMIT` | `20` | Maximum `workflow_type` series reported by `dw_workflow_task_consecutive_failures`; excess types are summarized. | `WORKFLOW_SERVER_METRICS_WORKFLOW_TASK_FAILURE_TYPE_LIMIT` |
| `DW_METRICS_PROMETHEUS_WORKFLOW_SERIES_LIMIT` | `100` | Maximum workflow series reported by `/api/system/prometheus-metrics` before excess series are summarized. | `WORKFLOW_SERVER_METRICS_PROMETHEUS_WORKFLOW_SERIES_LIMIT` |
| `DW_METRICS_PROMETHEUS_ACTIVITY_SERIES_LIMIT` | `100` | Maximum activity series reported by `/api/system/prometheus-metrics` before excess series are summarized. | `WORKFLOW_SERVER_METRICS_PROMETHEUS_ACTIVITY_SERIES_LIMIT` |
| `DW_METRICS_PROMETHEUS_TASK_QUEUE_SERIES_LIMIT` | `100` | Maximum task-queue runtime series reported by `/api/system/prometheus-metrics` before excess series are summarized. | `WORKFLOW_SERVER_METRICS_PROMETHEUS_TASK_QUEUE_SERIES_LIMIT` |
| `DW_MAX_HISTORY_EVENTS` | `50000` | Maximum history events per workflow run before continue-as-new is enforced. | `WORKFLOW_MAX_HISTORY_EVENTS` |
| `DW_HISTORY_RETENTION_DAYS` | `30` | Default number of days closed-run history is retained when a namespace does not override it. | `WORKFLOW_HISTORY_RETENTION_DAYS` |
| `DW_MAX_PAYLOAD_BYTES` | `2097152` | Maximum serialized bytes for one payload. | `WORKFLOW_MAX_PAYLOAD_BYTES` |
| `DW_EXTERNAL_PAYLOAD_MAX_BYTES` | `67108864` | Maximum encoded size in bytes (64 MiB) accepted by the authenticated external-payload upload and fetch transport. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_MAX_BYTES` |
| `DW_EXTERNAL_PAYLOAD_REQUEST_TIMEOUT` | `30` | Client-facing upload and fetch request-timeout budget advertised by the server, in seconds. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_REQUEST_TIMEOUT` |
| `DW_EXTERNAL_PAYLOAD_UPLOAD_EXPIRY` | `3600` | Seconds an uploaded external payload may remain unclaimed before its opaque reference expires. | `WORKFLOW_SERVER_EXTERNAL_PAYLOAD_UPLOAD_EXPIRY` |
| `DW_MAX_MEMO_BYTES` | `262144` | Maximum serialized bytes for a workflow memo. | `WORKFLOW_MAX_MEMO_BYTES` |
| `DW_MAX_SEARCH_ATTRIBUTES` | `100` | Maximum number of search attributes on one workflow. | `WORKFLOW_MAX_SEARCH_ATTRIBUTES` |
| `DW_MAX_SEARCH_ATTRIBUTE_KEY_LENGTH` | `128` | Maximum byte length for one search-attribute key. | `WORKFLOW_MAX_SEARCH_ATTRIBUTE_KEY_LENGTH` |
| `DW_MAX_SEARCH_ATTRIBUTE_VALUE_BYTES` | `2048` | Maximum byte size for one search-attribute string value. | `WORKFLOW_MAX_SEARCH_ATTRIBUTE_VALUE_BYTES` |
| `DW_MAX_OPERATION_NAME_LENGTH` | `256` | Maximum byte length for a signal, update, or query name. | `WORKFLOW_MAX_OPERATION_NAME_LENGTH` |
| `DW_MAX_PENDING_ACTIVITIES` | `2000` | Maximum pending activities per workflow run before rejecting a command batch. | `WORKFLOW_MAX_PENDING_ACTIVITIES` |
| `DW_MAX_PENDING_CHILDREN` | `2000` | Maximum pending child workflows per run before rejecting a command batch. | `WORKFLOW_MAX_PENDING_CHILDREN` |
| `DW_MAX_NEXUS_OPERATIONS_PER_CALLER` | `200` | Maximum Nexus operations returned per caller from the operations history surface before clients must paginate. | `WORKFLOW_MAX_NEXUS_OPERATIONS_PER_CALLER` |
| `DW_COMPRESSION_ENABLED` | `true` | Enable JSON response compression above the minimum size threshold. | `WORKFLOW_SERVER_COMPRESSION_ENABLED` |

## Docker Bootstrap And Provenance

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_EXPOSE_PACKAGE_PROVENANCE` | `false` | Include `package_provenance` in `/api/cluster/info` for admin requests. | `WORKFLOW_SERVER_EXPOSE_PACKAGE_PROVENANCE` |
| `DW_PACKAGE_PROVENANCE_PATH` | `<base_path>/.package-provenance` | Absolute path to the Docker build provenance file. | `WORKFLOW_SERVER_PACKAGE_PROVENANCE_PATH` |
| `DW_SERVICE_BOUNDARY_CROSS_NAMESPACE_DEFAULT` | `allow` | Default service-call boundary action for cross-namespace calls when no more-specific rule matches. | `WORKFLOW_SERVER_SERVICE_BOUNDARY_CROSS_NAMESPACE_DEFAULT` |
| `DW_SERVICE_BOUNDARY_RATE_LIMIT_PER_MINUTE` | unset | Optional per-minute service-call boundary rate limit. | `WORKFLOW_SERVER_SERVICE_BOUNDARY_RATE_LIMIT_PER_MINUTE` |
| `DW_SERVICE_BOUNDARY_MAX_IN_FLIGHT` | unset | Optional service-call boundary concurrency limit. | `WORKFLOW_SERVER_SERVICE_BOUNDARY_MAX_IN_FLIGHT` |

## Docker Bootstrap

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_ENV_AUDIT_STRICT` | `0` | Fail container boot when `env:audit` finds unknown or legacy `DW_*` variables. | - |
| `DW_BOOTSTRAP_RETRIES` | `30` | Bootstrap attempts before entrypoint gives up on migrations and default namespace seed. | `WORKFLOW_SERVER_BOOTSTRAP_RETRIES` |
| `DW_BOOTSTRAP_DELAY_SECONDS` | `2` | Seconds between bootstrap attempts. | `WORKFLOW_SERVER_BOOTSTRAP_DELAY_SECONDS` |

## Workflow Package Controls

The server image bundles `durable-workflow/workflow`, so it also exposes the
package-level `DW_V2_*` controls. These should match worker deployments that
execute the same workflows.

| Variable | Default | Purpose | Legacy alias |
| --- | --- | --- | --- |
| `DW_V2_NAMESPACE` | unset | Scope workflow instances to a namespace. | `WORKFLOW_V2_NAMESPACE` |
| `DW_V2_TENANCY_ORGANIZATION` | unset | Optional organization segment for the workflow package tenancy hierarchy exposed to readiness, discovery, and operator surfaces. | `WORKFLOW_V2_TENANCY_ORGANIZATION` |
| `DW_V2_TENANCY_PROJECT` | unset | Optional project segment for the workflow package tenancy hierarchy exposed to readiness, discovery, and operator surfaces. | `WORKFLOW_V2_TENANCY_PROJECT` |
| `DW_V2_TENANCY_ENVIRONMENT` | unset | Optional environment segment for the workflow package tenancy hierarchy exposed to readiness, discovery, and operator surfaces. | `WORKFLOW_V2_TENANCY_ENVIRONMENT` |
| `DW_V2_CURRENT_COMPATIBILITY` | unset | Worker-compatibility marker this worker advertises. | `WORKFLOW_V2_CURRENT_COMPATIBILITY` |
| `DW_V2_SUPPORTED_COMPATIBILITIES` | unset | Comma-separated compatibility markers this worker accepts, or `*`. | `WORKFLOW_V2_SUPPORTED_COMPATIBILITIES` |
| `DW_V2_COMPATIBILITY_NAMESPACE` | unset | Compatibility namespace for shared workflow databases with independent fleets. | `WORKFLOW_V2_COMPATIBILITY_NAMESPACE` |
| `DW_V2_COMPATIBILITY_HEARTBEAT_TTL` | `30` | Seconds a worker-compatibility heartbeat remains valid. | `WORKFLOW_V2_COMPATIBILITY_HEARTBEAT_TTL` |
| `DW_V2_PIN_TO_RECORDED_FINGERPRINT` | `true` | Resolve in-flight runs from the workflow fingerprint recorded at `WorkflowStarted`. | `WORKFLOW_V2_PIN_TO_RECORDED_FINGERPRINT` |
| `DW_V2_CONTINUE_AS_NEW_EVENT_THRESHOLD` | `10000` | History event count at which the package asks the workflow author to continue-as-new. | `WORKFLOW_V2_CONTINUE_AS_NEW_EVENT_THRESHOLD` |
| `DW_V2_CONTINUE_AS_NEW_SIZE_BYTES_THRESHOLD` | `5242880` | Serialized-history byte size at which the package asks for continue-as-new. | `WORKFLOW_V2_CONTINUE_AS_NEW_SIZE_BYTES_THRESHOLD` |
| `DW_V2_HISTORY_EXPORT_SIGNING_KEY` | unset | Optional HMAC key for authenticating history export archives. | `WORKFLOW_V2_HISTORY_EXPORT_SIGNING_KEY` |
| `DW_V2_HISTORY_EXPORT_SIGNING_KEY_ID` | unset | Key identifier recorded alongside signed history exports. | `WORKFLOW_V2_HISTORY_EXPORT_SIGNING_KEY_ID` |
| `DW_V2_UPDATE_WAIT_COMPLETION_TIMEOUT_SECONDS` | `10` | Seconds the server waits for an update to reach a terminal stage. | `WORKFLOW_V2_UPDATE_WAIT_COMPLETION_TIMEOUT_SECONDS` |
| `DW_V2_UPDATE_WAIT_POLL_INTERVAL_MS` | `50` | Milliseconds between update-stage polls. | `WORKFLOW_V2_UPDATE_WAIT_POLL_INTERVAL_MS` |
| `DW_V2_GUARDRAILS_BOOT` | `warn` | Boot-time structural guardrail mode: `warn`, `fail`, or `silent`. | `WORKFLOW_V2_GUARDRAILS_BOOT` |
| `DW_V2_LIMIT_PENDING_ACTIVITIES` | `2000` | Package-level pending-activity ceiling. | `WORKFLOW_V2_LIMIT_PENDING_ACTIVITIES` |
| `DW_V2_LIMIT_PENDING_CHILDREN` | `1000` | Package-level pending-child-workflow ceiling. | `WORKFLOW_V2_LIMIT_PENDING_CHILDREN` |
| `DW_V2_LIMIT_PENDING_TIMERS` | `2000` | Package-level pending-timer ceiling. | `WORKFLOW_V2_LIMIT_PENDING_TIMERS` |
| `DW_V2_LIMIT_PENDING_SIGNALS` | `5000` | Package-level pending-signal ceiling. | `WORKFLOW_V2_LIMIT_PENDING_SIGNALS` |
| `DW_V2_LIMIT_PENDING_UPDATES` | `500` | Package-level pending-update ceiling. | `WORKFLOW_V2_LIMIT_PENDING_UPDATES` |
| `DW_V2_LIMIT_COMMAND_BATCH_SIZE` | `1000` | Maximum commands accepted in one workflow-task completion. | `WORKFLOW_V2_LIMIT_COMMAND_BATCH_SIZE` |
| `DW_V2_LIMIT_PAYLOAD_SIZE_BYTES` | `2097152` | Package-level single-payload byte ceiling. | `WORKFLOW_V2_LIMIT_PAYLOAD_SIZE_BYTES` |
| `DW_V2_LIMIT_MEMO_SIZE_BYTES` | `262144` | Package-level workflow-memo byte ceiling. | `WORKFLOW_V2_LIMIT_MEMO_SIZE_BYTES` |
| `DW_V2_LIMIT_SEARCH_ATTRIBUTE_SIZE_BYTES` | `40960` | Package-level search-attribute byte ceiling. | `WORKFLOW_V2_LIMIT_SEARCH_ATTRIBUTE_SIZE_BYTES` |
| `DW_V2_LIMIT_HISTORY_TRANSACTION_SIZE` | `5000` | Package-level history-transaction event ceiling. | `WORKFLOW_V2_LIMIT_HISTORY_TRANSACTION_SIZE` |
| `DW_V2_LIMIT_WARNING_THRESHOLD_PERCENT` | `80` | Percent of a structural limit at which the package emits a warning. | `WORKFLOW_V2_LIMIT_WARNING_THRESHOLD_PERCENT` |
| `DW_V2_TASK_DISPATCH_MODE` | `queue` | Package-level task dispatch mode, usually overridden by `DW_TASK_DISPATCH_MODE` in server mode. | - |
| `DW_V2_MATCHING_ROLE_QUEUE_WAKE` | `true` | Whether queue workers run the in-worker matching-role wake on every Looping event. Set to `false` to opt execution-only nodes out of the broad-poll wake when a dedicated `php artisan workflow:v2:repair-pass --loop` daemon owns the sweep instead. | `WORKFLOW_V2_MATCHING_ROLE_QUEUE_WAKE` |
| `DW_V2_TASK_REPAIR_REDISPATCH_AFTER_SECONDS` | `3` | Seconds before an orphaned workflow task is redispatched by repair. | `WORKFLOW_V2_TASK_REPAIR_REDISPATCH_AFTER_SECONDS` |
| `DW_V2_TASK_REPAIR_LOOP_THROTTLE_SECONDS` | `5` | Minimum seconds between task-repair passes per queue. | `WORKFLOW_V2_TASK_REPAIR_LOOP_THROTTLE_SECONDS` |
| `DW_V2_TASK_REPAIR_SCAN_LIMIT` | `25` | Maximum tasks considered per task-repair pass. | `WORKFLOW_V2_TASK_REPAIR_SCAN_LIMIT` |
| `DW_V2_TASK_REPAIR_FAILURE_BACKOFF_MAX_SECONDS` | `60` | Maximum task-repair failure backoff in seconds. | `WORKFLOW_V2_TASK_REPAIR_FAILURE_BACKOFF_MAX_SECONDS` |
| `DW_V2_MULTI_NODE` | `false` | Declare a multi-node deployment so cache backends are validated for cross-node coordination. | `WORKFLOW_V2_MULTI_NODE` |
| `DW_V2_VALIDATE_CACHE_BACKEND` | `true` | Validate the long-poll cache backend at boot. | `WORKFLOW_V2_VALIDATE_CACHE_BACKEND` |
| `DW_V2_CACHE_VALIDATION_MODE` | `warn` | Cache-backend validation failure mode: `fail`, `warn`, or `silent`. | `WORKFLOW_V2_CACHE_VALIDATION_MODE` |
| `DW_V2_FLEET_VALIDATION_MODE` | `warn` | Fleet-compatibility validation mode: `warn` logs, `fail` blocks dispatch and fails closed when no compatible worker is available. | `WORKFLOW_V2_FLEET_VALIDATION_MODE` |
| `DW_SERIALIZER` | `avro` | Payload codec diagnostic input; final v2 resolves new-run payloads to Avro. | `WORKFLOW_SERIALIZER` |

Use `DW_V2_GUARDRAILS_BOOT` in CI and deployment manifests. The older `WORKFLOW_V2_GUARDRAILS_BOOT` name is retained only so `env:audit` can point alpha-era operators at the rename; the workflow package no longer reads it as a runtime fallback.

Use [Task Matching and Dispatch](/docs/polyglot/task-matching-dispatch)
when you configure `DW_V2_MATCHING_ROLE_QUEUE_WAKE` or a dedicated
`workflow:v2:repair-pass --loop` daemon. Those settings change where
ready-task discovery runs, not the worker-protocol contract itself.

## Runtime Infrastructure Variables

Runtime infrastructure variables are framework and dependency controls. The
server audit recognizes them so it does not warn, but they are not stable
Durable Workflow API knobs.

| Group | Variables | Use |
| --- | --- | --- |
| Application | `APP_NAME`, `APP_ENV`, `APP_KEY`, `APP_DEBUG`, `APP_URL`, `APP_VERSION`, locale, timezone, maintenance, and cipher settings | Laravel application runtime identity and boot behavior. |
| Logging | `LOG_CHANNEL`, `LOG_LEVEL`, stack, daily, Slack, and Papertrail settings | Laravel logging destinations and retention. |
| Database | `DB_CONNECTION`, `DB_HOST`, `DB_PORT`, `DB_DATABASE`, `DB_USERNAME`, `DB_PASSWORD`, `DB_SOCKET`, `DB_URL`, charset, collation, and foreign-key settings | SQL state store for workflow, namespace, worker, and projection tables. |
| Redis/cache/queue | `REDIS_*`, `QUEUE_CONNECTION`, `QUEUE_FAILED_DRIVER`, `CACHE_STORE`, `CACHE_PREFIX`, `SESSION_*` | Queue workers, cache locks, long-poll signaling, and web/session runtime support. |
| Filesystems/mail/broadcasting | `FILESYSTEM_DISK`, `MAIL_*`, `BROADCAST_*`, `PUSHER_*`, `AWS_*` | Framework integrations used by deployment-specific features. |
| Build/runtime | `MYSQL_VERSION`, `REDIS_VERSION`, `PHP_CLI_SERVER_WORKERS`, `VITE_APP_NAME`, `BCRYPT_ROUNDS` | Docker Compose images and framework runtime behavior. |

## Migration Notes

When migrating an older deployment, change the public name first and leave the
legacy name unset. If both are present, the `DW_*` value is the value operators
should reason about, and the legacy name should be removed.

```bash
# Before
WORKFLOW_SERVER_AUTH_DRIVER=token
WORKFLOW_SERVER_OPERATOR_TOKEN=operator-secret

# After
DW_AUTH_DRIVER=token
DW_OPERATOR_TOKEN=operator-secret
```

Run the server image with `DW_ENV_AUDIT_STRICT=1` after migration to catch
misspelled variables, stale aliases, and settings copied from older runbooks.

<!-- Source: docs/polyglot/cli-reference.md -->

# CLI Command Reference

This page documents the v2 `dw` command surface as an operator and automation
contract. Use the [CLI guide](/docs/polyglot/cli) for installation and
profile setup; use this page when wiring scripts, CI jobs, runbooks, or AI
agents to exact command shapes.

All server-backed commands target the standalone server control-plane protocol
version `2`. The CLI validates the server-published protocol manifests before
trusting canonical request and response fields.

## Global Options

Server-backed commands accept these options unless noted otherwise.

| Option | Meaning |
| --- | --- |
| `--server`, `-s` | Server base URL. Overrides profiles and environment variables for this invocation. |
| `--namespace` | Target namespace. Overrides the profile namespace. |
| `--token` | Bearer token for this invocation. Prefer profiles with `--token-env` for stored automation. |
| `--env` | Named CLI environment profile. Unknown profile names fail instead of falling back. |
| `--output=table|json|jsonl` | Output contract. `table` is human-readable, `json` is one JSON document, and `jsonl` is one JSON object per line for list commands. |
| `--json` | Command-local alias for JSON output on commands that expose it. |

Commands that accept caller payloads use one shared input contract:

| Option | Meaning |
| --- | --- |
| `--input`, `-i` | Inline input document. |
| `--input-file` | Read input from a file, or `-` for stdin. Mutually exclusive with `--input`. |
| `--input-encoding=json|raw|base64` | Decode input as JSON, pass raw text as one argument, or decode base64 as one argument. Defaults to `json`. |

JSON input for workflow starts, signals, queries, updates, schedules, and
activity completions represents the v2 positional argument array. Raw and
base64 input become one positional argument so the server still receives the
canonical `input` array.

## Connection And Diagnostics

| Command | Purpose | Important options |
| --- | --- | --- |
| `dw --version` | Print CLI build identity. When `DW_ENV` or `DURABLE_WORKFLOW_SERVER_URL` selects a target, also performs a short compatibility probe. | `-V`, `--version` |
| `dw server:health` | Check server health and auth reachability. | global options, `--json` |
| `dw server:info` | Show server version, the role-topology manifest, protocol manifests, request contract, worker protocol, worker-fleet facts, and compatibility metadata. | global options, `--json` |
| `dw doctor` | Explain the resolved profile/server/token/TLS state, remote compatibility warnings, and next steps. | global options, `--json` |
| `dw debug workflow <workflow-id>` | Capture stuck-run diagnostics for one workflow: state, pending tasks, queue facts, failures, and compatibility metadata. | `--run-id`, global options, `--json` |
| `dw server:start-dev` | Start a local development server for smoke work. | `--port`, `--db=sqlite|mysql|pgsql` |
| `dw watch workflow <workflow-id>` | Poll a workflow until terminal or until a configured polling limit. | `--run-id`, `--interval`, `--max-polls`, global options |
| `dw upgrade` | Replace the running standalone `dw` binary with a newer (or pinned) release. Refuses to rewrite Composer vendor, Homebrew cellar, and PHAR installs. | `--tag`, `--dry-run`, `--force`, `--output=table|json` |

Use `server:info` when validating contract shape, `doctor` when explaining why
a CLI cannot talk to a server, and `debug workflow` when support needs one
machine-readable run capture.

### Server Info And Role Topology

`dw server:info` is the CLI surface for `GET /api/cluster/info`. In table mode
it includes a `Topology:` section that summarizes the role-topology contract
the server published for the responding node:

- `Supported Shapes` and `Current Shape` identify which deployment shapes are
  legal and which one this node is serving right now.
- `Current Process Class` and `Current Roles` name the process class and role
  bundle on the responding node.
- `Matching Role`, `Matching Partitions`, and `Matching Backpressure` expose
  the matching-role shape, wake ownership, task-dispatch mode, and partition
  primitives that determine how ready work is claimed.
- `Matching Discovery Limits` summarizes the frozen numeric matching-role
  contract — `poll_batch_cap`, `availability_ceiling_seconds`,
  `wake_signal_ttl_seconds`, `workflow_task_lease_seconds`, and
  `activity_task_lease_seconds` — so operators can verify the deployment
  matches the documented matching-role contract without grepping the package
  source.
- `Current Write Boundaries` lists the durable write surfaces currently owned
  by the roles on this node.
- `Scaling Boundaries` and `Failure Domains` tell operators what load driver
  or first failure signal to expect for each role.

Use `--output=json` when scripts need the raw manifest. `topology.schema` and
`topology.version` pin the manifest contract revision so scripts can detect
shape drift; `topology.execution_mode` reports the dispatch mode the
responding node serves. The stable machine fields live under `topology`,
including `supported_shapes`, `current_shape`, `current_process_class`,
`current_roles`, `matching_role`, `role_catalog`, `authority_boundaries`,
`authority_surfaces`, `supported_topologies`, `scaling_boundaries`, and
`failure_domains`. The `topology.matching_role` block also publishes
`partition_primitives` and `backpressure_model` so scripts can check which
routing axes the responding node uses for ready-task discovery without
parsing prose. `topology.matching_role.discovery_limits` exposes the frozen
numeric matching-role contract — `poll_batch_cap`,
`availability_ceiling_seconds`, `wake_signal_ttl_seconds`,
`workflow_task_lease_seconds`, and `activity_task_lease_seconds` — so scripts
can pin the workflow package's matching-role numbers without scraping a
human-readable section.

```bash
dw server:info --output=json \
  | jq '.topology | {current_shape, current_process_class, current_roles, matching_role, scaling_boundaries, failure_domains}'
```

For the meaning of those fields, see
[Server Role Topology](/docs/polyglot/server-role-topology).

### Server Info And Coordination Health

`dw server:info` also publishes the server's all-namespaces rollout-safety
verdict so operators can read coordination health without standing up a
separate health surface. In table mode the CLI renders a `Coordination
Health:` section under `Topology:`. In `--output=json` the same data lives
under `coordination_health`, with these stable machine fields:

- `coordination_health.schema` and `coordination_health.version` pin the
  manifest contract revision.
- `coordination_health.namespace_scope` reports whether the verdict covers
  one namespace or the whole fleet.
- `coordination_health.status` and `coordination_health.http_status` report
  the top-level verdict and HTTP gate the server applies to readiness.
- `coordination_health.generated_at` records when the snapshot was taken.
- `coordination_health.categories` summarizes per-category counts (such as
  `correctness`, `safety`, `routing`).
- `coordination_health.warning_checks` and `coordination_health.error_checks`
  list the check names that pushed the verdict to warning or error so
  scripts can branch on the failing surfaces.
- `coordination_health.checks[]` is the per-check detail array, where each
  entry pins `name`, `status`, `category`, and `message` automation can use
  to explain a degraded verdict. The frozen check inventory always includes
  `worker_compatibility`, `task_transport`, `routing_health`,
  `durable_resume_paths`, the projection/scheduler checks, and `activity_path`.
  `activity_path` is the activity-side counterpart of `task_transport`: it
  surfaces activity executions whose schedule-to-start, start-to-close,
  schedule-to-close, or heartbeat deadline has passed without enforcement, plus
  the sustained activity retry backlog. Renaming `activity_path` is a
  protocol-level change.

```bash
dw server:info --output=json \
  | jq '.coordination_health | {status, http_status, namespace_scope, warning_checks, error_checks}'
```

For the underlying readiness contract behind these fields, see the
[Server API Reference](/docs/polyglot/server-api-reference).

### Self-Upgrade

`dw upgrade` downloads the matching platform asset from the
`durable-workflow/cli` GitHub release, verifies it against the release's
`SHA256SUMS`, and replaces the running binary only on a successful checksum
match. Use `--tag` to pin to a specific release tag, `--dry-run` to resolve
the target release and print the asset URLs without downloading, and `--force`
to re-download and replace even when the current and target versions match.

The command refuses to rewrite installations managed by another tool. JSON
output uses a stable `status` field so automation can branch without parsing
prose:

| `status` | Meaning |
| --- | --- |
| `upgraded` | Binary replaced with `target_version`. |
| `noop` | `current_version` already matches `target_version`; `--force` bypasses this. |
| `dry-run` | `--dry-run` resolved `target_version`, `asset_url`, and `checksum_url` without downloading. |
| `refused` | Install kind is not a standalone release binary (Composer vendor, Homebrew cellar, or PHAR), or the platform has no published asset. |
| `permission-denied` | The install directory is not writable; the payload includes a `hint` with the recommended next step. |
| `error` | Release catalog fetch, checksum mismatch, or filesystem error. |

For refusals, the JSON payload's `installation.kind` identifies the managing
tool (`composer-vendor`, `homebrew`, `phar`, or `binary`) and `reason` names
the right owner to use instead. For public, source-free automation, reinstall
a pinned release through `https://durable-workflow.com/install.sh`; for
tap-managed Homebrew installs, use `brew upgrade durable-workflow/tap/dw`.

## Environment Profiles

Profiles live under `~/.config/dw/config.json`, or
`$XDG_CONFIG_HOME/dw/config.json` when set. Set `DW_CONFIG_HOME` to isolate a
test or CI profile directory.

| Command | Purpose | Important options |
| --- | --- | --- |
| `dw env:set <name>` | Create or update a named profile. | `--server`, `--namespace`, `--token-env`, `--token`, `--tls-verify=true|false`, `--profile-output=table|json|jsonl`, `--make-default`, `--json` |
| `dw env:list` | List profiles with literal tokens redacted by default. | `--show-token`, `--json`, `--output=jsonl` |
| `dw env:show [name]` | Show one profile, defaulting to the current profile. | `--show-token`, `--json` |
| `dw env:use <name>` | Set the default profile. Unknown names fail. | `--json` |
| `dw env:delete <name>` | Delete a profile. | `--json` |

Prefer `--token-env=NAME` for production profiles so secrets stay in the
runtime environment and not in the profile file.

## Workflow Commands

| Command | Purpose | Important options |
| --- | --- | --- |
| `dw workflow:start` | Start a workflow through the control plane. | `--type`, `--workflow-id`, `--business-key`, `--task-queue`, `--duplicate-policy`, `--memo`, `--search-attr key=value`, `--execution-timeout`, `--run-timeout`, `--wait`, input options, `--json` |
| `dw workflow:list` | List workflow instances. | `--type`, `--status`, `--query`, `--limit`, global output options |
| `dw workflow:describe <workflow-id>` | Describe the current or selected run. | `--run-id`, `--follow`, `--json` |
| `dw workflow:list-runs <workflow-id>` | List runs for a workflow instance. | `--json` |
| `dw workflow:show-run <workflow-id> <run-id>` | Show one run. | `--follow`, `--json` |
| `dw workflow:history <workflow-id> <run-id>` | Read run history events. | `--follow`, `--page-size`, `--json` |
| `dw workflow:history-export <workflow-id> <run-id>` | Export the archival run-history payload. | `--output-file`, global options |
| `dw workflow:signal <workflow-id> <signal-name>` | Send a signal. | `--run-id`, input options, `--json` |
| `dw workflow:query <workflow-id> <query-name>` | Execute a read-only workflow query. | `--run-id`, input options, `--json` |
| `dw workflow:update <workflow-id> <update-name>` | Submit or execute a workflow update. | `--wait=accepted|completed`, `--run-id`, input options, `--json` |
| `dw workflow:cancel [workflow-id]` | Request cancellation for one workflow or a batch query. | `--reason`, `--run-id`, `--all-matching`, `--type`, `--status`, `--limit`, `--yes`, `--json` |
| `dw workflow:terminate <workflow-id>` | Force terminate a workflow. | `--reason`, `--run-id`, `--json` |
| `dw workflow:repair <workflow-id>` | Ask the server to repair a stuck or retryable run. | `--json` |
| `dw workflow:archive <workflow-id>` | Archive a closed run. | `--reason`, `--json` |

Examples:

```bash
dw workflow:start \
  --type=App\\Workflows\\ProcessOrder \
  --workflow-id=order-123 \
  --task-queue=payments \
  --input='["order-123"]' \
  --json

dw workflow:update order-123 approve --wait=completed --input='["manager"]'
dw workflow:cancel --all-matching='WorkflowType = "ImportJob"' --limit=25 --yes
```

`workflow:start`, `workflow:signal`, `workflow:query`, and `workflow:update`
validate canonical request fields against the server-published request
contract. Non-canonical legacy aliases are rejected before the request is sent.

## Bridge Adapter Commands

Bridge commands are bounded ingress and handoff tools for integration events.
They call the server bridge-adapter surface and return the
`durable-workflow.v2.bridge-adapter-outcome.contract` shape in JSON mode.
They do not execute workflow code or own workflow state transitions.

| Command | Purpose | Important options |
| --- | --- | --- |
| `dw bridge:webhook <adapter>` | Send one webhook bridge event that starts, signals, or updates a workflow through the control plane. | `--action=start_workflow|signal_workflow|update_workflow`, `--idempotency-key`, `--target`, input options, `--correlation`, `--json` |

Examples:

```bash
dw bridge:webhook stripe \
  --action=start_workflow \
  --idempotency-key=stripe-event-1001 \
  --target='{"workflow_type":"orders.fulfillment","task_queue":"external-workflows","business_key":"order-1001"}' \
  --input='{"order_id":"order-1001"}' \
  --json

dw bridge:webhook pagerduty \
  --action=signal_workflow \
  --idempotency-key=pd-event-3003 \
  --target='{"workflow_id":"wf-remediation-42","signal_name":"incident_escalated"}' \
  --input='{"severity":"critical"}'
```

Use the bridge outcome fields instead of inferring behavior from HTTP status
alone. `outcome`, `reason`, `control_plane_outcome`, `idempotency_key`, and
the redacted `target` summary are the automation contract for duplicates,
routing misses, malformed payloads, and accepted handoffs.

## Schedule Commands

| Command | Purpose | Important options |
| --- | --- | --- |
| `dw schedule:create` | Create a schedule. | `--schedule-id`, `--workflow-type`, `--cron`, `--interval`, `--task-queue`, `--timezone`, `--execution-timeout`, `--run-timeout`, `--overlap-policy`, `--jitter`, `--max-runs`, `--paused`, `--note`, input options, `--json` |
| `dw schedule:list` | List schedules. | global output options |
| `dw schedule:describe <schedule-id>` | Describe one schedule. | `--json` |
| `dw schedule:update <schedule-id>` | Update schedule spec or workflow input. | schedule create options, input options, `--json` |
| `dw schedule:pause <schedule-id>` | Pause a schedule. | `--note`, `--json` |
| `dw schedule:resume <schedule-id>` | Resume a schedule. | `--note`, `--json` |
| `dw schedule:trigger <schedule-id>` | Trigger a schedule immediately. | `--overlap-policy`, `--json` |
| `dw schedule:backfill <schedule-id>` | Backfill a time window. | `--start-time`, `--end-time`, `--overlap-policy`, `--json` |
| `dw schedule:delete <schedule-id>` | Delete a schedule. | `--json` |

Use either `--cron` or `--interval` when creating interval-based schedules.
Use `--paused` for deploy-time registration that should not start work yet.

## Worker And Task Queue Commands

| Command | Purpose | Important options |
| --- | --- | --- |
| `dw worker:register [worker-id]` | Register a worker with capacity and compatibility metadata. | `--task-queue`, `--runtime`, `--sdk-version`, `--build-id`, `--workflow-type`, `--activity-type`, `--max-workflow-tasks`, `--max-activity-tasks`, `--json` |
| `dw worker:list` | List workers. | `--task-queue`, `--status`, global output options |
| `dw worker:describe <worker-id>` | Describe one worker. | `--json` |
| `dw worker:deregister <worker-id>` | Deregister one worker. | `--json` |
| `dw task-queue:list` | List active task queues and admission status. | global output options |
| `dw task-queue:describe <queue>` | Describe worker capacity, leases, dispatch budgets, and pending query-task capacity. | `--json` |
| `dw task-queue:build-ids <queue>` | Inspect per-build-id cohort state and rollout status for one queue. | `--json` |
| `dw task-queue:drain <queue>` | Mark a build-id cohort as draining so it stops claiming new tasks. | `--build-id <value>`, `--unversioned`, `--json` |
| `dw task-queue:resume <queue>` | Clear a previous drain so the cohort can claim new tasks again. | `--build-id <value>`, `--unversioned`, `--json` |

The task queue commands are the preferred operator view for throttling,
capacity, and no-worker diagnoses. See
[Task Queue Admission](/docs/polyglot/task-queue-admission) for the
server-side policy behind those fields and
[Worker Build-Id Rollout](/docs/polyglot/worker-build-id-rollout) for the
full unversioned-to-versioned cutover, canary, drain, and rollback lifecycle.

`dw task-queue:drain` and `dw task-queue:resume` both require either
`--build-id <value>` to target a specific build cohort or `--unversioned` to
target the cohort of workers registered without a `build_id`. Combining the
two fails fast with an invalid-option error. Both commands are idempotent:
repeated drains do not shift the recorded `drained_at` timestamp, and
resuming an already-active cohort is a no-op.

## Worker Protocol Commands

These commands are low-level protocol tools for diagnostics, smoke tests, and
non-SDK worker experiments. Normal PHP and Python workers should use their SDK
worker loops.

| Command | Purpose | Important options |
| --- | --- | --- |
| `dw workflow-task:poll <worker-id>` | Poll one workflow task. | `--task-queue`, `--build-id`, `--poll-request-id`, `--history-page-size`, `--accept-history-encoding`, `--json` |
| `dw workflow-task:history <task-id> <page-token>` | Fetch the next history page for a leased workflow task. | `--lease-owner`, `--attempt`, `--json` |
| `dw workflow-task:complete <task-id> <attempt>` | Complete one workflow task with command payloads. | `--lease-owner`, `--complete-result`, `--command`, `--json` |
| `dw workflow-task:fail <task-id> <attempt>` | Report workflow-task execution failure for retry or diagnosis. Distinct from completing a task with a `fail_workflow` command. | `--lease-owner`, `--message`, `--type`, `--stack-trace`, `--json` |
| `dw query-task:poll <worker-id>` | Poll and lease one routed workflow query task. | `--task-queue`, `--json` |
| `dw query-task:complete <query-task-id> <attempt>` | Complete a leased query task with a JSON result and matching envelope. | `--lease-owner`, `--result`, `--json` |
| `dw query-task:fail <query-task-id> <attempt>` | Report a leased query task failure with a machine-readable reason. | `--lease-owner`, `--message`, `--reason`, `--type`, `--stack-trace`, `--json` |
| `dw activity:complete <task-id> <attempt-id>` | Complete one leased activity attempt. | `--lease-owner`, input options, `--json` |
| `dw activity:fail <task-id> <attempt-id>` | Fail one leased activity attempt. | `--lease-owner`, `--message`, `--type`, `--non-retryable`, `--json` |

Use `workflow-task:fail` for worker-side execution failures such as replay
mismatches or deserialization errors; it targets
`POST /worker/workflow-tasks/{taskId}/fail`. Completing a task with a workflow
command that fails the workflow is a separate concern and goes through
`workflow-task:complete` with the appropriate command payload.

`query-task:poll`, `query-task:complete`, and `query-task:fail` drive the
routed-query worker surface at `/worker/query-tasks/…`. Normal workers pull
query tasks through their SDK query handler; the CLI surface is for
diagnostics, CLI/SDK parity checks, and non-SDK worker experiments. Query-task
failures default to `--reason=query_rejected`; use a stable reason string such
as `unknown_query`, `decode_failure`, or a runtime-specific identifier so
callers can distinguish expected rejections from runtime errors.

### Workflow Task History Pages

`workflow-task:history` is the CLI diagnostic wrapper around the worker-plane
history-page endpoint. Use it only after `workflow-task:poll` returns a leased
workflow task with `next_history_page_token`; normal workers should let their
SDK fetch extra history pages.

<!-- docs-example id="cli.workflow-task-history.command" -->
```bash
dw workflow-task:history workflow-task-01 history-page-2 \
  --lease-owner=python-worker-1 \
  --attempt=2 \
  --json
```

JSON output is the server response without field renaming:

<!-- docs-example id="cli.workflow-task-history.response" -->
```json
{
  "history_events": [
    {"event_id": 2, "event_type": "ActivityScheduled", "payload": {}}
  ],
  "total_history_events": 4,
  "next_history_page_token": "history-page-3"
}
```

Automation should read `history_events`, `total_history_events`, and
`next_history_page_token`. The worker-history endpoint does not use the
control-plane run-history fields `events` or `next_page_token`.

## Namespace And Search Attribute Commands

| Command | Purpose | Important options |
| --- | --- | --- |
| `dw namespace:list` | List namespaces. | global output options |
| `dw namespace:create <name>` | Create a namespace. | `--description`, `--retention`, `--json` |
| `dw namespace:describe <name>` | Describe one namespace. | `--json` |
| `dw namespace:update <name>` | Update namespace metadata. | `--description`, `--retention`, `--json` |
| `dw namespace:set-storage-driver <name> <driver>` | Configure the namespace external payload storage policy used when encoded payloads exceed the offload threshold. | `--threshold-bytes`, `--uri`, `--disk`, `--bucket`, `--prefix`, `--region`, `--endpoint`, `--auth-profile`, `--disable`, `--json` |
| `dw storage:test` | Round-trip a small inline payload and a large offloaded payload through the selected namespace storage policy or driver override. | `--driver=local|s3|gcs|azure`, `--small-bytes`, `--large-bytes`, global options, `--json` |
| `dw search-attribute:list` | List search attributes. | global output options |
| `dw search-attribute:create <name> <type>` | Register a search attribute. | `--json` |
| `dw search-attribute:delete <name>` | Delete a search attribute. | `--json` |

Search attribute types are server-compatible values such as `keyword`, `text`,
`int`, `double`, `bool`, `datetime`, and `keyword_list`.

External payload storage commands call the server's namespace storage API. The
driver argument is one of `local`, `s3`, `gcs`, or `azure`; object-store drivers
use server-side filesystem configuration, so CLI flags describe the namespace
policy rather than carrying provider credentials. Use `--disk` to bind the
`s3`, `gcs`, or `azure` driver to a named server-side filesystem disk that
holds the actual provider credentials. Use `--disable` to keep the policy
record while preventing new offloads.

Examples:

```bash
dw namespace:set-storage-driver billing s3 \
  --disk=external-payload-objects \
  --bucket=dw-payloads \
  --prefix=billing/ \
  --threshold-bytes=2097152 \
  --json

dw namespace:set-storage-driver dev local \
  --uri=file:///var/lib/durable-workflow/payloads

dw storage:test --namespace=billing --large-bytes=2097152 --json
dw storage:test --driver=s3 --small-bytes=128 --large-bytes=3145728 --json
```

In JSON mode, `namespace:set-storage-driver` returns the namespace payload with
its `external_payload_storage` policy. `storage:test` returns the diagnostic
status plus `small_payload` and `large_payload` result objects; automation
should branch on those fields instead of parsing the human table.

The diagnostic is also the fastest operator check for the reference-envelope
contract. A passing large-payload result proves the selected policy can write
encoded bytes, return a `durable-workflow.v2.external-payload-reference.v1`
reference, read the object back, and verify `size_bytes` plus `sha256`. A failed
diagnostic should be treated as a storage-policy problem before workflows are
allowed to offload payloads through that namespace.

## System Commands

System commands expose server maintenance passes as explicit, scriptable
operations.

| Command | Purpose | Important options |
| --- | --- | --- |
| `dw system:repair-status` | Show workflow repair backlog/status. | `--json` |
| `dw system:repair-pass` | Run one repair pass. | `--run-id`, `--limit`, `--json` |
| `dw system:activity-timeout-status` | Show activity timeout backlog/status. | `--json` |
| `dw system:activity-timeout-pass` | Run one activity timeout pass. | `--task-id`, `--limit`, `--json` |
| `dw system:retention-status` | Show retention backlog/status. | `--json` |
| `dw system:retention-pass` | Run one retention cleanup pass. | `--run-id`, `--limit`, `--json` |

Prefer status commands before pass commands in runbooks so operators can see
the pending scope before mutating server state.

## Schema Commands

| Command | Purpose | Important options |
| --- | --- | --- |
| `dw schema:list` | List published machine-readable schemas. | no server connection required |
| `dw schema:show <command-name>` | Show the bundled JSON Schema for one command output. | `--output=json|jsonl`; no server connection required |
| `dw schema:manifest` | Show the schema manifest. | no server connection required |

Schema commands are useful when an AI client or CI job needs the current
control-plane, response, or output contract without scraping prose docs.
The [current v4 manifest](https://durable-workflow.github.io/cli-json-envelopes/v4/manifest.json)
binds every JSON envelope and record-level JSONL schema by public resolver and
digest. Its workflow start/run and query/update payload fields accept only the
`avro` codec. The retained
[v3 manifest](https://durable-workflow.github.io/cli-json-envelopes/v3/manifest.json)
and [v2 manifest](https://durable-workflow.github.io/cli-json-envelopes/v2/manifest.json)
remain available with their original bytes for revision-pinned consumers.

## Output And Exit Contract

Use `--output=json` when a command returns one object and `--output=jsonl` when
a list command feeds a stream processor. Human tables are allowed to improve
over time; JSON field names are the automation contract.

All commands use the stable exit-code policy documented in the
[CLI guide](/docs/polyglot/cli#exit-codes):

| Code | Meaning |
| --- | --- |
| `0` | Success. |
| `1` | Command ran but failed generically. |
| `2` | Invalid local usage or validation error. |
| `3` | Network or transport failure. |
| `4` | Authentication or authorization failure. |
| `5` | Resource not found. |
| `6` | Server-side `5xx` failure. |
| `7` | Timeout. |

For support bundles, collect `dw doctor --output=json`, `dw server:info
--output=json`, and `dw debug workflow <workflow-id> --output=json`.

## Related Guides

- [CLI](./cli.mdx) covers installation, profile setup, and exit-code behavior.
- [Server](./server.md) documents the HTTP control plane that server-backed
  commands call.
- [Client and Worker Capabilities](./cli-python-parity.md) compares
  CLI, PHP, Python, and Rust client and worker surfaces.

<!-- Source: docs/polyglot/invocable-carrier.md -->

# Invocable HTTP Carrier

The invocable HTTP carrier is the first concrete carrier defined under the
[external execution surface](./external-execution.md). It is published from
`GET /api/cluster/info` at
`worker_protocol.invocable_carrier_contract` with
`schema: durable-workflow.v2.invocable-carrier.contract`, `version: 1`, and
`carrier_type: invocable_http`.

When a configured handler mapping resolves to an `invocable_http` carrier, the
server invokes the configured endpoint with the carrier-neutral
[external task input envelope](./external-execution.md#published-contract-seams)
and reconciles the response against the
[external task result envelope](./external-execution.md#published-contract-seams).
Workflow tasks, signal/update ordering, replay, and history mutation stay
inside the server. The carrier only moves declared input and result envelopes
across an HTTPS boundary.

## What It Is

The invocable HTTP carrier exists for activity-grade work that an operator
team would rather host as an HTTPS handler than as a long-poll worker:

- operator maintenance activities and platform automation hosted in an internal
  HTTP service
- bounded integration handoffs that already terminate at an HTTPS endpoint
- serverless or container-based activity handlers that expose a single POST
  route per activity type

Activity execution is delegated to the configured endpoint. Durable workflow
state, history, and the activity-completion contract remain server-owned.

## Activity-Only Scope

The published manifest fixes the carrier scope to `task_kinds: [activity_task]`
and names explicit non-goals so the boundary cannot drift through configuration:

- `workflow_task_execution` — workflow tasks remain on real workflow runtimes
- `workflow_replay` — the carrier never replays workflow history
- `history_mutation` — handlers may not edit event history directly
- `generic_webhook_ingress` — generic ingress goes through a bridge adapter,
  not through this carrier

A handler mapping with a non-activity `kind`, or a carrier mapping with a
non-`activity_task` capability, fails configuration validation with
`invalid_invocable_carrier_scope` before it can appear on the activity poll
response.

## HTTPS Target And Method

Carrier `target_fields` are validated by the server before a mapping is
exposed:

| Field | Required | Allowed | Notes |
| --- | --- | --- | --- |
| `url` | yes | absolute HTTPS URL, or HTTP for loopback dev (`localhost`, `127.0.0.0/8`, `::1`) | URL credentials (`scheme://user:pass@host`) are forbidden. |
| `method` | no | `POST` | Defaults to `POST`. No other HTTP method is accepted. |
| `timeout_seconds` | no | integer 1–900 | Transport deadline for one handler attempt. The task deadline is enforced separately by the server/runtime. |
| `retry_policy` | no | object (see below) | Carrier-owned transport retry budget. |

Invalid targets fail closed with `invalid_carrier_target` and never appear in
discovery output.

## Auth Model

Non-loopback `invocable_http` mappings must resolve an `auth_ref` from the
external executor configuration. Unauthenticated invocable HTTP is allowed
only against loopback HTTP targets so a developer can iterate locally.

- `auth_refs` are declared once at the top of the config and referenced from
  defaults or per-mapping. Supported types include `profile`, `env`,
  `token_file`, `mtls`, and `signed_headers`.
- A mapping that resolves to a non-loopback target without an effective
  `auth_ref` fails configuration validation with `missing_invocable_auth_ref`.
- Tokens, secrets, signatures, and authorization headers are never echoed in
  cluster diagnostics, in `dw server:info` output, or on the activity poll
  response. The mapping diagnostics report which `auth_ref` resolved and the
  redacted summary, never the credential value.

The intent is that an operator can read the redacted runtime diagnostics and
confirm which auth secret will be used, without the server ever exposing the
secret itself.

## Transport Retry Versus Durable Activity Retry

The optional `retry_policy` on an invocable carrier is **transport-only**. It
governs the carrier's HTTP delivery before the handler reports a result:

| `retry_policy` field | Allowed | Default | Meaning |
| --- | --- | --- | --- |
| `max_attempts` | integer 1–5 | `1` | Maximum HTTP delivery attempts the carrier will make for one task lease. |
| `backoff_seconds` | array of integers, each 0–300, up to 5 entries | `[]` | Per-attempt backoff before the next HTTP try. |
| `retryable_status_codes` | subset of `[408, 425, 429, "5xx"]` | `[408, 429, "5xx"]` | Response codes that count as transport-retryable. |

Transport retries never become history events. The server only learns about
the carrier's final attempt — either the structured success/failure envelope
the handler returned, or a transport timeout that maps to
`failure.kind=timeout, classification=deadline_exceeded`, or one of the
malformed-output paths.

Once the handler reports a result, the **durable activity retry policy
remains the server/runtime authority**. Task-level retry, scheduling, and
backoff continue to follow the activity retry policy declared on the
workflow side. The carrier's `retry_policy` does not extend, override, or
substitute for it.

## Request And Response Envelope

The server sends the carrier-neutral input envelope and expects the
carrier-neutral result envelope back:

| Direction | Content type | Schema |
| --- | --- | --- |
| Request | `application/vnd.durable-workflow.external-task-input+json` | `external_task_input_contract` |
| Response | `application/vnd.durable-workflow.external-task-result+json` | `external_task_result_contract` |

The handler must preserve `task.id`, `task.attempt`, and
`task.idempotency_key` from the input envelope, and must respond with the
declared success or failure envelope (or fail-closed by emitting a
malformed-output mapping).

The carrier maps transport facts to result facts deterministically:

- transport timeout → `failure.kind=timeout`,
  `classification=deadline_exceeded`
- non-2xx response without a valid result envelope → `malformed_output`
- invalid JSON or schema mismatch on the response → `malformed_output`
- result envelope referencing an unsupported payload reference →
  `unsupported_payload`

Handlers must be idempotent. The same `task.id` and
`task.idempotency_key` may arrive more than once if the carrier retries
transport delivery or the runtime redelivers an unfinished lease.

## Configuration Example

The server reads handler mappings from
`DW_EXTERNAL_EXECUTOR_CONFIG_PATH`, optionally selecting an overlay with
`DW_EXTERNAL_EXECUTOR_CONFIG_OVERLAY`. See the
[Server Config Reference](./server-config-reference.md) for the full list of
environment variables.

Below is a minimal `durable-workflow.external-executor.config` document that
registers one `invocable_http` carrier and one activity mapping:

```json
{
  "schema": "durable-workflow.external-executor.config",
  "version": 1,
  "defaults": {
    "profile": "prod",
    "namespace": "operations",
    "task_queue": "operator-tasks",
    "auth_ref": "handler-token"
  },
  "auth_refs": {
    "handler-token": {
      "type": "env",
      "env": "DURABLE_WORKFLOW_HANDLER_TOKEN"
    }
  },
  "carriers": {
    "ops-invocable": {
      "type": "invocable_http",
      "url": "https://handlers.example.com/durable/activity",
      "method": "POST",
      "timeout_seconds": 60,
      "capabilities": ["activity_task"],
      "retry_policy": {
        "max_attempts": 3,
        "backoff_seconds": [2, 5],
        "retryable_status_codes": [408, 429, "5xx"]
      }
    }
  },
  "mappings": [
    {
      "name": "billing.reconcile-ledger",
      "kind": "activity",
      "task_queue": "operator-tasks",
      "activity_type": "billing.reconcile-ledger",
      "carrier": "ops-invocable",
      "handler": "billing.reconcile-ledger",
      "timeout_seconds": 60
    }
  ]
}
```

A loopback dev variant uses `"url": "http://127.0.0.1:8080/durable/activity"`
and may omit `auth_ref` because loopback HTTP is the one allowed
unauthenticated path.

## Inspection And Diagnostics

Two CLI commands surface the published invocable carrier contract from a
running server:

- `dw server:info` renders the schema name, contract version, carrier type,
  task kinds, allowed request/response content types, and the redacted
  external executor mapping diagnostics. Use it to confirm the server has
  loaded the expected mappings.
- `dw doctor` runs the cluster-info diagnostic, prints the
  `invocable_carrier_contract` block, and surfaces any
  `invalid_carrier_target`, `missing_invocable_auth_ref`,
  `invalid_invocable_carrier_scope`, `unknown_carrier`, `unknown_auth_ref`,
  or `unknown_handler` errors that blocked a mapping from being advertised.

The activity poll response itself reports the resolved mapping, the carrier
target (with auth redacted), and the effective `retry_policy`. Workers and
operator tooling read these fields as the source of truth instead of caching
their own copy of the configuration file.

## Coexistence And Rollout

The invocable carrier rollout boundary is published in the same manifest:

- A poll-based carrier and an `invocable_http` carrier may share a queue only
  when the mappings are activity-type specific. Two mappings cannot claim the
  same `(task_queue, activity_type)` pair.
- Operators must remove or overlay-disable invocable mappings before deleting
  the credentials they reference. Pulling credentials before the mapping is
  drained will move every new attempt to `failure.kind=auth`, not silently
  succeed.
- Carrier `retry_policy` is purely transport. The durable activity retry
  policy remains the only authority over how many task attempts the workflow
  observes.

## Related Surfaces

- [External Execution Surface](./external-execution.md) — the carrier-neutral
  product boundary that this carrier implements.
- [PHP Invocable Activity Handler](./invocable-php-handler.md) — the PHP
  helper an external process uses to parse input envelopes, dispatch
  activity callables, and emit the structured result envelopes this carrier
  expects.
- [Server Config Reference](./server-config-reference.md) — environment
  variables for `DW_EXTERNAL_EXECUTOR_CONFIG_PATH` and
  `DW_EXTERNAL_EXECUTOR_CONFIG_OVERLAY`.
- [External Payload Storage](../features/external-payload-storage.md) — how
  oversized request or result payloads are offloaded to a configured driver
  and represented as a verifiable reference envelope inside the same input
  and result schemas the invocable carrier uses.
- [Worker Protocol](./worker-protocol.md) — the broader worker-plane contract
  that publishes this carrier alongside poll-based handler shapes.

<!-- Source: docs/polyglot/namespace-auth-workers.md -->

# Namespace, Auth, And Worker Registration

Use this reference when provisioning a standalone server, issuing automation
credentials, or implementing a worker runtime. The server has three separate
contracts that must line up before work can flow:

- the request names a namespace through `X-Namespace`, `?namespace=`, or the
  server default namespace
- the credential has the role required by the route family
- workers register the same namespace, task queue, runtime, type keys, and
  capacity that workflow starts and task polls use

## Request Authority

Durable Workflow treats namespace, auth role, and protocol version as request
authority. Do not infer them from workflow ids, task ids, or display labels.

| Request family | Required credential role | Required version header | Namespace source |
| --- | --- | --- | --- |
| Discovery `GET /api/cluster/info` | `worker`, `operator`, or `admin` | none | optional `X-Namespace` for context |
| Namespace list/describe | `operator` or `admin` | `X-Durable-Workflow-Control-Plane-Version: 2` | route target or request context |
| Namespace create/update/storage policy | `admin` | `X-Durable-Workflow-Control-Plane-Version: 2` | route target or request body |
| Workflow, schedule, task queue, bridge adapter, worker visibility | `operator` or `admin` | `X-Durable-Workflow-Control-Plane-Version: 2` | `X-Namespace`, `?namespace=`, then default |
| System health, metrics, passes, and storage tests | `admin` | `X-Durable-Workflow-Control-Plane-Version: 2` | `X-Namespace`, `?namespace=`, then default |
| Worker registration, polling, heartbeats, completion | `worker` | `X-Durable-Workflow-Protocol-Version: 1.0` | `X-Namespace`, `?namespace=`, then default |

The server checks the route role before namespace existence on role-gated
endpoints. A wrong-role token receives an auth failure instead of a namespace
existence signal. After the role and version checks pass, namespace-scoped
routes reject unknown namespaces with `reason: "namespace_not_found"`.

## Auth Roles

Token auth is the default production path:

```bash
DW_AUTH_DRIVER=token
DW_WORKER_TOKEN=worker-secret
DW_OPERATOR_TOKEN=operator-secret
DW_ADMIN_TOKEN=admin-secret
```

If a deployment uses one shared `DW_AUTH_TOKEN`, that token effectively has all
route permissions. Prefer role-scoped tokens for production so a worker process
cannot mutate namespaces or run system maintenance passes.

The same role split exists for signature auth:

```bash
DW_AUTH_DRIVER=signature
DW_WORKER_SIGNATURE_KEY=worker-signature-secret
DW_OPERATOR_SIGNATURE_KEY=operator-signature-secret
DW_ADMIN_SIGNATURE_KEY=admin-signature-secret
```

`DW_AUTH_DRIVER=none` is for local development only. It removes the auth
boundary from every route and should never be exposed outside a trusted local
network.

## Namespace Contract

The bootstrap process seeds the default namespace. Set
`DW_DEFAULT_NAMESPACE` when omitted namespace headers should resolve somewhere
other than `default`:

```bash
DW_DEFAULT_NAMESPACE=default
```

Create every tenant or environment namespace before directing clients or
workers at it:

```bash
curl -sS -X POST "$DURABLE_WORKFLOW_SERVER_URL/api/namespaces" \
  -H "Authorization: Bearer $DW_ADMIN_TOKEN" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "orders-prod",
    "description": "production order workflows",
    "retention_days": 90
  }'
```

Namespace names are normalized to lowercase and may contain letters, numbers,
dot, underscore, and dash. They are unique after normalization. For example,
creating `Production` and then `production` is a conflict with
`reason: "namespace_already_exists"`.

A normal workflow start names the namespace with `X-Namespace`:

```bash
curl -sS -X POST "$DURABLE_WORKFLOW_SERVER_URL/api/workflows" \
  -H "Authorization: Bearer $DW_OPERATOR_TOKEN" \
  -H "X-Namespace: orders-prod" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" \
  -H "Content-Type: application/json" \
  -d '{
    "workflow_type": "orders.fulfillment",
    "workflow_id": "order-1001",
    "task_queue": "orders",
    "input": ["order-1001"]
  }'
```

Control-plane reads, workflow commands, schedule operations, search-attribute
operations, task-queue visibility, and worker visibility use that same
namespace context. Namespace administration routes themselves are the
exception: they target the namespace in the route or request body and do not
require that namespace to already exist before create or describe can run.

## Worker Registration Contract

Every worker process must register before polling. Registration is namespaced,
so the tuple `(namespace, worker_id)` identifies the worker record. The
registered `task_queue` must match future poll requests for that worker id.

```bash
curl -sS -X POST "$DURABLE_WORKFLOW_SERVER_URL/api/worker/register" \
  -H "Authorization: Bearer $DW_WORKER_TOKEN" \
  -H "X-Namespace: orders-prod" \
  -H "X-Durable-Workflow-Protocol-Version: 1.0" \
  -H "Content-Type: application/json" \
  -d '{
    "worker_id": "py-orders-1",
    "task_queue": "orders",
    "runtime": "python",
    "sdk_version": "<installed-sdk-version>",
    "build_id": "orders-worker-2026-04-22",
    "supported_workflow_types": ["orders.fulfillment"],
    "workflow_definition_fingerprints": {
      "orders.fulfillment": "sha256:definition-fingerprint"
    },
    "supported_activity_types": ["payments.capture"],
    "max_concurrent_workflow_tasks": 10,
    "max_concurrent_activity_tasks": 50
  }'
```

| Field | Required | Contract |
| --- | --- | --- |
| `worker_id` | no | Stable process identity. The server generates one when omitted, but long-running runtimes should set it for logs and task queue diagnostics. |
| `task_queue` | yes | Queue this worker polls. Poll requests for a different queue fail with `reason: "task_queue_mismatch"`. |
| `runtime` | yes | One of `php`, `python`, `typescript`, `go`, or `java`. |
| `sdk_version` | no | Runtime SDK version shown in worker visibility and diagnostics. |
| `build_id` | no | Deploy/build identity used by task queue build-id visibility and rollout cohorts. It should stay stable for one replay-compatible worker family. |
| `supported_workflow_types` | no | Workflow type keys this worker can replay. Empty means no workflow-type filter. |
| `workflow_definition_fingerprints` | no | Per-workflow deterministic definition fingerprints. Active re-registration with a changed fingerprint is rejected. |
| `supported_activity_types` | no | Activity type keys this worker can execute. Empty means no activity-type filter. |
| `max_concurrent_workflow_tasks` | no | Advertised local workflow-task slots. Defaults to `100`; minimum `1`. |
| `max_concurrent_activity_tasks` | no | Advertised local activity-task slots. Defaults to `100`; minimum `1`. |

Active re-registration with the same worker id is allowed when the advertised
definition fingerprints are unchanged. If a running worker changes workflow
code for an already advertised type, it must restart with a new `worker_id`;
otherwise registration fails with `reason: "workflow_definition_changed"`.

`build_id` is not just decorative metadata. It is the operator-facing cohort
identity used by task-queue rollout APIs. Keep one `build_id` for workers that
can safely replay the same in-flight work, and change it when a rollout creates
a new compatibility family. See
[Worker Compatibility and Routing](/docs/polyglot/worker-compatibility-routing)
for the pinning and rollback contract, and
[Worker Build-Id Rollout](/docs/polyglot/worker-build-id-rollout) for the
drain/resume workflow.

## Polling And Visibility

Worker poll requests must use the same namespace, worker id, and task queue
that registration used:

```bash
curl -sS -X POST "$DURABLE_WORKFLOW_SERVER_URL/api/worker/workflow-tasks/poll" \
  -H "Authorization: Bearer $DW_WORKER_TOKEN" \
  -H "X-Namespace: orders-prod" \
  -H "X-Durable-Workflow-Protocol-Version: 1.0" \
  -H "Content-Type: application/json" \
  -d '{
    "worker_id": "py-orders-1",
    "task_queue": "orders",
    "timeout_seconds": 30
  }'
```

Operators can inspect the same registration state through control-plane
visibility:

```bash
curl -sS "$DURABLE_WORKFLOW_SERVER_URL/api/task-queues/orders" \
  -H "Authorization: Bearer $DW_OPERATOR_TOKEN" \
  -H "X-Namespace: orders-prod" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2" | jq '.pollers, .admission'
```

Task queue visibility is the first place to check when workers receive no
tasks. It distinguishes missing workers from queue mismatches, unsupported
workflow/activity type filters, saturated worker slots, server active-lease
caps, dispatch-rate caps, and query-task backpressure.

Poll responses themselves expose the same machine-readable outcome through
`poll_status` when the server advertises
`worker_protocol.server_capabilities.poll_status = true` in
`GET /api/cluster/info`. Workflow-task, activity-task, and query-task poll
routes keep that field even when `task` is `null`, so worker runtimes can
branch on one stable surface:

- `leased`: a task was leased successfully.
- `empty`: no task was ready before the poll returned.
- `throttled`: queue admission limits withheld a new task for this poll.
- `unavailable`: the server could not safely coordinate the queue and returned
  a typed unavailable outcome instead of pretending the queue was empty.
- `draining`: the worker's build-id cohort is draining, so the poll fails with
  HTTP `409` and `reason: "worker_draining"` until the cohort resumes.

## Error Surface

Automation should branch on named reasons rather than prose messages.

| Reason | Where | Meaning | Operator action |
| --- | --- | --- | --- |
| `missing_control_plane_version` | control-plane route | The request omitted `X-Durable-Workflow-Control-Plane-Version: 2`. | Add the version header or upgrade the client profile. |
| `missing_protocol_version` | worker route | The request omitted `X-Durable-Workflow-Protocol-Version: 1.0`. | Fix the worker client or SDK version negotiation. |
| `namespace_not_found` | namespace-scoped route | The namespace from `X-Namespace`, query string, or server default does not exist. | Create the namespace or correct the client namespace. |
| `namespace_already_exists` | `POST /api/namespaces` | A normalized namespace name already exists. | Reuse it or choose a distinct name. |
| `task_queue_mismatch` | worker poll route | A worker id registered for one queue attempted to poll another. | Restart with a new worker id or poll the registered queue. |
| `worker_draining` | worker poll route | The worker's build-id cohort is marked draining, so it may finish in-flight work but cannot claim new tasks. | Resume the cohort for rollback, or stop the worker after its current leases drain. |
| `workflow_definition_changed` | `POST /api/worker/register` | Active worker id tried to advertise changed workflow fingerprints. | Restart the changed process with a new worker id. |
| `validation_failed` | any JSON route | A field is missing, malformed, too large, or outside allowed bounds. | Read `errors` or `validation_errors` and correct the payload. |

## See Also

- [Server API Reference](/docs/polyglot/server-api-reference)
- [Worker Protocol](/docs/polyglot/worker-protocol)
- [Task Queue Admission](/docs/polyglot/task-queue-admission)
- [CLI Command Reference](/docs/polyglot/cli-reference)

<!-- Source: docs/polyglot/python.md -->

# Python SDK

The Python SDK is a thin, async-first client for a self-hosted Durable Workflow
Server or Durable Workflow Cloud namespace runtime. It lets Python processes
start, observe, signal, and cancel workflows through the runtime's
control-plane API, and register as workers that execute workflow tasks and
activities.

## Try the local Sample App playground

For the shortest no-Cloud authoring journey, open the current Sample App
[`main` branch in GitHub Codespaces](https://codespaces.new/durable-workflow/sample-app?quickstart=1&ref=main)
and run:

<!-- docs-example id="sdk.python.sample-app-playground" -->
```bash
scripts/playground python
```

The local playground generates caller-owned workflow and activity source,
selects the current stable artifacts, and starts the published
Server and Waterline. It waits for a worker registration whose identity,
workflow type, activity type, and task queue match the generated contract before
starting the workflow. Success requires the expected completed result and
history; the terminal then prints the exact local Waterline run link and the
path to structured JSON evidence.

The package installation, inline quickstart, API reference, and repository
example below remain the direct paths for users who do not want Sample App.

The SDK targets the same durable model as the PHP package — instance IDs, run IDs, history events, task queues, and type keys are shared across languages. A Python worker can serve activities for a PHP-authored workflow, and vice versa.

For a capability comparison across the Python SDK, PHP SDK, Rust SDK, and
`dw`, see
[Client and Worker Capabilities](/docs/polyglot/cli-python-parity/).

For constructor signatures, return types, exception classes, and metric names,
see the generated [Python SDK API reference](https://python.durable-workflow.com/).

Cloud customers use the provisioned namespace's runtime URL and namespace with
separate client and worker credentials. See
[Cloud Managed Runtime](/docs/polyglot/cloud-control-plane) for that
connection boundary; the quickstart below uses local self-hosted values.

## Requirements

- Python 3.10 or later
- Docker (for the local Server used in this quickstart), an existing
  [self-hosted Server](/docs/polyglot/server), or a provisioned
  [Cloud namespace runtime](/docs/polyglot/cloud-control-plane)

## Installation

Install the <PythonPackageReleaseLink authority="qualified">stable Python SDK
release</PythonPackageReleaseLink>. The exact requirement is generated
from the same tuple as the Server quickstart, and a lock or constraints file can
retain it for reproducibility.

<!-- docs-example id="python.sdk.install" -->
```bash
pip install durable-workflow==2.0.0
```

The SDK depends on [httpx](https://www.python-httpx.org/) for HTTP and on the
`fastavro` Python package to encode the Apache Avro wire format used by the
only public v2 payload codec. Prometheus metrics support is optional.

## Quickstart

Here is a complete Python program that defines a workflow with one activity, starts it against a local Durable Workflow server, and waits for the result:

<!-- docs-example id="python.quickstart.worker" -->
```python
import asyncio
import uuid

from durable_workflow import Client, Worker, workflow, activity

@activity.defn(name="greet")
async def greet(name: str) -> dict:
    return {"greeting": f"Hello, {name}!", "length": len(name)}

@workflow.defn(name="greeter")
class GreeterWorkflow:
    def run(self, ctx, *args):
        result = yield ctx.schedule_activity("greet", list(args))
        return result

async def main():
    workflow_id = f"greeting-{uuid.uuid4().hex}"

    async with Client("http://localhost:8080", token="dev-token", namespace="default") as client:
        handle = await client.start_workflow(
            workflow_type="greeter",
            task_queue="default",
            workflow_id=workflow_id,
            input=["world"],
        )

        worker = Worker(
            client,
            task_queue="default",
            workflows=[GreeterWorkflow],
            activities=[greet],
        )

        await worker.run_until(workflow_id=workflow_id, timeout=30.0)
        result = await handle.result(timeout=10.0)

    print(result)  # {"greeting": "Hello, world!", "length": 5}

asyncio.run(main())
```

### Running against a local server

The program above assumes a Durable Workflow server reachable at
`http://localhost:8080` with the local `dev-token`. If you don't have one yet,
the fastest source-free path is the published server image:

```bash
export DW_SERVER_IMAGE=durableworkflow/server:2.0.0
export DW_AUTH_TOKEN=dev-token

docker volume create durable-workflow-python-quickstart

docker run --rm \
  -v durable-workflow-python-quickstart:/app/database \
  -e DW_AUTH_DRIVER=token \
  -e DW_AUTH_TOKEN="$DW_AUTH_TOKEN" \
  "$DW_SERVER_IMAGE" server-bootstrap

docker rm -f durable-workflow-server >/dev/null 2>&1 || true
docker run -d --name durable-workflow-server \
  -p 8080:8080 \
  -v durable-workflow-python-quickstart:/app/database \
  -e DW_AUTH_DRIVER=token \
  -e DW_AUTH_TOKEN="$DW_AUTH_TOKEN" \
  "$DW_SERVER_IMAGE"

until curl -sf http://localhost:8080/api/ready > /dev/null; do sleep 1; done
```

For production deployment — auth drivers, database config, TLS — see the [server setup guide](/docs/polyglot/server).

For a larger example, the SDK repository includes [`examples/order_processing`](https://github.com/durable-workflow/sdk-python/tree/main/examples/order_processing), a Docker Compose stack that runs a Python worker through an order workflow end to end.

## Client API Reference

The async `durable_workflow.Client` is the public entry point for control-plane
and worker-plane HTTP calls. Use it as an async context manager so the
underlying `httpx.AsyncClient` connection pool closes cleanly.

```python
from durable_workflow import Client

async with Client(
    "https://workflow.example.com",
    token="shared-token",
    namespace="default",
    timeout=60.0,
) as client:
    info = await client.get_cluster_info()
```

### Constructor

| Argument | Type | Default | Use when |
| --- | --- | --- | --- |
| `base_url` | `str` | required | Server origin, without `/api`. |
| `token` | `str | None` | `None` | One bearer token should authorize both control-plane and worker-plane calls. |
| `control_token` | `str | None` | `None` | Control-plane calls need a different bearer token than worker polling. |
| `worker_token` | `str | None` | `None` | Worker-plane calls need a different bearer token than operator calls. |
| `namespace` | `str` | `"default"` | Target a server namespace through `X-Namespace`. |
| `timeout` | `float` | `60.0` | Override the default HTTP timeout. |
| `retry_policy` | `TransportRetryPolicy | None` | default policy | Tune transport retries for transient HTTP failures. |
| `metrics` | `MetricsRecorder | None` | no-op | Emit client and worker metrics to a custom recorder. |
| `payload_size_limit_bytes` | `int` | SDK default | Match the server's max payload-byte contract. |
| `payload_size_warning_threshold_percent` | `int` | SDK default | Warn before a payload reaches the configured limit. |
| `payload_size_warnings` | `bool` | `True` | Disable local payload-size warnings in tests or controlled scripts. |

`token` is the simplest option. If both `control_token` and `worker_token` are
set, control-plane methods use `control_token` and worker-plane methods use
`worker_token`.

### External Payload Storage

The SDK exports the same external-payload reference contract used by the server
and CLI storage APIs. Use it when Python activity handlers or invocable
carriers need to decode large payload references from workflow history, or when
a Python process needs to create a language-neutral payload envelope without
embedding large bytes inline.

| API | Role | Failure surface |
| --- | --- | --- |
| `ExternalStorageDriver` | Protocol for `put(data, sha256=..., codec=...)`, `get(uri)`, and `delete(uri)`. | Driver-raised storage errors. |
| `LocalFilesystemExternalStorage` | Dependency-free `file://` driver for local development and tests. | `ValueError` when a referenced URI escapes the configured root. |
| `S3ExternalStorage` | Adapter for a boto3-compatible client. | `ValueError` for foreign bucket/prefix references or non-byte responses. |
| `GCSExternalStorage` | Adapter for a google-cloud-storage-style client. | `ValueError` for foreign bucket/prefix references or non-byte responses. |
| `AzureBlobExternalStorage` | Adapter for an Azure container client. | `ValueError` for foreign container/prefix references or non-byte responses. |
| `ExternalPayloadReference` | Immutable wire reference with `uri`, `sha256`, `size_bytes`, `codec`, and schema. | `ValueError` when `from_dict()` receives an unsupported schema or malformed fields. |
| `ExternalPayloadCache` | Bounded replay cache for already verified external payload bytes. | Constructor rejects non-positive entry or byte limits. |
| `store_external_payload()` | Stores encoded bytes through a driver and returns `ExternalPayloadReference`. | Driver errors. |
| `fetch_external_payload()` | Fetches referenced bytes, then verifies size and SHA-256 before decode. | `ExternalPayloadIntegrityError` on size/hash mismatch. |
| `delete_external_payload()` | Deletes the referenced object and evicts any cache entry. | Driver-raised storage errors. |
| `external_storage_envelope()` | Encodes a value inline until the threshold is crossed, then writes bytes through a driver. | `ValueError` when the threshold is invalid or no driver can resolve a reference. |
| `external_storage_driver_from_policy()` | Builds the matching driver from a server or Cloud `external_payload_storage` policy, given application-supplied provider clients. | `ValueError` when the policy is disabled, unsupported, or missing the required client/bucket/container. |

```python
from durable_workflow import (
    ExternalPayloadCache,
    LocalFilesystemExternalStorage,
    external_storage_envelope,
    to_avro_payload_value,
)
from durable_workflow.serializer import encode
from durable_workflow.external_storage import fetch_external_payload, store_external_payload

storage = LocalFilesystemExternalStorage("/var/lib/durable-workflow/payloads")
cache = ExternalPayloadCache(max_entries=256, max_bytes=32 * 1024 * 1024)

payload = to_avro_payload_value({"invoice_pdf": "x" * 1_000_000})
envelope = external_storage_envelope(
    payload,
    external_storage=storage,
    threshold_bytes=64 * 1024,
)

reference = store_external_payload(
    storage,
    encode({"archived": True}).encode("utf-8"),
    codec="avro",
)
payload_bytes = fetch_external_payload(storage, reference, cache=cache)
```

The reference schema is `EXTERNAL_PAYLOAD_REFERENCE_SCHEMA`
(`durable-workflow.v2.external-payload-reference.v1`). Object-store adapters do
not add cloud SDK dependencies to `durable-workflow`; applications pass their
already configured S3, GCS, or Azure clients.

The wire envelope fields are intentionally small and stable:

```json
{
  "schema": "durable-workflow.v2.external-payload-reference.v1",
  "uri": "s3://dw-payloads/billing/run-001/input.avro",
  "sha256": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
  "size_bytes": 1048576,
  "codec": "avro",
  "expires_at": "2026-05-22T00:00:00Z"
}
```

`ExternalPayloadReference.from_dict()` rejects unknown schemas and malformed
fields. `fetch_external_payload()` verifies `size_bytes` and `sha256` before
returning bytes, and raises `ExternalPayloadIntegrityError` when the object is
missing, truncated, or mutated. Worker replay should use `ExternalPayloadCache`
only after verification, so repeated history reads avoid refetching the same
blob without weakening integrity checks.

When the application has already read the namespace's
`external_payload_storage` policy from the server or Cloud control plane,
`external_storage_driver_from_policy()` returns the matching driver. Provider
SDK clients remain application-owned, so the SDK never adds boto3,
google-cloud-storage, or azure-storage-blob as runtime dependencies:

```python
from durable_workflow.external_storage import external_storage_driver_from_policy

namespace = await client.describe_namespace("billing")
driver = external_storage_driver_from_policy(
    namespace.external_payload_storage,
    s3_client=application_owned_s3_client,
)
```

The factory raises `ValueError` when the policy is disabled, when the named
driver is unsupported, or when the matching provider client is missing.
Disabled policies do not return a no-op driver; callers should branch on
`policy.enabled` before asking for a driver.

### Namespaces

Namespaces are the tenancy boundary for workflows, schedules, search
attributes, and external payload storage. The Client targets one namespace at a
time through the `namespace=` constructor argument, but the operator surface
below applies to any namespace the bearer token is authorized for.

| Method | Returns | Failure surface |
| --- | --- | --- |
| `await client.list_namespaces()` | `NamespaceList` | Auth/server errors. |
| `await client.describe_namespace(name)` | `NamespaceDescription` | `NamespaceNotFound`, auth/server errors. |
| `await client.create_namespace(name, description=None, retention_days=30)` | `NamespaceDescription` | `InvalidArgument` for duplicate names or invalid retention, auth/server errors. |
| `await client.update_namespace(name, description=None, retention_days=None)` | `NamespaceDescription` | `NamespaceNotFound`, `InvalidArgument`, auth/server errors. Only provided fields are sent. |
| `await client.set_namespace_external_storage(name, driver=..., enabled=True, threshold_bytes=None, config=None)` | `NamespaceDescription` | `InvalidArgument` when the policy fails server validation, auth/server errors. |
| `await client.test_external_storage(driver=None, small_payload_bytes=None, large_payload_bytes=None)` | `StorageTestResult` | `InvalidArgument` when the bound policy is missing required fields, auth/server errors. |

`set_namespace_external_storage` mirrors `dw namespace:set-storage-driver`. The
first positional argument is the namespace `name`, matching `describe_namespace`,
`create_namespace`, and `update_namespace`. The deprecated `namespace=` alias
emits a `DeprecationWarning`; new code should use the positional `name`. The
`config` dict carries driver-specific keys including the optional `disk` field
on `s3`, `gcs`, and `azure` so credentials stay on the server. The returned
`NamespaceDescription` reflects the policy the server actually persisted —
including the threshold the server fell back to when no caller value was
provided.

`test_external_storage` mirrors `dw storage:test`. The server round-trips a
small and large payload through the bound external storage driver and returns
`StorageTestResult`, which exposes per-payload `StoragePayloadTestResult`
records with `wrote`, `read`, `verified`, `latency_ms`, and `bytes` fields.

`NamespaceDescription` carries the namespace's `external_payload_storage`
policy as documented in [External Payload Storage](#external-payload-storage).
Pair it with `external_storage_driver_from_policy()` to build the matching
Python driver from the policy without re-reading credentials in application
code.

### Cluster and Task Queues

| Method | Returns | Notes |
| --- | --- | --- |
| `await client.health()` | `dict[str, Any]` | Calls the health endpoint for readiness checks. |
| `await client.get_cluster_info()` | `dict[str, Any]` | Reads server version, protocol, capability, and compatibility metadata. |
| `await client.list_task_queues()` | `TaskQueueList` | Lists task queues visible in the namespace. |
| `await client.describe_task_queue(name)` | `TaskQueueDescription` | Returns worker capacity, current leases, query admission, and dispatch-budget facts. |
| `await client.list_task_queue_build_ids(task_queue)` | `TaskQueueBuildIdRollout` | Snapshots the per-build-id cohort state for a queue, including unversioned workers under a cohort whose `build_id` is `None`. |
| `await client.drain_task_queue_build_id(task_queue, build_id)` | `TaskQueueBuildIdRolloutState` | Marks a build-id cohort as draining so it stops claiming new tasks. Pass `build_id=None` to drain unversioned workers. Idempotent. |
| `await client.resume_task_queue_build_id(task_queue, build_id)` | `TaskQueueBuildIdRolloutState` | Reverts a previous drain so the cohort can claim work again. Pass `build_id=None` to resume unversioned workers. Idempotent. |

Task queue return types expose nested `TaskQueueAdmission`,
`TaskQueueTaskAdmission`, `TaskQueueQueryAdmission`, `TaskQueueBuildIdCohort`,
`TaskQueueBuildIdRollout`, and `TaskQueueBuildIdRolloutState` dataclasses so
scripts can check server-side capacity and build-id rollout without parsing
prose output. See
[Worker Build-Id Rollout](/docs/polyglot/worker-build-id-rollout) for the
end-to-end rollout walkthrough; the CLI mirrors of these methods are
`dw task-queue:build-ids`, `dw task-queue:drain`, and `dw task-queue:resume`.

### Workflow Operations

| Method | Returns | Failure surface |
| --- | --- | --- |
| `await client.start_workflow(...)` | `WorkflowHandle` | `WorkflowAlreadyStarted`, `InvalidArgument`, `Unauthorized`, `ServerError` |
| `await client.describe_workflow(workflow_id)` | `WorkflowExecution` | `WorkflowNotFound`, auth/server errors |
| `await client.list_workflows(...)` | `WorkflowList` | Auth/server errors |
| `await client.list_workflow_runs(workflow_id)` | `WorkflowRunList` | `WorkflowNotFound`, auth/server errors |
| `await client.describe_workflow_run(workflow_id, run_id)` | `WorkflowRun` | `WorkflowNotFound`, auth/server errors |
| `await client.get_history(workflow_id, run_id)` | decoded history payload | `WorkflowNotFound`, auth/server errors |
| `await client.export_history(workflow_id, run_id)` | decoded archival history payload | `WorkflowNotFound`, auth/server errors |
| `await client.signal_workflow(workflow_id, signal_name, args=None)` | `None` | `WorkflowNotFound`, `InvalidArgument`, auth/server errors |
| `await client.query_workflow(workflow_id, query_name, args=None)` | decoded query result | `QueryFailed`, `WorkflowNotFound`, auth/server errors |
| `await client.update_workflow(workflow_id, update_name, args=None, ...)` | decoded update result | `UpdateRejected`, `WorkflowNotFound`, auth/server errors |
| `await client.cancel_workflow(workflow_id, reason=None)` | `None` | `WorkflowNotFound`, auth/server errors |
| `await client.terminate_workflow(workflow_id, reason=None)` | `None` | `WorkflowNotFound`, auth/server errors |
| `await client.repair_workflow(workflow_id)` | `WorkflowCommandResult` | `WorkflowNotFound`, auth/server errors |
| `await client.archive_workflow(workflow_id, reason=None)` | `WorkflowCommandResult` | `WorkflowNotFound`, auth/server errors |
| `await client.get_result(handle, poll_interval=0.5, timeout=30.0)` | decoded workflow output | `WorkflowFailed`, `WorkflowCancelled`, `WorkflowTerminated`, `TimeoutError` |

`start_workflow` accepts `workflow_type`, `task_queue`, optional
`workflow_id`, optional `input`, `duplicate_policy`, `memo`,
`search_attributes`, `business_key`, `execution_timeout_seconds`, and
`run_timeout_seconds`. All caller payloads are Avro-enveloped before they cross
the HTTP boundary.

Use `client.get_workflow_handle(workflow_id, run_id=None, workflow_type="")`
when a script already knows the workflow id and wants handle-style methods.

| `WorkflowHandle` method | Equivalent client method |
| --- | --- |
| `await handle.result(...)` | `client.get_result(handle, ...)` |
| `await handle.describe()` | `client.describe_workflow(handle.workflow_id)` |
| `await handle.signal(name, args=None)` | `client.signal_workflow(...)` |
| `await handle.query(name, args=None)` | `client.query_workflow(...)` |
| `await handle.update(name, args=None, ...)` | `client.update_workflow(...)` |
| `await handle.cancel(reason=None)` | `client.cancel_workflow(...)` |
| `await handle.terminate(reason=None)` | `client.terminate_workflow(...)` |

### Schedules

Schedules use `ScheduleSpec` for calendar/interval rules and `ScheduleAction`
for the workflow start request issued when a schedule fires.

| Method | Returns | Notes |
| --- | --- | --- |
| `await client.create_schedule(...)` | `ScheduleHandle` | Creates a schedule and returns a handle. |
| `await client.list_schedules()` | `ScheduleList` | Lists visible schedules. |
| `await client.describe_schedule(schedule_id)` | `ScheduleDescription` | Reads schedule status, action, next fire, and counters. |
| `await client.update_schedule(schedule_id, ...)` | `None` | Updates spec, action, overlap policy, jitter, memo, search attributes, or note. |
| `await client.pause_schedule(schedule_id, note=None)` | `None` | Pauses future fires. |
| `await client.resume_schedule(schedule_id, note=None)` | `None` | Resumes a paused schedule. |
| `await client.trigger_schedule(schedule_id, overlap_policy=None)` | `ScheduleTriggerResult` | Requests an immediate fire. |
| `await client.backfill_schedule(schedule_id, start_time=..., end_time=..., overlap_policy=None)` | `ScheduleBackfillResult` | Replays missed fire windows. |
| `await client.get_schedule_history(schedule_id, *, limit=None, after_sequence=None)` | `ScheduleHistoryPage` | Returns one page of the schedule's audit history stream, ordered by `sequence` ascending. `limit` is clamped server-side between 1 and 500 (default 100); `after_sequence` is a non-negative cursor obtained from the previous page's `next_cursor`. Raises `ScheduleNotFound` if the schedule id is unknown. History survives a `delete_schedule` call so operators can audit why a schedule was removed. |
| `client.iter_schedule_history(schedule_id, *, limit=None, after_sequence=None)` | `AsyncIterator[ScheduleHistoryEvent]` | Yields every audit event for the schedule, paging under the hood until the server reports `has_more=False`. Same failure surface as `get_schedule_history`. |
| `await client.delete_schedule(schedule_id)` | `None` | Deletes the schedule. |

`client.get_schedule_handle(schedule_id)` returns a `ScheduleHandle` with
`describe`, `update`, `pause`, `resume`, `trigger`, `backfill`, `history`,
`iter_history`, and `delete` methods that forward to the corresponding client
methods. `ScheduleHandle.history(...)` returns a `ScheduleHistoryPage` and
`ScheduleHandle.iter_history(...)` returns an `AsyncIterator[ScheduleHistoryEvent]`.

A `ScheduleHistoryPage` carries the ordered `events` list, a `has_more` flag,
a `next_cursor` integer (or `None` on the final page), the `schedule_id`, and
the owning `namespace`. Each `ScheduleHistoryEvent` carries `sequence`,
`event_type` (`ScheduleCreated`, `SchedulePaused`, `ScheduleResumed`,
`ScheduleUpdated`, `ScheduleTriggered`, `ScheduleTriggerSkipped`, or
`ScheduleDeleted`), `recorded_at`, optional `workflow_instance_id` and
`workflow_run_id` for fired workflows, and the raw `payload` dictionary the
control plane recorded for the transition.

### Bridge Events and Worker-Plane Methods

`await client.send_webhook_bridge_event(adapter, action=..., target=..., input=..., idempotency_key=..., correlation=None)`
returns `BridgeAdapterOutcome`, the same machine-readable outcome contract used
by `dw bridge:webhook`. It is the Python entry point for bounded ingress from
webhook-shaped systems.

The low-level worker-plane methods are public for custom workers and protocol
tests, but normal applications should use `Worker`:

| Method group | Methods |
| --- | --- |
| Worker registration | `register_worker` |
| Workflow tasks | `poll_workflow_task`, `complete_workflow_task`, `fail_workflow_task`, `workflow_task_history` |
| Query tasks | `poll_query_task`, `complete_query_task`, `fail_query_task` |
| Activity tasks | `poll_activity_task`, `complete_activity_task`, `fail_activity_task`, `heartbeat_activity_task` |

These methods send `X-Durable-Workflow-Protocol-Version` and use `worker_token`
when one is configured. Prefer the higher-level `Worker` unless you are writing
an SDK adapter or protocol conformance test.

`workflow_task_history(...)` pages replay history for one already leased
workflow task. Call it only after `poll_workflow_task(...)` returns
`next_history_page_token`:

```python
page = await client.workflow_task_history(
    task_id="workflow-task-01",
    next_history_page_token="history-page-2",
    lease_owner="python-worker-1",
    workflow_task_attempt=2,
)
```

The request body uses `next_history_page_token`, `lease_owner`, and
`workflow_task_attempt`. The decoded response uses the worker-protocol field
names `history_events`, `total_history_events`, and
`next_history_page_token`; it does not use the control-plane run-history names
`events` or `next_page_token`.

### Workers

The worker registry exposes which workers the server has seen recently and what
each worker can run. Use these methods to drive build-id rollouts, find stale
workers to deregister, and reconcile fleet capacity from operator scripts.

| Method | Returns | Failure surface |
| --- | --- | --- |
| `await client.list_workers(task_queue=None, status=None)` | `WorkerList` | Auth/server errors. |
| `await client.describe_worker(worker_id)` | `WorkerDescription` | `WorkerNotFound`, auth/server errors. |
| `await client.deregister_worker(worker_id)` | `dict[str, Any]` | `WorkerNotFound`, auth/server errors. |

`list_workers` filters server-side: pass `task_queue` to scope to one queue and
`status` to a single status string the server recognizes. The default returns
every registered worker for the namespace.

`WorkerDescription` carries the worker's runtime, SDK version, build id,
declared workflow and activity types, last heartbeat, and current task
admission so a script can decide which cohort to drain or which workers to
remove from the roster.

`deregister_worker` is idempotent on the server side; it removes a worker that
no longer heartbeats so capacity accounting and `list_workers` stay clean.
It does not interrupt in-flight leases.

### Search Attributes

Search attributes are typed namespace metadata that appear on workflow
executions for filtering and indexing. The Python client mirrors the same
control-plane surface as the CLI.

| Method | Returns | Failure surface |
| --- | --- | --- |
| `await client.list_search_attributes()` | `SearchAttributeList` | Auth/server errors. |
| `await client.create_search_attribute(name, attribute_type)` | `dict[str, Any]` | `InvalidArgument` for duplicate names or unsupported types, auth/server errors. |
| `await client.delete_search_attribute(name)` | `dict[str, Any]` | `SearchAttributeNotFound`, `InvalidArgument` when the attribute is system-defined, auth/server errors. |

`SearchAttributeList` separates `system` and `custom` attribute definitions so
operator scripts can verify that the engine-defined keys (workflow id, status,
type, start time, etc.) are present before refusing to create a clashing custom
key. Supported `attribute_type` values are `keyword`, `text`, `int`, `double`,
`bool`, `datetime`, and `keyword_list`.

### System Maintenance

Operator scripts and on-call automation drive the same maintenance loops as the
`dw system:*` commands through the Client. Each method requires the bearer
token to carry admin scope; without it the server returns `Unauthorized`.

| Method | Returns | CLI mirror |
| --- | --- | --- |
| `await client.repair_status()` | `dict[str, Any]` | `dw system:repair-status` |
| `await client.repair_pass(run_ids=None, instance_id=None)` | `dict[str, Any]` | `dw system:repair-pass` |
| `await client.retention_status()` | `dict[str, Any]` | `dw system:retention-status` |
| `await client.retention_pass(run_ids=None, limit=None)` | `dict[str, Any]` | `dw system:retention-pass` |
| `await client.activity_timeout_status()` | `dict[str, Any]` | `dw system:activity-timeout-status` |
| `await client.activity_timeout_pass(execution_ids=None, limit=None)` | `dict[str, Any]` | `dw system:activity-timeout-pass` |

`repair_pass` runs one task-repair sweep. With no filters the server runs a
full-scope pass over the namespace; pass `run_ids` to narrow the sweep to a
specific list of workflow runs, or `instance_id` to bound it to a single
running instance.

`retention_pass` enforces the namespace retention window on terminal runs.
With no filters the server prunes expired runs up to its scan limit; pass
`run_ids` to narrow the sweep, or `limit` to bound how many runs a single pass
processes. The companion `retention_status()` reports the namespace retention
window, the cutoff, and the run ids currently eligible for pruning up to the
server's scan limit.

`activity_timeout_pass` enforces start-to-close and schedule-to-close deadlines
on activity executions that have already passed their deadline. With no
filters the server processes any expired activity executions up to its scan
limit; pass `execution_ids` to target a specific list, or `limit` to bound a
single pass.

These methods do not raise on empty work — repeated calls during a quiet
period return the same `passes` / `repaired` / `pruned` counters with zero
deltas, so they are safe to call from cron-driven operator scripts.

## Defining Workflows

Workflows are Python classes decorated with `@workflow.defn`. The `run` method is a generator that yields commands to the server.

```python
from durable_workflow import workflow

@workflow.defn(name="order-processing")
class OrderWorkflow:
    def run(self, ctx, *args):
        order = args[0] if args else {}

        # Schedule an activity and wait for the result
        validated = yield ctx.schedule_activity(
            "validate_order", [order]
        )

        # Start a timer (durable sleep)
        yield ctx.start_timer(seconds=60)

        # Schedule another activity
        receipt = yield ctx.schedule_activity(
            "process_payment", [validated]
        )

        return receipt
```

The `name` in `@workflow.defn(name="...")` is the type key used across all languages. It must be a plain string — not a Python module path or class reference.

### Signals, Queries, and Updates

Signals are recorded in durable history and dispatched to Python handlers
during workflow replay:

```python
@workflow.defn(name="approval")
class ApprovalWorkflow:
    def __init__(self) -> None:
        self.approved = False
        self.approved_by = None

    @workflow.signal("approve")
    def approve(self, by: str) -> None:
        self.approved = True
        self.approved_by = by

    def run(self, ctx, *args):
        yield ctx.schedule_activity("wait_for_approval", [])
        return {"approved": self.approved, "approved_by": self.approved_by}
```

Query and update receiver decorators are available so workflow classes can
publish stable handler names. Python workers execute server-routed query tasks
by replaying the committed workflow history and invoking the registered query
handler against that replayed state. They also apply accepted updates delivered
through workflow tasks:

```python
@workflow.defn(name="approval")
class ApprovalWorkflow:
    def __init__(self) -> None:
        self.approved = False

    @workflow.query("status")
    def status(self) -> dict:
        return {"approved": self.approved}

    @workflow.update("set_approval")
    def set_approval(self, approved: bool) -> dict:
        self.approved = approved
        return {"approved": self.approved}

    @set_approval.validator
    def validate_set_approval(self, approved: bool) -> None:
        if not isinstance(approved, bool):
            raise ValueError("approved must be boolean")
```

Python workers complete server-routed queries by returning a query-task result
to the server. For a declared update validator, the worker instead receives a
dedicated validation task, replays authoritative workflow state, and invokes
only the validator. The server records no accepted update and dispatches no
update handler until that validation succeeds. Validator-bearing workers refuse
to register unless discovery advertises the synchronous pre-accept contract;
rejection, missing or incompatible workers, worker loss, timeout, and fenced
duplicate or stale completion remain explicit typed outcomes. PHP and Rust do
not currently expose update-validator authoring APIs and advertise that absence
in their workflow contracts rather than claiming validator parity.

### Workflow Context

The `WorkflowContext` passed to `run` provides deterministic operations:

| Method | Description |
|--------|-------------|
| `ctx.schedule_activity(type, args)` | Schedule an activity task, optionally with per-call retry and timeout options |
| `ctx.start_timer(seconds)` | Durable sleep |
| `ctx.start_child_workflow(type, args)` | Start a child workflow, optionally with per-call retry and workflow timeout options |
| `yield [command, [...]]` | Join a nested deterministic activity, child-workflow, timer, or mixed group in input order |
| `yield ctx.select({key: command, ...})` | Resume with the first durably committed member while retaining handles for every non-winner |
| `ctx.saga()` | Register and run reverse-order durable activity compensations |
| `ctx.throw_if_cancellation_requested()` | Observe cooperative cancellation at an author-controlled safe point |
| `ctx.side_effect(fn)` | Capture a non-deterministic value |
| `ctx.get_version(change_id, min, max)` | Safe workflow code versioning |
| `ctx.upsert_search_attributes(attrs)` | Update search attributes |
| `ctx.continue_as_new(*args)` | Restart the workflow with new input |
| `ctx.now()` | Deterministic clock (from history) |
| `ctx.random()` | Seeded random generator |
| `ctx.uuid4()` | Deterministic UUID |
| `ctx.logger` | Logger that is silent during replay |

### Activity Retries and Timeouts

Use `ActivityRetryPolicy` and timeout keyword arguments on `ctx.schedule_activity(...)`
when one activity call needs a different retry budget or deadline than the
default single-attempt behavior:

```python
from durable_workflow import ActivityRetryPolicy

receipt = yield ctx.schedule_activity(
    "process_payment",
    [validated],
    retry_policy=ActivityRetryPolicy(
        max_attempts=4,
        initial_interval_seconds=1,
        backoff_coefficient=2,
        maximum_interval_seconds=30,
        non_retryable_error_types=["ValidationError"],
    ),
    start_to_close_timeout=120,
    schedule_to_close_timeout=300,
    heartbeat_timeout=15,
)
```

`start_to_close_timeout` caps one attempt after a worker starts it.
`schedule_to_close_timeout` caps the whole activity execution across all
attempts. `heartbeat_timeout` requires long-running activities to call
`activity.context().heartbeat(...)` before the interval expires. The retry
policy is snapped onto the durable activity execution when it is scheduled, so
later deploys do not change already-running attempts.

### Determinism Rules

Workflow code is replayed from history. It must not perform any non-deterministic operations directly:

- No I/O (HTTP calls, file reads, database queries)
- No `datetime.now()` or `time.time()` — use `ctx.now()`
- No `random.random()` — use `ctx.random()`
- No `uuid.uuid4()` — use `ctx.uuid4()`

All non-determinism must flow through the context or be captured with `ctx.side_effect()`.

### Fan-Out

Yield a list of commands to run them concurrently:

```python
@workflow.defn(name="fan-out-example")
class FanOutWorkflow:
    def run(self, ctx, *args):
        items = args[0]

        # Nested lists may mix activities, children, and timers.
        results = yield [
            ctx.schedule_activity("process_item", [items[0]]),
            [
                ctx.start_child_workflow("process-batch", [items[1:]]),
                ctx.start_timer(1),
            ],
        ]

        return results  # same nested shape and input order
```

The worker emits one ordinary command per durable leaf and attaches the shared
stable `parallel_group_*` fields plus the full outer-to-inner path. History can
close members in any order; replay still binds results and failures by durable
input position. Exact duplicate terminal delivery is ignored. Pending history
after a worker restart and fully completed history reconstruct the same group
without rescheduling completed work.

### First-completion selection

Use `ctx.select()` when independent work should start together but the workflow
can make progress after one member completes:

```python
selected = yield ctx.select({
    "resolver": ctx.schedule_activity("resolve-request", [request_id]),
    "input": ctx.wait_condition(lambda: self.resolution is not None, key="resolution-ready"),
    "deadline": ctx.start_timer(2),
})

if selected.key == "deadline":
    yield selected.handles["resolver"].cancel()
    return {"status": "timed_out"}

yield selected.handles["deadline"].cancel()
return {"status": "resolved", "value": selected.result()}
```

The `SelectionResult` includes the winner's stable key, input index, operation
kind and durable identity, typed outcome, and all `DurableOperationHandle`
instances. `yield handle.await_result()` waits for a non-winner later; `yield
handle.cancel()` records an explicit void request. Only committed
`SelectionOperationCancelled` history proves cancellation won; replay advances
past an unmarked request, and `await_result()` still returns a completion that
committed first. Replay consumes the persisted winner even if later history
contains another completion first. Duplicate and out-of-order external
delivery cannot replace an already committed winner.

### Saga Compensation

```python
def forward(saga):
    flight = yield ctx.schedule_activity("trip.reserve-flight", [])
    saga.add_compensation("trip.cancel-flight", [flight])

    hotel = yield ctx.schedule_activity("trip.reserve-hotel", [])
    saga.add_compensation("trip.cancel-hotel", [hotel])

    ctx.throw_if_cancellation_requested()
    yield ctx.schedule_activity("trip.charge", [])
    return {"status": "booked"}

return (yield from ctx.saga().run(forward))
```

The saga executes ordinary activity commands sequentially in reverse
registration order after failure or cooperative cancellation. It stops on the
first compensation failure. `SagaCompensationFailed` preserves the initiating
failure, compensation failure, compensation activity type, and deterministic
registration order as structured diagnostics.

### Child Workflows

```python
from durable_workflow import ChildWorkflowRetryPolicy

result = yield ctx.start_child_workflow(
    "child-workflow-type",
    [{"input": "data"}],
    task_queue="child-queue",
    parent_close_policy="terminate",
    retry_policy=ChildWorkflowRetryPolicy(
        max_attempts=3,
        initial_interval_seconds=2,
        backoff_coefficient=2,
        non_retryable_error_types=["ValidationError"],
    ),
    execution_timeout_seconds=600,
    run_timeout_seconds=120,
)
```

`execution_timeout_seconds` caps the logical child workflow execution across
retries and continue-as-new runs. `run_timeout_seconds` caps each child run
attempt. Retry backoff is applied after a child run fails; invalid child start
commands are protocol errors and are not retried as child attempts.

### Continue-as-New

For long-running workflows, use continue-as-new to reset the history:

```python
from durable_workflow import ContinueAsNew

@workflow.defn(name="polling-workflow")
class PollingWorkflow:
    def run(self, ctx, *args):
        iteration = args[0] if args else 0

        result = yield ctx.schedule_activity("poll_source", [])

        if result.get("done"):
            return result

        # Continue with incremented iteration
        return ContinueAsNew(arguments=[iteration + 1])
```

## Defining Activities

Activities are async Python functions decorated with `@activity.defn`. Unlike workflows, activities can perform I/O freely.

```python
from durable_workflow import activity

@activity.defn(name="send_email")
async def send_email(to: str, subject: str, body: str) -> dict:
    # Activities can do I/O: HTTP calls, database queries, etc.
    response = await some_email_client.send(to=to, subject=subject, body=body)
    return {"message_id": response.id, "sent": True}
```

The `name` is the type key shared across languages. A PHP workflow can schedule an activity named `"send_email"` and a Python worker will pick it up, and vice versa.

### Activity Context

Inside an activity, access execution metadata and heartbeat via `activity.context()`:

```python
@activity.defn(name="long_running_task")
async def long_running_task(items: list) -> dict:
    ctx = activity.context()

    print(f"Attempt #{ctx.info.attempt_number}")
    print(f"Task queue: {ctx.info.task_queue}")

    for i, item in enumerate(items):
        # Check for cancellation
        if ctx.is_cancelled:
            return {"partial": True, "processed": i}

        await process(item)

        # Heartbeat to keep the task alive
        await ctx.heartbeat({"progress": i + 1, "total": len(items)})

    return {"processed": len(items)}
```

### Non-Retryable Errors

Raise `NonRetryableError` to fail the activity without retries:

```python
from durable_workflow import NonRetryableError

@activity.defn(name="validate")
async def validate(data: dict) -> dict:
    if "required_field" not in data:
        raise NonRetryableError("Missing required_field")
    return data
```

## Worker

The `Worker` registers with the server, polls for tasks, and dispatches them to your workflow and activity implementations.

```python
from durable_workflow import Client, Worker

async with Client("http://localhost:8080", token="secret") as client:
    worker = Worker(
        client,
        task_queue="default",
        workflows=[GreeterWorkflow, OrderWorkflow],
        activities=[greet, send_email, validate],
        max_concurrent_workflow_tasks=10,
        max_concurrent_activity_tasks=10,
    )

    await worker.run()  # blocks until worker.stop() is called
```

For smoke tests and one-workflow examples,
`await worker.run_until(workflow_id="...", timeout=60.0)` registers the same
worker and drives one workflow to a terminal state with sequential polling. Use
`run()` for deployed workers that should keep polling.

| Parameter | Default | Description |
|-----------|---------|-------------|
| `task_queue` | required | The task queue to poll |
| `workflows` | `()` | Workflow classes to register |
| `activities` | `()` | Activity functions to register |
| `worker_id` | auto-generated | Unique worker identifier |
| `poll_timeout` | `35.0` | Long-poll timeout in seconds |
| `max_concurrent_workflow_tasks` | `10` | Max parallel workflow tasks |
| `max_concurrent_activity_tasks` | `10` | Max parallel activity tasks |
| `shutdown_timeout` | `30.0` | Seconds to drain in-flight tasks on stop |
| `metrics` | client's recorder | Optional metrics recorder for poll and task counters/histograms |
| `interceptors` | `()` | Ordered worker task wrappers for instrumentation, tracing, and policy hooks |

The two `max_concurrent_*` values are advertised to the server during worker
registration and appear in task queue admission diagnostics. Treat them as the
worker's local capacity. Use server-side
[task queue admission](/docs/polyglot/task-queue-admission) caps when a
namespace, queue, or downstream budget group needs a hard shared budget across
multiple workers.

### Worker API Reference

| Method | Returns | Use when |
| --- | --- | --- |
| `await worker.run()` | `None` | Long-running process supervised by systemd, Docker, Kubernetes, or a local dev shell. Registers once, then polls workflow, activity, and query tasks until stopped or cancelled. |
| `await worker.run_until(workflow_id=..., timeout=60.0, poll_interval=0.5)` | `WorkflowExecution` | Smoke tests and examples that start one workflow and want the same process to drive it until a terminal status. |
| `await worker.stop()` | `None` | Cooperative shutdown. Stops new polls and drains in-flight tasks up to `shutdown_timeout`. |

`run()` validates server compatibility before it starts polling. The worker
requires the server's published `control_plane.version`,
`control_plane.request_contract`, `worker_protocol.version`, and
`auth_composition_contract` to match the SDK's supported contract versions. A
missing or incompatible manifest raises `RuntimeError` during registration, so
supervisors fail fast instead of running a worker that cannot safely complete
tasks.

`run_until()` uses the same registration and dispatch path as `run()`, but
polls sequentially and returns the final `WorkflowExecution` for the named
workflow. It raises `TimeoutError` when the workflow is still non-terminal
after the timeout.

During task execution:

- unknown workflow or activity types are reported back to the server as task
  failures, not hidden in local logs
- `NonRetryableError` marks activity failures as non-retryable
- `ActivityCancelled` propagates as a cancellation outcome
- unhandled activity exceptions are reported as retryable failures unless the
  activity retry policy or server deadline says otherwise
- query handler exceptions are reported as `QueryFailed`

### Worker Interceptors

Pass `interceptors=[...]` when worker execution needs tracing, metrics, audit
logging, or local policy checks around tasks. Interceptors run in the order
provided; the first interceptor is the outer wrapper and should call `next` to
continue the chain.

```python
from durable_workflow import (
    ActivityInterceptorContext,
    PassthroughWorkerInterceptor,
)

class AuditInterceptor(PassthroughWorkerInterceptor):
    async def execute_activity(self, context: ActivityInterceptorContext, next):
        print("activity started", context.activity_type, context.worker_id)
        return await next(context)

worker = Worker(
    client,
    task_queue="orders",
    workflows=[OrderWorkflow],
    activities=[charge_card],
    interceptors=[AuditInterceptor()],
)
```

| Hook | Context fields | `next` returns |
| --- | --- | --- |
| `execute_workflow_task(context, next)` | `worker_id`, `task_queue`, `task` | workflow commands, or `None` |
| `execute_activity(context, next)` | `worker_id`, `task_queue`, `task`, `activity_type`, `args` | decoded activity result |
| `execute_query_task(context, next)` | `worker_id`, `task_queue`, `task` | encoded query result string |

Use `PassthroughWorkerInterceptor` as a base class when you only need one
hook. Implement `WorkerInterceptor` directly when you want type checkers to
force all hooks to be present.

## Logging

The SDK uses Python's standard `logging` module with structured logger names:

| Logger Name | What It Logs |
|-------------|--------------|
| `durable_workflow.worker` | Worker registration, task polls, task completion, errors |
| `durable_workflow.workflow.replay` | Workflow replay events (silent during replay) |

### Configuring Logging

Set the logging level in your application's entry point:

```python
import logging

# Show INFO-level worker events (registration, task completion)
logging.basicConfig(level=logging.INFO)

# Or configure specific loggers
logging.getLogger("durable_workflow.worker").setLevel(logging.DEBUG)
logging.getLogger("durable_workflow.workflow.replay").setLevel(logging.INFO)
```

### Log Levels

- **INFO**: Worker registration, task starts/completions, workflow completion
- **DEBUG**: Detailed task payloads (truncated), poll cycles
- **WARNING**: Retryable errors (failed API calls, unknown workflow types)
- **ERROR**: Non-retryable failures, replay crashes

### Replay-Aware Logging

Inside workflows, use `ctx.logger` for replay-aware logging:

```python
@workflow.defn(name="order_processor")
class OrderProcessor:
    def run(self, ctx, order_id: str):
        ctx.logger.info("Processing order %s", order_id)  # Only logs during execution, not replay
        result = yield ctx.schedule_activity("process_order", [order_id])
        ctx.logger.info("Order processed: %s", result)
        return result
```

Log statements are **silent during replay** to avoid duplicate log spam when workflows recover or continue execution.

### Structured Logging

For JSON-structured logs, configure your application's root logger with a JSON formatter:

```python
import logging
import json

class JSONFormatter(logging.Formatter):
    def format(self, record):
        return json.dumps({
            "timestamp": self.formatTime(record),
            "level": record.levelname,
            "logger": record.name,
            "message": record.getMessage(),
        })

handler = logging.StreamHandler()
handler.setFormatter(JSONFormatter())
logging.getLogger("durable_workflow").addHandler(handler)
logging.getLogger("durable_workflow").setLevel(logging.INFO)
```

## Metrics

`Client(metrics=...)` and `Worker(metrics=...)` accept any recorder with two methods:

```python
def increment(name: str, value: float = 1.0, tags: dict[str, str] | None = None) -> None: ...
def record(name: str, value: float, tags: dict[str, str] | None = None) -> None: ...
```

The default recorder is no-op, so metrics collection has no setup cost. Use `InMemoryMetrics` for tests or custom exporter loops:

```python
from durable_workflow import Client, InMemoryMetrics, Worker

metrics = InMemoryMetrics()

async with Client("http://localhost:8080", token="secret", metrics=metrics) as client:
    worker = Worker(client, task_queue="default", workflows=[GreeterWorkflow], activities=[greet])
```

For Prometheus, install the optional extra and pass `PrometheusMetrics`:

```bash
pip install 'durable-workflow[prometheus]'
```

```python
from durable_workflow import Client, PrometheusMetrics

metrics = PrometheusMetrics()
client = Client("http://localhost:8080", token="secret", metrics=metrics)
```

The SDK records:

| Metric | Type | Tags |
|--------|------|------|
| `durable_workflow_client_requests` | Counter | `method`, `route`, `plane`, `status_code`, `outcome` |
| `durable_workflow_client_request_duration_seconds` | Histogram | `method`, `route`, `plane`, `status_code`, `outcome` |
| `durable_workflow_worker_polls` | Counter | `task_kind`, `task_queue`, `outcome` |
| `durable_workflow_worker_poll_duration_seconds` | Histogram | `task_kind`, `task_queue`, `outcome` |
| `durable_workflow_worker_tasks` | Counter | `task_kind`, `task_queue`, `outcome` |
| `durable_workflow_worker_task_duration_seconds` | Histogram | `task_kind`, `task_queue`, `outcome` |

## Schedules

Create and manage scheduled workflows through the client:

```python
from durable_workflow import ScheduleSpec, ScheduleAction

# Create a schedule
handle = await client.create_schedule(
    schedule_id="hourly-report",
    spec=ScheduleSpec(cron_expressions=["0 * * * *"]),
    action=ScheduleAction(
        workflow_type="generate-report",
        task_queue="default",
        input=[{"format": "pdf"}],
    ),
    overlap_policy="skip",
    jitter_seconds=30,
)

# List all schedules
schedule_list = await client.list_schedules()

# Describe a schedule
desc = await handle.describe()
print(f"Next fire: {desc.next_fire_at}")
print(f"Total fires: {desc.fires_count}")

# Pause and resume
await handle.pause(note="maintenance window")
await handle.resume(note="maintenance complete")

# Trigger immediately
result = await handle.trigger()

# Backfill missed runs
backfill = await handle.backfill(
    start_time="2024-01-01T00:00:00Z",
    end_time="2024-01-02T00:00:00Z",
)

# Update the schedule
await handle.update(
    spec=ScheduleSpec(cron_expressions=["*/30 * * * *"]),
    note="Changed to every 30 minutes",
)

# Read the audit history stream (paused/resumed/triggered/updated events)
page = await handle.history(limit=100)
for event in page.events:
    print(event.sequence, event.event_type, event.recorded_at)
while page.has_more and page.next_cursor is not None:
    page = await handle.history(limit=100, after_sequence=page.next_cursor)
    for event in page.events:
        print(event.sequence, event.event_type, event.recorded_at)

# Or iterate every event without managing cursors
async for event in handle.iter_history():
    print(event.sequence, event.event_type)

# Delete the schedule (history survives the delete for audit)
await handle.delete()
```

## Synchronous Client

For scripts, notebooks, and non-async contexts, use the synchronous wrapper:

<!-- docs-example id="python.sync-client" -->
```python
import uuid

from durable_workflow.sync import Client as SyncClient

client = SyncClient("http://localhost:8080", token="secret")
workflow_id = f"sync-greeting-{uuid.uuid4().hex}"

handle = client.start_workflow(
    workflow_type="greeter",
    task_queue="default",
    workflow_id=workflow_id,
    input=["world"],
)

execution = client.describe_workflow(workflow_id)
print(execution.status)
```

The synchronous client mirrors the async API but wraps each call with `asyncio.run`.

## Error Handling

The SDK maps server error codes to typed Python exceptions:

| Exception | When |
|-----------|------|
| `WorkflowNotFound` | Workflow ID does not exist |
| `WorkflowAlreadyStarted` | Duplicate workflow ID with conflicting policy |
| `WorkflowFailed` | Workflow execution failed |
| `WorkflowCancelled` | Workflow was cancelled (inherits from `BaseException`, not `Exception`) |
| `WorkflowTerminated` | Workflow was terminated |
| `ActivityCancelled` | Activity was cancelled during execution (inherits from `BaseException`, not `Exception`) |
| `ChildWorkflowFailed` | A child workflow failed |
| `QueryFailed` | Query handler returned an error |
| `UpdateRejected` | Update was rejected by the workflow |
| `ScheduleNotFound` | Schedule ID does not exist |
| `ScheduleAlreadyExists` | Duplicate schedule ID |
| `NamespaceNotFound` | Namespace does not exist |
| `InvalidArgument` | Invalid request parameters |
| `Unauthorized` | Authentication failed |
| `ServerError` | Server returned an unexpected error |

<!-- docs-example id="python.duplicate-start-recovery" -->
```python
from durable_workflow import WorkflowNotFound, WorkflowAlreadyStarted

try:
    handle = await client.start_workflow(
        workflow_type="greeter",
        task_queue="default",
        workflow_id="existing-id",
        input=["world"],
    )
except WorkflowAlreadyStarted:
    handle = client.get_workflow_handle("existing-id")
except WorkflowNotFound:
    print("Workflow type not registered on any worker")
```

### Cancellation is intentionally uncatchable by `except Exception`

`WorkflowCancelled` and `ActivityCancelled` inherit from `BaseException`, not `Exception`. A generic `except Exception:` block in an activity body — or in code that awaits `client.get_result()` — will **not** catch them. This is deliberate: cancellation is a control-plane outcome, and silently swallowing it in a catch-all would let an activity report success after its workflow asked it to stop.

If you need to run cleanup on cancellation, catch the class by name and re-raise:

```python
from durable_workflow import ActivityCancelled, activity

@activity.defn(name="long_task")
async def long_task(items: list) -> dict:
    ctx = activity.context()
    try:
        for i, item in enumerate(items):
            await process(item)
            await ctx.heartbeat({"progress": i + 1})
        return {"done": True}
    except ActivityCancelled:
        await cleanup_partial_state()
        raise
```

This mirrors the standard-library precedent set by `asyncio.CancelledError` and `KeyboardInterrupt`.

## Testing

Workflow authors should be able to test workflow code without a running server
or worker. The `durable_workflow.testing` module ships two entry points:

- `WorkflowEnvironment` drives a workflow to completion in a single Python
  process against user-registered activity mocks.
- `replay_history` and `replay_history_file` replay a captured production
  history against current workflow code and raise on any non-determinism.

Both entry points reuse the same `durable_workflow.workflow.replay` machinery
the worker uses at runtime, so a workflow that passes its test harness behaves
the same way under a real worker.

### WorkflowEnvironment

`WorkflowEnvironment` dispatches yielded workflow commands against registered
mocks and auto-fires timers, side effects, and search-attribute upserts. Tests
do not need a real clock, Redis, or server.

```python
from durable_workflow import workflow
from durable_workflow.testing import WorkflowEnvironment

@workflow.defn(name="greeter")
class Greeter:
    def run(self, ctx, name: str):
        greeting = yield ctx.schedule_activity("greet", [name])
        return greeting

def test_greeter_returns_activity_result() -> None:
    env = WorkflowEnvironment()
    env.register_activity_result("greet", "hello, world")

    result = env.execute_workflow(Greeter, "world")

    assert result == "hello, world"
```

| Method | Purpose |
| --- | --- |
| `register_activity_result(name, result)` | Return `result` for every call to activity `name`. Use this when the test does not care about arguments. |
| `register_activity(name, fn)` | Call `fn(*arguments)` for each scheduled invocation of activity `name`. Use this when the mock must vary with arguments or capture invocations. |
| `register_child_workflow_result(workflow_type, result)` | Return `result` when the workflow starts a child of type `workflow_type`. |
| `signal(name, args=None, run=None)` | Queue a signal to be delivered before the next replay iteration. The harness injects a `SignalReceived` event and dispatches it to the registered `@workflow.signal` handler. Pass `run=N` to target link `N` of a continue-as-new chain. |
| `register_workflow(workflow_cls)` | Make an additional workflow class resolvable by name. Required when a chain calls `continue_as_new(workflow_type=...)` with a type other than the starting workflow. |
| `execute_workflow(workflow_cls, *args, run_id="test-run")` | Drive the workflow to a terminal state and return its result. Follows `continue_as_new` links to the final run and returns that run's result. Raises `WorkflowFailed` when the workflow ends in the failed state. |
| `runs` / `run_count` | After `execute_workflow` returns, expose one `WorkflowRunRecord` per link in the chain (input, workflow type, history events, terminal command) so tests can assert on the full continuation chain. |

The harness fails loudly on missing fixtures:

- scheduling an activity that has no registered mock raises `KeyError`
- starting a child workflow that has no registered mock raises `KeyError`
- a workflow that never reaches a terminal state within the iteration limit
  (default `1000`) raises `RuntimeError`

Pass `iteration_limit=...` to `WorkflowEnvironment(...)` to tune the cap for
workflows that legitimately iterate more than the default.

#### Callable activity mocks

Use `register_activity` when the mock needs to respond based on arguments or
record calls:

```python
def test_callable_mock_captures_arguments() -> None:
    captured: list[str] = []

    def record_greet(name: str) -> str:
        captured.append(name)
        return f"greeted:{name}"

    env = WorkflowEnvironment()
    env.register_activity("greet", record_greet)

    assert env.execute_workflow(Greeter, "alice") == "greeted:alice"
    assert captured == ["alice"]
```

#### Signals

Signals queued with `env.signal(...)` are drained before the next replay
iteration. The signal payload is wrapped in the same `{codec, blob}` envelope
the worker sees at runtime and dispatched to the workflow's registered
`@workflow.signal` handler:

```python
@workflow.defn(name="approval")
class Approval:
    def __init__(self) -> None:
        self.approved_by: str | None = None

    @workflow.signal("approve")
    def on_approve(self, by: str) -> None:
        self.approved_by = by

    def run(self, ctx):
        yield ctx.schedule_activity("wait", [])
        return {"approved_by": self.approved_by}

def test_signal_is_delivered_before_run_returns() -> None:
    env = WorkflowEnvironment()
    env.register_activity_result("wait", None)
    env.signal("approve", ["alice"])

    result = env.execute_workflow(Approval)

    assert result == {"approved_by": "alice"}
```

#### Timers, side effects, and search attributes

The harness auto-fires the corresponding history event for each of these
commands, so workflows do not block on wall-clock time inside tests:

- `ctx.sleep(seconds)` → `TimerFired`
- `ctx.side_effect(...)` → `SideEffectRecorded`
- `ctx.upsert_search_attributes(...)` → `SearchAttributesUpserted`
- `workflow.version(...)` markers → `VersionMarkerRecorded`

#### Continue-as-new chains

When a workflow returns `ctx.continue_as_new(...)`, the harness appends a
`WorkflowContinuedAsNew` event to the completing run, resets history, and
starts a new run with the command's arguments. The return value of
`execute_workflow` is the terminal result of the final link.

```python
@workflow.defn(name="countdown")
class Countdown:
    def run(self, ctx, counter: int):
        yield ctx.schedule_activity("emit", [counter])
        if counter > 0:
            return ctx.continue_as_new(counter - 1)
        return {"final_counter": counter}

def test_chain_returns_final_run_result() -> None:
    env = WorkflowEnvironment()
    env.register_activity_result("emit", None)

    result = env.execute_workflow(Countdown, 3)

    assert result == {"final_counter": 0}
    assert env.run_count == 4
    assert [r.input for r in env.runs] == [[3], [2], [1], [0]]
```

When the chain switches workflow types, register the follow-on class first:

```python
@workflow.defn(name="stage-one")
class StageOne:
    def run(self, ctx):
        yield ctx.schedule_activity("stage_one", [])
        return ctx.continue_as_new(workflow_type="stage-two")

@workflow.defn(name="stage-two")
class StageTwo:
    def run(self, ctx):
        return (yield ctx.schedule_activity("stage_two", []))

def test_chain_can_switch_workflow_type() -> None:
    env = WorkflowEnvironment()
    env.register_workflow(StageTwo)
    env.register_activity_result("stage_one", None)
    env.register_activity_result("stage_two", "done")

    assert env.execute_workflow(StageOne) == "done"
    assert [r.workflow_type for r in env.runs] == ["stage-one", "stage-two"]
```

Target a signal at a specific link in the chain with `run=N`:

```python
env.signal("approve", ["alice"], run=2)  # delivered to the second run
```

The chain length is capped by `continue_as_new_limit` (default `50`).
Exceeding the limit raises `RuntimeError` so tests catch runaway
continuations instead of spinning forever; tune the limit with
`WorkflowEnvironment(continue_as_new_limit=...)` when a chain legitimately
runs longer.

#### Failure assertions

Workflows that raise a Python exception surface as `WorkflowFailed`:

```python
import pytest
from durable_workflow.errors import WorkflowFailed

@workflow.defn(name="failing")
class Failing:
    def run(self, ctx):
        yield ctx.schedule_activity("step", [])
        raise RuntimeError("boom")

def test_workflow_failure_surfaces_as_workflow_failed() -> None:
    env = WorkflowEnvironment()
    env.register_activity_result("step", None)

    with pytest.raises(WorkflowFailed) as exc_info:
        env.execute_workflow(Failing)

    assert "boom" in str(exc_info.value)
```

### Replay testing against production history

Use `replay_history` to regression-test a workflow change against a real
history captured from the server. The replayer runs the current workflow code
against the recorded event sequence and raises if it yields a different
command than the one history recorded — the definition of a non-determinism
bug.

```python
from durable_workflow import Client
from durable_workflow.testing import replay_history

async with Client("http://localhost:8080") as client:
    history = await client.get_history("order-42", run_id="...")

replay_history(OrderWorkflow, history["events"], start_input=["order-42"])
```

A workflow that previously completed must still complete when replayed
against the same history. If the workflow code changed in a way that diverges
from the recorded sequence (reordered activity calls, removed branches,
changed activity types), `replay_history` raises so the regression is caught
in CI rather than in production.

`replay_history_file` is a convenience wrapper that reads a JSON file in
either of two shapes: a top-level list of events, or a dict with an `events`
key matching the `get_history` response shape:

```python
from durable_workflow.testing import replay_history_file

replay_history_file(
    OrderWorkflow,
    "tests/histories/order-42.json",
    start_input=["order-42"],
)
```

Both functions accept an optional `payload_codec` validation hint. Leave it
unset to use the history's recorded tag. An explicit or recorded tag must be
`avro`; `json`, unknown, or untagged durable payloads fail before decode.

#### Class-based replayer

When one test pass replays histories for several different workflow types, or
when the test should let the captured `WorkflowStarted` event decide which
workflow to replay, use the `Replayer` class. Register every workflow class
up front and call `replay(...)` once per captured history:

```python
from durable_workflow import Replayer, ReplayOutcome

replayer = Replayer(workflows=[OrderWorkflow, RefundWorkflow])

# Explicit type and input — equivalent to replay_history(OrderWorkflow, ...).
outcome: ReplayOutcome = replayer.replay(
    history["events"],
    start_input=["order-42"],
    workflow_type="order",
)

# Type and start input inferred from a WorkflowStarted event in the history.
outcome = replayer.replay(history)  # history may be an events list or a
                                    # dict with an "events" key

for command in outcome.commands:
    # Inspect commands the replayed workflow would have emitted next.
    ...
```

`Replayer(workflows=[...])` rejects an empty workflow set or a duplicate
registration with `ValueError`. `replay(...)` raises `ValueError` when a
history asks for a workflow type that was not registered, or when multiple
workflows are registered and the caller does not provide `workflow_type` and
the history does not include a `WorkflowStarted` event.

`ReplayOutcome.commands` is the same command list the functional
`replay_history` helpers return; the class-based entry point exists so a test
suite can share one registration across many histories.

### Test Harness Reference

| Symbol | Purpose |
| --- | --- |
| `durable_workflow.testing.WorkflowEnvironment` | In-process test harness. Drive a workflow to completion against registered activity and child-workflow mocks. |
| `durable_workflow.testing.replay_history(workflow_cls, events, start_input=None, *, run_id="", payload_codec=None)` | Replay a history event sequence against current workflow code. Raises on non-determinism. |
| `durable_workflow.testing.replay_history_file(workflow_cls, path, start_input=None, *, run_id="", payload_codec=None)` | Load a JSON history from disk and replay it. Accepts either a top-level list of events or a dict with an `events` key. |
| `durable_workflow.Replayer(*, workflows=[...])` | Class-based replayer. Register one or more workflow classes, then call `replay(history, start_input=None, *, workflow_type=None, workflow_id=None, run_id="", payload_codec=None)` to replay each captured history. Infers `workflow_type` and `start_input` from a `WorkflowStarted` event when the history contains one. |
| `durable_workflow.ReplayOutcome` | Dataclass returned by `Replayer.replay`. Carries `commands: list[Command]` — the commands the replayed workflow would have emitted next. |
| `durable_workflow.errors.WorkflowFailed` | Raised by `execute_workflow` when the workflow terminates in the failed state. |
| `durable_workflow.errors.WorkflowCancelled` | Terminal state when the workflow was cancelled. Inherits from `BaseException`. |
| `durable_workflow.errors.WorkflowTerminated` | Terminal state when the workflow was terminated by the server. Inherits from `BaseException`. |

## Payload Codec

<div
  data-public-payload-codec-contract="avro-only"
  data-payload-codec="avro"
  data-authority-manifest="https://durable-workflow.github.io/sdk-neutrality-contract.json"
  data-legacy-v1-import-drain="none">
  Every payload that crosses the worker-protocol boundary is codec-tagged.
  Durable Workflow 2.0 has one public payload codec: <code data-payload-codec-field="codec">avro</code>.
  The Python SDK rejects JSON-tagged, unknown, and untagged durable payloads
  instead of selecting another decoder. See the <a href="/docs/polyglot/avro-value-protocol/">Avro Value protocol</a>.
</div>

### Avro support is built in

The pinned 2.0 SDK artifact pulls in `fastavro` as a runtime dependency, so
every outgoing surface (`start_workflow`, `signal_workflow`, `query_workflow`,
`update_workflow`, activity result encoding, schedule actions) emits
Avro-tagged payloads through the optimized production path. There is no
optional codec extra to install.

### Fixed typed values

The SDK explicitly selects named branches in the shared
`durable_workflow.protocol.Value` schema. It does not JSON-encode the value
inside Avro. Integers and doubles, strings and bytes, booleans and integers,
and lists and maps therefore remain distinct across SDKs. The wire uses Avro
single-object framing and the schema fingerprint; unknown fingerprints fail
with `unsupported_payload_schema`.

Every client and worker surface works end-to-end on the Avro default:

- **Client starts, signals, queries, updates** — `start_workflow`, `signal_workflow`, `query_workflow`, and `update_workflow` emit `payload_codec = "avro"` payloads through the fixed Value schema. A Python client can therefore drive workflows that PHP and other polyglot SDKs will replay, and vice-versa.
- **Activity worker** — Avro-tagged activity arguments decode transparently. The worker encodes activity results as Avro so PHP, Python, and future SDK workers share one payload boundary.
- **Activity failures** — `fail_activity_task(..., details=...)` sends `failure.details` as a `{codec, blob}` envelope. The server records the blob plus `details_payload_codec`, so diagnostic failure data from Python workers remains language-neutral in history exports and observability views.
- **Workflow worker history replay** — Avro-tagged start input and activity result events are decoded during replay, so a Python workflow can participate in an Avro-coded run.
- **Workflow query tasks** — Server-routed query tasks carry Avro-tagged workflow arguments, query arguments, and replay history. The worker returns the query result as an Avro envelope.

### Running a Python activity worker against a v2 run

No codec configuration is needed. The SDK validates `payload_codec` before
decoding every claim, accepts only `avro`, runs the activity, and returns an
Avro-tagged result. Any other tag fails closed with
`unsupported_payload_codec`; the worker never sniffs or guesses the format.

### Types that round-trip cleanly across Python and PHP

| Python type | Avro Value branch | PHP type |
|-------------|------|----------|
| `str` | `StringValue` | `string` |
| `bytes` | `BytesValue` | `AvroBinaryValue` |
| `int` | `LongValue` | `int` |
| `float` | `DoubleValue` | `float` |
| `bool` | `BooleanValue` | `bool` |
| `None` | `null` | `null` |
| `list` | `ArrayValue` | `array` (list) |
| `dict[str, ...]` | `MapValue` | `array` (string-keyed map) |

Adapt Python-specific types such as dataclasses, sets, tuples, and datetime
objects to canonical strings, integers, maps, or lists before using them as
workflow or activity values.

## Running Against a Shared Server

The [Quickstart](#quickstart) above shows how to bring up a local server with
the published Docker image. In a team environment you usually point the Python
worker at an existing server (staging, production, or a shared dev instance):

```python
from durable_workflow import Client

client = Client(
    "https://workflow.example.internal",
    control_token="team-orders-operator-token",
    worker_token="team-orders-worker-token",
    namespace="team-orders",
)
```

Set the `namespace` argument to whichever tenant namespace the shared server has
provisioned for your team, and use the credentials issued for that namespace.
The Server operator manages namespace creation; see the
[Server guide](/docs/polyglot/server) for details.

The same constructor supports a Cloud managed runtime. Use the namespace's
Cloud-provided runtime URL and namespace value, set `control_token` to the
client runtime credential, and set `worker_token` to the worker runtime
credential. Cloud provisions the namespace; do not substitute a self-hosted
Server address. See
[Cloud Managed Runtime](/docs/polyglot/cloud-control-plane).

<!-- Source: docs/polyglot/rust-cloud-quickstart.md -->

# Rust Cloud Quickstart

:::caution Controlled early access

Durable Workflow Cloud is available through controlled early access. Use this
guide only after Cloud has provisioned a namespace and two role-scoped runtime
credentials. The generally available Rust journey does not require Cloud; start
with the [Rust SDK guide](./rust.md) or run `scripts/playground rust` against
the playground's default local runtime.

:::

The Sample App exposes one symmetric playground for PHP, Python, and Rust. This
page selects Rust and changes only the runtime target. The corresponding
[PHP](./php.md) and [Python](./python.md) SDK paths use the same command with
`php` or `python`; the [Sample App playground
contract](https://github.com/durable-workflow/sample-app/blob/main/README.md#symmetric-sdk-playground)
documents all three choices.

The playground resolves the current stable artifact versions from the Sample
App's machine-owned metadata. Use those generated versions instead of copying
version numbers into these commands or adding a separate SDK installation step.

## 1. Open the prepared Sample App

[Create a Codespace from the Sample App `main`
branch](https://codespaces.new/durable-workflow/sample-app?quickstart=1&ref=main),
wait for setup to finish, and open a terminal at the repository root. The
prepared image contains the SDK toolchains and `dw` required by the shared
playground.

Cloud provides the runtime URL, runtime namespace, and two role credentials.
Choose an application task queue, then export these placeholders only after
replacing them with the corresponding values:

```bash
export DURABLE_WORKFLOW_RUNTIME_URL='<provisioned-runtime-url>'
export DURABLE_WORKFLOW_RUNTIME_NAMESPACE='<provisioned-runtime-namespace>'
export DURABLE_WORKFLOW_CLIENT_TOKEN='<client-runtime-credential>'
export DURABLE_WORKFLOW_WORKER_TOKEN='<worker-runtime-credential>'
export DURABLE_WORKFLOW_TASK_QUEUE='<rust-task-queue>'
```

The runtime URL is the namespace runtime root returned by Cloud. It must be an
absolute HTTPS URL without a query, fragment, or terminal `/api`; the SDK and
CLI append their own API routes. The runtime namespace is the separate value
returned with that URL, not a name inferred from its path.

The client credential starts, describes, and reads workflows. The worker
credential registers, polls, heartbeats, and completes tasks. They must be
different role-scoped secrets even though this one command launches both child
processes. A Cloud administration key manages namespaces and credentials; do
not export or pass it as either runtime credential.

## 2. Run the Rust journey

Run the shared external-runtime contract:

```bash
scripts/playground rust --runtime managed \
  --runtime-url "$DURABLE_WORKFLOW_RUNTIME_URL" \
  --namespace "$DURABLE_WORKFLOW_RUNTIME_NAMESPACE" \
  --task-queue "$DURABLE_WORKFLOW_TASK_QUEUE"
```

The command scaffolds missing files under `.playground/rust` without replacing
caller-owned source. It prints the effective workflow type, activity type,
queue, worker command, client command, input, and expected result before it
starts anything. The worker receives only the worker credential; after its
exact registration becomes visible, the client receives only the client
credential.

The registration wait is bounded to 60 seconds. Do not treat process startup
alone as readiness. Continue only after the command prints a checkpoint shaped
like this (identities are illustrative):

```text
Worker ready: target=managed runtime_url=<provisioned-runtime-url> namespace=<provisioned-runtime-namespace> id=<worker-id> queue=<rust-task-queue> workflow_type=sample-app.playground.rust.authored-workflow activity_type=sample-app.playground.rust.authored-activity
```

The same command starts one workflow, waits up to 120 seconds for the SDK
client result, confirms `status=completed` with `dw`, and checks the required
workflow and activity history. One successful result looks like:

```text
Completed rust workflow <workflow-id>: {"greeting":"Hello, Durable Workflow, from the Sample App Rust playground","input":{"name":"Durable Workflow"},"activity_runtime":"rust","workflow_runtime":"rust"}
```

The final `Playground success` line repeats the runtime target, namespace,
queue, registered types, and expected result shape without credential values.
The command also writes `storage/app/playground-rust-evidence.json` with the
selected artifact versions, exact registration, workflow/run identity,
`completed` status, result, and history event types. Managed Waterline remains
the operator surface for the provisioned namespace; the managed journey does
not start a local Server or Waterline.

## Bounded diagnosis

If the `Worker ready` checkpoint does not appear within 60 seconds, start with
the effective contract printed above the error:

- **Queue mismatch:** the runner queries the exact value passed to
  `--task-queue`. Confirm that Cloud admits that queue and that the printed
  registration uses the same value; then rerun the same command.
- **Type mismatch:** the error names the workflow and activity types the worker
  must advertise. Compare them with the effective contract. If caller-owned
  files contain older hard-coded registrations, update them or prove the
  current scaffold in a new directory with
  `--source "$HOME/durable-rust-worker"`.
- **Credential-role mismatch:** authorization before registration points to
  the worker credential; authorization while describing or starting the run
  points to the client credential. Do not swap the values or replace either
  with a Cloud administration key.
- **Runtime mismatch:** both roles must use the exact provisioned runtime URL
  and runtime namespace. Remove a terminal `/api`; do not substitute the Cloud
  administration URL or a self-hosted Server URL.

If registration succeeds but completion fails, keep the printed workflow/run
identity. Inspect that selected run in Managed Waterline and compare its
pending workflow or activity type and task queue with the effective contract.
Fix that mismatch before starting another run. The retained evidence path and
bounded worker output identify whether the client result, durable status,
expected result, or required history check failed.

Return to the broader [Rust SDK guide](./rust.md), or use the generated [Rust
API reference](https://rust.durable-workflow.com/durable_workflow/) for
individual types and methods.

<!-- Source: docs/polyglot/invocable-php-handler.md -->

# PHP Invocable Activity Handler

The [invocable HTTP carrier](./invocable-carrier.md) lets the server POST a
leased activity task to an HTTPS endpoint instead of waiting for a long-poll
worker. The PHP helper `Workflow\V2\Support\InvocableActivityHandler` is the
reference implementation that an external PHP process uses to turn that
request into a result envelope the server can reconcile.

Use it to wire an activity handler into:

- an AWS Lambda or Google Cloud Function invoked over HTTPS
- a Laravel controller sitting behind a thin HTTP service
- a container exposing one POST endpoint per activity queue

The helper parses the carrier-neutral external task input envelope, looks up
the registered callable, enforces the lease deadline, and emits the
carrier-neutral external task result envelope — including the failure shapes
the server expects.

## Quick Start

Install the Durable Workflow package in the external process, register one
callable per activity handler name, and hand the request body to
`handle()`:

```php
use Workflow\V2\Support\InvocableActivityHandler;

$handler = new InvocableActivityHandler([
    'billing.charge-card' => static function (int $amount, string $currency): array {
        // Real charge-card logic. Must be idempotent per task id.
        return [
            'approved' => true,
            'amount' => $amount,
            'currency' => $currency,
        ];
    },
]);

$envelope = json_decode(file_get_contents('php://input'), associative: true);
$result = $handler->handle($envelope);

header('Content-Type: application/vnd.durable-workflow.external-task-result+json');
echo json_encode($result, JSON_UNESCAPED_SLASHES);
```

The handler receives the input envelope and returns the result envelope.
The carrier contract (HTTPS, POST, auth, timeouts, retry budget) is owned by
the invocable HTTP carrier. The PHP helper owns argument decoding, handler
dispatch, deadline enforcement, result encoding, and the failure taxonomy.

## Registering Handlers

The first constructor argument is a map keyed by the `task.handler` value the
server sends on the input envelope. That value is the `handler` field on the
matching entry in the external executor config:

```php
new InvocableActivityHandler(
    handlers: [
        'billing.charge-card' => [$billingService, 'chargeCard'],
        'billing.refund' => [$billingService, 'refund'],
        'ops.rotate-key' => static fn (string $keyId): array => $keys->rotate($keyId),
    ],
    carrier: 'billing-lambda',
    resultCodec: 'avro',
);
```

An input with a `task.handler` that is not registered produces a
`failed` result with `failure.kind = application`,
`classification = application_error`, and
`type = UnknownActivityHandler`. The configured activity retry policy still
applies on the server side.

The optional `carrier` name is echoed in `metadata.carrier` on every result
envelope so operators can tell which external runtime produced the response.
Use a stable, redaction-safe identifier (`billing-lambda`,
`ops-cloud-run`, `laravel-admin-api`).

The optional `resultCodec` controls how the helper serializes the return
value into `result.payload.blob`. It defaults to `avro`, which matches the
codec that PHP workers already use for durable payloads. The result codec
must be one the server's `CodecRegistry` knows about; `protobuf` is not
accepted and fails fast in the constructor.

## Result Envelope

On success, `handle()` returns the carrier-neutral success envelope:

```json
{
  "schema": "durable-workflow.v2.external-task-result",
  "version": 1,
  "outcome": {
    "status": "succeeded",
    "recorded": true
  },
  "task": {
    "id": "acttask_01HV7D3G3G61TAH2YB5RK45XJS",
    "kind": "activity_task",
    "attempt": 1,
    "idempotency_key": "attempt_01HV7D3KJ1C8WQNNY8MVM8J40X"
  },
  "result": {
    "payload": {
      "codec": "avro",
      "blob": "<base64-encoded payload>"
    },
    "metadata": {
      "content_type": "application/vnd.durable-workflow.result+json"
    }
  },
  "metadata": {
    "handler": "billing.charge-card",
    "carrier": "billing-lambda",
    "duration_ms": 42
  }
}
```

The `task.id`, `task.attempt`, and `task.idempotency_key` fields are copied
from the input envelope so the server can reconcile the result with the
original lease.

On failure, `handle()` returns the failure envelope. The `failure` block
carries the kind, classification, message, originating PHP type, stack
trace, and whether the failure is retryable. A deadline failure also
includes the `deadline` name and `expires_at` value:

```json
{
  "schema": "durable-workflow.v2.external-task-result",
  "version": 1,
  "outcome": {
    "status": "failed",
    "retryable": true,
    "recorded": true
  },
  "task": {"id": "...", "kind": "activity_task", "attempt": 1, "idempotency_key": "..."},
  "failure": {
    "kind": "timeout",
    "classification": "deadline_exceeded",
    "message": "Invocable activity task received after lease.expires_at.",
    "type": "ExternalTaskDeadlineExceeded",
    "stack_trace": null,
    "timeout_type": "deadline_exceeded",
    "cancelled": false,
    "details": {
      "deadline": "lease.expires_at",
      "expires_at": "2026-04-22T15:14:02.000000Z"
    }
  },
  "metadata": {"handler": "billing.charge-card", "carrier": "billing-lambda", "duration_ms": 3}
}
```

## Failure Taxonomy

| `failure.kind` | `classification` | Retryable | When it fires |
| --- | --- | --- | --- |
| `timeout` | `deadline_exceeded` | yes | Lease or input deadline already expired when the envelope arrived, or the handler returned after one expired during execution. |
| `decode_failure` | `decode_failure` | no | Arguments could not be decoded with the declared codec, a deadline string was unparseable, the handler raised `TypeError` / `ValueError` on its parameters, or the success payload could not be re-encoded with the configured result codec. |
| `application` | `application_error` | depends on thrown exception | The handler threw. `retryable` is `false` when the thrown exception implements `Workflow\Exceptions\NonRetryableExceptionContract`; otherwise `true`. The message is the exception message and `type` is the exception class. |
| `application` | `application_error` | no | `task.kind` is not `activity_task`, or `task.handler` is not registered in the map. |

The helper never returns a bare exception. Every code path produces a
structured result envelope so the invocable carrier can reconcile the
response deterministically.

## Deadlines And Idempotency

The input envelope carries the active `lease.expires_at` plus the declared
activity deadlines (`schedule_to_start`, `start_to_close`,
`schedule_to_close`, `heartbeat`). The helper checks all of them:

- Before dispatching the handler, any expired deadline short-circuits into a
  `timeout` failure with `details.deadline` naming which field expired. The
  registered callable is not invoked.
- After the handler returns, the helper re-checks the deadlines. A handler
  that ran for longer than its lease produces the same `timeout` failure
  and the return value is dropped from the envelope.

Because the carrier retries transport delivery and the runtime redelivers
leases that were not reported, the same `task.id` and
`task.idempotency_key` can arrive more than once. Handler code must be
idempotent. The idempotency key on every envelope is stable across retries
for the same attempt.

## Payload Codecs And External Storage

Argument payloads arrive on `payloads.arguments` with a `codec` field. The
helper uses the server's `CodecRegistry` to decode them. The only public v2
codec is `avro`; `json`, unknown codecs, and malformed or untagged blobs fail
closed. JSON remains the HTTP carrier document, not a durable payload codec.

When workflow inputs exceed the configured
[external payload storage](../features/external-payload-storage.md)
threshold, the server stores the bytes in the configured driver and sends
a reference envelope instead of an inline blob. Pass an
`ExternalPayloadStorageDriver` into the constructor so the helper can
resolve references before calling the handler:

```php
use Workflow\V2\Contracts\ExternalPayloadStorageDriver;
use Workflow\V2\Support\InvocableActivityHandler;

$handler = new InvocableActivityHandler(
    handlers: $handlers,
    carrier: 'billing-lambda',
    resultCodec: 'avro',
    externalStorage: $driver,
);
```

The driver must satisfy the same contract the server uses to write the
payload so the reference, hash, and codec all match. A reference that the
external process cannot resolve produces a `decode_failure` instead of a
silent empty payload.

## Production Wiring

A Laravel controller that hosts the handler behind a single POST route
looks like this:

```php
use Illuminate\Http\JsonResponse;
use Illuminate\Http\Request;
use Workflow\V2\Support\InvocableActivityHandler;

final class BillingActivityController
{
    public function __construct(private readonly InvocableActivityHandler $handler) {}

    public function __invoke(Request $request): JsonResponse
    {
        $result = $this->handler->handle($request->all());

        return new JsonResponse(
            data: $result,
            headers: ['Content-Type' => 'application/vnd.durable-workflow.external-task-result+json'],
        );
    }
}
```

Route the external executor config at the HTTPS URL that fronts this
controller and declare an `auth_ref` so the route is authenticated. The
loopback HTTP exemption only applies to developer iteration.

## Related Surfaces

- [Invocable HTTP Carrier](./invocable-carrier.md) — the server-side carrier
  contract that delivers input envelopes and reconciles result envelopes.
- [External Execution Surface](./external-execution.md) — the carrier-neutral
  product boundary, including the input and result envelope schemas the
  helper implements.
- [External Payload Storage](../features/external-payload-storage.md) — how
  oversized arguments or results are offloaded to a configured driver and
  represented as a verifiable reference envelope.
- [Worker Protocol](./worker-protocol.md) — the broader worker-plane contract
  that publishes the invocable carrier contract alongside poll-based handler
  shapes.

<!-- Source: docs/polyglot/rust.md -->

# Rust SDK

Use this guide for the general-access SDK surface, then continue to the
generated API reference for individual types and methods. No Cloud account is
required: the primary path uses the published Rust crate with a self-hosted
Server. Developers already enrolled in Durable Workflow Cloud controlled early
access can instead choose the clearly separate [managed-runtime
quickstart](./rust-cloud-quickstart.md).

The first-party Rust SDK is a workflow-authoring surface, not only a protocol
compatibility client. Rust authors deterministic workflows, activities, and
long-running worker services against the same durable execution model used by
PHP and Python. The async control-plane client starts, signals, queries,
updates, cancels, terminates, and awaits executions; the worker runtime replays
workflow history, runs workflow/activity/update handlers, reports worker and
activity heartbeats, and exchanges language-neutral payloads with a self-hosted
Server or Durable Workflow Cloud namespace runtime.

The [Rust documentation landing page](https://rust.durable-workflow.com/)
provides the SDK index and general entry points. For crate modules, structs,
traits, and methods, continue to the generated [Rust SDK API
reference](https://rust.durable-workflow.com/durable_workflow/).

The stable Rust SDK supports durable
timers, child workflows, activity retries and timeouts, signals, replayed query
handlers, cancellation and termination, server-enforced workflow deadlines,
typed side effects, version markers, updates, and typed terminal/replay
failures. It does not yet claim schedule management. Use the
[2.0 Capability Index](/docs/capabilities/) instead of assuming every SDK
has identical feature breadth.

## Try the local Sample App playground

For the shortest no-Cloud authoring journey, open the current Sample App
[`main` branch in GitHub Codespaces](https://codespaces.new/durable-workflow/sample-app?quickstart=1&ref=main)
and run:

<!-- docs-example id="sdk.rust.sample-app-playground" -->
```bash
scripts/playground rust
```

The local playground generates caller-owned workflow and activity source,
selects the current stable artifacts, and starts the published
Server and Waterline. It waits for a worker registration whose identity,
workflow type, activity type, and task queue match the generated contract before
starting the workflow. Success requires the expected completed result and
history; the terminal then prints the exact local Waterline run link and the
path to structured JSON evidence.

This intentionally small, transport-first scaffold uses `serde_json::Value`;
it is not the only recommended Rust application contract. Continue to the
crate's existing [typed input/output
example](https://github.com/durable-workflow/sdk-rust/blob/main/examples/hello_world.rs),
[retry, timeout, heartbeat, and terminal-failure activity policy
example](https://github.com/durable-workflow/sdk-rust/blob/main/examples/activity_options.rs),
and [cooperative-cancellation heartbeat
example](https://github.com/durable-workflow/sdk-rust#heartbeats). The package,
repository example, and generated API reference below remain the direct paths
for users who do not want Sample App.

## Package and source

- [Crate on crates.io](https://crates.io/crates/durable-workflow)
- [Source repository](https://github.com/durable-workflow/sdk-rust)

Install the Rust SDK from the last passing qualified tuple. The exact requirement
is generated from the same machine-readable authority as the Server quickstart:

<!-- docs-example id="rust.sdk.install" -->
```bash
cargo add durable-workflow@=2.0.0
```

Or declare the same qualified requirement directly in `Cargo.toml`:

```toml
[dependencies]
durable-workflow = "=2.0.0"
```

The crate requires Rust 1.86 or newer. Its package metadata declares the exact
qualified Durable Workflow Server range, worker protocol 1.2, and control plane
2. During deployment, the protocol
manifests advertised by `GET /api/cluster/info` remain authoritative.

Server negotiates worker-protocol headers within major `1`: a server
advertising `1.N` accepts a worker header `1.M` only when `M <= N`. Rust SDK
workers send `X-Durable-Workflow-Protocol-Version: 1.2`, so they require
the same synchronized server train, which must also advertise worker
protocol `1.2` or newer. The current server advertises `1.13`, accepts the
Rust header, and returns `1.13` in its response header and body.

Negotiation fails closed. A missing or malformed header, a different major,
or a worker minor newer than the server's advertised minor is rejected. The
server version range selects the release family; it does not override the
runtime protocol manifest.

## Prepare the released repository example

The repository's
[`hello_world` example](https://github.com/durable-workflow/sdk-rust/blob/main/examples/hello_world.rs)
registers a Rust worker, starts a workflow, sends a signal, runs an activity,
reports an activity heartbeat, and waits for the completed result. Because that
example runs the application client and worker in one process, the no-account
path below connects it to a self-hosted Server. A provisioned Cloud namespace
is an optional secondary runtime with separate role-scoped credentials.

For a reproducible source exercise, obtain the exact crate source recorded by
the qualified tuple. The value below is generated from that machine-readable
authority. The example directory is absolute so either connection path can
enter it directly:

<!-- docs-example id="rust.sdk.repository-source" -->
```bash
export DURABLE_WORKFLOW_RUST_VERSION=2.0.0
export DURABLE_WORKFLOW_RUST_EXAMPLE_DIR="$PWD/durable-workflow-rust-${DURABLE_WORKFLOW_RUST_VERSION}"
git clone --depth 1 --single-branch --branch "$DURABLE_WORKFLOW_RUST_VERSION" \
  https://github.com/durable-workflow/sdk-rust.git "$DURABLE_WORKFLOW_RUST_EXAMPLE_DIR"
```

## Run the combined example with self-hosted Server

The unmodified `hello_world` example accepts one token for a self-hosted
Durable Workflow Server whose authentication policy allows the same credential
to make workflow commands and poll for work:

<!-- docs-example id="rust.sdk.self-hosted" -->
```bash
cd "$DURABLE_WORKFLOW_RUST_EXAMPLE_DIR"
DURABLE_WORKFLOW_SERVER_URL=http://localhost:8080 \
DURABLE_WORKFLOW_TOKEN=dev-token \
cargo run --example hello_world
```

The example uses the `default` namespace provisioned by the local Server
quickstart. Use `TASK_QUEUE` to override its default `rust-workers` task queue.
No Cloud enrollment or Cloud credential is involved in this path.

## Optional secondary path: connect to Durable Workflow Cloud

For the shortest managed-runtime proof, use the [Rust Cloud
quickstart](./rust-cloud-quickstart.md). It runs the same symmetric Sample App
playground available to PHP and Python, selects current stable artifacts
dynamically, verifies exact worker readiness, and waits for a completed result.

The lower-level combined SDK example below is useful when adapting an existing
Rust process. Export the values returned when Cloud provisions the namespace
and creates its two runtime credentials:

See [Cloud Managed Runtime](/docs/polyglot/cloud-control-plane) for the
managed connection boundary.

<!-- docs-example id="rust.sdk.cloud.environment" -->
```bash
export DURABLE_WORKFLOW_RUNTIME_URL='https://your-runtime-url'
export DURABLE_WORKFLOW_RUNTIME_NAMESPACE='orders'
export DURABLE_WORKFLOW_CLIENT_TOKEN='dwr_client_credential'
export DURABLE_WORKFLOW_WORKER_TOKEN='dwr_worker_credential'
```

In `$DURABLE_WORKFLOW_RUST_EXAMPLE_DIR/examples/hello_world.rs`, replace the
existing `server_url`, `token`, and `Client::builder(...)` setup with this
split-token builder configuration:

<!-- docs-example id="rust.sdk.cloud.client" -->
```rust
let runtime_url = std::env::var("DURABLE_WORKFLOW_RUNTIME_URL")
    .expect("DURABLE_WORKFLOW_RUNTIME_URL must be set");
let runtime_namespace = std::env::var("DURABLE_WORKFLOW_RUNTIME_NAMESPACE")
    .expect("DURABLE_WORKFLOW_RUNTIME_NAMESPACE must be set");
let client_token = std::env::var("DURABLE_WORKFLOW_CLIENT_TOKEN")
    .expect("DURABLE_WORKFLOW_CLIENT_TOKEN must be set");
let worker_token = std::env::var("DURABLE_WORKFLOW_WORKER_TOKEN")
    .expect("DURABLE_WORKFLOW_WORKER_TOKEN must be set");

let client = Client::builder(runtime_url)
    .namespace(runtime_namespace)
    .control_token(Some(client_token))
    .worker_token(Some(worker_token))
    .build()?;
```

Then run the example normally:

<!-- docs-example id="rust.sdk.cloud.run" -->
```bash
cd "$DURABLE_WORKFLOW_RUST_EXAMPLE_DIR"
cargo run --example hello_world
```

The example's workflow start, signal, describe, and result calls use
`control_token`; its `Worker` registration, polling, heartbeat, and completion
calls use `worker_token`. Both roles use the same Cloud-provided runtime URL,
runtime namespace, and task queue, but they do not reuse a credential. Do not
replace either Cloud credential with `.token(...)`: that method is the generic
single-token fallback used by the self-hosted configuration above.

## Start with server-enforced workflow timeouts

The Rust SDK provides `WorkflowStartOptions` and
`Client::start_workflow_with_options` for workflow deadlines that the server
enforces even after the starting process exits. Execution timeout covers the
whole workflow instance, including continue-as-new runs; run timeout covers one
run and is recomputed when a new run begins.

<!-- docs-example id="rust.workflow-start-timeouts" -->
```rust
use durable_workflow::{json, Client, Result, WorkflowStartOptions};

async fn start(client: &Client) -> Result<()> {
    let handle = client.start_workflow_with_options(
        "orders.await-payment",
        "orders",
        "order-42",
        WorkflowStartOptions::new()
            .execution_timeout_seconds(300)
            .run_timeout_seconds(30),
        json!([{"order_id": "order-42"}]),
    ).await?;

    println!("workflow={} run={:?}", handle.workflow_id, handle.run_id);
    Ok(())
}
```

Both values are seconds, must be positive, and the run timeout cannot exceed
the execution timeout. The existing `Client::start_workflow` convenience
method uses `WorkflowStartOptions::default()`: a 3600-second execution timeout
and a 600-second run timeout. See [Timeouts](/docs/features/timeouts) for
the server's deadline and continue-as-new semantics.

These are workflow policy, not HTTP or result-polling timeouts. In particular,
`WorkflowResultOptions::timeout` only stops the local `result()` call from
waiting. It does not close, cancel, or otherwise change the workflow run. The
caller can inspect the returned identity and wait again. An execution or run
deadline configured with `WorkflowStartOptions` is durable server state; when
it expires, the server closes the run with a terminal `timed_out` outcome.

## Deterministic parallel groups

`WorkflowContext::parallel` and its `join` alias compose activities, child
workflows, timers, mixed groups, and nested groups without adding a new Server
command. Constructors on `ParallelOperation` defer every leaf until the entire
tree is known:

```rust
use durable_workflow::{json, ChildWorkflowOptions, ParallelOperation};
use std::time::Duration;

let results = ctx.parallel(vec![
    ParallelOperation::activity("load-profile", json!(["customer-42"])),
    ParallelOperation::group(vec![
        ParallelOperation::child_workflow(
            "quote-shipping",
            ChildWorkflowOptions::new("shipping-workers"),
            json!(["customer-42"]),
        ),
        ParallelOperation::timer(Duration::from_secs(1)),
    ]),
]).await?;
```

Every leaf carries the shared stable group identity plus its complete
outer-to-inner path. Successful results retain nested input shape and order.
`Error::ParallelFailed` preserves the typed leaf cause, deterministic member
path, group path, and completed siblings. Pending and completed histories replay
without rescheduling; exact duplicates and late completions do not change the
chosen positional outcome.

## Durable first-completion selection

`WorkflowContext::select` and `select_keyed` start activities, child workflows,
timers, signals, condition waits, and nested ordinary groups together, then
resume from the winner recorded by the runtime:

```rust
use durable_workflow::{json, ParallelOperation, SelectionKey};
use std::time::Duration;

let selected = ctx.select_keyed(vec![
    ("resolver", ParallelOperation::activity("resolve-request", json!([request_id]))),
    ("input", ParallelOperation::signal("resolution.received")),
    ("deadline", ParallelOperation::timer(Duration::from_secs(2))),
]).await?;

if selected.key == SelectionKey::Name("deadline".to_string()) {
    if let Some(resolver) = selected.handle(&SelectionKey::Name("resolver".to_string())) {
        resolver.cancel().await?;
    }
}
```

`SelectionResult` preserves the stable key/index, operation kind and identity,
typed value or failure, and a `DurableOperationHandle` for every member. A loser
continues unless workflow code later awaits `handle.await_result()` or awaits
`handle.cancel()`. The cancellation future resolves to unit; only committed
`SelectionOperationCancelled` history proves cancellation won. Replay advances
past an unmarked request, and `await_result()` still returns a completion that
committed first. Restart and replay consume the persisted winner; later or
duplicate terminal events cannot change it.

## Saga compensation

Register a compensation only after the corresponding forward activity has
completed, then give the forward result to `Saga::finish`:

```rust
let mut saga = ctx.saga();
let outcome = async {
    let flight = ctx.activity("trip.reserve-flight", json!([])).await?;
    saga.add_compensation("trip.cancel-flight", json!([flight]))?;

    let hotel = ctx.activity("trip.reserve-hotel", json!([])).await?;
    saga.add_compensation("trip.cancel-hotel", json!([hotel]))?;

    ctx.throw_if_cancellation_requested()?;
    ctx.activity("trip.charge", json!([])).await?;
    Ok(json!({"status": "booked"}))
}.await;

saga.finish(outcome).await
```

Failure or cooperative cancellation runs the existing activity command in
reverse registration order, one compensation at a time. Compensation stops at
its first failure. `Error::SagaCompensationFailed` preserves both typed errors,
the compensation activity type, and its registration order through restart and
replay.

## Deterministic side effects and version markers

The Rust SDK records small non-deterministic values
with `WorkflowContext::side_effect` and derives deterministic UUIDv4 values
with `WorkflowContext::uuid_v4`. A cold replay decodes the recorded value
instead of invoking the callback again; reordered, missing, duplicate, or
codec-incompatible markers return a typed `Error::NonDeterministicReplay`.

Use `WorkflowContext::get_version(change_id, min_supported, max_supported)` to
keep old and new workflow branches replay-compatible during a rollout. New
runs record `max_supported`; existing runs reuse the durable marker.
`patched(change_id)` provides the boolean rollout form, and
`deprecate_patch(change_id)` preserves the marker after the legacy branch has
drained. See [Side Effects](/docs/features/side-effects/) and
[Versioning](/docs/features/versioning/) for the shared durable semantics.

## Cancel, terminate, and handle terminal outcomes

The 2.0 baseline separates cooperative cancellation from forced termination.
Cancellation is the normal lifecycle operation when workflow and
activity code should observe the stop request and clean up. Termination closes
the run without waiting for that cleanup and should be reserved for an
operator-enforced stop.

```rust
use durable_workflow::{Client, WorkflowCommandOptions};

# async fn cancel(client: &Client) -> durable_workflow::Result<()> {
client.cancel_workflow(
    "order-42",
    WorkflowCommandOptions::new()
        .reason("customer withdrew the order")
        .request_id("cancel-order-42"),
).await?;
# Ok(())
# }
```

Instance-targeted `cancel_workflow` and `terminate_workflow` resolve the
current run on the server. For selected-run safety, call
`cancel_workflow_run` or `terminate_workflow_run`, or use a handle's
`cancel_selected_run` and `terminate_selected_run` methods. If a selected run
is stale, `Error::WorkflowCommandRejected` exposes the stable
`historical_run_command_rejected` reason together with workflow ID, run ID,
target scope, HTTP status, and the response body.

Successful `WorkflowHandle::result` calls continue to return the decoded JSON
value. Match the typed terminal variants for every other outcome. Branch on
the stable reason and category fields instead of display text:

<!-- docs-example id="rust.typed-workflow-timeouts" -->
```rust
use durable_workflow::{Error, WorkflowHandle, WorkflowResultOptions};

# async fn wait(handle: WorkflowHandle) -> durable_workflow::Result<()> {
match handle.result(WorkflowResultOptions::default()).await {
    Ok(value) => println!("completed: {value}"),
    Err(Error::WorkflowCancelled(outcome)) => {
        println!("cancelled {:?}: {}", outcome.run_id, outcome.reason);
    }
    Err(Error::WorkflowTerminated(outcome)) => {
        println!("terminated: {}", outcome.reason);
    }
    Err(Error::WorkflowFailed(outcome)) => {
        println!("failure {:?}: {:?}", outcome.failure_id, outcome.exception_class);
    }
    Err(Error::WorkflowTimedOut(outcome)) => match (
        outcome.reason.as_str(),
        outcome.failure_category.as_deref(),
    ) {
        ("result_wait_timeout", Some("client_timeout")) => {
            println!(
                "caller deadline for {} / {:?}; the run may still be open",
                outcome.workflow_id, outcome.run_id,
            );
        }
        ("execution_timeout" | "run_timeout", category) => {
            println!(
                "server timeout for {} / {:?}: reason={} category={:?}",
                outcome.workflow_id, outcome.run_id, outcome.reason, category,
            );
        }
        (reason, category) => {
            println!("other typed timeout: reason={reason} category={category:?}");
        }
    }
    Err(error) => return Err(error),
}
# Ok(())
# }
```

Each terminal outcome carries workflow and run identity. It also retains the
public reason, failure category and identity, exception type and class,
non-retryable state, message, and exception payload when the server supplies
them. A local wait deadline has reason `result_wait_timeout` and category
`client_timeout`; a server timeout is a terminal `timed_out` run whose stable
reason is `execution_timeout` or `run_timeout`.

Handles returned by either start method retain the selected `run_id`.
`WorkflowHandle::result` describes that run-specific route, so reusing the same
workflow ID for a newer run cannot make a wait silently report the newer run's
outcome. Preserve both `outcome.workflow_id` and `outcome.run_id` in logs,
metrics, and retry records; use instance-level lookups only when following the
current run is intentional.

## Workflow updates

Rust supports durable updates across application-client, selected-handle, and
worker-authoring roles. Use `Client::update_workflow` or
`WorkflowHandle::update` for JSON-compatible values, and the matching
`update_workflow_avro_value` or `update_avro_value` methods for explicitly
typed Avro values.

Workers register named handlers with `Worker::register_update`; use
`register_update_avro_value` when the handler consumes and returns Avro values
directly. An update is a durable, result-bearing workflow mutation. It is
separate from fire-and-forget signals and read-only replayed queries.

## Payload envelope

Workflow input, signals, activities, queries, and results use the published
`PayloadEnvelope` contract: a `codec` plus encoded `blob`. The SDK's default
`avro` path uses its declared `apache-avro` dependency and the platform's
fixed versioned Value schema; do not hand-roll the blob or replace the envelope
with an implementation-specific record.

<!-- docs-example id="rust.avro-payload-envelope" -->
```rust
use durable_workflow::{decode_payload, json, PayloadEnvelope, Result, Value};

fn round_trip() -> Result<()> {
    let envelope = PayloadEnvelope::avro(&json!({"order_id": "order-42"}))?;
    assert_eq!(envelope.codec, "avro");

    let decoded: Value = decode_payload(&envelope)?;
    assert_eq!(decoded["order_id"], "order-42");
    Ok(())
}
```

`start_workflow` and `start_workflow_with_options` apply this envelope
automatically to their serializable input. Use the public helpers only when a
program needs to exchange an envelope directly.

Long-running activities should heartbeat and inspect `should_stop()`. On
cancellation, release temporary files, connections, or other process-local
resources and return promptly. A late completion is rejected by durable state
and cannot convert a cancelled or terminated run into success; managed workers
continue polling after that definitive rejection and after restart.

For server images, authentication, and production topology, continue with the
[server setup guide](/docs/polyglot/server).

<!-- Source: docs/polyglot/worker-compatibility-routing.md -->

# Worker Compatibility and Routing

Use this guide when you need to deploy a new worker build, canary one task
queue, roll a bad build back, or keep long-running workflows alive through a
worker build rollout. Durable Workflow v2 treats worker compatibility as a
routing contract, not as an informal deployment convention.

The key rule is simple: a run that was started under one compatibility family
must keep landing on workers that can safely replay and execute that run. The
system must surface "no compatible worker is available" as an explicit
operational state instead of silently handing the task to a different build.

## Two related identities

Durable Workflow exposes two related but different rollout surfaces:

- **Build id** is the operator-facing cohort identity on the standalone server.
  It is the value a worker registers as `build_id`, and it is what
  task-queue rollout APIs drain or resume.
- **Compatibility marker** is the routing identity for in-flight work. In
  embedded and server-hosted PHP workers it comes from
  `DW_V2_CURRENT_COMPATIBILITY` and
  `DW_V2_SUPPORTED_COMPATIBILITIES`. On the standalone server, build-id
  cohorts are the operator-facing way to inspect and control those compatible
  worker groups.

Treat both values as opaque strings such as `orders-2026-04-28` or
`api-v3`. Durable Workflow does not interpret semver, compare dates, or guess
that one string is newer than another.

## What gets pinned

Compatibility is attached to durable workflow work, not just to live worker
processes.

- **Workflow start** records the current compatibility family on the new run.
- **Workflow tasks** inherit that compatibility family and keep it through
  retries, lease expiry, and redispatch.
- **Activity tasks** inherit the same compatibility family as their parent run.
- **Retry runs** keep the source run's compatibility family.
- **Continue-as-new** keeps the current run's compatibility family.
- **Child workflows** inherit the parent run's compatibility family.

This means a long-running workflow does not drift onto a different executor
family just because a deployment changed. New builds affect new starts unless
you deliberately drain old cohorts and move traffic.

## How routing works

Compatibility enforcement happens at more than one layer.

### Poll-time narrowing

Workers narrow what they ask for by task queue and compatibility/build cohort
so the server or embedded engine can avoid offering obviously incompatible
tasks first.

This is an efficiency hint, not the final safety boundary.

### Claim-time enforcement

Claim-time enforcement is the correctness boundary. If a worker cannot safely
run a task, Durable Workflow rejects the claim with an explicit compatibility
reason instead of silently reassigning ownership.

That is the contract to rely on during compatibility-affecting deploys:

- tasks are never silently widened to an incompatible worker
- lease expiry and redelivery preserve the original compatibility family
- a missing compatible worker is observable state, not undefined behavior

## What operators should configure

Use stable, human-readable compatibility families for builds that can safely
replay the same in-flight workflows.

For embedded or server-hosted PHP workers, set:

```bash
DW_V2_CURRENT_COMPATIBILITY=orders-2026-04-28
DW_V2_SUPPORTED_COMPATIBILITIES=orders-2026-04-28,orders-2026-04-21
```

`DW_V2_CURRENT_COMPATIBILITY` names the family new runs are pinned to.
`DW_V2_SUPPORTED_COMPATIBILITIES` names the families this worker may still
claim while a rollout or rollback is in progress. Use `*` only for
single-build fleets or narrow test harnesses; do not use it to hide an
uncertain rollout policy.

For standalone-server workers, register a stable `build_id` and use the
build-id rollout APIs to drain or resume cohorts intentionally.

## Safe rollout pattern

Use this sequence for compatibility-affecting worker changes:

1. Start the new worker cohort with a new build id or compatibility marker.
2. Keep the old cohort live until you confirm the new cohort can claim work.
3. Drain the old cohort so it stops taking new tasks but can finish or release
   what it already leased.
4. Watch the compatibility and task-queue surfaces until the old cohort is no
   longer needed.
5. Resume the old cohort only if you need to roll back.

Keep the compatibility family stable across replay-compatible rebuilds. Change
it only when the new workers must no longer claim tasks created by the older
family.

## Explicit missing-worker state

When no worker can satisfy the required compatibility family, Durable Workflow
must show that plainly:

- worker fleet summaries show that required compatibility is not covered
- task-queue and rollout surfaces show which build cohorts are still active,
  draining, stale, or gone
- workflow diagnostics surface that the run is waiting for a compatible worker
- operator health and metrics expose compatibility-blocked work as a named
  condition

This is the expected signal during a partial rollout or an incomplete rollback.
Treat it as a real operating condition to fix, not as a random transient.

## How this relates to build-id rollouts

Build-id rollout state and compatibility routing solve different problems:

- **Build-id rollout** tells operators which worker cohorts are live and lets
  them drain or resume those cohorts intentionally.
- **Compatibility routing** decides whether a specific in-flight task is
  eligible to run on a specific worker.

You usually use both together. Build-id cohorts tell you which executors are
available; compatibility routing makes sure long-running work only lands on a
compatible cohort.

## Related references

- [Namespace, Auth, And Worker Registration](/docs/polyglot/namespace-auth-workers)
  for the worker registration payload, including `build_id`
- [Worker Build-Id Rollout](/docs/polyglot/worker-build-id-rollout) for
  drain/resume lifecycle on one task queue
- [Server Config Reference](/docs/polyglot/server-config-reference) for
  `DW_V2_CURRENT_COMPATIBILITY`,
  `DW_V2_SUPPORTED_COMPATIBILITIES`, and related config
- [Operator Operating Envelope](/docs/operator-operating-envelope) for
  the health, metrics, and run diagnostics that expose compatibility gaps

<!-- Source: docs/polyglot/worker-protocol.md -->

# Worker Protocol

Durable Workflow exposes a versioned worker protocol through two bridge contracts. These contracts define the complete set of verbs that external workers — including the standalone Durable Workflow server — use to poll, claim, execute, and complete workflow and activity tasks.

## Protocol Version

The current server-advertised protocol version is **1.13**. The protocol
follows semver-style numbering:

- **Major** bumps when a change is backwards-incompatible (new required fields, removed verbs, changed pagination semantics).
- **Minor** bumps for additive changes (new optional fields, new non-terminal command types).

Workers may use an older minor within the same major. A server advertising
`1.N` accepts `X-Durable-Workflow-Protocol-Version: 1.M` when `M <= N` and
returns its advertised `1.N` on the response. The default `1.13` server
therefore accepts request versions `1.0` through `1.13`, including Rust SDK
the Rust SDK on `1.2`. Missing or malformed headers, different majors, and worker
minors newer than the server fail closed.

You can retrieve the full protocol description programmatically:

```php
use Workflow\V2\Support\WorkerProtocolVersion;

$summary = WorkerProtocolVersion::describe();
// Returns version, verb lists, command types, and pagination defaults.
```

## Capability Discovery

The standalone server publishes worker-protocol capabilities under
`worker_protocol.server_capabilities` in `GET /api/cluster/info`. The same
object is echoed as `server_capabilities` on worker-plane responses, including
poll, heartbeat, complete, and fail responses.

Read these fields before sending optional command fields:

- `supported_workflow_task_commands`: command types accepted by workflow-task completion.
- `poll_status`: poll responses carry a machine-readable status even when no
  task is leased, distinguishing `leased`, `empty`, `throttled`, and
  `unavailable` without forcing clients to infer queue state from `task: null`.
- `activity_retry_policy` and `activity_timeouts`: activity command retry and timeout options.
- `worker_session_verbs` and `worker_sessions`: worker-session lifecycle verbs,
  activity command fields, renewal behavior, failure detection, and terminal
  statuses. The worker-session runtime shape is specified by
  [`worker-sessions-runtime.schema.json`](/platform-protocol-specs/worker-sessions-runtime.schema.json).
- `child_workflow_retry_policy` and `child_workflow_timeouts`: child workflow retry and timeout options.
- `parent_close_policy`: child workflow parent-close policy support.
- `query_tasks`: server-routed workflow query tasks for external runtimes.
- `non_retryable_failures`: workflow and activity failure metadata support.

Before polling, each worker must register its namespace, task queue, runtime,
supported type keys, and local capacity through `POST /api/worker/register`.
The [Namespace, Auth, And Worker Registration](/docs/polyglot/namespace-auth-workers)
reference freezes that registration payload and the role-scoped auth contract.
For the broader ready-task discovery and lease-assignment contract behind these
verbs, see [Task Matching and Dispatch](/docs/polyglot/task-matching-dispatch).
For activity affinity across multiple durable steps, see
[Worker Sessions](/docs/features/worker-sessions).

## Execution Semantics

Worker transport is at-least-once distributed coordination, not exactly-once
delivery:

- workflow-task replay rebuilds state from committed history; it is not
  workflow-level retry
- activity-task lease expiry can trigger redelivery to another worker
- late completion or failure reports can be rejected as stale because another
  attempt already won the durable race

Use `activity_execution_id` as the default remote idempotency key when a
worker or carrier talks to another system. Reach for `activity_attempt_id`
only when the downstream system must distinguish separate tries of the same
logical activity execution.

Read [Execution Guarantees and Idempotency](/docs/constraints/execution-guarantees)
for the authoritative replay, retry, lease-expiry, redelivery, and
exactly-once durable-history contract behind these verbs.

## Workflow Task Bridge

The `WorkflowTaskBridge` contract defines how an external worker interacts with durable workflow tasks:

| Verb | Description |
|------|-------------|
| `poll` | Find ready workflow tasks matching queue and compatibility criteria |
| `claim` / `claimStatus` | Claim a specific task, acquiring a 5-minute lease |
| `historyPayload` | Retrieve the full replay history for a claimed task |
| `historyPayloadPaginated` | Retrieve history in pages for large workflows |
| `execute` | Claim and execute a task in-process using the package executor |
| `complete` | Submit commands from an external worker to complete a task |
| `fail` | Record a task failure from an external worker |
| `heartbeat` | Extend the lease on a claimed task |

### Paginated History

For workflows with large histories, use `historyPayloadPaginated` to retrieve events in pages:

```php
use Workflow\V2\Contracts\WorkflowTaskBridge;

$bridge = app(WorkflowTaskBridge::class);

$afterSequence = 0;
$allEvents = [];

do {
    $page = $bridge->historyPayloadPaginated($taskId, $afterSequence, 500);
    $allEvents = array_merge($allEvents, $page['history_events']);
    $afterSequence = $page['next_after_sequence'] ?? $afterSequence;
} while ($page['has_more']);
```

The default page size is 500 events (matching `WorkerProtocolVersion::DEFAULT_HISTORY_PAGE_SIZE` and the `default_history_page_size` value the server publishes in its worker-protocol capabilities); the maximum is 1000. Servers can advertise a different effective default through the worker-protocol manifest, so prefer reading the published capability over hard-coding either number. The response includes `has_more` and `next_after_sequence` for cursor-based pagination.

### History Compression

For workflows with very large histories, the bridge or server can compress the history events payload to reduce transfer size. Compression is opt-in: the caller must request it via an `Accept-Encoding`-style parameter.

When the event count in a response exceeds the compression threshold (50 events), the bridge may return:

- `history_events`: `[]` (empty array, signalling events are in the compressed key)
- `history_events_compressed`: base64-encoded compressed payload
- `history_events_encoding`: the algorithm used (`gzip` or `deflate`)

The caller decompresses by decoding base64, inflating with the indicated algorithm, and JSON-decoding the result to recover the original `history_events` array.

```php
use Workflow\V2\Support\HistoryPayloadCompression;

// Compress a history payload for transfer (bridge/server side).
$compressed = HistoryPayloadCompression::compress($payload, 'gzip');

// Decompress on the worker side.
$original = HistoryPayloadCompression::decompress($compressed);
```

If the caller does not request compression, or the event count is below the threshold, the response contains the standard uncompressed `history_events` array.

### Long-Poll Semantics

Both `poll` verbs support an optional long-poll mode. When the caller includes a `timeout_seconds` parameter, the bridge or server holds the connection open for up to that duration waiting for a matching task to become ready, instead of returning an empty result immediately.

| Parameter | Default | Min | Max |
|-----------|---------|-----|-----|
| `timeout_seconds` | 30 | 1 | 60 |

Behavior:

- If a task becomes ready during the wait, it is returned immediately.
- If the timeout expires with no task, the response keeps the normal poll
  envelope and returns `task: null` with `poll_status: "empty"`.
- The client should retry immediately on an empty long-poll response unless shutting down.
- HTTP-level timeouts on the transport should be set above 60 seconds to avoid premature disconnects.

Every worker poll response includes `poll_status`:

| Value | Meaning |
|------|---------|
| `leased` | The server leased a task to this worker. |
| `empty` | No matching task was ready before the short poll or long-poll timeout ended. |
| `throttled` | Admission limits prevented the server from leasing work even though the queue may still have ready tasks. |
| `unavailable` | The queue or matching path was temporarily unavailable, so the worker should treat the poll as a transient infrastructure miss. |

Branch on `poll_status` before assuming an empty poll means idle capacity.
`task: null` plus `poll_status: "throttled"` is backpressure, not absence of
work.

```php
use Workflow\V2\Support\WorkerProtocolVersion;

$semantics = WorkerProtocolVersion::longPollSemantics();
// ['default_timeout_seconds' => 30, 'min_timeout_seconds' => 1, 'max_timeout_seconds' => 60]

// Clamp a caller-supplied timeout to the valid range.
$clamped = WorkerProtocolVersion::clampLongPollTimeout($userTimeout);
```

### Poll Response Status

The server keeps one poll-response contract across workflow-task,
activity-task, and query-task polling:

- `task`: the leased task payload, or `null` when the poll did not lease work.
- `poll_status`: the machine-readable outcome for the poll attempt.
- `protocol_version` and `server_capabilities`: the echoed worker-protocol
  manifest fields.

Workers should branch on `poll_status` before making route-specific
assumptions about `task`:

| `poll_status` | Typical HTTP status | Meaning |
| --- | --- | --- |
| `leased` | `200` | The server leased work and `task` contains the payload. |
| `empty` | `200` | No matching task was ready before the poll returned. |
| `throttled` | `200` | Queue admission limits withheld a new lease for this poll attempt. |
| `unavailable` | `503` or `200` | The server could not safely coordinate the queue and returned a typed unavailable outcome. |
| `draining` | `409` | The worker's build-id cohort is draining, so the server refuses to lease new work and returns `reason: "worker_draining"`. |

### Completion, Heartbeat, and Fail Requests

Workflow-task `complete`, `heartbeat`, and `fail` endpoints all require the
worker to echo two lease-identity fields from the poll or claim response:

| Field | Type | Description |
|------|------|-------------|
| `lease_owner` | string | Worker identity that holds the task lease. Must match the `lease_owner` returned from `poll` or `claim`. |
| `workflow_task_attempt` | integer ≥ 1 | Attempt number of the leased task. Must match the `workflow_task_attempt` returned from `poll` or `claim`. |

Stale attempts or wrong lease owners are rejected before any command is
applied, so an expired worker cannot commit replay commands against a
re-claimed task.

Request endpoints and bodies:

- `POST /api/worker/workflow-tasks/{task_id}/complete` — requires `lease_owner`, `workflow_task_attempt`, and a non-empty `commands` array. See [Command Types](#command-types) for the command shapes.
- `POST /api/worker/workflow-tasks/{task_id}/heartbeat` — requires `lease_owner` and `workflow_task_attempt`. Returns the renewed lease expiry and current run status.
- `POST /api/worker/workflow-tasks/{task_id}/fail` — requires `lease_owner`, `workflow_task_attempt`, and a `failure` object containing `message` (required) plus optional `type` and `stack_trace`.

Example complete request:

```json
{
  "lease_owner": "py-worker-1",
  "workflow_task_attempt": 1,
  "commands": [
    { "type": "complete_workflow" }
  ]
}
```

Example heartbeat request:

```json
{
  "lease_owner": "py-worker-1",
  "workflow_task_attempt": 1
}
```

Example fail request:

```json
{
  "lease_owner": "py-worker-1",
  "workflow_task_attempt": 1,
  "failure": {
    "message": "Replay mismatch at event 7",
    "type": "DeterminismFailed"
  }
}
```

### Command Types

When completing a workflow task, the external worker submits a list of typed commands. At most one terminal command is allowed per completion.

**Non-terminal commands** (zero or more, processed in order):

| Type | Required Fields | Description |
|------|----------------|-------------|
| `schedule_activity` | `activity_type` | Schedule an activity task for execution |
| `start_timer` | `delay_seconds` | Schedule a durable timer |
| `start_child_workflow` | `workflow_type` | Start a child workflow instance |
| `complete_update` | `update_id` | Mark an accepted update as applied and completed |
| `fail_update` | `update_id`, `message` | Mark an accepted update as failed |
| `record_side_effect` | `result` | Record a deterministic side-effect result |
| `record_version_marker` | `change_id`, `version`, `min_supported`, `max_supported` | Record a versioning decision |
| `upsert_search_attributes` | `attributes` | Upsert indexed metadata on the workflow run |

`schedule_activity` accepts optional `retry_policy`, `start_to_close_timeout`,
`schedule_to_start_timeout`, `schedule_to_close_timeout`, and
`heartbeat_timeout` fields. `retry_policy` uses `max_attempts`,
`backoff_seconds`, and `non_retryable_error_types`.

`start_child_workflow` accepts optional `parent_close_policy`, `retry_policy`,
`execution_timeout_seconds`, and `run_timeout_seconds` fields.
`parent_close_policy` is one of `abandon`, `request_cancel`, or `terminate`.
Child retry policy uses the same `max_attempts`, `backoff_seconds`, and
`non_retryable_error_types` object shape as activities. Retry backoff applies
after a child run fails; invalid child start commands are protocol errors and
do not consume child retry attempts.

`complete_update` closes the accepted update named by `update_id` after the
worker applies the update handler. It accepts an optional `result` payload
using the same `{codec, blob}` envelope as workflow completion results.
`fail_update` closes the accepted update as failed and accepts optional
`exception_class`, `exception_type`, and `non_retryable` fields in addition to
the required `message`.

**Terminal commands** (at most one):

| Type | Required Fields | Description |
|------|----------------|-------------|
| `complete_workflow` | — | Mark the run as completed (optional `result`) |
| `fail_workflow` | `message` | Mark the run as failed |
| `continue_as_new` | — | Close the run and start a new one (optional `arguments`, `workflow_type`) |

If a cancel or terminate command closes the run while a workflow task is
leased, workflow-task `history`, `heartbeat`, `complete`, and `fail` calls keep
the worker-protocol envelope but reject with `reason: "run_closed"`. The
response also includes `can_continue: false`, `cancel_requested: true`, and a
concrete `stop_reason` such as `run_cancelled` or `run_terminated`, so workers
can distinguish cancellation observation from a generic lease error. The same
response includes `run_closed_reason` and `run_closed_at` from the durable run
record so workers can log the exact closure state that stopped the leased task.

Workflow-task poll responses include stable resume context copied from the
durable task payload:

| Field | Meaning |
|------|---------|
| `workflow_wait_kind` | The wait being applied by this task: `update`, `signal`, `child`, `condition`, `timer`, or `null` for ordinary replay/start tasks |
| `open_wait_id` | Stable wait identity such as `update:{id}` or `signal-application:{id}` |
| `resume_source_kind` / `resume_source_id` | Durable source that woke the task, such as `workflow_update`, `workflow_signal`, `timer`, or `child_workflow_run` |
| `workflow_update_id` | Accepted update id when the task applies an update |
| `workflow_signal_id` | Accepted signal id when the task applies a signal |
| `signal_name` / `signal_wait_id` | Signal target and stable wait identity when the task applies a signal or a timer-backed signal wait |
| `workflow_command_id` | Control-plane command id that produced the task, when available |
| `activity_execution_id` / `activity_attempt_id` / `activity_type` | Activity identifiers when the task resumes after a completed or failed activity |
| `child_call_id` / `child_workflow_run_id` | Child wait identifiers when the task resolves a child workflow |
| `timer_id` / `condition_wait_id` | Pure timer and timer-backed condition identifiers when the task resumes after a timer |
| `condition_key` / `condition_definition_fingerprint` | Stable condition label and predicate fingerprint when a timer-backed condition wait recorded them |
| `workflow_sequence` / `workflow_event_type` | History sequence and event type for event-backed activity, child, and timer resolution tasks |

Fields that do not apply are `null`. SDK workers should prefer these fields
over scanning history when they need to correlate a leased task with an
accepted update, signal, activity result, child resolution, or timer-backed
wait. Pure timer resumes set `workflow_wait_kind: "timer"`,
`open_wait_id: "timer:{timer_id}"`, `resume_source_kind: "timer"`, and
`timer_id`. Signal-backed resumes set `workflow_wait_kind: "signal"` plus
`signal_name`; accepted-signal application tasks also set `workflow_signal_id`
and timer-backed signal waits set `signal_wait_id` with the firing `timer_id`.
Condition-timeout resumes set `workflow_wait_kind: "condition"`,
`condition_wait_id`, and, for keyed waits, `condition_key` plus
`condition_definition_fingerprint`.

## Query Tasks

When a control-plane query targets a workflow whose code is owned by an
external runtime, the standalone server cannot replay that workflow in the PHP
process. Instead, it creates an ephemeral query task and waits for an active
non-PHP worker on the workflow's task queue to execute it.

Query tasks are read-only. Workers replay the supplied history, invoke the
registered query handler, and then complete or fail the query task. They do not
write durable history events and they are not retried after the caller's
control-plane query times out.

| Endpoint | Description |
|----------|-------------|
| `POST /api/worker/query-tasks/poll` | Long-poll for a query task on a worker's registered task queue |
| `POST /api/worker/query-tasks/{query_task_id}/complete` | Submit the query result |
| `POST /api/worker/query-tasks/{query_task_id}/fail` | Reject or fail the query |

Poll request:

```json
{
  "worker_id": "py-worker-1",
  "task_queue": "orders"
}
```

Poll response:

```json
{
  "poll_status": "leased",
  "task": {
    "query_task_id": "01J...",
    "query_task_attempt": 1,
    "workflow_id": "order-123",
    "run_id": "01J...",
    "workflow_type": "order-processing",
    "query_name": "status",
    "payload_codec": "avro",
    "workflow_arguments": { "codec": "avro", "blob": "<base64-avro-bytes>" },
    "query_arguments": { "codec": "avro", "blob": "<base64-avro-bytes>" },
    "history_events": [],
    "task_queue": "orders",
    "lease_owner": "py-worker-1",
    "lease_expires_at": "2026-04-18T12:00:00.000000Z"
  },
  "protocol_version": "1.13",
  "server_capabilities": { "query_tasks": true }
}
```

`task` is `null` when the poll returns no lease. Use `poll_status` to
distinguish an ordinary empty wait from throttling or temporary queue
unavailability. The worker must echo `lease_owner` and `query_task_attempt` on
completion or failure; stale attempts and wrong lease owners are rejected.

Complete request:

```json
{
  "lease_owner": "py-worker-1",
  "query_task_attempt": 1,
  "result": { "status": "ready" },
  "result_envelope": { "codec": "avro", "blob": "<base64-avro-bytes>" }
}
```

Fail request:

```json
{
  "lease_owner": "py-worker-1",
  "query_task_attempt": 1,
  "failure": {
    "reason": "rejected_unknown_query",
    "message": "unknown query 'status'",
    "type": "QueryFailed"
  }
}
```

Use `reason: "rejected_unknown_query"` when the workflow type has no matching
query handler; the control-plane caller receives `404`. Other worker-side
query failures should use `reason: "query_rejected"` and return `409`.
If no active worker can accept the query, the control plane returns
`query_worker_unavailable`; if no result arrives before the configured timeout,
it returns `query_worker_timeout`.

## Activity Task Bridge

The `ActivityTaskBridge` contract defines how an external worker interacts with activity tasks:

| Verb | Description |
|------|-------------|
| `poll` | Find ready activity tasks matching queue and compatibility criteria |
| `claim` / `claimStatus` | Claim a specific activity task with lease |
| `complete` | Record activity completion with a result |
| `fail` | Record activity failure, with optional codec-tagged `failure.details` |
| `status` | Check liveness and cancellation state without renewing the lease |
| `heartbeat` | Extend the lease and report optional progress |

Activity heartbeat responses include `can_continue` and `cancel_requested`
fields, allowing long-running activities to respond to cancellation requests.
When a run-level cancel or terminate command stops a leased activity, heartbeat,
complete, and fail responses also include `run_closed_reason` and
`run_closed_at`.

## Payload Codecs

Every payload byte string that crosses the worker-protocol boundary is tagged with a **`payload_codec`** naming the format of the accompanying blob. v2 uses one language-neutral codec: **`avro`** — so any SDK (PHP, Python, Go, TypeScript, Rust) can encode and decode payloads without sharing a runtime or an app key. The running server advertises its codec support on `GET /api/cluster/info` under **`capabilities.payload_codecs`**.

### The `avro` codec

`avro` is the v2 typed-value codec. Every payload uses the fixed recursive
`durable_workflow.protocol.Value` schema and standard Avro single-object
framing. Its named branches preserve booleans, signed 64-bit integers, finite
doubles, bytes, UTF-8 strings, lists, and string-keyed maps without
workflow-specific schemas or a registry. See the
[Avro Value protocol](/docs/polyglot/avro-value-protocol/).

### Wire Format: Payload Envelope

On fields that carry payload bytes (`arguments`, `result`, `payload`, etc.), the worker protocol surfaces the codec alongside the opaque string. Poll responses look like:

```json
{
  "task_id": "...",
  "payload_codec": "avro",
  "arguments": {
    "codec": "avro",
    "blob": "<base64-avro-bytes>"
  },
  "history_events": [ ... ]
}
```

The worker reads `payload_codec` and confirms it is `avro` before decoding. An unrecognised codec value is an error — the worker should not attempt to sniff or guess.

Activity completions send `result` as the same `{codec, blob}` envelope.
Activity failures may send structured diagnostic payloads under
`failure.details`; when present, `failure.details` is also a `{codec, blob}`
envelope. The server stores the details blob verbatim and records
`details_payload_codec` with the durable failure payload so non-PHP workers can
round-trip diagnostic data without PHP serialization.

The stable cross-language failure surface is `activity_type`,
`failure_category`, `exception_type`, `message`, `code`, `non_retryable`, and
codec-tagged `details`. Runtime fields such as exception class names, source
file paths, line numbers, and stack traces are diagnostics only. SDKs should
not expose those runtime fields in their default `exception_payload`; they may
surface them only when a worker or server explicitly records a `diagnostics` or
`runtime_diagnostics` envelope.

### Starting a Workflow

`POST /api/workflows` accepts `input` in two shapes:

1. **Plain JSON array** — the server adapts each JSON value to the fixed Avro
   Value schema. JSON input cannot express the bytes branch; clients that need
   bytes send an explicit Avro envelope.

   ```json
   { "workflow_type": "MyWorkflow", "input": ["hello", 42] }
   ```

2. **Explicit envelope** — for clients that already hold pre-encoded bytes:

   ```json
   {
     "workflow_type": "MyWorkflow",
     "input": { "codec": "avro", "blob": "<base64-avro-bytes>" }
   }
   ```

The server stores the blob verbatim and tags the run with the `avro` codec.

The codec is stored on the `WorkflowRun` and **propagates for the life of the run**: activity arguments, results, signal/update arguments, and child-workflow inputs are all Avro-encoded.

Embedded/package starts (workflows kicked off from PHP via `WorkflowStub::make(...)->start(...)` rather than the HTTP API) also resolve the new-run default through final v2's Avro-only codec contract.

## Resolving the Bridges

Both bridges are registered in the Laravel container and can be resolved directly:

```php
use Workflow\V2\Contracts\WorkflowTaskBridge;
use Workflow\V2\Contracts\ActivityTaskBridge;

$workflowBridge = app(WorkflowTaskBridge::class);
$activityBridge = app(ActivityTaskBridge::class);
```

## Related Guides

- [Server](./server.md) documents the control-plane endpoints and deployment
  shape that host this protocol.
- [External Execution Surface](./external-execution.md) explains the
  activity-grade worker, bridge, and handler contracts that build on the worker
  protocol.

<!-- Source: docs/polyglot/task-matching-dispatch.md -->

# Task Matching and Dispatch

Use this guide when you need to reason about how Durable Workflow finds ready
work, assigns it to compatible workers, and scales that assignment path beyond
"every node polls everything." This is the contract behind workflow-task polls,
activity-task polls, queue wake behavior, and dedicated matching-role
deployments.

The core idea is that Durable Workflow separates two concerns:

- durable history and task state stay in the database
- task matching decides which live worker gets the next eligible task

That separation matters even before you introduce a separate matching service.
It lets operators talk about ready-task discovery, queue ownership, lease
churn, and backpressure as one explicit role instead of as incidental side
effects of every worker process.

## What the matching role does

The matching role is responsible for:

- discovering ready workflow and activity tasks
- narrowing the eligible set by namespace, connection, queue, and
  compatibility family
- surfacing tasks to one worker at a time
- converting a successful claim into a lease with expiry
- preserving the same task when a claim fails, a lease expires, or a worker
  disappears
- making wake and backlog state visible to operators

Matching does not replace durable history, workflow replay, or task execution.
It decides who gets a chance to execute next.

## Deployment shapes

Durable Workflow supports three practical shapes today:

### In-worker matching

This is the default shape. Worker processes long-poll for ready work and use
claim-time fencing to ensure only one worker owns a task lease at a time.

Use this when:

- you run a single node or a small fleet
- queue pressure is moderate
- you do not need a dedicated matching daemon yet

### In-server HTTP matching

In standalone-server deployments, external workers reach the same matching
contract through the worker protocol:

- `POST /api/worker/workflow-tasks/poll`
- `POST /api/worker/activity-tasks/poll`

The server becomes the network entrypoint for the same ready-task discovery,
claim, heartbeat, and completion flow.

### Dedicated matching-role deployment

Larger fleets can concentrate broad ready-task discovery into a dedicated
matching-role process. The documented operator shape today is:

```bash
php artisan workflow:v2:repair-pass --loop
```

Run that daemon in a dedicated process and set
`DW_V2_MATCHING_ROLE_QUEUE_WAKE=0` on execution-only nodes so they stop doing
the broad queue-worker wake on every loop tick.

Use this shape when:

- execution nodes should focus on replay and activity execution
- broad independent polling is creating unnecessary contention
- operators want an explicit place to own ready-task discovery and sweep
  cadence

The dedicated role changes where matching happens, not the worker-protocol
contract. Poll, claim, lease, and redelivery semantics stay the same.

## Ready-task discovery

The matching role looks for durable tasks that are actually ready to run now:

- the task kind matches the poller (`workflow` or `activity`)
- the task is in `ready` state
- the task's `available_at` has arrived
- namespace, queue, connection, and compatibility filters still match
- activity polls also require a matching advertised activity type

Long-poll wakeups are an acceleration path, not the correctness path. If the
wake signal is delayed or missing, the task still becomes visible through
durable polling. A wake failure should raise latency, not strand work.

## Claim, lease, and backpressure

Matching only offers an opportunity. Ownership starts when one worker
successfully claims the task and receives a lease.

That lease gives Durable Workflow its backpressure behavior:

- one worker owns the task until the lease is renewed, completed, released, or
  expired
- a failed or stale worker does not permanently trap the task
- redelivery is normal and must be handled as part of at-least-once execution
- incompatible workers do not steal the lease; they get an explicit rejection

Because work is lease-based, queue pressure shows up as backlog, stale leases,
or repeated redelivery instead of as hidden in-memory loss.

## Queue partitioning

The main partitioning primitives are:

- **namespace** for tenant or environment boundaries
- **connection** and **queue** for work-routing boundaries
- **compatibility family** for compatibility-marker routing safety
- **activity type filters** for workers that only execute selected activities

Use separate queues when you want stronger isolation, independent worker pools,
or different downstream budgets. Use compatibility families when the same
queue must stay live across a rollout or rollback without letting in-flight
work drift to incompatible executors.

## Wake signals and dedicated sweeps

Wake signals shorten the time between "task became ready" and "a poller noticed
it." They are not the durable source of truth.

In the default shape, queue workers can emit wake signals as part of their
normal loop. In a dedicated matching-role deployment, execution-only nodes
disable that broad wake path and the matching daemon owns the sweep cadence
instead.

Treat these as tuning levers:

- `DW_V2_MATCHING_ROLE_QUEUE_WAKE` controls whether execution nodes perform the
  in-worker wake
- `DW_WAKE_SIGNAL_TTL_SECONDS` controls how long wake markers remain visible
- long-poll timeout settings control how long pollers stay parked waiting for
  work

## What operators should watch

Use these surfaces together:

- `GET /api/cluster/info` for the per-node role topology:
  `current_shape`, `current_process_class`, current roles, and the full
  `matching_role` contract: `queue_wake_enabled`, deployment `shape`,
  `wake_owner`, `task_dispatch_mode`, frozen `partition_primitives`, and the
  current `backpressure_model`
- task-queue visibility for ready depth, active slots, and throttling
- worker fleet visibility for active vs stale pollers on each queue
- `dw system:operator-metrics --json` or `/api/system/operator-metrics` for
  the same node-local `matching_role` contract beside live backlog, repair,
  worker, and health counters from the responding process
- health and Waterline diagnostics for unhealthy tasks, stale leases, and
  compatible-worker gaps
- rolling-upgrade checks when an overlap window is live and matching must
  block unsafe claims

If the oldest ready-task age grows while compatible workers are available, the
matching path is not making forward progress quickly enough. If ready tasks are
preserved but no compatible worker can claim them, that is a compatibility
problem, not a matching-loss problem.

## When to introduce a dedicated matching role

Move from default in-worker matching to a dedicated matching-role shape when:

- broad polling is creating visible database or cache contention
- execution nodes should stop paying the overhead of ready-task sweeps
- queue fairness and dispatch ownership need to be reasoned about as one
  explicit subsystem
- operators want a distinct process to scale, supervise, and debug for
  ready-task discovery

Stay on the default shape when the current fleet is small and the main need is
clear rollout and compatibility behavior rather than a new topology.

## Related references

- [Worker Protocol](/docs/polyglot/worker-protocol) for poll, heartbeat,
  complete, and fail verbs
- [Task Queue Admission](/docs/polyglot/task-queue-admission) for slot,
  lease, and dispatch budgets on top of matching
- [Rolling Upgrades](/docs/rolling-upgrades) for rollout procedure when
  matching and compatibility are both in play
- [Server Config Reference](/docs/polyglot/server-config-reference) for
  `DW_V2_MATCHING_ROLE_QUEUE_WAKE`, poll timing, and wake-signal settings

<!-- Source: docs/polyglot/task-queue-admission.md -->

# Task Queue Admission

Task queue admission keeps one queue, tenant, or downstream dependency from consuming the whole worker fleet. Durable Workflow exposes admission in three layers:

- worker registrations advertise local workflow and activity slots
- the server can cap active workflow and activity leases per namespace and queue
- the server can cap workflow and activity dispatches per minute per namespace and queue
- the server can cap dispatches per minute for named downstream budget groups shared by several queues
- query tasks have a bounded pending queue so synchronous reads fail fast instead of growing without limit

Use admission controls when a queue is tied to a rate-limited dependency, tenants share the same server, or operators need to prove why a workflow is waiting.

Admission sits on top of the matching contract. Read
[Task Matching and Dispatch](/docs/polyglot/task-matching-dispatch) for
how ready work is discovered and leased before these budgets decide whether the
next task is allowed through.

## How The Budget Is Applied

Workflow and activity polling starts with the workers that are currently registered for a namespace and task queue. Each worker advertises `max_concurrent_workflow_tasks` and `max_concurrent_activity_tasks`; the server sums active, non-stale workers to calculate the queue's registered slot capacity.

Server-side active lease and dispatch-rate caps are optional. When configured, the server checks a short-lived cache lock before leasing the next workflow or activity task. If the active lease cap is full, polling returns no task for that poll instead of exceeding the in-flight budget. If the per-minute dispatch cap is full, polling returns no task until the next minute bucket has capacity. Downstream budget groups apply the same per-minute dispatch behavior across every queue in the namespace that shares a `dispatch_budget_group` name.

Query tasks are different: the control plane enqueues an ephemeral query task and waits for a worker response. `DW_QUERY_TASK_MAX_PENDING_PER_QUEUE` caps how many pending query tasks can exist for each namespace and task queue. When the queue is full, new queries return `query_task_queue_full` with HTTP `429`. If the cache store cannot provide the lock needed to mutate the query-task queue, queries return `query_task_queue_unavailable` with HTTP `503`.

## Server Configuration

Set global caps when every queue should share the same ceiling:

```bash
DW_WORKFLOW_TASK_MAX_ACTIVE_LEASES_PER_QUEUE=25
DW_ACTIVITY_TASK_MAX_ACTIVE_LEASES_PER_QUEUE=100
DW_WORKFLOW_TASK_MAX_ACTIVE_LEASES_PER_NAMESPACE=500
DW_ACTIVITY_TASK_MAX_ACTIVE_LEASES_PER_NAMESPACE=2000
DW_WORKFLOW_TASK_MAX_DISPATCHES_PER_MINUTE=600
DW_ACTIVITY_TASK_MAX_DISPATCHES_PER_MINUTE=1200
DW_WORKFLOW_TASK_MAX_DISPATCHES_PER_MINUTE_PER_NAMESPACE=12000
DW_ACTIVITY_TASK_MAX_DISPATCHES_PER_MINUTE_PER_NAMESPACE=24000
DW_QUERY_TASK_MAX_PENDING_PER_QUEUE=1024
```

Queue caps protect one task queue. Namespace caps protect the tenant-wide total across every task queue in the namespace, which is useful when a tenant can shard work across many queues but still shares one downstream quota. Budget-group caps protect a named downstream dependency across selected queues without throttling every queue in the namespace.

Use `DW_TASK_QUEUE_ADMISSION_OVERRIDES` when specific queues or namespaces need different budgets. Keys are checked in this order: `namespace:task_queue`, `namespace:*`, `task_queue`, then `*`.

```bash
DW_TASK_QUEUE_ADMISSION_OVERRIDES='{
  "production:payments": {
    "workflow_tasks": {
      "max_active_leases_per_queue": 8,
      "max_dispatches_per_minute": 120,
      "dispatch_budget_group": "downstream-openai",
      "max_dispatches_per_minute_per_budget_group": 600
    },
    "activity_tasks": {
      "max_active_leases_per_queue": 12,
      "max_dispatches_per_minute": 240
    }
  },
  "production:*": {
    "workflow_tasks": {
      "max_active_leases_per_namespace": 300,
      "max_dispatches_per_minute_per_namespace": 6000
    },
    "activity_tasks": {
      "max_active_leases_per_namespace": 1200,
      "max_dispatches_per_minute_per_namespace": 12000
    }
  },
  "email": {
    "activity_tasks": {
      "max_active_leases_per_queue": 4,
      "max_dispatches_per_minute": 60,
      "dispatch_budget_group": "downstream-sendgrid",
      "max_dispatches_per_minute_per_budget_group": 300
    }
  },
  "*": {
    "workflow_tasks": { "max_active_leases_per_queue": 50 }
  }
}'
```

The override value also accepts `max_active_leases` as an alias for `max_active_leases_per_queue` and `budget_group` as an alias for `dispatch_budget_group`.

Cache must support atomic locks for server-side active lease caps, dispatch-rate caps, and query-task admission. Dispatch-rate counters are short-lived minute buckets created only for capped queues that actually lease tasks. Redis is the recommended cache store for multi-node deployments.

## Worker Slot Registration

Python workers expose local semaphores through `Worker(...)` and send the same values during registration:

```python
worker = Worker(
    client,
    task_queue="payments",
    workflows=[PaymentWorkflow],
    activities=[charge_card, send_receipt],
    max_concurrent_workflow_tasks=8,
    max_concurrent_activity_tasks=12,
)
```

For custom HTTP workers, send the slot fields to `POST /api/worker/register`:

```json
{
  "worker_id": "payments-python-1",
  "task_queue": "payments",
  "runtime": "python",
  "supported_workflow_types": ["payments.PaymentWorkflow"],
  "supported_activity_types": ["payments.charge_card", "payments.send_receipt"],
  "max_concurrent_workflow_tasks": 8,
  "max_concurrent_activity_tasks": 12
}
```

Worker slots are not a hard tenant budget by themselves. They describe what active workers can currently process. Add server caps when you need a queue-wide ceiling that still holds if more workers are deployed.

## Inspect Admission

Use the CLI when debugging:

```bash
dw task-queue:list
dw task-queue:describe payments
dw task-queue:describe payments --json | jq '.admission'
```

The server exposes the same data through:

- `GET /api/task-queues`
- `GET /api/task-queues/{name}`

An admission payload has three sections:

```json
{
  "workflow_tasks": {
    "status": "throttled",
    "active_worker_count": 3,
    "configured_slot_count": 24,
    "leased_count": 8,
    "ready_count": 5,
    "available_slot_count": 16,
    "server_max_active_leases_per_queue": 8,
    "server_active_lease_count": 8,
    "server_remaining_active_lease_capacity": 0,
    "server_max_active_leases_per_namespace": 300,
    "server_namespace_active_lease_count": 149,
    "server_remaining_namespace_active_lease_capacity": 151,
    "server_max_dispatches_per_minute": 120,
    "server_dispatch_count_this_minute": 120,
    "server_remaining_dispatch_capacity": 0,
    "server_max_dispatches_per_minute_per_namespace": 6000,
    "server_namespace_dispatch_count_this_minute": 3520,
    "server_remaining_namespace_dispatch_capacity": 2480,
    "server_dispatch_budget_group": "downstream-openai",
    "server_max_dispatches_per_minute_per_budget_group": 600,
    "server_budget_group_dispatch_count_this_minute": 600,
    "server_remaining_budget_group_dispatch_capacity": 0,
    "server_lock_required": true,
    "server_lock_supported": true,
    "budget_source": "worker_registration.max_concurrent_workflow_tasks",
    "server_budget_source": "server.admission.queue_overrides"
  },
  "activity_tasks": {
    "status": "accepting",
    "configured_slot_count": 36,
    "server_max_active_leases_per_queue": 12,
    "server_remaining_active_lease_capacity": 4,
    "server_max_dispatches_per_minute": 240,
    "server_remaining_dispatch_capacity": 197
  },
  "query_tasks": {
    "status": "accepting",
    "max_pending_per_queue": 1024,
    "approximate_pending_count": 7,
    "remaining_pending_capacity": 1017,
    "lock_supported": true,
    "budget_source": "server.query_tasks.max_pending_per_queue"
  }
}
```

## Status Reference

| Section | Status | Meaning |
|---------|--------|---------|
| Workflow/activity | `accepting` | Active workers have available slots and no server cap is full. |
| Workflow/activity | `throttled` | The optional server-side active lease cap or dispatch-per-minute cap is full. |
| Workflow/activity | `saturated` | Registered worker slots are all leased, even if no server cap is configured. |
| Workflow/activity | `no_slots` | Active workers registered zero slots for that task kind. |
| Workflow/activity | `no_active_workers` | No active, non-stale worker is polling that queue. |
| Workflow/activity | `unavailable` | A configured server cap needs a cache lock, but the lock is unavailable. |
| Query | `accepting` | The pending query-task queue has remaining capacity. |
| Query | `full` | Pending query tasks reached `DW_QUERY_TASK_MAX_PENDING_PER_QUEUE`; new queries return HTTP `429`. |
| Query | `unavailable` | The query-task queue cannot acquire its cache lock; new queries return HTTP `503`. |

## Tuning Pattern

1. Start with worker slots sized to the process: CPU-bound workflow tasks are usually lower than I/O-heavy activity tasks.
2. Add active lease caps for queues that need an in-flight ceiling across all workers.
3. Add namespace-wide active lease caps when one tenant can create many queues but still needs a total in-flight ceiling.
4. Add dispatch-per-minute caps for queues that protect a rate-limited external API, database pool, tenant, or legacy service from bursts even when workers have free slots.
5. Add budget-group dispatch caps when several queues share one downstream dependency but unrelated queues in the namespace should keep flowing.
6. Add namespace-wide dispatch caps when the downstream quota is tenant-wide rather than queue-specific.
7. Inspect `dw task-queue:describe <queue>` during load. `saturated` means add worker capacity or lower workflow fan-out. `throttled` means an active lease or dispatch-rate cap is doing its job. `no_active_workers` means the queue has no healthy poller.
8. Keep query-task capacity large enough for normal operator reads, but low enough to fail fast during incidents. Query-task overflow is backpressure, not data loss.

## Related Guides

- [Server](/docs/polyglot/server)
- [CLI](/docs/polyglot/cli)
- [Python SDK](/docs/polyglot/python)
- [Worker Protocol](/docs/polyglot/worker-protocol)

<!-- Source: docs/polyglot/worker-build-id-rollout.md -->

# Worker Build-Id Rollout

Use this reference when you cut over from unversioned workers to build-tagged
workers, canary a new build onto a task queue, drain an older build before
decommissioning it, or roll a bad build back. The server records operator
intent alongside the live worker rows so the next poll, CLI describe, or
`list_task_queue_build_ids` call reflects the rollout state honestly even if
the old workers disappear before their backlog drains.

This guide is about cohort control, not the whole routing contract. Read
[Worker Compatibility and Routing](/docs/polyglot/worker-compatibility-routing)
for the rule that in-flight work must stay pinned to compatible executors and
that "no compatible worker is available" is explicit operator state.

The Durable Workflow server expresses a rollout on one task queue as a set of
**build-id cohorts**. A cohort groups every worker registration that reported
the same `build_id` when it called `POST /api/worker/register`. Workers that
omit `build_id` form the **unversioned cohort**, which is the pre-rollout
default and the one you migrate away from on the first cutover.

## Rollout State The Server Records

Each `(namespace, task_queue, build_id)` cohort carries the aggregated worker
state (active, draining, stale, total counts) plus operator intent:

| Field | Purpose |
| --- | --- |
| `build_id` | The registered build identity. `null` identifies the unversioned cohort. |
| `rollout_status` | Aggregate view of what the cohort will do with new tasks: `active`, `active_with_draining`, `draining`, `stale_only`, or `no_workers`. |
| `drain_intent` | Operator intent for the cohort: `active` or `draining`. |
| `drained_at` | When the cohort was first marked draining. Absent while the cohort is active. Repeated drain calls do not shift this timestamp. |
| `active_worker_count` | Live workers currently accepting new tasks. |
| `draining_worker_count` | Live workers that still hold in-flight tasks but no longer claim new work. |
| `stale_worker_count` | Workers whose last heartbeat is older than the stale cutoff. |
| `total_worker_count` | Sum of the three cohort populations. |
| `runtimes`, `sdk_versions` | Distinct runtime and SDK version strings observed across the cohort. |
| `last_heartbeat_at`, `first_seen_at` | Cohort-wide heartbeat window, useful for confirming quiet cohorts before deleting them. |

`drain_intent` is persistent: resuming a cohort, stopping every worker, or
letting the cohort go stale does not silently flip it back to `active`. Only
an explicit `POST .../build-ids/resume` clears `drain_intent` and `drained_at`.
This keeps `rollout_status` honest even after a cohort has no live workers.

## Inspect The Rollout

Before draining or deleting a build, confirm which cohorts are still
reachable on the queue:

```bash
curl -sS "$DURABLE_WORKFLOW_SERVER_URL/api/task-queues/orders-critical/build-ids" \
  -H "Authorization: Bearer $DW_OPERATOR_TOKEN" \
  -H "X-Namespace: orders-prod" \
  -H "X-Durable-Workflow-Control-Plane-Version: 2"
```

The same snapshot is available from the operator CLI and the Python SDK:

```bash
dw task-queue:build-ids orders-critical --json
```

```python
from durable_workflow import Client

async with Client("https://durable-workflow.example", token=operator_token) as client:
    rollout = await client.list_task_queue_build_ids("orders-critical")
    for cohort in rollout.build_ids:
        print(cohort.build_id, cohort.rollout_status, cohort.total_worker_count)
```

## First Cutover: Unversioned To Versioned

A queue that has always been served by unversioned workers reports a single
`build_id: null` cohort with `rollout_status: "active"`. The first cutover
introduces a new build-tagged cohort alongside it.

1. Deploy the new worker fleet with a stable `build_id` (for example,
   `orders-worker-2026-04-22`) registered through
   `POST /api/worker/register`.
2. Confirm both cohorts are active:
   ```bash
   dw task-queue:build-ids orders-critical --json
   ```
   You should see `null` and the new `build_id` each reporting
   `rollout_status: "active"` and non-zero `active_worker_count`.
3. Start the drain on the unversioned cohort once the new workers are
   handling work:
   ```bash
   dw task-queue:drain orders-critical --unversioned
   ```
   `drain_intent` flips to `draining` for the unversioned cohort. Workers
   that are still running process their in-flight tasks but stop claiming
   new ones. Future worker registrations or heartbeats that arrive without
   a `build_id` land as draining too.
4. Wait until `active_worker_count` and `draining_worker_count` are both
   zero for the unversioned cohort. The cohort stays listed with
   `drain_intent: "draining"` so you can confirm the cutover is permanent.

## Canary A New Build

A canary is a second build that takes a small fraction of traffic while the
primary build keeps serving. Use a separate `build_id` for the canary so each
cohort's state is individually inspectable.

1. Deploy the canary workers with `build_id: orders-worker-2026-04-22-canary`.
2. Inspect `list_task_queue_build_ids` to confirm both cohorts report
   `rollout_status: "active"` with the expected worker counts.
3. Promote by starting more workers on the new `build_id` and reducing the
   primary's worker count, or demote the canary by draining it:
   ```bash
   dw task-queue:drain orders-critical --build-id orders-worker-2026-04-22-canary
   ```

The server does not control the task split across cohorts. Operators size
the cohort populations and rely on polling distribution to weight traffic.
Build-id rollout state exists so operators can confirm which cohorts can still
claim work and trigger a clean handoff when one cohort is ready to stop.

## Drain An Older Build

Draining keeps already-leased tasks on the older build while new tasks go to
other active cohorts on the queue:

```bash
dw task-queue:drain orders-critical --build-id orders-worker-2026-04-21-z9
```

The server stamps `drain_intent: "draining"` on the cohort and marks every
worker registered under that `build_id` as draining on its next heartbeat.
The call is idempotent: rerunning it does not reset `drained_at`, so you can
safely retry it from automation.

Once a worker row is marked `draining`, the workflow-task, activity-task, and
query-task poll routes stop leasing new work to that worker. Polls fail with
HTTP `409`, `poll_status: "draining"`, and `reason: "worker_draining"` until
the cohort is resumed.

`draining` is part of the general poll-response contract, not a special-case
drain-only field. The same `poll_status` surface is how workers observe normal
`leased` and `empty` polls, admission `throttled` outcomes, and typed
`unavailable` coordination failures on other poll paths.

Monitor the drain by polling `list_task_queue_build_ids` and watching
`active_worker_count` and `draining_worker_count` fall to zero. At that point
the cohort shows `rollout_status: "draining"` with zero worker counts, meaning
no live workers remain and operator intent still records "drained". That is
the safe moment to stop the worker processes and delete the build artifact.

## Roll Back A Bad Build

Rollback is the reverse flow: resume a previously drained cohort, route new
traffic back to it, and drain the bad build.

1. Resume the known-good cohort:
   ```bash
   dw task-queue:resume orders-critical --build-id orders-worker-2026-04-21-z9
   ```
   The server clears `drain_intent`, wipes `drained_at`, and flips any
   worker rows that are still heartbeating in under that `build_id` back to
   `active` immediately so the read endpoint stops reporting draining state.
2. Drain the bad cohort:
   ```bash
   dw task-queue:drain orders-critical --build-id orders-worker-2026-04-22
   ```
3. Scale the known-good build back up or redeploy it if workers have
   already been stopped. Workers registering under its `build_id` pick up
   the cleared drain intent and land as `active`.

Resume is also idempotent. Rerunning it against an already-active cohort
is a no-op, so automated rollback flows can issue it safely.

## Endpoints And Commands Reference

| Intent | HTTP endpoint | CLI | Python SDK method |
| --- | --- | --- | --- |
| Inspect cohort state | `GET /api/task-queues/{taskQueue}/build-ids` | `dw task-queue:build-ids` | `Client.list_task_queue_build_ids` |
| Mark a cohort as draining | `POST /api/task-queues/{taskQueue}/build-ids/drain` | `dw task-queue:drain` | `Client.drain_task_queue_build_id` |
| Resume a previously drained cohort | `POST /api/task-queues/{taskQueue}/build-ids/resume` | `dw task-queue:resume` | `Client.resume_task_queue_build_id` |

Drain and resume both take a JSON body of `{"build_id": "..."}`, or
`{"build_id": null}` for the unversioned cohort. The CLI expresses the
unversioned cohort with `--unversioned` and any other build with
`--build-id <value>`; combining the two fails fast.

## Related References

- [Namespace, Auth, And Worker Registration](/docs/polyglot/namespace-auth-workers)
  for the `POST /api/worker/register` call that stamps `build_id` on every
  worker.
- [Task Queue Admission](/docs/polyglot/task-queue-admission) for the
  worker-slot and dispatch budgets that apply alongside rollout state.
- [Server API Reference](/docs/polyglot/server-api-reference) for the
  full list of control-plane routes and their required roles and protocol
  headers.
- [CLI Command Reference](/docs/polyglot/cli-reference) for the argument
  and flag shape of every `dw task-queue:*` subcommand.

<!-- Source: docs/polyglot/php.md -->

# PHP SDK

Use `durable-workflow/sdk` when a PHP application or remote worker connects to
the standalone Durable Workflow Server or a Durable Workflow Cloud namespace
runtime URL. This first-party SDK is framework-neutral: it provides the
control-plane client, authentication, transport, public payload codec, replay
handler, and managed remote-worker lifecycle without requiring Laravel or the
embedded engine package.

## Try the local Sample App playground

For the shortest no-Cloud authoring journey, open the current Sample App
[`main` branch in GitHub Codespaces](https://codespaces.new/durable-workflow/sample-app?quickstart=1&ref=main)
and run:

<!-- docs-example id="sdk.php.sample-app-playground" -->
```bash
scripts/playground php
```

The local playground generates caller-owned workflow and activity source,
selects the current stable artifacts, and starts the published
Server and Waterline. It waits for a worker registration whose identity,
workflow type, activity type, and task queue match the generated contract before
starting the workflow. Success requires the expected completed result and
history; the terminal then prints the exact local Waterline run link and the
path to structured JSON evidence.

The package-owned quickstart and API reference below remain the direct path for
users who want to add the SDK to an existing project without Sample App.

For step-by-step onboarding, framework paths, testing, deployment, and
troubleshooting, use the authored
[PHP developer portal](https://php.durable-workflow.com/). For exact
constructor signatures, return types, and exception classes, use its distinct
[generated API reference](https://php.durable-workflow.com/api/). The portal's
[machine-readable contract](https://php.durable-workflow.com/quickstart-contract.json)
identifies the package, runtime forms, role credentials, shipped source files,
expected result, and published-artifact smoke as one tested path.

Cloud customers use the runtime URL and namespace returned during provisioning,
with separate client and worker credentials. See
[Cloud Managed Runtime](/docs/polyglot/cloud-control-plane) for that
connection boundary; the examples below show the same SDK against local
self-hosted values.

Use `durable-workflow/workflow` for the separate embedded Laravel path, where
the application owns workflow state in its existing database and executes work
through its Laravel queues. See [Deployment Modes](/docs/polyglot/deployment-modes/)
for the complete ownership comparison.

## Requirements

- PHP 8.1 or later
- A reachable [self-hosted Server](/docs/polyglot/server/) or provisioned
  [Cloud namespace runtime](/docs/polyglot/cloud-control-plane)

## Install

Install the current published PHP SDK. The exact requirement below is generated
from the registry-refreshed published-artifact authority, and Composer records
the resolved package in `composer.lock`:

<!-- docs-example id="php.sdk.install" -->
```bash
composer require durable-workflow/sdk:2.0.0
```

The SDK uses the official `apache/avro` Composer package for the public payload
envelope. Its production dependency graph excludes Laravel, Illuminate,
`durable-workflow/workflow`, and `durable-workflow/server`.

## Start and inspect a workflow

The SDK-owned quickstart creates a clean Composer project, defines one
attributed workflow and activity, starts the worker, starts a unique workflow,
and waits for the result. The same shipped `bootstrap.php`, `worker.php`, and
`client.php` files are installed from the package and executed by the protected
published-artifact smoke, so this page does not maintain a second code listing.

Choose only the runtime connection value:

| Runtime | Value passed to `Client` | SDK request path behavior |
| --- | --- | --- |
| Self-hosted Server | Bare origin such as `http://localhost:8080` | Adds one `/api` segment; callers do not append `/api`. |
| Durable Workflow Cloud | Complete provisioned URI such as `https://cloud.example/api/runtime/v1/namespaces/<runtime-id>` | Preserves the namespace runtime path and appends endpoint `/api` after it. |

Open the [tested PHP path](https://php.durable-workflow.com/) for the exact
commands and visible source. Client operations read
`DURABLE_WORKFLOW_CLIENT_TOKEN`; worker polling reads
`DURABLE_WORKFLOW_WORKER_TOKEN`. The guide keeps those credentials in separate
processes without echoing or committing either value.

`Worker::register()` discovers `#[Workflow]` and `#[Activity]` handlers in the
same source file. The guide also documents the direct
`registerWorkflow()`/`registerActivity()` alternative for callable-first code.
The bootstrap removes autoloader-path selection from users when those shipped
files run in a standalone project, SDK checkout, installed package, or a
playground/container that places them beside its Composer `vendor/` directory.

`WorkflowHandle` follows the current run after a continue-as-new transition.
Use its selected-run methods when an operation must remain guarded to one
specific run.

## Lifecycle, updates, schedules, and visibility

The current public client is broader than selected-run result handling:

- `WorkflowHandle` exposes `describe`, `result`, `signal`, `query`, `cancel`,
  and `terminate`, with selected-run variants for run-specific safety.
- `Client` exposes `listWorkflows` with server filtering and pagination,
  `workflowHistory`, `updateWorkflow`, `cancelWorkflow`, and
  `terminateWorkflow`.
- Schedule methods cover create, describe, list, update, pause, resume,
  trigger, backfill, and delete.
- Operational visibility includes `listNamespaces`, `listWorkers`, and
  `listTaskQueues`, with matching describe methods.

Remote workers register workflow, activity, query, and update handlers through
`registerWorkflow`, `registerActivity`, `registerQuery`, and `registerUpdate`.
Use the generated [PHP SDK API reference](https://php.durable-workflow.com/api/)
for complete parameters and return types.

## Run a remote PHP worker

Workflow handlers are ordinary callables that run as straight-line code inside
a managed Fiber. Call operations such as `WorkflowContext::activity()`
directly; the SDK suspends the Fiber at durable decisions and returns recorded
results during replay without repeating external activity. Do not declare a
workflow as a Generator or yield `WorkflowContext` commands: Generator results
are rejected. The managed worker registers its workflow and activity type
names, polls the public worker protocol, heartbeats, completes or fails tasks,
and handles graceful shutdown when `pcntl` is available.

### First-completion selection

Use `WorkflowContext::select()` to start independent deferred activities,
child workflows, timers, condition waits, or nested ordinary barriers and
resume from the first durably committed winner. The returned `SelectionResult`
contains stable member keys and identities plus one handle per member:

```php
$selected = $ctx->select([
    'resolver' => fn () => $ctx->activity('resolve-request', [$requestId]),
    'input' => fn () => $ctx->waitCondition(
        fn (): bool => $this->resolution !== null,
        key: 'resolution-ready',
    ),
    'deadline' => fn () => $ctx->sleep(2),
]);

if ($selected->key === 'deadline') {
    $selected->handles['resolver']->cancel();
}
```

Selection does not cancel non-winners. Call `await()` on a handle to consume
its eventual result or `cancel()` to record explicit cancellation. A cold worker
or completed-history replay consumes the recorded winner even when duplicate or
later input and terminal events appear in another delivery order.
`cancel()` returns void and does not report its terminal outcome. Only
`SelectionOperationCancelled` history proves cancellation won; replay advances
past an unmarked cancel request, and if the operation completed first,
`await()` still returns that result.

## Framework service mode and embedded Laravel

The same package ships first-party
[Laravel service-mode](https://github.com/durable-workflow/sdk-php#laravel-service-mode)
and
[Symfony service-mode](https://github.com/durable-workflow/sdk-php#symfony-service-mode)
bridges. They retain framework dependency injection, configuration, logging,
console workers, and test fakes while connecting to Cloud or Server.

Those bridges are distinct from
[embedded Laravel workflows](/docs/installation/), where
`durable-workflow/workflow` makes the Laravel application itself own durable
state and execute through Laravel queues. See
[Laravel Adoption and Runtime Transition](/docs/laravel-adoption/) for the
same representative Laravel use case across v1, v2 embedded, and this shipped
service-mode bridge, including drain and rollback. Use
[Deployment Modes](/docs/polyglot/deployment-modes/) for the wider runtime
boundary comparison.

## Protocol and release boundary

The SDK declares its supported server range, worker protocol version,
control-plane version, and payload codecs in Composer metadata. The server also
publishes its accepted protocol and codec set from `GET /api/cluster/info`.
Check runtime discovery during deployment instead of inferring compatibility
from a server patch version.

The PHP SDK is versioned independently from the 2.0 Laravel package. Keep the
exact published pin in runnable prerelease examples and evaluate release notes
when moving between pre-1.0 SDK releases; no cross-release shim is implied.

## Related references

- [Standalone Server](/docs/polyglot/server/)
- [Deployment Modes](/docs/polyglot/deployment-modes/)
- [Worker Protocol](/docs/polyglot/worker-protocol/)
- [Capability Index](/docs/capabilities/)
- [PHP developer portal](https://php.durable-workflow.com/)
- [PHP API reference](https://php.durable-workflow.com/api/)
- [PHP executable quickstart contract](https://php.durable-workflow.com/quickstart-contract.json)
- [PHP SDK source](https://github.com/durable-workflow/sdk-php)

<!-- Source: docs/polyglot/portable-worker-affinity.md -->

# Portable Worker Affinity

This is a service-mode capability reference, not an embedded Laravel feature
guide. PHP service workers implement these features; Python and Rust service
workers do not yet implement them.

Local activities, worker sessions, and sticky execution share one portability
rule: a service worker must declare each feature as supported or explicitly
refused. Protocol version `1.18` is the floor for these declarations. The
server rejects a flat routing capability unless the worker's structured
manifest marks the same feature as supported.

## SDK support

| SDK worker | Local activities | Worker sessions | Sticky execution |
| --- | --- | --- | --- |
| PHP | Supported | Supported | Supported |
| Python | Not supported | Not supported | Not supported |
| Rust | Not supported | Not supported | Not supported |

Python and Rust advertise `supported: false` for all three features. That
prevents incompatible routing; it is not feature parity or a fallback
implementation. These workers remain usable for ordinary workflows and queued
activities, with complete durable-history replay.

## Local activity recording

PHP service workers run a local activity inside the workflow worker. The
workflow-task completion contains the arguments, attempt outcomes, retry and
timeout settings, heartbeat progress, and terminal result or failure. The
server records that sequence atomically as normal activity history marked
`execution_mode=local`.

Replay consumes the recorded terminal activity event. It does not invoke the
local handler again. The synchronous PHP handler is not preempted: cancellation
and elapsed heartbeat, per-attempt, and total timeouts are observed before an
attempt, when the handler calls `ActivityContext::heartbeat()`, or after the
handler returns. A handler that neither returns nor heartbeats cannot be
interrupted by these cooperative controls, so local activities must remain
short and divide blocking work with safe heartbeat boundaries.

## Worker session lifecycle

The PHP SDK exposes typed session options and create, use, renew, and close
operations. Options include requirements, queue, lease duration, total TTL,
maximum concurrent activities, and reacquisition policy. A worker closes the
sessions it holds during graceful shutdown.

If a holder disappears, its lease and concurrency reservation expire. A new
holder may reacquire the session when requirements match, but it must rebuild
worker-local resources before the first activity uses them. Session identity
never makes process memory durable.

## Sticky execution and cold replay

The PHP cache is bounded and keyed by the exact workflow ID, run ID, and worker
build ID. It reports `hit`, `miss`, `eviction`, and `forced_cold_replay`.
Expiry, eviction, worker replacement, holder loss, or a build mismatch discards
the optimization and replays complete durable history.

Sticky routing is an affinity optimization. A forced cold replay is diagnostic
evidence that the optimization was unavailable; it is not a workflow
correctness failure. Workflow code must remain deterministic with an empty
cache.

## Safe defaults and rolling fleets

Ordinary workflows require no session or sticky configuration. Mixed-version
fleets fail closed at the protocol floor: the server checks the negotiated
version, the flat capability, the structured manifest, and exact sticky cache
identity before accepting feature-specific completion data.

The published [cross-SDK scenario manifest](/platform-conformance/portable-worker-affinity-runtime-scenarios.json)
covers manifest truth, local-activity replay, session holder loss and
reacquisition, sticky hits and eviction, worker replacement, forced cold
replay, and zero-configuration workflows.

For embedded Laravel implementations, see
[Local Activities](/docs/features/local-activities),
[Worker Sessions](/docs/features/worker-sessions), and
[Sticky Execution](/docs/features/sticky-execution). Those APIs belong to the
workflow package, not to Python or Rust service workers.

<!-- Source: docs/polyglot/avro-value-protocol.md -->

# Avro Value protocol

Durable Workflow 2.0 uses one fixed recursive
[`durable_workflow.protocol.Value`](/schemas/v2/durable_workflow.protocol.Value.v1.avsc)
schema for every Avro payload. Workflow inputs and results, activity values and
failure details, signals, queries, updates, replay history, and externally
stored payloads all use the same schema. Applications do not publish their own
Avro schemas, and the platform does not require a network schema registry.

The value union has distinct named branches for null, boolean, signed 64-bit
integer, finite double, bytes, UTF-8 string, list, and string-keyed map. That
keeps `7` distinct from `7.0`, text distinct from bytes, and lists distinct from
maps in PHP, Python, and Rust.

## Wire frame

The `blob` field is base64 around standard Avro single-object bytes:

```text
C3 01 || 8-byte little-endian CRC-64-AVRO fingerprint || Avro datum
```

Schema v1 has fingerprint `e2a33dff55802237`. SDKs bundle the immutable schema
for each supported fingerprint, select the writer schema from the frame, and
resolve it against the current reader. An unknown fingerprint or incompatible
new branch fails as `unsupported_payload_schema`; decoders do not guess or
fall back to JSON.

Future value kinds are new uniquely named record branches appended to the
union. Released branches are never reordered or reused.

## Value policy

- Map keys must be strings. SDKs reject other keys instead of stringifying
  them.
- Integers must fit the signed 64-bit Avro `long` range.
- Doubles must be finite; NaN and infinities are rejected.
- Python `bytes` and Rust `AvroValue::Bytes` select Avro `bytes`. PHP callers
  use `AvroBinaryValue::fromBytes()` because a PHP string alone cannot declare
  whether it is text or binary.
- Decimal, arbitrary-precision integer, date/time, UUID, enum, dataclass,
  Pydantic, and domain objects require explicit adapters to a canonical value
  kind.

Avro is the only Durable Workflow 2.0 payload codec. HTTP request and response
documents remain JSON transport, but every durable value inside those
documents uses this fixed schema and single-object frame. A `json` codec tag,
an unknown codec, or an untagged raw durable blob fails closed with
`unsupported_payload_codec`; runtimes never transcode or guess.

## JSON inspection projection

Run descriptions retain `input_envelope`, `output_envelope`, and result
envelopes as the lossless payload authority. JSON-facing inspection surfaces
such as the CLI and Waterline render values that JSON cannot represent with a
typed projection:

```json
{"$type":"bytes","base64":"AP8="}
{"$type":"map","entries":[{"key":"0","value":"zero"}]}
```

The map projection is used for empty maps and numeric-looking string keys that
PHP arrays cannot retain without changing their type. Ordinary scalars, lists,
and unambiguous string-keyed maps remain ordinary JSON values. Consumers that
need the original typed value decode the accompanying envelope rather than the
display projection.

## Repeatable benchmark

Each SDK ships the same representative-value benchmark and enforces a budget
for its selected production path:

```bash
# PHP SDK checkout
composer benchmark-avro-value

# Python SDK checkout
python benchmarks/avro_value.py --enforce

# Rust SDK checkout
cargo run --release --example avro_value_benchmark -- --enforce
```

The JSON output compares compact JSON, the removed JSON-in-Avro wrapper, and
the fixed typed schema. It reports raw datum, framed payload, and actual
`{codec, blob}` HTTP-envelope sizes together with end-to-end adapter,
encode, and decode latency. `AVRO_VALUE_ENCODE_BUDGET_US` and
`AVRO_VALUE_DECODE_BUDGET_US` can tighten the defaults on a qualification
runner. Release CI executes these commands with budget enforcement; a
production-path regression must be explained or corrected before release.
The old wrapper implementation exists only inside the benchmark, not as a
runtime compatibility path.

## Deployment preflight

Server bootstrap inventories every persisted `payload_codec` and verifies the
single-object magic and fixed-schema fingerprint of inline frames,
nested-history envelopes, and external payload references. Deployment stops
before the new runtime starts if any active or replay-relevant non-Avro payload,
untagged payload, corrupt reference, or obsolete frame exists. Active runs may
drain on the current prerelease; retained terminal and replay-relevant state
must follow the backup-first
[prerelease history migration](/docs/polyglot/prerelease-history-migration).
Exporting a run does not alter the rejected database state. Never delete
history to bypass the preflight.

<!-- Source: docs/polyglot/cli-python-parity.md -->

# Client and Worker Capabilities

Choose a surface for what the process needs to do. The `dw` CLI and the
first-party PHP, Python, and Rust SDKs share the v2 control-plane and worker
contracts, but their operator, client, and worker roles are not identical.

This all-client guide remains at the original
`/docs/polyglot/cli-python-parity/` route so existing links keep working.

## Capability comparison

| Capability | `dw` CLI | PHP SDK | Python SDK | Rust SDK |
| --- | --- | --- | --- | --- |
| Workflow lifecycle | **Supported:** start, list, inspect, wait, cancel, terminate, and archive operator commands. [Commands](./cli-reference.md#workflow-commands) | **Supported:** start, describe, list, await results, cancel, and terminate through `Client` and `WorkflowHandle`. [Lifecycle evidence](./php.md#lifecycle-updates-schedules-and-visibility) | **Supported:** start, list/describe, await results, cancel, and terminate through the async client. [Workflow operations](./python.md#workflow-operations) | **Supported:** start, describe, await results, cancel, and terminate through `Client` and `WorkflowHandle`. [Terminal operations](./rust.md#cancel-terminate-and-handle-terminal-outcomes) |
| Signals | **Supported:** send by workflow ID or selected run. [Commands](./cli-reference.md#workflow-commands) | **Supported:** client and handle send methods plus worker signal history. [PHP API](./php.md#lifecycle-updates-schedules-and-visibility) | **Supported:** async client sends and workflow workers handle signals. [Messages](./python.md#signals-queries-and-updates) | **Supported:** client and handle sends plus worker signal handling. [Rust API](https://rust.durable-workflow.com/) |
| Queries | **Supported:** execute a named read-only query with structured output. [Commands](./cli-reference.md#workflow-commands) | **Supported:** client and handle queries plus registered worker query handlers. [PHP API](./php.md#lifecycle-updates-schedules-and-visibility) | **Supported:** async client queries plus replayed worker query handlers. [Messages](./python.md#signals-queries-and-updates) | **Supported:** client queries and replayed worker query handlers when runtime discovery advertises query-task support. [Rust API](https://rust.durable-workflow.com/) |
| Updates | **Supported:** submit and wait for accepted or completed outcomes. [Commands](./cli-reference.md#workflow-commands) | **Supported:** `updateWorkflow` and `registerUpdate`; the PHP SDK does not expose validator authoring and declares no validators. [PHP API](./php.md#lifecycle-updates-schedules-and-visibility) | **Supported:** clients, workflow handlers, and synchronous declared validators when discovery advertises the pre-accept contract. Validator-bearing workers refuse unsupported runtimes. [Messages](./python.md#signals-queries-and-updates) | **Supported:** client, handle, Avro payloads, and registered worker update surfaces; the Rust SDK does not expose validator authoring and declares no validators. [Updates](./rust.md#workflow-updates) |
| Schedules | **Supported:** complete schedule lifecycle, backfill, and audit history. [Commands](./cli-reference.md#schedule-commands) | **Supported:** complete schedule lifecycle and listing through `Client`. [PHP API](./php.md#lifecycle-updates-schedules-and-visibility) | **Supported:** complete async schedule lifecycle and audit-history paging. [Schedules](./python.md#schedules) | **Not supported:** the current Rust SDK does not claim a schedule-management API. Use CLI, PHP, Python, or the server API. |
| Visibility | **Supported:** workflow/run search, workers, task queues, history, and diagnostic JSON. [Commands](./cli-reference.md#workflow-commands) | **Supported:** workflow filtering/pagination, history, namespaces, workers, and task queues. [Visibility evidence](./php.md#lifecycle-updates-schedules-and-visibility) | **Supported:** workflow, schedule, namespace, worker, queue, history, and search-attribute client surfaces. [Client API](./python.md#client-api-reference) | **Different:** selected-run describe/result is supported; fleet-wide list/search and namespace administration are not claimed. [Rust client](./rust.md#package-and-source) |
| Worker execution | **Intentionally different:** low-level worker-protocol commands support diagnostics and conformance; `dw` is not an application worker runtime. [Worker commands](./cli-reference.md#worker-protocol-commands) | **Supported:** remote workflow, activity, query, and update handlers through `durable-workflow/sdk`. [PHP worker](./php.md#run-a-remote-php-worker) | **Supported:** deterministic workflow and activity workers. [Python worker](./python.md#worker) | **Supported:** native workflow, activity, query, and update handlers. [Rust worker API](https://rust.durable-workflow.com/) |

**Supported** means the named current release surface exposes the capability.
**Different** identifies an intentional scope boundary. **Not supported** is
an explicit current gap, not a hidden promise. Runtime protocol discovery
remains authoritative when a capability depends on a negotiated worker
protocol.

## Evidence by product surface

### CLI

The [CLI overview](./cli.mdx) defines installation, profiles, structured
output, and exit behavior. The
[command reference](./cli-reference.md) is the complete operator surface,
including lifecycle, messages, schedules, visibility, and the low-level
worker-protocol commands that are intentionally not an SDK worker loop.

### PHP SDK

The pinned PHP SDK exposes framework-neutral client and remote-worker APIs.
Its current public surface includes workflow lifecycle and result handles,
signals, queries, updates, schedules, workflow filtering/history, namespace,
worker, and task-queue visibility, plus registered workflow, activity, query,
and update handlers. See the [PHP SDK guide](./php.md) and generated
[PHP API reference](https://php.durable-workflow.com/api/).

### Python SDK

The Python SDK combines an async control-plane client with deterministic
workflow and activity workers. Its guide publishes
[client operations](./python.md#client-api-reference),
[message handlers](./python.md#signals-queries-and-updates),
[worker execution](./python.md#worker), and
[schedule management](./python.md#schedules). The generated
[Python API reference](https://python.durable-workflow.com/) carries exact
signatures and result types.

### Rust SDK

The published Rust SDK exposes control-plane and selected-run lifecycle,
signals, replayed queries, updates, and native workflow/activity workers.
Update support includes `Client::update_workflow`,
`WorkflowHandle::update`, `Worker::register_update`, and their Avro-value
variants. Schedule management and fleet-wide list/search remain explicit gaps.
See the [Rust SDK guide](./rust.md) and generated
[Rust API reference](https://rust.durable-workflow.com/).

## Shared contract evidence

All four products target the same versioned HTTP+JSON control plane and public
payload envelope. Evidence is split by what it proves:

- the [Capability Index](/docs/capabilities/) records exact artifact floors
  and current breadth;
- the [Platform Conformance Suite](/docs/platform-conformance/) records
  cross-client and cross-worker runtime scenarios;
- CLI and Python repositories retain shared request fixtures for operations
  whose semantic request bodies are byte-for-byte compared today;
- PHP and Rust public API references and release tests establish the additional
  supported methods listed above.

The existence of a shared fixture in two repositories does not imply that
other SDKs lack the operation. Conversely, a common endpoint does not imply
that every product exposes the same operator or worker role.

## Adding or extending a client surface

When adding a new CLI or SDK operation:

1. Keep paths, methods, semantic fields, payload envelopes, and error outcomes
   language-neutral.
2. Add request and runtime evidence for every participating client or worker.
3. Document deliberately different syntax or role boundaries.
4. Mark unsupported products explicitly.
5. Treat language-specific serialization, file paths, class names, or error
   shapes as bugs in the public contract.

<!-- Source: docs/polyglot/prerelease-history-migration.md -->

# Migrate retained prerelease history

Use this offline procedure when an Avro-only Durable Workflow 2.0 deployment
reports `unsupported_payload_codec` for state retained from an earlier public
2.0 prerelease. It converts the known JSON-tagged, untagged, and obsolete Avro
Value representations without making any legacy codec available to a running
v2 application.

Do not use history export as a substitute for this procedure. An export is a
portable copy of a run; it does not change the database rows that deployment
startup inspects. Never delete history to make the preflight pass.

## Decide what must be converted

Runs that are still active can finish on the currently deployed prerelease.
Drain them before the maintenance window when that is operationally safe.
Terminal runs, closed executions retained for replay, and state that cannot
drain must be converted. The migration inventories both groups, and does not
delete either one.

The dry run covers the protocol-owned surfaces checked by Server startup:

- workflow inputs and outputs, activity payloads, commands, signals, updates,
  service calls, update-validation tasks, and durable-stream references;
- inline single-object frames and the explicit payload envelopes nested in
  history events; and
- externally stored payload references, including reference shape, codec,
  object availability, byte length, and SHA-256 integrity.

Customer-owned memo, search-attribute, context, and diagnostic maps are not
codec declarations and are left unchanged. Unknown codecs, unknown schema
fingerprints, corrupt references, and values that the fixed Avro Value schema
cannot represent are reported as unsafe. An unsafe finding blocks apply.

## Ownership boundary

The command belongs to the Workflow PHP package, but it must run in the
process that owns the database connection and external-payload configuration:

- **Embedded Laravel:** install the target Workflow package in the application
  checkout, then run the application's `php artisan` command while its web,
  queue, scheduler, and maintenance processes are stopped.
- **Standalone Server:** use a one-shot shell or job from the target Server
  image, with the same database, namespace configuration, credentials, and
  external-payload mounts as the deployment. Keep the normal bootstrap hook
  and all Server processes stopped until conversion succeeds.

Do not point an unrelated application checkout at the database. Namespace
storage policy is part of the integrity check; a process that cannot read an
external object reports it as unsafe instead of guessing.

## Deployment order

1. Keep the old prerelease available long enough to drain any active runs you
   choose to finish. Then stop every database writer: API nodes, workers,
   schedulers, bootstrap jobs, and maintenance jobs.
2. Take the database and external-object-store snapshots required by your
   normal recovery policy. Retain the old application or Server artifact.
3. Install or pull the target artifact that contains
   `workflow:v2:migrate-prerelease-history`, but do not start it. Run the dry
   inventory from that artifact:

   ```bash
   php artisan workflow:v2:migrate-prerelease-history --dry-run --json
   ```

   A zero exit code means every affected value is safely convertible. Review
   the paths and affected run IDs. A nonzero exit code with `unsafe_fields`
   greater than zero blocks the upgrade; correct storage access or remain on
   the old prerelease.
4. Choose a new private backup path and a new private evidence directory on
   persistent storage. Neither may already exist. Apply once:

   ```bash
   php artisan workflow:v2:migrate-prerelease-history \
     --backup=/secure/dw-prerelease-history-backup.json \
     --evidence-dir=/secure/dw-prerelease-history-evidence
   ```

   The command writes the backup before beginning updates and refuses to
   overwrite evidence. It locks and updates database rows in one transaction,
   preserves workflow and run identities, verifies external-object hashes and
   sizes, and retains every original external object. Replacement external
   objects are content-addressed Avro copies. No history is deleted.
5. Wait for the command to finish. It exports and strict-replays every affected
   retained run, rebuilds its summary, wait, timeline, timer, and lineage
   projections, checks those projections for drift, and repeats the payload
   inventory. Success requires all three checks. Keep the generated replay and
   projection reports with the backup.
6. Run the normal target bootstrap only after migration succeeds:

   ```bash
   php artisan server:bootstrap --force
   ```

   For embedded Laravel, run the application's normal migration/bootstrap
   procedure instead. The Avro-only payload preflight must be clean before any
   new worker, API node, scheduler, or maintenance process starts.
7. Start the target deployment and verify readiness and a retained-run replay
   or query through the normal operator surface. Retain the migration packet
   for the duration of the rollback window.

## Rollback

If apply, replay, projection verification, or postflight fails, the upgrade is
blocked. Leave all writers stopped and run the rollback command printed by the
migration. With the paths above it is:

```bash
php artisan workflow:v2:migrate-prerelease-history \
  --rollback-from=/secure/dw-prerelease-history-backup.json \
  --evidence-dir=/secure/dw-prerelease-history-evidence
```

Rollback verifies the backup digest and migration state, refuses to overwrite
rollback evidence, and refuses to restore a row changed after conversion. It
restores the original database values atomically and retains replacement
external objects as evidence. Then restore the previous application or Server
artifact. If rollback refuses because another writer changed state, keep the
upgrade blocked and restore the coordinated database and object-store snapshot
from step 2.

After rollback, the Avro-only deployment is expected to fail preflight again.
Resolve the reported unsafe state, create new backup and evidence paths, and
repeat the complete procedure before retrying the upgrade.

<!-- Source: docs/polyglot/server-role-topology.md -->

# Server Role Topology

## Why This Manifest Exists

`GET /api/cluster/info` publishes the server's `topology` manifest under the
schema `durable-workflow.v2.role-topology`. Treat that manifest as the public
contract for role names, supported deployment shapes, durable-write authority,
failure-domain expectations, scaling boundaries, and the ordered migration path
from today's standalone distribution toward a split control/execution topology.

Use this page when you need to reason about server shape from scripts,
dashboards, runbooks, or rollout automation. Use the
[Server API Reference](/docs/polyglot/server-api-reference) for the raw
HTTP surface and the [Server Guide](/docs/polyglot/server) for deployment
setup.

## Reading The Topology Manifest

The `topology` object answers these contract questions:

| Field family | Question it answers |
| --- | --- |
| `schema`, `version` | Which topology contract revision are you reading? |
| `supported_shapes` | Which product deployment shapes are legal? |
| `role_vocabulary` | Which role names are valid on this contract? |
| `current_shape`, `current_process_class`, `current_roles`, `execution_mode` | What is the responding node doing right now? |
| `matching_role.*` | Who owns broad ready-task wake, which routing axes are frozen, and which dispatch/backpressure posture is active? |
| `role_catalog`, `authority_surfaces` | Which interfaces and durable mutation surfaces belong to each role? |
| `shape_assignments` | Which process classes are allowed for each supported shape? |
| `authority_boundaries`, `failure_domains`, `scaling_boundaries` | Which role is allowed to write what, how each role fails, and what load axis each role scales on? |
| `supported_topologies`, `migration_path` | What deployment families are product-supported, and what is the ordered path from the standalone shape toward more isolated roles? |
| `kernel_invariants` | Which durable-kernel guarantees the role split must preserve regardless of which shape is currently running? |

`current_shape`, `current_process_class`, and `current_roles` describe the node
that answered the HTTP request, not the full fleet. Use
`current_process_class` as the node's declared identity, then compare
`current_roles` against the process-class bundles in `shape_assignments` when
you need to verify that declaration.

```bash
curl -sS "$DURABLE_WORKFLOW_SERVER_URL/api/cluster/info" \
  -H "Authorization: Bearer $DURABLE_WORKFLOW_AUTH_TOKEN" \
  -H "X-Namespace: default" \
  | jq '{
    current_shape: .topology.current_shape,
    current_process_class: .topology.current_process_class,
    current_roles: .topology.current_roles,
    execution_mode: .topology.execution_mode,
    matching_role: .topology.matching_role,
    scaling_boundaries: .topology.scaling_boundaries,
    migration_path: .topology.migration_path
  }'
```

## Role Vocabulary

The public role names are fixed by `topology.role_vocabulary`:

| Role | Responsibility |
| --- | --- |
| `api_ingress` | Accept external HTTP traffic, including discovery and control-plane entrypoints. |
| `control_plane` | Start, signal, update, repair, cancel, terminate, archive, and otherwise mutate workflow lifecycle state. |
| `matching` | Discover ready work, own task leases, and coordinate dispatch pressure. |
| `history_projection` | Persist durable history and maintain derived run summaries and exports. |
| `scheduler` | Fire schedules and persist schedule-run state. |
| `execution_plane` | Run workflow and activity task work. |

Automation should treat these exact identifiers as the stable vocabulary.

## Supported Deployment Shapes

`topology.supported_shapes` names the legal product deployment shapes:

| Shape | Process classes published in `shape_assignments` | Contract meaning |
| --- | --- | --- |
| `embedded` | `application_process` | One application process owns control-plane, matching, history projection, scheduler, and execution. |
| `standalone_server` | `server_http_node`, `scheduler_node`, `worker_node` | The current standalone server distribution: HTTP ingress/control-plane on the server node, scheduler isolated as its own process class, and execution on worker nodes. |
| `split_control_execution` | `ingress_node`, `control_plane_node`, `scheduler_node`, `matching_node`, `execution_node` | The same product contract split into narrower role-specific process classes so scaling and failure boundaries can move by subsystem. |

`split_control_execution` is a supported topology, not a second engine or a
different API. The same discovery surface describes both the standalone and the
split-role shapes.

## Current Node Identity

Use `current_shape`, `current_process_class`, `current_roles`, and
`execution_mode` together:

- `current_shape` identifies the responding node's shape contract.
- `current_process_class` identifies the declared process class for that node.
- `current_roles` identifies the active role bundle on that node.
- `execution_mode` distinguishes `remote_worker_protocol` from
  `local_queue_worker`.

For the standalone server distribution, `current_shape` remains
`standalone_server` even when `DW_MODE=embedded` switches execution to local
queue workers. In that case, `execution_mode` changes to `local_queue_worker`
while the HTTP node keeps the standalone-server role contract.

### Hosted Route Gating

The same topology contract also tells callers when a node should reject hosted
traffic outright. Authenticated hosted routes fail closed unless the responding
node advertises the HTTP control bundle those routes require today:
`api_ingress` plus `control_plane`.

Wrong-node responses return `503` with
`reason: "topology_role_unavailable"` and the topology evidence needed to
reroute:

- `current_shape`
- `current_process_class`
- `current_roles`
- `required_roles`
- `missing_roles`

That gate runs before namespace resolution on hosted routes, so a request sent
to a `scheduler_node`, `matching_node`, or `execution_node` does not learn
whether the named namespace exists before it is redirected to the correct node
class. `GET /api/health`, `GET /api/ready`, and authenticated
`GET /api/cluster/info` remain available so automation can discover the node's
shape before retrying elsewhere.

### Workflow Bootstrap Gate

A second route-level gate fails closed when the responding node has unresolved
workflow v2 bootstrap blockers. While `checks.workflow_v2.status` is `blocked`,
authenticated workflow start/mutation, schedule mutation, bridge-adapter, and
worker-protocol routes return `503` with `reason: "workflow_v2_blocked"` plus:

- `blocked_by`: the ordered list of upstream readiness blockers, for example
  `migrations`.
- `remediation`: the operator-facing instruction for clearing the listed
  blockers, mirrored from `/api/ready` `checks.workflow_v2.remediation`.

The bootstrap gate sits in the same slot as the hosted-route topology gate:
after role and protocol-version validation, before namespace resolution. A
blocked request therefore never observes namespace existence.

The gated route families are workflow routes such as `/api/workflows`, schedule
mutations such as `POST /api/schedules`, `PUT /api/schedules/{scheduleId}`,
`DELETE /api/schedules/{scheduleId}`, `/pause`, `/resume`, `/trigger`, and
`/backfill`, bridge-adapter routes such as
`/api/bridge-adapters/webhook/{adapter}`, and worker-protocol routes under
`/api/worker`.

Schedule **reads** (`GET /api/schedules`, `GET /api/schedules/{scheduleId}`,
`GET /api/schedules/{scheduleId}/history`) are intentionally exempted so
operators can inspect schedule state during recovery. Worker-protocol routes
return the same bootstrap-gate payload in the worker-protocol envelope, so
worker SDKs branch on the same machine-readable `reason: "workflow_v2_blocked"`
the control plane returns.

## Matching Role Contract

`topology.matching_role` freezes the live matching and wake posture for the
responding node:

| Field | Meaning |
| --- | --- |
| `queue_wake_enabled` | Whether short-lived queue wake signals are currently enabled. |
| `shape` | Which matching deployment shape this node advertises: `in_worker` or `dedicated`. |
| `wake_owner` | Which implementation currently owns the broad wake sweep: `worker_loop` or `dedicated_repair_pass`. |
| `task_dispatch_mode` | Whether dispatch is happening through `poll`-driven remote workers or `queue`-driven local execution. |
| `partition_primitives` | The frozen routing axes the matching role reasons about, in order: `connection`, `queue`, `compatibility`, `namespace`. |
| `backpressure_model` | The durable admission boundary the matching role enforces. Current v2 reports `lease_ownership`. |
| `discovery_limits` | The frozen numeric matching-role contract the workflow package compiles in: `poll_batch_cap`, `availability_ceiling_seconds`, `wake_signal_ttl_seconds`, `workflow_task_lease_seconds`, and `activity_task_lease_seconds`. |

This lets operators and automation distinguish "matching exists but wake is
degraded" from "this node is intentionally running a different dispatch mode,"
and it gives the same routing and backpressure vocabulary the server itself
publishes in operator metrics.

`discovery_limits` is the matching-role numeric contract: `poll_batch_cap`
freezes the maximum batch of ready-task rows returned per poll,
`availability_ceiling_seconds` freezes the cross-backend tolerance applied to
`available_at` so freshly-available tasks survive sub-second timestamp drift,
`wake_signal_ttl_seconds` freezes the default `CacheLongPollWakeStore` signal
TTL, and `workflow_task_lease_seconds` / `activity_task_lease_seconds` freeze
the default workflow and activity task lease durations. Operators read these
values to verify the deployment matches the documented matching-role contract
without grepping the package source. Tightening any of these is a
protocol-level change because dispatch, worker, and acceleration timing
elsewhere in the contract depend on them; renaming a field is also a
protocol-level break.

## Authority Boundaries

`topology.authority_boundaries` names the durable write surfaces each role is
supposed to mutate:

| Role | Published writes |
| --- | --- |
| `api_ingress` | `worker_registrations` |
| `control_plane` | `workflow_instances`, `workflow_runs.status`, `workflow_tasks.lifecycle` |
| `matching` | `workflow_tasks.leases`, `activity_tasks.leases` |
| `history_projection` | `history_events`, `workflow_run_summaries`, `workflow_history_exports` |
| `scheduler` | `workflow_schedules.fire_state`, `workflow_starts.scheduled` |
| `execution_plane` | `workflow_tasks.outcomes`, `activity_attempts`, `worker_compatibility_heartbeats` |

Use this contract to catch cross-role drift before you split processes or add
new topology-specific automation.

## Failure And Scaling Boundaries

### Failure Domains

`topology.failure_domains` names the first degraded behavior and the first
operator-visible signal for each role outage:

| Failure domain | `effect` | `operator_signal` |
| --- | --- | --- |
| `control_plane_down` | `workers_continue_claimed_tasks_only_until_lease_expiry` | `operator_commands_fail_fast` |
| `execution_plane_down` | `ready_tasks_accumulate_without_loss` | `operators_see_ready_depth_growth` |
| `matching_down` | `claim_falls_back_to_direct_ready_task_discovery` | `ready_depth_rises_while_claim_rate_falls` |
| `history_projection_down` | `projection_reads_may_stale_while_durable_writes_continue` | `projection_lag_seconds_may_increase` |
| `scheduler_down` | `scheduled_workflows_stop_firing_and_record_missed_runs` | `operators_see_missed_schedule_state` |
| `api_ingress_down` | `external_http_traffic_stops_at_the_edge` | `embedded_in_process_calls_may_continue` |

These are product-facing expectations, not internal implementation trivia. Use
them to describe what should happen when a role is degraded before reading logs.

### Scaling Boundaries

`topology.scaling_boundaries` tells you which load axis each role primarily
scales on in the split-role model:

| Role | Scaling boundary |
| --- | --- |
| `api_ingress` | `incoming_http_request_rate` |
| `control_plane` | `operator_commands_and_run_lifecycle_transitions` |
| `matching` | `ready_task_rate_and_poller_count` |
| `history_projection` | `durable_event_rate` |
| `scheduler` | `active_schedule_count` |
| `execution_plane` | `workflow_and_activity_task_rate` |

This is the explicit answer to "what do we scale independently?" for the
split-role topology.

## Migration Path

`topology.migration_path` is ordered. Each step preserves one durable kernel
while isolating responsibilities more clearly:

1. `audit_role_boundaries`
   Result: tooling flags cross-role writes before runtime shape changes.
2. `expose_role_bindings`
   Result: container seams allow out-of-process adapters without patching the package.
3. `introduce_dedicated_matching_shape`
   Result: matching can run as its own process class without changing the claim contract.
4. `split_history_projection`
   Result: history and projections can move out of process without introducing a second writer.
5. `split_scheduler`
   Result: schedule firing can move behind leader election while single-replica deployments stay legal.
6. `optional_execution_partitioning`
   Result: workers can partition by namespace, connection, queue, and compatibility.

Read this list as the supported topology transition order, not as a separate
product roadmap detached from the current engine.

Each `topology.migration_path[]` entry carries an explicit
`reversible: true` flag. Treat the migration path as bidirectional: a
deployment that has reached `split_history_projection` MAY collapse the
history role back into the control-plane process and remain a legal
topology shape. Rollback is part of the contract, not an unmodelled
edge case.

## Durable Kernel Invariants

`topology.kernel_invariants` enumerates the guarantees the role split
preserves regardless of which supported shape is running. Use this list
when validating that a candidate topology change is product-supported
rather than a fork of the engine:

| Invariant | What it guarantees |
| --- | --- |
| `single_persistence_engine` | One workflow database backs every topology shape; role split does not introduce a second persistence engine. |
| `single_worker_protocol` | One HTTP worker protocol carries claim, complete, fail, and heartbeat traffic across every topology; role split does not fork the worker contract. |
| `single_history_writer` | `history_events` has exactly one durable writer per logical event regardless of where the history/projection role runs. |
| `single_control_authority_per_run` | Every mutation of a given workflow run routes through one control-plane authority; per-run row locks serialise transitions across replicas. |
| `embedded_topology_remains_supported` | The embedded shape where one process fills every role MUST stay legal; existing embedded hosts are never forced to migrate. |
| `role_split_is_topology_only` | Splitting roles is a topology change, not a product fork; collapsing the roles back onto a single process is always a legal topology. |

Each entry's `applies_to` field lists the shapes the invariant covers.
For the supported topology family, every invariant currently applies to
`embedded`, `standalone_server`, and `split_control_execution`. If an
upgrade adds a new shape, the invariants whose `applies_to` does not
include it MUST be reviewed before that shape is treated as
product-supported.

```bash
curl -sS "$DURABLE_WORKFLOW_SERVER_URL/api/cluster/info" \
  -H "Authorization: Bearer $DURABLE_WORKFLOW_AUTH_TOKEN" \
  -H "X-Namespace: default" \
  | jq '.topology.kernel_invariants[] | {id, applies_to}'
```

## Coordination Health

`/api/cluster/info` also publishes `coordination_health` beside `topology`.
Keep the distinction clear:

- `topology` tells you what the node is allowed to do and how the product shape
  is supposed to behave.
- `coordination_health` tells you whether rollout-safety and coordination checks
  are currently healthy across namespaces.
- `coordination_health.blocked_by`, `coordination_health.message`, and
  `coordination_health.remediation` appear when the server cannot evaluate
  rollout-safety health because readiness prerequisites such as migrations or
  database connectivity are missing.
- `coordination_health.routing_drains` summarizes draining build-id cohorts
  across queues and namespaces. `queues_with_drains` tells you whether rollout
  automation is intentionally holding traffic away from any cohort right now.
- `coordination_health.warning_checks`, `coordination_health.error_checks`, and
  `coordination_health.checks` remain the normalized check inventory once
  rollout-safety evaluation is running.
- `coordination_health.checks[]` always includes the frozen `activity_path`
  check next to `worker_compatibility`, `task_transport`, `routing_health`,
  `durable_resume_paths`, and the projection/scheduler checks. `activity_path`
  is the activity-side counterpart of `task_transport`: it surfaces activity
  executions whose schedule-to-start, start-to-close, schedule-to-close, or
  heartbeat deadline has passed without enforcement, plus the sustained activity
  retry backlog. Renaming the check is a protocol-level change.

Use both surfaces together when deciding whether a topology change is both
supported and currently safe.

## Related References

- [Server API Reference](/docs/polyglot/server-api-reference) for the
  authenticated `/api/cluster/info` HTTP contract.
- [Server Guide](/docs/polyglot/server) for deployment setup and the
  broader standalone server operating model.
- [Deployment Modes](/docs/polyglot/deployment-modes) for when to choose
  embedded, standalone server, or broader support-led topologies.

<!-- Source: docs/polyglot/workflow-streams.md -->

# Workflow Streams

Workflow Streams are named, run-scoped output logs owned by Durable Workflow
Server. PHP, Python, and Rust expose typed list, describe, subscribe, append,
close, and error operations, so applications do not need to assemble HTTP
requests. The runtime advertises
`durable-workflow.v2.workflow-streams.contract@1`; the
[machine-readable SDK matrix](/workflow-stream-capabilities.json) records the
exact support in each SDK.

Each stream assigns monotonically increasing offsets beginning at 0. A page's
`next_offset` is the next inclusive offset to request. Delivery to consumers is
at least once: finish the page's idempotent effects, then durably checkpoint
`next_offset`. A crash before that checkpoint may redeliver items.

## Emit from workflow code

The three SDKs emit through the same deterministic command boundary:
`record_side_effect.workflow_stream`. Each SDK derives the item idempotency key
from the task's durable workflow command identity, the stream-command ordinal,
and the batch index. Server commits the stream mutation and recorded side
effect together. Replay consumes that side effect and cannot create another
durable item for the same logical append.

PHP:

```php
use DurableWorkflow\Model\WorkflowStreamAppendItem;

$context->appendWorkflowStream('progress', [
    new WorkflowStreamAppendItem(['percent' => 50]),
]);
$context->closeWorkflowStream('progress');
```

Python:

```python
from durable_workflow import WorkflowStreamAppendItem

yield ctx.append_workflow_stream("progress", [
    WorkflowStreamAppendItem(payload={"percent": 50}),
])
yield ctx.close_workflow_stream("progress")
```

Rust:

```rust
let item = WorkflowStreamAppendItem::new(serde_json::json!({"percent": 50}))?;
ctx.append_workflow_stream("progress", &[item], None)?;
ctx.close_workflow_stream("progress", None)?;
```

The workflow helpers are for deterministic authoring. Code outside a workflow
can use each client's typed `append` operation and supply its own stable
idempotency key.

## Subscribe and resume

Subscription reads a bounded page, optionally waiting for new items for up to
60 seconds. PHP accepts a cancellation callback between bounded polls. Python
accepts normal task cancellation or an `asyncio.Event` that cancels the
in-flight poll. Dropping Rust's subscription future cancels its request.

| SDK | Page operation | Resume value | Typed iteration |
| --- | --- | --- | --- |
| PHP | `subscribeWorkflowStream(...)` | `WorkflowStreamPage::$nextOffset` | `iterateWorkflowStream(...)` |
| Python | `await subscribe_workflow_stream(...)` | `WorkflowStreamPage.next_offset` | `iter_workflow_stream(...)` |
| Rust | `subscribe_workflow_stream(...).await` | `WorkflowStreamPage::next_offset` | Repeat the bounded page future until `terminal`. |

`open`, `closed`, and `errored` are the lifecycle states. The typed description
also exposes `last_offset`, `total_items`, `pending_items`, and `error_reason`.
An append to a terminal stream is rejected. When the configured pending bound
is reached, append returns `stream_full`; producers must slow down or wait for
consumers to drain the stream.

## External payload references

Inline values use the shared Avro Value envelope. External references follow
each SDK's existing storage contract rather than inventing another transport:

| SDK | External reference behavior |
| --- | --- |
| PHP | Appends and returns an opaque `payload_reference`; storage upload and fetch remain application-owned. |
| Python | Uses the configured external-storage driver to upload, integrity-check, cache, fetch, and decode an external Avro envelope. Without a driver, callers can append or inspect an opaque `payload_reference`. |
| Rust | Appends and returns an opaque reference and its metadata. The current Rust SDK does not claim an external-storage driver. |

## Workflow Streams and embedded MessageStream

The names align where behavior aligns, but the models are not interchangeable.

| Concept | Service-mode Workflow Stream | Embedded Laravel MessageStream |
| --- | --- | --- |
| Address | Workflow run + stream name | Workflow instance/run + stream key |
| Direction | Workflow output only | Workflow inbox and outbox |
| First offset | 0 | 1 |
| Delivery | At least once; consumer-owned checkpoint | At least once; engine-owned message cursor |
| Continue as new | No stream cursor transfer | Inbox cursor transfers to the continued run |
| Inbound workflow messaging | Not provided; use signals or updates | Provided by `inbox()->receive()` |

Waterline uses one normalized table for both modes and shows the mode,
lifecycle, head/cursor offsets, pending count, direction, and error. It does not
present service output streams as an inbound workflow inbox.

## Qualification

The public
[Workflow Stream scenario manifest](/platform-conformance/workflow-stream-runtime-scenarios.json)
requires published-artifact runs for PHP producer to Python and Rust consumers,
producer worker restart, consumer reconnect, lifecycle/backpressure/cancellation,
and external payload references. A release result must record every required
scenario; source-unit tests alone are not conformance evidence.

<!-- Source: docs/rolling-upgrades.md -->

# Rolling Upgrades

Run a rolling upgrade when you want to replace API nodes, workers, or the
scheduler without taking the deployment offline. This contract covers the
small clustered shape from the [self-hosting deployments](/docs/deployment)
guide: two or three API nodes behind a load balancer, shared external MySQL
or PostgreSQL, shared Redis, independently scaled workers, and exactly one
scheduler or maintenance runner.

A rolling upgrade is supported when every guarantee on this page holds.
Outside that envelope, use the documented stop-the-world flow in the
deployment guide instead.

The role vocabulary and shape manifest behind this contract are documented in
[Server Role Topology](/docs/polyglot/server-role-topology). This guide
focuses on how the current `standalone_server` process classes roll in place.

## What rolling upgrade means here

A **rolling upgrade** replaces processes one at a time, draining each one
before stopping it, while the rest of the fleet keeps serving traffic. The
result is zero downtime for the deployment as a whole and a bounded overlap
window where old and new processes coexist.

The contract distinguishes four process classes in the current
`standalone_server` shape, each with its own rollout posture:

- **HTTP/API nodes**: stateless server processes serving HTTP traffic and
  currently hosting the `api_ingress`, `control_plane`, `matching`, and
  `history_projection` roles.
- **Workers**: SDK processes that poll the worker plane and execute
  activity and workflow tasks as the `execution_plane`.
- **Scheduler / maintenance runner**: the singleton process that fires
  schedules and drives activity-timeout and history retention as the
  `scheduler` role.
- **Bootstrap**: the one-shot process that runs database migrations and
  default-namespace seeding.

API nodes and workers roll independently. The scheduler is a singleton —
you stop the old one before starting the new one, but the rest of the
deployment keeps serving traffic across that gap.

## Compatible version-skew rules

The rolling-upgrade overlap window is the time during which more than one server
image, workflow package version, or worker SDK version is live. The
window must satisfy every rule in this section.

### Server image and workflow package

- **Adjacent versions only.** During a rolling upgrade, every API node
  and worker must run a server image whose workflow package version is
  the same major version as the cluster's previous package and within
  one minor version of every other live process. Skipping a major
  version requires a stop-the-world upgrade.
- **Forward-additive migrations.** Every Durable Workflow v2 schema
  change is additive within a major version. New nodes must not require
  a column or table that has not been migrated in. Old nodes must not
  break when a new column they do not read is present. The
  [migration order rules](#schemabootstrap-ordering) below enforce this.
- **Adjacent control-plane and worker-protocol versions.** Every node
  publishes its supported `control_plane.version` and
  `worker_protocol.version` ranges from `GET /api/cluster/info`. During
  a rolling upgrade, the new image's supported range must overlap with
  every live old node's supported range. Discover the range before the
  rollout, and abort if it does not overlap.

### Worker SDK and build identity

- **Workers tag every build.** Every worker that may participate in a
  rolling upgrade must register through `POST /api/worker/register`
  with a stable `build_id`. The unversioned cohort
  (`build_id: null`) is the pre-rollout default; the
  [worker build-id rollout guide](/docs/polyglot/worker-build-id-rollout)
  explains the first cutover.
- **Workflow definition fingerprints stay pinned.** Server images
  carrying `DW_V2_PIN_TO_RECORDED_FINGERPRINT=true` (the default) keep
  in-flight runs pinned to the workflow definition fingerprint
  recorded at `WorkflowStarted`. A new worker that ships a different
  fingerprint for the same workflow refuses to claim those runs until
  they finish.
- **Overlapping-build admission posture.** Choose the
  [`DW_V2_FLEET_VALIDATION_MODE`](/docs/polyglot/server-config-reference)
  posture before you start. `warn` lets the rollout proceed even when
  the required compatibility marker has no live worker;  `fail` blocks
  dispatch and fails the readiness contract closed during that window.
  Production rollouts that require a clean cutover should be on `fail`.

## Schema/bootstrap ordering

Schema changes ride a single bootstrap pass. Order matters.

1. **Run bootstrap first, exactly once.** Run `php artisan
   server:bootstrap --force` (or the published-image equivalent) from
   one container before starting any new API node, worker, or
   scheduler. Bootstrap runs `migrate` plus default-namespace seeding;
   it adopts any workflow package migrations whose tables already
   exist on the connection.
2. **New schema must be backwards-compatible with old code.** Every v2
   migration in this release path is additive (new tables, new
   columns, new indexes). Old API nodes and workers continue running
   against the new schema.
3. **Do not start new code before bootstrap completes.** Roll the new
   server image only after bootstrap exits successfully. New API nodes
   and workers may rely on the freshly migrated tables, and starting
   them before bootstrap finishes is the most common cause of a 5xx
   surge during the cutover.
4. **Bootstrap is idempotent.** Re-running it is safe. If bootstrap
   fails partway, fix the underlying error and re-run; the migration
   ledger picks up where it left off, and the namespace seed is a
   no-op when the row exists.

A migration that lands on one server before another MUST NOT corrupt
the readiness surface. If you discover a non-additive change during
planning, take a stop-the-world upgrade window for that release and
return to rolling upgrades on the next one.

## Drain and admission during the overlap window

Three admission surfaces enforce overlap-window safety automatically:

- **Boot-time admission.** Every server process loads
  `BackendCapabilities`, `LongPollCacheValidator`, `WorkflowModeGuard`,
  and the readiness contract at boot. A process whose backend or
  cache cannot satisfy the v2 contract refuses to mark itself ready.
- **Worker compatibility.** When
  `DW_V2_FLEET_VALIDATION_MODE=fail` and no live worker advertises the
  required compatibility marker for a task's connection and queue
  scope, the matching role blocks dispatch and the
  `worker_compatibility` health check escalates from `warning` to
  `error`. Tasks stay ready and visible — they are never dropped — and
  the readiness contract returns 503 on that node so the load balancer
  takes it out of rotation.
- **Routing safety.** A ready task whose required compatibility has no
  live worker is preserved and counted under the
  `compatibility_blocked_runs` backlog metric. Routing safety never
  silently escalates to task loss; the at-least-once execution
  guarantee still applies after a routing drain.

To keep the worker overlap window short, drain old
worker cohorts as new workers come online. Use the
[worker build-id rollout](/docs/polyglot/worker-build-id-rollout)
flow:

```bash
dw task-queue:drain orders-critical --build-id orders-worker-2026-04-21-z9
```

The cohort's `drain_intent` flips to `draining`. Workers under that
build keep finishing in-flight work but stop claiming new tasks. Wait
for the cohort's `active_worker_count` and `draining_worker_count` to
reach zero before stopping the old worker processes.

The scheduler does not run a long-lived task queue, so it does not need
a worker drain. Stop the old scheduler container, run bootstrap if it
has not run yet, and start the new one. The window between the two is
bounded by how long it takes the new container to come up; schedule
firing resumes from the persisted state on next tick.

If the rollout also introduces a dedicated matching-role deployment, make that
topology change explicit instead of assuming every execution node will keep
doing broad ready-task sweeps. See
[Task Matching and Dispatch](/docs/polyglot/task-matching-dispatch) for
the documented `workflow:v2:repair-pass --loop` plus
`DW_V2_MATCHING_ROLE_QUEUE_WAKE=0` shape. Verify the live node contract from
`GET /api/cluster/info`: `topology.current_shape` should still match the
deployment you are cutting over, `topology.current_roles` should still match
the documented role bundle for that node, and
`topology.matching_role.queue_wake_enabled`,
`topology.matching_role.shape`, and `topology.matching_role.wake_owner`
should show the expected broad-ready-task owner. The default shape reports
`queue_wake_enabled: true`, `shape: "in_worker"`, and
`wake_owner: "worker_loop"`; dedicated matching rollouts flip execution nodes
to `queue_wake_enabled: false`, `shape: "dedicated"`, and
`wake_owner: "dedicated_repair_pass"`.

## Readiness and cutover

Use the readiness contract — not just the liveness probe — to decide
when traffic flows to a node.

- `GET /api/health` proves the process is serving HTTP.
- `GET /api/ready` proves the process can use its configured runtime
  dependencies, including migrations, default namespace, and (under
  `DW_V2_FLEET_VALIDATION_MODE=fail`) the worker-compatibility
  admission check.
- `GET /api/cluster/info` proves an authenticated client can discover
  build identity, control-plane protocol, worker protocol, payload
  codecs, and server capabilities.
- `POST /api/worker/register` proves workers can authenticate into the
  expected namespace and task queue.

Cutover sequence for one API node:

1. Take the node out of the load balancer rotation. The simplest path
   is to fail the load balancer's readiness probe by stopping the new
   image's pre-start hook before bringing the new container up.
2. Drain in-flight HTTP requests. Most clients retry on connection
   reset; long-running connections (worker long-polls) reconnect
   against the rest of the fleet.
3. Stop the old container, start the new one.
4. Wait for `GET /api/ready` to return 200 and for
   `GET /api/cluster/info` to advertise the new build identity. When the
   rollout changes the matching topology, also confirm
   `topology.matching_role.task_dispatch_mode`,
   `topology.matching_role.queue_wake_enabled`,
   `topology.matching_role.shape`, `topology.matching_role.wake_owner`,
   `topology.matching_role.partition_primitives`, and
   `topology.matching_role.backpressure_model` match the intended deployment
   before returning the node to traffic. Use `/api/system/operator-metrics`
   when you want the same node-local matching-role contract alongside live
   backlog, repair, and worker counters from the responding process.
5. Return the node to rotation.

Repeat one node at a time. Do not roll the next node until the
previous one is back in rotation and serving traffic cleanly.

For workers, the cutover is per-cohort:

1. Bring the new worker cohort online with a new `build_id`.
2. Confirm both cohorts report `rollout_status: "active"` and non-zero
   `active_worker_count` from `dw task-queue:build-ids <queue> --json`.
3. Drain the old cohort with `dw task-queue:drain`.
4. Wait until the old cohort's `active_worker_count` and
   `draining_worker_count` reach zero.
5. Stop the old worker processes.

## Rollback

Every step is reversible. Plan for rollback before you start.

- **Bootstrap rollback.** v2 migrations are reversible by the standard
  Laravel `down()` path. A rollback that reverts a migration the new
  image relies on requires stopping every new node first; otherwise
  the new code observes a missing column and the readiness contract
  returns 503 on those nodes. Most rollbacks do not need to reverse
  migrations because schema changes are additive.
- **API node rollback.** Stop the new container, restart the old one
  on the same node. Take the node out of rotation while it boots and
  return it once `GET /api/ready` is green. Repeat for any other
  upgraded API nodes. Old code keeps reading the new schema cleanly
  because the schema change was additive.
- **Worker rollback.** Resume the previously drained cohort, drain
  the bad cohort, and scale the known-good build back up:

  ```bash
  dw task-queue:resume orders-critical --build-id orders-worker-2026-04-21-z9
  dw task-queue:drain  orders-critical --build-id orders-worker-2026-04-22
  ```

  Resume clears `drain_intent` and `drained_at`, and any worker
  heartbeating under the resumed `build_id` flips back to `active` on
  the next poll. Both calls are idempotent.

If the rollout exposed a non-additive schema problem, take a
stop-the-world upgrade window to roll back, run the corrective
migrations, and re-plan.

## Operator verification

Verify each phase of the rollout from operator surfaces, not from logs.

| Question | Surface |
| --- | --- |
| Is bootstrap finished? | `php artisan server:bootstrap --force` exit code 0; `migrate:status` shows every migration ran. |
| Is the new node ready? | `GET /api/ready` returns 200; `GET /api/cluster/info` reports the new build. |
| Is compatibility admission healthy? | `GET /api/system/operator-metrics` `workers.fleet`, `workers.active_workers`, and `workers.active_workers_supporting_required` agree on a non-zero supporter count for every required compatibility marker. |
| Is the worker drain progressing? | `dw task-queue:build-ids <queue> --json` shows `active_worker_count` and `draining_worker_count` falling for the draining cohort. |
| Is routing safe? | `GET /api/system/operator-metrics` `backlog.compatibility_blocked_runs` and `backlog.max_compatibility_blocked_age_ms` stay near zero; the `worker_compatibility` health check is not in `error`. |
| Is the scheduler caught up? | `GET /api/system/operator-metrics` `schedules.missed` is zero and `schedules.oldest_overdue_at` is null. |
| Are stuck runs piling up? | `GET /api/system/operator-metrics` `runs.repair_needed` and `runs.max_repair_needed_age_ms` stay near their pre-rollout baseline. |

`dw system:operator-metrics --json` exposes the same operator-metrics snapshot
on the console for the standalone-server fleet, so operators may pick whichever
surface matches their existing workflow. A separately deployed Waterline
service can read the same server-owned rollout state through the PHP SDK under
`/waterline/api/v2/health` and `/waterline/api/stats`; keep its endpoint,
namespace, and server token aligned with the fleet being rolled. Embedded
Laravel deployments expose those Waterline routes from their in-process
package instead. The server API and CLI remain available in either case, as
documented in the
[Operator Operating Envelope](/docs/operator-operating-envelope).

## Failure modes and what to do

| Symptom | Likely cause | Action |
| --- | --- | --- |
| New API node fails `GET /api/ready` after start | Bootstrap did not finish, or `DW_V2_FLEET_VALIDATION_MODE=fail` and no compatible worker is live yet | Rerun bootstrap; bring a compatible worker cohort online before adding the API node back to rotation. |
| `worker_compatibility` health check escalates to `error` mid-rollout | The required compatibility marker has no live supporting worker | Bring more workers under a supporting `build_id` online; resume a previously drained cohort if rollback is the right call. |
| `backlog.compatibility_blocked_runs` climbs and `max_compatibility_blocked_age_ms` grows | Tasks are queued for a marker no live worker supports | Same as above; tasks are preserved and will redispatch automatically once a compatible worker heartbeats. |
| `dw task-queue:drain` returns success but workers keep claiming tasks | The worker process did not heartbeat after the drain | Wait one heartbeat cycle; if the cohort stays active, restart the worker process so it picks up the drain intent. |
| Schedule fires stop after a scheduler restart | Old and new scheduler are both stopped | Start the new scheduler container; verify `schedules.missed` returns to zero on next operator-metrics scrape. |

If a symptom is not on this list, treat the rollout as failed: stop
adding new processes, drain whatever new cohorts you started, and
restore the previous build before debugging further.

## Related references

- [Self-Hosting Deployments](/docs/deployment) for the deployment
  shapes this contract assumes.
- [Worker Build-Id Rollout](/docs/polyglot/worker-build-id-rollout)
  for the per-cohort drain and resume calls.
- [Operator Operating Envelope](/docs/operator-operating-envelope)
  for the diagnostic, queue, and rebuild contract that operators read
  alongside the rollout signals.
- [Server Config Reference](/docs/polyglot/server-config-reference)
  for the rollout-safety environment variables (`DW_V2_FLEET_VALIDATION_MODE`,
  `DW_V2_PIN_TO_RECORDED_FINGERPRINT`, `DW_V2_GUARDRAILS_BOOT`,
  `DW_V2_CACHE_VALIDATION_MODE`, `DW_V2_MULTI_NODE`,
  `DW_V2_VALIDATE_CACHE_BACKEND`, `DW_V2_TASK_REPAIR_*`).
- [Server API Reference](/docs/polyglot/server-api-reference) for
  the readiness, cluster info, and operator-metrics endpoints used to
  verify each phase of the rollout.

<!-- Source: docs/compatibility.md -->

# Version Compatibility

This page is the **canonical compatibility and release-authority contract** for
the Durable Workflow public platform. It is the single source of truth for:

- which surfaces are public,
- the stability level of each public surface,
- which changes may ship in a patch, minor, or major release,
- whether a given field is part of the contract or is diagnostic-only,
- and the runtime version-negotiation protocol clients use to fail closed when
  the server advertises a surface they cannot speak.

Per-package stability documents (for example `docs/api-stability.md` in the
`durable-workflow/workflow` repository, the `dw` CLI reference, the Waterline
operator API page) are **downstream** of this page. They add per-package
detail under these rules; when a per-package document and this page disagree,
this page wins, and the disagreement is a bug in the per-package document.

The same contract is published in machine-readable form so SDKs, server
manifests, and CI gates can validate themselves against one source of truth:

- `surface_stability_contract` in the response body of
  `GET /api/cluster/info` on the standalone Durable Workflow server, schema
  `durable-workflow.v2.surface-stability.contract`, version `4`.
- A frozen mirror of the same manifest in this repository at
  `static/compatibility-contract.json`.
- The PHP class `Workflow\V2\Support\SurfaceStabilityContract`, which is
  the in-process source the server re-exports.

Artifact-channel admission is governed separately by
[`/public-artifact-release-policy.json`](pathname:///public-artifact-release-policy.json).
This independently reviewed policy controls which 2.0 channels may become the
canonical public tuple.

A release that changes any surface listed below — its stability level, its
field set, its breaking-change rules — must update this page, the JSON
mirror, the PHP manifest, and any per-package stability document in the same
change. Docs CI validates the machine-readable contract, artifact release-phase
policy, released Rust metadata when available, and worker-protocol specs.
Editorial alignment between this page and the manifest remains an explicit
release-review responsibility.

## Companion: Platform Protocol Spec Catalog

This page says *which* surfaces are public and *how* they may change.
The companion [Platform Protocol Specs](/docs/platform-protocol-specs)
catalog says *where* the normative machine-readable specification for
each surface lives, *which format* the spec uses (OpenAPI for HTTP APIs,
JSON Schema for object families, AsyncAPI for event-stream semantics),
*which repository* owns the spec, which object families it governs, and
which public URL resolves the artifact. SDK authors, agents, and operators
should validate against the spec catalog rather than re-reading prose or
depending on repository-local implementation details.

The catalog is advertised as `platform_protocol_specs` in
`GET /api/cluster/info`.

Every catalog entry's `surface_family` must exist in the contract above;
docs-site CI validates the catalog, resolves each public spec URL, and rejects
repository-local authority fields.

Every required platform protocol catalog entry is marked `published`; the
invocable carrier entry remains `in_progress`. Every available entry links
directly to a public OpenAPI, AsyncAPI, or JSON Schema document. The
[`cluster_info_envelope`](/docs/platform-protocol-specs#cluster-info-envelope-notes)
schema pins the discovery surface every other catalog entry can be
reached from.

## Stability Levels

Every public surface in Durable Workflow carries exactly one of these
stability levels. Levels are explicit; a surface that is not classified is
not public.

| Level | Meaning | When breaking changes are allowed |
|-------|---------|-----------------------------------|
| `frozen` | Wire-format or persisted shape that must decode the same way for the workflow lifetime. Renaming, removing, or repurposing a field is a protocol break, never a minor change. | Only by introducing a **parallel primitive** with a new type name. The original shape stays decodable indefinitely. |
| `stable` | Public surface covered by the platform semver guarantee. Additive changes ship in minor releases. | Major release only. |
| `prerelease` | Public surface that is feature-complete but still allowed to change before the matching `1.0.0` / `2.0.0` cut. | In clearly labelled prerelease versions; called out in release notes. |
| `experimental` | Public-but-unstable surface. May change in any release, including patch releases. Callers must opt in by reading the experimental flag on the surface. | Any release; release notes call out the change. |

## Public Surface Families

This is the complete list of public surface families. Adding, removing, or
re-classifying a family requires a contract change (`SurfaceStabilityContract`
version bump, this page, and the JSON mirror in the same commit).

| Family | Stability | Authority manifest in `/api/cluster/info` | What it covers |
|--------|-----------|-------------------------------------------|----------------|
| `server_api` | `stable` | `control_plane` | Standalone server HTTP API: control-plane routes, namespace routes, schedule routes, system routes, plus `/api/health`, `/api/ready`, `/api/cluster/info`. Per-route version is governed by `control_plane.request_contract` and `control_plane.response.contract`. The top-level server `version` is build identity, not the client compatibility authority. |
| `worker_protocol` | `stable` | `worker_protocol` | Worker-plane HTTP API used by external SDK workers to register, poll, heartbeat, manage worker-session leases, complete, and fail workflow, activity, and query tasks. Includes the `worker_sessions` and `local_activities` runtime contracts, `external_execution_surface_contract`, `external_executor_config_contract`, `invocable_carrier_contract`, `external_task_input_contract`, and `external_task_result_contract`. |
| `cli_json` | `stable` | n/a (see CLI reference) | The `--output=json` and `--output=jsonl` shapes emitted by `dw`. JSON exit codes and JSON field names are the durable surface; the human-readable `--output=table` form is documentation, not contract. |
| `waterline_api` | `stable` | n/a (see Waterline operator API) | Waterline observability HTTP API at `/waterline/api/v2/*`, the engine-source contract, and the dashboard JSON shapes. Waterline must match the workflow package major version. |
| `mcp_discovery_results` | `stable` | n/a (see MCP workflows page) | The `/mcp/*` Model Context Protocol surfaces and the `llms.txt` / `llms-2.0.txt` discovery files. MCP tool names, parameter schemas, and `payload_preview_limit_bytes` semantics are part of the contract; tool descriptions and discovery hints are diagnostic. |
| `official_sdks` | `stable` | `client_compatibility` | The first-party SDKs: PHP `durable-workflow/sdk`, the `durable_workflow` Python SDK, and the `durable-workflow` Rust SDK. The `dw` CLI is the official command client. Each SDK's public surface is governed by its own per-package stability document, which must defer to this page. |
| `history_event_wire_formats` | `frozen` | n/a (frozen shapes; see workflow `docs/api-stability.md`) | The persisted shape of every row in `workflow_history_events` and `workflow_schedule_history_events`. Once a workflow writes an event, every future SDK that replays it must decode the same field set. |
| `cluster_info_manifests` | `stable` | `surface_stability_contract`, `client_compatibility`, `control_plane`, `worker_protocol`, `auth_composition_contract`, `coordination_health` | The protocol manifests published by `GET /api/cluster/info` itself. Each nested manifest carries its own `schema` and `version` and evolves under its own contract rules. The envelope keys are stable. |

### Per-package stability documents

These documents add per-package detail under the rules on this page:

- `durable-workflow/workflow` (PHP) — [`docs/api-stability.md`](https://github.com/durable-workflow/workflow/blob/main/docs/api-stability.md). Authoritative for the PHP authoring API, the `Support\*` server-facing classes, and the frozen history-event wire-format tables.
- `durable-workflow/sdk` (PHP) — [`README.md`](https://github.com/durable-workflow/sdk-php/blob/main/README.md). Authoritative for the framework-neutral remote client and worker API distributed from Packagist.
- `durable-workflow/server` — [`README.md`](https://github.com/durable-workflow/server/blob/main/README.md) and `docs/contracts/*`. Authoritative for the standalone server's request/response contracts.
- `dw` CLI — [`/docs/polyglot/cli-reference`](/docs/polyglot/cli-reference). Authoritative for the JSON output shapes and exit codes.
- Python SDK — `README.md` in `durable-workflow/sdk-python`. Authoritative for the `durable_workflow` package public API.
- Rust SDK — `README.md` and `[package.metadata.durable-workflow]` in
  `durable-workflow/sdk-rust`. Authoritative for the `durable-workflow` crate
  public API and its package compatibility declaration.

## Release Rules

These rules apply to every public surface family above. They are reproduced in
the JSON mirror under `release_rules`.

### Patch releases

Allowed:
- bug fixes that preserve the documented contract
- documentation fixes
- dependency bumps that do not change the public surface
- changes to surfaces marked `experimental`

Forbidden:
- removing or renaming any `stable` or `frozen` field, route, command, or class
- narrowing accepted input on any `stable` route or command
- changing the meaning of an existing `stable` field

### Minor releases

Allowed:
- adding new fields, routes, commands, or classes to a `stable` surface
- adding new optional parameters with safe defaults
- adding new capability flags to discovery responses
- promoting a `prerelease` or `experimental` surface to `stable`

Forbidden:
- removing or renaming any `stable` or `frozen` field, route, command, or class
- changing the meaning of an existing `stable` or `frozen` field

### Major releases

Allowed:
- removing, renaming, or narrowing a `stable` surface
- increasing the required `control_plane.version` or `worker_protocol.version`
- dropping a previously supported SDK or CLI version range

Required:
- announce in release notes at least
  one minor release before cutting the major
- where feasible, ship the new surface alongside the old surface in a
  previous minor release so callers can migrate before the major
- document the migration path on the [migration guide](/docs/migration)
  before publish

## Diagnostic-Only Versus Guaranteed Fields

Every field in every `stable` or `frozen` surface is either **guaranteed** or
**diagnostic-only**. The two have different change rules:

- **Guaranteed fields** are part of the documented contract. Producers must
  keep emitting them in the documented shape; consumers may rely on their
  presence and meaning. Removing or renaming a guaranteed field on a `stable`
  surface is a major change.
- **Diagnostic-only fields** are emitted for human triage, debugging, and
  observability. They may be added, renamed, or removed in any minor
  release. They must be marked `diagnostic_only: true` (or the doc-page
  equivalent) wherever they are documented. **Consumers must not parse,
  persist, or branch on diagnostic fields in production decision logic.**

Unknown additive fields on a `stable` or `frozen` shape must be ignored by
older consumers (forward compatibility). Unknown required fields must fail
closed. SDKs and CLIs publish their own forward-compatibility behavior in
their per-package stability documents.

## Compatibility Matrix

This is the operational compatibility matrix. It records which client
versions are validated against which server protocol manifests. Components
validate the matrix at runtime via `GET /api/cluster/info` and fail closed
when the manifests do not agree.

### Last qualified reproducible tuple

The table is immutable compatibility evidence for the last jointly qualified
2.0 tuple. It is a reproducibility record, not a set of independently
maintained claims about the newest package in each registry:

| Component | Supported version | Install identity |
|-----------|-------------------|------------------|
| Server | `2.0.0` | `durableworkflow/server:2.0.0` |
| CLI | `2.0.0` | `VERSION=2.0.0` |
| Workflow engine | `2.0.1` | `durable-workflow/workflow:2.0.1` |
| Waterline operator | `2.0.0` | `durable-workflow/waterline:2.0.0` |
| PHP SDK | `2.0.0` | `durable-workflow/sdk:2.0.0` |
| Python SDK | `2.0.0` | `durable-workflow==2.0.0` |
| Rust SDK | `2.0.0` | `durable-workflow = "=2.0.0"` |

PyPI renders the Python distribution version as
`2.0.0`; the documented PEP 440 install spelling
`2.0.0` resolves to that same release. This
normalization does not create a second supported version.

Qualification is coordinated as a unit. A tuple is publishable only when
all seven entries share the same authority identifier, both server
registries agree, and the generated quickstart contract uses those exact
artifacts. The registry refresher fails closed instead of combining
independently newest packages. Its current authorized release phase is
`stable`; registry tags from later channels remain
ineligible until the release policy is reviewed and changed.

Earlier alpha and beta artifacts are historical. They are not alternative
onboarding choices, and release history remains intact.

Capabilities in this train are the 2.0 baseline and therefore have no
feature-introduction version matrix. New capabilities progress through
ordinary compatible releases: additive work advances the compatible version,
while a breaking public-surface change waits for the next major version.
Each stable component follows semantic versioning from this 2.0 baseline.

### Runtime protocol compatibility

Top-level package versions select the supported train. Runtime protocol
manifests provide a second, fail-closed check:

| Client | Product train | Control plane | Worker protocol request |
|--------|---------------|---------------|-------------------------|
| CLI | `2.0.0` | `2` | n/a |
| PHP SDK | `2.0.0` | `2` | `1.13` |
| Python SDK | `2.0.0` | `2` | `1.1` |
| Rust SDK | `2.0.0` | `2` | `1.2` |

The current published Server advertises the worker protocol version shown in
the authority-role table above. It accepts request headers from the same major
with a minor less than or equal to the advertised minor, then returns the
advertised version. Missing or malformed headers, different majors, and worker
minors ahead of the server fail closed. The CLI validates
`control_plane.version: "2"`.

The server's top-level `version` is build identity. Clients must use the
`control_plane`, `worker_protocol`, `client_compatibility`, and
`surface_stability_contract` manifests returned by
`GET /api/cluster/info` for protocol negotiation.

Workflow and Waterline must use compatible 2.x releases. Runtime discovery and
package constraints provide the compatibility boundary after the stable cut.

### Runtime validation examples

The SDKs validate discovery before registering a worker. An incompatible
server produces an explicit compatibility error rather than attempting a
legacy prerelease path. All worker requests send
`X-Durable-Workflow-Protocol-Version`; control-plane requests send
`X-Durable-Workflow-Control-Plane-Version: 2`.

Before a product train is promoted, release qualification must start from a
clean machine, install only the published artifacts named above, and complete
the PHP, Python, and Rust conformance paths. Source checkouts and unpublished
substitutions do not count as public-artifact evidence.

Stable package metadata and runtime discovery now define the supported 2.x
compatibility boundary. Release qualification still installs public artifacts
together before publication, while patch and minor releases follow the semantic
versioning rules below.

### Release progression

Patch releases preserve documented stable contracts. Minor releases may add
fields, routes, commands, classes, or optional parameters with safe defaults.
Breaking stable changes require a major release and a documented migration
path. Frozen history-event shapes remain decodable indefinitely; a new shape
uses a parallel primitive rather than mutating an existing event.

Every release must keep package metadata, release notes, installation commands,
and cross-language examples synchronized.

### Release review checklist

- Confirm the machine-readable compatibility contract matches the Workflow
  surface-stability manifest.
- Confirm every selected SDK version passes against the selected Server.
- Confirm installation examples use the stable release channel.
- Confirm package metadata identifies the exact SDK release and protocol
  versions; record later Server qualification in the compatibility evidence.
- Confirm clean-machine published-artifact conformance passes for PHP, Python,
  and Rust.
- Confirm release notes describe post-baseline additions and do not present
  older prereleases as supported choices.

## See Also

- [Server Setup](/docs/polyglot/server) — Deploying the standalone server
- [Server API Reference](/docs/polyglot/server-api-reference) — `GET /api/cluster/info` and the protocol manifests
- [PHP SDK](/docs/polyglot/php) — PHP client and worker
- [Python SDK](/docs/polyglot/python) — Python client and worker
- [Rust SDK](/docs/polyglot/rust) — Rust client and worker
- [CLI](/docs/polyglot/cli) — Command-line interface
- [Migration Guide](/docs/migration) — Migrating from v1 to v2
- [PHP workflow `docs/api-stability.md`](https://github.com/durable-workflow/workflow/blob/main/docs/api-stability.md) — per-package stability for the PHP workflow package

<!-- Source: docs/platform-protocol-specs.md -->

# Platform Protocol Specs

The platform protocol catalog tells SDK authors, agents, operators, and
third-party tooling exactly which machine-readable specification governs each
public Durable Workflow surface.

Use the
[machine-readable catalog](https://durable-workflow.github.io/platform-protocol-specs.json)
for automation. Every available entry provides both a stable specification
identifier and a public HTTPS URL that resolves directly to an OpenAPI,
AsyncAPI, or JSON Schema artifact.

Repository paths, implementation symbols, and test fixtures are intentionally
not consumer authority. They are excluded from the published catalog and kept
only as validation diagnostics.

## Catalog identity and discovery

The catalog is also available through:

- the public JSON catalog at
  https://durable-workflow.github.io/platform-protocol-specs.json;
- platform_protocol_specs in GET /api/cluster/info on the standalone server;
- this 2.0 guide for human-readable context.

The JSON URL is the machine-consumable catalog authority. The server re-exports
the same catalog so clients can discover the active protocol surface without
assuming a repository layout.

## Consumer fields

Each entry exposes the following contract:

| Field | Meaning |
|---|---|
| spec_id | Stable Durable Workflow protocol identifier. Published documents carry the matching identity. |
| spec_url | Direct public HTTPS URL for the machine-readable specification. |
| format | openapi, json_schema, or asyncapi. |
| status | Whether the referenced artifact is published, in progress, or planned. |
| surface_family | Compatibility-policy family that governs the surface. |
| authority_manifest | Discovery manifest through which a runtime advertises the surface. |
| owner_repo | Project responsible for the specification. |
| object_families | Public object-family names and their owning projects. |
| evolution_rule | Compatibility rule for additive and breaking changes. |
| breaking_change_release | Release boundary required by that evolution rule. |

Consumers should resolve spec_url and validate the document identity against
spec_id. They do not need source checkout access.

## Available specifications

This inventory is rendered from the public JSON catalog. Specification
identity, availability, public links, and object-family ownership therefore
remain catalog data rather than a second prose authority.

<!-- durable-workflow-platform-protocol-catalog:start -->
```json
{
  "catalog_schema": "durable-workflow.v2.platform-protocol-specs.catalog",
  "catalog_version": 16,
  "entries": [
    {
      "catalog_entry": "control_plane_api",
      "availability": "available",
      "spec_id": "durable-workflow.v2.control-plane-api",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/control-plane-api.openapi.yaml",
      "owner_repo": "durable-workflow/server",
      "format": "openapi",
      "status": "published",
      "object_families": [
        {
          "name": "control_plane_request_contract",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "control_plane_response_envelope",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "control_plane_operation_contract",
          "owner_repo": "durable-workflow/server"
        }
      ]
    },
    {
      "catalog_entry": "worker_protocol_api",
      "availability": "available",
      "spec_id": "durable-workflow.v2.worker-protocol-api",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/worker-protocol-api.openapi.yaml",
      "owner_repo": "durable-workflow/server",
      "format": "openapi",
      "status": "published",
      "object_families": [
        {
          "name": "worker_registration_request",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "worker_deregistration_result",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "worker_task_poll_request",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "worker_task_result",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "worker_query_task_poll_request",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "worker_query_task_result",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "external_task_input_contract",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "external_task_result_contract",
          "owner_repo": "durable-workflow/server"
        }
      ]
    },
    {
      "catalog_entry": "worker_protocol_stream",
      "availability": "available",
      "spec_id": "durable-workflow.v2.worker-protocol-stream",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/worker-protocol-stream.asyncapi.yaml",
      "owner_repo": "durable-workflow/server",
      "format": "asyncapi",
      "status": "published",
      "object_families": [
        {
          "name": "worker_poll_stream",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "worker_task_lease",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "worker_task_heartbeat",
          "owner_repo": "durable-workflow/server"
        }
      ]
    },
    {
      "catalog_entry": "worker_sessions_runtime",
      "availability": "available",
      "spec_id": "durable-workflow.v2.worker-sessions-runtime",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/worker-sessions-runtime.schema.json",
      "owner_repo": "durable-workflow/server",
      "format": "json_schema",
      "status": "published",
      "object_families": [
        {
          "name": "worker_session_runtime_contract",
          "owner_repo": "durable-workflow/workflow"
        },
        {
          "name": "worker_session_options",
          "owner_repo": "durable-workflow/workflow"
        },
        {
          "name": "worker_session_lifecycle",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "worker_session_visibility",
          "owner_repo": "durable-workflow/server"
        }
      ]
    },
    {
      "catalog_entry": "local_activity_runtime",
      "availability": "available",
      "spec_id": "durable-workflow.v2.local-activity-runtime",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/local-activity-runtime.schema.json",
      "owner_repo": "durable-workflow/workflow",
      "format": "json_schema",
      "status": "published",
      "object_families": [
        {
          "name": "local_activity_runtime_contract",
          "owner_repo": "durable-workflow/workflow"
        },
        {
          "name": "local_activity_options",
          "owner_repo": "durable-workflow/workflow"
        },
        {
          "name": "local_activity_history_markers",
          "owner_repo": "durable-workflow/workflow"
        },
        {
          "name": "local_activity_visibility",
          "owner_repo": "durable-workflow/workflow"
        }
      ]
    },
    {
      "catalog_entry": "history_event_payloads",
      "availability": "available",
      "spec_id": "durable-workflow.v2.history-event-payloads",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/history-event-payloads.schema.json",
      "owner_repo": "durable-workflow/workflow",
      "format": "json_schema",
      "status": "published",
      "object_families": [
        {
          "name": "workflow_history_events",
          "owner_repo": "durable-workflow/workflow"
        },
        {
          "name": "workflow_schedule_history_events",
          "owner_repo": "durable-workflow/workflow"
        }
      ]
    },
    {
      "catalog_entry": "history_export_bundle",
      "availability": "available",
      "spec_id": "durable-workflow.v2.history-export-bundle",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/history-export-bundle.schema.json",
      "owner_repo": "durable-workflow/workflow",
      "format": "json_schema",
      "status": "published",
      "object_families": [
        {
          "name": "history_export_bundle",
          "owner_repo": "durable-workflow/workflow"
        }
      ]
    },
    {
      "catalog_entry": "replay_bundle",
      "availability": "available",
      "spec_id": "durable-workflow.v2.replay-bundle",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/replay-bundle.schema.json",
      "owner_repo": "durable-workflow/workflow",
      "format": "json_schema",
      "status": "published",
      "object_families": [
        {
          "name": "replay_bundle",
          "owner_repo": "durable-workflow/workflow"
        }
      ]
    },
    {
      "catalog_entry": "waterline_read_api",
      "availability": "available",
      "spec_id": "durable-workflow.v2.waterline-read-api",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/waterline-read-api.openapi.yaml",
      "owner_repo": "durable-workflow/waterline",
      "format": "openapi",
      "status": "published",
      "object_families": [
        {
          "name": "waterline_read_envelope",
          "owner_repo": "durable-workflow/waterline"
        },
        {
          "name": "waterline_operator_action_envelope",
          "owner_repo": "durable-workflow/waterline"
        }
      ]
    },
    {
      "catalog_entry": "waterline_diagnostic_objects",
      "availability": "available",
      "spec_id": "durable-workflow.v2.waterline-diagnostic-objects",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/waterline-diagnostic-objects.schema.json",
      "owner_repo": "durable-workflow/waterline",
      "format": "json_schema",
      "status": "published",
      "object_families": [
        {
          "name": "waterline_run_detail",
          "owner_repo": "durable-workflow/waterline"
        },
        {
          "name": "waterline_health_diagnostics",
          "owner_repo": "durable-workflow/waterline"
        },
        {
          "name": "waterline_timeline_rows",
          "owner_repo": "durable-workflow/waterline"
        },
        {
          "name": "waterline_lineage_edges",
          "owner_repo": "durable-workflow/waterline"
        }
      ]
    },
    {
      "catalog_entry": "repair_actionability_objects",
      "availability": "available",
      "spec_id": "durable-workflow.v2.repair-actionability-objects",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/repair-actionability-objects.schema.json",
      "owner_repo": "durable-workflow/workflow",
      "format": "json_schema",
      "status": "published",
      "object_families": [
        {
          "name": "task_repair_policy",
          "owner_repo": "durable-workflow/workflow"
        },
        {
          "name": "task_repair_candidates",
          "owner_repo": "durable-workflow/workflow"
        },
        {
          "name": "operator_queue_visibility",
          "owner_repo": "durable-workflow/workflow"
        },
        {
          "name": "actionability",
          "owner_repo": "durable-workflow/waterline"
        },
        {
          "name": "agent_root_cause",
          "owner_repo": "durable-workflow/durable-workflow.github.io"
        },
        {
          "name": "agent_remediation",
          "owner_repo": "durable-workflow/durable-workflow.github.io"
        },
        {
          "name": "safe_mutation",
          "owner_repo": "durable-workflow/durable-workflow.github.io"
        }
      ]
    },
    {
      "catalog_entry": "cli_json_envelopes",
      "availability": "available",
      "spec_id": "durable-workflow.v2.cli-json-envelopes",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/cli-json-envelopes.schema.json",
      "owner_repo": "durable-workflow/cli",
      "format": "json_schema",
      "status": "published",
      "object_families": [
        {
          "name": "cli_output_schema_manifest",
          "owner_repo": "durable-workflow/cli"
        },
        {
          "name": "cli_command_output_schema",
          "owner_repo": "durable-workflow/cli"
        }
      ]
    },
    {
      "catalog_entry": "mcp_discovery",
      "availability": "available",
      "spec_id": "durable-workflow.v2.mcp-discovery",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/mcp-discovery.schema.json",
      "owner_repo": "durable-workflow/durable-workflow.github.io",
      "format": "json_schema",
      "status": "published",
      "object_families": [
        {
          "name": "mcp_tool_discovery",
          "owner_repo": "durable-workflow/durable-workflow.github.io"
        },
        {
          "name": "llms_txt_discovery",
          "owner_repo": "durable-workflow/durable-workflow.github.io"
        }
      ]
    },
    {
      "catalog_entry": "mcp_tool_results",
      "availability": "available",
      "spec_id": "durable-workflow.v2.mcp-tool-results",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/mcp-tool-results.schema.json",
      "owner_repo": "durable-workflow/durable-workflow.github.io",
      "format": "json_schema",
      "status": "published",
      "object_families": [
        {
          "name": "mcp_tool_result_envelope",
          "owner_repo": "durable-workflow/durable-workflow.github.io"
        },
        {
          "name": "agent_root_cause",
          "owner_repo": "durable-workflow/durable-workflow.github.io"
        },
        {
          "name": "agent_remediation",
          "owner_repo": "durable-workflow/durable-workflow.github.io"
        },
        {
          "name": "safe_mutation",
          "owner_repo": "durable-workflow/durable-workflow.github.io"
        }
      ]
    },
    {
      "catalog_entry": "cluster_info_envelope",
      "availability": "available",
      "spec_id": "durable-workflow.v2.cluster-info-envelope",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/cluster-info-envelope.schema.json",
      "owner_repo": "durable-workflow/server",
      "format": "json_schema",
      "status": "published",
      "object_families": [
        {
          "name": "cluster_info_envelope",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "client_compatibility_manifest",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "surface_stability_contract",
          "owner_repo": "durable-workflow/workflow"
        },
        {
          "name": "platform_protocol_specs_catalog",
          "owner_repo": "durable-workflow/workflow"
        },
        {
          "name": "platform_conformance_suite_manifest",
          "owner_repo": "durable-workflow/workflow"
        },
        {
          "name": "sdk_neutrality_contract",
          "owner_repo": "durable-workflow/workflow"
        }
      ]
    },
    {
      "catalog_entry": "invocable_carrier_execution",
      "availability": "available",
      "spec_id": "durable-workflow.v2.invocable-carrier-execution",
      "spec_url": "https://durable-workflow.github.io/platform-protocol-specs/invocable-carrier-execution.schema.json",
      "owner_repo": "durable-workflow/server",
      "format": "json_schema",
      "status": "in_progress",
      "object_families": [
        {
          "name": "invocable_carrier_contract",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "external_execution_surface_contract",
          "owner_repo": "durable-workflow/server"
        },
        {
          "name": "external_executor_config_contract",
          "owner_repo": "durable-workflow/server"
        }
      ]
    }
  ]
}
```
<!-- durable-workflow-platform-protocol-catalog:end -->

### Worker session runtime notes

This schema covers worker-session capability discovery, lifecycle envelopes,
task-affinity snapshots, and operator visibility. Its stable identifier is
durable-workflow.v2.worker-sessions-runtime.

### Local activity runtime notes

This schema covers local-activity capability discovery, option snapshots,
history markers, retry semantics, and operator visibility. Its stable
identifier is durable-workflow.v2.local-activity-runtime.

### Cluster-info envelope notes

This schema covers GET /api/cluster/info and the nested discovery manifests
available from that endpoint. Its stable identifier is
durable-workflow.v2.cluster-info-envelope.

## Formats

| Format | Use |
|---|---|
| OpenAPI 3.1 | HTTP and JSON request-response surfaces whose routes, methods, status codes, and envelopes are part of the contract. |
| JSON Schema 2020-12 | Persisted records, event payloads, result envelopes, runtime options, and related object families. |
| AsyncAPI 2.6 or newer | Poll, stream, lease-renewal, ordering, and delivery semantics. |

## Status levels

| Status | Meaning |
|---|---|
| published | The public machine-readable specification is available at spec_url and can be consumed directly. |
| in_progress | A public specification is available, but its coverage is partial. Listed fields and routes are normative. |
| planned | The entry has a stable catalog identity but no consumable specification yet. Planned entries do not advertise spec_url. |

## Evolution rules

additive_minor_breaking_major allows additive changes in minor releases.
Removals, renames, type narrowings, and semantic changes require a major
release and should use a parallel route or field where practical.

parallel_primitive_only governs frozen wire formats. A breaking shape must be
introduced under a new event type, command type, or schema identifier while
the original remains decodable.

experimental_any_release applies only to explicitly experimental
specifications and allows change in any release.

## Release validation

Docs-site CI enforces the following machine checks:

| Gate | What CI checks |
|---|---|
| catalog_aligned_with_surface_families | Every entry references a declared public compatibility family. |
| owner_repo_known | Entry and object-family owners use the catalog vocabulary. |
| format_known | Every available artifact parses as its declared format. |
| public_spec_references_resolve | Every available spec_url is an HTTPS URL in the public protocol-spec namespace and resolves to a shipped artifact whose identity matches spec_id. |
| repository_local_authority_fields_rejected | Published entries contain no repository-local paths, implementation symbols, test references, or legacy authority fields. |
| workflow_package_mirror_aligned | The public catalog matches the packaged Workflow catalog when that release input is available. |
| server_owned_spec_mirrors_aligned | Server-owned published artifacts match owner-repository mirrors when those inputs are available. |
| diagnostic_provenance_complete | Validation-only provenance covers every catalog entry and object family. |
| object_family_metadata_declared | Catalog entries and published documents agree on object-family names and owners. |
| rendered_retrieval_surfaces_aligned | The built 2.0 page and full 2.0 model-retrieval bundles expose every available entry's catalog identity, public URL, and object-family ownership. |
| breaking_change_release_consistent_with_evolution_rule | Every breaking-change release value matches its evolution rule. |
| deliverable_specs_published | Every required platform surface has a published, parseable specification. |

The machine check loads the public JSON catalog, validates its vocabulary and
consumer-safe references, parses shipped specifications, checks object-family
and embedded-schema metadata, and compares package mirrors when available.
After the site and model bundles are generated, a semantic check compares their
rendered catalog values to that same JSON authority. Documentation prose and
heading text are not part of the comparison.

When the contract changes, update the packaged Workflow catalog, this public
JSON mirror, and affected published specification artifacts together. Human
review confirms that explanatory prose remains useful without treating it as
machine authority.

Continue to the [Platform Conformance Suite](/docs/platform-conformance)
to resolve the active, byte-bound fixtures that exercise these protocol
specifications. Historical fixture evidence is identified separately in that
suite and does not replace a current catalog authority.

<!-- Source: docs/platform-conformance.md -->

# Platform Conformance Suite

This page is the public authority for the Durable Workflow **platform
conformance suite**. It defines the conformance target matrix, reusable
fixture catalog, harness contract, pass / fail rules, and release gates
for implementations that claim Durable Workflow v2 compatibility.

The same manifest is advertised by the standalone server from
`GET /api/cluster/info` under `platform_conformance_suite`. The
[Platform Protocol Specs](/docs/platform-protocol-specs) catalog names
that nested manifest as the `platform_conformance_suite_manifest`
object family in the `cluster_info_envelope` spec.

The suite is downstream of the
[Version Compatibility](/docs/compatibility) authority. Where the suite
enumerates a surface family or stability rule, it must match the
surface-stability contract. The compatibility authority defines what
the contract is; this page defines how an implementation proves it
follows that contract.

## Public Conformance Authorities

The suite registers stable machine-readable authorities by schema identity
and public URL. The framework-neutral PHP SDK authority is published as
[`durable-workflow.v2.php-sdk-conformance-contract`](pathname:///platform-conformance/php-sdk-conformance.json).
It declares the released-package topology, scenario and evidence requirements,
and public runner and result-schema identifiers used to evaluate
`durable-workflow/sdk` against the released standalone server.

## Target Matrix

A conformance **target** is a kind of implementation that can claim
Durable compatibility. An implementation may claim more than one target.
For example, the standalone server claims `standalone_server`,
`worker_protocol_implementation`, and `repair_actionability_surface`.

| Target | Required surface families | Required fixture categories |
| --- | --- | --- |
| `standalone_server` | `server_api`, `worker_protocol`, `cluster_info_manifests` | `control_plane_request_response`, `signal_query_runtime_contract`, `workflow_update_runtime_contract`, `search_attribute_runtime_contract`, `schedules_runtime_contract`, `namespace_runtime_contract`, `child_workflow_runtime_contract`, `saga_runtime_contract`, `worker_versioning_runtime_contract`, `migration_runtime_contract`, `skew_refusal_matrix_contract`, `principal_attribution_contract`, `worker_task_lifecycle`, `failure_repair_actionability` |
| `embedded_engine` | `history_event_wire_formats` | `history_replay_bundles` |
| `official_sdk` | `official_sdks`, `worker_protocol`, `history_event_wire_formats` | `control_plane_request_response`, `signal_query_runtime_contract`, `workflow_update_runtime_contract`, `search_attribute_runtime_contract`, `schedules_runtime_contract`, `namespace_runtime_contract`, `child_workflow_runtime_contract`, `saga_runtime_contract`, `worker_versioning_runtime_contract`, `migration_runtime_contract`, `skew_refusal_matrix_contract`, `principal_attribution_contract`, `worker_task_lifecycle`, `history_replay_bundles` |
| `worker_protocol_implementation` | `worker_protocol`, `history_event_wire_formats` | `worker_task_lifecycle`, `signal_query_runtime_contract`, `workflow_update_runtime_contract`, `search_attribute_runtime_contract`, `schedules_runtime_contract`, `namespace_runtime_contract`, `child_workflow_runtime_contract`, `saga_runtime_contract`, `worker_versioning_runtime_contract`, `migration_runtime_contract`, `skew_refusal_matrix_contract`, `history_replay_bundles` |
| `cli_json_client` | `cli_json` | `control_plane_request_response`, `signal_query_runtime_contract`, `workflow_update_runtime_contract`, `search_attribute_runtime_contract`, `schedules_runtime_contract`, `namespace_runtime_contract`, `child_workflow_runtime_contract`, `saga_runtime_contract`, `worker_versioning_runtime_contract`, `migration_runtime_contract`, `skew_refusal_matrix_contract`, `principal_attribution_contract`, `cli_json_envelopes` |
| `waterline_contract_surface` | `waterline_api` | `signal_query_runtime_contract`, `workflow_update_runtime_contract`, `search_attribute_runtime_contract`, `namespace_runtime_contract`, `saga_runtime_contract`, `worker_versioning_runtime_contract`, `migration_runtime_contract`, `skew_refusal_matrix_contract`, `principal_attribution_contract`, `waterline_observer_envelopes` |
| `repair_actionability_surface` | `worker_protocol`, `server_api` | `failure_repair_actionability` |
| `mcp_discovery_surface` | `mcp_discovery_results` | `mcp_discovery_envelopes` |
| `prerelease_release_candidate` | `server_api`, `official_sdks`, `cli_json`, `waterline_api`, `cluster_info_manifests` | `skew_refusal_matrix_contract`, `workflow_update_runtime_contract`, `principal_attribution_contract`, `prerelease_readiness_contract` |

Targets are stable. Adding a target, adding a required surface to an
existing target, adding a required fixture category, promoting a
provisional category to required, changing stable runtime scenario
`operations` or `pass_criteria`, changing a stable runtime scenario public
requirement field (`artifact_policy`, `common_result_evidence`,
`required_matrix`, `scenario_requirements`, or `host_runner_contract`), or
changing a pass / fail rule is a suite contract change and must advance the
manifest version.

## Fixture Catalog

Each stable source is a public artifact with three machine-readable bindings in
the suite catalog: `artifact_id`, `resolver_url`, and `sha256`. Runtime artifact
ids end in the suite version. Protocol artifact ids end in the platform
protocol catalog version. A consumer resolves an artifact by fetching its
`resolver_url` and accepting the bytes only when their SHA-256 digest matches
the catalog binding.

| Category | Status | Consumer-resolvable authority |
| --- | --- | --- |
| `control_plane_request_response` | `stable` | [`durable-workflow.v2.control-plane-api@catalog-16`](https://raw.githubusercontent.com/durable-workflow/durable-workflow.github.io/f781ced1ae33c8697835bd527a125bdf3eaf4321/static/platform-protocol-specs/control-plane-api.openapi.yaml) |
| `worker_task_lifecycle` | `stable` | [`durable-workflow.v2.worker-protocol-api@catalog-16`](https://durable-workflow.github.io/platform-protocol-specs/v1.19/worker-protocol-api.openapi.yaml) |
| `worker_task_lifecycle` | `stable` | [`durable-workflow.v2.worker-protocol-stream@catalog-16`](https://durable-workflow.github.io/platform-protocol-specs/v1.19/worker-protocol-stream.asyncapi.yaml) |
| `signal_query_runtime_contract` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/signal_query_runtime_contract@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/signal-query-runtime-scenarios.json) |
| `workflow_update_runtime_contract` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/workflow_update_runtime_contract@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/workflow-update-runtime-scenarios.json) |
| `search_attribute_runtime_contract` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/search_attribute_runtime_contract@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/search-attribute-runtime-scenarios.json) |
| `schedules_runtime_contract` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/schedules_runtime_contract@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/schedules-runtime-scenarios.json) |
| `history_replay_bundles` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/history_replay_bundles@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/replay-runtime-scenarios.json) |
| `history_replay_bundles` | `stable` | [`durable-workflow.v2.history-event-payloads@catalog-16`](https://raw.githubusercontent.com/durable-workflow/durable-workflow.github.io/f781ced1ae33c8697835bd527a125bdf3eaf4321/static/platform-protocol-specs/history-event-payloads.schema.json) |
| `history_replay_bundles` | `stable` | [`durable-workflow.v2.replay-bundle@catalog-16`](https://raw.githubusercontent.com/durable-workflow/durable-workflow.github.io/f781ced1ae33c8697835bd527a125bdf3eaf4321/static/platform-protocol-specs/replay-bundle.schema.json) |
| `namespace_runtime_contract` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/namespace_runtime_contract@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/namespace-runtime-scenarios.json) |
| `child_workflow_runtime_contract` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/child_workflow_runtime_contract@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/child-workflow-runtime-scenarios.json) |
| `worker_versioning_runtime_contract` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/worker_versioning_runtime_contract@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/worker-versioning-runtime-scenarios.json) |
| `saga_runtime_contract` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/saga_runtime_contract@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/saga-runtime-scenarios.json) |
| `migration_runtime_contract` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/migration_runtime_contract@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/migration-runtime-scenarios.json) |
| `skew_refusal_matrix_contract` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/skew_refusal_matrix_contract@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/skew-refusal-matrix-scenarios.json) |
| `prerelease_readiness_contract` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/prerelease_readiness_contract@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/prerelease-readiness-scenarios.json) |
| `failure_repair_actionability` | `stable` | [`durable-workflow.v2.repair-actionability-objects@catalog-16`](https://raw.githubusercontent.com/durable-workflow/durable-workflow.github.io/f781ced1ae33c8697835bd527a125bdf3eaf4321/static/platform-protocol-specs/repair-actionability-objects.schema.json) |
| `cli_json_envelopes` | `stable` | [`durable-workflow.cli.output-schema-manifest@3`](https://durable-workflow.github.io/cli-json-envelopes/v3/manifest.json) |
| `principal_attribution_contract` | `stable` | [`durable-workflow.v2.platform-conformance.runtime-scenarios/principal_attribution_contract@38`](https://raw.githubusercontent.com/durable-workflow/workflow/75dfd5c869823409ef3d6c4b009a7882159ae9a2/resources/conformance/suite-v38/platform-conformance/principal-attribution-scenarios.json) |

The suite row retains CLI schema revision v3 as immutable historical evidence.
The current CLI contract is the
[`durable-workflow.cli.output-schema-manifest@4`](https://durable-workflow.github.io/cli-json-envelopes/v4/manifest.json)
closure, which publishes the Avro-only payload fields without rewriting v2 or
v3 bytes.

The current conformance-suite worker-protocol target uses the lifecycle-neutral
protocol 1.19 bytes recorded by
`durable-workflow.v2.worker-protocol-api@catalog-16` and
`durable-workflow.v2.worker-protocol-stream@catalog-16`. Protocol revisions
1.15 through 1.18 remain available at the versioned resolver URLs retained by
the suite history. These versioned fixtures do not replace the unversioned
current published Server protocol authority. The former beta-worded bytes
remain available only as the
explicitly historical
[`durable-workflow.v2.worker-protocol-api@catalog-16-beta-history`](https://raw.githubusercontent.com/durable-workflow/durable-workflow.github.io/e990bc36731463cc5b2cb2a9175dbccfdea61704/static/platform-protocol-specs/worker-protocol-api.openapi.yaml)
binding recorded in the suite manifest.

The planned `waterline_observer_envelopes` and `mcp_discovery_envelopes`
categories remain `provisional`. Their planned source-tree locations are
non-normative placeholders: harnesses must not resolve them as fixtures or use
them for a stable conformance claim.

## Workflow Lifecycle Release Authority

The exact released workflow-lifecycle scenario authority is
[published as JSON](https://durable-workflow.github.io/platform-conformance/workflow-lifecycle-scenarios.json).
It records the lifecycle requirements exercised at the current published
server, PHP SDK, and Rust SDK release boundary.

The PHP shard installs the exact `durable-workflow/sdk` package named by the
current artifact tuple from Packagist into a disposable Composer project. It
runs separate PHP client and worker processes against the matching public
server image, records the Packagist distribution and official `apache/avro`
provenance, and must report
`local_product_source_checkouts_used=false`.

The Rust shard installs the exact crate named by the current artifact tuple
from crates.io and records the registry source and checksum for that crate and
the official `apache-avro` crate. The payload proof uses the SDK's published
Avro envelope backed by `apache-avro`; a custom codec implementation or local
product checkout is not acceptable provenance.

The Rust shard must execute and report all of these cells:

- `instance_cancel` and `instance_terminate` through the public SDK commands.
- `selected_run_guard` and `stale_run_rejection` so a selected run cannot be
  confused with the instance's current run.
- `typed_failed`, `typed_cancelled`, `typed_terminated`, and `typed_timed_out`
  as typed terminal outcomes carrying workflow and run identity.
- `cancellation_heartbeat` and `late_activity_completion_refused` so activity
  code observes cancellation and a late completion cannot overwrite the
  terminal result.
- `worker_restart_during_cancellation` while cancellation settlement is still
  pending, proving that a replacement worker does not reclaim the closed
  activity.

The manifest also requires the exact artifact and server versions, cluster
identity, install provenance, workflow identities, per-cell outcomes, stable
reasons, payload contract, executor topology, Rust shard contract version,
runner identity, and shard exit status. Missing, unsupported, runner-blocked,
or unsuccessful Rust evidence cannot satisfy the lifecycle category.

A fixture category is required for a target only when the target lists
it and the category status is not `provisional`. Provisional categories
emit advisory warnings and become load-bearing only when promoted to
`stable` in a later suite version.

The `signal_query_runtime_contract` category is stable. A result for it
must record concrete pinned published artifact versions and must name
every required scenario as `pass`, `fail`, `unsupported`, `not_covered`,
or `runner_blocked` with linked findings. Placeholder or unresolved
version tokens such as `latest`, `current`, `head`, `<latest>`,
`${VERSION}`, or `{{ version }}` fail the result gate. Only `pass` cells
count toward a passing category:
`published_artifact_install_only`,
`python_worker_cli_and_sdk_baseline`, `php_worker_cli_and_sdk_baseline`,
`python_worker_php_facing_and_cli_clients`,
`php_worker_python_and_cli_clients`,
`rust_worker_rust_php_python_clients`, `python_worker_rust_client`,
`php_worker_rust_client`, `rust_query_error_and_immutability`,
`ordered_signal_delivery`,
`dedup_contract_observation`, `signal_during_replay`,
`query_during_replay`,
`rust_replayed_instance_state_query_after_cold_restart`,
`completed_run_signal_and_query`,
`unknown_signal_and_query_errors`,
`malformed_signal_and_query_payloads`, and
`waterline_operator_visibility`.

Those scenario ids and their pass criteria are published as the
machine-readable runtime scenario manifest at
[`static/platform-conformance/signal-query-runtime-scenarios.json`](pathname:///platform-conformance/signal-query-runtime-scenarios.json).
Implementation tests may exercise the scenarios, but they are not stable
fixture sources for external harnesses.

The Rust cells install the exact `durable-workflow` crate version declared by
the runtime scenario manifest from crates.io and record the Cargo registry
source and checksum for both the SDK and its resolved `apache-avro` dependency.
Snapshot-derived query transport is graded by
`rust_worker_rust_php_python_clients` and
`rust_query_error_and_immutability`. It is not replayed workflow-instance
state. The separate
`rust_replayed_instance_state_query_after_cold_restart` cell uses
`register_replayed_workflow` and `register_replayed_query`, starts a fresh Rust
worker process after a cold stop, restores durable history, and compares
running, restored, and completed state through Rust, PHP, and Python callers.
Both successful and failed query sequences capture history and workflow-command
counts before the first successful measured query and must leave those counts
unchanged.

For `completed_run_signal_and_query`, a completed cleanly run with a
replayable declared query handler must return its final query state
through every claimed public query surface. Stable terminal-state errors
are valid only for explicitly unsupported terminal states or unavailable
handlers; a generic completed-run terminal error is not a passing result
for a replayable completed run.

The `workflow_update_runtime_contract` category is a stable runtime
scenario category. A result for it must use published artifacts, pin the
server, CLI, Python SDK, PHP SDK, and Waterline versions,
state whether any local product source checkouts were used, and name
every required update scenario as `pass`, `fail`, `unsupported`,
`not_covered`, or `runner_blocked` with linked findings. Passing update
evidence must exercise public control-plane, history, CLI/SDK, and
operator-readable surfaces; local source execution does not count.

Required workflow update scenarios cover published-artifact install,
declared update contract visibility, accepted update control-plane and
history evidence, running or waiting operator visibility, completed
result round trip, failed or refused outcomes, duplicate request or
idempotency behavior, unknown update refusal, invalid input refusal,
payload envelope round trip, terminal-workflow behavior, authenticated
principal attribution, PHP client/worker parity, Python client/worker
parity, and operator diagnostics. If an SDK lacks first-class update
support, the result must report a typed unsupported cell with a focused
SDK finding instead of silently omitting the language.

Those scenario ids and their pass criteria are published as the
machine-readable runtime scenario manifest at
[`static/platform-conformance/workflow-update-runtime-scenarios.json`](pathname:///platform-conformance/workflow-update-runtime-scenarios.json).
The current host-runner handoff is intentionally `runner_blocked` until a
registered host runner can install the pinned published artifacts and
drive the full workflow-update matrix.

The `search_attribute_runtime_contract` category is a stable runtime
scenario category. A result for it must use published artifacts, cover
PHP and Python workflow start/upsert behavior, CLI query and error
surfaces, Waterline operator visibility, cross-language codec round
trips, equality/range/bool queries, OR/NOT grammar, keyword-list
membership, type safety, indexing latency distribution, load latency,
namespace isolation, reserved-name refusal, and query injection
hardening. A Python/server smoke subset is nonconforming until every
required cell is recorded as `pass`, `fail`, `unsupported`,
`not_covered`, or `runner_blocked` with linked findings. Only `pass`
cells count toward a passing category.

Those search-attribute scenario ids and their pass criteria are
published at
[`static/platform-conformance/search-attribute-runtime-scenarios.json`](pathname:///platform-conformance/search-attribute-runtime-scenarios.json).

The `schedules_runtime_contract` category is a stable runtime scenario
category. A result for it must use published artifacts and cover cron
cadence, fixed-rate cadence, list and describe visibility, pause/resume
windows with no fires, delete stopping future fires, missed-fire policy,
restart survival, CLI schedule operations, Python SDK schedule
operations, PHP-facing schedule operations, Python-created PHP workflow
fires, PHP-created Python workflow fires, invalid cron refusal, and
nonexistent workflow type outcomes. A schedule lifecycle smoke subset is
nonconforming until every required cell is recorded as `pass`, `fail`,
`unsupported`, `not_covered`, or `runner_blocked` with linked findings.
Only `pass` cells count toward a passing category.

Those schedules scenario ids and their pass criteria are published at
[`static/platform-conformance/schedules-runtime-scenarios.json`](pathname:///platform-conformance/schedules-runtime-scenarios.json).

The `history_replay_bundles` category is also a stable runtime scenario
category. A result for it must use published artifacts, cover PHP and
Python replay for completed histories, worker restart, activity,
signal/update, wait-condition, version-marker, saga-compensation,
explicit code-divergence refusal, server-side history mutation refusal,
malformed history refusal, and in-flight signal restart timing. A
golden-history smoke subset is nonconforming until every required cell is
recorded as `pass`, `fail`, `unsupported`, `not_covered`, or
`runner_blocked` with linked findings. Only `pass` cells count toward a
passing category.

Those replay scenario ids and their pass criteria are published at
[`static/platform-conformance/replay-runtime-scenarios.json`](pathname:///platform-conformance/replay-runtime-scenarios.json).

The `namespace_runtime_contract` category is a stable runtime scenario
category. A result for it must use published artifacts, cover namespace
create/update/describe/list, lifecycle cleanup and recreate, workflow
visibility and mutation isolation, PHP worker task-queue isolation, CLI
namespace context and default-scope behavior, SDK namespace selection
parity, search-attribute schema and value query isolation, schedule
isolation, Waterline/operator scoped visibility, explicit Nexus
cross-namespace calls, reserved-name refusal, and result-record routing
for product findings. A namespace smoke subset is nonconforming until
every required cell is recorded as `pass`, `fail`, `unsupported`,
`not_covered`, or `runner_blocked` with linked findings. Only `pass`
cells count toward a passing category.

Non-normative implementation notes for Waterline/operator scoped
visibility can come from the published Waterline artifact or a focused
Waterline shard. Useful reviewer captures include scoped workflow list
and detail views, schedule views when the product exposes them,
search-attribute values, dashboard scope, operator API stats, and the
documented verdict for any default or unscoped view advertised by the
product; the normative pass criteria remain the
`waterline_operator_namespace_visibility` scenario below.

Those namespace scenario ids and their pass criteria are published at
[`static/platform-conformance/namespace-runtime-scenarios.json`](pathname:///platform-conformance/namespace-runtime-scenarios.json).

The `child_workflow_runtime_contract` category is a stable runtime
scenario category. A result for it must use published artifacts and cover
same-language PHP and Python parent/child runs, PHP-to-Python and
Python-to-PHP parent/child runs, child failure round-trip typing, parent
cancellation propagation to a child, direct child cancellation observed
by a parent, replay across parent worker restart while waiting on a
child, concurrent fan-out to five children, and namespace behavior. A
single parent/child smoke is nonconforming until every required cell is
recorded as `pass`, `fail`, `unsupported`, `not_covered`, or
`runner_blocked` with linked findings. Only `pass` cells count toward a
passing category.

Those child-workflow scenario ids and their pass criteria are published
at
[`static/platform-conformance/child-workflow-runtime-scenarios.json`](pathname:///platform-conformance/child-workflow-runtime-scenarios.json).

The `saga_runtime_contract` category is a stable runtime scenario
category. A result for it must use published artifacts and cover forward
success, failure after a later step with reverse-order compensation,
early-step failure with no extra compensation, compensation retry
idempotence, compensation-failure terminal visibility, mid-compensation
worker restart, PHP workflow to Python compensation, Python workflow to
PHP compensation, typed compensation error round trips, and
operator-visible in-progress compensation status. A saga smoke that only
proves one happy path or one SDK is nonconforming until every required
cell is recorded as `pass`, `fail`, `unsupported`, `not_covered`, or
`runner_blocked` with linked findings. Only `pass` cells count toward a
passing category. The PHP package is identified as `workflow-php` in the
runtime matrix and may also be recorded as `workflow` when comparing
against the platform release artifact set.

Those saga scenario ids and their pass criteria are published at
[`static/platform-conformance/saga-runtime-scenarios.json`](pathname:///platform-conformance/saga-runtime-scenarios.json).

The `worker_versioning_runtime_contract` category is a stable runtime
scenario category. A result for it must use published artifacts and
cover worker build-ID registration, operator rollout visibility,
drain/resume controls, per-run compatibility pins, replay only by
compatible workers, promoted-version routing for new starts, replay
after cache eviction, no-compatible-worker diagnostics, CLI and
Waterline visibility surfaces, PHP/Python cross-language pinning,
adversarial no-version-bump behavior, and history API version pins. A
worker-versioning smoke subset is nonconforming until every required
cell is recorded as `pass`, `fail`, `unsupported`, `not_covered`, or
`runner_blocked` with linked findings. Only `pass` cells count toward a
passing category.

Cross-language PHP/Python pinning evidence must record worker runtime
identities, workflow and run IDs, task-queue rollout state, the public
poll and rollout outcomes used to verify pinning, published worker
artifact install source and version, and confirmation that no local
product source checkout was used.

Those worker-versioning scenario ids and their pass criteria are
published at
[`static/platform-conformance/worker-versioning-runtime-scenarios.json`](pathname:///platform-conformance/worker-versioning-runtime-scenarios.json).

The `migration_runtime_contract` category is a stable runtime scenario
category. A result for it must use published artifacts and cover the
latest supported v1 state setup, the documented migration steps,
completed-history preservation and replay, in-flight workflow progress,
mid-activity retry state, signal-or-timer wait state, schedule cadence, worker registration
projection, Waterline operator visibility, CLI access to preupgrade
state, new v2 workflow starts, queue-aware rollback semantics, and loud
refusal for unsupported version skew. Rollback evidence must inventory ready,
delayed, and reserved v1 queue work; record whether the recovery boundary was
drained, captured as an application-consistent SQL-plus-queue cut, or accepted
as unrecoverable; and prove that each restored nonterminal v1 row has a
runnable queue or signal wake path. An eligible stale `pending` row may instead
use the supported v1.0.77 Watchdog only when evidence observes the worker loop
dispatching the enabled-by-default Watchdog, the Watchdog redispatching that
workflow after its five-minute stale bound, and the row advancing on its
recorded queue. That pending-only path does not replace preserved queue state
for retries, timers, or `waiting`/`running` work. SQL-only restore is passing
rollback evidence only when no other nonterminal queue-dependent v1 execution
exists at the recovery cut. The recovery manifest must record only the
`APP_KEY` secret-manager reference and version, with the key and recovery
credentials kept separately access-controlled from SQL and queue backups. A
fresh-install smoke or a migration run that
does not start from realistic v1 state is nonconforming until every
required cell is recorded as `pass`, `fail`, `unsupported`,
`not_covered`, or `runner_blocked` with linked findings. Only `pass`
cells count toward a passing category.

The documented-steps cell must record the live guide command list, the
commands actually executed in guide order, exit codes, and per-command
timings. Before and after state snapshots must include observed
completed-history, in-flight workflow, retrying-activity, signal-or-timer
wait, schedule, and worker-registration cells.

Those migration scenario ids and their pass criteria are published at
[`static/platform-conformance/migration-runtime-scenarios.json`](pathname:///platform-conformance/migration-runtime-scenarios.json).

The `skew_refusal_matrix_contract` category is a stable runtime scenario
category. A result for it must use published artifacts and cover
compatible, backward-skewed, forward-skewed, and outside-window pairings
for CLI, Python SDK, the standalone PHP SDK worker, and Waterline surfaces.
Workflow is present only as the embedded Laravel and Waterline engine. It
must also probe future-version boundaries, capture requests and
responses for every skewed operation, classify worker skew as
`register_refused`, `register_and_serve`, or `register_and_drop`, and
classify Waterline skew as `banner`, `render_refused`, or
`stale_render`. A protocol-manifest smoke subset is nonconforming until
every required cell is recorded as `pass`, `fail`, `unsupported`,
`not_covered`, or `runner_blocked` with linked findings. Only `pass`
cells count toward a passing category; `register_and_drop` and
`stale_render` without a loud warning are blocking product findings.

Those skew-refusal scenario ids and their pass criteria are published at
[`static/platform-conformance/skew-refusal-matrix-scenarios.json`](pathname:///platform-conformance/skew-refusal-matrix-scenarios.json).

The `principal_attribution_contract` category is a stable runtime
scenario category. A result for it must use published artifacts and
prove that workflow history records server-derived principals for start,
signal, query, cancellation, completion, failure, anonymous, and
server-originated event surfaces. It must exercise adversarial
payload/header spoofing, alice/bob named identities, credential
rotation, CLI operator visibility, Waterline operator visibility, and
authenticated start or signal operations through both the Python SDK and
the PHP `DurableWorkflow\Client`. SDK cells must record the
package version, operation outputs, history/API principal samples, and
raw HTTP reference principals so the harness can compare principal shape
and expected principal ids. The PHP cell must resolve `durable-workflow/sdk`
as an exact Packagist distribution and record its dist type, URL, and
reference. `durable-workflow/workflow` is resolved separately only for the
embedded Laravel and Waterline engine. A role-token smoke subset is nonconforming
until every required scenario is recorded as `pass`, `fail`,
`unsupported`, `not_covered`, or `runner_blocked` with linked findings.

Those principal-attribution scenario ids and their pass criteria are
published at
[`static/platform-conformance/principal-attribution-scenarios.json`](pathname:///platform-conformance/principal-attribution-scenarios.json).

The `prerelease_readiness_contract` category is a stable runtime
scenario category for the coordinated 2.0 release candidate. A result
for it must use only published artifacts and public user-facing docs,
record separate Workflow and Waterline GO / NO-GO verdicts, cover core
feature completeness, migration readiness, public API stability,
documentation accuracy, configuration understandability, and
cross-component compatibility, and evaluate server, CLI, PHP, Python, and
Rust SDKs, Workflow, Waterline, sample app, and public docs as one ecosystem
tuple. It must also execute each versioned 2.0 standalone quickstart path and
the separate embedded Laravel path from live public docs through observable
completed workflows within 10 minutes, recording exact commands, outputs,
artifact versions, package provenance, and wall-clock timings. A
discovery-only quickstart check is nonconforming until it records the PHP,
Python, Rust, and Laravel quickstart scenarios as `pass`, `fail`,
`unsupported`, `not_covered`, or `runner_blocked`.
Any missing artifact, stale docs page, undocumented migration step,
installability gap, API instability, cross-component breaking-change
risk, or release-channel mismatch must be recorded as a non-pass cell
with a linked finding. Runner-only evidence is nonconforming and cannot
make prerelease readiness green.

Those prerelease readiness scenario ids and their pass criteria are
published at
[`static/platform-conformance/prerelease-readiness-scenarios.json`](pathname:///platform-conformance/prerelease-readiness-scenarios.json).
The concise [2.0 quickstart](/docs/quickstart/) is the public onboarding path.

## Pass / Fail Rules

1. **`guaranteed_field_equality`.** Every field marked guaranteed in the
   fixture schema must be present, type-correct, and value-equal in the
   implementation response. Diagnostic-only fields are ignored.
2. **`unknown_additive_fields_tolerated`.** Extra fields pass only if
   they are documented diagnostic-only fields or the fixture is on a
   stability level that allows additive evolution.
3. **`frozen_shape_exact_match`.** Fixtures backed by a `frozen` surface
   family must match exactly. A frozen-shape mismatch is always a
   failure.
4. **`required_fixtures_must_pass`.** A release that claims a target must
   pass every required fixture category for that target. One failed
   required fixture means the release does not conform for that target.
5. **`stable_runtime_scenario_coverage`.** A stable runtime category
   must report every required scenario it declares with one of the
   statuses published by its runtime scenario manifest: `pass`, `fail`,
   `unsupported`, `not_covered`, or `runner_blocked`. Full conformance
   requires every required scenario to pass. A smoke-only subset,
   omitted scenario, unsupported public surface, uncovered cell, or
   runner-blocked cell is nonconforming and must link the owning
   finding. This status set and pass-only runtime rule are suite version
   5+ semantics.
6. **`provisional_categories_warn_only`.** A failed fixture in a
   provisional category emits a warning and does not block the release.
7. **`diagnostic_only_mismatches_pass`.** If only diagnostic-only fields
   differ, the harness records the difference in `diagnostic_diff` and
   the fixture passes.

The harness result declares one of four conformance levels:

| Level | Meaning |
| --- | --- |
| `full` | Every required fixture passes for every claimed target. |
| `partial` | Every required fixture passes for at least one claimed target, but another claimed target is failing. |
| `provisional` | Only provisional categories failed; required categories all passed. |
| `nonconforming` | At least one required fixture failed for every claimed target. |

## Harness Contract

A conforming harness:

- loads the suite manifest from `platform_conformance_suite` in
  `GET /api/cluster/info`, or from the static mirror for offline runs;
- loads each declared fixture from its source-of-truth path;
- drives the implementation through the fixture's documented operation;
- compares the response under the pass / fail rules above;
- emits one result document per run with schema
  `durable-workflow.v2.platform-conformance.result`, suite version,
  implementation identity, per-fixture results, diagnostic diff, and
  overall conformance level;
- exits non-zero if and only if the conformance level is
  `nonconforming`.

The result document is an artifact. A compatibility claim is valid only
when the result was produced against the implementation build and the
suite version named by that build.

## Release Gates

| Release | Required claimed target(s) | Required artifact |
| --- | --- | --- |
| `durable-workflow/server` | `standalone_server`, `worker_protocol_implementation`, `repair_actionability_surface` | Harness result document attached to the release. |
| `durable-workflow/workflow` | `embedded_engine` | Harness result document attached to the release. |
| `durable-workflow/sdk` | `official_sdk`, `worker_protocol_implementation` | Harness result document attached to the release. |
| `durable_workflow` | `official_sdk`, `worker_protocol_implementation` | Harness result document attached to the release. |
| `dw` | `cli_json_client` | Harness result document attached to the release. |
| `waterline` | `waterline_contract_surface` | Harness result document attached to the release. |
| `durable-workflow/2.0-release-candidate` | `prerelease_release_candidate` | Conformance record stores the published-artifact prerelease readiness result. |

Release reviewers confirm that the harness result is attached, the
conformance level is `full` or `provisional`, and the suite version in
the result matches the version exposed by the build under test. A
`nonconforming` result blocks the release.

## Release Check

The docs-site release check in
`scripts/check-platform-conformance-authority.js` fails the build if
the static manifest points at a missing, repo-local, version-alias-only,
or non-docs-site authority. It also validates every stable fixture-level
`authority_doc` value as a canonical current-version docs-site URL and
rejects repository-relative source trees, test fixtures, documentation,
schema directories, unversioned artifact ids, non-public resolvers, and
incorrect byte digests in every stable fixture category. Runtime categories
must additionally expose a suite-bound public scenario manifest.
The same check requires this page to list the manifest schema, target
names, fixture category names, pass / fail rules, and release gates from
the machine-readable mirror.

When the suite changes, update this page and
`static/platform-conformance-contract.json` together. If the change
adds a target, adds a required fixture category, promotes a provisional
category to stable, changes stable runtime scenario `operations` or
`pass_criteria`, changes a stable runtime scenario public requirement
field (`artifact_policy`, `common_result_evidence`, `required_matrix`,
`scenario_requirements`, or `host_runner_contract`), or changes a pass /
fail rule, advance the suite version in the same release change. The
release check pins stable runtime scenario criteria and public runtime
requirement snapshots by suite-versioned digests so external harnesses
cannot observe new criteria or evidence-policy requirements under an old
suite version.

Published runtime scenario criteria and public requirement digest entries
are append-only. To advance stable runtime scenario `operations` or
`pass_criteria`, leave every existing
`VERSIONED_RUNTIME_SCENARIO_CRITERIA_DIGESTS` entry unchanged, increase the
suite version, update the scenario manifest to that version, and add a new
criteria digest entry for the new current suite. To change public runtime
manifest evidence requirements, artifact policy, required matrix, scenario
requirement fields, or host-runner contract fields, leave every existing
`VERSIONED_RUNTIME_SCENARIO_PUBLIC_REQUIREMENT_DIGESTS` entry unchanged,
advance the suite version, update the scenario manifest to that version,
and add the corresponding public requirement digest entry for the new
current suite. The release check compares published digest entries from the
target branch against the current change, so editing or deleting an older
suite entry fails even when the current suite version adds a new digest.

<!-- Source: docs/sdk-neutrality.md -->

# SDK Neutrality

This page is the human-readable guide to the platform-wide SDK neutrality
contract. It enumerates the minimum neutrality rules that
every public Durable Workflow surface must satisfy and the standing
language-agnosticism audit that release reviewers apply to new server,
workflow, CLI, Waterline, and MCP surfaces.

The upstream architecture guide is published at
[`sdk-neutrality.md`](https://github.com/durable-workflow/workflow/blob/main/docs/architecture/sdk-neutrality.md).
The consumable machine-readable authority is published from this site at
[`/sdk-neutrality-contract.json`](/sdk-neutrality-contract.json) under
the schema id `durable-workflow.v2.sdk-neutrality.contract`. Its protocol and
SDK-breadth authority comes from the JSON shipped in the Workflow Composer
package at `resources/sdk-neutrality-contract.json`. The public mirror adds
current package-distribution metadata from the centralized docs artifact
tuple. The standalone Durable Workflow server re-exports the packaged base
manifest from `GET /api/cluster/info` under `sdk_neutrality_contract`. When
this page disagrees with the published JSON contract, the JSON contract wins
and the disagreement is a documentation bug.

This contract is downstream of the
[Version Compatibility](/docs/compatibility) authority (which says
*which* surfaces are public and *how* they may change) and the
[Platform Protocol Specs](/docs/platform-protocol-specs) catalog
(which says *where* the normative spec for each surface lives). It says
*what shape* those specs are allowed to take so that a future SDK outside the
current PHP, Python, and Rust roster can target them without requiring a
protocol redesign.

## Why this exists

Durable ships three first-party standalone SDKs: the PHP `durable-workflow/sdk`
package, Python `durable_workflow` package, and Rust `durable-workflow` crate.
The separate PHP `durable-workflow/workflow` package remains the embedded
Laravel engine and replay owner; it is not the framework-neutral PHP SDK.
Building or maintaining a wide first-party SDK roster is **not** a release goal.
Demand for SDKs in TypeScript, Go, Java, and .NET ecosystems has not
yet been demonstrated, and the maintenance cost of a broad official
roster is high.

What this contract protects against is a different failure mode: the
public contracts under those SDKs quietly hard-coding language-specific
assumptions. If a future TypeScript or Go SDK becomes worth
building, the work should be "write a new client against the published
wire protocol", not "redesign the protocol so another language can speak it
at all".

## Scope

| Scope key | Meaning |
| --- | --- |
| `goal` | Preserve protocol and contract neutrality so a future TypeScript, Go, Java, or .NET SDK does not require a protocol redesign to exist. |
| `non_goal` | Ship a broad official SDK portfolio. First-party SDK breadth is intentionally narrow and grows only when adoption demand justifies it. |
| `present_priority` | Python is the current highest-value non-PHP path for existing users and is treated as a priority surface for parity coverage. |
| `future_posture` | TypeScript, Go, Java, .NET and other languages are demand-driven. They have no reserved release slot, but every public contract must be shaped so a future SDK in those languages can be written without breaking the wire protocol. |

## Official SDK breadth policy

The official-SDK roster is intentionally narrow:

| Language | Posture | Published package | Conformance authority |
| --- | --- | --- | --- |
| PHP | `priority` | [`durable-workflow/sdk`](https://packagist.org/packages/durable-workflow/sdk) | [`signal_query_runtime_contract`](https://durable-workflow.github.io/platform-conformance/signal-query-runtime-scenarios.json), actors `sdk_php`, `php_sdk_client`, and `php_worker` |
| Python | `priority` | <PythonPackageReleaseLink>`durable_workflow` 2.0.0</PythonPackageReleaseLink> | [`history_replay_bundles`](https://durable-workflow.github.io/platform-conformance/replay-runtime-scenarios.json), actor `python_sdk_runtime` |
| Rust | `priority` | [`durable-workflow`](https://crates.io/crates/durable-workflow) | [`signal_query_runtime_contract`](https://durable-workflow.github.io/platform-conformance/signal-query-runtime-scenarios.json), actors `rust_sdk`, `rust_worker`, and `rust_sdk_client` |
| TypeScript | `demand_driven` | None | Public contracts must remain implementable in TypeScript without protocol redesign. |
| Go | `demand_driven` | None | Public contracts must remain implementable in Go without protocol redesign. |
| Java | `demand_driven` | None | Public contracts must remain implementable in Java without protocol redesign. |
| .NET | `demand_driven` | None | Public contracts must remain implementable in .NET without protocol redesign. |

The JSON contract lists the exact `scenario_ids` that constitute coverage for
`first_party.php_sdk`, `first_party.python_sdk`, and `first_party.rust_sdk`.
It tracks the Laravel package separately as
`embedded_engines.php_workflow_engine`, with replay coverage under actor
`workflow_php_runtime`. Consumers can resolve those identifiers directly in
the linked catalogs without cloning an SDK repository or reading its test
suite.

A new first-party SDK is added only when:

1. There is documented user demand the existing SDKs cannot serve.
2. A candidate maintainer team commits to keeping the SDK on the
   conformance harness, the protocol-spec catalog, and the release
   authority manifest.
3. Adding the SDK does not require breaking changes to the worker
   protocol, control plane, history-event wire formats, or replay
   fixtures. If it would, the protocol is the bug, not the SDK.

## Neutrality rules

Every public Durable contract must satisfy each of the seven neutrality
rules below. The JSON manifest is the authority for the exact field
shapes; the summaries are for reviewers. Each rule on the JSON manifest
carries a `requirement`, `rationale`, `how_to_apply` line, and an `authority`
array. Every authority entry has a stable catalog or schema ID and an absolute
URL to its published artifact.

| Rule | Requirement |
| --- | --- |
| `protocol_neutrality` | Public RPC and event surfaces use HTTP+JSON or AsyncAPI shapes that any HTTP-capable runtime can produce and consume. |
| `codec_neutrality` | Every durable payload that crosses a public boundary advertises `avro`. No other public or engine-specific v2 codec is offered. |
| `error_shape_neutrality` | Public failure objects use a structured envelope of (`code`, `message`, optional `details`). PHP and Python exception class names are diagnostic only. |
| `type_identity_neutrality` | Workflow, activity, child workflow, and exception types are identified by stable string names. Class FQCNs and module paths are SDK-input convenience, not contract. |
| `replay_fixture_neutrality` | Replay fixtures and golden history bundles are JSON conforming to the published `history_event_payloads` and `replay_bundle` schemas. |
| `discovery_neutrality` | Every public surface is reachable from `GET /api/cluster/info` and the `platform_protocol_specs` catalog. |
| `documentation_neutrality` | Public-contract docs describe shapes in schema, route, and field semantics. PHP and Python class behaviour appears as SDK examples, not as the normative contract. |

## Standing language-agnosticism audit

Every new server, workflow, CLI, Waterline, or MCP surface must clear
the neutrality audit before promotion to `stable`. The audit is a
standing review item on the release PR for every change that touches an
audit-scoped surface family. The audit-scoped families are:

- `server_api`
- `worker_protocol`
- `cli_json`
- `waterline_api`
- `mcp_discovery_results`
- `cluster_info_manifests`

The checklist has eight steps. Seven correspond to the neutrality rules
above. The eighth — the **future-SDK thought experiment** — asks the
reviewer to describe in two sentences how a TypeScript or Go SDK would
consume the new surface using only the published spec catalog and a
standard HTTP+JSON toolchain. If the answer requires a first-party SDK,
the surface is not neutral and either the surface is reshaped or the
neutrality gap is recorded as a known limitation before promotion.

## What a future SDK relies on

The contract identifies the surfaces a future SDK must be able to read
without inspecting any first-party SDK source:

- **Protocol** — `durable-workflow.v2.control-plane-api`,
  `durable-workflow.v2.worker-protocol-api`, and
  `durable-workflow.v2.worker-protocol-stream` in the published
  [protocol catalog](https://durable-workflow.github.io/platform-protocol-specs.json).
- **Codecs** — the universal codec set documented by
  `durable-workflow.v2.worker-protocol-api` and advertised through the
  `durable-workflow.v2.cluster-info-envelope` discovery schema.
- **Error shape** — the worker-protocol failure envelope and
  `durable-workflow.v2.repair-actionability-objects` schema.
- **Replay inputs** — the `durable-workflow.v2.history-event-payloads` and
  `durable-workflow.v2.replay-bundle` JSON Schemas plus scenario IDs in the
  public [`history_replay_bundles`](https://durable-workflow.github.io/platform-conformance/replay-runtime-scenarios.json)
  catalog.
- **Discovery** — `durable-workflow.v2.cluster-info-envelope` and the
  `durable-workflow.v2.platform-protocol-specs.catalog` itself.

If any of those surfaces is not reachable for a candidate SDK in a
given language, building the SDK requires protocol changes and the
language-agnosticism guarantee is not being honored.

## Release gates

A release that introduces a new public surface family or promotes an
existing surface from `prerelease` or `experimental` to `stable` must
record the audit outcome on the release PR. The release-gates section
of the manifest enumerates the specific checks:

- `audit_recorded` — the release PR description states which audit
  steps were applied and links the protocol-spec catalog entry, the
  conformance fixture, and the discovery entry for the new surface.
- `no_php_or_python_only_required_fields` — no guaranteed field on a
  `stable` surface requires a PHP-only or Python-only codec.
- `universal_codec_advertised` — worker protocol negotiation advertises
  exactly one universal codec, `avro`.
- `fixture_schema_validated` — new replay fixtures or golden history
  bundles validate against the published JSON Schemas.
- `discovery_entry_present` — new public surfaces have a
  `platform_protocol_specs` catalog entry with a non-empty
  `surface_family`, `owner_repo`, and `format`.

Enforcement is split between machine and human gates. Release CI resolves every
authority URL, protocol/schema ID, and conformance scenario ID in the public
contract; cross-references the audit scope against the surface stability
families; and rejects repository-local authority paths or implementation
symbols. Release reviewers tick the SDK-neutrality audit on every release PR
that adds or promotes a public surface and remain responsible for the
future-SDK thought experiment.

## Changing this contract

Adding a neutrality rule, tightening an existing rule, adding a
required audit step, adding a surface family to the audit scope, or
changing the official-SDK breadth policy is a contract change. Bump the
manifest version, update the architecture guide and this page in the same
change, and align the consumable contract at
[`/sdk-neutrality-contract.json`](/sdk-neutrality-contract.json).
Removing a neutrality rule or audit step is a major change.