Replay and recovery

Replay republishes stored artefacts back into pipeline queues so a downstream stage can be re-executed without re-fetching source data. Two flavours: same-pipeline (rerun a stage in the original config) and cross-pipeline (route the output into a different config). Both spin up a fresh execution and run the publisher detached from the parent.

When to replay

A downstream adapter failed; fix it; rerun from that stage
You updated adapter logic and want to test the new version on the same inputs
An LLM call timed out; retry with a different model
You want to fan the same upstream data out to a new pipeline

Prerequisite: storage still has the data

Replay reads the parent's stored stage output (the SRC_ID/from_stage prefix). If objects are missing, replay fails loudly before publishing. Storage is not a cache — retention is a deployment concern (see Storage model).

Verify before replay:

factflow storage list --prefix executions/SRC_ID/ | head

Every replay creates a new execution with the parent recorded as parent_execution_id. The original execution is never mutated. --from-stage and --to-route are both required; --config-id is what splits same-pipeline from cross-pipeline.

Same-pipeline replay

Rerun a stage within the original pipeline. Reads the parent's stored stage output and republishes it to a route in the same config, scoped under the new replay execution.

factflow execution replay SRC_ID \
  --from-stage markdown_segmented \
  --to-route embedding_generator

Routes resolve from the parent execution's config_snapshot, so the queue mapping matches exactly what produced the data. Upstream stages don't rerun — only the target route and what flows downstream of it.

Replay a subset by narrowing to a path within the stage:

factflow execution replay SRC_ID \
  --from-stage markdown_segmented \
  --to-route embedding_generator \
  --from-path batch-2024-06/

Cross-pipeline replay

Route the source's output into a different pipeline's route. Add --config-id to point at the target config. Used when:

Two pipelines share a stage output format (e.g., both consume segmented markdown)
You want to re-process existing scraped pages through an updated consolidation pipeline

factflow execution replay SRC_ID \
  --from-stage markdown \
  --to-route embeddings_route \
  --config-id TARGET_CONFIG_ID

Storage is still read from the parent (SRC_ID), but route → queue resolution comes from the target config's snapshot, not the parent's. This prevents a class of subtle bugs when the same route name appears in multiple configs with different queues — the resolver loads only the snapshot that owns the route you named, never the global config directory.

Via the API

curl -X POST http://localhost:8000/api/v1/executions/SRC_ID/replay \
  -H 'Content-Type: application/json' \
  -d '{"replay_source": "storage", "from_stage": "markdown", "to_route": "embeddings_route"}'

replay_source is required and accepts storage, lineage, or manual. The CLI sets it for you (its --source flag defaults to storage); over raw HTTP you must include it. Returns a new execution id immediately — the replay publisher runs asynchronously. Add "config_id" for cross-pipeline replay; "from_path" for a subset; "tag" to label the new execution. Poll or wait on the returned id as you would any other execution.

Failure modes

Source data is gone. Replay reads from the parent's stored from_stage output. If nothing is there, it raises before publishing — no execution proceeds. Check factflow storage list first.
Unknown route. The route name must exist in the resolving config snapshot. An unresolvable --to-route fails the new execution fast, before any messages are published.
Partial publish. The publisher collects per-object errors instead of aborting on the first one. If some objects fail to publish, the run completes with status partial_failure; a clean run reports publishing_complete. Inspect the execution's error info to see which keys failed.
Cancelled mid-replay. Cancelling or stopping the replay execution cancels the publisher task cleanly (see below) — no orphaned publishers, no leaked queue writes.

Recovery

Similar shape, different intent: recovery handles executions that were interrupted mid-run (server crash, manual kill). On startup the orchestrator marks every execution still in running as interrupted so stale records don't linger — recovery then replays their stored artefacts to resume processing.

Recovery isn't a CLI subcommand. It's engaged automatically at startup; the operator surface is the same storage and route-resolution mechanics described above.

Under the hood

The runtime behind replay lives in factflow_replay. Three pieces matter to an operator debugging a replay:

DetachedReplayService does the publishing. It lists the parent's storage prefix (skipping .meta.json sidecars), assigns a fresh correlation_id per object and one replay_batch_id for the run, and publishes each message with the new execution_id in message metadata and the object_key in the payload. It publishes everything before returning — the queue is the buffer, so routes consume at their own pace.
Manager-owned publisher. The server schedules the publisher as an asyncio task that OrchestratorManager owns (schedule_replay). Cancelling or stopping the replay execution cancels that task — this is why a cancelled replay leaves no orphaned publisher.
Recovery on restart. cleanup_stale_executions() runs at startup and flips orphaned running executions to interrupted; the recovery manager then drives storage replay for them.

Full API surface (signatures, result fields, the RouteQueueResolver) is on the package page — don't reach for these from pipeline YAML; replay is driven via CLI/API only.

factflow-replay package — the API surface: DetachedReplayService, RouteQueueResolver, recovery manager
Concept: Storage model — why storage is the replay source of truth
Concept: Lineage — use lineage to find the right --from-stage
Concept: Executions — how replay fits the multi-execution model
API: HTTP reference — the POST /executions/{id}/replay endpoint