Skip to content
Factflow

Storage operations

Every pipeline writes to storage via StorageProtocol. Two providers ship: filesystem (dev + single-node prod) and MinIO / S3 (multi-node prod). The CLI and API work identically across both.

Browse the hierarchy

Section titled “Browse the hierarchy”

The default key convention is executions/<exec-id>/<route>/<stage>/<msg-id>. Browse by prefix:

Terminal window
factflow storage browse executions/EXEC_ID/
factflow storage browse executions/EXEC_ID/sitemap_scraper/
factflow storage browse executions/EXEC_ID/sitemap_scraper/web_scraper/

browse groups by prefix; list gives a flat listing.

List by execution

Section titled “List by execution”

list has no execution flag — filter by key prefix instead:

Terminal window
factflow storage list --prefix executions/EXEC_ID/
factflow storage list --prefix executions/EXEC_ID/ --limit 100

Other filters: --correlation-id, --stage, --route-id, --extension, --offset.

Read an object

Section titled “Read an object”
Terminal window
factflow storage get KEY

Writes raw content to stdout. For binary content, redirect or use -o/--output (a file path, not a format selector):

Terminal window
factflow storage get executions/EXEC_ID/path/to/object > out.bin
factflow storage get executions/EXEC_ID/path/to/object -o out.bin

Inspect metadata

Section titled “Inspect metadata”

Every object has a sidecar (.meta.json on filesystem, object metadata on S3/MinIO):

Terminal window
factflow storage metadata KEY

Returns the full metadata blob — provenance, config hash, lineage reference, adapter-specific extras.

Download execution output

Section titled “Download execution output”

Bulk download every artefact for an execution:

Terminal window
factflow storage download EXEC_ID --output ./out/

The execution id (or tag) is positional. --output/-o is the output directory (default ./); --format zip saves a single archive instead, in which case -o is the zip path. Preserves the storage key hierarchy. Useful for handoff to external tooling.

Watch live writes

Section titled “Watch live writes”

During a running execution, stream writes as they happen:

Terminal window
factflow storage watch EXEC_ID

The execution id (or tag) is positional. Uses Server-Sent Events (SSE) under the hood. Exits when the execution completes or Ctrl+C.

Filter to a specific stage:

Terminal window
factflow storage watch EXEC_ID --stage markdown_crawled

Watch several executions at once by repeating --execution-id for the additional ones.

Via the API

Section titled “Via the API”
  • GET /api/v1/storage/objects?prefix=... — list
  • GET /api/v1/storage/objects/{key} — read bytes
  • GET /api/v1/storage/objects/{key}/metadata — read sidecar
  • POST /api/v1/content/archive/prepare + GET /api/v1/content/archive/{job}/status — async bulk download (the download CLI uses this)

Retention

Section titled “Retention”

Storage is the replay source of truth. If you delete an object, any execution that depended on it can no longer replay the downstream stage.

Safe retention policy:

  • Never expire objects referenced by executions you might replay (bounded by how far back you replay — usually 30–90 days)
  • Bulk-prune with an execution-level delete flow (not yet automated in the CLI; manual rm or bucket lifecycle rules)

Configuration

Section titled “Configuration”

Set the storage URI via config / env:

FACTFLOW_STORAGE_URI=file://./output/storage     # filesystem
FACTFLOW_STORAGE_URI=minio://factflow-bucket      # MinIO / S3

See factflow-infra for the full settings shape.

Section titled “Related”