Troubleshooting and exit codes

When something fails, start here: the CLI tells you only whether the command itself worked, so read the run status from the output, then narrow it down with lineage and the dead-letter queue.

CLI exit codes

Every factflow command returns one of two codes:

Code	Meaning
0	Success
1	Any error — API failure, validation failure, bad flags, or a timed-out HTTP call

There are no other codes. The exit status reflects whether the command ran, not the state of the run it touched.

In particular, execution wait <id> exits 0 for any terminal status — including failed and cancelled. It returns 1 only if the HTTP call itself fails. To branch on run outcome in a script, parse the printed status field rather than $?:

status=$(factflow execution get <id> --output json | jq -r .status)
if [ "$status" != "completed" ]; then
  echo "run did not complete: $status"
  exit 1
fi

Error classification

When an adapter reports a failure, two independent flags decide what happens — there is no severity enum:

• retryable. When true, the message is re-enqueued for another attempt with exponential backoff — use it for transient faults: network blips, rate limits, broker hiccups. When false, the message is dead-lettered instead of retried.
• fatal (result metadata). Aborts the whole execution: the processor emits a fatal_error event and the orchestrator stops the run. Use it for unrecoverable conditions — bad credentials, a missing model — where retrying any message is pointless. fatal wins over retryable: redelivery cannot fix a bad credential.

Set retryable either as the field on AdapterResult or as metadata["retryable"]. Both reach the acknowledgement decision, and where they disagree the retry wins — dead-lettering something transient strands recoverable work, while retrying something permanent costs a bounded number of attempts and dead-letters anyway.

HTTP-driven adapters add a status-code layer on top: each classifies response codes internally — pdf_fetcher has a retryable-status set, web_crawler has _is_retryable_error — and sets the flag accordingly. There is no config knob for this; the classification lives in the adapter.

Retry is not a NACK

A transient failure is re-enqueued by republishing the message with a scheduled delivery time, not by negatively acknowledging it. On ActiveMQ Artemis a STOMP NACK means "remove this message" rather than "redeliver it", so NACKing a transient failure destroyed it. Retry, backoff and the attempt limit are therefore owned by Factflow, not delegated to broker redelivery policy. See Queues and messaging.

Dead-letter inspection

A message that is rejected as permanently failed, or that exhausts its attempt limit, is moved to a dead-letter destination named after the queue it came from:

Provider	Dead-letter destination
Artemis	<queue>.dlq — e.g. /queue/<execution_id>/scraping.tasks.dlq
RabbitMQ	the dlx.failed queue, bound to the dead-letter exchange
Pulsar	<topic>-dlq, held by a dead-letter-retention subscription

Nothing drains these. There is no consumer and no CLI command to browse them, so a dead letter waits until someone looks. Every one carries x-dead-letter-reason, x-dead-letter-source and x-dead-lettered-at, so it is diagnosable on its own.

To find out whether anything was dead-lettered, read the counters below rather than hunting through destinations. To trace the failing hop, use Lineage debugging — the lineage row records which adapter failed and why.

Durability counters

factflow system health --debug reports what became of failures, under debug.queue.durability and debug.queue.ack_outcomes. These answer "is this run healthy" without grepping the process log:

Counter	Meaning when non-zero
dead_lettered	Messages moved aside for a human. Expected to match the number of genuinely failed messages — more than that means healthy work was moved.
retries_scheduled	Transient failures re-enqueued with backoff. Normal under load.
retry_duplicates	A retry was queued but its original could not be settled, so both copies will be delivered. Preserving beats losing, but the duplicate is real.
settlements_lost	An acknowledgement did not reach the broker, so work may be repeated. This is the alarm — investigate it.
ack_outcomes.written	Acknowledgements that reached the connection. The healthy majority.
ack_outcomes.superseded	Settlements correctly withheld because the delivery had been reissued. Not a failure.
ack_outcomes.connection_changed	Acknowledgements condemned because the connection changed.

Counters are per process and cumulative since it started. On a multi-instance deployment each instance reports only its own, and on a long-lived one you want the change across a run rather than the absolute value.

debug.queue.redelivery reports dispatch rather than work: redeliveries_observed counts what the broker flagged as redelivered, which includes messages it merely redispatched from a consumer window. Treat it as shape, not as a fault. The work-level signal is redeliveries_reexecuted on /api/v1/system/metrics — non-zero there means the same message was processed more than once.

When a run hangs

A run that never completes is almost always a stuck route or an un-acked message. Check Monitoring & metrics for queue depth, then the lineage chain for the last successful hop.