Queue diagnostics

When a pipeline seems stuck — messages accepted but no progress — the queue is the first place to look. This guide walks through the diagnostic tools.

Health check

Fast path — does the broker even respond?

factflow system health --debug

Look for the queue component. Status healthy means connection is live.

Broker connectivity diagnostic

For debugging a connection to a specific broker — typically AWS AmazonMQ, where the failure is in DNS, TLS or the STOMP handshake rather than in Factflow. It opens a one-shot connection with the credentials you pass and returns a step-by-step timeline. It does not use the configured queue provider and does not sweep the configured queues.

host, username and password are required; the port defaults to 61614 (AWS MQ STOMP+SSL) and SSL is on by default. API-only — there is no CLI command:

curl -X POST http://localhost:8000/api/v1/system/mq-diagnostic \
  -H 'Content-Type: application/json' \
  -d '{"host": "b-xxxxx.mq.eu-north-1.amazonaws.com", "username": "USER", "password": "PASS"}'

The response carries steps — one entry per phase (dns, tcp, tls, stomp) with elapsed milliseconds — plus the negotiated STOMP version, heartbeat and TLS details. A failure names the phase it stopped in and includes the broker's own ERROR frame body when there is one, which is usually the fastest way to tell an auth rejection from a network problem.

Add "test_pubsub": true to publish and read back real messages over that connection, with message_count, message_delay_ms and throttle_rps to shape the load. That measures the transport; it does not exercise the engine, so it cannot tell you anything about redelivery or duplicate work — see Verifying delivery at volume for that.

Verifying delivery at volume

The counters above tell you whether delivery is healthy now. To prove it stays healthy under load — the failure mode behind issue #425, where ~36,000 documents produced ~1.5M broker deliveries and only 5.3% of the work settled — run scenario s14 of the test-cli harness.

s14 pushes synthetic messages through the real engine (real processor, real lineage, real broker) using the synthetic_fanout and synthetic_sink adapters. No network, no LLM, no storage and no source data, so a lost or duplicated message can only have come from message delivery.

It runs against the durable compose broker, not an ephemeral test container, because the guarantees depend on the committed redelivery and dead-letter policy in backend/config/artemis/broker.xml:

docker compose up -d artemis postgresdb

# FACTFLOW_ENV=embedded is required: the server skips database initialisation when the
# host is localhost and the database is named factflow, which is exactly the compose setup.
cd backend && FACTFLOW_ENV=embedded FACTFLOW_DATABASE_HOST=localhost \
  uv run python -m factflow_server serve --port 8000

# Smoke run — 100 messages, about a minute
STORM_WAVES=10 STORM_WAVE_SIZE=10 AUTO=1 ./scripts/test-cli/run.sh s14

# Full scale — 36,000 messages, matching the incident
AUTO=1 ./scripts/test-cli/run.sh s14

Twelve steps. The four that decide it:

Step	Asserts	Incident measured
s14.6	every message reached a terminal state and none was abandoned	—
s14.7	settle rate ~100%	5.3%
s14.8	execution amplification ~1.0x — no work ran twice	2.98x
s14.9	delivery amplification bounded	42.7x

s14.10 is the one to read when something is wrong: it bounds acknowledgements that never reached the broker. Artemis replenishes a consumer's flow-control window on acknowledgement, so a consumer whose acks are discarded is granted one window and then stops consuming permanently — with every other queue counter still reading clean. s14.12 freezes the broker mid-flight with docker pause and requires that nothing is lost across the thaw.

This is a local gate. Scenarios at this scale take minutes and are deliberately excluded from CI.

Provider-specific issues

Artemis (STOMP) — header timestamps

Artemis rejects STOMP frames with malformed timestamps silently (no error, message just disappears). Factflow's StompNamingStrategy formats timestamps correctly; custom subscribers must match.

Format: strict ISO-8601 with timezone, e.g. 2026-04-22T18:30:00.123456+00:00.

RabbitMQ (AMQP) — name rules

AMQP queue names must match [a-zA-Z0-9_\-.:]+. Special characters cause the broker to reject publishes. AmqpNamingStrategy validates before publish.

Pulsar — topic topology

Pulsar has different semantics around ordering and subscription types. PulsarNamingStrategy handles the topology mapping. If you hit Pulsar-specific issues, the factflow-infra reference has the provider's quirks.

Execution isolation

Every queue name is wrapped by ExecutionScopedQueue — prefixed with the execution id. A message published for execution A under route web_scraper is NOT visible to execution B's web_scraper processor, even though they share the route name in config.

If you expect execution B to pick up something and it doesn't: verify the executions aren't using the same broker instance but different execution ids (which is correct — that's what isolation does).

Stuck message — common diagnoses

1. Processor isn't subscribed

factflow execution routes EXEC_ID

Every route should show a processor. If a route is missing, the orchestrator failed to start that processor. Check factflow execution get EXEC_ID for errors.

2. Backpressure applied

factflow system metrics shows in_flight count. If in_flight == concurrency for the affected route, the processor is saturated. Options: raise concurrency in YAML, find the slow adapter, reduce upstream rate.

3. Circuit breaker open

The engine's per-adapter circuit breaker may have opened after repeated failures. factflow lineage failures --execution-id EXEC_ID --route-id ROUTE_ID shows the pattern (or --stage-name STAGE to narrow by adapter stage).

4. Route is globally paused

factflow pipeline list

Shows paused state per route. Unstick with:

factflow pipeline resume ROUTE_ID

5. Broker queue depth

Direct broker inspection tools (Artemis web console, RabbitMQ management UI) show queue depth. If messages are piling up without being consumed, see #1 and #2.

6. It failed and was moved aside

A message that failed permanently is no longer on its queue — it was moved to a dead-letter destination, and nothing consumes those. Read debug.queue.durability from factflow system health --debug to find out whether that happened, before searching for the message itself.

The counter that matters most is settlements_lost: non-zero means an acknowledgement did not reach the broker, so the work may have been repeated. See Troubleshooting → Durability counters for the full table and Dead-letter inspection for where each provider puts them.

Related

• Concept: Queue isolation — how scoped queues work
• factflow-infra reference — provider details
• Running pipelines — pause/resume routes
• Troubleshooting — error classification, dead letters, durability counters