Running Scribe → Avalon

Avalon content is produced by two pipelines run in sequence, not one:

1. Scribe (scribe/scribe-knowledge-synthesis.yaml) — reads source markdown from storage and writes a concept_map (taxonomy-guided concepts, each tagged with a source identity).
2. Avalon consolidation (scribe/avalon-consolidation.yaml) — reads the concept_map and synthesizes avalon_pages (one consolidated page per concept).

Scribe is source-driven; consolidation is concept-map-driven and runs after scribe. Both make LLM calls. For the adapters themselves, see Knowledge & Avalon synthesis; this page is how to run the flow.

Prerequisite: the LLM profile must exist

Every LLM adapter in both pipelines names an llm_profile. If that profile is not configured on the server, the adapter fails at runtime and the pipeline silently produces nothing — concept detection drops every message, avalon_pages stays empty, yet the execution still reports a terminal status. This is the single most common reason an Avalon run yields zero pages.

Check the live profiles first and make the config match:

factflow system llm-profiles --type chat

The shipped configs use openrouter-gemini-flash-chat. If the model profiles were renamed and a config still points at an old name, every llm_profile: reference fails with Provider profile '<name>' not found in the server log. The config validator does not check profile validity, so this passes validation and only surfaces at runtime — always confirm against system llm-profiles.

Stage 1 — Scribe: markdown → concept map

Three routes (scribe-knowledge-synthesis.yaml):

scribe_enumerate          storage_enumerator (*.md, recursive)              → /queue/scribe.diff
scribe_knowledge_diff     storage_retriever → pii_redactor →
                          sensitivity_classifier → anti_bac_gate →
                          sc_redactor → knowledge_diff → scribe_diff_writer → /queue/scribe.detect
scribe_concept_detection  concept_detection → concept_map_writer            → concept_map (DB)

The init message names which storage prefixes to enumerate:

init_message:
  queue: "/queue/scribe.enumerate"
  payload:
    source_directories:
      - "boost/routines"

Run it, overriding source_directories for the corpus you want:

CID=$(factflow config create -f backend/config/pipelines/scribe/scribe-knowledge-synthesis.yaml -o json | jq -r .id)
EID=$(factflow config run "$CID" \
  --payload '{"source_directories": ["<prefix>/markdown_converted"]}' \
  --tag scribe-run -o json | jq -r .id)
factflow execution wait "$EID" --timeout 30m

concept_detection tags each concept with a source identity from source_prefixes (SBANKEN: "sbanken-"), falling back to default_source_prefix (dnb-). That identity is what consolidation uses to decide a concept is multi-source.

Stage 2 — Avalon consolidation: concept map → pages

Consolidation reads the concept_map table directly — its init payload is empty (no source directories):

avalon_consolidation_trigger  concept_fanout (min_sources: 2, relevance gates)  → /queue/avalon.consolidate.synthesize
avalon_batch_synthesizer      concept_source_loader → batch_synthesizer (LLM)   → /queue/avalon.consolidate.validate
avalon_structural_validator   structural_validator (citation/length/diversity)  → /queue/avalon.consolidate.write
avalon_writer                 avalon_writer → concept_map_updater               → avalon_pages (DB)

ACID=$(factflow config create -f backend/config/pipelines/scribe/avalon-consolidation.yaml -o json | jq -r .id)
AEID=$(factflow config run "$ACID" --tag avalon-consolidate -o json | jq -r .id)
factflow execution wait "$AEID" --timeout 30m

min_sources: 2 is the key gate — a concept is consolidated into an Avalon page only if it was detected in at least two sources. Single-source concepts never become pages, so Avalon coverage depends on running scribe across multiple source corpora first.

Multi-source

Every source ingest writes markdown to storage; scribe enumerates whichever prefixes you list in source_directories. List one or many:

source_directories:
  - "<web-exec-id>/markdown_converted"     # dnb.no / sbanken.no — execution-scoped
  - "<sharepoint-exec-id>/markdown"        # SharePoint documents → markdown
  - "<confluence-exec-id>/..."             # Confluence pages + attachments
  - "boost/routines"                       # Boost routines — fixed prefix

Source	Markdown produced by	Storage prefix	Identity
Web (dnb.no, sbanken.no)	Web ingest	<exec-id>/markdown_converted	dnb- / sbanken-
SharePoint	SharePoint ingest	execution-scoped converter output	dnb-
Boost routines	Boost workflow	boost/routines (fixed)	per package
Confluence	Confluence ingest	execution-scoped (via the markdown chain)	dnb-

Most sources write under their execution id (<exec-id>/markdown_converted); Boost routines use a fixed boost/routines prefix. When sources disagree on a fact, batch_synthesizer.source_precedence (dnb.no → sbanken.no → sharepoint → boost:*) decides which one wins.

Replay — reusing an existing ingest

You rarely re-ingest to regenerate Avalon. Instead you replay an existing markdown corpus into scribe. There are two mechanisms, and which one applies depends on whether the source execution still exists in the database:

A. The ingest execution is in the DB → execution replay

Resolves routes from the parent execution's frozen config_snapshot and re-emits its stored stage output into scribe. This is the cross-pipeline handoff (e.g. Boost → Scribe):

factflow execution replay <ingest-exec-id> \
  --from-stage markdown_converted \
  --to-route scribe_enumerate \
  --config-id <scribe-config-id>

See Replay for the mechanics and constraints (it reads from the parent's snapshot, not the live config directory).

B. Only the stored corpus survives → storage-driven run

If the ingest ran long ago and its DB record is gone (a fresh embedded database, a wiped volume), the stored markdown is still the source of truth. Point scribe's source_directories straight at the stored prefix — no execution record is needed, because storage_enumerator reads the object store directly:

factflow config run "$CID" \
  --payload '{"source_directories": ["<old-exec-id>/markdown_converted"]}' \
  --tag avalon-replay -o json

The stored markdown carries its own provenance sidecars (origin, source_ref, document_id), so concepts stay correctly attributed even though the original run no longer exists in the database. This is the path to use when you have a 2.3G storage tree but a freshly reset database.

Verify the output

psql "$DSN" -c "SELECT COUNT(*) FROM concept_map;"   # after stage 1
psql "$DSN" -c "SELECT COUNT(*) FROM avalon_pages;"  # after stage 2

# or a dry-run export — writes nothing, reports the diff against the corpus baseline
cd backend && uv run python -m factflow_knowledge.export --dry-run --work-dir /tmp/avalon-out --db-url "$DSN"

• concept_map is 0 after stage 1 → re-check the LLM profile.
• avalon_pages is 0 but concept_map is populated → too few multi-source concepts (the min_sources: 2 gate); add more source corpora.

Gotchas

• LLM profile mismatch → silent zero output. Always verify against system llm-profiles.
• min_sources: 2 → a single-source corpus produces no pages; consolidation needs concept overlap across sources.
• Classifier / anti_bac_gate schema validation errors — if the model returns output that does not fit the expected structured schema, those documents are dropped at the hygiene stage, lowering coverage without failing the run. Harden the model profile or the output contract.

Related

• Knowledge & Avalon synthesis — the adapters in detail
• Replay — replay mechanics and constraints
• Web ingest · SharePoint · Boost · Confluence — source pipelines
• factflow-knowledge reference