Confluence ingest

Three adapters crawl Confluence and feed it into the shared markdown pipeline: confluence_space_crawler enumerates a discovery unit and fans out one message per page, confluence_page_fetcher stores each page's HTML body and fans out a descriptor per attachment, and confluence_attachment_fetcher downloads each attachment binary for the document_converter.

Pipeline shape

version: "1.0"

init_message:
  queue: "/queue/confluence.crawl"
  payload:
    metadata:
      job_type: "confluence_ingest"
      source: "confluence_pipeline"

routes:
  crawl_confluence:
    inbound:
      queue: "/queue/confluence.crawl"
      subscription: "confluence-crawlers"
      concurrency: 1            # convention: predictable cursor advance
      prefetch: 1
      message_timeout: 1800
    adapters:
      - type: "confluence_space_crawler"
        config:
          mode: "spaces"
          space_keys: ["ITJC"]
    outbound:
      - queue: "/queue/confluence.fetch_page"

  fetch_page:
    inbound:
      queue: "/queue/confluence.fetch_page"
      subscription: "confluence-page-fetchers"
      concurrency: 4
      prefetch: 4
      message_timeout: 300
    adapters:
      - type: "confluence_page_fetcher"
        config: {}
    outbound:
      - queue: "/queue/html.ready"                   # stored HTML body → markdown chain
      - queue: "/queue/confluence.fetch_attachment"  # attachment descriptors → attachment fetcher

  fetch_attachment:
    inbound:
      queue: "/queue/confluence.fetch_attachment"
      subscription: "confluence-attachment-fetchers"
      concurrency: 4
      prefetch: 4
      message_timeout: 300
    adapters:
      - type: "confluence_attachment_fetcher"
        config: {}
    outbound:
      - queue: "/queue/confluence.document.convert"

  convert_document:
    inbound:
      queue: "/queue/confluence.document.convert"
      subscription: "confluence-document-converters"
      concurrency: 5
      prefetch: 10
      message_timeout: 300
    adapters:
      - type: "document_converter"
        config: {}
    outbound:
      - queue: "/queue/markdown.ready"
        transform:
          object_key: "object_key"
          document_id: "document_id"
          title: "title"

Both fetcher routes ship with config: {} — the defaults route the page body to /queue/html.ready and attachment descriptors to /queue/confluence.fetch_attachment. The crawler and page fetcher set each fan-out message's destination directly, so the engine routes by destination, not by outbound when-conditions.

Two configs ship in backend/config/pipelines/confluence/, differing only in where the stream stops:

confluence-ingest.yaml — the four routes above. Queues are execution-scoped, so with no route consuming /queue/html.ready or /queue/markdown.ready, page HTML and attachment markdown land in storage and stop. Use it for storage-only ingest, or as the front of a larger composed config.
confluence-markdown.yaml runs the knowledge path: crawler, space lister and page fetcher plus an inlined html_to_markdown route that writes the markdown_converted storage stage. Page bodies only, no attachments, and nothing downstream of markdown. Because queues are execution-scoped, the conversion stage must be inlined here rather than deployed separately.

Discovery modes

The crawler's mode selects how it enumerates pages. It is required — there is no default.

spaces — crawl named spaces. Set space_keys: ["ITJC", "..."] (required for this mode). Lists each space's pages newest-first.
all_visible — crawl every global, current space. Optionally narrow with exclude_space_keys: ["..."].
subtree — crawl a page and its descendants. Set space_key and root_page_id (both required); include_root defaults to true.

adapters:
  - type: "confluence_space_crawler"
    config:
      mode: "subtree"
      space_key: "ITJC"
      root_page_id: "123456"

Authentication

Auth is a pluggable provider selected by FACTFLOW_CONFLUENCE_AUTH_KIND. All env vars use the FACTFLOW_CONFLUENCE_ prefix and live in the package's gitignored .env.

api_token (default) — tenant-direct HTTP Basic auth with a classic or scoped Atlassian API token:

FACTFLOW_CONFLUENCE_AUTH_KIND=api_token
FACTFLOW_CONFLUENCE_BASE_URL=https://<tenant>.atlassian.net/wiki
FACTFLOW_CONFLUENCE_EMAIL=...
FACTFLOW_CONFLUENCE_API_TOKEN=...

oauth — OAuth 2.0 client-credentials (2LO) bearer against the Atlassian gateway. Tokens are minted and refreshed automatically:

FACTFLOW_CONFLUENCE_AUTH_KIND=oauth
FACTFLOW_CONFLUENCE_CLIENT_ID=...
FACTFLOW_CONFLUENCE_CLIENT_SECRET=...
FACTFLOW_CONFLUENCE_CLOUD_ID=...   # optional; auto-resolved when credentials grant exactly one site

FACTFLOW_CONFLUENCE_CLOUD_ID is resolved automatically when the credentials grant access to exactly one site; set it explicitly when they grant more than one. OAuth needs all five scopes — read:space:confluence, read:page:confluence, read:attachment:confluence, read:hierarchical-content:confluence, and the classic read:confluence-content.all. Do not strip the classic scope: attachment binary download depends on it.

Attachments

The page fetcher lists a page's attachments and emits one descriptor each; the attachment fetcher downloads, stores, and forwards the binary to document_converter.

v1 download — binaries download via the v1 endpoint /rest/api/content/{page_id}/child/attachment/{attachment_id}/download. The v2 _links.download servlet returns 401 for OAuth tokens, so v1 is the only working path (hence the classic scope above).
Size cap — max_attachment_bytes defaults to 50 MB. Size is checked twice: the declared size before download, and the streamed length during download. Oversized attachments are skipped — not stored, not emitted.
Conversion — the attachment filename (extension included) is preserved in the storage key so document_converter can infer the source format. Formats MarkItDown supports are converted; others are stored but skipped at conversion.

A missing page or attachment (404) is treated as deleted-between-crawl-and-fetch and skipped, not an error.

Incremental crawl

Each discovery unit keeps a high-water mark in the Postgres confluence_cursors table, so re-runs only emit pages modified since the last run. The anchor is each page's version.createdAt.

spaces and all_visible modes list newest-first, so the crawler stops at the first page older than the cursor.
subtree mode has no guaranteed sort, so it filters each page against the cursor instead of stopping early.

The cursor key is space:<space_key> for spaces / all_visible and subtree:<space_key>:<root_page_id> for subtree. The crawl route runs at concurrency: 1 by convention for predictable cursor advance; the UPSERT stays correct at higher values.

Page bodies and attachments get a stable document_id (confluence-<space>-<page> and confluence-<space>-<page>-<attachment>) carried through to the markdown sidecar, so replays map back to the same vectors.

Typical downstream

Page bodies (via html_to_markdown) and attachments (via convert_document) both converge on /queue/markdown.ready. From there the flow feeds the markdown workflow for segmentation, then embeddings, then knowledge. confluence-markdown.yaml inlines only the markdown conversion and stops at the markdown_converted storage stage, which is where the knowledge pipeline reads from.