Skip to content
Factflow

factflow-confluence

Confluence Cloud source workflow. Three auto-discovered adapters crawl the Confluence hierarchy — space(s) → page(s) → attachment(s) — and feed page bodies plus attachment binaries into the shared markdown / embedding pipeline. Discovery, listing, and metadata use the Confluence Cloud v2 REST API; the attachment binary download uses the v1 endpoint (v2 has no working OAuth binary-download path).

Tier and role

Section titled “Tier and role”

Source pipeline producing stored page HTML and attachment binaries from a Confluence Cloud site. Page bodies flow to the markdown chain; attachments flow through the shared document_converter. Both converge on /queue/markdown.ready, where segmentation / embedding / knowledge extraction run downstream.

Context

Section titled “Context”

One subpackage, confluence/:

  • adapter.py — the three pipeline adapters and a shared _ConfluenceClientMixin (the mixin deliberately does not inherit PipelineAdapter, so it is never registered):
    • ConfluenceSpaceCrawlerAdapter (type: confluence_space_crawler) — enumerates a discovery unit (spaces / subtree / all_visible) and fans out one message per page. Pure fan-out terminal (continue_pipeline=False); each emitted message carries an explicit destination so the engine routes by destination, not outbound when-conditions. The only adapter that takes db_pool.
    • ConfluencePageFetcherAdapter (type: confluence_page_fetcher) — fetches a page's storage-format body, stores it at <storage_prefix>/<space_key>/<page_id>.html, then fans out one HTML body message (to the markdown chain) plus one descriptor per attachment (to the attachment fetcher). Pure fan-out.
    • ConfluenceAttachmentFetcherAdapter (type: confluence_attachment_fetcher) — downloads one attachment binary, stores it, emits one document-convert message. The only one of the three that continues the pipeline (1:1 transform, continue_pipeline=True).
  • client.py — ConfluenceClient, an async httpx wrapper (HTTP/2, follow-redirects) with _RetryPolicy, Link-header / cursor pagination helpers, and the streaming download that raises AttachmentTooLargeError once the body exceeds the cap.
  • auth.py — the ConfluenceAuthProvider Protocol and its two concrete providers (ApiTokenBasicAuthProvider, OAuthAuthProvider), plus URL / timeout constants. The provider owns both the auth header and the API base URL, so switching flows is a contained provider swap.
  • cursor.py — ConfluenceCursor (frozen dataclass) and ConfluenceCursorStore, which persists per-discovery-unit cursors in the Postgres confluence_cursors table for incremental crawl.
  • models.py — the Pydantic adapter configs: ConfluenceCrawlerConfig, ConfluencePageFetcherConfig, ConfluenceAttachmentFetcherConfig.
  • settings.py — ConfluenceSettings, env-driven BaseSettings (prefix FACTFLOW_CONFLUENCE_).

Rationale

Section titled “Rationale”
  • v2 for discovery, v1 for binaries. The v2 surface drives space discovery, page listing, body fetch, and attachment metadata. The v2 _links.download servlet returns 401 for OAuth tokens, so the attachment binary is fetched via the v1 endpoint /rest/api/content/{pageId}/child/attachment/{attId}/download — the only working path (requires the classic read:confluence-content.all scope; do not strip it).
  • Pluggable auth behind one Protocol. ConfluenceAuthProvider abstracts both the header and the base URL. api_token (default) does tenant-direct HTTP Basic auth; oauth does 2LO client-credentials bearer against the Atlassian gateway. Moving from a classic API token to a scoped token to OAuth is a provider swap, not a rewrite.
  • Incremental by cursor, not full re-crawl. Each discovery unit keeps a cursor (space:<key> or subtree:<key>:<root>) anchored on version.createdAt. Newest-first spaces listing breaks at the first stale page; unordered subtree /descendants filters instead. The single UPSERT stays correct under concurrency, so the concurrency 1 on the crawl route is a convention for predictable cursor advance, not a correctness requirement.
  • Stable document_id for replay-safe vectors. Fetchers mint confluence-<space>-<page> (bodies) and confluence-<space>-<page>-<attachment> (attachments), persist it in the storage sidecar, and propagate it on the message so the markdown converter honors it instead of minting a random id. Replays map to the same vectors.
  • Converter stage is shared and separate. Attachments are stored as binaries with the original (sanitized) extension, then converted by the shared document_converter, so conversion can replay independently and only MarkItDown-supported formats are converted.

Public API

Section titled “Public API”

The top-level factflow_confluence/__init__.py re-exports the auth layer and its factory. __all__:

from factflow_confluence import (
    ApiTokenBasicAuthProvider,
    ConfluenceAuthProvider,   # Protocol
    ConfluenceSettings,
    OAuthAuthProvider,
    create_auth_provider,
)
  • create_auth_provider(settings: ConfluenceSettings) -> ConfluenceAuthProvider — module-level factory defined in __init__.py. Selects by settings.auth_kind (api_token → ApiTokenBasicAuthProvider; oauth → OAuthAuthProvider; anything else → ValueError). Mirrors the canonical factflow factory idiom (create_storage_provider, create_completion_client).

The three adapter classes are not re-exported at the package root — they live in factflow_confluence.confluence.adapter, are auto-discovered by factflow_engine.discovery, and are referenced from pipeline YAML by their type names (confluence_space_crawler, confluence_page_fetcher, confluence_attachment_fetcher), not imported. Likewise ConfluenceClient, the config models (ConfluenceCrawlerConfig, ConfluencePageFetcherConfig, ConfluenceAttachmentFetcherConfig), ConfluenceCursorStore, and ConfluenceCursor are internal and not re-exported.

Credentials come from env (gitignored .env), prefix FACTFLOW_CONFLUENCE_:

  • FACTFLOW_CONFLUENCE_AUTH_KIND — api_token (default) | oauth
  • api_token (all required): FACTFLOW_CONFLUENCE_BASE_URL (e.g. https://<tenant>.atlassian.net/wiki), FACTFLOW_CONFLUENCE_EMAIL, FACTFLOW_CONFLUENCE_API_TOKEN
  • oauth (CLIENT_ID + CLIENT_SECRET required): FACTFLOW_CONFLUENCE_CLIENT_ID, FACTFLOW_CONFLUENCE_CLIENT_SECRET, and optional FACTFLOW_CONFLUENCE_CLOUD_ID (auto-resolved from /oauth/token/accessible-resources when the grant has exactly one site; set it explicitly to disambiguate multiple sites)

OAuth needs all five scopes — read:space:confluence, read:page:confluence, read:attachment:confluence, read:hierarchical-content:confluence (subtree mode), and the classic read:confluence-content.all (attachment binary download). Dropping the classic scope breaks attachment download.

Dependencies

Section titled “Dependencies”
  • Workspace: factflow-protocols (AdapterResult, PipelineAdapter, PipelineContext, StorageProtocol), factflow-foundation, factflow-engine (auto-discovery + execution-scoped queues)
  • External (third-party): httpx[http2] >=0.28.1 — the only declared third-party runtime dep
  • External services: Confluence Cloud REST API (v2 for discovery / listing / metadata, v1 for attachment binaries), a storage provider for stored page HTML and attachment binaries, and Postgres for the confluence_cursors table (incremental crawl). The crawler is the only adapter injected with db_pool (AsyncConnectionPool), wrapped in ConfluenceCursorStore; the fetchers are injected with storage.

psycopg_pool / psycopg (cursor store) and pydantic / pydantic-settings (models, settings) are used at runtime as transitive deps, not declared directly in this package.

Testing

Section titled “Testing”

Tests at backend/packages/workflows/factflow-confluence/tests/. Integration tests that hit a live Confluence Cloud site require credentials in the package's gitignored .env and are skipped without them.

Section titled “Related”
  • factflow-markdown — downstream of the page-body path; confluence-ingest.yaml hands stored HTML to the markdown chain, confluence-markdown.yaml inlines it
  • factflow-sharepoint — sibling source workflow that shares the same document_converter stage for binary-to-markdown conversion
  • Build a Confluence pipeline — the two shipped configs (confluence-ingest.yaml stops at storage; confluence-markdown.yaml converts page bodies for the knowledge path)