Skip to content
Factflow

SharePoint ingest

Three adapters pull documents from SharePoint and convert them to markdown: sharepoint_enumerator lists files across sites, sharepoint_fetcher downloads each, and document_converter turns Office formats into markdown.

Pipeline shape

Section titled “Pipeline shape”
version: "1.0"


routes:
  enumerate:
    inbound:
      queue: "/queue/sharepoint.enumerate"
      subscription: "sp-enumerators"
    adapters:
      - type: "sharepoint_enumerator"
        config:
          sites:
            - "dnbasa.sharepoint.com,/sites/allcompany"
          file_types: [".docx", ".pdf", ".pptx"]


  fetch:
    inbound:
      queue: "/queue/sharepoint.fetch"
      subscription: "sp-fetchers"
    adapters:
      - type: "sharepoint_fetcher"
        config:
          max_file_size_mb: 500
      - type: "document_converter"


init_message:
  route: "enumerate"
  payload:
    full_sync: true

sites are "hostname,/sites/name" identifiers; file_types defaults to the converter's supported set.

Authentication

Section titled “Authentication”

Microsoft Graph via app registration (app-only client credentials). Env (prefix FACTFLOW_SHAREPOINT_):

FACTFLOW_SHAREPOINT_TENANT_ID=...
FACTFLOW_SHAREPOINT_CLIENT_ID=...
FACTFLOW_SHAREPOINT_CLIENT_SECRET=...

Required Graph permissions: Sites.Read.All, Files.Read.All (application, not delegated).

Local dev: pre-acquired delegated token

Section titled “Local dev: pre-acquired delegated token”

When a Conditional Access policy blocks app-only token issuance for the service principal (e.g. AADSTS53003), you can run a local crawl with a delegated token minted as yourself. Set it and the client-credentials path is skipped entirely:

FACTFLOW_SHAREPOINT_ACCESS_TOKEN=<bearer token>

The token must carry the Sites.Read.All + Files.Read.All delegated Graph scopes. An Azure CLI token (az account get-access-token) usually will not — its scopes are whatever the CLI app was consented for, which in many tenants excludes SharePoint, so /drives returns empty and the crawl enumerates nothing. Mint one from a dedicated public-client app registration instead.

1. Register a public-client app (one-time)

Section titled “1. Register a public-client app (one-time)”

Entra ID → App registrations → New registration:

  • Supported account types: single tenant.
  • Authentication → Add a platform → Mobile and desktop applications (public client, no secret), redirect URI http://localhost. Do not put the redirect under the Web platform — that is a confidential client and needs a secret.
  • Authentication → Allow public client flows: Yes.
  • API permissions → Microsoft Graph → Delegated: Sites.Read.All, Files.Read.All, then Grant admin consent (admin-only scopes). If you are not an admin, have one open https://login.microsoftonline.com/<TENANT_ID>/adminconsent?client_id=<CLIENT_ID>.

Note the Application (client) ID and Directory (tenant) ID.

2. Mint → export → restart → run

Section titled “2. Mint → export → restart → run”
Terminal window
cd backend
uv run python scripts/uv/get_delegated_token.py \
  --client-id <CLIENT_ID> --tenant-id <TENANT_ID>

This opens the system browser; on a managed device it satisfies device-compliance Conditional Access (the same path az login uses). The script prints the granted scopes and a ready-to-paste export FACTFLOW_SHAREPOINT_ACCESS_TOKEN=... line. Paste it into the shell that runs the server, restart the server so it re-reads the env, then run the sharepoint-ingest-test.yaml single-site pipeline to smoke-test before the full ingest.

Caveats — this is a development escape hatch, not the production auth story:

  • Bounded to your access. Delegated = the app's granted scopes ∩ your own permissions; the crawl sees only sites and files you can already open. It grants nothing you do not already have.
  • Short-lived (~1 h) and not refreshed. If a long crawl outlives the token, re-mint it and restart.
  • It is a bearer secret — keep it in the shell env / .env (gitignored), never commit it. Pasting the export …= line records the live token in your shell history; prefix the paste with a space (zsh setopt HIST_IGNORE_SPACE / bash HISTCONTROL=ignorespace) or source it from a gitignored .env instead.

Why separate fetch from conversion

Section titled “Why separate fetch from conversion”

The binary document is persisted between fetch and conversion, so you can re-run conversion — e.g. after upgrading the converter library — against the same source without re-hitting Graph.

Converter coverage

Section titled “Converter coverage”

document_converter converts the extensions in its supported_formats set (.docx, .pdf, .pptx, and similar) to markdown; anything outside the set is logged and skipped. Output suffix and size / timeout limits are configurable via output_suffix, max_content_size_mb, and conversion_timeout_seconds.

Typical downstream

Section titled “Typical downstream”

Feeds the markdown workflow for segmentation, then embeddings, then knowledge.

Section titled “Related”