Web ingest (scrape & crawl)

The webscraper workflow ingests content from websites that publish a sitemap. It ships two fetch engines that share the same discovery front-end and storage tail:

web_scraper — HTTP-only. Fetches the raw HTML over httpx, fast and cheap, with adaptive per-host rate limiting. Use it for server-rendered sites.
web_crawler — browser-based. Drives Chromium through crawl4ai/Playwright, renders JavaScript, and outputs markdown directly. Use it for client-rendered pages an HTTP fetch can't see.

Both sit behind the same sitemap_parser → url_expander discovery chain and write through web_content_storage.

Which engine when

Signal	Engine
Server-rendered HTML, content present in the initial response	`web_scraper`
`web_scraper` returns an empty SPA skeleton	`web_crawler`
Content appears only after JS runs	`web_crawler`
Site blocks non-browser user agents	`web_crawler`
High volume, latency-sensitive	`web_scraper` (one browser tab per crawl is expensive)

web_scraper is the default path. Reach for web_crawler only when an HTTP fetch comes back too thin — one web_crawler instance ≈ one Chromium tab, so keep its concurrency low (2–4 per CPU core). A cheap-path-first split (HTTP first, route only the thin results into a browser route via a condition) beats running every URL through the browser.

Discovery and storage chain

Type	Purpose
`sitemap_parser`	Fetch a sitemap, extract URLs; supports sitemap indexes (`max_urls` caps the list)
`url_expander`	Fan out: one incoming URL list → one outgoing message per URL (no config)
`web_scraper`	HTTP fetch one URL → HTML + metadata
`web_crawler`	Browser fetch one URL → markdown + metadata; emits discovered PDF links
`pdf_fetcher`	Fetch a PDF link discovered by `web_crawler`
`web_content_storage`	Persist content + metadata to storage

See the adapter catalog for each adapter's full config shape.

HTTP scraper pipeline

version: "1.0"

routes:
  sitemap_scraper:
    name: "Sitemap Web Scraper"
    inbound:
      queue: "/queue/webscraper.sitemap"
      subscription: "sitemap-processors"
      concurrency: 5
      prefetch: 10

    adapters:
      - type: "sitemap_parser"
        config:
          max_urls: 1000

      - type: "url_expander"

      - type: "web_scraper"
        config:
          follow_redirects: true
          http_timeout: 30.0
          rate_limit: "moderate"

      - type: "web_content_storage"

init_message:
  route: "sitemap_scraper"
  payload:
    sitemap_url: "https://example.com/sitemap.xml"

web_scraper returns HTML, so it feeds the markdown workflow: HTML → markdown → segments → embeddings.

Resilience

web_scraper wraps each host in a circuit breaker (circuit_failure_threshold, circuit_timeout_seconds, circuit_success_threshold) and bounds concurrent requests per instance with max_in_flight_requests (default 5). Failed fetches retry up to max_retries (default 3).

Browser crawler pipeline

web_crawler renders JavaScript through Chromium and emits markdown directly — crawl4ai does the HTML→Markdown conversion in-process, so the markdown workflow's HTML converter is not needed for this content. It also extracts PDF links from the rendered page and fans them out to a pdf_processor route via emit_messages.

version: "1.0"

routes:
  crawler:
    inbound:
      queue: "/queue/webcrawler.urls"
      subscription: "crawler-processors"
      concurrency: 2   # browser is heavy

    adapters:
      - type: "web_crawler"
        config:
          headless: true
          browser_type: "chromium"
          excluded_tags: ["nav", "header", "footer", "aside", "form", "script", "style"]
          word_count_threshold: 10
          page_timeout: 30.0

      - type: "web_content_storage"

init_message:
  route: "crawler"
  payload:
    url: "https://example.com/spa-page"

Key `web_crawler` config

Field	Default	Purpose
`headless`	`true`	Run Chromium without a visible window
`browser_type`	`chromium`	Browser engine (`chromium`/`firefox`/`webkit`)
`excluded_tags`	`nav header footer aside form script style noscript`	Tags stripped before extraction
`excluded_selectors`	`.skip-link .breadcrumb [class='navigation'] [class='cookie'] [class*='banner']`	CSS selectors stripped before extraction
`word_count_threshold`	`10`	Minimum words for a block to be kept
`page_timeout`	`30.0`	Page-load timeout (seconds)
`navigation_timeout`	`60.0`	Navigation timeout (seconds)
`cache_mode`	`bypass`	Crawl4AI cache mode (`enabled` skips re-fetching cached pages)
`text_only`	`false`	Skip images/CSS/fonts for faster content-only extraction
`skip_unchanged`	`false`	Skip URLs whose `source_modified_at` is unchanged (requires storage)
`use_service`	`true`	Use a shared Crawl4AI service instead of an in-process browser
`service_url`	—	Service URL; defaults to `FACTFLOW_CRAWL4AI_SERVICE_URL` or `http://crawl4ai:11235`
`api_token`	—	Operator token for the service; defaults to `CRAWL4AI_API_TOKEN`. Required when `use_service` is `true`

The full set lives in WebCrawlerConfig — see the package reference.

The service is only reachable when a token is configured. The unclecode/crawl4ai image binds 127.0.0.1 unless CRAWL4AI_API_TOKEN is set on the server, which makes its own health check pass while refusing every connection from another container. Set the same value on both sides — the adapter raises at construction when it is missing, rather than failing each message with an opaque connection error. docker-compose defaults both to factflow-local-token; deployed environments generate one per environment.

Dependencies and footprint

crawl4ai requires Chromium:

Embedded mode (use_service: false) auto-installs Chromium; expect ~500MB disk for browser caches and ~500MB memory per concurrent browser.
In production, run the unclecode/crawl4ai image, keep use_service: true, and point service_url (or FACTFLOW_CRAWL4AI_SERVICE_URL) at it. The service pools browsers so the route doesn't carry the per-instance memory cost.

Rate limiting

Both engines throttle per host. web_scraper's rate_limit defaults to "moderate" and accepts a named strategy, a numeric req/s, or a full adaptive config object; web_crawler's rate_limit defaults to the "default" preset. The strategies, the AIMD adaptive controller, and the full config shape are documented in rate-limit strategies. Register custom strategies through RateLimitStrategyRegistry.