Rate-limit strategies

The web_scraper adapter's rate_limit field throttles outbound requests per host. It takes three shapes, from terse to fully explicit. The web_crawler and pdf_fetcher adapters also have a rate_limit, but theirs is a named preset string only — they don't accept a numeric rate or an inline config object. Examples live in backend/config/rate_limiting_examples.yaml.

In a real pipeline rate_limit is a field inside the adapter's config: block (see Web ingest), not a top-level webscraper: wrapper — that wrapper is only the shorthand the standalone examples file uses to show the field in isolation.

1. Named strategy (string)

webscraper:
  rate_limit: "dnb"          # MODERATE-tuned preset for DNB.no, with bot detection

Name	Behaviour
conservative	2 req/s, adaptive within [0.5, 2.0]
moderate	5 req/s, adaptive within [1.0, 10.0]
aggressive	20 req/s, adaptive within [5.0, 50.0]
dnb	6 req/s, adaptive within [1.0, 10.0]; treats HTTP 403 as bot detection (DNB.no)

conservative, moderate, aggressive, plus adaptive and none, are the RateLimitStrategy enum values. dnb is a named preset registered in RateLimitStrategyRegistry — it builds on the MODERATE strategy, not a separate enum member.

2. Numeric rate (non-adaptive)

webscraper:
  rate_limit: 10.0           # 10 requests per second, fixed

3. Full configuration (object)

webscraper:
  rate_limit:
    enabled: true
    requests_per_second: 5.0
    burst_capacity: 10
    adaptive: true
    min_rate: 1.0
    max_rate: 20.0
    success_threshold: 50    # successful requests before increasing the rate
    increase_factor: 1.1     # +10% when consistently succeeding
    decrease_factor: 0.5     # −50% on a 429 response

You can also start from a named base and override individual fields:

webscraper:
  rate_limit:
    strategy: "moderate"     # moderate base…
    max_rate: 15.0           # …with a higher ceiling
    decrease_factor: 0.3     # …and gentler backoff

Adaptive behaviour

When adaptive: true, the limiter walks the rate between min_rate and max_rate: after success_threshold consecutive successes it multiplies by increase_factor; on a 429 it multiplies by decrease_factor. This is the AIMD-style controller the Web ingest workflow uses to stay under a host's tolerance without manual tuning.