Skip to content
Factflow

LLM and embedding providers

Every adapter that calls a language model talks to an LLMClientProtocol or EmbeddingClientProtocol. The concrete implementation is chosen at construction time by factflow-llm based on configuration. Adapter authors never import a specific provider — they declare a profile name and let the factory resolve it.

Why abstract?

Section titled “Why abstract?”

Three practical reasons:

  • Cost optimisation — point a profile at Claude Sonnet instead of Claude Opus via a config change, no redeploy
  • Vendor redundancy — if one provider is down, repoint the profile to another without touching adapter code
  • Testability — production code depends on a protocol; tests inject a mock satisfying the same protocol

Supported providers

Section titled “Supported providers”
provider_typeChatEmbeddingsClientRequired fields
openai✓✓OpenAIClientapi_key
azure✓✓AzureOpenAIClientapi_key, api_base, deployment_name, api_version
anthropic✓—AnthropicClientapi_key
bedrock—✓BedrockEmbeddingClient (Cohere Embed via Bedrock)AWS creds + region
huggingface—✓HuggingFaceEmbeddingClient (local, sentence-transformers)model name only

Each profile is typed chat or embedding (the ModelType enum). The factory infers it from the model name (text-embedding-* → embedding) unless MODEL_TYPE is set explicitly. Asking create_completion_client for an embedding profile (or vice versa) raises at construction time — fail fast.

Anthropic, Bedrock, and HuggingFace providers are gated by optional imports (ANTHROPIC_AVAILABLE, BEDROCK_AVAILABLE, SENTENCE_TRANSFORMERS_AVAILABLE). The factory skips a provider whose library isn't installed. Install everything with:

Terminal window
cd backend && uv sync --extra all-llm

Configuring provider profiles

Section titled “Configuring provider profiles”

Profiles are loaded from environment variables, not from a YAML block. LLMConfig.from_environment() parses numbered patterns — N runs from 1 to 99 — so you can define many profiles of the same provider type:

Terminal window
# OpenAI chat profile
OPENAI_PROFILE_NAME_1=default
OPENAI_API_KEY_1=sk-...
OPENAI_MODEL_1=gpt-4o


# OpenAI embedding profile (model name implies embedding type)
OPENAI_PROFILE_NAME_2=openai-small
OPENAI_API_KEY_2=sk-...
OPENAI_MODEL_2=text-embedding-3-small


# Azure chat profile
AZURE_PROFILE_NAME_1=rai-chat-azure-openai
AZURE_API_KEY_1=...
AZURE_API_BASE_1=https://your-resource.openai.azure.com
AZURE_DEPLOYMENT_NAME_1=gpt-4o
AZURE_API_VERSION_1=2024-02-01


# Bedrock embedding profile (EU-resident)
BEDROCK_PROFILE_NAME_1=cohere-eu
BEDROCK_MODEL_1=eu.cohere.embed-v4:0
BEDROCK_AWS_REGION_1=eu-west-1
BEDROCK_AWS_ACCESS_KEY_ID_1=...
BEDROCK_AWS_SECRET_ACCESS_KEY_1=...

The first profile loaded becomes default_provider, so adapters can reference "default" to get it. Set MODEL_TYPE per profile (OPENAI_MODEL_TYPE_2=embedding) when the model name doesn't make the type obvious.

Put these in backend/.env (gitignored) or the deployment's environment. Never commit credentials. App-wide config — database, queue, storage — lives in application config; provider profiles are read straight from the environment.

Referencing a profile from a pipeline

Section titled “Referencing a profile from a pipeline”

Adapters take a llm_profile string (a profile name), not a provider/model pair. The model is fixed on the profile, so the adapter config never restates it:

- type: "llm_translator"
  config:
    llm_profile: "rai-chat-azure-openai"   # resolves to a chat profile
    source_language: "Norwegian"
    target_language: "English"
    timeout: 120.0

Embedding adapters take a models list, each entry a profile reference:

- type: "embedding_generator"
  config:
    models:
      - profile_name: "openai-small"
        enabled: true
        batch_size: 100
      - profile_name: "cohere-eu"
        enabled: true
    storage_providers:
      - type: "pgvector18"

Swapping "rai-chat-azure-openai" from Azure to another chat provider (OpenAI or Anthropic) is an environment change — repoint the profile, no pipeline edit. (Bedrock has no chat client, so a chat profile can't point at it.) A few clustering adapters name the field embedding_profile; same concept, embedding-typed profile.

The factory

Section titled “The factory”
from factflow_llm import LLMClientFactory
from factflow_llm.settings import LLMConfig


config: LLMConfig = LLMConfig.from_environment()
factory = LLMClientFactory(config)


chat = factory.create_completion_client(profile_name="default")
emb  = factory.create_embedding_client(profile_name="openai-small")


response = await chat.complete(messages=[...])
vectors  = await emb.embed(texts=[...])

factflow-llm injects the factory into adapters as llm_factory; adapters call create_completion_client(self._llm_profile). Clients are cached per profile — first call constructs, subsequent calls reuse.

EU data residency

Section titled “EU data residency”

Several workflows require EU-resident inference and enforce it through profile choice. The factory routes provider_type=bedrock to _create_bedrock_embedding_client only — BedrockClient (chat) exists in factflow-llm but is not wired into the factory. So Bedrock yields embedding clients today; the provider table above marks Bedrock chat unsupported.

  • EU-resident embeddings — point an embedding profile at Bedrock eu-west-1 with an eu.-prefixed model (eu.cohere.embed-v4:0) that keeps data in-region, or run HuggingFace locally so embeddings never leave the host.
  • EU-resident chat — point a chat profile at an Azure OpenAI resource provisioned in an EU region (the AZURE_API_BASE_* endpoint determines residency). Bedrock chat is not an option until the chat client is wired.

The boost, hygiene, and knowledge adapters document EU residency on their llm_profile / embedding_profile fields. Residency is a property of the profile you point them at — pick an EU-region Azure profile for chat and an eu-west-1 Bedrock (or local HuggingFace) profile for embeddings when content falls under residency rules.

Verifying credentials

Section titled “Verifying credentials”

After startup, list the profiles the deployment actually loaded — secrets stripped:

Terminal window
factflow system llm-profiles

This hits /system/llm-profiles, which reads from the same LLMConfig.from_environment() the runtime uses and groups profiles by chat / embedding. A profile missing from the list means its env vars didn't parse (typo, missing API_KEY, or an uninstalled optional dependency).

Error classification

Section titled “Error classification”

Every provider exception is classified through classify_llm_error(exc), which returns an LLMErrorClassification carrying two booleans — fatal and retryable — plus status_code, error_type, and message. Adapters react to the booleans, so behaviour is consistent across vendors:

fatalretryableTriggered byCaller behaviour
TrueFalse401 auth, 403 permission, 404 not-found (AuthenticationError, PermissionDeniedError, NotFoundError)Don't retry; abort
FalseTrue429 rate limit, 500+ server error, connection error, timeout (RateLimitError, InternalServerError, APIConnectionError, APITimeoutError)Retry with backoff

Anything unrecognised defaults to fatal=False, retryable=True — the conservative path. is_fatal_llm_error(exc) is the one-line check for the fatal case, and get_error_metadata(exc) flattens the classification into the dict an AdapterResult carries. Adapter authors rarely write custom classification — the helper handles the per-provider quirks.

Adding a new provider

Section titled “Adding a new provider”
  1. Implement LLMClientProtocol (and/or EmbeddingClientProtocol) in a new file under factflow-llm/src/factflow_llm/
  2. Add a discovery flag (MYPROVIDER_AVAILABLE = try_import("myprovider"))
  3. Register the provider_type branch in LLMClientFactory._create_client_for_provider
  4. Add any provider-specific fields to LLMProviderConfig in settings.py
  5. Add a test that constructs the client without credentials — it should fail fast, not hang
Section titled “Related”