Skip to content
Factflow

Markdown & segmentation

The markdown workflow turns raw HTML into vector-ready text. Five adapters in a chain:

storage_retriever → html_to_markdown → smart_segmenter → segment_publisher → markdown_storage_writer

Typical upstream: webscraper. Typical downstream: embeddings.

Minimal pipeline

Section titled “Minimal pipeline”
version: "1.0"


routes:
  markdown_processor:
    inbound:
      queue: "/queue/markdown.input"
      subscription: "markdown-processors"
      concurrency: 10


    adapters:
      - type: "storage_retriever"
        config:
          key_field: "storage_key"


      - type: "html_to_markdown"
        config:
          preserve_images: false


      - type: "smart_segmenter"
        config:
          model: "text-embedding-3-small"
          max_tokens: 512
          overlap_tokens: 20


      - type: "segment_publisher"


      - type: "markdown_storage_writer"

Adapters

Section titled “Adapters”
TypePurpose
storage_retrieverRead HTML (or other input) from storage by lineage reference
html_to_markdownGitHub-flavoured conversion, preserves structure
smart_segmenterToken-aware splitting using tiktoken; each segment fits an LLM context
segment_publisherFan-out: one message per segment
markdown_storage_writerPersist canonical markdown + each segment

Tokens and segmentation

Section titled “Tokens and segmentation”

smart_segmenter respects max_tokens (default 512, ceiling 2000) while keeping markdown structure intact — it splits on section boundaries, not mid-sentence. The model field names the embedding model the segments are destined for (e.g. text-embedding-3-small); the segmenter derives the matching tiktoken encoding from it, so token counts match what the embedder will see.

Merge strategy

Section titled “Merge strategy”

If segments end up very small, MergeStrategy determines whether to merge adjacent ones back together. Configure per pipeline via the segmenter config.

Direct use of building blocks

Section titled “Direct use of building blocks”

Outside a pipeline, import the segmenter or token counter for ad-hoc work:

from factflow_markdown import MarkdownSegmenter, TokenCounter


counter = TokenCounter(model="text-embedding-3-small")
print(counter.count_tokens("Some text."))


segmenter = MarkdownSegmenter(max_chunk_size=512, chunk_overlap=50)
for segment in segmenter.segment(markdown_text):
    print(segment.text, segment.token_count)
Section titled “Related”