<img src="banner.png" width="100%" alt="document-chunker banner">

Deutsch — Offizielle Deutsch-Version / Documento Oficial en Deutsch.

Document Chunker (Deutsch)

Splits documents into overlapping token chunks. Optimized for RAG pipelines and LLM context windows. Zero dependencies — Python stdlib + re only.

Usage

As Library

from document_chunker import DocumentChunker

chunker = DocumentChunker(chunk_size=400, overlap=80)
chunks = chunker.chunk_text("Long text...")

for chunk in chunks:
    print(f"Chunk {chunk['chunk_id']}: {chunk['tokens']} tokens")

Chunking a File

chunks = chunker.chunk_document("document.md", source="My Project")

Chunking an Entire Directory

from document_chunker import chunk_corpus

chunks = chunk_corpus(["doc1.md", "doc2.txt"], source="Corpus")

CLI

python document_chunker.py document.md    # Single file
python document_chunker.py ./docs/        # Entire directory

Parameters

Parameter	Default	Description
chunk_size	400	Max tokens per chunk
overlap	80	Overlapping tokens between chunks

Supported File Types

.txt, .md, .py, .sh

Änderungsprotokoll

1.0.0 (2026-03-12)

• Ported from BACH system/tools/document_chunker.py