Fine-Tuning Dataset Curator

Prepare, filter, deduplicate, and format high-quality datasets for supervised fine-tuning (SFT), RLHF, and DPO training of language models.

Activation Triggers

Activate on: "fine-tuning dataset", "training data preparation", "RLHF data", "DPO pairs", "SFT data", "data quality filtering", "dedup training data", "curate instruction dataset", "preference data"

NOT for: Model training loop implementation (ai-engineer), prompt optimization without fine-tuning (prompt-engineer), or general ETL pipelines (data-pipeline-engineer)

Quick Start

1. Define the task — What behavior should the fine-tuned model exhibit? Write 10 gold-standard examples by hand first.
2. Collect raw data — Scrape, export from logs, use existing datasets, or generate synthetic examples with a stronger model.
3. Filter and clean — Remove duplicates, low-quality entries, PII, and off-topic examples. Target quality over quantity.
4. Format for training — Convert to the target format: chat-ml for SFT, chosen/rejected pairs for DPO, reward signals for RLHF.
5. Validate — Hold out 10-15% for evaluation, verify distribution balance, run a small training test before full fine-tune.

Core Capabilities

Domain	Technologies	Notes
Quality Filtering	fasttext classifiers, perplexity scoring, regex rules	Remove noise before it poisons the model
Deduplication	MinHash (datasketch), exact hash, SimHash	Near-dedup critical for training stability
Augmentation	LLM-generated paraphrases, backtranslation, persona variation	3-5x dataset size with diversity
Format Conversion	chat-ml, Alpaca, ShareGPT, OpenAI JSONL	Match target training framework
PII Removal	presidio, regex, spaCy NER	Legal requirement for most training data
RLHF/DPO Prep	Preference pair generation, reward model labeling	Chosen/rejected pairs with margin scoring

Architecture Patterns

Pattern 1: SFT Data Curation Pipeline

Raw Sources ──→ [Extract] ──→ [Filter] ──→ [Dedup] ──→ [Augment] ──→ [Format] ──→ [Validate]
    │               │            │            │            │              │            │
  logs, docs    parse to     quality       MinHash      paraphrase    chat-ml     hold-out
  APIs, CSVs    instruction/ scoring       near-dedup   via LLM       or JSONL    eval set
                response     remove <                   persona                   distribution
                pairs        threshold                  variation                 check

# Quality filtering pipeline
import hashlib
from datasketch import MinHash, MinHashLSH

def curate_sft_dataset(raw_examples: list[dict]) -> list[dict]:
    # Step 1: Basic quality filters
    filtered = []
    for ex in raw_examples:
        instruction, response = ex["instruction"], ex["response"]
        if len(response.split()) < 10:          # Too short
            continue
        if len(response.split()) > 2000:         # Too long (likely garbage)
            continue
        if instruction.strip() == "":            # Empty instruction
            continue
        if response.count("\n") > 50:            # Excessive formatting
            continue
        filtered.append(ex)

    # Step 2: Near-deduplication with MinHash
    lsh = MinHashLSH(threshold=0.8, num_perm=128)
    deduped = []
    for i, ex in enumerate(filtered):
        mh = MinHash(num_perm=128)
        for word in ex["response"].split():
            mh.update(word.encode("utf-8"))
        if not lsh.query(mh):  # No near-duplicate found
            lsh.insert(f"doc_{i}", mh)
            deduped.append(ex)

    # Step 3: Format for training
    formatted = []
    for ex in deduped:
        formatted.append({
            "messages": [
                {"role": "user", "content": ex["instruction"]},
                {"role": "assistant", "content": ex["response"]}
            ]
        })
    return formatted

Pattern 2: DPO Preference Pair Generation

Instruction ──→ [Generate N responses] ──→ [Score/Rank] ──→ [Select Pairs]
                      │                         │                  │
                  temperature=0.8            human eval         chosen: best
                  N=4 responses per          or LLM judge       rejected: worst
                  instruction                or heuristic        margin > threshold

Output format (DPO):
{
  "prompt": "Explain quantum computing",
  "chosen": "Quantum computing uses qubits...",    # High-quality response
  "rejected": "Quantum computing is computers..."  # Lower-quality response
}

Pattern 3: Synthetic Data Augmentation

10 Gold Examples ──→ [LLM Generator] ──→ [Quality Filter] ──→ 500 Examples
                          │                      │
                     "Generate 50            remove duplicates,
                      variations of          score perplexity,
                      this instruction       human spot-check
                      with different          10% sample
                      personas and
                      complexity levels"

Anti-Patterns

1. Quantity over quality — 1,000 high-quality examples outperform 50,000 noisy ones. Filter aggressively; a model trained on garbage outputs garbage.
2. No deduplication — Duplicate or near-duplicate examples cause the model to memorize specific phrasings instead of learning patterns. Always dedup.
3. Ignoring distribution balance — If 80% of your data is one category, the model will be biased. Track category distribution and balance or weight accordingly.
4. Training on the eval set — Accidentally including test examples in training data produces inflated metrics. Deduplicate between train/eval splits.
5. Skipping PII removal — Training on data with real names, emails, and phone numbers bakes PII into model weights. Use presidio or spaCy NER to scrub before training.

Quality Checklist

• Gold-standard examples written by hand (minimum 10) before scaling
• Raw data filtered: minimum response length, maximum length, non-empty instructions
• Near-deduplication applied (MinHash threshold 0.8 or tighter)
• Train/eval split with deduplication across splits (no leakage)
• PII removed from all examples (names, emails, phone numbers, addresses)
• Distribution balanced across categories/topics (or intentionally weighted)
• Format matches target training framework (chat-ml, Alpaca, OpenAI JSONL)
• Augmented data quality-checked via spot-sampling (10% human review)
• DPO pairs have sufficient quality margin between chosen and rejected
• Final dataset stats documented: size, categories, avg tokens, quality score distribution