Write a data pipeline design document. Use when the user says "pipeline design doc", "document this pipeline", "pipeline architecture doc", "data flow document", "how does this pipeline work", "design doc for ETL", "pipeline spec", or needs to capture the architecture, data flow, and operational details of a data pipeline - even if they don't explicitly say "design doc".
The English summary is being prepared. Showing the original description.
Review the instructions the agent will receive before you install.
Overview
Based on Fundamentals of Data Engineering (Reis & Housley) and Designing Data-Intensive Applications (Kleppmann). A pipeline design doc is the architectural contract for a data flow: what data moves, from where to where, how it transforms, and what guarantees it provides. Kleppmann's standard: document the data system's reliability, scalability, and maintainability properties - not just the happy path.
The test: can a new engineer understand what this pipeline does, debug it when it breaks, and extend it safely - without asking anyone?
Workflow
Step 1: Write the pipeline overview
Pipeline: [name]
Owner: [team or person]
Purpose: [1-2 sentences on why this pipeline exists and what decision it enables]
Source systems: [list of upstream data sources]
Destination: [where data lands - warehouse, lake, API, etc.]
Schedule: [batch frequency or streaming latency SLA]
Data volume: [approx rows/day or GB/day]
SLA: [when downstream consumers need data available by]
What enters the stage (schema, format, expected volume)
What the stage does (filter, join, aggregate, enrich)
What exits the stage (output schema, format)
Technology used (Spark, dbt, Airflow, Kafka, etc.)
Step 3: Document data contracts
For each source, specify the contract:
Source: [system name]
Schema: [fields and types]
Delivery method: [push/pull, API, file drop, CDC, etc.]
Delivery frequency: [real-time / hourly / daily]
SLA: [when source guarantees data is available]
Schema change policy: [how the source team notifies of changes]
Known quirks: [nulls in required fields, late-arriving data, duplicates, etc.]
For the output, specify the guarantee:
Output table/topic: [name]
Schema: [fields and types]
Freshness SLA: [data is no older than X]
Completeness guarantee: [how to detect missing data]
Deduplication: [how duplicates are handled]
Step 4: Document transformation logic
For each non-trivial transformation, explain:
What: what business logic is applied
Why: why this transformation exists (not obvious from code)
Assumptions: data quality or format assumptions baked in
Edge cases: known cases that need special handling
Example:
Transform: revenue_attribution
What: assigns each order to the last marketing touch within 30 days
Why: finance requires last-touch model per FY23 attribution policy
Assumptions: user_id is consistent across sessions (not always true for guests)
Edge cases: guest checkouts with no touch - attributed to "direct"
Step 5: Document failure modes and recovery
Failure mode: [source is late / schema change / job fails mid-run]
Impact: [what downstream consumers experience]
Detection: [how we know it's happening - alert name or monitoring link]
Recovery:
1. [first action]
2. [verify: command or query]
3. [reprocessing steps if data needs backfill]
Escalate if: [condition that exceeds this runbook]
Step 6: Document operational details
Monitoring: links to dashboards, key metrics and alert thresholds
Logs: where to find them, what to search for
Backfill procedure: how to rerun historical data safely
Idempotency: is the pipeline safe to rerun? If yes, how? If no, what guards exist?
Dependencies: what must run before this pipeline (orchestration DAG link)
Anti-Patterns
1. Happy path only
Bad: "The pipeline reads from Postgres and loads to BigQuery daily."
Good: Document what happens when Postgres is slow, when a schema column is dropped, when the daily job misses its window.
2. Undocumented assumptions
Bad: "Joins on user_id."
Good: "Joins on user_id. Assumes user_id is stable post-registration. Guest users (user_id = null) are excluded from this join and handled in the guest_orders pipeline."
3. No data contract
Bad: "Takes whatever the source sends."
Good: Define the expected schema, the schema change notification process, and how the pipeline handles unexpected fields.
4. No idempotency statement
Bad: Doc doesn't say whether reruns are safe.
Good: "This pipeline is idempotent. Reruns use MERGE (upsert) on event_id. Safe to rerun for any date range."
Rate any product, feature, or experience on the 11-star scale (Brian Chesky's Airbnb thought experiment). Use when user says "rate this experience", "11-star", "star rating", "experience audit", "how good is this", "experience rating", "product audit", "quality assessment", or wants to evaluate product quality and identify improvement paths. Also trigger when user wants to benchmark a feature, assess where a product stands, or map out what "great" looks like - even if they don't explicitly say "11-star".
The English summary is being prepared. Showing the original description.
Generate A/B copy variants grounded in Eugene Schwartz's awareness and market-sophistication stages and John Caples' tested-headline discipline. Every variant declares its angle, the awareness stage it targets, and a falsifiable hypothesis, and the skill recommends which pair to test first and what delta counts as signal. Use when the user says "A/B test this", "write variants", "give me options", "test different angles", "copy variants", "ab copy", "test this headline", "multiple versions", "which version is better", or wants to test copy before publishing. For reviewing copy that already exists, see copy-review. For sample size and stopping rules, see growth-experiment. For diagnosing a page rather than its words, see page-cro. For full paid ad sets by platform, see ad-campaign-writer.
The English summary is being prepared. Showing the original description.
Design statistically rigorous experiments to test data science hypotheses. Use when the user says "design an A/B test", "run an experiment", "test this hypothesis", "is this difference significant", "how many samples do I need", "randomized controlled trial", "causal inference", "uplift test", "holdout group", "significance test", "power calculation", "avoid p-hacking", or needs to determine whether an observed effect is real before making a product or model decision. For lean build-measure-learn product experiments without the statistics, use experiment-design instead.
The English summary is being prepared. Showing the original description.
Write measurable acceptance criteria for requirements or user stories. Use when the user says "write acceptance criteria", "definition of done", "how do we test this", "fit criteria", "given when then", "Gherkin scenarios", "what does done look like", "testable conditions", "how will we know this works" - even if they don't explicitly say "acceptance criteria".
The English summary is being prepared. Showing the original description.
Build a strategic account plan for winning, retaining, or expanding a named account. Use when the user says "build an account plan", "strategic account planning", "how do I grow this account", "account expansion strategy", "whitespace analysis", "QBR prep", "key account review", "map the stakeholders in this account", "land and expand plan", "how do I break into [company]", or wants a structured strategy for a specific account.
The English summary is being prepared. Showing the original description.
Write paid ad copy for LinkedIn, Google, Meta, and YouTube. Produces multiple variants per platform, each targeting a different Eugene Schwartz awareness level and Cialdini persuasion principle. Use when the user asks for ad copy, ad variants, "write LinkedIn ads", Google ads, Meta ads, paid social copy, ad headlines, or wants creative for a paid campaign. Reads brand voice, ICP, and positioning from knowledge/. For organic social posts, see linkedin-post or social-calendar. For the landing page the ad points at, see landing-page-writer. For testing variants, see ab-copy-writer.
The English summary is being prepared. Showing the original description.