DataFlow

Use this repo skill for open-dataflow / dataflow tasks that prepare data, build pipelines, launch serving backends, run evaluations, or wrap operators with RayOrch.

First reads

• references/repo-provenance.md — source commit, package version, evidence paths, and refresh baseline.
• references/installation-and-backends.md — install commands, verified import checks, and backend selection.
• references/api-overview.md — verified public surface for the most important classes, CLIs, and serving helpers.
• references/troubleshooting.md — cross-cutting install/import, key-mismatch, TTY, and backend failure guidance.
• references/repo-routing-metadata.json — routing metadata used by the managed repo-skill router.

Fast start

1. Install the package for local inspection or use the published distribution:
   • python -m pip install -e .
   • or python -m pip install open-dataflow
2. Run a safe environment smoke check:
   • python scripts/check_dataflow_env.py
3. Inspect the public API surface when you need signatures or command names:
   • python scripts/inspect_dataflow_surface.py
4. Read the focused sub-skill that matches the task family below.

Route map

pipeline-foundations

Use for operator, pipeline, storage, prompt, wrapper, and compile-time key-validation work.

Typical requests:

• create or debug a PipelineABC, BatchedPipelineABC, or StreamBatchedPipelineABC
• choose between FileStorage, LazyFileStorage, DummyStorage, batch storage, or MyScale storage
• fix input_* / output_* mismatches or Key Matching Error
• validate prompt_restrict, PromptABC, DIYPromptABC, or draw_graph

serving-cli

Use for CLI routing, dataflow init, chat, eval, pdf2model, text2model, webui, and serving class setup.

Typical requests:

• inspect command groups and help output
• choose an API, local, or hosted serving backend
• understand credential, timeout, or WebUI side effects
• diagnose missing optional serving dependencies

text-workflows

Use for text cleaning, filtering, reasoning, code, conversation, Text2SQL, prompt-driven generation, translation, and text2model prep.

Typical requests:

• adapt CPU-safe text filters or prompt-driven generators
• map columns such as raw_content, instruction, problem, generated_code, or golden_answer
• prepare offline text fixtures or text2model inputs
• separate API-backed stages from pure local filtering

document-vision-rag

Use for PDF/OCR, visual QA, knowledge-base cleaning, LightRAG, Agentic RAG, speech, chemistry, and pdf2model planning.

Typical requests:

• validate document inputs before OCR or retrieval
• choose between KBC, PDF VQA, or document-prep flows
• reason about MinerU, FlashMinerU, DataFlex, LlamaFactory, and audio extras
• diagnose missing documents, suffixes, or hardware/credential limits

rayorch-acceleration

Use for RayOrch wrapping, actor cleanup, and pipeline acceleration without changing the surrounding pipeline contract.

Typical requests:

• wrap an existing operator with RayAcceleratedOperator
• preserve order and storage behavior through CPU or GPU execution
• decide when to use replicas, num_gpus_per_replica, or env
• debug shutdown or optional Ray dependency issues

How to choose

• If the task is mainly about how DataFlow operators and pipeline storage work, start with pipeline-foundations.
• If the task is mainly about CLI commands, serving, or launch-time dependencies, start with serving-cli.
• If the task is mainly about text or tabular dataset transformation, start with text-workflows.
• If the task is mainly about documents, PDFs, OCR, RAG, or pdf2model, start with document-vision-rag.
• If the task is mainly about distributed acceleration of an existing operator, start with rayorch-acceleration.
• If the request spans multiple families, use the root references first, then hand off to the narrowest sub-skill that owns the final action.

What this root skill does not do

• It does not execute training, downloads, or backend-heavy workflows by default.
• It does not rely on the original checkout after generation; bundled references and scripts carry the reusable guidance.
• It does not replace the focused sub-skills when the request is already narrow.

When to revisit provenance

Read references/repo-provenance.md before deciding whether this skill is stale for a checkout of DataFlow or before running a refresh workflow.