Register Hugging Face datasets for Marin by inspecting their structure without downloading the full dataset. Identify text fields, configurations, and data splits, then add a dataset module and validate the mapping with a sample.
中文简介正在准备中。当前显示英语简介。
Register a named Hugging Face dataset for Marin by inspecting its schema and adding the appropriate experiments/datasets module.
安装前可以查看交给 AI 代理的指令内容。
Inspect the schema without downloading the full dataset:
uv run lib/marin/tools/get_hf_dataset_schema.py <dataset_name> [options]
For programmatic inspection:
from marin.tools.get_hf_dataset_schema import get_schema
schema = get_schema(dataset_name="wikitext", config_name="wikitext-103-v1")
Use repo-managed dependencies. For a one-off inspection without a provisioned
environment, add --with datasets --with pyyaml to uv run.
If the result says a config is required, select one of available_configs and
retry with --config_name. Add --trust_remote_code only after inspecting the
dataset repository and accepting its code-execution boundary. The tool streams;
do not replace it with a full dataset download.
If the dataset cannot be found, stop and report the identifier, path, or access failure instead of guessing a replacement.
Choose the text field from the reported schema. Prefer an exact text field,
then a field containing text, then another string field. Inspect sample_row
to verify the content; it may be empty for some datasets. The result also
reports splits, text_field_candidates, and features.
Add a leaf module under experiments/datasets/ using the lazy builders in
marin.experiment.data:
<name>_dataset() for one corpus;<name>_datasets() -> dict[str, ...] for a keyed family;experiments/datasets/nemotron.py and return
one keyed handle per subset.Validate the selected config, splits, text mapping, and one sample before adding tokenization or downstream experiment configuration.
还没有评价。欢迎分享你的使用感受。
无需登录即可发表。欢迎分享使用感受、提问，或说明在哪些环境下无法运行。
还没有评论。来写第一条吧。
Compare Zephyr changes using the same prepared dataset and matching execution settings. Analyze Finelog metrics for each processing stage to distinguish CPU efficiency, elapsed time, and peak memory changes, then report improvements, regressions, or tradeoffs.
中文简介正在准备中。当前显示英语简介。
Turn a specified GCP access request into a reviewable Pulumi change and pull request. Select the narrowest suitable role and encrypt personal email identities, covering resource permissions and Cloud Run viewer access through IAP, an access gateway.
中文简介正在准备中。当前显示英语简介。
Add or update Pallas/Mosaic kernels, specialized computation routines for JAX. The skill covers reference implementations, output and gradient checks, public wrappers, fallback behavior, performance benchmarks, and hardware-specific tuning when requested.
中文简介正在准备中。当前显示英语简介。
Preserve old experiment scripts with a dated Git tag, a named snapshot of the code, and add comments linking related GitHub issues to the archived files and cleanup pull request.
中文简介正在准备中。当前显示英语简介。
Launch and continuously monitor a specified Zephyr pipeline on Iris when asked. It checks coordinator and worker health, tracks stage progress, identifies stalled or repeated failures, and requires explicit approval before stopping or restarting jobs.
中文简介正在准备中。当前显示英语简介。
Try model or training changes in a copied Grug or Grugformer experiment, document the differences, and bring successful changes back into the base template while removing superseded experiment code and checking behavior.
中文简介正在准备中。当前显示英语简介。