Register Hugging Face datasets for Marin by inspecting their structure without downloading the full dataset. Identify text fields, configurations, and data splits, then add a dataset module and validate the mapping with a sample.
Register a named Hugging Face dataset for Marin by inspecting its schema and adding the appropriate experiments/datasets module.
Review the instructions the agent will receive before you install.
Inspect the schema without downloading the full dataset:
uv run lib/marin/tools/get_hf_dataset_schema.py <dataset_name> [options]
For programmatic inspection:
from marin.tools.get_hf_dataset_schema import get_schema
schema = get_schema(dataset_name="wikitext", config_name="wikitext-103-v1")
Use repo-managed dependencies. For a one-off inspection without a provisioned
environment, add --with datasets --with pyyaml to uv run.
If the result says a config is required, select one of available_configs and
retry with --config_name. Add --trust_remote_code only after inspecting the
dataset repository and accepting its code-execution boundary. The tool streams;
do not replace it with a full dataset download.
If the dataset cannot be found, stop and report the identifier, path, or access failure instead of guessing a replacement.
Choose the text field from the reported schema. Prefer an exact text field,
then a field containing text, then another string field. Inspect sample_row
to verify the content; it may be empty for some datasets. The result also
reports splits, text_field_candidates, and features.
Add a leaf module under experiments/datasets/ using the lazy builders in
marin.experiment.data:
<name>_dataset() for one corpus;<name>_datasets() -> dict[str, ...] for a keyed family;experiments/datasets/nemotron.py and return
one keyed handle per subset.Validate the selected config, splits, text mapping, and one sample before adding tokenization or downstream experiment configuration.
No reviews yet. Tell others what you thought.
No login needed. Share how it worked for you, ask a question, or note where it did not run.
No comments yet. Be the first to write one.
Compare Zephyr changes using the same prepared dataset and matching execution settings. Analyze Finelog metrics for each processing stage to distinguish CPU efficiency, elapsed time, and peak memory changes, then report improvements, regressions, or tradeoffs.
Turn a specified GCP access request into a reviewable Pulumi change and pull request. Select the narrowest suitable role and encrypt personal email identities, covering resource permissions and Cloud Run viewer access through IAP, an access gateway.
Add or update Pallas/Mosaic kernels, specialized computation routines for JAX. The skill covers reference implementations, output and gradient checks, public wrappers, fallback behavior, performance benchmarks, and hardware-specific tuning when requested.
Preserve old experiment scripts with a dated Git tag, a named snapshot of the code, and add comments linking related GitHub issues to the archived files and cleanup pull request.
Launch and continuously monitor a specified Zephyr pipeline on Iris when asked. It checks coordinator and worker health, tracks stage progress, identifies stalled or repeated failures, and requires explicit approval before stopping or restarting jobs.
Try model or training changes in a copied Grug or Grugformer experiment, document the differences, and bring successful changes back into the base template while removing superseded experiment code and checking behavior.