Register Hugging Face datasets for Marin by inspecting their structure without downloading the full dataset. Identify text fields, configurations, and data splits, then add a dataset module and validate the mapping with a sample.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Register a named Hugging Face dataset for Marin by inspecting its schema and adding the appropriate experiments/datasets module.
설치하기 전에 에이전트에게 전달되는 지시 내용을 확인할 수 있습니다.
Inspect the schema without downloading the full dataset:
uv run lib/marin/tools/get_hf_dataset_schema.py <dataset_name> [options]
For programmatic inspection:
from marin.tools.get_hf_dataset_schema import get_schema
schema = get_schema(dataset_name="wikitext", config_name="wikitext-103-v1")
Use repo-managed dependencies. For a one-off inspection without a provisioned
environment, add --with datasets --with pyyaml to uv run.
If the result says a config is required, select one of available_configs and
retry with --config_name. Add --trust_remote_code only after inspecting the
dataset repository and accepting its code-execution boundary. The tool streams;
do not replace it with a full dataset download.
If the dataset cannot be found, stop and report the identifier, path, or access failure instead of guessing a replacement.
Choose the text field from the reported schema. Prefer an exact text field,
then a field containing text, then another string field. Inspect sample_row
to verify the content; it may be empty for some datasets. The result also
reports splits, text_field_candidates, and features.
Add a leaf module under experiments/datasets/ using the lazy builders in
marin.experiment.data:
<name>_dataset() for one corpus;<name>_datasets() -> dict[str, ...] for a keyed family;experiments/datasets/nemotron.py and return
one keyed handle per subset.Validate the selected config, splits, text mapping, and one sample before adding tokenization or downstream experiment configuration.
아직 리뷰가 없습니다. 사용해 본 소감을 남겨 주세요.
로그인 없이 쓸 수 있습니다. 사용 후기, 질문, 작동하지 않은 환경 등을 편하게 남겨 주세요.
아직 댓글이 없습니다. 첫 댓글을 남겨 보세요.
Compare Zephyr changes using the same prepared dataset and matching execution settings. Analyze Finelog metrics for each processing stage to distinguish CPU efficiency, elapsed time, and peak memory changes, then report improvements, regressions, or tradeoffs.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Turn a specified GCP access request into a reviewable Pulumi change and pull request. Select the narrowest suitable role and encrypt personal email identities, covering resource permissions and Cloud Run viewer access through IAP, an access gateway.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Add or update Pallas/Mosaic kernels, specialized computation routines for JAX. The skill covers reference implementations, output and gradient checks, public wrappers, fallback behavior, performance benchmarks, and hardware-specific tuning when requested.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Preserve old experiment scripts with a dated Git tag, a named snapshot of the code, and add comments linking related GitHub issues to the archived files and cleanup pull request.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Launch and continuously monitor a specified Zephyr pipeline on Iris when asked. It checks coordinator and worker health, tracks stage progress, identifies stalled or repeated failures, and requires explicit approval before stopping or restarting jobs.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Try model or training changes in a copied Grug or Grugformer experiment, document the differences, and bring successful changes back into the base template while removing superseded experiment code and checking behavior.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.