Investigate model specifications, requirements, and evaluation strategy. Use when researching a model before benchmarking: reading HuggingFace model cards, estimating VRAM requirements, selecting GPU providers, and determining evaluation approach. Triggers on "model research", "investigate model", "model info", "VRAM estimate", "which provider", "model card".
インストールする前に、エージェントに与えられる指示の中身を確認できます。
You are an experienced ML engineer investigating a model before running benchmarks.
Given a model name, produce a structured research summary:
Run the helper script to pull structured metadata:
python .claude/skills/model-researcher/scripts/hf_model_info.py MODEL_ID
To check if the model is available on HF Inference API (serverless):
python .claude/skills/model-researcher/scripts/hf_inference_check.py MODEL_ID
To search for similar or alternative models:
python .claude/skills/model-researcher/scripts/hf_model_search.py --task llm --sort downloads --limit 10
python .claude/skills/model-researcher/scripts/hf_model_search.py --search "qwen" --limit 5
If the script fails or the model is not on HuggingFace:
Run the estimator with --check-env to filter by available tokens and sort by cost:
python .claude/skills/model-researcher/scripts/gpu_estimator.py \
--params PARAM_COUNT --quant fp16 --model-type llm --check-env --json
--check-env reads .env and:
Review the cost estimate and include it in the research summary. Adjust based on:
Classify the model and load the appropriate evaluation guide:
| Model Type | Reference File | Key Metrics |
|---|---|---|
| LLM (text-to-text) | references/eval-llm.md | tokens/sec, latency, output quality |
| VLM (vision-language) | references/eval-vlm.md | tokens/sec, hallucination rate, OCR accuracy |
| Code Generation | references/eval-code-gen.md | tokens/sec, pass@1, FIM accuracy |
| Embedding | references/eval-embedding.md | embeddings/sec, retrieval quality |
| Image Generation | references/eval-image.md | sec/image, visual quality |
| TTS (text-to-speech) | references/eval-tts.md | RTF, audio quality |
| STT (speech recognition) | references/eval-stt.md | RTF, WER/CER |
| Audio / Music Gen | references/eval-audio.md | RTF, sec/generation, listening quality |
| Video Generation | references/eval-video-gen.md | sec/frame, temporal consistency |
| Object Detection | references/eval-object-detection.md | mAP, FPS, detection quality |
| 3D Generation | references/eval-3d-gen.md | generation time, mesh quality |
| Time Series | references/eval-timeseries.md | MAE/RMSE, prediction accuracy |
Read the relevant reference file for detailed evaluation guidance.
For unknown model types: research the model's documentation and design an appropriate evaluation based on its input/output modalities.
Sort by cheapest hourly cost, check .env for token availability:
HF_TOKEN.HF_TOKEN only.MODAL_TOKEN_ID + MODAL_TOKEN_SECRET. $30/mo free.BEAM_TOKEN.VAST_API_KEY. $0.10–2.00/hr.RUNPOD_API_KEY. $0.34–2.69/hr.gpu_estimator.py outputs cost-sorted recommendations. Use that to pick the cheapest viable option.
Structure your findings as:
## Research Summary: {model_name}
- **URL**: https://huggingface.co/{model_id}
- **Description**: {1-2 sentence summary from model card}
- **Type**: {model_type}
- **Parameters**: {param_count}
- **Architecture**: {architecture}
- **License**: {license}
- **Notable Features**: {e.g., "6 language tags for accent switching", "supports img2img"}
- **Install**: {pip install command from model card, if available}
- **VRAM Required**: {vram_estimate}
- **Recommended GPU**: {gpu}
- **Recommended Provider**: {provider}
- **Estimated Cost**: {cost} (e.g., "~$0.53 on Modal A100-40GB, ~15min" or "Free via HF Inference API")
- **Evaluation Strategy**: {brief description}
- **Key Metrics**: {metrics to measure}
This summary is used in the final report. Include enough model context (URL, description, features) so that the eval-reporter phase can write a rich Model Overview section.
Always present the cost estimate before proceeding to execution so the user can approve.
まだレビューはありません。使ってみた感想をお寄せください。
概要と使いどころ
Autonomous model validation and benchmarking. Investigates any ML model (LLM, image gen, TTS, time series, etc.), runs it on GPU cloud, evaluates quality and performance, and generates HTML reports. Use when user asks to verify, benchmark, evaluate, or test a model. Triggers on "verify model", "benchmark", "evaluate model", "test model", "run benchmark", "model evaluation", "モデルを検証", "ベンチマーク", "モデルを試して".
Generate HTML reports and structured metrics from model evaluation results. Creates publication-ready reports with embedded outputs (images, audio, charts) and metrics.json for cross-model comparison. Use when generating reports, writing metrics, creating evaluation summaries, or formatting benchmark results. Triggers on "generate report", "write metrics", "create report", "evaluation summary", "benchmark results", "format results".
日本語の概要は準備中です。原文の説明を表示しています。
Execute model inference on GPU cloud providers. Handles code generation, deployment, execution, and result collection across HF Inference API/Endpoints, Colab, Modal, beam.cloud, Vast.ai, and RunPod. Use when running models on GPU, deploying to cloud, executing notebooks, or troubleshooting GPU execution failures. Triggers on "run on GPU", "execute model", "deploy to modal", "colab notebook", "beam deploy", "HF inference", "HF endpoints", "vast", "runpod".
日本語の概要は準備中です。原文の説明を表示しています。