本文へ移動
cccskills
無料GitHub で公開

fifa-vaila-continuation

Resume work on the FIFA Skeletal Tracking Light 2026 challenge using vailá modules (SAM 3, soccer pitch keypoints, calibration, per-frame DLT). Use whenever the user reopens the project after closing the terminal, asks "where did we stop on FIFA?", or wants the next concrete commands to run from `~/data/vaila` with `.venv` already created.

vaila-multimodaltoolbox/vaila
インストール方法を見る

含まれるファイル（1）

  • SKILL.md11.4 KB

SKILL.md（原文）

インストールする前に、エージェントに与えられる指示の中身を確認できます。

FIFA + vailá Continuation Skill

Companion files:

  • Maintenance rule: when you edit any *.py during this continuation, also update version/date metadata (global version from vaila.py), root README.md Last updated:, and matching help .md/.html. See AGENTS.md (“Mandatory: Update metadata on any script change”).

  • ~/data/vaila/AGENT_HISTORY.md — what was done, current state, decisions.

  • ~/data/vaila/HISTORY.md — chronological log of the session.

  • .claude/skills/fifa-skeletal-tracking/SKILL.md — full FIFA pipeline reference.

  • .claude/skills/sam3-video/SKILL.md — SAM 3 video segmentation reference.

  • .claude/skills/soccer-field-keypoints-yolo/SKILL.md — pitch-keypoint training/inference.

  • docs/fifa_workflow.md §4.5 — external merged unified/ dataset, QA export, dedupe, yolo pose train (weights → soccerfield_keypoints_ai).

  • .claude/skills/yolo-fb-gui-cli/SKILL.md — YOLO + FB chooser + GUI→CLI mirror.

  • docs/sessions/2026-07-06-yolo-fb-gui-cli-mirror.md — v0.3.72 session log.

  • docs/sessions/2026-07-07-sapiens-output-dir-fix.md — v0.3.76 Sapiens2 single output dir.

Companion repo (the official starter kit): ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026.

Objective

Continue from a known-good baseline and execute the improvement path:

  1. SAM 3 per-player video masks
  2. Per-frame soccer pitch keypoints for camera refinement
  3. Per-frame DLT export for rec2d.py / rec3d.py
  4. (Optional) fifa baseline (SAM 3D Body) → fifa pack for Codabench

Known-good baseline (validated 2026-04-25)

  • pyproject.toml (portable, one file for every OS) synced with the cuda dependency group (PyTorch cu128).
  • uv sync --no-group cpu --group cuda --extra sam succeeded.
  • vaila/models/sam3/sam3.pt and sam3.1_multiplex.pt present.
  • cv2.VideoCapture is back after uv pip install --reinstall opencv-python==4.10.0.84.
  • SAM 3 smoke test: 32 frames of ARG_CRO_000737.mp4 written to ~/data/FIFA/outputs_sam3_smoke/processed_sam_20260425_183925/.
  • SAM 3 60-frame test: 37 person IDs tracked, sam_points.csv produced. Output: ~/data/FIFA/outputs_sam3_test1/processed_sam_20260425_213120/ARG_CRO_000737/.

Pitch-keypoint detector status (2026-04-25 21:42)

ModelStatusPath
Old football_pitch32_best.ptCOLLAPSED — all 32 kp in ~30 px cluster (mosaic-trained)vaila/models/soccerfield_keypoints_yolo/football_pitch32_best.pt
New 50-ep recipe-A best.ptGeometry recovered — kp span full image; pose_mAP50=0.17, box_mAP50=0.83vaila/models/runs/pose_fifa/pitch32_recipeA_50ep/weights/best.pt
400-ep continuationTraining in background (PID 2070719)vaila/models/runs/pose_fifa/pitch32_recipeA_400ep/weights/best.pt (when done)

Training recipe that escapes the collapse:

uv run yolo pose train \
  model=~/data/vaila/yolo26s-pose.pt \
  data=vaila/models/hf_datasets/football-pitch-detection/data/data.yaml \
  epochs=800 imgsz=1280 batch=8 \
  mosaic=0.0 mixup=0.0 close_mosaic=0 erasing=0.0 \
  pose=25.0 kobj=2.0 device=0 patience=80 \
  project=vaila/models/runs/pose_fifa name=pitch32_recipeA

Monitoring the running 400-ep job:

tail -1 ~/data/vaila/vaila/models/runs/pose_fifa/pitch32_recipeA_400ep/results.csv \
  | awk -F',' '{printf "epoch=%s box_mAP50=%s pose_mAP50=%s pose_mAP50-95=%s\n",$1,$11,$15,$16}'

# live progress
tail -f ~/data/vaila/vaila/models/runs/pose_fifa/train_recipeA_400ep.log

Sanity test for a freshly trained best.pt (CPU-friendly, no GPU):

CUDA_VISIBLE_DEVICES="" uv run python -m vaila.soccerfield_keypoints_ai \
  --mode video \
  -i ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data/videos/ARG_CRO_000737.mp4 \
  -o ~/data/FIFA/outputs_pitch_kps_v3 \
  --backend ultralytics \
  --weights ~/data/vaila/vaila/models/runs/pose_fifa/pitch32_recipeA_400ep/weights/best.pt \
  --imgsz 1280 --conf 0.20 --draw-min-conf 0.30 \
  --device cpu --stride 30 --max-frames 5 --overlay-video

Look at field_keypoints_video.csv and confirm kp x ranges across ~the whole image width (not collapsed).

Resume Sequence (paste in this order)

0. Enter the project

cd ~/data/vaila
source .venv/bin/activate     # optional — `uv run` works without it

1. Re-confirm the CUDA group + extras (idempotent)

bash bin/setup_pyproject.sh --target=cuda --yes
uv sync --no-group cpu --group cuda --extra sam

2. Make sure SAM 3 weights are still there

uv run vaila/vaila_sam.py --download-weights   # no-op if already present
ls vaila/models/sam3/sam3.pt

3. Quick smoke test (1 video, 32 frames) — only if a fresh sanity check is needed

uv run vaila/vaila_sam.py \
  -i ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data/videos/ARG_CRO_000737.mp4 \
  -o ~/data/FIFA/outputs_sam3_smoke \
  -t person --max-frames 32

4. Run SAM 3 batch over all FIFA videos

Default uses subprocess-per-video isolation (clean GPU between clips):

uv run vaila/vaila_sam.py \
  -i ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data/videos \
  -o ~/data/FIFA/outputs_sam3 \
  -t person \
  --max-frames 64 \
  --max-input-long-edge 1280 \
  --postprocess-points foot

Useful flags:

  • --frame-by-frame — lower VRAM (CUDA only).
  • --no-isolate-batch — same-process batch (debug only).
  • -t "team in red", -t goalkeeper, etc. — text presets accepted by SAM 3.

5. Soccer-pitch keypoints (need EITHER Roboflow OR a YOLO best.pt)

5A. Roboflow backend (no local weights needed)

export ROBOFLOW_API_KEY="YOUR_KEY"
uv run python -m vaila.soccerfield_keypoints_ai \
  --mode video \
  -i ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data/videos/ARG_CRO_000737.mp4 \
  -o ~/data/FIFA/outputs_pitch_kps \
  --backend roboflow \
  --roboflow-model-id football-field-detection-f07vi/14 \
  --conf 0.3 --draw-min-conf 0.05 \
  --stride 1 --max-frames 300 \
  --overlay-video

5B. Local Ultralytics YOLO-pose

uv run python -m vaila.soccerfield_keypoints_ai \
  --mode video \
  -i ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data/videos/ARG_CRO_000737.mp4 \
  -o ~/data/FIFA/outputs_pitch_kps \
  --backend ultralytics \
  --weights /ABS/PATH/TO/best.pt \
  --imgsz 1280 --conf 0.3 --draw-min-conf 0.05 \
  --device 0 --stride 1 --overlay-video

To train a best.pt from the vendored dataset, follow .claude/skills/soccer-field-keypoints-yolo/SKILL.md (vaila/models/hf_datasets/football-pitch-detection/data/data.yaml, kpt_shape: [32, 3], imgsz=1280, mosaic=0, erasing=0).

6. Calibration (manual first, automatic later)

# List the semantic point names accepted by the calibrator
uv run python -m vaila.soccerfield_calib --list-keypoints

# Manual interactive calibration on a single FIFA clip
uv run python -m vaila.soccerfield_calib \
  -v ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data/videos/ARG_CRO_000737.mp4 \
  -o ~/data/FIFA/outputs_pitch_calib \
  --data-root ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data

The detector outputs generic p1, p2, …. Until a mapping (p_i → semantic name) is published, the safe path is manual calibration on a few key frames and reuse of the homography for the whole clip when the camera is roughly static.

7. Per-frame DLT for rec2d.py / rec3d.py

Once the FIFA cameras/*.npz is populated (either from the official dataset or from fifa baseline --export-camera):

uv run vaila/vaila_sam.py fifa dlt-export \
  --cameras-dir ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data/cameras \
  --output-dir   ~/data/FIFA/outputs_dlt_per_frame

Use the resulting per-frame .dlt2d / .dlt3d with vaila/rec2d.py and vaila/rec3d.py (broadcast / moving cameras). For genuinely fixed cameras, use rec2d_one_dlt2d.py / rec3d_one_dlt3d.py instead.

8. (Optional) Full FIFA pipeline + Codabench submission

# Once and only once (clones sam_3d_body + downloads gated weights)
uv run hf auth login                        # see workaround below if hf is broken
bash bin/setup_fifa_sam3d.sh
uv sync --no-group cpu --group cuda --extra sam --extra fifa

uv run vaila/vaila_sam.py fifa bootstrap \
  --videos-dir ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data/videos \
  --data-root  ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data
uv run vaila/vaila_sam.py fifa prepare    --data-root ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data
uv run vaila/vaila_sam.py fifa boxes      --data-root ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data
uv run vaila/vaila_sam.py fifa preprocess --data-root ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data
uv run vaila/vaila_sam.py fifa baseline   --data-root ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data \
                                          --output    ~/data/FIFA/outputs/submission_full.npz \
                                          --export-camera
uv run vaila/vaila_sam.py fifa dlt-export --cameras-dir ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data/cameras \
                                          --output-dir  ~/data/FIFA/outputs/dlt_per_frame
uv run vaila/vaila_sam.py fifa pack       --submission-full ~/data/FIFA/outputs/submission_full.npz \
                                          --data-root       ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data \
                                          --output-dir      ~/data/FIFA/outputs \
                                          --split val

Recovery / known issues

  • hf auth login fails inside .venv with ModuleNotFoundError: No module named 'typer'. Either set HF_TOKEN=... in .env, or run hf auth login from a separate pipx install huggingface_hub install.
  • cv2.VideoCapture missing after a fresh sync: uv pip install --reinstall opencv-python==4.10.0.84.
  • SAM 3 batch CUDA OOM cascade: already mitigated in vaila_sam.py via subprocess-per-video isolation. If you still OOM, add --frame-by-frame and lower --max-input-long-edge.

Next-step priorities (highest expected MPJPE win first)

  1. Per-frame camera refinement with pitch keypoints (Section 5+6).
  2. Cleaner player boxes/masks with SAM 3 batch (Section 4).
  3. Only then touch fifa baseline and 3D refinement (Section 8).

Fallbacks when the retrain plateaus

  1. Push pose loss higher (pose=40 kobj=4), reset from pitch32_recipeA_50ep/best.pt, 200–300 epochs.
  2. Bigger backbone (yolo26m-pose.pt / yolo26x-pose.pt) with lr0=1e-4 amp=False to dodge NaN.
  3. Keypoint-safe augmentation: degrees=10 translate=0.1 scale=0.5 — never mosaic or mixup.
  4. Roboflow drop-in (no local training): --backend roboflow --roboflow-model-id football-field-detection-f07vi/14 with ROBOFLOW_API_KEY.
  5. Manual homography: 4–6 keyframes/video with vaila/soccerfield_calib.py, cv2.findHomography, interpolate.
  6. External SOTA as alternative pitch detector:

レビュー

まだレビューはありません。使ってみた感想をお寄せください。

ログインしてレビューを書く

同じリポジトリのスキル

概要と使いどころ

agent-platform-alert-configuration

無料

Configures best-practice alerting policies for AI agents using OpenTelemetry (OTel) metrics, generating output as Terraform (.tf) configuration files. Use when analyzing, writing, or deploying alerting policies to monitor agent latency, error rates, token usage, and quality metrics. Don't use for standard infrastructure monitoring unrelated to AI agents, or when the agent is not instrumented with OpenTelemetry (for Reliability, Cost, Safety, Security alerts). NOTE: Reliability, Cost, Safety, and Security alerts use generic OTel metrics and work across runtimes (such as Cloud Run, Vertex AI). Quality alerts rely on Vertex AI Online Monitors and are strictly bound to Vertex AI deployments.

日本語の概要は準備中です。原文の説明を表示しています。

vaila-multimodaltoolbox/vaila192026年10月8日 更新

agent-platform-deploy

無料

Deploy open models or custom weights from Model Garden to Agent Platform endpoints, check the status of an in-progress deployment operation, or clean up resources by undeploying models and deleting endpoints. Use when asked to actively deploy a model, list the Model Garden CATALOG of available models, check if a specific model is deployable (`gcloud ai model-garden models list-deployment-config`), query deployment cost, troubleshoot deployment errors (like quota limits), or undeploy/clean up endpoints. Also use when copying and deploying a 1P Tuned Model. Don't use for pure listing/discovery questions of the form "is X deployed?", "list my endpoints", or "which regions have models running?" — for those use `agent-platform-endpoint-management`. Don't use for public Vertex AI deployments (use the `vertex-deploy` skill) or for running model evaluations (use the `agent-platform-eval-flywheel` skill).

日本語の概要は準備中です。原文の説明を表示しています。

vaila-multimodaltoolbox/vaila192026年10月8日 更新

agent-platform-endpoint-management

無料

Manages Agent Platform serving endpoints. Use when you need to create, list, describe, update, or delete serving endpoints for model deployment on Agent Platform. Also use when troubleshooting endpoint permission, quota, or resource busy errors. Don't use for deploying models to endpoints or for running model evaluations.

日本語の概要は準備中です。原文の説明を表示しています。

vaila-multimodaltoolbox/vaila192026年10月8日 更新

agent-platform-eval-flywheel

無料

Measures and improves the quality of AI models and agents on Google Cloud using the Eval Quality Flywheel methodology. Use when evaluating an agent or model, building an eval dataset, picking or writing evaluation metrics, analyzing failures, comparing results before and after a fix, or when guidance is needed on Agent Platform eval methodology — including dataset schema, LLM-as-judge scoring, and common failure causes. For fine-tuning, use agent-platform-tuning. For general production deployment, use agent-platform-deploy.

日本語の概要は準備中です。原文の説明を表示しています。

vaila-multimodaltoolbox/vaila192026年10月8日 更新

agent-platform-inference

無料

Connects to and performs inference with Google Cloud Agent Platform GenAI models, including First-Party Gemini models and Third-Party OpenMaaS models (Llama, DeepSeek, Qwen, etc.). Use when asked to perform inference, ask a model a question, run a test prompt, execute chat completions, or generate code for calling Gemini or OpenMaaS models, authenticate with GenAI SDK, OpenAI SDK, or legacy Agent Platform SDK, configure base URLs and global/regional endpoints, or troubleshoot 429 Resource Exhausted (DSQ), 400 User Validation, or 404 Not Found errors. Don't use for deploying models to endpoints or for running model evaluations.

日本語の概要は準備中です。原文の説明を表示しています。

vaila-multimodaltoolbox/vaila192026年10月8日 更新

agent-platform-migrate-from-ai-studio

無料

Guides agents and users through migrating from Gemini API in Google AI Studio to Gemini Enterprise Agent Platform (formerly Vertex AI). Use this skill when moving applications to Google Cloud, to leverage Cloud credits, or to unify inferencing with other Cloud infrastructure (IAM, billing, telemetry).

日本語の概要は準備中です。原文の説明を表示しています。

vaila-multimodaltoolbox/vaila192026年10月8日 更新

vaila-multimodaltoolbox のスキルをすべて見る

このスキルの問題を報告する