跳到正文
cccskills
免费在 GitHub 公开

fifa-vaila-continuation

Resume work on the FIFA Skeletal Tracking Light 2026 challenge using vailá modules (SAM 3, soccer pitch keypoints, calibration, per-frame DLT). Use whenever the user reopens the project after closing the terminal, asks "where did we stop on FIFA?", or wants the next concrete commands to run from `~/data/vaila` with `.venv` already created.

中文简介正在准备中。当前显示原文说明。

vaila-multimodaltoolbox/vaila

实测评分 未实测

查看安装方法

包含的文件（1）

  • SKILL.md11.4 KB

SKILL.md（原文）

安装前可以查看交给 AI 代理的指令内容。

FIFA + vailá Continuation Skill

Companion files:

  • Maintenance rule: when you edit any *.py during this continuation, also update version/date metadata (global version from vaila.py), root README.md Last updated:, and matching help .md/.html. See AGENTS.md (“Mandatory: Update metadata on any script change”).

  • ~/data/vaila/AGENT_HISTORY.md — what was done, current state, decisions.

  • ~/data/vaila/HISTORY.md — chronological log of the session.

  • .claude/skills/fifa-skeletal-tracking/SKILL.md — full FIFA pipeline reference.

  • .claude/skills/sam3-video/SKILL.md — SAM 3 video segmentation reference.

  • .claude/skills/soccer-field-keypoints-yolo/SKILL.md — pitch-keypoint training/inference.

  • docs/fifa_workflow.md §4.5 — external merged unified/ dataset, QA export, dedupe, yolo pose train (weights → soccerfield_keypoints_ai).

  • .claude/skills/yolo-fb-gui-cli/SKILL.md — YOLO + FB chooser + GUI→CLI mirror.

  • docs/sessions/2026-07-06-yolo-fb-gui-cli-mirror.md — v0.3.72 session log.

  • docs/sessions/2026-07-07-sapiens-output-dir-fix.md — v0.3.76 Sapiens2 single output dir.

Companion repo (the official starter kit): ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026.

Objective

Continue from a known-good baseline and execute the improvement path:

  1. SAM 3 per-player video masks
  2. Per-frame soccer pitch keypoints for camera refinement
  3. Per-frame DLT export for rec2d.py / rec3d.py
  4. (Optional) fifa baseline (SAM 3D Body) → fifa pack for Codabench

Known-good baseline (validated 2026-04-25)

  • pyproject.toml (portable, one file for every OS) synced with the cuda dependency group (PyTorch cu129).
  • uv sync --no-group cpu --group cuda --extra sam succeeded.
  • vaila/models/sam3/sam3.pt and sam3.1_multiplex.pt present.
  • cv2.VideoCapture is back after uv pip install --reinstall opencv-python==4.10.0.84.
  • SAM 3 smoke test: 32 frames of ARG_CRO_000737.mp4 written to ~/data/FIFA/outputs_sam3_smoke/processed_sam_20260425_183925/.
  • SAM 3 60-frame test: 37 person IDs tracked, sam_points.csv produced. Output: ~/data/FIFA/outputs_sam3_test1/processed_sam_20260425_213120/ARG_CRO_000737/.

Pitch-keypoint detector status (2026-04-25 21:42)

ModelStatusPath
Old football_pitch32_best.ptCOLLAPSED — all 32 kp in ~30 px cluster (mosaic-trained)vaila/models/soccerfield_keypoints_yolo/football_pitch32_best.pt
New 50-ep recipe-A best.ptGeometry recovered — kp span full image; pose_mAP50=0.17, box_mAP50=0.83vaila/models/runs/pose_fifa/pitch32_recipeA_50ep/weights/best.pt
400-ep continuationTraining in background (PID 2070719)vaila/models/runs/pose_fifa/pitch32_recipeA_400ep/weights/best.pt (when done)

Training recipe that escapes the collapse:

uv run yolo pose train \
  model=~/data/vaila/yolo26s-pose.pt \
  data=vaila/models/hf_datasets/football-pitch-detection/data/data.yaml \
  epochs=800 imgsz=1280 batch=8 \
  mosaic=0.0 mixup=0.0 close_mosaic=0 erasing=0.0 \
  pose=25.0 kobj=2.0 device=0 patience=80 \
  project=vaila/models/runs/pose_fifa name=pitch32_recipeA

Monitoring the running 400-ep job:

tail -1 ~/data/vaila/vaila/models/runs/pose_fifa/pitch32_recipeA_400ep/results.csv \
  | awk -F',' '{printf "epoch=%s box_mAP50=%s pose_mAP50=%s pose_mAP50-95=%s\n",$1,$11,$15,$16}'

# live progress
tail -f ~/data/vaila/vaila/models/runs/pose_fifa/train_recipeA_400ep.log

Sanity test for a freshly trained best.pt (CPU-friendly, no GPU):

CUDA_VISIBLE_DEVICES="" uv run python -m vaila.soccerfield_keypoints_ai \
  --mode video \
  -i ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data/videos/ARG_CRO_000737.mp4 \
  -o ~/data/FIFA/outputs_pitch_kps_v3 \
  --backend ultralytics \
  --weights ~/data/vaila/vaila/models/runs/pose_fifa/pitch32_recipeA_400ep/weights/best.pt \
  --imgsz 1280 --conf 0.20 --draw-min-conf 0.30 \
  --device cpu --stride 30 --max-frames 5 --overlay-video

Look at field_keypoints_video.csv and confirm kp x ranges across ~the whole image width (not collapsed).

Resume Sequence (paste in this order)

0. Enter the project

cd ~/data/vaila
source .venv/bin/activate     # optional — `uv run` works without it

1. Re-confirm the CUDA group + extras (idempotent)

bash bin/setup_pyproject.sh --target=cuda --yes
uv sync --no-group cpu --group cuda --extra sam

2. Make sure SAM 3 weights are still there

uv run vaila/vaila_sam.py --download-weights   # no-op if already present
ls vaila/models/sam3/sam3.pt

3. Quick smoke test (1 video, 32 frames) — only if a fresh sanity check is needed

uv run vaila/vaila_sam.py \
  -i ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data/videos/ARG_CRO_000737.mp4 \
  -o ~/data/FIFA/outputs_sam3_smoke \
  -t person --max-frames 32

4. Run SAM 3 batch over all FIFA videos

Default uses subprocess-per-video isolation (clean GPU between clips):

uv run vaila/vaila_sam.py \
  -i ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data/videos \
  -o ~/data/FIFA/outputs_sam3 \
  -t person \
  --max-frames 64 \
  --max-input-long-edge 1280 \
  --postprocess-points foot

Useful flags:

  • --frame-by-frame — lower VRAM (CUDA only).
  • --no-isolate-batch — same-process batch (debug only).
  • -t "team in red", -t goalkeeper, etc. — text presets accepted by SAM 3.

5. Soccer-pitch keypoints (need EITHER Roboflow OR a YOLO best.pt)

5A. Roboflow backend (no local weights needed)

export ROBOFLOW_API_KEY="YOUR_KEY"
uv run python -m vaila.soccerfield_keypoints_ai \
  --mode video \
  -i ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data/videos/ARG_CRO_000737.mp4 \
  -o ~/data/FIFA/outputs_pitch_kps \
  --backend roboflow \
  --roboflow-model-id football-field-detection-f07vi/14 \
  --conf 0.3 --draw-min-conf 0.05 \
  --stride 1 --max-frames 300 \
  --overlay-video

5B. Local Ultralytics YOLO-pose

uv run python -m vaila.soccerfield_keypoints_ai \
  --mode video \
  -i ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data/videos/ARG_CRO_000737.mp4 \
  -o ~/data/FIFA/outputs_pitch_kps \
  --backend ultralytics \
  --weights /ABS/PATH/TO/best.pt \
  --imgsz 1280 --conf 0.3 --draw-min-conf 0.05 \
  --device 0 --stride 1 --overlay-video

To train a best.pt from the vendored dataset, follow .claude/skills/soccer-field-keypoints-yolo/SKILL.md (vaila/models/hf_datasets/football-pitch-detection/data/data.yaml, kpt_shape: [32, 3], imgsz=1280, mosaic=0, erasing=0).

6. Calibration (manual first, automatic later)

# List the semantic point names accepted by the calibrator
uv run python -m vaila.soccerfield_calib --list-keypoints

# Manual interactive calibration on a single FIFA clip
uv run python -m vaila.soccerfield_calib \
  -v ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data/videos/ARG_CRO_000737.mp4 \
  -o ~/data/FIFA/outputs_pitch_calib \
  --data-root ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data

The detector outputs generic p1, p2, …. Until a mapping (p_i → semantic name) is published, the safe path is manual calibration on a few key frames and reuse of the homography for the whole clip when the camera is roughly static.

7. Per-frame DLT for rec2d.py / rec3d.py

Once the FIFA cameras/*.npz is populated (either from the official dataset or from fifa baseline --export-camera):

uv run vaila/vaila_sam.py fifa dlt-export \
  --cameras-dir ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data/cameras \
  --output-dir   ~/data/FIFA/outputs_dlt_per_frame

Use the resulting per-frame .dlt2d / .dlt3d with vaila/rec2d.py and vaila/rec3d.py (broadcast / moving cameras). For genuinely fixed cameras, use rec2d_one_dlt2d.py / rec3d_one_dlt3d.py instead.

8. (Optional) Full FIFA pipeline + Codabench submission

# Once and only once (clones sam_3d_body + downloads gated weights)
uv run hf auth login                        # see workaround below if hf is broken
bash bin/setup_fifa_sam3d.sh
uv sync --no-group cpu --group cuda --extra sam --extra fifa

uv run vaila/vaila_sam.py fifa bootstrap \
  --videos-dir ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data/videos \
  --data-root  ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data
uv run vaila/vaila_sam.py fifa prepare    --data-root ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data
uv run vaila/vaila_sam.py fifa boxes      --data-root ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data
uv run vaila/vaila_sam.py fifa preprocess --data-root ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data
uv run vaila/vaila_sam.py fifa baseline   --data-root ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data \
                                          --output    ~/data/FIFA/outputs/submission_full.npz \
                                          --export-camera
uv run vaila/vaila_sam.py fifa dlt-export --cameras-dir ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data/cameras \
                                          --output-dir  ~/data/FIFA/outputs/dlt_per_frame
uv run vaila/vaila_sam.py fifa pack       --submission-full ~/data/FIFA/outputs/submission_full.npz \
                                          --data-root       ~/data/FIFA/FIFA-Skeletal-Tracking-Starter-Kit-2026/data \
                                          --output-dir      ~/data/FIFA/outputs \
                                          --split val

Recovery / known issues

  • hf auth login fails inside .venv with ModuleNotFoundError: No module named 'typer'. Either set HF_TOKEN=... in .env, or run hf auth login from a separate pipx install huggingface_hub install.
  • cv2.VideoCapture missing after a fresh sync: uv pip install --reinstall opencv-python==4.10.0.84.
  • SAM 3 batch CUDA OOM cascade: already mitigated in vaila_sam.py via subprocess-per-video isolation. If you still OOM, add --frame-by-frame and lower --max-input-long-edge.

Next-step priorities (highest expected MPJPE win first)

  1. Per-frame camera refinement with pitch keypoints (Section 5+6).
  2. Cleaner player boxes/masks with SAM 3 batch (Section 4).
  3. Only then touch fifa baseline and 3D refinement (Section 8).

Fallbacks when the retrain plateaus

  1. Push pose loss higher (pose=40 kobj=4), reset from pitch32_recipeA_50ep/best.pt, 200–300 epochs.
  2. Bigger backbone (yolo26m-pose.pt / yolo26x-pose.pt) with lr0=1e-4 amp=False to dodge NaN.
  3. Keypoint-safe augmentation: degrees=10 translate=0.1 scale=0.5 — never mosaic or mixup.
  4. Roboflow drop-in (no local training): --backend roboflow --roboflow-model-id football-field-detection-f07vi/14 with ROBOFLOW_API_KEY.
  5. Manual homography: 4–6 keyframes/video with vaila/soccerfield_calib.py, cv2.findHomography, interpolate.
  6. External SOTA as alternative pitch detector:

评价

还没有评价。欢迎分享你的使用感受。

评论

无需登录即可发表。欢迎分享使用感受、提问，或说明在哪些环境下无法运行。

还没有评论。来写第一条吧。

提交后会立即公开。请勿填写邮箱、电话等个人信息。URL 最多 2 个。

0 / 1000 字

此表单受 reCAPTCHA 保护，适用 Google 的隐私权政策和服务条款。

同一仓库的技能

agent-platform-alert-configuration

免费

Configures best-practice alerting policies for AI agents using OpenTelemetry (OTel) metrics, generating output as Terraform (.tf) configuration files. Use when analyzing, writing, or deploying alerting policies to monitor agent latency, error rates, token usage, and quality metrics. Don't use for standard infrastructure monitoring unrelated to AI agents, or when the agent is not instrumented with OpenTelemetry (for Reliability, Cost, Safety, Security alerts). NOTE: Reliability, Cost, Safety, and Security alerts use generic OTel metrics and work across runtimes (such as Cloud Run, Vertex AI). Quality alerts rely on Vertex AI Online Monitors and are strictly bound to Vertex AI deployments.

中文简介正在准备中。当前显示原文说明。

vaila-multimodaltoolbox/vaila212026年10月24日 更新

agent-platform-deploy

免费

Deploy open models or custom weights from Model Garden to Agent Platform endpoints, check the status of an in-progress deployment operation, or clean up resources by undeploying models and deleting endpoints. Use when asked to actively deploy a model, list the Model Garden CATALOG of available models, check if a specific model is deployable (`gcloud ai model-garden models list-deployment-config`), query deployment cost, troubleshoot deployment errors (like quota limits), or undeploy/clean up endpoints. Also use when copying and deploying a 1P Tuned Model. Don't use for pure listing/discovery questions of the form "is X deployed?", "list my endpoints", or "which regions have models running?" — for those use `agent-platform-endpoint-management`. Don't use for public Vertex AI deployments (use the `vertex-deploy` skill) or for running model evaluations (use the `agent-platform-eval-flywheel` skill).

中文简介正在准备中。当前显示原文说明。

vaila-multimodaltoolbox/vaila212026年10月24日 更新

agent-platform-endpoint-management

免费

Manages Agent Platform serving endpoints. Use when you need to create, list, describe, update, or delete serving endpoints for model deployment on Agent Platform. Also use when troubleshooting endpoint permission, quota, or resource busy errors. Don't use for deploying models to endpoints or for running model evaluations.

中文简介正在准备中。当前显示原文说明。

vaila-multimodaltoolbox/vaila212026年10月24日 更新

agent-platform-eval-flywheel

免费

Measures and improves the quality of AI models and agents on Google Cloud using the Eval Quality Flywheel methodology. Use when evaluating an agent or model, building an eval dataset, picking or writing evaluation metrics, analyzing failures, comparing results before and after a fix, or when guidance is needed on Agent Platform eval methodology — including dataset schema, LLM-as-judge scoring, and common failure causes. For fine-tuning, use agent-platform-tuning. For general production deployment, use agent-platform-deploy.

中文简介正在准备中。当前显示原文说明。

vaila-multimodaltoolbox/vaila212026年10月24日 更新

agent-platform-inference

免费

Connects to and performs inference with Google Cloud Agent Platform GenAI models, including First-Party Gemini models and Third-Party OpenMaaS models (Llama, DeepSeek, Qwen, etc.). Use when asked to perform inference, ask a model a question, run a test prompt, execute chat completions, or generate code for calling Gemini or OpenMaaS models, authenticate with GenAI SDK, OpenAI SDK, or legacy Agent Platform SDK, configure base URLs and global/regional endpoints, or troubleshoot 429 Resource Exhausted (DSQ), 400 User Validation, or 404 Not Found errors. Don't use for deploying models to endpoints or for running model evaluations.

中文简介正在准备中。当前显示原文说明。

vaila-multimodaltoolbox/vaila212026年10月24日 更新

agent-platform-migrate-from-ai-studio

免费

Guides agents and users through migrating from Gemini API in Google AI Studio to Gemini Enterprise Agent Platform (formerly Vertex AI). Use this skill when moving applications to Google Cloud, to leverage Cloud credits, or to unify inferencing with other Cloud infrastructure (IAM, billing, telemetry).

中文简介正在准备中。当前显示原文说明。

vaila-multimodaltoolbox/vaila212026年10月24日 更新

查看 vaila-multimodaltoolbox 的全部技能

报告此技能的问题