Skip to content
cccskills
FreePublished on GitHub

ai-cli-benchmark

Design, run, and review fair performance, token, and cost comparisons across GitHub Copilot CLI, Codex CLI, and direct model APIs. Use for AI CLI benchmark, Copilot vs Codex, prompt-cache diagnosis, cache read/write analysis, fresh-process latency measurement, token-cost comparison, or when a benchmark must separate provider caching from local CLI caches.

The English summary is being prepared. Showing the original description.

aktsmm/Agent-Skills
Works withGitHubOpenAI
See how to install

Files included (6)

  • SKILL.md4.7 KB
  • LICENSE.txt264 B
  • scripts/helper.py3.6 KB
  • scripts/requirements.txt25 B
  • scripts/test_helper.py2.4 KB
  • skill-license.json712 B

SKILL.md (original)

Review the instructions the agent will receive before you install.

AI CLI Benchmark

When to Use

  • Copilot CLI、Codex CLI、直接 API の速度・token・費用を比較する。
  • prompt cache の read / write を区別し、cache-offを主張できるか確認する。
  • fresh process、temporary home、instruction/tool isolationを揃えた再現可能なbenchmarkを作る。

過去sessionの事後分析だけなら対象外。このskillは比較条件の設計、実行、証拠化を担当する。

Cache Boundaries

  • Copilot CLI / Codex CLIには、provider側model prompt cacheを無効化する公開optionがない。fresh process、一時home、ephemeral、nonceでもcache offと呼ばない。
  • COPILOT_CACHE_HOMEはMarketplace、自動更新packageなどのローカル一時データ用。COPILOT_MCP_TOOL_CACHEはローカルMCP serverのtool-list snapshot用。model prompt cacheではない。
  • OpenAI prompt cacheは手動消去できない。GPT-5.6以降をResponses APIから直接呼ぶ場合だけ、explicit-only modeでbreakpointを置かず、そのrequestのcache writeを避けられる。CLIへ推測適用しない。
  • nonceはuser prompt全体の一致を避ける補助であり、その前のhidden system / agent prefixのcache readを防がない。

Workflow

  1. Freeze the task

    • dataset、expected labels、prompt、schema、input order、model、reasoningを固定し、hashを取る。
    • accuracyと性能を同時に比較する場合、同じ合格条件を全providerへ適用する。

  2. Define measurement boundaries

    • one request、serial total、fresh-process end to endを区別する。
    • API直呼びとCLI起動込み時間をmodel speedとして並べない。

  3. Isolate each CLI

    • repo外のowned temp directory、fresh process、一時COPILOT_HOME / auth-only CODEX_HOMEを使う。
    • custom instructions、tools、MCP、session reuseを無効にし、secretを子processとlogへ渡さない。

  4. Observe cache; do not assume control

    • 必要ならprompt先頭へ一意nonceを入れ、nonce本体ではなくhashだけ保存する。
    • zero-cache-read gateは診断用。失敗はprovider cacheの観測であり、benchmark失敗として隠さない。

  5. Record raw categories

    • normal input、cache read、cache write、output、reasoningを別fieldで保存する。
    • provider schemaで包含・排他関係を確認するまでtoken区分を足さない。

  6. Repeat before claiming performance

    • 単発差をcacheの因果効果にしない。median / p95 / rangeとaccuracyを報告する。

  7. Validate the artifact

    python scripts/helper.py result.json `
  --require-cache-metadata `
  --report validation.json
    • Required shape: test_design.prompt_sha256, optional test_design.cache_observation (nonce_added_at_prompt_start, nonce_sha256), and providers[] with provider, elapsed_ms, passed, cases, usage (input_tokens, cached_input_tokens, cache_write_tokens, output_tokens, optional reasoning_tokens).
    • Add --require-zero-cache-read only for the diagnostic gate in step 4.

Reporting Rules

  • cache writeはcache hitではない。readと分け、該当modelの料金区分で費用を計算する。
  • uncached runではなく、観測どおりcache read 0 runまたはcache-observed runと書く。
  • CLI version、model、reasoning、prompt hash、nonce hash、fresh-process条件、測定境界、usage schemaを記載する。
  • cache read 0でも将来のwriteやhidden prefixの制御を保証しない。

Done Criteria

  • dataset/prompt/schema/model/reasoningが固定され、hashがある。
  • 全providerでfresh processと一時homeを使う。
  • cache read/writeを別fieldで保存する。
  • 測定境界とaccuracyを一緒に報告する。
  • 複数runなしの速度差へ因果を割り当てない。
  • benchmark JSON validatorがPASSする。

References

Reviews

No reviews yet. Tell others what you thought.

Comments

No login needed. Share how it worked for you, ask a question, or note where it did not run.

No comments yet. Be the first to write one.

Comments are published right away. Do not include personal information such as email addresses or phone numbers. Up to 2 URLs.

0 / 1,000 characters

This form is protected by reCAPTCHA, and the Google Privacy Policy and Terms of Service apply.

More skills from this repository

agentic-workflow-guide

Free

Design, review, and debug agent workflows, and decide when a request should use a prompt, instruction, skill, agent, or hook before escalating to multi-agent design. Use for .agent.md / .instructions.md / .prompt.md / AGENTS.md work, workflow architecture, orchestration planning, scheduled automation model allocation, or when agent workflows may be overkill. Triggers on 'agent workflow', 'create agent', 'automation models', 'ワークフロー設計', 'orchestrator'.

The English summary is being prepared. Showing the original description.

aktsmm/Agent-Skills26Updated Oct 18, 2026

analyze-copilot-sessions

Free

Analyze historical VS Code GitHub Copilot Chat sessions by model, reasoning effort, AIU, time, reliability, workflow behavior, and external quality evidence, or safely prune workspace-scoped local chat history by age. Use for session analysis, repeated-task retrospectives, model evaluation, cost/performance analysis, and old session cleanup.

The English summary is being prepared. Showing the original description.

aktsmm/Agent-Skills26Updated Oct 18, 2026

animated-infographic

Free

Create reproducible animated infographic GIFs from structured data or measured results, with audience-language labels, explicit timing semantics, representative-frame QA, and a static fallback. Use when asked for GIF生成, animated infographic, 動的インフォグラフィック, 結果が流れる図, animated benchmark, or a blog/SNS animation whose displayed metrics must remain traceable to source data.

The English summary is being prepared. Showing the original description.

aktsmm/Agent-Skills26Updated Oct 18, 2026

archify

Free

Create polished, validated architecture, workflow, sequence, data-flow, and lifecycle/state diagrams as explorable standalone HTML with inline SVG, dark/light themes, optional trace motion, and PNG/JPEG/WebP/SVG/WebM export. Accept plain-language requirements or pasted Mermaid flowchart, sequenceDiagram, and stateDiagram input; inspect repository evidence when the diagram must reflect real code. Use when the user asks to visualize system architecture, infrastructure, cloud/security/network topology, technical workflows, API call sequences, request lifecycles, data pipelines, ETL/ELT, data lineage, state machines, or to convert/beautify Mermaid. Also use for Archify, interactive or animated HTML diagrams, インタラクティブ図, 動く構成図. When an editable .drawio file is required, use the draw.io workflow instead and treat Archify HTML only as a supplement.

The English summary is being prepared. Showing the original description.

aktsmm/Agent-Skills26Updated Oct 18, 2026

ascii-banner

Free

Generate ASCII-art text banners for terminals, READMEs, and CLI startup messages. Use when a user asks for an ASCII banner, text art, バナー, アスキーアート, or a decorative header for a script/README.

The English summary is being prepared. Showing the original description.

aktsmm/Agent-Skills26Updated Oct 18, 2026

azure-advisor-report

Free

Generate Azure environment monthly report (Markdown + PowerPoint) from Azure Advisor and Cost Management API. Use when creating Advisor recommendations report, cost trend analysis, or security/reliability assessment for a customer subscription. Triggers on 'Advisor report', 'Azure monthly report', 'subscription report', 'Azure環境レポート', '簡易月次レポート'.

The English summary is being prepared. Showing the original description.

aktsmm/Agent-Skills26Updated Oct 18, 2026

See all skills by aktsmm

Report a problem with this skill