本文へ移動
cccskills
無料GitHub で公開

rag-evaluation-matrix

Use when comparing basic, enhanced, GraphRAG, or agentic RAG designs, evaluating domain-specific RAG quality, tuning retrieval components, or deciding whether agentic RAG is worth its cost.

インストール方法を見る

含まれるファイル(3)

  • SKILL.md3.0 KB
  • references/rag-metrics.md990 B
  • scripts/rag_eval_matrix.py1.6 KB

SKILL.md(原文)

インストールする前に、エージェントに与えられる指示の中身を確認できます。

RAG Evaluation Matrix

Do not choose agentic RAG by fashion. Compare retrieval designs against domain questions, answerability, correctness, cost, latency, and failure reasons.

Use When

  • The user asks whether to use basic RAG, enhanced RAG, GraphRAG, or agentic RAG.
  • A RAG system works on demos but fails on domain-specific questions.
  • You need to compare embedding models, rerankers, chunking, query rewriting, or tool orchestration.
  • LLM-as-judge scores need alignment with human review.

Evaluation Flow

  1. Build a domain question set with answerable, unanswerable, ambiguous, multi-hop, and adversarial cases.
  2. For each candidate pipeline, log retrieved evidence, final answer, citations, latency, token usage, and cost.
  3. Score answer correctness and answerability separately.
  4. Add evidence quality checks: citation support, contradiction handling, source freshness, and missing-source diagnosis.
  5. Segment results by question type instead of reporting only one aggregate score.
  6. Inspect low-correctness failures and map them to retrieval, synthesis, tool orchestration, or corpus gaps.
  7. Pick the simplest design that meets quality, latency, and cost constraints.

Matrix

DimensionBasic RAGEnhanced RAGAgentic RAG
Best forStable FAQ and narrow corporaNoisy corpora, query mismatch, rerankingMulti-step, ambiguous, tool-rich tasks
Main riskWeak recall and unsupported answersPipeline complexityCost, latency, loops, tool misuse
Eval focusRetrieval recall and citation supportComponent ablationsTrajectory, action choice, stopping behavior
Ship gateCorrectness and answerability meet thresholdAblation proves each module helpsAgentic gains justify extra cost

Script

Use the helper to combine per-run JSON metrics into a decision table:

python scripts/rag_eval_matrix.py results/*.json

Expected JSON fields: pipeline, question_type, correct, answerable_correct, latency_ms, cost_usd.

Common Mistakes

MistakeFix
Optimizing average score onlyBreak down by question type and domain
Judging unanswerable questions as wrong by defaultScore answerability separately
Skipping human calibrationSample judge disagreements and tune rubrics
Choosing agentic RAG without ablationCompare against enhanced RAG at equal budget
Ignoring failure reasonsClassify each miss before tuning

References

レビュー

まだレビューはありません。使ってみた感想をお寄せください。

同じリポジトリのスキル

概要と使いどころ

Use when long-lived agents, project memories, system prompts, or playbooks are repeatedly summarized, rewritten, or optimized and risk losing domain detail, provenance, or execution learnings.

日本語の概要は準備中です。原文の説明を表示しています。

mouadja02/skills102026年10月16日 更新

Use this skill when the user explicitly asks to map, document, or onboard into an existing codebase. Trigger for prompts like "map this codebase", "document this architecture", "onboard me to this repo", or "create codebase docs". Do not trigger for routine feature implementation, bug fixes, or narrow code edits unless the user asks for repository-level discovery.

日本語の概要は準備中です。原文の説明を表示しています。

mouadja02/skills102026年10月16日 更新

Adds educational comments to a code file so it becomes a learning resource, asking for the file (with a list of close matches) if none is given. Use when preparing code for learners, onboarding, or teaching.

日本語の概要は準備中です。原文の説明を表示しています。

mouadja02/skills102026年10月16日 更新

Write, debug, and optimize Adobe Illustrator automation scripts using ExtendScript (JavaScript/JSX). Use when creating or modifying scripts that manipulate documents, layers, paths, text frames, colors, symbols, artboards, or any Illustrator DOM objects. Covers the complete JavaScript object model, coordinate system, measurement units, export workflows, and scripting best practices.

日本語の概要は準備中です。原文の説明を表示しています。

mouadja02/skills102026年10月16日 更新

Patterns and techniques for adding governance, safety, and trust controls to AI agent systems. Use this skill when: - Building AI agents that call external tools (APIs, databases, file systems) - Implementing policy-based access controls for agent tool usage - Adding semantic intent classification to detect dangerous prompts - Creating trust scoring systems for multi-agent workflows - Building audit trails for agent actions and decisions - Enforcing rate limits, content filters, or tool restrictions on agents - Working with any agent framework (PydanticAI, CrewAI, OpenAI Agents, LangChain, AutoGen)

日本語の概要は準備中です。原文の説明を表示しています。

mouadja02/skills102026年10月16日 更新

Check any AI agent codebase against the OWASP Agentic Security Initiative (ASI) Top 10 risks. Use this skill when: - Evaluating an agent system's security posture before production deployment - Running a compliance check against OWASP ASI 2026 standards - Mapping existing security controls to the 10 agentic risks - Generating a compliance report for security review or audit - Comparing agent framework security features against the standard - Any request like "is my agent OWASP compliant?", "check ASI compliance", or "agentic security audit"

日本語の概要は準備中です。原文の説明を表示しています。

mouadja02/skills102026年10月16日 更新

mouadja02 のスキルをすべて見る

このスキルの問題を報告する