본문으로 건너뛰기
cccskills
무료GitHub에 공개

rag-evaluation-matrix

Use when comparing basic, enhanced, GraphRAG, or agentic RAG designs, evaluating domain-specific RAG quality, tuning retrieval components, or deciding whether agentic RAG is worth its cost.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

연동 도구Hugging Face
설치 방법 보기

포함된 파일(3)

  • SKILL.md3.0 KB
  • references/rag-metrics.md990 B
  • scripts/rag_eval_matrix.py1.6 KB

SKILL.md(원문)

설치하기 전에 에이전트에게 전달되는 지시 내용을 확인할 수 있습니다.

RAG Evaluation Matrix

Do not choose agentic RAG by fashion. Compare retrieval designs against domain questions, answerability, correctness, cost, latency, and failure reasons.

Use When

  • The user asks whether to use basic RAG, enhanced RAG, GraphRAG, or agentic RAG.
  • A RAG system works on demos but fails on domain-specific questions.
  • You need to compare embedding models, rerankers, chunking, query rewriting, or tool orchestration.
  • LLM-as-judge scores need alignment with human review.

Evaluation Flow

  1. Build a domain question set with answerable, unanswerable, ambiguous, multi-hop, and adversarial cases.
  2. For each candidate pipeline, log retrieved evidence, final answer, citations, latency, token usage, and cost.
  3. Score answer correctness and answerability separately.
  4. Add evidence quality checks: citation support, contradiction handling, source freshness, and missing-source diagnosis.
  5. Segment results by question type instead of reporting only one aggregate score.
  6. Inspect low-correctness failures and map them to retrieval, synthesis, tool orchestration, or corpus gaps.
  7. Pick the simplest design that meets quality, latency, and cost constraints.

Matrix

DimensionBasic RAGEnhanced RAGAgentic RAG
Best forStable FAQ and narrow corporaNoisy corpora, query mismatch, rerankingMulti-step, ambiguous, tool-rich tasks
Main riskWeak recall and unsupported answersPipeline complexityCost, latency, loops, tool misuse
Eval focusRetrieval recall and citation supportComponent ablationsTrajectory, action choice, stopping behavior
Ship gateCorrectness and answerability meet thresholdAblation proves each module helpsAgentic gains justify extra cost

Script

Use the helper to combine per-run JSON metrics into a decision table:

python scripts/rag_eval_matrix.py results/*.json

Expected JSON fields: pipeline, question_type, correct, answerable_correct, latency_ms, cost_usd.

Common Mistakes

MistakeFix
Optimizing average score onlyBreak down by question type and domain
Judging unanswerable questions as wrong by defaultScore answerability separately
Skipping human calibrationSample judge disagreements and tune rubrics
Choosing agentic RAG without ablationCompare against enhanced RAG at equal budget
Ignoring failure reasonsClassify each miss before tuning

References

리뷰

아직 리뷰가 없습니다. 사용해 본 소감을 남겨 주세요.

댓글

로그인 없이 쓸 수 있습니다. 사용 후기, 질문, 작동하지 않은 환경 등을 편하게 남겨 주세요.

아직 댓글이 없습니다. 첫 댓글을 남겨 보세요.

보내면 바로 공개됩니다. 이메일 주소나 전화번호 같은 개인 정보는 쓰지 마세요. URL은 2개까지입니다.

0 / 1,000자

이 양식은 reCAPTCHA로 보호되며 Google 개인정보처리방침 및 서비스 약관이 적용됩니다.

같은 저장소의 스킬

Use when long-lived agents, project memories, system prompts, or playbooks are repeatedly summarized, rewritten, or optimized and risk losing domain detail, provenance, or execution learnings.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

mouadja02/skills102026년 10월 17일 업데이트

Use this skill when the user explicitly asks to map, document, or onboard into an existing codebase. Trigger for prompts like "map this codebase", "document this architecture", "onboard me to this repo", or "create codebase docs". Do not trigger for routine feature implementation, bug fixes, or narrow code edits unless the user asks for repository-level discovery.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

mouadja02/skills102026년 10월 17일 업데이트

Adds educational comments to a code file so it becomes a learning resource, asking for the file (with a list of close matches) if none is given. Use when preparing code for learners, onboarding, or teaching.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

mouadja02/skills102026년 10월 17일 업데이트

Write, debug, and optimize Adobe Illustrator automation scripts using ExtendScript (JavaScript/JSX). Use when creating or modifying scripts that manipulate documents, layers, paths, text frames, colors, symbols, artboards, or any Illustrator DOM objects. Covers the complete JavaScript object model, coordinate system, measurement units, export workflows, and scripting best practices.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

mouadja02/skills102026년 10월 17일 업데이트

Patterns and techniques for adding governance, safety, and trust controls to AI agent systems. Use this skill when: - Building AI agents that call external tools (APIs, databases, file systems) - Implementing policy-based access controls for agent tool usage - Adding semantic intent classification to detect dangerous prompts - Creating trust scoring systems for multi-agent workflows - Building audit trails for agent actions and decisions - Enforcing rate limits, content filters, or tool restrictions on agents - Working with any agent framework (PydanticAI, CrewAI, OpenAI Agents, LangChain, AutoGen)

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

mouadja02/skills102026년 10월 17일 업데이트

Check any AI agent codebase against the OWASP Agentic Security Initiative (ASI) Top 10 risks. Use this skill when: - Evaluating an agent system's security posture before production deployment - Running a compliance check against OWASP ASI 2026 standards - Mapping existing security controls to the 10 agentic risks - Generating a compliance report for security review or audit - Comparing agent framework security features against the standard - Any request like "is my agent OWASP compliant?", "check ASI compliance", or "agentic security audit"

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

mouadja02/skills102026년 10월 17일 업데이트

mouadja02의 스킬 모두 보기

이 스킬의 문제 신고