Authors and runs Ragas - RAG-pipeline evaluation framework with metrics organized into RAG (Faithfulness, Response Relevancy, Context Precision/Recall, Context Entities Recall, Noise Sensitivity), Natural Language Comparison (Factual Correctness, Semantic Similarity, BLEU/ROUGE/CHRF/Exact Match), Agents/Tool-Use (Topic Adherence, Tool Call Accuracy/F1, Agent Goal Accuracy), General Purpose (Aspect Critic, Rubrics-based Scoring), Nvidia (Answer Accuracy, Context Relevance, Response Groundedness), and Summarization. Use when the user evaluates a RAG pipeline (retriever + generator) and needs the deepest metric variety in the OSS LLM-eval space.
日本語の概要は準備中です。原文の説明を表示しています。
実行スコア 未検証
インストール方法を見るインストールする前に、エージェントに与えられる指示の中身を確認できます。
The model: assemble a dataset (question + answer + retrieval
contexts + ground truth), import the metrics relevant to the
evaluation goal, run evaluate(), and inspect per-metric per-row
scores (per rg-gh).
For non-RAG prompt evals, prefer promptfoo-evaluation.
For pytest-native LLM evals with a managed dashboard, prefer
deepeval-evaluation.
Per rg-gh:
pip install ragas
Or from source:
pip install git+https://github.com/explodinggradients/ragas
Per rg-gh (verbatim):
import asyncio
from openai import AsyncOpenAI
from ragas.metrics import DiscreteMetric
from ragas.llms import llm_factory
# Setup your LLM
client = AsyncOpenAI()
llm = llm_factory("gpt-4o", client=client)
# Create a custom aspect evaluator
metric = DiscreteMetric(
name="summary_accuracy",
allowed_values=["accurate", "inaccurate"],
prompt="""Evaluate if the summary is accurate and captures
key information.
Response: {response}
Answer with only 'accurate' or 'inaccurate'."""
)
# Score your application's output
async def main():
score = await metric.ascore(
llm=llm,
response="The summary of the text is..."
)
print(f"Score: {score.value}")
print(f"Reason: {score.reason}")
if __name__ == "__main__":
asyncio.run(main())
DiscreteMetric is the pattern for custom rubric-based scoring;
the built-in metrics in Step 3 follow a similar shape but are
preconfigured.
Ragas ships 30+ metrics across RAG, Natural Language Comparison, Agents/Tool-Use, SQL, General Purpose, Nvidia, and Summarization families. The RAG core: Faithfulness (claims grounded in retrieved context), Response Relevancy, Context Precision, and Context Recall. Pick 3 - 5 per pipeline.
Full per-family catalog with each metric's use: references/metrics.md, sourced from docs.ragas.io/en/stable/concepts/metrics/available_metrics/.
Ragas accepts a Hugging Face Dataset or pandas.DataFrame with
columns matching the metrics being run:
| Column | Required by |
|---|---|
question | All RAG metrics |
answer | Response Relevancy, Faithfulness, NL Comparison |
contexts (list of strings) | Context Precision/Recall, Faithfulness |
ground_truth | Context Recall, Factual Correctness, Answer Accuracy |
reference_contexts | Context-comparison metrics |
See the per-metric pages on docs.ragas.io for exact required-column lists.
Ragas integrates with LangChain + LlamaIndex retrieval pipelines -
the integration code captures contexts from the retriever and
answer from the generator into the evaluation dataset
automatically. Consult the per-framework integration docs on
docs.ragas.io when wiring; APIs evolve faster than this
skill body and the canonical doc is the source of truth.
Ragas does not ship a first-party CI action. Pattern: run
evaluate() in a pytest fixture or a CLI script, compare per-metric
scores against thresholds, fail CI on regression.
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy
result = evaluate(dataset, metrics=[faithfulness, answer_relevancy])
assert result["faithfulness"] >= 0.85
assert result["answer_relevancy"] >= 0.80
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Run all 30+ metrics on every PR | Cost + latency explode | Pick 3 - 5 metrics per pipeline (Step 3) |
Faithfulness without contexts column | Metric returns NaN / errors | Pass contexts per dataset spec (Step 4) |
| Pin nothing | Ragas + judge-model versions both drift | Pin both in requirements + CI env |
| Skip Aspect Critic for product-specific concerns | Built-in metrics miss the requirement | Custom Aspect Critic + rubric (Step 3) |
pip install ragas[multimodal]); check the per-metric doc on
docs.ragas.io.deepeval-evaluation,
promptfoo-evaluation -
sister tools (different framework styles)まだレビューはありません。使ってみた感想をお寄せください。
ログインなしで書き込めます。使ってみた感想、質問、動かなかった環境などを気軽にどうぞ。
まだコメントはありません。最初のコメントを書いてみませんか。
Builds a CI gate that fails the build on **new** WCAG / a11y violations introduced by a PR while grandfathering pre-existing violations on a per-rule / per-page baseline. Aggregates verdicts from axe-core / pa11y / Lighthouse a11y / WAVE / IBM Equal Access scans. Use when a project has accumulated a11y debt and a strict "zero violations" gate would block every PR - the ratchet pattern lets the team ship while preventing regressions.
日本語の概要は準備中です。原文の説明を表示しています。
Workflow skill that builds an A/B-test validity checklist from an experiment proposal, walking the canonical design-correctness gates - pre-registered OEC/power/guardrails, randomization unit + SRM check, assignment integrity, telemetry, peeking discipline, novelty/primacy, post-experiment SRM re-check - into a per-experiment checklist + sign-off form. Use when launching, auditing, or governing an experiment. For pitfall mechanics (guardrails, peeking) see experiment-results-interpreter's references; to read an already-valid result use experiment-results-interpreter; for per-SDK harness tests use experiment-sdk-testing - this gates DESIGN, not SDK code.
日本語の概要は準備中です。原文の説明を表示しています。
The single install-flow skill: the reference contract for the PWA install flow (installability gate fields, `beforeinstallprompt` handshake, per-platform paths - Android WebAPK / iOS Share menu / Firefox no-op - and the `display-mode` post-install signal, in references/install-flow-reference.md) plus the build-an-X workflow that emits the Add-to-Home-Screen suite. Walks the four-stage timeline, emitting one test per gate cell per [web.dev/articles/install-criteria][install-criteria], the deferred-prompt → `prompt()` → `userChoice` chain per [web.dev/articles/customize-install][customize-install], the iOS Safari manual-metadata branch (`apple-touch-icon`) per [web.dev/learn/pwa/installation][learn-pwa], and the post-install `(display-mode: standalone)` MQ assertion. Output: a Playwright spec with per-stage cells plus a coverage matrix. Use when a PWA's manifest, icons, or install handler change, when install conversion drops at an unknown stage, or when triaging a flaky install assertion.
日本語の概要は準備中です。原文の説明を表示しています。
Build-an-X workflow that uses an LLM to map existing tests to spec sections - given a spec doc + the test suite, the LLM identifies which tests cover which sections, surfaces uncovered sections (gap), and recommends specific tests to add. Output is a coverage matrix per spec ID. Scope is mapping tests that already exist and naming the gaps, not authoring tests for new acceptance criteria. Use when a spec doc and a test suite both exist but nobody can say which requirements are actually covered - before a release sign-off, an audit, or a decision about where to spend the next round of test effort.
日本語の概要は準備中です。原文の説明を表示しています。
Generates tests from natural-language specs (acceptance criteria, user stories, requirements) using an LLM, with confidence scoring per test case (LLM self-assessment plus heuristics: assertion quality, naming, completeness), batching uncertain cases for human review, and integration with the team's existing test framework. Use when the user asks to generate unit tests from acceptance criteria, convert user stories to test cases, automate test creation from requirements, or augment a spec-driven test suite with AI-generated stubs that are then curated before merge.
日本語の概要は準備中です。原文の説明を表示しています。
Configures Allure Report (test-runner adapter install, `allure-results` directory wiring, `categories.json` for failure classification, `history-trend.json` retention via the copy-history-between-runs pattern), runs the Allure CLI to convert `allure-results` to a static HTML site, and uploads the report as a CI artifact. Use when the team needs richer test reporting than JUnit XML - step-level attachments, per-test history, retry tracking, and severity / epic / feature labeling across framework-agnostic adapters (pytest, Jest, JUnit, TestNG, NUnit, Mocha). As a rich static HTML report generator, it is the open-source alternative to the sunset ExtentReports (JVM/.NET per-test HTML narrative); for hosted cross-run flakiness analytics rather than a static per-run report use currents-integration.
日本語の概要は準備中です。原文の説明を表示しています。