Authors and runs DeepEval - pytest-native LLM eval framework with `LLMTestCase` (input + actual_output + expected_output + retrieval_context) and ~11 built-in metrics (G-Eval, Answer-Relevancy, Faithfulness, Contextual-Recall / Precision / Relevancy, Hallucination, Bias, Toxicity, Summarization, JSON-Correctness); runs via `deepeval test run {file.py}` with `assert_test()` per test or `evaluate()` for batch; integrates Confident-AI dashboard. Use when the user prefers pytest workflow, works with RAG and needs faithfulness/contextual metrics out-of-the-box, or wants a managed dashboard.
日本語の概要は準備中です。原文の説明を表示しています。
実行スコア 未検証
インストール方法を見るインストールする前に、エージェントに与えられる指示の中身を確認できます。
Per de-start, DeepEval is "an open-source LLM eval
package" enabling "evaluation of LLM applications locally through
test cases and metrics." The model: each test constructs an
LLMTestCase, applies one or more Metric instances, and either
asserts (assert_test) or batch-evaluates (evaluate). Pytest
discovery + reporting works unchanged.
pip install -U deepeval
Optional Confident-AI login (for dashboard):
deepeval login
Per de-start: after login "Confident AI will generate testing reports and automate regression testing whenever you run a test run."
Per de-gh (verbatim quickstart):
import pytest
from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams
def test_case():
correctness_metric = GEval(
name="Correctness",
criteria="Determine if the 'actual output' is correct based on the 'expected output'.",
evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
threshold=0.5
)
test_case = LLMTestCase(
input="What if these shoes don't fit?",
actual_output="You have 30 days to get a full refund at no extra cost.",
expected_output="We offer a 30-day full refund at no extra costs.",
retrieval_context=["All customers are eligible for a 30 day full refund at no extra costs."]
)
assert_test(test_case, [correctness_metric])
Run it:
deepeval test run test_chatbot.py
(Per de-gh.)
Per de-start, LLMTestCase fields:
| Field | Required | Notes |
|---|---|---|
input | yes | The user prompt / query |
actual_output | yes | What the LLM produced |
expected_output | optional | Reference answer (used by metrics that compare) |
retrieval_context | optional | List of retrieved chunks for RAG metrics |
context | optional | Ground-truth context for hallucination metric |
Per de-gh the available metrics include:
| Metric | Use |
|---|---|
GEval | Custom rubric-based scoring (LLM-as-judge with chain-of-thought) |
AnswerRelevancyMetric | Does actual_output answer input? |
FaithfulnessMetric | Does actual_output only state facts in retrieval_context? |
ContextualRecallMetric | Does retrieval_context contain enough info to produce expected_output? |
ContextualPrecisionMetric | Are relevant chunks ranked higher in retrieval_context? |
ContextualRelevancyMetric | Are chunks in retrieval_context relevant to input? |
HallucinationMetric | Does actual_output contradict context? |
BiasMetric | Bias detection in actual_output |
ToxicityMetric | Toxic-content detection |
SummarizationMetric | Summary quality vs source |
JsonCorrectnessMetric | Valid + schema-conformant JSON output |
Each metric takes a threshold parameter; the test passes if score
≥ threshold.
GEval is the universal escape hatch when no built-in metric fits:
from deepeval.metrics import GEval
from deepeval.test_case import SingleTurnParams
professionalism = GEval(
name="Professionalism",
criteria="Determine if the response uses professional language without slang or contractions.",
evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT],
threshold=0.7,
)
The criteria string is the rubric; the judge model evaluates and
returns a 0 - 1 score with reasoning attached.
For dataset-driven runs without pytest:
from deepeval import evaluate
evaluate(test_cases=[case1, case2, case3], metrics=[g_eval, faithfulness])
Returns scores per metric per case; useful for regression sweeps across a CSV/JSONL of historical inputs.
deepeval test run tests/llm/ --run-async --workers 4
# pytest exit code propagates: nonzero if any assert_test fails
Combine with Confident-AI for a dashboard view of run history; Confident-AI is the company behind DeepEval (per de-gh).
| Anti-pattern | Why it fails | Fix |
|---|---|---|
threshold=0.0 on every metric | Tests never fail; eval theater | Pick real thresholds (0.5 - 0.8 typical) |
Hallucination metric without context | Metric has nothing to compare against | Always pass context (Step 3) |
Faithfulness metric without retrieval_context | Same problem | Pass retrieval chunks (Step 3) |
| Custom GEval criteria too vague | Judge produces inconsistent scores | Concrete criteria with examples (Step 5) |
Skip --workers in CI | Sequential runs slow + costly | --workers 4 parallelization (Step 7) |
promptfoo-evaluation
is lower-friction.まだレビューはありません。使ってみた感想をお寄せください。
ログインなしで書き込めます。使ってみた感想、質問、動かなかった環境などを気軽にどうぞ。
まだコメントはありません。最初のコメントを書いてみませんか。
Builds a CI gate that fails the build on **new** WCAG / a11y violations introduced by a PR while grandfathering pre-existing violations on a per-rule / per-page baseline. Aggregates verdicts from axe-core / pa11y / Lighthouse a11y / WAVE / IBM Equal Access scans. Use when a project has accumulated a11y debt and a strict "zero violations" gate would block every PR - the ratchet pattern lets the team ship while preventing regressions.
日本語の概要は準備中です。原文の説明を表示しています。
Workflow skill that builds an A/B-test validity checklist from an experiment proposal, walking the canonical design-correctness gates - pre-registered OEC/power/guardrails, randomization unit + SRM check, assignment integrity, telemetry, peeking discipline, novelty/primacy, post-experiment SRM re-check - into a per-experiment checklist + sign-off form. Use when launching, auditing, or governing an experiment. For pitfall mechanics (guardrails, peeking) see experiment-results-interpreter's references; to read an already-valid result use experiment-results-interpreter; for per-SDK harness tests use experiment-sdk-testing - this gates DESIGN, not SDK code.
日本語の概要は準備中です。原文の説明を表示しています。
The single install-flow skill: the reference contract for the PWA install flow (installability gate fields, `beforeinstallprompt` handshake, per-platform paths - Android WebAPK / iOS Share menu / Firefox no-op - and the `display-mode` post-install signal, in references/install-flow-reference.md) plus the build-an-X workflow that emits the Add-to-Home-Screen suite. Walks the four-stage timeline, emitting one test per gate cell per [web.dev/articles/install-criteria][install-criteria], the deferred-prompt → `prompt()` → `userChoice` chain per [web.dev/articles/customize-install][customize-install], the iOS Safari manual-metadata branch (`apple-touch-icon`) per [web.dev/learn/pwa/installation][learn-pwa], and the post-install `(display-mode: standalone)` MQ assertion. Output: a Playwright spec with per-stage cells plus a coverage matrix. Use when a PWA's manifest, icons, or install handler change, when install conversion drops at an unknown stage, or when triaging a flaky install assertion.
日本語の概要は準備中です。原文の説明を表示しています。
Build-an-X workflow that uses an LLM to map existing tests to spec sections - given a spec doc + the test suite, the LLM identifies which tests cover which sections, surfaces uncovered sections (gap), and recommends specific tests to add. Output is a coverage matrix per spec ID. Scope is mapping tests that already exist and naming the gaps, not authoring tests for new acceptance criteria. Use when a spec doc and a test suite both exist but nobody can say which requirements are actually covered - before a release sign-off, an audit, or a decision about where to spend the next round of test effort.
日本語の概要は準備中です。原文の説明を表示しています。
Generates tests from natural-language specs (acceptance criteria, user stories, requirements) using an LLM, with confidence scoring per test case (LLM self-assessment plus heuristics: assertion quality, naming, completeness), batching uncertain cases for human review, and integration with the team's existing test framework. Use when the user asks to generate unit tests from acceptance criteria, convert user stories to test cases, automate test creation from requirements, or augment a spec-driven test suite with AI-generated stubs that are then curated before merge.
日本語の概要は準備中です。原文の説明を表示しています。
Configures Allure Report (test-runner adapter install, `allure-results` directory wiring, `categories.json` for failure classification, `history-trend.json` retention via the copy-history-between-runs pattern), runs the Allure CLI to convert `allure-results` to a static HTML site, and uploads the report as a CI artifact. Use when the team needs richer test reporting than JUnit XML - step-level attachments, per-test history, retry tracking, and severity / epic / feature labeling across framework-agnostic adapters (pytest, Jest, JUnit, TestNG, NUnit, Mocha). As a rich static HTML report generator, it is the open-source alternative to the sunset ExtentReports (JVM/.NET per-test HTML narrative); for hosted cross-run flakiness analytics rather than a static per-run report use currents-integration.
日本語の概要は準備中です。原文の説明を表示しています。