Authors and runs Promptfoo evals for LLM prompts and RAG pipelines - wires `promptfooconfig.yaml` providers + prompts + tests + assertions (deterministic `equals` / `contains` / `is-json` / `regex`, semantic `similar`, model-graded `llm-rubric` / `factuality` / `g-eval`, performance `latency` / `cost`, custom `javascript` / `python`), runs `npx promptfoo eval`, views HTML report via `promptfoo view`, and integrates CI for regression gating. Use when the user runs Promptfoo, asks about prompt regression suites, or needs an eval-driven workflow for LLM-backed features.
日本語の概要は準備中です。原文の説明を表示しています。
実行スコア 未検証
インストール方法を見るインストールする前に、エージェントに与えられる指示の中身を確認できます。
A promptfooconfig.yaml declares providers (LLMs under test),
prompts (templates with {{var}} placeholders), tests
(input variables + assertions); promptfoo eval runs the
cross-product (per pf-config).
promptfooconfig.yaml or the user wants to
author one.Per github.com/promptfoo/promptfoo:
npm install -g promptfoo
# or
brew install promptfoo
# or one-off
npx promptfoo@latest
Per pf-gh: "Node.js 20.20+ or 22.22+ is required for npm and npx usage."
Initialize from a built-in example, then run:
promptfoo init --example getting-started
cd getting-started
promptfoo eval
promptfoo view # opens HTML report
(Commands per pf-gh.)
A minimal promptfooconfig.yaml per pf-config:
prompts:
- file://prompt1.txt
providers:
- openai:gpt-5-mini
- anthropic:claude-haiku-4-5
tests:
- vars:
language: French
input: Hello world
assert:
- type: contains-json
Provider syntax <vendor>:<model> works for OpenAI, Anthropic,
Vertex (vertex:gemini-2.0-flash-exp), Ollama (ollama:llama2), and
30+ others (per pf-config).
Vars interpolate into prompts using {{variable}}. Arrays create
combinations (per pf-config):
tests:
- vars:
language: [French, German, Spanish]
input: [Hello world, Good morning]
This generates 6 test rows (3 languages × 2 inputs). Vars can also load from files:
tests:
- vars:
var3: file://path/to/var3.txt
context: file://fetch_from_vector_database.py
Per promptfoo.dev/docs/configuration/expected-outputs/:
Deterministic (string + structure):
| Type | Example |
|---|---|
equals | value: 'expected string' |
contains / icontains | value: 'substring' (case-sensitive / insensitive) |
starts-with | value: 'prefix' |
contains-any / contains-all | value: ['a', 'b'] |
regex | value: '^pattern$' |
is-json / contains-json | (validates / locates JSON) |
is-sql / contains-sql | (SQL syntax) |
is-xml / is-html / contains-xml / contains-html | (markup) |
is-valid-openai-tools-call / is-valid-openai-function-call | (tool calls) |
Custom logic:
- type: javascript
value: 'output.length > 10'
- type: python
value: 'file://script.py'
Text-quality metrics (default thresholds per pf-asserts):
| Type | Default threshold |
|---|---|
rouge-n | 0.75 |
bleu | 0.5 |
gleu | 0.5 |
meteor | 0.5 |
levenshtein | 5 (edit distance max) |
Performance + cost:
- type: latency
threshold: 200 # milliseconds
- type: cost
threshold: 0.001 # dollars per response
Model-graded (LLM-as-judge):
| Type | Use |
|---|---|
llm-rubric | Free-form rubric: value: 'Is helpful and accurate' |
model-graded-closedqa | Closed-QA evaluation method |
factuality | Compares against reference facts |
g-eval | Chain-of-thought scoring |
answer-relevance | Checks output relates to query |
context-faithfulness / context-recall / context-relevance | RAG-specific |
Semantic similarity:
- type: similar
value: 'reference text'
threshold: 0.8 # cosine similarity via embeddings
Negation: all deterministic assertions support a not- prefix
(not-equals, not-contains, not-regex, etc.) per pf-asserts.
Shared assertions/vars across all tests:
defaultTest:
vars:
shared_var: 'shared content'
assert:
- type: llm-rubric
value: does not describe self as AI
options:
provider: openai:gpt-5-mini-0613
tests:
- vars:
unique_var: value1
(Per pf-config.)
Modify LLM output before assertions execute (per pf-config):
tests:
- vars:
body: Hello world
options:
transform: output.toUpperCase()
Or load from a file:
options:
transform: file://transform.js:customTransform
Group assertions and require a threshold percentage to pass (per pf-asserts):
assert:
- type: assert-set
threshold: 0.5 # 50% of grouped asserts must pass
assert:
- type: cost
threshold: 0.001
- type: latency
threshold: 200
Provider API keys via env vars (per pf-gh: export OPENAI_API_KEY=sk-abc123).
GitHub Actions pattern (per promptfoo.dev/docs/integrations/github-action):
- uses: promptfoo/promptfoo-action@v2
with:
openai-api-key: ${{ secrets.OPENAI_API_KEY }}
config: 'promptfooconfig.yaml'
cache-path: '~/.cache/promptfoo'
use-config-prompts: false
no-share: true
promptfoo-version: 'latest'
The action posts a PR comment with regression diff vs the base branch. Caching reuses LLM responses for unchanged tests (per promptfoo.dev/docs/configuration/caching/).
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Only deterministic assertions on creative outputs | LLM responses vary; rigid asserts produce flake | Use llm-rubric or similar (Step 4) |
| Single provider in config | Misses cross-provider regression | At least 2 providers per eval (Step 2) |
| No cost / latency caps | Eval cost balloons per PR | cost + latency asserts (Step 4) |
assert-set with threshold: 0 | Effectively disables grouping | Pick a real threshold (Step 7) |
| Fresh provider model on every run without pinning | Output drifts when vendor updates model | Pin specific snapshot (e.g., openai:gpt-5-mini-0613) |
| Skip caching in CI | Re-runs every test on every PR | cache-path in GHA action (Step 8) |
まだレビューはありません。使ってみた感想をお寄せください。
ログインなしで書き込めます。使ってみた感想、質問、動かなかった環境などを気軽にどうぞ。
まだコメントはありません。最初のコメントを書いてみませんか。
Builds a CI gate that fails the build on **new** WCAG / a11y violations introduced by a PR while grandfathering pre-existing violations on a per-rule / per-page baseline. Aggregates verdicts from axe-core / pa11y / Lighthouse a11y / WAVE / IBM Equal Access scans. Use when a project has accumulated a11y debt and a strict "zero violations" gate would block every PR - the ratchet pattern lets the team ship while preventing regressions.
日本語の概要は準備中です。原文の説明を表示しています。
Workflow skill that builds an A/B-test validity checklist from an experiment proposal, walking the canonical design-correctness gates - pre-registered OEC/power/guardrails, randomization unit + SRM check, assignment integrity, telemetry, peeking discipline, novelty/primacy, post-experiment SRM re-check - into a per-experiment checklist + sign-off form. Use when launching, auditing, or governing an experiment. For pitfall mechanics (guardrails, peeking) see experiment-results-interpreter's references; to read an already-valid result use experiment-results-interpreter; for per-SDK harness tests use experiment-sdk-testing - this gates DESIGN, not SDK code.
日本語の概要は準備中です。原文の説明を表示しています。
The single install-flow skill: the reference contract for the PWA install flow (installability gate fields, `beforeinstallprompt` handshake, per-platform paths - Android WebAPK / iOS Share menu / Firefox no-op - and the `display-mode` post-install signal, in references/install-flow-reference.md) plus the build-an-X workflow that emits the Add-to-Home-Screen suite. Walks the four-stage timeline, emitting one test per gate cell per [web.dev/articles/install-criteria][install-criteria], the deferred-prompt → `prompt()` → `userChoice` chain per [web.dev/articles/customize-install][customize-install], the iOS Safari manual-metadata branch (`apple-touch-icon`) per [web.dev/learn/pwa/installation][learn-pwa], and the post-install `(display-mode: standalone)` MQ assertion. Output: a Playwright spec with per-stage cells plus a coverage matrix. Use when a PWA's manifest, icons, or install handler change, when install conversion drops at an unknown stage, or when triaging a flaky install assertion.
日本語の概要は準備中です。原文の説明を表示しています。
Build-an-X workflow that uses an LLM to map existing tests to spec sections - given a spec doc + the test suite, the LLM identifies which tests cover which sections, surfaces uncovered sections (gap), and recommends specific tests to add. Output is a coverage matrix per spec ID. Scope is mapping tests that already exist and naming the gaps, not authoring tests for new acceptance criteria. Use when a spec doc and a test suite both exist but nobody can say which requirements are actually covered - before a release sign-off, an audit, or a decision about where to spend the next round of test effort.
日本語の概要は準備中です。原文の説明を表示しています。
Generates tests from natural-language specs (acceptance criteria, user stories, requirements) using an LLM, with confidence scoring per test case (LLM self-assessment plus heuristics: assertion quality, naming, completeness), batching uncertain cases for human review, and integration with the team's existing test framework. Use when the user asks to generate unit tests from acceptance criteria, convert user stories to test cases, automate test creation from requirements, or augment a spec-driven test suite with AI-generated stubs that are then curated before merge.
日本語の概要は準備中です。原文の説明を表示しています。
Configures Allure Report (test-runner adapter install, `allure-results` directory wiring, `categories.json` for failure classification, `history-trend.json` retention via the copy-history-between-runs pattern), runs the Allure CLI to convert `allure-results` to a static HTML site, and uploads the report as a CI artifact. Use when the team needs richer test reporting than JUnit XML - step-level attachments, per-test history, retry tracking, and severity / epic / feature labeling across framework-agnostic adapters (pytest, Jest, JUnit, TestNG, NUnit, Mocha). As a rich static HTML report generator, it is the open-source alternative to the sunset ExtentReports (JVM/.NET per-test HTML narrative); for hosted cross-run flakiness analytics rather than a static per-run report use currents-integration.
日本語の概要は準備中です。原文の説明を表示しています。