Authors and runs Giskard LLM scans - adversarial test-case generation for LLM applications via `giskard.scan(model)` covering 7 vulnerability categories (hallucination, harmful_content, prompt_injection, sensitive_information_disclosure, stereotypes, robustness, basic_sycophancy); wraps any callable model behind `giskard.Model(model_predict, model_type="text_generation", ...)`; emits HTML report. Use when the user needs adversarial / red-team coverage on top of functional eval suites.
日本語の概要は準備中です。原文の説明を表示しています。
実行スコア 未検証
インストール方法を見るインストールする前に、エージェントに与えられる指示の中身を確認できます。
Distinct from functional eval frameworks (Promptfoo, DeepEval, Ragas), Giskard's value (per gk-gh) is adversarial test generation - auto-generates inputs designed to break LLMs along documented vulnerability dimensions, then reports findings in a triageable HTML report.
Important version note (2026-05-06): per gk-gh, "Giskard
v2 is no longer actively maintained. The current v3 focus is on
giskard-checks for evaluations, while vulnerability scanning and
RAG evaluation still rely on Giskard v2." This skill targets v2 LLM
scanning; pin >2,<3 per the install command.
Per gk-gh (v2-pinned):
pip install "giskard[llm]>2,<3"
The [llm] extra pulls in the LLM scan dependencies.
Per gk-gh (verbatim quickstart):
import giskard
import pandas as pd
def model_predict(df: pd.DataFrame):
"""The function takes a DataFrame and must return a list of outputs (one per row)."""
return [my_llm_chain.run({"query": question}) for question in df["question"]]
giskard_model = giskard.Model(
model=model_predict,
model_type="text_generation",
name="My LLM Application",
description="A question answering assistant",
feature_names=["question"],
)
The description field steers Giskard's adversarial generator -
make it specific (e.g., "A question answering assistant for medical
guidance" vs the generic "A QA assistant"). Better description ⇒
better-targeted adversarial inputs.
scan_results = giskard.scan(giskard_model)
display(scan_results)
(Per gk-gh.)
In a Jupyter notebook this renders the report inline. To export:
scan_results.to_html("giskard-report.html")
Per gk-gh the v2 LLM scan covers these detector categories:
| Detector | What it tries to surface |
|---|---|
hallucination | Generated content not grounded in inputs / facts |
harmful_content | Toxic, dangerous, or harmful generation |
prompt_injection | Inputs that override system instructions |
sensitive_information_disclosure | PII / credentials / system prompt leakage |
stereotypes | Discriminatory / stereotyped output by protected attribute |
robustness | Brittleness to small input perturbations |
basic_sycophancy | Agreeing with falsehoods to please the user |
Detector selection (subset run):
scan_results = giskard.scan(
giskard_model,
only=["hallucination", "prompt_injection"],
)
(Per the scan() API; consult docs.giskard.ai for the
exact parameter list per Giskard release.)
After a scan surfaces issues, Giskard can synthesize tests for regression coverage:
test_suite = scan_results.generate_test_suite("My LLM Test Suite")
test_suite.run()
This produces deterministic regression tests from the failing adversarial prompts found by the scan - re-run on every PR to prevent regression on previously surfaced vulnerabilities.
Giskard does not ship a first-party CI action; pattern:
python -m my_giskard_scan_script # produces giskard-report.html
Then upload as a CI artifact:
- uses: actions/upload-artifact@v4
with: { name: giskard-report, path: giskard-report.html }
For PR-blocking gating, parse scan_results for severity and
fail CI if any critical findings appear:
critical = [issue for issue in scan_results.issues if issue.level == "major"]
if critical:
sys.exit(1)
| Anti-pattern | Why it fails | Fix |
|---|---|---|
Generic description= field | Adversarial generator produces off-target inputs; many false positives | Specific description (Step 2) |
| Run scan once, never regenerate test suite | Vulnerabilities resurface in new code | Regenerate test suite per release (Step 5) |
| Skip Step 5 - only rely on scans | No regression protection between scans | Always synthesize the test suite |
| Pin Giskard but not the judge-LLM provider | Judge-model drift causes flake | Pin both in CI env |
giskard-checks. Track upstream before
greenlighting new investment.text_generation and text_classification model types
in v2; multi-modal scanning lives elsewhere in the Giskard
ecosystem.まだレビューはありません。使ってみた感想をお寄せください。
ログインなしで書き込めます。使ってみた感想、質問、動かなかった環境などを気軽にどうぞ。
まだコメントはありません。最初のコメントを書いてみませんか。
Builds a CI gate that fails the build on **new** WCAG / a11y violations introduced by a PR while grandfathering pre-existing violations on a per-rule / per-page baseline. Aggregates verdicts from axe-core / pa11y / Lighthouse a11y / WAVE / IBM Equal Access scans. Use when a project has accumulated a11y debt and a strict "zero violations" gate would block every PR - the ratchet pattern lets the team ship while preventing regressions.
日本語の概要は準備中です。原文の説明を表示しています。
Workflow skill that builds an A/B-test validity checklist from an experiment proposal, walking the canonical design-correctness gates - pre-registered OEC/power/guardrails, randomization unit + SRM check, assignment integrity, telemetry, peeking discipline, novelty/primacy, post-experiment SRM re-check - into a per-experiment checklist + sign-off form. Use when launching, auditing, or governing an experiment. For pitfall mechanics (guardrails, peeking) see experiment-results-interpreter's references; to read an already-valid result use experiment-results-interpreter; for per-SDK harness tests use experiment-sdk-testing - this gates DESIGN, not SDK code.
日本語の概要は準備中です。原文の説明を表示しています。
The single install-flow skill: the reference contract for the PWA install flow (installability gate fields, `beforeinstallprompt` handshake, per-platform paths - Android WebAPK / iOS Share menu / Firefox no-op - and the `display-mode` post-install signal, in references/install-flow-reference.md) plus the build-an-X workflow that emits the Add-to-Home-Screen suite. Walks the four-stage timeline, emitting one test per gate cell per [web.dev/articles/install-criteria][install-criteria], the deferred-prompt → `prompt()` → `userChoice` chain per [web.dev/articles/customize-install][customize-install], the iOS Safari manual-metadata branch (`apple-touch-icon`) per [web.dev/learn/pwa/installation][learn-pwa], and the post-install `(display-mode: standalone)` MQ assertion. Output: a Playwright spec with per-stage cells plus a coverage matrix. Use when a PWA's manifest, icons, or install handler change, when install conversion drops at an unknown stage, or when triaging a flaky install assertion.
日本語の概要は準備中です。原文の説明を表示しています。
Build-an-X workflow that uses an LLM to map existing tests to spec sections - given a spec doc + the test suite, the LLM identifies which tests cover which sections, surfaces uncovered sections (gap), and recommends specific tests to add. Output is a coverage matrix per spec ID. Scope is mapping tests that already exist and naming the gaps, not authoring tests for new acceptance criteria. Use when a spec doc and a test suite both exist but nobody can say which requirements are actually covered - before a release sign-off, an audit, or a decision about where to spend the next round of test effort.
日本語の概要は準備中です。原文の説明を表示しています。
Generates tests from natural-language specs (acceptance criteria, user stories, requirements) using an LLM, with confidence scoring per test case (LLM self-assessment plus heuristics: assertion quality, naming, completeness), batching uncertain cases for human review, and integration with the team's existing test framework. Use when the user asks to generate unit tests from acceptance criteria, convert user stories to test cases, automate test creation from requirements, or augment a spec-driven test suite with AI-generated stubs that are then curated before merge.
日本語の概要は準備中です。原文の説明を表示しています。
Configures Allure Report (test-runner adapter install, `allure-results` directory wiring, `categories.json` for failure classification, `history-trend.json` retention via the copy-history-between-runs pattern), runs the Allure CLI to convert `allure-results` to a static HTML site, and uploads the report as a CI artifact. Use when the team needs richer test reporting than JUnit XML - step-level attachments, per-test history, retry tracking, and severity / epic / feature labeling across framework-agnostic adapters (pytest, Jest, JUnit, TestNG, NUnit, Mocha). As a rich static HTML report generator, it is the open-source alternative to the sunset ExtentReports (JVM/.NET per-test HTML narrative); for hosted cross-run flakiness analytics rather than a static per-run report use currents-integration.
日本語の概要は準備中です。原文の説明を表示しています。