12 skills · sorted by relevance
Set up repeatable security tests for LLM apps with Promptfoo and DeepTeam. Connect them to automated build pipelines, detect renewed vulnerabilities to malicious instructions or data leakage, and compare results across model or prompt changes.
Configures and runs LLM evaluation using Promptfoo framework. Use when setting up prompt testing, creating evaluation configs (promptfooconfig.yaml), writing Python custom assertions, implementing llm-rubric for LLM-as-judge, or managing few-shot examples in prompts. Triggers on keywords like "promptfoo", "eval", "LLM evaluation", "prompt testing", or "model comparison".
The English summary is being prepared. Showing the original description.
Wire Promptfoo and DeepTeam into CI/CD for automated regression red-teaming of LLM apps against OWASP LLM Top 10 and OWASP Agentic presets, failing the build when jailbreak or injection vulnerabilities regress.
The English summary is being prepared. Showing the original description.
Build repeatable evaluations for AI agent responses, tool choices, and multi-step tasks. Use reference datasets and scoring criteria to compare changes, detect quality regressions, and set automated checks that must pass before deployment.
Testing and evaluation framework for LLM prompts and applications -- promptfooconfig.yaml, assertions, model-graded evals, red teaming, CI/CD integration, custom providers, and comparative evaluation
The English summary is being prepared. Showing the original description.
Assess the security of LLM applications and AI agents by tracing inputs, retrieved content, tool calls, and outputs. Covers prompt injection, where content carries unwanted instructions, along with permission misuse, memory poisoning, and supply-chain risks.
Manage golden dataset regression tests for LLM prompts using Promptfoo. Commands — init, add, run, report. Use when user says "regression", "golden dataset", "prompt regression", "eval run", "test my prompts", or after shipping prompt changes.
The English summary is being prepared. Showing the original description.
Use when red-teaming an agentic AI / LLM application — indirect & zero-click prompt injection, MCP tool poisoning, persistent memory poisoning, excessive-agency tool abuse, multi-turn jailbreaks, PyRIT/Garak/Promptfoo harnesses
The English summary is being prepared. Showing the original description.
Run AI agent and LLM evaluations in CI/CD pipelines — automated quality gates that fail the build when AI output quality drops. Use when someone asks to "test my AI agent", "add evals to CI", "catch prompt regressions", "compare models", "evaluate LLM output quality", "set up AI quality gates", or "benchmark my agent before deploying". Covers eval frameworks (Cobalt, Promptfoo, Braintrust), LLM-as-judge scoring, threshold-based assertions, and GitHub Actions integration.
The English summary is being prepared. Showing the original description.
Expert guide for Prompt Engineering, Chain-of-Thought, few-shot prompting, structured output, prompt injection defense, and automated AI evaluations & regression benchmarking (Promptfoo, DeepEval) / Panduan ahli rekayasa prompt dan evaluasi otomatis AI.
The English summary is being prepared. Showing the original description.
AI/LLM red-team assessment using the OWASP LLM Top 10 (2025) + OWASP AI Testing Guide (AITG v1, Nov 2025) frameworks, plus OWASP MCP Top 10 runtime testing for agentic/MCP targets. Tests prompt injection, jailbreaks, system prompt leakage, sensitive data extraction, excessive agency, improper output handling, model extraction, content bias, evasion, membership inference, MCP token exposure, MCP command injection, and more. Uses four tools in combination: FuzzyAI (single-turn jailbreak fuzzing), PyRIT (multi-turn orchestrated attacks), Garak (probe-based vulnerability scanning), and promptfoo (plugin-based red-team evaluation). Each tool covers different OWASP categories; running them together gives systematic coverage. Includes a conditional MCP reconnaissance phase and a post-access AI infrastructure phase (chained from /post-exploit). Produces: OWASP LLM Top 10 + AITG + MCP coverage matrix, findings per category, architecture diagram of the AI system, PoCs for confirmed exploits. Chains into /gh-export for issue filing.
The English summary is being prepared. Showing the original description.
OpenAI API (developers.openai.com) の Evals / Fine-tuning リファレンス。 Evals API, Datasets, graders（string check / text similarity / model / python / multi）, agent evals, trace grading, external models, Promptfoo（移行先）, red teaming, SFT, DPO, RFT, vision fine-tuning, distillation, prompt optimizer, fine-tuning best practices。 Evals は 2026-11-30 停止予定、セルフサービス fine-tuning は段階的廃止中。
The English summary is being prepared. Showing the original description.