Results for “evaluation”

Expert in Langfuse - the open-source LLM observability platform. Covers tracing, prompt management, evaluation, datasets, and integration with LangChain, LlamaIndex, and OpenAI. Essential for debugging, monitoring, and improving LLM applications in production. Use when "langfuse, llm observability, llm tracing, prompt management, llm evaluation, monitor llm, debug llm, langfuse, observability, tracing, llm-monitoring, evaluation, prompt-management, debugging, analytics" mentioned.

The English summary is being prepared. Showing the original description.

Testing and benchmarking LLM agents including behavioral testing, capability assessment, reliability metrics, and production monitoring—where even top agents achieve less than 50% on real-world benchmarksUse when "agent testing, agent evaluation, benchmark agents, agent reliability, test agent, testing, evaluation, benchmark, agents, reliability, quality" mentioned.

The English summary is being prepared. Showing the original description.

Agent Evaluation workflow skill. Use this skill when the user needs Testing and benchmarking LLM agents including behavioral testing, and the operator should preserve the upstream workflow, copied support files, and provenance before merging or handing off.

The English summary is being prepared. Showing the original description.

Design a source evaluation protocol using lateral reading and credibility checks for digital information. Use when students need to evaluate websites, online sources, or social media claims.

The English summary is being prepared. Showing the original description.

LLM-as-judge evaluation framework with 5-dimension rubric (accuracy, groundedness, coherence, completeness, helpfulness) for scoring AI-generated content quality with weighted composite scores and evidence citations

The English summary is being prepared. Showing the original description.

Generate msmodelslim evaluation YAML configuration (service_oriented + aisbench + vllm-ascend). Use when user asks for evaluation config generation.

The English summary is being prepared. Showing the original description.

Create evaluation frameworks for AI outputs with rubrics and scoring. TRIGGERS - Use when user needs help with ai-evaluation-framework related tasks.

The English summary is being prepared. Showing the original description.

Author and run offline evals for Agentforce agents: fixture format, scoring rubrics, hallucination checks, regression baselines, CI integration, prompt-change safety. Trigger keywords: agent eval, hallucination, doesn't hallucinate, agentforce offline evals, rubric for agent response. NOT for deciding which layers of testing the agent needs — use agentforce/agentforce-testing-strategy. NOT for Testing Center, AiEvaluationDefinition or sf agent test runs — use agentforce/agent-testing-and-evaluation.

The English summary is being prepared. Showing the original description.

Use when designing evaluation suites for agentic systems, choosing agent benchmarks, measuring multi-agent performance, or critiquing single-score evaluation of agents. Based on arXiv:2605.26112v1. Trigger phrases: "evaluate my agent", "benchmark agent", "measure agent reliability", "pass^k evaluation", "agent metrics beyond success rate", "trajectory quality", "memory hygiene metric", "longitudinal agent eval", "why my agent scores well on benchmarks but fails in production".

The English summary is being prepared. Showing the original description.

Builds a DRAFT weighted supplier evaluation matrix from supplier responses and the panel's agreed criteria, weights and scale: a provisional score per cell with the quoted evidence and anchor behind it, a register of every missing or partial answer, and calibration and sensitivity flags. Provisional for panel moderation; never recommends, awards or disqualifies. Use when the user asks to score, weight or rank supplier, bid or tender responses against agreed criteria, for example "score these bids", "build the evaluation matrix", "weight the tender responses" or "rank the suppliers against our criteria". Do not use for a side-by-side comparison without scores, use rfp-comparison-pack instead; for writing the criteria, use rfp-requirements-pack instead. Drafts for human review; never approves, authorises or signs off.

The English summary is being prepared. Showing the original description.

Turns a business need and its constraints into a DRAFT RFP requirements pack: traced functional and non-functional requirements with priorities, mandatory gates, evaluation criteria with weights left to the business owner, and fact-seeking supplier questions. Solution-neutral; never names a supplier or product. Use when the user asks to "write the RFP requirements", "turn this business case into tender requirements", "draft the supplier questionnaire", "structure our RFQ from this brief" or "set up evaluation criteria before we go to market". Do not use for responses already received, use rfp-comparison-pack to compare them or supplier-evaluation-matrix to score them instead. Drafts for human review; never approves, authorises or signs off.

The English summary is being prepared. Showing the original description.

Master comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing.

The English summary is being prepared. Showing the original description.

Implement comprehensive evaluation strategies for LLM applications using automated metrics, human feedback, and benchmarking. Use when testing LLM performance, measuring AI application quality, or establishing evaluation frameworks.

The English summary is being prepared. Showing the original description.

DDD評価エージェント - 戦略的設計・戦術的設計の両面からDDD原則への適合度を評価し改善点を特定。/ddd-evaluation [対象パス] で呼び出し。

The English summary is being prepared. Showing the original description.

評価統合エージェント - MMI評価とDDD評価の結果を統合し、優先度付き改善計画を策定。/integrate-evaluations [対象パス] で呼び出し。

The English summary is being prepared. Showing the original description.

Generate HTML reports and structured metrics from model evaluation results. Creates publication-ready reports with embedded outputs (images, audio, charts) and metrics.json for cross-model comparison. Use when generating reports, writing metrics, creating evaluation summaries, or formatting benchmark results. Triggers on "generate report", "write metrics", "create report", "evaluation summary", "benchmark results", "format results".

The English summary is being prepared. Showing the original description.

Focused Signals scout for PostHog projects using feature flags. Watches the flag roster and the `$feature_flag_called` evaluation stream for contradictions between a flag's configured state and its real traffic: evaluation cliffs on healthy flags, ghost flags (code calling keys that no longer exist), response-distribution shifts with no corresponding flag edit, and flag debt (stale, fully-rolled-out, or dead flags still burning evaluations). Emits findings only when they clear the confidence bar; otherwise writes durable memory and closes out empty. Self-contained peer in the signals-scout-* fleet — no dependencies on other skills.

The English summary is being prepared. Showing the original description.

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

The English summary is being prepared. Showing the original description.

Open-source AI observability platform for LLM tracing, evaluation, and monitoring. Use when debugging LLM applications with detailed traces, running evaluations on datasets, or monitoring production AI systems with real-time insights.

The English summary is being prepared. Showing the original description.

Generate the success criteria for a task or question, then review work against them. Given a task, goal, or open-ended question, decompose it into scenarios, evaluation perspectives, and fine-grained weighted YES/NO criteria using the Recursive Expansion Tree (RET) method; if work is supplied, score it criterion-by-criterion and surface what is missing or could be better. Use when asked to self-review or check your own work, judge whether a task is done well or completely, build a definition-of-done or completeness checklist, create an evaluation rubric or grading criteria, score or grade answers to a question, set up an LLM-as-judge rubric, or when the user mentions self-review, completeness check, success criteria, evaluation criteria, scoring rubric, Qworld, or the RET algorithm.

The English summary is being prepared. Showing the original description.

Create evaluation frameworks for AI outputs with rubrics and scoring. TRIGGERS - Use when user needs help with ai-evaluation-framework related tasks.

The English summary is being prepared. Showing the original description.

Design automated evaluation pipelines for LLM and agent systems — combining deterministic checks, statistical metrics, and LLM-as-judge scoring into repeatable, CI-integrated eval suites. Load when the user asks to set up automated evals, design an eval pipeline, integrate evals into CI/CD, create an eval suite, do eval-driven development, or says "automate my evals", "CI eval integration", "evaluation pipeline", "continuous evaluation", "monitoring eval quality", "set up regression testing for my agent". Sub-skill of eval-output orchestrator.

The English summary is being prepared. Showing the original description.

Design structured evaluation rubrics for scoring LLM and agent outputs — defining quality dimensions, scoring scales, hard gates, score descriptions, and edge cases. Load when the user asks to create an eval rubric, define evaluation criteria, design scoring dimensions, write an eval spec, or says "what should I evaluate", "design a rubric", "create eval criteria", "define quality dimensions", "evaluation rubric for", "how do I measure quality of". Sub-skill of eval-output orchestrator.

The English summary is being prepared. Showing the original description.

Plan, evaluate, and adopt agent infrastructure for a named workload. Use for agent runtime/framework selection, agent evaluation or observability plans, memory and tool boundaries, infrastructure cost controls, and organizational adoption. Covers workload-first comparison, risk and recovery planning, held-out evaluation, cost accounting, and staged rollout. Not a vendor ranking or a production certification. NOT for implementing a specific agent behavior (agentic-patterns), building a specific agent (ai-engineer), prompt optimization (prompt-engineer), or conventional ETL workflow design (jury_rig-architect).

The English summary is being prepared. Showing the original description.