Results for “lua”

Evaluate a machine learning GPU cluster during a cloud trial or acceptance test. Measure GPU computing performance, communication within and between machines, and local and shared storage, then compile the results into a dated Markdown report.

• Evaluating trial GPU machines from a cloud provider
• Running acceptance tests for a GPU cluster
• Measuring GPU performance under load from neighboring GPUs

Design Redis caching, leaderboards, notifications, and message processing. Helps choose data structures, write Lua scripts for operations that run together without interference, and plan expiration, memory use, and persistence settings.

• Designing caches with expiration times
• Building leaderboards and request rate limits
• Processing messages with Redis Streams

Assess language models with over 60 academic benchmark tests using consistent prompts and scoring. Compare models, track performance during training, and organize research results with support for HuggingFace models, vLLM, and APIs.

• Compare models on standard knowledge and reasoning tests
• Track model performance across training checkpoints
• Organize evaluation results for research reports

Add team-specific evaluation criteria to plugin-eval by designing a metric pack, a set of custom checks and measurements. Create its manifest and JSON-producing script, then run it through the evaluator using a consistent output structure.

• Add team-specific evaluation criteria
• Define custom checks and numerical metrics
• Build a metric-pack manifest and script

Evaluate AI agents by preparing test cases, scoring responses and tool use, investigating failures, and comparing results after changes. Supports built-in checks and custom scoring rules, including evaluations performed by a language model.

• Create test cases for an AI agent
• Check response quality and tool use
• Investigate failed evaluation cases

Advanced Evaluation workflow skill. Use this skill when the user needs This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment and the operator should preserve the upstream workflow, copied support files, and provenance before merging or handing off.

The English summary is being prepared. Showing the original description.

Evaluate Jira ticket complexity + inconsistencies via the jira-evaluator agent. Read-only. Triggers: 'evaluate KEY', 'analyze ticket complexity for KEY', 'check ticket quality for KEY'. NOT for estimation (mk:jira-estimator); NOT for full RCA (mk:jira-analyst).

The English summary is being prepared. Showing the original description.

Investigate AI observability evaluations of both types — `hog` (deterministic code-based) and `llm_judge` (LLM-prompt-based). Find existing evaluations, inspect their configuration, run them against specific generations, query individual pass/fail results, and generate AI-powered summaries of patterns across many runs. Use when the user asks to debug why an evaluation is failing, surface common failure modes, compare results across filters, dry-run a Hog evaluator, prototype a new LLM-judge prompt, or manage the evaluation lifecycle (create, update, enable/disable, delete).

The English summary is being prepared. Showing the original description.

Evaluate large language models on shared tests of knowledge, reasoning, and code generation with lm-evaluation-harness. Save scores and individual predictions, compare models in tables, and track performance across saved stages of model training.

• Evaluating models on MMLU and GSM8K
• Building comparison tables for multiple models
• Tracking scores across training checkpoints

Select, inspect, validate, and report Paperclip evaluations for Runner protocol behavior or complete product workflows. Interpret results with evidence, source revisions, costs, and failure categories, and refresh reports from retained evidence without new provider calls.

• Choosing between Runner and Product E2E evaluations
• Inspecting evaluation catalogs without credentials
• Interpreting failures using evidence and execution records

Estimate a public company's intrinsic value per share using discounted cash flow, a method that values future cash in today's terms. Produce a Markdown report documenting financial inputs, assumptions, five-year forecasts, and sensitivity to valuation assumptions.

• Estimate a public company's fair value per share
• Compare the current share price with estimated value
• Check how valuation changes under different assumptions

Train and evaluate models that estimate human poses from WiFi signals. Compare results against a model that always predicts the average pose, check for data leakage, and report measured improvements with clear evaluation conditions.

• Train a pose model using camera labels
• Evaluate pose estimation without a camera at inference
• Compare accuracy against an average-pose baseline

Create a fixed JSON evaluation suite from a repository's existing tests and check its structure for errors. Use the suite with harness-evolve to compare variants against consistent tasks, independently of the repository's normal test command.

• Create a fixed evaluation suite from existing tests
• Check evaluation suite structure in CI
• Compare variants across commits using consistent tasks

Add support for a programming language to codegraph, from connecting its parser and implementing code extraction to writing tests. Evaluate extraction quality and compare agent effort with and without codegraph across three real codebases.

• Adding a new programming language to codegraph
• Testing extraction of functions, imports, and calls
• Comparing agent effort across three real codebases

Build financial models for valuation and investment analysis using discounted future cash flows, assumption testing, and probability simulations. Outputs include Excel models, comparison tables, charts, and valuation ranges to examine possible outcomes and risks.

• Build a company valuation model from financial statements
• Test how growth and discount rates affect valuation
• Simulate acquisition valuation under uncertain assumptions

Build and evaluate time-to-event models with scikit-survival, including records whose events remain unobserved when follow-up ends. Keep preprocessing separate from evaluation data, compare models, predict survival probabilities, and summarize censoring-aware metrics.

• Validate event indicators and follow-up times in local CSV data
• Train survival models without leaking evaluation data
• Evaluate survival probabilities with censoring-aware metrics

Create consistent records for evaluating AI agents, including task success, invalid tool calls, token use, and actual costs. Produce CSV tables, database definitions, and JSON validation schemas to support reproducible comparisons and checks for performance declines.

• Recording agent evaluation results in a shared format
• Comparing a candidate agent with a baseline
• Tracking actual cost per solved task

Turns supplied AI agent evaluation results into reports that distinguish autonomous success, human-assisted success, failure, timeout, and invalid runs. Makes metric denominators, retries, timing, costs, and comparison conditions explicit to support evidence-based decisions.

• Separating autonomous and human-assisted success rates
• Reporting evaluations with retries and invalid runs
• Comparing agents or releases under matched conditions

Evaluate LLM applications using automated scores, human reviews, and model-based judging. Build repeatable checks to compare models or prompts, assess answer quality, and detect declines in performance before deployment.

• Comparing models and prompts
• Checking quality after prompt changes
• Detecting performance declines before deployment

Build and run evaluations for AI applications using Phoenix. Start with observed failures, create checks in code or use a language model as a judge, run experiments, and validate automated judgments against human labels.

• Turning observed failures into evaluation criteria
• Building code-based or language-model evaluators
• Evaluating retrieval and answer faithfulness in RAG

Build an end-to-end evaluation pipeline for Python AI applications using pixie-qa and real language-model calls. Define criteria, prepare realistic test data, score application runs, and analyze the evidence to produce a prioritized improvement plan.

• Setting up quality checks for Python AI applications
• Creating realistic scenarios for known failure modes
• Scoring full application runs with real language-model calls

Run repeated change-and-evaluation cycles for one predefined improvement mission within an explicit runtime limit. Save each evaluation as JSON and each decision in readable Markdown logs, with support for periodic reruns through Claude Code native cron.

• Repeating improvements under a fixed evaluation method
• Keeping evaluation results and decision logs
• Running experiments within a time limit

Evaluate LLM application quality using automated scores, human reviews, and another language model as a judge. Compare models or prompts, establish baseline results, and detect quality drops through repeatable tests and statistical comparisons.

• Comparing models and prompts
• Checking quality before deployment
• Evaluating search quality in RAG applications

Use Phoenix to record execution traces, detailed records of an AI application's operations, and investigate problems. Evaluate answers on datasets, compare prompts and models, and set up monitoring through a self-hosted server.

• Investigate AI application problems using execution traces
• Evaluate answer quality across a test dataset
• Compare prompts and models through experiments