What it does
This skill evaluates response speed and practical task performance on a running local model server. It can find models served by Ollama, LM Studio, llama.cpp, MLX, vLLM, and SGLang, then measure time to first response, context loading speed, and generation speed. It evaluates tasks such as math, coding, following instructions, tool calls, and retrieving information from long inputs. When a task fails, it examines the model's actual output to identify the cause. It can also run follow-up experiments with changed conditions, such as model identity prompts or generation settings, and summarize results in comparison tables.
When it helps
Use it to compare local models, quantization methods, execution engines, or machines. It can help determine whether speed, coding ability, or tool use meets your needs. It can also assess agent suitability in a simulated environment that includes tasks such as file operations.
Example requests
- “Evaluate the speed and practical performance of the model running in LM Studio.”
- “Compare the coding and tool-call results for Q4 and Q8.”
- “Assess whether this model fits my agent tasks.”
Notes
The target model server must already be running. This skill does not install execution engines or download models. It uses an included Python script for evaluation. For coding tasks, model-generated code is run in a temporary directory.
This overview was written by AI (OpenAI Codex) from the public SKILL.md. Check the original below for the exact behavior.