Skip to content
cccskills
FreePublished on GitHubEnglish overview

run-model-eval

Benchmark a running local model server for response speed and practical tasks such as coding, math, instruction following, and tool use. Compare models, quantizations, or engines, inspect failed outputs, and run targeted experiments to explain surprising results.

OriginalShow the original description

Test a local model server that is already running (Ollama, LM Studio, llama.cpp, MLX, vLLM, SGLang) for speed and practical capability, and dig into anything that looks off.

holaboss-ai/holaOS11,442 stars(GitHub 11,442 + cccskills 0)

Run score Not tested

See how to install

Good for

  • Compare the speed of two local model setups
  • Check whether a model handles coding, math, and tool use
  • Investigate why a model gives unreliable or malformed answers
  • Assess a model for agent tasks in a simulated workspace

Overview

What it does

This skill evaluates response speed and practical task performance on a running local model server. It can find models served by Ollama, LM Studio, llama.cpp, MLX, vLLM, and SGLang, then measure time to first response, context loading speed, and generation speed. It evaluates tasks such as math, coding, following instructions, tool calls, and retrieving information from long inputs. When a task fails, it examines the model's actual output to identify the cause. It can also run follow-up experiments with changed conditions, such as model identity prompts or generation settings, and summarize results in comparison tables.

When it helps

Use it to compare local models, quantization methods, execution engines, or machines. It can help determine whether speed, coding ability, or tool use meets your needs. It can also assess agent suitability in a simulated environment that includes tasks such as file operations.

Example requests

  • “Evaluate the speed and practical performance of the model running in LM Studio.”
  • “Compare the coding and tool-call results for Q4 and Q8.”
  • “Assess whether this model fits my agent tasks.”

Notes

The target model server must already be running. This skill does not install execution engines or download models. It uses an included Python script for evaluation. For coding tasks, model-generated code is run in a temporary directory.

This overview was written by AI (OpenAI Codex) from the public SKILL.md. Check the original below for the exact behavior.

Document check

AI checked the SKILL.md for operations that need care and for setup requirements. The score comes only from actually running the skill (hands-on test).

SafetyAre sending data, deleting files, and running commands disclosed?
BPartly lackingGenerated code runs locally; vendor pack unit tests must be read before execution.
Easy setupDoes it need few tools or API keys, and are they documented?
AGoodIt requires Python and explicitly says the target server must already be running.

Not counted in the score. Rated by AI (OpenAI Codex) as of Nov 16, 2026. Treat it as a guide and read the original before installing.

Disagree? Request a re-evaluation

Tell us which point is wrong and why. No login needed. We will check it against the SKILL.md, redo the evaluation, and show the result here.

This form is protected by reCAPTCHA, and the Google Privacy Policy and Terms of Service apply.

Files included (56)

  • SKILL.md6.8 KB
  • holaeval.py192.9 KB
  • packs/agent-v1.json6.8 KB
  • packs/agent-v1/system-full.md19.7 KB
  • packs/agent-v1/system-lean.md1.2 KB
  • packs/agent-v1/tools-full.json53.6 KB
  • packs/agent-v1/tools-lean.json19.6 KB
  • packs/general-v1.json11.0 KB
  • packs/suite-coding-1.0.json4.5 KB
  • packs/suite-coding-1.0/calc-power/hidden/tests/test_hidden.py1.3 KB
  • packs/suite-coding-1.0/calc-power/repo/calc/__init__.py318 B
  • packs/suite-coding-1.0/calc-power/repo/calc/evaluate.py716 B

44 more files are not shown. View source on GitHub

SKILL.md (original)

Review the instructions the agent will receive before you install.

Reviews

No reviews yet. Tell others what you thought.

Comments

No login needed. Share how it worked for you, ask a question, or note where it did not run.

No comments yet. Be the first to write one.

Comments are published right away. Do not include personal information such as email addresses or phone numbers. Up to 2 URLs.

0 / 1,000 characters

This form is protected by reCAPTCHA, and the Google Privacy Policy and Terms of Service apply.

More skills from this repository

app-builder-sdk

FreeSummary

Build holaOS apps for connecting external services or managing work through a dashboard. The skill uses shared SDK building blocks for stored records, actions, and synchronization, with the dashboard and agent tools working on the same data.

  • Building an external service integration
  • Creating a CRM dashboard
  • Building a content planning workspace
holaboss-ai/holaOS11.4KUpdated Oct 28, 2026

audience-analyst

FreeSummary

Analyze audience demographics and engagement to identify distinct groups, understand what they want, and prioritize valuable segments. Turn those findings into recommendations for content topics, formats, posting times, and tone, while explaining limits in the data.

  • Identify distinct groups within an audience
  • Prioritize groups that engage or convert
  • Plan topics and formats for each audience group
holaboss-ai/holaOS11.4KUpdated Oct 28, 2026

Guide an agent through efficient use of an embedded browser by choosing targeted actions, searches, and waits. Reduce repeated tool calls and unnecessary page snapshots, while using more detailed inspection when the target or result remains unclear.

  • Act on known buttons or fields with fewer browser calls
  • Locate a known target on a large page
  • Wait for navigation or downloads to finish
holaboss-ai/holaOS11.4KUpdated Oct 28, 2026

browser-qa

FreeSummary

Validate workflows and reproduce issues in an embedded browser. Distinguish application problems from incorrect targets, missing waits, or session state, then report the shortest reliable reproduction steps with only the evidence needed to support the finding.

  • Reproducing a browser workflow failure
  • Checking for regressions after a fix
  • Distinguishing timing issues from application failures
holaboss-ai/holaOS11.4KUpdated Oct 28, 2026

build-dashboard

FreeSummary

Build the user interface for a holaOS app whose data resources are already connected through the SDK. Choose a feed, table, kanban board, form, or calendar layout using shared components and workspace styling.

  • Display scheduled items in a chronological feed
  • Show CRM contacts or tickets in a table
  • Organize issues by status on a kanban board
holaboss-ai/holaOS11.4KUpdated Oct 28, 2026

Package workspace skills and their required service integrations into one named, installable capability. Create a manifest that references existing skills, install the capability, and report which integrations still need to be connected.

  • Bundle multiple skills for a shared outcome
  • Declare required external service integrations
  • Create a manifest that references existing skills
holaboss-ai/holaOS11.4KUpdated Oct 28, 2026

See all skills by holaboss-ai

Report a problem with this skill