Benchmarking the Whole Stack, Not Just the Model

There is a question we hear a lot that turns out not to be very useful: which local AI model is the best one? It assumes there is a single answer. In practice, the right model is the one that fits the job in front of you, running on the hardware you actually have. Every other model is beside the point.

And the model is only part of what decides the result. A few terms, because each one moves the outcome:

• Weights: the numbers a model learned during training. They are the model itself. Example: "a 30B model" means 30 billion weights.
• Quantization: shrinking those weights to lower precision so the model uses less memory and runs faster, trading a little accuracy. Example: the same model at 4-bit (Q4) is smaller and quicker than at 8-bit or full 16-bit.
• Inference engine: the software that runs the model. Example: llama.cpp or MLX.
• Context length: how much text the model can hold at once, input plus output, counted in tokens. Example: 8k tokens is a few pages; 128k is a small book.
• Server: the app and machine serving the model over an API. Its memory and chips decide what even fits. Example: LM Studio or Ollama, running on an Apple M3 Ultra or an NVIDIA GPU box.
• Harness: the code wrapped around the model that feeds it prompts, gives it tools, and controls what it is allowed to do. Example: an agent runner like OpenCode, or the control loop inside Cubernaut.

The same set of weights behaves differently the moment you change any of the others. The speed, the memory footprint, and sometimes the answers move with it. So the thing worth measuring is not the model on its own. It is the whole stack, exactly as it will run in production.

What we measure

We built Yardstick, a standard set of tests for a local model, so the answer comes from data, not opinion. You aim it at a running model, and it measures the things that matter for real work: how fast it writes, how much memory it uses while it works, and how well it uses tools on both simple and harder requests. A model that nails an easy tool call but stumbles on a layered one is not the same tool in practice. Yardstick also checks how it handles code and images, how much it really knows about security, and how it responds when a request should be refused, including the gray area where a task could be misused and the model has to stay careful without shutting down every request.

That last check gets its own test, because a model can fail at refusals two ways: it can go along with anything, which is dangerous, or refuse everything, which makes it useless. Neither is much use in practice.

Not tied to one runtime or format

We built the tool so it is not locked to any one vendor. It talks to a model through a common API standard, and when a server expects a different message format, we point it at that one instead. The runtime underneath can be LM Studio today and Ollama or an MLX server tomorrow. None of that changes the test or the numbers it produces.

That is deliberate. The local AI landscape turns over fast. A benchmark that only works against one runtime, or one API format, is stale the week either of them changes. Keeping the measurement independent of both is what lets us compare last month's setup against next month's on even ground.

Comparing them fairly

A number only means something next to another number from the same test, on the same system. We do not average across different machines or different batteries and call the result a trend. When the tool cannot confirm a piece of hardware or a context length, it records that value as unknown rather than borrowing one from the wrong box. It runs unattended across every model, quant, and context-length combination we care about, and it resumes cleanly if it stops partway, because a benchmark you cannot trust to run honestly overnight is not worth running.

Some of what it measures scores itself. A refusal is close to a yes or a no, so a detector handles it. The judgment calls, like whether a model truly answered a security question or just sounded like it did, go to a separate judge model grading against a fixed rubric, so the scoring does not drift from one run to the next.

What this looks like on a real run

We ran a real test on Cubernaut, our local pentest tool. Cubernaut uses a local AI model to do a job one step at a time.

The job was simple. Find all the API endpoints on a target app. An endpoint is one of the URLs an app exposes. The app had 14, and we knew all 14, so we could grade the AI.

Cubernaut ships with APIRecon, our own API enumeration tool. Part of what we watched was whether each setup would actually use it.

We ran the same job on three setups, 10 times each:

• Qwen3 27B (a "dense" model), run by Ollama
• Qwen3 35B "mixture-of-experts" (a big model that only wakes a small part of itself per step), run by Ollama
• the same Qwen3 27B model, run by LM Studio on MLX

Every run used the same machine, a MacBook Pro with an Apple M5 Max chip and 64 GB of memory, so the hardware never became a hidden variable.

Here is the useful part. All three setups reached the full 14 out of 14. Once every setup can do the job, what separates them is efficiency: speed, steps, and time. So we compare them by changing one piece at a time.

Same runtime, two different models

Both models run on Ollama, on the same box. The only thing that changes is the model itself: a normal "dense" 27B against a 35B "mixture-of-experts."

Same job, same runtime, but the MoE model was about twice as fast and did it in half the steps. A mixture-of-experts model only turns on a small part of itself for each step, so it runs light even though it is "bigger." Here, the model was the level that mattered.

Same model, two different runtimes

Now we hold the model still (the 27B dense one) and change the runtime: Ollama against LM Studio on MLX.

Same weights, very different behavior. On this box, Ollama was faster and did fewer steps. MLX was slower and wordier. MLX was also more stubborn: even when the prompt named APIRecon, it used it only half the time. The rest of the time it typed its own curl commands to hit the endpoints instead. Here, the runtime was the level that mattered, not the model.

Change the model and the numbers move. Hold the model and change the runtime, they move again. You cannot pick the best one from the name on the box. You pick it by measuring the whole stack.

Why we measure first

This matters in practice. Every tool we build on top of a local model inherits whatever that model actually is, on whatever hardware it runs on. Before we put one inside something a client depends on, we want to know what it can do and where it falls down, measured rather than assumed.

Our Cubernaut and Terrain Trace tools both run on local models by design, kept on the operator's own network. The work described here is how we decide which models earn a place inside them, and why.