Local inference on developer laptops did not pay for itself in agent SDLC

Observation

Running an agent's SDLC loop against a hosted model costs money on every call, and every classification test during development pays it. The question was whether that cost could be designed away by swapping in a small open-weight model. Most of the team already carried a 16GB M1 MacBook Pro, so the plan was to run inference there and iterate for free.

The task was narrow on purpose - Classify a document against a taxonomy held in a separate guidelines file. It is the low-hanging fruit amongst the use-cases the agent under development actually needs.

Local inference on 16GB developer laptops missed the performance bar for this loop. In addition, the hosted control it was meant to replace matched its accuracy for pennies.

Context

Two open-weight quantised models were benchmarked for throughput, then scored for classification accuracy against a minimal seven-document corpus covering the taxonomy. A hosted model ran as the control. Both context sizes here are real measurements of the same task: 10k tokens with the taxonomy guidelines pre-processed, 30k with the taxonomy left raw. At 30k context, with the guidelines left raw, most runs failed by exhausting system resources irrespective of the model used.

The 16GB baseline sets the constraint. Every run started with 8 to 10GB already claimed by the IDE, browser and other resident software. Some swap is also already in use, and well under half the machine's memory free before the model even loaded.

The pattern

Throughput first, at the pre-processed 10k guidelines as input showing mean values from the benchmark run,

Prompt-processing and time-to-first-token readings are excluded here because they are not comparable across the two columns. The models ran on different inference servers, LM Studio and oMLX, which report those metrics differently. At 30k, with the guidelines left raw, processing times multiplied. Most runs failed regardless of model, exhausting system resources before completion. Hence, the tables do not carry benchmark figures at 30k context.

Then classification accuracy against the seven-document corpus:

On paper the 4-bit ~4B model looks like a clean win as it matches the hosted model's accuracy at zero marginal cost. But the win is partial. Classification of the corpus takes 55 seconds against 2.7 seconds hosted, a twentyfold tax on every iteration of the loop the exercise meant to speed up. The cost being avoided is worth stating too - the hosted control ran the same seven documents at full accuracy for $0.07. That corpus was deliberately small, so the figure sets a scale rather than a budget.

Completing a run meant shutting other applications down first, which defeats the point of running locally alongside normal development work. The verdict was that local inference did not clear the bar on this hardware. It failed on two counts, and only one of them is hardware. Memory headroom can be bought. The twentyfold per-turn latency gap, at this model class, cannot.

Generalised lesson

This is not a claim that one model loses to another, or that 30k tokens is a threshold. What mattered was a threefold difference in input size, down to whether the reference document was pre-processed first. That choice decided whether the task ran at all on 16GB of shared memory.

Two costs stack against local inference on developer hardware. The first is a flat per-turn latency tax that erodes the "iterate for free" premise for the model that wins on accuracy and cost. The second is a memory ceiling that depends as much on input preparation as on model choice.

Cost them separately; a hardware budget answers one, not the other.

The wider lesson is about the premise, not the hardware. Price a cost worth engineering away before the engineering starts, at realistic volume. Nobody had established that figure here. The control run that finally produced one came to pennies, a number local inference did not beat.

Evidence / Sources

• Internal benchmark and eval run against a minimal seven-document classification corpus, using llama-benchy for throughput and DeepEval for output quality.
• Model inference servers: LM Studio and oMLX.
• Hardware baseline: MacBook Pro M1, 16GB RAM, taken as the team's lowest common specification.