Private LLM

Self-Hosted LLM: The Technical Picture

What actually determines whether a self-hosted deployment performs well and what it costs to run. Mostly it is not the model you pick.

Model selection

The open-weight families worth considering are Llama, Mistral, Qwen and Phi, plus their instruction-tuned and domain-tuned derivatives. The differences between them on any specific task are smaller than the marketing suggests, and they change with every release, which is why we do not publish a recommendation that would be stale within a quarter.

What matters more is sizing. The instinct is to pick the largest model that fits the budget. The better approach is to build an evaluation set from real examples of your task, then start from the smallest model and move up only when the evaluation says you have to. On extraction, classification and retrieval-grounded answering, small models frequently pass, and every step up the parameter count is paid for monthly in GPU capacity.

Serving

The serving layer decides your throughput far more than the model does. vLLM and TGI use continuous batching and paged attention to serve many concurrent requests from one GPU efficiently. Ollama is simpler to operate and appropriate for low-concurrency internal tools, but it will not give you the same throughput per GPU under load.

Pick based on your concurrency profile. A tool used by fifteen people intermittently and a pipeline processing fifty thousand documents overnight have almost nothing in common operationally, even running the same weights.

Quantisation

Quantisation reduces the numeric precision of the weights, cutting memory use and increasing speed at some cost to output quality. Eight-bit quantisation is usually close to free in practice. Four-bit is a genuine trade: some tasks barely notice, others degrade in ways that only show up on the harder examples in your evaluation set, which is exactly where you were relying on the model.

Evaluate it on your own data. Published benchmark deltas will not tell you how your specific task behaves.

What actually drives the hardware bill

Three things, in order: the memory footprint of the weights, the memory consumed by the key-value cache during generation, and your peak concurrency. The first is easy to calculate and the one everybody plans for. The second scales with context length and concurrent requests and is the one that surprises teams, because a long-context retrieval application can spend more memory on cache than on weights.

This is why we size against a measured workload rather than a formula. Over-provisioning is expensive and under-provisioning shows up as queueing under exactly the load you built the system for.

Retrieval is where the work is

On most engagements, model deployment is a minority of the effort. The majority goes into retrieval: getting documents in, chunking them sensibly, keeping the index fresh, and enforcing access control at retrieval time rather than by asking the model to be careful. A prompt is not an access control, and treating it as one is the most common serious design error we see in self-hosted deployments.

Related

• The business case for on-premise deployment
• Whether you need fine-tuning at all
• How Senteras runs a deployment