Self-Hosting Open-Source LLMs: What You Need to Know Before Getting Started

Getting a language model to answer a prompt on your laptop is often the easy part. The questions start when a second developer wants access. How much GPU memory will you need? What happens when an editor sends a very long request? Who can use the endpoint, and how will you know when it fails?

Before choosing a model or copying an installation command, it helps to know what you are actually putting together. We will cover the basics here, then use Centizen’s Qwen and vLLM setup in the next article to show how these decisions play out on a real server.

What is a self-hosted LLM?

A self-hosted large language model (LLM) runs inference on infrastructure you control, instead of sending every request to an external model API. That infrastructure might be a server in your office, a rented GPU, or a private cloud environment.

In practice, you need the model weights and tokenizer, a compatible inference engine, enough CPU or GPU memory for the workload, and an interface your applications can call. You also need to look after access, updates, monitoring, and failures. Self-hosting gives you control over these choices, but it makes your team responsible for them.

Start with a task you can test. The model, hardware, and serving setup should follow from what your users actually need.

What does self-hosting an LLM actually mean?

An LLM generates text by predicting tokens, which may be words or pieces of words. Running an already-trained model to produce an answer is called inference. When you self-host, that work runs on hardware you manage or rent, rather than through a third-party model API.

You usually do not train the model yourself. You download its weights (the learned numerical values), use a tokenizer to turn text into model inputs, and run an inference engine to produce responses. Your application talks to that engine, often over HTTP.

One terminology point is worth getting right: open-source and open-weight do not always mean the same thing. Downloadable weights do not automatically grant every commercial or redistribution right. Read the license, especially if the model will become part of a customer-facing product.

Choose the workload before choosing the model

Start with the job, not a leaderboard. A small model answering routine questions may be enough for an internal tool. A coding editor that sends source files, conversation history and tool definitions is a different workload. So is an application that must handle many users at once.

There is no reason to start with the largest model you can afford. Choose a few tasks that matter, test candidate models against them, and include the actual client in the test. A model that looks good in a chat demo may behave differently when an editor expects tool calls and streaming responses.

Understand parameters, model memory, and the context window

Parameters are not the same as active compute

A parameter is one of the numerical values learned during training. In a dense model, the main network participates in generating each token. A mixture-of-experts (MoE) model selects a subset of experts for each token. That can reduce computation, but the other experts still have weights that must be stored or made available. A model with 3 billion active parameters is not necessarily a model that occupies only 3 billion parameters’ worth of memory.

Quantization reduces weight storage, not every memory cost

Quantization stores model weights using fewer bits. It can make a model practical on smaller GPUs, although the format, hardware and runtime must support it. The tradeoff is not always the same across models or tasks, so quality needs testing rather than guessing.

Take a 30-billion-parameter model as a simple example. At 16 bits per weight, the weights alone work out to roughly 60 GB. At an idealized 4 bits, they work out to about 15 GB. Neither number includes the rest of what the inference engine needs. Treat these as arithmetic for planning, not a promise that the model will load.

The KV cache is the memory many first-time deployments miss

The GPU also has to remember information from tokens it has already processed. Many transformer runtimes keep that information in a key-value (KV) cache. The amount grows with the requests being served, and it depends on the model’s attention design, context length, concurrency and cache precision.

This is where first deployments can go wrong. You may have enough memory to load the weights and still have too little room for a useful context window or several users. Leave space for the cache and runtime overhead, then measure the real workload.

Know the software layers before installing them

On a Linux GPU server, several things must work together: the operating system, the GPU driver, compatible compute libraries, the model files, and the inference engine. Docker can package the application environment, but the host still needs a working driver. Above the inference engine sits the API your editor or application calls.

For a first local test, Ollama or llama.cpp can be a straightforward route. For a shared GPU service, vLLM is one option worth evaluating because of its serving and scheduling features. Do not choose solely from a feature list: check support for your exact model, hardware, request format and expected traffic.

Plan installation as a series of checks

A tutorial command can get you started, but it will not tell you whether your setup is suitable for the job. Record the following checks as you go. They are also the details you will want when something stops working a month later.

1. Define a test task. Pick several representative prompts and decide what a useful answer looks like. Include a long prompt and, if relevant, a tool-calling request.
2. Check the model license and artifacts. Confirm the exact repository, revision, tokenizer, model format, and runtime compatibility. Treat model files and custom code as software supply-chain inputs.
3. Inventory the machine. Record GPU model and VRAM, system RAM, CPU, free disk, operating system, and driver version. Check whether the runtime supports the available accelerator.
4. Choose the runtime and precision. Match the model to an inference engine and a supported weight format. Estimate memory for weights and active requests.
5. Install in an isolated environment. Prefer a documented Python environment or pinned container image. Avoid unreviewed installation scripts on shared infrastructure.
6. Start locally first. Bind the test endpoint to localhost or a restricted network. Verify model loading, a simple response, and streaming before exposing it to other users.
7. Measure realistic requests. Record prompt and output tokens, latency, errors, memory use, and concurrency. Repeat tests with the actual editor or application, not only a one-line chat prompt.
8. Add shared-service controls. Introduce authentication, TLS, access restrictions, request limits, audit policy, health checks, and a rollback procedure before broad internal use.

On an NVIDIA Linux machine, nvidia-smi is a useful first check. It shows whether the driver can see the GPU and reports memory and driver information. It does not prove that the Python libraries or inference runtime are compatible; test those separately.

nvidia-smi
free -h
df -h

These commands tell you what hardware and free space you have. They do not install anything. We are deliberately leaving version-specific installation commands to the implementation article, where the exact environment can be documented.

Design for application behavior, not just model output

A model answering “Hello” tells you almost nothing about how it will work inside a coding editor. Editors can resend long conversations and large tool definitions with every request. They also expect particular streaming and tool-call formats. A successful chat response can still turn into an editor error.

Test the whole route, from the editor through any gateway to the inference engine and back. Check how tokens are counted, how partial responses are streamed, and how tool calls are returned. Be careful with validation limits you invent yourself: a tidy-looking cap on tools or message length can break a legitimate client.

Keeping inference on your own server gives you more control over where requests are processed. It does not automatically keep every copy of the data private. Prompts and source code may still end up in application logs, editor extensions, monitoring tools or backups. Decide what gets retained and who can see it.

When does self-hosting make sense?

Self-hosting can make sense when you need tighter control of the data path, expect enough sustained usage to justify the infrastructure, or need to tailor the serving environment. It is less attractive when usage is occasional and nobody has time to operate the system. A managed API may be the sensible first choice.

Do the cost comparison with the whole system in view. Hardware or GPU rental is only one line. Add idle time, engineering work, security reviews, monitoring, upgrades and incident handling. Some teams will end up with a hybrid setup, using local models for selected tasks and hosted APIs for others.

What engineering teams should take away

The useful starting point is a workload you can describe and test. From there, model selection, memory sizing, runtime choice and API design become much more concrete. Installing the weights is a step in that process, not the finish line.

For a real implementation example, read Self-Hosting Qwen with vLLM: Our Architecture, Configuration, and Engineering Decisions. We explain how we approached model selection, GPU memory, configuration, and developer-tool integration on our own infrastructure.

In Part 3, we examine what our request records revealed once developers began using the service.

Frequently asked questions

Do you need a GPU to self-host an LLM?

Not always. Smaller models can run on CPUs, although GPUs usually provide better performance for many inference workloads. The hardware you need depends on the model, its precision, the response time you expect, and how many people will use it at once.

How much GPU memory is needed to run an LLM?

There is no single requirement. You need room for the model weights, KV cache, runtime overhead, and active requests. Quantization can reduce weight memory, but a model that loads successfully may still leave too little space for long contexts or concurrent users.

Is self-hosting an LLM cheaper than using an API?

It depends on how much you use it. Include GPU purchase or rental, idle capacity, maintenance, power, monitoring, and engineering time in the comparison. A hosted API may be simpler and cheaper for occasional requests. Dedicated infrastructure may be worth evaluating for steady workloads.

What is the difference between Ollama, llama.cpp, and vLLM?

Ollama makes local model setup and management relatively straightforward. llama.cpp supports efficient inference across a range of hardware, including CPU-based setups. vLLM focuses on serving models and handling shared inference workloads. Choose by testing your model, hardware, API requirements, and traffic rather than assuming one runtime suits every situation.

Technical references

• Hugging Face: Models on the Hub
• vLLM documentation
• Ollama documentation
• llama.cpp project documentation
• Open Source Definition