Self-Hosting Qwen with vLLM: Our Architecture, Configuration, and Engineering Decisions

We wanted to make language models available to our development team and use them in product AI features without sending every request to an external model provider. Getting Qwen to answer a prompt was only the first step. Making it useful through the tools our developers already use took more work.

This is the second article in our self-hosting series. In Part 1, we covered the fundamentals. Here, we will walk through our own Qwen and vLLM setup: the pieces we chose, the settings we revisited, and the integration details that mattered. This is a description of our implementation, not a universal deployment recipe.

How we self-hosted Qwen3 with vLLM

At Centizen, we run quantized Qwen3 models using vLLM on our own GPU infrastructure. Our setup uses a GPU with 32 GB of VRAM and an internal gateway for authentication and usage tracking.

Our deployment work has included Qwen3-30B-A3B for general tasks and Qwen3-Coder-30B-A3B-Instruct for coding-related work, both with AWQ quantization. We wanted to support our development tools and product AI features without depending on an external model provider for every request.

Getting a model to answer a prompt was straightforward compared with making it work through the tools our developers use. Context limits, tool calling, streaming responses and client compatibility all required engineering attention.

Why we started with an internal service

Our developers use coding tools, and we also have applications that can benefit from model-backed features. We wanted a shared service that both could call, rather than a separate model running on every developer machine. An internal gateway gave us a place to authenticate callers and track usage.

We chose to run inference on our own GPU hardware. That gives us direct control over the model and serving stack, but it also means the team is responsible for configuration, compatibility and troubleshooting. Self-hosting moves work; it does not make that work disappear.

The architecture we put together

The request path is straightforward on paper. A developer's editor or an application sends a request to our internal gateway. The gateway handles access and metering, then forwards the request to the inference service. vLLM loads and serves the Qwen model, and the response travels back to the client.

The gateway matters because an API endpoint that works for one engineer is not yet a team service. You need to know which client made a request, how much it used, and what failed. We implemented authentication and usage metering at this layer. We are not claiming that every enterprise security control is already in place.

Two Qwen models for different work

Our deployment work has included two quantized Qwen3 models: Qwen3-30B-A3B AWQ for general tasks and Qwen3-Coder-30B-A3B-Instruct AWQ 4-bit for coding-oriented use. Both are mixture-of-experts models. The roughly 30-billion-parameter name describes total parameters, not the number activated for each token.

That distinction matters when planning hardware. Fewer active parameters can reduce compute per generated token, but you still need to account for the model weights and the memory used while serving requests.

Our on-premises setup uses a GPU with 32 GB of VRAM. We used AWQ-quantized weights to make this class of model practical within that constraint. The exact hardware and software versions matter when reproducing this configuration, so we recommend recording them before deployment.

The memory calculation is more than the model file

Quantization reduces the space needed for weights, but it does not remove the memory used by the inference engine, temporary buffers or the KV cache. That cache holds information from tokens already processed. Longer prompts and more simultaneous requests put additional pressure on it.

We worked with a gpu_memory_utilization setting around 0.92 in reconstructed configuration notes. That is a high allocation target, not a guarantee that 92% of memory is available to the model weights. The useful test is whether representative long prompts and parallel requests complete without destabilizing the service.

Our notes do not include a controlled concurrency benchmark or a measured GPU-memory breakdown. We would not turn this setup into a blanket claim that a 32 GB GPU is enough for every team running a 30B model.

Serving Qwen through vLLM

We used vLLM in Docker to expose an API that our applications and editor tools could call. The serving configuration evolved as we tested actual requests. Our notes show a maximum model length of 40,960 tokens in one iteration and 65,536 later. Prefix caching was enabled in the reconstructed settings.

Our configuration at a glance

These are the components and settings recorded during our implementation. Some settings come from reconstructed notes rather than a verified snapshot of the currently running service.

The table does not imply that both models were loaded on the GPU at the same time or that these values are a reproducible launch command.

Increasing the context limit is not a free improvement. It affects how much cache the engine may need to reserve and how much work a large prompt creates. For coding tools, the prompt can contain much more than the developer's visible question: source files, conversation history and tool descriptions all count.

We also had to pay attention to tool-call and reasoning parsers. The general model and the coder model did not necessarily need identical parsing behavior. Our reconstructed notes reference a Hermes-style tool parser for the general setup and a Qwen3 XML-oriented parser for the coder configuration. These are compatibility decisions to validate against the exact vLLM build and model revision, not flags to copy blindly.

Where the editor integration became difficult

A successful API response from a terminal did not tell us whether a coding editor would work properly. Editors send their own request shapes and expect responses in specific formats. That difference caused several of the problems we investigated.

We encountered mismatches involving tool schemas, tool-description length limits, token estimates and streaming response fields. One integration expected a different shape for reasoning or tool-call deltas. We also had to check the context budget advertised to the editor; the client setting and the server's actual limit must agree.

Some problems were outside the model itself. HTTPS access, private-network browser restrictions and proxy path rewriting affected whether requests could reach the right endpoint. Those are ordinary application-integration issues, but they can look like model failures until you follow the request through every layer.

One lesson was particularly useful: test from the actual editor, not only with a direct HTTP request. Include a simple completion, a tool call, a long-context request and a streamed response. Inspect both what the server sent and what the client accepted.

What we learned about operating it

Our internal traffic confirmed that the service was being used primarily through development tools, rather than as a standalone chat interface. We will cover the request counts, failures and limitations of those observations in Part 3. For this article, the operational lesson is simpler: a running container and a healthy HTTP endpoint do not prove the full workflow is healthy.

We also learned to verify that a configuration change had actually reached the running process. Updating a file or rebuilding an image is not the same as confirming the active server is using the new values. When behavior did not match expectations, checking the running process, route and client request often saved more time than changing model settings again.

What we would do differently next time

• Record the environment on day one. Save the GPU model, driver, container digest, model revision, launch arguments and parser versions together.
• Test with real client payloads early. A coding editor is a better compatibility test for coding work than a generic chat prompt.
• Keep model-serving and gateway issues separate. Trace authentication, routing, token counting and streaming before blaming inference.
• Change one setting at a time. Keep a short record of why the context limit, parser or cache settings changed and what improved.
• Measure before promising capacity. GPU memory, context and concurrency interact. Our setup is evidence of feasibility for our workload, not a benchmark for every workload.

Would we self-host again? For the use cases we are pursuing, having a model service under our control is useful. The tradeoff is the engineering attention required to make it dependable for the people and products using it.

Frequently asked questions

Can Qwen3-30B-A3B run on a GPU with 32 GB of VRAM?

We have run an AWQ-quantized version of Qwen3-30B-A3B using a GPU with 32 GB of VRAM. That does not mean every 32 GB deployment will work the same way. Available memory depends on the model weights, KV cache, context length, concurrent requests and runtime overhead.

Why did we use vLLM for Qwen3?

We used vLLM to serve Qwen3 through an API that our internal applications and development tools could call. The work also involved tuning context settings and checking streaming responses and tool-call parsing for compatibility with our clients.

What problems did we encounter with coding editors?

Some editor requests and responses did not match what the serving stack initially expected. We investigated tool schemas, tool-description limits, token estimates, streaming fields, HTTPS access and proxy routing. Testing directly from the editor exposed problems that a basic API request did not.

Can the same Qwen model handle coding and general AI tasks?

A general-purpose model can handle coding requests, but editor workflows have their own requirements, particularly around structured tool calls and response formats. We worked with both general-purpose Qwen3 and Qwen3 Coder for different tasks rather than assuming one model was the best fit for everything.