What 2,500+ LLM Requests Taught Us About Self-Hosting AI for Software Development

November 9, 2026 | Artificial Intelligence

2,508 requests. 13.1 million tokens. Six people using the service. Those numbers gave us something more useful than a successful model demo: a record of what happened when our developers started using a self-hosted LLM through their normal tools.

This is the third article in our series on self-hosting open-weight models. Part 1 explains the fundamentals. Part 2 covers our Qwen and vLLM architecture. Here, we focus on what our request records and engineering notes actually tell us—and what they do not.

What did we learn from 2,508 self-hosted LLM requests?

From October 9 through November 3, 2026, Centizen recorded 2,508 requests and approximately 13.1 million tokens through our internal Qwen3 and vLLM service. Most requests came through developer-tool API keys rather than standalone chat. The logs recorded 97.1% of requests as successful, but that figure does not measure code correctness or developer productivity.

The issues we investigated were often at the boundaries between the editor, gateway, proxy and inference engine: tool schemas, token limits, parsing, streaming responses and routing. The practical lesson was to test the complete developer workflow, not just whether the model endpoint returns a response.

What we measured, and over what period

Our available records cover October 9 through November 3, 2026. They show 2,508 requests and approximately 13.1 million tokens. Of those requests, 2,436 were recorded as successful, which is 97.1%. The other 72 were not recorded as successful. We have not established a complete cause-by-cause breakdown for those 72 outcomes.

Recorded activity, October 9–November 3, 2026
MetricObserved value
Observation periodOctober 9–November 3, 2026
Total requests2,508
Recorded successful requests2,436
Other recorded outcomes72
Recorded request success rate97.1%
Total recorded tokensApproximately 13.1 million
Editor-key requests2,466 (98.3%)
Chat requests42 (1.7%)
Active requesters6
Live API keys3
Busiest recorded dayOctober 22: 2,057 requests

These are descriptive request-log figures, not controlled benchmarks. They do not measure code quality, model accuracy or sustained capacity.

Centizen request log summary showing 2,508 total requests, 2,436 successful requests, 72 other outcomes and 13.1 million recorded tokens.
These are observed counts for the stated period, not a benchmark run.

There were six active requesters and three live API keys in the records. The busiest recorded day was October 22, with 2,057 requests. A peak of six requests per second also appears in the notes. Neither number should be treated as a tested sustained capacity limit; a busy day and a controlled load test answer different questions.

October 22 accounted for approximately 82% of all recorded requests in this period. The traffic was therefore concentrated rather than evenly distributed across the dates. The records used for this article do not establish why that day was unusually busy.

This was mainly developer-tool traffic

Of the 2,508 requests, 2,466 (98.3%) were associated with editor API keys. Only 42 (1.7%) were recorded as chat requests. That split changed how we interpreted the service. We were not primarily evaluating a chatbot. We were learning how a model behaves when an editor packages source code, instructions, tool descriptions and conversation history into requests.

Comparison of 2,466 editor API key requests and 42 chat requests, representing 98.3 percent and 1.7 percent of recorded traffic.
Editor-key traffic dominated the observed period. It does not tell us how many coding tasks were completed.

It also explains why a basic chat prompt was a weak acceptance test. A model might answer a direct question while an editor still fails to parse a tool call or rejects a streamed response.

A 97.1% success rate needs context

The recorded success rate is useful for spotting operational trends, but it is not a measure of code correctness, developer productivity or response quality. We did not run a controlled comparison against a hosted model, and we do not have a validated per-request latency distribution in these records.

Recorded 97.1 percent request success rate alongside cautions that the figure does not measure model accuracy, latency or service-level guarantees.
A successful request is one signal. It is not the same as a useful or correct answer.

Similarly, 13.1 million tokens is total recorded token volume. Dividing it by elapsed time would not give a meaningful decoding-speed benchmark. Input and output tokens, prompt length, concurrency and idle periods all affect the interpretation.

Where the integration work went

Some of the most time-consuming issues were not about whether Qwen could generate an answer. They appeared at the boundaries between the coding editor, our internal gateway, proxy routing and the vLLM response format.

Troubleshooting flow from coding editor through gateway, proxy and vLLM back to the client, showing possible failure locations.
Tracing the whole request path helped us avoid treating every client error as a model error.

We investigated editor tool-schema mismatches, arbitrary limits on tool-description lengths, incorrect token estimators, and parser settings that did not match the expected reasoning or tool-call format. Streaming introduced another problem: some clients expected different reasoning or tool-call delta fields than the server emitted.

We also encountered issues outside inference: a VS Code maxInputTokens setting, HTTPS private-network CORS behavior, and HAProxy path rewriting. A healthy inference endpoint did not guarantee that the actual editor workflow was healthy.

Four categories of integration issues: tool schemas, token estimation, streaming response compatibility and networking.
These categories are documented examples, not a frequency ranking of all failed requests.

What changed in our testing approach

We started treating the client request as the unit of testing. Instead of asking only whether the server responds, we checked what the editor sends, what the gateway forwards, what vLLM streams, and what the editor can interpret.

Token budgeting is a good example. The text visible in the editor is only part of the request. Tool definitions and surrounding context consume tokens too. A rough estimate can be useful early, but near a hard limit we need an exact tokenization check against the serving model and tokenizer.

We also learned to confirm that a configuration change reached the running process. Editing a launch file is not proof that the active container picked up the new parser, context limit or routing setting.

Six-step validation sequence covering API calls, real editor payloads, streaming, long prompts, proxy access and parallel-load recovery.
A practical regression sequence to repeat when the model, client or inference configuration changes.

What we can—and cannot—conclude about capacity

Our setup uses an on-premises GPU with 32 GB of VRAM, quantized Qwen3 models and vLLM. It was used for real internal requests. That supports a narrow conclusion: this arrangement was workable for the traffic we observed. It does not establish a universal GPU requirement or a concurrency promise for another team.

The source notes do not document a controlled throughput benchmark, p95 latency, measured cost per request, or a complete GPU utilization series. We also have no recorded out-of-memory, CUDA mismatch or model-load incidents in this evidence set; that is not proof such incidents could never occur.

The lessons we would carry into the next deployment

  • Test the real editor early. A direct API call misses tool schemas, streaming and client token settings.
  • Log enough to locate a failure. Separate editor, gateway, proxy and inference behavior without exposing private prompts or credentials.
  • Record the exact runtime. Pin model revision, vLLM image digest, parser settings, GPU driver and launch command.
  • Use precise metrics. Request success is not answer quality; token volume is not tokens per second.
  • Load-test before expanding. Reproduce long-context and parallel workloads instead of extrapolating from a single busy day.

The most useful outcome of this work was not a headline benchmark. It was a clearer picture of the integration work required to make self-hosted inference useful to a development team. We now know which boundaries need repeatable tests—and which performance claims still need evidence.

Frequently asked questions

Can a self-hosted Qwen3 model support software development teams?

Centizen's records show that developers used a Qwen3/vLLM service through their coding tools. The deployment was workable for the requests we observed, but those records do not establish capacity or performance requirements for every development team.

What were the main problems with self-hosting Qwen3 using vLLM?

The documented issues were primarily integration-related: tool schemas, token estimation, tool-call and reasoning parser settings, streaming response formats, HTTPS access and proxy routing. We cannot attribute every unsuccessful request to a particular cause.

Does a 97.1% request success rate mean the model produced correct code?

No. The 97.1% figure describes how the service recorded request outcomes. It does not measure code correctness, response quality, development speed or task completion.

How should teams test a self-hosted LLM for coding tools?

Use real editor requests, not only direct API calls. Include tool calls, long prompts, streamed responses, token limits, and parallel workloads. Trace each request from the editor through the gateway and inference engine, then check what the editor actually accepted.

Technical references

Centizen request counts and integration observations in this article come from our internal records, not from these external references.

Explore the series: Start with Part 1: Self-Hosting Open-Source LLMs, then read Part 2: Our Qwen/vLLM architecture and engineering decisions. This third article covers our observed request activity and the integration lessons behind it.

Our services:

  • Staffing: Contract, contract-to-hire, direct hire, remote global hiring, SOW projects, and managed services.
  • Remote hiring: Hire full-time IT professionals from our India-based talent network.
  • Custom software development: Web/Mobile Development, UI/UX Design, QA & Automation, API Integration, DevOps, and Product Development.

Our products:

  • ZenBasket: A customizable ecommerce platform.
  • Zenyo payroll: Automated payroll processing for India.
  • Zenyo workforce: Streamlined HR and productivity tools.