vLLM Serving Guide for Hermes Agent (MLOps Skill)

vLLM is a serving engine for large language models that trades the simplicity of a plain transformers loop for throughput you can actually put behind a production endpoint. This Hermes Agent skill walks you through installing it, standing up an OpenAI-compatible server, running offline batch jobs, and squeezing large models into limited GPU memory with quantization. Reach for it when you are past prototyping and need to answer many concurrent requests without the GPU sitting idle.

What it does

The skill covers the three shapes vLLM work usually takes. First, offline inference: you load a model into a Python process and call generate on a list of prompts. Second, an OpenAI-compatible HTTP server, so existing clients that speak the OpenAI chat completions API can point at your own hardware. Third, quantized serving, which lets a model that would not otherwise fit share a GPU with its KV cache.

The reason vLLM is worth the extra moving parts is PagedAttention, which manages the KV cache in blocks rather than one contiguous allocation per sequence, plus continuous batching, which mixes prefill and decode requests in the same batch. The source states this combination gives 24x higher throughput than standard transformers. That number is the whole argument for the skill: if you are serving one user at a time, the machinery buys you nothing.

Before you start

Install the package with pip:

pip install vllm

The skill is optional in Hermes Agent and installed on demand. It is version 1.0.1, authored by Orchestra Research, MIT licensed, and listed for linux and macos. The upstream tags tell you what to expect: vLLM, Inference Serving, PagedAttention, Continuous Batching, High Throughput, Production, OpenAI API, Quantization, Tensor Parallelism.

On hardware, the source is specific. Small models in the 7B to 13B range want a single A10 (24GB) or A100 (40GB). Medium models at 30B to 40B want two A100 (40GB) cards with tensor parallelism. Large models at 70B and above want four A100 (40GB) or two A100 (80GB), and the source recommends AWQ or GPTQ at that size. Supported platforms are NVIDIA as the primary target, plus AMD ROCm, Intel GPUs, and TPUs.

One practical implication: tensor parallelism is a multi-GPU feature, so the tensor-parallel examples below only make sense on a machine where those GPUs are visible to the process. If you are on a laptop, the single-GPU paths are the ones that apply.

Quick start

Offline inference in Python

The shortest path to a generated token is the LLM class with SamplingParams:

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Meta-Llama-3-8B-Instruct")
sampling = SamplingParams(temperature=0.7, max_tokens=256)

outputs = llm.generate(["Explain quantum computing"], sampling)
print(outputs[0].outputs[0].text)

You construct the engine once, describe the sampling behaviour once, then pass a list of prompts. The result is a list of request outputs, and the generated string lives under the first output of the first request. This is the pattern to use for scripts, evaluation runs, and anything where an HTTP hop is overhead you do not need.

OpenAI-compatible server

To expose the model over HTTP instead:

vllm serve meta-llama/Meta-Llama-3-8B-Instruct

# Query with OpenAI SDK
python -c "
from openai import OpenAI
client = OpenAI(base_url='http://localhost:8000/v1', api_key='EMPTY')
print(client.chat.completions.create(
    model='meta-llama/Meta-Llama-3-8B-Instruct',
    messages=[{'role': 'user', 'content': 'Hello!'}]
).choices[0].message.content)
"

The serve subcommand starts the server, and the OpenAI SDK talks to it by pointing base_url at the local /v1 path. The API key is a placeholder string because the server is not authenticating you; it exists so the client library is satisfied. The model name you pass to chat.completions.create matches the one you served.

Common workflows

Workflow 1: Production API deployment

Copy this checklist and track progress:

Deployment Progress:
- [ ] Step 1: Configure server settings
- [ ] Step 2: Test with limited traffic
- [ ] Step 3: Enable monitoring
- [ ] Step 4: Deploy to production
- [ ] Step 5: Verify performance metrics

Step 1: Configure server settings

Choose configuration based on your model size:

# For 7B-13B models on single GPU
vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
  --gpu-memory-utilization 0.9 \
  --max-model-len 8192 \
  --port 8000

# For 30B-70B models with tensor parallelism
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9 \
  --quantization awq \
  --port 8000

# For production with caching (Prometheus metrics are exposed
# automatically at /metrics on the API port)
vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
  --gpu-memory-utilization 0.9 \
  --enable-prefix-caching \
  --port 8000 \
  --host 0.0.0.0

The three blocks map to three situations. A single-GPU 7B to 13B deployment sets how much of the card vLLM may claim and caps the context length. A 30B to 70B deployment adds --tensor-parallel-size to spread the model across cards and --quantization awq to shrink it. The production variant adds prefix caching and binds to all interfaces with --host 0.0.0.0, which is what you want behind a load balancer and not what you want on an untrusted network.

Step 2: Test with limited traffic

Run load test before production:

# Install load testing tool
pip install locust

# Create test_load.py with sample requests
# Run: locust -f test_load.py --host http://localhost:8000

Verify TTFT (time to first token) < 500ms and throughput > 100 req/sec.

The source gives you the acceptance bar directly: time to first token under 500ms and more than 100 requests per second. Writing the actual test_load.py is on you; the skill only tells you the tool and the invocation.

Step 3: Enable monitoring

vLLM exposes Prometheus metrics at /metrics on the API port (default 8000):

curl http://localhost:8000/metrics | grep vllm

Key metrics to monitor:

vllm:time_to_first_token_seconds - Latency
vllm:num_requests_running - Active requests
vllm:gpu_cache_usage_perc - KV cache utilization

These three cover the questions you will actually ask during an incident. Latency tells you whether users are waiting. Active requests tells you whether the server is busy or starved. KV cache utilization tells you how close you are to the memory ceiling that causes preemption and, eventually, out-of-memory failures.

Step 4: Deploy to production

Use Docker for consistent deployment:

# Run vLLM in Docker
docker run --gpus all -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --gpu-memory-utilization 0.9 \
  --enable-prefix-caching

The container image takes the same server flags after the image name, so the configuration you tested outside Docker carries over. --gpus all is what gives the container access to the cards.

Step 5: Verify performance metrics

Check that deployment meets targets:

TTFT < 500ms (for short prompts)
Throughput > target req/sec
GPU utilization > 80%
No OOM errors in logs

Note the qualifier on the first item: the 500ms TTFT target is stated for short prompts. Long prompts change the arithmetic, which is why the next workflow section covers chunked prefill.

Workflow 2: Offline batch inference

For processing large datasets without server overhead.

Copy this checklist:

Batch Processing:
- [ ] Step 1: Prepare input data
- [ ] Step 2: Configure LLM engine
- [ ] Step 3: Run batch inference
- [ ] Step 4: Process results

Step 1: Prepare input data

# Load prompts from file
prompts = []
with open("prompts.txt") as f:
    prompts = [line.strip() for line in f]

print(f"Loaded {len(prompts)} prompts")

One prompt per line, stripped of whitespace. Nothing exotic, but it keeps the batch job's input format obvious.

Step 2: Configure LLM engine

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    tensor_parallel_size=2,  # Use 2 GPUs
    gpu_memory_utilization=0.9,
    max_model_len=4096
)

sampling = SamplingParams(
    temperature=0.7,
    top_p=0.95,
    max_tokens=512,
    stop=["</s>", "\n\n"]
)

Here the engine is configured with the same knobs the CLI exposes, just as constructor arguments: tensor parallelism across two GPUs, a memory fraction, and a context cap. The sampling parameters add top_p and a stop list, which matters for batch jobs where you want generation to end at a natural boundary rather than run to the token limit.

Step 3: Run batch inference

vLLM automatically batches requests for efficiency:

# Process all prompts in one call
outputs = llm.generate(prompts, sampling)

# vLLM handles batching internally
# No need to manually chunk prompts

This is the part that surprises people coming from manual batching code. You hand over the whole list and the engine schedules it. Do not write your own chunking loop on top; the source explicitly says it is unnecessary.

Step 4: Process results

# Extract generated text
results = []
for output in outputs:
    prompt = output.prompt
    generated = output.outputs[0].text
    results.append({
        "prompt": prompt,
        "generated": generated,
        "tokens": len(output.outputs[0].token_ids)
    })

# Save to file
import json
with open("results.jsonl", "w") as f:
    for result in results:
        f.write(json.dumps(result) + "\n")

print(f"Processed {len(results)} prompts")

Each output carries the original prompt back, the generated text, and the token ids, which is how you get a token count without a separate tokenizer pass. Writing one JSON object per line gives you a file that streams well and appends cleanly if you later split the job across runs.

Workflow 3: Quantized model serving

Fit large models in limited GPU memory.

Quantization Setup:
- [ ] Step 1: Choose quantization method
- [ ] Step 2: Find or create quantized model
- [ ] Step 3: Launch with quantization flag
- [ ] Step 4: Verify accuracy

Step 1: Choose quantization method

The source gives three options with a clear recommendation each:

AWQ: Best for 70B models, minimal accuracy loss
GPTQ: Wide model support, good compression
FP8: Fastest on H100 GPUs

Pick AWQ when you are pushing a very large model onto cards you already have. Pick GPTQ when the model you want is only available in that format. Pick FP8 when the hardware is H100 and speed is the priority.

Step 2: Find or create quantized model

Use pre-quantized models from HuggingFace:

# Search for AWQ models
# Example: TheBloke/Llama-2-70B-AWQ

This step is a pointer rather than a command. The convention is that the quantization method appears in the repository name, so you can identify a compatible checkpoint before downloading tens of gigabytes.

Step 3: Launch with quantization flag

# Using pre-quantized model
vllm serve TheBloke/Llama-2-70B-AWQ \
  --quantization awq \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.95

# Results: 70B model in ~40GB VRAM

The payoff line is in the comment: a 70B model fitting in roughly 40GB of VRAM. That is the difference between needing a multi-GPU node and needing one card, which is usually the entire reason someone opens this workflow.

Step 4: Verify accuracy

Test outputs match expected quality:

# Compare quantized vs non-quantized responses
# Verify task-specific performance unchanged

Quantization is a trade, and this step is where you check what you paid. The source frames it as comparing quantized against non-quantized responses and confirming task-specific performance has not moved. Do this before you route real traffic, not after.

When not to use it

Use vLLM when you are deploying production LLM APIs at 100+ requests per second, serving OpenAI-compatible endpoints, fitting large models into limited GPU memory, running multi-user applications like chatbots and assistants, or you need low latency alongside high throughput.

Use alternatives instead in these cases, as the source lists them:

llama.cpp: CPU or edge inference, single-user
HuggingFace transformers: Research, prototyping, one-off generation
TensorRT-LLM: NVIDIA-only, when you need absolute maximum performance
Text-Generation-Inference: When you are already in the HuggingFace ecosystem

The pattern is that vLLM's advantages are concurrency advantages. If your workload is one request at a time, or you are still deciding what to build, the alternatives are a better fit and the operational surface is smaller.

Limits and gotchas

The source collects its failure modes under common issues, and they are worth reading before you deploy rather than after.

Out of memory during model loading. Lower the memory fraction and the context cap:

vllm serve MODEL \
  --gpu-memory-utilization 0.7 \
  --max-model-len 4096

Or use quantization:

vllm serve MODEL --quantization awq

Slow first token (TTFT > 1 second). Enable prefix caching for repeated prompts:

vllm serve MODEL --enable-prefix-caching

For long prompts, enable chunked prefill:

vllm serve MODEL --enable-chunked-prefill

Model not found error. Use --trust-remote-code for custom models:

vllm serve MODEL --trust-remote-code

Low throughput (<50 req/sec). Increase concurrent sequences:

vllm serve MODEL --max-num-seqs 512

Check GPU utilization with nvidia-smi - should be >80%.

Inference slower than expected. Verify tensor parallelism uses power of 2 GPUs:

vllm serve MODEL --tensor-parallel-size 4  # Not 3

Enable speculative decoding for faster generation (pass config as JSON; --speculative-model was removed in favor of --speculative-config):

vllm serve MODEL \
  --speculative-config '{"model": "DRAFT_MODEL", "num_speculative_tokens": 5, "method": "draft_model"}'

Two of these deserve emphasis. The power-of-two rule for tensor parallelism is a hard constraint, not a suggestion, and a three-GPU configuration is a mistake you will only notice as unexplained slowness. And the speculative decoding flag is a migration trap: if you are following older material, --speculative-model no longer exists and the configuration now goes through --speculative-config as a JSON string.

Related references

The skill ships with deeper material for each of these areas:

Server deployment patterns: See references/server-deployment.md for Docker, Kubernetes, and load balancing configurations.
Performance optimization: See references/optimization.md for PagedAttention tuning, continuous batching details, and benchmark results.
Quantization guide: See references/quantization.md for AWQ/GPTQ/FP8 setup, model preparation, and accuracy comparisons.
Troubleshooting: See references/troubleshooting.md for detailed error messages, debugging steps, and performance diagnostics.

Upstream, the project lives at https://docs.vllm.ai with source at https://github.com/vllm-project/vllm, a community forum at https://discuss.vllm.ai, and the underlying design described in the paper "Efficient Memory Management for Large Language Model Serving with PagedAttention" (SOSP 2023).

vLLM Serving for Hermes Agent: High-Throughput LLM APIs