vLLM Provider Setup for OpenClaw

vLLM exposes open-source (and some custom) models via an HTTP API that follows the OpenAI specification. OpenClaw talks to it through the openai-completions API, and can auto-discover available models when you enable that with VLLM_API_KEY.

Getting started

Start vLLM with an OpenAI-compatible server

The /v1 endpoints (/v1/models, /v1/chat/completions) must be reachable at your base URL. Typical deployment targets for vLLM are:

http://127.0.0.1:8000/v1

Set the API key environment variable

When your server doesn't enforce authentication, any non-empty value is sufficient:

export VLLM_API_KEY="vllm-local"

Select a model

Use one of your vLLM model IDs in place of the example:

{
  agents: {
    defaults: {
      model: { primary: "vllm/your-model-id" },
    },
  },
}

Verify the model is available

openclaw models list --provider vllm

Tip

For automated environments (CI, scripting), supply the base URL, key, and model directly:

openclaw onboard --non-interactive --accept-risk --skip-health \
  --mode local \
  --auth-choice vllm \
  --custom-base-url "http://127.0.0.1:8000/v1" \
  --custom-api-key "vllm-local" \
  --custom-model-id "your-model-id"

Model discovery (implicit provider)

With VLLM_API_KEY configured (or an auth profile in place) and models.providers.vllm left unset, OpenClaw calls GET http://127.0.0.1:8000/v1/models and turns the model IDs it returns into entries in the model list.

Note

Declaring models.providers.vllm explicitly restricts OpenClaw to only those models you list. To have OpenClaw additionally query that provider's /models endpoint and pull in every advertised vLLM model, add "vllm/*": {} to agents.defaults.models.

Explicit configuration

Set up explicit configuration when vLLM runs on a different host or port, you want to lock in contextWindow/maxTokens, your server demands a genuine API key, or you're connecting to a trusted loopback, LAN, or Tailscale address:

{
  models: {
    providers: {
      vllm: {
        baseUrl: "http://127.0.0.1:8000/v1",
        apiKey: "${VLLM_API_KEY}",
        api: "openai-completions",
        timeoutSeconds: 300, // Optional: extend request timeout for slow local models
        models: [
          {
            id: "your-model-id",
            name: "Local vLLM Model",
            reasoning: false,
            input: ["text"],
            cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
            contextWindow: 128000,
            maxTokens: 8192,
          },
        ],
      },
    },
  },
}

To keep the provider dynamic without enumerating each model, place a wildcard in the visible model catalog:

{
  agents: {
    defaults: {
      models: {
        "vllm/*": {},
      },
    },
  },
}

Advanced configuration

Proxy-style behavior

OpenClaw handles vLLM as a proxy-style OpenAI-compatible /v1 backend, not as a native OpenAI endpoint:

Qwen thinking controls

For Qwen models, put compat.thinkingFormat: "qwen-chat-template" on the model row when the server expects Qwen chat-template kwargs. These models present a binary /think profile (off, on) because Qwen chat-template thinking is a simple on/off switch, not an OpenAI-style graduated effort scale.

{
  models: {
    providers: {
      vllm: {
        models: [
          {
            id: "Qwen/Qwen3-8B",
            name: "Qwen3 8B",
            reasoning: true,
            compat: { thinkingFormat: "qwen-chat-template" },
          },
        ],
      },
    },
  },
}

OpenClaw translates /think off into:

{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "preserve_thinking": true
  }
}

Thinking levels that aren't off result in enable_thinking: true being sent. If your endpoint prefers DashScope-style top-level flags, switch to compat.thinkingFormat: "qwen" to place enable_thinking at the request root.

Nemotron 3 thinking controls

For vllm/nemotron-3-* models with thinking disabled, the bundled plugin transmits:

{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "force_nonempty_content": true
  }
}

To override these values, set chat_template_kwargs under the model params. If params.extra_body.chat_template_kwargs is also present, it takes precedence because extra_body acts as the final request-body override.

{
  agents: {
    defaults: {
      models: {
        "vllm/nemotron-3-super": {
          params: {
            chat_template_kwargs: {
              enable_thinking: false,
              force_nonempty_content: true,
            },
          },
        },
      },
    },
  },
}

Qwen tool calls appear as text

Start by confirming vLLM was launched with the correct tool-call parser and chat template for the model. vLLM's docs recommend hermes for Qwen2.5 models and qwen3_xml for Qwen3-Coder models.

Symptoms: skills or tools never execute, the assistant outputs raw JSON/XML like {"name":"read","arguments":...}, or vLLM responds with an empty tool_calls array when OpenClaw sends tool_choice: "auto".

Certain Qwen/vLLM setups only deliver structured tool calls when the request includes tool_choice: "required". Enforce this per model using params.extra_body:

{
  agents: {
    defaults: {
      models: {
        "vllm/Qwen-Qwen2.5-Coder-32B-Instruct": {
          params: {
            extra_body: {
              tool_choice: "required",
            },
          },
        },
      },
    },
  },
}

Substitute the model id with the exact one from openclaw models list --provider vllm, or apply the same override via the CLI:

openclaw config set agents.defaults.models '{"vllm/Qwen-Qwen2.5-Coder-32B-Instruct":{"params":{"extra_body":{"tool_choice":"required"}}}}' --strict-json --merge

This workaround is opt-in: it forces every tool-enabled turn to produce a tool call, so reserve it for a dedicated model entry where that behavior is acceptable. Avoid setting it globally for all vLLM models, and never combine it with a proxy that turns arbitrary assistant text into executable tool calls.

Custom base URL

When your vLLM server listens on a custom host or port, specify baseUrl in the explicit provider config:

{
  models: {
    providers: {
      vllm: {
        baseUrl: "http://192.168.1.50:9000/v1",
        apiKey: "${VLLM_API_KEY}",
        api: "openai-completions",
        timeoutSeconds: 300,
        models: [
          {
            id: "my-custom-model",
            name: "Remote vLLM Model",
            reasoning: false,
            input: ["text"],
            contextWindow: 64000,
            maxTokens: 4096,
          },
        ],
      },
    },
  },
}

Troubleshooting

Slow first response or remote server timeout

For large local models, remote LAN hosts, or tailnet connections, define a provider-scoped request timeout:

{
  models: {
    providers: {
      vllm: {
        baseUrl: "http://192.168.1.50:8000/v1",
        apiKey: "${VLLM_API_KEY}",
        api: "openai-completions",
        timeoutSeconds: 300,
        models: [{ id: "your-model-id", name: "Local vLLM Model" }],
      },
    },
  },
}

timeoutSeconds only governs vLLM model HTTP requests: connection establishment, response headers, body streaming, and the overall guarded-fetch abort. It also lifts the LLM idle/stream watchdog cap above the implicit ~120s default for this provider. Prefer this over raising agents.defaults.timeoutSeconds, which controls the entire agent run.

Server not reachable

Verify that the vLLM server is up and reachable:

curl http://127.0.0.1:8000/v1/models

On a connection error, confirm the host, port, and that vLLM is running in OpenAI-compatible server mode. OpenClaw trusts the exact configured models.providers.vllm.baseUrl origin for guarded model requests on loopback, LAN, and Tailscale endpoints. Metadata, link-local, and local-use NAT64 (64:ff9b:1::/48) origins stay blocked without explicit opt-in. Set models.providers.vllm.request.allowPrivateNetwork: true only when vLLM requests must reach another private origin, or use false to disable exact-origin trust.

Auth errors on requests

If auth errors appear, provide a real VLLM_API_KEY matching your server configuration, or set up the provider explicitly under models.providers.vllm.

Tip

When your vLLM server does not enforce auth, any non-empty VLLM_API_KEY value works as an opt-in signal for OpenClaw.

No models discovered

Auto-discovery needs VLLM_API_KEY to be configured. If models.providers.vllm is defined, OpenClaw relies solely on your declared models unless agents.defaults.models includes "vllm/*": {}.

Tools render as raw text

When a Qwen model emits JSON/XML tool syntax instead of running a skill:

• Launch vLLM with the correct parser/template for that model.
• Verify the exact model id with openclaw models list --provider vllm.
• Add a dedicated per-model params.extra_body.tool_choice: "required" override only if tool_choice: "auto" still returns empty or text-only tool calls.

Warning

Further assistance: Troubleshooting and FAQ.

Related

• Model selection, Picking providers, model refs, and failover behavior.

• OpenAI, Native OpenAI provider and OpenAI-compatible route behavior.

• OAuth and auth, Auth details and credential reuse rules.

• Troubleshooting, Common issues and how to resolve them.