Neura Market Logo

Run OpenClaw through llmman: OpenAI-compatible local server

Configure the llmman provider so OpenClaw can talk to a self-hosted OpenAI-compatible server. Covers provider setup, base URL, auth behavior, and the openai-completions adapter.

Read this when

  • You want to run OpenClaw against local GGUF or safetensors models through llmman
  • You want hybrid inference that keeps small requests local and overflows large ones to a hosted model
  • You need llmman setup, configuration, vision, or troubleshooting guidance

llmman fetches GGUF and safetensors models out of OCI registries, then serves them through APIs compatible with Ollama, OpenAI, and Anthropic. GGUF models are handled with llama-server, while safetensors models rely on vllm or mlx_lm.server. Communication from OpenClaw goes through the generic openai-completions adapter.

PropertyValue
Provider idllmman (custom; configure under models.providers.llmman)
Pluginnone, not a bundled OpenClaw provider plugin
Auth env varnone required; any value works, llmman serve has no auth
APIOpenAI-compatible (openai-completions)
Default base URLhttp://127.0.0.1:17434/v1

Note

llmman is a self-hosted OpenAI-compatible backend you configure yourself, not a dedicated OpenClaw provider plugin: setup happens under models.providers.llmman rather than by choosing an onboarding auth option. If you want a bundled plugin that discovers itself automatically, look at SGLang or vLLM.

Info

Version scope: the content here is verified against llmman b315, commit 0e7a3ed.

Getting started

Start llmman with a model

LLMMAN_CONTEXT_LENGTH=65536 llmman serve gemma4

By default, llmman serve binds to 127.0.0.1:17434. To change the bind address, set LLMMAN_HOST before starting up; no --host/--port flags exist for this. CUDA, ROCm, Vulkan, and Metal acceleration are detected automatically; because no --device flag is available, use LLMMAN_LLM_LIBRARY to override the detection. Supplying the model argument is optional: leave it out and the server starts anyway, loading models when the first request names them.

In the example, the server context is pinned to 65,536 tokens, and OpenClaw below uses that same number. Should you alter LLMMAN_CONTEXT_LENGTH, make sure the OpenClaw model's contextWindow does not exceed it.

Verify the server is reachable

curl http://127.0.0.1:17434/v1/models
curl http://127.0.0.1:17434/api/version

At the top level, llmman serve offers no dedicated /health route; for a readiness probe, reach for /v1/models or /api/version instead.

Add an OpenClaw provider entry

Declare a provider entry explicitly, then aim your default model at it. The config example below shows how.

Full config example

Gemma 4 running on a local llmman server:

{
  agents: {
    defaults: {
      model: { primary: "llmman/gemma4" },
      models: {
        "llmman/gemma4": {
          alias: "Gemma 4 (llmman)",
        },
      },
    },
  },
  models: {
    mode: "merge",
    providers: {
      llmman: {
        baseUrl: "http://127.0.0.1:17434/v1",
        apiKey: "llmman-local",
        api: "openai-completions",
        models: [
          {
            id: "gemma4",
            name: "Gemma 4 (llmman)",
            reasoning: false,
            input: ["text"],
            cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
            contextWindow: 65536,
            maxTokens: 4096,
          },
        ],
      },
    },
  },
}

On-demand startup

Only when an llmman/... model is selected can OpenClaw launch llmman on its own. Put localService into that same provider entry:

{
  models: {
    providers: {
      llmman: {
        baseUrl: "http://127.0.0.1:17434/v1",
        apiKey: "llmman-local",
        api: "openai-completions",
        timeoutSeconds: 300,
        localService: {
          command: "/opt/homebrew/bin/llmman",
          args: ["serve", "gemma4"],
          env: { LLMMAN_CONTEXT_LENGTH: "65536" },
          healthUrl: "http://127.0.0.1:17434/v1/models",
          readyTimeoutMs: 180000,
          idleStopMs: 0,
        },
        models: [
          {
            id: "gemma4",
            name: "Gemma 4 (llmman)",
            reasoning: false,
            input: ["text"],
            cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
            contextWindow: 65536,
            maxTokens: 4096,
          },
        ],
      },
    },
  },
}

The path given for command has to be absolute. Execute which llmman on the Gateway host, then use the resulting path. For the complete field reference, see Local model services.

Advanced configuration

Why requiresStringContent might matter

The requested model gets resolved and loaded by llmman, its id is rewritten for whichever backend was chosen, and generation defaults like repeat_penalty are added. Message content and tool schemas pass through without normalization, so whether those fields work depends on the backend and model you picked.

Warning

Should OpenClaw runs fail with:

messages[1].content: invalid type: sequence, expected a string

then put compat.requiresStringContent: true in the model entry. After that, OpenClaw collapses pure text content parts into plain strings prior to sending the request.

Tool-schema caveat

When a model handles small direct /v1/chat/completions requests fine but breaks on full OpenClaw agent-runtime turns, start by turning off the tool schema surface:

compat: {
  supportsTools: false
}

Doing so lowers prompt pressure for stricter local backends. If tiny direct requests continue to succeed while ordinary OpenClaw agent turns keep crashing inside llama-server, consider it a limitation of the upstream model or server, not something wrong with OpenClaw's transport.

Manual smoke test

Once configured, exercise both layers:

curl http://127.0.0.1:17434/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"gemma4","messages":[{"role":"user","content":"What is 2 + 2?"}],"stream":false}'

openclaw infer model run \
  --model llmman/gemma4 \
  --prompt "What is 2 + 2? Reply with one short sentence." \
  --json

When the first command succeeds but the second one does not, check the Troubleshooting section that follows.

Proxy-style behavior

Since llmman relies on the generic openai-completions adapter rather than openai-responses, request shaping that is exclusive to native OpenAI never kicks in. Nothing gets sent for service_tier, no Responses store, no prompt-cache hints, and no OpenAI reasoning-compat payload shaping either.

Troubleshooting

curl /v1/models fails

llmman serve is either not running, or it cannot be reached at the address you configured. 127.0.0.1:17434 is the default; should you set LLMMAN_HOST, you must also update the OpenClaw baseUrl and healthUrl so they line up.

messages[].content expected a string

Add compat.requiresStringContent: true to the model entry (described earlier).

Direct /v1/chat/completions calls pass but openclaw infer model run fails

Neither probe uses tools, so compat.supportsTools has no bearing on this failure. Verify the base URL and model id you configured, look through the llmman/backend logs, and put the two request payloads and their responses side by side.

Model run passes but a normal agent turn fails

A larger prompt is involved in the agent turn, and tool schemas may be part of it. Run compat.supportsTools: false to isolate pressure coming from tool schemas (refer to the tool-schema caveat above).

llama-server still crashes on larger agent turns

When schema errors disappear but the spawned llama-server still crashes on bigger agent turns, treat the cause as an upstream llama.cpp or a model limitation. Lower the prompt pressure, or move to another backend or model.

Tip

For general help, see Troubleshooting and FAQ.

1,083 words · updated Sep 18, 2026