llama.cpp Provider: Managed & Existing llama-server Setup

The llama-cpp plugin exposes a single llama-cpp model provider. OpenClaw can either run a local llama-server itself or attach to one already under your control. Both paths rely on llama-cpp/<model> references and the OpenAI-compatible transport.

openclaw plugins install @openclaw/llama-cpp-provider
openclaw onboard

Choose server ownership

Ownership is determined by models.providers.llama-cpp.localService. When that value is present, OpenClaw takes charge of the process. In its absence, baseUrl points to a live endpoint. Switching between the two modes rewrites ownership-specific state on the same provider, without ever spawning a separate provider namespace.

Managed local server

Pick Managed local server when you want OpenClaw to handle installation, startup, and shutdown of the server. After you give consent, setup validates a pinned llama.cpp build, fetches verified chat and embedding models, writes the loopback endpoint and localService definition, and tests the result before committing it.

The default chat model is Gemma 4 E4B IT Q4_K_M, roughly 5.0 GB, with a 65,536 token context cap. OpenClaw only offers it on systems with 16 GiB of RAM or more. The managed EmbeddingGemma model sits around 0.3 GB. Setup discovery stays read-only, so nothing gets installed or downloaded during that phase.

Use another managed GGUF

Add a model under models.providers.llama-cpp.models, pick its llama-cpp/<id> reference, and rerun managed setup:

{
  id: "my-local-model",
  name: "My local GGUF",
  reasoning: false,
  input: ["text"],
  cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
  contextWindow: 65536,
  maxTokens: 2048,
  params: {
    modelPath: "~/Models/my-model.Q4_K_M.gguf",
    contextSize: 65536,
  },
  compat: { supportsTools: true },
}

modelPath handles local paths, cache-relative filenames, full hf: file URIs, and HTTPS GGUF URLs that expose a SHA-256 response digest. The default cache lives at ~/.openclaw/models/llama.cpp; a configured modelCacheDir takes precedence for managed setup.

Existing llama-server

Choose Existing llama-server when another terminal, container, service manager, or machine owns the process.

Start llama-server

Assign the model a stable alias:

llama-server \
  --model /path/to/model.gguf \
  --alias my-model \
  --host 127.0.0.1 \
  --port 8080

Configure OpenClaw

Launch openclaw onboard, select Existing llama-server, and provide the endpoint. Turn on API-key authentication only if the server or proxy demands it.

Select the model

openclaw models list --provider llama-cpp
openclaw models set llama-cpp/my-model

OpenClaw consults /health, /models (with /v1/models as the fallback), and /props. Router property probes rely on autoload=false; discovery never loads, wakes, unloads, downloads, or reloads models. Explicitly configured model rows override discovered rows that share the same ID.

Authentication and endpoint replacement

Existing endpoints support no auth, API keys, SecretRefs, auth profiles, and explicit authorization headers. An explicit Authorization header takes priority over ambient API-key discovery unless setup receives a new key. Omitting an API key removes the default llama.cpp auth profile and stale inline key fields while keeping an explicit Authorization header and unrelated headers intact. Endpoint URLs that embed a username or password get rejected.

export LLAMA_SERVER_API_KEY="<API_KEY>"
openclaw onboard

When the endpoint changes, setup withholds the old endpoint's environment, profile, configured key, and header credentials from the replacement. Moving out of managed mode also clears localService, managed model/cache parameters, and the managed request timeout before discovery runs.

For non-interactive setup:

openclaw onboard \
  --non-interactive \
  --accept-risk \
  --auth-choice llama-cpp-existing-server \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id my-model

Reach for --llama-server-api-key <API_KEY> when a replacement endpoint needs a fresh credential. LLAMA_SERVER_API_KEY stays available for initial setup and endpoints that have not changed.

Manual configuration

Guided setup is the recommended route because it verifies discovery. The minimal manual shape looks like this:

{
  models: {
    mode: "merge",
    providers: {
      "llama-cpp": {
        baseUrl: "http://127.0.0.1:8080/v1",
        api: "openai-completions",
        request: { allowPrivateNetwork: true },
        models: [],
      },
    },
  },
}

Custom provider IDs can also target llama-server through the generic OpenAI-compatible path. They remain custom providers and should declare the llamacpp tool-schema profile explicitly; see custom provider capability declarations.

Requests and local embeddings

Both ownership choices use OpenClaw's standard chat, image, streaming, and tool transport. The llama.cpp compatibility family strips unsupported tool-schema constraints, translates thinking-off requests into the Qwen chat-template flag, and adjusts JSON Schema requests for older llama-server builds.

Local memory embeddings require managed mode:

{
  memory: {
    search: {
      provider: "local",
      local: {
        modelPath: "hf:ggml-org/embeddinggemma-300m-qat-q8_0-GGUF/embeddinggemma-300m-qat-Q8_0.gguf",
      },
    },
  },
}

The plugin keeps the historical local embedding provider and index identity. Run openclaw memory status --index after deliberately changing the embedding model.

Troubleshooting

• Managed setup: execute openclaw doctor and openclaw memory status --deep.
• Existing server: check /health, /models, and /props; HTTP 503 indicates the model is still loading.
• Missing tools: confirm both tool capability flags in /props and apply a tool-capable Jinja chat template.
• Managed Linux builds need glibc 2.34 on x64 or 2.38 on arm64. Windows builds require the Microsoft Visual C++ 2015-2022 Redistributable.
• Platforms without a verified managed build should fall back to an existing server.

OpenClaw never auto-selects CUDA, ROCm, SYCL, OpenVINO, or Vulkan archives.

Related

• Local model services
• Model providers
• LM Studio