llama.cpp Provider Plugin for Local GGUF Inference & Embeddings

llama-cpp is the official external provider plugin that handles local GGUF text inference and embeddings within the same process. It registers llama-cpp as a text provider, local as an embedding provider, and manages the node-llama-cpp native runtime.

Before you can use local inference or local memory embeddings, install the plugin:

openclaw plugins install @openclaw/llama-cpp-provider

The openclaw npm package does not ship with node-llama-cpp bundled. Keeping the native dependency inside this plugin prevents routine OpenClaw npm updates from overwriting a manually installed runtime located in the OpenClaw package directory.

Local text inference

During interactive setup, pick Local model (llama.cpp). OpenClaw will ask for permission before downloading the default model:

hf:bartowski/Qwen_Qwen3-4B-Instruct-2507-GGUF/Qwen_Qwen3-4B-Instruct-2507-Q4_K_M.gguf

The Qwen3 4B Instruct 2507 Q4_K_M file is roughly 2.5 GB. Plan for about 3 GB of RAM to hold the model weights, plus additional memory for context and OpenClaw runtime overhead. The default context is automatically configured with a cap of 8,192 tokens, keeping it usable on machines with 8 GB of RAM. Increase the context size only when the system has sufficient memory.

The discovery check performed during onboarding is read-only. llama.cpp is offered automatically only when the default or configured GGUF file is already present in the model cache; no download occurs during discovery. Ollama and LM Studio remain separate local service options and follow their own discovery flows. Selecting llama.cpp manually is the step that triggers a prompt to download the default model.

This provider uses the chat template embedded in the GGUF model and native function calling from node-llama-cpp. Text is streamed token by token. Tool calls are sent back to OpenClaw for execution rather than being handled inside node-llama-cpp.

Use another GGUF model

Add a model to models.providers.llama-cpp. Provide either a local path or a full hf: file URI in params.modelPath:

{
  models: {
    mode: "merge",
    providers: {
      "llama-cpp": {
        baseUrl: "local://llama-cpp",
        api: "openai-completions",
        params: {
          modelCacheDir: "~/.node-llama-cpp/models",
        },
        models: [
          {
            id: "my-local-model",
            name: "My local GGUF",
            reasoning: false,
            input: ["text"],
            cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
            contextWindow: 8192,
            maxTokens: 2048,
            params: {
              modelPath: "~/Models/my-model.Q4_K_M.gguf",
              contextSize: 8192,
            },
            compat: { supportsTools: true },
          },
        ],
      },
    },
  },
  agents: {
    defaults: {
      model: { primary: "llama-cpp/my-local-model" },
    },
  },
}

Inference never downloads a missing model automatically. For a custom hf: URI, first download the GGUF file into modelCacheDir. Discovery relies on node-llama-cpp's own read-only cache resolver, which handles repository, branch, and split-file naming.

Memory embedding configuration

Set memory.search.provider to local:

{
  memory: {
    search: {
      provider: "local",
      local: {
        modelPath: "hf:ggml-org/embeddinggemma-300m-qat-q8_0-GGUF/embeddinggemma-300m-qat-Q8_0.gguf",
      },
    },
  },
}

By default, local.modelPath points to the hf: URI shown above (embeddinggemma-300m-qat-Q8_0.gguf). To use a different model, point it at another hf: URI or a local .gguf file. local.modelCacheDir overrides the download cache location (default: ~/.node-llama-cpp/models), and local.contextSize accepts either an integer or "auto".

When local.contextSize is a number, the provider passes that requirement to node-llama-cpp for automatic GPU-layer placement. This allows node-llama-cpp to fit the model and embedding context together while still performing its memory-safety checks. With "auto", node-llama-cpp uses its normal automatic placement.

Native runtime

For the smoothest native installation, use Node 24. Source checkouts using pnpm may require approval and a rebuild of the native dependency:

pnpm approve-builds
pnpm rebuild node-llama-cpp

Memory runtime diagnostics

Run openclaw memory status --deep after the provider has loaded to see the selected backend and build, device names, GPU offloaded layers, requested context size, and the most recent VRAM or unified-memory snapshot. VRAM values include a timestamp because passive status reads do not reload the model or query the device.

The same last-known information can appear in openclaw doctor if the running Gateway has already used the local provider. A normal status or doctor command does not load a model solely for diagnostic purposes.

Troubleshooting

If node-llama-cpp is missing or fails to load, OpenClaw reports the failure with:

1. Install the plugin: openclaw plugins install @openclaw/llama-cpp-provider.
2. Use Node 24 for native installations or updates.
3. From a pnpm source checkout: run pnpm approve-builds, then pnpm rebuild node-llama-cpp.

For local inference without an in-process native dependency, use the Ollama or LM Studio provider instead. For simpler local embeddings, set memory.search.provider to a remote embedding provider such as lmstudio, ollama, openai, or voyage.