Llama.cpp Skill for Hermes Agent: GGUF Inference Guide

This skill gives Hermes Agent a working knowledge of llama.cpp and the GGUF format: how to find a quantized model on the Hugging Face Hub, pick the right quant for the hardware in front of you, and turn that choice into a runnable llama-server or llama-cli command. It is aimed at people running models locally on CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs, and at anyone who needs to answer "which GGUF should I download?" without guessing. If your work involves edge deployment, memory-constrained machines, or simply avoiding a hosted inference bill, this is the skill you install.

What it does

The skill covers three connected jobs. First, model discovery: searching the Hub for repos that already advertise llama.cpp support, then reading the repo's local-app view to see what the maintainers recommend. Second, quant selection: mapping a hardware profile onto a quant label, preferring whatever the Hub page says over generic rules of thumb. Third, command construction: producing the exact llama-server or llama-cli invocation, either copied from the Hub snippet or rebuilt from the repo name plus a chosen quant.

It also covers the Python side. The llama-cpp-python bindings let you load a GGUF directly in a script, run plain generation, use the chat completion interface with streaming, and produce embeddings. That matters when the model is a component inside a larger pipeline rather than a chat endpoint.

A deliberate design choice runs through the whole skill: prefer URL workflows over asking for hf, Python, or custom scripts. The Hub's own pages and its tree API are treated as the source of truth, and the skill's job is to read them correctly rather than to reimplement them.

Before you start

This is an optional skill, installed on demand rather than shipped by default. Its path is optional-skills/mlops/inference/llama-cpp, version 2.1.2, authored by Orchestra Research under the MIT license. It runs on linux, macos, and windows.

You need llama.cpp itself available. The skill documents three install routes, and which one you pick depends on your platform and whether you want a package-managed binary or a build you control.

Install llama.cpp

# macOS / Linux (simplest)
brew install llama.cpp

winget install llama.cpp

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release

Homebrew is the shortest path on macOS and Linux. Winget covers Windows. The source build is the one to choose when you need a configuration the packaged binaries do not give you, and it requires a working CMake toolchain plus the compiler for your platform.

For the Python bindings, the base install is a plain pip command. GPU acceleration is enabled through CMake arguments passed at install time, and the skill notes the CUDA and Metal variants explicitly: pip install llama-cpp-python (CUDA: CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python --force-reinstall --no-cache-dir; Metal: CMAKE_ARGS="-DGGML_METAL=on" ...). The force-reinstall and no-cache flags matter because a cached wheel built without the GPU flag will otherwise be reused.

Model discovery workflow

The skill's ordering is explicit: reach for URL workflows before asking for hf, Python, or custom scripts. That keeps the process inspectable and avoids a dependency on tooling the user may not have.

Search for candidate repos on the Hub. The base URL is https://huggingface.co/models?apps=llama.cpp&sort=trending. Add search= for a model family, and add num_parameters=min:0,max:24B or similar when the user has size constraints.
Open the repo with the llama.cpp local-app view: https://huggingface.co/?local-app=llama.cpp.
Treat the local-app snippet as the source of truth when it is visible. Copy the exact llama-server or llama-cli command, and report the recommended quant exactly as HF shows it.
Read the same ?local-app=llama.cpp URL as page text or HTML and extract the section under Hardware compatibility. Prefer its exact quant labels and sizes over generic tables, and keep repo-specific labels such as UD-Q4_K_M or IQ4_NL_XL. If that section is not visible in the fetched page source, say so and fall back to the tree API plus generic quant guidance.
Query the tree API to confirm what actually exists: https://huggingface.co/api/models//tree/main?recursive=true. Keep entries where type is file and path ends with .gguf. Use path and size as the source of truth for filenames and byte sizes. Separate quantized checkpoints from mmproj-*.gguf projector files and BF16/ shard files. Use https://huggingface.co//tree/main only as a human fallback.
If the local-app snippet is not text-visible, reconstruct the command from the repo plus the chosen quant. Shorthand quant selection is llama-server -hf :; the exact-file fallback is llama-server --hf-repo --hf-file .
Only suggest conversion from Transformers weights if the repo does not already expose GGUF files.

The reason step 5 exists is that a quant chip on a page is a label, not a filename. The tree API is what turns one into the other.

Quick start

Run directly from the Hugging Face Hub

llama-cli -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0

llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0

The -hf form takes a repo and a quant suffix, so you never touch a local path. llama-cli is the interactive route; llama-server starts the HTTP server.

Run an exact GGUF file from the Hub

Use this when the tree API shows custom file naming or the exact HF snippet is missing.

llama-server \
    --hf-repo microsoft/Phi-3-mini-4k-instruct-gguf \
    --hf-file Phi-3-mini-4k-instruct-q4.gguf \
    -c 4096

Here the repo and the filename are given separately, which is what you need when the file does not follow the naming convention the shorthand assumes. The -c flag sets the context size.

OpenAI-compatible server check

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "user", "content": "Write a limerick about Python exceptions"}
    ]
  }'

This is the sanity check that the server is up and speaking the OpenAI chat completions shape. If you are wiring llama.cpp into an existing client that already targets that API, this is the endpoint it will hit.

Python bindings (llama-cpp-python)

Basic generation

from llama_cpp import Llama

llm = Llama(
    model_path="./model-q4_k_m.gguf",
    n_ctx=4096,
    n_gpu_layers=35,     # 0 for CPU, 99 to offload everything
    n_threads=8,
)

out = llm("What is machine learning?", max_tokens=256, temperature=0.7)
print(out["choices"][0]["text"])

The constructor takes the local GGUF path plus the knobs that matter most: context length, how many layers go to the GPU, and thread count. The comment on n_gpu_layers is the useful part: zero keeps everything on CPU, 99 pushes the whole model to the GPU.

Chat + streaming

llm = Llama(
    model_path="./model-q4_k_m.gguf",
    n_ctx=4096,
    n_gpu_layers=35,
    chat_format="llama-3",   # or "chatml", "mistral", etc.
)

resp = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is Python?"},
    ],
    max_tokens=256,
)
print(resp["choices"][0]["message"]["content"])

# Streaming
for chunk in llm("Explain quantum computing:", max_tokens=256, stream=True):
    print(chunk["choices"][0]["text"], end="", flush=True)

The chat_format argument is what tells the runtime how to wrap messages in the model's expected template. Getting it wrong is a common source of degraded output, so match it to the model family. Streaming works the same way as the non-chat call, with stream=True and a loop over chunks.

Embeddings

llm = Llama(model_path="./model-q4_k_m.gguf", embedding=True, n_gpu_layers=35)
vec = llm.embed("This is a test sentence.")
print(f"Embedding dimension: {len(vec)}")

Setting embedding=True switches the instance into embedding mode, and embed returns the vector for a string.

You can also load a GGUF straight from the Hub:

llm = Llama.from_pretrained(
    repo_id="bartowski/Llama-3.2-3B-Instruct-GGUF",
    filename="*Q4_K_M.gguf",
    n_gpu_layers=35,
)

from_pretrained accepts a repo id and a filename pattern, so the same wildcard matching you would use on the Hub works here.

Choosing a quant

The rule is Hub page first, generic heuristics second. Prefer the exact quant that HF marks as compatible for the user's hardware profile. For general chat, start with Q4_K_M. For code or technical work, prefer Q5_K_M or Q6_K if memory allows. For very tight RAM budgets, consider Q3_K_M, IQ variants, or Q2 variants only if the user explicitly prioritizes fit over quality. For multimodal repos, mention mmproj-*.gguf separately, because the projector is not the main model file. And do not normalize repo-native labels: if the page says UD-Q4_K_M, report UD-Q4_K_M.

That last point is easy to get wrong. Renaming a label to something tidier breaks the link between your answer and the file the user will actually download.

Extracting available GGUFs from a repo

When the user asks what GGUFs exist, return the filename, the file size, the quant label, and whether it is a main model or an auxiliary projector. Ignore unless requested: README, BF16 shard files, and imatrix blobs or calibration artifacts.

The tree API is the tool for this step: https://huggingface.co/api/models//tree/main?recursive=true.

A worked example from the source makes the distinction concrete. For a repo like unsloth/Qwen3.6-35B-A3B-GGUF, the local-app page can show quant chips such as UD-Q4_K_M, UD-Q5_K_M, UD-Q6_K, and Q8_0, while the tree API exposes exact file paths such as Qwen3.6-35B-A3B-UD-Q4_K_M.gguf and Qwen3.6-35B-A3B-Q8_0.gguf with byte sizes. Use the tree API to turn a quant label into an exact filename.

Search patterns

Use these URL shapes directly:

https://huggingface.co/models?apps=llama.cpp&sort=trending
https://huggingface.co/models?search=<term>&apps=llama.cpp&sort=trending
https://huggingface.co/models?search=<term>&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending
https://huggingface.co/<repo>?local-app=llama.cpp
https://huggingface.co/api/models/<repo>/tree/main?recursive=true
https://huggingface.co/<repo>/tree/main

Keeping these as literal URL shapes rather than prose descriptions means they can be pasted straight into a browser or a fetch call.

Output format

When answering discovery requests, prefer a compact structured result like:

Repo: <repo>
Recommended quant from HF: <label> (<size>)
llama-server: <command>
Other GGUFs:
- <filename> - <size>
- <filename> - <size>
Source URLs:
- <local-app URL>
- <tree API URL>

The shape is deliberate. It puts the recommendation and the runnable command at the top, lists the alternatives below, and cites the URLs the answer came from so the user can verify it.

When not to use it

The skill's own guidance is to prefer URL workflows before asking for hf, Python, or custom scripts. That is an ordering rule rather than an exclusion, but it implies the boundary: if the Hub pages and tree API already answer the question, do not reach for a script. Conversion from Transformers weights is the fallback of last resort, suggested only when the repo does not already expose GGUF files.

Limits and gotchas

Several constraints recur in the source and are worth stating plainly. The local-app snippet is authoritative only when it is visible; if the Hardware compatibility section is not present in the fetched page source, say so rather than inventing a recommendation, and fall back to the tree API plus generic quant guidance. The tree API is the source of truth for filenames and byte sizes, not the quant chips on the page. Projector files and BF16 shards are not main model files and should be separated out. Repo-native quant labels must be reported unchanged. And low-bit quants such as Q3_K_M, IQ variants, and Q2 variants are for cases where the user has explicitly chosen fit over quality, not defaults.

What pairs with this

The skill ships with reference documents that go deeper on specific areas. hub-discovery.md covers URL-only Hugging Face workflows, search patterns, GGUF extraction, and command reconstruction. advanced-usage.md covers speculative decoding, batched inference, grammar-constrained generation, LoRA, multi-GPU, custom builds, and benchmark scripts. quantization.md covers quant quality tradeoffs, when to use Q4/Q5/Q6/IQ, model size scaling, and imatrix. server.md covers direct-from-Hub server launch, OpenAI API endpoints, Docker deployment, NGINX load balancing, and monitoring. optimization.md covers CPU threading, BLAS, GPU offload heuristics, batch tuning, and benchmarks. troubleshooting.md covers install, convert, quantize, inference, and server issues, plus Apple Silicon and debugging.

Upstream, the llama.cpp project lives at https://github.com/ggml-org/llama.cpp. Hugging Face publishes GGUF and llama.cpp documentation at https://huggingface.co/docs/hub/gguf-llamacpp, Local Apps documentation at https://huggingface.co/docs/hub/main/local-apps, and Local Agents documentation at https://huggingface.co/docs/hub/agents-local.