Neura Market Logo
BundledMLOpsVersion 2.1.2

Local GGUF Inference with llama.cpp and Hugging Face Hub Discovery

llama.cpp local GGUF inference + HF Hub model discovery.

Written by Neura Market from the official Hermes Agent documentation for Llama Cpp. Commands, paths, and version numbers are reproduced from the source unchanged.

Read the official documentation

This skill lets you run GGUF models locally using llama.cpp and discover the right quantized model on the Hugging Face Hub without leaving your terminal. You would reach for it when you need to serve a model on CPU, Apple Silicon, or a GPU, find the correct GGUF file for a repo, or build a llama-server command from a Hugging Face URL.

What it does

The skill covers two workflows that often go together. First, it walks you through discovering GGUF models on the Hugging Face Hub using URL-based queries, the local-app view, and the tree API. Second, it shows you how to run those models with llama.cpp, either from the Hub directly or from a local file, and how to use the Python bindings for generation, chat, streaming, and embeddings.

Before you start

You need llama.cpp installed. The skill supports Linux, macOS, and Windows. For GPU acceleration, you will need the appropriate backend: CUDA for NVIDIA, ROCm for AMD, Metal for Apple Silicon, or the Vulkan-based backend for Intel GPUs. The Python bindings require llama-cpp-python and may need backend-specific build flags.

Install llama.cpp

Three install paths are available depending on your platform and preference.

# macOS / Linux (simplest)
brew install llama.cpp

winget install llama.cpp

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release

Run directly from the Hugging Face Hub

Once llama.cpp is installed, you can run a model straight from the Hub without downloading anything manually. The shorthand syntax is -hf <repo>:<quant>.

llama-cli -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0

llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0

llama-cli runs a single prompt and exits. llama-server starts an HTTP server that exposes an OpenAI-compatible API.

Run an exact GGUF file from the Hub

When the tree API shows custom file naming or the exact HF snippet is missing, use the long form with --hf-repo and --hf-file.

llama-server \
    --hf-repo microsoft/Phi-3-mini-4k-instruct-gguf \
    --hf-file Phi-3-mini-4k-instruct-q4.gguf \
    -c 4096

OpenAI-compatible server check

After starting llama-server, test it with curl.

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "user", "content": "Write a limerick about Python exceptions"}
    ]
  }'

Python bindings (llama-cpp-python)

Install the Python package. For CUDA, pass the build flag; for Metal, use the Metal flag.

pip install llama-cpp-python (CUDA: CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python --force-reinstall --no-cache-dir; Metal: CMAKE_ARGS="-DGGML_METAL=on" ...).

Basic generation

from llama_cpp import Llama

llm = Llama(
    model_path="./model-q4_k_m.gguf",
    n_ctx=4096,
    n_gpu_layers=35,     # 0 for CPU, 99 to offload everything
    n_threads=8,
)

out = llm("What is machine learning?", max_tokens=256, temperature=0.7)
print(out["choices"][0]["text"])

Chat + streaming

llm = Llama(
    model_path="./model-q4_k_m.gguf",
    n_ctx=4096,
    n_gpu_layers=35,
    chat_format="llama-3",   # or "chatml", "mistral", etc.
)

resp = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is Python?"},
    ],
    max_tokens=256,
)
print(resp["choices"][0]["message"]["content"])

# Streaming
for chunk in llm("Explain quantum computing:", max_tokens=256, stream=True):
    print(chunk["choices"][0]["text"], end="", flush=True)

Embeddings

llm = Llama(model_path="./model-q4_k_m.gguf", embedding=True, n_gpu_layers=35)
vec = llm.embed("This is a test sentence.")
print(f"Embedding dimension: {len(vec)}")

You can also load a GGUF straight from the Hub:

llm = Llama.from_pretrained(
    repo_id="bartowski/Llama-3.2-3B-Instruct-GGUF",
    filename="*Q4_K_M.gguf",
    n_gpu_layers=35,
)

Model Discovery workflow

When you need to find the right model, prefer URL workflows before asking for hf, Python, or custom scripts.

  1. Search for candidate repos on the Hub:

    • Base: https://huggingface.co/models?apps=llama.cpp&sort=trending
    • Add search= for a model family
    • Add num_parameters=min:0,max:24B or similar when the user has size constraints

  2. Open the repo with the llama.cpp local-app view:

    • https://huggingface.co/?local-app=llama.cpp

  3. Treat the local-app snippet as the source of truth when it is visible:

    • copy the exact llama-server or llama-cli command
    • report the recommended quant exactly as HF shows it

  4. Read the same ?local-app=llama.cpp URL as page text or HTML and extract the section under Hardware compatibility:

    • prefer its exact quant labels and sizes over generic tables
    • keep repo-specific labels such as UD-Q4_K_M or IQ4_NL_XL
    • if that section is not visible in the fetched page source, say so and fall back to the tree API plus generic quant guidance

  5. Query the tree API to confirm what actually exists:

    • https://huggingface.co/api/models//tree/main?recursive=true
    • keep entries where type is file and path ends with .gguf
    • use path and size as the source of truth for filenames and byte sizes
    • separate quantized checkpoints from mmproj-*.gguf projector files and BF16/ shard files
    • use https://huggingface.co//tree/main only as a human fallback

  6. If the local-app snippet is not text-visible, reconstruct the command from the repo plus the chosen quant:

    • shorthand quant selection: llama-server -hf :
    • exact-file fallback: llama-server --hf-repo --hf-file

  7. Only suggest conversion from Transformers weights if the repo does not already expose GGUF files.

Choosing a quant

Use the Hub page first, generic heuristics second.

  • Prefer the exact quant that HF marks as compatible for the user's hardware profile.
  • For general chat, start with Q4_K_M.
  • For code or technical work, prefer Q5_K_M or Q6_K if memory allows.
  • For very tight RAM budgets, consider Q3_K_M, IQ variants, or Q2 variants only if the user explicitly prioritizes fit over quality.
  • For multimodal repos, mention mmproj-*.gguf separately. The projector is not the main model file.
  • Do not normalize repo-native labels. If the page says UD-Q4_K_M, report UD-Q4_K_M.

Extracting available GGUFs from a repo

When the user asks what GGUFs exist, return:

  • filename
  • file size
  • quant label
  • whether it is a main model or an auxiliary projector

Ignore unless requested:

  • README
  • BF16 shard files
  • imatrix blobs or calibration artifacts

Use the tree API for this step:

  • https://huggingface.co/api/models//tree/main?recursive=true

For a repo like unsloth/Qwen3.6-35B-A3B-GGUF, the local-app page can show quant chips such as UD-Q4_K_M, UD-Q5_K_M, UD-Q6_K, and Q8_0, while the tree API exposes exact file paths such as Qwen3.6-35B-A3B-UD-Q4_K_M.gguf and Qwen3.6-35B-A3B-Q8_0.gguf with byte sizes. Use the tree API to turn a quant label into an exact filename.

Search patterns

Use these URL shapes directly:

https://huggingface.co/models?apps=llama.cpp&sort=trending
https://huggingface.co/models?search=<term>&apps=llama.cpp&sort=trending
https://huggingface.co/models?search=<term>&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending
https://huggingface.co/<repo>?local-app=llama.cpp
https://huggingface.co/api/models/<repo>/tree/main?recursive=true
https://huggingface.co/<repo>/tree/main

Output format

When answering discovery requests, prefer a compact structured result like:

Repo: <repo>
Recommended quant from HF: <label> (<size>)
llama-server: <command>
Other GGUFs:
- <filename> - <size>
- <filename> - <size>
Source URLs:
- <local-app URL>
- <tree API URL>

When not to use it

If the repo already exposes GGUF files, do not suggest converting from Transformers weights. The skill is designed for discovery and inference, not for training or fine-tuning. For speculative decoding, batched inference, grammar-constrained generation, LoRA, multi-GPU setups, or custom builds, see the advanced-usage reference.

Limits and gotchas

  • The local-app snippet is the source of truth when visible, but it may not be text-accessible in all page fetches. In that case, fall back to the tree API and generic quant guidance.
  • Do not normalize repo-native quant labels. If the page says UD-Q4_K_M, report it exactly as UD-Q4_K_M.
  • The tree API returns all files, including BF16 shards and calibration artifacts. Filter to .gguf files only, and separate projector files from main model files.
  • For multimodal repos, the mmproj-*.gguf file is a projector, not the main model. Do not confuse them.

What pairs with this

  • hub-discovery.md - URL-only Hugging Face workflows, search patterns, GGUF extraction, and command reconstruction
  • advanced-usage.md, speculative decoding, batched inference, grammar-constrained generation, LoRA, multi-GPU, custom builds, benchmark scripts
  • quantization.md, quant quality tradeoffs, when to use Q4/Q5/Q6/IQ, model size scaling, imatrix
  • server.md, direct-from-Hub server launch, OpenAI API endpoints, Docker deployment, NGINX load balancing, monitoring
  • optimization.md, CPU threading, BLAS, GPU offload heuristics, batch tuning, benchmarks
  • troubleshooting.md, install/convert/quantize/inference/server issues, Apple Silicon, debugging

More MLOps skills

Huggingface AccelerateRun PyTorch training across GPUs with minimal changes.ChromaEmbedding database for RAG and semantic search.ClipZero-shot image classification and image-text search.Evaluating Llms Harnesslm-eval-harness: benchmark LLMs (MMLU, GSM8K, etc.).Weights And BiasesW&B: log ML experiments, sweeps, model registry, dashboards.FaissFast vector similarity search at billion scale.