deploy-llms-with-ansible

Free

Easily deploy any LLM on a VM with minimal configuration, using Ansible.

FreeFree tier

Inputs: textOutputs: text

Type

Open Source

Visit Website GitHub (38 stars)

About deploy-llms-with-ansible

deploy-llms-with-ansible is an open-source project that automates the deployment and serving of large language models (LLMs) on Debian-based virtual machines using Ansible. It supports two popular backends: Ollama and llama.cpp, each running inside Docker containers for isolation and ease of management. The deployment is secured via IP whitelisting, and the served model provides an OpenAI-compatible API endpoint on port 8080. Configuration is minimal—users define variables such as the model URL (for llama.cpp) or model name (for Ollama), SSH credentials, and allowed IPs in an inventory file. With a single Ansible playbook command, the tool sets up the chosen backend, pulls the model, and exposes it for chat completions.

Key Features

Deploy LLMs with Ansible playbooks

Supports both Ollama and llama.cpp backends

Uses Docker containers for isolation

IP whitelisting for server security

OpenAI-compatible API endpoint on port 8080

Minimal configuration via inventory files (model URL/name, SSH, allowed IPs)

Works on Debian-based VMs with Docker installed

Pros & Cons

Pros

Straightforward setup with Ansible automation
Supports two of the most popular open-source LLM backends
Built-in security via IP whitelisting
OpenAI-compatible API simplifies client integration
Fully open source and customizable
Uses Docker for consistent runtime environment

Cons

Limited to Debian-based VMs with Docker pre-installed
Requires manual configuration of inventory files
Only supports GGUF models (llama.cpp) or models from Ollama library
No built-in support for model versioning, scaling, or load balancing
Lacks a graphical interface or dashboard for management

Best For

Hosting a private LLM API on a personal or cloud VMRapid prototyping of LLM-powered applications with minimal infrastructure effortSecure internal deployment of open-source language models within an organizationEducational or research environments requiring repeatable model serving setups

FAQ

What are the prerequisites for using this tool?

You need Ansible installed on your local machine, a Debian-based VM with Docker installed, and SSH access to that VM.

How do I choose between Ollama and llama.cpp?

Copy the appropriate example inventory file from the 'ollama' or 'llamacpp' folders, then configure the variables. For llama.cpp you need a URL to a GGUF model, for Ollama you specify a model name from the Ollama library.

Is the API compatible with the OpenAI SDK?

Yes, the served endpoint follows the OpenAI chat completions API specification, so you can use standard tools like curl or the OpenAI Python client.

How do I restrict access to the deployed LLM?

You set the 'allowed_ip_for_8080' variable in your inventory file to whitelist specific IP addresses. Only those IPs will be able to send requests to the API.