Guidance

Free

A handy looking Python library from Microsoft that uses Handlebars templating to interleave generation, prompting, and logical control.

FreeFree tier

Inputs: textOutputs: text

Type

Open Source

Visit Website GitHub (38 stars)

About Guidance

Guidance is an efficient open-source Python library for steering large language models. It provides a Pythonic interface using common idioms like context managers (system, user, assistant) and functions (gen, select) to interleave control flow (conditionals, loops, tool use) with text generation. Guidance allows constrained generation with regular expressions and context-free grammars (CFGs), ensuring structured and high-quality output while reducing latency and cost compared to conventional prompting or fine-tuning. It supports multiple backends including Transformers, llama.cpp, and OpenAI, and is installable via pip. Originally developed by Microsoft, it is now maintained under the guidance-ai organization on GitHub.

Key Features

Pythonic interface using context managers (system, user, assistant) and functions (gen, select)

Constrained generation with regular expressions (regex) and context-free grammars (CFGs)

Seamless interleaving of control flow (conditionals, loops, tool use) with text generation

Captures generated text into named variables for easy access

Supports multiple backends: Transformers, llama.cpp, OpenAI, and more

Reduces latency and cost vs. conventional prompting or fine-tuning

Pros & Cons

Pros

Reduces latency and cost through constrained generation
Familiar Python syntax lowers learning curve for developers
Supports powerful constraints like regex and select for precise output control
Works with popular LLM backends (Transformers, llama.cpp, OpenAI)
Active open-source community with 21.7k stars and 1.2k forks on GitHub

Cons

Primarily designed for Python users, limiting cross-language adoption
May require understanding of regex and grammar concepts for advanced constraints
As an open-source library, lacks official enterprise support or SLAs

Best For

Structuring LLM output with specific formats (e.g., regex patterns, dropdown selections)Building interactive assistants with guided conversation flowsReducing token usage and cost by constraining generation to valid outputsPrototyping prompt chains with conditional logic and tool integration

FAQ

What is Guidance?

Guidance is an open-source Python library for steering large language models. It allows users to constrain generation with regex and CFGs, interleave control flow, and reduce latency and cost compared to conventional prompting.

What backends does Guidance support?

Guidance supports multiple backends including Transformers, llama.cpp, and OpenAI. You can specify the backend when initializing a model (e.g., Transformers or LlamaCpp).

How do I install Guidance?

Guidance is available on PyPI. You can install it with pip install guidance, provided you already have the backend required for your model.

Can Guidance be used in Jupyter notebooks?

Yes, Guidance provides a widget for richer user experience when running in a Jupyter notebook.