Guidance: Constrained LLM Generation with Grammars in Hermes Agent

{ "TITLE": "Guidance Python Library Reference", "BODY": "# Guidance Python Library Reference\n\nGuidance is a Python library for constrained LLM generation that enforces output syntax via regex, grammars, and selection, reduces latency, and supports multi-step workflows with Pythonic control flow.\n\n## When to Use Guidance\n\nUse Guidance when you need to control LLM output syntax with regex or grammars, guarantee valid JSON, XML, or code generation, reduce latency compared to traditional prompting, enforce structured formats such as dates, emails, or IDs, build multi-step workflows with Pythonic control flow, or prevent invalid outputs through grammatical constraints.\n\n## Installation\n\nInstall the base package and optional backend dependencies.\n\n\n# Base installation\npip install guidance\n\n# With specific backends\npip install guidance[transformers] # Hugging Face models\npip install guidance[llama_cpp] # llama.cpp models\n\n\n## Prerequisites\n\nYou need a Python environment with pip. For local models, ensure sufficient GPU memory (or CPU) and model files. For OpenAI, set an API key as an environment variable or pass it directly. For Transformers, install the Hugging Face transformers library. For LlamaCpp, install llama-cpp-python and have a GGUF model file.\n\n## Capabilities\n\nGuidance provides regex constraints for token-level output control, grammar-based generation (CFG) via composable functions or guidance.json(), selection constraints with select() for fixed categories, token healing to fix tokenization boundary issues (enabled by default), Pydantic schema-based JSON generation via guidance.json(), reusable generation patterns with @guidance decorator (stateless or stateful), context managers for chat-style interactions (system, user, assistant), support for local backends (Transformers, LlamaCpp) with full constraint support, support for remote API backends (OpenAI, Azure) with unconstrained gen()/chat only, multi-step reasoning and ReAct agent patterns, and data extraction with structured fields.\n\n## Backend Support\n\n### Local Backends (Full Constraint Support)\n\nLocal backends support regex, select(), and grammar-based constrained generation. They require logit access for constraints.\n\nTransformers backend:\n\n\nfrom guidance.models import Transformers\n\nlm = Transformers(\n \"microsoft/Phi-4-mini-instruct\",\n device=\"cuda\" # Or \"cpu\"\n)\n\n\nLlamaCpp backend:\n\n\nfrom guidance.models import LlamaCpp\n\nlm = LlamaCpp(\n model_path=\"/path/to/model.gguf\",\n n_ctx=4096,\n n_gpu_layers=35\n)\n\n\n### Remote API Backends (Unconstrained Only)\n\nRemote API backends (OpenAI, Azure) support only unconstrained gen() and chat; no token-level constraints. regex, select(), and grammar-based constrained generation do not work with these backends.\n\n\nfrom guidance import models\n\nlm = models.OpenAI(\n model=\"gpt-4o-mini\",\n api_key=\"your-api-key\" # Or set OPENAI_API_KEY env var\n)\n\n\nNote: Guidance 0.3.x has no models.Anthropic class. The grammar= form is not part of current guidance; use composable functions or guidance.json().\n\n## Basic Structured Generation\n\nImport models and gen() from guidance, load a model, concatenate a prompt string with gen('name', max_tokens=N), and access the result via lm['name'].\n\n\nfrom guidance import models, gen\n\n# Load model (supports OpenAI, Transformers, llama.cpp)\nlm = models.OpenAI(\"gpt-4\")\n\n# Generate with constraints\nresult = lm + \"The capital of France is \" + gen(\"capital\", max_tokens=5)\n\nprint(result[\"capital\"]) # \"Paris\"\n\n\n## Chat Format with Local Model\n\nUse context managers for chat-style interactions with system, user, and assistant roles.\n\n\nfrom guidance import models, gen, system, user, assistant\n\n# Local model (supports constrained generation)\nlm = models.Transformers(\"microsoft/Phi-4-mini-instruct\")\n\n# Use context managers for chat format\nwith system():\n lm += \"You are a helpful assistant.\"\n\nwith user():\n lm += \"What is the capital of France?\"\n\nwith assistant():\n lm += gen(max_tokens=20)\n\n\n\nfrom guidance import system, user, assistant, gen\n\nlm = models.Transformers(\"microsoft/Phi-4-mini-instruct\")\n\n# System message\nwith system():\n lm += \"You are a JSON generation expert.\"\n\n# User message\nwith user():\n lm += \"Generate a person object with name and age.\"\n\n# Assistant response\nwith assistant():\n lm += gen(\"response\", max_tokens=100)\n\nprint(lm[\"response\"])\n\n\n## Regex Constraint Generation\n\nRegex constraints work only with local backends. Append a prompt with gen('name', regex=r'pattern'). The result is guaranteed to match the regex.\n\n\nfrom guidance import models, gen\n\nlm = models.Transformers(\"microsoft/Phi-4-mini-instruct\")\n\n# Constrain to valid email format\nlm += \"Email: \" + gen(\"email\", regex=r\"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}\")\n\n# Constrain to date format (YYYY-MM-DD)\nlm += \"Date: \" + gen(\"date\", regex=r\"\\d{4}-\\d{2}-\\d{2}\")\n\n# Constrain to phone number\nlm += \"Phone: \" + gen(\"phone\", regex=r\"\\d{3}-\\d{3}-\\d{4}\")\n\nprint(lm[\"email\"]) # Guaranteed valid email\nprint(lm[\"date\"]) # Guaranteed YYYY-MM-DD format\n\n\n## Selection Constraint Generation\n\nSelection constraints work only with local backends. Use select(['option1','option2',...], name='name') to constrain output to one of the listed options.\n\n\nfrom guidance import models, gen, select\n\nlm = models.Transformers(\"microsoft/Phi-4-mini-instruct\")\n\n# Constrain to specific choices\nlm += \"Sentiment: \" + select([\"positive\", \"negative\", \"neutral\"], name=\"sentiment\")\n\n# Multiple-choice selection\nlm += \"Best answer: \" + select(\n [\"A) Paris\", \"B) London\", \"C) Berlin\", \"D) Madrid\"],\n name=\"answer\"\n)\n\nprint(lm[\"sentiment\"]) # One of: positive, negative, neutral\nprint(lm[\"answer\"]) # One of: A, B, C, or D\n\n\n## Token Healing\n\nToken healing fixes tokenization boundary issues and is enabled by default. No special action is needed.\n\n\n# Without token healing\nprompt = \"The capital of France is \"\n# Last token: \" is \"\n# First generated token might be \" Par\" (with leading space)\n# Result: \"The capital of France is Paris\" (double space!)\n\n\n\nfrom guidance import models, gen\n\nlm = models.Transformers(\"microsoft/Phi-4-mini-instruct\")\n\n# Token healing enabled by default\nlm += \"The capital of France is \" + gen(\"capital\", max_tokens=5)\n# Result: \"The capital of France is Paris\" (correct spacing)\n\n\n\n# Token healing is enabled by default\n# No special action needed - just concatenate naturally\nlm += \"The capital is \" + gen(\"capital\") # Automatic healing\n\n\n## JSON Generation with Pydantic Schema\n\nUse guidance.json() to generate valid JSON matching a Pydantic schema. The schema compiles to a grammar.\n\n\nfrom guidance import models, gen\nfrom guidance import json as gen_json\nfrom pydantic import BaseModel, Field\n\nlm = models.Transformers(\"microsoft/Phi-4-mini-instruct\")\n\n# JSON via a Pydantic schema (guidance.json compiles the schema to a grammar)\nclass Person(BaseModel):\n name: str = Field(pattern=r\"[A-Za-z ]+\")\n age: int\n email: str = Field(pattern=r\"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}\")\n\nlm += gen_json(name=\"person\", schema=Person)\n\nprint(lm[\"person\"]) # Guaranteed valid JSON matching the schema\n\n# Or compose grammar functions directly:\ngrammar = \"name=\" + gen(\"name\", regex=r\"[A-Za-z ]+\") + \" age=\" + gen(\"age\", regex=r\"[0-9]+\")\nlm += grammar\n\n\n## Reusable Guidance Functions\n\nDecorate a function with @guidance (or @guidance(stateless=False) for stateful). The function takes lm as the first argument and returns lm. Inside, append generation calls with gen(). Call the function by passing lm.\n\n\nfrom guidance import guidance, gen, models\n\n@guidance\ndef generate_person(lm):\n \"\"\"Generate a person with name and age.\"\"\"\n lm += \"Name: \" + gen(\"name\", max_tokens=20, stop=\"\\n\")\n lm += \"\\nAge: \" + gen(\"age\", regex=r\"[0-9]+\", max_tokens=3)\n return lm\n\n# Use the function\nlm = models.Transformers(\"microsoft/Phi-4-mini-instruct\")\nlm = generate_person(lm)\n\nprint(lm[\"name\"])\nprint(lm[\"age\"])\n\n\n## Multi-Step Reasoning (Chain-of-Thought)\n\nDefine a @guidance function that loops over steps, generating reasoning steps and a final answer.\n\n\nfrom guidance import models, gen, guidance\n\n@guidance\ndef chain_of_thought(lm, question):\n \"\"\"Generate answer with step-by-step reasoning.\"\"\"\n lm += f\"Question: {question}\\n\\n\"\n\n # Generate multiple reasoning steps\n for i in range(3):\n lm += f\"Step {i+1}: \" + gen(f\"step_{i+1}\", stop=\"\\n\", max_tokens=100) + \"\\n\"\n\n # Final answer\n lm += \"\\nTherefore, the answer is: \" + gen(\"answer\", max_tokens=50)\n\n return lm\n\nlm = models.Transformers(\"microsoft/Phi-4-mini-instruct\")\nlm = chain_of_thought(lm, \"What is 15% of 200?\")\n\nprint(lm[\"answer\"])\n\n\n## ReAct Agent Pattern\n\nUse @guidance(stateless=False) for stateful agent functions. Initialize tools, loop over thought/action/action input/observation, check done condition, and generate a final answer.\n\n\n@guidance(stateless=False)\ndef react_agent(lm, question, tools, max_rounds=5):\n \"\"\"ReAct agent with tool use.\"\"\"\n lm += f\"Question: {question}\\n\\n\"\n\n for i in range(max_rounds):\n # Thought\n lm += f\"Thought {i+1}: \" + gen(\"thought\", stop=\"\\n\")\n\n # Action\n lm += \"\\nAction: \" + select(list(tools.keys()), name=\"action\")\n\n # Execute tool\n tool_result = tools[lm[\"action\"]]()\n lm += f\"\\nObservation: {tool_result}\\n\\n\"\n\n # Check if done\n lm += \"Done? \" + select([\"Yes\", \"No\"], name=\"done\")\n if lm[\"done\"] == \"Yes\":\n break\n\n # Final answer\n lm += \"\\nFinal Answer: \" + gen(\"answer\", max_tokens=100)\n return lm\n\n\n\nfrom guidance import models, gen, select, guidance\n\n@guidance(stateless=False)\ndef react_agent(lm, question):\n \"\"\"ReAct agent with tool use.\"\"\"\n tools = {\n \"calculator\": lambda expr: eval(expr),\n \"search\": lambda query: f\"Search results for: {query}\",\n }\n\n lm += f\"Question: {question}\\n\\n\"\n\n for round in range(5):\n # Thought\n lm += f\"Thought: \" + gen(\"thought\", stop=\"\\n\") + \"\\n\"\n\n # Action selection\n lm += \"Action: \" + select([\"calculator\", \"search\", \"answer\"], name=\"action\")\n\n if lm[\"action\"] == \"answer\":\n lm += \"\\nFinal Answer: \" + gen(\"answer\", max_tokens=100)\n break\n\n # Action input\n lm += \"\\nAction Input: \" + gen(\"action_input\", stop=\"\\n\") + \"\\n\"\n\n # Execute tool\n if lm[\"action\"] in tools:\n result = tools[lm[\"action\"]](lm[\"action_input\"])\n lm += f\"Observation: {result}\\n\\n\"\n\n return lm\n\nlm = models.Transformers(\"microsoft/Phi-4-mini-instruct\")\nlm = react_agent(lm, \"What is 25 * 4 + 10?\")\nprint(lm[\"answer\"])\n\n\n## Data Extraction\n\nDefine a @guidance function that takes a text parameter, appends the text, and generates structured fields with stop sequences or regex.\n\n\nfrom guidance import models, gen, guidance\n\n@guidance\ndef extract_entities(lm, text):\n \"\"\"Extract structured entities from text.\"\"\"\n lm += f\"Text: {text}\\n\\n\"\n\n # Extract person\n lm += \"Person: \" + gen(\"person\", stop=\"\\n\", max_tokens=30) + \"\\n\"\n\n # Extract organization\n lm += \"Organization: \" + gen(\"organization\", stop=\"\\n\", max_tokens=30) + \"\\n\"\n\n # Extract date\n lm += \"Date: \" + gen(\"date\", regex=r\"\\d{4}-\\d{2}-\\d{2}\", max_tokens=10) + \"\\n\"\n\n # Extract location\n lm += \"Location: \" + gen(\"location\", stop=\"\\n\", max_tokens=30) + \"\\n\"\n\n return lm\n\ntext = \"Tim Cook announced at Apple Park on 2024-09-15 in Cupertino.\"\n\nlm = models.Transformers(\"microsoft/Phi-4-mini-instruct\")\nlm = extract_entities(lm, text)\n\nprint(f\"Person: {lm['person']}\")\nprint(f\"Organization: {lm['organization']}\")\nprint(f\"Date: {lm['date']}\")\nprint(f\"Location: {lm['location']}\")\n\n\n## Classification with Confidence\n\nCombine select() with regex for numeric confidence.\n\n\nfrom guidance import models, gen, select\n\nlm = models.Transformers(\"microsoft/Phi-4-mini-instruct\")\n\ntext = \"This product is amazing! I love it.\"\n\nlm += f\"Text: {text}\\n\"\nlm += \"Sentiment: \" + select([\"positive\", \"negative\", \"neutral\"], name=\"sentiment\")\nlm += \"\\nConfidence: \" + gen(\"confidence\", regex=r\"[0-9]+\", max_tokens=3) + \"%\"\n\nprint(f\"Sentiment: {lm['sentiment']}\")\nprint(f\"Confidence: {lm['confidence']}%\")\n\n\n## Inline JSON Generation with Regex\n\nGenerate valid JSON directly in chat by combining regex constraints.\n\n\nfrom guidance import models, gen, system, user, assistant\n\nlm = models.Transformers(\"microsoft/Phi-4-mini-instruct\")\n\nwith system():\n lm += \"You generate valid JSON.\"\n\nwith user():\n lm += \"Generate a user profile with name, age, and email.\"\n\nwith assistant():\n lm += \"\"\"{\n \"name\": \"\"\" + gen(\"name\", regex=r'\"[A-Za-z ]+\"', max_tokens=30) + \"\"\",\n \"age\": \"\"\" + gen(\"age\", regex=r\"[0-9]+\", max_tokens=3) + \"\"\",\n \"email\": \"\"\" + gen(\"email\", regex=r'\"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}\"', max_tokens=50) + \"\"\"\n}\"\"\"\n\nprint(lm) # Valid JSON guaranteed\n\n\n## Parameters\n\n| Parameter | Meaning | Required |\n|-----------|---------|----------|\n| model | Model identifier string (e.g., 'gpt-4', 'microsoft/Phi-4-mini-instruct') or path to GGUF file | Yes |\n| gen(name, ...) | Generates a value for the named variable with optional constraints | Yes for generation |\n| max_tokens | Maximum number of tokens to generate | No |\n| regex | Regular expression pattern the output must match (local backends only) | No |\n| stop | Stop sequence(s) to end generation (e.g., '\n') | No |\n| select(options, name) | Constrains output to one of the given options (local backends only) | No |\n| schema | Pydantic model class for JSON generation via guidance.json() | Yes for gen_json |\n| device | Device for Transformers backend ('cuda' or 'cpu') | No |\n| n_ctx | Context size for LlamaCpp backend | No |\n| n_gpu_layers | Number of GPU layers for LlamaCpp backend | No |\n| api_key | API key for OpenAI backend (or set OPENAI_API_KEY env var) | No |\n| stateless | If False, function can maintain state across calls (for agents) | No (default True) |\n\n## Constraints and Caveats\n\nRegex, select(), and grammar-based constrained generation only work with local backends (Transformers, LlamaCpp). Remote API backends (OpenAI, Azure) support only unconstrained gen() and chat; no token-level constraints. Guidance 0.3.x has no models.Anthropic class. The grammar= form is not part of current guidance; use composable functions or guidance.json(). Token healing is enabled by default and requires no special action. Grammar compilation is cached after first use. Overly strict regex (e.g., exact match to few strings) may cause generation failure or slowness. Pydantic validation is not built-in; use Instructor for that. Local models require logit access for constraints.\n\n## Failure Modes\n\nUsing regex/select with remote API backends will not enforce constraints. Too strict regex may cause generation to fail or be very slow. Missing stop sequences may produce multi-line output when single-line expected. Free generation without constraints may produce invalid formats (e.g., malformed emails). Incorrect model loading (e.g., wrong path) will raise import or runtime errors.\n\n## Best Practices\n\nUse regex constraints to ensure valid formats rather than relying on free generation.\n\n\n# ✅ Good: Regex ensures valid format\nlm += \"Email: \" + gen(\"email\", regex=r\"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}\")\n\n# ❌ Bad: Free generation may produce invalid emails\nlm += \"Email: \" + gen(\"email\", max_tokens=50)\n\n\nUse select() to guarantee valid categories instead of free generation.\n\n\n# ✅ Good: Guaranteed valid category\nlm += \"Status: \" + select([\"pending\", \"approved\", \"rejected\"], name=\"status\")\n\n# ❌ Bad: May generate typos or invalid values\nlm += \"Status: \" + gen(\"status\", max_tokens=20)\n\n\nAlways include stop sequences for single-line outputs.\n\n\n# ✅ Good: Stop at newline for single-line outputs\nlm += \"Name: \" + gen(\"name\", stop=\"\\n\")\n\n# ❌ Bad: May generate multiple lines\nlm += \"Name: \" + gen(\"name\", max_tokens=50)\n\n\nReuse patterns with @guidance functions.\n\n\n# ✅ Good: Reusable pattern\n@guidance\ndef generate_person(lm):\n lm += \"Name: \" + gen(\"name\", stop=\"\\n\")\n lm += \"\\nAge: \" + gen(\"age\", regex=r\"[0-9]+\")\n return lm\n\n# Use multiple times\nlm = generate_person(lm)\nlm += \"\\n\\n\"\nlm = generate_person(lm)\n\n\nSet reasonable constraints to avoid generation failure.\n\n\n# ✅ Good: Reasonable constraints\nlm += gen(\"name\", regex=r\"[A-Za-z ]+\", max_tokens=30)\n\n# ❌ Too strict: May fail or be very slow\nlm += gen(\"name\", regex=r\"^(John|Jane)$\", max_tokens=10)\n\n\n## Performance Notes\n\nToken healing is enabled by default and requires no special action. Grammar compilation is cached after first use. Overly strict regex may cause generation failure or slowness. The library supports models with up to 18000 tokens of context and has been tested with models released in 2024. Constrained generation can reduce latency by up to 50% compared to traditional prompting with retries.\n\n## Reference Files\n\nFor additional reference material, consult the following files: references/constraints.md, references/backends.md, and references/examples.md." }