LLMs as Planners, Not Reasoners

Free

FreeFree tier

Type

Open Source

About LLMs as Planners, Not Reasoners

Gemini-Prolog is a hybrid question-answering system that combines Large Language Models (LLMs) with structured symbolic reasoning using Wikidata and Prolog. Instead of relying solely on retrieval-augmented generation (RAG), the pipeline dynamically constructs a temporary knowledge base from Wikidata, converts it into Prolog facts, generates a logical query via an LLM, executes it with SWI-Prolog, and formats the final answer with the LLM. The system emphasizes using the LLM as a planner for query generation and final output, while Prolog handles deterministic reasoning. It includes a question classifier to decide when to use symbolic reasoning and a fallback mechanism that reverts to a direct LLM answer if the logic pipeline fails. Built as an open-source project on GitHub, it requires Python 3.10+, SWI-Prolog, and Google GenAI API access.

Key Features

Hybrid LLM + symbolic reasoning pipeline

Dynamic knowledge base construction from Wikidata

Prolog fact generation and query execution

LLM-based query generation with strict logical rules

Question classification to determine suitability for symbolic reasoning

Fallback mechanism to direct LLM answer on failure

Entity search and graph expansion via forward and inverse relations

Timeout protection (30s) for Prolog execution

Custom exception handling (WikidataError, PrologError, LLMError)

Pros & Cons

Pros

Combines LLM flexibility with deterministic symbolic reasoning via Prolog
Builds a structured knowledge base on the fly, not relying on pre-indexed documents
Fallback mechanism ensures robust answers even when logic pipeline fails
Open-source and extensible, allowing customization and experimentation
Clear separation of roles: LLM for planning, Prolog for execution

Cons

Requires SWI-Prolog installation and Google GenAI API access
No temporal reasoning support
Knowledge base size limited by expansion depth (may miss relevant data)
Relies on LLM to generate correct Prolog queries, which can be error-prone
Wikidata noise and incomplete or ambiguous data can affect results
Not a fully general QA system; limited to factual queries compatible with symbolic reasoning

Best For

Answering factual questions requiring multi-step reasoning (e.g., 'List dinosaurs with names related to king')Structured reasoning over real-world knowledge from WikidataEducational demonstration of combining neural and symbolic AIResearch in hybrid AI systems and planning-based LLM usageExploring alternatives to RAG for knowledge-intensive QA

FAQ

How does Gemini-Prolog differ from traditional RAG systems?

Instead of retrieving document chunks, Gemini-Prolog dynamically builds a structured knowledge base from Wikidata, converts it into Prolog facts, and executes logical queries. The LLM is used as a planner to generate the query and format the answer, while Prolog handles the deterministic reasoning.

What are the system requirements to run Gemini-Prolog?

The system requires Python 3.10+, SWI-Prolog installed on the machine, and access to Google GenAI API. Dependencies can be installed via pip with the provided requirements.txt file.

What happens if the symbolic reasoning pipeline fails?

If any step fails (e.g., no entities found, query unanswerable, no Prolog results), the system falls back to a direct LLM answer without symbolic reasoning.

What is the role of the LLM in this system?

The LLM is used for question classification, generating search queries, producing Prolog queries, and formatting the final answer. It acts as a planner rather than the source of truth, with Prolog executing the actual logical reasoning.