How to put a base64 image in a ChatPromptTemplate in LangChain

To put a base64 image in a ChatPromptTemplate, you must use a HumanMessage with an image content block, not a string placeholder. The error you saw occurs because MessagesPlaceholder expects a string for variable_name, not a list, and because base64 image data cannot be passed as plain text. The correct approach is to define a MessagesPlaceholder with a single string variable name, then pass a list of HumanMessage objects containing the image data when invoking the prompt.

The Full Answer

The core issue in the original code is twofold. First, MessagesPlaceholder takes a single string for variable_name, not a list. The line MessagesPlaceholder(variable_name = ["im64",str(image_data)]) is invalid because variable_name expects a string like "image_messages". Second, even if you fix that, putting base64 image data directly into a system prompt string or a placeholder will not be interpreted as an image. LangChain treats the string as plain text, not as image content. To send an image, you must use a HumanMessage with a structured content list that includes an image block.

Step 1: Understand MessagesPlaceholder

MessagesPlaceholder is a placeholder for a list of message objects (like HumanMessage, AIMessage, etc.) that you supply later when invoking the prompt. It is not a way to inject raw text or base64 strings into a prompt. According to the LangChain documentation, the correct usage is:

MessagesPlaceholder(variable_name="history")

Then later:

prompt.format_messages(history=[...])

or

prompt.invoke({"history": [...]})

In your case, you need a placeholder for the image messages, so you would use:

MessagesPlaceholder(variable_name="image_messages")

Step 2: Create a HumanMessage with image content

LangChain supports sending images as part of a HumanMessage content list. The format depends on your LangChain version. For version 0.3.x (which you are using), the recommended format uses image_url with a data URI. For newer versions (1.x), you can use either image_url or a base64 key with mime_type. The accepted solution on Stack Overflow confirms both work, but image_url is more universally compatible.

For LangChain 0.3.x and 1.x:

from langchain.schema import HumanMessage

image_data = "..."  # your base64 string (without the data: prefix)

message = HumanMessage(
    content=[
        {
            "type": "image_url",
            "image_url": {"url": f"data:image/jpeg;base64,{image_data}"},
        },
    ]
)

Alternative for newer LangChain 1.x (also works):

message = HumanMessage(
    content=[
        {
            "type": "image",
            "base64": image_data,
            "mime_type": "image/jpeg",
        },
    ]
)

Note: The base64 key format is documented in LangChain's Messages documentation but may require a newer version. The image_url approach is more widely tested.

Step 3: Build the full prompt template

Here is a complete working example based on the accepted solution (tested with LangChain 1.2.13, but the pattern applies to 0.3.x with adjusted imports):

import base64
from langchain.prompts.chat import ChatPromptTemplate, MessagesPlaceholder
from langchain.schema import SystemMessage, HumanMessage

# Load and encode image
with open("food.jpg", "rb") as f:
    image_bytes = f.read()
image_data = base64.b64encode(image_bytes).decode("utf-8")

# Define the prompt template
prompt = ChatPromptTemplate.from_messages(
    [
        SystemMessage(
            "You are an expert assistant. Your task is to analyze the food items "
            "displayed in the image and provide detailed information about them."
        ),
        MessagesPlaceholder(variable_name="image_messages"),
    ]
)

# Create the image message
image_message = HumanMessage(
    content=[
        {
            "type": "image_url",
            "image_url": {"url": f"data:image/jpeg;base64,{image_data}"},
        },
    ]
)

# Format the prompt
formatted_prompt = prompt.format_prompt(
    image_messages=[image_message]
)

Step 4: Invoke with a chat model

Once you have the formatted prompt, you can pass it to a chat model. The accepted solution uses ChatOllama from langchain_ollama, but the same pattern works with any LangChain chat model (OpenAI, Anthropic, etc.):

from langchain_ollama import ChatOllama

llm = ChatOllama(model="llama3.2-vision")  # or any multimodal model
result = llm.invoke(formatted_prompt)
print(result.content)

If you are using OpenAI, import ChatOpenAI from langchain_openai and ensure the model supports vision (e.g., gpt-4o).

Why the original code failed

1. variable_name was a list: MessagesPlaceholder(variable_name=["im64", str(image_data)]) is invalid. variable_name must be a string.
2. Image data was treated as text: Putting {{im64}} in the system prompt and then passing image_data as a string does not create an image message. The model sees the base64 string as text, not as an image.
3. No HumanMessage with image content: The image must be wrapped in a HumanMessage with a proper content block, not injected into a system message or placeholder string.