LLaVA Reference: Vision-Language Model for Conversional Image Understanding

LLaVA is an open-source vision-language model designed for conversational image understanding. It accepts an image and a text prompt, then generates a text response that describes, answers questions about, or discusses the image content. The model supports multi-turn conversations, maintaining context across several exchanges about the same image.

When to Use LLaVA

Use LLaVA for building vision-language chatbots, visual question answering (VQA), image description and captioning, multi-turn image conversations, visual instruction following, and document understanding with images.

Capabilities

Image captioning: describe an image in detail.
Visual question answering: answer questions about image content.
Textual object detection: list objects visible in an image.
Scene understanding: describe what is happening in a scene.
Document understanding: extract the main topic from document images.
Multi-turn conversations: maintain context across multiple exchanges about the same image.
Integration with LangChain and Gradio.
Training custom models via a two-stage process (feature alignment then instruction tuning).
Quantization support (4-bit and 8-bit) to reduce VRAM.

Prerequisites

Python environment
Git
GPU with sufficient VRAM (see constraints)
PyTorch and other dependencies installed via pip install -e .
Internet access to download models from Hugging Face

Installation

Clone the repository and install the package.

# Clone repository
git clone https://github.com/haotian-liu/LLaVA
cd LLaVA

# Install
pip install -e .

Basic Usage (Python)

The following example loads a model, processes an image, creates a conversation, and generates a response.

from llava.model.builder import load_pretrained_model
from llava.mm_utils import get_model_name_from_path, process_images, tokenizer_image_token
from llava.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN
from llava.conversation import conv_templates
from PIL import Image
import torch

# Load model
model_path = "liuhaotian/llava-v1.5-7b"
tokenizer, model, image_processor, context_len = load_pretrained_model(
    model_path=model_path,
    model_base=None,
    model_name=get_model_name_from_path(model_path)
)

# Load image
image = Image.open("image.jpg")
image_tensor = process_images([image], image_processor, model.config)
image_tensor = image_tensor.to(model.device, dtype=torch.float16)

# Create conversation
conv = conv_templates["llava_v1"].copy()
conv.append_message(conv.roles[0], DEFAULT_IMAGE_TOKEN + "\nWhat is in this image?")
conv.append_message(conv.roles[1], None)
prompt = conv.get_prompt()

# Generate response
input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors='pt').unsqueeze(0).to(model.device)

with torch.inference_mode():
    output_ids = model.generate(
        input_ids,
        images=image_tensor,
        do_sample=True,
        temperature=0.2,
        max_new_tokens=512
    )

response = tokenizer.decode(output_ids[0], skip_special_tokens=True).strip()
print(response)

Loading Different Model Sizes

You can load different model sizes. The 7B model uses approximately 14 GB VRAM in FP16, the 13B model uses about 28 GB, and the 34B model uses about 70 GB. Quantization reduces these requirements.

# Load different models
model_7b = "liuhaotian/llava-v1.5-7b"
model_13b = "liuhaotian/llava-v1.5-13b"
model_34b = "liuhaotian/llava-v1.6-34b"

# 4-bit quantization for lower VRAM
load_4bit = True  # Reduces VRAM by ~4×

Multi-Turn Conversation (Python)

To maintain context across multiple turns, update the conversation object with each response before appending the next user message.

# Initialize conversation
conv = conv_templates["llava_v1"].copy()

# Turn 1
conv.append_message(conv.roles[0], DEFAULT_IMAGE_TOKEN + "\nWhat is in this image?")
conv.append_message(conv.roles[1], None)
response1 = generate(conv, model, image)  # "A dog playing in a park"

# Turn 2
conv.messages[-1][1] = response1  # Add previous response
conv.append_message(conv.roles[0], "What breed is the dog?")
conv.append_message(conv.roles[1], None)
response2 = generate(conv, model, image)  # "Golden Retriever"

# Turn 3
conv.messages[-1][1] = response2
conv.append_message(conv.roles[0], "What time of day is it?")
conv.append_message(conv.roles[1], None)
response3 = generate(conv, model, image)

CLI Usage

Single Image Query

Run the CLI with a query.

# Single image query
python -m llava.serve.cli \
    --model-path liuhaotian/llava-v1.5-7b \
    --image-file image.jpg \
    --query "What is in this image?"

# Multi-turn conversation
python -m llava.serve.cli \
    --model-path liuhaotian/llava-v1.5-7b \
    --image-file image.jpg
# Then type questions interactively

Web UI (Gradio)

Launch a Gradio web interface.

# Launch Gradio interface
python -m llava.serve.gradio_web_server \
    --model-path liuhaotian/llava-v1.5-7b \
    --load-4bit  # Optional: reduce VRAM

# Access at http://localhost:7860

Example Queries

The following examples show typical queries for different tasks. The ask function is a placeholder for your inference code.

Image captioning: question = "Describe this image in detail." then response = ask(model, image, question)
Visual question answering: question = "How many people are in the image?" then response = ask(model, image, question)
Textual object detection: question = "List all the objects you can see in this image." then response = ask(model, image, question)
Scene understanding: question = "What is happening in this scene?" then response = ask(model, image, question)
Document understanding: question = "What is the main topic of this document?" then response = ask(model, document_image, question)

Training Custom Models

Training uses a two-stage process. Stage 1 performs feature alignment using 558K image-caption pairs. Stage 2 performs visual instruction tuning using 150K instruction data.

# Stage 1: Feature alignment (558K image-caption pairs)
bash scripts/v1_5/pretrain.sh

# Stage 2: Visual instruction tuning (150K instruction data)
bash scripts/v1_5/finetune.sh

Quantization

Quantization reduces VRAM usage. 4-bit quantization reduces VRAM by about 4x, and 8-bit quantization reduces it by about 2x.

# 4-bit quantization
tokenizer, model, image_processor, context_len = load_pretrained_model(
    model_path="liuhaotian/llava-v1.5-13b",
    model_base=None,
    model_name=get_model_name_from_path("liuhaotian/llava-v1.5-13b"),
    load_4bit=True  # Reduces VRAM ~4×
)

# 8-bit quantization
load_8bit=True  # Reduces VRAM ~2×

LangChain Integration

Subclass the LangChain LLM class and implement the _call method.

from langchain.llms.base import LLM

class LLaVALLM(LLM):
    def _call(self, prompt, stop=None):
        # Custom LLaVA inference
        return response

llm = LLaVALLM()

Gradio Integration

Create a ChatInterface with an additional image input.

import gradio as gr

def chat(image, text, history):
    response = ask_llava(model, image, text)
    return response

demo = gr.ChatInterface(
    chat,
    additional_inputs=[gr.Image(type="pil")],
    title="LLaVA Chat"
)
demo.launch()

Parameters

model_path (required): Hugging Face model identifier or local path to the LLaVA model.
model_base (optional): Base model path (usually None for LLaVA models).
model_name (required): Name derived from model_path via get_model_name_from_path.
load_4bit (optional): Boolean flag to enable 4-bit quantization (reduces VRAM about 4x).
load_8bit (optional): Boolean flag to enable 8-bit quantization (reduces VRAM about 2x).
temperature (optional): Controls randomness in generation (0.2-0.7 recommended).
max_new_tokens (optional): Maximum number of tokens to generate (512-1024 recommended for detailed responses).
do_sample (optional): Boolean flag to enable sampling (set to True for varied responses).
conv_template (required): Conversation template name (e.g., 'llava_v1').
image_file (required for CLI image query): Path to the input image file.
query (required for CLI single query): Text query.

Constraints and Caveats

GPU required; CPU inference is extremely slow.
VRAM requirements: 7B model about 14 GB (FP16), 13B about 28 GB (FP16), 34B about 70 GB (FP16). 4-bit quantization reduces by about 4x, 8-bit by about 2x.
Inference speed on A100: 7B model about 20 tokens/s, 13B about 12 tokens/s, 34B about 5 tokens/s.
Start with the 7B model for manageable VRAM.
Use 4-bit quantization to significantly reduce VRAM.
Clear, specific prompts yield better answers.
Multi-turn conversations maintain context.
Temperature 0.2-0.7 balances creativity and consistency.
max_new_tokens 512-1024 for detailed responses.
Batch processing: process multiple images sequentially.
Licensed under Apache 2.0.
Alternatives: GPT-4V (highest quality, API-based), CLIP (simple zero-shot classification), BLIP-2 (better for captioning only), Flamingo (research, not open-source).

Failure Modes

Hallucinations: may describe objects not present in the image.
Spatial reasoning: struggles with precise locations.
Small text: difficulty reading fine print.
Object counting: imprecise for many objects.
VRAM requirements: need powerful GPU.
Inference speed: slower than CLIP.

Performance Benchmarks

On standard benchmarks, LLaVA achieves the following scores: 78.5% on VQAv2, 62.0% on GQA, 35.4% on VizWiz, 64.3% on SQA, and 23000 on MMBench.

LLaVA Vision-Language Assistant: Setup, Usage, and Best Practices