Visual Instruction Tuning

Free

Multimodal instruction tuning with GPT-4-generated data

FreeFree tier

Inputs: image, textOutputs: text

Type

Open Source

Visit Website GitHub (38 stars)

About Visual Instruction Tuning

Visual Instruction Tuning introduces LLaVA (Large Language and Vision Assistant), an end-to-end trained large multimodal model that connects a vision encoder with a large language model (LLM) for general-purpose visual and language understanding. It is the first attempt to leverage language-only GPT-4 to generate multimodal language-image instruction-following data, enabling instruction tuning for multimodal tasks. LLaVA demonstrates impressive multimodal chat abilities, sometimes exhibiting behaviors similar to multimodal GPT-4 on unseen images/instructions, achieving an 85.1% relative score compared to GPT-4 on a synthetic multimodal instruction-following dataset. When fine-tuned on Science QA, LLaVA combined with GPT-4 achieves a new state-of-the-art accuracy of 92.53%. The generated visual instruction tuning data, model, and code are publicly available. Presented as a NeurIPS 2023 Oral.

Key Features

First approach to use GPT-4 for generating multimodal language-image instruction-following data

End-to-end trained large multimodal model connecting a vision encoder and LLM

Demonstrates impressive multimodal chat abilities, often resembling multimodal GPT-4 behaviors

Achieves 85.1% relative score compared to GPT-4 on synthetic multimodal instruction-following dataset

New state-of-the-art accuracy of 92.53% on Science QA when fine-tuned with GPT-4

Publicly available visual instruction tuning data, model, and code

Pros & Cons

Pros

Open-source model and code publicly available
State-of-the-art performance on Science QA (92.53%)
Uses GPT-4 to generate high-quality training data efficiently
Demonstrates strong generalization to unseen images and instructions
NeurIPS 2023 Oral presentation

Cons

Relies on GPT-4 for data generation, which is a proprietary and costly API
Performance may be limited compared to GPT-4 itself on complex tasks
Requires substantial computational resources for training and inference
The synthetic data may not cover all real-world multimodal scenarios

Best For

Multimodal visual question answeringGeneral-purpose visual and language instruction followingScience question answering (Science QA)Multimodal chat and dialogue systems

FAQ

What is LLaVA?

LLaVA (Large Language and Vision Assistant) is an end-to-end trained large multimodal model that connects a vision encoder and a large language model for general-purpose visual and language understanding.

How is the training data generated?

The training data is generated by using language-only GPT-4 to produce multimodal language-image instruction-following data, a first-of-its-kind approach.

Is LLaVA publicly available?

Yes, the GPT-4 generated visual instruction tuning data, the LLaVA model, and the code base are all publicly available.