Visual-RFT: Visual Reinforcement Fine-Tuning

Free

Reinforcement fine-tuning for vision-language models using verifiable rewards

FreeFree tier

Type

Open Source

Visit Website GitHub (40 stars)

About Visual-RFT: Visual Reinforcement Fine-Tuning

Visual-RFT (Visual Reinforcement Fine-Tuning) extends reinforcement fine-tuning (RFT) to visual tasks, adapting the R1-style reinforcement learning approach from language models to multi-modal domains. It uses large vision-language models (LVLMs) to generate multiple responses containing reasoning tokens and final answers for each input, then applies verifiable reward functions (e.g., Intersection over Union for object detection) and the Group Relative Policy Optimization (GRPO) algorithm to update the model. On fine-grained image classification, few-shot object detection, reasoning grounding, and open-vocabulary object detection benchmarks, Visual-RFT shows competitive performance and improved generalization compared to supervised fine-tuning (SFT). For example, it improves accuracy by 24.3% over baseline in one-shot fine-grained image classification with around 100 samples, and exceeds baseline by 21.9 on COCO two-shot detection and 15.4 on LVIS. Visual-RFT represents a data-efficient, reward-driven paradigm for fine-tuning LVLMs that enhances reasoning and adaptability for domain-specific tasks.

Key Features

Extends Reinforcement Fine-Tuning (RFT) from language to multi-modal visual tasks

Uses Large Vision-Language Models (LVLMs) to generate multiple responses with reasoning tokens

Employs verifiable reward functions (e.g., Intersection over Union for object detection) tailored to visual perception tasks

Optimizes model parameters via Group Relative Policy Optimization (GRPO)

Achieves data-efficient fine-tuning with small sample sizes (e.g., ~100 samples for one-shot classification)

Shows significant accuracy improvements over supervised fine-tuning on fine-grained classification and few-shot detection

Pros & Cons

Pros

Data-efficient: achieves strong results with few training samples (e.g., 24.3% accuracy gain with ~100 samples)
Outperforms supervised fine-tuning across multiple visual benchmarks
Extends proven R1-style reinforcement learning to multimodal domains
Provides task-specific verifiable rewards (e.g., IoU) for precise optimization
Enhances reasoning and generalization for domain-specific visual tasks

Cons

Requires careful design of verifiable reward functions for each new visual task
Limited to tasks where a verifiable reward can be defined (not suitable for open-ended generation)
Potential high computational cost due to generating multiple responses and policy optimization
Not yet tested on broader vision-language tasks beyond perception benchmarks

Best For

Fine-grained image classification (especially one-shot and few-shot scenarios)Few-shot object detection (e.g., COCO two-shot, LVIS)Reasoning grounding tasksOpen-vocabulary object detection

FAQ

What is Visual-RFT?

Visual-RFT is a method that extends Reinforcement Fine-Tuning (RFT) from language models to visual tasks, using verifiable reward functions and policy optimization to fine-tune large vision-language models.

How does Visual-RFT work?

It first uses a vision-language model to generate multiple responses with reasoning tokens for each input. Then it applies task-specific verifiable reward functions (e.g., Intersection over Union for detection) and updates the model using Group Relative Policy Optimization (GRPO).

What tasks does Visual-RFT improve?

Visual-RFT shows improvements on fine-grained image classification (especially few-shot), few-shot object detection, reasoning grounding, and open-vocabulary object detection.

How much improvement does Visual-RFT offer over supervised fine-tuning?

In one-shot fine-grained image classification with about 100 samples, Visual-RFT improves accuracy by 24.3% over baseline. In few-shot object detection, it exceeds baseline by 21.9 on COCO's two-shot setting and 15.4 on LVIS.

Is Visual-RFT open-source?

The paper is available on arXiv and the method is described in detail; code availability is indicated via a project page link but not explicitly stated as open-source in the provided content.