Code Guided Synthetic data generation system (CoSyn)

Analysis

Why This Paper Matters

Vision-language models (VLMs) have made remarkable strides, but their ability to understand text-rich images—like charts, documents, diagrams, and tables—remains a critical bottleneck. The challenge is twofold: collecting diverse, high-quality multimodal data is expensive and time-consuming, and existing datasets often lack the variety needed for robust generalization. CoSyn tackles this head-on by turning the problem on its head: instead of collecting images and then annotating them, it uses the coding capabilities of text-only LLMs to generate synthetic images from code, and then uses that same code as a rich context to produce instruction-tuning data. This approach is elegant because it leverages the strengths of LLMs in code generation and language understanding, while bypassing the need for actual image processing during data creation. The result is a scalable, cost-effective pipeline that can produce hundreds of thousands of diverse, high-quality examples, enabling VLMs to achieve state-of-the-art performance on text-rich benchmarks and even generalize to novel tasks.

How It Works

At its core, CoSyn uses code as an intermediate representation to bridge the gap between images and text. Given a text query like "generate a dataset of book covers," the system first prompts an LLM (Claude-3.5-Sonnet) to generate executable code in languages such as Python, HTML, or LaTeX. This code is then executed to render a synthetic image. Crucially, the same code—without the image—is fed back into an LLM (GPT-4o-mini) to generate textual instruction-tuning data, including questions, answers, and chain-of-thought explanations. The mathematical formulation is: P(I, T|q) = P_LM(C|q) · P(I|C) · P_LM(T|C), where C is the code.

The system architecture is modular and pipeline-based. CoSyn selects from 20 generation pipelines built on 11 rendering tools, each tailored to a specific image category. For example, Matplotlib, Plotly, and Vega-Lite generate charts; LaTeX and HTML produce documents and tables; Mermaid and Graphviz create diagrams; SVG and Asymptote handle vector graphics and math; Lilypond generates music sheets; and RDKit produces chemical structures. Each pipeline follows a four-stage procedure: topic generation (conditioned on a sampled persona from Persona Hub to enhance diversity), data generation (populating detailed content), code generation (creating executable code), and instruction generation (producing QA pairs and explanations). The use of 200K personas ensures that the synthetic data covers a wide range of styles and contexts.

Results

Using CoSyn, the authors generated CoSyn-400K, a dataset of 400K images across nine categories (charts, documents, math problems, tables, diagrams, vector graphics, music sheets, electrical circuits, and chemical structures) with 2.7M instruction-tuning pairs. They trained a 7B VLM (CLIP ViT-L/14 336px vision encoder + Mistral-7B language model) using a two-stage strategy: pre-training on dense captions from PixMo-Cap, followed by supervised fine-tuning on CoSyn-400K combined with 12 auxiliary datasets (about 1M images).

The results are striking. The 7B model achieved the highest average performance across seven text-rich benchmarks, surpassing the second-best model (Llama 3.2 11B) by 3.9%. It ranked first on four of the seven datasets and second on the remaining three. Even the zero-shot version outperformed most open and closed models, despite not being trained on any evaluation dataset data—a significant achievement given that competing models often used benchmark training data.

Ablation studies confirmed that synthetic data is a key driver of performance. The model trained with synthetic data alone surpassed GPT-4V in some zero-shot settings. On a novel task (NutritionQA), the synthetic-data-trained model matched GPT-4V's performance, demonstrating strong zero-shot generalization. Chain-of-thought reasoning from synthetic explanations improved performance on multi-hop reasoning tasks (ChartQA, TableVQA, NutritionQA) but slightly hurt performance on benchmarks biased toward short answers (DocVQA, InfoVQA).

An additional experiment on synthetic pointing data showed that CoSyn can also generate click-prediction training data. By providing the LLM with the source code of generated images and asking it to create a pointing question and modify the code to draw points, the authors extracted coordinates for training. A model trained on 65K synthetic samples achieved comparable performance to one trained on 155K human-annotated samples (PixMo-point). Combining synthetic and human data led to state-of-the-art performance on the ScreenSpot benchmark, outperforming UGround (trained on 1.3M screenshots).

Significance

CoSyn represents a paradigm shift in how we think about multimodal data generation. By using code as a bridge, it decouples image generation from text annotation, allowing each step to be optimized independently. This not only reduces the cost and effort of data collection but also enables the creation of diverse, high-quality datasets that can be tailored to specific tasks or domains. The ability to generate synthetic pointing data further extends the framework's utility to interactive and GUI-based applications. The paper's findings—that synthetic data can match or exceed human-annotated data in performance, and that it enables zero-shot generalization to novel tasks—have profound implications for the future of VLM training. As LLMs continue to improve in code generation and language understanding, frameworks like CoSyn will become even more powerful, potentially democratizing access to high-quality multimodal training data and accelerating progress in AI.