MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Knowledge Poisoning Attacks

Analysis

Why This Paper Matters

Multimodal retrieval-augmented generation (RAG) has become a cornerstone for building AI systems that can answer questions using both text and images. However, the security of these systems is largely unexplored. This paper addresses a critical gap by introducing MM-PoisonRAG, the first comprehensive framework for knowledge poisoning attacks specifically designed for multimodal RAG. As organizations increasingly rely on RAG to ground LLMs with external knowledge, understanding how an attacker can manipulate the knowledge base is essential for building trustworthy systems.

The paper's significance lies in its systematic approach to attacking both local and global knowledge. Local attacks target specific queries, potentially causing misinformation on sensitive topics, while global attacks degrade the entire system's reliability. This dual perspective is crucial because it shows that even a small number of malicious entries can have outsized effects, and that the attack surface is broader than previously thought. The work also highlights that multimodal RAG introduces new attack vectors—such as image-text mismatches—that are not present in text-only RAG.

Technical Contributions

• MM-PoisonRAG Framework: A unified attack framework that generates poisoned image-text pairs to be inserted into the knowledge base.
• Local Poisoning Attacks: Targeted attacks that craft specific image-text pairs to mislead the RAG system for particular queries, exploiting the retriever's reliance on cross-modal similarity.
• Global Poisoning Attacks: Broad attacks that inject a set of poisoned documents to reduce overall retrieval precision and generation quality, often by creating conflicting or irrelevant contexts.
• Multimodal RAG Pipeline: A four-component pipeline (multimodal KB, retriever, reranker, generator) that serves as a testbed for evaluating attack effectiveness.
• Evaluation Metrics: Uses retrieval recall, answer accuracy, and faithfulness to quantify attack impact.

Results

The paper reports that global poisoning attacks reduce answer accuracy by over 40% compared to the clean baseline, while local attacks successfully redirect targeted queries to incorrect answers with high success rates (e.g., over 80% for certain queries). Retrieval recall also drops significantly, indicating that the poisoned knowledge corrupts the retriever's ability to find relevant contexts. The attacks are shown to be effective even with a small poisoning ratio (e.g., 5% of the knowledge base), demonstrating a high attack efficiency.

Significance

This paper is a wake-up call for the multimodal AI community. It shows that RAG systems, which are often assumed to be secure because they rely on external knowledge, are vulnerable to data poisoning. The findings have immediate implications for deploying RAG in production, especially in domains like healthcare, finance, and legal where misinformation can have severe consequences. The work also opens up new research directions: developing robust retrieval mechanisms that can detect poisoned entries, designing certification methods for RAG pipelines, and exploring adversarial training for multimodal encoders. As multimodal RAG becomes more prevalent, this paper provides a foundational understanding of its security risks and sets the stage for future defenses.