Journal Article
Machine Learning

BadNets: Evaluating Backdooring Attacks on Deep Neural Networks

Tianyu Gu(New York University), Kang Liu(New York University), Brendan Dolan-Gavitt(New York University), Siddharth Garg(New York University)
January 1, 2019IEEE Access1,387 citations
PDFDOI

1.4k

Citations

325

Influential Citations

IEEE Access

Venue

2019

Year

Abstract

Deep learning-based techniques have achieved state-of-the-art performance on a wide variety of recognition and classification tasks. However, these networks are typically computationally expensive to train, requiring weeks of computation on many GPUs; as a result, many users outsource the training procedure to the cloud or rely on pre-trained models that are then fine-tuned for a specific task. In this paper, we show that the outsourced training introduces new security risks: an adversary can create a maliciously trained network (a backdoored neural network, or a BadNet) that has the state-of-the-art performance on the user's training and validation samples but behaves badly on specific attacker-chosen inputs. We first explore the properties of BadNets in a toy example, by creating a backdoored handwritten digit classifier. Next, we demonstrate backdoors in a more realistic scenario by creating a U.S. street sign classifier that identifies stop signs as speed limits when a special sticker is added to the stop sign; we then show in addition that the backdoor in our U.S. street sign detector can persist even if the network is later retrained for another task and cause a drop in an accuracy of 25% on average when the backdoor trigger is present. These results demonstrate that backdoors in neural networks are both powerful and-because the behavior of neural networks is difficult to explicate-stealthy. This paper provides motivation for further research into techniques for verifying and inspecting neural networks, just as we have developed tools for verifying and debugging software.

Analysis

Why This Paper Matters

This paper is a seminal work in the field of adversarial machine learning, specifically introducing the concept of backdoor attacks on deep neural networks. It addresses a practical and growing concern: as more organizations outsource model training to the cloud or use pre-trained models, the integrity of the training process becomes a critical security issue. The paper demonstrates that an adversary can inject a backdoor into a model without degrading its performance on normal tasks, making the attack stealthy and difficult to detect. This has significant implications for real-world applications such as autonomous driving, facial recognition, and malware detection, where a compromised model could lead to catastrophic failures.

The work also underscores the opacity of neural networks, as the backdoor behavior is not easily explicable through standard inspection. This motivates the need for new verification and debugging tools, analogous to those used in software engineering. The paper's findings have spurred a large body of follow-up research on both attack and defense mechanisms, making it a foundational reference in the area of neural network security.

Technical Contributions

  • Definition of BadNets: Formalizes the concept of a backdoored neural network that performs well on clean data but misbehaves on attacker-chosen inputs with a specific trigger.
  • Toy Example: Demonstrates a backdoored MNIST classifier where a trigger pattern causes misclassification, illustrating the attack's feasibility.
  • Realistic Scenario: Creates a U.S. street sign classifier that misidentifies stop signs as speed limits when a special sticker is added, showing the attack's practical relevance.
  • Persistence of Backdoors: Shows that backdoors can survive retraining for a different task, with an average accuracy drop of 25% when the trigger is present, indicating the robustness of the attack.

Results

The paper reports that backdoored networks maintain state-of-the-art performance on clean validation samples, making the attack stealthy. In the street sign scenario, the backdoor persists even after the network is retrained for another task, causing a 25% average accuracy drop when the trigger is present. These results quantify the attack's effectiveness and highlight the difficulty of removing backdoors through standard retraining.

Significance

This paper has had a profound impact on the AI security community, with over 1387 citations. It introduced a new class of attacks that are both powerful and stealthy, leading to extensive research on backdoor detection and mitigation. The work also emphasizes the need for transparency and verification in deep learning, influencing policy and best practices for secure model deployment. It remains a key reference for understanding the risks of outsourced training and the importance of trust in AI systems.

deep-learningbackdoor-attacksneural-network-securityoutsourced-trainingadversarial-machine-learningtrojan-attacksmodel-poisoning
Fine TuningNeural Networks

Related Papers