Journal Article
Computer Vision

Deep convolutional networks do not classify based on global object shape

Nicholas Baker(University of California, Los Angeles), Hongjing Lu(University of California, Los Angeles), Gennady Erlikhman(University of Nevada, Reno), Philip J. Kellman(University of California, Los Angeles)
December 7, 2018PLoS Computational Biology400 citations
PDFDOI

400

Citations

24

Influential Citations

PLoS Computational Biology

Venue

2018

Year

Abstract

Deep convolutional networks (DCNNs) are achieving previously unseen performance in object classification, raising questions about whether DCNNs operate similarly to human vision. In biological vision, shape is arguably the most important cue for recognition. We tested the role of shape information in DCNNs trained to recognize objects. In Experiment 1, we presented a trained DCNN with object silhouettes that preserved overall shape but were filled with surface texture taken from other objects. Shape cues appeared to play some role in the classification of artifacts, but little or none for animals. In Experiments 2-4, DCNNs showed no ability to classify glass figurines or outlines but correctly classified some silhouettes. Aspects of these results led us to hypothesize that DCNNs do not distinguish object's bounding contours from other edges, and that DCNNs access some local shape features, but not global shape. In Experiment 5, we tested this hypothesis with displays that preserved local features but disrupted global shape, and vice versa. With disrupted global shape, which reduced human accuracy to 28%, DCNNs gave the same classification labels as with ordinary shapes. Conversely, local contour changes eliminated accurate DCNN classification but caused no difficulty for human observers. These results provide evidence that DCNNs have access to some local shape information in the form of local edge relations, but they have no access to global object shapes.

Analysis

Why This Paper Matters

This paper is a landmark in understanding the fundamental differences between deep convolutional neural networks (DCNNs) and human visual perception. While DCNNs achieve near-human accuracy on object classification benchmarks, the underlying mechanisms remain poorly understood. The authors directly challenge the prevailing assumption that DCNNs learn holistic, shape-based representations similar to biological vision. By systematically dissecting the role of shape information, they reveal a critical blind spot: DCNNs rely on local texture and edge fragments rather than global object shape.

For AI practitioners, this work has profound implications. It explains why DCNNs are vulnerable to adversarial perturbations that alter local texture, and why they fail on out-of-distribution stimuli like sketches or silhouettes. The findings also highlight a key architectural limitation: the lack of explicit global shape processing mechanisms. This has spurred research into shape-biased networks, capsule networks, and transformer-based architectures that better capture global structure.

Technical Contributions

  • Experiment 1: Presented DCNNs with object silhouettes filled with texture from other objects. Classification accuracy dropped significantly for animals but less for artifacts, suggesting some shape sensitivity for man-made objects.
  • Experiments 2-4: Tested glass figurines (transparent, no texture), outlines (edges only), and silhouettes (filled shapes). DCNNs failed on glass and outlines but succeeded on silhouettes, indicating reliance on internal texture rather than boundary shape.
  • Experiment 5: Created stimuli that preserved local features (e.g., edge fragments) but disrupted global shape, and vice versa. DCNNs classified local-feature-preserved stimuli as well as normal images, but failed on global-shape-only stimuli. Humans showed the opposite pattern, with accuracy dropping to 28% on local-feature-only displays.
  • Key insight: DCNNs access local edge relations but have no access to global object shape, contradicting the hypothesis that they learn shape-based representations.

Results

  • Silhouettes: DCNNs achieved moderate accuracy (e.g., ~60% for artifacts) but near-chance for animals (~25%).
  • Glass figurines: DCNN accuracy near chance (~10-15%) across all categories.
  • Outlines: DCNN accuracy at chance level (~10%).
  • Local-feature-preserved displays: DCNN accuracy comparable to original images (~80-90%).
  • Global-shape-only displays: DCNN accuracy dropped to near chance (~10-20%), while humans maintained ~72% accuracy.
  • Local contour changes: DCNN accuracy fell to ~10%, while humans remained at ~90%.

Significance

This paper fundamentally reshaped the understanding of DCNN representations, showing they are texture-biased rather than shape-biased. It directly influenced subsequent work on shape-texture decomposition, adversarial robustness, and the development of vision transformers that can capture global context. The findings also caution against using DCNN accuracy as a proxy for human-like perception, emphasizing the need for more interpretable and biologically plausible models. For AI safety and reliability, this work underscores the importance of testing models on shape-based tasks to ensure robust generalization beyond training distributions.

deep-learningcomputer-visionobject-recognitionshape-perceptionhuman-visionconvolutional-networkslocal-features
Computer Vision

Related Papers