A comprehensive index of artificial intelligence and machine-learning research with AI-generated summaries, citation metrics, and direct links to papers and code.
Mingzhi Chen, Taiming Lu, Jiacheng Zhu, et al.
Introduces Derf, a normalization-free pointwise function that outperforms LayerNorm, RMSNorm, and DyT across vision, speech, and DNA tasks.
Unknown
Dynamic Tanh (DyT) replaces normalization layers in Transformers with a simple element-wise operation, achieving comparable or better performance across vision, language, and speech tasks.
Unknown
ConvNets (NFNets) match Vision Transformers in performance when pre-trained on large datasets and fine-tuned on ImageNet, challenging the assumption that ViTs are inherently superior at scale.
Amin Ghiasi, Hamid Kazemi, Eitan Borgnia, et al.
This paper investigates the internal mechanisms of Vision Transformers, revealing their spatial information preservation, last-layer pooling behavior, and reliance on background features compared to CNNs.
Unknown
OWL ViT enables open-vocabulary object detection by adapting Vision Transformers with CLIP-style pre-training and bipartite matching loss.
Unknown
DETR redefines object detection as a direct set prediction problem using transformers, eliminating hand-crafted components like NMS and anchor boxes.
Unknown
SoViT introduces compute-optimal shape scaling for vision transformers, achieving performance of models twice its size with equivalent compute.
Unknown
FastViT introduces RepMixer, a reparameterized token mixer, achieving fast inference with minimal accuracy loss in hybrid vision transformers.
Ze Liu, Han Hu, Yutong Lin, et al.
Swin Transformer V2 introduces post-norm, cosine attention, log-spaced position bias, and self-supervised pretraining to scale vision transformers to 3B parameters.
Unknown
Self-supervised Vision Transformers learn explicit semantic segmentation and excel as k-NN classifiers, unlike convolutional networks.
Unknown
PaLI is a joint language-vision model that generates multilingual text from visual and textual inputs using large pre-trained encoder-decoder language models and Vision Transformers.
Unknown
Open Pre-trained Transformers (OPT) provides a suite of decoder-only models from 125M to 175B parameters, with OPT-175B matching GPT-3 performance.