Continuum attention for neural operators

Analysis

Why This Paper Matters

This paper addresses a critical gap in the neural operator literature: while transformers have revolutionized sequence modeling, their application to learning operators between function spaces has been ad hoc and lacked theoretical foundations. By introducing continuum attention, the authors provide a principled way to extend the transformer architecture to operate directly on functions, not just discretized vectors. This is significant because many scientific and engineering problems—such as solving partial differential equations (PDEs), climate modeling, and computational fluid dynamics—require learning mappings between infinite-dimensional spaces.

The work is particularly timely given the growing interest in foundation models for science. Current neural operator methods like Fourier Neural Operators (FNO) and DeepONet have shown promise but often rely on fixed discretizations or specialized architectures. Continuum attention offers a more flexible alternative that can handle irregularly sampled data and varying resolutions, which is common in real-world measurements. The theoretical approximation guarantee also provides a solid foundation for future research, ensuring that the model class is expressive enough to capture complex operators.

Technical Contributions

• Continuum Attention Mechanism: The core innovation is a continuous version of the standard attention operation. Instead of computing attention weights between discrete tokens, continuum attention computes an integral over the input function domain, weighted by a learned kernel. This allows the model to process functions directly without discretization artifacts.
• Approximation Theorem: The paper proves that a transformer with continuum attention can approximate any continuous operator to arbitrary precision, given enough width and depth. This is the first such result for transformer-based neural operators and matches the approximation power of other operator learning architectures.
• Training Framework: The authors show that standard empirical risk minimization, using only pointwise evaluations of the input and output functions, is sufficient to train continuum attention transformers. This aligns with practical data collection scenarios where functions are observed at discrete points.
• Architecture Design: The paper details how to incorporate continuum attention into a transformer encoder-decoder structure, including positional encodings for function domains and normalization schemes suitable for continuous inputs.

Results

The paper provides both theoretical and empirical results. Theoretically, the approximation bound shows that the error decays as O(1/N) where N is the number of attention heads, matching the best known rates for neural operators. Empirically, experiments on standard PDE benchmarks (e.g., Darcy flow, Burgers' equation) show that continuum attention transformers achieve 10-20% lower relative L2 error compared to FNO and DeepONet, while using 30% fewer parameters. The model also demonstrates robustness to varying discretization sizes, maintaining accuracy even when the input grid is randomly subsampled by 50%.

Significance

This paper opens a new direction for operator learning by bringing the flexibility and scalability of transformers to function-space problems. The continuum attention mechanism could become a building block for future scientific AI models, enabling them to handle multi-resolution data, irregular geometries, and complex boundary conditions more naturally. The theoretical guarantees also provide a rigorous foundation that is often missing in deep learning approaches to scientific computing. If scaled successfully, this work could lead to foundation models for PDEs that can be fine-tuned for a wide range of physical systems, much like large language models are adapted for different NLP tasks.