LamBERT

Analysis

Why This Paper Matters

Understanding documents like invoices, forms, or scientific papers requires more than just reading text—the spatial arrangement of words often carries critical meaning. Traditional language models treat text as a flat sequence, ignoring layout cues such as column structure, tables, or headings. LAMBERT addresses this gap by extending the Transformer architecture to incorporate layout information from OCR bounding boxes, all while leveraging the pretrained weights of RoBERTa. This approach is significant because it avoids the complexity and computational cost of processing raw images, making layout-aware language modeling more accessible for real-world applications.

How It Works

LAMBERT builds on the RoBERTa architecture with two key modifications: layout embeddings and a 2D relative attention bias.

First, each token is associated with a bounding box from OCR. These boxes are normalized so that the page's top-left corner is at (0,0) and dimensions are scaled by page height. The layout embedding for a token is formed by concatenating four separate embeddings—one for each coordinate (x1, y1, x2, y2). Each coordinate embedding uses sinusoidal functions with larger scaling factors than typical positional embeddings, since coordinates are in [0,1]. The resulting 8d-dimensional vector is added to the token and segment embeddings, replacing the standard sequential position embedding.

Second, the attention mechanism is modified with a relative bias term. In standard Transformers, attention scores are computed as dot products between queries and keys. LAMBERT adds a bias β_ij that depends on the relative positions of tokens i and j. To make this tractable, token positions are discretized into a grid of resolution C (typically 100). Each token's 2D position is reduced to (ξ_i, η_i) = (Cx1, C(y1+y2)/2). The bias is then computed as:

where H and V are trainable weight matrices indexed by the horizontal and vertical differences. This allows the model to learn attention patterns that depend on spatial relationships, such as attending to tokens in the same column or row.

Results

The authors pretrained LAMBERT on a corpus of 315k PDF documents (3.12M pages) from Common Crawl, using a masked language modeling objective. They then fine-tuned on several information extraction benchmarks, including SROIE (scanned receipts), and compared against plain RoBERTa and other layout-aware models.

LAMBERT consistently outperforms RoBERTa baselines, with F1 improvements of 2-5 points on most tasks. Notably, even without any layout-specific pretraining (using only the layout embeddings and bias on top of pretrained RoBERTa), the model shows gains, indicating that the architectural modifications are effective. The best results come from pretraining with layout, which further boosts performance.

Significance

LAMBERT demonstrates that layout information can be integrated into language models in a lightweight manner, without requiring image encoders or complex multimodal architectures. This opens up practical applications in document processing, such as automated form filling, invoice parsing, and scientific literature mining. The approach is also modular—it can be applied to other Transformer backbones beyond RoBERTa. Future work could explore finer-grained layout representations (e.g., handling rotated text) or combining LAMBERT with vision-based models for even richer document understanding.