Stop Your LLMs from Forgetting: How a 2016 String Algorithm…

fig1a

Have you ever tried to read a massive pile of reports and summarize them in under 50 words? It’s hard. Now, imagine asking a cutting-edge Large Language Model (LLM)—like Gemini—to do it.

You might think AIs have perfect memories, but they don't. When forced to aggregate information from dozens of documents under strict length constraints, AIs suffer from severe "memory loss" biases. They either ignore the middle of your documents or completely forget the older information they read first.

In this article, we’ll introduce a simple yet powerful solution called Pyramid Aggregation. Intriguingly, this method is adapted from a 10-year-old string concatenation algorithm that was originally designed to make basic programming languages run faster. By applying it to modern AI, we solved the forgetting problem and achieved a 95% speedup in processing time.

Let's dive in!

The Motivation: From Google Apps Script to Generative AI

On October 13, 2016, I published a technical post on my blog titled "Improved Algorithms for Summation of Array Elements". Recently, I officially archived this work on Zenodo.

Back in 2016, the goal of that paper was simple: find a highly efficient way to join (concatenate) thousands of text strings together. In the older version of Google Apps Script (before the V8 engine), standard sequential text joining was incredibly slow and memory-intensive because the computer had to rebuild the text footprint over and over again ($O(N^2)$ complexity). By proposing a hierarchical tree-like method—which I called the Pyramid Method—we restricted the active memory growth to a linear scale ($O(N)$), resulting in a massive 99.7% reduction in execution costs.

While review-proofing that paper for Zenodo recently, a spark went off in my head:

"Could this exact same pyramid tree algorithm be used to optimize how today's Generative AIs aggregate multiple documents?"

When an LLM processes long texts, the computer's attention calculations scale quadratically ($O(L^2)$) relative to the input length—just like the old string joining problem!

To test this theory, I designed an experiment comparing three text aggregation workflows: Batch-Concatenate, Sequential Update, and our proposed Pyramid Aggregation. The results were published as a preprint on Zenodo: "Pyramid Aggregator: Mitigating Information Loss in Multi-Document Fact Extraction via Hierarchical Merging".

The Core Problem: Why AIs Have "Short-Term Memory Loss"

To understand why we need a pyramid, we first need to look at how AIs behave when you feed them a lot of documents under a strict word limit (e.g., summarizing 32 system logs in under 50 words without just listing names).

Traditionally, developers use one of two methods, both of which suffer from severe positional biases:

Batch-Concatenate (The "Lost in the Middle" Effect): You staple all 32 documents together and throw them into the AI in one go. Because AIs pay more attention to the very beginning and the very end of a prompt, they suffer from Primacy Bias. The AI remembers the first few logs, ignores the middle, and gives you a summary that completely misses half the data.
Sequential Update (The "Information Drift" Effect): You feed the documents to the AI one by one. You ask the AI to read Doc 1, make a summary, then read Doc 2 to update that summary, and so on. Because the AI must respect the strict 50-word limit at every step, it has to discard old information to make room for new details. By the time it reaches Doc 32, it has forgotten almost everything from the first half of the sequence. This is Recency Bias.

Introducing the Solution: Pyramid Aggregator

Instead of processing everything at once or step-by-step, Pyramid Aggregation organizes document merging into a balanced tree topology.

Here is how the three workflows compare visually:

Figure 1: Workflows of the 3 text aggregation methods

Figure 1 displays the structural difference between the three approaches:

Batch-Concatenate (Left Panel) takes all 32 documents, forces them into a single massive block, and sends them through one LLM call to get the final summary. It is simple but causes severe information loss.
Sequential Update (Middle Panel) creates a linear pipeline (Doc 1 -> Doc 2 -> ... -> Doc 32) where each step depends on the previous output. It acts as an unavoidable bottleneck because the AI can never process the next document until the current step is finished.
Pyramid Aggregation (Right Panel) structures the 32 documents into a balanced hierarchy. Here, the 32 documents are processed in three sequential levels: first, at Level 1 (Local Extraction), they are partitioned into 8 groups of 4 and summarized; next, at Level 2 (Intermediate Merging), the 8 intermediate summaries are paired into 2 groups of 4 and merged; finally, at Level 3 (Final Merger), the 2 remaining summaries are merged into a single "Final Summary" at the very top.

The Magic of Parallel Processing with the Antigravity SDK

In a basic implementation, running 11 separate LLM calls (8 for Level 1, 2 for Level 2, and 1 for Level 3) would be painfully slow if executed one by one. This is where the Antigravity Python SDK's concurrency control comes into play.

In the Pyramid workflow, the 8 nodes in Level 1 are completely independent of each other. The Antigravity SDK leverages event-driven asynchronous execution to process all 8 Level-1 calls at the exact same time (in parallel). Once those return, the SDK immediately triggers the 2 Level-2 merge calls in parallel.

Using the SDK's async context (async with Agent(config)) combined with asyncio.gather, we can orchestrate this entire multi-agent hierarchy concurrently. The SDK handles connection pooling and rate limits automatically behind the scenes, allowing us to transition from a linear time complexity ($O(\theta)$ rounds) to a highly efficient logarithmic scale ($O(\log_{\phi} \theta)$ rounds). Every document maintains an identical 3-step depth to the top, completely eliminating position bias while ensuring near-instant execution.

The Experiment and Mind-Blowing Results

We put these three methods to the test using 32 synthetic system logs (each reporting an error in a specific software module) and a lightweight LLM (gemini-3.1-flash-lite) orchestrated via the Antigravity Python SDK.

We set a strict limit: the final summary had to be under 50 words and could not list more than 3 module names in a single sentence (preventing the AI from cheating by just listing everything).

Here are the actual results from our benchmark:

Figure 2: Experimental results comparing the three methods

Looking at Figure 2, we can draw three critical conclusions:

Panel A: Tracking AI Memory Loss

In Panel A, we tracked which documents (from 0 to 31) made it into the final summary:

Batch-Concatenate (orange line) only captured the very first documents (Primacy Bias).
Sequential Update (purple line) only captured the very last documents (Recency Bias).
Pyramid Aggregation (green line) didn't prioritize any single index, keeping a flat, unbiased profile.

Panel B: The Quality Trade-Off (Micro vs. Macro)

Under a strict 50-word limit, it is mathematically impossible to list all 32 names. So what does the AI prioritize? In Panel B, each method displays two bars: a blue bar for Macro Domain Coverage (%) and an orange bar for Micro Facts Counted (%):

The baseline methods (Batch and Sequential) tried to preserve specific names, showing short orange bars (recovering 9.4% and 25.0% of micro facts). However, because they truncated the middle or start of the context, their blue bars only reached 50.0% coverage of the overall system failure domains, completely missing the other half.
Pyramid Aggregation chose a different path. It abstracted the specific names into broader categories (e.g., summarizing specific module names as "memory, network, and database failures"). This resulted in an orange bar of 0.0% (recovering no exact names), but successfully pushed its blue bar to 100.0% Macro Domain Coverage. It gave a balanced, complete view of the entire system status without leaving any document block out.

An Analogy: The Classroom Roll Call

Why is a 0.0% Micro Recovery actually a victory for the Pyramid method? Think of it this way:

The Baseline Approach: A teacher is asked to introduce 32 students in under 50 words. To do this, the teacher just reads the names of the first 3 students ("Golem, Avalon, Titan") and ignores the remaining 29. They get a few exact names right (a short orange bar), but completely ignore the rest of the class (50% blue bar).
The Pyramid Approach: The teacher groups the students by their activities and reports: "This class consists of 10 athletes, 12 artists, and 10 programmers." No individual names are spoken (0.0% orange bar), but every single student is represented in the summary (100% blue bar).

Here is a visual representation of this analogy:

Figure 3: Classroom Analogy of Text Aggregation

As shown in Figure 3, the baseline approach (left) only keeps a few active students (like Sarah, Mike, and Ben) while leaving the rest of the class greyed out and forgotten. On the other hand, the Pyramid approach (right) synthesizes the entire classroom by dividing all 32 students into functional categories (Athletes, Artists, and Programmers). In other words, the 0% in Panel A/B isn't a failure to remember—it is a proof of high-level semantic synthesis. The AI successfully summarized the data rather than lazily cropping it.

Panel C: Speeding Up with Parallel Execution (Lower is Better)

Perhaps the most dramatic result is the execution time, shown in Panel C (where lower is better):

Batch-Concatenate (light blue bar) finished in a rapid 5.44 seconds because it only executed a single API call. However, it achieved this speed at the cost of losing half of the overall document content.
Sequential Update (pink bar) took a whopping 166.02 seconds (almost 3 minutes) because it had to wait for each step to finish before starting the next one.
Pyramid Aggregation (greenish-blue bar) completed the entire process in just 8.10 seconds!
By leveraging the Antigravity SDK's asynchronous framework to run Level 1 and Level 2 in parallel, we compressed the time complexity to a logarithmic scale ($O(\log \theta)$), achieving a 95.1% speedup compared to the sequential method. It performs almost as fast as a single batch call while maintaining 100% information coverage.

Conclusion

By taking a simple tree-reduction algorithm written in 2016 for basic string concatenations and mapping it to the attention constraints of modern Large Language Models, we solved a fundamental issue in AI document synthesis.

Pyramid Aggregation acts as a hierarchical semantic filter, turning raw, massive text inputs into balanced, high-level summaries without dropping historical context. Thanks to modern asynchronous orchestration SDKs like Antigravity, we can deploy this tree structure at scale, achieving near-instant results.

If you are building AI agents, RAG pipelines, or log analysis tools, stop throwing all your text into a single prompt or chaining them sequentially. Build a pyramid instead!