Stripe's engineering team has automated database incident recovery for its MongoDB infrastructure by modeling the global system as a graph and applying graph search algorithms alongside state machines. The approach, detailed in a company blog post and reported by InfoQ's Renato Losio on Aug 09, 2026, replaces hard-coded remediation sequences with dynamic pathfinding that adapts to different shard layouts.
The results are measurable. Database-related pager alerts dropped by about 30%, which translates to 200 fewer pages per year. The system also eliminates an estimated 12 days of unhealthy shard states annually. These gains come after a period when the old control plane paged operators 124 times in six months for misconfigured shards, and 32 times for single-node-down scenarios complicated by additional health issues.
From Hard-Coded Rules to Dynamic Pathfinding
Stripe's original remediation system was hard-coded and plugin-based. It did not scale well, according to the engineering team. Fragile dependencies, complex multi-failure scenarios, layout-specific logic, and unhandled intermediate states frequently required manual intervention. Critical operations like index builds and planned maintenance were blocked for an average of one hour per incident under that system.
The new approach models global infrastructure as a graph. Nodes represent infrastructure components, edges capture relationships between them, and node attributes describe the current state of each component. Graph search algorithms and state machines compute and execute remediation plans automatically, without human input.
Stripe removed the hard-coded sequences entirely. Instead, the system relies on graph traversal to identify valid recovery paths. The same remediation logic adapts automatically to different database layouts and evolving infrastructure, a capability the team says was impossible with the previous design.
BFS First, Then Dijkstra's Algorithm
Stripe initially used breadth first search (BFS) to find valid remediation paths. The team later adopted Dijkstra's algorithm to prioritize lower-cost recovery plans. Dijkstra's reduces unnecessary operations while preserving correctness, according to the blog post.
The algorithm explores paths to all reachable states, not only the goal state. When no complete path exists, Dijkstra's returns the path to the least misconfigured state, enabling partial remediation. That means the system can still improve a broken shard even when full recovery is not immediately possible.
Remediation is modeled as composable rules with explicit state transitions. The planner can combine operations dynamically as infrastructure evolves. This design allows the system to discover novel recovery procedures, unlike runbooks which only encode known procedures.
A Daily Reality for Global Database Fleets
Scott MacVicar, head of developer infrastructure at Stripe, described the operational challenge on LinkedIn. "When you operate a global database fleet, hardware degradation and unhealthy shards are daily occurrences," he wrote. "The challenge is fixing these issues without burning out on-call engineers."
MacVicar's comments frame the motivation behind the automation. The system's ability to handle partial remediation and adapt to different MongoDB shard layouts directly addresses the pain points that caused frequent manual intervention in the past.
Stay ahead of the AI curve
The most important updates, news, and content — delivered weekly.
No spam. Unsubscribe anytime.
The team states that state machine modeling, simulation-based planning, and runtime pathfinding offer an alternative to accumulating runbooks. Instead of documenting every possible failure scenario, the system computes recovery paths on the fly.
Beyond Failure Recovery
Stripe plans to extend the framework beyond failure recovery. Planned extensions include automating topology changes and blue-green deployments, as well as orchestrating planned maintenance alongside reactive healing. These ambitions suggest the graph-based approach could become a general-purpose infrastructure orchestration layer.
The company's engineering team recently published articles on fund segregation in Stripe Connect and on event notification handlers for processing thin events. Those pieces, along with the remediation work, show a broader push toward automated, self-managing infrastructure.
Industry-Wide Investment in Automation
Stripe is not alone in this direction. Uber recently described its declarative, self-healing Odin platform, which similarly aims to reduce manual operational work. Meta has detailed AI-assisted tooling to accelerate incident response. Both companies, like Stripe, are investing heavily in automated infrastructure operations.
The InfoQ article places Stripe's work in the AI, ML & Data Engineering topic, with related coverage in Architecture & Design, DevOps, GraphDB, MongoDB, State-machine, Database, Site Reliability Engineering, and Incident Response. The breadth of those tags reflects how the approach touches multiple disciplines.
The shift from runbooks to state-machine-driven discovery is notable. Runbooks encode known recovery procedures; a state machine discovers novel ones. That distinction matters as infrastructure grows more complex and failure modes multiply.
Stripe's system now dynamically adapts to different MongoDB shard layouts, a claim the team makes explicitly. The reduction in pager alerts and unhealthy shard states provides concrete evidence that the approach works in production.
The 30% reduction in alerts and the 200 fewer pages per year are the headline numbers. But the elimination of 12 days of unhealthy shard states annually may matter more for long-term reliability. Each hour of blocked critical operations under the old system represented real downtime for customers.
The six-month window of 124 pages for misconfigured shards and 32 pages for complicated single-node failures shows how often the old system struggled. Those incidents frequently required manual intervention, which is exactly what the new system aims to eliminate.
Stripe's engineering team concludes that simulation-based planning and runtime pathfinding offer a compelling alternative to accumulating runbooks. The evidence so far supports that conclusion, though the system's long-term performance will depend on how well it handles the next generation of failure modes.