Nobody Alerts on Silence: Wiring Sentry Into an LLM Pipeline

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry. 🔨 #bugsmash,...

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

🔨 #bugsmash, week by week: a 390% CPU hour nobody noticed, a state machine with no exit, a backup that leaked 156 GB. Week four is the finale: I wired monitoring into the pipeline that produced all three — and its best catch was itself.

Project Overview

TextStack is an open-source reader for technical books, built in .NET: an ASP.NET Core API, a background Worker, PostgreSQL + pgvector, React on top. The LLM pipeline does translation, word explanations, "Ask this book" RAG, and three production agents (Enrichment, Librarian, Tutor), routed between a local Ollama and OpenAI by a config-driven router. The code is public: github.com/mrviduus/textstack.

Bug Fix or Performance Improvement

Three weeks ago a user's PDF fell through my LLM router onto a CPU-only Ollama container instead of GPT-4.1, and my CPU sat at 390% for an hour. Zero exceptions. Zero error logs. Zero alerts. And when I went to see what my existing observability had recorded, the answer was nothing at all: the OTLP exporter pointed at an Aspire dashboard container that is profile-gated and doesn't run in production. Every span my services had ever produced in prod had been fired into a closed socket.

Observability you never read is indistinguishable from observability you never installed.

The one-line config fix was submission #1. This submission is the fix for the class of bug — a system that has no way to make a sound when it does the wrong thing successfully:

• the router now records why it picked a provider, not just which one;
• expensive tasks landing on the default provider fire a throttled Sentry alert;
• provider failures the client deliberately swallows now report before returning their empty response;
• the Worker probes provider reachability at startup and a circuit breaker stops a dead provider from eating 50 × 90 s of wall-clock per start;
• and an environment-tag fix so a laptop can never masquerade as production again (that story is below — it earned its own PR).

Code

Four PRs, all merged to main; 1,363 unit tests, full CI green:

• #445 — the Sentry integration: SDK for API + Worker, route-reason spans, agent transactions, allowlist scrubber
• #446 — the first leak it found in itself: EF Core SQL riding in breadcrumb messages, past a green-tested scrubber
• #447 — the reader-facing race it found in me: 23505 on reading-progress upserts, real users losing their place in books
• #448 — the readiness probe, the circuit breaker, and the environment-tag fix

My Improvements

The router now says why. Route resolution was a ?? chain that produced a string — identical whether an operator deliberately routed a task or it fell off the end onto the default. That chain doesn't just fail to record intent; it destroys it. So it returns two things now:

private RouteDecision ResolveRoute(string? featureTag)
{
    var matched = RegistryKey(featureTag) ?? ConfigRouteKey(featureTag);
    return matched is not null
        ? new RouteDecision(matched, RouteReason.RouteMatched)
        : new RouteDecision(config["Ai:DefaultProvider"] ?? "openai",
                            RouteReason.DefaultFallback);
}

Every LLM call tags its span with ai.task, ai.provider.resolved, and ai.provider.reason = route_matched | default_fallback. "Which model answered this, and did anyone choose it on purpose?" is now a trace query instead of a CPU graph.

Alert arithmetic. pdf.parse resolves a route once per page with parallelism six — my first version would have turned the original incident into 106 identical Sentry events. Every alarm goes through a throttle keyed on (task, provider, reason): first hit fires immediately, then one event per hour per distinct problem. The unit test literally counts to 106 and asserts one claim.

No silent fallback, ever — in either direction. When the breaker finds Ollama dead, tasks are skipped and stay queued; nothing auto-switches to a paid provider, because that converts an outage into unbounded spend. Provider choice stays 100% config-driven.

And the first live run found a hole in my own fix. The startup probe opens the circuit on a one-minute backoff; the backfill worker wakes after a two-minute start delay — by then the circuit is legitimately half-open, and my single up-front gate waved the whole batch through. A per-book re-check turned 38 calls into one:

Metadata backfill: enriching 38 user books
Metadata backfill: aborting after 0 enriched / 1 failed — provider 'ollama'
  is unavailable; the remaining candidates stay queued

Tests check what you imagined; a live run checks what's there.

Best Use of Sentry

Error Monitoring — what the first 24 hours in production caught:

• The OpenAI account was out of credits. HTTP 429 (insufficient_quota) on /translate and /explain — the entire paid surface had been failing for readers for twelve hours. No version of my logs would have surfaced that before a user complained.
• Readers were losing their place in books. PUT /me/progress threw 23505: duplicate key value violates unique constraint ten times in four hours: a textbook read-then-insert race (session heartbeat + sendBeacon on unload + second device), milliseconds wide, invisible in tests. Fixed in #447.

Custom tags (ai.task, ai.provider, ai.failure, agent.name, agent.outcome) go through an allowlist scrubber — every tag not explicitly blessed dies at the edge, so a future SetTag("prompt", userText) can never leak. A Sentry issue answers "which feature, on which model, is broken?" without opening a trace.

Tracing covers agent runs and RAG indexing at 100% sampling (they're the reason I installed this), HTTP at 20%, health checks at 0%. I rejected the deprecated OTel bridge specifically because spans leaving through the OpenTelemetry SDK bypass BeforeSend — my OTel pipeline carries raw client IPs and full SQL text that must never leave the box. A tiny TraceScope dual-writes an Activity and a Sentry span instead, so everything Sentry receives passes my scrubber.

Breadcrumbs caught my scrubber lying — twice. A live event's breadcrumb trail contained SQL: EF Core interpolates the query into the breadcrumb message, not the structured data bag my scrubber nulled (and my unit tests were green the whole time, asserting exactly the wrong thing). Fixed by dropping EF command breadcrumbs outright — then production found the same leak in a second channel: EF logs a failed command at Error level and Sentry's ILogger integration promotes it to an event, SQL in the message again. A scrubber written against one egress path will be bypassed by the next one. Both doors are closed in #446, and dropping loses no signal — the exception middleware already reports the same failure with the SQLSTATE and constraint name, no SQL.

Release + environment tags as forensics. The most interesting issue of the first day showed a dead Ollama starving a metadata pipeline: thirty events, tagged environment: Production. I read it as an outage and started writing the fix. It was my laptop — a dev .env with ASPNETCORE_ENVIRONMENT=Production plus the production DSN I'd pasted in to verify the integration. What broke the spell was Sentry's own metadata: linux-arm64 runtime on an x86_64 prod, and a release tag pointing at a commit that had never been deployed. An environment tag is a claim a process makes about itself, not a fact. Now SENTRY_RELEASE comes from the GIT_SHA build arg — every CI-built image has one, no dotnet run ever does — and a Production claim without a release gets renamed production-unverified (#448).

And the meta-lesson that justified the whole exercise: I verified the integration by sending real events at the real DSN and reading the captured payloads in the UI — that's how the leaks, the inferred-geo surprise, and the middleware capture path all surfaced. A monitoring system whose first act is to indict itself is one you can start trusting.

What's the most embarrassing thing your monitoring has ever caught — and was it in the code, or in you?

I build TextStack, an open-source reader for technical books, in .NET. The full write-up lives on my blog. github.com/mrviduus/textstack