At QCon AI, a practitioner-led conference focused on scaling AI workloads safely, DoorDash software engineer Bruna Pereira walked attendees through the design, implementation, and eventual dismantling of SafeChat, an AI-powered content moderation system for the company's real-time marketplace. The talk, published by InfoQ, traced how a narrowly built chat safety tool evolved into a content-agnostic moderation platform used across the company. Pereira, who leads the trust and safety engineering team from São Paulo, Brazil, brought more than a decade of software engineering experience to the stage, including three years spent building a startup and a background in FinTech.
DoorDash operates a marketplace where consumers order food, Dashers deliver it, and merchants prepare it. These three groups interact through chat, calls, and in person. The company treats being safe and feeling safe as equal product metrics. A meaningful number of safety incidents on the platform stem from verbal abuse. Relationships between the parties last anywhere from 40 minutes to 6 minutes max, a short window that shapes how moderation must work.
The scale of these interactions is enormous. Every day, the chat system handles over 4 million messages. Voice calls between consumers and Dashers during deliveries exceed 400,000 daily. Images exchanged in chat or SMS top 200,000 per day. For chat, the goal is to classify every message as safe before it reaches its destination. The system has only a fraction of a second to make that call.
The Initial Idea That Failed
The business side of DoorDash initially suggested using an LLM to classify every message as safe or unsafe. On paper, that approach seemed straightforward. In practice, it collapsed under real-world constraints. LLM latency varies from 2 to 10 seconds on average for their use case. With over 4 million messages flowing daily, waiting several seconds per message would destroy the chat experience. Calling an LLM on all messages would have worked theoretically but not in practice due to latency and cost.
Pereira explained the core tension plainly. The economics of using an LLM only work if you ask it only the hard questions. If every message, including the vast majority that are perfectly fine, triggers a slow, expensive LLM call, the system becomes unusable. The team needed a different architecture.
Before building anything, the team spent a couple of months understanding what unsafety means in their context. They instrumented the chat and used a free moderation API asynchronously to understand categories of unsafe messages. This data-gathering phase proved essential. Only a small single-digit percent of messages were found to be unsafe. That number shaped everything that followed. If nearly all messages are safe, the system should spend almost no effort on them and reserve heavy processing for the rare problematic ones.
The team built a small in-house trained ML classifier to serve as that cheap first layer. The classifier had to meet strict requirements. It needed to respond in less than 100 milliseconds at 90% of the time. It had to be cheap, meaning no per-call costs. And it had to be good at identifying what is obviously safe. The classifier is not a final judge, but a filter, like a metal detector at an airport. It lets the vast majority of messages pass through without further scrutiny.
Messages not classified as safe by the internal model, which amounts to less than 10% of all traffic, get sent to an LLM. This two-tier design keeps LLM usage minimal while ensuring nothing dangerous slips through. The LLM is not asked for a simple yes or no. Instead, it is asked to score messages across multiple axes, such as threatening, profane, or sexual. This scoring approach proved more stable than a Boolean safe or unsafe answer.
The data-gathering phase also revealed the distribution of abuse categories. The free moderation API, used asynchronously, helped the team see which types of unsafety were most common. This informed the axes the LLM would later score. The team learned that threats, profanity, and sexual content each required different handling. A single binary label could not capture that nuance.
The team also learned that context matters. A message that looks unsafe in isolation might be fine in a longer conversation. The scoring approach allowed the LLM to consider the message in context, producing a more accurate assessment. This was another reason scores beat Booleans.
Why Scores Beat Booleans
Pereira emphasized a key design decision during her talk. A Boolean is a flag and a score is a knob. Scores allow for graduated actions, adding new categories later, and moving thresholds without redesigning the system. With a score, the team can decide that a message scoring 0.3 on profanity gets censored, while a message scoring 0.9 on threats triggers a full block and order cancellation. A Boolean forces a binary choice with no middle ground.
The architecture that emerged follows a clear pipeline. A message comes in, the system strips noise such as empty messages, image attachments, and common pleasantries. The small model classifier then evaluates it. If the message is safe, it ships immediately. If not, the LLM takes over. The LLM produces scores, and the system applies a graduated action based on severity.
Low-severity content, such as swearing, leads to censoring the message and letting it go through. Mid-severity content, like insulting someone, leads to blocking the message entirely. High-severity content, such as a threat, leads to blocking the message and offering the affected party the option to cancel the order without paying. Very high-severity content escalates further. The system blocks the message, cancels the order, warns the offender, and removes the affected party from the loop.
This graduated approach gives the system flexibility. Not every unsafe message deserves the same response. A mild curse word is not the same as a death threat. The scoring system allows the team to tune responses precisely.
The scoring system also made it easier to add new categories later. If the team discovered a new type of abuse, they could add a new axis without redesigning the pipeline. This extensibility was a direct result of choosing scores over Booleans. The team could also adjust thresholds over time as they learned more about the abuse patterns on the platform.
Pereira noted that the LLM's scoring stability was a pleasant surprise. Early experiments showed that asking for a score on a scale produced more consistent results than asking for a yes or no answer. The LLM was better at ranking severity than at making binary judgments. This finding reinforced the design choice.
The noise-stripping step also proved important. Empty messages, image attachments, and common pleasantries were filtered out before the classifier even saw them. This reduced the load on the cheap model and the LLM. It also prevented the system from wasting resources on messages that did not need moderation.
Handling Images and Voice
Chat messages are not the only vector for abuse. Images and voice calls also carry risk. For images, the team uses a commercial vision API as the cheap layer instead of the internal model. The vision API screens images before they reach recipients. Voice presents a different challenge entirely. The message cannot be prevented from being delivered because it is already heard. The system can hang up the call and cancel the order, but it cannot un-say what was said.
The voice channel handles over 400,000 calls daily between consumers and Dashers. With that volume, abuse can occur frequently. The system detects abusive language in real time and takes action by terminating the call and canceling the order. This is a reactive measure, but it still reduces harm by stopping the interaction and offering the affected party an out.
Images add another layer of complexity. Over 200,000 images flow through chat or SMS each day. A commercial vision API screens these for inappropriate content. The API acts as the cheap filter, catching obvious violations before they reach the recipient. Anything ambiguous can be escalated, though the talk did not detail a second stage for images specifically.
The voice detection system operates differently from the chat system. Because the message is already spoken, the system cannot block it. Instead, it listens in real time and reacts when it detects abuse. The reaction is immediate termination of the call and cancellation of the order. This stops the abuse from continuing and gives the affected party a way out.
The image screening also has its own flow. The commercial vision API is integrated as the cheap layer, similar to how the internal model serves chat. The API screens images for inappropriate content before they are delivered. This prevents harmful images from reaching recipients in the first place.
Pereira noted that the voice and image channels were added after the chat system proved successful. The pattern of a cheap filter followed by a smart judge applied to all three channels. The team did not need to build a separate architecture for each. The same principles guided the design.
The voice channel's reactive nature means it cannot prevent the initial harm. But it can stop the harm from continuing. Terminating the call and canceling the order removes the affected party from the abusive situation. This is a meaningful reduction in harm, even if it is not perfect prevention.
Measurable Impact
After implementing SafeChat, DoorDash measured roughly a 50% reduction in incidents driven by verbal abuse. Pereira stressed that this is a real reduction in human harm, not just a model accuracy improvement. The number represents fewer people experiencing abuse on the platform. That is the metric that matters.
The 50% figure came after the full SafeChat system was live, including the cheap classifier, the LLM scorer, and the graduated actions. The system did not eliminate abuse entirely, but it cut it in half. For a platform handling over 4 million chat messages daily, that is a substantial improvement in user safety.
Pereira noted that the system had to operate within tight latency constraints. The cheap model responds in less than 100 milliseconds at 90% of the time. The LLM, when invoked, takes 2 to 10 seconds on average. That latency is acceptable for the less than 10% of messages that reach the LLM, but it would be catastrophic if applied to every message.
The reduction in incidents was measured across the platform. The team tracked verbal abuse incidents before and after the system went live. The 50% reduction was consistent across different time periods and user segments. This gave the team confidence that the improvement was real and not a statistical fluke.
Pereira also noted that the system's impact went beyond the headline number. The graduated actions meant that low-severity abuse was handled differently from high-severity abuse. This reduced the overall harm even when incidents still occurred. A censored message causes less harm than a delivered threat.
The latency constraints were a constant consideration. The cheap model's sub-100-millisecond response time was critical for the chat experience. Users expect messages to appear instantly. Any delay would be noticeable and frustrating. The two-tier design kept the average latency low while still catching the rare unsafe message.
The team also monitored the system's performance over time. They tracked false positives and false negatives. False positives, where safe messages were blocked, were rare but annoying. False negatives, where unsafe messages slipped through, were more serious. The team used this data to refine the model and the thresholds.
Throwing Away SafeChat
Despite the success, the team made a bold decision. They threw away the SafeChat system entirely. The learnings, model, and data were kept, but the codebase was discarded. The reason was simple. Other teams at DoorDash wanted to use the same pattern for different use cases. They wanted to moderate profile pictures, names at signup, food reviews, and identify fraud in chat and phone calls. The SafeChat system was too narrowly built for chat. It could not be easily adapted.
Pereira explained that the pattern, not the system itself, was what other teams wanted. The pattern of a cheap filter, a smart judge, and graduated action applies broadly. A profile picture moderator needs the same architecture as a chat moderator. A food review moderator needs it too. The team recognized that building a separate system for each use case would be wasteful.
The team built a content-agnostic moderation platform that allows teams to configure moderation without writing code. This platform abstracts away the specifics of chat, images, or reviews. Teams define their moderation needs through configuration, and the platform handles the rest.
The platform has three kinds of models. Internal models are trained, fine-tuned, and deployed on DoorDash's own servers. External models come from vendors with contracts and are integrated once into the platform. External prompts are prompts written to use in any LLM from any vendor, accessed through an LLM gateway. The LLM gateway integrates with models from various vendors and allows declaring fallback and retry strategies without writing code.
The decision to throw away the codebase was not easy. SafeChat was working and delivering measurable results. But the team saw that the code was too specific to chat. Every new use case would require significant refactoring. A configurable platform would serve many teams with less effort.
Pereira emphasized that the learnings, model, and data were preserved. The team did not start from scratch. They carried forward the knowledge of what worked and what did not. The model, in its 9th version, was reused. The data from the original data-gathering phase informed the platform's design.
The platform's configurable nature means teams can define their own moderation pipelines. They choose the models, the thresholds, and the actions. They do not need to write code. This lowers the barrier to entry for moderation across the company.
Building Blocks and Agents
The platform allows composing building blocks into moderation agents, which are essentially pipelines or workflows. Conditions between steps are expressed via the UI using the output from the previous model. This means a team can define a pipeline like this. If the internal model scores a message above 0.5 on the unsafe label, send it to the LLM prompt. Otherwise, apply a default action. All of this is configured visually, with no code required.
Stay ahead of the AI curve
The most important updates, news, and content — delivered weekly.
No spam. Unsubscribe anytime.
The SafeChat pipeline serves as a canonical example. Internal model, if unsafe label is greater than 0.5, go to LLM prompt, then action. Otherwise, action. This simple composition captures the essence of the two-tier moderation approach.
Moderation agents can run synchronously or asynchronously. Synchronous agents make an HTTP call and hold the connection open until the result is ready. These are used when gating a decision, such as blocking a chat message before delivery. Asynchronous agents acknowledge the request, run in the background, and publish the result to a Kafka topic. These are preferred when avoiding latency caps on individual steps. For example, a food review might be moderated asynchronously, with the result published later.
The platform also includes a backtesting feature. Teams can test agents against historical data before deploying them. Backtesting allows testing a single step or the entire agent. This feature makes test it before trust it a real built-in workflow. Pereira noted that a thousand examples is a good number for backtesting with human review. That dataset size provides enough signal to evaluate an agent's performance.
During backtesting, humans can classify results as correct or incorrect, or as true positive, true negative, false positive, or false negative. Metrics are calculated from this human input to decide if the agent is good enough for production. This closes the loop between model development and deployment.
The building blocks are modular. Teams can mix and match internal models, external models, and external prompts. They can add conditions between steps. They can choose synchronous or asynchronous execution. This flexibility is what makes the platform content-agnostic.
The backtesting feature is particularly valuable for teams new to moderation. They can test their configuration against historical data before going live. This reduces the risk of shipping a broken pipeline. The human review process ensures that the metrics reflect real-world performance.
Pereira noted that the platform's UI makes configuration accessible. Teams do not need to be ML experts to build a moderation agent. They can see the pipeline visually and understand how messages flow through it. This democratizes moderation across the company.
Model Evolution
The internal model for SafeChat is in its 9th version, meaning it has been trained nine times. The cheap model is scored from 0 to 1 on how unsafe or safe it is. This scoring approach mirrors the LLM's scoring, keeping the two layers consistent.
The first round of training for the cheap model was almost binary. It was trained on very bad messages and ok messages, which produced a model that struggled with nuance. A second round of training used the free moderation API to get more gradual data. This gave the model a better sense of the spectrum between clearly safe and clearly unsafe.
Training and retraining of internal models is handled by the ML platform team, not the moderation platform team. The moderation platform is a client of the ML platform. This separation of concerns keeps the moderation team focused on configuration and workflows while the ML team handles model training.
Retraining decisions are based on random data analysis and feedback from agents on incidents that were not caught. If the system misses certain types of abuse, that feedback triggers a retraining cycle. An example of retraining was on abbreviations, which the first layer missed because it thought they were safe. Abbreviations like certain shorthand for profanity or threats can slip past a model trained on full words. Retraining on these cases improved the model's recall.
Pereira shared several lessons learned during the talk. Put a cheap model in front of an LLM. Ask for scores not labels. Know when to throw a system away. These lessons apply beyond DoorDash's specific context. Any company building moderation systems can benefit from the same architecture.
The model's evolution shows the importance of iterative training. The first version was too binary. It could not distinguish between a mild curse and a serious threat. The second version, trained on more gradual data, performed better. Subsequent versions continued to improve as the team added more training data and refined the approach.
The retraining on abbreviations was a specific example of the feedback loop. The model missed certain shorthand terms because it was trained on full words. Once the team identified this gap, they added examples of abbreviations to the training data. The next version of the model caught these cases.
The separation between the ML platform team and the moderation platform team is a key architectural decision. The moderation team does not need to worry about model training. They focus on configuration and workflows. The ML team handles the technical details of training and deploying models.
Pereira also noted that the platform's model registry allows teams to see which version of a model they are using. This transparency is important for debugging and auditing. If a moderation agent behaves unexpectedly, the team can check which model version is in use.
Q&A Insights
The talk included a Q&A session with audience members. One participant expressed interest in adopting the approach. "I like what you're doing. I want to do it myself as well for my use case." This sentiment captured the broader appeal of the platform. The pattern is transferable.
Another participant asked about cost. Pereira explained that the cheap model has no per-call costs, which keeps the system economical. The LLM is only invoked for less than 10% of messages, so LLM costs remain manageable. The commercial vision API for images adds some cost, but it is still cheaper than running an LLM on every image.
A participant asked about quantifying backtesting. Pereira noted that a thousand examples is a good number for human review. More examples would be better, but a thousand provides enough signal to make a decision. The human review process classifies results, and metrics are calculated from those classifications.
Another question touched on model retraining decisions. Pereira explained that the team relies on random data analysis and feedback from agents on missed incidents. The ML platform team handles the actual training. The moderation team provides the data and the feedback loop.
The talk runs 42:22 in length and includes slides and a transcript. InfoQ published the full presentation, making it available to a wider audience.
The Q&A revealed that the audience was engaged with the practical details. The participant who wanted to adopt the approach was interested in the transferability of the pattern. The cost question showed that economics are a major consideration for moderation systems. The backtesting question highlighted the importance of validation before deployment.
Pereira's answers were direct and practical. She did not speculate about future features or hypothetical scenarios. She stuck to what the team had built and what they had learned. This grounded approach resonated with the audience.
The retraining question led to a discussion of the feedback loop. Pereira explained that the team does not retrain on a fixed schedule. Instead, they retrain when they see gaps in coverage. This data-driven approach ensures that training resources are spent where they are needed most.
The Platform's Broader Reach
The content-agnostic moderation platform now serves multiple teams at DoorDash. Profile pictures are moderated at signup. Names are screened for inappropriate content. Food reviews are checked for abuse. Fraud detection in chat and phone calls uses the same infrastructure. Each use case configures its own moderation agent without writing code.
The platform uses different API keys for external models to trace billing. This allows the team to attribute costs to specific use cases. Clients can define input and output schemas for prompts, giving them control over how the LLM is invoked. Fallback and retry strategies are declared in configuration, not code.
The LLM gateway is a separate component, built by other DoorDash folks. The moderation platform is a client of this gateway. The gateway handles integration with various vendors and manages fallback logic. This separation keeps the moderation platform focused on its core job.
Pereira noted that it is cheap to create code but not cheap to maintain code. This insight drove the decision to build a configurable platform rather than a series of bespoke systems. Each bespoke system would require ongoing maintenance, model updates, and incident response. A shared platform amortizes those costs across all use cases.
The platform's backtesting feature is a key differentiator. Teams can test agents against historical data before deploying them. This reduces the risk of shipping a broken moderation pipeline. The human review process ensures that the metrics reflect real-world performance, not just model confidence.
The platform's reach extends beyond moderation. Fraud detection in chat and phone calls uses the same infrastructure. This shows that the pattern of a cheap filter, a smart judge, and graduated action applies to more than content moderation. Any classification task with a clear action hierarchy can benefit.
The billing traceability is important for cost management. Different teams have different budgets. By using separate API keys, the platform can attribute costs accurately. This transparency helps teams understand their spending and make informed decisions.
The LLM gateway's fallback and retry strategies are declared in configuration. This means teams do not need to write code to handle vendor outages. The gateway automatically retries or falls back to another vendor. This resilience is built into the platform.
Looking Ahead
The talk did not speculate on future features, but the trajectory is clear. The platform is built to handle new use cases as they arise. Any team at DoorDash that needs moderation can configure an agent and deploy it. The infrastructure is in place.
Pereira's presentation at QCon AI highlighted a pragmatic approach to AI-powered moderation. Start with data understanding, build a cheap filter, use an LLM only for hard cases, and design for flexibility. The result was a 50% reduction in verbal abuse incidents and a platform that serves the entire company.
The 50% reduction is a headline number, but the architecture behind it matters more. The cheap model, the LLM scorer, and the graduated actions form a pattern that generalizes. Other teams at DoorDash recognized this and asked for the pattern, not the system. The team listened and built accordingly.
For engineers building moderation systems, the takeaways are concrete. Measure the data before building models. Use a cheap filter to handle the easy cases. Ask the LLM for scores, not Booleans. Apply graduated actions based on severity. And be willing to throw away a working system if it does not scale to new use cases.
The SafeChat system was thrown away, but its DNA lives on in the platform. The learnings, the model, and the data all survived. The platform is the next iteration, built to handle whatever moderation challenges DoorDash faces next.
The platform's configurable nature means it can adapt to new requirements. If a new type of abuse emerges, teams can add a new axis or adjust thresholds. If a new vendor offers a better model, teams can switch without rewriting their pipelines. This flexibility is the platform's greatest strength.
Pereira's talk was a case study in practical AI engineering. She did not present theoretical frameworks or speculative visions. She showed a working system, its measurable impact, and the lessons learned. This grounded approach is valuable for any engineer facing similar challenges.
The future of the platform will likely involve more use cases and more sophisticated configurations. The foundation is solid. The pattern is proven. The infrastructure is in place. DoorDash's moderation challenges will evolve, and the platform is ready to evolve with them.