Discord
FreeScaling message storage to trillions
FreeFree tier
About Discord
Discord is a communication platform designed for creating communities, with a focus on voice, video, and text chat. This article details the engineering behind Discord's message storage system, which evolved from MongoDB to Cassandra and later to a custom solution to handle trillions of messages. The post highlights challenges with Cassandra such as hot partitions, compaction backlogs, and latency spikes, and discusses the architectural changes made to improve scalability and reliability.
Key Features
Scalable message storage handling trillions of messages
Migration from MongoDB to Cassandra for better scalability and fault tolerance
Custom storage architecture to overcome Cassandra limitations
Partitioning by channel and time bucket for efficient queries
Snowflake IDs for chronological ordering
Quorum consistency for read/write operations
Pros & Cons
Pros
- Handles massive scale (trillions of messages across 177 nodes)
- Fault-tolerant distributed database design
- Chronologically sortable message IDs enable efficient queries
Cons
- Cassandra cluster exhibited hot partitions causing latency spikes
- High operational toil with frequent paging due to performance issues
- Compaction and garbage collection overhead impacted latency
- Uneven message distribution between small and large servers challenges performance
Best For
Storing and retrieving chat messages in large communitiesHigh-throughput messaging for servers with hundreds of thousands of usersHandling uneven message distribution across channels
FAQ
How did Discord initially store messages?
Discord initially used MongoDB, then migrated to Cassandra for better scalability and fault tolerance.
What caused performance issues in Discord's Cassandra cluster?
Hot partitions from high-traffic channels, compaction backlogs, and JVM garbage collection pauses led to unpredictable latency and operational challenges.
How does Discord partition messages?
Messages are partitioned by channel and a static time bucket, with Snowflake IDs ensuring chronological ordering.
What is quorum consistency?
Quorum consistency means reads and writes require agreement from a majority of replicas, ensuring data accuracy but also affecting latency when nodes are overloaded.