Going Global: Distributed Systems and Real-Time Data
Part 4 of 5 in the System Design Series
In October 2021, Facebook went down for six hours.
Not just the app, Facebook, Instagram, and WhatsApp all went dark simultaneously. Three billion people lost access. Facebook's own employees couldn't badge into buildings because the access control systems relied on the same infrastructure. Internal tools to diagnose the problem were also down.
The cause? A configuration update that accidentally severed all communication between Facebook's data centers. The entire global system lost the ability to coordinate.
This is the nature of distributed systems. When things are spread across the world, they fail in ways that a single server never could. Let's talk about how engineers think about this.
The CAP Theorem: An Unavoidable Tradeoff
When you build a system that spans multiple servers, or multiple data centers, you run into a fundamental constraint first described by computer scientist Eric Brewer. It's called the CAP Theorem, and it states that a distributed system can only guarantee two of these three properties at once:
- Consistency, Every read returns the most recent data. Every node in the system sees the same thing at the same time.
- Availability, The system always responds to requests, even if some nodes are down.
- Partition Tolerance, The system keeps working even when network failures prevent some nodes from communicating with others.
That means the real decision is between Consistency and Availability:
- Prefer Consistency (like a bank), If two servers can't agree on the current account balance, the system refuses to answer rather than risk showing the wrong number.
- Prefer Availability (like a social media platform), If two servers can't sync, you still see a result, maybe your follower count is off by a few for a moment, but the app keeps working.
Storing Files: Blob Storage
Most databases are optimised for structured data, rows and columns, numbers and strings. They're terrible at storing a 4K video file or a high-resolution image.
For that, engineers use blob storage, services like Amazon S3 or Google Cloud Storage. "Blob" stands for Binary Large Object: essentially any file that doesn't fit neatly into a database table.
Files get uploaded to cloud storage buckets and are assigned unique URLs. Your database doesn't store the video, it stores the link to the video. When a user requests it, your app hands them that URL and the storage service delivers the file directly.
This keeps your database lean and fast, while the storage layer handles the heavy lifting of serving large files reliably.
CDNs: Solving the Speed of Light Problem
But here's the issue with storing files in one location: if your storage bucket lives in a data center in Virginia, a user in Lagos has to wait for data to travel across the Atlantic. Every single time.
That latency adds up, especially for media-heavy applications. A Nigerian student loading a YouTube lecture shouldn't have to wait for bytes to make a transatlantic roundtrip.
This is what Content Delivery Networks (CDNs) are for. A CDN is a global network of servers, called edge servers, distributed across dozens or hundreds of locations worldwide. When you upload your video to a CDN-backed platform, copies are cached at edge servers in Lagos, Nairobi, London, São Paulo, and everywhere else.
When that student hits play, they're pulling the video from the nearest edge server, maybe a few hundred kilometres away instead of thousands. The experience is dramatically faster.
Cloudflare, Akamai, and AWS CloudFront are all CDNs. Any application serving media or static assets to a global audience needs one.
Real-Time Data: Beyond the Request-Response Cycle
Standard HTTP has a fundamental limitation: the client asks, the server answers. That's it. The server can't volunteer information, it can only respond when asked.
For most applications, this is fine. But consider a live chat app, a stock ticker, or a multiplayer game. In these contexts, you can't wait for the user to refresh to see new data.
One naïve solution is polling, the client sends a request every few seconds: "Anything new?" This works, but it's wasteful. The vast majority of those requests come back empty, and you're hammering your server with constant traffic just to detect the occasional update.
The better solution is WebSockets.
A WebSocket creates a persistent, two-way connection between the client and the server. Once established, either side can push data to the other at any time, no request needed. When a new chat message arrives, the server pushes it to every connected client instantly. The connection stays open until one side closes it.
This is how WhatsApp, Slack, and Discord deliver messages in real time. It's how live sports scores update without you refreshing. It's how multiplayer games keep every player's screen in sync.
Webhooks: When Servers Need to Notify Each Other
WebSockets solve real-time communication between a server and a user's browser. But what about server-to-server communication?
Say you're building a payment integration with Flutterwave. Your user clicks "Pay" on your site. Flutterwave processes the payment. You need to know whether it succeeded.
You could poll Flutterwave's API every few seconds: "Did it go through yet?" But that's slow and wasteful.
Instead, Flutterwave uses a webhook. You give them a URL on your server. When the payment completes, whether that's 2 seconds or 20 seconds later, Flutterwave sends an HTTP POST request to your URL with the result. Your server receives it, updates the order status, and moves on.
Webhooks are essentially the inverse of an API call: instead of you asking for data, the other system pushes data to you the moment something happens.
Facebook's 2021 outage, YouTube buffering in Lagos, WhatsApp messages arriving instantly, these are all manifestations of the same engineering challenges. Distributed systems are inherently complex, and the techniques covered in this post are how engineers manage that complexity.
In Part 5, the final post in this series, we'll look at how modern applications are architected, the shift from monolithic codebases to microservices, and the infrastructure that ties them together.
Previous: Part 3 → Managing the Data: Databases and Optimisation Techniques Next: Part 5 → Modern Architecture: Microservices to API Gateways