Notes for Scalability, Availability & Stability Patterns
A must-read for Software Engineers: “Scalability, Availability & Stability Patterns” by Jonas Boner
Availability Patterns
Failover
active-passive: one system is active, while the other remains idle until the active one fails. Cost effective but standby takes time to warm up
active-active: both systems run simultaneously, sharing the load and providing redundancy. Quick switch over but higher cost.
Replication
Replication Modes: Push (master pushes data replicas) vs Pull (replicas periodically pull data from master)
Replication Patterns: master-slave (one master acts as writer, slaves act as reader) vs master-master (multi-master replications)
Scalability Patterns — State
Partitioning
horizontal partitioning (divide a table by row) vs vertical partitioning (divide a table by column)
HTTP Caching
first request is slower but subsequent requests are faster using Reverse Proxy, CDN
Sharding
divides a table into smaller segments (by rows or columns) across different nodes (partitioning) and maintains multiple copies (replication) for redundancy and availability.
NoSQL
- key-value databases (like in-memory hash table)
- column databases (for read-heavy analytical workloads)
- document databases (for flexible documents with id)
- graph databases (for relationship data)
Distributed Caching
cache-aside: client manages the logic of cache check, DB check, update cache. Simple but a hit miss involves 3 round trips.
write-through: client read/write cache and cache updates database as one atomic operation. Strong consistent but slow overall
write-behind: client read/write cache, then update database asynchronously. Fast but may lose data.
refresh-ahead: warm up cache upfront. Must know what to warm up.
eviction policies: TTL (time to live), FIFO (first in first out), LIFO (last in first out), explicit cache invalidation
Concurrency
shared-state: everyone can access anything anytime with lock
message-passing: share nothing, communicate through messages
dataflow: immutable data like functional programming but for data
software transactional memory: like database transactions with begin > commit > rollback
Scalability Patterns — Behaviour
Event-Driven Architecture (EDA)
Event Sourcing, CQRS, PubSub, Request-Reply
Load-Balancing
Load balancing algorithm
- Random allocation
- Round robin allocation
- Weighted allocation
- Dynamic allocation (least connection, more overhead)
Load balancer Tools
- DNS: round robin or weighted
- Reverse Proxy: Nginx, HAProxy
- Hardware Load Balancer (costly)
Parallel Computing
Unit of Execution (UE): process for better isolation, thread for sharing memory, abstractions of concurrency for different runtimes
Stability Patterns
Timeouts
Always use timeouts. If many concurrent requests are waiting for slow responses, the system can quickly run out of these resources, leading to degraded performance or even crashes.
Circuit Breaker
To prevent an application from a downstream API that is highly likely to fail. Circuit Breaker Pattern
Let it crash
Instead of preventing it, manage it via process supervision like PM2 for NodeJS
Fail fast
Verify resource availability before starting. Verify user input before processing. Test before deploying. It’s aligned with shift-left concept
Bulkheads
It’s about isolation. Separate an application into multiple micro services. Bulkheads Pattern
Steady State
The application behaves as normal: logging, monitoring, alerting, etc.
Throttling
Put limits to maintain a steady pace. Limit the maximum resource one can use. Avoid bursts (like DDoS). Throttling Pattern, Rate Limiting Pattern
Consistency Patterns
Strong Consistency
Data is replicated synchronously, hence slower write but read will return the latest write.
Eventual Consistency
Data is replicated asynchronously (eventually) hence faster write but read may not always get the latest write. Normally changes will propagate through the system in due time.
Read-your-writes Consistency
See their own updates
Trade-Offs
Performance vs Scalability
If your system is slow for a single user, it’s a performance issue. If your system is fast for a single user but slow under heavy load, it’s scalability issue.
Latency vs Throughput
You should maximise throughput with acceptable latency
Availability vs Consistency
It is the CAP Theorem — at a given point in time, you can only pick 2 out of Consistency (data are consistent), Availability (always up), Partition tolerance (cluster instead of single node).
In a centralised system, we have CA and we don’t have P. For example, a relational database on a single server. We rarely see it in the real world.
In distributed system, we have P and
- If we pick AP, sacrifice Consistency, meaning during node failures, different nodes may have inconsistent data
- If we pick CP, sacrifice Availability, meaning during node failures, the system may reject some operations
A quick recap
From Google I/O 2009, x-axis is how distributed your system is, from single node with backups, master-slave, master-master, two-phase commit, a full distributed consensus protocol. y-axis is your trade-offs.
