Concurrency on a single machine has well-understood primitives — locks, channels, atomics. Concurrency across machines is harder because you can't trust your peers, the network isn't reliable, and there's no shared memory.
The patterns are different. The failure modes are worse. This page is the working set for getting it right.
Single-machine concurrency primitives assume:
None of these hold across machines. Network packets are lost; clocks drift; "is that machine slow or dead?" is undecidable in finite time; delays are unbounded.
This is why distributed concurrency requires different patterns.
"Only one process holds this resource at a time, even across machines."
def acquire_lock(redis, key, value, timeout):
return redis.set(key, value, nx=True, ex=timeout)
def release_lock(redis, key, value):
# Lua script that releases only if value matches
return redis.eval(release_script, 1, key, value)
Cheap; works for "we'd prefer not to run two of these at once."
Pitfalls (Martin Kleppmann's analysis, 2017):
Mitigations:
For "best-effort" locks (cron singletons, rate-limit-related coordination): Redis is fine. For correctness-critical locks (financial transactions, irreversible operations): use etcd or ZooKeeper.
etcd and ZooKeeper both provide strongly-consistent locks via consensus protocols (Raft / Zab).
etcd lease + lock pattern:
session, err := concurrency.NewSession(client, concurrency.WithTTL(10))
mutex := concurrency.NewMutex(session, "/my-lock")
mutex.Lock(context.Background())
defer mutex.Unlock(context.Background())
// ... critical section
The session gives a lease; the lock is held while the lease is alive; if the holder dies, the lease eventually expires and the lock releases.
For irreversible operations (charging cards, sending shipments, entering data into systems of record), this is the pattern.
"Exactly one node is the leader at any time."
Use cases:
Approach:
Most "we need a leader" problems are solved by deploying etcd / Consul and using their primitives. Don't roll your own.
"Increment a number; multiple machines may increment concurrently; never miss an increment; never count one twice."
Approach:
UPDATE counters SET value = value + 1. Postgres transactions handle this; rate limited by row contention at high write rates.For high-volume metrics, exact counters bottleneck; approximate or sharded designs are necessary.
The defence against retry-induced double-effects.
Pattern:
def charge(idempotency_key, amount):
# Has this idempotency key been used?
existing = lookup(idempotency_key)
if existing:
return existing # return previous result
# Atomically: charge, record key+result.
with transaction:
result = charge_impl(amount)
record(idempotency_key, result)
return result
Idempotency requires:
For any retried mutation, this is non-negotiable. See SagaPattern.
Two-phase commit (2PC): a coordinator polls participants for vote; if all yes, commit; if any no, abort.
Limitations:
In modern distributed systems, 2PC is rare for cross-service work. Sagas (compensating transactions) are preferred. See SagaPattern, DistributedComputingAlgorithms.
For within-database distributed transactions (one Postgres cluster across nodes), the database handles this internally — Postgres uses 2PC for cross-shard with partition managers.
In SQL: optimistic via WHERE version = $expected_version on UPDATE.
In application code: use atomic CAS where available (Redis, etcd compare-and-swap); use database row locks (SELECT FOR UPDATE) where strong serialisation is needed.
For most application-level concurrency: optimistic with retry on conflict. Conflicts are rare; the retry cost is low.
See ApiRateLimitingAlgorithms. The interesting concurrency aspect: counters per-user shared across N application instances. Centralised counter (Redis) is the simplest approach. Decentralised approaches gain throughput at the cost of approximation.
A core tradeoff:
For most workloads, eventual consistency is fine — caches, social timelines, profile pictures. For specific subsystems (account balances, inventory at zero, identity), strong consistency is necessary.
Most modern distributed databases let you choose per-operation: strong reads vs eventual reads. Use strong where it matters; eventual where it doesn't. The default to "everything strong" is pessimistic over-engineering.
The opposite of locks: design so coordination isn't required.
When you can design coordination out, you scale better. The hard part: many problems don't fit these patterns naturally.
| Need | Substrate |
|---|---|
| Best-effort lock | Redis SET NX EX |
| Correctness-critical lock | etcd / ZooKeeper |
| Leader election | etcd / Consul / Raft library |
| Atomic counter | Redis INCR / SQL UPDATE |
| Distributed semaphore | etcd / Redis |
| Idempotency | App-level table |
| Cross-service transaction | Saga (compensating transactions) |
| Eventual consistency | CRDTs / quorum reads/writes |
| Strong consistency | Consensus (Spanner, CockroachDB, etc.) |
For most production teams: a Redis instance and a Postgres database cover most needs. Add etcd / Consul when correctness-critical coordination is required.
Split brain. Network partition; both sides think they're "the active one." Defended by quorum (majority must agree).
Phantom locks. Lock holder dies without releasing; lock TTL is the safety. Choose TTLs carefully — too short and active holders lose their lock; too long and crashed holders block work.
Clock skew. Two nodes' clocks differ; lease expiries differ; surprises. Use logical clocks where possible.
Cascading retries. Service A fails; B retries; A's downstream gets retry storm. Add jitter, circuit breakers, exponential backoff.
Thundering herd. Cache key expires; 1000 requests miss simultaneously; all hit the database. Single-flight pattern (one fetcher; others wait); stale-while-revalidate; jittered TTLs.
For most distributed services in 2026:
This stack handles 90% of distributed concurrency needs. The remaining 10% require deeper understanding of consensus, CRDTs, and the specific algorithms in DistributedComputingAlgorithms.