In modern distributed systems engineering, load testing has evolved far beyond the basic exercise of firing HTTP requests at a server until it crashes. Today, it is an empirical discipline of subjecting a system to anticipated peak operational volumes to observe its behavior, identify latency bottlenecks, validate capacity planning, and ensure graceful degradation under stress. A mature load testing strategy doesn't simply ask, "Can the system handle 10,000 requests per second?" Instead, it seeks to answer critical architectural questions: How does the system behave when it hits its absolute limits? Does it shed load cleanly, or does it collapse into a cascading failure? And crucially, how quickly does it recover once the traffic spike subsides?
This guide dives deeply into the mechanics of load testing, covering the mathematics of workload models, the hidden dangers of metric collection, the art of interpreting percentiles, the integration of continuous profiling, and the structural implementation of performance testing within CI/CD pipelines.
A fundamental, yet incredibly common error in performance engineering is selecting the wrong workload model for the system under test. The workload model defines how virtual users are generated and how they interact with the target system.
In a closed workload system, the load generator maintains a strictly fixed number of concurrent virtual users (often implemented as threads). Each virtual user executes a request, waits for the response, and only then executes the next request. This creates a synchronous feedback loop between the system under test and the load generator.
The primary trap of a closed model is that it naturally throttles itself. If your database experiences a sudden lock contention issue and query times increase from 10 milliseconds to 500 milliseconds, the virtual users will spend more time waiting and less time sending new requests. The load on your system drops precisely at the moment it begins to struggle. This artificially caps concurrency and heavily masks the true failure point of the system. While closed models are entirely appropriate for testing systems where users genuinely wait in a queue (such as a call center or a database connection pool), they are dangerously misleading when applied to public internet traffic.
In an open workload system, requests arrive at a predefined arrival rate (for example, a strict 500 requests per second), regardless of how fast or slow the server processes them. New requests are generated independently of the completion of previous requests.
This model accurately simulates the chaotic, uncoordinated nature of internet traffic. If your server slows down, the open workload generator does not care—it continues to fire requests at the predefined rate. Consequently, requests queue up at the server level, realistically exposing thread pool exhaustion, memory leaks, and the exact threshold where the system tips into failure. For any public-facing web service or API, utilizing an open workload model is considered a non-negotiable best practice.
When engineering teams run a load test, they typically look at a generated report and celebrate a 99th percentile (p99) latency of 50 milliseconds. However, if they used a closed workload model, they are likely falling victim to a mathematical illusion known as Coordinated Omission.
Coordinated omission, a term heavily popularized by Gil Tene, occurs when a load testing tool inadvertently coordinates with the system under test to completely omit latency spikes from its measurements. Imagine a scenario where a test tool expects to send a request every 10 milliseconds. Suddenly, the server experiences a "Stop the World" Garbage Collection pause that freezes the application for 100 milliseconds. Because the test tool is waiting for a response before sending the next request, it sits idle. It completely skips sending the 9 requests that should have occurred during that 100-millisecond window.
When the server wakes up and responds, the tool records one slow request (100ms) and then immediately resumes its fast pace. The 9 requests that would have sat in a queue and experienced massive delays were never sent, and therefore, never measured. The resulting report completely hides the severity of the pause, presenting a falsely optimistic latency distribution.
To mitigate coordinated omission, Site Reliability Engineers (SREs) must utilize modern testing tools specifically engineered to correct for it (such as wrk2 or specifically configured open-model tools) that account for queueing time. Furthermore, latency should always be measured and verified from the server-side metrics (via Prometheus, Datadog, or OpenTelemetry) rather than blindly trusting the aggregates generated by the client-side load testing tool.
When analyzing load test results, looking at the average (mean) response time is an anti-pattern. Averages are mathematically misleading because they are easily skewed by a handful of extreme outliers or entirely masked by a high volume of fast, simple requests.
Instead of averages, performance engineers must focus on percentiles, which describe the value at or below which a specific percentage of requests fall.
In a monolithic architecture, a p99 latency spike only affects 1% of your users. However, in a modern microservices architecture, a single user request to a frontend gateway might "fan out" into 50 internal microservice calls. If even one of those internal services has a poor p99 latency, the mathematical probability of the user experiencing a delay skyrockets. In highly distributed systems, tail latency essentially becomes median latency from the user's perspective. Controlling the p99 is not about optimizing for an edge case; it is about ensuring basic stability for the majority of your traffic.
A standard load test is excellent at telling you that your system is failing, but it is notoriously bad at telling you why. When p99 latency spikes during a test, engineers are often left guessing whether the culprit is CPU throttling, network latency, database lock contention, or inefficient application code.
This is where Continuous Profiling bridges the gap. Modern profiling tools (such as Pyroscope, PolarSignals, or Datadog Continuous Profiler) run in the background with negligible overhead, continuously capturing CPU cycles, memory allocations, and thread blocking states.
When executing a load test, the best practice is to annotate your profiling data with specific test identifiers. If you observe a massive spike in p95 latency at minute 15 of your test, you can seamlessly pivot to your continuous profiling dashboard, filter for that exact timestamp, and view a flame graph showing the precise lines of application code that were consuming CPU at that moment. This transforms performance tuning from a reactive, guessing-game methodology into a proactive, evidence-based engineering discipline.
Historically, load testing was a localized event performed in the final staging phase before a major release. This approach is fundamentally flawed because it catches architectural performance regressions far too late in the development cycle, making them incredibly expensive to fix. The modern standard is to "shift-left," integrating performance validation directly into the Continuous Integration and Continuous Deployment (CI/CD) pipeline.
Integrating load tests into CI/CD is fraught with challenges, primarily the risk of creating "flaky" pipelines where tests randomly fail due to noisy network neighbors rather than actual code regressions. To implement this successfully:
main branch.Modern performance testing is treated as "tests as code," version-controlled and peer-reviewed alongside application logic. Developers should be deeply familiar with the nuances of these industry-standard tools:
wrk2 fork is particularly notable as it was explicitly built to solve the coordinated omission problem by utilizing a constant throughput load generation model.