Testing Performance And Load

Testing Performance and Load

Use this skill when a system's speed or capacity is in question and the answer needs to be a number someone can act on.

Most performance testing produces numbers nobody uses. The two causes are always the same: a workload that does not resemble reality, and a result reported as an average. A test that hammers one endpoint with a flat 500 virtual users tells you how the system responds to something that will never happen, and a mean response time of 200ms is compatible with one user in twenty waiting four seconds.

When to Use

• a feature has a stated latency or throughput requirement
• "the app feels slow" needs to become a measurement
• a launch, campaign, or migration needs a capacity check
• a performance regression is suspected between two releases
• a performance result exists and needs interpreting
• a CI check should catch obvious regressions before they ship

Operating Principles

• Model the workload before writing the script. Test shape comes from real traffic: which endpoints, in what ratio, with what think time, at what concurrency.
• Percentiles, never averages. Report p50, p95, p99. The mean describes nobody's experience and hides the tail entirely.
• A threshold without an SLO is a guess. Derive the number from what users need or what the business promised, and say which.
• One variable per run. Comparing two runs that differ in load, data volume, and code changes tells you nothing about any of them.
• Environment differences invalidate the number, not just weaken it. A result from a quarter-size environment is a shape, not a capacity.
• Find the bottleneck, do not just report the symptom. "p95 is 3 seconds" is an observation; "p95 is 3 seconds because the product query has no index on category_id" is a finding.

Workflow

Phase 0: Name the question

Different questions need different tests. Pick one per run.

A test that tries to answer all six answers none. If "it is slow" is all you have, start with a load test at expected concurrency and let the result narrow the question.

Phase 1: Model the workload

The phase that decides whether the result means anything. Work through ./resources/workload-model.md.

From real traffic where possible:

• Endpoint mix: the actual ratio, from access logs or analytics. Reads usually outnumber writes by an order of magnitude, and synthetic tests usually get this wrong.
• Concurrency: derived from throughput and think time, not guessed. concurrent users ≈ arrival rate × session duration.
• Think time: real users pause. A test with zero think time is a different system entirely.
• Data distribution: most requests hit a small hot set. A test with uniformly random ids has a cache hit rate nothing like production.
• Session shape: users log in, browse, and act. A test that only calls the checkout endpoint skips the state that makes checkout slow.
• Peak profile: daily peak, weekly peak, and the campaign or event shape if there is one.

When real traffic is unavailable, state the assumptions explicitly and mark the result as assumption-dependent. An assumed workload is workable; an unstated one is not.

Phase 2: Set thresholds from the SLO

Each threshold traces to something: a user need, a documented SLO, a competitor benchmark, or a current baseline you intend to hold.

./resources/thresholds-and-slos.md covers deriving them. The shape:

thresholds: {
  'http_req_duration{name:checkout}': ['p(95)<800', 'p(99)<2000'],
  'http_req_failed': ['rate<0.001'],
  'checkout_completed': ['count>0'],
}

Rules:

• per-endpoint, not global. A global p95 mixes a health check with a report export and describes neither.
• include an error rate threshold. A fast 500 is not a pass, and a load test with no error threshold will report one as success.
• include at least one business-outcome counter. "Requests succeeded" is not the same as "orders were created", and a test that never completes a checkout can look perfect.

Phase 3: Prepare the environment

Record every difference from production before running anything. The differences bound what the result can claim.

• Sizing: instance count, CPU, memory, relative to production
• Data volume: a database with 2,000 rows and one with 4 million behave differently, and the difference is not linear
• Data shape: long-lived accounts, large carts, deep histories
• Caches: cold or warm, and warmed how
• Dependencies: real, stubbed, or rate-limited. A stubbed dependency that responds in 1ms removes the queueing that production has.
• Network: the load generator's location relative to the system
• Autoscaling: enabled or fixed, and with what policy

Then decide where the load generator runs. A generator on the same machine as the system under test competes with it for CPU, and the result measures the contention.

Phase 4: Warm up, then measure

• Warmup: run at low load until JIT compilation, connection pools, and caches settle. Exclude the warmup from the results, and say how long it was.
• Ramp: reach target load over minutes, not instantly, unless the spike is the subject.
• Steady state: long enough for the metric to stabilize. Ten minutes is usually the minimum for anything meaningful.
• Ramp-down: watch for errors during the descent; connection pool problems often surface here.

Monitor the system, not only the client. Client-side response times tell you something is slow; server CPU, memory, database connections, queue depth, and GC pauses tell you why. A run with no server-side observation produces a symptom and no cause.