The table below maps outcomes -- the results that you may want to achieve in your validation efforts -- to one or more techniques that can be used to accomplish that outcome.
To use the table, either eyeball-browse or search for keywords.
| When I am working on... | I want to get this outcome... | ...so I should consider |
|---|---|---|
| Development | Prove backward compatibility with existing callers and clients | Shadow testing |
| Development; Integration testing | Ensure telemetry is sufficiently detailed and complete to trace and diagnose malfunction in End-to-End testing flows | Distributed Debug challenges ; Orphaned call chain analysis |
| Development | Ensure program logic is correct for a variety of expected, mainline, edge and unexpected inputs | Unit testing; Functional tests; Integration testing |
| Development | Prevent regressions in logical correctness; earlier is better | Unit testing; Functional tests; Integration testing; Rings (each of these are expanding scopes of coverage) |
| Development | Quickly validate mainline correctness of a point of functionality (e.g. single API), manually | Manual smoke testing Tools: postman, powershell, curl |
| Development; Integration testing | Validate that multiple components function together across multiple interfaces in a call chain, incl network hops | Integration testing; End-to-end (End-to-End testing) tests; Segmented end-to-end (End-to-End testing) |
| Development | Prove disaster recoverability – recover from corruption of data | DR drills |
| Development | Find vulnerabilities in service Authentication or Authorization | Scenario (security) |
| Development | Prove correct RBAC and claims interpretation of Authorization code | Scenario (security) |
| Development | Document and/or enforce valid API usage | Unit testing; Functional tests |
| Development | Prove implementation correctness in advance of a dependency or absent a dependency | Unit testing (with mocks); Unit testing (with emulators) |
| Development | Ensure that the user interface is accessible | Accessibility |
| Development | Ensure that users can operate the interface | UI testing (automated) (human usability observation) |
| Development | Prevent regression in user experience | UI automation; End-to-End testing |
| Development | Detect and prevent 'noisy neighbor' phenomena | Load testing |
| Development | Detect availability drops | Synthetic Transaction testing; Outside-in probes |
| Development | Prevent regression in 'composite' scenario use cases / workflows (e.g. an ecommerce system might have many APIs that used together in a sequence perform a "shop-and-buy" scenario) | End-to-End testing; Scenario |
| Development; Operations | Prevent regressions in runtime performance metrics e.g. latency / cost / resource consumption; earlier is better | Rings; Synthetic Transaction testing / Transaction; Rollback Watchdogs |
| Development; Optimization | Compare any given metric between 2 candidate implementations or variations in functionality | Flighting; A/B testing |
| Development; Staging | Prove production system of provisioned capacity meets goals for reliability, availability, resource consumption, performance | Load testing (stress); Spike; Soak; Performance testing |
| Development; Staging | Understand key user experience performance characteristics – latency, chattiness, resiliency to network errors | Load; Performance testing; Scenario (network partitioning) |
| Development; Staging; Operation | Discover melt points (the loads at which failure or maximum tolerable resource consumption occurs) for each individual component in the stack | Squeeze; Load testing (stress) |
| Development; Staging; Operation | Discover overall system melt point (the loads at which the end-to-end system fails) and which component is the weakest link in the whole stack | Squeeze; Load testing (stress) |
| Development; Staging; Operation | Measure capacity limits for given provisioning to predict or satisfy future provisioning needs | Squeeze; Load testing (stress) |
| Development; Staging; Operation | Create / exercise failover runbook | Failover drills |
| Development; Staging; Operation | Prove disaster recoverability – loss of data center (the meteor scenario); measure MTTR | DR drills |
| Development; Staging; Operation | Understand whether observability dashboards are correct, and telemetry is complete; flowing | Trace Validation; Load testing (stress); Scenario; End-to-End testing |
| Development; Staging; Operation | Measure impact of seasonality of traffic | Load testing |
| Development; Staging; Operation | Prove Transaction and alerts correctly notify / take action | Synthetic Transaction testing (negative cases); Load testing |
| Development; Staging; Operation; Optimizing | Understand scalability curve, i.e. how the system consumes resources with load | Load testing (stress); Performance testing |
| Operation; Optimizing | Discover system behavior over long-haul time | Soak |
| Optimizing | Find cost savings opportunities | Squeeze |
| Staging; Operation | Measure impact of failover / scale-out (repartitioning, increasing provisioning) / scale-down | Failover drills; Scale drills |
| Staging; Operation | Create/Exercise runbook for increasing/reducing provisioning | Scale drills |
| Staging; Operation | Measure behavior under rapid changes in traffic | Spike |
| Staging; Optimizing | Discover cost metrics per unit load volume (what factors influence cost at what load points, e.g. cost per million concurrent users) | Load (stress) |
| Development; Operation | Discover points where a system is not resilient to unpredictable yet inevitable failures (network outage, hardware failure, VM host servicing, rack/switch failures, random acts of the Malevolent Divine, solar flares, sharks that eat undersea cable relays, cosmic radiation, power outages, renegade backhoe operators, wolves chewing on junction boxes, …) | Chaos |