Red Hat turns Krkn experiments into repeatable resilience gates

Red Hat has published a four-stage method for turning Krkn chaos experiments into repeatable resilience checks for Kubernetes and OpenShift. The blueprint covers strategy, controlled execution, exploratory testing and eventual CI integration.

Begin with a service outcome

The method starts with a critical user journey rather than a fault injector. Application, platform and operations teams map its compute, network, storage and service dependencies, then convert service-level objectives into measurable test criteria for request success, latency, throughput, data correctness and recovery time.

A useful hypothesis links one failure to one expected outcome. Red Hat's example asks whether transactions continue and replicas recover within 60 seconds after an API pod fails. The exact threshold belongs to the service; a short experiment does not establish monthly availability compliance.

Test recovery, not object state

The second stage injects a small, controlled failure under representative load. Teams prepare a healthy baseline, disrupt the target, measure degradation and recovery, restore the environment and verify cleanup before another run.

The important warning is that a Ready pod does not prove the service recovered. Transactions can still fail when connections to dependencies have not re-established. OpenShift Virtualization tests similarly need to check the guest application alongside virtual-machine and host health.

Krkn can evaluate Prometheus-based SLO checks, but the blueprint recommends adding application checks for transaction correctness. Test setup, targets, criteria and results should be versioned so teams can reproduce failures and distinguish intermittent behavior from fixes.

Explore, then stabilize

The third stage uses Krkn-AI's evolutionary search to explore combinations such as latency, packet loss, storage disruption, resource pressure and worker loss under load. Findings should retain configurations, parameters and seeds where supported. Once a weakness is fixed, the experiment moves out of exploration and into a saved regression workflow.

The final stage uses Krkn Operator and Chaos Studio to build replayable sequences, attach PromQL checks and compare results. Stable workflows can begin as reporting-only CI jobs, then become blocking gates after teams have verified that missed faults, absent measurements and incomplete scenarios fail correctly.

Krkn's beta resilience score weights critical and warning SLO checks, but Red Hat advises using it alongside mandatory service, data-integrity and recovery checks. A score describes only the tested conditions; it is not evidence that every outage mode is covered.

Put chaos before the change

The blueprint's most practical use is pre-flight validation. Run a known baseline, apply an OpenShift upgrade, application release, tuning change or infrastructure modification in staging, then replay the same workflow under comparable load. A change that passes an aggregate score but violates a recovery-time target should still be blocked.

That discipline turns chaos engineering from a demonstration into change evidence: one critical service, one controlled failure, explicit success criteria, a verified fix and a workflow that can be run again.