Red Hat’s Quay testing finds storage and client bottlenecks before registry limits

Red Hat engineers testing container-image pulls against Red Hat Quay found that storage, the load generator and OpenShift node limits became bottlenecks before the registry application itself. The work provides a repeatable way to size Quay deployments, but stops short of claiming a universal capacity number.

A customer problem became a reference test

The study began with an enterprise deployment spanning 160 nodes across two edge clusters that was seeing image-pull failures under load. Red Hat built a reference environment on self-managed OpenShift in AWS, using two Quay replicas, external PostgreSQL and Redis, and NooBaa object storage on OpenShift Data Foundation.

The engineers used the open-source quay-performance-scripts framework to generate pulls of 171 MB images with 100 layers. They monitored Quay, PostgreSQL, NooBaa, Redis and HAProxy with Prometheus-backed dashboards.

Early tests appeared to show little benefit from adding Quay CPU: moving from 2 to 8 vCPU per pod increased reported throughput only from 0.17 to 0.22 images per second. Part of that result came from the test harness averaging 300-second timeouts with successful pulls; the team corrected the framework to calculate durations from successful pulls.

Storage scaling changed the result

NooBaa endpoints saturated before the Quay pods did. In an 8-vCPU test at 500 concurrent pulls, four NooBaa endpoints completed 245 of 500 pulls at 0.22 images per second. Increasing the endpoint count to six completed all 500 pulls at 0.52 images per second, more than doubling throughput without scaling Quay.

That finding makes the storage back end part of Quay capacity planning rather than an interchangeable dependency. Red Hat cautioned that deployments using S3, Google Cloud Storage, Ceph or another supported back end will produce different limits.

The client could not keep up

A later 16-vCPU test collapsed at 400 concurrent pulls even though an 8-vCPU configuration had completed 750. Packet captures showed TCP zero-window events: the single Python load-generator pod could not consume data as quickly as Quay served it. Dividing the same 400-pull load across four client pods restored throughput to about 1.5 images per second, a 21-fold improvement over the single-client result.

At still higher concurrency, the team also hit OpenShift’s kubelet podPidsLimit of 16,384 and raised it to 65,536 for the test. Long-running tests exposed expiring authentication tokens as another source of false failure; the framework now refreshes tokens after HTTP 401 responses.

A baseline, not a guarantee

For the tested configuration, Red Hat recommended maximum concurrency ranging from 25 pulls with 2 vCPU per Quay pod to 750 with 8 vCPU and six NooBaa endpoints. The published 16-vCPU figure of 300 carries an explicit warning that it reflects the single-client limit rather than Quay capacity.

The useful result is the method: document the complete configuration, observe every component and remove bottlenecks one layer at a time. Red Hat said it does not currently publish official Quay capacity-planning guidance, so these figures should be treated as deployment-specific baselines rather than product limits.