LeaderWorkerSet 1.0 changes the upgrade calculus for disaggregated AI serving

LeaderWorkerSet has reached 1.0 with changes aimed at making multi-host and disaggregated AI workloads easier to roll out, scale and recover. The version number is not the only reason operators should pay attention: the v1.0.0 release notes identify changed rollout defaults and remove one compatibility path for older DisaggregatedSet objects.

The project is part of the upstream model-serving stack around llm-d. The llm-d 0.10.0 release pinned LeaderWorkerSet 0.11.0 for its tested infrastructure, so llm-d operators should not infer compatibility with LeaderWorkerSet 1.0 merely from the upstream release.

What changed

LeaderWorkerSet 1.0 adds fractional rollout windows for DisaggregatedSet roles. Instead of forcing one role to finish becoming Ready before another advances, rollout progress can be proportional to each role’s replica count while remaining within maxSurge and maxUnavailable budgets. The release also allows scale-up during rolling updates and offers an opt-in policy for scale-down before an update finishes.

DisaggregatedSet can now divide a role into independently scalable sub-roles that share the same workload configuration. The release supports both ordinal and hash group identities for that arrangement. Recovery controls also become more explicit: operators can cap automatic group recreations with maxGroupRestarts and limit retry frequency with restartBackoff.

The controller configuration API graduates to config.lws.x-k8s.io/v1. The earlier v1alpha1 version remains supported but is deprecated. Native gang scheduling remains alpha and opt-in, although 1.0 includes fixes for PodGroup recreation and group-size changes during rollouts.

Who is affected

The sharpest upgrade edge concerns DisaggregatedSet roles that omit rollingUpdateConfiguration. In 1.0, those roles default to maxSurge: 0 and maxUnavailable: 1, matching LeaderWorkerSet behavior. Earlier DisaggregatedSet behavior used maxSurge: 1 and maxUnavailable: 0. Workloads that rely on bringing up a replacement before draining the old revision must set those earlier values explicitly.

There is also a compatibility boundary for DisaggregatedSets created with LeaderWorkerSet 0.9.x. Version 1.0 removes support for the slice-less LeaderWorkerSets generated by that line. The project directs affected operators to install 0.10.x or 0.11.x and complete at least one template rollout before moving to 1.0.

The project changed revision hashing for newly created DisaggregatedSets so that startup policy, network configuration, labels and annotations participate in rollout detection. Existing objects retain the previous hash behavior to avoid an upgrade-triggered rollout.

What to do

Before upgrading, inspect every DisaggregatedSet role for an explicit rollout policy. If availability depends on starting the new revision before removing the old one, set maxSurge: 1 and maxUnavailable: 0 rather than relying on the old implicit behavior.

Operators coming from 0.9.x should stage through 0.10.x or 0.11.x and perform the required template rollout. Teams using llm-d should stay with the dependency versions in their tested llm-d release matrix until the project publishes a combination that includes LeaderWorkerSet 1.0, or validate the newer controller independently in a non-production environment.

Finally, plan the configuration API migration from v1alpha1 to v1. The old API still works in this release, but its deprecation makes the stable version the safer target for configuration management and future upgrades.