Gang Scheduling (Co-Scheduling)
In plain English
Plain definition
Gang scheduling, also called co-scheduling, reserves enough resources for every related task before starting the group. It is useful for tightly coupled HPC and machine-learning jobs, but poor queue design can cause head-of-line blocking, fragmented capacity and long waits when one required placement cannot be satisfied.
Answer first: Gang scheduling admits a tightly coupled group only when at least its required member count can be placed together. On current Kubernetes, native PodGroup gang scheduling is alpha and disabled by default, so verify the cluster version, API version, feature gates, workload controller, and scheduler name before choosing it over Volcano, YuniKorn, or another implementation.
Primary references: Kubernetes PodGroup scheduling and Kubernetes v1.36 workload-aware scheduling. Related KBY concept: bin packing.
Kubernetes ordinarily evaluates Pods sequentially. Kubernetes v1.35 introduced native alpha PodGroup scheduling behind the disabled-by-default GenericWorkload feature gate, and v1.36 expanded workload-aware scheduling; clusters without those gates or an external gang-aware scheduler still bind Pods independently. For tightly-coupled parallel workloads (MPI collectives, parameter-server or all-reduce based ML training, Spark executors requiring a minimum quorum), this creates a classic resource deadlock scenario: pod A gets scheduled and starts consuming its allocation while waiting on a barrier or rendezvous with pod B, but pod B is stuck in the pending queue because a competing job has consumed the remaining capacity. The cluster ends up with resources locked by half-started jobs that can make no forward progress, while other jobs are also partially admitted, leading to a convoy of mutually blocking allocations.
Gang scheduling solves this by treating the job’s task set as an atomic scheduling unit — a PodGroup in schedulers like Volcano and Apache YuniKorn, or a PGID concept in Slurm/PBS batch systems. The scheduler evaluates whether the aggregate resource requirement (CPU, memory, GPU, RDMA NICs) for all members of the gang can be satisfied simultaneously across the cluster before committing any bindings. If sufficient capacity is not currently available, none of the pods are bound; they remain collectively pending until the scheduler can perform the full atomic placement. This requires the scheduler to reason about a `minMember` or quorum threshold, since some frameworks tolerate partial gangs (e.g., elastic training with a minimum worker count) rather than requiring strict 100% co-location.
Implementation details matter significantly at scale. Naive gang scheduling can itself induce scheduler-level head-of-line blocking: a large gang waiting for capacity can stall smaller, otherwise schedulable jobs behind it in the queue, so production implementations pair gang scheduling with backfill algorithms or preemption policies (e.g., Volcano’s preempt action, YuniKorn’s hierarchical queue fairness) to avoid cluster-wide throughput collapse. Another critical edge case is partial failure during steady-state — if one gang member is evicted (node pressure eviction, spot instance reclamation) after the job is running, most gang-aware controllers will tear down and reschedule the entire gang rather than leave a crippled partial job consuming resources indefinitely, since the remaining members are typically blocked on collective communication primitives (NCCL all-reduce, MPI barriers) that hang rather than fail cleanly. This tear-down/requeue behavior must be coordinated with the job controller (e.g., Kubeflow’s PyTorchJob or MPIJob operator) to avoid split-brain state between the scheduler’s view of the gang and the workload controller’s view of task lifecycle.
Gang scheduling is fundamentally an admission-control and bin-packing problem layered on top of the base scheduler, and it interacts poorly with default Kubernetes primitives that assume independent pod lifecycles — PodDisruptionBudgets, HPA, and even basic liveness probes need to be gang-aware or explicitly disabled for these workloads. Engineers building ML platforms or HPC-on-Kubernetes systems should treat gang scheduling not as a scheduler feature flag but as a cluster capacity-planning discipline: without adequate slack capacity or dedicated node pools, gang-scheduled jobs will starve indefinitely under a first-come-first-served default scheduler, and the correct mitigation is queue-level admission control combined with preemption, not merely enabling co-scheduling.