Gang Scheduling (Co-Scheduling)

Standard Kubernetes scheduling is task-oriented: the default scheduler binds pods to nodes independently, one at a time, with no awareness of sibling pods belonging to the same logical job. For tightly-coupled parallel workloads (MPI collectives, parameter-server or all-reduce based ML training, Spark executors requiring a minimum quorum), this creates a classic resource deadlock scenario: pod A gets scheduled and starts consuming its allocation while waiting on a barrier or rendezvous with pod B, but pod B is stuck in the pending queue because a competing job has consumed the remaining capacity. The cluster ends up with resources locked by half-started jobs that can make no forward progress, while other jobs are also partially admitted, leading to a convoy of mutually blocking allocations.

Gang scheduling solves this by treating the job’s task set as an atomic scheduling unit — a PodGroup in schedulers like Volcano and Apache YuniKorn, or a PGID concept in Slurm/PBS batch systems. The scheduler evaluates whether the aggregate resource requirement (CPU, memory, GPU, RDMA NICs) for all members of the gang can be satisfied simultaneously across the cluster before committing any bindings. If sufficient capacity is not currently available, none of the pods are bound; they remain collectively pending until the scheduler can perform the full atomic placement. This requires the scheduler to reason about a `minMember` or quorum threshold, since some frameworks tolerate partial gangs (e.g., elastic training with a minimum worker count) rather than requiring strict 100% co-location.

Implementation details matter significantly at scale. Naive gang scheduling can itself induce scheduler-level head-of-line blocking: a large gang waiting for capacity can stall smaller, otherwise schedulable jobs behind it in the queue, so production implementations pair gang scheduling with backfill algorithms or preemption policies (e.g., Volcano’s preempt action, YuniKorn’s hierarchical queue fairness) to avoid cluster-wide throughput collapse. Another critical edge case is partial failure during steady-state — if one gang member is evicted (node pressure eviction, spot instance reclamation) after the job is running, most gang-aware controllers will tear down and reschedule the entire gang rather than leave a crippled partial job consuming resources indefinitely, since the remaining members are typically blocked on collective communication primitives (NCCL all-reduce, MPI barriers) that hang rather than fail cleanly. This tear-down/requeue behavior must be coordinated with the job controller (e.g., Kubeflow’s PyTorchJob or MPIJob operator) to avoid split-brain state between the scheduler’s view of the gang and the workload controller’s view of task lifecycle.

Gang scheduling is fundamentally an admission-control and bin-packing problem layered on top of the base scheduler, and it interacts poorly with default Kubernetes primitives that assume independent pod lifecycles — PodDisruptionBudgets, HPA, and even basic liveness probes need to be gang-aware or explicitly disabled for these workloads. Engineers building ML platforms or HPC-on-Kubernetes systems should treat gang scheduling not as a scheduler feature flag but as a cluster capacity-planning discipline: without adequate slack capacity or dedicated node pools, gang-scheduled jobs will starve indefinitely under a first-come-first-served default scheduler, and the correct mitigation is queue-level admission control combined with preemption, not merely enabling co-scheduling.