API Priority and Fairness (APF)

APF sits inside the kube-apiserver request-handling chain, positioned after authentication/authorization but before the request reaches its handler. Every incoming request is classified by a FlowSchema into a PriorityLevelConfiguration (e.g. system, leader-election, workload-high, catch-all). Each priority level owns a slice of the API server’s total concurrency budget, expressed in seats rather than raw request counts — a seat roughly corresponds to a unit of estimated cost, with list/watch requests weighted heavier than simple gets. Within a priority level, requests are further split into flows via a distinguisher (typically the requesting user or namespace), and a shuffle-sharded set of FIFO queues implements a fair-queuing algorithm so a single noisy tenant cannot monopolize the level’s seats even if it floods requests.

The core algorithm is a variant of Fair Queuing with virtual finish times, borrowed conceptually from network packet scheduling. Each queue tracks a virtual start/finish time; when concurrency frees up, the scheduler dequeues from the queue with the earliest virtual finish time, approximating max-min fairness across flows without requiring per-flow rate limiting configuration. Requests that cannot be admitted within a bounded queue-wait time (configurable per priority level) are rejected with HTTP 429 and a Retry-After header rather than blocking indefinitely, which is what allows APF to function as effective admission control under overload rather than just a scheduling nicety.

• Seat estimation drift: list requests against large collections can be underestimated at admission time, causing actual memory/CPU cost to exceed the seats reserved, which is why APF pairs with --max-requests-inflight and watch-cost heuristics tuned per cluster size.
• Priority inversion via misclassification: a poorly written FlowSchema (e.g. matching on a wildcard subject) can route a bursty controller into the workload-high level and starve legitimate high-priority traffic that shares the level.
• Cascading queue rejection: during a control-plane incident, a client that retries 429s aggressively without honoring Retry-After and exponential backoff amplifies load precisely on the subsystem meant to shed it.
• Observability gap: without scraping apiserver_flowcontrol_* metrics, operators often diagnose APF-induced 429s as generic API server unavailability rather than a fairness/queue-depth problem.

Architecturally, APF matters because it decouples the API server’s stability guarantees from any single client’s behavior pattern, which is essential once a cluster hosts dozens of controllers, operators, and CI pipelines all calling the same control plane. Getting FlowSchema and PriorityLevelConfiguration tuning wrong is a common source of mysterious 429 storms in large multi-tenant clusters, making APF metrics a mandatory part of control-plane SLO dashboards rather than an optional enhancement.