The Internal Developer Platform – Part 2: Designing the IDP Control Plane with CRDs

Most Internal Developer Platform initiatives stall not because of poor developer portal UX, but because the underlying IDP control plane is architected as a glorified templating engine rather than a proper declarative system. Platform teams that reach for Helm-behind-a-button, Terraform-triggered-by-CI, or bash scripts wrapped in a pipeline inherit every imperative failure mode that Kubernetes was designed to eliminate: partial applies, no drift detection, no idempotency guarantees, and zero self-healing. A production-grade IDP control plane treats developer intent as a first-class Kubernetes API object, extended via Custom Resource Definitions (CRDs), and continuously reconciled by an Operator running a level-triggered control loop. This article covers the exact mechanics: CRD schema modelling, operator scaffolding, reconciliation semantics, and the failure modes you must engineer around before exposing this to several hundred engineers.

#The Case for a Kubernetes-Native IDP Control Plane

Building the IDP control plane directly on the Kubernetes API machinery is not an aesthetic choice, it is a pragmatic reuse of infrastructure you already operate. You inherit etcd as a strongly consistent object store, RBAC for authorisation, admission webhooks for policy enforcement, and audit logging for compliance, all without writing a bespoke database layer. Every consumer of the platform, whether it is a raw kubectl apply, a GitOps reconciler like Argo CD, or a Backstage scaffolder plugin, writes to the same API surface. This mirrors broader architectural patterns found in distributed systems, where a single source of truth with multiple projections is preferred over N systems each holding partial state.

The alternative, a bespoke REST API backed by a relational database that shells out to kubectl or terraform apply, discards all of this for free. You end up re-implementing optimistic concurrency control, watch/notify semantics, and reconciliation, badly, in application code. The IDP control plane pattern avoids that tax entirely.

#Architectural Breakdown: CRDs as the API Contract

A CRD is the schema contract between the developer and the platform. It is not a YAML abstraction layer, it is an OpenAPI v3-validated resource type registered directly into the Kubernetes API server. Field validation, enums, and required properties are enforced server-side before your operator ever sees the object, which eliminates an entire class of “garbage in” reconciliation bugs. Below is a claim-style CRD that models an application environment request:

1apiVersion: apiextensions.k8s.io/v1
2kind: CustomResourceDefinition
3metadata:
4  name: applicationclaims.platform.kby.io
5spec:
6  group: platform.kby.io
7  names:
8    kind: ApplicationClaim
9    plural: applicationclaims
10    singular: applicationclaim
11    shortNames: [appclaim]
12  scope: Namespaced
13  versions:
14    - name: v1alpha1
15      served: true
16      storage: true
17      subresources:
18        status: {}
19      schema:
20        openAPIV3Schema:
21          type: object
22          properties:
23            spec:
24              type: object
25              required: ["tier", "runtime"]
26              properties:
27                tier:
28                  type: string
29                  enum: ["dev", "staging", "prod"]
30                runtime:
31                  type: string
32                  enum: ["nodejs18", "go1.22", "python3.11"]
33                database:
34                  type: object
35                  properties:
36                    engine:
37                      type: string
38                      enum: ["postgres", "mysql"]
39                    sizeGb:
40                      type: integer
41                      minimum: 1
42                      maximum: 500
43                ingress:
44                  type: object
45                  properties:
46                    public:
47                      type: boolean
48                    hostname:
49                      type: string
50            status:
51              type: object
52              properties:
53                phase:
54                  type: string
55                observedGeneration:
56                  type: integer
57                conditions:
58                  type: array
59                  items:
60                    type: object
61                    properties:
62                      type: {type: string}
63                      status: {type: string}
64                      reason: {type: string}
65                      lastTransitionTime: {type: string}

Note the status subresource. Splitting spec and status into separate subresources means updates to observed state do not increment the resource’s metadata.generation, which is the field your reconciler uses to detect actual user-intent changes versus status churn. Skipping this is one of the most common mistakes in early-stage IDP control plane builds, and it results in reconcile loops that fire on their own status writes.

#Operators and the Reconciliation Loop

The Operator is a controller process built on controller-runtime that watches the API server via an informer cache, populates a work queue keyed by namespace/name, and invokes a Reconcile function per item. Kubernetes reconciliation is level-triggered, not edge-triggered: the function receives no diff, only a request telling it “something changed on this object, go look at the world and converge it”. This is a deliberate design decision documented in the upstream Kubernetes Operator pattern, and it is what makes the loop idempotent and safe to replay after crashes, restarts, or missed events.

1func (r *ApplicationClaimReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
2    log := log.FromContext(ctx)
3    var claim platformv1alpha1.ApplicationClaim
4    if err := r.Get(ctx, req.NamespacedName, &claim); err != nil {
5        if apierrors.IsNotFound(err) {
6            return ctrl.Result{}, nil
7        }
8        return ctrl.Result{}, err
9    }
10
11    if !claim.DeletionTimestamp.IsZero() {
12        return r.handleFinalizer(ctx, &claim)
13    }
14
15    if !controllerutil.ContainsFinalizer(&claim, finalizerName) {
16        controllerutil.AddFinalizer(&claim, finalizerName)
17        if err := r.Update(ctx, &claim); err != nil {
18            return ctrl.Result{}, err
19        }
20    }
21
22    ns := buildNamespace(&claim)
23    if err := r.reconcileResource(ctx, ns); err != nil {
24        r.setCondition(&claim, "NamespaceReady", "False", err.Error())
25        return ctrl.Result{RequeueAfter: 15 * time.Second}, err
26    }
27
28    deploy := buildDeployment(&claim)
29    if err := r.reconcileResource(ctx, deploy); err != nil {
30        return ctrl.Result{RequeueAfter: 30 * time.Second}, err
31    }
32
33    claim.Status.Phase = "Ready"
34    claim.Status.ObservedGeneration = claim.Generation
35    if err := r.Status().Update(ctx, &claim); err != nil {
36        return ctrl.Result{}, err
37    }
38
39    return ctrl.Result{}, nil
40}

Every branch that fails returns a RequeueAfter rather than an immediate retry, which prevents the operator from hammering downstream APIs (a cloud database provisioning endpoint, for example) during transient failures. The controller-runtime work queue additionally applies exponential backoff on repeated errors regardless of the explicit requeue value, so cascading failures in the IDP control plane degrade gracefully rather than causing a request storm.

#Implementation Logic: From Developer Claim to Provisioned Environment

The end-to-end flow through the IDP control plane looks like this in practice:

• Developer submits an ApplicationClaim via kubectl, a Backstage scaffolder action, or a Git commit reconciled by Argo CD.
• A validating admission webhook checks cross-field constraints the OpenAPI schema cannot express, for example rejecting tier: prod claims without an assigned cost centre label.
• The API server persists the object to etcd and the operator’s informer receives the watch event within milliseconds.
• The Reconcile function computes desired child resources (Namespace, NetworkPolicy, Deployment, database claim via a Crossplane composition) and applies them using server-side apply to avoid clobbering fields owned by other controllers.
• Status conditions are written back, and Backstage or the CLI polls status.phase until it reaches Ready.

1apiVersion: platform.kby.io/v1alpha1
2kind: ApplicationClaim
3metadata:
4  name: checkout-service
5  namespace: team-payments
6spec:
7  tier: staging
8  runtime: go1.22
9  database:
10    engine: postgres
11    sizeGb: 20
12  ingress:
13    public: true
14    hostname: checkout-staging.internal.kby.io

Note that the developer never touches Terraform state, Helm values files, or cloud IAM policies directly. Every downstream primitive is owned and labelled with an ownerReference back to the ApplicationClaim, which gives you cascading garbage collection for free when the claim is deleted, provided finalizers are handled correctly.

#Failure Modes and Edge Cases in the IDP Control Plane

A control plane serving hundreds of engineers will surface failure modes that never appear in a demo cluster with five test objects:

• Thundering herd on relist: when the operator pod restarts, the informer performs a full LIST against the API server and re-queues every existing object. At 5,000+ claims, this can spike API server CPU and cause request timeouts for unrelated controllers sharing the cluster.
• Finalizer deadlocks: if handleFinalizer fails silently (for example, a downstream cloud database deletion API returns a 5xx indefinitely), the object never leaves Terminating state and the namespace becomes stuck, blocking kubectl delete namespace entirely.
• Webhook unavailability as a single point of failure: a failurePolicy: Fail validating webhook that goes down blocks all writes to the CRD cluster-wide, not just the affected tenant. Many platform teams learn this the hard way during a webhook pod OOM-kill event.
• Stale cache reads: the informer cache resyncs on an interval (commonly 10 hours by default) and reconciles are otherwise event-driven; if a watch connection silently drops without the client noticing, drift between actual and desired state can persist far longer than expected.
• Split-brain reconciliation: running multiple operator replicas without leader election means two processes can concurrently reconcile the same object, issuing conflicting writes to child resources and triggering an infinite update loop as each replica “corrects” the other’s changes.
• Schema migration breakage: promoting v1alpha1 to v1beta1 without a conversion webhook silently breaks any client still requesting the old version, including CI pipelines that pin an API version string.

#Scaling and Security Trade-offs

Once the IDP control plane is serving production traffic, every design decision above becomes a trade-off you must actively manage rather than a default you can ignore:

• Namespaced vs cluster-scoped CRDs: namespaced claims give you native RBAC tenancy boundaries per team, but require a per-namespace admission webhook configuration; cluster-scoped resources simplify platform-wide policy but widen the blast radius of a single misconfigured RBAC binding.
• Leader election overhead vs availability: enabling leader election (--leader-elect=true) prevents split-brain reconciliation but introduces a brief reconciliation gap during operator pod upgrades while the new leader acquires the lease, typically 10-15 seconds with default lease durations.
• Admission webhook latency budget: every kubectl apply now pays the round-trip cost of your webhook. Keep validation logic under 100ms p99 and always set a sane timeoutSeconds, or you risk API server request timeouts platform-wide during load spikes.
• Watch cache memory at scale: at 10,000+ custom resources, the informer’s in-memory cache on each operator replica becomes non-trivial; consider sharding by splitting a monolithic CRD into narrower, purpose-specific resource types rather than scaling replica count, since unsharded work queues do not parallelise across replicas without leader-per-shard patterns.
• Cascading deletion vs custom cleanup: ownerReferences give free garbage collection for Kubernetes-native children, but any resource outside the cluster (a managed database, a DNS record, an IAM role) requires explicit finalizer-based cleanup logic with its own retry and dead-letter handling.
• Audit logging retention vs compliance cost: every write to the IDP control plane is auditable at the API server level, which satisfies most SOC 2 and ISO 27001 change-tracking requirements out of the box, but high-cardinality audit logs at scale carry non-trivial storage and indexing cost if shipped to a SIEM unfiltered.