Part 3: Reconciling the Agentic Action Ledger

Part 2 established the Credential Broker: a system that mints single-use, 30-90 second TTL credentials for autonomous agents, each cryptographically bound to a specific Action Ledger hash. That solves the authorisation problem. It does not solve the audit problem. An agent orchestrator can claim it performed an action, write that claim to its own ledger, and yet never have consumed a matching credential — or worse, consumed one and then diverged from the claimed outcome. This is the exact gap this part addresses: Action Ledger reconciliation, the process of proving that what an agent says it did matches what the credential subsystem cryptographically witnessed it doing.

#The Reconciliation Gap

In a human-operated system, a mismatched audit trail is usually a logging bug. In an agentic system, it is a security incident. Large language model orchestrators are non-deterministic; they retry silently, they hallucinate tool success, and under load they can duplicate function calls without surfacing an error to the planning loop. If the orchestrator’s own log is treated as the source of truth for compliance reporting, none of this is detectable. Action Ledger reconciliation exists precisely because the orchestrator’s claimed sequence and the broker’s issued-credential sequence are two independent, adversarially-untrusted data sources that must be forced into agreement before either is trusted for SOC 2 or ISO 27001 evidence.

#Architectural Breakdown: Two Streams, One Truth

The reconciliation engine sits between two append-only, hash-chained streams:

• Orchestrator Action Ledger (OAL) — intent-plus-outcome records written by the agent runtime: action ID, tool manifest reference, claimed timestamp, claimed result hash.
• Credential Issuance Log (CIL) — the broker’s ground-truth record from Part 2: every credential minted, its TTL, the resource ARN it was scoped to, and the ledger hash it was bound against at issuance time.

Neither stream is permitted write access to the other. The CIL is written exclusively by the broker process under its own SPIFFE identity; the OAL is written exclusively by the orchestrator. This separation is what makes Action Ledger reconciliation meaningful — if the orchestrator could edit the CIL, reconciliation would be theatre. For teams designing this boundary from scratch, it is worth reviewing established architectural patterns for append-only event sourcing before implementing the join logic below, since the same idempotency and ordering guarantees apply.

#Why a Simple Join Is Insufficient

A naive reconciliation job would join OAL and CIL on (action_id) and flag mismatches. In practice this produces excessive false positives because credential issuance and action completion are not atomic — there is a window, typically (50-400,ms), between credential mint and the orchestrator’s write of the outcome record. Reconciliation must therefore be a stateful three-way comparison: claimed action, issued credential, and consumed credential (the STS/Vault usage event), each timestamped and compared against a tolerance window rather than exact equality.

#Implementation Logic

1. Both OAL and CIL entries are written with a monotonic sequence number per orchestrator instance, not wall-clock time alone, to survive clock skew across regions.
2. The reconciliation service consumes both streams from an append-only store (Kinesis, or a Postgres table with logical replication) and buffers entries in a sliding window keyed by (action_id).
3. For each window, the service computes reconciliation status: MATCHED, ORPHAN_CREDENTIAL (issued, never claimed), ORPHAN_CLAIM (claimed, no credential ever issued), or DIVERGENT_OUTCOME (result hash mismatch).
4. Anything other than MATCHED after the tolerance window expires is written to a quarantine table and triggers a broker-side session freeze for that orchestrator identity.
5. Reconciled records are sealed into a Merkle batch every (N) minutes and the root hash is exported to an external, append-only audit sink for tamper evidence.

#Action Ledger Entry Schema

1{
2  "action_id": "act_7f21c9",
3  "orchestrator_id": "spiffe://kby.internal/agent/planner-03",
4  "sequence_no": 44821,
5  "tool_manifest_ref": "s3-object-put-v2",
6  "claimed_ts": "2024-05-14T11:02:31.220Z",
7  "claimed_result_hash": "sha256:9ac0...e21f",
8  "credential_hash_expected": "sha256:44bd...9a01"
9}

#Credential Issuance Log Entry

1{
2  "credential_id": "cred_88df2a",
3  "broker_id": "spiffe://kby.internal/broker/prod",
4  "sequence_no": 44819,
5  "issued_ts": "2024-05-14T11:02:30.910Z",
6  "ttl_seconds": 45,
7  "resource_arn": "arn:aws:s3:::kby-artifacts/build-4471.tar.gz",
8  "bound_ledger_hash": "sha256:44bd...9a01",
9  "consumed_ts": "2024-05-14T11:02:31.140Z"
10}

#Reconciliation Query (Postgres)

1SELECT o.action_id,
2       CASE
3         WHEN c.credential_id IS NULL THEN 'ORPHAN_CLAIM'
4         WHEN o.claimed_result_hash  c.bound_ledger_hash THEN 'DIVERGENT_OUTCOME'
5         WHEN c.consumed_ts IS NULL
6              AND now() - c.issued_ts > interval '90 seconds' THEN 'ORPHAN_CREDENTIAL'
7         ELSE 'MATCHED'
8       END AS reconciliation_status
9FROM orchestrator_action_ledger o
10LEFT JOIN credential_issuance_log c
11  ON o.credential_hash_expected = c.bound_ledger_hash
12WHERE o.claimed_ts > now() - interval '10 minutes';

The service configuration governs tolerance windows and quarantine behaviour rather than hardcoding them, since tolerance is workload-dependent:

1reconciliation:
2  window_seconds: 120
3  clock_skew_tolerance_ms: 500
4  quarantine_on:
5    - ORPHAN_CREDENTIAL
6    - DIVERGENT_OUTCOME
7  freeze_orchestrator_on_quarantine: true
8  merkle_seal_interval_minutes: 5
9  audit_sink: "s3://kby-audit-sealed/reconciliation/"

#Failure Modes and Edge Cases

Action Ledger reconciliation introduces its own failure surface, distinct from the broker failures covered in Part 2.

• Reconciliation lag under burst load — if the orchestrator issues hundreds of parallel tool calls, the sliding window can fill faster than the join completes, producing transient ORPHAN_CREDENTIAL flags that resolve once the consumer catches up. Alerting thresholds must debounce on window closure, not on first detection.
• Partial writes on orchestrator crash — if the orchestrator process dies after a credential is consumed but before the OAL entry is flushed, this is a legitimate ORPHAN_CREDENTIAL, not an attack. The quarantine handler must distinguish crash-recovery replays (identified by a resumed sequence number) from genuine gaps.
• Replay of a stale ledger hash — an attacker with a captured credential could attempt to bind it to a fabricated OAL entry after the fact. Because credentials are hash-bound at mint time per Part 2, any post-hoc claim produces a hash mismatch and surfaces as DIVERGENT_OUTCOME immediately.
• Sequence number wraparound or restart — orchestrator redeploys that reset sequence counters to zero will generate false ORPHAN_CLAIM storms unless the reconciliation service keys on a composite of orchestrator instance ID plus sequence, not sequence alone.
• Clock skew exceeding tolerance — cross-region orchestrators with skew beyond (500,ms) will produce persistent false divergence; NTP/Chrony drift must be monitored independently of the reconciliation pipeline, since the pipeline cannot distinguish skew from tampering.

#Scaling and Security Trade-offs

Action Ledger reconciliation can run as a streaming job or a batch job, and the choice has direct consequences for both blast radius and cost.

• Streaming reconciliation (Kinesis/Flink) detects DIVERGENT_OUTCOME within seconds, enabling near-real-time orchestrator freeze, but costs materially more at high call volumes and requires exactly-once semantics on the consumer side to avoid double-quarantine.
• Batch reconciliation (5-15 minute windows) is cheaper and simpler to operate but extends the attack window during which a divergent orchestrator continues operating with live credentials.
• Fail-closed quarantine (freezing the orchestrator identity on any ORPHAN_CREDENTIAL) maximises security but risks halting legitimate work during load-induced lag; this trade-off should mirror the fail-closed broker behaviour established in Part 2 rather than introduce a second, inconsistent policy.
• Merkle-sealed audit export adds storage and compute overhead per batch but is the only practical way to prove to an external auditor that reconciliation records were not altered after the fact — a requirement increasingly cited in vendor due-diligence questionnaires for autonomous systems.
• Sequence-keyed windows scale linearly with orchestrator count but degrade if a single orchestrator identity is shared across many parallel agent instances; per-instance SPIFFE IDs from Part 2 avoid this by keeping the sequence space partitioned.

For teams building audit tooling on top of this, HashiCorp’s own guidance on structured audit devices is a useful reference point for log integrity design: Vault Audit Devices documentation.

With reconciliation in place, the platform can now cryptographically prove agreement between claimed and credentialed action sequences. Part 4 moves up a layer, addressing how Crossplane Compositions expose these guarantees as consumable infrastructure primitives for agent-driven provisioning.