The explainability problem in automated audits
Every automated design audit has the same failure mode. The gate blocks a PR. The status check says BLOCKED. No further information follows — which architectural principle was violated, which file triggered it, what question the AI was asked, what it decided and why.
For a developer, this is a friction problem. They cannot confirm the verdict, cannot fix their code with confidence, cannot distinguish a legitimate block from a false positive. For an AI coding agent, the same opacity is a termination condition — an agent that receives BLOCKED with no structured rationale has no path forward.
Explainability is the condition under which automation can be trusted — by the humans it governs and by the agents that submit code alongside them.
The three questions developers and agents ask
When a CDA blocks a PR, three questions follow immediately and in order — whether the author is a human reading a UI or an agent reading a structured payload.
Which specific check failed, what did the AI decide, what file or line triggered it, and what is the full reasoning behind the verdict.
Which lenses ran against my code and why, which lenses exist in the org but did not fire and why not, and what would change that.
What exactly needs to change in the code to conform to the principle — or whether this is a case that requires a Change Intention and architect approval instead.
Driving determinism in the pipeline
Determinism is not a natural property of LLMs — the same input can yield a different verdict across model versions, temperature settings, or context orderings. rkito drives determinism through two complementary strategies: procedurally coded knowledge for everything that can be expressed as rules, and narrow, focused context for the parts that genuinely require semantic judgment.
Lens selection — which checks run against your PR — is entirely rule-based: six fixed channels with no AI involved. AI is confined to the binary semantic judgment inside each lens evaluation, where the context submitted is the smallest slice that can answer the question. Every AI call is fully logged — system prompt, user prompt, raw response — so verdicts are auditable and replayable.
How rkito drives determinism — techniques and research areasLens selection starts with ORG_WIDE lenses, which always run on every PR. These enforce architectural concerns that are not localized to specific files or components — often the foundational Concepts of your system, though any org-wide principle can be expressed as one. Because they are non-local by definition, they do not gate on file paths or component taxonomy.
Targeted lenses are then selected by five deterministic channels: glob patterns, component taxonomy, steering artifact binding, pillar overlap, and CHI entity association. After that, a focused AI call reviews the remaining unmatched lenses against the PR context and may suggest additional ones the rule engine could not infer. At no point does AI decide which ORG_WIDE lenses run — that is always fixed.
The five-stage explainability timeline
Every audit is a structured decision record across five stages — each independently inspectable, organized for interrogation by both humans and agents.
The incoming webhook event: PR metadata, branch, author, full list of changed files. The raw input the entire pipeline worked from.
Each changed file mapped to a component, capability, or domain entity in the Architecture Ledger. Deterministic for classified paths; AI-inferred with full reasoning for unclassified files.
The complete lens selection record: which lenses fired and the exact channels that triggered each match, which lenses exist but did not fire and the exact reason. Fully deterministic.
Per-lens, per-checklist, per-prompt: the question asked, the AI answer, the full reasoning paragraph, the evidence citation (file and line), and the complete AI call — system prompt, user prompt, raw response.
Result aggregation, conformance score, GitHub check status, PR comment. The output the developer sees on the PR.
Closing the loop for AI agents
AI agents are now first-class PR authors. The CDA pipeline evaluates their PRs identically — the same lenses, the same five-stage record. The difference is what happens when the audit returns BLOCKED. A developer reads the UI. An agent needs a machine-readable result it can reason over and act on.
The critical enabler is that the audit result is structured, not narrative. The agent receives which specific checks failed, which lenses fired, the exact file and line cited as evidence, the deterministic reason each lens was selected, and the binary AI verdict with its reasoning paragraph. This is enough for the agent to reason precisely about what to change.
When the lens check caught a genuine violation, the agent can either fix it architecturally or determine that a Change Intention is required and surface that to the developer. The loop is not fully autonomous in the CHI case — structural decisions that require architectural approval must still involve a human. But the agent narrows the problem to the decision that genuinely requires one.
Why missing a deviation is worse than catching one
When a CDA passes a PR that should have been caught, the explainability record is the only way to understand why. After the merge, the question shifts from “why is my PR blocked?” to “why wasn't it caught?”.
The same five-stage record answers this question: which lenses ran, which did not run, what the AI was shown, what it answered. If a lens should have fired but its glob patterns were too narrow — visible. If the AI returned INCONCLUSIVE because the diff context was ambiguous — visible. If the lens was not yet configured for the component that changed — visible.
Every missed deviation is a specific, traceable gap in lens configuration or classification — not a general systemic failure. The audit trail makes the gap nameable and fixable.
Ask rkito — inline audit interrogation
Ask rkito is integrated directly into the audit detail page — not as a separate chatbot, but as a context-aware assistant that knows the full audit record. When a developer clicks “Ask rkito” on a specific failed check, the conversation is pre-seeded with the exact failing question, the AI's answer, the reasoning, and the evidence citation.
The audit context includes: the conformance score, the list of matched lenses, the failed checks, the steering principles those checks operationalize, and the architecture ledger entities involved. The AI has the same information the audit had. It can explain the audit's reasoning without having to reconstruct it.
An audit that cannot be interrogated is not a governance mechanism — it is a black-box gate. Explainability is the condition under which automated design governance can be trusted by the developers it governs and by the agents that author code alongside them.