Why this is hard
LLMs and reasoning engines are fundamentally probabilistic. Temperature, token sampling, context ordering, and model version all influence outputs. The same architectural question about the same code change can produce different answers across runs. For a documentation tool, this is an acceptable trade-off. For a governance gate that blocks or passes PRs, it is not.
The goal is deterministic governance outcomes — a conformant PR should always pass, a non-conformant PR should always fail, and the reason should always be the same. Achieving this with AI components in the critical path requires pushing probabilism to the edges and building deterministic structure around it.
rkito's approach has two parts. First: encode as much knowledge as possible procedurally — in rules, taxonomies, and match channels — so that AI is never asked to decide what it doesn't need to decide. Second: where AI judgment is genuinely required, constrain it aggressively through prompt design, context scoping, structured output, and logging.
What is implemented today
LiveLens selection starts deterministically. ORG_WIDE lenses are always included — they apply to every PR because the architectural concerns they enforce are not localized to specific files or components. They are often authored to enforce Concepts (the foundational architecture artifacts), though any org-wide architectural principle can be expressed as one. Targeted lenses are then selected via five deterministic channels: glob patterns, component taxonomy, steering artifact binding, pillar overlap, and CHI entity association. After that base selection is complete, a focused AI call reviews the remaining unmatched lenses and the PR context — changed files, classified components, PR title — and suggests any additional lenses the rule engine could not infer. The deterministic base always applies. The AI expansion is additive and conservative.
Files in known component paths are classified deterministically from the Architecture Ledger. The AI is only called for files that fall outside any registered path — and even then, the context submitted is limited to the file's extension, imports, and the component path candidates.
Each lens prompt is written to elicit a yes/no answer, not open-ended reasoning. The prompt asks a single architectural question: 'Does this change violate principle X?' Binary outputs are easier to test, easier to audit, and more consistent across model versions.
The context submitted to an AI lens evaluation is the narrowest slice that can answer the question: the relevant file diff, the lens question, and the component's architectural role. Not the PR description, not the full repository, not adjacent files unless the lens explicitly requires them.
AI calls return structured fields — answer (YES/NO/INCONCLUSIVE), reasoning (one paragraph), evidence (file:line citations) — enforced at the schema level. The model cannot produce a malformed or uninterpretable response without triggering a retry.
File and line citations are extracted from the AI's reasoning paragraph via a pattern matcher — not trusted as-is from the model output. The extracted evidence is stored separately and verified against the actual PR diff.
Every AI call stores the exact system prompt, user prompt, and raw response. A verdict can be replayed against a different model or a newer version of the same model to detect verdict drift. The audit record is the ground truth, not a reconstruction.
Research areas and roadmap
UpcomingThese are the open problems. Some are engineering challenges with clear solutions. Some are active research questions. All of them matter to the reliability of automated design governance at scale.
Before a new lens prompt is activated, run it against a golden dataset of known-conformant and known-non-conformant PR diffs and measure the precision/recall. A prompt with less than 95% accuracy on the benchmark should not reach production.
Track how often the same PR diff produces different verdicts across time and model versions. A lens prompt whose verdict distribution shifts after a model update is a signal that the prompt was relying on model-specific behavior rather than stable semantic reasoning.
For lenses marked BLOCKER severity — where a false positive stops a PR from merging — run the evaluation against multiple models and require majority agreement before returning BLOCKED. A single model's verdict on a high-stakes check is insufficient.
Before submitting to AI, run deterministic AST-level or static analysis to pre-classify obvious cases. A function that calls no external services can be deterministically excluded from an external-integration lens without an AI call. Submit to AI only when static analysis is genuinely inconclusive.
Use embeddings to rank which sections of a PR diff are most relevant to the lens question, and submit only the top-N sections rather than the full diff. This shrinks the context window, improves consistency, and reduces the surface area for irrelevant content to influence the verdict.
Systematically apply small mutations to lens prompts — rephrasing, negation, synonym substitution — and measure whether verdicts on known examples change. A prompt that is sensitive to minor wording changes is semantically fragile and will produce inconsistent verdicts in the field.
For lenses that consistently fire on simple syntactic criteria — a specific import, a missing annotation, a function signature pattern — automatically extract a formal rule that replaces the AI call entirely. The AI's historical responses become the training signal for a deterministic rule.
Verify that the file:line evidence cited in the AI's reasoning actually contains the pattern described. If the model says 'line 47 exposes an unauthenticated endpoint' and line 47 is a comment, the citation is wrong. Deterministic post-processing catches these before the result reaches the developer.
Track how often different reasoning models agree on the same lens evaluation. Persistent disagreement between models on the same PR diff is a signal that the prompt is ambiguous, not that one model is wrong. High disagreement rate is a lens quality metric.
The same PR diff submitted at different times — before and after a model update, a prompt change, or a lens reconfiguration — should produce the same verdict. Deviations are flagged as consistency violations and surfaced to the lens author for review.
The deeper question
The techniques above are ways to get closer to determinism using AI components. But the deeper question is: for how many lens checks does the AI add value that a deterministic rule cannot provide?
The answer is: for checks that require semantic understanding of intent — “does this change violate the principle of single-responsibility for this service?”, “does this external call handle data classification appropriately?” — a rule cannot substitute for judgment. These checks require understanding what the code does, not just what pattern it matches.
The long-term trajectory for rkito is a growing procedural surface area and a shrinking, better-constrained AI surface area. Every lens check that can be migrated from AI judgment to a deterministic rule should be. The AI calls that remain should be the ones that are genuinely irreplaceable — and those should be subject to the full suite of reliability techniques above.
This is not a solved problem in the industry. It is one of the defining engineering challenges of building reliable AI-assisted governance systems. rkito's position is that it is solvable — not by avoiding AI, but by building deterministic structure around it.
The goal is not to eliminate AI from the audit pipeline. It is to know precisely where AI is making a decision, why that decision cannot be made deterministically, and what evidence was submitted to produce it.