Determinism is not a natural property of LLMs. Achieving deterministic governance outcomes from probabilistic AI components requires deliberate architecture: procedurally coded knowledge for everything that can be a rule, and the narrowest possible context for everything that cannot. This is an open engineering problem — and a core research area for rkito.
Why this is hard
LLMs and reasoning engines are fundamentally probabilistic. Temperature, token sampling, context ordering, and model version all influence outputs. The same architectural question about the same code change can produce different answers across runs. For a documentation tool, this is an acceptable trade-off. For a governance gate that blocks or passes PRs, it is not.
The goal is deterministic governance outcomes — a conformant PR should always pass, a non-conformant PR should always fail, and the reason should always be the same. Achieving this with AI components in the critical path requires pushing probabilism to the edges and building deterministic structure around it.
rkito’s approach has two parts. First: encode as much knowledge as possible procedurally — in rules, taxonomies, and match channels — so that AI is never asked to decide what it doesn’t need to decide. Second: where AI judgment is genuinely required, constrain it aggressively through prompt design, context scoping, structured output, and logging.
What is implemented today
Research areas and roadmap
These are the open problems. Some are engineering challenges with clear solutions. Some are active research questions. All of them matter to the reliability of automated design governance at scale.
Before a new lens prompt is activated, run it against a golden dataset of known-conformant and known-non-conformant PR diffs and measure the precision/recall. A prompt with less than 95% accuracy on the benchmark should not reach production.
Track how often the same PR diff produces different verdicts across time and model versions. A lens prompt whose verdict distribution shifts after a model update is a signal that the prompt was relying on model-specific behavior rather than stable semantic reasoning.
For lenses marked BLOCKER severity — where a false positive stops a PR from merging — run the evaluation against multiple models and require majority agreement before returning BLOCKED. A single model's verdict on a high-stakes check is insufficient.
Before submitting to AI, run deterministic AST-level or static analysis to pre-classify obvious cases. A function that calls no external services can be deterministically excluded from an external-integration lens without an AI call. Submit to AI only when static analysis is genuinely inconclusive.
Use embeddings to rank which sections of a PR diff are most relevant to the lens question, and submit only the top-N sections rather than the full diff. This shrinks the context window, improves consistency, and reduces the surface area for irrelevant content to influence the verdict.
Systematically apply small mutations to lens prompts — rephrasing, negation, synonym substitution — and measure whether verdicts on known examples change. A prompt that is sensitive to minor wording changes is semantically fragile and will produce inconsistent verdicts in the field.
For lenses that consistently fire on simple syntactic criteria — a specific import, a missing annotation, a function signature pattern — automatically extract a formal rule that replaces the AI call entirely. The AI's historical responses become the training signal for a deterministic rule.
Verify that the file:line evidence cited in the AI's reasoning actually contains the pattern described. If the model says 'line 47 exposes an unauthenticated endpoint' and line 47 is a comment, the citation is wrong. Deterministic post-processing catches these before the result reaches the developer.
Track how often different reasoning models agree on the same lens evaluation. Persistent disagreement between models on the same PR diff is a signal that the prompt is ambiguous, not that one model is wrong. High disagreement rate is a lens quality metric.
The same PR diff submitted at different times — before and after a model update, a prompt change, or a lens reconfiguration — should produce the same verdict. Deviations are flagged as consistency violations and surfaced to the lens author for review.
The deeper question
The techniques above are ways to get closer to determinism using AI components. But the deeper question is: for how many lens checks does the AI add value that a deterministic rule cannot provide?
The answer is: for checks that require semantic understanding of intent — “does this change violate the principle of single-responsibility for this service?”, “does this external call handle data classification appropriately?” — a rule cannot substitute for judgment. These checks require understanding what the code does, not just what pattern it matches.
The long-term trajectory for rkito is a growing procedural surface area and a shrinking, better-constrained AI surface area. Every lens check that can be migrated from AI judgment to a deterministic rule should be. The AI calls that remain should be the ones that are genuinely irreplaceable — and those should be subject to the full suite of reliability techniques above.
This is not a solved problem in the industry. It is one of the defining engineering challenges of building reliable AI-assisted governance systems. rkito’s position is that it is solvable — not by avoiding AI, but by building deterministic structure around it.
The goal is not to eliminate AI from the audit pipeline. It is to know precisely where AI is making a decision, why that decision cannot be made deterministically, and what evidence was submitted to produce it.