AI-generated illustration. Evidence, AI recommendations, and release authorization serve different roles. A recommendation cannot substitute for records or authorize a release.
An AI code review policy should keep three things separate: recorded evidence, AI recommendations, and release authorization. Let AI tools explain results and request checks. Require inspectable records tied to the proposed change, and give approval authority to a designated person or explicitly governed gate.
Cloudflare’s October 7 publisher excerpt describes specialized AI agents analyzing security alerts. It reports that deterministic evidence collection is separated from model inference to provide recommendations to Managed Defense Analysts. See Cloudflare’s description of its evidence-grounded security operations harness. This is a reported design for security-alert analysis, not evidence of software-release outcomes or validation of an AI code assurance product.
The transferable idea is a control boundary: a recommendation may explain evidence, but it must not replace the records or grant itself approval authority.
Define three separate layers
Recorded evidence: what happened?
Evidence includes test results, security-check output, architecture-policy evaluations, and documented human assessments. An admissible record should identify:
- The commit or immutable artifact evaluated.
- The check, its version or configuration, and its scope.
- The relevant environment, fixtures, and dependencies.
- The result, including failures, skipped checks, and execution errors.
- The collection owner and a retrievable source record.
Collection ownership means accountability for the record’s provenance and completeness, not necessarily manual execution. An AI agent may request an approved CI job; the CI system records the execution, and the named owner remains responsible for collection.
AI-written tests can contribute evidence when executed through an approved process. Generated test text is not a result. Nor does a pass establish that the assertions are adequate. For critical behavior, a reviewer should assess whether the tests cover the requirement rather than reproduce the implementation’s assumptions.
Model interpretation: what might it mean?
The model may summarize failures, identify gaps, suggest checks, and recommend approval or rejection. Its interpretation should link to supporting records and distinguish observations from inferences.
“The tenant-isolation test passed” is a bounded observation if the record supports it. “The authorization change is secure” is a broader judgment that the same record cannot establish.
A favorable narrative must not turn a failed check into a pass, hide a skipped check, or treat an old result as current.
Accountable authorization: who approves?
Authorization applies policy to the evidence and the change’s risk. In the starter policy below, AI tools may recommend but may not authorize.
If you permit automated authorization for a narrow class of changes, define its eligibility and gates separately. A model calling a change “low risk” must not be enough to qualify it.
A reusable evidence policy
This is a proposed template, not a universal standard. Replace role labels with named owners or maintained approval groups, and specify the required checks for each change class.
| Evidence type | Collection owner | Commit and environment binding | AI interpretation permissions | Accountable approver | Missing-evidence disposition |
|---|---|---|---|---|---|
| Unit and integration results | Repository owner, using approved CI | Proposed commit; runtime, fixtures, and check configuration | Summarize results and suggest coverage gaps; cannot infer a pass from missing output | Change maintainer | Hold if a required suite is absent, skipped, or tied to another commit |
| Required security checks | Security-check owner | Proposed commit or artifact; rule set and configuration | Explain findings and suggest remediation; cannot dismiss findings or waive the check | Designated security approver | Hold until current, inspectable results exist |
| Architecture and dependency-policy checks | Architecture-policy owner | Proposed commit; policy version and dependency manifest or lockfile | Map findings to policy and propose alternatives; cannot redefine policy | Architecture delegate where required | Hold on unmet mandatory controls; route eligible exceptions to the named authority |
| Human assessment of critical behavior | Assigned domain reviewer | Reviewed diff and commit; stated assumptions and scope | Prepare questions and summarize the assessment; cannot impersonate reviewer sign-off | Domain owner, plus security approver where required | Hold until the required assessment is recorded |
| Exception record | Control owner | Exact change, waived control, environment, and expiry | Draft a rationale; cannot grant or extend exceptions | Named exception authority | Apply the normal gate if no valid exception exists |
A collection owner and approver may be the same person where policy allows it. For sensitive changes, explicitly decide whether separation of duties is required. Two role labels do not guarantee independence.
AI-generated illustration. In the hypothetical example, current unit-test results do not fill the security-evidence gap. The earlier security result belongs to the older commit, so approval remains on hold.
Worked example: passing tests, stale security evidence
Hypothetical scenario: A team changes authorization code controlling access to tenant-specific records. Its policy requires current unit tests, a security check, and security approval. Assume the review contains:
- Proposed commit
B, which includes the authorization change. - Passing unit-test results for commit
B. - A passing required security check for earlier commit
A. - An AI recommendation to approve, citing both results.
Decision: hold approval. The security result is valid evidence about commit A, but it does not satisfy the requirement for commit B. Passing unit tests remain useful within their scope; they do not replace the missing current security result.
- The maintainer records the gap: the required security check is stale, not failed.
- The security-check owner reruns the check against commit
Bwith the required configuration and retains the record. - The AI may update its recommendation, identifying any remaining limitations.
- The designated security approver assesses the current evidence and authorization behavior before recording a decision.
- Any subsequent code change triggers reassessment of which records must be refreshed.
This prevents evidence from one code state from silently satisfying a requirement for another. It does not prove that commit B is secure. Current passing checks establish only that their defined assertions passed under the recorded conditions.
Resolve conflicts without choosing the friendliest result
Passing unit tests and a failing security check may describe different properties, not contradictory results. Start by checking scope.
- Compare identity and scope. Are the records about the same commit, environment, configuration, and requirement?
- Preserve the originals. A rerun should add evidence, not erase an inconvenient result.
- Assign an owner. Record whether the discrepancy reflects a defect, execution problem, coverage difference, or unresolved question.
- Require an explicit disposition. The approver needs an explanation and supporting records, not merely a later green status.
Keep approval on hold while material discrepancies in required evidence remain unresolved. Model confidence cannot resolve them.
Decide what can be waived
Separate non-waivable requirements from controls that allow bounded exceptions. For this starter policy, make these requirements non-waivable:
- The proposed change and applicable requirements must be identifiable.
- Source records supporting the decision must remain inspectable.
- A release decision must have an accountable authority.
- An AI summary cannot replace a missing record or grant an exception.
List waivable controls explicitly. An exception should identify the exact control and change, the authorizing person, the rationale, residual risk, compensating measures, and an expiry or revalidation condition. Record the waiver; do not present an unperformed check as passed.
An exception to one check is not blanket release approval.
If emergency releases need a different path, define it in advance. Do not improvise it from an AI recommendation or retroactively declare missing evidence acceptable.
The approval record can be short: change identity, required records, unresolved findings, applicable exceptions, and accountable decision. Keeping these elements separate makes the decision inspectable. It does not, by itself, establish safe code.
Explore AI code assurance with OpenThunder.