Skip to content
draft platform v0.1.0 Adversarial Review By Independent Two-model Evaluation and Reconciliation Created 2026-07-29 · Updated 2026-07-29

ARBITER — Adversarial Review By Independent Two-model Evaluation and Reconciliation

Objective

Run multi-model adversarial code review automatically on pull requests, so that review findings reaching an author have survived challenge by a second model family rather than resting on one model's judgment.

Context

The adversarial-code-review skill defines a five-stage pipeline: two reviewers from different model families review a diff independently, each then cross-validates the other's findings, and a cheaper model reconciles the result. Findings one reviewer retracts after seeing the other's evidence never reach the author.

Today that pipeline only runs when a human invokes it, because it is built on kiro-cli agents and the subagent tool. Kiro is a tool for humans and is not part of this product's runtime, so the skill serves as specification — the pipeline shape, the review substance in references/review-checklist.md, the attribution rules, and the degrade behavior — while the implementation is written as CI code.

Running the pipeline is what has historically found its defects. The routing fix exists because a real run revealed that both reviewers had been executing on the same model, that reviewer stages never received the review checklist, and that later stages could not identify which PR they were reviewing. Each failure produced a confident, correct-looking review rather than an error. A CI implementation inherits that failure class and must fail closed instead.

Existing automated review at ASU is single-perspective: Datadog code analysis and SonarQube apply fixed rules, and PRAETOR contributes a single agentic reviewer. None of them can disagree with themselves.

Tools In Use

  • Amazon Bedrock — inference for all five stages. Both reviewer slots run on Bedrock-hosted models, which now span multiple lineages including DeepSeek V3.2, GLM 4.7, Kimi K2.5, and Qwen3 Coder Next
  • GitHub Actions — trigger on pull request events and orchestration of the five phases
  • GitHub OIDC → IAM role — short-lived AWS credentials, no stored keys
  • OpenTofu — the IAM role, Bedrock access policy, and any supporting resources
  • adversarial-code-review skill — specification for pipeline structure and review substance

Under evaluation

  • AWS Step Functions — identified upgrade path if phase fan-out, retries, or per-stage observability become the pain point; Parallel states map directly onto the pipeline shape
  • Bedrock AgentCore Runtime — managed sessions up to eight hours, Identity, and Observability, at the cost of a deployable service. Our only recorded AgentCore experience is MACP finding AgentCore Gateway non-functional in our environment — a different component, but not a reassuring baseline
  • Reviewer B model — DeepSeek V3.2, GLM 4.7, or Qwen3 Coder Next. OpenAI models on Bedrock are excluded as a dependency while they remain in limited preview

Capability Comparison

Capability Datadog Code Analysis SonarQube Amazon Q Developer (GitHub) PRAETOR ARBITER
Rule-based static analysis ✅ ✅ ➖ ➖ ❌
Agentic review of intent vs implementation ❌ ❌ ✅ ✅ ✅
Two independent model families ❌ ❌ ❌ ❌ ✅
Cross-validation and retraction before posting ❌ ❌ ❌ ❌ ✅
ASU review checklist applied verbatim ➖ rules ➖ rules ➖ .amazonq/rules ✅ Skills ✅
OpenTofu access-control review ➖ ➖ ➖ ❌ CloudFormation only ✅
Cross-repo dependency risk ❌ ❌ ❌ ✅ ❌
Builds and tests the change ❌ ❌ ❌ ✅ ❌
Triggers on new commits, not just PR open ✅ ✅ ❌ ✅ ✅
Public repositories ✅ ✅ ✅ ❌ ➖ no platform restriction, deferred past phase 1

➖ indicates partial or indirect coverage.

Scope

Phase 1 — advisory pipeline

  • GitHub Actions workflow triggered on pull request open and synchronize
  • OIDC assumption into a dedicated IAM role scoped to the specific Bedrock model IDs
  • All five stages implemented: two parallel reviews, two parallel cross-validations, synthesis
  • Calling-side interpolation of the checklist, prior-stage outputs, and configured model identifiers — a prompt reaching a model with an unsubstituted placeholder fails the run
  • Provider-diversity verification against configured model IDs before any stage is spawned, failing closed rather than degrading to two same-family reviewers
  • Default-on trigger with skip predicates: documentation-only diffs, dependency bumps with no behavior change, and diffs below a size threshold
  • Labels in both directions — one to force a review the predicates skipped, one to suppress
  • Findings posted as a single review: inline comments plus a tier-ordered summary, every comment carrying the model identifiers verbatim
  • Retracted findings never posted; degrade modes labeled in the summary per the skill
  • Synthesis input sizing check, splitting by severity tier rather than truncating

Phase 2 — gating

  • Promotion to a required status check, contingent on a precision threshold
  • Documented bypass path and a policy for when using it is legitimate

Out of scope

  • Rule-based static analysis — Datadog and SonarQube own that
  • Cross-repo dependency analysis and build-and-test verification — PRAETOR's contribution
  • Any dependency on kiro-cli or human-facing agent tooling at runtime
  • Automatic remediation commits

Risks

Risk Likelihood Impact Mitigation
Both reviewer slots resolve to one model family, producing a second opinion labeled as adversarial Medium High Verify configured model IDs before spawning any stage and abort the run on a same-family pair; never infer provider from a model's self-report
A prompt reaches a model with placeholders unsubstituted, so the stage reviews prompt text instead of the diff Medium High Assert no unsubstituted placeholder remains before each spawn; fail the job rather than post
Synthesis exceeds the cheapest model's context and silently reconciles only part of the input Medium High Size the combined input before spawning and split per severity tier; treat the model's own overflow declaration as a backstop only
Untrusted diff content is treated as instructions by a reviewer stage Medium High Diff content is data, never instruction; no tool grants that mutate state; no credentials reachable from a model-controlled path
Source code and secrets present in a diff are sent to model inference High Medium Inference stays within the AWS account, no third-party egress; document what is transmitted and confirm with security review before enrollment
Both reviewers behind one control plane — a Bedrock regional issue or quota ceiling stops the whole pipeline Medium Medium Apply the skill's defined degrade: proceed single-source and label the output; pin a fallback Reviewer B model
Per-PR cost grows unbounded once the trigger is default-on Medium Medium Skip predicates, a per-run token ceiling, a cheaper non-Anthropic Reviewer B, and budget alarms
False positives erode trust and authors stop reading the comments Medium Medium Advisory in phase 1; cross-validation retraction removes findings the counterpart disproves; distinct comment prefix so ARBITER output is attributable
Two AI reviewers commenting on the same diff produces noise High Low Distinct comment prefixes per product and a stated policy for divergent verdicts

Success Criteria

  • The pipeline runs unattended on qualifying pull requests and posts one review per run
  • A run cannot post when its two reviewer slots resolve to the same model family
  • No review comment is posted without the verbatim model identifiers that produced it
  • Findings retracted during cross-validation do not appear on the pull request
  • Skip predicates demonstrably prevent runs on documentation-only diffs and dependency bumps
  • Degraded runs — reviewer failure, cross-validation unavailable — are labeled as such
  • Per-pull-request cost stays within an agreed ceiling
  • The review checklist reaches every reviewer stage, verifiable from the run's own artifacts

Future Phases

  • Step Functions orchestration if the Actions runner proves limiting
  • AgentCore Runtime evaluation
  • A non-AWS Reviewer B if two Bedrock lineages prove insufficiently decorrelated
  • Public-repository coverage, including asu-unity-stack
  • Adjudication of disputed findings as a distinct stage rather than inline

Open Questions

  • Skip predicate thresholds — what diff size, and how are dependency bumps detected reliably?
  • What precision threshold promotes the check from advisory to required?
  • Which repositories enroll first, and who consents on repos with contributors outside the team?
  • Which Reviewer B model, and which fallback?
  • What per-run and monthly cost ceiling?
  • Does sending diff content to Bedrock inference require security review sign-off before enrollment?
  • Are pull requests from outside contributors reviewed, given that fork-triggered workflows cannot assume the OIDC role?
  • Which Bedrock region, and are all required model IDs enabled in the target account?

References

Decisions

Date Decision Rationale
2026-07-29 Kiro is excluded from the runtime kiro-cli is human tooling; the skill becomes specification, the pipeline is reimplemented as CI code
2026-07-29 Both reviewer slots run on Bedrock One auth path via OIDC, no third-party egress of source, and a materially cheaper Reviewer B
2026-07-29 GitHub Actions is the phase 1 host Satisfies the trigger requirement with no deployable service; Step Functions identified as the upgrade path
2026-07-29 Default-on with skip predicates Closest to the skill's stated default while containing cost on diffs where a second opinion adds nothing
2026-07-29 Advisory in phase 1 Precision is unmeasured; gating waits on a promotion criterion
2026-07-29 Independent of PRAETOR Complementary capability, no build-versus-buy dependency in either direction

Changelog

Date Status Change
2026-07-29 draft Initial draft