Skip to content
draft platform v0.1.0 Pre-Release Automated Evaluation, Testing, and Operational Readiness Created 2026-07-29 · Updated 2026-07-29

PRAETOR — Pre-Release Automated Evaluation, Testing, and Operational Readiness

Objective

Adopt AWS DevOps Agent release management so that every pull request in enrolled repositories is evaluated against ASU standards, cross-repository dependency risk, and actual build and test behavior before it merges.

Context

Code review today depends on reviewer availability and reviewer attention. Rule-based tooling — Datadog code analysis, SonarQube — catches known patterns but cannot tell whether a change will break a consumer in another repository or fail when built.

AWS DevOps Agent release management addresses that directly, and requires no orchestration to be built. It reviews changes through three lenses — standards compliance defined as natural-language Skills, cross-repository dependency analysis from an indexed knowledge graph, and access-control verification — then performs automated verification testing by building, running, and testing the change in a managed environment. Reviews auto-trigger when a pull request opens and when new commits are pushed, findings post as inline comments, and the result can be configured as a required status check.

This follows the pattern AVATAR established for AWS Security Agent: adopt an AWS frontier agent, encode ASU-specific standards into it, and govern rollout as an ASU product.

Two constraints shape phase 1. Release management is in preview and available only in us-east-1. Automatic pull request triggers are restricted to private repositories, because anyone can open a pull request against a public one. Measured in July 2026 that excluded 7 of roughly 918 repositories in the ASU organization, so the practical cost is small — a ratio worth re-checking rather than assuming.

Boundary against SIRIWAIT. DevOps Agent also has a production-operations half covering autonomous incident investigation and coordination. That is deferred (see Future Phases) and concerns operational incidents — availability, latency, deployment regressions. Security incident response remains SIRIWAIT's scope. The two overlap in agentic investigation tooling and in alert routing through Slack and PagerDuty, and that overlap needs resolving before production operations is taken up here.

Tools In Use

  • AWS DevOps Agent — release readiness code review on GitHub pull requests
  • DevOps Agent Skills — ASU review standards expressed in natural language, no policy-as-code required
  • AGENTS.md — per-repository tuning of what verification testing builds and runs
  • GitHub organization integration — Agent Space connection to enrolled repositories

Under evaluation

  • Automated verification testing — enabling it means AWS builds and executes our code in a managed environment; a per-repository decision, not a default
  • Private VPC connection — required only if verification builds need Artifactory or other internal artifact stores, which the default allowlist does not reach
  • Required status check integration — deferred to phase 2

Capability Comparison

Capability Datadog Code Analysis SonarQube Amazon Q Developer (GitHub) ARBITER PRAETOR
Rule-based static analysis ✅ ✅ ➖ ❌ ➖
Standards defined in natural language ❌ ❌ ✅ .amazonq/rules ✅ checklist ✅ Skills
Cross-repo dependency risk ❌ ❌ ❌ ❌ ✅
Builds, runs, and tests the change ❌ ❌ ❌ ❌ ✅
Access-control verification ➖ ➖ ➖ ✅ any IaC ✅ CloudFormation only
Triggers on new commits ✅ ✅ ❌ ✅ ✅
Native required status check ✅ ✅ ❌ ➖ custom ✅
Multi-model cross-validation ❌ ❌ ❌ ✅ ❌
Public repositories ✅ ✅ ✅ ➖ deferred past phase 1 ❌
Build effort — — none pipeline none

➖ indicates partial or indirect coverage.

Note the access-control row: DevOps Agent's verification is documented as CloudFormation-only. ASU's IaC is OpenTofu, so that lens does not apply to our highest-risk diffs.

Scope

Phase 1 — advisory reviews

  • Agent Space created in us-east-1, GitHub organization integration connected
  • Enrollment of a small set of private repositories, with auto trigger change review enabled
  • ASU review standards authored as DevOps Agent Skills, version-controlled rather than console-only
  • Findings advisory: inline comments and a status comment, no merge blocking
  • Per-repository decision on automated verification testing, with a VPC connection only if builds need internal artifact stores
  • Concurrency and quota baseline established — the default is 4 concurrent release readiness reviews per Agent Space
  • Cost observed while free during preview, and modeled for post-preview rates

Phase 2 — gating

  • Promotion to a required status check on Blocking findings, contingent on a precision threshold
  • Documented bypass path and a policy for when using it is legitimate

Out of scope

  • Public repositories — no automatic trigger is available
  • Production operations — see Future Phases
  • Replacing Datadog code analysis or SonarQube
  • Automatic remediation merges

Risks

Risk Likelihood Impact Mitigation
Release management is preview and could change or regress without notice Medium High Advisory only in phase 1; no merge depends on it until phase 2
Single-region availability makes us-east-1 a hard dependency High Medium Accept for phase 1 and record it; review if release management expands regions
Access-control verification does not cover OpenTofu High Medium Do not present this lens as covering our IaC; ARBITER's checklist-driven review covers that gap
Post-preview pricing turns an unmetered pilot into a per-review bill High Medium Model cost before preview ends; confirm whether AWS Support plan credits apply; set budget alarms
Concurrency quota of 4 reviews per Agent Space throttles review throughput at scale Medium Medium Baseline during pilot; request an increase before broad enrollment; keep advisory so throttling never blocks a merge
Verification testing builds and runs our code in an AWS-managed environment High Medium Per-repository opt-in; review the documented network allowlist and vendor guardrails before enabling on anything sensitive
Findings post publicly on pull requests in repos owned by other teams High Medium Owner consent before enrollment, and a heads-up to contributors
Duplicate AI review comments where ARBITER also runs High Low Distinct comment prefixes per product and a stated policy for divergent verdicts
Repository indexing exposes cross-repository source to the service Medium Medium Confirm data handling against the service's security documentation before enrollment

Success Criteria

  • Reviews trigger automatically on pull request open and on new commits in enrolled repositories
  • ASU standards authored as Skills are visibly reflected in findings, not just generic advice
  • Findings arrive as inline comments with an overall status comment, and block nothing in phase 1
  • Skills are version-controlled and reviewable, not edited only in a console
  • Where verification testing is enabled, builds succeed on enrolled repositories
  • Concurrency and cost baselines are recorded before broad enrollment
  • The OpenTofu access-control gap is documented so no one assumes coverage that does not exist

Future Phases

  • Production operations: autonomous incident investigation, mitigation plans, proactive prevention, incident coordination through Slack, ServiceNow, and PagerDuty, on-demand SRE tasks — contingent on resolving the SIRIWAIT boundary
  • Broad enrollment beyond the pilot set
  • Release testing against deployed instances as a pipeline stage
  • Coding-agent integrations that surface findings before a commit exists

Open Questions

  • What is post-preview pricing for release management, and do AWS Support plan credits offset it?
  • Which repositories enroll first, and who consents where contributors sit outside the team?
  • Is automated verification testing enabled in phase 1, and on which repositories?
  • Which AWS account owns the Agent Space, and who administers it?
  • Is us-east-1 acceptable for this workload's data residency?
  • Where do DevOps Agent Skills live in version control, and how are they promoted?
  • What precision threshold promotes the check from advisory to required?
  • How does this interact with existing required checks from Datadog and SonarQube?

References

Decisions

Date Decision Rationale
2026-07-29 Phase 1 covers release management only Answers the pull request review requirement; production operations is a separate rollout touching on-call tooling and SIRIWAIT's scope
2026-07-29 Public repositories are out of scope DevOps Agent does not auto-trigger on public repos, which affected 7 of roughly 918 ASU repositories when measured in July 2026
2026-07-29 Advisory in phase 1 Avoids making merges depend on a preview feature in a single region
2026-07-29 ASU standards encoded as DevOps Agent Skills Native customization surface, natural language, no policy-as-code expertise required
2026-07-29 Independent of ARBITER Complementary capability, no build-versus-buy dependency in either direction

Changelog

Date Status Change
2026-07-29 draft Initial draft