The Ironies of A² I² – Architecting for Autonomous Reliability

InfoQ – QCon San Francisco, June 25 2026
Speaker: J. Paul Reed, Staff Incident Operations Manager, Chime
Video: 45 min

---

Service update → Use cases → Trade‑offs

1. Service update – AI‑augmented incident tooling is now mainstream

Over the past year the major cloud providers have added managed AI agents to their observability suites:

Provider	Service	Pricing change (effective 2026‑07)
AWS	Amazon CodeGuru Agent (interactive code‑fix bot)	$0.025 per execution, 20 % discount for ≥ 10 M calls
Azure	Azure Monitor AI Insights (anomaly detection + auto‑remediation)	$0.015 per GB of telemetry processed, free tier up to 5 GB
GCP	Vertex AI Agent Assist (chat‑driven runbook execution)	$0.03 per prompt, volume discount after 1 M prompts
Datadog	AI‑Powered Log Summarizer (Claude‑based)	$0.02 per log GB, bundled with Pro plan

These services promise instantaneous root‑cause suggestions, automated roll‑backs, and even code generation for hot‑fixes. The hype is that human operators can “step back” while the AI runs the playbook.

---

2. Use cases – Where the ironies appear

a) Speed vs. correctness

When a scaling event triggers an auto‑remediation rule, the system can spin up 200 new instances in under a second. The trade‑off is that the rule only checks statistical health metrics; it does not verify downstream dependencies. In a recent incident at a fintech firm, an aggressive autoscale caused a thundering‑herd on a downstream cache, leading to a 12‑minute outage. The AI‑driven rule saved minutes of manual work but doubled the overall recovery time because the root cause was hidden behind the automation.

b) Skill erosion

Developers who rely on Claude‑based agents for routine refactoring quickly forget the exact CLI flags needed for a manual kubectl rollout undo. When a production rollback failed, the on‑call engineer spent 8 minutes searching the docs—a classic illustration of the first irony: manual skills deteriorate when not exercised.

c) Camouflage of system state

An autopilot‑style health check in Azure Monitor AI Insights was silently fixing a memory leak by restarting a container every 5 minutes. The UI showed a green status, masking the underlying degradation. When the leak finally exhausted the host’s memory, the system crashed, and the incident commander had no visibility into the corrective loop that had been running.

d) Opaque decision trees

AI agents built on large language models (LLMs) do not expose a traditional if/else flow. In a post‑mortem, the team could not reconstruct why the Vertex AI Agent Assist suggested a database schema change that introduced a deadlock. The lack of explainability forced a costly forensic effort.

---

3. Trade‑offs – How to balance autonomy and human control

Irony	Architectural implication	Mitigation pattern
Manual skills decay	Over‑reliance on AI for routine tasks reduces hands‑on experience.	Shadow‑runbooks – require a human to execute the same command after the AI, logging both actions.
Speed vs. correctness	Faster automation often means fewer validation steps.	Staged automation – a fast path for low‑risk changes, followed by a slower verification stage for high‑impact services.
Camouflage of state	Automation can mask symptoms until a hard failure occurs.	Observability of automation – emit dedicated metrics (automation.latency, automation.error_rate) and surface them in dashboards.
Opaque AI decisions	LLM‑based agents lack deterministic traces.	Explainability wrappers – use tools like OpenAI’s function calling or Claude’s tool use logs to capture a structured decision record.
Efficiency‑Thoroughness Trade‑off (ETTO)	Incident responders are forced into an efficiency bet already made by the system.	Buy‑time tactics – inject deliberate pauses (await human_acknowledge()) in automated remediation to give engineers a window for manual assessment.

---

4. Practical guidance for incident teams

1. Register AI usage – Every AI‑driven action must be tagged (e.g., source:vertex_ai_agent). Incident commanders need this context to allocate expertise (e.g., a Go specialist when an AI‑generated Go patch is deployed).
2. Maintain mental models – Encourage regular simulation drills that force engineers to solve incidents without AI assistance. This keeps the underlying system knowledge alive.
3. Deploy explainability hooks – For each AI service, enable the audit‑log feature (e.g., aws:codeguru:agent:trace). Store the logs alongside the incident timeline.
4. Separate authority from autonomy – Grant AI agents only the permissions they need (least‑privilege). Use IAM roles that can be revoked instantly if the automation behaves unexpectedly.
5. Iterate runbooks – Treat AI suggestions as proposals rather than final actions. A runbook step might read: “Run AI‑generated fix only after human review of the diff and unit‑test results.”

---

5. What comes next?

The research cited by Reed – from Bainbridge (1983) to Mica Endsley’s situational‑awareness work – shows that the paradox of automation is not new, but the scale of AI amplifies it. Emerging standards such as ISO/IEC 42010‑AI (architectural description for AI systems) and the CNCF AI‑Ops Working Group are beginning to codify best practices around coordination (the missing piece between connectivity and control).

By treating AI as a joint cognitive system rather than a black‑box, architects can design observability stacks that keep humans in the loop, preserve critical skills, and still reap the speed benefits of automation.

---

For a deeper dive, see the full transcript and the slides linked in the QCon archive. The talk is a reminder that the most reliable systems are those where automation and human expertise are deliberately balanced, not where one silently replaces the other.