Moving Beyond the Runbook: How We Solved a Silent Infrastructure Failure Using AI and MCPs

As Head of Engineering, I still take my turn on the weekend on-call rotation. It keeps me grounded in the day-to-day realities of our system, and more importantly, it serves as a continuous feedback loop for our engineering discipline. Every on-call shift is an opportunity for a retrospective—not just to patch a symptom, but to figure out how we can elevate our engineering practice.

This past weekend, an alert tripped. It was an automated check-in monitor flagging a missed run for one of our critical high-frequency background workers.

Historically, this would trigger a tedious, manual fact-finding mission for the on-call engineer. They would open up a dozen browser tabs, cross-reference application log buckets, scan error tracking platforms, and query database replicas to reconstruct the timeline.

This time, I didn't open a terminal to hunt for logs. Instead, I handed the investigation over to an AI development harness.

The Diagnostics Hunt: Reconstructing the Crime Scene

What made this investigation remarkable wasn't just the AI's reasoning; it was the AI's access. By utilizing a suite of targeted Model Context Protocol (MCP) servers, the LLM had a secure, real-time window into our environment. It could query cloud system logs, inspect our error-tracking platform, and safely run read-only queries against our relational database replicas.

The diagnostic process unfolded with a level of rigor that usually takes an engineer 30 to 45 minutes of deep context-switching to achieve:

• The Application Ghost Town: The harness scanned our application error tracker and log groups during the specific 15-minute window of the failure. It found absolutely nothing. No exceptions, no aborted tasks, no fatal errors.
• Checking Business Logic: It opened a secure, read-only tunnel to our database replica to check if a global feature flag or "kill switch" had been toggled to intentionally halt the worker. The flags were all normal.
• Dipping into the Infra Layer: Recognizing that a total lack of application-level footprint meant the code never even started executing, the AI dropped down a layer. It queried the underlying container orchestration API logs.

There, it found the verbatim smoking gun in the container deployment layer response:

"failures": [
  {
    "reason": "RESOURCE:MEMORY",
    "detail": "Insufficient memory available (3072 requested, 763 available)."
  }
]

At the exact second the job was scheduled to fire, a "thundering herd" of seven other scheduled tasks had triggered simultaneously. They dogpiled onto the shared container instances, starving our worker of its required memory allocation. The task failed placement, never launched a container, and silently skipped its cycle.

Pausing for Engineering Discipline

Before jumping to a fix, I asked the harness to pause. In legacy workflows, a common fix for a resource spike is throwing money at it—bumping instance sizes or increasing the auto-scaling headroom.

Instead, we analyzed the root cause. The harness parsed our core configuration templates and mapped out every scheduled task in the system. It discovered that we had a massive scheduling collision at the top of the hour.

Crucially, we asked a vital safety question before changing anything: Do any of these tasks have implicit time-window dependencies tied to the exact top of the hour? If a legacy task relies on a hardcoded clock calculation (like looking back exactly 60 minutes from :00), moving its execution time blindly breaks data integrity.

The AI safely audited the source code of the conflicting tasks, proved they used relative time-deltas or snapped boundaries mathematically, and proposed a zero-cost, staggered cron schedule that completely flattened the concurrency wave.

The Paradigm Shift: The Death of the Static Runbook

This incident highlighted a massive inflection point for our engineering organization.

Right now, we have hundreds of static documentation pages indexed into hundreds of alerts, explicitly telling engineers what steps to take when a specific alert fires. Those guides were written in a pre-LLM era.

Today, those documents are fundamentally obsolete.

Static runbooks are rigid, reactive, and age like milk. What we are moving toward is a paradigm where the on-call engineer doesn't follow a recipe; they pair with an AI harness that has deep, contextual system access via secure MCPs.

This changes the entire shape of engineering culture:

• From Scripting to Systems Thinking: On-call shifts are notorious for producing burnout because engineers spend their time executing repetitive mitigation steps. When the tedious data aggregation is offloaded to a harness, the engineer can focus on system architecture and long-term remediation.
• Empowering the Whole Team: A junior engineer on-call armed with an MCP-enabled harness can safely diagnose complex, multi-layered infrastructure failures that previously required escalating to a principal engineer.
• Standardizing with Guardrails: The next milestone for our leadership team isn't writing more documentation—it is standardizing the safe deployment of these AI tools. We are defining the exact guardrails, permission boundaries, and audit logs required to ensure that our team can use MCPs securely across the infrastructure.

We aren't just adopting an AI tool to fix a cron job. We are completely rewriting how we train, empower, and scale our engineering talent to build a more resilient organization.