Cutting geo out of the monolith: moving GPS ingestion from Rails to Go

Solo's mobile app streams a GPS point to our backend every few seconds while someone is driving. For years, those points landed inside our main Rails application — one worker thread tied up per incoming point, each one waiting almost entirely on a fast in-memory database in Redis rather than doing any real computation. That's a wasteful pairing: dedicating a whole thread to sit and wait is expensive at scale, and this one piece of the app does it hundreds of millions of times a day. Over June and July, we pulled this one piece out of the monolith, rebuilt it as a small, dedicated service written in Go, and rolled it into production gradually rather than switching it on all at once.

This post walks through why we made that call, how we tested it safely in production before fully committing to it, what actually went wrong along the way, and — with real numbers pulled from our own monitoring, not estimates — what changed as a result.

The approach

The key decision was to change as little as possible. GPS processing is really two very different jobs: writing new location points as they arrive, and reading that location history back out later, for trip breakdown, mileage, trip routes, dashboards and other features that depend on where a user has been. We only touched the writing side. Every part of the system that reads location data back out — mileage calculation, dashboards, analytics, other backend jobs — was left completely untouched.

That was possible because the new service writes into the exact same underlying data store, in the exact same shape, that the old code used. Nothing downstream needed to know a location point came from a different program. To everything else in our system, a GPS point looks identical whether the old code wrote it or the new code did — so we could swap out how the data gets in, without touching a single line of the much larger, much more complex code that reads it back out. That's the leverage: change the narrow, high-volume path; leave the wide, already-working one alone.

Making the two writers indistinguishable took real care in two specific places:

• How we encode a location. We store GPS points using a well-known technique called geohashing, which converts a latitude/longitude pair into a single compact code — designed so that looking up "who was near this spot recently" later is fast. Both the old code and the new code feed that same code into the same lookup tables, so they have to produce byte-for-byte the identical result for the same coordinate, or the two systems would encode the same real-world location differently and downstream lookups would quietly return wrong or missing results. We didn't just write the new version and eyeball it — we generated a large set of known-correct answers straight from the original code, and the new version is checked against every one of them automatically, every time either side changes. If a future edit would ever produce a different code for the same input, that gets caught immediately, instead of quietly writing bad location data somewhere.
• How we recognize a logged-in user. When someone logs in, our backend issues their app a token — a piece of proof that says, in effect, "this is user #12345, logged in until such-and-such time." To stop anyone from forging a fake one, that token is stamped using a private piece of data that only our backend knows (never sent to the app, never visible to the user); checking a token later means re-doing that same stamping and confirming it still matches. Before accepting a location point, the backend checks that stamp, plus a fast lookup against a marker in Redis confirming the token hasn't since been revoked. That checking logic already existed. The new service had to check the exact same stamp, against the exact same marker, using that exact same private piece of data — not a new or separate one of its own — so that someone who logged in through the (unchanged) login flow would be recognized as valid the instant their app started sending location pings to the new service, with nothing for the end user to notice, and certainly no forced re-login.

One more piece worth calling out: the first GPS point of the day triggers a small downstream job (flagging that today's first location update happened). The old code enqueued that job with a single line, using a client library that hands the job straight to the job queue's own storage. The new service, written in a different programming language, doesn't have that same convenient library available. The easy way out would have been to add a new web endpoint back on the old side, and have the new service call that over the network to enqueue the job on its behalf. We deliberately avoided that: it adds an extra network call to the busiest path in the whole app for no real benefit, and worse, it would make a fundamentally simple write depend on the old application's web server being up and running, when the only thing actually needed is the job queue's storage itself. It also would have meant anyone testing the new service on their own machine would first need to stand up the entire old application just to exercise something that's really just a write to a shared store. Instead, we taught the new service to write a job directly into that shared storage, in the exact format the job queue expects to read — no dependency on the old application at all.

Operational methodology: rolling out traffic gradually, not flipping a switch

The centerpiece of how we tested this safely is a piece of infrastructure called a load balancer. Every request from the mobile app already passes through one on its way to our servers — its job is to receive incoming traffic and decide which pool of backend servers should actually handle each request. Normally that's a simple decision: one pool, one set of servers, done.

A load balancer can also be configured to split incoming requests between two different pools of servers, by percentage — send, say, 90% of requests to pool A and 10% to pool B. That capability is what made this rollout possible. We stood up the new service as a second pool, sitting right alongside the existing one, behind the very same load balancer our users were already talking to — same address, same security, completely invisible to the app itself. Then we dialed the percentage of live, real, production location-tracking traffic sent to the new pool up in stages — 0%, then 10%, then 50%, then 100% — watching error rates and response times closely at every step before moving on to the next one.

That gradual dial is the entire safety mechanism, and it's worth sitting with for a moment, because it's really the whole strategy. At 0%, we're only proving the new service is up and reachable — no real user is touched by it yet. At 10%, a small, deliberately bounded slice of real traffic starts hitting the new code; small enough that if something's wrong, the blast radius is small and the problem is easy to spot. By the time we reach 100%, we've already watched the new service handle a growing share of real, unpredictable production traffic, under real conditions, for days — not guessed at how it might behave from a staging environment that never quite matches reality. And at any point along the way, undoing it is a single number: turn the percentage back down to 0%, and every request goes back to the pool that was already working, instantly, with nothing else to change or clean up. That's what let us treat this as routine, low-stakes work, rather than a scheduled, all-hands cutover event.

Two more things made it safe to run this loosely rather than as a white-glove event:

• Watching for real data loss, not just crashes. Because the new service is designed to tell the phone "success" even in the rare case that an internal save failed (matching the old behavior — see above), simply watching for error responses wouldn't have told us if location points were quietly going missing; everything would have looked perfectly healthy from the outside. So we added a separate alert that watches specifically for the internal signal that a save genuinely failed, deliberately ignoring the common, harmless case of a phone just disconnecting mid-request, so it only pages someone when a location point may actually have been lost.
• Only paying for detailed logging when we could afford it. Because this endpoint is called so often — hundreds of millions of times a day — logging details about every single request was never something we could do on the old side; the sheer volume would have made the logging bill for this one piece of the app alone prohibitively expensive, so the old code never logged anything about individual requests. But the moment we started sending a small percentage of real traffic to the new service, we needed exactly that kind of visibility — a way to trace individual points and be certain none were quietly getting lost in the switch. At low traffic percentages, that same detailed logging was cheap enough to justify, so we turned it on. As we grew more confident and increased the percentage of traffic going to the new service, we deliberately turned it back off again — because once we were also handling full production volume, keeping it on would have recreated the exact cost problem the old side always avoided.

What broke on the way

None of this was a clean cutover. In order:

• A silent way to lose data that only showed up under real mobile conditions. Mobile networks are unreliable — phones lose signal, apps get backgrounded, connections drop mid-request, constantly. Our old code was written in a way that, once it started processing a request, it finished the work even if the phone on the other end had already disconnected — the location point still got saved. The new service, written using the standard way Go handles web requests, did the opposite by default: the moment it noticed the connection was gone, it immediately abandoned whatever it was doing, including a save to Redis that hadn't finished yet. That's a perfectly reasonable default in most situations — why keep working for a client that's no longer listening — but for us it meant a real GPS point could quietly disappear any time a phone's connection blipped mid-request, which on cellular networks is often. The fix was to explicitly tell the service: for this specific piece of work, finish it regardless of whether the client is still there, but never let it run forever — cap it at a firm time limit, so a genuinely stuck save still gets abandoned rather than hanging indefinitely. That matched what the old code had been doing all along, without reintroducing the risk of a request that never ends.
• Chasing a timeout that wasn't the actual problem. Early on, a small share of requests were taking multiple seconds to complete — far slower than normal. The most obvious suspect was the login-check step: it's the one part of handling a request that reaches out over the network to Redis, and a network call is naturally where you look first for network-shaped slowness. So we added a strict time limit there, on the theory that if this check was ever the culprit, we wanted it to fail fast rather than let it drag a request out for seconds. But once we added detailed, per-step timing to a request — so we could see exactly how long each internal step was taking, not just the request as a whole — the numbers showed the login check was never the problem at all: it was consistently finishing in under a millisecond, every single time. The real slowness turned out to be something else entirely — the time spent simply waiting to receive the full request from a slow mobile connection, which has nothing to do with our backend at all. Once we knew that, we relaxed the login-check limit back up, since it was never actually risky, and a limit that tight was itself a hazard — capable of failing a perfectly healthy check on nothing more than an ordinary, brief network hiccup. In its place, we added a separate limit on how long we'd wait to receive a request's data in the first place, which is what was actually needed.
• A bigger connection pool made things worse, not better. Our service keeps a small pool of already-open connections to Redis ready to go, so it doesn't have to pay the cost of opening a brand-new connection for every single incoming request — reusing a warm connection is much cheaper than establishing a new one each time. It's tempting to assume a bigger pool is simply better: more open connections should mean more work can happen at once. But Redis itself is one system with a finite amount of work it can actually do at any given moment — like a single checkout counter, where however many customers are waiting in line, only so many can be served in parallel, no matter how many people are in line. Handing that one counter an enormous pool of open connections doesn't create more real capacity to serve them; it just means more requests are competing for the same limited processing power at once, plus the connections themselves add memory and bookkeeping overhead on both sides. Past a certain size, a larger pool wasn't relieving pressure on Redis, it was adding overhead of its own — so we deliberately shrank it back down (from 200 connections to 64), which reduced the contention rather than the throughput.
• A new program that thought it had more computer than it actually did. When you run a service in the cloud, it doesn't get an entire physical machine to itself — it's given a specific, limited slice of a much bigger shared machine's processing power (in our case, the equivalent of 2 processor cores), enforced by a hard limit called a CPU quota, even though the actual hardware underneath might have dozens of cores available. The Go language spreads a program's work across multiple internal "lanes" running in parallel — controlled by a setting called GOMAXPROCS — and by default it decides how many lanes to open based on how many processor cores it believes it has access to. The version of Go we started with wasn't aware of the cloud-specific CPU quota our container had actually been given — it read the full underlying machine's core count instead, and set GOMAXPROCS far higher than our 2-core allocation could realistically run at once. That's a bit like scheduling a shift of thirty people to share two desks: everyone still shows up, but they spend more time fighting over the desks than getting anything done. Upgrading to Go 1.25 — whose runtime correctly reads the container's actual CPU quota instead of the whole machine's core count — fixed this by setting GOMAXPROCS to match what we'd genuinely been given.
• Our capacity planning was a little too optimistic. Our infrastructure automatically adds more running copies of the service whenever traffic per copy crosses a set threshold — that's the actual mechanism behind autoscaling: a target requests-per-second number baked into the scaling policy, which the system watches to decide when it's time to spin up another copy. We originally set that threshold at 1,500 requests per second per copy. A real evening traffic peak on July 8 — the same day we moved fully over to the new service in production — showed per-copy latency actually starting to degrade closer to 1,300 requests per second (the bottleneck was concurrent round-trips to Redis, not raw CPU). So we lowered the autoscaling threshold itself to 1,000 requests per second per copy, giving ourselves comfortable headroom below where we'd actually seen trouble start, rather than tuning it to sit right at that edge.
• Deciding, deliberately, when it's OK to give up on a write — and never running just one copy. We built in what we call a write budget: a hard one-second ceiling, enforced as a request deadline in code, on how long the service will keep trying to save a single location point to Redis. If Redis is ever briefly overloaded and a particular save is taking unusually long, the service abandons that one save the instant the deadline hits, rather than let the request hang indefinitely — since a new GPS point arrives from the same phone every few seconds anyway, occasionally shedding one under rare, extreme load is a far better trade than a request that never finishes. Separately, our infrastructure is configured with a hard minimum of two running copies of the service at all times (autoscale_min = 2), even at the quietest, lowest-traffic moments — never just one. With only one copy running, anything that takes it down even briefly, like a routine restart during a deploy, would momentarily stop the entire mobile fleet from being able to record its location at all. Running two means the load balancer always has a working copy to send traffic to, even while the other is restarting.

The new state — real numbers, not estimates

Both the old and new code paths sit behind that same load balancer described above, which happens to record exactly how long every request takes. That gives us a genuinely fair, apples-to-apples comparison — not an estimate, not a lab benchmark, but real production response times, before and after, off the same infrastructure.

Three numbers matter here, and they're worth defining plainly:

• p50 — the typical request. Half of all requests were faster than this, half were slower.
• p90 — the slower-than-usual request. Only 10% of requests were slower than this.
• p99 — the worst request that still counts as normal. Only 1% of requests were slower than this — but that 1% is exactly where a real person starts to notice something feels off.

	p50	p90	p99
Before — old code, geo traffic included (avg, Jun 18 – Jul 3)	40 ms	269 ms	467 ms
After — old code, geo traffic removed (avg, Jul 20 – Aug 9)	35 ms	69 ms	384 ms
After — new dedicated Go service (avg, Jul 20 – Aug 9)	11 ms	33 ms	98 ms

The "before" row reflects everything that pool of servers was handling at the time, not a geo-only number — we don't have a breakdown that granular going back that far. But GPS tracking was by far the highest-volume thing that pool handled (the mobile app pings it every few seconds, for every active driver), so it dominates that mix — and the shape of the change lines up exactly with the rollout: the drop starts July 7–8, the day the new service went to 100% of traffic in production, and it happens as a clean step, not a gradual trend.

That step tells two different stories at once:

• The geo path itself got much faster. The typical request time dropped by about 3.6×, the slower-than-usual tail dropped by about 8×, and the worst-case tail dropped by about 4.8×, once it moved off the old shared pool of workers and onto its own dedicated service.
• Everything else got faster too, just from geo leaving. Even looking only at the traffic that stayed on the old code, its own slower-than-usual tail dropped by roughly 4× (269ms → 69ms) the moment geo traffic left. That tells us the geo write path, at that volume, had quietly been eating into the limited pool of workers available to handle every other kind of request on that side of the system, the whole time. Carving it out was as much a win for the old application as it was for the new one.

On footprint: this one endpoint carries the large majority of all traffic our backend receives — every active driver's phone pings it every few seconds, continuously, all day — and today that runs on a small, automatically-scaling fleet of just 2 to 8 lightweight cloud servers, each given a modest, fixed slice of processing power. Before the migration, the equivalent slice of that same traffic rode along inside the old application's much larger server fleet. And this isn't just a theoretical before-and-after: watching that fleet's own running server count over time, its floor held at a steady 29 servers, every single day, through July 7 — then dropped to a hard floor of 20 servers exactly on July 8, the same day the new service went fully live in production — and it's stayed there, flat, for the month since. The daily peak it used to burst up to under load shrank as well, from the 40s–60s down to the high 20s. That's the fleet visibly getting smaller in production, in lockstep with the cutover — not a config value nobody ever revisits.

That difference in footprint shows up directly on the bill — and it's worth pulling real numbers here rather than hand-waving "cheaper." Both fleets happen to run on the exact same family of processor: Amazon's own ARM-based "Graviton" chips. The old web tier runs on m6g.large instances (2 processor cores, 8 GB of memory each); the new geo service runs on Fargate tasks sized to roughly half of that (2 processor cores, 4 GB of memory each). So this is a genuinely fair, same-hardware-family comparison — not an old-generation-chip-vs-new-generation-chip trick. At AWS's public on-demand pricing for those exact instance and task types, in the region we run in: a single m6g.large costs about $56/month; a single geo task, on Fargate, costs about $58/month — nearly identical per-unit pricing, since Fargate carries a small convenience premium over renting the same hardware directly and managing it yourself.

Put the real before/after server counts against those prices, and the old web tier's own bill dropped on its own — before counting the new service at all. 29 always-on servers at ~$56/month each comes to roughly $1,630/month; 20 comes to roughly $1,124/month — a real, measured cut of about $506/month, or 31%, just from the web tier needing fewer machines once geo traffic left it. That's not free money, of course — the new geo service itself now costs something where it used to run as part of the same shared fleet: at its own baseline of 2 tasks, roughly $115/month. Net it out, and the combined baseline bill for both systems together still dropped by roughly $391/month — about 24% — even after fully paying for the new dedicated service on top. Scaled up to a full traffic peak (the old tier's configured ceiling of up to 90 servers, against the new service's up to 8 tasks), the same shape holds: roughly $5,060/month against roughly $460/month for the geo side alone. As with the server-count comparison above, that peak-side figure stacks the entire web tier's bill (which does far more than just geo) against the entire geo service's bill, not a precise per-endpoint attribution — but the baseline number is a real, measured before-and-after on the exact same fleet, and it moved in exactly the direction you'd hope.

Takeaways

The riskiest part of a project like this was never really the new code itself — making sure it produced identical output to the old code, byte for byte, was a problem we solved once, upfront, with automated checks generated straight from the original production code. Every actual incident came from somewhere else entirely: subtle differences in how two different pieces of technology behave by default, under real conditions, that simply don't show up until you're handling real production traffic — one system finishing a job after a client disconnects and the other not, a resource pool that helps up to a point and then starts hurting, a program that doesn't realize how much computer it's actually been given. Rolling traffic over gradually, behind the same load balancer, with an instant one-number rollback, is what let us find every one of those in production, at a small, safe percentage of traffic, instead of all at once in a single high-stakes cutover — and monitoring that didn't simply trust "the request succeeded" as proof the data was actually saved is what would have caught it immediately if any of them had actually cost us real location data.

Securing the Chain: Why Open Source Math Isn’t Enough for Enterprise Trust

Securing your data from attackers using public/private keys is a generally well-understood concept in software engineering. Yet, many engineers—especially those involved in implementing or maintaining complex payment gateways—do not fully grasp the mechanics behind certificate chaining. They often miss how a payload is made tamper-proof using a mixture of multiple public/private key pairs, and the architectural necessity of managing leaked or revoked certificates.

The Ubiquity of the Chain

To start with, certificate chains are used widely across the Internet. When your browser connects securely to a web server via TLS/SSL, the server responds with a certificate chain. Your browser unwraps this chain to determine that the server is legitimate, ensuring you aren't falling victim to a "Man in the Middle" (MITM) attack.

If you receive server-to-server notifications from the Apple App Store or Google Play Store, these platforms send a certificate chain right within the webhook payload for payment-related events. While the webhook payload itself travels as unencrypted plain text, the accompanying certificates ensure the data was genuinely created by the platform store.

Other examples involve signing software packages. A .dmg or .pkg file that you run on your Mac is signed with a cryptographic chain. The operating system will—rightly so—complain if it doesn't find a trusted corporation at the top of that chain (a familiar, and increasingly restrictive, security enhancement on modern macOS).

The Two Pillars of Enterprise Security

This is the perfect lens through which to view how security must be addressed under two distinct pillars when building an enterprise-grade stack:

1. Design: This involves the cryptographic math, the packaging of hashed signatures signed by private keys, and the open-source libraries that provide the raw tools to verify that data has not been tampered with.

2. Policy: This involves operational realities that open-source library developers intentionally leave out. It relates to how your organization securely handles local root certificate stores, OS patches, certificate renewals, and the architecture required for real-time certificate revocation checks.

But first, we need a thorough understanding of how a certificate chain actually achieves tamper-proof safety for your data.

Anatomy of a Webhook Signature

Let's look at a payment event webhook sent to your backend by the Apple App Store. The payload might read: "User 67834342 bought a yearly subscription for Gardening Tips, expires in 2027/04/08".

To send this securely, Apple's servers use a Leaf Certificate to sign the payload.

The Leaf certificate contains a short-lived public key (valid for a few days to a week) backed by a private key kept top-secret inside Apple's infrastructure. Apple takes a one-way cryptographic hash of the payload using a well-known algorithm like SHA-256. This unique "fingerprint" is then signed (encrypted) using the Leaf Private Key to generate the digital signature.

Both the unencrypted payload and the signature are packaged together to be sent to your payments server.

The Missing Link: Trusting the Leaf

If you think about this from the receiving end, you will spot an immediate architectural problem: your server needs the Leaf's public key to verify the signature. Because this Leaf certificate is short-lived and constantly changing, your server cannot know it a priori. Apple must send it to you dynamically.

To do this safely, Apple doesn't just send the Leaf certificate; they send a complete Certificate Chain containing a Leaf cert, an Intermediate cert, and a Root cert at the top. The complete packet hitting your server looks like this:

Using just the Leaf certificate, your server can easily verify the payload's integrity. It extracts the public key from the Leaf certificate, decrypts the signature to reveal the original hash (h1), hashes the plain-text payload locally using SHA-256 to get a second hash (h2), and checks if h1 == h2. If they match, the data hasn't been altered.

But wait. What if an attacker generated their own arbitrary public/private key pair, stuffed their public key into a fake Leaf certificate, signed a fraudulent payload with their private key, and sent it to your endpoint? The math would pass perfectly.

This is exactly why the certificate chain exists.

The Fortress at the Root

Apple maintains a Root Certificate whose private key is kept inside a highly secure, air-gapped vault. This physical machine has no network access and is locked behind multiple biometric access doors. Activating it requires a quorum of trusted employees (Key Custodians) using separate physical smart cards simultaneously. In the rare event that this key must be used, Apple executes a highly scripted, heavily audited "Key Ceremony" witnessed by an external firm, with zero recording devices permitted.

Because the Root private key is protected so fiercely, Apple never uses it to sign daily webhook traffic. Instead, they use it to sign an Intermediate Certificate. That Intermediate Certificate's private key is then used to sign the short-lived Leaf Certificates that handle daily operations.

When Apple sends you the chain, each link is structurally identical. The certificate data (containing its public key and identity claims) is hashed, and that hash is signed by the private key of the certificate immediately above it. To validate the chain, your server works upward: it uses the Intermediate public key to verify the Leaf certificate, and the Root public key to verify the Intermediate certificate.

Each cert is packaged and sent with the same mechanism used to send the payload. (check the first diagram)

Here is a block diagram outlining the complete process:

Bridging the Gap into Policy

You might ask: "What stops an attacker from forging the Root certificate itself and throwing it into the payload header?"

This is where the operating system and internal infrastructure policy come into play. Your payments server should completely ignore any root certificate sent over the wire. Instead, it must validate the chain against a trusted copy of the Apple Root certificate stored locally on your server's filesystem (managed via Linux or OS package management). Root certificates change only once every 20 to 30 years, and these updates are planned and deployed by infrastructure teams well in advance.

Exactly where your OS pulls these trusted roots from, and how it ensures they remain current and secure, are policy concerns that open-source cryptographic libraries deliberately ignore.

The same applies to key compromises. If a private key leaks, Apple will publish its serial number to a global Certificate Revocation List (CRL) or an OCSP responder. However, querying a third-party server over the network on every single webhook adds unacceptable latency to a critical payment endpoint. Resolving this requires architectural trade-offs—such as building local memory caches with strict time-to-live (TTL) boundaries.

The Strategic Takeaway

This operational boundary is the exact gap your Platform Engineering team must bridge to maintain an enterprise-grade software stack.

Proper architectural security design shouldn't be reinvented by every product developer building a new billing feature. Infrastructure leaders must ensure that platform engineers have the deep training required to build secure, reusable middleware modules that abstract these messy details away.

For the rest of your development team, securely consuming third-party webhooks should be as simple as invoking a verified method call and checking its boolean return value.

Moving Beyond the Runbook: How We Solved a Silent Infrastructure Failure Using AI and MCPs

As Head of Engineering, I still take my turn on the weekend on-call rotation. It keeps me grounded in the day-to-day realities of our system, and more importantly, it serves as a continuous feedback loop for our engineering discipline. Every on-call shift is an opportunity for a retrospective—not just to patch a symptom, but to figure out how we can elevate our engineering practice.

This past weekend, an alert tripped. It was an automated check-in monitor flagging a missed run for one of our critical high-frequency background workers.

Historically, this would trigger a tedious, manual fact-finding mission for the on-call engineer. They would open up a dozen browser tabs, cross-reference application log buckets, scan error tracking platforms, and query database replicas to reconstruct the timeline.

This time, I didn't open a terminal to hunt for logs. Instead, I handed the investigation over to an AI development harness.

The Diagnostics Hunt: Reconstructing the Crime Scene

What made this investigation remarkable wasn't just the AI's reasoning; it was the AI's access. By utilizing a suite of targeted Model Context Protocol (MCP) servers, the LLM had a secure, real-time window into our environment. It could query cloud system logs, inspect our error-tracking platform, and safely run read-only queries against our relational database replicas.

The diagnostic process unfolded with a level of rigor that usually takes an engineer 30 to 45 minutes of deep context-switching to achieve:

• The Application Ghost Town: The harness scanned our application error tracker and log groups during the specific 15-minute window of the failure. It found absolutely nothing. No exceptions, no aborted tasks, no fatal errors.
• Checking Business Logic: It opened a secure, read-only tunnel to our database replica to check if a global feature flag or "kill switch" had been toggled to intentionally halt the worker. The flags were all normal.
• Dipping into the Infra Layer: Recognizing that a total lack of application-level footprint meant the code never even started executing, the AI dropped down a layer. It queried the underlying container orchestration API logs.

There, it found the verbatim smoking gun in the container deployment layer response:

"failures": [
  {
    "reason": "RESOURCE:MEMORY",
    "detail": "Insufficient memory available (3072 requested, 763 available)."
  }
]

At the exact second the job was scheduled to fire, a "thundering herd" of seven other scheduled tasks had triggered simultaneously. They dogpiled onto the shared container instances, starving our worker of its required memory allocation. The task failed placement, never launched a container, and silently skipped its cycle.

Pausing for Engineering Discipline

Before jumping to a fix, I asked the harness to pause. In legacy workflows, a common fix for a resource spike is throwing money at it—bumping instance sizes or increasing the auto-scaling headroom.

Instead, we analyzed the root cause. The harness parsed our core configuration templates and mapped out every scheduled task in the system. It discovered that we had a massive scheduling collision at the top of the hour.

Crucially, we asked a vital safety question before changing anything: Do any of these tasks have implicit time-window dependencies tied to the exact top of the hour? If a legacy task relies on a hardcoded clock calculation (like looking back exactly 60 minutes from :00), moving its execution time blindly breaks data integrity.

The AI safely audited the source code of the conflicting tasks, proved they used relative time-deltas or snapped boundaries mathematically, and proposed a zero-cost, staggered cron schedule that completely flattened the concurrency wave.

The Paradigm Shift: The Death of the Static Runbook

This incident highlighted a massive inflection point for our engineering organization.

Right now, we have hundreds of static documentation pages indexed into hundreds of alerts, explicitly telling engineers what steps to take when a specific alert fires. Those guides were written in a pre-LLM era.

Today, those documents are fundamentally obsolete.

Static runbooks are rigid, reactive, and age like milk. What we are moving toward is a paradigm where the on-call engineer doesn't follow a recipe; they pair with an AI harness that has deep, contextual system access via secure MCPs.

This changes the entire shape of engineering culture:

• From Scripting to Systems Thinking: On-call shifts are notorious for producing burnout because engineers spend their time executing repetitive mitigation steps. When the tedious data aggregation is offloaded to a harness, the engineer can focus on system architecture and long-term remediation.
• Empowering the Whole Team: A junior engineer on-call armed with an MCP-enabled harness can safely diagnose complex, multi-layered infrastructure failures that previously required escalating to a principal engineer.
• Standardizing with Guardrails: The next milestone for our leadership team isn't writing more documentation—it is standardizing the safe deployment of these AI tools. We are defining the exact guardrails, permission boundaries, and audit logs required to ensure that our team can use MCPs securely across the infrastructure.

We aren't just adopting an AI tool to fix a cron job. We are completely rewriting how we train, empower, and scale our engineering talent to build a more resilient organization.

The Pragmatic FinOps Playbook: How We Slashed Our Cloud Footprint by 50% Without Vendor Overhead

When a scaling company realizes its cloud footprint has gotten bloated, the corporate reflex is entirely predictable: purchase an enterprise FinOps dashboard, lock into a multi-year subscription, or hire an outside autonomous rate-optimization vendor.

But those paths introduce a quiet tax of their own. Dedicated third-party cost optimization vendors generally use two aggressive pricing models that create massive overhead:

1. The “Percentage of Savings” Model (The Contingency Fee)

Platforms like nOps (specifically their autonomous rate and commitment management programs) and tools like CloudFix often use a ShareSave / contingency model.

The Cost: They take anywhere from 15% to 25% of the verified savings they deliver.

The Reality: If an organization successfully optimizes its infrastructure to save $10,000 every month, a vendor taking a standard 20% cut will invoice that company $2,000 every single month indefinitely—just to keep those optimization toggles turned on.

2. The Enterprise Tier / Fixed Subscription Model

For visibility, governance, and SaaS/Kubernetes-heavy cost-tracking (like Finout, CloudZero, or Cloudability), pricing scales directly with your total monthly cloud spend.

The Cost: For an environment with a mid-market or scaling cloud bill, subscriptions typically start at a baseline of $1,500 to $3,000+ per month ($18K to $36K+ annualized), often locked into rigid 12-to-36-month contracts.

The Reality: These tools only provide dashboards and visibility. They point at the problem but still require your internal engineering team to manually execute and maintain the actual fixes.

The Hidden Costs of Partnering with Outside Vendors

Beyond the software licensing or contingency fees, bringing in an outside optimization vendor or consulting team introduces massive operational friction:

• The Integration & Security Tax: To get started, you have to grant deep, sweeping IAM access permissions to your core cloud environments, configure complex billing exports, and clear lengthy internal security reviews.
• The Timeline Lag: Onboarding a vendor, running their “complimentary analysis phase,” and sitting through endless alignment calls easily burns 2 to 4 weeks before a single line of infrastructure is actually changed.
• The “Context Blindness” Bottleneck: Automated platforms excel at macro-level rate optimization (like automated buying of Savings Plans), but they are completely blind to your unique architecture. An automated third-party tool can never engineer a nuanced, context-aware workaround. It applies generic, blunt rules that don't capture your real product constraints.

Recently, we chose a different path. We treated our infrastructure bill not as an administrative hurdle or a vendor procurement task, but as a strict data-engineering and context problem. By exporting our comprehensive cloud usage data and feeding it to an LLM prompted to act as a ruthless cloud economist, we armed a lean, internal engineering sprint with high-signal analysis.

We didn't spend weeks signing contracts or sacrificing a chunk of our margins. Because our internal engineers owned both the data and the business context, we bypassed the vendor pipeline entirely and executed immediate wins instantly. With zero production downtime and a few targeted adjustments, we cut our total cloud footprint by roughly 50%—keeping 100% of the financial upside within our business from day one.

Here is the exact playbook of where the AI acted as a scout, and where human engineering judgment took over.

Cost by service, pre-tax (exact dollar figures blurred). Relational Database Service was the single largest line item on our bill, followed by CloudWatch and Elastic Compute Cloud.

1. Database Storage Evolution: The Zero-Downtime Migration (RDS)

Our single largest cost driver was database infrastructure, devouring nearly 48% of our entire cloud footprint. In the early stages of a platform, over-provisioning storage to guarantee absolute performance is a common and entirely defensible trade-off. However, maintaining legacy, premium tiers past their necessary shelf life is pure waste.

The AI scout flagged an immediate operational optimization: our database clusters were still utilizing legacy provisioned IOPS SSD storage (io1). Historically, engineering teams hesitate to touch database storage layers out of a deeply ingrained fear of maintenance windows, indexing bottlenecks, or catastrophic production downtime. We hadn't done this earlier because we believed it would require an operational maintenance window.

But modern cloud infrastructure has quietly evolved.

By transitioning our relational database instances from legacy io1 storage configurations to modern, general-purpose gp3 volumes, we unlocked massive savings. The setup was instant, the cutover required zero maintenance windows, and it resulted in absolutely zero downtime for our live application.

The Lesson: Some of the highest-leverage cost savings are gated behind legacy infrastructure assumptions we simply haven't re-tested recently.

2. Observability Rightsizing: Balancing Analytics with Cost (CloudWatch)

Observability is essential, but unmanaged log retention is an exponential cost trap. Our billing analysis surfaced an unsustainable volume spike originating from CloudWatch Logs Insights queries and log group retention.

A third-party, automated tool would have been entirely blind to our operational context; it would have simply recommended a sweeping, blunt truncation of our logs to save money. This is exactly where human engineering context became mandatory. A blind reduction in retention strips downstream product and analytics teams of historical operational context.

Instead of executing a simple truncation or undertaking a complex, multi-week archival pipeline project, our engineers designed a pragmatic compromise:

• High-Volume Retention Cut: We aggressively reduced the retention windows of 5 high-throughput log groups down to exactly one week. These groups generated massive noise but held minimal long-term analytical value.
• Analytical Preservation: We maintained a longer two-month retention window for the 2 core log groups critical to our team's regular operational analysis.
• Query Indexing Optimization: Our engineers knew exactly which two log groups were critical to analytics, so they built dedicated indices explicitly tailored to the regular, repetitive queries the team actually uses.

Notably, this hybrid strategy did not come from an automated AI suggestion—it was engineered entirely by our team. This custom compromise allowed us to completely bridge the gap between cost and product velocity, bypassed a heavy data pipeline project, and still slashed total CloudWatch costs by a staggering 63%.

3. Traffic Architecture: Eliminating the Silent NAT Gateway Tax

One of the most insidious line items on any modern enterprise cloud bill is data transfer. Our data analysis surfaced an immense cost signature originating from NAT Gateways processing massive volumes of outbound public traffic. The root cause was systemic: internal application clusters were routing heavy, recurring data reads through public avenues.

Specifically, our services perform highly frequent, large-scale data reads from Amazon S3 buckets. Under the default VPC configuration, these requests travel out through the NAT Gateway across the public internet to reach S3, racking up steep data processing fees on every single gigabyte transferred.

We systematically replaced these costly routes by deploying Amazon VPC Gateway Endpoints for S3. This architectural change rerouted our high-frequency data pipeline directly through the internal AWS network routing fabric, entirely bypassing the NAT Gateways. The application traffic never changed, data latency decreased, and the processing costs dropped off a cliff.

The Executive Summary

Optimization Area	The Action Taken	The In-House Result
Database (RDS)	Migrated legacy io1 to modern gp3 storage	Instant setup, 0 downtime, immediate cost reduction
Observability (CloudWatch)	Truncated 5 groups; indexed 2 core groups	63% cost reduction with zero impact on analytics data
Networking (VPC)	Deployed VPC Gateway Endpoints for S3 reads	Eliminated massive NAT Gateway data processing fees

The right organizational model for modern infrastructure management is AI as the scout, and the engineer as the decision-maker.

Infrastructure cost optimization is fundamentally a leadership challenge, not a software procurement challenge. If you rely purely on automated tools or third-party vendors, you will either implement generic changes that disrupt your team's velocity or miss context-specific architectural fixes entirely. Real operational leverage occurs when you equip talented, internal engineers with high-signal analysis, trust their internal product context, and empower them to build pragmatic, metric-driven solutions.

Engineering Notes  ·  Go  ·  GenAI  ·  Infrastructure

How Claude + Gemini Gave Me a Crash Course in Go, H3 Hexagons, and Cloud Infrastructure — In Under 3 Hours

Thushara Wijeratna  ·  June 2026

There’s a moment in any engineering project where you hit a wall that isn’t really a technical wall — it’s a knowledge wall. You know what you want to build. You can see the shape of the solution. But the specific language, library, or paradigm you need is outside your comfort zone, and the time cost of getting up to speed feels prohibitive.

I hit exactly that wall recently. And I got through it in about two and a half hours, with a working solution, a complete deployment pipeline, and a meaningful education. Here’s how.

---

The Background: A Cluster Migration and an Unexpected Opportunity

We’re in the middle of migrating a geo-processing cluster from Rails to Go. The new cluster is more efficient by an order of magnitude, but it’s barely past proof-of-concept — just enough tested to know it works, not enough to bet production on. It runs on ECS and has access to our Redis Time Series infrastructure.

Then a separate project came up: mapping driver density into H3 hexagons. The idea is to take raw GPS driving data and visualize where drivers are concentrated — a spatial density problem. It’s the kind of analysis that screams for Python or Go. Not Ruby. Definitely not Rails.

The problem: I barely understand Go. And while Python is friendlier territory, the natural home for this work was the new Go cluster I was already running — it had the right access to the time series data, and using it here would be a meaningful first real exercise for the cluster.

So I had a choice: spend days self-teaching Go spatial libraries, Docker packaging, ECS task definitions, and IAM policies — or ask for help. I asked for help.

---

Phase 1: Claude Builds the Foundation

I opened a conversation with Claude and described what I was after: take a sample of driving data, bin the GPS coordinates into H3 hexagons at an appropriate resolution, and produce a density visualization. I explained the Go context and asked for a working implementation.

What I got back wasn’t just code. It was reasoned code. Claude walked through the H3 resolution tradeoffs (too fine and you get sparse, noisy hexagons; too coarse and you lose geographic meaning), chose a sensible default, and produced a complete Go program that:

• Parsed the GPS data
• Mapped each coordinate to its H3 cell index
• Counted occurrences per hexagon
• Exported the density data ready for visualization

The code compiled. The logic made sense when I read it. When I hit small integration issues, Claude adjusted immediately when I fed back the errors.

But the analysis code was only part of the problem.

---

Phase 1b: The Infrastructure Claude Also Built

This is the part of the story I want to make sure doesn’t get lost — because it’s where the time savings were arguably even more dramatic.

To run this Go script in the new cluster, I needed a complete infrastructure stack built from scratch:

A Dockerized Go application. The script had to be packaged as a container, with the right base image, dependency management, and build configuration to run cleanly inside ECS. Claude produced a working Dockerfile and explained each layer.

An ECS Task Definition. The container needed to be provisioned as a task in the new geo cluster — with the right CPU/memory allocation, environment variables, logging configuration, and networking setup to reach the Redis Time Series instance. This meant understanding how ECS task definitions are structured and how they interact with the surrounding infrastructure.

IAM roles and policies. The tricky part: the driving data lives in a locked-down production environment. To process it safely, the task needed read access to an S3 bucket containing that data, without being given any broader permissions it didn’t need. Claude helped design the IAM policy with least-privilege principles — specific bucket ARNs, specific actions, nothing more. It also helped reason through the trust relationship between the ECS task execution role and the task role itself, which is a subtle distinction that trips up a lot of people.

S3 bucket configuration. A staging bucket was needed to land the processed data safely — isolated from production, with appropriate access controls so the analysis job could write results without touching anything it shouldn’t.

What I got wasn’t just “here’s a Dockerfile.” It was a coherent infrastructure design: here’s how the pieces connect, here’s why we’re scoping the IAM policy this way, here’s what would go wrong if we didn’t. The guardrails weren’t an afterthought — they were built in from the start.

By the end of that session, I had a traffic density plot from real driving data, running inside the new cluster, reading from production data safely through a well-scoped IAM boundary, writing results to an isolated S3 bucket. Working. Deployed. Safe.

If the story ended there, it would already be remarkable. A non-Go developer, unfamiliar with ECS task definitions and spatial indexing libraries, with a working, deployed pipeline in a couple of hours. That’s the headline.

But the story didn’t end there.

Kepler.gl rendering of H3 resolution-5 driver density over the Dallas–Fort Worth metro. The yellow hexagon (Garland/East Dallas) logged 1,856,072 trips in the sample — the output of the Go pipeline built in this session.

---

Phase 2: Gemini Pressure-Tests and Extends the Vision

I took the Claude-generated solution to Gemini — not to debug it, but to challenge it. I wanted to know: is this approach actually good? And what would make it better?

This is a workflow pattern worth naming explicitly: use one AI to build, use another to audit and extend. Different models have different strengths, different training emphases, different tendencies in how they approach problems. Running your solution past a second model is a cheap form of peer review.

Gemini’s response surprised me. It didn’t just validate the approach — it pushed it forward. The suggestions fell into two categories:

1. Statistical enrichment at scale. The density map I’d built was a point-in-time snapshot. Gemini pointed out that with more data, you could extract genuinely useful statistics: density gradients, hotspot persistence over time, anomaly detection for unusual clustering patterns. These aren’t just nice-to-haves — for a driver dispatch system, they’re operational intelligence.

2. Redis Time Series as the efficient backbone. Here’s where it got interesting. Gemini connected the density problem to infrastructure I already had running in the cluster: Redis Time Series. With compaction rules, you can store high-frequency GPS events efficiently and roll them up into time-windowed density summaries without blowing out memory or query latency. The density map becomes a living view into an efficiently maintained time series, not a batch job.

This wasn’t something I would have reached on my own in a single session. It required knowing that Redis Time Series could do this, understanding how compaction rules work, and seeing the connection between spatial density and temporal aggregation. Gemini handed me all three in one exchange.

---

What I Actually Learned

The output of this session wasn’t just a working program. It was an accelerated education across several domains I wasn’t strong in:

Go development — not just syntax, but idiomatic patterns for data processing pipelines, error handling, and module organization. Reading and debugging Claude’s code taught me more about Go in two hours than I’d absorbed from documentation in weeks.

H3 spatial indexing — the geometry of hexagonal hierarchical indexing, resolution tradeoffs, and why H3 is better suited for density mapping than geohash or simple grid approaches.

ECS + IAM design — how to structure a task definition, how to reason about least-privilege IAM policies for ECS workloads, and how to isolate production data access safely. This alone would have taken a full day to research and implement from scratch.

Stats at scale — how to think about density not as a static count but as a statistical distribution with temporal dimensions, and how to maintain that data efficiently in a system you’re already running.

This is the thing I keep coming back to: Gen AI isn’t just a code generator. Used well, it’s a compressed curriculum. You get the code, but you also get the reasoning, the tradeoffs, the “here’s why this approach and not that one.” If you pay attention, you learn.

---

The Workflow, Distilled

1. Bring a concrete problem, not an abstract question. “Build me a density map of GPS data in Go using H3, packaged for ECS, reading from S3 with a scoped IAM role” is better than “how do I do spatial analysis in Go.” Specificity gets specific answers.

2. Don’t just run the code — read it. The code is the lesson. When something compiles and runs, go back and understand why each piece is there. Ask the AI to explain sections that aren’t clear. This is where the actual learning happens.

3. Use a second model as a peer reviewer. After you have something working, bring it to a different model and ask: “Is this approach sound? What am I missing? How would this scale?” You’ll get different angles, different critiques, different extensions.

4. Follow the threads. Gemini’s Redis Time Series suggestion wasn’t something I asked for. It emerged from a conversation about making the approach more robust. Let the conversation go where it wants to go — some of the best insights come from tangents.

---

A Note on What This Isn’t

This isn’t a story about AI replacing engineering judgment. The H3 resolution I used, the decision to pursue Redis Time Series compaction, the choice of which statistics actually matter for driver dispatch, the architecture of which IAM boundaries made sense given our production setup — those required domain understanding that I brought to the table. The AI provided implementation and expanded my knowledge of available tools. The judgment about what to do with those tools remained mine.

That division of labor — AI handles implementation and education, engineer handles domain judgment and architectural decisions — is the productive pattern. Collapse it in either direction and you lose something important.

---

Closing Thought

Two and a half hours. One working density visualization. A complete Docker + ECS + IAM + S3 deployment pipeline. A crash course in Go, spatial indexing, cloud infrastructure, and time-series-backed analytics. A roadmap for how to make the analysis genuinely production-grade.

The combination of Claude for building and Gemini for pressure-testing and extending isn’t a fluke. It’s a repeatable workflow for the class of problems where you know what you want but don’t yet have the specific technical fluency to get there alone.

The 10x output claim isn’t about volume. It’s about the ratio of what you can accomplish to what your current knowledge would otherwise limit you to. That ratio, in the right problem, can be extraordinary.

---

The Claude transcript showing how the Go/H3 solution and infrastructure were built is available here. The Gemini conversation on statistical extensions and Redis Time Series is here.

Shipping with Claude: What a Production Incident Taught Us About LLMs and Engineering Fundamentals

We've been migrating our Rails backend from encrypted credentials files to Chamber — a more operationally flexible approach to secret management that lets us rotate and audit secrets. It was a deliberate, multi-PR migration, and we used Claude as a pairing partner throughout. The migration went smoothly right up until it didn't. Here's what happened, what we missed, and what we now see in our own codebase as a result.

---

Why we migrated in the first place

Rails encrypted credentials solve a real problem: secrets don't live in plaintext on disk. But they introduce a different problem that only becomes painful at scale — you cannot review changes to them.

When a developer updated a credential, the PR diff looked like this:

config/credentials/production.yml.enc
@@ -1 +1 @@
-UC84OmUIw4yscrhH+RHdY1F+FIodrBqcr9dawCAzpcqU3bmUGre898PmiOzr61s8mRbDGbgesoDn6RDX38r
-tdSKI8y544h8jLcEKx2cKkPN9wchYS/nVH1ONPVbZFaAg9wSeNjOuLONPImiKFcFLaWPJH32MP0v4R5YP
-uatJ3alZ9l40CjqUm0c5c/9O+jd7EcDuwzl/X/3WuZ93z1ylJ1cp8oKcnsOq39MJNj3DK48rymsuqvgy5
-CgEMv0QgxCWuRb7Ss61f/vV3VBxoyXPtLghnapvUQcqdJXj5VHSrwzZGoBKjB64Aw7+frcy4pHJ6p7CMd
-0advbhFD5hhfkJkFmOoHJz1RXMYRLgBcSAv9vAOpAqGct/1FPudP6ZNgYm/YbTp/MrxllgEqI+L3u1OnL
+mRbDGbgesoDn6RDX38rtdSKI8UC84OmUIw4yscrhH+RHdY1F+FIodrBqcr9dawCAzpcqU3bmUGre898Pmi
+Ozr61y544h8jLcEKx2cKkPN9wchYS/nVH1ONPVbZFaAg9wSeNjOuLONPImiKFcFLaWPJH32MP0v4R5YPu
+atJ3alZ9l40CjqUm0c5c/9O+jd7EcDuwzl/X/3WuZ93z1ylJ1cp8oKcnsOq39MJNj3DK48rymsuqvgy5C
+gEMv0QgxCWuRb7Ss61f/vV3VBxoyXPtLghnapvUQcqdJXj5VHSrwzZGoBKjB64Aw7+frcy4pHJ6p7CMd0
+advbhFD5hhfkJkFmOoHJz1RXMYRLgBcSAv9vAOpAqGct/1FPudP6ZNgYm/YbTp/MrxllgEqI+L3u1OnL9
 [... 7KB ...]

One line in, one line out. The entire file is a single blob of ciphertext that gets re-encrypted every time any value inside changes. A reviewer looking at this diff cannot tell:

• Which key was added, removed, or modified
• Whether the change applied to the right environment
• Whether an existing key was accidentally dropped during re-encryption
• Whether the value being set is structurally correct

The PR description is the only source of truth, and only as trustworthy as the author's summary. Approving a credentials change is an act of faith, not review.

It compounds further when two engineers touch unrelated keys in the same file. Because the entire file re-encrypts as a single blob, any two concurrent changes produce a merge conflict that is literally unresolvable without the master key — and even then, the "conflict" is invisible at the diff level. There is no way to see whose change is present, whose was lost, or whether both survived.

Compare that to the equivalent change expressed as a Chamber migration — moving Twilio's credentials out of production.yml.enc and into config/settings/twilio.yml:

# config/environments/production.rb
   config.twilio = {
-    verification_service_id: ENV['TWILIO_VERIFICATION_SERVICE_ID'] ||
-        Rails.application.credentials.config.dig(:twilio, :verification_service_id),
-    account_sid: ENV['TWILIO_ACCOUNT_SID'] ||
-        Rails.application.credentials.config.dig(:twilio, :account_sid),
-    auth_token: ENV['TWILIO_AUTH_TOKEN'] ||
-        Rails.application.credentials.config.dig(:twilio, :auth_token)
+    verification_service_id: ENV['TWILIO_VERIFICATION_SERVICE_ID'] || Chamber.dig(:twilio, :verification_service_id),
+    account_sid:              ENV['TWILIO_ACCOUNT_SID']             || Chamber.dig(:twilio, :account_sid),
+    auth_token:               ENV['TWILIO_AUTH_TOKEN']              || Chamber.dig(:twilio, :auth_token)
   }

# config/settings/twilio.yml (new file)
+default:
+  twilio: &default
+    _secure_verification_service_id: qcex...
+    _secure_account_sid:             HXeg...
+    _secure_auth_token:              pPZA...
+
+development:
+  twilio:
+    _secure_verification_service_id: vZtC...
+    _secure_account_sid:             gwhj...
+    _secure_auth_token:              UZQo...
+
+staging:
+  twilio:
+    _secure_verification_service_id: OH7L...
+    _secure_account_sid:             CTwp...
+    _secure_auth_token:              ajul...
+
+production:
+  twilio:
+    _secure_verification_service_id: TTFX...
+    _secure_account_sid:             IcZ1...
+    _secure_auth_token:              Axyj...

The values are still encrypted — nobody can read the secrets from the diff. But a reviewer can now verify that all three keys exist for all four environments, that production has its own separate values, that the application code reads them in the right order (ENV override → Chamber), and that no key was silently dropped. That's a reviewable change.

That was the core motivation. Secret management you can actually audit.

---

The migration

Pave runs a Rails 8.0 API backend. The migration happened across several PRs, moving one integration at a time: Stripe, Redis, Twilio, Braze, the database connection URLs, and finally the active_record encryption keys. Each followed the same pattern. Before:

Stripe.api_key = Rails.application.credentials.dig(:stripe, :api_key)

After:

# Chamber has no decryption key (CHAMBER_KEY) in test and raises on access;
# skip it there (Stripe is stubbed in specs).
Stripe.api_key = ENV['STRIPE_API_KEY'] || (Rails.env.test? ? nil : Chamber.dig(:stripe, :api_key))

The Rails.env.test? ? nil : Chamber.dig(...) guard was necessary: test and CI environments don't have a CHAMBER_KEY, so calling Chamber.dig at boot raises Chamber::Errors::DecryptionFailure. Every initializer got this guard. Claude co-authored most of this work. The PRs were clean, the pattern was consistent, and the earlier changes deployed without incident.

---

The last PR

PR #7245 was the finish line: retire the encrypted credentials files entirely. It migrated the database connection URLs and active_record.encryption keys to Chamber, then deleted production.yml.enc, staging.yml.enc, and feature.yml.enc. Reviewed, merged.

Fifty-three minutes later, we had 431 ArgumentError: Missing master key errors in production, all coming from Api::V1::UserEventsController#create.

The culprit was this line in the controller:

current_user&.user_setting&.update!(user_ip: request.remote_ip) if current_user&.user_setting.present?

user_ip is a Lockbox-encrypted attribute on UserSetting:

class UserSetting < ApplicationRecord
  has_encrypted :device_ip
  has_encrypted :user_ip

  belongs_to :user
end

Lockbox's default key lookup, when no explicit initializer sets the key, falls through to Rails.application.credentials.lockbox[:master_key]. There was no config/initializers/lockbox.rb in the codebase — Lockbox had been reading from credentials silently, by convention, the whole time. When we deleted the credentials files, that implicit dependency snapped.

PR #7285 — the hotfix — added the missing initializer:

Lockbox.master_key = Rails.env.test? ? nil : Chamber.dig(:lockbox, :master_key)

Done in minutes. But in the pressure of the moment, this used the same guard pattern that was already everywhere. And that guard quietly created a new problem.

---

What Claude missed

Claude co-authored PR #7245. The PR summary noted that "Development and test environments are unaffected — they use local DB config with no Chamber calls." That was true for the database config. But neither Claude nor the human reviewer connected that observation to Lockbox, which had no explicit initializer and therefore no Chamber call that anyone could see to guard.

This is worth saying plainly: Claude didn't catch it. Nothing in the diff was wrong. The deletions of the .yml.enc files were the stated goal. There was no failing test, no linting rule, no static analysis warning that could have surfaced an implicit gem-level dependency on a file that was about to disappear.

We're not saying this to criticise the tool — we kept using it through the incident and the subsequent fix, and it was genuinely helpful. We're saying it because the failure mode matters: LLMs reason about what's in the diff and the context window. Implicit dependencies — framework defaults, convention-over-configuration gem behaviours, transitive lookup chains that were never written down — are precisely what an LLM is likely to miss. The Lockbox key was never written anywhere in the code we touched. That's exactly why it wasn't caught.

The practical upshot: treat Claude like a very fast, very capable engineer who hasn't yet internalised your codebase's hidden contracts. Pair it with the kind of review that asks "what else reads from this file?" before deleting it.

---

The central lesson: test bypasses that look like pragmatism

Here is the line from the hotfix that made the incident survivable but embedded the longer-term problem:

Lockbox.master_key = Rails.env.test? ? nil : Chamber.dig(:lockbox, :master_key)

Look at it from a test's point of view. With nil as the master key, Lockbox doesn't raise — it silently skips encryption. Attributes get stored as plaintext in their _ciphertext columns and read back the same way. Every test that touches an encrypted field passes. And we already had a request spec that appeared to cover this code path:

context 'when valid event is passed' do
  let(:params) { { event_name: 'user_sign_up', session_id: session_id } }

  it { is_expected.to be 200 }

  it 'saves the user ip in user settings' do
    expect(user.reload.user_ip).to be
  end
end

This test passed before the incident (Lockbox stored user_ip as plaintext, readable). It passed after the hotfix (same). It would pass today if the production key were accidentally set to nil. The test was measuring persistence, not encryption. With a nil master key, those two things are indistinguishable.

---

Why engineers write this code

It is easy to read that guard and think "obvious mistake." It is harder to explain why experienced engineers keep writing it. There are two honest reasons.

Time pressure is the visible one. Under incident pressure — hundreds of errors per minute in production — the path to fixing the immediate breakage is all that matters. The hotfix author added the guard because every other Chamber initializer in the codebase already had it. Consistency under pressure is not irrational. But each time the pattern is copied without questioning it, the next gap becomes slightly harder to see.

Complexity and incomplete mental models are the more insidious reason, and the one that actually applied here. PR #7245 was not written under incident pressure. It was a planned migration, reviewed carefully. The nil guard was added because the engineer genuinely did not know that Lockbox was reading from credentials at all — there was no explicit initializer, no comment, nothing to grep for. When you don't know what a subsystem depends on, you don't know what your test environment is silently bypassing.

This is the more dangerous case. The engineer is not cutting corners. They believe nil is equivalent to a key for test purposes — that "test doesn't need this" is a true statement. For some things it is. For an encryption key, it is not: nil doesn't mean "use a test key," it means "skip the encryption entirely."

---

Not all test forks are the same

The broader audit this incident prompted surfaced several Rails.env.test? forks in our initializers. They are not all the same problem.

# devise.rb
config.stretches = Rails.env.test? ? 1 : 12

# 1_redis.rb
$redis_aws = Rails.env.test? ? Test::MockRedisEnhanced.new : Redis.new(url: Chamber.dig(:redis, :redis_aws_url))

# stripe.rb
Stripe.api_key = ENV['STRIPE_API_KEY'] || (Rails.env.test? ? nil : Chamber.dig(:stripe, :api_key))

# lockbox.rb (before fix)
Lockbox.master_key = Rails.env.test? ? nil : Chamber.dig(:lockbox, :master_key)

The useful question is: does the test code path exercise equivalent behaviour to production?

Bcrypt stretches — acceptable. bcrypt with 1 stretch instead of 12 is a well-documented Rails practice. The hash is still computed; the algorithm is identical. The fork reduces test runtime by several seconds without changing what is being tested.

MockRedis — defensible. MockRedisEnhanced implements the Redis command interface in memory. Tests verify the same application logic; they just don't make network calls. The gap is that MockRedis and Redis are not identical in every edge case, but for the operations under test the equivalence holds. Network isolation is a legitimate reason to use a test double.

Stripe nil key — ambiguous. Stripe calls in tests are all stubbed with VCR cassettes or allow/receive mocks, so the nil key never causes a real API call. The nil is effectively inert. But there's also no spec asserting that the Stripe configuration itself is valid — so if someone misconfigures the Chamber key, nothing in the test suite would catch it before a real charge fails in production.

Lockbox nil key — not acceptable. nil does not replace an encryption key. It changes what the application does: encrypt-and-store becomes store-as-plaintext. A test that passes with nil is not testing Lockbox at all.

The decision rule: if nil changes behaviour rather than just routing around infrastructure, the fork is hiding a gap. A mock Redis or a stubbed HTTP client exercises the same logic with a different transport. A nil encryption key turns off the encryption logic entirely.

---

The fix pattern

The Lockbox fix sidesteps Rails.env.test? entirely. Rails environment files load before initializers, so we set an env var in test.rb — right alongside the existing active_record.encryption test keys:

# config/environments/test.rb

config.active_record.encryption.primary_key         = 'ZEjSqIThOjtppHyOdsPZzReeEhUmG1mH'
config.active_record.encryption.deterministic_key   = 'lv4L60ar5hfA9fZ9U33W6YQ3RvFuDxCm'
config.active_record.encryption.key_derivation_salt = 'Io4oojz3zpY2KoIxdIZmWJlh9CUtwf7y'
ENV["LOCKBOX_MASTER_KEY"] ||= "0" * 64  # real dummy key — exercises actual Lockbox code path

The initializer becomes:

# config/initializers/lockbox.rb

Lockbox.master_key = ENV["LOCKBOX_MASTER_KEY"] || Chamber.dig(:lockbox, :master_key)

No Rails.env.test?. In test, the env var provides a real (dummy) key and Chamber.dig is never called. In production, the env var is not set, so Chamber provides the real key. The same code path executes in every environment.

The companion spec makes future encryption bypass detectable:

# spec/models/user_setting_spec.rb

describe "Lockbox encrypted attributes" do
  let(:user) { create(:user) }

  it "encrypts user_ip at rest and round-trips correctly" do
    setting = user.user_setting
    setting.update!(user_ip: "1.2.3.4")
    expect(setting.user_ip_ciphertext).not_to eq("1.2.3.4")  # proves encryption ran
    expect(setting.reload.user_ip).to eq("1.2.3.4")           # proves decryption works
  end

  it "encrypts device_ip at rest and round-trips correctly" do
    setting = user.user_setting
    setting.update!(device_ip: "5.6.7.8")
    expect(setting.device_ip_ciphertext).not_to eq("5.6.7.8")
    expect(setting.reload.device_ip).to eq("5.6.7.8")
  end
end

The first assertion in each example is the one that would have caught the original incident. With a nil master key and plaintext storage, user_ip_ciphertext equals "1.2.3.4" and the spec fails. With a real key, the ciphertext is unreadable binary — proof that Lockbox actually ran.

---

The audit

The incident prompted us to grep our initializer directory systematically:

grep -rn "Rails\.env\.test?" config/initializers/

Results:

config/initializers/stripe.rb:2:   Stripe.api_key = ENV['STRIPE_API_KEY'] || (Rails.env.test? ? nil : Chamber.dig(:stripe, :api_key))
config/initializers/sidekiq.rb:10: if Rails.env.test? # redis is mocked in test anyway
config/initializers/devise.rb:127: config.stretches = Rails.env.test? ? 1 : 12
config/initializers/1_redis.rb:5:  $redis_general = Rails.env.test? ? MockRedis.new : Redis.new(...)
config/initializers/1_redis.rb:6:  $redis_auth    = Rails.env.test? ? MockRedis.new : Redis.new(...)
config/initializers/1_redis.rb:7:  $redis_aws     = Rails.env.test? ? Test::MockRedisEnhanced.new : Redis.new(...)
config/initializers/lockbox.rb:1:  Lockbox.master_key = ... # fixed

For each one, we're applying the same question: what is this test actually asserting, and is the answer "that the real thing works" or "that nil doesn't crash"? Where the answer is the latter, we apply the fix pattern — give test a real dummy value, keep the same initializer logic, add an assertion that proves the mechanism ran.

The goal is not to make every test environment identical to production. Tests need to be fast and isolated. The goal is narrower: when a test passes, it should be evidence that the production code path ran — not evidence that nil is harmless.

---

What we're taking forward

Explicit over implicit, always. Lockbox worked for months without anyone explicitly configuring it. That was a liability. Every gem that touches security should have an explicit initializer that makes its key source visible. If you cannot grep for where the key comes from, someone will delete it.

nil is not a test double. A mock provides equivalent behaviour through a different mechanism. A nil disables the behaviour. The difference is a test that gives you confidence versus a test that gives you false confidence.

Complexity is a more dangerous bypass trigger than time pressure. Time pressure is visible — everyone knows the engineer is cutting corners. Incomplete mental models are invisible — the engineer genuinely believes nil is equivalent. Standard review asks "does this look right?" The review this incident called for is "what does nil actually do to this subsystem?" That needs to be a habitual question, not a post-mortem one.

LLMs shift the bottleneck without removing it. Claude helped us write and migrate code faster throughout this project. What it did not do is hold the complete mental model of the system — the knowledge that a particular gem reads from credentials by convention, that deleting a file might snap a dependency that was never written down. That knowledge lives in engineers who have read the source, survived the previous incident, or thought carefully enough to ask the right question before merging. Building faster with AI makes that judgment more valuable, not less.

The Chamber migration is complete. The audit is underway. And we have a new heuristic for every initializer we write from here: if test gets nil where production gets a real value, the test is probably not testing what you think it is.

---

Thushara Wijeratna, WorkSolo Engineering