A service shouldn’t need a platform expert to get a canary, an analysis gate, and automatic rollback. IKS AIR treats operations as done-for-you: those best practices and guardrails are on by default, and progressive delivery with Argo Rollouts is how that shows up on every deploy.
That’s the third piece of IKS AIR, after the application spec and managed autoscaling. When developers can’t see the underlying machinery, the platform has to earn trust with real safety nets. We’ll end on where that goes next: Agentic Operations.
I work on the platform engineering team at Intuit. This post is reposted, with light edits, from the Intuit engineering blog, co-written with my colleague Avni Sharma.
In Part 1 we took away the YAML. In Part 2 we showed how IPA, the Intelligent Pod Autoscaler, sizes the service from real load. This post is what happens when it’s time to ship new code, or debug a pod you can’t exec into.
Progressive delivery, out of the box#
Pushing code directly to production is risky. So the paved path builds progressive delivery in with Argo Rollouts.
The canary#
Production starts the new version at 10% of live traffic, then 30, 50, 80, 100, with five minutes between steps. A healthy deploy takes about twenty minutes. Earlier environments move faster: 25, 50, 100, about ten. Some skip the canary entirely so you can iterate.
The analysis gate#
Where the canary does run, Argo Rollouts queries HTTP 5xx every sixty seconds. Cross 4% and the rollout fails: traffic goes back to the last stable revision. Production waits for three failed samples before it aborts, so a one-minute blip doesn’t undo the deploy. Pre-production aborts on one. Mean time to recovery from a bad deploy drops from however long it takes someone to notice and react, to however long the analysis gate takes to trip.
The default gate is deliberately simple. Error rate is easy to reason about and hard to get wrong. It also misses real regressions: a canary that’s still returning 200s while p99 triples, or memory climbing toward an OOM. Teams that want that broader signal can opt into extra analysis templates (latency, CPU, memory, pod restarts) and the rollback still trips the same way. There’s also an AIOps path, built on Numalogic (the Apache 2.0 anomaly-detection library from Numaproj, a project Intuit originated and contributes to), which scores the canary against the service’s own historical behavior instead of a fixed threshold. Rollback works identically; only the signal that trips it changes.
Who owns the config#
Those weights and thresholds are platform-managed. You don’t retune them per service the way teams used to retune HPA. On the old kustomize path, analysis lived in each team’s Rollout-patch.yaml (often four to seven templates, each with its own thresholds). That was power. It was also another of those 47 infrastructure commits. AIR generates the templates; the paved path uses error rate.
Rollback after promote#
Once a version is fully promoted, the portal still allows user-initiated rollback to a prior revision. That’s the human path for “this made it through canary and we still don’t want it in production.” The automatic path is the one that fires during the canary, without waiting for that click.
Debugging the black box#
A common concern with abstractions is that they become a black box when things go wrong. How do you debug a service when you don’t have access to the underlying container or namespace? This ended up being a bigger problem than we originally scoped in 2022. We hadn’t anticipated that our application containers would also become hardened enough to lack basic tools, sometimes not even a shell, which meant “debug without knowing your cluster” needed more than a good dashboard.
Debug shell#
To get around those hardened, shell-less containers, we provide a debug shell that launches a temporary, ephemeral container within the same pod. This debug container comes loaded with the vetted tools an engineer needs for interactive troubleshooting (a shell, a package manager, tcpdump, curl, language-specific utilities) without compromising the security of the main application container. When the debug session ends, the container goes away, and the app image was never changed. This approach wasn’t available to us back in 2022. It relies on ephemeral containers, which graduated to stable in Kubernetes 1.25, after that writeup, and let you attach a debug container to a running pod without rebuilding or restarting it. AIR pods already share a process namespace, so the debug container can see the app’s processes.
Dumps, profiles, and log levels#
We built one-click debugging workflows directly into our developer portal. For Java services, an engineer can trigger a thread dump or a heap dump on a specific pod with a single click. In production, heap dump needs extra approval first: a .hprof is production data, and we treat it that way. Go services get goroutine profiles. Python services get stack traces. It’s the same click in the same portal either way.
Flipping the log level to DEBUG used to mean a restart. Now you can do it on a running Java or Python pod. It reverts on its own after eight hours, or you revert it. You don’t rebuild the image and you don’t bounce the pod.
Individually these are small things. Together they’re the difference between filing a ticket with the platform team and just fixing your own problem.
What’s next: making operations agentic#
And we’re not done. Everything we’ve described so far (autoscaling, progressive delivery, debugging) still relies on a human deciding when to act, even if the platform does the heavy lifting once they do. The next stage we’re exploring is closing that loop: an agent that decides how carefully to ship a change, and that can recognize a problem and resolve it, not just surface it.
A few directions we’re exploring:
- Risk-scored canaries. We’d score the code change itself, then write the Argo Rollouts steps from that score: traffic weights and how long to soak at each step. A trivial change keeps a fast canary. A risky one gets smaller increments and longer pauses, so it picks up more signal before it owns production traffic. Today’s paved path is one conservative default for every deploy. This would keep the platform owning the Rollout config, but pick the pace from the change instead of a single template.
- Drift detection and remediation. We’d have an agent spot when a running service has drifted from its intended or approved spec (a manual hotfix, a stale config, an untracked change, an unsupported pattern) and reconcile it automatically, instead of waiting for an engineer to notice, or for it to surface as an incident down the road.
- Agentic debuggability. We’d extend today’s one-click dumps and log-level flips into an agent that proactively pulls the right diagnostic artifacts at the right moment, correlates them against recent deploys and metrics, and proposes a root cause, rather than leaving an engineer to reactively debug an issue to resolution.
- Agentic rollback. We’d go beyond a fixed error-rate threshold during canary, with an agent that observes the service, reasons about why it looks unhealthy, and decides the right response, rather than just paging someone to take manual action.
None of this is shipped, but it’s the direction we’re excited about: agents taking on more of the judgment calls that today still land on an on-call engineer at 2 a.m.
I like working on this kind of problem: a platform that lets thousands of engineers move faster with less friction. If you’re curious what else we’re building, check out the Intuit engineering blog or the software engineering careers page.
Image Credits#
A control room filled with lots of monitors image by Tom Donders
