HPA is hard to get right, and most teams keep the vended defaults until something breaks. IKS AIR turns 30–40 YAML files into one spec, IPA drives HPA and VPA together from real traffic, and infrastructure commits dropped 77%.
That’s the first pillar of IKS AIR: abstracted compute and AI-powered autoscaling. The Intelligent Pod Autoscaler is a Kubernetes controller that recommends replica counts and pod size from historical load — with a human still in the loop for baseline changes.
I work on the platform engineering team at Intuit. This post is reposted, with light edits, from the Intuit engineering blog — co-written with my colleague Avni Sharma.
From 40 YAMLs to a Single Intent#
In Part 1 we covered the leftover YAML tax in kustomize repos. This post is how IKS AIR actually takes that away for compute and autoscaling.
Traditionally, deploying a new service could mean configuring anywhere from 30 to 40 (or more) YAML files. You have deployments, services, ingresses, HPAs, and a dozen other custom resources. It’s a lot to manage and a prime source of errors.
IKS AIR abstracts this complexity away into a single, high-level application specification. That’s the difference between a typo in one field and a misconfigured NetworkPolicy that quietly breaks connectivity in production. Using an OAM-like model of components and traits, this spec lets developers define what their application requires in terms of components (like a web service or a worker) and traits (which modify the component’s behavior), without needing to understand the low-level Kubernetes implementation.
We chose this model deliberately over exposing raw Kubernetes CRDs. Components describe what an application is — a web service, a cron job, a worker — while traits describe how it behaves: how it scales, how it’s exposed, how its rollouts are staged. A developer declaring a sizing trait doesn’t need to know that it ultimately configures an HPA and a VPA together; they just describe the outcome they want, and the platform figures out the Kubernetes-native way to deliver it. That separation is what lets us evolve the underlying implementation — swapping autoscaling strategies, hardening manifests, changing defaults — without ever touching a developer’s spec.
The platform takes this simple spec and generates all the necessary, battle-hardened Kubernetes manifests under the hood.
Networking, Without the Networking Headaches#
Compute isn’t the only thing IKS AIR abstracts. Developers no longer need to configure ingress rules, load balancers, or gateways by hand. They simply declare whether an API is internal or external to Intuit, and the platform handles the rest. Service-to-service connectivity across clusters and accounts is discovered and wired up automatically, with the platform injecting and managing the necessary networking components behind the scenes.
Letting AI Handle the Scaling#
As we mentioned in our 2022 post, we’d already laid out a three-level maturity model for how far we thought resource sizing could go: static pre-defined configurations for services with predictable load, dynamic right-sizing driven by real-time CPU utilization, and — the long-term bet — fully dynamic runtime sizing based on memory utilization or custom metrics, with no developer intervention required.
Level 1 and Level 2 shipped as described. Level 3 is where the plan changed shape — and where we learned just how far off our starting point was. Fleet-wide, average CPU utilization sat in the low single digits. Not because teams were careless, but because the only feedback they had was pre-production load testing that rarely matched real traffic, so the rational move for any individual team was to over-provision defensively and never revisit it. Multiply that across thousands of services and you get a platform running at a fraction of its real capacity.
Closing that gap meant more than “wire up a controller to watch memory in real time.” We built IPA, our Intelligent Pod Autoscaler, as a Kubernetes custom resource with its own controller — which decoupled autoscaling configuration from the Deployment manifest entirely. That separation mattered operationally: it meant we could evolve how recommendations get computed and applied without asking every service owner to touch their manifests again.
IPA’s long-term replica recommender analyzes a rolling window of historical performance trends — refreshed every few days — to determine the optimal min and max replicas, feeding those recommendations directly into the HPA controller. Simultaneously, it analyzes historical CPU and memory usage to right-size each pod vertically, the VPA half of the job. Both run together, so a service isn’t over-provisioned horizontally to cover for being under-provisioned vertically, or vice versa.
The part the 2022 plan didn’t anticipate: full autonomy turned out to be the wrong target for production. IPA’s recommendations go through developer approval before rolling out, because these are baseline changes, not reactive ones — the kind of change you want a human to glance at once, not approve every time. And we enforce a conservative floor on minimum replicas in production, so no recommendation, however confident, can leave a service under-provisioned for a traffic spike it didn’t anticipate. “No developer intervention at all” became “no developer toil, but still a human in the loop for the decisions that matter.”
That’s the shape the 2022 vision actually took: not a fully autonomous system, but one that takes capacity planning and performance tuning off developers’ plates while keeping a human in the approval loop for the calls that matter. We rolled IPA out to all 900+ existing production AIR services without service disruption — nobody had to redeploy or reconfigure anything to get the benefit. High-traffic services that were once routinely over-provisioned by 5x or more are now trending toward a much higher fleet-wide utilization target, well above where the fleet started.
The next frontier is scaling ahead of demand instead of just right-sizing after the fact — we’re piloting a real-time predictor that forecasts traffic changes per service so HPA can scale up before a spike hits, not after.
That payoff shows up directly in the numbers: each service that moves to IKS AIR saves an estimated 25 engineering days per year in ongoing outer-loop work — paying back the roughly four days it takes to migrate in just a few weeks, then compounding as a recurring efficiency gain for the life of the service. Concretely, that’s teams going from an average of 47 infrastructure commits a year down to about 11 — a 77% drop in the day-to-day tinkering it takes to keep a service running well.
Next time, we’ll cover the final piece of the puzzle: how we provide “done for you” operational excellence with built-in progressive delivery and intuitive debugging tools.
If you’re curious what else Intuit’s engineering org is building, check out the Intuit engineering blog or their software engineering careers page.
Photo Credits#
Aerial view of a large corn maze with paths and stars image by Bob Brewer
