Kubernetes became the default platform for cloud-native applications by making CPU and memory easy to share. AI workloads have exposed the limits of that model. Training jobs need many GPUs at once, inference services need predictable latency, notebooks sit idle for hours while holding expensive accelerators, and clusters now mix several generations of GPU hardware. The result, in many organizations, is a paradox: teams complain that they cannot get GPUs, while finance sees GPU bills for capacity that is mostly idle.
The Kubernetes ecosystem has changed substantially to address this. Dynamic Resource Allocation reached general availability in 2025, queueing and gang scheduling have matured, and GPU sharing techniques are better understood. This article explains how these pieces fit together, how to choose between them and how to build a practical roadmap from a basic GPU node pool to efficient, multi-tenant AI infrastructure.
The GPU Utilization Problem
Cast AI’s 2026 State of Kubernetes Optimization report, which analyzed tens of thousands of production clusters on AWS, Google Cloud and Azure before any optimization was applied, found average GPU utilization of just 5% in the first year it measured GPUs, alongside CPU utilization of 8% and memory utilization of 20%. The same report noted that fewer than 2% of GPUs ran on Spot capacity for most of 2025. An opinion piece in CIO, drawing on community and vendor experience, describes typical enterprise GPU cluster utilization in the range of 10–30%. Whatever the precise number in a given environment, the pattern is consistent: GPUs are allocated far more than they are used.
The causes are largely structural:
- Whole-device allocation. Traditional Kubernetes device plugins expose GPUs as integer counts. A small inference model using a fraction of a GPU’s memory still blocks the entire device.
- Static reservation. Teams reserve dedicated node pools or GPUs “just in case,” and interactive notebooks hold GPUs while users are away.
- Poor scheduling for batch work. Without queueing, distributed training jobs either wait indefinitely for all GPUs to be free or start partially and waste the resources they do hold.
- Heterogeneous fleets. Mixing GPU models, memory sizes and interconnects makes it hard to match workloads to the right hardware with simple labels and node selectors.
- Little visibility. Many teams do not measure actual GPU utilization per workload, so waste remains invisible.
Dynamic Resource Allocation: A New Foundation
Dynamic Resource Allocation (DRA) is the most significant change to how Kubernetes handles specialized hardware since device plugins were introduced. The Kubernetes project announced that DRA graduated to general availability in Kubernetes v1.34, released in 2025, with the core APIs in the resource.k8s.io group moving to GA and several related features progressing through beta and alpha.
Conceptually, DRA works more like storage provisioning than the old device count model. Hardware drivers publish descriptions of available devices and their attributes. Administrators define device classes, for example “GPUs with at least a given amount of memory.” Workloads create resource claims that describe what they need, and the scheduler finds devices that satisfy those claims. As CIO’s analysis puts it, workloads can declare requirements such as GPU type, memory or interconnect instead of requesting a fixed number of GPUs.
Why DRA Matters for AI Platforms
- Richer matching. Claims can express attributes, so a job can request a GPU with sufficient memory rather than a specific node label, which simplifies mixed fleets.
- Sharing and partitioning. DRA provides a framework for sharing devices between containers and for partitioned devices, which supports more efficient use of large accelerators.
- Better lifecycle handling. Resource claims are first-class objects with their own lifecycle, making it easier to prepare, allocate and release devices cleanly.
- Vendor-neutral model. The same API model applies to GPUs, other accelerators and specialized network devices, reducing the need for vendor-specific scheduling extensions.
In practice, DRA depends on drivers from hardware vendors and support in managed Kubernetes services, which is maturing at different speeds. Platform teams should check the status of DRA drivers for their GPU vendor and their cloud provider’s managed Kubernetes version before planning a production rollout.
Choosing a GPU Sharing Strategy
Even with DRA, you still need to decide how GPUs are shared between workloads. The three main techniques have different isolation and performance characteristics.
Multi-Instance GPU (MIG)
MIG partitions a supported GPU, available on NVIDIA Ampere and newer data center GPUs, into hardware-isolated instances, each with dedicated memory and compute. It provides predictable performance and strong isolation, making it well suited to production inference for small and medium models and to multi-tenant clusters where noisy neighbors are unacceptable. The trade-off is fixed partition sizes, which must be planned.
Multi-Process Service (MPS)
MPS allows multiple processes to share a GPU concurrently with software-level partitioning. It offers better utilization for workloads that individually underuse a GPU, with weaker isolation than MIG. It is a reasonable choice for latency-tolerant inference and batch scoring.
Time-Slicing
Time-slicing lets workloads take turns on a GPU. It is simple to configure and works on a wide range of hardware, but provides no memory isolation and introduces latency jitter. It fits development, experimentation and notebook environments best.
A sensible default is: MIG for production inference that needs predictable latency, time-slicing for development and notebooks, and full GPUs, often many at once, for large-scale training.
The fastest way to cut AI infrastructure cost is rarely buying cheaper GPUs. It is making the GPUs you already pay for do useful work, through right-sized sharing, queue-based admission and visibility into actual utilization.
Queueing and Scheduling for AI Workloads
The default Kubernetes scheduler places pods one at a time. Distributed training needs all of its workers to start together, and multi-tenant clusters need fair sharing of scarce accelerators. Two open-source projects have become central to solving this.
Kueue
Kueue is a Kubernetes-native job queueing system. It adds cluster-wide queues, quotas per team or tenant, borrowing of unused quota between teams in a cohort, preemption and all-or-nothing admission, so a distributed job is admitted only when all the resources it needs are available. This prevents partial starts that hold GPUs without making progress and lets platform teams guarantee each team a share of capacity while still lending idle resources.
KAI Scheduler
KAI Scheduler is an open-source scheduler from NVIDIA, released under the Apache 2.0 license and originating from Run:ai. According to CIO, it supports fractional GPU allocation, topology-aware scheduling and hierarchical queues across the AI lifecycle. It is particularly relevant for organizations running large, mixed AI workloads on NVIDIA hardware that need more sophisticated placement than the default scheduler offers.
Topology Awareness
Large training jobs are sensitive to the interconnect between GPUs. Placing workers on GPUs connected by high-bandwidth links, within the same node or across nodes with fast networking, can significantly reduce communication overhead. Topology-aware scheduling, through scheduler features or placement policies, should be part of any platform that runs multi-node training.
A Reference Architecture for AI on Kubernetes
The following layered architecture applies to Amazon EKS, Azure Kubernetes Service, Google Kubernetes Engine or self-managed clusters.
- Node layer: separate node pools for training, inference and interactive work, with autoscaling, a mix of on-demand, reserved and Spot capacity, and GPU drivers managed through an operator such as the NVIDIA GPU Operator.
- Device layer: DRA drivers or device plugins exposing GPUs, MIG partitions or shared devices, with device classes defined by the platform team.
- Scheduling layer: Kueue or an AI-aware scheduler providing queues, quotas, gang admission and preemption; topology-aware placement for training.
- Workload layer: training operators, inference servers with autoscaling including scale-to-zero, and managed notebook environments with idle culling.
- Observability layer: GPU metrics from DCGM exported to your monitoring stack, utilization dashboards per team and workload, and cost allocation linked to your FinOps reporting.
- Governance layer: policies for which teams can use which GPU classes, maximum job durations, required checkpointing for Spot-based training and approval for large reservations.
Managed AI Services or Self-Managed Kubernetes?
Not every AI workload belongs on Kubernetes. Before investing in a GPU platform, decide which workloads genuinely benefit from it. Managed model APIs and fully managed training and inference services remove the need to operate GPU infrastructure at all, and they are often the right choice for teams consuming foundation models through prompts or light fine-tuning.
Kubernetes earns its complexity when one or more of the following apply:
- You self-host open-weight models for cost, latency, data residency or customization reasons.
- Several teams share a GPU budget and need fair, policy-driven access to it.
- You run sustained training or batch inference volumes where the economics of reserved or Spot GPUs beat per-request pricing.
- You need portability across clouds or between cloud and on-premises infrastructure.
- Your AI services must integrate closely with existing containerized applications, observability and security tooling.
Many organizations end up with a hybrid: managed model APIs for general-purpose generative AI, and a Kubernetes-based platform for self-hosted models, fine-tuning and high-volume inference. Making that split explicit keeps the platform focused on the workloads where it adds value.
Implementation Roadmap
Phase 1: Make Utilization Visible (Month 1)
Deploy GPU telemetry and build dashboards that show allocated versus actually used GPU capacity by namespace, team and workload type. CIO’s guidance is to observe utilization for about a month before making major architecture decisions, and we agree: data almost always reveals surprises, such as idle notebooks or inference services sized for peak traffic that never arrives.
Phase 2: Introduce Queues and Quotas (Months 2–3)
Deploy Kueue with a small number of queues, for example one each for production inference, training and experimentation. Give teams guaranteed quotas with borrowing enabled, so idle capacity is shared rather than stranded. Move batch training to queue-based submission.
Phase 3: Share GPUs Appropriately (Months 3–4)
Enable MIG on suitable GPUs for production inference, time-slicing for development and notebooks, and introduce idle culling for interactive sessions. Review inference services and consolidate small models onto shared or partitioned devices.
Phase 4: Optimize Capacity and Adopt DRA (Months 4–6)
Move checkpointed training to Spot or preemptible capacity, size reservations to the observed baseline and enable scale-to-zero for infrequently used inference endpoints. Pilot DRA on a non-critical node pool once your vendor’s drivers and managed Kubernetes version support it, and plan a gradual transition from device plugins to resource claims.
Worked Example: Consolidating Fragmented GPU Pools
Consider a company with three data science teams, each of which had requested its own GPU node pool on a managed Kubernetes service. Each pool was sized for that team’s peak needs. Teams regularly complained of waiting for GPUs, yet monthly GPU spend kept growing.
Telemetry showed that most GPUs were allocated to long-running notebooks and to inference services serving small models with modest traffic, while training jobs queued for full GPUs. The platform team consolidated the three pools into shared pools by workload type, enabled MIG on the inference pool so several small models could share each GPU with hardware isolation, enabled time-slicing and idle culling for notebooks, and introduced Kueue with a guaranteed quota per team plus borrowing.
Training jobs were moved to a queue with gang admission, and checkpointing was made mandatory so that non-urgent training could run on Spot capacity. Within a few months, teams reported shorter waits for GPUs because idle capacity was now shared, and the platform team was able to defer a planned expansion of reserved GPU capacity. The utilization dashboards became part of the monthly FinOps review, giving finance visibility into how GPU investment was used.
Common Pitfalls
- Buying more GPUs before measuring. Capacity shortages are often allocation problems, not hardware shortages.
- Using time-slicing for latency-sensitive production. Lack of isolation can cause unpredictable latency and memory contention.
- Running training on Spot without checkpoints. Preemptions then waste hours of compute.
- Rolling out DRA without driver readiness. Verify vendor drivers and managed service support, and pilot first.
- Treating AI infrastructure separately from FinOps. GPU cost and utilization should be reported alongside the rest of cloud spend, with clear ownership per team.
Outlook: Kubernetes as the AI Operating Layer
Kubernetes is steadily becoming the operating layer for AI, not just for stateless services. DRA provides a common model for accelerators, queueing systems bring batch scheduling capabilities that once required separate high-performance computing schedulers, and inference platforms increasingly build on Kubernetes primitives. Expect continued progress on DRA features such as partitionable devices and consumable capacity, closer integration between schedulers and cost tooling, and more managed services exposing these capabilities by default. For platform teams, the opportunity is to build one shared, well-governed AI platform rather than a collection of team-specific GPU islands.
Conclusion
Running AI on Kubernetes efficiently is less about any single feature than about combining several: visibility into real utilization, queue-based admission with fair quotas, sharing strategies matched to workload types, smart use of Spot and reserved capacity, and, increasingly, Dynamic Resource Allocation as the foundation for describing and allocating accelerators. Start by measuring, introduce queues and sharing, then adopt DRA as your ecosystem supports it. The result is AI infrastructure that gives teams faster access to GPUs while making every accelerator you pay for earn its keep.
How Hanoi Technology Can Help
Hanoi Technology designs and operates Kubernetes-based AI platforms with GPU sharing, queueing and utilization visibility on AWS, Azure and Google Cloud.