KEDA Gets GPU Autoscaling: Reference Implementation Solves Kubernetes Cost Problem

In Brief

What: A new open-source reference implementation enables KEDA to autoscale Kubernetes workloads based on GPU metrics (utilization, memory, temperature, power) via an external scaler architecture, solving the CGO incompatibility that blocked native GPU scaling.

Why it matters: GPU compute is the dominant cost driver for LLM inference and ML training workloads, yet standard Kubernetes autoscaling ignores it entirely. Teams can now scale to zero on idle GPUs and right-size inference deployments based on actual accelerator load.

What it means for Bulgaria and the region: Bulgarian teams running GPU workloads on managed Kubernetes (GKE, EKS, AKS) or on-prem clusters can immediately reduce inference costs. With Sofia emerging as a hub for AI/ML engineering, this pattern is directly applicable to local startups and enterprise AI teams alike.

If this pattern looks relevant to your inference stack, ISTA 2026 in September is where Bulgarian engineers compare notes in person; speaker applications close May 31.

Pattern Name: GPU-Aware Kubernetes Autoscaling via External Scalers

Why This Is Important: GPU workloads are now the dominant cost center for AI/ML infrastructure, yet most teams still autoscale on CPU metrics. This pattern is adoptable today with production-ready tooling.

Domain: DevOps/SRE/Platform

Who Should Care: DevOps/SRE, Backend, Manager

Level: Intermediate

Evidence Type: Release

The Problem

Kubernetes autoscaling was designed for a world where CPU and memory were the expensive resources. The Horizontal Pod Autoscaler (HPA) watches these metrics by default, and most custom metrics pipelines still funnel through Prometheus exporters that treat GPU utilization as an afterthought. This made sense in 2018. It makes no sense when a single A100 costs more per hour than an entire node of CPU workers.

The mismatch creates three distinct failure modes. First, inference deployments stay scaled up long after traffic drops because the HPA sees idle CPU and assumes the workload is still active. Second, scale-up events lag behind actual demand because memory pressure on the GPU (the real bottleneck for LLM serving) is invisible to the scheduler. Third, teams resort to manual scaling or crude time-based rules, which defeats the entire point of running on Kubernetes.

The problem compounds for scale-to-zero scenarios. vLLM and similar inference servers can cold-start in seconds, but only if the autoscaler knows when to wake them. Without GPU-aware activation thresholds, teams either keep expensive replicas running 24/7 or accept cold-start latency spikes that break SLOs.

What Changed This Week

A reference implementation for GPU-aware KEDA autoscaling was published on the CNCF blog on May 27, 2026. The architecture uses a DaemonSet that runs on every GPU node, calls NVML via go-nvml to read local GPU metrics, and exposes them over gRPC using KEDA’s ExternalScaler interface.

This is not a theoretical proposal. The repository includes Helm charts, pre-built profiles for common workloads (vLLM, Triton, training, batch), and an e2e test suite with 11 tests covering profiles, error paths, streaming, and aggregation modes. The tests run in CI without GPU hardware using a mock collector mode.

The key insight is architectural: KEDA’s operator runs as a single deployment and cannot make local NVML calls to GPUs on other nodes. The DaemonSet pattern solves this by placing a scaler pod on every GPU node, which then serves metrics to the central KEDA operator over the network. This is the same pattern Kubernetes uses for device plugins and the metrics server.

Why Now

Three forces converged to make this pattern urgent. First, LLM inference costs have become the dominant line item in AI infrastructure budgets. A single H100 at on-demand pricing runs €3-4 per hour; a cluster of eight cards serving a production model costs more per month than most engineering teams’ salaries. Autoscaling that ignores GPU utilization is autoscaling that ignores the bill.

Second, the agentic inference pattern is spreading. Agentic workloads are bursty by nature: an agent might spin up multiple inference calls in parallel, then go idle for minutes. Traditional request-based autoscaling cannot capture this pattern because the GPU memory stays allocated even when the agent is thinking. Memory-based scaling can.

Third, GreenOps is no longer a nice-to-have. Wasted GPU cycles translate directly into wasted energy. Scope 3 emissions reporting is becoming mandatory for large enterprises, and “we left the GPUs running overnight because the autoscaler didn’t know they were idle” is not a defensible answer to an auditor.

The New Practice

The new practice has three components: deploy the DaemonSet, configure ScaledObjects with GPU-aware triggers, and tune profiles for your workload type.

Deployment is straightforward. The Helm chart targets nodes with nvidia.com/gpu.present=true and tolerates the nvidia.com/gpu taint. It uses the NVIDIA container runtime by default; for clusters without it, set nvmlHostMounts.enabled=true to mount device files directly.

ScaledObject configuration uses KEDA’s external trigger type. The scaler address points to the DaemonSet service, and the profile parameter selects pre-tuned defaults:

Profile selection depends on workload characteristics. The vllm-inference profile scales on memory_used_percent with a target of 80% and an activation threshold of 5%, enabling scale-to-zero. The training profile scales on gpu_utilization at 90% with no scale-to-zero (training jobs should not be interrupted). The batch profile uses aggressive scale-down with a 70% memory target and 1% activation threshold.

For multi-GPU nodes, aggregation modes (max, min, avg, sum) let teams choose how to combine metrics across cards. Targeting a specific GPU index is also supported.

Tooling Implications

KEDA remains the control plane. No changes to KEDA core are required; the external scaler interface was designed for exactly this use case. Teams already running KEDA for Prometheus or Kafka scaling can add GPU triggers without architectural changes.

DCGM Exporter (NVIDIA’s official Prometheus exporter for GPU metrics) is not replaced but complemented. DCGM Exporter is still useful for dashboards and alerting; the keda-gpu-scaler is specifically optimized for the low-latency, high-frequency polling that autoscaling requires.

Cluster Autoscaler and Karpenter work alongside this pattern. The GPU scaler handles pod-level scaling; node-level scaling still depends on the cluster autoscaler recognizing pending pods with GPU requests. Teams should verify that their node autoscaler is configured to provision GPU node pools on demand.

vLLM and Triton configurations may need adjustment. Scale-to-zero requires that the inference server can cold-start within the HPA’s scale-up window. For vLLM, this typically means pre-loading model weights into a shared volume or using a model cache. Triton’s model repository should be on fast storage (NVMe or network-attached SSD).

Evidence

The primary source is the CNCF blog post published May 27, 2026, which includes the architecture rationale, code snippets, and links to the reference implementation.

The repository’s e2e test suite provides evidence of functional correctness: 11 tests covering the full IsActive → GetMetricSpec → GetMetrics flow, including error paths and streaming. Tests run without GPU hardware using mock collectors, which means CI validation is possible on standard runners.

No production benchmark data is published yet. The pattern is adoptable but not yet standardizing; teams should expect to tune activation thresholds and aggregation modes based on their specific workload profiles.

Failure Modes

Metric lag is the most common failure. NVML polling has latency; if the scaler polls every 15 seconds but traffic spikes in 5-second bursts, the HPA will always be behind. Reduce the polling interval (at the cost of higher CPU usage on the DaemonSet) or use predictive scaling for known traffic patterns.

Aggregation mismatch causes silent waste. If a multi-GPU node has one card at 95% utilization and three at 10%, using avg aggregation will report 31% and prevent scale-up. Use max aggregation for inference workloads where any single card hitting capacity is a bottleneck.

Scale-to-zero thrashing happens when the activation threshold is too low. A 1% threshold on memory_used_percent will trigger scale-up from background VRAM allocation (driver overhead, CUDA context). The vllm-inference profile uses 5% for this reason; adjust based on your baseline memory footprint.

DaemonSet scheduling failures occur when the node selector or tolerations do not match the cluster’s GPU node configuration. Verify that nvidia.com/gpu.present=true is actually set on GPU nodes; some managed Kubernetes providers use different labels.

gRPC connectivity issues between the KEDA operator and the DaemonSet can cause scaling to stop silently. Monitor the keda-operator logs for connection errors and ensure network policies allow traffic on port 6000.

Metrics to Watch

GPU utilization delta between actual and target. If the HPA consistently overshoots or undershoots, the target percentage needs adjustment.

Scale-to-zero success rate. Track how often idle deployments actually scale to zero versus staying at minReplicas. A low success rate indicates the activation threshold is too sensitive.

Cold-start latency after scale-from-zero events. Measure p99 latency for the first request after a scale-up. If it exceeds SLO, pre-warming or model caching is required.

DaemonSet CPU usage. The scaler pods should use minimal resources; if CPU spikes, reduce polling frequency or check for NVML call failures.

Cost per inference request. The ultimate metric. Track GPU-hours per 1000 requests before and after adopting GPU-aware scaling.

Do This Next Sprint

1. Audit your current GPU autoscaling. Run kubectl get hpa -A and check which metrics your GPU workloads scale on. If the answer is “cpu” or “memory” (not GPU memory), you have the problem this pattern solves.
2. Deploy the DaemonSet in a staging cluster. Use the Helm chart with default settings. Verify that pods schedule on GPU nodes and that the gRPC endpoint responds.
3. Create a ScaledObject for one inference deployment. Start with the vllm-inference profile and minReplicaCount: 1 (not zero) until you validate cold-start behavior.
4. Run a load test and watch the HPA. Use kubectl get hpa -w while generating traffic. Verify that scale-up and scale-down events correlate with GPU metrics, not CPU.
5. Enable scale-to-zero after validation. Set minReplicaCount: 0 and verify that the deployment scales down during idle periods and scales up within your SLO window.

What This Means for Bulgaria

Bulgarian teams running GPU workloads on managed Kubernetes (GKE, EKS, AKS) can deploy this pattern today. The DaemonSet works on any cluster with NVIDIA GPUs and the standard device plugin; no special cloud provider integration is required.

For teams using on-prem GPU clusters (common in Bulgarian enterprises with data residency requirements), the pattern is equally applicable. Verify that your nodes have the NVIDIA container runtime installed; if not, use the nvmlHostMounts option.

Action items for Bulgarian engineers:

1. Check if your cloud provider’s GPU node pools use the nvidia.com/gpu.present=true label. GKE and EKS do by default; AKS may require manual labeling.
2. If you are interviewing at Bulgarian AI/ML companies (Chaos Group, Gtmhub/Quantive, or the growing Sofia startup scene), ask about their GPU autoscaling strategy. “We scale on CPU” is a red flag for cost management maturity.
3. For teams subject to EU sustainability reporting (CSRD), document your GPU utilization metrics before and after adopting this pattern. The delta is directly relevant to Scope 3 emissions calculations.
4. Bulgarian universities with GPU clusters (Sofia University, TU-Sofia) can use this pattern to share expensive hardware across research groups more efficiently. The scale-to-zero capability is particularly valuable for batch training jobs.

Dig Deeper

• KEDA External Scaler Specification - Official documentation for the gRPC interface used by the GPU scaler. Essential reading before customizing the scaler behavior.
• go-nvml Library - The Go bindings for NVML used by the reference implementation. Useful for understanding which GPU metrics are available and their semantics.
• Kubernetes Device Plugin Framework - Background on how Kubernetes exposes GPU resources to pods. Explains why the DaemonSet pattern is necessary.
• vLLM Production Deployment Guide - Covers model caching and cold-start optimization, which are prerequisites for effective scale-to-zero with GPU workloads.

Many of the patterns covered in the Content Hub will take center stage at ISTA Conference this September, where practitioners and tech leaders discuss them live, debate the trade-offs, and put them in the context of the latest industry shifts. Stay tuned for the program announcement.