Scaling stateful Gemini 4 Argon agent sessions on GKE: Connection draining, startup boost, and session persistence

Architect GKE for long-running Gemini 4 Argon agent sessions using VPA CPU startup boost, GCPBackendPolicy connection draining, and session checkpointing.

title: "Scaling stateful Gemini 4 Argon agent sessions on GKE: Connection draining, startup boost, and session persistence" published: true description: "Architect GKE for long-running Gemini 4 Argon agent sessions using VPA CPU startup boost, GCPBackendPolicy connection draining, and session checkpointing." tags: kubernetes, ai, gke, googlecloud cover_image: https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/7xhrstgm9dxsbn3z63ps.png

Most Kubernetes architectures are built around a comfortable assumption: HTTP requests arrive, execute stateless logic in under 200 milliseconds, and return a response. Multi-turn Gemini 4 Argon agent workflows invert every part of that profile. A single user request can initiate a 3-minute bidirectional streaming session where the agent plans, executes external tools, ingests multi-megabyte documents into its context window, and streams intermediate reasoning steps back to the client.

When you deploy these long-running, stateful agent orchestrators onto Google Kubernetes Engine (GKE) using default web-service configurations, three production failure modes appear almost immediately:

1. Startup CPU throttling: Python and Node.js agent runtimes that initialize heavy AI SDKs, OpenTelemetry instrumentation, and local tokenizer caches spike CPU during boot, causing new Pods to fail readiness probes or stall Horizontal Pod Autoscaler (HPA) scale-out events.
2. Dropped in-flight reasoning loops during rollouts: Default 30-second load balancer timeouts and abrupt SIGTERM shutdowns sever active WebSocket or Server-Sent Events (SSE) streams mid-turn when GKE scales down or rolls out a new image.
3. Lost conversation and checkpoint state: Storing multi-turn reasoning history purely in Pod memory means any container restart forces the agent to replay expensive Gemini 4 Argon token contexts from scratch.

In this article, I will show you how to architect GKE for stateful Gemini 4 Argon agent runtimes by combining VPA CPU Startup Boost, long-horizon connection draining via GCPBackendPolicy, and external state checkpointing.

The lifecycle of a stateful Gemini 4 Argon agent Pod

An agent runtime Pod traverses three distinct operational phases across its lifecycle:

1. Admission and cold boot (0s to 15s): GKE's VPA admission webhook injects a temporary CPU Startup Boost so the container initializes AI SDKs, tokenizers, and tracing pipelines without CPU throttling. Once it passes its readinessProbe, GKE uses In-Place Pod Resize (IPPR) to step CPU down to baseline without restarting the container.
2. Steady-state streaming (up to 10 minutes): The GKE Gateway routes long-lived SSE or WebSocket streams with a 600-second backend timeout and cookie-based session affinity. The worker executes multi-turn tool loops with Gemini 4 Argon, checkpointing turn state to Cloud Memorystore.
3. Graceful shutdown and draining: On rolling updates or scale-down events, a preStop hook and a 300-second termination grace period allow active reasoning loops to finish and flush final checkpoints before the container exits.

Each GKE primitive addresses a specific stage of this lifecycle:

Eliminating cold-start throttling with VPA CPU startup boost

Agent orchestrator images frequently bundle the google-genai SDK, gRPC C-extensions, Pydantic schema validators, and document parsing libraries. If you size your container's baseline CPU request for steady-state I/O waiting (where the Pod spends most of its time awaiting network streams from the Gemini platform), the container only needs 500m CPU at steady state, but requires 1500m CPU during the first 10 seconds of startup.

Using VPA CPU startup boost (available in Public Preview on GKE 1.36.0-gke.4447000 and later in the Rapid Channel), GKE temporarily boosts the container's CPU request at admission and uses Kubernetes In-Place Pod Resize (IPPR) to lower the CPU allocation back to 500m without restarting the container once it passes its readinessProbe.

GKE Autopilot vs. GKE Standard sizing considerations

• GKE Autopilot: Autopilot validates CPU-to-memory ratios at admission. Ensure your Pod's memory request supports the boosted CPU value (on the general-purpose Autopilot compute platform, memory must fall between 1 GiB and 8 GiB per vCPU), or assign the workload to a Performance ComputeClass.
• GKE Standard: Ensure your worker nodes have sufficient allocatable CPU headroom to accommodate the temporary startup boost factor.

Create a VerticalPodAutoscaler policy named agent-runtime-startup-boost.yaml that triples the baseline 500m CPU allocation to 1500m during boot and unboosts immediately (durationSeconds: 0) upon readiness. Setting updateMode: "Off" applies the startup boost and in-place downscale without enabling continuous VPA actuation, avoiding conflicts with Horizontal Pod Autoscaler (HPA):

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: gemini-agent-startup-boost
  namespace: agent-runtime
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: gemini-session-orchestrator
  updatePolicy:
    updateMode: "Off"
  startupBoost:
    cpu:
      type: Factor
      factor: 3
      durationSeconds: 0

Configuring long-lived streaming and graceful termination

To prevent GKE from severing active Gemini 4 Argon streaming sessions during a rolling deployment, you need to coordinate timeouts across two layers: the Kubernetes Pod lifecycle (terminationGracePeriodSeconds and preStop hooks) and the GKE Gateway load balancer (GCPBackendPolicy).

Save the following manifest as agent-session-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gemini-session-orchestrator
  namespace: agent-runtime
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
  selector:
    matchLabels:
      app: gemini-session-orchestrator
  template:
    metadata:
      labels:
        app: gemini-session-orchestrator
    spec:
      serviceAccountName: gemini-agent-sa
      terminationGracePeriodSeconds: 300
      containers:
      - name: agent-server
        image: us-central1-docker.pkg.dev/production-ai-platform/agents/gemini-4-session-server:v1.0.0
        ports:
        - containerPort: 8080
        env:
        - name: GOOGLE_CLOUD_PROJECT
          value: "production-ai-platform"
        - name: REDIS_HOST
          value: "10.128.0.45"
        resources:
          requests:
            cpu: "500m"
            memory: "2Gi"
          limits:
            memory: "2Gi"
        readinessProbe:
          httpGet:
            path: /healthz/ready
            port: 8080
          periodSeconds: 5
          failureThreshold: 2
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 10"]
---
apiVersion: v1
kind: Service
metadata:
  name: gemini-session-svc
  namespace: agent-runtime
spec:
  selector:
    app: gemini-session-orchestrator
  ports:
  - port: 80
    targetPort: 8080
---
apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
  name: gemini-streaming-backend-policy
  namespace: agent-runtime
spec:
  default:
    timeoutSec: 600
    connectionDraining:
      drainingTimeoutSec: 300
    sessionAffinity:
      type: GENERATED_COOKIE
      cookieTtlSec: 3600
  targetRef:
    group: ""
    kind: Service
    name: gemini-session-svc

Notice three critical settings in this configuration:

• maxUnavailable: 0: Ensures GKE brings new, startup-boosted Pods into a Ready state before terminating any existing Pod holding active sessions.
• preStop sleep hook (sleep 10): Because Kubernetes endpoint removal and Google Cloud load balancer programming happen asynchronously with SIGTERM, sleeping for 10 seconds before the application stops accepting new connections prevents in-flight requests at the load balancer edge from hitting a closing socket.
• timeoutSec: 600 and drainingTimeoutSec: 300: Allows streaming SSE or WebSocket responses from Gemini 4 Argon to run for up to 10 minutes and gives draining Pods 5 minutes to finish active turns.

Implementing per-turn state checkpointing and graceful drain

Even with a 300-second drain window, an agent session might span multiple user interactions over an hour, or a node preemption on Spot capacity might give only 30 seconds of warning. By checkpointing the conversation history after every Gemini 4 Argon turn into Redis (such as Memorystore for Redis), any replica in the Deployment can transparently resume the session if the original Pod terminates.

Here is the core session server (session_server.py) showing graceful SIGTERM tracking and per-turn Redis checkpointing with the google-genai SDK:

import asyncio
import json
import os
import signal
from contextlib import asynccontextmanager
from fastapi import FastAPI, HTTPException
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
import redis.asyncio as redis
from google import genai
from google.genai import types

draining = False
active_turns = 0


class TurnRequest(BaseModel):
    user_message: str


def handle_sigterm(*_args):
    global draining
    draining = True


@asynccontextmanager
async def lifespan(_app: FastAPI):
    signal.signal(signal.SIGTERM, handle_sigterm)
    yield
    # Wait for active Gemini 4 Argon turns to finish and checkpoint before exiting
    while active_turns > 0:
        await asyncio.sleep(1)
    await redis_client.aclose()


app = FastAPI(lifespan=lifespan)
genai_client = genai.Client(
    vertexai=True,
    project=os.environ["GOOGLE_CLOUD_PROJECT"],
    location=os.environ.get("VERTEX_LOCATION", "us-central1"),
)
redis_client = redis.Redis(host=os.environ["REDIS_HOST"], port=6379, decode_responses=True)
MODEL_ID = os.environ.get("GEMINI_MODEL", "gemini-4-argon")


@app.get("/healthz/ready")
async def readiness_check():
    if draining:
        raise HTTPException(status_code=503, detail="Pod is draining active sessions")
    return {"status": "ready"}


@app.post("/v1/sessions/{session_id}/stream")
async def stream_agent_turn(session_id: str, request: TurnRequest):
    # Load prior conversation history from Redis checkpoint
    raw_history = await redis_client.get(f"session:{session_id}")
    history = json.loads(raw_history) if raw_history else []
    history.append({"role": "user", "parts": [{"text": request.user_message}]})

    contents = [
        types.Content(role=turn["role"], parts=[types.Part.from_text(text=p["text"]) for p in turn["parts"]])
        for turn in history
    ]

    async def event_generator():
        global active_turns
        active_turns += 1
        assembled_response = []
        try:
            async for chunk in await genai_client.aio.models.generate_content_stream(
                model=MODEL_ID,
                contents=contents,
            ):
                if chunk.text:
                    assembled_response.append(chunk.text)
                    yield f"data: {json.dumps({'text': chunk.text})}\n\n"

            # Checkpoint completed assistant turn back to Redis with a 24-hour TTL
            history.append({"role": "model", "parts": [{"text": "".join(assembled_response)}]})
            await redis_client.set(f"session:{session_id}", json.dumps(history), ex=86400)
        finally:
            active_turns -= 1

    return StreamingResponse(event_generator(), media_type="text/event-stream")

Verifying startup boost and connection draining

Once deployed, confirm that VPA CPU startup boost is accelerating Pod initialization and resizing CPU in place without restarting the container:

# Inspect the startup boost annotation injected on the Pod
kubectl get pods -n agent-runtime \
    -l app=gemini-session-orchestrator \
    -o custom-columns="NAME:.metadata.name,CPU_REQ:.spec.containers[0].resources.requests.cpu,BOOST_ANNOTATION:.metadata.annotations.vpaCpuStartupBoost\/agent-server"

# Watch for the InPlaceResizedByVPA event once the Pod passes readinessProbe
kubectl get events -n agent-runtime --field-selector reason=InPlaceResizedByVPA

Expected event output:

LAST SEEN   TYPE     REASON                 OBJECT                                             MESSAGE
14s         Normal   InPlaceResizedByVPA    pod/gemini-session-orchestrator-6b8f9c7d4-k29lm    Pod was in-place resized by VPA Updater (CPU: 1500m -> 500m)

To verify that rolling updates drain cleanly without dropping streams, trigger a rollout while streaming a multi-step Gemini 4 Argon response and monitor active backend connections in Cloud Monitoring using the metric loadbalancing.googleapis.com/https/backend_request_count alongside kubernetes.io/container/cpu/core_usage_time.

Operational trade-offs to watch

• Memory growth on long context windows: While CPU usage drops to near zero while waiting on Gemini platform network streams, assembling 500K-token contexts in Python memory can spike heap allocations. VPA CPU Startup Boost only boosts CPU; size your container memory requests and limits to accommodate the largest concurrent payload your agent processes.
• Avoid Spot VMs for long-running streaming gateways: Spot VMs (or spot: true in a GKE ComputeClass) only provide a 30-second preemption notice, which is too short for a 3-minute agent reasoning turn to drain gracefully. Run the lightweight session orchestrator Pods on standard on-demand capacity.
• HPA scaling metric selection: Because steady-state agent orchestrators are I/O-bound rather than CPU-bound, scaling HPA solely on CPU utilization will under-scale during concurrent streaming bursts. Pair HPA with custom load balancer concurrency metrics or active session gauges exported to Cloud Monitoring.

Next steps

Running Gemini 4 Argon agents reliably in production requires adapting Kubernetes from short-lived request/response defaults to stateful, long-horizon execution. By combining VPA CPU Startup Boost for fast initialization, GCPBackendPolicy for long streaming timeouts, and per-turn state checkpointing, your agent deployments can scale and roll out new versions without interrupting active users.

To dive deeper into these GKE capabilities, read the VPA CPU Startup Boost guide, review the GKE Gateway GCPBackendPolicy reference, and check the Gemini API documentation on Google Cloud.