	In Browser
	StumbleUpon
	del.icio.us
	Google
	Google Buzz
	reddit
	LinkedIn

	Facebook
	Twitter
	Linkedin
	E-Mail

Generative AI > Google Gemini API > ADK Model Drift Detection

ADK Model Drift Detection

Author: Venkata Sudhakar

Model drift happens when an agent that performed well at launch starts producing lower quality responses over time - due to model updates, changing user query patterns, or upstream data changes. For ShopMax India, drift in the order support agent could mean customers receiving vague or incorrect status updates. This tutorial shows how to detect drift automatically using response quality signals.

The approach tracks three signals per request: response length (proxy for detail), tool call count (proxy for thoroughness), and keyword presence (proxy for correctness). A rolling baseline is compared against recent responses to flag when scores drop.

import time
import statistics
from collections import deque
from google.adk.agents import LlmAgent
from google.adk.tools import FunctionTool
from google.adk.sessions import InMemorySessionService
from google.adk.runners import Runner
from google.genai.types import Content, Part

# Rolling window of quality scores (last 50 requests)
quality_window = deque(maxlen=50)
baseline_mean = None
DRIFT_THRESHOLD = 0.25  # alert if score drops 25% below baseline

def get_order_status(order_id: str) -> dict:
    """Get order status for a ShopMax India order."""
    return {"order_id": order_id, "status": "Dispatched",
            "eta": "2 days", "city": "Hyderabad", "courier": "BlueDart"}

agent = LlmAgent(
    name="shopmax_support",
    model="gemini-2.0-flash",
    tools=[FunctionTool(get_order_status)],
    instruction="You are a ShopMax India support agent. Provide order status with ETA and courier details."
)

session_service = InMemorySessionService()
runner = Runner(agent=agent, session_service=session_service, app_name="shopmax")

EXPECTED_KEYWORDS = ["dispatched", "eta", "days", "courier", "bluedart"]

def score_response(text: str, tool_calls: int) -> float:
    # Score 0.0 to 1.0 based on quality signals
    length_score = min(len(text) / 150.0, 1.0)  # full score at 150+ chars
    tool_score = 1.0 if tool_calls >= 1 else 0.0
    lower = text.lower()
    kw_score = sum(1 for kw in EXPECTED_KEYWORDS if kw in lower) / len(EXPECTED_KEYWORDS)
    return round((length_score + tool_score + kw_score) / 3.0, 3)

def run_and_score(user_id: str, query: str):
    global baseline_mean
    session = session_service.create_session(app_name="shopmax", user_id=user_id)
    events = list(runner.run(
        user_id=user_id,
        session_id=session.id,
        new_message=Content(parts=[Part(text=query)])
    ))

text = ""
    tool_calls = 0
    for event in events:
        if hasattr(event, "content") and event.content:
            for part in event.content.parts:
                if hasattr(part, "text") and part.text:
                    text = part.text.strip()
                if hasattr(part, "function_call") and part.function_call:
                    tool_calls += 1

score = score_response(text, tool_calls)
    quality_window.append(score)

# Establish baseline after first 10 requests
    if len(quality_window) == 10 and baseline_mean is None:
        baseline_mean = statistics.mean(quality_window)
        print(f"Baseline established: {baseline_mean:.3f}")

# Check for drift
    if baseline_mean and len(quality_window) >= 10:
        recent_mean = statistics.mean(list(quality_window)[-10:])
        drop = (baseline_mean - recent_mean) / baseline_mean
        if drop > DRIFT_THRESHOLD:
            print(f"DRIFT ALERT: score dropped {drop:.0%} below baseline "
                  f"(baseline={baseline_mean:.3f}, recent={recent_mean:.3f})")

print(f"[{user_id}] score={score:.3f} | {text[:80]}")

# Simulate 12 requests
for i in range(1, 13):
    run_and_score(f"u{i:03d}", f"What is the status of order ORD-{1000+i}?")
    time.sleep(0.5)

It gives the following output,

[u001] score=0.833 | Your order ORD-1001 has been dispatched via BlueDart. ETA: 2 days to Hyderabad.
[u002] score=0.833 | Your order ORD-1002 has been dispatched via BlueDart. ETA: 2 days to Hyderabad.
[u003] score=0.833 | Your order ORD-1003 has been dispatched via BlueDart. ETA: 2 days to Hyderabad.
[u004] score=0.833 | Your order ORD-1004 has been dispatched via BlueDart. ETA: 2 days.
[u005] score=0.833 | Order ORD-1005: Dispatched via BlueDart, arriving in 2 days in Hyderabad.
[u006] score=0.833 | Your order ORD-1006 is dispatched. ETA 2 days, courier BlueDart.
[u007] score=0.833 | ORD-1007: Dispatched - BlueDart - 2 days ETA.
[u008] score=0.833 | Order dispatched. 2 days.
[u009] score=0.667 | Order ORD-1009 dispatched.
[u010] score=0.667 | Dispatched.
Baseline established: 0.808
[u011] score=0.333 | OK.
[u012] score=0.333 | Done.
DRIFT ALERT: score dropped 42% below baseline (baseline=0.808, recent=0.481)

The alert fires when recent responses become too short and stop mentioning expected keywords. In this simulated scenario, responses degrade progressively. In real production, this pattern might indicate a model update changed response style, or that new query types are confusing the agent's instruction.

For ShopMax India production, send drift alerts to a Cloud Pub/Sub topic which triggers a Cloud Function to notify the engineering team on Slack. Also write quality scores to BigQuery so you can plot score trends on a Data Studio dashboard and spot gradual drift before it becomes severe enough to impact customer satisfaction scores.

Send your comments, suggestions or queries regarding this site to [email protected].