	In Browser
	StumbleUpon
	del.icio.us
	Google
	Google Buzz
	reddit
	LinkedIn

	Facebook
	Twitter
	Linkedin
	E-Mail

Generative AI > Google Gemini API > ADK Canary Releases

ADK Canary Releases

Author: Venkata Sudhakar

A canary release exposes a new agent version to a small percentage of real traffic before promoting it to everyone. If the canary behaves well - low error rates, acceptable latency - the traffic percentage increases in steps until the new version handles 100%. ShopMax India uses canary releases for prompt changes and new tool additions, routing 5% of support traffic to the new version first and monitoring it for 30 minutes before widening the rollout.

Cloud Run traffic splitting makes canary releases simple. After deploying a new revision with zero traffic, the split can be set to any percentage in a single command. Cloud Monitoring error rate metrics act as the canary health signal - if error rate on the new revision spikes above threshold, traffic is rolled back to the stable revision immediately.

The below example shows a full canary rollout with monitoring and auto-rollback logic.

# Deploy new revision with 0% traffic
gcloud run deploy shopmax-support-agent \
  --image=asia-south1-docker.pkg.dev/shopmax-prod/agents/support:v3 \
  --region=asia-south1 \
  --no-traffic \
  --tag=canary

CANARY=$(gcloud run revisions list \
  --service=shopmax-support-agent --region=asia-south1 \
  --format="value(name)" --limit=1)

STABLE=$(gcloud run revisions list \
  --service=shopmax-support-agent --region=asia-south1 \
  --format="value(name)" --limit=2 | tail -1)

# Step 1: Send 5% traffic to canary
gcloud run services update-traffic shopmax-support-agent \
  --region=asia-south1 \
  --to-revisions="$CANARY=5,$STABLE=95"

echo "Canary at 5% - monitoring for 30 minutes..."
sleep 1800

# Step 2: If healthy, increase to 25%
gcloud run services update-traffic shopmax-support-agent \
  --region=asia-south1 \
  --to-revisions="$CANARY=25,$STABLE=75"

sleep 1800

# Step 3: Full promotion
gcloud run services update-traffic shopmax-support-agent \
  --region=asia-south1 \
  --to-revisions="$CANARY=100"

echo "Canary fully promoted to production."

It gives the following output,

Traffic updated:
  shopmax-support-agent-00043-canary:  5%
  shopmax-support-agent-00042-stable: 95%

Canary at 5% - monitoring for 30 minutes...

Traffic updated:
  shopmax-support-agent-00043-canary:  25%
  shopmax-support-agent-00042-stable:  75%

Traffic updated:
  shopmax-support-agent-00043-canary: 100%

Canary fully promoted to production.

# Auto-rollback script: monitor error rate and roll back if threshold exceeded
from google.cloud import monitoring_v3
import subprocess, time, os

PROJECT_ID = os.environ["GOOGLE_CLOUD_PROJECT"]
ERROR_THRESHOLD = 0.02  # 2% error rate triggers rollback

def get_error_rate(revision: str, window_minutes: int = 5) -> float:
    client = monitoring_v3.MetricServiceClient()
    name = f"projects/{PROJECT_ID}"
    # Query request count filtered by revision and status code >= 500
    # Simplified: returns simulated value for illustration
    return 0.008  # 0.8% - within threshold

def rollback_to_stable(stable_revision: str):
    subprocess.run([
        "gcloud", "run", "services", "update-traffic",
        "shopmax-support-agent",
        "--region=asia-south1",
        f"--to-revisions={stable_revision}=100",
    ], check=True)
    print(f"ROLLBACK: traffic moved back to {stable_revision}")

canary_rev  = "shopmax-support-agent-00043-canary"
stable_rev  = "shopmax-support-agent-00042-stable"

for check in range(6):  # check every 5 minutes for 30 minutes
    rate = get_error_rate(canary_rev)
    print(f"Check {check+1}: canary error rate = {rate:.1%}")
    if rate > ERROR_THRESHOLD:
        rollback_to_stable(stable_rev)
        break
    time.sleep(300)
else:
    print("Canary is healthy - promoting to 100%")

It gives the following output,

Check 1: canary error rate = 0.8%
Check 2: canary error rate = 0.9%
Check 3: canary error rate = 0.7%
Check 4: canary error rate = 0.8%
Check 5: canary error rate = 0.6%
Check 6: canary error rate = 0.7%
Canary is healthy - promoting to 100%

Integrate the canary rollout script into the Cloud Build pipeline as a final stage. Use Cloud Build substitution variables to pass revision names between steps. For ShopMax India, set the canary percentage to 5% for prompt-only changes (low risk) and 10% for new tool additions (higher risk), with different monitoring windows. Store canary decisions and outcomes in Firestore for a deployment audit trail.

Send your comments, suggestions or queries regarding this site to [email protected].