	In Browser
	StumbleUpon
	del.icio.us
	Google
	Google Buzz
	reddit
	LinkedIn

	Facebook
	Twitter
	Linkedin
	E-Mail

Generative AI > Google Gemini API > Gemini API for Content Moderation

Gemini API for Content Moderation

Author: Venkata Sudhakar

Content moderation is a critical requirement for any platform that accepts user-generated content. The Gemini API can classify content across multiple policy dimensions simultaneously in a single call - far more efficient than running separate classifiers for spam, toxicity, misinformation, and policy violations. It understands context, so it correctly handles edge cases that keyword-based filters miss entirely.

ShopMax India accepts seller product listings, customer reviews, and Q&A responses. Before Gemini moderation, fake reviews and spam listings slipped through keyword filters because they were written to evade detection. With Gemini, the moderation pipeline catches 94% of policy violations at submission time, with false positive rates low enough that legitimate sellers are rarely affected. The remaining 6% go to a human review queue.

The below example shows a content moderation pipeline that evaluates user-submitted reviews against multiple policy dimensions.

import vertexai
from vertexai.generative_models import GenerativeModel
import json

vertexai.init(project="shopmax-india", location="asia-south1")
model = GenerativeModel("gemini-2.0-flash")

def moderate_review(review_text: str, product_name: str) -> dict:
    prompt = f"""Moderate this customer review for ShopMax India marketplace.
Return ONLY a JSON object with:
- action: "approve" / "reject" / "human_review"
- violations: list of policy violations found (empty list if none)
- is_fake: true if review appears fabricated or incentivised
- is_spam: true if review is promotional spam
- is_off_topic: true if review is not about the product
- contains_pii: true if review contains personal info (phone, email, address)
- quality_score: integer 1-5 (1=very low quality, 5=excellent)
- rejection_reason: brief reason if action is reject, else null

Product: {product_name}
Review: {review_text}

JSON:
"""

response = model.generate_content(
        prompt,
        generation_config={"temperature": 0.1, "response_mime_type": "application/json"}
    )
    return json.loads(response.text)

test_reviews = [
    ("Great headphones! Best purchase this year. 5 stars.",
     "Sony WH-1000XM5"),
    ("Call me at 9876543210 for cheap duplicate products. Whatsapp only.",
     "Sony WH-1000XM5"),
    ("I received cash from the seller to write this review. Product is ok.",
     "Samsung Galaxy S24")
]

for review, product in test_reviews:
    result = moderate_review(review, product)
    print(f"Review: {review[:50]}...")
    print(f"Action: {result['action']}, Fake: {result['is_fake']}, Spam: {result['is_spam']}")
    if result["violations"]:
        print(f"Violations: {result['violations']}")
    print()

It gives the following output,

Review: Great headphones! Best purchase this year. 5 stars....
Action: approve, Fake: False, Spam: False

Review: Call me at 9876543210 for cheap duplicate products....
Action: reject, Fake: False, Spam: True
Violations: ['spam', 'contains_pii', 'off_platform_solicitation']

Review: I received cash from the seller to write this review....
Action: reject, Fake: True, Spam: False
Violations: ['incentivised_review', 'policy_violation']

The below example shows how to batch-moderate submissions and route them to the appropriate queue.

def process_moderation_queue(submissions: list) -> dict:
    approved, rejected, human_review = [], [], []

for item in submissions:
        result = moderate_review(item["text"], item["product"])
        item["moderation"] = result

if result["action"] == "approve":
            approved.append(item)
        elif result["action"] == "reject":
            rejected.append(item)
        else:
            human_review.append(item)

return {
        "approved": len(approved),
        "rejected": len(rejected),
        "human_review": len(human_review),
        "total": len(submissions),
        "auto_resolution_rate_pct": round(
            (len(approved) + len(rejected)) / len(submissions) * 100, 1
        )
    }

# Simulate a batch of 5 submissions
batch = [
    {"text": "Excellent product, fast delivery to Hyderabad!", "product": "Samsung TV"},
    {"text": "Waste of money. Stopped working in 2 days.", "product": "Boat Earbuds"},
    {"text": "Buy from my shop instead: shopname.com", "product": "JBL Speaker"},
    {"text": "Average product. Nothing special.", "product": "Philips Iron"},
    {"text": "Contact 9999888877 for best price", "product": "Apple iPhone"}
]

stats = process_moderation_queue(batch)
print(f"Batch results: {stats}")

It gives the following output,

Batch results:
{"approved": 3, "rejected": 2, "human_review": 0,
 "total": 5, "auto_resolution_rate_pct": 100.0}

A 100% auto-resolution rate on clean batches means the human review team can focus their attention on genuinely ambiguous cases. ShopMax India configures Gemini to send borderline cases - where violations are suspected but confidence is low - to the human queue rather than auto-rejecting them. This balances enforcement with fairness, ensuring legitimate sellers are not penalised by false positives while maintaining marketplace integrity across thousands of daily submissions.

Send your comments, suggestions or queries regarding this site to [email protected].