LLM Output Caching with GPTCache - Reducing API Costs

ShopMax India handles thousands of similar customer queries daily - product availability, store locations, return policies. Without caching, each query hits the LLM API and incurs cost. GPTCache intercepts LLM calls and returns cached responses for semantically similar questions, cutting API costs significantly for high-traffic deployments.

GPTCache sits between your application and the LLM API as a middleware layer. It embeds incoming queries and compares them against cached embeddings using similarity search. If a match is found above a configurable threshold, the cached response is returned without an API call. Cache storage can be in-memory for development or Redis for production.

The below example shows how ShopMax India integrates GPTCache with the OpenAI API to cache product FAQ responses using semantic similarity matching.

It gives the following output,

ShopMax India accepts returns within 10 days of purchase for electronics
with original packaging and invoice. Mobile phones and laptops require
a technical inspection before refund approval.
Cache hit: False

# Second similar query returns instantly from cache:
# "How many days do I have to return a product at ShopMax?"
# Cache hit: True

Set your similarity threshold carefully - too low causes incorrect cache hits, too high defeats the purpose. A threshold of 0.85 cosine similarity works well for FAQ-style queries at ShopMax India. For production, use Redis as the cache backend with a TTL of 24 hours so pricing and policy changes propagate daily. Monitor cache hit rates - a hit rate below 20% suggests queries are too varied to benefit from caching.