Sign in Subscribe
AI

How to Build a Tenant-Scoped AI Agent Output Caching Layer Using Semantic Similarity Deduplication to Cut Multi-Tenant LLM Inference Costs in 2026

Scott Miller

• 10 min read
How to Build a Tenant-Scoped AI Agent Output Caching Layer Using Semantic Similarity Deduplication to Cut Multi-Tenant LLM Inference Costs in 2026

LLM inference bills have a way of arriving like a cold shower. You architect a beautiful multi-tenant AI product, onboard a few hundred customers, and suddenly your monthly token spend looks like a phone number. The culprit, more often than not, is not complex reasoning chains or massive context windows. It is redundancy: dozens of tenants asking semantically identical questions and each one triggering a fresh, expensive round-trip to your model provider.

In 2026, with frontier model pricing still a meaningful line item for SaaS companies running agentic workloads, a well-designed tenant-scoped semantic caching layer is one of the highest-leverage engineering investments you can make. This tutorial walks you through building one from scratch, covering tenant isolation, embedding-based similarity matching, TTL-aware freshness policies, and the subtle gotchas that will bite you if you skip them.

By the end, you will have a production-ready architecture that can realistically eliminate 30 to 60 percent of redundant LLM calls without ever serving stale or cross-contaminated responses to your tenants.

Why Standard Key-Value Caching Falls Short for LLM Outputs

Traditional caches work on exact key matches. Cache key equals hash of input, cache hit returns stored output, done. That model breaks completely for natural language because two prompts can be semantically identical while being lexically different:

  • "Summarize last quarter's revenue performance" vs. "Give me a summary of Q3 revenue results"
  • "What is the refund policy?" vs. "How do I get a refund?"
  • "Draft a follow-up email to the Johnson account" vs. "Write a follow-up for Johnson"

An exact-match cache treats these as entirely different queries and fires three separate LLM calls. A semantic cache recognizes the intent overlap and serves a cached response for the near-duplicate, saving tokens, latency, and money.

The second failure mode of naive caching in multi-tenant systems is data leakage. If you build a single shared cache without tenant scoping, Tenant A's cached response about their proprietary pricing strategy could theoretically be served to Tenant B. That is not a performance bug; that is a security incident.

Architecture Overview

Before diving into code, here is the high-level component map of what we are building:

  • Embedding Service: Converts incoming prompts into dense vector representations.
  • Tenant-Partitioned Vector Store: Stores prompt embeddings and their associated LLM outputs, namespaced per tenant.
  • Similarity Lookup Engine: Queries the vector store for semantically similar cached entries above a configurable cosine similarity threshold.
  • TTL and Freshness Manager: Enforces time-to-live policies per tenant, per query category, or per agent type.
  • Cache Write-Back Layer: On a cache miss, calls the LLM, writes the result back to the cache, and returns the response.
  • Eviction and Invalidation Controller: Handles manual invalidation, tenant offboarding, and cache warming.

The entire flow sits as a middleware layer between your AI agent orchestrator (LangChain, LlamaIndex, a custom agentic loop, or whatever you are running in 2026) and your model provider API.

Step 1: Choose Your Embedding Model Wisely

Your cache quality is only as good as your embedding model. A poor embedder will either produce too many false positives (serving wrong cached responses) or too many false negatives (missing obvious duplicates). For a production semantic cache in 2026, you have three solid options:

  • A dedicated small embedding model running locally (e.g., a fine-tuned sentence-transformer variant): lowest latency, zero per-call cost, but requires infra to host.
  • A hosted embedding API (OpenAI text-embedding-3-small, Cohere embed-v4, or equivalent): easiest to integrate, minimal per-call cost, slight network overhead.
  • Your primary LLM's own embedding endpoint: Convenient but often overkill for this use case and more expensive per token.

For most multi-tenant SaaS products, a hosted small embedding model hits the sweet spot. The embedding call costs a fraction of a full inference call, and the latency overhead (typically 20 to 50ms) is easily justified by the savings when you get a cache hit.

Critical rule: Use the same embedding model consistently. Mixing embedding models across cache writes and reads will produce nonsensical similarity scores and corrupt your cache entirely.

Step 2: Design Tenant-Scoped Namespaces in Your Vector Store

Tenant isolation is non-negotiable. Every vector store worth using in 2026 supports namespace or collection-level partitioning. Here is how to implement it correctly using Pinecone, Weaviate, Qdrant, or pgvector as your backend.

Namespace Strategy

The simplest and most robust approach is to prefix every vector ID and namespace with a deterministic, non-guessable tenant identifier:


namespace = f"tenant_{sha256(tenant_id.encode()).hexdigest()[:16]}"

Do not use raw tenant IDs or human-readable slugs as namespace keys in production. Use a hashed or UUID-derived identifier to prevent enumeration attacks.

Schema for Each Cached Entry

Each vector in your store should carry the following metadata payload alongside the embedding:

  • tenant_id: The internal tenant identifier (for audit logging).
  • original_prompt_hash: SHA-256 of the original prompt text (for exact-match fast-path lookup).
  • original_prompt: The raw prompt text (for human-readable cache inspection).
  • cached_response: The full LLM output string.
  • agent_type: The agent or workflow that generated this response (important for TTL segmentation).
  • created_at: Unix timestamp of cache write.
  • ttl_seconds: Time-to-live for this specific entry.
  • hit_count: Number of times this cached entry has been served (useful for eviction scoring).

Sample Vector Upsert (Python, Qdrant)


from qdrant_client import QdrantClient
from qdrant_client.models import PointStruct
import hashlib, time, uuid

client = QdrantClient(url="http://localhost:6333")

def write_to_cache(tenant_id: str, prompt: str, response: str,
                   embedding: list[float], agent_type: str, ttl_seconds: int = 3600):
    namespace = hashlib.sha256(tenant_id.encode()).hexdigest()[:16]
    point_id = str(uuid.uuid4())

    client.upsert(
        collection_name=f"cache_{namespace}",
        points=[
            PointStruct(
                id=point_id,
                vector=embedding,
                payload={
                    "tenant_id": tenant_id,
                    "original_prompt_hash": hashlib.sha256(prompt.encode()).hexdigest(),
                    "original_prompt": prompt,
                    "cached_response": response,
                    "agent_type": agent_type,
                    "created_at": int(time.time()),
                    "ttl_seconds": ttl_seconds,
                    "hit_count": 0,
                }
            )
        ]
    )

Step 3: Build the Similarity Lookup Engine

On every incoming prompt, your cache layer needs to run a fast approximate nearest-neighbor search against the tenant's namespace before forwarding the request to the LLM. Here is the complete lookup function:


def lookup_cache(tenant_id: str, prompt: str, prompt_embedding: list[float],
                 similarity_threshold: float = 0.92) -> str | None:
    namespace = hashlib.sha256(tenant_id.encode()).hexdigest()[:16]
    collection = f"cache_{namespace}"
    now = int(time.time())

    results = client.search(
        collection_name=collection,
        query_vector=prompt_embedding,
        limit=5,
        with_payload=True,
        score_threshold=similarity_threshold,
    )

    for result in results:
        payload = result.payload
        # TTL check: discard expired entries
        age = now - payload["created_at"]
        if age > payload["ttl_seconds"]:
            # Optionally delete expired entry asynchronously
            schedule_delete(collection, result.id)
            continue

        # Valid hit: increment hit count and return cached response
        increment_hit_count(collection, result.id, payload["hit_count"])
        return payload["cached_response"]

    return None  # Cache miss

A few important design decisions embedded in this function:

  • The similarity threshold of 0.92 is a starting point, not a gospel number. You will need to tune this per domain. Customer support agents can often tolerate a threshold as low as 0.88. Code generation agents may need 0.97 or higher to avoid serving subtly wrong code.
  • TTL is checked at read time, not just at write time. This gives you a soft expiry without requiring a background sweep to be perfectly reliable.
  • Returning the top result above the threshold (not just the absolute top result) prevents serving a mediocre match when the best available result is still semantically distant.

Step 4: Implement Freshness Policies Without Killing Your Hit Rate

The most common objection to semantic caching is: "What if the underlying data changes and the cached answer becomes stale?" This is a valid concern, and the solution is a layered TTL policy rather than a single global expiry.

TTL Tiers by Query Volatility

Classify your agent's query types into volatility tiers and assign TTLs accordingly:

  • Static knowledge queries (product documentation, FAQ, policy explanations): TTL of 24 to 72 hours. These change rarely and have the highest cache value.
  • Aggregated analytics queries (summarize last month's data, trend analysis): TTL of 1 to 4 hours, aligned with your data pipeline refresh cadence.
  • Near-real-time operational queries (current ticket status, live inventory): TTL of 60 to 300 seconds. Short enough to stay fresh, long enough to absorb burst traffic spikes.
  • Personalized or stateful queries (anything referencing "my account," "my recent activity"): TTL of 0 or cache bypass entirely. Do not cache these.

Tenant-Level TTL Overrides

Some tenants have stricter freshness requirements than others. A healthcare SaaS tenant may require near-zero TTL for clinical decision support queries, while a marketing analytics tenant is happy with 6-hour-old summaries. Build TTL overrides into your tenant configuration:


TENANT_TTL_CONFIG = {
    "tenant_abc123": {"default_ttl": 1800, "analytics_ttl": 900},
    "tenant_def456": {"default_ttl": 86400, "analytics_ttl": 3600},
}

def resolve_ttl(tenant_id: str, agent_type: str) -> int:
    config = TENANT_TTL_CONFIG.get(tenant_id, {})
    if agent_type == "analytics":
        return config.get("analytics_ttl", 3600)
    return config.get("default_ttl", 3600)

Soft Invalidation via Event Hooks

For data-driven freshness, connect your cache invalidation to your application's event bus. When a tenant's underlying dataset is updated (a new data ingestion job completes, a document is uploaded, a CRM record changes), publish an invalidation event that clears or marks stale the relevant cache entries for that tenant:


def on_tenant_data_updated(tenant_id: str, affected_agent_types: list[str]):
    namespace = hashlib.sha256(tenant_id.encode()).hexdigest()[:16]
    collection = f"cache_{namespace}"
    for agent_type in affected_agent_types:
        client.delete(
            collection_name=collection,
            points_selector={"filter": {"must": [{"key": "agent_type", "match": {"value": agent_type}}]}}
        )

Step 5: Wire It All Together as Agent Middleware

Now let us assemble the complete cache middleware that wraps your LLM call. This is the function your agent orchestrator calls instead of hitting the model provider directly:


from openai import OpenAI
import asyncio

openai_client = OpenAI()

async def cached_llm_call(
    tenant_id: str,
    prompt: str,
    agent_type: str,
    model: str = "gpt-4o",
    similarity_threshold: float = 0.92,
) -> dict:

    # Step 1: Fast-path exact match check (no embedding needed)
    prompt_hash = hashlib.sha256(prompt.encode()).hexdigest()
    exact_hit = check_exact_match_cache(tenant_id, prompt_hash)
    if exact_hit:
        return {"response": exact_hit, "cache_status": "exact_hit", "tokens_saved": True}

    # Step 2: Generate embedding for semantic lookup
    embedding_response = openai_client.embeddings.create(
        input=prompt,
        model="text-embedding-3-small"
    )
    prompt_embedding = embedding_response.data[0].embedding

    # Step 3: Semantic similarity lookup
    cached_response = lookup_cache(tenant_id, prompt, prompt_embedding, similarity_threshold)
    if cached_response:
        return {"response": cached_response, "cache_status": "semantic_hit", "tokens_saved": True}

    # Step 4: Cache miss - call the LLM
    llm_response = openai_client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}]
    )
    response_text = llm_response.choices[0].message.content

    # Step 5: Write result to cache asynchronously (non-blocking)
    ttl = resolve_ttl(tenant_id, agent_type)
    asyncio.create_task(
        async_write_to_cache(tenant_id, prompt, response_text, prompt_embedding, agent_type, ttl)
    )

    return {"response": response_text, "cache_status": "miss", "tokens_saved": False}

Notice the two-tier lookup strategy: an exact hash match runs first (zero vector store overhead), and the semantic search only runs when the exact match fails. In high-traffic deployments, this alone can save 15 to 25 percent of your vector store query costs.

Step 6: Tune Your Similarity Threshold Per Tenant and Agent Type

Shipping a single hardcoded similarity threshold is the most common mistake teams make when deploying semantic caches. Different tenants have different tolerance for "close enough" answers, and different agent types have wildly different semantic sensitivity.

A/B Testing Your Threshold

Run a shadow mode for the first two weeks after deployment. Log every semantic hit alongside the original prompt and the matched cached prompt. Have a small internal review panel (or an LLM-as-judge setup) score whether the cached response was actually appropriate for the incoming query. Use those scores to calibrate your threshold per agent type.

Adaptive Threshold by Confidence Score

A more sophisticated approach is to implement a confidence band rather than a hard threshold:

  • Score above 0.97: Serve cache hit immediately, no review.
  • Score between 0.90 and 0.97: Serve cache hit but append a soft freshness disclaimer in your UI layer (e.g., "Based on a recent similar query").
  • Score below 0.90: Treat as a cache miss, call the LLM.

Step 7: Observability and Cost Attribution

A caching layer without observability is a black box that will eventually erode trust. Instrument your cache with these key metrics, emitted per tenant:

  • Cache hit rate (exact + semantic, broken out separately)
  • Estimated tokens saved per tenant per day (multiply cache hits by average prompt + completion token count)
  • Average similarity score of semantic hits (a sudden drop signals prompt distribution drift)
  • TTL expiry rate (high expiry rate means your TTLs may be too aggressive)
  • Cache write latency (async write-back should never block your response path)
  • Tenant-level cost savings dashboard (this is a feature you can surface to tenants as a value-add)

Emit these metrics to your observability stack (Datadog, Grafana, OpenTelemetry, or whatever you are running) and set alerts for hit rate drops below your baseline. A sudden hit rate collapse usually means either a prompt format change in your agent or a tenant whose query patterns have fundamentally shifted.

Common Pitfalls and How to Avoid Them

Pitfall 1: Caching Agent Outputs That Contain Dynamic Tool Calls

If your agent's response includes the result of a live tool call (a database query, an API lookup, a web search), caching the final output is dangerous because the tool result may be stale. The fix: cache at the pre-tool-call reasoning step only, or tag tool-augmented responses with a very short TTL and a "live data included" flag that bypasses semantic matching.

Pitfall 2: Not Handling Tenant Offboarding

When a tenant churns or is deleted, their cache namespace must be purged. Build an explicit delete_tenant_cache(tenant_id) function into your offboarding workflow. Orphaned vector collections are not just a storage cost; they are a compliance liability under data deletion regulations.

Pitfall 3: Embedding Model Version Drift

If your embedding model provider releases a new version and you upgrade without invalidating existing cache entries, your old embeddings and new embeddings will live in incompatible vector spaces. The result is garbage similarity scores. Always version-stamp your embeddings in metadata and run a full cache flush on embedding model upgrades.

Pitfall 4: Ignoring System Prompt Variations

In multi-tenant systems, different tenants often have different system prompts that fundamentally change how the LLM should respond to the same user query. Cache keys must incorporate a hash of the system prompt, not just the user-facing prompt. Two identical user messages with different system prompts are not the same query.

Expected Results: What to Realistically Expect

Based on production patterns observed across multi-tenant AI SaaS products in 2026, a well-tuned semantic cache layer typically delivers:

  • 30 to 50 percent reduction in LLM API calls for customer support and FAQ-style agent workloads with high query repetition.
  • 15 to 30 percent reduction for analytics and reporting agents where queries are more varied but structurally similar.
  • Under 5 percent reduction for highly creative or open-ended generation tasks (document drafting, code generation from scratch). Do not over-invest in caching these.
  • P50 latency improvement of 200 to 800ms on cache hits, since you are eliminating the full LLM inference round-trip and replacing it with a vector search (typically 5 to 30ms) plus an embedding call.

Conclusion

Building a tenant-scoped semantic caching layer is not a micro-optimization. For multi-tenant AI products running at any meaningful scale in 2026, it is table-stakes infrastructure. The combination of tenant namespace isolation, embedding-based similarity deduplication, tiered TTL freshness policies, and two-tier exact-plus-semantic lookup gives you a system that is simultaneously cost-efficient, secure, and fresh enough for production use.

The key mindset shift is to stop thinking of your LLM as a function you call and start thinking of it as an expensive resource you protect with a smart cache. Your model provider charges you for every token whether it is redundant or not. Your semantic cache does not.

Start with a single high-volume agent type, instrument aggressively, tune your similarity threshold over two to four weeks of production traffic, and then roll the pattern across your full agent fleet. The ROI tends to be visible within the first billing cycle, and the architectural discipline it enforces, particularly around tenant isolation and TTL hygiene, pays dividends well beyond the cost savings.

Sign up for more like this.

Enter your email
Subscribe