Semantic cache

Answer repeated and similar requests from a cache, to cut cost and wait time, without crossing governance boundaries.

This is in development. Details may change before it is available.

The semantic cache answers a request from an earlier answer when the request is the same as, or close enough in meaning to, one Altrum has answered before. The answer comes back faster and costs no model tokens. It works for text, chat and Responses requests, streamed or not.

Switch it on

The cache is off by default. Switching it on keeps your organization’s answers in its own cache for the time you set. That is separate from keeping prompts and answers in your logs, which has its own switch.

• For organization and personal keys: an owner or admin opens AI Gateway, Settings, Cache and switches on Answer repeated requests from the cache.
• For an AI system’s keys: switch it on or off on the AI system’s AI Gateway tab, or leave it to follow the organization’s setting.

These choices belong to the organization and apply to every AI system under it:

Setting	Values	Default
Find similar requests with	A provider you hold OpenAI, Azure, Bedrock, Gemini or Vertex credentials for, and its Embedding model, which measures how close two requests are. None: exact matches only finds only requests that match word for word.	None
Similarity	0.5 to 1. A higher number needs a closer match.	0.8
Keep answers for	5 minutes, 1 hour, 1 day or 7 days.	5 minutes

Boundaries

• Each AI system has its own cache, and your organization’s other keys share one. A personal key reads and writes only its own person’s answers. An answer is never served across those lines, and never to another organization.
• A cached answer still passes your guardrails on answers. If the guardrail changed after the answer was stored, the new rules apply to it.
• Entries are deleted when their time is up. Owners and admins can also Clear cache at any time: in Settings, Cache for the organization’s keys, or on an AI system’s AI Gateway tab for that AI system. Each asks first.
• If the cache is not available, requests are answered as if it were off, and the Cache settings say so.

What it costs and how it shows

• A request answered from the cache is marked cache_hit in the request log and has no model cost.
• Measuring similarity uses your provider credentials. Its small cost is added to the request that needed it and counts against your budgets. A request that repeats an earlier one word for word needs no measuring and costs nothing.

Request headers

A request can only make the cache stricter than your settings.

Header	Effect
x-alt-cache-key	Keeps this request’s answers apart from others in the same cache, for example per customer.
x-alt-cache-ttl	Keeps the answer for less time than your setting: 30s, 5m, 1h or a number of seconds.
x-alt-cache-threshold	Asks for a closer match than your setting.
x-alt-cache-type	direct to match word for word only, or semantic to match by meaning only.
x-alt-cache-no-store	true serves from the cache if it can but does not store this answer.