Auditing Prompt Caching

Analysis

Why This Paper Matters

Prompt caching is a widely used optimization in large language model (LLM) APIs to reduce latency and cost by reusing cached responses for identical or similar prompts. However, this paper reveals a critical and often overlooked security implication: caching introduces data-dependent timing variations that can be exploited as side-channel attacks. By measuring API response times, an attacker can determine whether a prompt was cached, potentially leaking sensitive information about other users' queries. This is particularly concerning when caches are shared globally across all users, as detected in seven major API providers including OpenAI.

The paper's significance extends beyond privacy leakage. The same timing variations can inadvertently reveal details about the underlying model architecture. The authors provide evidence that OpenAI's embedding model is a decoder-only Transformer, a fact that was previously not publicly known. This demonstrates that side-channel information can bypass corporate secrecy around model design, which has implications for competitive intelligence and model security.

Technical Contributions

The paper introduces a novel statistical auditing framework to detect prompt caching in black-box LLM APIs. Key innovations include:

• Statistical tests for caching detection: The authors design experiments that compare response times for repeated versus novel prompts, using statistical hypothesis testing to determine if caching is present.
• Global cache sharing detection: By sending prompts from different user accounts and measuring timing correlations, they can infer whether the cache is shared across users.
• Model architecture inference: Timing patterns from caching behavior are used to deduce architectural properties, such as the decoder-only nature of OpenAI's embedding model.

Results

The audits revealed that seven out of the tested API providers, including OpenAI, employ global cache sharing across users. This means that an attacker could potentially learn about other users' prompts by observing fast response times for cached inputs. Additionally, the timing variations provided evidence that OpenAI's embedding model is a decoder-only Transformer, contradicting prior assumptions or lack of public disclosure. The paper does not provide exact metrics like false positive rates or effect sizes, but the statistical significance of the findings is established through rigorous auditing.

Significance

This work has broad implications for the AI industry. It underscores the need for transparency from API providers regarding their caching policies, as well as the potential privacy risks of shared caches. The findings also highlight that side-channel attacks are not limited to traditional cryptographic systems but extend to modern LLM services. For practitioners, this means that deploying prompt caching without careful consideration of privacy implications could lead to data leakage. The paper calls for more robust auditing tools and potentially for cache isolation mechanisms to protect user privacy. Furthermore, the ability to infer model architecture through timing side channels opens new avenues for model reverse engineering, which could impact intellectual property and security.