RAG vs Long-Context: Cost Tradeoffs

Question

How should I compare RAG against long-context prompts for cost and margin?

Quick answer

Formula: architecture_delta = cost_long_context - cost_rag

Formula: request_input_tokens_delta = long_context_input_tokens - (base_prompt_tokens + retrieved_chunks * tokens_per_chunk)

• Assumption: compare both architectures on the same task quality bar, not raw prompt size alone.
• Assumption: long-context removes retrieval, reranking, vector-query, and embedding-refresh stack terms.
• Assumption: embedding refresh is a fixed monthly term and should not be multiplied by active users.

Example: if RAG sends 2,980 request input tokens and the long-context alternative sends 3,400, you still compare whether the extra prompt cost is smaller than the retrieval stack it replaces.

When RAG Usually Wins

• Retrieved context is narrower than the long prompt you would otherwise stuff into every request.
• Corpus freshness matters enough that targeted retrieval is still cheaper than very large prompts.
• Cache reuse is high and retrieval quality keeps retries low.

When Long-Context Can Win

• The relevant context is already compact enough to fit in one explicit prompt without wide retrieval.
• You can remove reranking, vector lookups, and monthly embedding refresh without quality loss.
• Model quality and latency stay acceptable even with the larger prompt window.

What To Compare Before Switching

1. Total request input tokens under both architectures.
2. Retrieval-stack cost removed by the long-context alternative.
3. Latency, answer quality, and failure-mode differences on sampled traffic.