What Is an AI Agent?

Question

What is an AI agent and what are its main cost components?

Quick answer

Formula: cost_per_workflow = llm_calls + retrieval + reranking + tool_calls + memory + infra

• Assumption: not every agent uses every block, so only model the steps that actually execute.
• Assumption: multi-step loops can multiply LLM and retrieval cost faster than teams expect.
• Assumption: tool calls and memory lookups should stay explicit instead of being hidden inside one average.

Example: if llm_calls=$0.024, retrieval=$0.002, tool_calls=$0.006, memory=$0.000, and infra=$0.001, cost_per_workflow=$0.033 before monthly aggregation.

Common Building Blocks

• Planning or routing call: decides which step happens next.
• Retrieval: fetches relevant context from a vector store or search index.
• Reranking: improves relevance before synthesis when recall is noisy.
• Tool calling: hits APIs, databases, or actions outside the model.
• Memory or state: stores prior workflow data or fetches prior context.
• Synthesis: final answer generation after the earlier steps complete.

How Agents Differ From Prompt-only Apps

• They usually execute multiple paid steps instead of one model call.
• Branching logic makes cost less intuitive because not every request follows the same path.
• Retries, fallbacks, and tool failures can widen the spread between average and p90 cost.

Why Unit Economics Matter

• Small changes in retrieval depth or model choice compound across multi-step workflows.
• Gross margin can look healthy on light traffic and collapse when retries or long sessions increase.
• Explicit cost blocks make pricing and rollout decisions defensible before launch.