Building an AI product has taught me to ask one question much earlier than most teams do:
What does one successful request actually cost?
Not just: How much did the model API call cost?
The true cost of an AI workflow can include model inference, input and output tokens, embeddings, retrieval, reranking, tool calls, retries, fallback models, observability, storage, infrastructure, and sometimes several model calls before the user receives one useful result.
That distinction becomes increasingly important as an AI product moves from prototype to production.
A workflow that looks inexpensive at 100 requests per day can become a very different business at 100,000 requests per day.
And this is where AI engineering becomes a unit-economics problem.
Cost per API call is not the same as cost per user task
One of the easiest mistakes to make when estimating AI costs is treating a single LLM call as the unit of work.
Imagine a user asks an AI application:
“Review this contract and identify the major risks.”
From the user's perspective, that is one request. But internally, the system might:
- 01Parse the uploaded document.
- 02Create or retrieve embeddings.
- 03Search a vector database.
- 04Rerank retrieved passages.
- 05Send context to an LLM.
- 06Call another model to classify identified risks.
- 07Use a tool to retrieve company policy.
- 08Ask the LLM to produce the final response.
- 09Retry one step because the first response failed validation.
- 10Record traces, metrics, and outputs for monitoring.
The user completed one task.
Your infrastructure may have executed several billable operations.
That means the more useful metric is often cost per successful task rather than cost per model call.
A practical AI cost model
At a high level, the variable cost of an AI workflow can be represented as:
Let's break that down.
This is usually the most visible cost. For token-priced models, the basic calculation is: Input Cost = Input Tokens × Input Token Rate. Output Cost = Output Tokens × Output Token Rate. Then: Model Cost = Input Cost + Output Cost.
But even this is more complicated than it first appears. A request may contain far more than the user's question.
- →System instructions
- →Conversation history
- →Retrieved documents
- →Tool definitions
- →User input
- →Structured output instructions
- →Examples
- →Previous agent state
A 30-word user question can therefore become a request containing thousands of tokens. The visible user prompt is not necessarily what you are paying to process.
Repeated context creates another important optimization opportunity. If every request includes the same long system prompt, tool definitions, schemas, instructions, or document prefixes, repeatedly processing all of that context can become expensive.
Major model providers now support forms of prompt caching that allow repeated prompt prefixes or context to be processed at different rates from entirely new input. OpenAI documents cached-input pricing and prompt caching for supported models, while Anthropic similarly supports prompt caching and documents separate cache behavior and pricing.
Caching works especially well when a meaningful portion of the request stays consistent.
- →Large system prompts
- →Policy documents
- →Tool definitions
- →Long application instructions
- →Stable reference material
- →Repeated document context
But caching should be an architectural decision, not simply a checkbox.
- →What portion of the prompt is reusable?
- →How frequently is it reused?
- →How long does the cache remain useful?
- →What happens when the underlying information changes?
- →Does caching reduce cost enough to justify additional complexity?
Retrieval costs in RAG systems
A RAG application has costs that exist outside the final LLM call.
Depending on the architecture, a request may require:
The query may need to be converted into an embedding.
The application queries a vector database or search infrastructure.
Keyword search and vector search may both run.
A reranking model may score the retrieved documents before sending them to the primary LLM.
Retrieved chunks are assembled into the final model prompt.
Each individual component may appear inexpensive.
At production volume, they accumulate.
This is why RAG optimization is not only about improving retrieval quality. You should be evaluating retrieval quality + latency + cost together.
Retrieving 30 documents and sending all 30 to an expensive model might increase the likelihood that the answer is grounded, but it may also dramatically increase input-token consumption.
The engineering problem becomes:
What is the smallest amount of high-quality context required to reliably complete the task?
Agent and tool-call costs
Agentic systems introduce another layer of cost.
A simple LLM request might require one inference call. An agent might:
- →Plan
- →Search
- →Call a database
- →Inspect the result
- →Call another tool
- →Reflect
- →Replan
- →Call another model
- →Validate
- →Generate a final response
One user action may therefore produce five, ten, or more model interactions.
That does not automatically make agents bad.
But it makes execution depth an important economic metric. Track:
- →Average model calls per task
- →Average tool calls per task
- →Average tokens per completed task
- →Average retries per task
- →Average cost per completed task
You may discover that the expensive part of the product is not your primary model.
It is uncontrolled orchestration.
Retries and failure costs
This is one of the most overlooked parts of AI unit economics. Suppose the first model response fails schema validation. Your system retries.
The second response calls a tool incorrectly. It retries again.
Eventually, the user receives a correct answer.
From the user's perspective: one successful request. From your billing perspective: three attempts.
This is why I like separating cost per attempt from cost per successful task. Suppose:
- → Average attempt cost = $0.04
- → Average attempts per completed task = 1.4
before adding other infrastructure.
Multiply small differences like that across millions of requests and reliability becomes a financial concern, not merely a technical one.
Observability is part of the cost stack
Production AI needs observability. Teams may need to capture:
- →Traces
- →Prompt versions
- →Model responses
- →Tool activity
- →Retrieval results
- →Token consumption
- →Latency
- →Evaluation results
- →Safety events
- →Error logs
- →User feedback
Those systems also cost money.
The mistake is not paying for observability.
The mistake is excluding observability from your product economics and then being surprised by infrastructure spend later.
If a capability is required to safely operate the product, it belongs in the cost model.
What should you actually measure?
A useful AI cost dashboard should move beyond total monthly API spend.
I would track several levels.
- →Input tokens per request
- →Output tokens per request
- →Cached input
- →Model calls per request
- →Tool calls per request
- →Retrieval operations
- →Average latency
- →Cost per attempt
These help engineering understand what is driving spend.
- →Successful completion rate
- →Retries per task
- →Failure rate
- →Escalation rate
- →Average cost per successful task
- →Cost by workflow
- →Cost by feature
This is where the economics become much more useful.
Imagine two features:
Cost per request: $0.03
Successful completion: 60%
Cost per request: $0.04
Successful completion: 95%
Feature A looks cheaper if you only examine request cost.
The more expensive model call produced the cheaper successful outcome.
This is exactly why optimization cannot be separated from evaluation.
AI cost optimization must be evaluated against quality
One of the worst optimization strategies is: move everything to the cheapest model.
Lower model cost does not necessarily mean lower system cost. A cheaper model may generate:
The correct comparison is not: Model A costs less than Model B. It is:
Which model gives us the best cost-quality tradeoff for this specific task?
This is where evaluation becomes part of financial optimization.
Model routing: not every task needs your best model
One of the strongest optimization patterns is routing workloads based on complexity.
Your most capable model does not necessarily need to handle every task.
- ✓Classify basic requests
- ✓Extract simple structured fields
- ✓Detect language
- ✓Route support requests
- ✓Summarize short text
- ✓Perform straightforward transformations
- →Complex reasoning
- →Difficult synthesis
- →Ambiguous decisions
- →High-value generation
- →Complex agent steps
- →Cases where smaller models fail evaluation thresholds
This turns model selection from a one-time architecture decision into an optimization system.
But model routing needs evaluation
Routing only works if you know when each model is good enough.
For every task category, establish:
- →Quality threshold
- →Accuracy requirement
- →Latency requirement
- →Safety threshold
- →Cost threshold
- →Escalation conditions
Then measure the tradeoff. For example:
| Model | Task Success | Avg. Cost | Avg. Latency |
|---|---|---|---|
| Model A | 96% | $0.08 | 2.8 sec |
| Model B | 93% | $0.03 | 1.7 sec |
| Model C | 78% | $0.01 | 1.1 sec |
The answer is not automatically Model A.
If 93% meets the product's quality threshold, Model B could create dramatically better economics. For another workflow, that 3-point difference may be unacceptable.
Optimization is contextual.
Reduce context before reducing capability
Large context windows can make it tempting to send everything to the model.
That can quickly become expensive.
Before downgrading model quality, examine whether you are paying to process unnecessary context. Look for:
- →Repeated instructions
- →Excessive conversation history
- →Duplicate retrieved chunks
- →Overly large documents
- →Unnecessary few-shot examples
- →Tool definitions irrelevant to the current task
- →Verbose metadata
- →Large structured payloads
- →Retrieval results the model never needs
A better retrieval strategy or tighter prompt can sometimes produce more savings than switching models.
Control output length
Output tokens can be materially more expensive than input tokens depending on the model and provider.
Do users actually need 1,500-word responses? For many workflows, they may need:
- →Five extracted fields
- →Three recommendations
- →A 100-word summary
- →A JSON object
- →A classification
- →A short explanation
Product design influences inference cost. Clear output constraints can improve:
- →Cost
- →Latency
- →Consistency
- →Evaluation
- →User experience
Do not pay the model to generate content nobody needs.
Use batch processing when real-time responses are not required
Not every AI workload needs synchronous inference. Examples include:
- →Overnight document classification
- →Bulk summarization
- →Periodic content enrichment
- →Dataset labeling
- →Offline evaluations
- →Large-scale extraction
- →Scheduled reporting
Both OpenAI and Anthropic currently document batch-processing options designed for asynchronous workloads, with pricing advantages for supported workloads.
The product question is:
Does this task truly need to happen while the user waits?
If not, asynchronous processing may provide both operational and economic benefits.
Cost per user matters more than cost per request
Once you understand request and task economics, move one level higher. Suppose:
- → Average successful task cost = $0.05
- → Active user completes 40 tasks per month
Variable AI cost per active user: $0.05 × 40 = $2.00/month
If the customer pays $20/month, that may look healthy. But now include:
- →Hosting
- →Databases
- →Search infrastructure
- →Monitoring
- →Payment fees
- →Customer support
- →Other variable services
Maybe true variable cost is $4 per user. Your gross contribution before fixed operating expenses becomes $20 – $4 = $16.
Now consider a power user completing 500 AI tasks per month. At the same $0.05: 500 × $0.05 = $25.
Your $20 subscription is already underwater before considering anything else.
That is why unlimited AI usage can become dangerous.
AI products need usage distribution, not just average usage
Averages hide expensive users.
Suppose your average customer generates 40 requests. But usage actually looks like:
- →60% generate fewer than 20
- →30% generate 20–100
- →8% generate 100–500
- →2% generate more than 1,000
That final 2% can materially change product economics. Analyze:
- →Median usage
- →P75
- →P90
- →P95
- →P99
- →Maximum usage
- →Cost by customer segment
Data science becomes extremely useful here.
You can model usage distributions, identify expensive cohorts, forecast future spend, and understand whether pricing reflects actual consumption.
From AI cost analysis to monetization strategy
Once cost per successful task and cost per user are known, pricing becomes much easier to reason about. Common AI monetization models include:
Users pay a fixed recurring fee.
Best when usage is relatively predictable.
Risk: Heavy users can destroy margins.
Customers pay based on consumption.
Examples:
- →Requests
- →Documents
- →Minutes
- →Tokens
- →Generated assets
- →Completed workflows
This directly links revenue to usage but can make customer bills less predictable.
Users purchase or receive a fixed number of credits.
Different workflows can consume different credit amounts.
Credits provide flexibility without exposing raw token economics to customers.
Example: Basic — 100 AI tasks/month. Pro — 500 AI tasks/month. Business — 2,500 AI tasks/month.
This creates predictable packaging while controlling exposure.
A subscription includes baseline usage, followed by overage pricing. For many AI SaaS products, this can balance predictable recurring revenue with variable infrastructure cost.
The metric I would put on every AI product dashboard
I would want teams to know:
Cost per successful user outcome
Not: How many tokens did we spend?
But: How much did it cost us to deliver the thing the customer came here to accomplish?
That metric connects engineering performance directly to product economics.
Connect cost to business value
Cheap AI is not necessarily good AI.
An AI workflow costing $2 per task could be extremely profitable if it replaces $50 of manual work. Another workflow costing $0.03 may be financially pointless if nobody values the output.
The relevant question is:
What economic value does this workflow create relative to what it costs to deliver?
A simple framework is: Value Created per Task ÷ Cost per Successful Task
This can help teams prioritize which AI workflows deserve continued investment.
AI cost optimization is also a governance problem
Cost optimization is usually treated as an engineering concern.
It also belongs in AI governance. Why?
Because unchecked autonomy can create unchecked spend.
An agent allowed to repeatedly:
- →Search
- →Reason
- →Call tools
- →Generate
- →Retry
- →Delegate to sub-agents
can consume substantial resources from a single request.
Operational controls may therefore include:
- →Maximum model calls
- →Maximum retries
- →Token budgets
- →Tool-call limits
- →Workflow timeouts
- →Cost thresholds
- →Escalation rules
- →Human approval for high-cost actions
- →Per-user usage limits
- →Per-workflow budgets
A practical AI unit economics framework
Before scaling an LLM-powered product, I would want clear answers to five levels of cost.
What does one execution attempt cost?
What does completing the user's intended task cost after accounting for retries and failures?
How much variable AI cost does an average active user generate?
Which cohorts generate the highest cost and the most value?
What happens when users double, requests per user increase, model pricing changes, context grows, agent complexity increases, or usage shifts toward power users?
Then run scenarios.
This gives leadership a much clearer picture than looking at this month's API invoice.
A simple optimization checklist
Before switching models, review the entire workflow.
- →Are we using more model capability than the task requires?
- →Can tasks be routed across models?
- →Are fallback models necessary?
- →Is repeated context being resent?
- →Can prompt caching help?
- →Is retrieved context larger than necessary?
- →Are outputs unnecessarily verbose?
- →Are we retrieving too many chunks?
- →Do we need reranking for every query?
- →Can retrieval quality improve enough to reduce context?
- →How many model calls does one task require?
- →Are tool calls actually necessary?
- →Are agents looping or retrying unnecessarily?
- →Can deterministic logic replace some agent steps?
- →Which workloads need real-time inference?
- →Which can run asynchronously?
- →Are we storing unnecessary traces or outputs?
- →Are observability costs proportional to their value?
- →What does one successful task cost?
- →Which users generate the highest variable cost?
- →Does pricing reflect usage?
- →Are usage limits appropriate?
- →What margin does each plan generate?
The bigger lesson
AI cost optimization is not simply: Use fewer tokens.
And it is not: Choose the cheapest model.
The real problem is finding the best combination of:
Quality + Cost + Latency + Reliability + Safety + Business Value
A technically impressive AI system can still be a bad product if its economics do not work.
And an inexpensive system can still be a bad product if quality is too low to create value.
The strongest AI systems are designed so that technical performance and economic performance can be evaluated together.
Before scaling, understand:
That is where AI engineering becomes AI product economics.
About the Author. Tobe Awosanya — AI Engineering Leader · Data Scientist · Technical AI Advisor · Founder, Data Techcon. Tobe works across data science, machine learning, AI engineering, responsible AI, product analytics, and production AI systems. Her work focuses on connecting technical decisions — including model selection, evaluation, architecture, governance, observability, and AI economics — to measurable product and business outcomes. Through Data Techcon, she advises organizations and builds practical Data & AI learning experiences for professionals, founders, and technical teams.
Download the AI Unit Economics Worksheet
Work through cost per attempt, cost per successful task, cost per active user, cost by segment, and margin at scale for your own AI workflow.
Get the worksheetNeed to evaluate the economics of an AI system?
If your team is building or scaling an AI product and needs help understanding model costs, routing decisions, token usage, architecture tradeoffs, unit economics, or monetization strategy, Data Techcon AI Strategy Advisory provides focused strategic and technical support. Our AI Cost Optimization & Unit Economics advisory work helps teams connect LLM architecture decisions to quality, scalability, pricing, and sustainable product economics.