Skip to content

Cost Engineering for AI

Introduction

Cost engineering designs a system to meet economic as well as technical limits. AI cost depends on tokens, model choice, traffic shape, hardware, and retries. This course covers token economics, caching, inference optimization, and deployment models. Afterward, you can estimate unit cost and choose architecture levers before implementation.

Why this matters for the AI SA

The AI Solutions Architect (AI SA) turns demand forecasts into capacity and cost decisions. They set token budgets, routing rules, cache boundaries, and buy-versus-build criteria. Small prompt choices can change annual spend by millions at scale. Designing for cost before build is the AI SA's signature move. Late cost cuts often damage quality because the architecture assumed unlimited inference.

Key concepts

Token economics

A token is a model's unit of text processing. Input tokens include instructions, history, retrieved context, and user text. Output tokens are generated by the model and often cost more. Providers publish prices per million tokens and may charge separately for cached input.

For illustration, assume $3 per million input tokens and $15 per million output tokens. A request with 8,000 input and 800 output tokens costs $0.036. One million such requests cost $36,000 before other services. These are illustrative figures, not current vendor quotes. Include retries, evaluations, moderation, embeddings, storage, and network transfer.

Caching

Prompt caching reuses computation for an identical stable prompt prefix. It works well for long system instructions or repeated reference material. Exact-response caching returns a previous answer for the same normalized request. Semantic caching returns an answer for a meaningfully similar request.

Semantic caching needs a similarity threshold and freshness policy. It can return the wrong tenant's or an outdated answer if keys are poorly designed. Include model version, permissions, locale, and source version in cache identity. Do not cache sensitive output without encryption and retention rules. Measure hit rate and correctness, not only saved calls.

Inference optimization levers

Use the smallest model that meets the quality gate. Distillation trains a smaller model to imitate a larger one. Quantization stores model numbers at lower precision to reduce memory and often increase speed. Both can reduce quality and need task-specific tests.

Batching combines requests so hardware processes them efficiently. It improves throughput but adds queueing latency. Limit output length and trim irrelevant context. Route easy tasks to a small model and difficult tasks to a stronger model. Difficulty routing needs a classifier and a safe fallback. Speculative decoding and optimized serving engines can improve self-hosted throughput.

Managed APIs and self-hosted GPUs

A managed API charges by usage and removes hardware operations. It is attractive for uncertain or bursty demand. Self-hosting converts much cost into hourly GPU capacity and engineering work. It can win at sustained high utilization or where control and data location dominate.

Model total cost of ownership, or TCO, not GPU rent alone. Include replicas, idle capacity, failures, serving software, monitoring, energy, and staff. Use measured tokens per second and expected concurrency. Plan headroom for peaks and maintenance. Compare equivalent quality, latency, context length, and availability.

Design before build

Start with requests per day, token distributions, peak concurrency, and quality targets. Calculate cost per successful task, not cost per call. Set budgets by feature and tenant. Expose token use and cache status in telemetry.

Design graceful degradation before overload. Options include shorter context, delayed batch processing, smaller models, and human queues. Revisit the forecast when prompts, traffic, or prices change. Avoid binding business logic to one provider's request format.

A worked example

A company expects 500,000 document summaries each month. Each starts with 20,000 input tokens and produces 1,000 output tokens. At illustrative prices of $3 and $15 per million tokens, one summary costs $0.075. The monthly model cost is $37,500.

The architect first removes boilerplate and reduces average input to 12,000 tokens. Prompt caching applies to 2,000 stable tokens with a 70% hit rate. Routing sends 80% of simple documents to a model costing one-third as much. After quality testing, estimated monthly model cost falls to about $15,000. The exact result depends on cache pricing and routing performance.

A self-hosted option needs two GPUs for availability at an illustrative $2.50 per GPU-hour. Raw monthly compute is about $3,650, but four GPUs are needed at peak. Operations, idle capacity, and a second region raise TCO. The team chooses managed APIs until demand is stable and measured.

Reference architecture for this example

One valid answer. Draw it yourself first, then compare: what did you include that this omits, and what does this include that you missed?

flowchart LR
  A["500,000 docs per month 20,000 input tokens"] --> B["Remove boilerplate 12,000 tokens"] --> C["Cache 2,000 stable tokens 70% hit rate"] --> D["Router"]
  D -->|80% simple| E["Cheap model one-third price"]
  D -->|20% complex| F["Premium model"]
  E --> G["Quality spot checks"]
  F --> G
  G --> H["Hosting decision"]
  H --> I["Managed API"]
  H --> J["Self-hosted 2 GPUs baseline 4 peak"]

Lab

Build a free spreadsheet or Python cost model.

  1. Define low, expected, and high monthly request volumes.
  2. Record p50 and p95 input and output token counts from 50 sample requests.
  3. Enter illustrative prices for a small and a large model.
  4. Include a 5% retry rate and one evaluation call per 20 requests.
  5. Calculate cost per request, successful task, month, and peak day.
  6. Add prompt-cache hit rates of 0%, 30%, and 70%.
  7. Add routing rates between the two models.
  8. Reject any route that fails a fixed quality threshold.
  9. Model one rented GPU with hourly price, measured throughput, and 50% utilization.
  10. Add a second replica for availability and 30% peak headroom.
  11. Find the utilization where self-hosting reaches cost parity.
  12. Run sensitivity analysis on traffic, output length, and GPU utilization.
  13. Write a recommendation with assumptions and a review date.

Note

Provider prices change. Keep prices as editable inputs and date every estimate.

Check your understanding

  1. Why should output tokens be forecast separately from input tokens?
  2. Which fields belong in a safe semantic-cache key?
  3. How can batching reduce cost while increasing latency?
  4. What costs are missing from a simple GPU hourly comparison?
  5. Why is cost per successful task stronger than cost per call?

Further reading

Conclusion

AI cost is an architecture property, not a purchasing afterthought. The AI SA makes unit economics visible early and preserves quality while applying the right optimization lever.