← The AI Builder Stack

AI Features Inside Your Product

Your app doesn't need to be built by AI to be smart — it needs to call AI at the right moment, for the right price.

00 — The Shift You're Making

Every layer before this one was about using AI tools to build your product — prompting an assistant to write code, scaffold a database, wire up a deployment pipeline. The AI was your co-builder, and once it finished, it stepped back. Your app shipped as ordinary software.

This layer is different. Here, AI doesn't leave when the building is done — it stays in the product, running every time a real user hits a button. A support tool that drafts replies, a search bar that understands plain English, a form that fills itself in from a photo: these call an AI model live, in production, on your dime, for every single user action.

That's a new kind of decision. Up to now you were choosing a coding assistant. Now you're choosing a vendor whose model sits between your user and the value they came for — and whose bill scales with your success. Get this layer right and AI becomes a real feature. Get it wrong and you've bolted an expensive, flaky dependency onto your product for a feature nobody asked for.

01 — Two Different Questions, Easy to Confuse

"Should I use AI to build this app?" (layers 00–07) and "should this app use AI as a feature?" (this layer) are unrelated questions, and beginners routinely answer the second one on autopilot just because the answer to the first was yes. Building your app with an AI coding tool tells you nothing about whether your finished app should be calling an AI model at runtime.

A useful gut check: if you removed the AI, would the product still solve the user's problem, just less delightfully — or would it stop working entirely? A to-do list app with an AI-generated "motivational nudge" survives fine without it. A tool that summarizes long contracts does not. Knowing which one you're building changes everything downstream — how much reliability you need, how much you can afford to spend per use, and whether a launch delay from an API outage is a shrug or a crisis.

02 — What an LLM API Actually Is, in Plain Terms

An LLM (large language model — the type of AI behind ChatGPT, Claude, and Gemini) is, from your product's point of view, just a service you send text to and get text back. You send a request — some instructions plus the user's input — over the internet to a provider's servers, and a few seconds later you get a response. That request-response exchange is called an API call (API = application programming interface, the technical doorway your app uses to talk to someone else's software), and providers charge for it based on how much text goes in and comes out, measured in tokens (roughly ¾ of a word each).

That's the whole mental model you need: your app is a customer of a text-generation utility, billed by the token, the same way a hosting provider bills by the gigabyte. Everything else in this layer is about which utility to buy, how much of it to buy, and when to add more machinery around it.

03 — Landscape: Who Sells This, and How

As of September 2026, three providers dominate the mainstream LLM API market: Anthropic, OpenAI, and Google. Each sells a range of models trading off intelligence against speed and cost, and each has cut prices repeatedly as the field commoditizes — so treat exact numbers as a snapshot, not gospel, and always check the provider's own pricing page before budgeting a real feature.

Provider Notable current model(s) Roughly how pricing works Best for
Anthropic (Claude) Claude Sonnet 5 (balanced flagship), Claude Haiku 4.5 (fast/cheap), Claude Fable 5.1 (newest top-tier model, released Sept 2026) Pay per million tokens, separate input/output rates (Sonnet 5 around $2 in / $10 out per million tokens); cheaper "cached" and "batch" rates for repeat or non-urgent requests Careful reasoning, longer documents, coding-adjacent and agentic tasks
OpenAI (GPT) GPT-5.6 family, sold as three tiers — Sol (flagship), Terra (balanced), Luna (cheap/fast) Per-million-token, tiered by model — Sol priced highest, Luna roughly 25x cheaper than Sol per OpenAI's own numbers Broad general-purpose use, huge ecosystem/tooling, cheap high-volume tasks via Luna
Google (Gemini) Gemini 3.5 Flash / Flash-Lite (cheap, fast), Gemini 3.1 Pro (higher reasoning) Per-million-token; Flash-Lite is the cheapest tier of the three providers, Pro models cost more above certain context lengths High-volume, latency-sensitive features; tight Google Cloud/Workspace integration

Two things matter more than memorizing this table. First, every provider now sells a "cheap and fast" tier alongside its flagship — Haiku, Luna, and Flash-Lite all exist because most product features don't need the smartest model, they need the cheapest one that's good enough. Second, prices here are falling and models are renaming fast (note how odd some of these 2026 names — Fable, Sol, Terra, Luna — would have looked a year earlier); don't hard-code a model choice into your product plan, budget for a category ("a fast, cheap model" or "a frontier reasoning model") and let your engineer or AI coding tool pick the specific SKU at build time.

04 — How to Actually Choose a Model (Not by Benchmark)

Beginners default to picking whatever model scored highest on a benchmark they saw in a tweet. That's the wrong axis. A PM should choose along three dimensions, in this order:

1. Quality bar — how wrong can this be? A feature that drafts an internal Slack summary can tolerate an occasional miss. A feature that tells a user their medication interaction status cannot. Set your quality bar first, because it's the only thing that should ever push you toward a more expensive model.

2. Latency — how long can the user wait? A live chat reply needs a response in a second or two, which usually means a fast/cheap-tier model. A "generate my weekly report" button that runs in the background can tolerate a slower, smarter model because nobody is staring at a spinner.

3. Cost — what can this feature earn or save, per use? This is where most beginners skip straight to "use the best model" without ever running the math in Section 06. Don't.

The practical move: start with the cheapest, fastest model from any provider (Haiku-tier, Luna-tier, Flash-Lite-tier) and only upgrade if it visibly fails your quality bar in testing. Nearly every credible engineering source on this — including Anthropic's own published guidance on building agents — makes the same point: start simple, add capability only when you can point to the specific failure that justifies it.

05 — RAG, in Product Terms

RAG stands for Retrieval-Augmented Generation. Strip the jargon and it means: before the AI answers, your app first fetches the specific, relevant pieces of your own data and hands them to the model along with the question. Without RAG, the model only knows what it learned during training — which is generic, dated, and knows nothing about your company's documents, your product's data, or last week's events. With RAG, the model gets to read the right paragraph from your knowledge base before it answers.

You need RAG when your AI feature has to be right about your specific stuff — your company's policies, your customers' account history, your product catalog, documents your users upload. A support bot that answers questions about your actual refund policy needs RAG. A feature that just rephrases whatever text the user pastes in does not — the user is already supplying all the context the model needs.

The product-level tell: if you catch yourself wanting to say "make sure the AI only talks about our stuff" or "make sure it doesn't make up answers about things it can't know," that's a RAG need. If the model has everything it needs in the user's own message, skip it — RAG adds real infrastructure (a place to store and search your documents) and real cost, and it's not free complexity to carry for a feature that doesn't need grounding. The engineering mechanics of doing this well — chunking documents, storing them as searchable embeddings in a vector database, tuning retrieval quality — are exactly what the companion guide below goes deep on; your job at this layer is just to recognize when the need exists.

06 — Agentic Behavior: The Expensive Upgrade

An AI agent is a system where the model doesn't just answer once — it runs in a loop, deciding what to do next, calling tools (searching the web, querying a database, sending an email) and using the results to decide its next step, sometimes many steps in a row, before finally responding. Compare that to a single LLM call: one request in, one response out, done.

Agentic behavior is genuinely powerful for tasks that require multiple steps and judgment calls a human would otherwise have to babysit — researching a topic across several sources, booking something that requires checking availability then confirming, triaging a support ticket by pulling data from three systems before drafting a reply. But it is also slower, far less predictable, and dramatically more expensive, because a single "answer" might now involve five, ten, or more model calls chained together, each with its own cost and its own chance of going wrong.

The decision rule that matters for a PM: default to a single LLM call. Reach for agentic behavior only when you can name the specific multi-step task that a single call genuinely cannot do — not because agents are exciting, but because they're the load-bearing complexity of last resort. Anthropic's own published engineering guidance on this — aimed at the practitioners building these systems — opens with almost exactly this advice: find the simplest solution possible, and only increase complexity when it demonstrably improves outcomes. If your feature is "summarize this," "classify this," or "draft a reply to this," that's one call. If it's "figure out what's wrong and fix it across three systems," that might genuinely need an agent — but prove the simple version fails first.

07 — Cost-Per-Call Thinking

This is the habit most non-technical PMs skip, and it's the one that will bite hardest post-launch. Every AI feature has a marginal cost — the cost that recurs every single time a user triggers it — and unlike most software costs, it doesn't shrink to nearly zero at scale the way serving a static webpage does. Every AI call costs real money, every time.

Build a simple back-of-envelope model before you ship:

  • Estimate tokens per call. A short chat reply might be a few hundred tokens in and a few hundred out. A "summarize this 10-page document" feature might be several thousand tokens in.
  • Multiply by the per-million-token price for the model tier you've chosen (see Section 03) to get cost-per-call — often fractions of a cent for a cheap-tier model on a short exchange, but climbing fast with document length, chat history, or a smarter model.
  • Multiply by expected usage — calls per user per day/month, times your expected user count — to get a monthly AI bill.
  • Compare that bill to what the feature is worth — what it saves in support time, what it lets you charge for, what it does for retention. If the feature costs more per user than that user pays you, you don't have a feature, you have a subsidy.

Two levers keep this sane as you scale: cheaper models for the bulk of traffic (reserve the expensive model for the few cases that truly need it), and provider discounts like prompt caching (paying less to reuse context the model has already seen) and batch processing (a discount, often around half price, for requests that don't need an instant answer). None of this requires you to implement anything yourself — it requires you to ask your engineer or AI coding tool "what's our cost per call, and what's it going to look like at 10x our current usage?" before you commit to a pricing plan built on top of it.

08 — Common Mistakes Beginners Make

Reaching for agentic complexity before proving the simple version fails. Multi-step, tool-using AI agents are the most talked-about and least necessary starting point for most features. Start with one LLM call; add steps only when you've watched the simple version fail at something specific.

Ignoring cost until the bill arrives. Teams ship an AI feature, love the demo, and discover at 10,000 users that the "smart" model they picked for quality is now their largest infrastructure line item. Run the math in Section 07 before launch, not after.

Chasing the newest model instead of the right one. A new flagship model drops seemingly every month, and it's tempting to always want "the best one." Most product features don't need the newest frontier model — they need the cheapest one that clears your quality bar, which is very often last quarter's model, or the cheap tier of this quarter's.

Adding RAG (or a whole knowledge base) for a feature that doesn't need grounding. If the user's own input already contains everything the model needs to answer, retrieval infrastructure is pure overhead.

Treating "the AI model" as a fixed, permanent choice. Pricing and capability shift fast enough that a model you pick today may be replaced or repriced within the year. Ask your engineer to keep the model swappable rather than hard-wired into the product, so a better or cheaper option later is a config change, not a rebuild.

Where this leads

You now know how to decide whether and how your product should call AI, and roughly what that will cost — but a decision on paper isn't the same as a feature that reliably works for real users. The next layer, 09 — Testing, Debugging & Quality Control, covers how to actually verify an AI feature behaves the way you expect before it ships, and how to keep catching its failures — the wrong answers, the weird edge cases, the silent quality drift — after it's live.


For the engineering-depth version of everything in this layer — how RAG and vector databases actually work under the hood, how AI agents are architected, what MCP (Model Context Protocol) is, and how teams build guardrails and run evals to keep these systems safe and accurate — see the companion 7-layer guide, The AI Factory Stack.