AI2 min read
Adding an LLM feature without betting the product on it
Large language models can remove hours of busywork — if you pick a narrow task, measure quality honestly and design for the cases where the model is wrong.
Almost every client conversation now includes the question "should we add AI?" Sometimes the answer is yes. Sometimes it's "not yet". Either way, the projects that succeed tend to follow the same pattern.
Start with a narrow, checkable task
"An assistant that knows everything about our business" is a research project. "Extract the renewal date, notice period and total value from a supplier contract" is a feature. Narrow tasks are easier to evaluate, cheaper to run and far easier to explain to users.
Keep a person in the loop where mistakes are expensive
Models are confident even when they're wrong. For anything with financial, legal or safety consequences, the model should propose and a person should approve. Good interface design makes that review fast: highlight the source text, show the model's confidence, and make corrections one click away.
Measure before and after
Before building the feature, we assemble an evaluation set — real examples with known correct answers — and score every prompt or model change against it.
{
"input": "contracts/acme-2025.pdf",
"expected": {
"renewal_date": "2026-03-01",
"notice_period_days": 60,
"total_value": 48000
}
}
Without an evaluation set, "it seems better" is the only metric you have, and it won't survive the first awkward edge case.
Plan for cost and latency
Model calls are slower and pricier than a database query. A few habits keep them under control:
- Cache results for identical inputs
- Use the smallest model that passes your evaluation set
- Stream responses so users see progress immediately
- Set hard per-user and per-day limits from the start
Ship it like any other feature
Put it behind a feature flag, roll it out gradually, and monitor it: error rates, latency, cost per request and how often users accept or correct the output. That last number is the most honest quality signal you'll get.
AI features don't need special treatment so much as ordinary engineering discipline applied to an unusually unpredictable component. Do that, and the "AI" part becomes the least risky thing in your release.
About the author
Ravi Menon · ML Lead
Ravi builds practical AI features and the evaluation pipelines that keep them honest, from document extraction to search and recommendations.