Skip to content
Algolithic
All articles

AI2 min read

Adding an LLM feature without betting the product on it

Large language models can remove hours of busywork — if you pick a narrow task, measure quality honestly and design for the cases where the model is wrong.

Ravi MenonML Lead

Almost every client conversation now includes the question "should we add AI?" Sometimes the answer is yes. Sometimes it's "not yet". Either way, the projects that succeed tend to follow the same pattern.

Start with a narrow, checkable task

"An assistant that knows everything about our business" is a research project. "Extract the renewal date, notice period and total value from a supplier contract" is a feature. Narrow tasks are easier to evaluate, cheaper to run and far easier to explain to users.

Keep a person in the loop where mistakes are expensive

Models are confident even when they're wrong. For anything with financial, legal or safety consequences, the model should propose and a person should approve. Good interface design makes that review fast: highlight the source text, show the model's confidence, and make corrections one click away.

Measure before and after

Before building the feature, we assemble an evaluation set — real examples with known correct answers — and score every prompt or model change against it.

{
  "input": "contracts/acme-2025.pdf",
  "expected": {
    "renewal_date": "2026-03-01",
    "notice_period_days": 60,
    "total_value": 48000
  }
}

Without an evaluation set, "it seems better" is the only metric you have, and it won't survive the first awkward edge case.

Plan for cost and latency

Model calls are slower and pricier than a database query. A few habits keep them under control:

  • Cache results for identical inputs
  • Use the smallest model that passes your evaluation set
  • Stream responses so users see progress immediately
  • Set hard per-user and per-day limits from the start

Ship it like any other feature

Put it behind a feature flag, roll it out gradually, and monitor it: error rates, latency, cost per request and how often users accept or correct the output. That last number is the most honest quality signal you'll get.

AI features don't need special treatment so much as ordinary engineering discipline applied to an unusually unpredictable component. Do that, and the "AI" part becomes the least risky thing in your release.

  • LLMs
  • AI
  • Evaluation

About the author

Ravi Menon · ML Lead

Ravi builds practical AI features and the evaluation pipelines that keep them honest, from document extraction to search and recommendations.

↳Keep reading

More from the workbench.

Engineering2 min read

Shipping weekly without breaking things

Small pull requests, feature flags and a deliberately boring CI pipeline let us release every week — even on products with thousands of daily users.

By Theo Park

Product2 min read

How we scope an MVP in two weeks

A short, structured discovery turns a big idea into a plan you can build, budget and test — without a 60-page spec nobody reads.

By Mara Quinn