Engineering · 28 Aug 2026

Moving our chat to a cheaper model

We set a quality bar before we switched, measured against it for 57 days, and kept the change.

We were defaulting Finautor AI's chat to Opus, the premium tier, and it was the largest line in our inference bill. On 1 July 2026 we moved chat to Claude Sonnet 5. Before we flipped it, we wrote down what would make us switch back.

We set the bar before we flipped, and it held

The gates were declared on 1 July 2026 — the same day the switch went live, before the measurement window opened — with a named decision date and a revert that was a single environment variable. Over the following 57 days and 695 primary chat turns:

Verdict: keep. We never used the revert.

The pre-registration is the point. Anyone can look back after a change and say nothing broke. Deciding the pass conditions in advance, and publishing them with a revert already wired, is what makes this evidence rather than reassurance.

What it saved

Cost fell about 40% per chat turn, from $0.20 to $0.12. That number is per chat turn, at list price, measured against a 7-day baseline, and it excludes cache-tier and unlogged background calls.

We are deliberately not quoting the drop in our weekly bill. Chat volume moved materially over the same period, so week-over-week totals are not like-for-like and would flatter us. Cost per turn is the comparison that holds.

Shorter answers are part of that number

Average output fell from 3,499 to 2,777 tokens — about 21% shorter. That was not one of the gates, and it is not a separate win to bank: a cheaper model that also answers more briefly is two effects inside one cost number, and we would rather say so than have someone find it. The four gates above are the check on whether the shorter answers still finished the job. They did.

A second experiment, with a result we cannot yet claim

Later, and separately, we tested prompt compression as a randomised concurrent control — 37 treated requests against 36 untreated, over 130.8 hours, at roughly a 49/51 split.

• Prompt tokens: −12.2%, 95% CI [−20.7%, −2.8%]. The interval excludes zero, so the token reduction is a real effect.
• Cost per request: −6.9%, 95% CI [−26.0%, +17.1%]. The interval contains zero, so the cost effect is not yet resolvable.

We are not claiming a cost saving from this experiment. The token reduction is measured; the cost effect is not. The prediction going in was a 15% prompt reduction; we measured 12.2%.

Why we built Bayeto

Doing this properly — a cost basis you can defend, a stated baseline window, a scope you can put in one sentence — was most of the work. That is why we built Bayeto.

Bayeto is our own project, built by the same people, so it is not an outside auditor and we will not present it as one. What it does is measure cost. It does not measure quality: the four gates above are ours, run in our own system, and they are the part that says the cheaper model still did the job.

Bayeto's case study covers this same migration from the cost side. Its headline figure is measured on a different basis from ours — a different unit, cost basis and baseline window — and its page sets out that difference rather than leaving you to reconcile two numbers. Run today against our pre-switch data, its engine proposes the same move and over-predicts the saving: €0.1883 per call against a measured €0.1377. It publishes that miss on its own page.