The half-second we refused: RAG latency vs. accuracy

Why we turned down an AI chatbot that was twice as fast: RAG latency versus accuracy

In early October, I had two ways to make Attlas nearly twice as fast within reach. The first was a single line of code. The second meant switching AI models. I turned on neither.

The first one cut the thinking time the model takes before it writes. The first word of the answer arrived in just over half a second. Then I read the answers. A visitor was asking which package to choose for an 80-guest party. The fast version recommended the "up to 75 people" package.

The second one, Claude Haiku 5.5, answered 35% faster than our current model, at roughly a quarter of the cost per response. But on some questions, instead of answering, it threw the question back at the visitor and asked what they were interested in...

Here's why I turned both down, and the ten days of measurement that came before those decisions:

• 13 leads explored
• 4 changes shipped to production
• 7 ideas abandoned

The abandoned ones are the part I'm proudest of

The rules of the game: measure before you touch anything

An AI chatbot never answers the same way twice. If you change a setting and hand-test three questions, you end up seeing what you want to see. So I set three rules before changing anything.

1. No change without a baseline. Every idea starts with a reference number, taken from the live version.
2. The pass mark is written before the exam. For example: "we keep it if accuracy doesn't drop by more than one point." If the result lands just short, we don't adopt it. We don't rerun the test to see if it passes the second time.
3. We test on real questions, alongside production. I put together a set of 120 real visitor questions, drawn from 26 client chatbots. They stay out of the source code and never leave my machine.

To score answer relevance at scale, I use a "judge" AI. I didn't trust it from the start. I checked it against cases where I knew the right score, then compared it with a second judge from a different model family. The two agreed in 95% of cases.

One last, very concrete constraint: Attlas is self-funded, and its database runs on a monthly data-transfer cap. Every test had to stay light. That constraint forced me to have the database do the computing itself rather than pulling data over, which actually made the tests cleaner.

What measuring turned up first

Before I gained a single millisecond, measuring surfaced bugs nobody had spotted. Not me, not the users.

Corrections weren't a priority. They became just another routine item, like any other. But a feature we'd promised wasn't doing what it said it did. That's fixed now.

Duplicates were pushing the right answers out. For each question, Attlas gets a limited number of passages to read, like a reading budget. When duplicates take up the space, the passage that holds the answer can end up outside the budget. After the cleanup, the share of questions where the right passage actually reaches the AI went from 96.4% to 98.8%.

What shipped to production

Two speed optimizations passed their exam, plus the fixes described above. None of them touches answer quality.

The first message no longer waits on the database

To find the right passages, Attlas compares the visitor's question against questions prepared in advance. Each one is stored as an embedding, a long string of numbers that represents its meaning: 1,536 numbers per question.

All told, those records were quite a weight... That's more than the server's fast memory, which has to hold everything else too. The result: when a chatbot hadn't been used for a while, its first search went to the disk to fetch the records. The visitor could wait 2 to 5 seconds before Attlas could even start "working".

The OpenAI embedding models we use pack most of the meaning into the first numbers. I verified that keeping only the first 512 gives exactly what the API would return in its shortened version. The records drop from 223 MB to 30 MB and fit in memory.

• Accuracy: unchanged, even slightly better on our 120 questions.
• First access to a chatbot: 95% of searches now take under 40 ms, versus 205 ms before — 5x faster.
• Worst case measured on the largest chatbots: from 2.6 s to 0.33 s.

The same idea had to apply to the suggested questions that appear under each answer.

Button clicks no longer recompute anything

On an Attlas page, a visitor can type a question or click one the owner has suggested. Those clicks account for half of all messages. Before, every click triggered a call to OpenAI to turn the text into embeddings — about 400 ms each time — even though the button text never changes.

What we dropped, and why that's good news

7 of the 13 ideas failed their exam. Each one seemed obvious on paper.

The first row counts as two tests: first we tried cutting the thinking everywhere, then only on chatbots with no tools. In both cases, quality dropped. The mistakes weren't about style. They were genuine reasoning errors, like offering the 75-person package to 80 guests. Keeping the thinking costs about half a second. It prevents that kind of error, and I'd rather pay that half second.

Every rejection is also less code. An optimization that delivers nothing stays in the product forever — to maintain, to understand, to debug. Not adding it is a win.

And some failures teach you something. Digging into keyword search, I looked closely at the questions the chatbot found nothing for. Almost all of them were about information missing from the client's database. The problem wasn't the search, it was the content. So the right answer isn't to search harder, but to flag those gaps to the chatbot owner. That's the next lead I want to pursue.

What this changes for you

If you have a chatbot on Attlas, here's what has changed:

• A visitor's first message no longer hangs for several seconds, even on a large chatbot
• Clicks on your suggested questions get going faster
• Your corrections finally take precedence over the rest of the database, as promised
• An accuracy of 98.8% revealed by testing.
• Attlas still takes the time to think before answering. It's a deliberate choice, backed by numbers.

There are plenty of impressive AI demos. Far fewer AI products where you can actually say, numbers in hand, whether they answer correctly. The difference between the two is measurement: a baseline before you touch anything, a pass mark written in advance, and the honesty to say no when the results don't follow. Even when the lead is tempting, and even when it saves you half a second.