Context recall and token cost: how K-Lake cuts what an assistant spends per answer

Summary

For the same three questions on the same nine filings, an assistant working through K-Lake put about 21,500 tokens into its context. Careful command-line searching needed 2.4 times that, a standard vector-chunk store 2.1 times, and the naive command-line searching most people actually do 47 times.

Two mechanisms account for it. K-Lake ranks by meaning and by exact term across every file at once, so fewer wrong passages come back. And large results stay on disk rather than in the conversation: of everything K-Lake returned in the session, 99.3 percent never reached the assistant's context. Because most assistants re-send the whole conversation on every turn, the gap widens with each question asked.

All four approaches were run, not estimated. The corpus is small and the vector store used a keyword-based similarity as a stand-in for dense embeddings, which if anything flatters that alternative. Both points are set out under limits.

What we set out to measure

When people compare ways of connecting an assistant to documents, they usually compare answer quality. Cost is rarely measured, yet it decides whether a rollout is affordable at scale. Every token that enters the assistant's context is billed, and with most assistants the whole conversation is re-sent on every turn, so an approach that floods the context early pays for that flood repeatedly.

The question was therefore simple: for the same questions on the same files, how many tokens enter the assistant's context under each retrieval approach, and what does that cost at published model prices?

Corpus, questions and agent

The estate was 40 SEC filings in PDF form, about 80 MB, from five listed companies between 2015 and 2023, all extracted and searchable in K-Lake before the session began. The measured comparison used the nine 3M filings within it: eight annual reports and one quarterly, which together come to about 6.3 million characters of extracted text. These are public documents, so the study can be repeated by anyone.

Three questions were asked, chosen to exercise different kinds of recall:

• A concept-level question. "What are 3M's risk factors?" The answer is a whole section that appears, with changes, in several years of filings.
• An exact-entity question. "Who was the Executive Vice President, Chief Financial and Transformation Officer at 3M?" The answer is one name, findable only by exact terms.
• A section-level question. "What are the risks related to legal and regulatory proceedings?" The answer spans a named sub-section and its supporting notes across filings.

The agent was a GLM-5.2 model driven by an agent harness, connected to K-Lake through the same MCP server any assistant uses. For every approach, the figure recorded is the number of characters that actually entered the assistant's context, converted to tokens at four characters per token.

The four approaches

K-Lake

The assistant called K-Lake's content search in hybrid mode, which ranks by meaning and by exact term together and then re-ranks the candidates. Each call returned a short preview and metadata for every hit, including page numbers and a link to the original file. The full results were written to disk by the agent harness, and the assistant then ran small scripts to read only the passages it needed. Five searches and ten such reads answered all three questions.

Careful command-line searching

This is what a skilled operator does when an assistant is pointed at a folder of extracted text. Each question becomes a search for likely words in each file, followed by a second command that extracts the section between the matching heading and the next one. It is precise, but it is one file at a time, and it returns whole sections rather than the paragraphs that answer the question. Forty-three tool calls answered the three questions.

Naive command-line searching

This is what most people actually do: search for likely words and pull a generous window of lines around every match, in this case twenty to forty lines either side. Every table of contents entry, financial statement and cross-reference that happens to sit near a match comes back too. Forty-two tool calls answered the three questions.

A standard vector-chunk store

This is the common "chunk, embed and retrieve" pattern. The nine filings were cut into 3,514 pieces of about 2,000 characters with a small overlap, and each question answered by fetching the twenty most similar pieces plus neighbouring pieces for the top five. Similarity was scored with a keyword-based measure standing in for dense embeddings. Six tool calls answered the three questions.

Results

Tokens into context are what the assistant had to read to answer. Tool calls matter separately, because with most assistants every call re-sends the growing conversation.

The careful and vector approaches both needed a little over twice K-Lake's context. The naive approach pulled over four million characters of filing text into the conversation, most of it irrelevant, because a window of lines around a match has no idea what it contains.

Where the data went

The largest single effect in the study is not ranking but plumbing. Across the session, K-Lake returned about 13.6 million characters of search results and file text. Almost none of it reached the assistant.

K-Lake returned about 75 times more data than the careful command-line approach, and put less than half as much into the conversation. The other three approaches have no equivalent step: everything their tools return goes straight into context, every time.

Why the gap widens

Most assistants re-send the entire conversation, including every prior tool result, on every turn. So what enters the context on the first question is paid for again on the second, and again on the third. Counting that resend, the three-question session cost about 41,000 input tokens through K-Lake, about 107,000 with careful command-line searching, and about 99,000 with the vector store. By the third question the command-line and vector approaches were each re-sending roughly 130,000 characters of earlier results that K-Lake had never held in context at all.

The effect compounds with every further question in the session. A ten-question session widens the gap further than a three-question one, which is why the cost table below grows faster for the alternatives than for K-Lake.

What it means in money

Applying published per-million-token list prices at the time of the study, with about 5,000 output tokens per response in every case, gives the cost of the three-question session under each approach. The three price bands stand in for a mainstream hosted model, a frontier hosted model and a low-cost open-weight model.

The per-session numbers are small. The ratio is what matters, and it holds whatever you multiply it by. On the mainstream price band, counting conversation resend:

At ten thousand ten-question sessions a month, K-Lake saves between $4,800 and $5,500 a month against the two careful alternatives, before counting the naive approach at all.

Limits of this study

We would rather you trusted a modest, measured number than a large, vague one. The limits are these.

• The corpus is small. Nine filings and three questions. The mechanisms behind the ratios get stronger as an estate grows, since there is more to rank and more to keep out of context, but we have not yet measured that.
• The vector store was flattered. Its similarity scoring was keyword-based, which returns almost no irrelevant pieces on these questions. Dense embeddings, which is what a real deployment would use, typically return 30 to 60 percent near-miss pieces on questions like these, and cannot answer a question whose words do not appear in the text. The 2.1x figure is therefore a floor.
• Keeping results on disk depends on the assistant's harness. The 99.3 percent figure relies on the harness writing large tool results to disk and passing the assistant a preview, which the one used here does. An assistant that pushes every tool result straight into context would see a smaller benefit from this mechanism, though it would still gain from K-Lake's ranking, previews and bounded file-window reads.
• One agent, one session. A different model or harness would make different choices about how many calls to issue and how much to read.
• Prices move. The cost tables use list prices at the time of the study. The token ratios do not depend on them.
• Command-line tools still win for pure pattern matching. Counting how many files mention a term, or listing them, is instant on extracted text. K-Lake's advantage is in recall, context and synthesis for question answering, not in bare matching.

The site states the result as "up to 47x fewer tokens per answer". That is the measured ratio against naive command-line searching, which is what an unassisted rollout most often looks like. Against the two carefully tuned alternatives the measured ratio is 2.1x to 2.4x, and we state that alongside it wherever the larger figure appears.

Reproduce it on your own files

The evaluation installer stands up K-Lake on a single VM with a 30-day licence. Point it at a representative share, connect the assistant you already use, ask your own questions, and compare the token counts your assistant reports against the same questions answered without K-Lake. We are happy to help design the comparison and will publish updated figures as we run the study at larger scale.