A code map made our AI agents use 25% more tokens. Test any token saver before you trust it.

What happened to us

Our team of AI agents works in code all day. We pay for that work in tokens: the chunks of text an AI model reads and writes.

An AI agent works in steps: search, open a file, read, answer. On every step, it reads the whole conversation so far again. So any text that goes in early gets paid for again and again.

Graphify is a tool that turns a folder of code into a map. An AI agent asks the map where things are, instead of searching the files. We wanted to know if that would save us tokens.

Quinn, the AI agent that runs our systems work, set up a fair test:

• 12 questions about one of our live apps, about 2,700 files. 4 easy, 4 medium, 4 hard.
• The right answer to each was written down before any run.
• Each question ran 4 times from a fresh start: twice with Graphify and twice without. That's 48 runs.
• Same AI model every time, Claude Sonnet 5.5. Same frozen copy of the code.
• A separate AI judge graded every answer against the key, without knowing which side wrote it.
• We set the bar first: keep Graphify only if it used at least 15% fewer tokens.

What came back:

• Both sides got all 24 of their answers right.
• With Graphify, the typical question used about 242,000 tokens. Without it, about 194,000. That's 25% more.
• With Graphify, answers took about 20 seconds on average. Without it, about 14.
• Graphify used fewer tokens on only 2 of the 12 questions. Both were hard ones that trace a path through many files. The best one used 25% fewer tokens and took 6 steps instead of 11.
• Easy questions cost the most extra: about a third more tokens on average. One easy question used 77% more.

Why the map cost more

Every one of the 24 Graphify runs asked the map first. Then every one still opened the files. After the map, they took at least 2 more steps, and 4 for the typical run.

The number of steps came out about even. Graphify cut steps on 6 questions and added steps on the other 6.

Each step got heavier. The map's answer stays in the conversation, so the AI agent reads it again on every step after. A typical step carried about 45,000 tokens with the map and about 36,000 without.

Another hard question shows this. The map cut the steps from 11 to 7 or 8. Those runs still used 10% more tokens, because each step carried more text.

Why our plain setup won

Our AI agents search code with plain tools. They find a file by name, search for a word, then open just the file they need. Each search brings back a few lines. A map's answer is bigger, and it arrives before the AI agent knows what it needs.

An AI agent that can open the file itself will check the file. Checking is the right call, because the code is the truth and a map is only a summary of the code. So the map's answer became extra text on top of the search.

Keeping loaded text small is how a team of AI agents stays cheap. Anything an AI agent loads at the start, it reads again on every step.

What we changed

We did not add Graphify. Our AI agents keep searching the code directly.

Any tool that says it saves tokens now gets this same test before we add it.

This was one codebase, 12 questions and one AI model. A much bigger codebase could come out differently. Run the same test on your own code before you trust our result.

The lesson

Text an AI agent loads gets paid for again on every step. A tool that adds text only saves tokens if it cuts more searching than it adds. If the AI agent checks the source anyway, a summary on top is extra cost.

Test a tool on your own code

1. Write 10 to 12 questions about your own code, with the right answers. Pick your keep line too, such as "15% fewer tokens, same right answers." Do both before you test anything.
2. Run each one at least twice with the tool and twice without. Same model, same copy of the code, a fresh start every time.
3. Have a separate AI grade the answers against your key, without telling it which side wrote them.
4. Count every token the AI reads and writes, including text it reads again. Your AI provider's usage logs show these counts.
5. Read what the AI agent did after it used the tool. If it still opened the files, the tool is extra text.
6. Trim what every AI agent loads. Cut any rule nobody can explain. We cut the instructions our own AI agents load by about a fifth. Each word you cut saves on every step.