Development team decentralizedbee discovered a counterintuitive phenomenon while building AI agents for clients (such as code assistants and data analysis tools): the biggest cost driver isn't model inference, but context length. And the main culprit in the context is often the output from tools.

Imagine this scenario: an agent searches through a codebase, and the `grep` command returns 500 file matches. The agent stuffs all 500 results into the context and then asks the model, "Which ones are relevant?" The user pays for tokens for all 500 items, while the model might only select 5 of them. In this case, the model is essentially acting as a JSON filter.

This pattern is everywhere—search results, database queries, API responses. The amount of data returned by tools typically far exceeds what the model actually needs, but for convenience, agents often just dump everything into the prompt.

To address this, they developed a compression layer called **Headroom**. Its core idea is to perform statistical analysis on tool outputs before they reach the model, keeping only the key information.

**Retention strategies include:**

* **Error messages**: Any content containing error keywords will not be discarded.

* **Statistical outliers**: If a value in a numeric field deviates from the mean by more than 2 standard deviations, it will be retained.

* **Query matches**: Uses the BM25 algorithm to score relevance to the user's actual question, keeping highly matching items.

* **High-scoring items**: If the data has relevance or score fields, it keeps the top N ranked items.

* **First and last items**: Keeps a small number of items from the beginning and end to provide context and the latest information.

**Discard strategies mainly include:**

* **Repetitive middle sections**: If there are 500 search results where 480 look basically the same, there's no need to keep all of them.


The real challenge isn't compression itself, but knowing **when not to compress**. For example, when searching for a specific user ID in a database, each row is unique and has no ranking signal, so compressing would result in information loss. Therefore, Headroom first performs a 'compressibility' analysis. If the data is highly unique and lacks importance signals, it skips compression and passes through the raw data.

Based on their actual workloads, the tool can achieve **60% to 90% token reduction**. The compression effect is significant for code searches (containing hundreds of file matches) and log analysis (containing many repetitive entries), while database results with unique rows typically compress little—which is the expected correct behavior.

The latency overhead from compression is minimal, around 1-5 milliseconds. The compression process is fast, and model inference remains the primary performance bottleneck.

**The project is open-source with two usage options:**

1. **Proxy server**: Can point to any OpenAI-compatible client.

2. **Python SDK wrapper**: Suitable for users who need more control.

It can work with OpenAI, Anthropic, Google's models, and local models (like llama.cpp using an OpenAI-compatible server) through LiteLLM.

GitHub repository: [https://github.com/chopratejas/headroom](https://github.com/chopratejas/headroom)

Additionally, this compression is **reversible**. The tool caches the original content (with TTL) and injects retrieval markers into the compressed output. If the model needs the compressed data, it can request its restoration. Although this functionality is rarely needed in practice, it provides a good safety net.

One commenter noted: "This is indeed clever. I've been hitting walls with agent costs lately, just accepting this 'token tax' like a fool. The compressibility analysis is smart—I've seen too many 'optimization' solutions that silently fail when edge cases appear."

Another commenter suggested an alternative approach: "I think the best solution to this problem is to use a cheaper model to handle token-intensive tool usage, like Claude Code does."

The development team noted that most agent frameworks seem to blindly truncate context, which always felt wrong to them. Either information is lost randomly, or users pay for tokens they don't need. There should be a middle ground. They're also looking forward to community feedback.