Infinite Context

Last week, my friend Porter told me about Victor Taelin's OptMem system. The system is elegantly simple:

1. Append each memory to a file
2. Compress memories together recursively

In his system, you have a fixed prefix context, and most of it doesn't change. The oldest parts of the context are heavily compressed, the newest parts are as they are. When you need a fact from the older parts, you ask to zoom into that piece and it's expanded out into context. The reason this is good for a model is that most of it is constant and cached, while allowing the entire system to still attend to facts from the far past. The model is not forgetful out of context.

Our Family Assistant

One of our shared chats with the home assistant

Julie and I have a family assistant. We talk with it about mundane things, about trip planning, to help us plan our grocery list, or to buy things, or to ask when we did something. It's got its own email inbox that we have auto-forwarding for some emails, and it is hooked up to our free/busy calendars, and so on. The assistant is also plugged into our home systems, our knowledge base on things, and knows when our daughter has left her bedroom and so on. We have a companion app that sits on our phone and notifies us about relevant tasks when we're going places and so on.

For a family assistant to be useful, it has to do things. The most useful thing in our experience is to be able to surface relevant tasks and act as a search agent across our lives.

Context Building

The home assistant builds a base-level of things and then fills the context with recent messages attached to an anchor point. At the point where the latest messages will no longer fit into the 80k context with the prior anchor, a new anchor is chosen that will fit the message. So there's a point where a prefill kicks in, but cache hit rates are 90% plus across all jobs simultaneously running (including coding omp sessions and background jobs), and chat messages are 90-95%.

The Knowledge Base

For a long time, the KB has been the tool that we use for storing information about our family and activities. Most agents are pretty good about recording information in the KB when provided with the appropriate skill. The hard part, as always, is retrieval. I wrote about this a little in its earlier implementation and that has mostly held up with the slight variation that I now use vec1 instead of the default vector implementation. It requires some parameterization but overall works quite well. Combined with fulltext search, agents are able to retrieve what they want to talk about.

The problem with this entire thing is obvious: an agent doesn't actually know that anything has happened until it inspects the KB, and it doesn't know what it has to search the KB for. Agents exist in a world where they have their baked in intelligence and knowledge and the context and nothing else. So what is in the context matters a lot for the actions that they will take.

In our use-case what was happening was that the model would simply not know that something else had been done or added or so on, so it would talk in general terms when I said something about plans that were already made and so on. With some prompt engineering, I was able to make it look up before saying things, but because it didn't really know what to search for it would often use keywords that don't land it in the right place.

The Summarization Trick

One part of OptMem's insight is that logarithmically folding gives one a fixed context size with great replayability. That means it's very context-efficient even though it could theoretically be rebuilding it all the time. The theory of this is that the single context has everything you could possibly need.

The other part of OptMem's insight is that having these things in context tells the model the universe of things it can know about, i.e. what is possible to search for.

Differences

Each task-stream is called a 'run' in our system. You can steer a run at any time.

However, Taelin is using a top-grade Claude while I'm using a much smaller DeepSeek model. In his case, intermediate results are not that useful, so he has the agent choose when to save a memo and when not to. In the home assistant case, any conversation I have is attention-worthy.

His agent also uses a single storage engine because it is used as an interactive terminal agent that builds things. My agent is also given many other skills to store information that make sense from their own data-modeling standpoint, though perhaps a common (lack of) structure is possible that is ETL'd live into the various views: our personal EMR, the KB that I have, our rolodex, and so on.

Another thing is that temporality is something inherently important to the home assistant use-case. Things from the distant past are less important than things from the recent past.

A third thing is that we want to run local things pretty fast but we also want to expose the agent to other chats and results that are going on. We don't use explicit memo and both Julie and I are talking to the agent at all sorts of times so this matters.

One exception is automated runs. While it would seem useful to put their results into the context as well, I found that it was effectively fairly noisy and the significance of a user interaction was substantially more than that of a background run.

Final Implementation

Lots of things are fixed, but newly generated addons go into the history. Total new prefill is 400 ms worth

So finally, what I did was keep the existing context building logic mostly the same and added a history digest block that informs the agent about what happened. The agent has the ability to zoom into those blocks like in OptMem, but in practice it just uses that as input to scan the message history over a time span for information. That works for me because ultimately I just need recognition and a long-running chat.

• Every user run becomes a memo
• Days summarize, rolled to weeks, rolled to months - all written once
• Today's runs are shortened and placed into running context past the cache break
• The agent is given a zoom and a search tool. It usually uses the latter
• Skip automation runs. They persist manually into the KB etc.
• Reuse is ~94%

Next Steps

What I'm going to experiment on next is adding in automated runs somehow. Their information is not useful as-is, but perhaps all auto-runs should be given the memo tool so they can add concrete fixed memories in. Without that, auto-runs that didn't process anything because of other things would leak "overdue" and "pending" into the fold which I had trouble engineering out without damaging performance - and a pre-release judge with Rune sounded hard.