AI agent memory: we handed the sorting to Jev

Why our AI agents' memory added recollections to every message

Since the setup described in the article on Cognee, an agent opening one of our projects can retrieve the decisions made and their reasons. We can therefore hand it tasks that previously required a long reminder of the context. Claude Code and Codex query the same server, which holds our ways of working and the history of our projects. After a month of use, we wanted to keep this shared memory, but what Cognee's official plugin added to conversations needed a second look.

The problem lay in how this plugin queried Cognee. In the version we used, it sent the session's exchanges to the server, then ran a search on every message to add the passages closest to the request to the context. It checked neither their usefulness for the work in progress nor their presence in the conversation. The agent could therefore receive a reminder it had read a few messages earlier. This behaviour may suit some uses, such as those of autonomous agentic workflows, but it did not fit our internal practice, which is closer to conventional development.

Measured over several hundred messages, this added about 5,700 characters to every prompt, after six seconds of waiting. Even a reaction to the agent's answer triggered the processing meant for a question about a past choice, while most of the passages already sat in what the Claude Code or Codex harness was sending to the model.

The cost of these useless reminders extended beyond the search itself. As long as the tool kept a passage in the context, the agent had to process it on the following turns, after the seconds of waiting for the initial call. The server and the memory calls cost a few cents a day, to which was added the cost of that text in the model's input, depending on its pricing and its cache. With a large model, or several agents receiving the same passages, we could end up paying more for that surplus than for the search itself. These costs may look negligible, but they did not seem scalable to us at the level of a mid-sized organisation.

To that cost was added a concern about the quality of the answers. In the Context Rot study, researchers at Chroma evaluated eighteen models and observed performance dropping as the context grew longer. Their experiments on distractors also showed that a passage close to the subject, but not answering the question, was enough to degrade the results. By sorting reminders before passing them on, we wanted to avoid leaving that work to the agent while it was supposed to be looking after the project.

What an AI agent memory should keep

This sorting starts when the memory is written. Over the sessions, exchanges with an agent produce thousands of lines, a part of which serves to pick a project up again, for instance the choice of a payment provider and its reason, or a language version imposed by the host. That information fits in a few dozen lines. Before having Jev select them, we had to define what deserved to enter the memory.

What deserves to be kept

Our first criterion was to keep what the code did not say. We dropped it during the tests, because many choices can be read in the repository, including the payment provider chosen or the absence of a framework. The agent still has to know where to look, without redoing a search carried out in a previous session. We therefore settled on a criterion tied to that time spent searching or reasoning, keeping a piece of information when it will save repeating the same work if the subject comes back.

With this criterion, we keep a technical choice and its reason, a constraint likely to cause trouble in the next session, or an incident together with its fix. Reactions and one-off tasks stay in the conversation. UI details belong in the design system, while habits shared across projects go in the CLAUDE.md. Those habits do not need to go through the memory, since the agent loads the instruction files in full.

This selection means accepting a few losses, among them the reason for a choice recorded in the repository. Requiring it to be kept every time would force us to justify our decisions systematically, which is not always necessary. We keep the explanations present in the exchanges, without imposing that extra work, and the agent can consult the repository to check the implementation.

Which messages call for a reminder

Once that information is kept, it remains to choose when to bring it back. Most messages sent to an agent do not call for it, since a thank-you, a commit request or an instruction that carries everything it needs are handled with the ongoing conversation. We split messages into eleven kinds to tell these cases apart. Five trigger no search, while the other six lead to checking whether a note or a passage from the memory answers the request.

the message
     │
     ▼
Jev, what kind is it?
     │
     ├─ a reaction           ┐
     ├─ an action on the     │
     │  finished work        │
     ├─ a complete           ├► nothing
     │  instruction          │
     ├─ general knowledge    │
     ├─ a question about     │
     │  the conversation     ┘
     │
     ├─ a new decision       ┐
     ├─ a past choice        │
     ├─ resuming work        │
     ├─ a recurring problem  ├► notes,
     ├─ a task that builds   │  then
     │  on what exists       │  Cognee
     └─ a reference to an    │
        earlier exchange     ┘

Where to look first

For these six kinds, the plugin looks first in the local notes, written on the machine over the sessions and reachable in a few milliseconds. If it finds an answer there, it passes it to the agent and stops the search. Otherwise it consults Cognee, provided the subject matches the information worth memorising. After a final sorting, the agent receives the passages kept, with pointers to past choices and to where to look. It must still check critical information in the code before acting, since the project may have changed since a note was written.

Jev, TypeSafe's decision model, tested on memory sorting

TypeSafe presented Jev on 15 September with several demonstrations, including an agent playing Doom while making decisions in a fraction of a second. Sorting the memory lent itself to a trial on our own work, with precise decisions to submit and a call frequency that made its cost and speed matter. It also gave us a way to check the results, by rereading the exchanges to see whether we would have kept the same information or triggered the same searches.

Jev receives a context and questions with possible answers. From these, it returns for each question a choice, a probability or a score, without writing any text. With a listed price of 0.042 dollars per million input tokens for Jev 1.13 and free output, a decision over a few thousand tokens costs a fraction of a cent. We could therefore submit every message to the sorting without the cost of those calls becoming an obstacle.

In the first questions, we asked Jev whether the agent lacked a piece of information the memory could supply. In one of our trials, the user mentioned a database set up several weeks earlier, while the recent conversation explained how that type of database works in general. Jev stopped the search, although the agent needed the details of our setup. Several rewordings were not enough to fix the error, because it kept mistaking a conversation that talks about the subject for a conversation that contains the answer.

We then changed the task, asking Jev to classify the message by its kind, then the subject and the exchange according to the categories useful at each step. Its answers improved with these questions about the text it received. In the database case, it only had to recognise a request tied to a past setup to trigger the search. The decision no longer depended on a deduction about what another agent knew or did not know.

The content given to Jev also mattered when assessing recollections. With the five passages returned by Cognee in one context, it gave the useful passages scores between 0.40 and 0.67, below our threshold. By presenting the passages one at a time, each against the message, we obtained scores between 0.88 and 0.92 for the good ones, while the bad ones did not exceed 0.63 in that trial. This result is in line with TypeSafe's documentation, which states that Jev's accuracy varies with the size of the context and drops beyond its default limits. We therefore kept the separate assessment and the single-item questions, with the option of reusing this approach for other sorting tasks, provided the results are checked on their own examples.

memsift, an open source memory plugin for Claude Code and Codex

To use this sorting in our sessions, we built memsift from the plugin published by Cognee's authors. We kept its history and its Apache-2.0 licence, while rewriting most of how it works. This version runs with Claude Code and Codex, on an existing Cognee server. We publish it with no affiliation with Cognee.

npx skills add nualt/memsift

How memsift decides what to give the agent

the message
 │
 ▼
Jev, 2 s at most
what kind of message?
 ├─ nothing to look up
 │  (most of them) ────► nothing
 ▼
Jev, each note judged alone
does it answer the message?
 ├─ yes ──────────► the note
 ▼
Jev
a subject worth keeping?
(choice, constraint, problem)
 ├─ no ───────────────► nothing
 ▼
Cognee, 12 s at most
its closest passages
 ▼
Jev, 2 s at most, one by one
do they answer the message?
 └─ yes ─────► a few lines
               (often none)

The plugin stops the path as soon as a step answers the message or rules out the search. Each sorting call to Jev costs a fraction of a cent and, for most messages, the path ends with nothing added from memory. At the end of the exchange, the plugin calls Jev to choose what deserves to be kept, by the same criterion of work not to be done twice. It appends the exchanges kept to the project's local notes, which a language model then revises in the background, before sending them to a dataset reserved for memsift in Cognee.

That revision calls for a precaution, since the model could alter passages it has no reason to touch while rewriting the whole file. To limit that risk, we ask it to name the lines to replace or remove. The code applies those changes and keeps the other lines word for word.

What it changes in use

To compare the text added, the waiting time and the reminders found, we ran an eighty-message conversation through the original plugin and then through memsift, with a real Cognee server in both cases. Another model had prepared the messages and the expected answers before the test, without access to memsift's code. That choice was meant to avoid picking examples to suit our implementation, while putting both plugins through the same conversation.

80 messages, same conversation
                 original   memsift
added            80 / 80    13 / 80
characters      204,297     11,352
wait              3.0 s      0.8 s
added for nothing 64 / 64    6 / 64
reminders found   9 / 16     7 / 16

The original plugin adds a passage to all eighty messages, including the sixty-four that called for no reminder. With memsift, thirteen messages out of eighty receive an addition from memory, six of them with no need for one, which brings the total volume down from 204,297 to 11,352 characters. The results on useful reminders are lower, however. Of the sixteen expected, memsift finds seven, against nine with the original plugin.

The reduction in useless text and waiting therefore comes with two more useful reminders missed. Those missed reminders come from the search towards the server, which memsift called nine times during this test without obtaining a useful passage, and that is the step that use will allow us to adjust.

In use, memsift installs with no dependency and pauses the memory when a service goes down. The agent then reports the interruption, so work can go on in the knowledge that reminders are unavailable. Before sending anything to the services, the plugin redacts secrets. The installation and operating details are in the documentation on Nualt and the code on GitHub.

At Nualt, a tool gets completed when it falls short

A tool that does not meet our need does not stop us, nor do we replace it at the first flaw. Cognee's plugin fed and queried the memory without sorting, so we spelled out what we expected from the memory, revised the questions put to the model and rewrote the plugin until it worked well enough on our projects. We then published it with its documentation, as we had done for the Medusa authentication module when Medusa lacked one. Cognee stays on our server, and memsift has been running on our projects since.

We apply the same rule to our clients' projects. When a part is missing or a tool misbehaves, we build what is needed rather than impose a workaround, and the client remains the owner, from the code to the credentials, and down to their memory when they use one. With memsift, they keep their notes and their Cognee server, while sorting and revision go through external services that we document, so they can choose with full knowledge whom they entrust their data to.