Your agent needs two memories, not one
The usual advice is to extract the facts and throw the conversation away. Keep both.
You and your partner have somewhere to be at eight, and you are out running errands. You text them: “I’ll let you know if I’m running late.” They text back, “ok, don’t forget, we have to leave by 7:30.”
Seven-thirty comes and goes. You are not home and you never sent a word. They are standing by the door, keys in hand, calling you for the third time, and by now they are furious.
You finally get in the car, already defending yourself. “I told you I might be late.” They stop you. “No. Go back and read what you actually sent.” You scroll up, and there it is in your own words: “I’ll let you know if I’m running late.” You never told them you’d be late. You promised to warn them, and then you didn’t. That was the entire point, and your memory had quietly turned it inside out.
Nobody was lying. Your memory did what memory does. It kept the gist and smoothed off the part that mattered, a promise to warn remembered as a warning already given. What saved the conversation was that the actual message was still there to go back to.
That gap is the whole point for today.
A conversation like that becomes several kinds of memory at once. A few facts you carry. A moment you look back on. A place in the longer story between you. Each of those gets its own piece later in this series. But there is one more memory, the one many people building AI tell you to skip: the actual words.
There are two ways people build agent memory, and they fail in opposite directions. One keeps every message and searches the pile, and never builds a real model of the person. The other builds the model and throws the conversation away, because once the facts are out, the raw log looks like noise, or an ever-growing pile of tokens to keep dragging around.
Last time I argued for the model over the pile. This time, the part almost everyone drops: keep the raw conversation too. Team0, the AI Chief of Staff, keeps it word for word, in its own place, tagged and categorized underneath the model even though it is never the working memory.
Why? Because everything distilled from a conversation is an interpretation, and an interpretation can be wrong. It drops the qualifier that changes everything. It merges two things that were not the same. It remembers a warning you never sent, the same trick your own memory pulled in the car. The model is what the agent understood. The raw record is what was actually said. When the two disagree, you want to be able to go back to what was actually said.
So the model is what the agent thinks with every day, and the raw conversation is the ground truth beneath it. Kept so a wrong reading is recoverable instead of permanent, and so that as the agent gets better at understanding, it can return to the real words and look again.
You already live by this. You don’t fully trust your own summary of an important conversation, which is exactly why you scrolled up to find the message.
A good therapist, for example, does the same thing on purpose. They record a session and listen back to it later, not because they forgot the hour, but to check that what they took away matches what was actually said. The recording is the truth. Their notes are a reading of it, and even a careful reading gets checked against the tape.
An agent should keep both, and treat the tidy version as a reading, not the record.