I like the overall idea of deliberate forgetting. And I found checkpointing into artifacts (code, design, intent) to start with fresh context to work well for that purpose.
Interestingly, the artifacts include not just code but alao retroactive design and retroactive “execution plan”. I feed them to another agent to validate the approach and resume work with fresh context and update mission statement.
Unexpectedly, design doc and an execution plan are now outcomes of coding process, not inputs to it. I’m not sure how to think about this.
> When building intelligent applications, harnesses, and workflows, our job is to choose when to be deterministic (what to put in code) and when to be probabilistic (when to let the model drive).
I really like this final sentence from the presentation. When I read this phrase, I realized that it captured one of the core dynamics that I've been using to figure out when and how to apply AI to a task. I often find myself trying to figure out if it is better to A) ask the AI to do a task for me or B) ask the AI to build something that can do the task for me. A major component of that answer comes down to how deterministic or probabilistic the task is.
Until a few months ago I did it this way, but OpenAI and Anthropic built their own harnesses, and since those end up conflicting with the harness I built myself, having no setup at all actually ended up performing better. As a result, it doesn't seem necessary to set up rules or memory separately, and if I do, using skills or agents made of only logging and flow (without things like rules or guardrails) seems like the best option going forward.
> The tea task for FireRed was in its weights, so it poisoned its context.
Mmm, to my mind, isn't this mixing completely different things? I thought poisoned context is stuff in the context that makes the model behave poorly. But if it's in the weights, that's just... model issues.
> For me, around 200,000 or 300,000 is where I start looking. I’ll check, see it’s at 250,000, and think: I’ve got a big job coming up. I’m going to reset and start fresh
I don't find this to be true, at least with the newer models. I also find that the models fill up to 250k very quickly (presumably mostly by looking through the codebase). Without that context I find the models perform badly. I also don't really notice that large of a change until around 700-800k context and even then it isn't as bad as it used to be. I suspect, that (especially for coding), the long horizon RLVR tasks have helped with this quite a bit.
I thought the price reduction was because both companies implemented Deepseek's published research which demonstrate effective approaches to lowering the cost of inference while maintaining quality. Thanks to Deepseek, they did lower their costs and they passed that in to their customers.
Anecdotally (sample size == 1), I have observed that as my context grows longer, the model seems to get dumber. I'm not an advanced user with harnesses, etc.; I just use the chat interface, as that's all I can afford, but after a few turns the model (I'm talking frontier models here from OAI and Ant) seems to just do stupider things.
DeepSeek v4.1 flash seems to do this thing where it starts saying that we should wrap up the current work and start again "next session" (whatever that means), that the "context budget" is almost up, or some other wording I'm forgetting, as the context gets longer.
I've taken to just manually making it write a handoff file and clearing context completely. I'm pretty much manually compacting instead of my harness doing it. Feels weird, but it does work.
I try to manually manage as much of the context as possible. Most of the time, I limit myself to a context window of 200k, and it works great for my use cases.
So like- I considered this idea of using a fast decision model to prune before compaction, but I scrapped it because A; im not bottlenecked by speed, and B; cache invalidation dark magic
From my understanding, if we have turns [A, B, C, D, E] and we decide that turn C is useless context, then we lose our cache for the [D, E] segment. (plz correct me if im wrong, I havent tested this)
Personally, I pay no mind to session state nowadays. I exclusively use provider native hosted compaction, which if you believe the providers, is the way to go. For context management- I spend most of my effort on the things which dump output back to the model.
If anyone has any good reads on benchmarking OpenAI and Anthropic hosted/native compaction against custom stuff, I would love to read!
Yeah, I also think decision models for pruning context isn't worthwhile. You're paying to load it into context (of your decision model), in order to save on loading the same data into context (of your generation model). The only way the economics makes sense is if you need a much smarter model for the compaction, but can't pay for it to load all of that context (because it's uncached or something). That sounds like a niche but it really doesn't exist. Cheap models are good enough, and smart models are cheap enough.
Personally, I use a harness extension that auto-compacts my context just before the cache expires. Combined with the default compact-when-full, this ends up being pretty much always what I want. It's one less thing to think about.
Yes, that's my understanding- though I realize that OpenAI has made changes to what you can do with caching over the API recently, so you may be able to do some fancy stuff to get around this.
I use dynamic context compaction in open code plugin. Then modified it further to include eviction.
Since I run everything local, speed benefits from keeping context low ~128k. This gets me through a single agent on multiple tasks through the same repo. It seems to break around 6M but that's more probablistic.
KV cache is definitely valuable but context poisoning happens regardless of size and it's probability increases regardless of intelligence. So at some point you're going to suffer from context overload regardless.
>This is the advice I keep giving now. Someone says, “My coding agent’s doing something weird. Claude’s been nerfed.” And I ask, “How big is your context?”
That’s really true, but on the other hand, if there’s too little information, it also ends up working in strange ways. Most AGENT code anti-patterns work well at a medium amount of context, but when there’s too much or too little information, they don’t behave properly.
Interestingly, the artifacts include not just code but alao retroactive design and retroactive “execution plan”. I feed them to another agent to validate the approach and resume work with fresh context and update mission statement.
Unexpectedly, design doc and an execution plan are now outcomes of coding process, not inputs to it. I’m not sure how to think about this.
I really like this final sentence from the presentation. When I read this phrase, I realized that it captured one of the core dynamics that I've been using to figure out when and how to apply AI to a task. I often find myself trying to figure out if it is better to A) ask the AI to do a task for me or B) ask the AI to build something that can do the task for me. A major component of that answer comes down to how deterministic or probabilistic the task is.
Mmm, to my mind, isn't this mixing completely different things? I thought poisoned context is stuff in the context that makes the model behave poorly. But if it's in the weights, that's just... model issues.
I don't find this to be true, at least with the newer models. I also find that the models fill up to 250k very quickly (presumably mostly by looking through the codebase). Without that context I find the models perform badly. I also don't really notice that large of a change until around 700-800k context and even then it isn't as bad as it used to be. I suspect, that (especially for coding), the long horizon RLVR tasks have helped with this quite a bit.
I've taken to just manually making it write a handoff file and clearing context completely. I'm pretty much manually compacting instead of my harness doing it. Feels weird, but it does work.
I have auto-compact on at 400k
I try to manually manage as much of the context as possible. Most of the time, I limit myself to a context window of 200k, and it works great for my use cases.
From my understanding, if we have turns [A, B, C, D, E] and we decide that turn C is useless context, then we lose our cache for the [D, E] segment. (plz correct me if im wrong, I havent tested this)
Personally, I pay no mind to session state nowadays. I exclusively use provider native hosted compaction, which if you believe the providers, is the way to go. For context management- I spend most of my effort on the things which dump output back to the model.
If anyone has any good reads on benchmarking OpenAI and Anthropic hosted/native compaction against custom stuff, I would love to read!
Personally, I use a harness extension that auto-compacts my context just before the cache expires. Combined with the default compact-when-full, this ends up being pretty much always what I want. It's one less thing to think about.
Since I run everything local, speed benefits from keeping context low ~128k. This gets me through a single agent on multiple tasks through the same repo. It seems to break around 6M but that's more probablistic.
KV cache is definitely valuable but context poisoning happens regardless of size and it's probability increases regardless of intelligence. So at some point you're going to suffer from context overload regardless.
That’s really true, but on the other hand, if there’s too little information, it also ends up working in strange ways. Most AGENT code anti-patterns work well at a medium amount of context, but when there’s too much or too little information, they don’t behave properly.