I’ve also found Claude to work quite well. Some hallucinations aside, it does a good job assimilating project knowledge I’ve thrown at it. Though, it seems to suffer some random inability to handle something it was able to handle last week with no problem at all. Opus seems to suffer this hindrance most often.
As I've been using Claude a lot today, I've been thinking about their token windows, and I have a better idea of how I think they are implementing it, based on how my sessions tend to go.
Starting out, you have you full token space for storage, and a "working memory" credit, lets say they both start at 100 and go to 0, and they are measuring different things. These are temporary definitions just to study how it works as a black box.
The token space is their unique storage for processing this session, which increases with your input and it's own processing output, and final output.
The "working memory" is how many tokens they allow you to use each question, maximum. They will save on processing if they can, but they have a max limit where it has to stop and use the last-best answer.
Your first query, you have 100 storage, and 100 working mem.
Your second query, say you have 95 storage, and 100 working mem.
At some point, maybe 50% storage, they start to reduce the the amount of working mem, because each new question will cost more storage, which increases the speed at which the session will end.
When the session ends, many people have lost all their work, because they werent done, and they didnt properly, get a summary-detail report back out to feed to the next session, so they will be frustrated.
My guess is they user test this a lot, and they have tuned it to try to extend sessions, because you also usually cant get what you want immediately. So it wont be useful if it's too short.
Mine hit the limit constantly, and my work often has like 5-30 sessions in a row to get it done, for small detailed precision single task assessment type work, so having to keep renewing the session is a regular activity.
One of the annoying things about the usage though is that if you are running a session and Claude gets interrupted because of what "may be a network issue" or some other such error on their end–I know it is not my network–whatever tokens Claude has used up at that point are eaten and just gone for that session.
So, you run into problems not based on your actual usage, but by server-side error on Claude's end that makes you hit your limit even though you never actually did anything and just kept hitting a wall. It would be nice if Anthropic could somehow correct that, but I don't know how feasible that actually would be on a per user basis.
I’m not that fancy. Claude was the first AI sub I tried out after having run deepseek and ollama locally, so I went with Pro and threw some money into the API to use internally with another app. I find if I’m using Opus, I can throw four or five heavy processing jobs at it before I hit the limit and have to wait for it to reset.
It definitely goes through retard periods. Anthropic says they dont limit tokens, and instead just slow it down, but I dont think thats correct. I think they limit tokens during shortages turning it extra dumb for a while, which would be the normal resource allocation method when most of your users think its magic anyway.
Even at it's best, its like an 85 IQ savant saboteur. It can spit out something useful, and then proceed to make every bad suggestion possible from then on.
I use Claude for coding, and it's performance is usually very good, sometimes having a 10-20s lag, but usually sub-1s.
It's total accepted file sizes is a huge problem in coding. Most of my files are beneath 30k, but generated ones are too big for it to look at, and even adding 20 files uses a lot of it's total token so there are only a few questions capable each session.
I assume they keep reseting these in the non-chat versions, but since AI is the worst programmer out of all programmers alive, it will never touch my code again.
Overall, Claud is the best I found. I used the https://t3.chat/ wrapper to test all the different models, and still use it when I want to see what a different model things, but Claude seems to respond the best, even though it's awful and makes me hate using it more every time I do.
Worst phrase in the AI universe: "You're absolutely right!"
I’ve also found Claude to work quite well. Some hallucinations aside, it does a good job assimilating project knowledge I’ve thrown at it. Though, it seems to suffer some random inability to handle something it was able to handle last week with no problem at all. Opus seems to suffer this hindrance most often.
As I've been using Claude a lot today, I've been thinking about their token windows, and I have a better idea of how I think they are implementing it, based on how my sessions tend to go.
Starting out, you have you full token space for storage, and a "working memory" credit, lets say they both start at 100 and go to 0, and they are measuring different things. These are temporary definitions just to study how it works as a black box.
The token space is their unique storage for processing this session, which increases with your input and it's own processing output, and final output.
The "working memory" is how many tokens they allow you to use each question, maximum. They will save on processing if they can, but they have a max limit where it has to stop and use the last-best answer.
Your first query, you have 100 storage, and 100 working mem.
Your second query, say you have 95 storage, and 100 working mem.
At some point, maybe 50% storage, they start to reduce the the amount of working mem, because each new question will cost more storage, which increases the speed at which the session will end.
When the session ends, many people have lost all their work, because they werent done, and they didnt properly, get a summary-detail report back out to feed to the next session, so they will be frustrated.
My guess is they user test this a lot, and they have tuned it to try to extend sessions, because you also usually cant get what you want immediately. So it wont be useful if it's too short.
Mine hit the limit constantly, and my work often has like 5-30 sessions in a row to get it done, for small detailed precision single task assessment type work, so having to keep renewing the session is a regular activity.
One of the annoying things about the usage though is that if you are running a session and Claude gets interrupted because of what "may be a network issue" or some other such error on their end–I know it is not my network–whatever tokens Claude has used up at that point are eaten and just gone for that session.
So, you run into problems not based on your actual usage, but by server-side error on Claude's end that makes you hit your limit even though you never actually did anything and just kept hitting a wall. It would be nice if Anthropic could somehow correct that, but I don't know how feasible that actually would be on a per user basis.
I get those sometime and can keep working after it comes back, im sure it costs session tokens bit hasn't ended my session yet i think.
With the way software works these days, we cant be sure were running similar patterns. They may have one of us in a feature flag pool.
Im on the 5x max plan.
I’m not that fancy. Claude was the first AI sub I tried out after having run deepseek and ollama locally, so I went with Pro and threw some money into the API to use internally with another app. I find if I’m using Opus, I can throw four or five heavy processing jobs at it before I hit the limit and have to wait for it to reset.
It definitely goes through retard periods. Anthropic says they dont limit tokens, and instead just slow it down, but I dont think thats correct. I think they limit tokens during shortages turning it extra dumb for a while, which would be the normal resource allocation method when most of your users think its magic anyway.
Even at it's best, its like an 85 IQ savant saboteur. It can spit out something useful, and then proceed to make every bad suggestion possible from then on.
Have used Grok and ChatGPT to reformat long documents but they always fail. No problems with Claude.
I use Claude for coding, and it's performance is usually very good, sometimes having a 10-20s lag, but usually sub-1s.
It's total accepted file sizes is a huge problem in coding. Most of my files are beneath 30k, but generated ones are too big for it to look at, and even adding 20 files uses a lot of it's total token so there are only a few questions capable each session.
I assume they keep reseting these in the non-chat versions, but since AI is the worst programmer out of all programmers alive, it will never touch my code again.
Overall, Claud is the best I found. I used the https://t3.chat/ wrapper to test all the different models, and still use it when I want to see what a different model things, but Claude seems to respond the best, even though it's awful and makes me hate using it more every time I do.
Worst phrase in the AI universe: "You're absolutely right!"