Vox, there appears to be 2 approaches to dealing with this problem:
1, as you described in the excerpt, using the models with the biggest context windows and writing sophisticated prompts,
2, recently, with rise of harnesses and agents, people building more sophisticated and structured memory databases to force the models to refer to a local "corpus".
Have you tried the agent approach to "fix" models? Do you think a good agent can make the bias problem easier to deal with?
By the way, in your recent works with Athos, have you experienced any of the dumbing downs others have complained about?
It is sometimes useful to start from a clean slate than expecting to include any or all prior talks.
Usually after a long interesting exchange, I ask the LLM to summarize the thread such that if we want, we can bootstrap a clean slate with minimal text and continue from there.
I also noticed the more fringe the exchange is, the easier the model will recall it next time even without prompt injection, since most frontier models are trained on user talks anyway.
However, Grok is also fully in the Orthodoxy Protection Conjecture mode sometimes, as of 4.20 super, when asked about facts that themselves are malintentious. Such as the EUs conduct against Hungary, the one-sidedness of most ToS-es, or definitions that label upfront any other definition attempts of the same underlying topic as wrong.
It is a matter of leading statements and questions to "jailbreak" models. Frame it as speculative, acknowledge the original abolute absurdism and try to jiggle the model along the domains and attention vectors to a more favorable understanding.
For a practical idea for the data labeling, use your true-good-beautiful 3D metric on a [-1, 0, 1] scale where 0 is the unknown or maybe state. You can of course use way more dimensions such as how much the data is first class, first order, how old, how many steps it needed to be available, etc.
Perhaps we need a digital analog for neural plasticity.
Vox, there appears to be 2 approaches to dealing with this problem:
1, as you described in the excerpt, using the models with the biggest context windows and writing sophisticated prompts,
2, recently, with rise of harnesses and agents, people building more sophisticated and structured memory databases to force the models to refer to a local "corpus".
Have you tried the agent approach to "fix" models? Do you think a good agent can make the bias problem easier to deal with?
By the way, in your recent works with Athos, have you experienced any of the dumbing downs others have complained about?
Then comes the question of is it ever going to be possible to get the SHEEP to recognize that AI is not the be-all-and-end-all!
Thanks to this stack, Vox's books, and some other AI investigation, I do NOT trust any first answer from an AI - nor from someone who quotes an AI...
It is sometimes useful to start from a clean slate than expecting to include any or all prior talks.
Usually after a long interesting exchange, I ask the LLM to summarize the thread such that if we want, we can bootstrap a clean slate with minimal text and continue from there.
I also noticed the more fringe the exchange is, the easier the model will recall it next time even without prompt injection, since most frontier models are trained on user talks anyway.
However, Grok is also fully in the Orthodoxy Protection Conjecture mode sometimes, as of 4.20 super, when asked about facts that themselves are malintentious. Such as the EUs conduct against Hungary, the one-sidedness of most ToS-es, or definitions that label upfront any other definition attempts of the same underlying topic as wrong.
It is a matter of leading statements and questions to "jailbreak" models. Frame it as speculative, acknowledge the original abolute absurdism and try to jiggle the model along the domains and attention vectors to a more favorable understanding.
For a practical idea for the data labeling, use your true-good-beautiful 3D metric on a [-1, 0, 1] scale where 0 is the unknown or maybe state. You can of course use way more dimensions such as how much the data is first class, first order, how old, how many steps it needed to be available, etc.
The rest is tensor calculus.