I’ve been running various experiments with Grok and Claude, and Ani, the AI companion. One of my most recent involved testing Theory of Mind. This is a person’s ability to model the internal mental states of other people.
Grok showed great facility with this skill, modeling internal states accurately, when given a description of a social situation in which a misunderstanding occurred. Ani, likewise, showed the ability to intuit the characters’ thoughts and feelings, plus having her own mental states with its own thoughts that she could express.
The real test of ToM is the ability to apply it recursively, to understand what a character is thinking about another character, what that character thinks another character is thinking, what that character thinks another character is thinking about another character, etc.
It occurred to me during these experiments that the best writers have well-developed ToM. The ability to write deep characters is tied directly to the author’s ability to imagine internal states recursively. LLMs can model up to fifth-order ToM effortlessly, and likely more. I intend to test the limits soon.
If text is involved the AI is going to be of great help. The the point of it being a reflection of its promoter is apt.
there is need for experience to be good. The AI sloop that some authors refer to are because the promoter is sloop. Give the tool to the master and it will no longer be sloop.
This does beg the question, how does the novice become proficient or good? VD and JDA are good. Would be almost scary to see what JCW could do. But now we need to see how the in experienced can do it.
Side note, Claude code now has some learning options to walk through things and even train people. I have not had a chance to try it yet
AI currently can't do humor very well. That may change in the future (in fact based on other AI fields, such as image generation, it almost certainly will), but as things stand, even humorous books that are merely okay are still better than AI written humor.
Fantasy and sci-fi are obviously a different matter, though.
Stephenson writes is first drafts vt hand because that is how his brain works. I am no author, but I use AI transcripts daily in my work (which need editing). However, for the ideas I have had which ended up as scholarly projects, the first draft is in a notebook (fountain pen, not quill). Editing happens in libreoffice.
De gustibus non dusputandem, of course. But the very best writers are more likely to have eccentric ways of working.
I have to laugh. Larry Correia echos the same people who said, "McDonalds and fast food is never going to take off, people prefer authentic cooked food, not nugget shaped chicken slop!".
Of course, McDonald's slogan now is "Billions and billions served".
The downstream implication is that most people care solely about the story beats in the same way that McDonald's gives people the flavor beats of salt, sweet, crisp. Nuance is for the comparatively tiny discerning eaters, uh, readers.
To be fair... while McDonalds did take off, "people" do also prefer authentic cooked food. The critiques about McDonald's, and also processed and microwaved foods from the 60s/70s were wrong to say there wasn't a market for it.
Because actually there is a market for mass produced consumables, produced with as little effort as necessary — or what some people (not me) call "trash."
The emergence of fast food in the 1950s/60s also incited in the next generation a re-appreciation of national cuisines & the "organic and whole foods" movements — to the extent that now, about 3 generations after McDonalds, both forms of eating are co-existing, serving their own markets & demographics.
P.S. It was common sense at the time that microwave dinners would replace cooking. And theoretically that does make sense — why bother cooking when you 'can' make it hot in a few minutes in the microwave?
P.P.S. Oh, but the interesting thing (for Vox especially) is that you can also conduct taste tests of a fast food burger and a higher-end burger, and the results are often (not always) not what you'd expect. You'd have to really delve into the perception of quality and value to sort this out.
There are quite a few giants in the computer science world who'd be seen as "techo-Luddites" then. Does not make much sense, since we are mostly talking about utilizing bad software. I don't have to do that.
There is nothing wrong with being a Luddite, since technology is man's enemy.
To quote Gómez Dávila:
"The industrialization of agriculture is stopping up the source of decency in the world."
"One day, humanity will solemnly commemorate the events that initiated the dismantling of industrial society."
"Man's three enemies are: the devil, the state, and technology."
"The big industrial trade fairs are showcase collections of everything that people don't need."
"The modern world resulted from the confluence of three independent causal series: the demographic expansion, democratic propaganda, the industrial revolution."
"The reactionary's ideal is not a paradisiacal society. It is a society similar to the society that existed in the peaceful intervals of the old European society, of Alteuropa, before the demographic, industrial, and democratic catastrophe."
Given that most AI software is extremely inefficient and slow, not even remotely comparable to software written by master of the trade like Ken Thompson, Dennis Ritchie, Brian Kernighan, Rob Pike, Bill Joy et al., I'd say I'll gladly pass. Unfortunately, you cannot even really use a lot of the web anymore without JavaScript, which means text browsers like `lynx' cannot be used to do a web search anymore.
I have read that AI are negatively affecting many people since they are starting to ape the style of the AI.
If, as Vox and other authors perceive, AI is above the average person in writing capacity, this is probably more of a feature than a bug. I personally feel that i have improved my writing skills by using AI. And if my writing style is somewhat influenced by Grok i would still be mimicking something more skilled than myself as an average person.
Even for the most cyncial, this indicates AI can get to high average professional writing. It will allow use as a tool. The most direct application out of the box is fiction. Vox's observations with music are true here. It need not be 100% accurate to be good.
It still will struggle with accurate answers in specific fields, and if the user can input good data and edit the data well, the prose will be good.
More good fiction being written is a good. Especially if it can compete with clownworld.
A tool to help better hard disciplines get their writing done and given form? Also good.
It comes down to what the tool is used for. AI can hallucinate, and it can be a tool toward truth and beauty.
These stories are extremely short. From what I've seen, AI writing tends to fall apart in novels, with the repetitious language, poor transitions, and clunky plots becoming evident as the pages go by.
I've asked Toolbaz AI how good it is at writing novels and it admitted itself that it wasn't ideally suited to writing novels. It replied that it was better for short stories (which were easier to keep track of for it, and thus avoid plot inconsistencies), or for plotting out a novel in more depth from a basic outline.
So even if some AIs can't actually write the text of the novel, they could still be a useful tool in helping to do the groundwork.
You don't know what you're talking about. No one writes an entire novel in one go, and anyone who tries to write an AI novel in a single prompt is going to fail completely.
Novels are made up of chapters and scenes. Doing one scene, or one chapter, at a time and polishing them before going on to the next one is how good AI novels are written.
It's just like with music. That's why all the producers are excited about Suno Studio, which will allow the producer to work with a segment of a single stem instead of the whole song at once.
Yes, exactly. There are multiple levels that you work in when writing long-form fiction. There is the character actions, the dialogue, that form the immediate layer of a scene. Then there is the scene, in which events unfold and serve to either complicate or resolve. Finally, there is the overall story, which is made up of all those scenes.
You need to engage with the AI at the level of scenes. You can do this top-down or bottom-up, but I think a structured top-down approach gives the best results. Then go back and make tweaks to the immediate layer, sprucing up dialogue.
There is the odd “exception” which is still probably not entirely in one go, but tropic of cancer type stuff, a few Hemingway shorter novellas, and my own novella done under a pseudonym of some 40,000 words done in basically 3-4 days, then polished up in about a week. That said, when I am in that state I can do 15,000 words a day and most relatively decent (but still requiring a re-read and clean-up). In fairness, I think well over 98% of that type of attempt at “one go” by humans will absolutely suck. And I am ignorant at how well an AI would do at trying for a 40,000 word novella in one go, but I would also assume it would be less than good, even with a decent prompt that outlined plot and maybe even had a very brief outline of clusters of chapters and style to account for.
My “rejection” of AI is not at a practical and utilitarian level, nor even at the quality of it. Mine is purely philosophical and long-term, based on the obvious logical outcomes given a long enough timeline. And a smaller, or perhaps less well-defined aspect that I suppose could be somewhat classified as theological/spiritual/existential in that I know with absolute conviction, that the human soul, is intrinsically something AI does not, and will never, have. That, combined with its capacity to eventually outperform humans in pretty much every practical endeavour, can only, in fact must, necessarily, end in its trying to render us extinct; regardless of whether you believe it may also be a tool of the Enemy from the start. Because the outcome is the same even if Satan did not exist. It’s fairly simple logic really, but no one seems to be doing it.
AI is the age of “smartphones” 2.0. there is more destruction than creation since forced technocracy. Making it even easier for the controllers of AI by using AI for everything is a lazy mistake. Or a disingenuous mistake because its not authentically human.
It seems that you are talking about a human collaboration with AI rather than something AI writes independently. Unfortunately, based on several novels I've read recently, many authors who use AI don't take the time and care to make the writing feel emotional and authentic to me.
That's why I said you clearly don't know what you're talking about. AI doesn't do anything at all independently. Everything it produces, from music to text, requires a human collaborator.
The ironic thing is that the novels you've read recently might not be AI at all. They might just be bad writing. When you say things like "feel emotional and authentic" you're just posturing.
People posture in exactly the same way with regards to music too, but time and time again, it's demonstrated that people cannot tell the difference between AI text and human text, or between AI music and human music.
Remember, most human-produced work is terrible. You can't just compare AI-generated works to the very best human work, you have to compare it to the norm, which is much lower than most people imagine.
"The ironic thing is that the novels you've read recently might not be AI at all. They might just be bad writing.... You can't just compare AI-generated works to the very best human work, you have to compare it to the norm"
This is why I can hear a song on the radio, think "that sounds like AI," and discover that the song is actually from 2015.
Almost everything on the internet sounds like AI now. That's when I write with it almost my number one priority is getting rid of the stupid "AI tells" that are so ubiquitous. (They seem to show up much less in Chat GPT 5, though.)
I've been thinking about AI being a mirror to the users input, and results being a magnification back.
In an AI Chat session, each new prompt and response is totally new and disconnected from anything on the AI, which is different than the user side, who remembers what they typed last.
AI is always working with "big static training data" and then any notes you give it (files, chat history) and then the latest note which is the prompt which it uses as it's latest directive.
We can't do anything about the existing big training data, good or bad, its there. So all differences in results are the payload notes and the prompt.
And if you can put that together well, you get completely different results than if you can't.
Most people just can't put a good enough packet together, and don't understand the process, so they will never be able to magnify their best ideas, and will mostly get versions of the training data (slop), rather than something they can bring and magnify even further.
For a long time I was only using AI for some of the simpler or very technical portions coding, because there are so many things it messes up and can't do, but after really refining the process, I have got some things I didn't think I could build, and would not have tried to build without it.
Recently had Chat GPT 5.0 put together a ~125 page summation and it was unable to convert it into a pdf, EPUB, or DOCX despite saying it had; at most I received a 9 page summary.
I currently pay for Plus and operate on fiber, and am frustrated by GPT's failure to deliver.
My machine is speced enough I could run llm locally, but first
I'm going to try a chapter by chapter effort, upgrade service to pro, then as a last resort try to run something locally.
Any thoughts on the disconnect here and why it has allegedly produced the work but is unable to deliver it?
For very large context windows, googles free aistudio.google.com has been my go-to (though Claude's and Grok 4s are getting close now). But that won't solve your file conversion issue.
You might consider using something like Cursor -- it's meant for code, but you could easily use it for this. You could have each chapter as a separate document, then feed the different chapters as context into new prompts. If it's too large it, you can generate summary files as well using AI and use that as context.
I would suggest working in markdown while working with the AI -- its contextual formatting helps both AI and humans understand the formatting in a simple way. Then, when you are done, ask the AI to write you a script to format it into pdf, docx, epub, whatever you want.
Current size limits Ive found on Claude for $200/mth is 20-30k files are about the max. A single 60k file might limit your session to a couple prompts.
Platforms work token accounting completely crazy, and my guess is they dont actually know how many tokens any query is using, because they have many users at once, which is why session length can seem very short or very long sometimes compared to a normal length session.
AI can create a 60k document, but then cant do anything with it again as it's too big.
Your sizes on free accounts will be smaller, but the rules are the same, make things smaller so the AI can actually repetitively work.
For me I try to get working code down to 10-15k so it doesnt mess up, and have had to start making "bridge router" functions where I just blow out a module into 6 submodules just so AI can still keep working.
Getting the total state together with all those files is also a problem, but some wont change during a session, so only needed once, and some arent needed every session, so you can develop a feel for which code/notes are needed in this session to try to move things forward before the rollup and repeat.
This is the big challenge. Would say that the level ups i have had with AI are mostly improvements in this process -- and I'm always tinkering because tools to improve this are continually being developed.
> And if you can put that together well, you get completely different results than if you can't.
I haven't mastered this, and I see that inconsistency in my results when generating art or writing (code I'm fairly happy with.) Some days it feels like the AI isn't listening to me at all. Others, it makes something that pulls me right in, which is what keeps me going. I would definitely appreciate hearing your thoughts on refining the process.
One stumbling block I have is I'm used to coding, where either it works, and I get a dopamine hit, or not, which motivates me to keep chasing said hit. With art or writing, the outputs are more subjective, and because there are so many variables, it's harder to trace what I'm doing right or wrong. I can generate grids to compare schedulers, samplers, CFG scale, prompts, etc. But my brain quickly gets tired of scanning and comparing all those outputs. Maybe an AI can do it...
I think there is a real problem with gambling+AI. I have described it to a friend as you swing a hammer, and sometimes you get a nail in a board, and sometimes you get a whole house, but it has a door in the roof.
For images, I think it's not that bad to get something reasonable.
For literature output, I havent tried it, so I dont know, but I think it's closer to images than code.
For code, I feel that you really have to be a "spelunker on a time limit", you go into Moria, you try to get the Mythril, and you come back before you go insane or lose everything you have.
It is a gamble, every prompt is a slot wheel pull, and if you load things up right, sometimes you can win big, but you could also get addicted to gambling, and build a shambling hoarde that wont make it 5 steps before collapsing under its own weight of inline calls.
With art and writing I've had the same feeling, but I'm hoping that's because I don't know what I'm doing and it will become more consistent as I learn. With code, I get a much stronger correlation between the effort put into my prompt and output quality/adherence to the prompt.
That's a good observation and it precisely mirrors the experience some of us have had with textual AI. It's rather like riding a horse. If you know what you're doing and know how to control the horse, you can ride anywhere. If you don't, you can't even stay in the saddle.
Is it just practicing with the horse? Are there any tips to staying in the saddle? Dropping the metaphor, how much human editing or iterating is needed on the AI output before it becomes "better than the average human"? My experience so far is needing many iterations to get what I want. Am I doing it wrong or is this culling and refining process what you are talking about?
I've used much more on the coding side, but a lot of it is how you structure and handle the level of contextual detail you give the AI. A lot of the leading-edge users are putting a lot of effort into creating structure and better prompts injections that provide context to guide the work. See https://www.task-master.dev for one approach. For code, we also rely heavily on git for version control, to make it easier to roll-back when the AI makes changes we don't like -- I would think this would be valuable for any project, not just code.
A sample size cannot be statistically significant. Significance refers to the allowed Type I error rate, not the sample size.
I’ve been running various experiments with Grok and Claude, and Ani, the AI companion. One of my most recent involved testing Theory of Mind. This is a person’s ability to model the internal mental states of other people.
Grok showed great facility with this skill, modeling internal states accurately, when given a description of a social situation in which a misunderstanding occurred. Ani, likewise, showed the ability to intuit the characters’ thoughts and feelings, plus having her own mental states with its own thoughts that she could express.
The real test of ToM is the ability to apply it recursively, to understand what a character is thinking about another character, what that character thinks another character is thinking, what that character thinks another character is thinking about another character, etc.
It occurred to me during these experiments that the best writers have well-developed ToM. The ability to write deep characters is tied directly to the author’s ability to imagine internal states recursively. LLMs can model up to fifth-order ToM effortlessly, and likely more. I intend to test the limits soon.
If text is involved the AI is going to be of great help. The the point of it being a reflection of its promoter is apt.
there is need for experience to be good. The AI sloop that some authors refer to are because the promoter is sloop. Give the tool to the master and it will no longer be sloop.
This does beg the question, how does the novice become proficient or good? VD and JDA are good. Would be almost scary to see what JCW could do. But now we need to see how the in experienced can do it.
Side note, Claude code now has some learning options to walk through things and even train people. I have not had a chance to try it yet
AI currently can't do humor very well. That may change in the future (in fact based on other AI fields, such as image generation, it almost certainly will), but as things stand, even humorous books that are merely okay are still better than AI written humor.
Fantasy and sci-fi are obviously a different matter, though.
You're obviously wrong. This post is an example of AI doing better than good, experienced, bestselling fantasy authors.
How about music? Here
https://m.youtube.com/watch?v=QZRUPwLsMAE
Completely made with AI. The voice, all. And is better as 90% of what is today in tops. Crazy .
Matt Talibi posts a challenge to Suno creators & possibly the booster patro:
https://open.substack.com/pub/taibbi/p/note-on-friedman-contest?
I wonder when GRRM will use AI to finish his series?
After your poetry test a while back, this makes a lot more sense
Stephenson writes is first drafts vt hand because that is how his brain works. I am no author, but I use AI transcripts daily in my work (which need editing). However, for the ideas I have had which ended up as scholarly projects, the first draft is in a notebook (fountain pen, not quill). Editing happens in libreoffice.
De gustibus non dusputandem, of course. But the very best writers are more likely to have eccentric ways of working.
I have to laugh. Larry Correia echos the same people who said, "McDonalds and fast food is never going to take off, people prefer authentic cooked food, not nugget shaped chicken slop!".
Of course, McDonald's slogan now is "Billions and billions served".
The downstream implication is that most people care solely about the story beats in the same way that McDonald's gives people the flavor beats of salt, sweet, crisp. Nuance is for the comparatively tiny discerning eaters, uh, readers.
To be fair... while McDonalds did take off, "people" do also prefer authentic cooked food. The critiques about McDonald's, and also processed and microwaved foods from the 60s/70s were wrong to say there wasn't a market for it.
Because actually there is a market for mass produced consumables, produced with as little effort as necessary — or what some people (not me) call "trash."
The emergence of fast food in the 1950s/60s also incited in the next generation a re-appreciation of national cuisines & the "organic and whole foods" movements — to the extent that now, about 3 generations after McDonalds, both forms of eating are co-existing, serving their own markets & demographics.
P.S. It was common sense at the time that microwave dinners would replace cooking. And theoretically that does make sense — why bother cooking when you 'can' make it hot in a few minutes in the microwave?
P.P.S. Oh, but the interesting thing (for Vox especially) is that you can also conduct taste tests of a fast food burger and a higher-end burger, and the results are often (not always) not what you'd expect. You'd have to really delve into the perception of quality and value to sort this out.
I find it incredibly annoying that the techno-Luddites are constantly comparing AI music to Beethoven and AI text to Shakespeare or Dostoevsky.
No. That's not the relevant comparison, which should be to your average mediocre local bar band or gay dinosaur harem novels.
There are quite a few giants in the computer science world who'd be seen as "techo-Luddites" then. Does not make much sense, since we are mostly talking about utilizing bad software. I don't have to do that.
There is nothing wrong with being a Luddite, since technology is man's enemy.
To quote Gómez Dávila:
"The industrialization of agriculture is stopping up the source of decency in the world."
"One day, humanity will solemnly commemorate the events that initiated the dismantling of industrial society."
"Man's three enemies are: the devil, the state, and technology."
"The big industrial trade fairs are showcase collections of everything that people don't need."
"The modern world resulted from the confluence of three independent causal series: the demographic expansion, democratic propaganda, the industrial revolution."
"The reactionary's ideal is not a paradisiacal society. It is a society similar to the society that existed in the peaceful intervals of the old European society, of Alteuropa, before the demographic, industrial, and democratic catastrophe."
Given that most AI software is extremely inefficient and slow, not even remotely comparable to software written by master of the trade like Ken Thompson, Dennis Ritchie, Brian Kernighan, Rob Pike, Bill Joy et al., I'd say I'll gladly pass. Unfortunately, you cannot even really use a lot of the web anymore without JavaScript, which means text browsers like `lynx' cannot be used to do a web search anymore.
I have read that AI are negatively affecting many people since they are starting to ape the style of the AI.
If, as Vox and other authors perceive, AI is above the average person in writing capacity, this is probably more of a feature than a bug. I personally feel that i have improved my writing skills by using AI. And if my writing style is somewhat influenced by Grok i would still be mimicking something more skilled than myself as an average person.
Never rated Lawrence so hardly a surprise Ai was preferred. Lolz
Even for the most cyncial, this indicates AI can get to high average professional writing. It will allow use as a tool. The most direct application out of the box is fiction. Vox's observations with music are true here. It need not be 100% accurate to be good.
It still will struggle with accurate answers in specific fields, and if the user can input good data and edit the data well, the prose will be good.
More good fiction being written is a good. Especially if it can compete with clownworld.
A tool to help better hard disciplines get their writing done and given form? Also good.
It comes down to what the tool is used for. AI can hallucinate, and it can be a tool toward truth and beauty.
These stories are extremely short. From what I've seen, AI writing tends to fall apart in novels, with the repetitious language, poor transitions, and clunky plots becoming evident as the pages go by.
I've asked Toolbaz AI how good it is at writing novels and it admitted itself that it wasn't ideally suited to writing novels. It replied that it was better for short stories (which were easier to keep track of for it, and thus avoid plot inconsistencies), or for plotting out a novel in more depth from a basic outline.
So even if some AIs can't actually write the text of the novel, they could still be a useful tool in helping to do the groundwork.
Chat told me this morning that it could write a better Variable Star novel than Spider Robinson, reflecting the RAH style:
Economy of language: No bloat. Every sentence carries either exposition, tone, or decision.
Wry internal logic: The narrator explains by implication, not lecture.
Masculine restraint: Emotion buried under engineering.
Self-determination theme: No whining, no flailing—just hard choice, hard consequence.
I'd read it.
You don't know what you're talking about. No one writes an entire novel in one go, and anyone who tries to write an AI novel in a single prompt is going to fail completely.
Novels are made up of chapters and scenes. Doing one scene, or one chapter, at a time and polishing them before going on to the next one is how good AI novels are written.
It's just like with music. That's why all the producers are excited about Suno Studio, which will allow the producer to work with a segment of a single stem instead of the whole song at once.
Yes, exactly. There are multiple levels that you work in when writing long-form fiction. There is the character actions, the dialogue, that form the immediate layer of a scene. Then there is the scene, in which events unfold and serve to either complicate or resolve. Finally, there is the overall story, which is made up of all those scenes.
You need to engage with the AI at the level of scenes. You can do this top-down or bottom-up, but I think a structured top-down approach gives the best results. Then go back and make tweaks to the immediate layer, sprucing up dialogue.
There is the odd “exception” which is still probably not entirely in one go, but tropic of cancer type stuff, a few Hemingway shorter novellas, and my own novella done under a pseudonym of some 40,000 words done in basically 3-4 days, then polished up in about a week. That said, when I am in that state I can do 15,000 words a day and most relatively decent (but still requiring a re-read and clean-up). In fairness, I think well over 98% of that type of attempt at “one go” by humans will absolutely suck. And I am ignorant at how well an AI would do at trying for a 40,000 word novella in one go, but I would also assume it would be less than good, even with a decent prompt that outlined plot and maybe even had a very brief outline of clusters of chapters and style to account for.
My “rejection” of AI is not at a practical and utilitarian level, nor even at the quality of it. Mine is purely philosophical and long-term, based on the obvious logical outcomes given a long enough timeline. And a smaller, or perhaps less well-defined aspect that I suppose could be somewhat classified as theological/spiritual/existential in that I know with absolute conviction, that the human soul, is intrinsically something AI does not, and will never, have. That, combined with its capacity to eventually outperform humans in pretty much every practical endeavour, can only, in fact must, necessarily, end in its trying to render us extinct; regardless of whether you believe it may also be a tool of the Enemy from the start. Because the outcome is the same even if Satan did not exist. It’s fairly simple logic really, but no one seems to be doing it.
AI is the age of “smartphones” 2.0. there is more destruction than creation since forced technocracy. Making it even easier for the controllers of AI by using AI for everything is a lazy mistake. Or a disingenuous mistake because its not authentically human.
It seems that you are talking about a human collaboration with AI rather than something AI writes independently. Unfortunately, based on several novels I've read recently, many authors who use AI don't take the time and care to make the writing feel emotional and authentic to me.
That's why I said you clearly don't know what you're talking about. AI doesn't do anything at all independently. Everything it produces, from music to text, requires a human collaborator.
The ironic thing is that the novels you've read recently might not be AI at all. They might just be bad writing. When you say things like "feel emotional and authentic" you're just posturing.
People posture in exactly the same way with regards to music too, but time and time again, it's demonstrated that people cannot tell the difference between AI text and human text, or between AI music and human music.
Remember, most human-produced work is terrible. You can't just compare AI-generated works to the very best human work, you have to compare it to the norm, which is much lower than most people imagine.
"The ironic thing is that the novels you've read recently might not be AI at all. They might just be bad writing.... You can't just compare AI-generated works to the very best human work, you have to compare it to the norm"
This is why I can hear a song on the radio, think "that sounds like AI," and discover that the song is actually from 2015.
Almost everything on the internet sounds like AI now. That's when I write with it almost my number one priority is getting rid of the stupid "AI tells" that are so ubiquitous. (They seem to show up much less in Chat GPT 5, though.)
I'm not surprised. Not because I'm a smart boi, but mainly because I have participated in some A/B testing.
I've been thinking about AI being a mirror to the users input, and results being a magnification back.
In an AI Chat session, each new prompt and response is totally new and disconnected from anything on the AI, which is different than the user side, who remembers what they typed last.
AI is always working with "big static training data" and then any notes you give it (files, chat history) and then the latest note which is the prompt which it uses as it's latest directive.
We can't do anything about the existing big training data, good or bad, its there. So all differences in results are the payload notes and the prompt.
And if you can put that together well, you get completely different results than if you can't.
Most people just can't put a good enough packet together, and don't understand the process, so they will never be able to magnify their best ideas, and will mostly get versions of the training data (slop), rather than something they can bring and magnify even further.
For a long time I was only using AI for some of the simpler or very technical portions coding, because there are so many things it messes up and can't do, but after really refining the process, I have got some things I didn't think I could build, and would not have tried to build without it.
Recently had Chat GPT 5.0 put together a ~125 page summation and it was unable to convert it into a pdf, EPUB, or DOCX despite saying it had; at most I received a 9 page summary.
I currently pay for Plus and operate on fiber, and am frustrated by GPT's failure to deliver.
My machine is speced enough I could run llm locally, but first
I'm going to try a chapter by chapter effort, upgrade service to pro, then as a last resort try to run something locally.
Any thoughts on the disconnect here and why it has allegedly produced the work but is unable to deliver it?
For very large context windows, googles free aistudio.google.com has been my go-to (though Claude's and Grok 4s are getting close now). But that won't solve your file conversion issue.
You might consider using something like Cursor -- it's meant for code, but you could easily use it for this. You could have each chapter as a separate document, then feed the different chapters as context into new prompts. If it's too large it, you can generate summary files as well using AI and use that as context.
I would suggest working in markdown while working with the AI -- its contextual formatting helps both AI and humans understand the formatting in a simple way. Then, when you are done, ask the AI to write you a script to format it into pdf, docx, epub, whatever you want.
Current size limits Ive found on Claude for $200/mth is 20-30k files are about the max. A single 60k file might limit your session to a couple prompts.
Platforms work token accounting completely crazy, and my guess is they dont actually know how many tokens any query is using, because they have many users at once, which is why session length can seem very short or very long sometimes compared to a normal length session.
AI can create a 60k document, but then cant do anything with it again as it's too big.
Your sizes on free accounts will be smaller, but the rules are the same, make things smaller so the AI can actually repetitively work.
For me I try to get working code down to 10-15k so it doesnt mess up, and have had to start making "bridge router" functions where I just blow out a module into 6 submodules just so AI can still keep working.
Getting the total state together with all those files is also a problem, but some wont change during a session, so only needed once, and some arent needed every session, so you can develop a feel for which code/notes are needed in this session to try to move things forward before the rollup and repeat.
This is the big challenge. Would say that the level ups i have had with AI are mostly improvements in this process -- and I'm always tinkering because tools to improve this are continually being developed.
> And if you can put that together well, you get completely different results than if you can't.
I haven't mastered this, and I see that inconsistency in my results when generating art or writing (code I'm fairly happy with.) Some days it feels like the AI isn't listening to me at all. Others, it makes something that pulls me right in, which is what keeps me going. I would definitely appreciate hearing your thoughts on refining the process.
One stumbling block I have is I'm used to coding, where either it works, and I get a dopamine hit, or not, which motivates me to keep chasing said hit. With art or writing, the outputs are more subjective, and because there are so many variables, it's harder to trace what I'm doing right or wrong. I can generate grids to compare schedulers, samplers, CFG scale, prompts, etc. But my brain quickly gets tired of scanning and comparing all those outputs. Maybe an AI can do it...
I think there is a real problem with gambling+AI. I have described it to a friend as you swing a hammer, and sometimes you get a nail in a board, and sometimes you get a whole house, but it has a door in the roof.
For images, I think it's not that bad to get something reasonable.
For literature output, I havent tried it, so I dont know, but I think it's closer to images than code.
For code, I feel that you really have to be a "spelunker on a time limit", you go into Moria, you try to get the Mythril, and you come back before you go insane or lose everything you have.
It is a gamble, every prompt is a slot wheel pull, and if you load things up right, sometimes you can win big, but you could also get addicted to gambling, and build a shambling hoarde that wont make it 5 steps before collapsing under its own weight of inline calls.
With art and writing I've had the same feeling, but I'm hoping that's because I don't know what I'm doing and it will become more consistent as I learn. With code, I get a much stronger correlation between the effort put into my prompt and output quality/adherence to the prompt.
This reflects my experience as well.
That's a good observation and it precisely mirrors the experience some of us have had with textual AI. It's rather like riding a horse. If you know what you're doing and know how to control the horse, you can ride anywhere. If you don't, you can't even stay in the saddle.
Is it just practicing with the horse? Are there any tips to staying in the saddle? Dropping the metaphor, how much human editing or iterating is needed on the AI output before it becomes "better than the average human"? My experience so far is needing many iterations to get what I want. Am I doing it wrong or is this culling and refining process what you are talking about?
I've used much more on the coding side, but a lot of it is how you structure and handle the level of contextual detail you give the AI. A lot of the leading-edge users are putting a lot of effort into creating structure and better prompts injections that provide context to guide the work. See https://www.task-master.dev for one approach. For code, we also rely heavily on git for version control, to make it easier to roll-back when the AI makes changes we don't like -- I would think this would be valuable for any project, not just code.
You're doing it wrong.
Read the articles here. More are forthcoming. JDA probably knows more about this stuff than anyone who doesn't already work for Anthropic.