The infrastructure around AI models has become the most important component of the tools they power.
Agents get more iffy the more steps between base reality and output.
Each layer needs to be rigorously tested for each use case.
The LLM under are systematic. If you run local models, learn the fail gates, you can design good tools.
If you just say "I want it to do stuff." It will. And you get the fail rate im the article.
My first agent was 97% successful on its test run this week, super grep basically (text search a document). It looks for references, writes a report.
So all I can say, is that one narrow use case, great. Any more requires rigor in testing and slow iterated design.
Most errors are design errors.
Human errors.
There is no replacement for human testing and human checking the base output.
Was there a human baseline the AI performance was compared to?
Agents get more iffy the more steps between base reality and output.
Each layer needs to be rigorously tested for each use case.
The LLM under are systematic. If you run local models, learn the fail gates, you can design good tools.
If you just say "I want it to do stuff." It will. And you get the fail rate im the article.
My first agent was 97% successful on its test run this week, super grep basically (text search a document). It looks for references, writes a report.
So all I can say, is that one narrow use case, great. Any more requires rigor in testing and slow iterated design.
Most errors are design errors.
Human errors.
There is no replacement for human testing and human checking the base output.
Was there a human baseline the AI performance was compared to?