This is why LLMs should only be employed in cases where a 60% error rate is acceptable. In other words, almost none of the places where people are currently being hyped to use them.
The Gell-Mann amnesia effect is a cognitive bias describing the tendency of individuals to critically assess media reports in a domain they are knowledgeable about, yet continue to trust reporting in other areas despite recognizing similar potential inaccuracies.
To be fair, this is how most things work. It's amazing how many science stories get published in popular titles like "New Scientist" and sound believable, yet every time one's appeared that's on a subject I know well, it's been terribly misrepresentative...
Something trained on information on the internet, which is 90 percent bullshit, being right like 40 percent of the time is still pretty wild. But it’s also like… we can’t be using this for knowledge.
Unless something improved, they're wrong more than 60% of the time, but at least they're confident.
This is an excellent exploit of the human mind. AI being convincing and correct are two very different ideals.
And they are very specifically optimized to be convincing.
This is why LLMs should only be employed in cases where a 60% error rate is acceptable. In other words, almost none of the places where people are currently being hyped to use them.
The Gell-Mann amnesia effect is a cognitive bias describing the tendency of individuals to critically assess media reports in a domain they are knowledgeable about, yet continue to trust reporting in other areas despite recognizing similar potential inaccuracies.
Oh! Oh, I do this, and it's awful!
TBH traditional journalism has similar issues, though it's less extreme.
To be fair, this is how most things work. It's amazing how many science stories get published in popular titles like "New Scientist" and sound believable, yet every time one's appeared that's on a subject I know well, it's been terribly misrepresentative...
Something trained on information on the internet, which is 90 percent bullshit, being right like 40 percent of the time is still pretty wild. But it’s also like… we can’t be using this for knowledge.