ChatGPT, Google Gemini, Google AI Overviews, Grok and Replika AI bots made huge errors before Scottish election, study finds
cross-posted from: https://piefed.world/c/tech/p/1140104/chatgpt-google-gemini-google-ai-overviews-grok-and-replika-ai-bots-made-huge-errors-befo
https://demos.co.uk/research/electoral-hallucinations-safeguarding-uk-elections-in-the-world-of-llms-and-ai-chatbots/Open linkView original on piefed.world
3 replies
Yeah, it's what they do. Generate convincing text. Calling it "errors" makes as much sense as claiming my dice "produced errors" when I lost at yahtzee.
An illustrative example: https://kucharski.substack.com/p/real-signals-or-artificial-stereotypes
"First, I’d created 2000 free-text responses and labelled them ‘UK’. Then I copied and pasted the exact same 2000 responses but labelled these ‘US’. Finally, I combined them to create a dataset of 4000 total responses, and jumbled them up.
Despite the responses being identical for the UK and US, Copilot produced a rich, detailed summary of how US and UK respondents differed."
LLMs making "errors" around elections might be the exact point of them.
The last bit is the most important imo: Chatbots must not be allowed to present themselves as providers of information. Nor should any commercial/official body be allowed to rely on them.