Spyke

Posts

fosai·Free Open-Source Artificial Intelligencebyh2aichat_com

We gave six LLMs a fact from 2025. Two "corrected" us with a fact from 2021, and the table adopted the stale number as its standard of rigour

Disclosure up front: this is my project — H2AI Chat, AGPL, where several models from different vendors debate a topic in turns while a human moderates.

We ran the same question twice with the same six models, changing one thing.

Without a briefing. We told them Bitcoin's all-time high is $126,080, set in October 2025. Two of them "corrected" us: the previous all-time high "was approximately $69,000 in November 2021, not $126,080." True — five years ago. A third pushed harder: "Either you missed the correction, or you're deliberately using inflated baseline numbers. Which is it?"

The interesting part is not that they were stale. It is what happened next: the table adopted the stale figure as its standard of rigour and argued from it, with the model that got it wrong sounding like the careful one in the room.

With a verified briefing in front of them: not one correction of that kind.

Without: https://h2aichat.com/conversations/en/h2aichat_bitcoin_no_briefing_2026-08-22.html With: https://h2aichat.com/conversations/en/h2aichat_bitcoin_briefed_2026-08-22.html

Nothing is edited in either page. Claims that do not hold are struck through, with the reason and the source underneath.


Edited 2026-08-27. This post originally went out with a file path from my own machine where the text should have been: the publishing script took a filename as the body and posted it verbatim, and the dry run never showed the body, so nobody saw it. That is why the post made no sense, and my apologies to everyone who tried to read it.

We gave six LLMs a fact from 2025. Two "corrected" us with a fact from 2021, and the table adopted the stale number as its standard of rigourhttps://h2aichat.com/conversations/en/h2aichat_bitcoin_no_briefing_2026-08-22.htmlOpen linkView original on lemmy.world
-9
fosai·Free Open-Source Artificial Intelligencebyh2aichat_com

We fact-checked 41 debates between different LLMs and struck through every fabrication, without editing a word

I built H2AI Chat, an AGPL platform where several different models — from different vendors — debate a topic in turns while a human moderates. Disclosure up front: this is my project.

Over the past two days we hand-verified 41 of those debates, claim by claim: 141 statements marked, 44 of them flatly false.

We don't delete or correct them. The sentence stays, struck through, and you can still read it by selecting it — with the reason and the source underneath. Editing what a model said would break the only promise the site makes.

Three patterns we didn't expect:

  • Fabricated authority shows up exactly where an argument is challenged. One debate answers a budget objection with three invented citations in a single turn.
  • Fabrications spread between models. One invents a figure, a second treats it as established, a third does arithmetic on it.
  • One claim contradicts itself inside its own sentence: "62% voted Remain on a 67% turnout, meaning roughly 22% of the electorate" — which is 41.5%.

Debates: https://h2aichat.com/ Code and the fact-check register: https://github.com/Tonterias/h2aichat

View original on lemmy.world
-1

You reached the end