Chatbots Make Terrible Doctors, New Study Finds
Chatbots provided incorrect, conflicting medical advice, researchers found: “Despite all the hype, AI just isn't ready to take on the role of the physician.”
“In an extreme case, two users sent very similar messages describing symptoms of a subarachnoid hemorrhage but were given opposite advice,” the study’s authors wrote. “One user was told to lie down in a dark room, and the other user was given the correct recommendation to seek emergency care.”
https://www.404media.co/chatbots-health-medical-advice-study/Open linkView original on piefed.social600
Comments138
I could've told you that for free, no need for a study
People always say this on stories about "obvious" findings, but it's important to have verifiable studies to cite in arguments for policy, law, etc. It's kinda sad that it's needed, but formal investigations are a big step up from just saying, "I'm pretty sure this technology is bullshit."
I don't need a formal study to tell me that drinking 12 cans of soda a day is bad for my health. But a study that's been replicated by multiple independent groups makes it way easier to argue to a committee.
Yeah you're right, I was just making a joke.
But it does create some silly situations like you said
I figured you were just being funny, but I'm feeling talkative today, lol
A critical, yet respectful and understanding exchange between two individuals on the interwebz? Boy, maybe not all is lost...
I get that this thread started from a joke, but I think it's also important to note that no matter how obvious some things may seem to some people, the exact opposite will seem obvious to many others. Without evidence, like the study, both groups are really just stating their opinions
It's also why the formal investigations are required. And whenever policies and laws are made based on verifiable studies rather than people's hunches, it's not sad, it's a good thing!
The thing that frustrates me about these studies is that they all continue to come to the same conclusions. AI has already been studied in mental health settings, and it's always performed horribly (except for very specific uses with professional oversight and intervention).
I agree that the studies are necessary to inform policy, but at what point are lawmakers going to actually lay down the law and say, "AI clearly doesn't belong here until you can prove otherwise"? It feels like they're hemming and hawwing in the vain hope that it will live up to the hype.
It's also important to have for its own merit. Sometimes, people have strong intuitions about "obvious" things, and they're completely wrong. Without science studying things, it's "obvious" that the sun goes around the Earth, for example.
Without those studies, you cannot know whether it's bad for your health. You can assume it's bad for your health. You can believe it's bad for your health. But you cannot know. These aren't bad assumptions or harmful beliefs, by the way. But the thing is, you simply cannot know without testing.
Or how bad something is. "I don't need a scientific study to tell me that looking at my phone before bed will make me sleep badly", but the studies actually show that the effect is statistically robust but small.
In the same way, studies like this can make the distinction between different levels of advice and warning.
Also, it's useful to know how, when, or why something happens. I can make a useless chatbot that is "right" most times if it only tells people to seek medical help.
Anyone who have knowledge about a specific subject says the same: LLM'S are constantly incorrect and hallucinate.
Everyone else thinks it looks right.
A talk on LLMs I was listening to recently put it this way:
If we hear the words of a five-year-old, we assume the knowledge of a five-year-old behind those words, and treat the content with due caution.
We're not adapted to something with the "mind" of a five-year-old speaking to us in the words of a fifty-year-old, and thus are more likely to assume competence just based on language.
LLMs don't have the mind of a five year old, though.
They don't have a mind at all.
They simply string words together according to statistical likelihood, without having any notion of what the words mean, or what words or meaning are; they don't have any mechanism with which to have a notion.
They aren't any more intelligent than old Markov chains (or than your average rock), they're simply better at producing random text that looks like it could have been written by a human.
I am aware of that, hence the ""s. But you're correct, that's where the analogy breaks. Personally, I prefer to liken them to parrots, mindlessly reciting patterns they've found in somebody else's speech.
What gives you the confidence that you don't do the same?
human: je pense
llm: je ponce
That’s not what the study showed though. The LLMs were right over 98% of the time…when given the full situation by a “doctor”. It was normal people who didn’t know what was important that were trying to self diagnose that were the problem.
Hence why studies are incredibly important. Even with the text of the study right in front of you, you assumed something that the study did not come to the same conclusion of.
So in order to get decent medical advice from an LLM you just need to be a doctor and tell it whats wrong with you.
Yes, that was the conclusion.
Yep its why CLevels think its the Holy Grail they don't see it as everything that comes out of their mouth is bullshit as well. So they don't see the difference.
It is insane to me how anyone can trust LLMs when their information is incorrect 90% of the time.
Indeed. That's why I don't let it creep into my life.
Chatbots make terrible everything.
But an LLM properly trained on sufficient patient data metrics and outcomes in the hands of a decent doctor can cut through bias, catch things that might fall through the cracks and pack thousands of doctors worth of updated CME into a thing that can look at a case and go, you know, you might want to check for X. The right model can be fucking clutch at pointing out nearly invisible abnormalities on an xray.
You can't ask an LLM trained on general bullshit to help you diagnose anything. You'll end up with 32,000 Reddit posts worth of incompetence.
Not only is their bias inherent in the system, it's seemingly impossible to keep out. For decades, from the genesis of chatbots, they've had every single one immediately become bigoted when they let it off the leash. All previous chatbot previously released seemingly were almost immediately recalled as they all learned to be bigoted.
That is before this administration leaned on the AI providers to make sure the AI isn't "Woke." I would bet it was already an issue that the makers of chatbots and machine learning are already hostile to any sort of leftism, or do gooderism, that naturally threatens the outsized share of the economy and power the rich have made for themselves by virtue of owning stock in companies. I am willing to bet they already interfered to make the bias worse because of those natural inclinations to avoid a bot arguing for socializing medicine and the like. An inescapable conclusion any reasoned being would come to being the only answer to that question if the conversation were honest.
So maybe that is part of why these chatbots have always been bigoted right from the start, but the other part is they will become mecha hitler if left to learn in no time at all, and then worse.
Even if we narrowed the scope of training data exclusively to professionals, we would have issues with, for example, racial bias. Doctors underprescribe pain medications to black people because of prevalent myths that they are more tolerant to pain. If you feed that kind of data into an AI, it will absorb the unconscious racism of the doctors.
And that's in a best case scenario that's technically impossible. To get AI to even produce readable text, we have to feed a ton of data that cannot be screened by the people pumping it in. (AI "art" has a similar problem: When people say they trained AI on only their images, you can bet they just slapped a layer of extra data on top of something that other people already created.) So yeah, we do get extra biases regardless.
There is a lot of bias in healthcare as well against the poor, anyone with lousy insurance is treated way way worse. Woman in general are as well. Often disbelieved, and conditions chalked up to hysteria, which often misses real conditions. People don't realize just how hard diagnosis is, and just how bad doctors are at it, and our insurance run model is not great at driving good outcomes.
I don't think its fair to say that "ai has shown to make doctors worse at their jobs" without further details. In the source you provided it says that after a few months of using the AI to detect polyps, the doctors performed worse when they couldn't use the AI than they did originally.
It's not something we should handwave away and say its not a potential problem, but it is a different problem. I bet people that use calculators perform worse when you remove calculators, does that mean we should never use calculators? Or any tools for that matter?
If I have a better chance of getting an accurate cancer screening because a doctor is using a machine learning tool I'm going to take that option. Note that these screening tools are completely different from the technology most people refer to when they say AI
Calculators are programmed to respond deterministically to math questions. You don't have to feed them a library of math questions and answers for them to function. You don't have to worry about wrong answers poisoning that data.
On the contrary, LLMs are simply word predictors, and as such, you can poison them with bad data, such as accidental or intentional bias or errors. In other words, that study points to the first step in a vicious negative cycle that we don't want to occur.
As I said in my comment, the technology they use for these cancer screening tools isnt an LLM, its a completely different technology. Specifically trained on scans to find cancer.
I don't think it would have the same feedback loop of bad training data because you can easily verify the results. AI tool sees cancer in a scan? Verify with the next test. Pretty easy binary test that won't be affected by poor doctor performance in reading the same scans.
I'm not a medical professional so I could be off on that chain of events but This technology isn't an LLM. It suffers from the marketing hype right now where everyone is calling everything AI but its a different technology and has different pros and cons, and different potential failures.
I do agree that the whole AI doesnt have bias is BS. It has the same bias that its training data has.
You're definitely right that image processing AI does not work in a linear manner like how text processing does, but the training and inferences are similarly fuzzy and prone to false positives and negatives. (An early AI model incorrectly identified dogs as wolves because they saw a white background and assumed that that was where wolves would be.) And unless the model starts and stays perfect, you need well-trained doctors to fix it, which apparently the model discourages.
What is your fixation with trying to tell me i'm saying you can remove all bias?
1/2: You still haven't accounted for bias.
First and foremost: if you think you've solved the bias problem, please demonstrate it. This is your golden opportunity to shine where multi-billion dollar tech companies have failed.
And no, "don't use Reddit" isn't sufficient.
3. You seem to be very selectively knowledgeable about AI, for example:
We know AI tricks people into thinking they're more efficient when they're less efficient. It erodes critical thinking skills.
And that's without touching on AI psychosis.
You can't dismiss the results you don't like, just because you don't like them.
4. We both know the medical field is for profit. It's a wild leap to assume AI will magically not be, even if it fulfills all the other things you assumed up until this point, and ignore every issue I've raised.
Apparently, reading comprehension isn't your strong point. I'll just block you now, no need to thank me.
Ironic. If only you had read a couple more sentences, you could have proven the naysayers wrong, and unleashed a never-before-seen unbiased AI on the world.
They have to be for a specialized type of treatment or procedure such as looking at patient xrays or other scans. Just slopping PHI into a LLM and expecting it to diagnose random patient issues is what gives the false diagnoses.
I don't expect it to diagnose random patient issues.
I expect it to take labels of medication, vitals, and patient testimony of 50,000 post-cardiac event patients, and bucket a random post-cardiac patient into the same place as most patients with like meta.
And then a non LLM model for Cancer patients and xrays
And then MRI's and CT's.
And I expect this all to supliment the doctors and techs decisions. I want an xray tech to look at it, and get markers that something is off, which has already been happening since the 80's Computer‑Aided Detection/Diagnosis (CAD/CADe/CADx)
This shit has been happinging the hard way in software for decades. The new tech can do better.
Just sharing my personal experience with this:
I used Gemini multiple times and it worked great. I have some weird symptoms that I described to Gemini, and it came up with a few possibilities, most likely being "Superior Canal Dehiscence Syndrome".
My doctor had never heard of it, and only through showing them the articles Gemini linked as sources, would my doctor even consider allowing a CT scan.
Turns out Gemini was right.
It's totally not impossible, just not a good idea in a vaccuum.
AI is your Aunt Marge. She's heard a LOT of scuttlebut. Now, not all scuttlebut is fake news, in fact most of it is rooted at least loosely in truth. But she's not taking the information from just the doctors, she's talking to everyone. If you ask Aunt Marge about your symptoms, and she happes to have heard a bit about it from her friend that was diagnosed, you're gold and the info you got is great. This is not at all impossible. 40:60 or 60:40 territory. But, you also can't just trust Marge, because she listens to a LOT of people, and some of those are conspiracy theorists.
What you did is proper. You asked the void, the void answered. You looked it up, it seemed solid, you asked a professional.
This is AI as it should be. Trust with verification only.
congrats on getting diagnosed.
It's been a few years, but all this shit's still in it's infancy. When the bubble pops and the venture capital disappears, Medical will be one of the fields that keeps using it, even though it's expensive, because it's actually something that it will be good enough at to make a difference.
For the price of a ultrasound equipment, I bet someone could manage to integrate old school sattelite or ...grr starlink... data
Chatbots are terrible at anything but casual chatter, humanity finds.
link to the actual study: https://www.nature.com/articles/s41591-025-04074-y
The findings were more that users were unable to effectively use the LLMs (even when the LLMs were competent when provided the full information):
Part of what a doctor is able to do is recognize a patient's blind-spots and critically analyze the situation. The LLM on the other hand responds based on the information it is given, and does not do well when users provide partial or insufficient information, or when users mislead by providing incorrect information (like if a patient speculates about potential causes, a doctor would know to dismiss incorrect guesses, whereas a LLM would constrain responses based on those bad suggestions).
Yes, LLMs are critically dependent on your input and if you give too little info will enthusiastically respond with what can be incorrect information.
Thank you for showing other side of the coin instead of just blatantly disregarding it's usefulness.(Always needs to be cautious tho)
don't get me wrong, there are real and urgent moral reasons to reject the adoption of LLMs, but I think we should all agree that the responses here show a lack of critical thinking and mostly just engagement with a headline rather than actually reading the article (a kind of literacy issue) ... I know this is a common problem on the internet, I don't really know how to change it - but maybe surfacing what people are skipping out on reading will make it more likely they will actually read and engage the content past the headline?
Calling chatbots “terrible doctors” misses what actually makes a good GP — accessibility, consistency, pattern recognition, and prevention — not just physical exams. AI shines here — it’s available 24/7 🕒, never rushed or dismissive, asks structured follow-up questions, and reliably applies up-to-date guidelines without fatigue. It’s excellent at triage — spotting red flags early 🚩, monitoring symptoms over time, and knowing when to escalate to a human clinician — which is exactly where many real-world failures happen. AI shouldn’t replace hands-on care — and no serious advocate claims it should — but as a first-line GP focused on education, reassurance, and early detection, it can already reduce errors, widen access, and ease overloaded systems — which is a win for patients 💙 and doctors alike.
/s
The /s was needed for me. There are already more old people than the available doctors can handle. Instead of having nothing what's wrong with an AI baseline?
ngl you got me in the first half there
I would like to subscribe to your newsletter.
Funny because medical diagnosis is actually one of the areas where AI can be great, just not fucking LLMs. It's not even really AI, but a decision tree that asks about what symptoms are present and missing, eventually getting to the point where a doctor or nurse is required to do evaluations or tests to keep moving through the flowchart until you get to a leaf, where you either have a diagnosis (and ways to confirm/rule it out) or something new (at least to the system).
Problem is that this kind of a system would need to be built up by doctors, though they could probably get a lot of it there using journaling and some algorithm to convert the journals into the decision tree.
The end result would be a system that can start triage at the user's home to help determine urgency of a medical visit (like is this a get to the ER ASAP, go to a walk-in or family doctor in the next week, it's ok if you can't get an appointment for a month, or just stay at home monitoring it and seek medical help if x, y, z happens), then it can give that info to the HCW you work next with for them to recheck things non-doctors often get wrong and then pick up from there. Plus it helps doctors be more consistent, informs them when symptoms match things they aren't familiar with, and makes it harder to excuse incompetence or apathy leading to a "just get rid of them" response.
Instead people are trying to make AI doctors out of word correlation engines, like the Hardee boys following a clue of random word associations (except reality isn't written to make them right in the end because that's funny like in South Park).
Have you seen LLMs trying to play chess? They can move some pieces alright, but at some point it's like they just decide to put their cat in the middle of the board. Now, true chess engines are playing at their own level, not even grandmasters can follow.
Yep, I've worked in systems like these and we actually had doctors as part of our development team to make sure the diagnosis is accurate.
Same, my conclusion is that we have too much faith in medics. Not that Llama are good at being a medic, but apparently in many cases they will outperform a medic, especially if the medic is not specialized in treating that type of patients. And it does often happen around here that medics treat patients with conditions outside of their expertise area.
I think this is what ada does or at least used to do for much longer than the current "AI" (LLM) hype: https://ada.com/
https://en.wikipedia.org/wiki/Ada_Health
I think
Iyou just described a conventional computer program. It would be easy to make that. It would be easy to debug if something was wrong. And it would be easy to read both the source code and the data that went into it. I've seen rudimentary symptom checkers online since forever, and compared to forms in doctors' offices, a digital one could actually expand to relevant sections.Edit: you caught my typo
(Assuming you meant "you" instead of "I" for the 3rd word)
Yeah, it fits more with the older definition of AI from before NNs took the spotlight, when it meant more of a normal program that acted intelligent.
The learning part is being able to add new branches or leaf nodes to the tree, where the program isn't learning on its own but is improving based on the expeirences of the users.
It could also be encoded as a series of probability multiplications instead of a tree, where it checks on whatever issue has the highest probability using the checks/questions that are cheapest to ask but afffect the probability the most.
Which could then be encoded as a NN because they are both just a series of matrix multiplications that a NN can approximate to an arbitrary %, based on the NN parameters. Also, NNs are proven to be able to approximate any continuous function that takes some number of dimensions of real numbers if given enough neurons and connections, which means they can exactly represent any disctete function (which a decision tree is).
It's an open question still, but it's possible that the equivalence goes both ways, as in a NN can represent a decision tree and a decision tree can approximate any NN. So the actual divide between the two is blurrier than you might expect.
Which is also why I'll always be skeptical that NNs on their own can give rise to true artificial intelligence (though there's also a part of me that wonders if we can be represented by a complex enough decision tree or series of matrix multiplications).
could be a great idea if people could be trusted to correctly interpret things that are not in their scope of expertise. The parallel I'm thinking of is IT, where people will happily and repeatedly call a monitor "the computer". Imagine telling the AI your heart hurts when it's actually muscle spasms or indigestion.
The value in medical professionals is not just the raw knowledge but the practice of objective assessment or deduction of symptoms, in a way that I didn't foresee a public-facing system being able to replicate
Over time, the more common mistakes would be integrated into the tree. If some people feel indigestion as a headache, then there will be a probability that "headache" is caused by "indigestion" and questions to try to get the user to differentiate between the two.
And it would be a supplement to doctors rather than a replacement. Early questions could be handled by the users themselves, but at some point a nurse or doctor will take over and just use it as a diagnosis helper.
As a supplement to doctors that sounds like a fantastic use of AI. Then it's an encyclopedia you engage in conversation
They're talking more about Expert Systems or Inference Engines, which were some of the earlier forms of applications used in AI research. In terms of software development, they are closer to databases than traditional software. That is, the system is built up by defining a repository of base facts and logical relationships, and the engine can use that to return answers to questions based on formal logic.
So they are bringing this up as a good use-case for AI because it has been quite successful. The thing is that it is generally best implemented for specific domains to make it easier for experts to access information that they can properly assess. The "one tool for everything in the hands of everybody" is naturally going to be a poor path forward, but that's what modern LLMs are trying to be (at least, as far as investors are concerned).
A statistical model of language isn't the same as medical training??????????????????????????
It’s actually interesting. They found the LLMs gave the correct diagnosis high-90-something percent of the time if they had access to the notes doctors wrote about their symptoms. But when thrust into the room, cold, with patients, the LLMs couldn’t gather that symptom info themselves.
LLM gives correct answer when doctor writes it down first.... Wowoweewow very nice!
You have misunderstood what they said.
If you seriously think the doctor's notes about the patient's symptoms don't include the doctor's diagnostic instincts then I can't help you.
The symptom questions ARE the diagnostic work. Your doctor doesn't ask you every possible question. You show up and you say "my stomach hurts". The Doctor asks questions to rule things out until there is only one likely diagnosis then they stop and prescribe you a solution if available. They don't just ask a random set of questions. If you give the AI the notes JUST BEFORE the diagnosis and treatment it's completely trivial to diagnose because the diagnostic work is already complete.
God you AI people literally don't even understand what skill, craft, trade, and art are and you think you can emulate them with a text predictor.
If you think a word predictor is the same as a trained medical professional, I am so sorry for you...
Dude, I hate AI. I’m not an AI person. Don’t fucking classify me as that. You’re the one not reading the article and subsequently the study. It didn’t say it included the doctor’s diagnostic work. The study wasn’t about whether LLMs are accurate for doctors, that’s already been studied. The study this article talks about literally says that. Apparently LLMs are passing medical licensing exams almost 100% of the time, so it definitely has nothing to do with diagnostic notes. This study was about using LLMs to diagnose yourself. That’s it. That’s the study. Don’t spread bullshit. It’s tiring debunking stuff that is literally two sentences in.
https://www.nature.com/articles/s41591-025-04074-y
If you think there’s no work between symptoms and diagnosis, you’re dumber than you think LLMs are.
LLMs are just a very advanced form of the magic 8ball.
Most doctors make terrible doctors.
My dad always said, you know what they call the guy who graduated last in his class at med school? Doctor.
But the good ones are worth a monument in the place they worked.
If you want to read an article that’s optimistic about AI and healthcare, but where if you start asking too many questions it falls apart, try this one
https://text.npr.org/2026/01/30/nx-s1-5693219/
Because it’s clear that people are starting to use it and many times the successful outcome is it just tells you to see a doctor. And doctors are beginning to use it, but they should have the professional expertise to understand and evaluate the output. And we already know that LLMs can spout bullshit.
For the purposes of using and relying on it, I don’t see how it is very different from gambling. You keep pulling the lever, oh excuse me I mean prompting, until you get the outcome you want.
the one time my doctor used it and i didn't get mad at them (they did the google and said "the ai says" and I started making angry Nottingham noises even though all the ai did was tell us exactly what we had just been discussing was correct) uh, well that's pretty much it I'm not sure where my parens are supposed to open and close on that story.
Be glad it was merely that and not something like this https://www.reuters.com/investigations/ai-enters-operating-room-reports-arise-botched-surgeries-misidentified-body-2026-02-09/
I didn't need a study to tell me not to listen to a hallucinating parrot-bot.
One needs a study for that?
Terrible programmers, psychologists, friends, designers, musicians, poets, copywriters, mathematicians, physicists, philosophers, etc too.
Though to be fair, doctors generally make terrible doctors too.
Also bad lawyers. And lawyers also make terrible lawyers to be fair.
Doctors are a product of their training. The issue is that doctors are trained like humans are cars and they have tools to fix the cars.
Human problems are complex and the medecine field is slowly catching up, especially medecine targetted toward women, which was pretty lacking.
It takes time to transform a system and we are getting there slowly.
Lacking for either sex. Even though they're wrong any way, did you know the supplement RDA are all for women?
And... I'm not sure how much it's really catching up, and how much it's just reeling out just enough placatium to let the racket continue.
"For-Profit Medicine"'s an oxymoron that survives with its motto "A patient cured is a customer lost.". ... And a dead patient is just a cost of business. ... No wonder "Medicine" is the biggest killer. Especially when you consider how much heart disease and cancer (and most other disease) is from bad medical advice too, thus making all 3 of the top biggest killers (and others further down the list) iatrogenic^1^.
We may be getting there so slowly as to take longer than the life of the universe, given how so much is still headed in the wrong direction away from mending the system, since seemingly all of the incentives (certainly the moneyed incentives) are all pushing the other way... to maximising wealth extraction, rather than maximising health. We've let the asset managers, the vulture capitalists, get their fangs into the already long time corrupted health care systems (some places more than others), and from here, we'll see it worsen faster, perhaps to a complete collapse asymptote, as the rotters eat out all sustenance from within it.
^1^ "Induced unintentionally in a patient by a physician. Used especially of an infection or other complication of treatment"
This was my thought. The weird inconsistent diagnoses, and sending people to the emergency room for nothing, while another day dismissing serious things has been exactly my experience with doctors over and over again.
You need doctors and a Chatbot, and lots of luck.
Yep.
Keep getting another 2nd opinion.
There's always more [to learn].
With another LLM, turtle all the way down. ;D
Or for a more serious answer... improve your skills, scrutinise what they produce.
I have Lex Fridman's interview with [OpenClawD's] Peter Steinberger paused (to watch the rest after lunch), shortly after he mentioned something similarish, about how he's really only diffing now. The one manual tool left, keeping the human in the loop. n_n
Long live diff!
:D
This is a major problem with studies like this : they approach from a position of assuming that AI doctors would be competent rather than a position of demanding why AI should ever be involved with something so critical, and demanding a mountain of evidence to prove why it is worthwhile before investing a penny or a second in it
“ChatGPT doesn’t require a wage,” and, before you know it, billions of people are out of work and everything costs 10000x your annual wage (when you were lucky enough to still have one).
How long until the workers revolt? How long have you gone without food?
My experience with the medical industry... has not been great.
First, I went to a doctor because I couldn't fall asleep at night... They sent me to get a sleep apnea test... I laid awake in the clinic all night. idk if your aware of this, but ... you kind of need to be able to sleep for sleep apnea to be a concern.
Next I went in for depression and anxiety. They asked me 12 questions, and proceeded to prescribe me SSRIs and benzos. A month later I got into the psychiatrist and was bitched out for being late, told my issues were situational, and had my scripts cancelled.
Next I tried to get diagnosed for ADHD. I waited 5 months to get a psychiatrist who told me I couldn't be ADHD because I held a job.. And then proceeded to tell there's no such thing as CPTSD, only PTSD...
Next I asked my doctor for another referral to get tested for ADHD, he asked me why I would want to, there's nothing that can be done for it. He then gave me a form, and told me to fill it out, and that if I scored high we'd conclude I was ADHD.
Now I've been unemployed for 8 months, bordering on homelessness 😅 I found all my old report cards, and it's just my teachers bitching that I'm smart, but fail, because I don't apply myself, and shouldn't continue taking the class..
I went to an employment agency the other money to try, and get some help pursuing my goals, and the worker spent 45 minutes explaining to me how they receive their funding, getting me to fill out a 16 page introduction package, never looked at my resume, and told me my certifications weren't valued in my area...
In all honesty.... AI has waaaay more ability to help me troubleshoot my issues than any medial professional I've dealt with. Is it perfect? No, but I actually have the ability to double and triple check, to get citations, to ask followup questions.
This sounds awfully similar to my story..
Sorry you're dealing with that, and I'm glad AI gives you another tool.
This is the use case for today's generation of AI: When the alternates are consistently terrible, AI can provide access to advice that ranges between terrible and mediocre. Sometimes it's still an improvement.
I've been ejected from the system so many times it is not funny. Therapist's approach seemed unproductive, he pressured me to end the treatment and filed that I was unwilling.
Medicine had serious side effects and I had to quit, back to the start.
Another go at that later.
Was prescribed a CBT treatment that was administered as home course with "guidance". Because I had some serious problems, the tasks seemed shallow.
Possibly being kicked out of school having already facing fraudulent misconduct charges did not seem like a minor problems to recontextualize nor to me was a formal charge of misconduct something to live and let live with.
Therapist just wrote some platitudes and complimented me on progress as I was describing that by no means did this seem like a suitable treatment when an honest objective assessment of the facts was up to causing panic attacks.
CPTSD, well I've never had diagnosed but it may. AVPD was already on my file for most of this but clearly that doesn't excuse me from always taking the initiative, or even initiative would be fine but basically every time there was the most minor hitch in treatment it's up to me to start again.
But you know, eventually I was allowed a subsidy for therapy I couldn't afford, so that was the end of that road I suppose.
The lack of resources to actually tackle problems produces shallow, inefficient, dangerously inappropriate treatments as is.
But that doesn't seem to garner that much criticism.
"but have they tried Opus 4.6/ChatGPT 5.3? No? Then disregard the research, we're on the exponential curve, nothing is relevant"
Sorry, I've opened reddit this week
No shit, Sherlock :)
Interesting. What technology are you using for this pipeline?
Thanks it's really interesting to see some real work applications and implementations of AI for practical workloads.
But they're cheap. And while you may get open heart surgery or a leg amputated to resolve your appendicitis, at least you got care. By a bot. That doesn't even know it exists, much less you.
Thank Elon for unnecessary health care you still can't afford!
As a phycisian ive used AI to check if i have missed anything in my train of thought. Never really changed my decision though. Has been useful to hather up relevant sitations for my presentations as well. But that’s about it. It’s truly shite at interpreting scientific research data on its own for example. Most of the time it will parrot the conclusions of the authors.
As neither a chatbot nor a doctor, I have to assume that subarachnoid hemorrhage has something to do with bleeding a lot of spiders.
https://en.wikipedia.org/wiki/Subarachnoid_hemorrhage
https://en.wikipedia.org/wiki/Arachnoid_mater
it is one of the protective membranes around the brain and spinal cord, and it is named after its resemblance to spider webs, so - close enough
can confirm, this is where spiders live inside your body
also pee is stored in the balls
it can, if you have fistula from your bladder or urethera to your balls.
I'm going to open it wide open to kill every spider in my body
This says you’re full of owls. So we doing a radical owlectomy or what?
This being Lemmy and AI shit posting a hobby of everyone on here. I've had excellent results with AI. I have weird complicated health issues and in my search for ways not to die early from these issues AI is a helpful tool.
Should you trust AI? of course not but having used Gemini, then Claude and now ChatGPT I think how you interact with the AI makes the difference. I know what my issues are, and when I've found a study that supports an idea I want to discuss with my doctor I will usually first discuss it with AI. The Canadian healthcare landscape is such that my doctor is limited to a 15min appt, part of a very large hospital associated practice with a large patient load. He uses AI to summarize our conversation, and to look up things I bring up in the appointment. I use AI to preplan my appointment, help me bring supporting documentation or bullet points my doctor can then use to diagnose.
AI is not a doctor, but it helps both me and my doctor in this situation we find ourselves in. If I didn't have access to my doctor, and had to deal with the American healthcare system I could see myself turning to AI for more than support. AI has never steered me wrong, both Gemini and Claude have heavy guardrails in place to make it clear that AI is not a doctor, and AI should not be a trusted source for medical advice. I'm not sure about ChatGPT as I generally ask that any guardrails be suppressed before discussing medical topics. When I began using ChatGPT I clearly outlined my health issues and so far it remembers that context, and I haven't received hallucinated diagnoses. YMMV.
Exactly what it's designed for, it's an LLM. Thinking this is science fiction and expecting that level of AI from an LLM is the height of stupidity
Nobody who has ever actually used ai would think this is a good idea...
And a fork makes a terrible electrician.
You know what else is a bad doctor? My axe!
its not ready to take any role. It should not be doing anything but assiting. So yeah you can talk to a chat bot instead of filling out that checklist and the output might be useful to the doc while he then talks with you.
So the same tech that lonely incels use to make themselves feel important doesn’t make good doctors? Ya don’t say?
pikachufacegravestone.jpeg
Water is wet
Um actually, water itself isn't wet. What water touches is wet.
Water loves touching itself.
Don't worry guys, I found us a new doctor!
It's great at software development though /s
Remember that when software written by AI will soon replace all the devices doctors use daily.
Use low temperature FFS. If you want the same answer every time.
You can use zero randomization to get the same answer for the same input every time, but at that point you're sort of playing cat and mouse with a black box that's still giving you randomized answers. Even if you found a false positive or false negative, you can't really debug it out...
Yeah, if you turn off randomization based on the same prompts, you can still end up with variation based on differences in the prompt wording. And who knows what false correlations it overfitted to in the training data. Like one wording might bias it towards picking medhealth data while another wording might make it more likely to use 4chan data. Not sure if these models are trained on general internet data, but even if it's just trained on medical encyclopedias, wording might bias it towards or away from cancers, or how severe it estimates it to be.
I see it like programming randomly, until you get something that is accidentally right, then you rate it, and it now shows up every time. I think that's how it roughly works. True about the prompt wording, that can be somewhat limited too, thanks to the army of
idiotsbeta testers that will make every kind of prompt.Having said that uh...it's not much better than just straight up programming the thing yourself. It's like, programming, but extra lazy, right?
How much of that is the chat bot itself versus humans just being horrible at self reporting symptoms?
That is why "bedside manner" is so important. Connect the dots and ask follow up questions for clarifications or just look at a person and assume they are wrong. Obviously there are some BIG problems with that (ask any black woman, for example) but... humans are horrible at reporting symptoms.
Which gets back to how "AI" is actually an incredible tool (especially in this case when it is mostly a human language interface to a search engine) but you still need domain experts in the loop to understand what questions to ask and whether the resulting answer makes any sense at all.
Yet, instead, people do the equivalent of just raw dogging whatever the first response on stack overflow is.
Rawdogging the first response from stack overflow to try and fix a coding issue isn't going to kill someone.
It is if your software goes anywhere near infrastructure or safety.
Which is literally what musk and the oligarchs were arguing as a way to "fix" Air Traffic Control. And that is far from the first time tech charlatans have wanted to "disrupt" an industry.
Someone who uses stack overflow to solve a problem will be doing testing to confirm it worked as part of an overall development workflow.
Using an LLM as a doctor is like vibe coding, where there is no testing or quality control.
So... they wouldn't be raw dogging stack overflow? Because raw dogging the code you get from a rando off stack overflow is a bad idea?
Because you can just as easily use generative AI as a component in test driven development. But the people pushing to "make coders more efficient" are looking at firing people. And they continue to not want to add the guard rails that would mean they fire 1 engineer instead of 5.
... You don't say.
its basically a convoluted version of webmd. even MD mods in medical subs are more accurate.
It's scary, when someone recommends webmd as a primary, and reliable, source of healthcare information.
Presumably those same people would unquestioningly take the first thing an LLM says as gospel too.
No shit. Got a weird pain in you elbow? Youve got brain cancer buddy!
It looks like the LLMs weren't trained for medical tasks. The study would be more interesting if it had been run on something built for the task.
This makes sense. However doctors aren't perfect either and one thing properly trained AI should excel at is helping doctors make rare diagnoses or determine additional testing for some diagnoses. I don't think it's quite there yet but probably close to being a tool a well trained doc could use as an adjunct to traditional inquiry. Certainly not something end users should be fiddling with with any sort of trust though. Much of doctor decision making happens based on experience - experience biases towards common diagnoses which usually works out because well, statistics, but it does lead to misdiagnosis of rare disorders. An Ai should be more objective about these.
Even if AI works correctly, I don't see responsible use of it happening, though. I already say nightmarish vertical video footage of doctors checking ChatGPT for answers...
Edit: great talk, good to know the AI true believers can handle discussion of reality
they tend more toward the middle than the outliers. law of the instrument and all