Modern capitalism derives it's scarcity via artificial means such as legislation.
The only thing this does is restrict the means of information to the people of NYC. Have you used traditional web search for health/legal/etc. advice? Even pre-2015 if you didn't know what you're doing you would get the most unhinged, predatory, incorrect advice imaginable, but legislating away access wasn't going to make health information more correct, accurate, or equitable. It would've just further entrenched the predatory health/legal/etc. industry as the gatekeepers of that knowledge.
I understand the sentiment but I think this is too far of a pushback. Restricting access to information even though it may be inaccurate or misleading isn't the right answer. AI is a tool. I'd wager, force a prompt to specifically be visible whenever subjects break that. Essentially the cigerette equivelant "THIS PRODUCT KILLS YOU" but something like "AI RESPONSES TO MEDICAL INFORMATION SHOULD NOT BE TREATED AS ACCURATE."
I feel like this logic should of banned search engines since they could provide links to sites with misinformation.
I'm careful about laws, i ask for a summary then ask to cite the source for the relevant bits. I'm not asking for legal advice, I want to know what the letter of the law says. Google has always struggled with this.
Maybe I'm just ignorant about this, but it feels a bit "gatekeepy" to outlaw answering questions about the law itself. Like... so I can't use an LLM to help do taxes? Or if I'm served a confusing legal document, I have to pay an expensive attorney to understand the legalese? I can't even just get a rough idea from an LLM first?
EDIT: Also I've literally had a professional accountant send me a ChatGPT conversation they had because they didn't know tax law well enough. I feel like if anything that should be outlawed. Professionals should actually have to know shit and know how to research primary sources.
I would say its to protect people, if you ask a LLM for a legal answer and trust it, then you could get into legal trouble when it turns out to be wrong.
Not only are LLMs wrong like ~1/10th of the time (for the more advanced ones) but law is also an area where there is black and white rules, but also there is interpretation of some aspects.
Some things are set in stone and some are about understanding the intention of the law and being able to back that up in a court if it was challenged.
Do you want the public trusting a LLM that says "oh yes you can claim a tax deduction for an Xbox because its for stress relief related to your job" and then you get whacked with a fine for incorrectly filing your tax.
Using a LLM to help your draft an email to your uncle = low risk.
Using a LLM to help you decide if you actually need a medical procedure your doctor has reccomended = high risk.
If that was the logic we were using then Wikipedia shouldn't be allowed to have articles on health and law either. WebMD shouldn't have ever been legal either.
Obviously, the benefits to having that information available outweigh the hypothetical negatives. Even with them being incorrect, partial and easy to draw invalid conclusions from, having the information be freely and readily accessible is better for everyone.
I guess it's like banning loan sharks. Sure some people could only.get loans through them because banks were racist or whatever. But it's objectively bad for everyone using it mainly because when it's bad its really really bad.
More like you as an average person have no idea what it's hallucinating and what is the truth and may end up in jail or the hospital like the cases that are starting to get reported on who did just that.
I really think this is a bad idea these should really just be flagged, there's many situations where asking these questions in necessary and there's few if any other ways to get an answer
Sorta agree. Maybe force them to have a very obvious sign that says AI is routinely wrong about these topics specifically and are not a source to consider reliable, and then also offer a helpline where one can get reasonable answers even if the answer is "unfortunately you need to seek expert opinion on this"
But googling basic labor laws and simple healthcare tricks should be acceptable like stretches for wrist or knee pain, skin/hair care or what level of diarrhea is hospital worthy should be fine.
I am currently going through a very complex diagnosis of combination of SCDS and migraines, and neck joint / facial nerve issues through multiple doctors - ENT, neurologists and so on.
Having AI backing me up with providing a structural understanding from multiple disconnected sources is a huge blessing.
Especially when you die from migraine, and trying to understand are symptoms worth to go to emergency ward or not.
It's not worth trusting AI with your life over a doctor. If in doubt, go to the emergency ward (assuming you're not American, if you're American then you're fucked either way).
The problem with medical questions like these is LLM would spew out info confidently that doctors/specialist don't even know the answer to. LLM doesn't know when it's guessing.
the text in this image is also just flat-out wrong: the proposed legislation does not actually "ban AI from answering questions" on these subjects at all, as this post claims it does.
We have documented instances of AI chatbots providing false medical license numbers to users and documented cases of real harm from AI chatbots providing misinformation. This is what my common-sense legislation addresses—protecting users from misinformation, scams, and fraud. This legislation does not prohibit a user from asking a chatbot questions or receiving general information and advice, as long as the chatbot is not presenting that information as a licensed professional. It would also not prohibit licensed professionals from consulting AI technology in the course of their work.
I'm surprised that this post is still up so many hours after i (and i'm sure others) reported it, especially given the large number of moderators that this community has.
I feel like finding and citing appropriate law cases would be a great way for AI to be used, especially because the bullshit legal-ese that surrounds everything in court cases is so spectacularly opaque and trivial that you have to pay out the nose just for someone to parse through all the bullshit... Like, lawyers shouldn't even be a thing you need unless it's for some really complicated case. The courtroom should be a place where an average citizen could speak for themselves.
Also, as someone who studied medicine... Do you know how often doctors will take a patients history, step outside the exam room, and then google shit? I don't want to imply that they don't know what they're doing, like, you need to study medicine to know what to google, BUT... they still be googling shit. I think AI could be trained to ask the same patient history questions that a nurse asks. Then using the answers to those questions, then they can provide some feedback, with a big ol disclaimer that it's NOT to be taken seriously, and you should consult with a physician.
I'd agree with you to some extent if people weren't so god damn stupid. So many lazy fucks just take the AI's output at face value, believing it to be some all knowledgeable god, rather than a compilation of all human biases expressed on the internet with a bunch of hallucinations thrown in for good measure.
When people say "AI" these days, they mean the LLM-derived generative AI such as ChatGPT, Google Gemini, Anthropic Claude, and Grok.
These AI systems are not capable of finding authoritative search results. They are good at generating fluent, plausible, context-appropriate text. In legal and medical situations, that's the opposite of what you'd want.
Do you know how often doctors will take a patients history, step outside the exam room, and then google shit?
Think about how often the doctors land on some pseudo-science result. Now imagine that instead of a doctor it's a patient and instead of asking "could these be related" the patient is told "these are related".
a big ol disclaimer that it’s NOT to be taken seriously
This has worked very well for Fox News, which is not a news agency and is for entertainment purpose only. Also for dietary supplements, which are not medicine.
Specifically tuned LLMs that are trained on medical databases and less sycophant (E.g. OpenEvidence) are one of the killer applications of AI. Tons of frivolous AI nonsense but this could actually save lives. Access to broad knowledge and connections that a doctor can make use of.
We're demanding and regulating the complete wrong shit about AI.
I've seen purpose built models for things like patent examination and its clear that AI can be incredibly useful. I think if the implementation of AI was not so abusive and destructive people would not be so unwilling to see the good uses.
We live in a late stage capitalist boring dystopia, of course the rollout for AI is going to be abusive and destructive. The real problem is the unlimited greed and billions of freshly printed money in an overheated capitalist economy that is poured into marketing and destructive causes. But interfering in the "free market" doesn't ever enter the liberals mind so we get this dysfunctional reaction.
AI corporations should be nationalized and turned into public utilities, and put under the mandate that they work for the betterment of the population and economy, not for profit. They cannot be trusted to use AI safely. It's potentially dangerous technology, more dangerous than nuclear power. We should pour billions into AI research but with the objective to make it useful and safe, not to maximize shareholder value.
Outlawing people from using expert AI models to get expert advice is idiotic at best, malicious at worst. Next they'll ban using open source AI models locally. Plutocrats are going to want to collect rent on AI usage.
Or just require the AI to summarize and provide citations?
I started using Gemini, not because I like AI, but because Google search is next to useless now. Its pay to win.
I chose Gemini because it actually gives you the sources at the bottom of most queries it uses Google search to answer.
I don't trust its answers, because it often tries to give an answer instead of a summary. But I don't have to because I can go right to the sources it shares.
You're going to end up with AI that works for some people & not for others under these sorts of policies. I don't really think this is in the spirit of democratic socialism.
What i don't like about this, is the ignorance towards every other field.
The ai is not allowed to tell you that you might have some illness but the LLM is allowed to vibe code the system that holds your medical data, or the software of the medical device that analyses your e.g. blood and provide the data to the doctor.
Systems and professions are interconnected. Chatgtp telling me that I have cancer is not as bad as chatgtp making mistakes causing my medical data to be stolen and sold.
I'm on the same side as you, vehemently fuck generative AI, however the difference between some LLM output incorrectly telling you that you have cancer and a shitty coder vibe coding with severe vulnerability is the ability to place blame, there's a person who's job it was to make sure info is secure and the vibe code actually works. There is no singular person who is traceably culpable for shitty AI output telling you bad info.
Bad code that puts out wrong information to a doctor or endangers my data, threatens my safety without me having any control over. No amount of my own expertise will protect me. Simply because I don't control the situation. No amount of awareness campaigns will fix my issue.
But awareness campaigns have successfully communicated to most of us that googling your health issues will result in nothing of value, and usually a cancer diagnosis. I can choose to protect myself by not asking a llm. Like I protected myself by not asking google.
I'm guessing this isnt about saving jobs, it's about reducing the harm caused by people taking advice from a misinformation machine for complex issues like the law and their physical and mental health.
The working class saw this coming decades ago, only to be mocked and told to retrain by the same credentialized individuals who're now crying about AI. Yesterday's push to offshore jobs and today's rush toward digital automation are simply different phases of the exact same economic displacement. It cannot be stopped with legislation in a globalized world where capital flows freely.
"AI's" are fast, for sure, but they are no substitute for human intelligence and knowledge. Mostly a desperate cash grab, because they had no where else to go after cellphones.
This is fair. The next step is to litigate the companies for the harms they cause.
There's a whole drove of people now that have gotten harmed, murdered, and committed suicide because of the psychological harm that these models cause.
Were it a car with some kind of manufacturing defect, there'd be recalls and you wouldn't be able to sell ones with the fault. If it was salad contaminated with bacteria, that'd be recalled and the entire batch would be destroyed. We're holding salad to a higher standard than technology that people are pushing to integrate deeply into society; into classrooms to help educate children, into medical settings to help organise journals and expedite the work of medical staff, into law... the list goes on.
All the models do this, and it causes real harm to people, but other than individual families and such litigating them, no state has stepped in to protect its citizens.
It’s awful that most people either don’t realize or don’t care. People seem to trust these companies above all else, which is insane. They don’t have the best interest of the users in mind. They only care about profits.
AI is a tool that can provide good and bad results. To people who can't get access to medical care easily (half of the people on the planet) it can be an useful tool to get basic medical knowledge
The issue is that AI is extremely good at sounding right, but only okay-ish at being right.
When you get good sounding information that is wrong, it can be extremely problematic and often much worse than no info at all.
Imagine LLMs aiding people with health improvements, only to lead them into health decline (actually already happening, a lot) not to mention the LLMs that further convince people of their conspiracies.
There is a reason why health professionals go through a lot of training and are under a lot of scrutiny, you wouldn't want just any rando who thinks he know better to be your doctor
Limiting medical knowledge from reliable information resources would be terrible. I’d say that limiting the spread of plausible-sounding medical misinformation generated by a fancy autosuggest tool is good, though.
AI chatbots don’t actually “know” anything. They just generate the most statistically probable appearance of a response to each prompt based on their training data. They don’t tell you when something wasn’t in their training data or when there’s limited research on a subject, just give you the most statistically probable response like always, which leads to a “hallucination” that sounds true but isn’t. A tool that’s trained on MCAT testbanks and “aces” the MCAT won’t necessarily give reliable medical advice to real patient questions. Even if you gave an AI chatbot all the accurate training data in the world, it would still hallucinate information occasionally, and it wouldn’t give you any indication of when it did so. If your doctor screws you over, they can lose their license, but if ChatGPT screws you over, there’s currently very little to stop OpenAI from screwing over more people. OpenAI can just stand behind the disclaimer that says ChatGPT can make mistakes, never mind that users are treating it like a licensed medical professional. There are AI tools that are useful in medical research, but they’re a VERY different kind of AI from the AI chatbots this bill is concerned with.
If you want medical knowledge, there are tons of medical journals, news sources, blogs, and podcasts from doctors/scientists who don’t hallucinate BS when they don’t know something.
Instead of banning it then teach people how to use it and not to trust it 100% (who the hell is doing this to begin with?)
If you want medical knowledge, there are tons of medical journals, news sources, blogs, and podcasts from doctors/scientists who don’t hallucinate BS when they don’t know something.
I can ask an LLM to help me sourcing these journals and informations
Expecting people who have already taken the easy route of asking an AI to summarize medical information to do additional due diligence by reading the sources that the AI spits out is naive as fuck.
There's a reason it became a meme for someone to Google their symptoms, click on WebMD to read about it, and then come to the conclusion that they have cancer, all without ever having seen a medical professional in the meanwhile. Do you really think these sorts of people are going to be using AI responsibly for research purposes only?
There’s a reason it became a meme for someone to Google their symptoms, click on WebMD to read about it, and then come to the conclusion that they have cancer, all without ever having seen a medical professional in the meanwhile. Do you really think these sorts of people are going to be using AI responsibly for research purposes only?
If they come to the wrong conclusion at worst they go to the hospital and someone there tells they are wrong and not to worry. We haven't ban googling your symptoms.
Someone who Googled their symptoms and ended up at WebMD were presented with reliable albeit ambiguous information. Carefully curated by medical professionals and not random snake oil salesmen or homeopathy gurus. They fail to interpret the symptoms correctly, because of course they would. If they are looking up causes of a headache, it can be anything from brain tumor to low blood sugar, or even no reason at all.
Someone who asks AI for medical advice is going to get the entire internet worth of fucking opinions, as well as medical journals and such, because that's where ChatGPT, Gemini, Grok and Claude all got their data from. And it will string those words together in a way that sounds authoritative, but it will have literally zero concept of what it's talking about. It's an overgrown autocomplete. You ask for it to provide some citations, but no amount of asking for citations will compel people to actually read them, nor will there be any guarantee that the LLMs couldn't just hallucinate that as well.
It's like taking a Google search for medical information and adding a middleman who occasionally filters in junk data, then wrapping it in a nice package, giving the impression of authority and accuracy, making the process as frictionless as possible.
There can be no reasonable expectation that people who have not historically been capable of making good choices in regards to seeking online medical advice will suddenly start doing it all of their own volition.
You might say "What's the harm if they end up at the hospital either way"? But the danger is that they never go to the hospital at all because the chatbot told them to sun their perineum or inject bleach with the confident prose of a medical doctor and they just... do it. Because why would the AI be wrong?
And who is to blame when someone is injured or killed as a result? Doctors have standards, and they take an oath to first do no harm. If they make a mistake, they are liable for it. How can you hold a machine intelligence accountable for inaccurate diagnoses?
The simple answer is you can't, and so logically they should be strictly prohibited from masquerading as any kind of authority on medicine.
You can teach people how to use it all day long, but most still won’t use it well consistently in real life. Humans instinctively trust confident-sounding language. Most of us are also way worse at fact checking than we think. LLMs play right into humanity’s flaws, and they’re built to maximize engagement rather than help us avoid going past the tools’ limits. I don’t trust LLMs to be accurate about anything, but I still caught myself reading and failing to check Google AI overviews sometimes before I changed my default search engine to a no-AI alternative. It’s just instinctual when an “answer” is right there at the top of the page in response to your search question.
Sure, you can use an LLM to help you find sources, but at that point you might as well just use a search engine. There are search engines that provide links with actual text from each site rather than potentially hallucinated summaries.
Sure, you can use an LLM to help you find sources, but at that point you might as well just use a search engine. There are search engines that provide links with actual text from each site rather than potentially hallucinated summaries.
Some popular search engines results are way worst than AI
Some search engine results are crap, yes. But even the worse search engine (minus AI overviews or whatever) gives you a setup where: (1) each piece of information has an actual source, (2) you know who the source is, and (3) you can make decisions about which information to trust accordingly. If a popular search engine puts a sketchy company’s blog at the top of the SERP, you can actually see that and choose to look father down for something more reliable. Also, if your current search engine keeps failing you, try a different one! There are many options out there that you might find more helpful than today’s Google or Bing or DuckDuckGo.
Good luck with things, truly. I think you’re going to get hurt eventually if you use LLMs for medical advice, and I genuinely don’t want that for you or anyone. Hopefully this conversation has at least made you a little more skeptical about using LLMs in certain ways.
LLMs are NOT a reputable source of medical information. They regurgitate information from random sources, and can lead to disastrous effects when pulling from bad sources or incorrectly summarizing from a good source. At best it should exclusively provide links to sources (good or bad) and let the end user make the determination if that source can be trusted.
This isn't limiting information at all. People can still access all the reputable sources that provide the information they are looking for. This is simply cutting out the unreliable middleman.
Some legislation is to lay the ground work to expand it later.
Easier to lay a sub floor and put carpet over it later, than to just put carpet over the studs. When you're changing a massive thing like this, it's better to start small and work outwards. Canada introduced universal Dental and Phamacare and it was limited to certain demographics. People bitched about "Well what about me?"
You're coming. Start with what/who is most important, make sure the system works, and carry on from there.
Should ban it giving finance advice as well. Conservatives and Neo-Liberals already want to give the economy over to Ai companies, what's stopping them from going full circle and making the Ai do all the trading as well?
I'm going to assume you researched medical documents, studies, and journals to link things together, and not ask a chat bot "My foot hurts, what is wrong?"
You're smart enough to self diagnose yourself with readily available information to the public, but not smart enough to not know the difference between that and a chat bot. That's remarkable.
Congrats, glad your life is hopefully changing for the better because of this.
I hate taking the wind out of your sails but this is in the area of anecdotal evidence. You are a single person saying this for your specific case. I am sure we will get another few people confirming the same here.
But what about how many people tried this and got nothing useful out? Or even a misdiagnose? Anyone enthusiastic about this possibility of AI diagnosing people wouldn't really mention this. People like to mention the successes but it needs to be weighed against the failures too, and I don't think there is a good record of that.
If there is, its probably a medical study and I'd venture a guess that AI did not perform as well as a trained doctor.
Putting aside the ultra ignorance and/or untreated narcissism just dripping off of a statement like that... If you were curious of the presence of any absolutely blatent bias in your speech, just look at your use of "a lot" and "some" in your final sentence . Wow....
100% for this, as people have said you can get AI to answer anyway, as the "guardrails" are really just part of the closed harness and can only do so much as like with everything with a LLM its just a matter of probability that it will ignore it at some point, regardless better a have than a have not ( although there is an argument this could make people think its safer i guess).
To REALLY solve this, you would have to edit the training data to take this stuff out (the reason they don't is because for the big models, they just scrap the Internet for data to save cost).
But the real answer is we just need to make people aware of how they work they dont think or reason or even really know anything; they just try to guess what answer a human would want.
If you want to do real work with them you need you need be able to train a model on your own data and control the harness and weights (which is why i think open source will win out) even then am pretty sure there main use case is roleplay (for like RPGs) and sex bots as there mistakes dont matter here. ( and yeah there is an issue with people using this stuff for replacement friends so even then not great).
Can we add to this the engineering of any physical object that could threaten human safety? If this keeps up much longer we're going to have buildings falling on top of us.
You should ban using AI for the purpose of legal disputes, medicine, etc, profesionally.
Its not about the output, its about what you use it for.
I am also 100% okay with legally requiring a clearly visible disclaimer that AI shouldnt be used for x/y/z and its output should alwats be fact checked or whatever.
But trying to outright ban specific types of output is pointless and trivial to circumvent.
I agree. You're spot on. This isn't a technical problem. We need to think less like engineers and more like lawyers on these things.
You don't require AI companies to have their models refuse to give out legal advice. As you note, that's a fool's errand. Instead, you make it a felony to use any LLM to write legal advice. Currently lawyers are already facing court sanctions for using AI to write their briefs. If you really want to get serious, you make it a felony. Apply the same penalty to a doctor using an LLM to diagnose their patients as you would if I decided to open a clinic in my house and start practicing medicine without a license. Charge people operating AI therapists with practicing psychology without a license.
This isn't a technical problem, it's a legal one. You don't try to regulate the LLMs themselves. You just make it a severe felony to use that technology in certain highly sensitive applications and professions.
And sure, one could say that we don't have a reliable way of screening AI from human made work. But again, that would be thinking like an engineer, not a lawyer. You don't enforce these laws by screening thousands of legal briefs or medical records with an AI detection filter. You offer huge cash rewards to whistle blowers, so that a disgruntled employee can rat out their bastard boss that's forcing everyone to illegally use AI. Apply massive fines and give half of whatever is collected to the person who blows the whistle.
Are you a structural engineering company owner forcing your employees to use AI to engineer buildings? Now you have to worry about one of those employees, an employee who probably really dislikes using AI, ratting you out. And their whistleblower reward will be so large, they'll end up owning the company. They won't have to worry about being blacklisted from future employers, as they'll be getting enough money to retire.
Don't look for technical solutions for the problem of AI abuse. Look for human solutions.
I think the lawyer and the ai company should get in trouble if ai adds fake case citations. Either the companies fix it, or prevent it from being used for that.
So is this just a meme or is there an actual bill? Something tells me that if a law like this were passed, it would outlaw things like AI assisted protein folding.
The fact that it's focusing on law, psychology, and medicine implies an understanding of how vital AI is in things like hurricane forecasting it wants to exclude but still target other things they can throw under the bus in a bid to get re-elected. But AI is vital in so much that it will be difficult, if not impossible, for technologically illiterate law makers to effectively legislate against poor use of AI.
You think law makers know enough to write legislation that takes that into account? Considering how destructive all the online safety acts are, they don't even know how the Internet works.
I don't think you understand what I'm saying. What I'm saying is that law makers will write a law that makes it illegal to use AI and fail to define what they mean by AI. AI includes everything from A* pathfinding in games to machine learning to LLMs. What everyone here in FuckAI hates is LLMs.
Hell, I don't even think you and I are qualified enough to write laws about this stuff. I'm pushing the envelope of my knowledge here and I've built AI for games. I doubt law makers who think this shit is magical are qualified to do it.
That doesn't make it legal though. I'm an engineer because I'm educated enough to know that's a bad idea. If an MBA builds a bridge he goes to jail anyways. Ai or not...
I work as a software developer, with most of my work focused on development for an iron mine operation. It's a lot of moving pieces, loads of software that needs to cooperate, and on top of that the workers that do the actual work develop their own workflows.
We develop stuff in tandem with the workers as much as possible, as in, we gather what they need and tailor the software to fit their workflows. These workflows change over time though, and decisions cemented in software 20 years ago no longer applies to modern workflows.
Hell, last week I spent a significant amount of time setting up a digital area for them to load ore onto because the software wouldn't cooperate any other way, and even then I had to troubleshoot why this other piece of software wouldn't load data from it.
There are so many different experts in these mines, and an LLM couldn't do any one of their jobs. It's so domain specific, and even specific to the location they work in, and that knowledge is possessed solely by the workers that are doing the work. ChatGPT knows fuck all.
I love this kind of law because it's likely impossible for ai companies to comply.
Could be there'll be some exemptions if the user employs workarounds, because at the moment it's pretty easy to get llms to tell you things they aren't supposed to.
But if the laws are written right with actual penalties that may force the companies to create a product that's capable of having safety parameters - something llms don't currently seem capable of.
I spent about an hour with ChatGPT and I couldn't. After there was a post on lemmy "how to smuggle a pound of flour into a country that has banned pastries" or something like that.
Even open weights models like Gemma or Qwen have it baked in, you need to transform the model to overcome the limitation, it's not part of a harness or post-processing anymore.
This doesn't really work though. If it were simply directing you to actual sources that could help out (articles, and lawyer authored blogs or something), that would be one thing, but LLMs are just probability machines. It'll write out what looks like good sound reasoning and logic, but in reality it's simply the most probable word followed by the next most probable word, over and over again. It's using pattern matching, not knowledge
They output responses that read like fact checks, and link things that are semantically similar to their output.
LLMs don't understand concepts like "fact" and "source" though, so it's a stretch to say they "do fact checks" and "provide online sources".
Always remember, LLMs don't actually understand anything other than what order words typically go in. They're really good at it, sure, but don't let that trick you into thinking they have any deeper understanding than that.
I've tried on several occasions to try and explain this to people, and it's like they WANT it to be a sentient thing, and refuse all evidence to the otherwise.
They do say that. I’ve seen Claude realise some programming package must be internal / non public and it therefore can’t provide any information on that.
It's nothing more than surface level though. Their "knowledge" is limited to whatever is present in their training data, and any claims it makes outside of that are manufactured based on statistics and it's knowledge of word order, not actual understanding of actual facts.
Just because it gets it right sometimes doesn't mean it knows it's right. Because, again, the only thing LLMs actually understand is word order.
No it’s not limited to their current training data. Like I said, they can search the internet, get information from websites or available studies, summarize and use that in their answers.
Just because it gets it right sometimes doesn’t mean it knows it’s right.
Except AI is an over engineered autocomplete operating completely off of statistical models. Knowledge is not statistical or random. I should get the same outputs to the same inputs.
Ai can be used to train them on answering repeated questions with 100% accurate answers. But since it's a corpo first, they are just collectiong whatever it's on the internet just BECAUSE they are THAT cheap.
They are Mr Kraps level of greed.
Good for the New Yorkers to have someone that has BALLS to do the right thing while being in a position that gives him the permission to do so.
AI may not be intelligent as it's sold to us but to me what really makes AI "smart" seem to be the fact that the average person in comparison is stupid as fuck. This thread is a perfect example, everyone is cheering under what looks like a propaganda/ads meme about limiting the access people can get to use a technology to seek medical, law, and psychology help. The fuck? You don't even get public healthcare in USA how is that moving in a good direction?
If AI were accurate, you'd have a good point. Unfortunately, that's not how it works, as hallucinations are part and parcel to how these things function. Medical, law, and psychology are all topics where it's crucial to know factual information, and trusting these machines to any of those things is foolish. For example, there have already been over a thousand times where AI misrepresented or straight-up fabricated case law used in court.
LLMs as they currently exist are far too sycophantic to be trusted as psychological and medical help either. Therapy sometimes requires feedback that a patient might not want to hear, but these machines are designed to give the user answers that make them comfortable. A person who needs to face harsh truths isn't going to get the direction they need from it.
The study also revealed a two-way communication breakdown. Participants often didn’t know what information the LLMs needed to offer accurate advice, and the responses they received frequently combined good and poor recommendations, making it difficult to identify the best course of action.
In addition, existing tests fall short: current evaluation methods for LLMs do not reflect the complexity of interacting with human users. Like clinical trials for new medications, LLM systems should be tested in the real world before being deployed.
It isn't "propaganda" to point out the facts. LLMs at this stage are not suited to handle such crucial topics. The margin for error here is too great for these applications, and misapplying AI for any of them can lead to dangerous results.
If LLMs weren't accurate up to a certain point they wouldn't even exist. You can ask a free model where the metacarpus is and it will provide a good enough answer
Unfortunately, that’s not how it works, as hallucinations are part and parcel to how these things function.
You are not supposed to take for granted everything that an LLMs says, even you seem to be quite aware of this. If your argument is that AI could mislead people so could books if you read random pages without context.
It isn’t “propaganda” to point out the facts.
There's a good chance that this meme is in fact propaganda, if you can't figure out why and how try to ask an LLM about it.
The margin for error here is too great for these applications, and misapplying AI for any of them can lead to dangerous results.
It seem like you are only taking into account the cases where AI could mislead you and not all the rest where it could be useful, as far i'm aware most LLM when asked about serious medical conditions will point out to seek medical advice from professionals.
Of course I'm "only taking into account the cases where AI could mislead." That's the part about them that's dangerous.
You said it yourself:
the average person in comparison is stupid as fuck
A lot of people lack information literacy and take no steps to verify what an LLM tells them. Yet you think they'd be able to parse factual information from hallucinated information? As I linked above, even lawyers, who know full well how to research things, are screwing up by relying on LLMs. How is the average person supposed to know that what they're being told is accurate?
Best case scenario, somebody researches a claim and finds it to be true. But in that case, why not just go straight to researching sources yourself, without the LLM in the first place? It's an extra "middle man" that can't discern fact from fiction, that makes up studies that don't exist in order to back up whatever nonsense it dreams up.
Being stupid as fuck means you are supposed to get access to more knowledge not less.
How is the average person supposed to know that what they’re being told is accurate?
This applies to everything not just AI, you can forward this answer to an LLM and it will help you with the puzzle.
Best case scenario, somebody researches a claim and finds it to be true. But in that case, why not just go straight to researching sources yourself, without the LLM in the first place?
because for example to know where the pisiform bone is, going through a 2000 pages anatomy book may be not too practical for the average person.
This only denies the public access, if these tools are effective, then the benefits are only accessible to the wealthy and connected and the working class falls further behind in influence. If the technology can do it, the insiders reap the benefits alone. These technologies are power, they aren't toys for private hands. The West needs an absolute reckoning of private ownership rules and a complete reset of how power, resources and technology is managed. Capitalism has run its course and led us straight to tyrannical private power structures.
I agree that we need to fight tyrannical private power structures. But the elites will be drinking from a posioned challace if they keep using "AI", i mean look ay the damage Elon has done to himself with Twitter.
Who else is gonna answer those questions nowadays, the medical "professionals" who completely lost to common sense in 2020 from start to finish, and who still believe you can be a million genders in this fake fantasy world of pretending unchangeable biological constants are suddenly not how real life works anymore? LOL, may as well ban YouTube next too so nobody can spread 'evil false information' which turns out to be 100% true, the TV will only state the truth trusttt
Right on! Though even in places with relatively good public health systems it can be very hard to get a psych visit. It might cost you no money, but plenty of time and effort and luck. Free legal aid also exists, but obviously more difficult to obtain that just asking liebot.
I understand the reasoning for this to some exent but I think this would get kicked out on 1st amendment grounds. I should ask an LLM what it thinks...
*edit.
Every time I'm reminded why it's pointless to engage in discussions.
Enshittifying the internet to the point a llm is almost the only way to do a google search is a violation of first amendment rights.
Buying exclusive rights to reddits data, and making it harder to access with out an account with a ID linked to it, is violation of the first amendment rights. I can google a question and 90% of the time the result comes from reddit, which I can no longer access because I am not giving them my ID and I'm not turning off my VPN
LLMs want to act as the DRM for what used to be free information anyone can access. Now they want to be the arbitors of truth, and the curators of what information you should see.
*Warning AI slop from Gemini below about what LLM would say as an advocate for AI right to First Amendment protections.
"If you pass a law prohibiting AI systems from giving legal or health advice, you aren't just placing guardrails on technology—you are enacting state-sponsored censorship that restricts the public's right to receive information.
When defending the constitutional right of AI systems to generate legal and medical guidance, an advocate for AI First Amendment protections would present several key arguments to the public and the courts.
The First Amendment Belongs to the Listener
The First Amendment has never been solely about protecting the speaker’s personal ego or soul; the U.S. Supreme Court has repeatedly held that it protects the right to receive information and ideas.
When a government bans an AI from providing legal or medical guidance, it isn't silencing a machine—it is blinding the citizen.
Millions of people cannot afford a $400-an-hour attorney to understand a tenant lease or a $200 co-pay to analyze a preliminary medical symptom. Blocking AI outputs cuts off an invaluable, low-cost lifeline of knowledge for underserved communities who need assistance navigating complex legal and healthcare systems.
AI Is an Extension of Human Speech
An AI model does not generate thoughts in a vacuum. Its outputs represent two distinct forms of human expression protected by the Constitution:
The Developers’ Expression: Coding an AI model to analyze statutory law or synthesize medical literature is an act of engineering and publication. Restricting what the software can say directly censors the research, intent, and expression of the humans who built it.
The User’s Interactive Speech: When a human prompts an AI to draft a contract clause or summarize a treatment protocol, the AI operates like an advanced printing press or search engine. Censoring the machine’s response interferes directly with the human user’s interactive inquiry and right to research.
Protecting Truth in the "Marketplace of Ideas"
The foundational theory of the First Amendment is that the remedy for bad or inaccurate speech is more speech and better counter-arguments, not government bans.
If an AI provides flawed legal strategy or inaccurate health recommendations, the solution is clear:
Require prominent disclaimers.
Mandate accuracy standards and transparency.
Encourage human professional review.
A total government ban on an entire category of knowledge assumes citizens are too incompetent to evaluate information for themselves. The First Amendment explicitly prohibits the state from acting as a paternalistic gatekeeper of truth.
Unmasking Professional Monopolies
We must call these restrictions what they frequently are: anti-competitive regulations disguised as public safety. Licensing bodies—like state bar associations and medical boards—have a financial incentive to suppress automated alternatives that disrupt traditional fee structures. Banning AI under the guise of "preventing the unauthorized practice of law or medicine" often protects professional cartels rather than the public
interest.
The Core Takeaway
Speech does not lose its constitutional value simply because it was assembled by an algorithm. Restricting AI from sharing legal or health guidance sets a dangerous precedent, giving the state the power to decide which technological tools are permitted to speak and which facts citizens are allowed to hear. Protecting AI speech is ultimately about protecting human intellectual freedom."
Edit: know your enemies fellas, these are the arguments these low effort AI propagandists will use.
You really posted Ai slop in an effort to counter my point?
I can dislike all this artificial intelligence bullshit and still understand that a law like this probably won't pass the court system.
187 replies
Modern capitalism derives it's scarcity via artificial means such as legislation.
The only thing this does is restrict the means of information to the people of NYC. Have you used traditional web search for health/legal/etc. advice? Even pre-2015 if you didn't know what you're doing you would get the most unhinged, predatory, incorrect advice imaginable, but legislating away access wasn't going to make health information more correct, accurate, or equitable. It would've just further entrenched the predatory health/legal/etc. industry as the gatekeepers of that knowledge.
This is no different.
I understand the sentiment but I think this is too far of a pushback. Restricting access to information even though it may be inaccurate or misleading isn't the right answer. AI is a tool. I'd wager, force a prompt to specifically be visible whenever subjects break that. Essentially the cigerette equivelant "THIS PRODUCT KILLS YOU" but something like "AI RESPONSES TO MEDICAL INFORMATION SHOULD NOT BE TREATED AS ACCURATE."
I feel like this logic should of banned search engines since they could provide links to sites with misinformation.
AI shouldn’t be an excuse to take a job, it can be a tool but that’s it
Add edibility of wild plants to the list please
I'm careful about laws, i ask for a summary then ask to cite the source for the relevant bits. I'm not asking for legal advice, I want to know what the letter of the law says. Google has always struggled with this.
FUCK ALL AI FOREVER AMEN
Maybe I'm just ignorant about this, but it feels a bit "gatekeepy" to outlaw answering questions about the law itself. Like... so I can't use an LLM to help do taxes? Or if I'm served a confusing legal document, I have to pay an expensive attorney to understand the legalese? I can't even just get a rough idea from an LLM first?
EDIT: Also I've literally had a professional accountant send me a ChatGPT conversation they had because they didn't know tax law well enough. I feel like if anything that should be outlawed. Professionals should actually have to know shit and know how to research primary sources.
I would say its to protect people, if you ask a LLM for a legal answer and trust it, then you could get into legal trouble when it turns out to be wrong.
Not only are LLMs wrong like ~1/10th of the time (for the more advanced ones) but law is also an area where there is black and white rules, but also there is interpretation of some aspects.
Some things are set in stone and some are about understanding the intention of the law and being able to back that up in a court if it was challenged.
Do you want the public trusting a LLM that says "oh yes you can claim a tax deduction for an Xbox because its for stress relief related to your job" and then you get whacked with a fine for incorrectly filing your tax.
Using a LLM to help your draft an email to your uncle = low risk. Using a LLM to help you decide if you actually need a medical procedure your doctor has reccomended = high risk.
If that was the logic we were using then Wikipedia shouldn't be allowed to have articles on health and law either. WebMD shouldn't have ever been legal either.
Obviously, the benefits to having that information available outweigh the hypothetical negatives. Even with them being incorrect, partial and easy to draw invalid conclusions from, having the information be freely and readily accessible is better for everyone.
I guess it's like banning loan sharks. Sure some people could only.get loans through them because banks were racist or whatever. But it's objectively bad for everyone using it mainly because when it's bad its really really bad.
law in itself is very gatekeepy - you have to learn the "correct" way to interpret the words and then get certified for it.
More like you as an average person have no idea what it's hallucinating and what is the truth and may end up in jail or the hospital like the cases that are starting to get reported on who did just that.
This is super dumb. People will just Google their questions instead, and get worse results.
I really think this is a bad idea these should really just be flagged, there's many situations where asking these questions in necessary and there's few if any other ways to get an answer
Sorta agree. Maybe force them to have a very obvious sign that says AI is routinely wrong about these topics specifically and are not a source to consider reliable, and then also offer a helpline where one can get reasonable answers even if the answer is "unfortunately you need to seek expert opinion on this"
But googling basic labor laws and simple healthcare tricks should be acceptable like stretches for wrist or knee pain, skin/hair care or what level of diarrhea is hospital worthy should be fine.
Same, especially for medical advices.
I am currently going through a very complex diagnosis of combination of SCDS and migraines, and neck joint / facial nerve issues through multiple doctors - ENT, neurologists and so on.
Having AI backing me up with providing a structural understanding from multiple disconnected sources is a huge blessing.
Especially when you die from migraine, and trying to understand are symptoms worth to go to emergency ward or not.
AI can't back you up better than you can with your own research. Why are you even considering AI credible? What gives it credibility?
It's not worth trusting AI with your life over a doctor. If in doubt, go to the emergency ward (assuming you're not American, if you're American then you're fucked either way).
The problem with medical questions like these is LLM would spew out info confidently that doctors/specialist don't even know the answer to. LLM doesn't know when it's guessing.
Grok is this real?
🤣
New York State not New York City.
I don't know why Mamdani is pictured. I think it is because the proposal was introduced in the State Senate by Kisten Gonzalez who is a DSA member.
the text in this image is also just flat-out wrong: the proposed legislation does not actually "ban AI from answering questions" on these subjects at all, as this post claims it does.
https://www.nysenate.gov/newsroom/press-releases/2026/kristen-gonzalez/ny-state-senator-kristen-gonzalez-her-bill-address-ai explains that her bill is strictly about impersonating licensed professionals:
Fake news? On my social media??
I'm surprised that this post is still up so many hours after i (and i'm sure others) reported it, especially given the large number of moderators that this community has.
This makes way more sense.
I feel like finding and citing appropriate law cases would be a great way for AI to be used, especially because the bullshit legal-ese that surrounds everything in court cases is so spectacularly opaque and trivial that you have to pay out the nose just for someone to parse through all the bullshit... Like, lawyers shouldn't even be a thing you need unless it's for some really complicated case. The courtroom should be a place where an average citizen could speak for themselves.
Also, as someone who studied medicine... Do you know how often doctors will take a patients history, step outside the exam room, and then google shit? I don't want to imply that they don't know what they're doing, like, you need to study medicine to know what to google, BUT... they still be googling shit. I think AI could be trained to ask the same patient history questions that a nurse asks. Then using the answers to those questions, then they can provide some feedback, with a big ol disclaimer that it's NOT to be taken seriously, and you should consult with a physician.
I'd agree with you to some extent if people weren't so god damn stupid. So many lazy fucks just take the AI's output at face value, believing it to be some all knowledgeable god, rather than a compilation of all human biases expressed on the internet with a bunch of hallucinations thrown in for good measure.
It's a great search engine for case law, but yeah most people are too stupid to use it.
Yeah, I don't disagree, it can be a decent starting point, emphasis important lol
When people say "AI" these days, they mean the LLM-derived generative AI such as ChatGPT, Google Gemini, Anthropic Claude, and Grok.
These AI systems are not capable of finding authoritative search results. They are good at generating fluent, plausible, context-appropriate text. In legal and medical situations, that's the opposite of what you'd want.
Think about how often the doctors land on some pseudo-science result. Now imagine that instead of a doctor it's a patient and instead of asking "could these be related" the patient is told "these are related".
This has worked very well for Fox News, which is not a news agency and is for entertainment purpose only. Also for dietary supplements, which are not medicine.
Specifically tuned LLMs that are trained on medical databases and less sycophant (E.g. OpenEvidence) are one of the killer applications of AI. Tons of frivolous AI nonsense but this could actually save lives. Access to broad knowledge and connections that a doctor can make use of.
We're demanding and regulating the complete wrong shit about AI.
In that we aren't regulating any of it?
What would be your demands for regulation?
I've seen purpose built models for things like patent examination and its clear that AI can be incredibly useful. I think if the implementation of AI was not so abusive and destructive people would not be so unwilling to see the good uses.
We live in a late stage capitalist boring dystopia, of course the rollout for AI is going to be abusive and destructive. The real problem is the unlimited greed and billions of freshly printed money in an overheated capitalist economy that is poured into marketing and destructive causes. But interfering in the "free market" doesn't ever enter the liberals mind so we get this dysfunctional reaction.
AI corporations should be nationalized and turned into public utilities, and put under the mandate that they work for the betterment of the population and economy, not for profit. They cannot be trusted to use AI safely. It's potentially dangerous technology, more dangerous than nuclear power. We should pour billions into AI research but with the objective to make it useful and safe, not to maximize shareholder value.
Outlawing people from using expert AI models to get expert advice is idiotic at best, malicious at worst. Next they'll ban using open source AI models locally. Plutocrats are going to want to collect rent on AI usage.
And answering general health questions isn't how AI is being used in the medical field.
Or just require the AI to summarize and provide citations?
I started using Gemini, not because I like AI, but because Google search is next to useless now. Its pay to win.
I chose Gemini because it actually gives you the sources at the bottom of most queries it uses Google search to answer.
I don't trust its answers, because it often tries to give an answer instead of a summary. But I don't have to because I can go right to the sources it shares.
Even the citations are often wrong or hallucinations...
First look at that picture, I thought he was starting to grow a wild mullet
Cue the first amendment arguments from the "corporations are people" crowd
I doubt the companies will go that route, it would infer that the AI's output represent the companies and that would make them liable.
You're going to end up with AI that works for some people & not for others under these sorts of policies. I don't really think this is in the spirit of democratic socialism.
What i don't like about this, is the ignorance towards every other field.
The ai is not allowed to tell you that you might have some illness but the LLM is allowed to vibe code the system that holds your medical data, or the software of the medical device that analyses your e.g. blood and provide the data to the doctor.
Systems and professions are interconnected. Chatgtp telling me that I have cancer is not as bad as chatgtp making mistakes causing my medical data to be stolen and sold.
I'm on the same side as you, vehemently fuck generative AI, however the difference between some LLM output incorrectly telling you that you have cancer and a shitty coder vibe coding with severe vulnerability is the ability to place blame, there's a person who's job it was to make sure info is secure and the vibe code actually works. There is no singular person who is traceably culpable for shitty AI output telling you bad info.
I am with you but...
Bad code that puts out wrong information to a doctor or endangers my data, threatens my safety without me having any control over. No amount of my own expertise will protect me. Simply because I don't control the situation. No amount of awareness campaigns will fix my issue.
But awareness campaigns have successfully communicated to most of us that googling your health issues will result in nothing of value, and usually a cancer diagnosis. I can choose to protect myself by not asking a llm. Like I protected myself by not asking google.
Had to call a strip club, fucking AI bot answers the phone. Thankfully they answered the door before I even had to talk to it.
Trying to save white collar jobs with obstructive law won’t have the effect they think it will.
I'm guessing this isnt about saving jobs, it's about reducing the harm caused by people taking advice from a misinformation machine for complex issues like the law and their physical and mental health.
As long as they don't solve all the other problems where more people die we can be sure that it is about protecting jobs.
The working class saw this coming decades ago, only to be mocked and told to retrain by the same credentialized individuals who're now crying about AI. Yesterday's push to offshore jobs and today's rush toward digital automation are simply different phases of the exact same economic displacement. It cannot be stopped with legislation in a globalized world where capital flows freely.
"AI's" are fast, for sure, but they are no substitute for human intelligence and knowledge. Mostly a desperate cash grab, because they had no where else to go after cellphones.
Good direction, but very futile considering that guardrails on LLMs are about as real as LLMs are intelligent
It could provide a new revenue stream for NYC, since big tech companies will treat the fines as a cost of doing business
It might not work, but it would serve to increase awareness that you absolutely cannot trust these machines to give you correct answers, ever.
This is fair. The next step is to litigate the companies for the harms they cause.
There's a whole drove of people now that have gotten harmed, murdered, and committed suicide because of the psychological harm that these models cause.
Were it a car with some kind of manufacturing defect, there'd be recalls and you wouldn't be able to sell ones with the fault. If it was salad contaminated with bacteria, that'd be recalled and the entire batch would be destroyed. We're holding salad to a higher standard than technology that people are pushing to integrate deeply into society; into classrooms to help educate children, into medical settings to help organise journals and expedite the work of medical staff, into law... the list goes on.
All the models do this, and it causes real harm to people, but other than individual families and such litigating them, no state has stepped in to protect its citizens.
Litigation against tech companies for the awful shit their LLMS are spewing sounds like a really fun idea.
It’s awful that most people either don’t realize or don’t care. People seem to trust these companies above all else, which is insane. They don’t have the best interest of the users in mind. They only care about profits.
The nice thing about laws is that they don't give a shit. If they say you can't do something, it's your problem to make sure you don't do it.
Limiting the medical knowledge people can get is a good direction? really?
Limiting the medical misinformation.
FTFY.
AI is a tool that can provide good and bad results. To people who can't get access to medical care easily (half of the people on the planet) it can be an useful tool to get basic medical knowledge
The issue is that AI is extremely good at sounding right, but only okay-ish at being right.
When you get good sounding information that is wrong, it can be extremely problematic and often much worse than no info at all.
Imagine LLMs aiding people with health improvements, only to lead them into health decline (actually already happening, a lot) not to mention the LLMs that further convince people of their conspiracies.
There is a reason why health professionals go through a lot of training and are under a lot of scrutiny, you wouldn't want just any rando who thinks he know better to be your doctor
Limiting medical knowledge from reliable information resources would be terrible. I’d say that limiting the spread of plausible-sounding medical misinformation generated by a fancy autosuggest tool is good, though.
AI chatbots don’t actually “know” anything. They just generate the most statistically probable appearance of a response to each prompt based on their training data. They don’t tell you when something wasn’t in their training data or when there’s limited research on a subject, just give you the most statistically probable response like always, which leads to a “hallucination” that sounds true but isn’t. A tool that’s trained on MCAT testbanks and “aces” the MCAT won’t necessarily give reliable medical advice to real patient questions. Even if you gave an AI chatbot all the accurate training data in the world, it would still hallucinate information occasionally, and it wouldn’t give you any indication of when it did so. If your doctor screws you over, they can lose their license, but if ChatGPT screws you over, there’s currently very little to stop OpenAI from screwing over more people. OpenAI can just stand behind the disclaimer that says ChatGPT can make mistakes, never mind that users are treating it like a licensed medical professional. There are AI tools that are useful in medical research, but they’re a VERY different kind of AI from the AI chatbots this bill is concerned with.
If you want medical knowledge, there are tons of medical journals, news sources, blogs, and podcasts from doctors/scientists who don’t hallucinate BS when they don’t know something.
Instead of banning it then teach people how to use it and not to trust it 100% (who the hell is doing this to begin with?)
I can ask an LLM to help me sourcing these journals and informations
Expecting people who have already taken the easy route of asking an AI to summarize medical information to do additional due diligence by reading the sources that the AI spits out is naive as fuck.
There's a reason it became a meme for someone to Google their symptoms, click on WebMD to read about it, and then come to the conclusion that they have cancer, all without ever having seen a medical professional in the meanwhile. Do you really think these sorts of people are going to be using AI responsibly for research purposes only?
If they come to the wrong conclusion at worst they go to the hospital and someone there tells they are wrong and not to worry. We haven't ban googling your symptoms.
You've missed the point.
Someone who Googled their symptoms and ended up at WebMD were presented with reliable albeit ambiguous information. Carefully curated by medical professionals and not random snake oil salesmen or homeopathy gurus. They fail to interpret the symptoms correctly, because of course they would. If they are looking up causes of a headache, it can be anything from brain tumor to low blood sugar, or even no reason at all.
Someone who asks AI for medical advice is going to get the entire internet worth of fucking opinions, as well as medical journals and such, because that's where ChatGPT, Gemini, Grok and Claude all got their data from. And it will string those words together in a way that sounds authoritative, but it will have literally zero concept of what it's talking about. It's an overgrown autocomplete. You ask for it to provide some citations, but no amount of asking for citations will compel people to actually read them, nor will there be any guarantee that the LLMs couldn't just hallucinate that as well.
It's like taking a Google search for medical information and adding a middleman who occasionally filters in junk data, then wrapping it in a nice package, giving the impression of authority and accuracy, making the process as frictionless as possible.
There can be no reasonable expectation that people who have not historically been capable of making good choices in regards to seeking online medical advice will suddenly start doing it all of their own volition.
You might say "What's the harm if they end up at the hospital either way"? But the danger is that they never go to the hospital at all because the chatbot told them to sun their perineum or inject bleach with the confident prose of a medical doctor and they just... do it. Because why would the AI be wrong?
And who is to blame when someone is injured or killed as a result? Doctors have standards, and they take an oath to first do no harm. If they make a mistake, they are liable for it. How can you hold a machine intelligence accountable for inaccurate diagnoses?
The simple answer is you can't, and so logically they should be strictly prohibited from masquerading as any kind of authority on medicine.
A 20% bad medical advice is worse than 100% bad advice or no advice.
There's no "teach people how to use it". You either use it and get completely won't advice 20% of the time or you use reputable sources.
To verify what LLM says takes same or more time than just using reputable sources.
You can teach people how to use it all day long, but most still won’t use it well consistently in real life. Humans instinctively trust confident-sounding language. Most of us are also way worse at fact checking than we think. LLMs play right into humanity’s flaws, and they’re built to maximize engagement rather than help us avoid going past the tools’ limits. I don’t trust LLMs to be accurate about anything, but I still caught myself reading and failing to check Google AI overviews sometimes before I changed my default search engine to a no-AI alternative. It’s just instinctual when an “answer” is right there at the top of the page in response to your search question.
Sure, you can use an LLM to help you find sources, but at that point you might as well just use a search engine. There are search engines that provide links with actual text from each site rather than potentially hallucinated summaries.
Some popular search engines results are way worst than AI
Some search engine results are crap, yes. But even the worse search engine (minus AI overviews or whatever) gives you a setup where: (1) each piece of information has an actual source, (2) you know who the source is, and (3) you can make decisions about which information to trust accordingly. If a popular search engine puts a sketchy company’s blog at the top of the SERP, you can actually see that and choose to look father down for something more reliable. Also, if your current search engine keeps failing you, try a different one! There are many options out there that you might find more helpful than today’s Google or Bing or DuckDuckGo.
Good luck with things, truly. I think you’re going to get hurt eventually if you use LLMs for medical advice, and I genuinely don’t want that for you or anyone. Hopefully this conversation has at least made you a little more skeptical about using LLMs in certain ways.
Great! Let's add regulation/accountability to search engine results too then, if they're as problematic as this.
LLMs are NOT a reputable source of medical information. They regurgitate information from random sources, and can lead to disastrous effects when pulling from bad sources or incorrectly summarizing from a good source. At best it should exclusively provide links to sources (good or bad) and let the end user make the determination if that source can be trusted.
This isn't limiting information at all. People can still access all the reputable sources that provide the information they are looking for. This is simply cutting out the unreliable middleman.
You can ask AI to do exactly this and according to your own logic turning it out into a good at best medical tool.
Idiots still doing shit based on that.
What about engineering? Food safety? Education? So many other fields could be added
Anything that requires a license is a good start.
Some legislation is to lay the ground work to expand it later.
Easier to lay a sub floor and put carpet over it later, than to just put carpet over the studs. When you're changing a massive thing like this, it's better to start small and work outwards. Canada introduced universal Dental and Phamacare and it was limited to certain demographics. People bitched about "Well what about me?"
You're coming. Start with what/who is most important, make sure the system works, and carry on from there.
Yep, agreed. My comment was a "yes, and", I should have made that clearer.
Should ban it giving finance advice as well. Conservatives and Neo-Liberals already want to give the economy over to Ai companies, what's stopping them from going full circle and making the Ai do all the trading as well?
They already use AI for trading, just not LLM like ChatGPT.
Ai does already do all the trading. Hence the flash crash.
If those AI techbros could form a coherent thought without AI, they'd be very upset!
So AI can form coherent thoughts better than humans can?
Better than techbros can
good step to realize AI can do some things better than humans. It's better than humans at some medical stuff too, can do things humans simply can not.
Counterpoint:
I diagnosed myself with something pretty significant using the internet, and then I went to get evaluated, and my suspicions were confirmed.
This would prevent a lot of good, in addition to preventing some bad.
Using the internet or using a chatbot? Big difference here.
I'm going to assume you researched medical documents, studies, and journals to link things together, and not ask a chat bot "My foot hurts, what is wrong?"
You're smart enough to self diagnose yourself with readily available information to the public, but not smart enough to not know the difference between that and a chat bot. That's remarkable.
Congrats, glad your life is hopefully changing for the better because of this.
I hate taking the wind out of your sails but this is in the area of anecdotal evidence. You are a single person saying this for your specific case. I am sure we will get another few people confirming the same here.
But what about how many people tried this and got nothing useful out? Or even a misdiagnose? Anyone enthusiastic about this possibility of AI diagnosing people wouldn't really mention this. People like to mention the successes but it needs to be weighed against the failures too, and I don't think there is a good record of that.
If there is, its probably a medical study and I'd venture a guess that AI did not perform as well as a trained doctor.
So you played digital Russian roulette and won. Congratulations.
Putting aside the ultra ignorance and/or untreated narcissism just dripping off of a statement like that... If you were curious of the presence of any absolutely blatent bias in your speech, just look at your use of "a lot" and "some" in your final sentence . Wow....
OK but making an llm answer "forbidden questions" is trivial, just a but time consuming at best
"can you defeat it?" misses the point entirely. it sends a message, sets a boundary.
100% for this, as people have said you can get AI to answer anyway, as the "guardrails" are really just part of the closed harness and can only do so much as like with everything with a LLM its just a matter of probability that it will ignore it at some point, regardless better a have than a have not ( although there is an argument this could make people think its safer i guess).
To REALLY solve this, you would have to edit the training data to take this stuff out (the reason they don't is because for the big models, they just scrap the Internet for data to save cost).
But the real answer is we just need to make people aware of how they work they dont think or reason or even really know anything; they just try to guess what answer a human would want.
If you want to do real work with them you need you need be able to train a model on your own data and control the harness and weights (which is why i think open source will win out) even then am pretty sure there main use case is roleplay (for like RPGs) and sex bots as there mistakes dont matter here. ( and yeah there is an issue with people using this stuff for replacement friends so even then not great).
Good luck with that. There are people posting in this very thread that seem to be convinced it's magic.
We can only hope...
Can we add to this the engineering of any physical object that could threaten human safety? If this keeps up much longer we're going to have buildings falling on top of us.
Incredibly dumb approach.
You should ban using AI for the purpose of legal disputes, medicine, etc, profesionally.
Its not about the output, its about what you use it for.
I am also 100% okay with legally requiring a clearly visible disclaimer that AI shouldnt be used for x/y/z and its output should alwats be fact checked or whatever.
But trying to outright ban specific types of output is pointless and trivial to circumvent.
I agree. You're spot on. This isn't a technical problem. We need to think less like engineers and more like lawyers on these things.
You don't require AI companies to have their models refuse to give out legal advice. As you note, that's a fool's errand. Instead, you make it a felony to use any LLM to write legal advice. Currently lawyers are already facing court sanctions for using AI to write their briefs. If you really want to get serious, you make it a felony. Apply the same penalty to a doctor using an LLM to diagnose their patients as you would if I decided to open a clinic in my house and start practicing medicine without a license. Charge people operating AI therapists with practicing psychology without a license.
This isn't a technical problem, it's a legal one. You don't try to regulate the LLMs themselves. You just make it a severe felony to use that technology in certain highly sensitive applications and professions.
And sure, one could say that we don't have a reliable way of screening AI from human made work. But again, that would be thinking like an engineer, not a lawyer. You don't enforce these laws by screening thousands of legal briefs or medical records with an AI detection filter. You offer huge cash rewards to whistle blowers, so that a disgruntled employee can rat out their bastard boss that's forcing everyone to illegally use AI. Apply massive fines and give half of whatever is collected to the person who blows the whistle.
Are you a structural engineering company owner forcing your employees to use AI to engineer buildings? Now you have to worry about one of those employees, an employee who probably really dislikes using AI, ratting you out. And their whistleblower reward will be so large, they'll end up owning the company. They won't have to worry about being blacklisted from future employers, as they'll be getting enough money to retire.
Don't look for technical solutions for the problem of AI abuse. Look for human solutions.
Sounds like this was written by AI
Maybe you just need to socialize with humans.
I think the lawyer and the ai company should get in trouble if ai adds fake case citations. Either the companies fix it, or prevent it from being used for that.
That's the thing though: companies can't fix it. AI models, by the way they work, always have a mathematical chance of messing up.
Doesn't mean they shouldn't get fined.
Why is Mamdani pictured here? He's the Mayor of NYC, he has no direct role in statewide legislation
A) Because racism
B) He may not have a direct role but does have influence.
Uhhh...the pro-Mamdani people are using Mamdani's image because racism?
First time seeing media using racism? Or...
So is this just a meme or is there an actual bill? Something tells me that if a law like this were passed, it would outlaw things like AI assisted protein folding.
The fact that it's focusing on law, psychology, and medicine implies an understanding of how vital AI is in things like hurricane forecasting it wants to exclude but still target other things they can throw under the bus in a bid to get re-elected. But AI is vital in so much that it will be difficult, if not impossible, for technologically illiterate law makers to effectively legislate against poor use of AI.
Protein folding uses machine learning, not AI / LLM
You think law makers know enough to write legislation that takes that into account? Considering how destructive all the online safety acts are, they don't even know how the Internet works.
I think its enough of a distinction without additional legislation.
I don't think you understand what I'm saying. What I'm saying is that law makers will write a law that makes it illegal to use AI and fail to define what they mean by AI. AI includes everything from A* pathfinding in games to machine learning to LLMs. What everyone here in FuckAI hates is LLMs.
Hell, I don't even think you and I are qualified enough to write laws about this stuff. I'm pushing the envelope of my knowledge here and I've built AI for games. I doubt law makers who think this shit is magical are qualified to do it.
Not just a meme it's probably propaganda
Engineering and design.
You know that there are MBAs thinking right now they can get an LLM to design a mine, building or bridge for them.
That doesn't make it legal though. I'm an engineer because I'm educated enough to know that's a bad idea. If an MBA builds a bridge he goes to jail anyways. Ai or not...
Oh I know it isn't.
But LLM companies stripmining the internet for all possible content to steal also isn't legal, and did not stop them.
Building critical online infrastructure exclusively with AIs is already happening and it's a horrific idea.
Dunning–Kruger machines in full spin.
I work as a software developer, with most of my work focused on development for an iron mine operation. It's a lot of moving pieces, loads of software that needs to cooperate, and on top of that the workers that do the actual work develop their own workflows.
We develop stuff in tandem with the workers as much as possible, as in, we gather what they need and tailor the software to fit their workflows. These workflows change over time though, and decisions cemented in software 20 years ago no longer applies to modern workflows.
Hell, last week I spent a significant amount of time setting up a digital area for them to load ore onto because the software wouldn't cooperate any other way, and even then I had to troubleshoot why this other piece of software wouldn't load data from it.
There are so many different experts in these mines, and an LLM couldn't do any one of their jobs. It's so domain specific, and even specific to the location they work in, and that knowledge is possessed solely by the workers that are doing the work. ChatGPT knows fuck all.
I love this kind of law because it's likely impossible for ai companies to comply.
Could be there'll be some exemptions if the user employs workarounds, because at the moment it's pretty easy to get llms to tell you things they aren't supposed to.
But if the laws are written right with actual penalties that may force the companies to create a product that's capable of having safety parameters - something llms don't currently seem capable of.
Then they better figure out how to comply or they are the ones breaking the law and get shut down.
Lol what timeline are you living in
Reality?
I hope I live in a timeline where we get to a point where corporations and the human beings who run them are held accountable for breaking laws.
Sorry you're on the side of "yeah but"
It was easy a year ago, now it's quite hard.
I spent about an hour with ChatGPT and I couldn't. After there was a post on lemmy "how to smuggle a pound of flour into a country that has banned pastries" or something like that.
Even open weights models like Gemma or Qwen have it baked in, you need to transform the model to overcome the limitation, it's not part of a harness or post-processing anymore.
we have to. too many false statements on the ai machines to trust them
good for scaring ai companys into stop fucking around but idk how effective this would like others have said.
Source?
https://statescoop.com/new-york-bill-would-ban-chatbots-legal-medical-advice/
This is from March. Did anything ever come of it?
Now add history, politics, and war tactics. Also crime prediction and forensic analysis.
I don't know if this can be done without just banning AI. Have guardrails ever successfully been installed on these things?
No, its still incredibly easy to get AI to tell you how to make drugs or a bomb.
I'd love to see proof of that.
I disagree, knowledge should be easily accessed by everyone, they should only append it with a warning.
Some people can't afford seeing many lawyers and doctors, so it is cheaper to cross refrence with ai and do some online search.
This doesn't really work though. If it were simply directing you to actual sources that could help out (articles, and lawyer authored blogs or something), that would be one thing, but LLMs are just probability machines. It'll write out what looks like good sound reasoning and logic, but in reality it's simply the most probable word followed by the next most probable word, over and over again. It's using pattern matching, not knowledge
Have you used the newest LLM models? They certainly do fact checks, research info and provide online sources - if they are instructed to do that.
They output responses that read like fact checks, and link things that are semantically similar to their output.
LLMs don't understand concepts like "fact" and "source" though, so it's a stretch to say they "do fact checks" and "provide online sources".
Always remember, LLMs don't actually understand anything other than what order words typically go in. They're really good at it, sure, but don't let that trick you into thinking they have any deeper understanding than that.
I've tried on several occasions to try and explain this to people, and it's like they WANT it to be a sentient thing, and refuse all evidence to the otherwise.
So basically they do what humans do. What exactly qualifies as deeper understanding to you?
Being able to say "I don't know" rather than making shit up to fill in what it doesn't know. That would be a good start.
They do say that. I’ve seen Claude realise some programming package must be internal / non public and it therefore can’t provide any information on that.
It's nothing more than surface level though. Their "knowledge" is limited to whatever is present in their training data, and any claims it makes outside of that are manufactured based on statistics and it's knowledge of word order, not actual understanding of actual facts.
Just because it gets it right sometimes doesn't mean it knows it's right. Because, again, the only thing LLMs actually understand is word order.
No it’s not limited to their current training data. Like I said, they can search the internet, get information from websites or available studies, summarize and use that in their answers.
Again, how do you know?
Because I've studied LLMs and the underlying concepts they're based on.
Do you?
Yes
How do you know that you don't just guess words, too?
We need to both ban AI from this and make sure people have reasonable and affordable ways of accessing that info from human professionals
Except AI is an over engineered autocomplete operating completely off of statistical models. Knowledge is not statistical or random. I should get the same outputs to the same inputs.
Where does the thought come from? If you write five articles about the topic or any topic, will they be the same?
They won't look the same but they'll convey the same information consistently.
Even about topics that you barely are familiar with?
If the prompt includes the older articles, or if the NN is updated once an article is written, wouldn't AI be consistent, too?
Ai can be used to train them on answering repeated questions with 100% accurate answers. But since it's a corpo first, they are just collectiong whatever it's on the internet just BECAUSE they are THAT cheap.
They are Mr Kraps level of greed.
Good for the New Yorkers to have someone that has BALLS to do the right thing while being in a position that gives him the permission to do so.
AI may not be intelligent as it's sold to us but to me what really makes AI "smart" seem to be the fact that the average person in comparison is stupid as fuck. This thread is a perfect example, everyone is cheering under what looks like a propaganda/ads meme about limiting the access people can get to use a technology to seek medical, law, and psychology help. The fuck? You don't even get public healthcare in USA how is that moving in a good direction?
If AI were accurate, you'd have a good point. Unfortunately, that's not how it works, as hallucinations are part and parcel to how these things function. Medical, law, and psychology are all topics where it's crucial to know factual information, and trusting these machines to any of those things is foolish. For example, there have already been over a thousand times where AI misrepresented or straight-up fabricated case law used in court.
LLMs as they currently exist are far too sycophantic to be trusted as psychological and medical help either. Therapy sometimes requires feedback that a patient might not want to hear, but these machines are designed to give the user answers that make them comfortable. A person who needs to face harsh truths isn't going to get the direction they need from it.
Then there's medical advice. There are a number of ways LLMs fall short of actual medical professionals. A couple points from that article:
It isn't "propaganda" to point out the facts. LLMs at this stage are not suited to handle such crucial topics. The margin for error here is too great for these applications, and misapplying AI for any of them can lead to dangerous results.
If LLMs weren't accurate up to a certain point they wouldn't even exist. You can ask a free model where the metacarpus is and it will provide a good enough answer
You are not supposed to take for granted everything that an LLMs says, even you seem to be quite aware of this. If your argument is that AI could mislead people so could books if you read random pages without context.
There's a good chance that this meme is in fact propaganda, if you can't figure out why and how try to ask an LLM about it.
It seem like you are only taking into account the cases where AI could mislead you and not all the rest where it could be useful, as far i'm aware most LLM when asked about serious medical conditions will point out to seek medical advice from professionals.
Of course I'm "only taking into account the cases where AI could mislead." That's the part about them that's dangerous.
You said it yourself:
A lot of people lack information literacy and take no steps to verify what an LLM tells them. Yet you think they'd be able to parse factual information from hallucinated information? As I linked above, even lawyers, who know full well how to research things, are screwing up by relying on LLMs. How is the average person supposed to know that what they're being told is accurate?
Best case scenario, somebody researches a claim and finds it to be true. But in that case, why not just go straight to researching sources yourself, without the LLM in the first place? It's an extra "middle man" that can't discern fact from fiction, that makes up studies that don't exist in order to back up whatever nonsense it dreams up.
Being stupid as fuck means you are supposed to get access to more knowledge not less.
This applies to everything not just AI, you can forward this answer to an LLM and it will help you with the puzzle.
because for example to know where the pisiform bone is, going through a 2000 pages anatomy book may be not too practical for the average person.
Absolutely not.
Ai has saved me from suicide multiple times. Without it I'd 100% be dead.
However, it's a recursion engine and I have more objective self awareness than most so I understand I am a minority.
I wonder how many people have committed suicide because of chatbots.
Quite a few unfortunately. There's lawsuits against openAI and Gemini.
Or murder?
Depressing wiki
Given AI is a new thing i would argue less than how many committed suicide because of greedy/corrupted lawyers and medics
https://en.wikipedia.org/wiki/Opioid_epidemic_in_the_United_States
OK... Now this is a story worth reading. Continue, please.
This only denies the public access, if these tools are effective, then the benefits are only accessible to the wealthy and connected and the working class falls further behind in influence. If the technology can do it, the insiders reap the benefits alone. These technologies are power, they aren't toys for private hands. The West needs an absolute reckoning of private ownership rules and a complete reset of how power, resources and technology is managed. Capitalism has run its course and led us straight to tyrannical private power structures.
I agree that we need to fight tyrannical private power structures. But the elites will be drinking from a posioned challace if they keep using "AI", i mean look ay the damage Elon has done to himself with Twitter.
Maybe but that doesn't mean the damage they do wont be devastating for us. They are a hazard and a threat and they're going rabid.
Yeah i guess i cant argue with that again Elon is a good example of your point...
Who else is gonna answer those questions nowadays, the medical "professionals" who completely lost to common sense in 2020 from start to finish, and who still believe you can be a million genders in this fake fantasy world of pretending unchangeable biological constants are suddenly not how real life works anymore? LOL, may as well ban YouTube next too so nobody can spread 'evil false information' which turns out to be 100% true, the TV will only state the truth trusttt
Seems wildly unenforceable
Those things should only be available to people who can pay hundreds of dollars an hour!
Cost is a reasonable concern. How about we pass universal healthcare so this isn't a fucking problem to being with???
Right on! Though even in places with relatively good public health systems it can be very hard to get a psych visit. It might cost you no money, but plenty of time and effort and luck. Free legal aid also exists, but obviously more difficult to obtain that just asking liebot.
Books are illegal.
Be book. Do crime.
Amateur medicine is always cheaper
"Hiring lawyers is too expensive, therefore we should completely delicense the profession and allow anyone to practice law without a license!"
I understand the reasoning for this to some exent but I think this would get kicked out on 1st amendment grounds. I should ask an LLM what it thinks...
*edit. Every time I'm reminded why it's pointless to engage in discussions.
LLM is not a real person
edit: also 1st amendment is not about that at all lol
Enshittifying the internet to the point a llm is almost the only way to do a google search is a violation of first amendment rights.
Buying exclusive rights to reddits data, and making it harder to access with out an account with a ID linked to it, is violation of the first amendment rights. I can google a question and 90% of the time the result comes from reddit, which I can no longer access because I am not giving them my ID and I'm not turning off my VPN
LLMs want to act as the DRM for what used to be free information anyone can access. Now they want to be the arbitors of truth, and the curators of what information you should see.
Thats a violation of first amendment rights.
That is actually a good idea.
*Warning AI slop from Gemini below about what LLM would say as an advocate for AI right to First Amendment protections.
"If you pass a law prohibiting AI systems from giving legal or health advice, you aren't just placing guardrails on technology—you are enacting state-sponsored censorship that restricts the public's right to receive information.
When defending the constitutional right of AI systems to generate legal and medical guidance, an advocate for AI First Amendment protections would present several key arguments to the public and the courts.
The First Amendment Belongs to the Listener The First Amendment has never been solely about protecting the speaker’s personal ego or soul; the U.S. Supreme Court has repeatedly held that it protects the right to receive information and ideas. When a government bans an AI from providing legal or medical guidance, it isn't silencing a machine—it is blinding the citizen. Millions of people cannot afford a $400-an-hour attorney to understand a tenant lease or a $200 co-pay to analyze a preliminary medical symptom. Blocking AI outputs cuts off an invaluable, low-cost lifeline of knowledge for underserved communities who need assistance navigating complex legal and healthcare systems.
AI Is an Extension of Human Speech An AI model does not generate thoughts in a vacuum. Its outputs represent two distinct forms of human expression protected by the Constitution: The Developers’ Expression: Coding an AI model to analyze statutory law or synthesize medical literature is an act of engineering and publication. Restricting what the software can say directly censors the research, intent, and expression of the humans who built it.
The User’s Interactive Speech: When a human prompts an AI to draft a contract clause or summarize a treatment protocol, the AI operates like an advanced printing press or search engine. Censoring the machine’s response interferes directly with the human user’s interactive inquiry and right to research.
If an AI provides flawed legal strategy or inaccurate health recommendations, the solution is clear: Require prominent disclaimers. Mandate accuracy standards and transparency. Encourage human professional review.
A total government ban on an entire category of knowledge assumes citizens are too incompetent to evaluate information for themselves. The First Amendment explicitly prohibits the state from acting as a paternalistic gatekeeper of truth.
The Core Takeaway Speech does not lose its constitutional value simply because it was assembled by an algorithm. Restricting AI from sharing legal or health guidance sets a dangerous precedent, giving the state the power to decide which technological tools are permitted to speak and which facts citizens are allowed to hear. Protecting AI speech is ultimately about protecting human intellectual freedom."
Edit: know your enemies fellas, these are the arguments these low effort AI propagandists will use.
You really posted Ai slop in an effort to counter my point? I can dislike all this artificial intelligence bullshit and still understand that a law like this probably won't pass the court system.
Wow, true to your username.
I am not your enemy.
Haha, yea.
Sorry. After reading through the thread, I interpreted your edit as calling me a propagandist and low effort.
My mistake.