I've said it time and time again: AIs aren't trained to produce correct answers, but seemingly correct answers. That's an important distinction and exactly what makes AIs so dangerous to use. You will typically ask the AI about something you yourself are not an expert on, so you can't easily verify the answer. But it seems plausible so you assume it to be correct.
Thankfully, AI is bad at maths for exactly this reason. You don't have to be an expert on a very specific topic to be able to verify a proof and - spoiler alert - most of the proofs ChatGPT 5 has given me are plain incorrect, despite OpenSlop's claims that it is vastly superior to previous models.
I've been through the cycle of the AI companies repeatedly saying "now it's perfect" only admitting it's complete trash when they release the next iteration and claim "yeah it was broken, we admit, but now it's perfect" so many times now...
Problem being there's a massive marketing effort to gaslight everyone and so if I point it out in any vaguely significant context, I'm just not keeping up and most only have dealt with the shitty ChatGPT 5.1, not the more perfect 5.2. Of course in my company they are about the Anthropic models so it is instead Opus 4.5 versus 4.6 now. Even proving the limitations in trying to work with 4.6 gives anthropic money, and at best I earn a "oh, those are probably going to be fixed in 4.7 or 5 or whatever".
Outsiders are used to traditional software that has mistakes, but those are straightforward to address so a close but imperfect software can hit the mark in updates. LLMs not working that way doesn't make sense. They use the same version number scheme after all, so expectations should be similar.
My own advise for people starting to use AI is to use it for things you know very well. Using it for things you do not know well, will always be problematic.
The problem is that we've had a culture of people who don't know things very well control the purse strings relevant to those things.
So we have executives who don't know their work or customers at all and just try to bullshit while their people frantically try to repair the damage the executive does to preserve their jobs. Then they see bullshit generating platforms and see a kindred spirit, and set a goal of replacing those dumb employees with a more "executive" like entity that also can generate reports and code directly. No talking back, no explaining that the request needs clarification, that the data doesn't support their decision, just a "yes, and..." result agreeing with whatever dumbass request they thought would be correct and simple.
Finally, no one talking back to them and making their life difficult and casting doubt on their competency. With the biggest billionaires telling them this is the right way to go, as long as they keep sending money their way.
The problem is that we've had a culture of people who don't know things very well control the purse strings relevant to those things.
I mean that has been the case for a long time, AI may enhance the effect of it, but human stupidity is nothing new.
So we have executives who don't know their work or customers at all and just try to bullshit while their people frantically try to repair the damage the executive does to preserve their jobs. Then they see bullshit generating platforms and see a kindred spirit, and set a goal of replacing those dumb employees with a more "executive" like entity that also can generate reports and code directly. No talking back, no explaining that the request needs clarification, that the data doesn't support their decision, just a "yes, and..." result agreeing with whatever dumbass request they thought would be correct and simple.
Once again, yes men have also been a historic phenomenon, and yes AI might speed this up, but it is nothing new per se.
Ai is a tool, not a perfect one, heck most of the time, barely functional, but it is a tool and in order to use it, you need to understand what it can do, and what it can't do.
I think if you're aware of the environmental impact, learn how to use it responsibly and avoid many of it pitfalls, together with a critical mindset, it can be usable for some cases.
It can be useful, sure, and yes, the myopic, self-centered lying executive is nothing new, but there are big groups now thinking they can remove whatever semblance of a check on executive decisions might be there.
The problem is, every time you use it, you become more passive. More passive means less alert to problems.
Look at all the accidents involving "safety attendants" in self-driving cars. Every minute they let AI take the wheel, they become more complacent. Maaaybe I'll sneak a peak at my phone. Well, haven't gotten into an accident in a month, I'll watch a video. In the corner of my vision. Hah, that was good, gotta leave a commen — BANG!
Even worse is that over time, the seemingly correct answers will drift further away from actually correct answers. I'm the best case, it's because people expect the wrong answers as that's all they've been exposed to. Worse cases would be the answers skew toward a specific end that AI maker wants people to think.
Depending on the material, the LLM can be faster. I have used an LLM to extract viable search terms to then go and read the material myself.
I never trust the summary, but it frequently gives me clues as to what keywords could take me to the right area of a source material. Internet articles that stretch brief content into tedious mess, documentation that is 99% something I already know, but I need something buried in the 1%.
Was searching for a certain type of utility and traditional Internet searches were flooded with shitware that wasn't meeting the criteria I wanted, LLM successfully zeroed in on just the perfect GitHub project.
Then as a reminder to never trust the results, I queried how to make it do a certain thing and it mentioned a command option that seemed like a dumb name that was opposite of what I asked for if it did work and not only would it have been opposite, no such option existed.
My work is technically dense and I read all day.
It's sometimes nice when I'm mentally exhausted to see if it's worth the effort to dig deeper in a 10 second upload. That's all I'm getting at.
They are designed to convince people. That's all they do. True, or false, real or fake, doesn't matter, as long as it's convincing. They're like the ultimate, idealized sociopath and con artist. We are being conned by a software designed to con people.
What they can do is generate code that is totally deterministic, and then defer to those results. The fact these people aren't doing that just makes them negligent.
I just got around to watching some of the ads that the big AI companies aired during the Superbowl. Each time I was thinking "wow, if this is true, this person is an idiot and is in for a world of trouble".
Like, there was one where a young farmer was supposedly taking over the family farm from her grandfather or something. She said something like "I uploaded all our data to ChatGPT and now I do what it tells me to do." If that's the case, wow. That farm is going to fail.
Another one was some guy who ran some kind of a machinist's shop, and was claiming that the bookkeeping and inventory control the shop used was really old fashioned. So, he had ChatGPT create him a whole bunch of new part numbers to make online ordering easier. Again, wow. You're trusting this key part of your business to a machine that just randomly makes stuff up?
AI is literally trained to get the right answer but not actually perform the steps to get to the answer. It's like those people that trained dogs to carry explosives and run under tanks, they thought they were doing great until the first battle they used them in they realized that the dogs would run under their own tanks instead of the enemy ones, because that's what they were trained with.
It's not trained to get the right answer. It's trained to know what sequence of words tends to follow another sequence of words, and then a little noise is added to that function to make it a bit creative. So, if you ask it to make a legal document, it has been trained on millions of legal documents, so it knows exactly what sequences of words are likely. But, it has no concept of whether or not those words are "correct". It's basically making a movie prop legal document that will look really good on camera, but should never be taken into court.
Probably more this, the idea to use LLMs is good, but the employees just didn't know how to use it right. To say the LLMs are the problem is to admit being wrong, or worse, being gullible in the face of marketing material. The one thing that is drilled in as a first principle of business leadership is to never ever look weak by being wrong or tricked.
They haven't drifted apart, they were never close in the first place. People have been increasingly confident in the models because they've increasingly sounded more convincing, but the tenuous connection to reality has been consistently off.
Yeah it's not even drift. It's just smoke and mirrors that looks convincing if you don't know what you're talking about. It's why you see writers say "AI is great at coding, but not writing" and then you see coders say "AI is great at writing, but not coding."
If you have any idea what good looks like, you can immediately recognize AI ain't it.
For a fun example, at my company we had a POC done by a very well known AI company. It was supposed to analyze a MS Project schedule, then compare tasks in that schedule to various data sources related to to tasks, then flag potential schedule risks. In the demo to the COO, they showed the AI look at a project schedule and say "Task XYZ could be at risk due to vendor quality issues or potential supply chain issues."
The COO was amazed. Wow it looked through all this data and came back with such great insight. Later I dug under the hood and found that it wasn't looking at any data behind the scenes at all. It was just answering specifically "what could make a project task at risk?" and then giving a hypothetical answer.
Anyone using AI to make any sort of decision is basically doing the equivalent of Googling your issue and then taking the top response as gospel. Yeah that might work for a few basic things, but anything important that requires any thought whatsoever is going to fail spectacularly.
It's why you see writers say "AI is great at coding, but not writing" and then you see coders say "AI is great at writing, but not coding."
I've always thought of this as being just like Hollywood. If you have expertise in whatever field they present an expert in, it's painful how off they are but it lookks fine for everyone outside the field of expertise.
It was just answering specifically "what could make a project task at risk?" and then giving a hypothetical answer
It wasn't even doing that. It was "looking" at training data for what a an analysis like that might look like, and then inventing a sequence of words that matched that training data. Maybe "vendor quality issues" is something that appears in the training data, so it's a good thing to put in its output.
I raised this as a concern at the corporate role I work in when an AI tool that was being distributed and encouraged for usage showed two hallucinated data points that were cited in a large group setting. I happened to know my area well, the data was not just marginally wrong but way off, and I was able to quickly check the figures. I corrected it in the room after verifying on my laptop and the reaction in the room was sort of a harmless whoops. The rest of the presentation continued without a seeming acknowledgement that the rest of the figures should be checked.
When I approached the head of the team that constructed the tool after the meeting and shared the inaccuracies and my concerns, he told me that he'd rather have more data fluency through the ease of the tool and that inaccuracies were acceptable because of the convenience and widespread usage.
I suspect stories like this are happening across my industry. Meanwhile, the company put out a press release about our AI efforts (literally using Gemini's Gem tool and custom ChatGPTs seeded with Google Drive) as something investors should be very excited about.
When I approached the head of the team that constructed the tool after the meeting and shared the inaccuracies and my concerns, he told me that he’d rather have more data fluency through the ease of the tool and that inaccuracies were acceptable because of the convenience and widespread usage.
"I prefer more data that's completely made up over less data that is actually accurate."
This tells you everything you need to know about your company's marketing and data analysis department and the whole corporate leadership.
Honestly this is not a new problem and is a further expression of the larger problem.
"Leadership" becomes removed from the day to day operations that run the organization and by nature the "cream" that rises tend to be sycophantic in nature. Our internal biases at work so it's no fault of the individual.
It is not a new problem and that has been the case for a long time. But it's a good visualization of it.
Everyone in a company has their own goals, from the lowly actual worker who just wants to pay the bills and spend as little effort on it as possible, to departments which want to justify their useless existence, to leadership who mainly wants to look good towards the investors to get a nice bonus.
That some companies end up actually making products that ship and that people want to use is more of an unintended side effect than the intended purpose of anyone's work.
You're thinking like a person who values accurate information more than feeling some kind of 'cool' and 'trendy' because now you can vibe code and we are a forward thinking company that embraces new paradigms and synergizes our expectations with the potential reality our market disprupting innovations could bring.
... sorry, I lapsed back into corpo / had a stroke.
The board room is more concerned with the presentation than the data, because presentations make sales.
What a lot of people fail.to understand is that for the C-Suite, the product isn't what's being manufactured, or the service being sold. The product is the stock, and anything that makes the number go up in the short term is good.
Lots of them have fiduciary duties, meaning they're legally prohibited from doing anything that doesn't maximize the value of the stock from moment to moment.
Someone please show me the criminal lawsuit against the CEO that made the moral decision and the stock went down! I'm so sick of the term fiduciary duty being used as a bullshit shield for bad behavior. When Tesla stock tanked because musk threw a Nazi salute, where were the fiduciary duty people!?
Further, as you hinted, long term is not their problem. They get a bump, cash in a few million dollars worth of RSUs, and either saddle the next guy with the fallout, it of they haven't left yet "whoopsie, but I can blame the LLM and I was just following best practices in the industry at the time". Either way they have enough to not even pretend to work another day of their life, even ignoring previous grifts, and they'll go on and do the same thing to some other company when they bail or the company falls over.
At the moment, nothing will be done. There's no way the current SEC chair will give a fuck about this sort of stuff.
But assuming a competent chair ever gets in charge, I expect there to be a shit show of lawsuits. It really doesn't matter that "the LLM did it" lying on those mandatory reports can lead to big fines.
Lots of them have fiduciary duties, meaning they’re legally prohibited from doing anything that doesn’t maximize the value of the stock from moment to moment.
Overall, I agree with you that stock price is their motivation, but the notion of shareholder supremacy binding their hands and preventing them from doing things that they want to otherwise do is incorrect. For one, they aren't actually mandated to do this by law, and secondarily, even if they were -- which to reiterate, they aren't -- just about any action they take on any single issue can be portrayed as them attempting to maximize company value.
Not really, no. This is mostly a myth. Unless the executives are deliberately causing the company to lose money, they really can't be sued based on this fiduciary duty to shareholders. They have to act in the shareholders' best interest, but "shareholder interest" is entirely up to interpretation. For example, it's perfectly fine to say, "we're going to lose money over the next five years because we believe it will ensure maximum profits over the long term." In order to sue a CEO for failing to protect shareholders, they would have to be doing something deliberately and undeniably against shareholder interest. Like if they embezzle money into their own bank account, or if they hold a Joker-style literal money burning.
If it were that easy to sue executives for violating their fiduciary duty to shareholders, golden parachutes and inflated executive compensation packages wouldn't exist. But good luck suing a CEO because he's paid too much. He can just claim in court that his compensation will ensure the company attracts the best talent to perform the best they can.
Executives are given wide latitude in how they define the best financial interest of shareholders. Shareholders ultimately do have the ability to remove executives from their positions. This is supposed to be the default way of dealing with incompetent executives. As shareholders already possess the ability to fire a CEO at any time, there is a very high bar to clear before shareholders can also sue executives. It's generally assumed if they really are doing that bad a job, you should just fire them.
Yes, that's correct, it's not an issue of legal liability, it's an issue of their interests converging. the CEO holds stock, he is a stock holder, and the execs are stock holders, they don't need any motivation to put the stocks first, they know where their interests converge, and precious few executives make more money in salary than they take away in stocks, in practicality, every corporation that offers stocks is stock focused, the is why we had daily and weekly meetings in retail stores on the absolute bottom of the ladder to talk primarily about stock prices, and why the main information displayed on price guns is the sale price/cost/ current quarter sales/last quarter sales/and last year to date quarter sales, and the sales numbers daily/monthly/quarterly are what you see posted around the office, it's always about beating last year to date numbers, and last quarter's numbers, and what always drove me fucking nuts is that the store made TWENTY TWO MILLION FUCKING DOLLARS in profit, but "your store is failing because you didn't make twenty two million and one penny. they don't care that we were making money hand over fist, because that's not the game. that game is dead. don't worry. they'll still cut payroll, and you can't like... spend that money or keep that money, but it doesn't matter. it only maters if it makes the stocks move. it's stocks all the way down. because that's where the interests converge. also as a side note, golden parachutes are an internal security measure against hostile take over, it means if someone does successfully raid your business and performs a hostile takeover, they have to pay your executives staff when they fire them and loot the company more money than the company could be looted for. it's never actually intended to be paid out.
it's why capitalism is over. they do not care about making a profit at all. they only care about the stocks. there is only one outcome to this approach, and that's dissolving the company slowly until it fails because your willing to saw your legs off for a small spike in quarterly earning. You eventually run out of legs to saw off.
Coming from science to industry taught me one thing: numbers (and rationality as a whole) serves only one goal. And the goal is to persuade the opponents: colleagues, investors, regulators.
In this broken sense, your head of the team is right: hallucinations are acceptable if supervisors believe the output.
Sounds like the people who are realistic about AI are going to end up having a huge advantage over people who use it naively.
Like with statistics, there are a lot of tools out there that can handle them perfectly accurately, you just don't want an LLM doing "analysis" because the NN isn't encoded for that. Consider how often our own NNs get addicted to gambling while not being fully specialized for processing language. An LLM might not get caught up in a gambler's fallacy, but that's more on account of being too simple than being smarter.
I wonder if this will break the trust in MBAs because LLMs are deceptively incompetent and from the sound of this comment and other things I've seen, that deception works well enough that their ego around being involved in the tool's development clashes with the experts telling them it's not as useful as it seems.
You should have ask what would happen if the figures were wrong, let them make an excuse and then eat shit later. AI is taking our jobs. Never interrupt an enemy making a mistake.
As someone who has to deal with LLMs/AI daily in my work in order to fix the messes they create, this tracks.
AI's sole purpose is to provide you a positive solution. That's it. Now that positive solution doesn't even need to be accurate or even exist. It's built to provide a positive "right" solution without taking the steps to get to that "right" solution thus the majority of the time that solution is going to be a hallucination.
you see it all the time. you can ask it something tech related and in order to get to that positive right solution it'll hallucinate libraries that don't exist, or programs that don't even do what it claims they do. Because logically to the LLM this is the positive right solution WITHOUT utilizing any steps to confirm that this solution even exists.
So in the case of OPs post I can see it happening. They told the LLM they wanted analytics for 3 months and rather than take the steps to get to an accurate solution it ignored said steps and decided to provide positive solution.
Don't use AI/LLMs for your day to day problem solving. you're wasting your time. OpenAI, Anthropic, Google, etc have all programmed these things to provide you with "positive" solutions so you'll keep using them. they just hope you're not savvy enough to call out their LLM's when they're clearly and frequently wrong.
Probably the skepticism is around someone actually trusting the LLM this hard rather than the LLM doing it this badly. To that I will add that based on my experience with LLM enthusiasts, I believe that too.
I have talked to multiple people who recognize the hallucination problem, but think they have solved it because they are good "prompt engineers". They always include a sentence like "Do not hallucinate" and thinks that works.
The gaslighting from the LLM companies is really bad.
There are ways to get more relevant info (when using terms that have different meanings based on context), to reduce the needless ass kissing, and to help ensure you get response in formats more useful to you. But being able to provide it context is not some magic fix for the underlying problems of the way this tech is constructed and its limitations. It will never be trustworthy.
Edit: God forbid anyone want our criticism to be based of an understanding of this shit rather than pure vitriol and hot takes.
Having worked in departments providing data all my career, I'm not surprised in the slightest that people do not care in any way about where the numbers they got come from.
Base line level of trust in co-worker comperence combined with either too much workload to give everything a fine toothed comb through, or too much laziness to bother.
Presented by F, slide deck created by E, based off conclusions made by D, from data formatted to look good to them by C, from work that they asked B to do, which was ultimately done by low man on the totem pole A.
All it takes is for one person in that chain to be considered trustworthy for every level above it to consider it trustworthy info by default.
I work in a regulated sector and our higher ups are pushing AI so much. And there response to AI hallucinations is to just put a banner on all internal AI tools to cross verify and have some quarterly stupid "trainings" but almost everyone I know never checks and verifies the output. And I know of atleast 2 instances where because AI hallucinated some numbers we sent out extra money to a third party.
My workplace (finance company) bought out an investments company for a steal because they were having legal troubles, managed to pin it on a few individuals, then fired the individuals under scrutiny.
Our leadership thought the income and amount of assets they controlled was worth the risk.
This new group has been the biggest pain in the ass. Complete refusal to actually fold into the company culture, standards, even IT coverage. Kept trying to sidestep even basic stuff like returning old laptops after upgrades.
When I was still tech support, I had two particularly fun interactions with them. One was when it was discovered that one of their top earners got fired for shady shit, then they discovered a month later that he had set his mailbox to autoreply to every email pointing his former clients to his personal email. Then, they hired back this guy and he lasted a whole day before they caught him trying to steal as much private company info as he could grab. The other incident was when I got a call from this poor intern they hired, then dumped the responsibility for this awful home grown mess of Microsoft Access, Excel, and Word docs all linked over ODBC on this kid. Our side of IT refused to support it and kept asking them to meet with project management and our internal developers to get it brought up into this century. They refused to let us help them.
In the back half of last year, our circus of an Infosec Department finally locked down access to unapproved LLMs and AI tools. Officially we had been restricted to one specific one by written policy, signed by all employees, for over a year but it took someone getting caught by their coworker putting private info into a free public chatbot for them to enforce it.
Guess what sub-company is hundreds of thousands of dollars into a shadow IT project that has went through literally none of the proper channels to start using an explicitly disallowed LLM to process private customer data?
My last job was with a very large west coast tech giant (its name is a homonym with an equally-large food services company). The mandatory information security training was a series of animated shorts featuring talking bears which you could fast-forward through and still get credit for completing. Not surprisingly, we had major data thefts every few months -- or more accurately we admitted to major data thefts that often.
It reminds me of when the internet exploded in the 90s and everyone "needed" a website. Even my corner gas station had a web presence for some reason. Then with smartphones everyone needed their own app. Now with AI everyone MUST use AI everywhere! If you don't you are a fool and going to get left behind! Do you know what you actually need it for? Not really but some article you read said you could fire 50% of your staff if you do.
If they have to verify the results every time, what is the point?
have some quarterly stupid “trainings”
Feeling this in my bones, executive just sent out a plan for 'fixing' the fact that the AI tools they are paying for us to use are getting roasted for sucking, they are giving the vendor more money to provide 200 hours of mandatory training for us to take. That's more training than they have required for anything before, and using LLM tools isn't exactly a difficulty problem.
Nah though, its really fine, my quality of life is enormously superior barely surviving off of SSDI and not having to explain data analytics to thumb sucking morons (VPs, 90% of other team leads), and either fix or cover all their mistakes.
Yep, without analytics you at least are likely going on anecdotal feel for things which while woefully incomplete is at least probably based on actual indirect experience, like number of customers you've spoken with, how happy they have seemed, how employees have been feeling, etc.
Could be horribly off the mark without actual study of the data, but it is at least roughly directed by reality rather than just random narrative made by a word generator that has nothing to do with your company at all.
I'm not sure, because you see I'm not C-level by far, but I feel the decisions in such cases are made based on imaginary version of clients, and what tops feel the clients want (that is what they think they would want if they were clients)
And they may guess right or wrong, though I agree that they may be more likely to guess right than an LLM, being humans and all
That is spot on. Also usually thinking the customer "wants" stuff that would be awfully convenient for the company. They want subscription fees and reduced functionality.
But they at least can tell when a customer is actively pissed when they actually have to face them, and have some takeaway from that. Often it's "that customer was dumb anyway" but there at least a chance of maybe a course correction. It may be by some other executive using that feedback to snipe a current decision maker and take his job. Note I'm told that scenario may be playing out at work, as one executive made a call that lost a 60 million dollar a year customer and a junior executive got sent a copy of the client feedback and is now going over his boss's head to try to take his job because it was directly tied to the current executive being a complete idiot.
When you delegate, to a person, a tool or a process, you check the result. You make sure that the delegated tasks get done and correctly and that the results are what is expected.
Finding that it is not the case after months by luck shows incompetence. Look for the incompetent.
Problem being is that whoever is checking the result in this case had to do the work anyway, and in such a case... why bother with the LLM that can't be trusted to pull the data anyway?
I suppose they could take the facts and figures that a human pulled and have an LLM verbose it up for people who for whatever reason want needlessly verbose BS. Or maybe an LLM can do a review of the human generated report to help identify potential awkward writing or inconsistencies. But delegating work that you have to do anyway to double check the work seems pointless.
Like someone here said "trust is also thing". Once you check a few time that the process is right and the result are right, you don't need to check more than ponctually. Unfortunatly, that's not what happened in this story.
Before anything else: whether the specific story in the linked post is literally true doesn’t actually matter. The following observation about AI holds either way. If this example were wrong, ten others just like it would still make the same point.
What keeps jumping out at me in these AI threads is how consistently the conversation skips over the real constraint.
We keep hearing that AI will “increase productivity” or “accelerate thinking.” But in most large organizations, thinking is not the scarce resource. Permission to think is. Demand for thought is. The bottleneck was never how fast someone could draft an email or summarize a document. It was whether anyone actually wanted a careful answer in the first place.
A lot of companies mistook faster output for more value. They ran a pilot, saw emails go out quicker, reports get longer, slide decks look more polished, and assumed that meant something important had been solved. But scaling speed only helps if the organization needs more thinking. Most don’t. They already operate at the minimum level of reflection they’re willing to tolerate.
So what AI mostly does in practice is amplify performative cognition. It makes things look smarter without requiring anyone to be smarter. You get confident prose, plausible explanations, and lots of words where a short “yes,” “no,” or “we don’t know yet” would have been more honest and cheaper.
That’s why so many deployments feel disappointing once the novelty wears off. The technology didn’t fail. The assumption did. If an institution doesn’t value judgment, uncertainty, or dissent, no amount of machine assistance will conjure those qualities into existence. You can’t automate curiosity into a system that actively suppresses it.
Which leaves us with a technology in search of a problem that isn’t already constrained elsewhere. It’s very good at accelerating surfaces. It’s much less effective at deepening decisions, because depth was never in demand.
Very well put. This is a dimension to the ongoing AI nonsense that I haven't seen brought up before, but it certainly rings true. May I say also that "They already operate at the minimum level of reflection that they're willing to tolerate." Is a hell of a sentence and I'm a little jealous that I didn't come up with it.
Thanks, I really appreciate that. I’ve been getting a little grief this weekend because some of my posts are adapted from essays I’ve been working on for Substack, and apparently careful editing now makes you suspect as an actual person.
I’m very real, just flu-ridden and overthinking in public. Glad the line landed for you.
I have been saying for years now that the kind of work that LLMs are best suited for replacing and also would by far be their most cost effective use case from a business stand point is...
Well its the most expensive employees who basically just spend most of their time having meetings or writing emails about things they only understand at a very birds eye view level.
I'm a data analyst and primary authority on the data model of a particular source system. Most questions for figures from that system that can't be answered directly and easily in the frontend end up with me.
I had a manager show me how some new LLM they were developing (which I had contributed some information about the model to) could quickly answer some questions that usually I have to answer manually, as part of a pitch to make me switch to his department so I can apply my expertise for improving this fancy AI instead of answering questions manually.
He entered a prompt, got a figure that I knew wasn't correct and I queried my data model for the same info, with a significantly different answer. Given how much said manager leaned on my expertise in the first place, he couldn't very well challenge my results and got all sheepish about how the AI still in development and all.
I don't know how that model arrived at that figure. I don't know if it generated and ran a query against the data I'd provided. I don't know if it just invented the number. I don't know how the devs would figure out the error and how to fix it. But I do know how to explain my own queries, how to investigate errors and (usually) how to find a solution.
Anyone who relies on a random text generator - no matter how complex that generation method to make it sound human - to generate facts is dangerously inept.
I don’t know how the devs would figure out the error and how to fix it.
This is like the biggest factor that people don't get when thinking of these models in the context of software. "Oh it got it wrong, but the developers will fix it in an update". Nope, they can fix traditional software mistakes, LLM output and machine learning things... They can throw more training data at it (which sometimes just changes what it gets wrong) and hope for the best, they can do better job at curating the context window to give the model the best shot at outputting the right stuff (e.g. the guy who got Opus to generate a slow crappy buggy compiler had to traditionally write a filter to find and show only the 'relevent' compiler output back to the models), they can try to generate code to do what you want and have you review the code and correct issues. But debugging and fixing the model itself... that's just not a thing at all.
I was in a meeting where a sales executive was bragging about the 'AI sales agent' they were working, but admitting frustration with the developres and a bit confused why the software developers weren't making progress when those same developers always made decent progress before, and they should be able to do this even faster because they have AI tools to help them... It eternally seemed in a state that almost worked but not quite no matter what model or iteration they went to, no matter how much budget they allocated, when it came down to the specific facts and figures it would always screw up.
I cannot understand how long these executives wade in the LLM pool and still believes in capabilities beyond what anyone has experienced.
I cannot understand how long these executives wade in the LLM pool and still believes in capabilities beyond what anyone has experienced.
They leave the actual work to the boots on the ground so they don't see how shitty the output is. They listen to marketing about how great it is and mandate everyone use it and then any feedback is filtered through all the brownnosers that report to them.
It eternally seemed in a state that almost worked but not quite no matter what model or iteration they went to, no matter how much budget they allocated, when it came down to the specific facts and figures it would always screw up.
This is probably the biggest misunderstanding since "Project Managers think three developers can produce a baby in three months": Just throw more time and money at AI model "development" for better results. It supposes predictable, deterministic behaviour that can be corrected, but LLMs aren't deterministic ny design, since that wouldn't sound human anymore.
Sure, when you're a developer dedicated to advancing the underlying technology, you may actually produce better results in time, but if you're just the consumer, you may get a quick turnaround for an alright result (and for some purposes, "alright" may be enough) but eventually you'll plateau at the limitations of the model.
Of course, executives universally seem to struggle with the concept of upper limits, such as sustainable growth or productivity.
To everyone I've talked to about AI, I've suggested a test. Take a subject that they know they are an expert at. Then ask AI questions that they already know the answers to. See what percentage AI gets right, if any. Often they find that plausible sounding answers are produced however, if you know the subject, you know that it isn't quite fact that is produced. A recovery from an injury might be listed as 3 weeks when it is average 6-8 or similar. Someone who did not already know the correct information, could be damaged by the "guessed" response of AI. AI can have uses but it needs to be heavily scrutinized before passing on anything it generates. If you are good at something, that usually means you have to waste time in order to use AI.
I had a very simple script. All it does is trigger an action on a monthly schedule.
I passed the script to Copilot to review.
It caught some typos. It also said the logic of the script was flawed and it wouldn't work as intended.
I didn't need it to check the logic of the script. I knew the logic was sound because it was a port of a script I was already using. I asked because I was curious about what it would say.
After restating the prompt several times, I was able to get it to confirm that the logic was not flawed, but the process did not inspire any confidence in Copilot's abilities.
Happy cake day, and this absolutely. I figured out its game the first time I asked it a spec for an automotive project I was working on. I asked it the torque specs for some head bolts and it gave me the wrong answer. But not just the wrong number, the wrong procedure altogether. Modern engines have torque to yield specs, meaning essentially you torque them to a number and then add additional rotation to permanently distort the threads to lock it in. This car was absolutely not that and when I explained back to it the error it had made IT DID IT AGAIN. It sounded very plausible but someone following those directions would have likely ruined the engine.
Except the ceo and executives ultimately responsible will blame their underlings that will be fired, even though it was an executive level decision. They didn't get to the pinnacle of corporate governance by admitting mistakes. That's not what they were taught at their ivy league schools, they were taught to lie and cheat to steal, and further slander their victims to excuse it.
It was bad before the current president set his outstanding example for the rest of the country. See what being a lying cheating piece of shit gets you? Everything. Nothing matters. We have the wrong people in charge across the board, from business to government to institutions.
And you're not wrong. I work for a law firm and we were tracking his EO's until mid-2025, and they were so riddled with typos, and errors, and URL's pointing to the wrong EO, that we actually ended up having to hide the URL's in the database we built so clients wouldn't think it was us making these errors.
LLMs can't really do math, so if there is any analysis being done, the numbers will typically be junk. Unless the LLM is writing the code to do the math, but then you have to validate the code.
Bro, just give us a few trillion dollars, bro. I swear bro. It'll be AGI this time next year, bro. We're so close, bro. I just need need some money, bro. Some money and some god-damned faith, bro.
You can make something AI based that does this, but it's not cheap or easy. You have to make agents that handle data retrieval and programmatically make the LLM to chose the right agent. We set one up at work, it took months. If it can't find the data with a high certainty, it tells you to ask the analytics dept.
My workplace, the senior management, is going all in on Copilot. So much so that at the end of last year to told us to use Copilot for year end reviews! Even provided a prompt to use, told us to link it to Outlook (not sure why, since our email retention isn't very long)... but whatever.
I tried it, out of curiosity because I had no faith. It started printing out stats for things that never happened. It provided a 35% increase here, a 20% decress there, blah blah blah. It didn't actually highlight anything I do or did. And I'm banking that a human will partially read my review, not just use AI.
If someone read it, I'm good. If AI reads it, I do wonder if I screwed myself. Since senior mgmt is just offloading to AI...
Ah, the fun of performance reviews. No one actually cares what is written there, the result is decided ignoring the actual content.
So everyone pretends that what you write in there is important and pretends that the written response is important, but nothing you or they will write has any chance of changing promotions and raises. Those may come, but when they come, it's never because someone read your write up and thought 'OMG, give that person a raise and promotion'.
So it's all an act so I can see why management wants to take any opportunity to shuffle people off to even more token efforts.
Every year I try to convince my coworker that his hours and hours of scrutinizing his records and crafting just the perfect performance review that captures the essence of his entire year is wasted, compared to me logging into the tool and spending 10 minutes writing some vague stuff off the top of my head. I don't lie or anything, just have a relatively brief and vague review, because I know they already know how much they cared about what I did and I'm not talking them into more.
Have any evidence of that? The only thing I saw was commentors in that thread (who were obvious AI-bros) claiming it must be AI generated because "it just wouldn't happen"...
This is how I’m starting to see it too. Stock market is just the gambling statistics of the ownership class. Line goes down and we’re supposed to pretend it’s harder to grow food and build houses all of a sudden.
There's a difference. If I go and gamble away my life savings, then I'm on the street. If they gamble away their investments, the government will say 'poor thing' and give them money to keep the economy ok.
Our AI that monitors customer interactions sometimes makes up shit that didn't happen during the call. Any agent smart enough could probably fool it into giving the wrong summary with the right key words. I only caught on when I started reading the logs carefully, but I don't know if management cares so long as the business client is happy.
Sounds like material that is generated that the executives demand be generated but never actually uses. My work has a ton of this, because the executives want people to feel like they are accountable and being reviewed even as they know the executives don't understand the direct output of their work, so people have to do the technical thing and separately eternally do non-technical writeups of what the technical work meant. I think someone checked and the executives didn't even log into the system they demanded.
So LLM to generate the bullshit that no one wants to write or read but wants to pretend it's important.
I guarantee you this is how several, if not most, fortune 500 companies currently operate.
The 50k DOW is not just propped up by the circlejerk spending on imaginary RAM. There are bullshit reports being generated and presented every day.
I patiently wait. There is a diligent bureaucrat sitting somewhere going through fiscal reports line by line. It won't add up.. receipts will be requested.. bubble goes pop
Surely this is just fraud right? Seeing they have a board directors they have shareholders probably? I feel they should at least all get fired, if not prosecuted. This lack of competency is just criminal to me.
I see this happening more and more as corporate USA throws itself blindly into AI dependency. Basic facts and information will become corrupted, maybe hopelessly so, as it infuses itself into our systems.
I hope they sue whoever sold it to them. it's not artificial intelligence, it's a machine learning chat bot. they may as well be running their company with a magic eight ball.
What dumbass decided to implement an experimental technology and not test it for 5 minutes to make sure it's accurate before giving it to the whole company and telling them to rely upon it?
Someone who probably thought it would coast them through a bonus/vesting/promotion cycle and get out of dodge before consequences might happen. If this specific story is true, the big disaster is that it fell apart too soon, not that it fell apart.
How much do you want to bet they also rolled out bonsuses based on this bogus data? The one saving grace is they started using the new LLM tooling mid-Q4 so any quarterlies would at least be partially based on real data
This is why I hate search engines promoting AI results when you are researching for something. It is confidently giving incorrect responses. I asked for sources on one LLM model before while using Duckduckgo, and it just told me that there are no sources and the information is based on broad knowledge. At one point, I challenged the AI that it is wrong, but it insisted it doesn't. It turns out that it is citing a years old source written by a different bot long ago. But on the one hand, most of you are probably familiar that on occasions that the AI is incorrect and you challenge it, it will relent, although it will be a sycophant even though you yourself are actually incorrect. This is Schrödinger's AI.
If true they’re all idiots, but I don’t believe the story anyway. All the data question answering LLMs I’ve seen use the LLM to write SQL queries on your databases and then wrap the output in a summary. So the summary is easy to check and very unlikely to be significantly wrong. AI/ML/statistics and code is a tool, use it for what it’s good for, don’t use it for what it’s not, treat hype with skepticism
Honestly, I was leaning toward "funny but probably fake" myself until I checked out OP's post history, which mentions "startups" and namedrops a few SaaS tools used heavily in marketing. If you've worked with marketers (or a fair few startup bros, honestly), you'll know this isn't beyond the bounds of reason for some of them 😂
Heshmati told the student he had used Excel’s autofill function to mend the data. He had marked anywhere from two to four observations before or after the missing values and dragged the selected cells down or up, depending on the case. The program then filled in the blanks. If the new numbers turned negative, Heshmati replaced them with the last positive value Excel had spit out.
Of course that guy didn't need fancy autofill to act like an idiot, he used good old fashion autofill.
The problem is you've got people using the tools that don't understand the output or the method to get there.
Take the Excel copilot function. You need to pass in a range of cells for the slop prompt to work on, but it's an optional parameter. If you don't pass that in, it returns results anyway. They're just complete bollocks.
It's even worse than that. The ones that should understand the tools decide that the ease is good enough and just become AI brain rot.
I've watched co-workers go from good co-workers to people I can't trust anything from because I know they just slapped at an AI and didn't check it.
What's worse is, when you come to them as an engineer and tell them they're wrong, you have to prove to them the AI is wrong, not they have to prove to you the AI is right.
Moreover, when you refer to documentation, they can't be bothered and say the AI didn't say that, so it must be wrong.
I'm on the fence, but will say that if, for whatever reason, it was never actually connected to the data or the connection had some flaw, I could totally believe it would just fabricate a report that looks consistent with what the request asked for. Maybe it failed to ever convey that an error occurred. Maybe it conveyed the lack of data and the user thought he could just tell the AI to fix the problem without trying to understand it himself and triggered it to generate a narrative consistent with fixing it without actually being able to fix it.
Sure if you do a sanity check it should fall apart, but that assumes they bother. Some people have crazy confidence in LLM and didn't even check.
Clearly you've never worked as a data analyst, or you would know that the vast majority of upper management and C Suite are, in fact, all fucking idiots.
They're generally where they are because of mutual secrets and nepotism, for who else is on their contact list.
Even when it does pull numeric data, it gets very confused.
I asked about rough price of something and of course the AI summary came back and said something like:
It typically costs 400-500 but could cost up to $200 in extreme circumstances, with 750 being the average
Basically did get three figures from three different internet results and combined them into a single sentence in a nonsense way.
At least in such a scenario, someone with at least a couple of active brain cells would stop and recognize some bullshittery going on, but the executive probably TLDRs the sentence and stops after '400-500'.
Most ai stuff I use include a list of relevant sources next to the results. Do you not ever click in?
For me it’s critical to confirm, for example, the detail of that vendor api I want to use. However even then any hallucinations would mostly waste my time since if it doesn’t work it won’t get released
You’re telling me that people make actually business decisions without ever checking sources?
I see you've never worked in a medium to large business. Very often the average person will assume that someone else did that.
Also, what AI tools do you use?
CoPilot Chat is particularly popular right now due to Microsoft including it with Office 365 subscriptions, it having a ton of security controls built into the same places in Azure where you're already configuring shit, and terms about not training off of data input to it. Popular for sysadmins already drowning in Microsoft's bullshit who don't want to spend a lot of time on managing the slop generator(s).
Most commonly I use an aggregator. It runs a chat window in a browser and you can choose from a variety of models or let it pick. While it’s not integrated with anything, it does really well for general purpose writing. Every result comes with citations and a few suggestions for next steps.
My company just gave up on copilot as useless, but we were explicitly using it for coding and it was just not effective. Sometimes “free” costs too much
Currently management is really pushing Cursor/Claude for coding, which I really hate. While Claude is much better at coding than copilot and does cite sources, cursor is way too aggressive at spraying arbitrary changes across the code base. I’ve had to do way too much damage control from junior devs blindly accepting when it makes arbitrary changes across the code base. For example one of my guys used it to generate unit tests, which it is good at, but they generated an order of magnitude too many tests of dubious value, that now need to run in every build and be maintained forever …. And in all that slop just arbitrarily introduced a new mocking tool. The intelligence part is pretty good but it needs to get much better at keeping the human in the loop. For example, I really like it for code reviews, it makes good catches and suggestions, but is horrible at presenting them to the developer for individual approval. Current effort is trying to use the agent.md to establish a sensible base for useful code reviews
Other than that, we’re spending a lot of time with mcp agents, which I’m still trying to decide on. All too often it’s just a more complex and dangerous way to do a text search, but it has a lot of potential to bring active data into the ai decision space
This is why AI will only be used in subjective things that dont need to be perfect, like art. You cant create intelligence by regurgitating reddit posts via an LLM.
AI is already useful in some areas, such as identifying patterns in medical data (which leads to breakthroughs in diagnosing rare diseases, identifying genetic factors in disease, targeting new treatments, etc).
Sure I just mean its not really AI, its simply more machine learning. Its not sentient, its not replacing the bulk of workers, it doesnt understand what its doing so its just a more advanced tool for workers to take advantage of.
You say "already" useful like there are going to be more uses in the future.
Nope. That's what it's useful for, and that's all it's going to be useful for until we progress beyond pattern recognition engines. Nothing available right now is ever going to be any more useful than it already is, it'll be something totally new.
Your high tech tool produces wrong data?
You cant not use it, your competitor will.
So you must do something else to make it more reliable, something small that the Firm can do.
And if you only caught it 3 months in, that firm is incompetent as fuck.
But its probably not even a real post.
I've said it time and time again: AIs aren't trained to produce correct answers, but seemingly correct answers. That's an important distinction and exactly what makes AIs so dangerous to use. You will typically ask the AI about something you yourself are not an expert on, so you can't easily verify the answer. But it seems plausible so you assume it to be correct.
Thankfully, AI is bad at maths for exactly this reason. You don't have to be an expert on a very specific topic to be able to verify a proof and - spoiler alert - most of the proofs ChatGPT 5 has given me are plain incorrect, despite OpenSlop's claims that it is vastly superior to previous models.
I've been through the cycle of the AI companies repeatedly saying "now it's perfect" only admitting it's complete trash when they release the next iteration and claim "yeah it was broken, we admit, but now it's perfect" so many times now...
Problem being there's a massive marketing effort to gaslight everyone and so if I point it out in any vaguely significant context, I'm just not keeping up and most only have dealt with the shitty ChatGPT 5.1, not the more perfect 5.2. Of course in my company they are about the Anthropic models so it is instead Opus 4.5 versus 4.6 now. Even proving the limitations in trying to work with 4.6 gives anthropic money, and at best I earn a "oh, those are probably going to be fixed in 4.7 or 5 or whatever".
Outsiders are used to traditional software that has mistakes, but those are straightforward to address so a close but imperfect software can hit the mark in updates. LLMs not working that way doesn't make sense. They use the same version number scheme after all, so expectations should be similar.
Both of those can be true.
I mean yeah, but they specifically mentioned its amazing performance in tasks requiring reasoning
My own advise for people starting to use AI is to use it for things you know very well. Using it for things you do not know well, will always be problematic.
The problem is that we've had a culture of people who don't know things very well control the purse strings relevant to those things.
So we have executives who don't know their work or customers at all and just try to bullshit while their people frantically try to repair the damage the executive does to preserve their jobs. Then they see bullshit generating platforms and see a kindred spirit, and set a goal of replacing those dumb employees with a more "executive" like entity that also can generate reports and code directly. No talking back, no explaining that the request needs clarification, that the data doesn't support their decision, just a "yes, and..." result agreeing with whatever dumbass request they thought would be correct and simple.
Finally, no one talking back to them and making their life difficult and casting doubt on their competency. With the biggest billionaires telling them this is the right way to go, as long as they keep sending money their way.
And on top of that, the people who don’t know things very well generated lots of the material the LLMs were trained on in the first place.
Can’t really blame the models for realizing much of human knowledge is bullshit and acting accordingly.
I mean that has been the case for a long time, AI may enhance the effect of it, but human stupidity is nothing new.
Once again, yes men have also been a historic phenomenon, and yes AI might speed this up, but it is nothing new per se.
Ai is a tool, not a perfect one, heck most of the time, barely functional, but it is a tool and in order to use it, you need to understand what it can do, and what it can't do.
I think if you're aware of the environmental impact, learn how to use it responsibly and avoid many of it pitfalls, together with a critical mindset, it can be usable for some cases.
It can be useful, sure, and yes, the myopic, self-centered lying executive is nothing new, but there are big groups now thinking they can remove whatever semblance of a check on executive decisions might be there.
The problem is, every time you use it, you become more passive. More passive means less alert to problems.
Look at all the accidents involving "safety attendants" in self-driving cars. Every minute they let AI take the wheel, they become more complacent. Maaaybe I'll sneak a peak at my phone. Well, haven't gotten into an accident in a month, I'll watch a video. In the corner of my vision. Hah, that was good, gotta leave a commen — BANG!
I prefer to say “algorithmically common” instead of “seemingly correct” but otherwise agree with you.
I use "mathmatical approximations of correct answers"
But that's wrong. It's not trained on correct answers. It's trained on whatever happens to be out there in the world.
It's mathematical approximations of words that are likely to be found near that question.
Even worse is that over time, the seemingly correct answers will drift further away from actually correct answers. I'm the best case, it's because people expect the wrong answers as that's all they've been exposed to. Worse cases would be the answers skew toward a specific end that AI maker wants people to think.
I use it to summarize stuff sometimes, and I honestly spend almost as much time checking it's accurate than I would if I had just read and summarized.
It is useful for 'What does this contain?' so I can see if I need to read something. Or rewording something I have made a pig's ear out of.
I wouldn't trust it for anything important.
The most important thing to do if you do use AI is to not ask leading questions. Keep them simple and direct
Skimming and scanning texts is a skill that achieves the same goal more quickly than using an unreliable bullshit generator.
Depending on the material, the LLM can be faster. I have used an LLM to extract viable search terms to then go and read the material myself.
I never trust the summary, but it frequently gives me clues as to what keywords could take me to the right area of a source material. Internet articles that stretch brief content into tedious mess, documentation that is 99% something I already know, but I need something buried in the 1%.
Was searching for a certain type of utility and traditional Internet searches were flooded with shitware that wasn't meeting the criteria I wanted, LLM successfully zeroed in on just the perfect GitHub project.
Then as a reminder to never trust the results, I queried how to make it do a certain thing and it mentioned a command option that seemed like a dumb name that was opposite of what I asked for if it did work and not only would it have been opposite, no such option existed.
Lol. Your advice: learn to read, noob
My work is technically dense and I read all day. It's sometimes nice when I'm mentally exhausted to see if it's worth the effort to dig deeper in a 10 second upload. That's all I'm getting at.
They are designed to convince people. That's all they do. True, or false, real or fake, doesn't matter, as long as it's convincing. They're like the ultimate, idealized sociopath and con artist. We are being conned by a software designed to con people.
Plausible confabulation machines.
What they can do is generate code that is totally deterministic, and then defer to those results. The fact these people aren't doing that just makes them negligent.
I just got around to watching some of the ads that the big AI companies aired during the Superbowl. Each time I was thinking "wow, if this is true, this person is an idiot and is in for a world of trouble".
Like, there was one where a young farmer was supposedly taking over the family farm from her grandfather or something. She said something like "I uploaded all our data to ChatGPT and now I do what it tells me to do." If that's the case, wow. That farm is going to fail.
Another one was some guy who ran some kind of a machinist's shop, and was claiming that the bookkeeping and inventory control the shop used was really old fashioned. So, he had ChatGPT create him a whole bunch of new part numbers to make online ordering easier. Again, wow. You're trusting this key part of your business to a machine that just randomly makes stuff up?
I suspect this will happen all over with in a few years, AI was good enough at first, but over time reality and the AI started drifting apart
AI is literally trained to get the right answer but not actually perform the steps to get to the answer. It's like those people that trained dogs to carry explosives and run under tanks, they thought they were doing great until the first battle they used them in they realized that the dogs would run under their own tanks instead of the enemy ones, because that's what they were trained with.
Holy shit, that's what they get for being so evil that they trained dogs as suicide bombers.
It's not trained to get the right answer. It's trained to know what sequence of words tends to follow another sequence of words, and then a little noise is added to that function to make it a bit creative. So, if you ask it to make a legal document, it has been trained on millions of legal documents, so it knows exactly what sequences of words are likely. But, it has no concept of whether or not those words are "correct". It's basically making a movie prop legal document that will look really good on camera, but should never be taken into court.
And then, the very same CEOs that demanded the use of AI in decision making will be the ones that blame it for bad decisions.
while also blaming employees
Of course, it is the employees who used it. /s
Probably more this, the idea to use LLMs is good, but the employees just didn't know how to use it right. To say the LLMs are the problem is to admit being wrong, or worse, being gullible in the face of marketing material. The one thing that is drilled in as a first principle of business leadership is to never ever look weak by being wrong or tricked.
What employees?
They haven't drifted apart, they were never close in the first place. People have been increasingly confident in the models because they've increasingly sounded more convincing, but the tenuous connection to reality has been consistently off.
Yeah it's not even drift. It's just smoke and mirrors that looks convincing if you don't know what you're talking about. It's why you see writers say "AI is great at coding, but not writing" and then you see coders say "AI is great at writing, but not coding."
If you have any idea what good looks like, you can immediately recognize AI ain't it.
For a fun example, at my company we had a POC done by a very well known AI company. It was supposed to analyze a MS Project schedule, then compare tasks in that schedule to various data sources related to to tasks, then flag potential schedule risks. In the demo to the COO, they showed the AI look at a project schedule and say "Task XYZ could be at risk due to vendor quality issues or potential supply chain issues."
The COO was amazed. Wow it looked through all this data and came back with such great insight. Later I dug under the hood and found that it wasn't looking at any data behind the scenes at all. It was just answering specifically "what could make a project task at risk?" and then giving a hypothetical answer.
Anyone using AI to make any sort of decision is basically doing the equivalent of Googling your issue and then taking the top response as gospel. Yeah that might work for a few basic things, but anything important that requires any thought whatsoever is going to fail spectacularly.
I've always thought of this as being just like Hollywood. If you have expertise in whatever field they present an expert in, it's painful how off they are but it lookks fine for everyone outside the field of expertise.
It wasn't even doing that. It was "looking" at training data for what a an analysis like that might look like, and then inventing a sequence of words that matched that training data. Maybe "vendor quality issues" is something that appears in the training data, so it's a good thing to put in its output.
I raised this as a concern at the corporate role I work in when an AI tool that was being distributed and encouraged for usage showed two hallucinated data points that were cited in a large group setting. I happened to know my area well, the data was not just marginally wrong but way off, and I was able to quickly check the figures. I corrected it in the room after verifying on my laptop and the reaction in the room was sort of a harmless whoops. The rest of the presentation continued without a seeming acknowledgement that the rest of the figures should be checked.
When I approached the head of the team that constructed the tool after the meeting and shared the inaccuracies and my concerns, he told me that he'd rather have more data fluency through the ease of the tool and that inaccuracies were acceptable because of the convenience and widespread usage.
I suspect stories like this are happening across my industry. Meanwhile, the company put out a press release about our AI efforts (literally using Gemini's Gem tool and custom ChatGPTs seeded with Google Drive) as something investors should be very excited about.
"I prefer more data that's completely made up over less data that is actually accurate."
This tells you everything you need to know about your company's marketing and data analysis department and the whole corporate leadership.
Potemkin leadership.
Honestly this is not a new problem and is a further expression of the larger problem.
"Leadership" becomes removed from the day to day operations that run the organization and by nature the "cream" that rises tend to be sycophantic in nature. Our internal biases at work so it's no fault of the individual.
Humanity is their own worst enemy lol
It is not a new problem and that has been the case for a long time. But it's a good visualization of it.
Everyone in a company has their own goals, from the lowly actual worker who just wants to pay the bills and spend as little effort on it as possible, to departments which want to justify their useless existence, to leadership who mainly wants to look good towards the investors to get a nice bonus.
That some companies end up actually making products that ship and that people want to use is more of an unintended side effect than the intended purpose of anyone's work.
That makes no sense. The inaccuracies are even less acceptable with widespread use!
You're thinking like a person who values accurate information more than feeling some kind of 'cool' and 'trendy' because now you can vibe code and we are a forward thinking company that embraces new paradigms and synergizes our expectations with the potential reality our market disprupting innovations could bring.
... sorry, I lapsed back into corpo / had a stroke.
🤫
It’s technological astrology. We’re doomed.
You need to know the words to properly wake the machine spirit
The board room is more concerned with the presentation than the data, because presentations make sales.
What a lot of people fail.to understand is that for the C-Suite, the product isn't what's being manufactured, or the service being sold. The product is the stock, and anything that makes the number go up in the short term is good.
Lots of them have fiduciary duties, meaning they're legally prohibited from doing anything that doesn't maximize the value of the stock from moment to moment.
Someone please show me the criminal lawsuit against the CEO that made the moral decision and the stock went down! I'm so sick of the term fiduciary duty being used as a bullshit shield for bad behavior. When Tesla stock tanked because musk threw a Nazi salute, where were the fiduciary duty people!?
Not criminal
Synopsys, Inc. (NASDAQ: SNPS) Investor Securities Class Action Lawsuit 10/31/2025
But that's over false or misleading statements. I'm not saying you can lie, just that you don't have to throw orphans in the meat grinder.
Further, as you hinted, long term is not their problem. They get a bump, cash in a few million dollars worth of RSUs, and either saddle the next guy with the fallout, it of they haven't left yet "whoopsie, but I can blame the LLM and I was just following best practices in the industry at the time". Either way they have enough to not even pretend to work another day of their life, even ignoring previous grifts, and they'll go on and do the same thing to some other company when they bail or the company falls over.
At the moment, nothing will be done. There's no way the current SEC chair will give a fuck about this sort of stuff.
But assuming a competent chair ever gets in charge, I expect there to be a shit show of lawsuits. It really doesn't matter that "the LLM did it" lying on those mandatory reports can lead to big fines.
Overall, I agree with you that stock price is their motivation, but the notion of shareholder supremacy binding their hands and preventing them from doing things that they want to otherwise do is incorrect. For one, they aren't actually mandated to do this by law, and secondarily, even if they were -- which to reiterate, they aren't -- just about any action they take on any single issue can be portrayed as them attempting to maximize company value.
https://pluralistic.net/2024/09/18/falsifiability/#figleaves-not-rubrics
No, not illegal, but they can be sued by the shareholder for failing to maximize value.
Sure, but since it's an unfalsifiable proposition, good luck proving it in court for any specific action.
Apparently, it does happen: https://tempusfugitlaw.com/real-life-breach-of-fiduciary-duty-case-examples-outcomes/
Particularly of note is the descision around AA's ESG investments.
I think this is mixing things up a bit. At least some of the cases there were fraud based.
Not really, no. This is mostly a myth. Unless the executives are deliberately causing the company to lose money, they really can't be sued based on this fiduciary duty to shareholders. They have to act in the shareholders' best interest, but "shareholder interest" is entirely up to interpretation. For example, it's perfectly fine to say, "we're going to lose money over the next five years because we believe it will ensure maximum profits over the long term." In order to sue a CEO for failing to protect shareholders, they would have to be doing something deliberately and undeniably against shareholder interest. Like if they embezzle money into their own bank account, or if they hold a Joker-style literal money burning.
If it were that easy to sue executives for violating their fiduciary duty to shareholders, golden parachutes and inflated executive compensation packages wouldn't exist. But good luck suing a CEO because he's paid too much. He can just claim in court that his compensation will ensure the company attracts the best talent to perform the best they can.
Executives are given wide latitude in how they define the best financial interest of shareholders. Shareholders ultimately do have the ability to remove executives from their positions. This is supposed to be the default way of dealing with incompetent executives. As shareholders already possess the ability to fire a CEO at any time, there is a very high bar to clear before shareholders can also sue executives. It's generally assumed if they really are doing that bad a job, you should just fire them.
Yes, that's correct, it's not an issue of legal liability, it's an issue of their interests converging. the CEO holds stock, he is a stock holder, and the execs are stock holders, they don't need any motivation to put the stocks first, they know where their interests converge, and precious few executives make more money in salary than they take away in stocks, in practicality, every corporation that offers stocks is stock focused, the is why we had daily and weekly meetings in retail stores on the absolute bottom of the ladder to talk primarily about stock prices, and why the main information displayed on price guns is the sale price/cost/ current quarter sales/last quarter sales/and last year to date quarter sales, and the sales numbers daily/monthly/quarterly are what you see posted around the office, it's always about beating last year to date numbers, and last quarter's numbers, and what always drove me fucking nuts is that the store made TWENTY TWO MILLION FUCKING DOLLARS in profit, but "your store is failing because you didn't make twenty two million and one penny. they don't care that we were making money hand over fist, because that's not the game. that game is dead. don't worry. they'll still cut payroll, and you can't like... spend that money or keep that money, but it doesn't matter. it only maters if it makes the stocks move. it's stocks all the way down. because that's where the interests converge. also as a side note, golden parachutes are an internal security measure against hostile take over, it means if someone does successfully raid your business and performs a hostile takeover, they have to pay your executives staff when they fire them and loot the company more money than the company could be looted for. it's never actually intended to be paid out.
By Brother in Christ, Paragraphs. Periods. Capitalization.
proofreading is the last retreat of cowards!!!
it's why capitalism is over. they do not care about making a profit at all. they only care about the stocks. there is only one outcome to this approach, and that's dissolving the company slowly until it fails because your willing to saw your legs off for a small spike in quarterly earning. You eventually run out of legs to saw off.
Coming from science to industry taught me one thing: numbers (and rationality as a whole) serves only one goal. And the goal is to persuade the opponents: colleagues, investors, regulators.
In this broken sense, your head of the team is right: hallucinations are acceptable if supervisors believe the output.
Sounds like the people who are realistic about AI are going to end up having a huge advantage over people who use it naively.
Like with statistics, there are a lot of tools out there that can handle them perfectly accurately, you just don't want an LLM doing "analysis" because the NN isn't encoded for that. Consider how often our own NNs get addicted to gambling while not being fully specialized for processing language. An LLM might not get caught up in a gambler's fallacy, but that's more on account of being too simple than being smarter.
I wonder if this will break the trust in MBAs because LLMs are deceptively incompetent and from the sound of this comment and other things I've seen, that deception works well enough that their ego around being involved in the tool's development clashes with the experts telling them it's not as useful as it seems.
You should have ask what would happen if the figures were wrong, let them make an excuse and then eat shit later. AI is taking our jobs. Never interrupt an enemy making a mistake.
I somehow hope this is made up, because doing this without checking and finding the obvious errors is insane.
This is probably real, as it isn't the first time it happened: https://www.theguardian.com/technology/2025/jun/06/high-court-tells-uk-lawyers-to-urgently-stop-misuse-of-ai-in-legal-work
It is a thing that is happening, but the OP instance probably didn't, since it is just a reddit post.
As someone who has to deal with LLMs/AI daily in my work in order to fix the messes they create, this tracks.
AI's sole purpose is to provide you a positive solution. That's it. Now that positive solution doesn't even need to be accurate or even exist. It's built to provide a positive "right" solution without taking the steps to get to that "right" solution thus the majority of the time that solution is going to be a hallucination.
you see it all the time. you can ask it something tech related and in order to get to that positive right solution it'll hallucinate libraries that don't exist, or programs that don't even do what it claims they do. Because logically to the LLM this is the positive right solution WITHOUT utilizing any steps to confirm that this solution even exists.
So in the case of OPs post I can see it happening. They told the LLM they wanted analytics for 3 months and rather than take the steps to get to an accurate solution it ignored said steps and decided to provide positive solution.
Don't use AI/LLMs for your day to day problem solving. you're wasting your time. OpenAI, Anthropic, Google, etc have all programmed these things to provide you with "positive" solutions so you'll keep using them. they just hope you're not savvy enough to call out their LLM's when they're clearly and frequently wrong.
Probably the skepticism is around someone actually trusting the LLM this hard rather than the LLM doing it this badly. To that I will add that based on my experience with LLM enthusiasts, I believe that too.
I have talked to multiple people who recognize the hallucination problem, but think they have solved it because they are good "prompt engineers". They always include a sentence like "Do not hallucinate" and thinks that works.
The gaslighting from the LLM companies is really bad.
"Prompt engineering" is the astrology of the LLM world.
There are ways to get more relevant info (when using terms that have different meanings based on context), to reduce the needless ass kissing, and to help ensure you get response in formats more useful to you. But being able to provide it context is not some magic fix for the underlying problems of the way this tech is constructed and its limitations. It will never be trustworthy.
Edit: God forbid anyone want our criticism to be based of an understanding of this shit rather than pure vitriol and hot takes.
Use of AI in companies would not save any time if you were checking each result.
Using an LLM for anything other than language, weird software questions, and really terrible low-stakes coding just seems like a stupid idea to me.
It can save time in certain contexts, but pretty much only in writing you some headings for a report or something. Using it for NUMBERS is insane.
It has happened in the open, so I don't see why it wouldn't happen even more behind closed doors:
Deloitte will provide a partial refund to the federal government over a $440,000 report that contained several errors, after admitting it used generative artificial intelligence to help produce it..
Yeah.
Kinda surprised there isn't already a term for submitting / presenting AI slop without reviewing and confirming.
Negligence and fraud come to mind
Slop flop seems like it would work. He’s flopped the slop. That slop was flopped out without checking.
Having worked in departments providing data all my career, I'm not surprised in the slightest that people do not care in any way about where the numbers they got come from.
Base line level of trust in co-worker comperence combined with either too much workload to give everything a fine toothed comb through, or too much laziness to bother.
Presented by F, slide deck created by E, based off conclusions made by D, from data formatted to look good to them by C, from work that they asked B to do, which was ultimately done by low man on the totem pole A.
All it takes is for one person in that chain to be considered trustworthy for every level above it to consider it trustworthy info by default.
I work in a regulated sector and our higher ups are pushing AI so much. And there response to AI hallucinations is to just put a banner on all internal AI tools to cross verify and have some quarterly stupid "trainings" but almost everyone I know never checks and verifies the output. And I know of atleast 2 instances where because AI hallucinated some numbers we sent out extra money to a third party.
My workplace (finance company) bought out an investments company for a steal because they were having legal troubles, managed to pin it on a few individuals, then fired the individuals under scrutiny.
Our leadership thought the income and amount of assets they controlled was worth the risk.
This new group has been the biggest pain in the ass. Complete refusal to actually fold into the company culture, standards, even IT coverage. Kept trying to sidestep even basic stuff like returning old laptops after upgrades.
When I was still tech support, I had two particularly fun interactions with them. One was when it was discovered that one of their top earners got fired for shady shit, then they discovered a month later that he had set his mailbox to autoreply to every email pointing his former clients to his personal email. Then, they hired back this guy and he lasted a whole day before they caught him trying to steal as much private company info as he could grab. The other incident was when I got a call from this poor intern they hired, then dumped the responsibility for this awful home grown mess of Microsoft Access, Excel, and Word docs all linked over ODBC on this kid. Our side of IT refused to support it and kept asking them to meet with project management and our internal developers to get it brought up into this century. They refused to let us help them.
In the back half of last year, our circus of an Infosec Department finally locked down access to unapproved LLMs and AI tools. Officially we had been restricted to one specific one by written policy, signed by all employees, for over a year but it took someone getting caught by their coworker putting private info into a free public chatbot for them to enforce it.
Guess what sub-company is hundreds of thousands of dollars into a shadow IT project that has went through literally none of the proper channels to start using an explicitly disallowed LLM to process private customer data?
My last job was with a very large west coast tech giant (its name is a homonym with an equally-large food services company). The mandatory information security training was a series of animated shorts featuring talking bears which you could fast-forward through and still get credit for completing. Not surprisingly, we had major data thefts every few months -- or more accurately we admitted to major data thefts that often.
It reminds me of when the internet exploded in the 90s and everyone "needed" a website. Even my corner gas station had a web presence for some reason. Then with smartphones everyone needed their own app. Now with AI everyone MUST use AI everywhere! If you don't you are a fool and going to get left behind! Do you know what you actually need it for? Not really but some article you read said you could fire 50% of your staff if you do.
I would quite honestly prefer every place to have their own web site instead of the ginormous amount of places that have facebook pages.
If they have to verify the results every time, what is the point?
Feeling this in my bones, executive just sent out a plan for 'fixing' the fact that the AI tools they are paying for us to use are getting roasted for sucking, they are giving the vendor more money to provide 200 hours of mandatory training for us to take. That's more training than they have required for anything before, and using LLM tools isn't exactly a difficulty problem.
Self solving problem!
As an unemployed data analyst / econometrician:
lol, rofl, perhaps even... lmao.
Nah though, its really fine, my quality of life is enormously superior barely surviving off of SSDI and not having to explain data analytics to thumb sucking morons (VPs, 90% of other team leads), and either fix or cover all their mistakes.
Yeah, sure, just have the AI do it, go nuts.
I am enjoying my unexpected early retirement.
Joke's on you, we make our decisions without asking AI for analytics. Because we don't ask for analytics at all
I feel like no analytics is probably better than decisions based on made-up analytics.
Yep, without analytics you at least are likely going on anecdotal feel for things which while woefully incomplete is at least probably based on actual indirect experience, like number of customers you've spoken with, how happy they have seemed, how employees have been feeling, etc.
Could be horribly off the mark without actual study of the data, but it is at least roughly directed by reality rather than just random narrative made by a word generator that has nothing to do with your company at all.
I'm not sure, because you see I'm not C-level by far, but I feel the decisions in such cases are made based on imaginary version of clients, and what tops feel the clients want (that is what they think they would want if they were clients)
And they may guess right or wrong, though I agree that they may be more likely to guess right than an LLM, being humans and all
That is spot on. Also usually thinking the customer "wants" stuff that would be awfully convenient for the company. They want subscription fees and reduced functionality.
But they at least can tell when a customer is actively pissed when they actually have to face them, and have some takeaway from that. Often it's "that customer was dumb anyway" but there at least a chance of maybe a course correction. It may be by some other executive using that feedback to snipe a current decision maker and take his job. Note I'm told that scenario may be playing out at work, as one executive made a call that lost a 60 million dollar a year customer and a junior executive got sent a copy of the client feedback and is now going over his boss's head to try to take his job because it was directly tied to the current executive being a complete idiot.
Wow, that story is completely awful and pretty believable
It almost certainly is, in as much as the one time and ongoing cost of not having some kind of data analytics team or system is $0.
I'd say the cost is much higher than that, it's just not because of the salary for the non-existent analytics team
I don't need AI to fabricate data. I can be stupid on my own, thank you.
When you delegate, to a person, a tool or a process, you check the result. You make sure that the delegated tasks get done and correctly and that the results are what is expected.
Finding that it is not the case after months by luck shows incompetence. Look for the incompetent.
Yeah. Trust is also a thing, like if you delegate to a person that you've seen getting the job done multiple times before, you won't check as closely.
But this person asked to verify and was told not to. Insane.
100%
Hallucinations are widely known, this is a collective failure of the whole chain of leadership.
Problem being is that whoever is checking the result in this case had to do the work anyway, and in such a case... why bother with the LLM that can't be trusted to pull the data anyway?
I suppose they could take the facts and figures that a human pulled and have an LLM verbose it up for people who for whatever reason want needlessly verbose BS. Or maybe an LLM can do a review of the human generated report to help identify potential awkward writing or inconsistencies. But delegating work that you have to do anyway to double check the work seems pointless.
Like someone here said "trust is also thing". Once you check a few time that the process is right and the result are right, you don't need to check more than ponctually. Unfortunatly, that's not what happened in this story.
Before anything else: whether the specific story in the linked post is literally true doesn’t actually matter. The following observation about AI holds either way. If this example were wrong, ten others just like it would still make the same point.
What keeps jumping out at me in these AI threads is how consistently the conversation skips over the real constraint.
We keep hearing that AI will “increase productivity” or “accelerate thinking.” But in most large organizations, thinking is not the scarce resource. Permission to think is. Demand for thought is. The bottleneck was never how fast someone could draft an email or summarize a document. It was whether anyone actually wanted a careful answer in the first place.
A lot of companies mistook faster output for more value. They ran a pilot, saw emails go out quicker, reports get longer, slide decks look more polished, and assumed that meant something important had been solved. But scaling speed only helps if the organization needs more thinking. Most don’t. They already operate at the minimum level of reflection they’re willing to tolerate.
So what AI mostly does in practice is amplify performative cognition. It makes things look smarter without requiring anyone to be smarter. You get confident prose, plausible explanations, and lots of words where a short “yes,” “no,” or “we don’t know yet” would have been more honest and cheaper.
That’s why so many deployments feel disappointing once the novelty wears off. The technology didn’t fail. The assumption did. If an institution doesn’t value judgment, uncertainty, or dissent, no amount of machine assistance will conjure those qualities into existence. You can’t automate curiosity into a system that actively suppresses it.
Which leaves us with a technology in search of a problem that isn’t already constrained elsewhere. It’s very good at accelerating surfaces. It’s much less effective at deepening decisions, because depth was never in demand.
If you’re interested, I write more about this here: https://tover153.substack.com/
Not selling anything. Just thinking out loud, slowly, while that’s still allowed.
Very well put. This is a dimension to the ongoing AI nonsense that I haven't seen brought up before, but it certainly rings true. May I say also that "They already operate at the minimum level of reflection that they're willing to tolerate." Is a hell of a sentence and I'm a little jealous that I didn't come up with it.
Thanks, I really appreciate that. I’ve been getting a little grief this weekend because some of my posts are adapted from essays I’ve been working on for Substack, and apparently careful editing now makes you suspect as an actual person.
I’m very real, just flu-ridden and overthinking in public. Glad the line landed for you.
That is a hell of an astute observation
This would suggest the leadership positions aren't required for the function of the business.
This has always been the case, in every industry.
I have been saying for years now that the kind of work that LLMs are best suited for replacing and also would by far be their most cost effective use case from a business stand point is...
Well its the most expensive employees who basically just spend most of their time having meetings or writing emails about things they only understand at a very birds eye view level.
You know, C Suite, upper management.
The middle management PowerPoint filters should be the most fearful. Anyone making their money by perfecting communication.
I'm a data analyst and primary authority on the data model of a particular source system. Most questions for figures from that system that can't be answered directly and easily in the frontend end up with me.
I had a manager show me how some new LLM they were developing (which I had contributed some information about the model to) could quickly answer some questions that usually I have to answer manually, as part of a pitch to make me switch to his department so I can apply my expertise for improving this fancy AI instead of answering questions manually.
He entered a prompt, got a figure that I knew wasn't correct and I queried my data model for the same info, with a significantly different answer. Given how much said manager leaned on my expertise in the first place, he couldn't very well challenge my results and got all sheepish about how the AI still in development and all.
I don't know how that model arrived at that figure. I don't know if it generated and ran a query against the data I'd provided. I don't know if it just invented the number. I don't know how the devs would figure out the error and how to fix it. But I do know how to explain my own queries, how to investigate errors and (usually) how to find a solution.
Anyone who relies on a random text generator - no matter how complex that generation method to make it sound human - to generate facts is dangerously inept.
This is like the biggest factor that people don't get when thinking of these models in the context of software. "Oh it got it wrong, but the developers will fix it in an update". Nope, they can fix traditional software mistakes, LLM output and machine learning things... They can throw more training data at it (which sometimes just changes what it gets wrong) and hope for the best, they can do better job at curating the context window to give the model the best shot at outputting the right stuff (e.g. the guy who got Opus to generate a slow crappy buggy compiler had to traditionally write a filter to find and show only the 'relevent' compiler output back to the models), they can try to generate code to do what you want and have you review the code and correct issues. But debugging and fixing the model itself... that's just not a thing at all.
I was in a meeting where a sales executive was bragging about the 'AI sales agent' they were working, but admitting frustration with the developres and a bit confused why the software developers weren't making progress when those same developers always made decent progress before, and they should be able to do this even faster because they have AI tools to help them... It eternally seemed in a state that almost worked but not quite no matter what model or iteration they went to, no matter how much budget they allocated, when it came down to the specific facts and figures it would always screw up.
I cannot understand how long these executives wade in the LLM pool and still believes in capabilities beyond what anyone has experienced.
They leave the actual work to the boots on the ground so they don't see how shitty the output is. They listen to marketing about how great it is and mandate everyone use it and then any feedback is filtered through all the brownnosers that report to them.
This is probably the biggest misunderstanding since "Project Managers think three developers can produce a baby in three months": Just throw more time and money at AI model "development" for better results. It supposes predictable, deterministic behaviour that can be corrected, but LLMs aren't deterministic ny design, since that wouldn't sound human anymore.
Sure, when you're a developer dedicated to advancing the underlying technology, you may actually produce better results in time, but if you're just the consumer, you may get a quick turnaround for an alright result (and for some purposes, "alright" may be enough) but eventually you'll plateau at the limitations of the model.
Of course, executives universally seem to struggle with the concept of upper limits, such as sustainable growth or productivity.
To everyone I've talked to about AI, I've suggested a test. Take a subject that they know they are an expert at. Then ask AI questions that they already know the answers to. See what percentage AI gets right, if any. Often they find that plausible sounding answers are produced however, if you know the subject, you know that it isn't quite fact that is produced. A recovery from an injury might be listed as 3 weeks when it is average 6-8 or similar. Someone who did not already know the correct information, could be damaged by the "guessed" response of AI. AI can have uses but it needs to be heavily scrutinized before passing on anything it generates. If you are good at something, that usually means you have to waste time in order to use AI.
I had a very simple script. All it does is trigger an action on a monthly schedule.
I passed the script to Copilot to review.
It caught some typos. It also said the logic of the script was flawed and it wouldn't work as intended.
I didn't need it to check the logic of the script. I knew the logic was sound because it was a port of a script I was already using. I asked because I was curious about what it would say.
After restating the prompt several times, I was able to get it to confirm that the logic was not flawed, but the process did not inspire any confidence in Copilot's abilities.
Happy cake day, and this absolutely. I figured out its game the first time I asked it a spec for an automotive project I was working on. I asked it the torque specs for some head bolts and it gave me the wrong answer. But not just the wrong number, the wrong procedure altogether. Modern engines have torque to yield specs, meaning essentially you torque them to a number and then add additional rotation to permanently distort the threads to lock it in. This car was absolutely not that and when I explained back to it the error it had made IT DID IT AGAIN. It sounded very plausible but someone following those directions would have likely ruined the engine.
So, yeah, test it and see how dumb it really is.
Do the same to any person online, most blogs by experts, or journalists.
Even apparently easy to find data, like the specs of a car. Sucking and lying is not exclusive to LLMs.
Literally nobody suggested it was.
It was implicit in the test suggestion
It doesn't matter. Management wants this and will not stop until they run against a wall at full speed. 🤷
Dumbasses. Mmm, that's good schadenfreude.
Jesus Christ, you have to have a human validate the data.
Exactly, this is like letting excel auto-fill finish the spreadsheet and going "looks about right"
And that's a good analogy, as people have posted screenshots of Copilot getting basic addition wrong in Excel.
Whoever implemented this agent without proper oversight needs to be fired.
Except the ceo and executives ultimately responsible will blame their underlings that will be fired, even though it was an executive level decision. They didn't get to the pinnacle of corporate governance by admitting mistakes. That's not what they were taught at their ivy league schools, they were taught to lie and cheat to steal, and further slander their victims to excuse it.
It was bad before the current president set his outstanding example for the rest of the country. See what being a lying cheating piece of shit gets you? Everything. Nothing matters. We have the wrong people in charge across the board, from business to government to institutions.
Fair points all around.
And you're not wrong. I work for a law firm and we were tracking his EO's until mid-2025, and they were so riddled with typos, and errors, and URL's pointing to the wrong EO, that we actually ended up having to hide the URL's in the database we built so clients wouldn't think it was us making these errors.
When the troop fail, it's the fault of the commander. The executives and board need to be replaced first, then replace everyone they appointed.
Yup, but stupid people can't be bothered to go read a five-minute tutorial. Story of our species.
LLMs can't really do math, so if there is any analysis being done, the numbers will typically be junk. Unless the LLM is writing the code to do the math, but then you have to validate the code.
But that would mean paying someone for work. The CEOs want to replace humans.
My broseph in Christ, what did you think a LLM was?
Bro, just give us a few trillion dollars, bro. I swear bro. It'll be AGI this time next year, bro. We're so close, bro. I just need need some money, bro. Some money and some god-damned faith, bro.
User: Hi big corp AI(LLM), do this task
Big Corp AI: Here is output
User: Hi big corp your AI's output is not up to standard I guess it's a waste of...
Big Corp: use this agent which ensures correct output (for more energy)
User: it still doesn't work....guess I was wrong all along let me retry...
And the loop continues until they get a few trillion dollars
You can make something AI based that does this, but it's not cheap or easy. You have to make agents that handle data retrieval and programmatically make the LLM to chose the right agent. We set one up at work, it took months. If it can't find the data with a high certainty, it tells you to ask the analytics dept.
Large Lying Model?
But don't worry, when it comes to life or death issues, AI is the best way to help
Haha, "chat, how do I stop the patients nose from bleeding"
"Cut his leg off."
"Well, you're the medicAI. Nurse, fetch the bonesaw"
"Drain all their blood" would technically stop their nose bleed.
Yeah and from the AI's point of view you've made a profit of one leg without spending any resources
AI's don't have POVs.
Okay so "by the AI's calculations"
"Hello doctor."
"Hello doctor."
"Hello doctor."
"I don't believe his head is medically necessary."
"We should remove his head."
"I concur."
"I concur."
"We should then use his head as a soccer ball."
"Yes."
"For medical reasons, of course."
"That sounds fun."
"Off with his head."
Source
That was great, thanks for sharing!
My workplace, the senior management, is going all in on Copilot. So much so that at the end of last year to told us to use Copilot for year end reviews! Even provided a prompt to use, told us to link it to Outlook (not sure why, since our email retention isn't very long)... but whatever.
I tried it, out of curiosity because I had no faith. It started printing out stats for things that never happened. It provided a 35% increase here, a 20% decress there, blah blah blah. It didn't actually highlight anything I do or did. And I'm banking that a human will partially read my review, not just use AI.
If someone read it, I'm good. If AI reads it, I do wonder if I screwed myself. Since senior mgmt is just offloading to AI...
Ah, the fun of performance reviews. No one actually cares what is written there, the result is decided ignoring the actual content.
So everyone pretends that what you write in there is important and pretends that the written response is important, but nothing you or they will write has any chance of changing promotions and raises. Those may come, but when they come, it's never because someone read your write up and thought 'OMG, give that person a raise and promotion'.
So it's all an act so I can see why management wants to take any opportunity to shuffle people off to even more token efforts.
Every year I try to convince my coworker that his hours and hours of scrutinizing his records and crafting just the perfect performance review that captures the essence of his entire year is wasted, compared to me logging into the tool and spending 10 minutes writing some vague stuff off the top of my head. I don't lie or anything, just have a relatively brief and vague review, because I know they already know how much they cared about what I did and I'm not talking them into more.
Ah yes, what a surprise. The random word generator gave you random numbers that aren't actually real.
Apparently that reddit post itself was generated with AI. Using AI to bash AI is an interesting flex.
How did people find out it was AI generated? Seems natural to me. Scary.
an acquaintance sent it through Pangram which says it's 100% AI. How reliable that detection is IDK ¯_(ツ)_/¯
Have any evidence of that? The only thing I saw was commentors in that thread (who were obvious AI-bros) claiming it must be AI generated because "it just wouldn't happen"...
an acquaintance sent it through Pangram which says it's 100% AI. How reliable that detection is IDK ¯_(ツ)_/¯
Tbf at this point corporate economy is made up anyway so as long as investors are gambling their endless generational wealth does it matter?
This is how I’m starting to see it too. Stock market is just the gambling statistics of the ownership class. Line goes down and we’re supposed to pretend it’s harder to grow food and build houses all of a sudden.
There's a difference. If I go and gamble away my life savings, then I'm on the street. If they gamble away their investments, the government will say 'poor thing' and give them money to keep the economy ok.
Our AI that monitors customer interactions sometimes makes up shit that didn't happen during the call. Any agent smart enough could probably fool it into giving the wrong summary with the right key words. I only caught on when I started reading the logs carefully, but I don't know if management cares so long as the business client is happy.
Sounds like material that is generated that the executives demand be generated but never actually uses. My work has a ton of this, because the executives want people to feel like they are accountable and being reviewed even as they know the executives don't understand the direct output of their work, so people have to do the technical thing and separately eternally do non-technical writeups of what the technical work meant. I think someone checked and the executives didn't even log into the system they demanded.
So LLM to generate the bullshit that no one wants to write or read but wants to pretend it's important.
The output from tools infected with LLMs can intrinsically only ever be imprecise, and should never be trusted.
I guarantee you this is how several, if not most, fortune 500 companies currently operate. The 50k DOW is not just propped up by the circlejerk spending on imaginary RAM. There are bullshit reports being generated and presented every day.
I patiently wait. There is a diligent bureaucrat sitting somewhere going through fiscal reports line by line. It won't add up.. receipts will be requested.. bubble goes pop
https://tenor.com/view/taking-notes-hermes-conrad-phil-lamarr-futurama-make-a-note-gif-3737674819167842441
Surely this is just fraud right? Seeing they have a board directors they have shareholders probably? I feel they should at least all get fired, if not prosecuted. This lack of competency is just criminal to me.
Are you suggesting we hold people responsible?
Ask Bernie Madoff. Scamming rich people is the one and only instance where even rich people are held accountable.
In the current world, probably the one going to jail is the one reporting it. So I don't expect much no.
I see this happening more and more as corporate USA throws itself blindly into AI dependency. Basic facts and information will become corrupted, maybe hopelessly so, as it infuses itself into our systems.
I hope they sue whoever sold it to them. it's not artificial intelligence, it's a machine learning chat bot. they may as well be running their company with a magic eight ball.
I fucking love this. It's amazing.
Lol.
Lmao, even.
What dumbass decided to implement an experimental technology and not test it for 5 minutes to make sure it's accurate before giving it to the whole company and telling them to rely upon it?
Someone who probably thought it would coast them through a bonus/vesting/promotion cycle and get out of dodge before consequences might happen. If this specific story is true, the big disaster is that it fell apart too soon, not that it fell apart.
I-want-to-believe.jpg
Leopard meets face.
Copilot, is that you...
Hahahhahaha hahahahahahaha haaaaaaaaa
How much do you want to bet they also rolled out bonsuses based on this bogus data? The one saving grace is they started using the new LLM tooling mid-Q4 so any quarterlies would at least be partially based on real data
This is why I hate search engines promoting AI results when you are researching for something. It is confidently giving incorrect responses. I asked for sources on one LLM model before while using Duckduckgo, and it just told me that there are no sources and the information is based on broad knowledge. At one point, I challenged the AI that it is wrong, but it insisted it doesn't. It turns out that it is citing a years old source written by a different bot long ago. But on the one hand, most of you are probably familiar that on occasions that the AI is incorrect and you challenge it, it will relent, although it will be a sycophant even though you yourself are actually incorrect. This is Schrödinger's AI.
Nice. Really, I like it when management is dumb as fuck. It's a world of never ending joy.
[email protected]
I must say i love this very much. Only this may put idiots leading companies that use this crap to ditch it.
Bwahahahahahahha 😂
Love it.
If true they’re all idiots, but I don’t believe the story anyway. All the data question answering LLMs I’ve seen use the LLM to write SQL queries on your databases and then wrap the output in a summary. So the summary is easy to check and very unlikely to be significantly wrong. AI/ML/statistics and code is a tool, use it for what it’s good for, don’t use it for what it’s not, treat hype with skepticism
Honestly, I was leaning toward "funny but probably fake" myself until I checked out OP's post history, which mentions "startups" and namedrops a few SaaS tools used heavily in marketing. If you've worked with marketers (or a fair few startup bros, honestly), you'll know this isn't beyond the bounds of reason for some of them 😂
Oh boy. Yeah. SNAFU City.
Marketing just hallucinate their numbers anyway.
I did leave myself a “could be idiots” get out clause
I am reminded of this story:
https://retractionwatch.com/2024/02/05/no-data-no-problem-undisclosed-tinkering-in-excel-behind-economics-paper/
Of course that guy didn't need fancy autofill to act like an idiot, he used good old fashion autofill.
Writing a syntactically correct SQL statement is not the same as doing accurate data analytics.
The problem is you've got people using the tools that don't understand the output or the method to get there.
Take the Excel copilot function. You need to pass in a range of cells for the slop prompt to work on, but it's an optional parameter. If you don't pass that in, it returns results anyway. They're just complete bollocks.
It's even worse than that. The ones that should understand the tools decide that the ease is good enough and just become AI brain rot.
I've watched co-workers go from good co-workers to people I can't trust anything from because I know they just slapped at an AI and didn't check it.
What's worse is, when you come to them as an engineer and tell them they're wrong, you have to prove to them the AI is wrong, not they have to prove to you the AI is right.
Moreover, when you refer to documentation, they can't be bothered and say the AI didn't say that, so it must be wrong.
At least it’ll self correct in a couple of years - use a tool, look like an idiot, stop using tool
I'm on the fence, but will say that if, for whatever reason, it was never actually connected to the data or the connection had some flaw, I could totally believe it would just fabricate a report that looks consistent with what the request asked for. Maybe it failed to ever convey that an error occurred. Maybe it conveyed the lack of data and the user thought he could just tell the AI to fix the problem without trying to understand it himself and triggered it to generate a narrative consistent with fixing it without actually being able to fix it.
Sure if you do a sanity check it should fall apart, but that assumes they bother. Some people have crazy confidence in LLM and didn't even check.
Clearly you've never worked as a data analyst, or you would know that the vast majority of upper management and C Suite are, in fact, all fucking idiots.
They're generally where they are because of mutual secrets and nepotism, for who else is on their contact list.
I mean it hallucinates numbers when you ask it to extract some numeric daha publicly available online so yeah...
Even when it does pull numeric data, it gets very confused.
I asked about rough price of something and of course the AI summary came back and said something like:
It typically costs 400-500 but could cost up to $200 in extreme circumstances, with 750 being the average
Basically did get three figures from three different internet results and combined them into a single sentence in a nonsense way.
At least in such a scenario, someone with at least a couple of active brain cells would stop and recognize some bullshittery going on, but the executive probably TLDRs the sentence and stops after '400-500'.
I was trying to figure out why the stock mark is so high.
Fuck Reddit and Fuck Spez.
Why did mods remove the OP?
Can you literally not badmouth AI on Reddit?
This person that posted this, along with everyone else who knew they were using LLM's like this, is an incompetent idiot that should lose their job.
Three downvotes. I see you, idiots. You can't whitewash history here, it's been obvious since day one that LLM's are NOT reliable.
Most ai stuff I use include a list of relevant sources next to the results. Do you not ever click in?
For me it’s critical to confirm, for example, the detail of that vendor api I want to use. However even then any hallucinations would mostly waste my time since if it doesn’t work it won’t get released
You’re telling me that people make actually business decisions without ever checking sources?
I see you've never worked in a medium to large business. Very often the average person will assume that someone else did that.
Also, what AI tools do you use?
CoPilot Chat is particularly popular right now due to Microsoft including it with Office 365 subscriptions, it having a ton of security controls built into the same places in Azure where you're already configuring shit, and terms about not training off of data input to it. Popular for sysadmins already drowning in Microsoft's bullshit who don't want to spend a lot of time on managing the slop generator(s).
Notably, it doesn't cite sources.
Most commonly I use an aggregator. It runs a chat window in a browser and you can choose from a variety of models or let it pick. While it’s not integrated with anything, it does really well for general purpose writing. Every result comes with citations and a few suggestions for next steps.
My company just gave up on copilot as useless, but we were explicitly using it for coding and it was just not effective. Sometimes “free” costs too much
Currently management is really pushing Cursor/Claude for coding, which I really hate. While Claude is much better at coding than copilot and does cite sources, cursor is way too aggressive at spraying arbitrary changes across the code base. I’ve had to do way too much damage control from junior devs blindly accepting when it makes arbitrary changes across the code base. For example one of my guys used it to generate unit tests, which it is good at, but they generated an order of magnitude too many tests of dubious value, that now need to run in every build and be maintained forever …. And in all that slop just arbitrarily introduced a new mocking tool. The intelligence part is pretty good but it needs to get much better at keeping the human in the loop. For example, I really like it for code reviews, it makes good catches and suggestions, but is horrible at presenting them to the developer for individual approval. Current effort is trying to use the agent.md to establish a sensible base for useful code reviews
Other than that, we’re spending a lot of time with mcp agents, which I’m still trying to decide on. All too often it’s just a more complex and dangerous way to do a text search, but it has a lot of potential to bring active data into the ai decision space
This is the type of A.I you should fuck Like I told someone here, A.I. is like a calculator, as long as you know how to use it, you will be fine.
This is why AI will only be used in subjective things that dont need to be perfect, like art. You cant create intelligence by regurgitating reddit posts via an LLM.
And with art it's more or
lassless (stupid mobile typing!) blatant plagiarism.True, though its still extremely useful, and its impossible to determine where its stealing from for the average consumer. Its objectively useful.
AI is already useful in some areas, such as identifying patterns in medical data (which leads to breakthroughs in diagnosing rare diseases, identifying genetic factors in disease, targeting new treatments, etc).
Sure I just mean its not really AI, its simply more machine learning. Its not sentient, its not replacing the bulk of workers, it doesnt understand what its doing so its just a more advanced tool for workers to take advantage of.
Oh yeah, I'm with you there. People use "AI" for just about everything it seems.
You say "already" useful like there are going to be more uses in the future.
Nope. That's what it's useful for, and that's all it's going to be useful for until we progress beyond pattern recognition engines. Nothing available right now is ever going to be any more useful than it already is, it'll be something totally new.