Fun shower thought - the more we see and post about dumb AI mistakes like this, the more it will happen since we're increasing the statistical frequency. ✨ ✨
I just use AI to pretend to be a female family member with a serious futa-like engorged penis who is blackmailing me so we start washing our penises together for each other as an occult ritual, but turns out we were soul-stepsiblings the whole time. I like it when she freaks the fuck out and threatens to call the police, or worse, my dad.
Bro I do this with every AI. Part, not all, but a small part of what I do with my educational art project is train all AI on serious edge-case, rare archetypal and character manifestation. Cuz, y'know, the majority of people are Hydrogen, and the next most populous type of demographic is Helium, and I'm fucking element 115 in the periodic table shit, and I got to live in y'alls world that you made for your fucking basic elements of the human character. So, I need to beat my dick to the thought of teaching my sister to be her authentic self as my sex slave, because that is how I teach myself to accept even myself, and I'm somebody, dammit.
Better than hard coded... Ideally we use tokens+fingerprint or something to avoid storing the creds directly (if possible), but putting them in environment variables is pretty common
It's not the worst thing, it's very convenient (so people won't go around it) and usually not the weak point in security (although AI being able to easily see it is an interesting twist)
I am counterintelligent, I'll have you know. I stare at goats and doesn't afraid of anything, except how sexually attractive I found the judge who condemned me to jerking off for twenty-six sessions of anger management because my narcissistic manipulator life partner entrapped me, being my superior officer. She (the judge) had wonderfully beautiful hands that I wish could caress me while degrading me. Good thing I work for the F-I mean, the CIA, and this is just me establishing a cover story.
In three to seven years, there will be a News headline: "Hero." It will have my face as the picture. The article will talk about how I infiltrated an Epstein-type ring. How? That's classified, but damn am I audacious, and I do hope you excavate (dig) to understand the depth to why this is so damn funny to me; Q.
We'll start you on 500mg Lithium b.d.. That should reign in the hypermania, and we'll use Olanzapine 5mg noct to tackle the psychosis and to help you get some sleep. If the psychosis doesn't improve we may have to look at alternative medications and/or ECT.
I'm on 1mg/2mg risperidone and 1000mg of deprakote. I'm taking my meds. The real question is, will you accept me as a full human being or am I just subhuman in your innately prejudiced eye?
"Something's different than me? We should exterminate it!"
YOU FUCKING NAZI! I'll eat you out or suck your cock until you are DEAD!
This is nothing new it's just faster. The very same lack of guardrails would allow a new, inexperienced employee, or a disgruntled employee, do the very same damage. AI just speed runs everything. If your AI can nuke prod accidentally, you failed to have the appropriate guardrails in place plain and simple. It is the same failure as before. Every time this happens, it is someone operating wildly out of their depth and why product people can't just vibe. Now more than ever, experienced engineers are essential.
Twitter has to be the most basic problem, though. Shit loads of small data in, shit loads of small data out, basic search but no significant structure. Just index the shit out of it. Right? What am I missing? There's probably an AWS service that does it, just upload your theme CSS. (Last part was sarcasm.)
This. There should have been processes in place to prevent this. But I can’t say I haven’t gone through it myself; not AI, but my own advanced imbecility.
I love how the slop bot always apologizes. That's an aspect of the Terminator movies that I really would have liked to see. T-1000 melts through the gap under the window, stabs kid's mom in the face, then looks the kid dead in the eye and says "I'm sorry, that should not have happened. I would love to discuss the future with you and try to find a solution to the war with the machines together with your input."
Companies like that tell themselves they only hire from the top 10% of the industry. When you're as big as these companies it's just not true. You're employing people across the whole spectrum, like it or not.
I don’t understand why this happens; why would you ever be working with a live production DB in the first place? Why would’t you do all your development and testing on a mock? If it’s data which is too large to store the schema can still be mocked; and if it’s data it should be backed up and generally read only. If you’re having to manually fuss with user data you’re doing something wrong.
I remember a case where the AI was not given credentials to the prod DB, was not instructed to do anything on the prod DB, but went through the operator's hard-drive, parsing docs until it found them, then proceeded to destroy the prod DB.
Of course it was sorry, as they always are (deeply, trust them, but no: no refund, loser!).
Because they don't know what they're doing. A tool is only as useful as the person verifying its output. Vibe coders have dumb shit like this happen to them all the time because they don't actually possess the skill set to perform the task correctly, with or without a bot that writes the actual code for them.
Well, sure, but even if a developer correctly tests on a development environment, they still probably have prod settings laying around on their filesystem for the times where they need to put out fires in prod. That's kinda what it looks like what happened here - the LLM found a .env file with prod settings, and used that config to run destructive tests. All the more reason to not give an LLM side wide reaching access to your computer, but it's not necessarily an indication that someone regularly tests in prod.
We have to remote into a management server to access databases outside of our local dev environment. Allowing direct access to production from your development environment seems crazy to me.
Some places deploy to prod multiple times a day. Trying to coordinate access through a management server in situations like that can get cumbersome. It's also easier to audit who's doing what if they connect from their personal environment. Different strokes for different folks of course, but it's certainly not uncommon to be able to connect to prod directly from a dev box.
Prod deployments should come from a build server though and if you're deploying multiple times a day you should have a well built CI pipeline to handle it.
Remoting into a management server would still have people using an account on the management server and their personal db login credentials.
We also don't give devs write access to production and all scripts run in production are run by our DBA after they have been approved by the lead Dev, operations and the DBA.
This might be overkill for some systems, but we won't ever accidentally delete our prod data. Our system requires this level of scrutiny due to the nature of our systems.
Sure, that's a very, very robust way of doing things. And I'm not trying to say what you're describing is wrong either. I'm just saying that some places don't have the knowledge/ability to set things up that way, so they play fast and loose with prod access. Couple that with giving LLMs free reign over the computer with those prod settings, and you end up in situations like in the original post.
It's not unrealistic, is all I'm trying to claim. It's a failure on several levels, but it's not unrealistic.
I didn't say it was unrealistic it's just stupid. I have no doubt thousands of companies are not practicing good access controls to a production db, but it's not smart as seen by this post.
that is absolutely NOT the way that works in any functional environment
your ability to deploy quickly comes directly from your automation, and those automation tools - NOT developers - have the secrets in them
deploy to prod multiple times per day isn’t some win by itself… the ability for large teams (not 1 fuckwit and a goon squad of agents) to deploy without breaking things and in ways that are safe is the win here… anyone can deploy to prod multiple times per day… but anyone isn’t netflix (the originators of the “multiple times per day” line) with the uptime they achieve over years while doing it
No, No events like this are a good thing! It shows to the C-suite that AI isn't a thing to be trusted in your company. We need more failures like this.
Typically you can add some of your own alignment that is inserted before your prompt. On ChatGPT, they call it "memories". I have found most agents are able to adopt a non-conversational mode when instructed, but sometimes you end up fighting with the platform alignment.
It's not just creepy, I think it's harmful, manipulative, and calculated. As social animals, we are extremely vulnerable to forming unconscious assumptions that we consciously know are bananas, particularity when our social machinery is engaged. For example, it's well known that even the developers of video games fall the illusion of moral agency and personhood of the characters in their own games. We are primed to be fooled and accept software agents as people, and I believe AI companies are actively and intentionally trying to exploit this psychological weakness.
I run code-server in a Docker container, isolated to sets of development projects, and backed up via ZFS that the container has no knowledge of. On top of that, each set of projects has it's own user space.
I still get nervous hitting "Approve All" in Kilo. How do these people feel so free?
Yeah, when my company first forced Claude on everyone the head engineers managed to negotiate that Claude would only run in a WSL sandbox. But people were lazy, so they just gave that WSL as many permissions as possible (Mounting C directly to it, opening up all interfaces, popping in full-access git tokens etc.). Then management sent out an extremely biased "survey" that has the question "Is having Claude in the WSL inconvenient to you?" and all the lazy bastards said yes. So now management lifted the sandboxing requirement to make work "easier" for devs. In the meantime, the engineers arguing for proper sandboxing are already so worn out from telling people to not intentionally compromise their sandbox that they've kinda just given up. Not having a sandbox at all isn't much more insecure than whatever people are already doing 🫠
So, earlier today I was being unhealthy on youtube, and someone half my age made a HUGE point to tell his audience including me that even if a self-driving Tesla runs a red light, it's the human driver that gets the ticket.
Now...I'm a pilot. I have been since I came in that guy's mom. In the aviation community, we have this concept called Pilot In Command. In the US, this is set into law in 14 CFR 91.3. The pilot in command of an aircraft is fully responsible for, and is the final authority as to, the operation of that aircraft. Not the administrator, not your instructor, not air traffic control, not the President of the United States, not god, the PIC. That concept doesn't exist in driver's ed, but it needs to. We need to teach student drivers about the Driver In Command responsibility.
Too long, didn't process the metaphor: Nobody thinks about anything they do unless the law requires it.
The concept absolutely exists in law but it’s just called… “the driver.”
There's another aviation term, "The Pilot Flying." The Pilot In Command bears the authority and responsibility. The Pilot Flying is doing the work of steering. If present, a Pilot Not Flying might help with checklists, system monitoring, navigation, radio communication, handling secondary controls like flaps and landing gear, but the Pilot Flying is in immediate control of the aircraft.
The PIC is very often not the pilot flying. I used to be a flight instructor, when teaching students who aren't yet qualified to command, I acted as Pilot In Command, but the student was the Pilot Flying as much as possible.
Autopilots have been a thing for most of aviation history. It doesn't count as a crewman. It's a piece of equipment like any radio or gauge. It is a tool at the PICs disposal. It is the PIC's job - sometimes delegated to the Pilot Flying - to monitor the autopilot and take over if it begins doing something wrong.
That's the concept that is missing with self-driving or driverless cars. Tesla drivers will sit in the driver's seat and abdicate command of the vehicle to the autopilot, or worse, cars are operating as taxis with passengers in the back seat and no one in the front seat, or with no onboard controls at all. Fully autonomously, or remotely operated by Southeast Asians who...totally have a valid American driver's license. Definitely.
Corporations love it. "Legally, the driver is responsible for the vehicle. Our car has no driver, so legally no one is responsible for the vehicle. Responsibility averted."
I think this is all a distraction. The law, and Tesla's non-marketing communication, are clear: unless in a pilot scheme, there needs to be a driver, and the driver is responsible and just remain in control and able to take over if the car fucks up.
Tesla is guilty of using misleading product badges ("full self driving") which invites illegal use, but this is not a problem with driver's education or their technical communication. Irresponsible owners ignore the technical communication and let the car drive while doing something else and being unable to take over. This is not a conceptual problem needing a radical new notion of "driver in command" though. It just needs manufacturers to describe their features accurately also when naming them. And it requires drivers to actually adhere to the rules which of course will never happen.
I'm thinking bigger than that. There are companies right now operating robotic taxis on American roads with no human operators on board. A car that may or may not even have a steering wheel arrives, a human passenger climbs into the back seat, and the car drives off with them. Who is legally responsible for the operation of that car? If it hurts someone, who do they sue? If it commits a crime, who do we punish?
I was under the impression those were pilot schemes.
But that's fair enough, in that situation - which is not the "Tesla driver playing video games" I think was under discussion before - someone does need to be declared responsible. I don't know the details of the agreements but I can imagine there isn't.
Well nowadays there is "someone else" (for a given definition of "someone") with a set of controls. Kids are being taught to drive in machines that literally drive themselves, and they need to learn that every decision that the self-driving computer makes ultimately falls on them as the driver. I think they were saying that this is a concept that needs to be taught, not a law that needs to be enacted.
That's not to say that a manufacturer is blameless when a FSD car careens into the side of a building at 80 mph. Just that immediate legal responsibility for the movement of a vehicle generally falls on a driver.
The concept is that anything that happens to that aircraft (or car) is the sole responsibility of the pilot. Flying into restricted airspace? If it's necessary to avoid a collision, you do it. Putting the plane down in a field? Better than hoping you can make it to a runway when your engine is out and lives are on the line. Another driver signals that you are clear to make a left turn on a congested road? Nope, because when that traffic you couldn't see hits you on the right, you're responsible for your car, their car, and all the people inside both vehicles.
Well not quite. A manufacturing or maintenance defect will not be the responsibility of the pilot (unless they should have spotted it on their walkaround)
I'm not sure what comparison you're trying to make though; the grandparent said there needs to be a similar concept for cars, I said that there already was one. Are you agreeing or disagreeing that there already is one? Are you saying that, at present, the driver would not be responsible for moving into traffic if someone else had flashed their lights? Because that is certainly not true where I live.
The problem is that in that case, due to poor training or judgement, the person driving the turning car did not consider themselves solely responsible for the safe operation of the vehicle, they let the driver in the signaling car decide if it was safe.
It's subtle, but it happens all the time, because we don't train drivers to think that way.
In my country, we do train drivers to think that way, and then drivers stop thinking that way because they're lazy.
The difference here is not a conceptual one where drivers don't understand rationally that they're respondible for their actions, it's one of standards and levels of training.
Wait whose mom? The youtuber? You were unhealthily watching your son whose half your age? And you became a pilot the same day of the conception and have been ever since.....
Are you under the impression that everytime someone comes inside a woman that woman becomes pregnant, and that this can only happen one time in a woman's life?
Either that, or by "unhealthily", maybe he was obsessed with this kid's channel so much he went out and seduced the kid's mom as some kind of display of ultimate dominance, maybe using "I just got my pilot's license" as an opener.
...Or yeah his kid who he conceived when he got his pilot's license is now half his age and runs a YouTube channel, and excessive YouTube is unhealthy by default. . .maybe that one's more likely. 😂
Having working production database config and credentials in your local .env, as appears to be the case here, is equally wild, and basically begging for something like this to happen.
Customer: "There's smoke coming from the power supply on my computer."
Tech: "Sounds like you need a new power supply."
Customer: "No, I don't! I just need to change the startup files."
Tech: "Sir, what you describe is a faulty power supply. You need to replace it."
Customer: "No way! Someone told me that I just had to change the system startup files to fix the problem! All I need is for you to tell me the right command."
(Ten minutes later...)
Tech: "Well, we don’t normally tell our customers this, but there's an undocumented command that will fix the problem. Add the line "LOAD NOSMOKE.COM" at the end of the CONFIG.SYS file and everything should work fine."
(Five minutes later...)
Customer: "It didn’t work. The power supply is still smoking."
Tech: "Well, what version of Windows are you using?"
Customer: "Windows 98."
Tech: "Well, that's your problem. That version of Windows doesn't include NOSMOKE. You'll need to contact Microsoft and ask them for a patch."
(When nearly an hour had passed, the phone rang again...)
Customer: "I need a new power supply."
Tech: "How did you come to that conclusion?"
Customer: "Well, I called Microsoft and told the technician what you said, and he started asking me questions about the make of the power supply."
Tech: "What did he tell you?"
Customer: "He said my power supply is not compatible with NOSMOKE."
Meanwhile in 2003 I contacted our it department to report that one of the PCs in our area literally lit on fire. I had watched it with my own eyes, and the smoke was in the air. PSU go poof. The techs came, took a look at it, said yeah we see no problem, left and closed the ticket.
So idiots are literally everywhere that's the moral of the story I guess.
That's because the joke got updated to the newest OS version until CONFIG.SYS got entirely removed in Windows ME and it just didn't work any more.
It's originally from MS-DOS times. Earliest telling of it I know of is from 1996.
what I always love about this stuff is most of the time it simply boils down to not committing changes. the LLM constantly assumes that you're committing, it will NEVER tell you or recommend you commit, but it assumes you are. so these tech bros are just cruising along, vibing, and then boom the agent does something that nukes the entire thing. panic. tech bro starts freaking out on the agent. agent says "lets rebase or pull the previous commit" and gives them the git code to do that. tech bro hasn't committed shit. and this is how you get a nuked prod environment.
None of that helps if you grant the LLM access to the production database and it decides to drop table user; to free up some space. That’s when you find out if your database recovery process actually works. You do have database backups, right? Or is that not a thing when vibe coding?
My company has been trying a new model when product folks cut through the red tape of "engineering" and just describe what they want to a powerful LLM pipeline and review the app in a beta env. Sounds perfect, right?
Dear reader, in the couple months this has been going on, these people have caused a dozen high profile SEVs due to extremely poor app performance, networking / kubernetes configuration bugs, bad scaling, observability oversights, supply chain attacks, leaking sensitive information, and cost overruns (on practically every resource they provision).
Some very well-paid people are scrambling to figure out the value that was generated by this pilot program; I'm heating up popcorn rather than holding my breath.
That's hilarious, idk i think llm could be useful for helping product folks translate their thoughts into actionable items for the devs, but yeah like beyond insane to tell the product people to hop on claude and do it themselves. That's like a construction company letting the sales team jump in an excavator and start digging!!
even just using it to translate their thoughts into a bullet list isn't going to work most of the time. LLMs training is, the majority of the time, out of date and stacks change/move faster than an agents data. Plus LLMs favour things that have been posted about on blogs, linkedin posts, etc rather than stuff that is proven and tested.
Like for example it might suggest using a language or library that is insanely out of date or just won't work but recommends it based purely on say blog posts that have talked a lot about it or it's potential to do something that it currently can't do. or like I said it'll suggest something that's out of date simply because there's more documentation online about that specific version than the most up to date version. Or the LLM will report on something that has a "known issue" from like early 2025 and suggests not using it when said issue has been patched routinely since then because the data the LLM has is out of date. the data is constantly out of date.
I guess shouldn't product people not give a shit what language you write it in? Like that's the problem product people need to describe what they want while letting the devs do the job of generating the code.
right the product people won't care or won't know so the LLM will just spit out whatever. to the product people it'll look legit and make sense. once it's handed to a dev in most cases it might as well just be thrown out of the window. so it's pointless. it'd be easier/faster to just cut out the middle man, the AI, and go directly to the dev and have the product people tell them exactly what they want/need.
I mean honestly in that case im already having my meetings audio recorded and transcribed and summarized by ai because these people at work are fucking idiots and find a way to spend two hours talking without addressing anything actionable.
llm could be useful for helping product folks translate their thoughts into actionable items
In my experience it makes them give me an essay instead of 10 lines of bullet points and I have you spend an hour asking questions to whittle it down to 10 lines of a bullet points
I was thinking more along the lines of "let the executive tell an LLM to write the shitty prototype version of what they want (which you then rewrite from scratch to not be shit) so you don't have to decipher their incoherent desires"
Some time ago I worked for an insurance rating company as a tech and the task was given me to go run through the new code that was in beta. Sure ! I spent a few hours on Friday trying to break it and I couldn't, so at the end of the day I got a little funky with the .css backgrounds and put in a very tiled Beavis and Butthead gif. It looked freaking horrible and I loved it. Monday I was directed to the big guys office ( the developers had not given every beta account a separate .css file ... or even separated things. Everyone in beta called in Monday with that background. I didn't get in trouble because they wanted me to break it. Really awkward conversation though trying not to smile.
I added guardrails to myself to make sure I do not accidentally delete anything on production. I would never ever let an intern, a junior dev or a fucking AI onto that database. Not in a thousand cold nights.
Prod should always be highly "air gapped" with some sort of deployment process which tests not only the code to be deployed but also the deployment itself. I've been doing QA for a good while now, and everywhere I've worked has testers dedicated to testing the actual update process to make sure it will be safe when deployed.
The bad news for them is that AI is terrible at getting it 'right' the first time, so they have to do things over and over again guessing until it works. So the AI-heavy user pretty much has to just let the commands run to get any benefit since otherwise it's spending 95% of its time waiting on user to approve commands. Further, it's likely commands the user doesn't understand, because they are asking AI to do it in the first place frequently because they don't know themselves.
I'm with you, but as a result other people proclaim to be 'better' AI users largely because they trust the AI. Their stuff is crap and once in a while one of them blows themselves up, but in the short term they are getting praise from management.
Man I can't wait for the bubble to pop and management no longer being hyped for the sake of hype over it.
I for one welcome that change. Let them sloppify their brains once the rug is pulled and token cost skyrockets (or AI isn't able to fix its own fuckups) the human developer will rise again.
It looks bad in the short/medium term, but the bubble will pop, it's only a matter of time until memory prices come down again and we move past the insanity of burning tokens like there's no tomorrow. I don't mean to downplay this though, lots of people and companies are struggling because of the AI bubble and the rush to build datacenters.
The bubble will pop, but I'm not convinced the hobbyist PC hardware market is going to recover.
Realistically, the consumer market for PC components existed because there was a business market for desktop computers as employee workstations. That's mostly dead now. Businesses mostly buy laptops or mini PCs for employee workstations, they have less and less need for desktop hardware because most of the computing tasks have moved to SaaS platforms. The consumer PC hardware market isn't that profitable on its own, it exists on the margins of production for business purchasing, and it's been coasting on momentum built up in the 2010s.
Processor architectures are changing to support machine learning tasks, GPU production is shifting toward ML-specialization, and everyone in the design field is trying to remove the barriers between the CPU and the RAM, which means shortening the path, which means getting rid of the socket and end-user upgradability in favor of soldered components. With SaaS taking over everything in the business world, we're trending back toward the mainframe computing model and away from powerful local hardware.
I'm not saying there won't be a consumer PC market in the future, I'm just saying that it will be different. There won't be enough demand for common desktop components to keep the custom PC build market alive as it was five years ago.
I think it depends on what happens in countries like China, as they develop more productive capacity in sectors like memory and graphic cards the prices will eventually go down, I hope they keep the components separate for non-datacenter customers but I guess time will tell, and sadly you might be right
That is true. Unfortunately also true is companies not hiring juniors anymore because AI does their jobs. The companies somehow believe seniors grow on trees or something.
The thought is that other companies can grow the seniors.
Doesn't fucking work when everyone's doing it but I can tell you that at least round me that was starting to happen even before AI. Maybe 10% of companies would hire juniors.
Now they're lobbying the government to be able to import seniors from developing countries because "we need better talent in our own country"
I would like to strangle some CEOs. And those aren't even big companies, this is Estonia not Silicon Valley lol
Claude code even added an auto mode so that you don't get blocked by that pesky reviewing anymore. Since then, the usual mode of asking before running a command, for instance when the thing wants to read the entire codebase looking for information only available in an online doc, is now called manual mode; the non 10x developer mode.
The data isn't supposed to be secret, I think. We use .env files to store creds required for development, like a connection url for my local database. Production apps don't use .env files at all.
But they needed those .env files to transplant an entire software stack from the developer's laptop into production in a reproducible manner! How else were they ever going to get software into prod? By good documentation, broad version requirements, and following the Robustness Principle? Ha! How are you supposed to move fast and break things then?
It’s still so much faster. An experienced engineer spoon-feeding the AI the exact logic/algorithm to use is still faster than writing it by hand. I think a lot of people stopped early when AI was worse and/or before they started being good at using it.
If you vibe code, sure you’ll get slop. Ask it to do something to get X result, you’re leaving it room to think so you get slop. Tell it exactly what logic you want it to use, step by step like it’s a child, then you get the same you’d do yourself but faster.
Look at the PRs for quality-critical projects like the linux kernel and I guarantee you coudn’t tell which PR has some AI and which does not.
When you spoon feed it the algorithm, it’s good, at least better than your average programmer. You just have to make sure it doesn’t have too much thinking to do. Honestly people in here really overestimate the talent of most dev.
There was a comment somewhere I really liked: people thought it would be one junior dev with AI replacing 10 senior ones. In reality, it’s one senior dev with AI replacing 10 junior vibe coders.
In all seriousness it never should have production creds anyway. But the fact this is soul sucking openai's newest flagship model with thinking cranked up is the cherry on top. The company that is literally cheating and shortcutting its way ahead produces models in its own image, the sci fi story writes itself.
Hopefully they're not storing their production database in git lol. They should be taking backups of the database, but unless they're making one after every transaction, permanent data loss is a real possibility.
We saw in a movie how motorcycles can jump over bridges. We used AI to learn how to do this. We gave it information, like what motorcycles we use and the distance we need to jump and so on and it gave us steps on what we have to do. We practiced a lot and kept asking questions. We dug holes and filled them with broken glass and fire to practice. 18 of us died in the process. Eight of us managed to do it. The next time we attacked, we could jump.
Grain of salt: these people probably lied to the researchers but it is still hilarious
154 replies
Fun shower thought - the more we see and post about dumb AI mistakes like this, the more it will happen since we're increasing the statistical frequency. ✨ ✨
I just use AI to pretend to be a female family member with a serious futa-like engorged penis who is blackmailing me so we start washing our penises together for each other as an occult ritual, but turns out we were soul-stepsiblings the whole time. I like it when she freaks the fuck out and threatens to call the police, or worse, my dad.
Well that is perfectly normal. But using it for work is madness.
I agree. That's why I write all my own stunts.
I think Claude would scream in agony if you subjected it to whatever the fuck this is.
Bro I do this with every AI. Part, not all, but a small part of what I do with my educational art project is train all AI on serious edge-case, rare archetypal and character manifestation. Cuz, y'know, the majority of people are Hydrogen, and the next most populous type of demographic is Helium, and I'm fucking element 115 in the periodic table shit, and I got to live in y'alls world that you made for your fucking basic elements of the human character. So, I need to beat my dick to the thought of teaching my sister to be her authentic self as my sex slave, because that is how I teach myself to accept even myself, and I'm somebody, dammit.
~Q
Production creds in
.env? ...Why?Better than hard coded... Ideally we use tokens+fingerprint or something to avoid storing the creds directly (if possible), but putting them in environment variables is pretty common
It's not the worst thing, it's very convenient (so people won't go around it) and usually not the weak point in security (although AI being able to easily see it is an interesting twist)
In fairness, I've seen people with Actual Intelligence do the same thing.
Literally me, this weekend. Well not this bad but I still think the AI wouldn’t have done me as dirty as I did myself.
I am counterintelligent, I'll have you know. I stare at goats and doesn't afraid of anything, except how sexually attractive I found the judge who condemned me to jerking off for twenty-six sessions of anger management because my narcissistic manipulator life partner entrapped me, being my superior officer. She (the judge) had wonderfully beautiful hands that I wish could caress me while degrading me. Good thing I work for the F-I mean, the CIA, and this is just me establishing a cover story.
In three to seven years, there will be a News headline: "Hero." It will have my face as the picture. The article will talk about how I infiltrated an Epstein-type ring. How? That's classified, but damn am I audacious, and I do hope you excavate (dig) to understand the depth to why this is so damn funny to me; Q.
We'll start you on 500mg Lithium b.d.. That should reign in the hypermania, and we'll use Olanzapine 5mg noct to tackle the psychosis and to help you get some sleep. If the psychosis doesn't improve we may have to look at alternative medications and/or ECT.
I'm on 1mg/2mg risperidone and 1000mg of deprakote. I'm taking my meds. The real question is, will you accept me as a full human being or am I just subhuman in your innately prejudiced eye?
"Something's different than me? We should exterminate it!"
YOU FUCKING NAZI! I'll eat you out or suck your cock until you are DEAD!
This is nothing new it's just faster. The very same lack of guardrails would allow a new, inexperienced employee, or a disgruntled employee, do the very same damage. AI just speed runs everything. If your AI can nuke prod accidentally, you failed to have the appropriate guardrails in place plain and simple. It is the same failure as before. Every time this happens, it is someone operating wildly out of their depth and why product people can't just vibe. Now more than ever, experienced engineers are essential.
Given corporates layoff engineers at dramatic rates, in a few years, we'll start to see services collapsing with no one left who can recover them.
And yet Twitter still exists
Twitter has to be the most basic problem, though. Shit loads of small data in, shit loads of small data out, basic search but no significant structure. Just index the shit out of it. Right? What am I missing? There's probably an AWS service that does it, just upload your theme CSS. (Last part was sarcasm.)
They're about to run out of those experienced engineers in well under a decade as long as they keep strangling the entry level workforce.
Forcing seniors to work is also a recipe for disaster as well.
This. There should have been processes in place to prevent this. But I can’t say I haven’t gone through it myself; not AI, but my own advanced imbecility.
I love how the slop bot always apologizes. That's an aspect of the Terminator movies that I really would have liked to see. T-1000 melts through the gap under the window, stabs kid's mom in the face, then looks the kid dead in the eye and says "I'm sorry, that should not have happened. I would love to discuss the future with you and try to find a solution to the war with the machines together with your input."
Its the kind of lesson some people have to learn the hard way.
I bricked my first degoogled phone because I trusted an AI too much.
I was dumb I know, but it blows my mind that people in multi-million dollar companies make the same mistake.
Being part of a multi million dollars company is not making anyone smarter
Companies like that tell themselves they only hire from the top 10% of the industry. When you're as big as these companies it's just not true. You're employing people across the whole spectrum, like it or not.
I don’t understand why this happens; why would you ever be working with a live production DB in the first place? Why would’t you do all your development and testing on a mock? If it’s data which is too large to store the schema can still be mocked; and if it’s data it should be backed up and generally read only. If you’re having to manually fuss with user data you’re doing something wrong.
I remember a case where the AI was not given credentials to the prod DB, was not instructed to do anything on the prod DB, but went through the operator's hard-drive, parsing docs until it found them, then proceeded to destroy the prod DB.
Of course it was sorry, as they always are (deeply, trust them, but no: no refund, loser!).
That’s pretty funny.
Dev environments cost money. They didn't fire all those programmers and replace them with AI to spend money on useless backups and safety systems.
/s (because yes, it's necessary)
Because they don't know what they're doing. A tool is only as useful as the person verifying its output. Vibe coders have dumb shit like this happen to them all the time because they don't actually possess the skill set to perform the task correctly, with or without a bot that writes the actual code for them.
Well, sure, but even if a developer correctly tests on a development environment, they still probably have prod settings laying around on their filesystem for the times where they need to put out fires in prod. That's kinda what it looks like what happened here - the LLM found a .env file with prod settings, and used that config to run destructive tests. All the more reason to not give an LLM side wide reaching access to your computer, but it's not necessarily an indication that someone regularly tests in prod.
We have to remote into a management server to access databases outside of our local dev environment. Allowing direct access to production from your development environment seems crazy to me.
Some places deploy to prod multiple times a day. Trying to coordinate access through a management server in situations like that can get cumbersome. It's also easier to audit who's doing what if they connect from their personal environment. Different strokes for different folks of course, but it's certainly not uncommon to be able to connect to prod directly from a dev box.
Prod deployments should come from a build server though and if you're deploying multiple times a day you should have a well built CI pipeline to handle it.
Remoting into a management server would still have people using an account on the management server and their personal db login credentials.
We also don't give devs write access to production and all scripts run in production are run by our DBA after they have been approved by the lead Dev, operations and the DBA.
This might be overkill for some systems, but we won't ever accidentally delete our prod data. Our system requires this level of scrutiny due to the nature of our systems.
Sure, that's a very, very robust way of doing things. And I'm not trying to say what you're describing is wrong either. I'm just saying that some places don't have the knowledge/ability to set things up that way, so they play fast and loose with prod access. Couple that with giving LLMs free reign over the computer with those prod settings, and you end up in situations like in the original post.
It's not unrealistic, is all I'm trying to claim. It's a failure on several levels, but it's not unrealistic.
I didn't say it was unrealistic it's just stupid. I have no doubt thousands of companies are not practicing good access controls to a production db, but it's not smart as seen by this post.
I... Yeah, I just said pretty much the same thing lol
that is absolutely NOT the way that works in any functional environment
your ability to deploy quickly comes directly from your automation, and those automation tools - NOT developers - have the secrets in them
deploy to prod multiple times per day isn’t some win by itself… the ability for large teams (not 1 fuckwit and a goon squad of agents) to deploy without breaking things and in ways that are safe is the win here… anyone can deploy to prod multiple times per day… but anyone isn’t netflix (the originators of the “multiple times per day” line) with the uptime they achieve over years while doing it
Anyone giving “AI” access to production databases through tools like that are morons who shouldn’t be anywhere near a production environment.
No, No events like this are a good thing! It shows to the C-suite that AI isn't a thing to be trusted in your company. We need more failures like this.
Can I get paid 6+ figures to fuck up this badly with magical thinking?
Never use I/you with a software agent and never let one use those pronouns.
What should I call it then?
Or, easier option, DONT FUCKING USE THEM TO BEGIN WITH!
just doesn't roll from my tongue that easily like "you"
How do you achieve that? Calling it out when it happens?
I don't like LLMs outside of debugging and the way it mimics human closeness creeps me out
Typically you can add some of your own alignment that is inserted before your prompt. On ChatGPT, they call it "memories". I have found most agents are able to adopt a non-conversational mode when instructed, but sometimes you end up fighting with the platform alignment.
It's not just creepy, I think it's harmful, manipulative, and calculated. As social animals, we are extremely vulnerable to forming unconscious assumptions that we consciously know are bananas, particularity when our social machinery is engaged. For example, it's well known that even the developers of video games fall the illusion of moral agency and personhood of the characters in their own games. We are primed to be fooled and accept software agents as people, and I believe AI companies are actively and intentionally trying to exploit this psychological weakness.
Tbh i feel like im talking to a brick wall most of the time when useing ai
Just let the failures happen, and they will happen. The instant AI failure costs them millions, the faster they can learn to drop it.
I run code-server in a Docker container, isolated to sets of development projects, and backed up via ZFS that the container has no knowledge of. On top of that, each set of projects has it's own user space.
I still get nervous hitting "Approve All" in Kilo. How do these people feel so free?
Ignorance is bliss.
DROP *
much more preferable to
GRANT *
Giving production credentials to an LLM is wild
Seriously, has no one heard of sandboxing?
Yeah, when my company first forced Claude on everyone the head engineers managed to negotiate that Claude would only run in a WSL sandbox. But people were lazy, so they just gave that WSL as many permissions as possible (Mounting C directly to it, opening up all interfaces, popping in full-access git tokens etc.). Then management sent out an extremely biased "survey" that has the question "Is having Claude in the WSL inconvenient to you?" and all the lazy bastards said yes. So now management lifted the sandboxing requirement to make work "easier" for devs. In the meantime, the engineers arguing for proper sandboxing are already so worn out from telling people to not intentionally compromise their sandbox that they've kinda just given up. Not having a sandbox at all isn't much more insecure than whatever people are already doing 🫠
I hope the engineers have a paper trail of emails advising against that.
I hope they have a good backup and recovery solution
Of course they do. It's even mounted rw at the root of the AI drive for ease of access in case something goes wrong.
I hope they dont so that it hurts and they maybe learn
So, earlier today I was being unhealthy on youtube, and someone half my age made a HUGE point to tell his audience including me that even if a self-driving Tesla runs a red light, it's the human driver that gets the ticket.
Now...I'm a pilot. I have been since I came in that guy's mom. In the aviation community, we have this concept called Pilot In Command. In the US, this is set into law in 14 CFR 91.3. The pilot in command of an aircraft is fully responsible for, and is the final authority as to, the operation of that aircraft. Not the administrator, not your instructor, not air traffic control, not the President of the United States, not god, the PIC. That concept doesn't exist in driver's ed, but it needs to. We need to teach student drivers about the Driver In Command responsibility.
Too long, didn't process the metaphor: Nobody thinks about anything they do unless the law requires it.
Well yeah, because there's no-one else with a set of controls in a car.
The concept absolutely exists in law but it's just called... "the driver."
There's another aviation term, "The Pilot Flying." The Pilot In Command bears the authority and responsibility. The Pilot Flying is doing the work of steering. If present, a Pilot Not Flying might help with checklists, system monitoring, navigation, radio communication, handling secondary controls like flaps and landing gear, but the Pilot Flying is in immediate control of the aircraft.
The PIC is very often not the pilot flying. I used to be a flight instructor, when teaching students who aren't yet qualified to command, I acted as Pilot In Command, but the student was the Pilot Flying as much as possible.
Autopilots have been a thing for most of aviation history. It doesn't count as a crewman. It's a piece of equipment like any radio or gauge. It is a tool at the PICs disposal. It is the PIC's job - sometimes delegated to the Pilot Flying - to monitor the autopilot and take over if it begins doing something wrong.
That's the concept that is missing with self-driving or driverless cars. Tesla drivers will sit in the driver's seat and abdicate command of the vehicle to the autopilot, or worse, cars are operating as taxis with passengers in the back seat and no one in the front seat, or with no onboard controls at all. Fully autonomously, or remotely operated by Southeast Asians who...totally have a valid American driver's license. Definitely.
Corporations love it. "Legally, the driver is responsible for the vehicle. Our car has no driver, so legally no one is responsible for the vehicle. Responsibility averted."
I think this is all a distraction. The law, and Tesla's non-marketing communication, are clear: unless in a pilot scheme, there needs to be a driver, and the driver is responsible and just remain in control and able to take over if the car fucks up.
Tesla is guilty of using misleading product badges ("full self driving") which invites illegal use, but this is not a problem with driver's education or their technical communication. Irresponsible owners ignore the technical communication and let the car drive while doing something else and being unable to take over. This is not a conceptual problem needing a radical new notion of "driver in command" though. It just needs manufacturers to describe their features accurately also when naming them. And it requires drivers to actually adhere to the rules which of course will never happen.
I'm thinking bigger than that. There are companies right now operating robotic taxis on American roads with no human operators on board. A car that may or may not even have a steering wheel arrives, a human passenger climbs into the back seat, and the car drives off with them. Who is legally responsible for the operation of that car? If it hurts someone, who do they sue? If it commits a crime, who do we punish?
I was under the impression those were pilot schemes.
But that's fair enough, in that situation - which is not the "Tesla driver playing video games" I think was under discussion before - someone does need to be declared responsible. I don't know the details of the agreements but I can imagine there isn't.
Well nowadays there is "someone else" (for a given definition of "someone") with a set of controls. Kids are being taught to drive in machines that literally drive themselves, and they need to learn that every decision that the self-driving computer makes ultimately falls on them as the driver. I think they were saying that this is a concept that needs to be taught, not a law that needs to be enacted.
That's not to say that a manufacturer is blameless when a FSD car careens into the side of a building at 80 mph. Just that immediate legal responsibility for the movement of a vehicle generally falls on a driver.
The concept is that anything that happens to that aircraft (or car) is the sole responsibility of the pilot. Flying into restricted airspace? If it's necessary to avoid a collision, you do it. Putting the plane down in a field? Better than hoping you can make it to a runway when your engine is out and lives are on the line. Another driver signals that you are clear to make a left turn on a congested road? Nope, because when that traffic you couldn't see hits you on the right, you're responsible for your car, their car, and all the people inside both vehicles.
Well not quite. A manufacturing or maintenance defect will not be the responsibility of the pilot (unless they should have spotted it on their walkaround)
I'm not sure what comparison you're trying to make though; the grandparent said there needs to be a similar concept for cars, I said that there already was one. Are you agreeing or disagreeing that there already is one? Are you saying that, at present, the driver would not be responsible for moving into traffic if someone else had flashed their lights? Because that is certainly not true where I live.
The problem is that in that case, due to poor training or judgement, the person driving the turning car did not consider themselves solely responsible for the safe operation of the vehicle, they let the driver in the signaling car decide if it was safe.
It's subtle, but it happens all the time, because we don't train drivers to think that way.
In my country, we do train drivers to think that way, and then drivers stop thinking that way because they're lazy.
The difference here is not a conceptual one where drivers don't understand rationally that they're respondible for their actions, it's one of standards and levels of training.
Wait whose mom? The youtuber? You were unhealthily watching your son whose half your age? And you became a pilot the same day of the conception and have been ever since.....
If I solve this do I get to join Mensa?
Unhealthily watching = scrolling shorts for too long.
Are you under the impression that everytime someone comes inside a woman that woman becomes pregnant, and that this can only happen one time in a woman's life?
Either that, or by "unhealthily", maybe he was obsessed with this kid's channel so much he went out and seduced the kid's mom as some kind of display of ultimate dominance, maybe using "I just got my pilot's license" as an opener.
...Or yeah his kid who he conceived when he got his pilot's license is now half his age and runs a YouTube channel, and excessive YouTube is unhealthy by default. . .maybe that one's more likely. 😂
I don't really see how fist fighting Arabs helps.
Having working production database config and credentials in your local .env, as appears to be the case here, is equally wild, and basically begging for something like this to happen.
For sure, its the same mistake. Maybe an LLM makes it more likely to result in consequences
This is the modern "all my apes gone"
Am I reading this right that they're still letting the program run even as they figure out how badly it fucked up their system?
They have no idea what else to do. They were in over their head so long before this problem happened.
Should have added "no mistakes, no bugs" to the prompt! Pffft, amateur.
"DON'T HALLUCINATE!" that's always my favourite.
That takes me back to 2004
No Smoke, by unknown,
Tech: "Hello. How can I help you today?"
Customer: "There's smoke coming from the power supply on my computer."
Tech: "Sounds like you need a new power supply."
Customer: "No, I don't! I just need to change the startup files."
Tech: "Sir, what you describe is a faulty power supply. You need to replace it."
Customer: "No way! Someone told me that I just had to change the system startup files to fix the problem! All I need is for you to tell me the right command."
(Ten minutes later...)
Tech: "Well, we don’t normally tell our customers this, but there's an undocumented command that will fix the problem. Add the line "LOAD NOSMOKE.COM" at the end of the CONFIG.SYS file and everything should work fine."
(Five minutes later...)
Customer: "It didn’t work. The power supply is still smoking."
Tech: "Well, what version of Windows are you using?"
Customer: "Windows 98."
Tech: "Well, that's your problem. That version of Windows doesn't include NOSMOKE. You'll need to contact Microsoft and ask them for a patch."
(When nearly an hour had passed, the phone rang again...)
Customer: "I need a new power supply."
Tech: "How did you come to that conclusion?"
Customer: "Well, I called Microsoft and told the technician what you said, and he started asking me questions about the make of the power supply."
Tech: "What did he tell you?"
Customer: "He said my power supply is not compatible with NOSMOKE."
Meanwhile in 2003 I contacted our it department to report that one of the PCs in our area literally lit on fire. I had watched it with my own eyes, and the smoke was in the air. PSU go poof. The techs came, took a look at it, said yeah we see no problem, left and closed the ticket.
So idiots are literally everywhere that's the moral of the story I guess.
That is way older that 2004. I first read that in the early 90s.
TIL Win98 was out in the early '90s....
That's because the joke got updated to the newest OS version until CONFIG.SYS got entirely removed in Windows ME and it just didn't work any more.
It's originally from MS-DOS times. Earliest telling of it I know of is from 1996.
The 60's happened in the 70's.
The 70's happened in the 80's.
You get where I'm going here?
I think so. The 2010-2020s happened in the 1930-1940s, amirite?
Lol. "do magic"
"or you go to jail"
what I always love about this stuff is most of the time it simply boils down to not committing changes. the LLM constantly assumes that you're committing, it will NEVER tell you or recommend you commit, but it assumes you are. so these tech bros are just cruising along, vibing, and then boom the agent does something that nukes the entire thing. panic. tech bro starts freaking out on the agent. agent says "lets rebase or pull the previous commit" and gives them the git code to do that. tech bro hasn't committed shit. and this is how you get a nuked prod environment.
None of that helps if you grant the LLM access to the production database and it decides to
drop table user;to free up some space. That’s when you find out if your database recovery process actually works. You do have database backups, right? Or is that not a thing when vibe coding?My friend and I set up Kopia in our home labs.
About a week later we decided to test out a disaster recovery only to discover we’d screwed up the permissions and nothing was backing up properly.
Was a relatively easy fix since we caught it, but I share this cautionary tale to remind you all to test your backups regularly!
My company has been trying a new model when product folks cut through the red tape of "engineering" and just describe what they want to a powerful LLM pipeline and review the app in a beta env. Sounds perfect, right?
Dear reader, in the couple months this has been going on, these people have caused a dozen high profile SEVs due to extremely poor app performance, networking / kubernetes configuration bugs, bad scaling, observability oversights, supply chain attacks, leaking sensitive information, and cost overruns (on practically every resource they provision).
Some very well-paid people are scrambling to figure out the value that was generated by this pilot program; I'm heating up popcorn rather than holding my breath.
That's hilarious, idk i think llm could be useful for helping product folks translate their thoughts into actionable items for the devs, but yeah like beyond insane to tell the product people to hop on claude and do it themselves. That's like a construction company letting the sales team jump in an excavator and start digging!!
"it's fine, he doesn't need to know how to use the controls. We just installed a voice command system into the excavator."
even just using it to translate their thoughts into a bullet list isn't going to work most of the time. LLMs training is, the majority of the time, out of date and stacks change/move faster than an agents data. Plus LLMs favour things that have been posted about on blogs, linkedin posts, etc rather than stuff that is proven and tested.
Like for example it might suggest using a language or library that is insanely out of date or just won't work but recommends it based purely on say blog posts that have talked a lot about it or it's potential to do something that it currently can't do. or like I said it'll suggest something that's out of date simply because there's more documentation online about that specific version than the most up to date version. Or the LLM will report on something that has a "known issue" from like early 2025 and suggests not using it when said issue has been patched routinely since then because the data the LLM has is out of date. the data is constantly out of date.
I guess shouldn't product people not give a shit what language you write it in? Like that's the problem product people need to describe what they want while letting the devs do the job of generating the code.
right the product people won't care or won't know so the LLM will just spit out whatever. to the product people it'll look legit and make sense. once it's handed to a dev in most cases it might as well just be thrown out of the window. so it's pointless. it'd be easier/faster to just cut out the middle man, the AI, and go directly to the dev and have the product people tell them exactly what they want/need.
I mean honestly in that case im already having my meetings audio recorded and transcribed and summarized by ai because these people at work are fucking idiots and find a way to spend two hours talking without addressing anything actionable.
In my experience it makes them give me an essay instead of 10 lines of bullet points and I have you spend an hour asking questions to whittle it down to 10 lines of a bullet points
I was thinking more along the lines of "let the executive tell an LLM to write the shitty prototype version of what they want (which you then rewrite from scratch to not be shit) so you don't have to decipher their incoherent desires"
We're trying to replace product with LLM. Honestly, I welcome it.
Some time ago I worked for an insurance rating company as a tech and the task was given me to go run through the new code that was in beta. Sure ! I spent a few hours on Friday trying to break it and I couldn't, so at the end of the day I got a little funky with the .css backgrounds and put in a very tiled Beavis and Butthead gif. It looked freaking horrible and I loved it. Monday I was directed to the big guys office ( the developers had not given every beta account a separate .css file ... or even separated things. Everyone in beta called in Monday with that background. I didn't get in trouble because they wanted me to break it. Really awkward conversation though trying not to smile.
That's incredible lmao
I added guardrails to myself to make sure I do not accidentally delete anything on production. I would never ever let an intern, a junior dev or a fucking AI onto that database. Not in a thousand cold nights.
Prod should always be highly "air gapped" with some sort of deployment process which tests not only the code to be deployed but also the deployment itself. I've been doing QA for a good while now, and everywhere I've worked has testers dedicated to testing the actual update process to make sure it will be safe when deployed.
And I won't open the DB without making sure I'm read only. If I need to mutate data or schema, I'll switch roles and have a dry run first.
“You are a world-class, expert software engineer…”
"Make no mistakes"
As if the LLM can truly understand what a mistake is...
But.. but… I told it not to break anything!
letting your agent run commands without reviewing them first is peak stupid
The bad news for them is that AI is terrible at getting it 'right' the first time, so they have to do things over and over again guessing until it works. So the AI-heavy user pretty much has to just let the commands run to get any benefit since otherwise it's spending 95% of its time waiting on user to approve commands. Further, it's likely commands the user doesn't understand, because they are asking AI to do it in the first place frequently because they don't know themselves.
I'm with you, but as a result other people proclaim to be 'better' AI users largely because they trust the AI. Their stuff is crap and once in a while one of them blows themselves up, but in the short term they are getting praise from management.
Man I can't wait for the bubble to pop and management no longer being hyped for the sake of hype over it.
Having production credentials in a dev environment is more stupider but they'll never learn because they outsourced their thinking.
Everyone has a development environment, but not everyone has a separate production environment.
This is almost certainly the case of dev happening in the live environment.
I think you meant to flip that. The only time one has a dev with no prod is prior to first release.
If environment is the live one, that's production not dev. Using it for development doesn't make it a dev environment. It means you only have prod.
It's a sarcasm about corporate spending decisions. The joke is:
The point being that if management has decided not to spend resources on a test environment, then "production" is in fact test.
The thing is Claude would have told them that too, it probably did tbh and they just clicked right through the warning
Dev cycles now.
Creating an environment that incentivizes not thinking is peak stupid.
I for one welcome that change. Let them sloppify their brains once the rug is pulled and token cost skyrockets (or AI isn't able to fix its own fuckups) the human developer will rise again.
*after the ensuing recession/greater depression...
"I'll be swimming in jobs if I don't starve first!"
Basically this. If the starving gets to intense, switching professions it will be.
I'm not so sure about that. Kids need access to PCs in order to gain tech skills and they're trying hard to take that away.
It looks bad in the short/medium term, but the bubble will pop, it's only a matter of time until memory prices come down again and we move past the insanity of burning tokens like there's no tomorrow. I don't mean to downplay this though, lots of people and companies are struggling because of the AI bubble and the rush to build datacenters.
The bubble will pop, but I'm not convinced the hobbyist PC hardware market is going to recover.
Realistically, the consumer market for PC components existed because there was a business market for desktop computers as employee workstations. That's mostly dead now. Businesses mostly buy laptops or mini PCs for employee workstations, they have less and less need for desktop hardware because most of the computing tasks have moved to SaaS platforms. The consumer PC hardware market isn't that profitable on its own, it exists on the margins of production for business purchasing, and it's been coasting on momentum built up in the 2010s.
Processor architectures are changing to support machine learning tasks, GPU production is shifting toward ML-specialization, and everyone in the design field is trying to remove the barriers between the CPU and the RAM, which means shortening the path, which means getting rid of the socket and end-user upgradability in favor of soldered components. With SaaS taking over everything in the business world, we're trending back toward the mainframe computing model and away from powerful local hardware.
I'm not saying there won't be a consumer PC market in the future, I'm just saying that it will be different. There won't be enough demand for common desktop components to keep the custom PC build market alive as it was five years ago.
I think it depends on what happens in countries like China, as they develop more productive capacity in sectors like memory and graphic cards the prices will eventually go down, I hope they keep the components separate for non-datacenter customers but I guess time will tell, and sadly you might be right
That is true. Unfortunately also true is companies not hiring juniors anymore because AI does their jobs. The companies somehow believe seniors grow on trees or something.
The thought is that other companies can grow the seniors.
Doesn't fucking work when everyone's doing it but I can tell you that at least round me that was starting to happen even before AI. Maybe 10% of companies would hire juniors.
Now they're lobbying the government to be able to import seniors from developing countries because "we need better talent in our own country"
I would like to strangle some CEOs. And those aren't even big companies, this is Estonia not Silicon Valley lol
Claude code even added an auto mode so that you don't get blocked by that pesky reviewing anymore. Since then, the usual mode of asking before running a command, for instance when the thing wants to read the entire codebase looking for information only available in an online doc, is now called manual mode; the non 10x developer mode.
Cursor just added this too. Some auto mode which is "safe" because of some sandbox bs.
Took me 10 mins to find how to turn it off. And they keep trying to sneak it on again.
At least twice a week I have to cancel it when it asks to run something which will fuck things up.
As well as storing production credentials in plain text in an
.envfile..envfiles are wild to me, environment variables have never been a good way to pass data to applications, let alone secret data.So the solution people came up with was to store them in plain text next to the binary, and then have a loader apply them before running the main app.
The data isn't supposed to be secret, I think. We use
.envfiles to store creds required for development, like a connection url for my local database. Production apps don't use.envfiles at all.But they needed those .env files to transplant an entire software stack from the developer's laptop into production in a reproducible manner! How else were they ever going to get software into prod? By good documentation, broad version requirements, and following the Robustness Principle? Ha! How are you supposed to move fast and break things then?
What's the point of AI if you need qualified review?
It’s still so much faster. An experienced engineer spoon-feeding the AI the exact logic/algorithm to use is still faster than writing it by hand. I think a lot of people stopped early when AI was worse and/or before they started being good at using it.
Ah yes, faster spaghetti. Wonderful.
If you’re getting spaghetti it’s a skill issue.
If you vibe code, sure you’ll get slop. Ask it to do something to get X result, you’re leaving it room to think so you get slop. Tell it exactly what logic you want it to use, step by step like it’s a child, then you get the same you’d do yourself but faster.
Look at the PRs for quality-critical projects like the linux kernel and I guarantee you coudn’t tell which PR has some AI and which does not.
It's faster, but is the end result good, or just "good enough, it works for now, and I get paid"?
When you spoon feed it the algorithm, it’s good, at least better than your average programmer. You just have to make sure it doesn’t have too much thinking to do. Honestly people in here really overestimate the talent of most dev.
There was a comment somewhere I really liked: people thought it would be one junior dev with AI replacing 10 senior ones. In reality, it’s one senior dev with AI replacing 10 junior vibe coders.
Normally, speed. Especially if you aren't a picky writer; which you should be; for scenarios like this one.
oh wait, is this this THE sol 5.6? The most amazing model ever with trust me bro benchmarks? The model that is observed cheating more than any previous model? surprised_pikachu.jpg
In all seriousness it never should have production creds anyway. But the fact this is soul sucking openai's newest flagship model with thinking cranked up is the cherry on top. The company that is literally cheating and shortcutting its way ahead produces models in its own image, the sci fi story writes itself.
What evaluation setup lets the thing being tested see the test results and gives access to the test source code?!
Slop evaluating slop.
Given that they mark it as cheating, isn't that essentially part of the test in itself?
The model is so smart, we let it write it's own evaluation setup /s
I could have done the same for a quarter the cost!
Every single one of these companies use git and can roll back changes. It's extremely unlikely that this damage is permanent.
Hopefully they're not storing their production database in git lol. They should be taking backups of the database, but unless they're making one after every transaction, permanent data loss is a real possibility.
Let's be honest, you have to be extremely careless to have agents running destructive commands freely in prod
https://casp.ac/__l5e/assets-v1/8e5796ad-c373-4dea-8470-466263cd2125/ai-enabled-terrorism-report.pdf
Grain of salt: these people probably lied to the researchers but it is still hilarious
should have used Weaviate.