I've been processing a couple of billion rows of data on my machine, the fans didn't even come on. WTF are they teaching "experts" these days, or has Elmo only hired people who claim that they can "wrangle data" and say "yes" ?
Even if querying data was processing-heavy and even if somehow the ‘hard drive’ got warm during this, then there still would need to be a hardware defect in order for the drive to overheat.
Yes, but this may be a symptom of an issue I've been seeing with younger programmers; they've siloed themselves so specifically into whatever programming they "specialize" in, that they become absolutely useless at dealing with absolutely anything else related to their job. And exasperating this issue is the fact that they've grown up with systems that "just work". Windows, iOS, and android are all at the point where fucking around with hardware issues is very uncommon for the average person.
Asking this guy to solve a hardware problem is like asking hime to tune a carburetor. He likely has not the slightest clue how to start.
In my experience, a lot of software dev degree paths basically don't even have relevant classes on hardware at all. Classes on hardware are all in IT Helpdesk and Network Admin degree paths whereas the software dev students are dumped straight into Visual Studio right off the bat with no relevant understanding of the underlying hardware or OS.
My experience does not reflect yours. Computer Architecture, Discrete Math (logic gate math), and Operating System Concepts were all required classes in my CS degree from just a few years ago.
Honestly that's good to hear. I've run into some devs who are completely mystified on how to connect to a remote database and couldn't tell a socket from sandwich.
My CS degree had a hardware/IT support class, but A) it was entirely simulation based. We never touched any actual hardware. We "built" PC's or identified physical issues in 3d sim software, set up RAID arrays in software, etc. B) it was super hand holdy and you only ever go over a problem once, so nothing on the class has stuck. I know much more from having built, troubleshot and maintained my own computers and network than I ever learned from that class, then learned more by doing in an actual IT support position before becoming an engineer.
I mean to be fair the sheer amount of material most university engineering programs require these days makes spending significant time on specific problems almost impossible. They try to shove so much theory into your head they lose track of practical implementation. Basically everyone I went to school with complained about the lack of practical application relative to theory, and I studied mechanical engineering which is theoretically and literally chiefly concerned with hardware.
You don’t teach a farmer how an internal combustion engine works. Computers are tools to software engineers. What they need to know is how to operate them, not how to maintain them.
I’m not sure how well that analogy holds up. Farmers are usually pretty well versed in mechanical systems. To the point that now that John Deere has been screwing them over on right to repair that some farmers are even becoming versed in computer programming so they can flash the firmware on their tractors.
I never said that it was impossible for a farmer to learn things outside their immediate field. Just like computer programmers often have knowledge of hardware and the general technology stack.
My point, to make it explicit to a few of the illiterates who’ve replied to my comment so far, is that it is not necessary to teach a web developer how a goddamn CPU works. They can gain nothing from that knowledge because there are at least 3 levels of abstraction between JavaScript and assembly.
And my point is that the example you used does not make the point you are trying to make, but rather the opposite. I get what you’re saying, it just doesn’t apply to farmers and mechanics.
Operating your tools and being able to maintain and repair your tools are the unequivocally essential skills for everyone in every single industry.
If you can't, you are not a professional.
The concepts of machine logic, registers/lookups/etc are essential for every programmer. If you don't have a clear idea about how the simplest CPU functions, you don't have any basis of understanding the abstractions in front of you, scripting in JS. Not a professional.
no but a web dev should have some knowledge basis on what the ever living fuck their AIDs code fuelled by nothing but the cheapest source of caffeine and brain damage they have even does.
This is the entire reason why half of the internet is just broken, stupid developers who don't know how anything works, but know how to code, making dogshit implementations of anything and everything they can get their hands on.
It doesn't matter that the learning is segmented, you should STILL be learning about computer hardware and it's architectural choices, it's literally the reason why programming languages work the way that they do.
No, not really. Programming requires understanding of the underlying hardware, at least to a certain extent. Otherwise performance issues will look like dark magic and optimizing anything would be impossible.
Where do you start debugging if something goes wrong with the software and your information level is this low/ do you look at network stats? CPU utilization, paging/swapping? Is the hard disk bandwidth the bottleneck? Without at least some passable understanding of a computer architecture people like this just throw up their hands, or throw whatever tricks they know at the wall and see what sticks.
Horseshit. Computers aren't tools for a software engineer. Computers are tools to an administrator, an accountant. Computers are the sandbox you are building castles in as a software engineer. If you don't understand the system upon which you build, its abilities and features, its limitations, it's dependencies, you are going to make some stupid mistakes.
You need to understand discrete mathematics as a consequence of computer computation. You need to understand parallel processing and threading for muli-core processors. You need to understand networking, package management, security vulnerabilities, etc. from different architectures and protocols. And it ALWAYS helps to understand the very basics of a computer's functioning, from hardware, CPU architecture, machine code, assembly/low level programming, memory management, etc.
print('Hello, World!) is day one shit for a reason. Programming language and logic is the basics. The real expertise comes from your 3rd and 4th year materials. Databases, architecture, theory of computation, discrete mathematics, networking, operating systems, compilers, etc.
If you're using a tool, it goes without saying, you should probably have at the very least, a cursory understanding of it's function. Lest you injure yourself gravely.
A lot of farmers are learning how they work cause the companies that sell them the equipment keep fucking them over. I would argue that farmers nowadays needs to know how that works along with basic programming to get past the anti-consumer bullshit companies put in to make it nigh impossible to fix things yourself.
doesnt matter if you know how to program, john deere is just going to put some autistic encryption and ID locking on their shit, what needs to happen is for john deere to stop fucking doing this.
Most tractors are walking computers anyway, farmers are genuinely the most multi talented people you will ever meet in your life.
I mean every programmer says they intend to quit and pick up farming. Might as well give them the knowledge to be successful at their late career while they're at it
That’s the price of specialization. Don’t ask a software engineer to troubleshoot hardware. Don’t ask a backend dev to write a frontend. Don’t ask a proctologist to look at your cough.
You simply cannot be proficient at every sub-sub-specialty. That’s why we collaborate and hand the ‘my computer gets hot’ problems to the hardware people. The alternative would be only moderately useful generalist.
I'm not asking everyone to be able to become a hardware specialist, but if you can't even figure out "my computer gets hot" I'm not going to be able to trust anything you do. Identifying a heat issue does not take a rocket surgeon.
How on earth is it offensive to say they're "not experts"? They're not prodigies with PhDs. These specific young men are just technical enough and ideologically aligned.
First year? That shit is like A+ cert level knowledge or below, and A+ is damn near worthless. They would know that in the first few hours of a study guide
Meh, some of them won some hackathons and scholarships, it's pretty clear they're otherwise at least somewhat bright but they don't have any relevant domain knowledge.
In other words, the type of person most likely to be prone to hubris and catastrophic failures.
There's not really anything wrong with being incompetent, so long as you have the humility to admit it and learn from people who know better, and try not to cause harm. That's not Musk's minions though.
I think it's important to differentiate incompetence from ignorance. Ignorance is not knowing. Incompetence is not being able to fulfill the requirements for your assigned task. If you cannot fulfill the requirements for your given task, then you should not be given said task.
has Elmo only hired people who claim that they can "wrangle data" and say "yes" ?
There's two issues going on:
Elmo's sociopathic approach to laying people off is public knowledge, and top experts have the luxury of not even applying for his jobs.
Elmo's ability to judge engineering talent has likely been wildly exaggerated thanks to how he has successfully bought organizations full of talented people, in the past.
I've read a story on the forbidden website where a "database" was a single table with a single column holding a single row that contained the actual data as a CSV blob. I'm willing to bet the muskies are not beyond such acts of genius.
You have to understand that the average Trump voter probably knows everything they know about computers from watching the 'wacky-zaney hacker with personality issues/quirks' "hack" into things by tippity tapping their fingies on a keyboard in your average copaganda performance.
This is something those types of people will believe.
You're on the mark. I'm like Help Desk Level 2, I wouldnt even consider myself an actual wizard. The average person in my office thinks I'm Gandalf. Its scary how much these people dont know. And each one of them is out there on the internet.
From the same group that doesn't understand joins and thinks nobody uses SQL this is hardly surprising .
Probably got an LLM running locally and asking it to get data which is then running 10 level deep sub queries to achieve what 2 inner joins would in a fraction of the time.
You're giving this person a lot of credit. It's probably all in the same table and this idiot is probably doing something like a for-loop over an integer range (the length of the table) where it pulls the entire table down every iteration of the loop, dumps it to a local file, and then uses plain text search or some really bad regex's to find the data they're looking for.
Considering that is nearly exactly some of the answers I've received during the technical part of interviews for jr data eng, you're probably not far off.
Shit I've seen solutions done up that look like that, fighting the optimiser every step (amongst other things)
I think you're still giving them too much credit with the for loop and regex and everything. I'm thinking they exported something to Excel, got 60k rows, then tried to add a lookup formula to them. Since you know, they don't use SQL. I've done ridiculous things like that in Excel, and it can get so busy that it slows down your whole computer, which I can imagine someone could interpret as their "hard drive overheating".
I have to admit I still have some legacy code that does that.
Then I found pandas. Life changed for the better.
Now I have lots if old code that I'll update, "one day".
However, even my old code, terrible as it is, does not overheat anything, and can process massively larger sets of data than 60,000 rows without any issue except poor efficiency.
60k isn't that much, I frequently run scripts against multiple hundreds of thousands at work. Wtf is he doing? Did he duplicate the government database onto his 2015 MacBook Air?
I have an sqlite db that is a few GB in size, game saves using the format. Sadly almost all blob data, would love to play with it if it was a bit more readable
Don't know what Elmos minions are doing, but I've written code at least equally unefficient. It was quite a few years ago (the code was in written in perl) and I at least want to think that I'm better now (but I'm not paid to code anymore). The task was to pull in data from a CSV (or something like that, as I mentioned, it's been a while) and it needed conversion to XML (or something similar).
The idea behind my code was that you could just configure which fields you want from arbitary source data and on where to place them on the whatever supported destination format. I still think that the basic idea behind that project is pretty neat, just throw in whatever you happen to have and have something completely else out of the other end. And it worked as it should. It was just stupidly hungry for memory. 20k entries would eat up several gigabytes of memory from a workstation (and back then it was premium to have even 16G around) and it was also freaking slow to run (like 0.2 - 0.5 seconds per entry).
But even then I didn't need to tweet that my hard drive is overheating. I well understood that my code is just bad and I even improved it a bit here and there, but it was still so very slow and used ridiculous amounts of RAM. The project was pretty neat and when you had few hundred items to process at a time it was even pretty good, there was companies who relied on that code and paid for support. It just totally broke down with even a slightly bigger datasets.
But, as I already mentioned, my hard drive didn't overheat on that load.
Seriously - I can parse multiple tables of 5+ million row each... in EXCEL... on a 10 year old desktop and not have the fan even speed up. Even the legacy Access database I work with handles multiple million+ row tables better than that.
Sounds like the kid was running his AI hamsters too hard and they died of exhaustion.
You're correct - the standard tabs can only hold roughly 1.2 million rows.
The way to get around that limitation is to use the Data Model within Power Pivot:
It can accept all of the data connections a standard Power Query can (ODBC, Sharepoint, Access, etc):
You build the connection in Power Pivot to your big tables and it will pull in a preview. If needed, you can build relationship between tables with the Relationship Manager. You can also use DAX to build formulas just like in a regular Excel tab (very similar to Visual Basic). You can then run Pivot Tables and charts against the Data Model to pull out the subsets of data you want to look at.
The load times are pretty decent - usually it takes 2-3 minutes to pull a table of 4 million rows from an SQL database over ODBC, but your results may vary depending on datasource. It can get memory intensive, so I recommend a machine with a decent amount of RAM if you're going to build anything for professional use.
The nice thing about building it out this way (as opposed to using independent Power Queries to bring out your data subsets) is that it's a one-button refresh, with most of the logic and formulas hidden back within the Data Model, so it's a nice way to build reports for end-users that's harder for them to fuck up by deleting a formula or hiding a column.
Oh yes, I remember using power query for a few months once I started working with bigger databases, but I saw that moving to Python would be better carrer wise and never came back to excel to do actual work (but at the end everything get exported to excel)
My fucking events table of my synapse DB in postgres is nearly ten times as large, and I ported that from sqlite no long ago, in a matter of minutes. All of the data is on a 2*3 cluster of old 256GB SSDs, equaling about 1.5TB with Raid 0. That's neither really fast, nor cool. But stable.
I mean if we were to sort of steelman this thing, there sure can be database relations and queries that hit only 60k rows but are still hteavy as fuck.
Unless I'm misreading it which is possible it's awfully late, he said he processed 60,000 rows didn't find what he was looking for but his hard drive overheated on the full pass.
Discs don't overheat because there was load. Even if he f***** up and didn't index the data correctly (I assume it's a relational database since he's talking about rows) The disc isn't just going to overheat because the job is big. It's going to be lack of air flow or lack of heatsink.
I guarantee you he was running on an external NVMe, and one of those little shitty-ass Chinese enclosures. Or maybe one of those self immolating SanDisk enclosures. Hell, maybe he's on a desktop and he slept a raw NVMe on his motherboard without a heatsink
There are times when you want a brilliant college student on your team, But you need seasoned professionals to help them through the things they've never seen before and never done before.
yes but also why say 60K when you could have literally said anything? I mean surely the fact that he thinks 60K rows a big number is already explaining alot lol.
They probably have an explanation tweet at the ready to make more sense of it. They just want enough 'hurr durr these idiot" comments before they reverse Uno card this with more context.
Based on all that has been going on, I feel like they don't really have the capacity to think more than one step ahead. They do sth stupid and then they usually follow up with "lol joke" or "lol you can't understand"
I don't get how people don't see this stuff. Yea the average trumpet isn't out there planning Jack shit. But there are think tanks that are. Remember anti-smoking campaigns. Anti climate change campaign. The precision to purchase advertising into key areas and specific demographics that would spread a message. This is ancient knowledge with modern technology.
Imagine a room full of former Wall Street, quants, established experts from fields like behavioral science and psychology. All with the singular goal to decide where to dedicate a dragons horde of wealth to maximize effect in a world where we all have anonymous pipes directly into our eyes and ears. We never stood a chance. There's no rich socialist funding think tanks. There's no counter. We can laugh at the yokel all we want. But the yokel is being puppeteered by some scary fuckers with intention to seize power with the new shifting Zeitgeist. Soldiers don't need to think. But their generals are. The left are like guerilla fighters going up against an imperial army full of Patons and Eisenhower's.
Cambridge analytical, heritage foundation, international democracy Union. We're fucked until we actually recognize why we're fucked
I keep hearing things about these hires he has, I don't think they're naive, At least not as such. They seem to be more power hungry trust fund babies.
But yeah, people with a few years in them would be a moral liability in that line of work.
So, this means they either have a local copy on disk of whatever database they're querying, or they're dumping a remote db to disk at some point before/during/after their query, right?
So yeah, she's apparently toting around an external hard drive with a copy of the "multiple terabytes" large US spending database, running queries against it, then dumping the 60k-row result set to CSV for further processing.
I'm still confused at what point the external drive overheats, even if she is doing all this in a "hot humid" hotel room that she can't run any fans I guess because her kids were asleep?
But like, all of that just adds more questions, and doesn't really answer the first one - why?
I run queries throughout the day that can return 8 million+ rows easily. Granted, it takes few minutes to run, but it has never caused a single issue with overheating even on slim pc’s.
This makes no fucking sense. 60k rows would return in a flash even on shitty hardware. And if it taxes anything, it’s gonna be the ram or cpu- not the hard drive.
In my experience, the only time that I've taxed a drive when doing a database query is either when dumping it, or with SQLite's vacuum, which copies the whole thing.
For a pretty simple search like OP seems to be doing, the indices should have taken care of basically all the heavy lifting.
I literally work with ~750,000 line exports on the daily on my little Lenovo workbook. It gets a little cranky, especially if I have a few of those big ones open, but I have yet to witness my hard drive melting down over it. I'm not doing anything special, and I have the exact same business-economy tier setup 95% of our business uses. While I'm doing this, that little champion is also driving 4 large monitors because I'm actual scum like that. Still no hardware meltdowns after 3 years, but I'll admit the cat likes how warm it gets.
750k lines is just for the branch specific item preferences table for one of our smaller business streams, too - FORGET what our sales record tables would look like, let alone the whole database! And when we're talking about the entirety of the social security database, which should contain at least one line each in a table somewhere for most of the hundreds of millions of people currently living in the US, PLUS any historical records for dead people??
Your hard drive melting after 60k lines, plus the attitude that 60k lines is a lot for a major database, speaks to GLARING IT incompetence.
You've got it all wrong, in traditional computer terminology the "hard drive" is the box that sits under the desk that collects cat fluff and cigarette tar.
I don't think I've seen a brand new computer in the past decade that even had a mechanical hard drive at all unless it was purpose-built for storing multiple terabytes, and 60K rows wouldn't even take multiple gigabytes.
I have when a misconfigured spark job I was debugging was filling hard drives with tb of error logs and killing the drives.
That was a pretty weird edge case though, and I don’t think the drives were melting, plus this was closer to 10 years ago when SSD write lifetimes were crappy and we bought a bad batch of drives.
Same as Elon when he confidently told off engineers during his takeover of Twitter or gestures broadly at the Mr. Dunning Kruger himself
Wonder if it’s an SQL DB
Elon probably hired confident right wingers whose parents bought and paid their way through prestigious schools. If he hired anyone truly skilled and knowledgeable, they’d call him out on his bullshit. So the people gutting government programs and passing around private data like candy are just confidently incorrect
What the hell are you doing that your hard drives are overheating? How do you even know it's overheating as I'm like 90% certain hard drives (except NVMe if we're being liberal with the meaning of hard drive) don't even have temperature sensors?
The only conclusion I can come to is that everything he's saying is just bullshit.
Can we think of any device someone might have that would struggle with 60k? Certainly an ESP32 chip could handle it fine, so most IoT devices would work...
Unless the database was designed by someone who only knows of data as that robot from Star Trek, most would be absolutely fine with 60k rows. I wouldn't be surprised if the machine they're using caches that much in RAM alone.
Imo if they can't max out their harddrive for at least 24 hours without it breaking, their computer was already broken. They just didn't know it yet.
Any reasonable SSD would just throttle if it was getting too hot, and I've never heard of a HDD overheating on its own, only if there's some external heat sources, like running it in a 60°C room
Hard Drives might do it if the enclosure is poorly designed (no ventilation), but I can't imagine a situation where it would overheat like that that quickly, even in a sealed box. 30k is nothing in database terms, and if their query was that heavy, it would bottleneck on the CPU, and barely heat the drive at all.
Unless they actually mean the hard drive, and not the computer. I've definitely had a cheap enclosure overheat and drop out on me before when trying to seek the drive a bunch, although it's more likely the enclosure's own electronics overheating. Unless their query was rubbish, a simple database scan/search like that should be fast, and not demanding in the slightest. Doubly so if it's dedicated, and not using some embedded thing like SQLite. A few dozen thousand queries should be basically nothing.
I'm a data engineer that processes 2 billion row 3000 column datasets every day, and I open shit in Excel with more than 60k rows. What the hell is this chick talking about?
If you work for a boss that fundamentally misunderstands what you are doing, then misleading them into thinking you're 'hard at work, making decisions with consequences' is the theatre you put up to keep the cash flowing.
It's one of the fundamental flows of autocracy, people try and represent what you want them to
It sounds like Hollywood tech lingo. Like when you're watching a movie or a TV show and the designated techy character starts just saying computer words that make no actual sense in the real world, but I guess in CSI: Idiottown the hard drives have severe overheating issues.
The row limitation seems, to me, like an actually-good thing. Excel is for data where you might conceivably scroll up and down looking at it and 1M is definitely beyond the ability of a human even to just skim looking for something different.
An older version of Excel could only handle 64k rows and I had a client who wanted large amounts of data in Excel format. "Oh sorry, it's a Microsoft limitation," I was thrilled to say. "I have no choice but to give you a useful summarization of the data instead of 800k rows (each 1000 columns wide) of raw data."
Some time ago, I heard a story of CS and Econ professors having lunch together. The Econ professor was excited that Excel was going to release a version that blew out the 64k row limit. The CS professor nearly choked on his lunch.
Dependence on Excel has definitely caused bad papers to be published in the Econ space, and has had real world consequences. There was a paper years ago that stated that once a country's debt gets above 120% of GDP, its economy goes into a death spiral. It was passed around as established fact by the sorts of politicians who justify austerity. Problem was, nobody could reproduce the results. Then an Econ undergrad asked the original author for their Excel spreadsheet, and they found a coding error in the formulas. Once corrected, the conclusion disappeared.
It completely screws up numbers if the column is a number and the number is over 15 digits long.
I work in insurance in Brazil, by standards of our regulatory body, claims numbers must be a string of 20 numbers (zfill(20) if needed). You can't imagine the amount of times excel had fucked me up rounding down the claim numbers, this is one of the first things I teach to my interns and juniors when they're working with the claims databases.
Sure, but sometimes you need to do a pivot table and get transformed, or you while working you accidentally made it a number, or just because excel being excel it import the number as scientific notation, or any other case.
This sounds like trying to do stuff in Excel? The computer isn't overheating but the amount of memory needed is very high which would make it run poorly. They might interpret that as overheating?
I think my Pi could process 60k rows without overheating. And the poor thing is dangling behind my bookshelf from its power cord with a fine layer of dust coating every inch of it.
This shit sounds like when your mom tells you that the Facebook printed out her bank statement on the fax machine. I'm not smart enough to even guess how you did something dumb enough to make that happen.
How bad are you at writing queries? How does your hard drive overheat even under 100% load? Do you have it smothered under a blanket? Did you crack it up and expose it to cheeto dust? What does running a query on your, presumably, remote database even have to do with your harddrive in the first place? Are you trying to copy the entire database locally to a laptop? Do you know how to tie your shoes yet, or are you still on the velcro?
This shit sounds like when you’re mom tells you that the Facebook printed out her bank statement on the tax machine.
My dear sweet mother asked me somewhere around 2005-06 "If we can fax paper, why not groceries, or pizza delivery?"
Apparently she had believed, for decades, that fax machines literally transported physical paper over phone lines. She has a college degree, and my family is wealthy.
Do not underestimate the mind boggling technical and scientific ignorance of old people who should know better.
I remember thinking that when I was a kid and saw a clip on PBS Kids of a kid using their home fax machine to fax their dad a picture they drew at work. I was also like 4 so teleportation was still distinctly possible to my brain
I've been told violence isn't the answer and we shouldn't just shoot nazis and nazi enablers dead.
The way most people change their mind isn't based on facts or figures, but emotions. Specifically, in-group belonging. For most people, and this certainly includes me and you some of the time, what our in-group believes is more compelling than an out-groups supposed facts.
They see that guy as someone in their group so they believe him. They see you as a bad outside bad bad bad liar, so nothing you say is likely to get through. (This comic is worth reading on this topic: https://theoatmeal.com/comics/believe )
If you want to change someone's mind, they have to see you as in-group. Not necessarily the same group as what you're arguing with. We all belong to many groups. American, new yorker, white guy, middle aged, yankees fan, etc etc there are many such slices. Like how you can't get a republican to recycle by appealing to environmental concerns (because environmentalists are out-group, so fuck them), but you might be able to get them to recycle via something like "only american ingenuity can turn trash into bridges and tanks!"
This takes a lot of time and effort, and if you don't get them to stop hanging out with the other group, you won't make any lasting changes.
So I think you'd need a multi prong approach:
Get them off bad media. Facebook, fox news, etc. This is reinforcing their bad beliefs. Because they see this stuff as trustworthy in-group, it goes right into the worldview.
Get them to stop hanging out with their shitty maga-hat friends. This is the social in-group that's reinforcing bad beliefs.
Get them to trust you.
Gently introduce the idea that maybe the extreme right doesn't have their interests at heart, etc
All of which takes a lot of time and effort, and your opposite number is basically trying to do the same thing. Except they have fox news, trump, and such in their corner.
And, again, I'm told we definitely shouldn't just shoot extreme right wingers and other nazi sympathizers dead. Nor should we burn their houses down. If we're an emergency responder, we definitely shouldn't let them die while thinking to ourselves "they would let so many die. without a thought, their passing deserves no mourning" or similar.
You should definitely nullify if you're on a jury and someone allegedly did violence to a shitty ceo or red-hat, though, bu that's getting off topic.
Wow, that was an awesome rabbit hole, thank you for the link.
If you want to change someone’s mind, they have to see you as in-group.
Maybe a less manipulative-sounding way to phrase that might be that we should remind people that we're all in it together. The far right media and their billionaire buddies have spent the past decade and a half dividing us, and they succeeded. Idk what it would take to unite this country again, but it at least is a little comforting to have a clear problem statement.
I've been thinking about this a lot lately. I'm from the UK, and whilst things are less politically dire here than the US, it's still pretty grim. Both the Conservatives and Labour seem reluctant to actually meaningfully tax the rich, even as the working class (and to a lesser extent, the middle class) are being squeezed by a cost of living crisis and general hopelessness. Parties like Reform are taking the racist "things are bad because we have too many immigrants" and I've recently realised that I need to stop resenting people for being taken in by that rhetoric; people are desperate and there aren't people in the mainstream pushing for alternatives (besides Reform). These people have a lot in common with me, such as recognising that we're being fucked but the system, but we just disagree on the solution. It's hard, but ultimately necessary to be able to be in solidarity with people like Reform' voters
I agree some form of consistent opposition messaging is needed.
The maga world talks in consistent themes and terminology, which creates a psychological advantage. Unfortunately, it's playground psychology, but if that's the game being played you need to find a way to win at it.
I can't remember the particular phrase that was used, but I heard an argument recently that we need to be more like politicians going on an interview and ensure that we're more on message. For example, it's fairly obvious by now that economically, the problem is wealth inequality, but I see fairly surprisingly few people discussing that.
Compelling point. I just found that arguing with „these kind of people“ (livibg in europe, so no MAGA‘s here but like-minded, conservative fundamentalists etc.) leads to nowhere. It‘s kind of like the covid-conversations.
And often I heard „you can‘t make them change their minds, so just let them be“.
Still, I think this behaviour leads to isolation and separates us as a people even more.
Long story short: good question. If you found the answer, let me know.
Doesn't actually say that 60k overheated his drive. He says that he ran a run on 60k, and that he couldn't do the whole database due to overheating. Two unrelated statements except that 60k is the lower bound for what he could process.
Doesn't mean he knows what he's doing though, as pretty huge datasets are processable on quite modest hardware if you do it right.
They make nothing. They're compensated for destroying things, and considering it's musk, they're likely given relatively little money in return for their time.
Even if the only thing you do all day is sit on the toilet and yell at the Internet, you're already a bigger net positive on society.
The reason he recruited a bunch of incompetent college kids is because nobody with any experience or wisdom would touch this rolling crime wave with a 10 foot pole
They're dumb kids. They probably see Musk as some kind of god. He gave them some line like "Hey, you guys wanna save the United States?" and they jumped at the chance.
Either she knows something novel, where processing data using voice coils is somehow beneficial, or is someone who calls their computer a ‘hard drive’, which summarily negates any legitimacy of technical competence.
What is this, a table for ants? Because that’s the average number of ants in an ant colony and it’s nowhere near an impressive amount of rows to be doing any sort of processing on. It wouldn’t be an impressive amount of rows if your rig was an i386DX-33 running off a 5” floppy.
Exactly, 60k rows is negligible enough in most cases that you can just treat it as free unless you're doing a cross join on it or something, unless he's doing something like using an unordered text file as his database with no ram or cache
I used to perform data analysis of robotics firmware logs which would generate several million log lines per hour and that was my second job out of college.
I don't know how you fuck up 60k lines that bad. Is he nesting 150 for loops and loading a copy of the data set in each one while mining crypto??
If there's something you want to search by in a database, you should index it.
Indexing will create an ordered data structure that will allow much faster queries. If you were looking for the username gazter in an unindexed column, it would have to check literally every username entry. In a table of 1000000 entries it would check 1000000 times.
In an indexed column it might do something like ask to be pointed to every name beginning with "g", then of those ask to be pointed to every name with the second letter "a" and so on. It would find out where in the database gazter is by checking only six times.
Substring matching is much more computationally difficult as it has to pull out each potentially matching value and run it through a function that checks if gazter exists somewhere in that value. Basically if you find yourself doing it you need to come up with a better plan.
Cartesian explosion would be when your query ends up doing a shit load of redundant work. Like if the query to load this thread were to look up all the posters here, get all their posts, get the threads from those posts and filter on the thread id.
I'm guessing you'd have to search the database to make the index, right? To search for 'gazter' you'd have had to go over the whole dataset and assigned each entry with a starting letter value, and so on?
When it comes to searching the database, the index will have already been created. When you create an index, it might take a while as the database engine reads all the data and creates a structure to shadow it. Each engine is probably different and I don't know if any work exactly like that, but it's an intuitive way to understand the basics of how B-trees work. You don't really need to think much about how it works, just that if you want to use a column as a filter, you want to index it.
However, when you're thinking about the structure of a database it's a good idea to think what you'll want to do with it before hand and how you'll structure queries. Sometimes searching columns without an index is unavoidable and then you've got to come up with other tricks to speed up your search. Like your doctor might find you (i'm presuming gaz is sort for gary and/or gareth here) with a query like
SELECT * FROM patients WHERE birthdate = "01-01-1980" AND firstname LIKE "gar%"
The db engine will first filter by birthdate which will massively reduce the amount of times it has to do the more intensive LIKE operation.
Storing large volumes of a text in a database column without optimization, then searching for small strings within it. It causes the database to basically search character by character to find a match by reading everything from disk. If you use indexes the database can do a lot of really incredible optimization to make finding values mich faster, and honestly string searching is better suited to a non-relational DB engine (which is why search engines don't use relational DBs).
Cartesian explosion is where you join related data together in a way that causes your result set to be wayyyy bigger than you expect. For example if you try to search through blog posts, but then also decide to bring in comments to search, then bring in the authors of those comments and all their comments from other posts. Result sets start to grow exponentially in that way, so maybe if you only search a few thousand blog posts you might be searching through millions of records because you designed your queries poorly.
The only time I've even seen drive temp sensor alarms is on server raid arrays and other similar hard drives/SSDs.... Never in my life have I seen one available on a consumer device, nor have I seen any alarm for and drive temp, go off. It just doesn't happen.
IMO, this is one of those language barriers where people call their computer chassis (and everything in it) the "hard drive".
Applying that assumption, their updated statement is: His computer over heated.
Idk what kind of shit system he's running on that 60k rows would cause overheating, but ok.
I didn't know hard drive overheating was a thing. Should I be worried that my 5 year old hard drive is about to overheat. I mean is this actually a floppy disk or something?
no. but generally spinning things that spin at several thousands of RPM that are spinning on a bearing, that no longer have a bearing usually sort of uh, tend to be VERY noisy.
Ok, but do you know anyone this has happened to? I don't and I have some pretty old drives I still use. I tend to just buy more. Also most drives these days are solid state aren't they? This just feels like a low probability event to me. Overheating RAM or the CPU or GPU, sure, but hard drive?
I had it happen to a random hard drive I bought, old bastard found in a bin at a local thrift store. Anyways had a big dent in it and while it started up, even booting into windows XP it cooked itself made a screeching noise and was hot to the touch. Don't think I need to explain that the big dent was probably the source of the overheating, anyways had to use an oven mitt to relocate it to a metal bucket of water where it boiled for a minute.
Sure if they physically damaged the hard drive, though at that point the failure and the code were largely unrelated. Though if they were using a computer with an SSD only I would have no clue. Assuming they aren't lying id have to look at the computer physically, perhaps even pry it open to see if there's any noticable damage.
There's also the possibility that they are just stupid and overheated their CPU by having too many apps open. Then they blamed their hard drive overheating because they are stupid.
not personally, i may have seen a video or two of it happening, but it's hard to tell whether the head is dragging against the platter, or it's the bearing, either one of those makes horrendous noise.
If you're worried about it happening on a drive you own, you should copy that data somewhere else as a backup, ideally sooner rather than later. If you're curious about the health of the drive you do stuff like SMART tests as well.
Yeah, most drives are solid state now, unless you're buying hdds for archival purposes, still cheaper and denser in most cases. It's a low probability failure, until the drive meets EOL, in which case it's a mechanical wear part, either the motor or the bearing fails. One of them will fail first, probably the bearing.
The bearing failing would likely result in the HDD overheating as a result. Assuming the platter still spins, but that's the only scenario i can think of where that would happen, unless you dump a very specific amount of continuous current into the read arm coils. That might also cause it, but it's not likely at all.
An ssd "overheating" is more likely, but it shouldn't cause too many issues, maybe premature degradation over long term use, and slowing of read/write speeds, or in some cases, an improvement, but other than that it should be business as normal. You would have to hit it with like a heat gun, to get a hardware failure or something like that.
I'm not actually worried about my drives. In fact that was kind of the point. I was kidding around because this excuse that the hard drive overheated sounds a little like a car running out of blinker fluid.
When an HDD works continuously it can heat up to above 60 °C if proper air circulation is not allowed, which can cause a very premature failure. In fact, it should be kept under 40 °C to achieve the intended lifespan. Unfortunately, PC cases are usually not great at removing heat from the HDD by default.
As for your drive, it most likely has a temperature sensor so it can be displayed by various utilities.
What in the fuck is this idiot doing? I've process datasets far larger than that and never once have I run into a hard drive "overheat". I mean what level of incompetence do you have to have to get a hard drive to overheat processing a measley 60K rows of data?
If you're running a pcie nvme ssd, one of the modern ones, and you're doing a SHIT ton of reads, like threadripper level amount of reads, i guess "overheating" isn't unexpected? Shouldn't do much other than slow down the SSD though?
dumbass probably loaded them into memory, and OOM'd, and thought it was the drive.
NVMEs will throttle if they are inadequately cooled. Pretty much only folks who are new at buiding computers and don't adequately cool their NVMEs experience this.
Either way it shows what a clown show this is. I'm pretty sure they're doing some babuki theatre to distract us and then if the scheme being done here is found, they'll point fingers at the dipshit kid and he'll be punished for whatever the con is.
Either way the billionaires will be better off for it, barring judicial, congressional, or military revolt; which is looking less likely by the day.
Bunch of traitors forgot their oath is to the constitution. The constitution IS the soul of the country and is to be respected. The country was founded by the constitution and it defines the system to be decentralized power giving control to the populace through three different branches of government. Not some false idol king or an evil oligarchy propped up by foreign and domestic robber barrons
NVMEs will throttle if they are inadequately cooled. Pretty much only folks who are new at buiding computers and don’t adequately cool their NVMEs experience this.
yeah, but it shouldn't be that significant though right? You're talking like 10% slower speeds to anywhere probably like 50% of the speed, which for modern NVME ssds is basically perfectly usable in most cases. But i'm not up to speed on pcie 4.0 and 5.0 ssds, so idk the specifics.
Or it's a cheap external nvme chassis with a Samsung 980 Pro. Had to run that when I was copying files from one of my old machines and boy, it will absolutely overheat to the point of failure.
Gave me quite the scare when I started getting read errors and then it dropped off the bus. It shutdown to protect itself but it certainly didn't seem like that at the time.
Out of memory/overheating in 60k rows? I've had a few multi-million row databases that could fit into a few gigs of memory, and most modern machines have that much in RAM. A 60k query that overheats the machine might only happen if you're doing something weird with joins.
Plus a lot of reads is nothing really, for basically all databases, unless you're doing an unsmart thing with how you're reading it (like scanning the whole database over and over). If you're not processing the data, it'd be I/O bottlenecked.
if they wrote good code yeah, evidently they didn't write good code if they're struggling to process 60k lines of a database lmao.
They must either be O(n^10) complexity or something retarded like that for this to be the case. I wouldn't put it past them.
Plus a lot of reads is nothing really, for basically all databases, unless you’re doing an unsmart thing with how you’re reading it (like scanning the whole database over and over). If you’re not processing the data, it’d be I/O bottlenecked.
again, i'm assuming they aren't very smart, since this is an issue in the first place.
Wow.
I've been processing a couple of billion rows of data on my machine, the fans didn't even come on. WTF are they teaching "experts" these days, or has Elmo only hired people who claim that they can "wrangle data" and say "yes" ?
Even if querying data was processing-heavy and even if somehow the ‘hard drive’ got warm during this, then there still would need to be a hardware defect in order for the drive to overheat.
Yes, but this may be a symptom of an issue I've been seeing with younger programmers; they've siloed themselves so specifically into whatever programming they "specialize" in, that they become absolutely useless at dealing with absolutely anything else related to their job. And exasperating this issue is the fact that they've grown up with systems that "just work". Windows, iOS, and android are all at the point where fucking around with hardware issues is very uncommon for the average person.
Asking this guy to solve a hardware problem is like asking hime to tune a carburetor. He likely has not the slightest clue how to start.
In my experience, a lot of software dev degree paths basically don't even have relevant classes on hardware at all. Classes on hardware are all in IT Helpdesk and Network Admin degree paths whereas the software dev students are dumped straight into Visual Studio right off the bat with no relevant understanding of the underlying hardware or OS.
My experience does not reflect yours. Computer Architecture, Discrete Math (logic gate math), and Operating System Concepts were all required classes in my CS degree from just a few years ago.
Honestly that's good to hear. I've run into some devs who are completely mystified on how to connect to a remote database and couldn't tell a socket from sandwich.
In my degree, we had to write kernel mods and device drivers
Can I have my socket with rye. I like rye.
My CS degree had a hardware/IT support class, but A) it was entirely simulation based. We never touched any actual hardware. We "built" PC's or identified physical issues in 3d sim software, set up RAID arrays in software, etc. B) it was super hand holdy and you only ever go over a problem once, so nothing on the class has stuck. I know much more from having built, troubleshot and maintained my own computers and network than I ever learned from that class, then learned more by doing in an actual IT support position before becoming an engineer.
I mean to be fair the sheer amount of material most university engineering programs require these days makes spending significant time on specific problems almost impossible. They try to shove so much theory into your head they lose track of practical implementation. Basically everyone I went to school with complained about the lack of practical application relative to theory, and I studied mechanical engineering which is theoretically and literally chiefly concerned with hardware.
You don’t teach a farmer how an internal combustion engine works. Computers are tools to software engineers. What they need to know is how to operate them, not how to maintain them.
I’m not sure how well that analogy holds up. Farmers are usually pretty well versed in mechanical systems. To the point that now that John Deere has been screwing them over on right to repair that some farmers are even becoming versed in computer programming so they can flash the firmware on their tractors.
Farmers can build a two stroke from parts.
No, but if a farmer’s tractor is overheating (as in the gard drive conparison), I’m sure they could diagnose it.
I never said that it was impossible for a farmer to learn things outside their immediate field. Just like computer programmers often have knowledge of hardware and the general technology stack.
My point, to make it explicit to a few of the illiterates who’ve replied to my comment so far, is that it is not necessary to teach a web developer how a goddamn CPU works. They can gain nothing from that knowledge because there are at least 3 levels of abstraction between JavaScript and assembly.
And my point is that the example you used does not make the point you are trying to make, but rather the opposite. I get what you’re saying, it just doesn’t apply to farmers and mechanics.
Operating your tools and being able to maintain and repair your tools are the unequivocally essential skills for everyone in every single industry.
If you can't, you are not a professional.
The concepts of machine logic, registers/lookups/etc are essential for every programmer. If you don't have a clear idea about how the simplest CPU functions, you don't have any basis of understanding the abstractions in front of you, scripting in JS. Not a professional.
no but a web dev should have some knowledge basis on what the ever living fuck their AIDs code fuelled by nothing but the cheapest source of caffeine and brain damage they have even does.
This is the entire reason why half of the internet is just broken, stupid developers who don't know how anything works, but know how to code, making dogshit implementations of anything and everything they can get their hands on.
It doesn't matter that the learning is segmented, you should STILL be learning about computer hardware and it's architectural choices, it's literally the reason why programming languages work the way that they do.
No, not really. Programming requires understanding of the underlying hardware, at least to a certain extent. Otherwise performance issues will look like dark magic and optimizing anything would be impossible.
Where do you start debugging if something goes wrong with the software and your information level is this low/ do you look at network stats? CPU utilization, paging/swapping? Is the hard disk bandwidth the bottleneck? Without at least some passable understanding of a computer architecture people like this just throw up their hands, or throw whatever tricks they know at the wall and see what sticks.
Horseshit. Computers aren't tools for a software engineer. Computers are tools to an administrator, an accountant. Computers are the sandbox you are building castles in as a software engineer. If you don't understand the system upon which you build, its abilities and features, its limitations, it's dependencies, you are going to make some stupid mistakes.
You need to understand discrete mathematics as a consequence of computer computation. You need to understand parallel processing and threading for muli-core processors. You need to understand networking, package management, security vulnerabilities, etc. from different architectures and protocols. And it ALWAYS helps to understand the very basics of a computer's functioning, from hardware, CPU architecture, machine code, assembly/low level programming, memory management, etc.
print('Hello, World!) is day one shit for a reason. Programming language and logic is the basics. The real expertise comes from your 3rd and 4th year materials. Databases, architecture, theory of computation, discrete mathematics, networking, operating systems, compilers, etc.
computers are a tool to anybody who uses them?
If you're using a tool, it goes without saying, you should probably have at the very least, a cursory understanding of it's function. Lest you injure yourself gravely.
What the fuck
How is he going to fix his tractor? Wait days for John Deere to send somebody? Let the crop rot on the vine?
A lot of farmers are learning how they work cause the companies that sell them the equipment keep fucking them over. I would argue that farmers nowadays needs to know how that works along with basic programming to get past the anti-consumer bullshit companies put in to make it nigh impossible to fix things yourself.
doesnt matter if you know how to program, john deere is just going to put some autistic encryption and ID locking on their shit, what needs to happen is for john deere to stop fucking doing this.
Most tractors are walking computers anyway, farmers are genuinely the most multi talented people you will ever meet in your life.
You've never met a farmer in your life.
It is good for the programmer to know how the computer operates, as well.
CS departments were doing poorly, but now they're putting out farmers? No wonder all these new graduates can't find a job.
I mean every programmer says they intend to quit and pick up farming. Might as well give them the knowledge to be successful at their late career while they're at it
Ooh wait 'til Musk realises he can improve US agricultural efficiency.
Just keep trying to justify your own lack of competency I guess. ¯\_(ツ)_/¯
the only reason farmers are afloat financially is BECAUSE they can rebuild an engine if needed.
Just look at the john deere right to repair shit. It's literally a huge problem.
That’s the price of specialization. Don’t ask a software engineer to troubleshoot hardware. Don’t ask a backend dev to write a frontend. Don’t ask a proctologist to look at your cough.
You simply cannot be proficient at every sub-sub-specialty. That’s why we collaborate and hand the ‘my computer gets hot’ problems to the hardware people. The alternative would be only moderately useful generalist.
I'm not asking everyone to be able to become a hardware specialist, but if you can't even figure out "my computer gets hot" I'm not going to be able to trust anything you do. Identifying a heat issue does not take a rocket surgeon.
If it was an nvme ssd i could almost believe it. Some come with totally underspecced heatsinks
He hired a bunch of 19-25 year old. Not experts
Hey! Thats offensive to 19-25 year olds, there are many who just finished college/university and are more than aware.
They’re just role playing like in movies, with no idea of the consequences.
How on earth is it offensive to say they're "not experts"? They're not prodigies with PhDs. These specific young men are just technical enough and ideologically aligned.
Except they’re not, as you will know their tweet would be false after your first year of any technical (IT oriented) education.
First year? That shit is like A+ cert level knowledge or below, and A+ is damn near worthless. They would know that in the first few hours of a study guide
I was being generous when you consider the people in school who somehow pass, even when they don’t know a thing 🥲
Technical enough to be hired, is all I meant. 🙄
Apologies, if I came over as hostile. I did not get your meaning through text.
Even then, no. These were all obviously nepotism hires who would not have otherwise qualified.
Meh, some of them won some hackathons and scholarships, it's pretty clear they're otherwise at least somewhat bright but they don't have any relevant domain knowledge.
In other words, the type of person most likely to be prone to hubris and catastrophic failures.
Your original comment was ambiguous as to if being an "expert" and "being 19-25" are mutually exclusive.
lol a 19-21 yo isnt going to have a degree lol,
If they went into uni straight out of high school, they could. A lot of Bachelor holders would be around that age, since they start at 18.
There is nothing wrong with being 19-25. There's something wrong with being wholly incompetent.
There's not really anything wrong with being incompetent, so long as you have the humility to admit it and learn from people who know better, and try not to cause harm. That's not Musk's minions though.
I think it's important to differentiate incompetence from ignorance. Ignorance is not knowing. Incompetence is not being able to fulfill the requirements for your assigned task. If you cannot fulfill the requirements for your given task, then you should not be given said task.
Bunch of 1337 hax0rs script kiddies who don't understand anything but they suck elon's balls or something idk.
These are the type of people that have deleted the French language from their GNU/Linux system.
There's two issues going on:
I've read a story on the forbidden website where a "database" was a single table with a single column holding a single row that contained the actual data as a CSV blob. I'm willing to bet the muskies are not beyond such acts of genius.
It's terrifying that this is plausible.
We call it New Redis!
You have to understand that the average Trump voter probably knows everything they know about computers from watching the 'wacky-zaney hacker with personality issues/quirks' "hack" into things by tippity tapping their fingies on a keyboard in your average copaganda performance.
This is something those types of people will believe.
You're on the mark. I'm like Help Desk Level 2, I wouldnt even consider myself an actual wizard. The average person in my office thinks I'm Gandalf. Its scary how much these people dont know. And each one of them is out there on the internet.
60k rows is generally very usable with even wide tables in row formats.
I’ve had pandas work with 1M plus rows with 100 columns in memory just fine.
After 1M rows move on to something better like Dask, polars, spark, or literally any DB.
The first thing I’d do with whatever data they’re running into issues with is rewrite it as partitioned and sorted parquet.
My go-to tool of late is
duckdb, comes with binaries for most platforms, works out of the box, loads any number of database formats and is FAST.Yes, his Boy Harem.
From the same group that doesn't understand joins and thinks nobody uses SQL this is hardly surprising .
Probably got an LLM running locally and asking it to get data which is then running 10 level deep sub queries to achieve what 2 inner joins would in a fraction of the time.
You're giving this person a lot of credit. It's probably all in the same table and this idiot is probably doing something like a for-loop over an integer range (the length of the table) where it pulls the entire table down every iteration of the loop, dumps it to a local file, and then uses plain text search or some really bad regex's to find the data they're looking for.
Considering that is nearly exactly some of the answers I've received during the technical part of interviews for jr data eng, you're probably not far off.
Shit I've seen solutions done up that look like that, fighting the optimiser every step (amongst other things)
I think you're still giving them too much credit with the for loop and regex and everything. I'm thinking they exported something to Excel, got 60k rows, then tried to add a lookup formula to them. Since you know, they don't use SQL. I've done ridiculous things like that in Excel, and it can get so busy that it slows down your whole computer, which I can imagine someone could interpret as their "hard drive overheating".
I have to admit I still have some legacy code that does that.
Then I found pandas. Life changed for the better.
Now I have lots if old code that I'll update, "one day".
However, even my old code, terrible as it is, does not overheat anything, and can process massively larger sets of data than 60,000 rows without any issue except poor efficiency.
They don't understand joins? How..
60k isn't that much, I frequently run scripts against multiple hundreds of thousands at work. Wtf is he doing? Did he duplicate the government database onto his 2015 MacBook Air?
i mean its even excel sized depending on how many columns. This is seriously sad and alarming
Holy shit if this ids lm issue that’s too funny
Sqlite can easily handle millions of rows. Don't sell it short
How about a 6.4TB sqlite database?
https://boyter.org/posts/searchcode-bigger-sqlite-than-you/
Should be enough to hold 60k rows
I have an sqlite db that is a few GB in size, game saves using the format. Sadly almost all blob data, would love to play with it if it was a bit more readable
60k is single json file
A TI-86 can query 60k rows without breaking a sweat.
If his hard drive overheated from that, he is doing something very wrong, very unhygienic, or both.
He probably mining crypto on top of running his SQL queries.
What? You don't run your hard drives in the oven while baking brownies? It makes them zesty.
There must be more join statements than column names
Don't know what Elmos minions are doing, but I've written code at least equally unefficient. It was quite a few years ago (the code was in written in perl) and I at least want to think that I'm better now (but I'm not paid to code anymore). The task was to pull in data from a CSV (or something like that, as I mentioned, it's been a while) and it needed conversion to XML (or something similar).
The idea behind my code was that you could just configure which fields you want from arbitary source data and on where to place them on the whatever supported destination format. I still think that the basic idea behind that project is pretty neat, just throw in whatever you happen to have and have something completely else out of the other end. And it worked as it should. It was just stupidly hungry for memory. 20k entries would eat up several gigabytes of memory from a workstation (and back then it was premium to have even 16G around) and it was also freaking slow to run (like 0.2 - 0.5 seconds per entry).
But even then I didn't need to tweet that my hard drive is overheating. I well understood that my code is just bad and I even improved it a bit here and there, but it was still so very slow and used ridiculous amounts of RAM. The project was pretty neat and when you had few hundred items to process at a time it was even pretty good, there was companies who relied on that code and paid for support. It just totally broke down with even a slightly bigger datasets.
But, as I already mentioned, my hard drive didn't overheat on that load.
Seriously - I can parse multiple tables of 5+ million row each... in EXCEL... on a 10 year old desktop and not have the fan even speed up. Even the legacy Access database I work with handles multiple million+ row tables better than that.
Sounds like the kid was running his AI hamsters too hard and they died of exhaustion.
Excel have a limit of 2^20 rows, something more that 1M. Curious what version of excel are you using for that.
You're correct - the standard tabs can only hold roughly 1.2 million rows.
The way to get around that limitation is to use the Data Model within Power Pivot:
It can accept all of the data connections a standard Power Query can (ODBC, Sharepoint, Access, etc):
You build the connection in Power Pivot to your big tables and it will pull in a preview. If needed, you can build relationship between tables with the Relationship Manager. You can also use DAX to build formulas just like in a regular Excel tab (very similar to Visual Basic). You can then run Pivot Tables and charts against the Data Model to pull out the subsets of data you want to look at.
The load times are pretty decent - usually it takes 2-3 minutes to pull a table of 4 million rows from an SQL database over ODBC, but your results may vary depending on datasource. It can get memory intensive, so I recommend a machine with a decent amount of RAM if you're going to build anything for professional use.
The nice thing about building it out this way (as opposed to using independent Power Queries to bring out your data subsets) is that it's a one-button refresh, with most of the logic and formulas hidden back within the Data Model, so it's a nice way to build reports for end-users that's harder for them to fuck up by deleting a formula or hiding a column.
Oh yes, I remember using power query for a few months once I started working with bigger databases, but I saw that moving to Python would be better carrer wise and never came back to excel to do actual work (but at the end everything get exported to excel)
I'd do that if I was given so much stupid access
I've run searches over 60k lines of raw JSON on a 2015 MacBook air without any problems.
No, its an external drive, appearently.
My fucking events table of my synapse DB in postgres is nearly ten times as large, and I ported that from sqlite no long ago, in a matter of minutes. All of the data is on a 2*3 cluster of old 256GB SSDs, equaling about 1.5TB with Raid 0. That's neither really fast, nor cool. But stable.
I mean if we were to sort of steelman this thing, there sure can be database relations and queries that hit only 60k rows but are still hteavy as fuck.
Unless I'm misreading it which is possible it's awfully late, he said he processed 60,000 rows didn't find what he was looking for but his hard drive overheated on the full pass.
Discs don't overheat because there was load. Even if he f***** up and didn't index the data correctly (I assume it's a relational database since he's talking about rows) The disc isn't just going to overheat because the job is big. It's going to be lack of air flow or lack of heatsink.
I guarantee you he was running on an external NVMe, and one of those little shitty-ass Chinese enclosures. Or maybe one of those self immolating SanDisk enclosures. Hell, maybe he's on a desktop and he slept a raw NVMe on his motherboard without a heatsink
There are times when you want a brilliant college student on your team, But you need seasoned professionals to help them through the things they've never seen before and never done before.
Can't be a relational database, Musk said the government doesn't use SQL.
He said many things.
Please remember that he is a genius. Only geniuses say a lot of things.
I rest my case.
Lol he also said cybertrucks don't suck ;)
yes but also why say 60K when you could have literally said anything? I mean surely the fact that he thinks 60K rows a big number is already explaining alot lol.
It's bait.
They probably have an explanation tweet at the ready to make more sense of it. They just want enough 'hurr durr these idiot" comments before they reverse Uno card this with more context.
Based on all that has been going on, I feel like they don't really have the capacity to think more than one step ahead. They do sth stupid and then they usually follow up with "lol joke" or "lol you can't understand"
"You don't know what you're talking about 💩💩💩"
Wait, seriously?
I don't get how people don't see this stuff. Yea the average trumpet isn't out there planning Jack shit. But there are think tanks that are. Remember anti-smoking campaigns. Anti climate change campaign. The precision to purchase advertising into key areas and specific demographics that would spread a message. This is ancient knowledge with modern technology.
Imagine a room full of former Wall Street, quants, established experts from fields like behavioral science and psychology. All with the singular goal to decide where to dedicate a dragons horde of wealth to maximize effect in a world where we all have anonymous pipes directly into our eyes and ears. We never stood a chance. There's no rich socialist funding think tanks. There's no counter. We can laugh at the yokel all we want. But the yokel is being puppeteered by some scary fuckers with intention to seize power with the new shifting Zeitgeist. Soldiers don't need to think. But their generals are. The left are like guerilla fighters going up against an imperial army full of Patons and Eisenhower's.
Cambridge analytical, heritage foundation, international democracy Union. We're fucked until we actually recognize why we're fucked
Somehow I feel over clicking without understanding of the consequences sounds like something a techbro would do
I keep hearing things about these hires he has, I don't think they're naive, At least not as such. They seem to be more power hungry trust fund babies.
But yeah, people with a few years in them would be a moral liability in that line of work.
music theaters also have rows, and they run on sql so logic checks out.
60k of rows is nothing. Fuck, where do you find these “geniuses”?
Tbf we don't know how many columns there are /s
The IRS just switched columns and rows. So there's 60k rows and 330 million columns /s
There's only one reason he wants kids and not experienced adults.
He’s attracted to them?
So, this means they either have a local copy on disk of whatever database they're querying, or they're dumping a remote db to disk at some point before/during/after their query, right?
Either way, I have just one question - why?
Edit: found the thread with a more in-depth explanation elsewhere in the thread: https://xcancel.com/DataRepublican/status/1900593377370087648#m
So yeah, she's apparently toting around an external hard drive with a copy of the "multiple terabytes" large US spending database, running queries against it, then dumping the 60k-row result set to CSV for further processing.
I'm still confused at what point the external drive overheats, even if she is doing all this in a "hot humid" hotel room that she can't run any fans I guess because her kids were asleep?
But like, all of that just adds more questions, and doesn't really answer the first one - why?
Have you ever heard of case of overheating hard drives within the last decade?
Plus, 60k is nothing. One of our customers had a database that was over 3M records before it got some maintenance. No issue with overheating lol
I run queries throughout the day that can return 8 million+ rows easily. Granted, it takes few minutes to run, but it has never caused a single issue with overheating even on slim pc’s.
This makes no fucking sense. 60k rows would return in a flash even on shitty hardware. And if it taxes anything, it’s gonna be the ram or cpu- not the hard drive.
In my experience, the only time that I've taxed a drive when doing a database query is either when dumping it, or with SQLite's vacuum, which copies the whole thing.
For a pretty simple search like OP seems to be doing, the indices should have taken care of basically all the heavy lifting.
I literally work with ~750,000 line exports on the daily on my little Lenovo workbook. It gets a little cranky, especially if I have a few of those big ones open, but I have yet to witness my hard drive melting down over it. I'm not doing anything special, and I have the exact same business-economy tier setup 95% of our business uses. While I'm doing this, that little champion is also driving 4 large monitors because I'm actual scum like that. Still no hardware meltdowns after 3 years, but I'll admit the cat likes how warm it gets.
750k lines is just for the branch specific item preferences table for one of our smaller business streams, too - FORGET what our sales record tables would look like, let alone the whole database! And when we're talking about the entirety of the social security database, which should contain at least one line each in a table somewhere for most of the hundreds of millions of people currently living in the US, PLUS any historical records for dead people??
Your hard drive melting after 60k lines, plus the attitude that 60k lines is a lot for a major database, speaks to GLARING IT incompetence.
Pretty sure I run updates or inserts that count over 60k fairly often. No overheats. Select queries sometimes way higher.
You've got it all wrong, in traditional computer terminology the "hard drive" is the box that sits under the desk that collects cat fluff and cigarette tar.
/s .....?
I don't think I've seen a brand new computer in the past decade that even had a mechanical hard drive at all unless it was purpose-built for storing multiple terabytes, and 60K rows wouldn't even take multiple gigabytes.
Reminds me of those 90s ads about hackers making your pc explode.
Musk gonna roll up in a wheelchair, "the attempt on my life has left me ketamine addicted and all knowing and powerful."
I have when a misconfigured spark job I was debugging was filling hard drives with tb of error logs and killing the drives.
That was a pretty weird edge case though, and I don’t think the drives were melting, plus this was closer to 10 years ago when SSD write lifetimes were crappy and we bought a bad batch of drives.
I'd much sooner assume that they're just fucking stupid and talking out of their ass tbh.
Same as Elon when he confidently told off engineers during his takeover of Twitter or gestures broadly at the Mr. Dunning Kruger himself
Wonder if it’s an SQL DB
Elon probably hired confident right wingers whose parents bought and paid their way through prestigious schools. If he hired anyone truly skilled and knowledgeable, they’d call him out on his bullshit. So the people gutting government programs and passing around private data like candy are just confidently incorrect
My one question would be "How?"
What the hell are you doing that your hard drives are overheating? How do you even know it's overheating as I'm like 90% certain hard drives (except NVMe if we're being liberal with the meaning of hard drive) don't even have temperature sensors?
The only conclusion I can come to is that everything he's saying is just bullshit.
They have temp sensors. But have never heard of a overheating drive.
Hard drives do get hot and need some cooling but not at 60k rows. Its either made up or their computer case is made of thermal cladding
You could query 60,000 rows on a low tier smart phone. Makes no sense at all.
Can we think of any device someone might have that would struggle with 60k? Certainly an ESP32 chip could handle it fine, so most IoT devices would work...
Right? There's no part of that xeet that makes any real sense coming from a "data engineer."
Terrifying, really.
Unless the database was designed by someone who only knows of data as that robot from Star Trek, most would be absolutely fine with 60k rows. I wouldn't be surprised if the machine they're using caches that much in RAM alone.
Imo if they can't max out their harddrive for at least 24 hours without it breaking, their computer was already broken. They just didn't know it yet.
Any reasonable SSD would just throttle if it was getting too hot, and I've never heard of a HDD overheating on its own, only if there's some external heat sources, like running it in a 60°C room
Hard Drives might do it if the enclosure is poorly designed (no ventilation), but I can't imagine a situation where it would overheat like that that quickly, even in a sealed box. 30k is nothing in database terms, and if their query was that heavy, it would bottleneck on the CPU, and barely heat the drive at all.
dude is 100% talking about ssds. NVME ones at that, he's just stupid.
Even if it was local, a raspberry pi can handle a query that size.
Edit - honestly, it reeks of a knowledge level that calls the entire PC a "hard drive".
Unless they actually mean the hard drive, and not the computer. I've definitely had a cheap enclosure overheat and drop out on me before when trying to seek the drive a bunch, although it's more likely the enclosure's own electronics overheating. Unless their query was rubbish, a simple database scan/search like that should be fast, and not demanding in the slightest. Doubly so if it's dedicated, and not using some embedded thing like SQLite. A few dozen thousand queries should be basically nothing.
Yeah, no matter what way you disorganize 60,000 rows, the data is still going to read into memory once.
Why? Because they feel the need to have local copies of sensitive financial information because... You know... They are computer security experts.
Or they're doing it on a Diamondmax 9.
I'm a data engineer that processes 2 billion row 3000 column datasets every day, and I open shit in Excel with more than 60k rows. What the hell is this chick talking about?
Seems like a good excuse to someone who doesn't know what they're doing and needs an excuse because why they haven't completed it yet?
The whole post is complete bs in multiple ways. So weird.
If you work for a boss that fundamentally misunderstands what you are doing, then misleading them into thinking you're 'hard at work, making decisions with consequences' is the theatre you put up to keep the cash flowing.
It's one of the fundamental flows of autocracy, people try and represent what you want them to
It sounds like Hollywood tech lingo. Like when you're watching a movie or a TV show and the designated techy character starts just saying computer words that make no actual sense in the real world, but I guess in CSI: Idiottown the hard drives have severe overheating issues.
Some interesting facts about excel I learned the hard way.
Not really related to what you said, but I'm still sore about the bad data import that caused me days of work to clean up.
The row limitation seems, to me, like an actually-good thing. Excel is for data where you might conceivably scroll up and down looking at it and 1M is definitely beyond the ability of a human even to just skim looking for something different.
An older version of Excel could only handle 64k rows and I had a client who wanted large amounts of data in Excel format. "Oh sorry, it's a Microsoft limitation," I was thrilled to say. "I have no choice but to give you a useful summarization of the data instead of 800k rows (each 1000 columns wide) of raw data."
Some time ago, I heard a story of CS and Econ professors having lunch together. The Econ professor was excited that Excel was going to release a version that blew out the 64k row limit. The CS professor nearly choked on his lunch.
Dependence on Excel has definitely caused bad papers to be published in the Econ space, and has had real world consequences. There was a paper years ago that stated that once a country's debt gets above 120% of GDP, its economy goes into a death spiral. It was passed around as established fact by the sorts of politicians who justify austerity. Problem was, nobody could reproduce the results. Then an Econ undergrad asked the original author for their Excel spreadsheet, and they found a coding error in the formulas. Once corrected, the conclusion disappeared.
Gotta love it when the boss asks for everything and the reply is something like "are you sure? 99% of it wouldn't mean anything to you."
I work in insurance in Brazil, by standards of our regulatory body, claims numbers must be a string of 20 numbers (zfill(20) if needed). You can't imagine the amount of times excel had fucked me up rounding down the claim numbers, this is one of the first things I teach to my interns and juniors when they're working with the claims databases.
Could you import the column in as text to preserve it?
Sure, but sometimes you need to do a pivot table and get transformed, or you while working you accidentally made it a number, or just because excel being excel it import the number as scientific notation, or any other case.
Diming on her own stupidity without realizing it, like all of them do
This sounds like trying to do stuff in Excel? The computer isn't overheating but the amount of memory needed is very high which would make it run poorly. They might interpret that as overheating?
It also makes sense if they are on calling the entire computer "the hard drive" like grandma and the fans kicked on.
Yeah, everyone commenting about being able to handle billions of rows easily, which obviously very true if you are worming with sql or similar.
But this is probably some finance kid, investment banker analyst, and only knows how to use Excel.
60,000 rows in excel with formulas, if not done efficiently, can for sure make you computer a little toasty.
Could be a gen 5 nvme drive without adequate cooling. Them bastards can run hot. Especially the early gen 5 drives.
But with only 60k rows would one even have enough time to overheat?
This cannot be real, wtf. This is cartoon levels of ineptitude.
Or sabotage by someone heading out? Please let this be resistance sabotage they haven’t noticed yet.
You guys arent running your software off raspberry pi's with sdcards from the gas station?
My allowance is 5$ a month!
I think my Pi could process 60k rows without overheating. And the poor thing is dangling behind my bookshelf from its power cord with a fine layer of dust coating every inch of it.
Why would you do this to that poor rPi?
Electronics get what they deserve.
Look, all I’m saying is give Pis a chance.
This shit sounds like when your mom tells you that the Facebook printed out her bank statement on the fax machine. I'm not smart enough to even guess how you did something dumb enough to make that happen.
How bad are you at writing queries? How does your hard drive overheat even under 100% load? Do you have it smothered under a blanket? Did you crack it up and expose it to cheeto dust? What does running a query on your, presumably, remote database even have to do with your harddrive in the first place? Are you trying to copy the entire database locally to a laptop? Do you know how to tie your shoes yet, or are you still on the velcro?
A laptop should easily handle a database of 60,000 rows. I run much bigger databases on my own laptop for development purposes.
Not if it is filled to the brim with 3 TB of porn
Delete Ass Master volume 7 to make room - that one wasn't any good anyways.
Really went downhill after volume 3. You hope that they'll get back on course by volume 6 at least, but alas.
I can personally vouch for that not causing a problem either.
My dear sweet mother asked me somewhere around 2005-06 "If we can fax paper, why not groceries, or pizza delivery?"
Apparently she had believed, for decades, that fax machines literally transported physical paper over phone lines. She has a college degree, and my family is wealthy.
Do not underestimate the mind boggling technical and scientific ignorance of old people who should know better.
🤣
I remember thinking that when I was a kid and saw a clip on PBS Kids of a kid using their home fax machine to fax their dad a picture they drew at work. I was also like 4 so teleportation was still distinctly possible to my brain
You're not supposed to place your laptop directly in the lap of your fur suit. Always leave an air gap for ventilation, smh.
Maybe
Maybe
They are just making shit up and doing jack shit
I really hope so.
Even a gamer knows that ssdd heat up but never to that level, lol.
What kind of cheap temu ssd does he have in his laptop?
When the only thing that is stopping kids from dismantling your government is an O(N^N) algorithm
Are you telling me there’s a difference between an inner and a cross join?
Cross join is obviously faster, I don’t even have to write “on”
As a reasonably experienced "data guy," this seems obviously laughable, but the discussion on X is scary. This guy is a savior in the MAGA world.
We can criticize and poke fun all day, but it doesn't matter much if our message isn't challenging the mindset of those with other opinions.
How do we make better use of our time to impact outside opinion?
I've been told violence isn't the answer and we shouldn't just shoot nazis and nazi enablers dead.
The way most people change their mind isn't based on facts or figures, but emotions. Specifically, in-group belonging. For most people, and this certainly includes me and you some of the time, what our in-group believes is more compelling than an out-groups supposed facts.
They see that guy as someone in their group so they believe him. They see you as a bad outside bad bad bad liar, so nothing you say is likely to get through. (This comic is worth reading on this topic: https://theoatmeal.com/comics/believe )
If you want to change someone's mind, they have to see you as in-group. Not necessarily the same group as what you're arguing with. We all belong to many groups. American, new yorker, white guy, middle aged, yankees fan, etc etc there are many such slices. Like how you can't get a republican to recycle by appealing to environmental concerns (because environmentalists are out-group, so fuck them), but you might be able to get them to recycle via something like "only american ingenuity can turn trash into bridges and tanks!"
This takes a lot of time and effort, and if you don't get them to stop hanging out with the other group, you won't make any lasting changes.
So I think you'd need a multi prong approach:
All of which takes a lot of time and effort, and your opposite number is basically trying to do the same thing. Except they have fox news, trump, and such in their corner.
And, again, I'm told we definitely shouldn't just shoot extreme right wingers and other nazi sympathizers dead. Nor should we burn their houses down. If we're an emergency responder, we definitely shouldn't let them die while thinking to ourselves "they would let so many die. without a thought, their passing deserves no mourning" or similar.
You should definitely nullify if you're on a jury and someone allegedly did violence to a shitty ceo or red-hat, though, bu that's getting off topic.
Wow, that was an awesome rabbit hole, thank you for the link.
Maybe a less manipulative-sounding way to phrase that might be that we should remind people that we're all in it together. The far right media and their billionaire buddies have spent the past decade and a half dividing us, and they succeeded. Idk what it would take to unite this country again, but it at least is a little comforting to have a clear problem statement.
I've been thinking about this a lot lately. I'm from the UK, and whilst things are less politically dire here than the US, it's still pretty grim. Both the Conservatives and Labour seem reluctant to actually meaningfully tax the rich, even as the working class (and to a lesser extent, the middle class) are being squeezed by a cost of living crisis and general hopelessness. Parties like Reform are taking the racist "things are bad because we have too many immigrants" and I've recently realised that I need to stop resenting people for being taken in by that rhetoric; people are desperate and there aren't people in the mainstream pushing for alternatives (besides Reform). These people have a lot in common with me, such as recognising that we're being fucked but the system, but we just disagree on the solution. It's hard, but ultimately necessary to be able to be in solidarity with people like Reform' voters
By the very same people asking the question: What are you gonna do about it?
Where 'it' is your oppression.
We must make better memes
I'm not even joking, the world runs on memes now. It's fucking stupid, but we must shitpost to save ourselves
I agree some form of consistent opposition messaging is needed.
The maga world talks in consistent themes and terminology, which creates a psychological advantage. Unfortunately, it's playground psychology, but if that's the game being played you need to find a way to win at it.
I can't remember the particular phrase that was used, but I heard an argument recently that we need to be more like politicians going on an interview and ensure that we're more on message. For example, it's fairly obvious by now that economically, the problem is wealth inequality, but I see fairly surprisingly few people discussing that.
That's probably who I'm remembering; I recently discovered his work.
Compelling point. I just found that arguing with „these kind of people“ (livibg in europe, so no MAGA‘s here but like-minded, conservative fundamentalists etc.) leads to nowhere. It‘s kind of like the covid-conversations. And often I heard „you can‘t make them change their minds, so just let them be“. Still, I think this behaviour leads to isolation and separates us as a people even more.
Long story short: good question. If you found the answer, let me know.
Good lord.
These people cannot do half of what she can do.
@[email protected]
Literally every time someone dismisses Wikipedia, it's because they believe something crazy that Wikipedia told them is wrong.
I checked conservapedia once, and its actually unhinged. If someone tells you to look at that, or reccommends it, they're crazy.
Did they ever finish their own bible translation? The one they started because King James was too woke.
“I read a book with a typo once. Libraries are a scam.”
Libraries are a scam if they weren't they'd still have VHS rentals.
heh
Molly White is very bright, and she makes them feel inadequate so they "have to" attack her. It's truly pathetic.
But her last name is White so it's a real dilemma for them.
“YOU’RE JUST JEALOUS” is such a fucking pussy-ass response, too.
God they picked out the ONE possible thing they could criticize her for, there's like 3 other things RIGHT NEXT TO THAT
"Software Engineer" was literally right next to it.
She is vehemently against crypto. She runs the newsletter "web3 is going great" - https://www.web3isgoinggreat.com/
Doesn't actually say that 60k overheated his drive. He says that he ran a run on 60k, and that he couldn't do the whole database due to overheating. Two unrelated statements except that 60k is the lower bound for what he could process.
Doesn't mean he knows what he's doing though, as pretty huge datasets are processable on quite modest hardware if you do it right.
I don't want to take away from the valid point of your comment, but rocket science is almost exclusively math
I will now attempt a full comment ... brace your hard drives
Bro seriosuly fuck off my phone is overheating now. Thanks
I cannot believe these people make more than me lol.
They make nothing. They're compensated for destroying things, and considering it's musk, they're likely given relatively little money in return for their time.
Even if the only thing you do all day is sit on the toilet and yell at the Internet, you're already a bigger net positive on society.
We're going to see all of these clowns in front of a grand jury one day.
The reason he recruited a bunch of incompetent college kids is because nobody with any experience or wisdom would touch this rolling crime wave with a 10 foot pole
I would be absolutely shocked if we had anything approaching justice for what this administration is doing.
We barely got anything for that whole ass insurrection attempt.
Most likely compensation is the promise of being part of privatization of whatever the fuck they're destroying.
They're dumb kids. They probably see Musk as some kind of god. He gave them some line like "Hey, you guys wanna save the United States?" and they jumped at the chance.
You get to be a piece of history! Whheeeeeee
As a former dumb kid this tracks
60k lol.
I regularly work with data in the 16tb range and weirdly my computer is fine. Git gud, doge scrubs.
Either she knows something novel, where processing data using voice coils is somehow beneficial, or is someone who calls their computer a ‘hard drive’, which summarily negates any legitimacy of technical competence.
Or wrote the code using AI without checking what it exactly does.
The AI probably used bogosort or something equivalent
What is this, a table for ants? Because that’s the average number of ants in an ant colony and it’s nowhere near an impressive amount of rows to be doing any sort of processing on. It wouldn’t be an impressive amount of rows if your rig was an i386DX-33 running off a 5” floppy.
Exactly, 60k rows is negligible enough in most cases that you can just treat it as free unless you're doing a cross join on it or something, unless he's doing something like using an unordered text file as his database with no ram or cache
Buddy’s probably running code he got from GitHub Copilot that is used to do a visualization of a bubble sort for learning purposes.
Even then, I can crank through orders of magnitude more data on my craptop
Skill issue, as the kids like to say.
I bet a million bucks the harddrive didnt "overheat".
Its just someone who doesnt know anything about computer hardware.
Its like me saying my car overheated if there is smoke coming out of it. I know nothing about cars.
I’ve found a lot of people call the computer itself the “hard drive”.
Yeah... That's what I think the idiot is likely doing.Anyone doing so has no fucking business touching code.Just read that the fuckstick is copying government data onto an external.
60k rows of anything will be pulled into the file cache and do very little work on the drive. Possibly none after the first read.
Not if each row is pi!
You can put 60k rows in Excel 95.
Hard drive was made by Tesla
Look up the Fremont Seagate factory history
I used to perform data analysis of robotics firmware logs which would generate several million log lines per hour and that was my second job out of college.
I don't know how you fuck up 60k lines that bad. Is he nesting 150 for loops and loading a copy of the data set in each one while mining crypto??
Substring searches in unindexed large string columns or cartesian explosion caused by shitty joins would be my initial guess.
Largely ignorant, but data-curious person here.
...what?
If there's something you want to search by in a database, you should index it.
Indexing will create an ordered data structure that will allow much faster queries. If you were looking for the username gazter in an unindexed column, it would have to check literally every username entry. In a table of 1000000 entries it would check 1000000 times.
In an indexed column it might do something like ask to be pointed to every name beginning with "g", then of those ask to be pointed to every name with the second letter "a" and so on. It would find out where in the database gazter is by checking only six times.
Substring matching is much more computationally difficult as it has to pull out each potentially matching value and run it through a function that checks if gazter exists somewhere in that value. Basically if you find yourself doing it you need to come up with a better plan.
Cartesian explosion would be when your query ends up doing a shit load of redundant work. Like if the query to load this thread were to look up all the posters here, get all their posts, get the threads from those posts and filter on the thread id.
That's very clear, thanks.
I'm guessing you'd have to search the database to make the index, right? To search for 'gazter' you'd have had to go over the whole dataset and assigned each entry with a starting letter value, and so on?
When it comes to searching the database, the index will have already been created. When you create an index, it might take a while as the database engine reads all the data and creates a structure to shadow it. Each engine is probably different and I don't know if any work exactly like that, but it's an intuitive way to understand the basics of how B-trees work. You don't really need to think much about how it works, just that if you want to use a column as a filter, you want to index it.
However, when you're thinking about the structure of a database it's a good idea to think what you'll want to do with it before hand and how you'll structure queries. Sometimes searching columns without an index is unavoidable and then you've got to come up with other tricks to speed up your search. Like your doctor might find you (i'm presuming gaz is sort for gary and/or gareth here) with a query like
SELECT * FROM patients WHERE birthdate = "01-01-1980" AND firstname LIKE "gar%"The db engine will first filter by birthdate which will massively reduce the amount of times it has to do the more intensive LIKE operation.Storing large volumes of a text in a database column without optimization, then searching for small strings within it. It causes the database to basically search character by character to find a match by reading everything from disk. If you use indexes the database can do a lot of really incredible optimization to make finding values mich faster, and honestly string searching is better suited to a non-relational DB engine (which is why search engines don't use relational DBs).
Cartesian explosion is where you join related data together in a way that causes your result set to be wayyyy bigger than you expect. For example if you try to search through blog posts, but then also decide to bring in comments to search, then bring in the authors of those comments and all their comments from other posts. Result sets start to grow exponentially in that way, so maybe if you only search a few thousand blog posts you might be searching through millions of records because you designed your queries poorly.
I would love to see the output of EXPLAIN on his query to see what he’s doing.
just a lame-ass excuse for not finding whatever evidence they were looking for.
elsewhere, some seeding was done.
now they'll do the 'full' data grab and 'find' what they were looking for.
Yeah. Hiring inexperienced children into government isn't fraud, by itself. But I bet it makes fraud way easier.
No matter what actually happened here we can confidently state this clown is full of shit and has no idea what he's doing.
They even doubled down: https://xcancel.com/DataRepublican/status/1900565922831618202#m
Well thanks for trying to keep us from catching it I guess
thank god, i am going to be ok, i was worried it might be contagious (it's not, he just has brain damage)
Can you screenshot or something? I can’t load that link
Does this one work? https://nitter.net/DataRepublican/status/1900565922831618202#m Its a bit too long to screenshot.
That works, thanks
So it was a probably shitty external HD… that’s a whole other can of worms
What’s the bet the software they downloaded is malware and it’s crypto mining?
That's pretty plausible. Inexperienced engineer plus overheating...good chance that adds up to a malware infection.
Is this a real post? I can’t seemed to find it on that website “X, formerly known as Twitter.”
some poeple have linked the discussion in other comment threads
I've used local hard drives from like 1992 and I have never ever gotten them to overheat.
“I store my records on vinyl. You’ve probably never heard of them.”
Maybe he left his shed lizard skin on top of the hard drive that caused the overheating?
IT guy checking in.
The only time I've even seen drive temp sensor alarms is on server raid arrays and other similar hard drives/SSDs.... Never in my life have I seen one available on a consumer device, nor have I seen any alarm for and drive temp, go off. It just doesn't happen.
IMO, this is one of those language barriers where people call their computer chassis (and everything in it) the "hard drive".
Applying that assumption, their updated statement is: His computer over heated.
Idk what kind of shit system he's running on that 60k rows would cause overheating, but ok.
I didn't know hard drive overheating was a thing. Should I be worried that my 5 year old hard drive is about to overheat. I mean is this actually a floppy disk or something?
it is, in the select event that your platter bearing fails, in which case it would be very, very obvious.
So you've had this happen?
no. but generally spinning things that spin at several thousands of RPM that are spinning on a bearing, that no longer have a bearing usually sort of uh, tend to be VERY noisy.
Ok, but do you know anyone this has happened to? I don't and I have some pretty old drives I still use. I tend to just buy more. Also most drives these days are solid state aren't they? This just feels like a low probability event to me. Overheating RAM or the CPU or GPU, sure, but hard drive?
I had it happen to a random hard drive I bought, old bastard found in a bin at a local thrift store. Anyways had a big dent in it and while it started up, even booting into windows XP it cooked itself made a screeching noise and was hot to the touch. Don't think I need to explain that the big dent was probably the source of the overheating, anyways had to use an oven mitt to relocate it to a metal bucket of water where it boiled for a minute.
So the story is plausible then?
Sure if they physically damaged the hard drive, though at that point the failure and the code were largely unrelated. Though if they were using a computer with an SSD only I would have no clue. Assuming they aren't lying id have to look at the computer physically, perhaps even pry it open to see if there's any noticable damage.
There's also the possibility that they are just stupid and overheated their CPU by having too many apps open. Then they blamed their hard drive overheating because they are stupid.
not personally, i may have seen a video or two of it happening, but it's hard to tell whether the head is dragging against the platter, or it's the bearing, either one of those makes horrendous noise.
If you're worried about it happening on a drive you own, you should copy that data somewhere else as a backup, ideally sooner rather than later. If you're curious about the health of the drive you do stuff like SMART tests as well.
Yeah, most drives are solid state now, unless you're buying hdds for archival purposes, still cheaper and denser in most cases. It's a low probability failure, until the drive meets EOL, in which case it's a mechanical wear part, either the motor or the bearing fails. One of them will fail first, probably the bearing.
The bearing failing would likely result in the HDD overheating as a result. Assuming the platter still spins, but that's the only scenario i can think of where that would happen, unless you dump a very specific amount of continuous current into the read arm coils. That might also cause it, but it's not likely at all.
An ssd "overheating" is more likely, but it shouldn't cause too many issues, maybe premature degradation over long term use, and slowing of read/write speeds, or in some cases, an improvement, but other than that it should be business as normal. You would have to hit it with like a heat gun, to get a hardware failure or something like that.
I'm not actually worried about my drives. In fact that was kind of the point. I was kidding around because this excuse that the hard drive overheated sounds a little like a car running out of blinker fluid.
oh no they're definitely stupid, that's for sure. I'm just providing the two available scenarios in which they would be right lmao.
When an HDD works continuously it can heat up to above 60 °C if proper air circulation is not allowed, which can cause a very premature failure. In fact, it should be kept under 40 °C to achieve the intended lifespan. Unfortunately, PC cases are usually not great at removing heat from the HDD by default.
As for your drive, it most likely has a temperature sensor so it can be displayed by various utilities.
I have a 12v fan running at 5v spitting air on my hdds, and that's enough for them to go from 55°C to 29°C, lol.
I did that too, the tiny fan is pretty much silent at 5V. The HDD has so much surface area it only needs a little air circulation.
Technically they could have had been using a computer without any temperature sensors or fans and each row was actually a 1GB file.
And also they had a hairdryer pointed the the drive.
It's possible!
Elon isn't sending his best.
Hard drive? How old is his computer?
The trick is to take the hard drive out of the EVA foam.
Instructions unclear, dick stuck in GPU.
What in the fuck is this idiot doing? I've process datasets far larger than that and never once have I run into a hard drive "overheat". I mean what level of incompetence do you have to have to get a hard drive to overheat processing a measley 60K rows of data?
I've met people who think their desktop PC is the "hard drive" but they're all over 60.
does elon only hire chip from sales guy vs web dude or something
No he also hire people who created a script to make fake ballots with a bias.
https://bsky.app/profile/denisedwheeler.bsky.social/post/3lhowh3ijgs2f
lol guys I think this is a shit post
As in, a dumb piece of shit is posting dumb shit.
i mean...
If you're running a pcie nvme ssd, one of the modern ones, and you're doing a SHIT ton of reads, like threadripper level amount of reads, i guess "overheating" isn't unexpected? Shouldn't do much other than slow down the SSD though?
dumbass probably loaded them into memory, and OOM'd, and thought it was the drive.
NVMEs will throttle if they are inadequately cooled. Pretty much only folks who are new at buiding computers and don't adequately cool their NVMEs experience this.
Either way it shows what a clown show this is. I'm pretty sure they're doing some babuki theatre to distract us and then if the scheme being done here is found, they'll point fingers at the dipshit kid and he'll be punished for whatever the con is.
Either way the billionaires will be better off for it, barring judicial, congressional, or military revolt; which is looking less likely by the day.
Bunch of traitors forgot their oath is to the constitution. The constitution IS the soul of the country and is to be respected. The country was founded by the constitution and it defines the system to be decentralized power giving control to the populace through three different branches of government. Not some false idol king or an evil oligarchy propped up by foreign and domestic robber barrons
yeah, but it shouldn't be that significant though right? You're talking like 10% slower speeds to anywhere probably like 50% of the speed, which for modern NVME ssds is basically perfectly usable in most cases. But i'm not up to speed on pcie 4.0 and 5.0 ssds, so idk the specifics.
Or it's a cheap external nvme chassis with a Samsung 980 Pro. Had to run that when I was copying files from one of my old machines and boy, it will absolutely overheat to the point of failure.
Gave me quite the scare when I started getting read errors and then it dropped off the bus. It shutdown to protect itself but it certainly didn't seem like that at the time.
Out of memory/overheating in 60k rows? I've had a few multi-million row databases that could fit into a few gigs of memory, and most modern machines have that much in RAM. A 60k query that overheats the machine might only happen if you're doing something weird with joins.
Plus a lot of reads is nothing really, for basically all databases, unless you're doing an unsmart thing with how you're reading it (like scanning the whole database over and over). If you're not processing the data, it'd be I/O bottlenecked.
if they wrote good code yeah, evidently they didn't write good code if they're struggling to process 60k lines of a database lmao.
They must either be O(n^10) complexity or something retarded like that for this to be the case. I wouldn't put it past them.
again, i'm assuming they aren't very smart, since this is an issue in the first place.