Why is searching the Internet so frustrating now?
A few years ago anything you wanted to search for would generate hundreds of pages of results. From sources all over. If you structured your query well you could dramatically cut down the irrelevant results and then spend a bit of time skimming through the offerings until you found your target.
Searching felt like a hunt, it could take you to far flung corners of the web, and hitting the jackpot on some obscure forum was part of the fun.
Now searching anything feels incredibly frustrating. You know what you are looking for is out there somewhere but it feels impossible getting to it. Basically we have only two main routes now, Google and Bing (and I swear Bing is just copying Google's homework). I've tried a handful of others and they all fall very short.
Something has changed dramatically under the hood.
It seems like the results are very shallow and no amount of prodding will get either search engine to produce anything besides the laziest output. Any subject has a limited pool of sites they lean on and then pretend that that's all of the Internet.
It generalizes very quickly and if you try to counter by putting things in quotations it gives up.
Even in the rare moments I've buckled and used their stupid AI, it takes even more time browbeating the program to just deliver the damned specifics.
What is going on??
How do we get our old search back?
I swear every day of trying to navigate the web feels worse than the day before.
How did we get here?
There's got to be something better, right? Or something we can do to pave a new structure that benefits the denizens of the web better than the lazy slop our tech overlords scrape off the table for us. What can we do?
101 replies
As others have mentioned, it was engineered to be worse for the capitalists to make profits from ads and the statistics to look better to shareholders. This in turn has made people to create a shit ton of garbage articles and blogs that say a ton while saying nothing, so even if you used a new search engine with the funding of Google but the algorithm of back then it would still be shit probably, because the volume of shit on the internet has now reached a level where finding anything in it is impossible.
The explosion of AI has come right at this moment where search engines have been lobotomized and gives people an option to be able to find things again, which is also another factor contributing to the profits of the AI companies, but even that is limited to what an LLM can average out from the totality of information that it has been fed and has the obvious danger of making stuff up and treating its biased training data as truth.
Generally, I think we're fucked. I really don't know if we're ever going to get back the internet experience that once existed and what it's going to take to get there, but it would probably involve a massive societal change to come before it.
At this point I think the only way forward is for enough like-minded people to sort of stake their claim on some semi-walled off portion of the internet and greatly limit access from swaths of IP addresses.
Sort of like web rings 2.0. Everyone, or enough people, will have to chip in on costs and provide either money or hardware.
There will have to be some sort of regulatory body with voting members.
To be 'in' the site has to abide by certain advertising restrictions and privacy agreements.
A decent use (finally) for llms would be to audit the private index and remove slop. I think we're just at the beginning of what will be a resource hogging AI arms race.
It's sort of what we've been trying to avoid since the beginning but it seems inevitable at this point.
I'd add that traditional search can't reasonably hope to compete with AI in this environment. If you're running a traditional search engine right now, there's a real argument that the best business decision is to accelerate the enshitification to capture as much value as you can from your remaining audience while you still have it.
This is the usual cycle with these sorts of technologies - once a replacement arrives, the natural choice is to shift from growth to exploitation.
This is going to get worse before it gets better.
I have a bunch of Search engines where they say different things about your search (I combine it with noscript because fuck JS)
Startpage: you need JS to verification, basicaly is an API from google.
Mojeek: also JS verification, their results are focus in local websites (.es .nl .de).
Searx.be: you need write a code before start use it, index all the fediverse, opensource frontends and scientific works in an anti-massmedia site, also can fail in specific searchs.
I trying different ones, because as you say is not realiable anymore so I have to act as if I asking help to a bunch of greddy friends with their own interests
I have not had good luck with any search engine I've used lately, even Kagi. I pretty much have to go directly on the website I expect to find what I'm looking for and search there now. Also I suspect that with AI people might be posting in forums less now.
SEO. Everybody is blaming corporations but nobody today can develop a search engine that can't be spammed. If they could, they would.
If only we (the collective users) spent more time creating a framework and vision of how the Internet should develop and aggressively pushed back on attempts to deviate from it...
When the people refuse to convene a governing body with teeth, the vacuum is filled with opportunists.
In a sense we are suffering from our own success. A place where anybody can build anything they want digitally. Left a ripe field for unrestrained commercial interests to infest the whole thing. Now it is bloated with human and AI slop to such a degree it smothers everything else and every major tech push is tuned towards data scraping every morsel.
True, but that reeks of victim blaming. The spammers are to blame here, not the mathematical inevitability of antispammers being behind on the arms race. Also, fucking AI.
No. It’s a challenge to all the brainiacs out there hacking high security servers with a freaking TV remote. I hope that they’ll read “Nobody today can develop a search engine that can’t be spammed,” and say to themselves, “I accept that challenge.”
Also, AI has been spamming forever. One of the tricks they did in the 2000s was to spam forums with links to websites so that Google would think those sites were more important than they actually were. In 2009, I hosted a forum that went nowhere, left it unattended for a month and came back to find bots had spammed it into oblivion with links. AI isn’t a new problem.
People with bad info put more effort into SEO than people with good info. Google changing focus from getting people good results to getting people to stay on their site. Powers that be filling the web with misinformation. There's always been conspiracies and misinformation on the internet, but now there's major money in pushing it.
There are leaked emails from Google (from many years ago) where it was proposed to purposely degrade search quality to increase ad clicks. While some at Google responded with dismay as this, it is not a great leap to assume that this faction eventually won and they did in fact deliberately do that. Advertising is Google's main revenue source, they are clearly motivated to do so.
SEO.
I think they intentionally made search engines worse to push people into using LLM's
Which are amazing now, but not to worry, LLMs will be complete dog shit too once they pull the rug.
They already are dogshit.
If your LLM prompt isn't about something common or mainstream is has an incorrect response most of the time. Unfortunately due to how they work, they will present the incorrect answers with the same confidence as correct answers.
LLMs seem like their only use is writing reports or resumes that no one needs to read. Everything else is better done by hand.
Ever since the deployment of machine generated results for search engines, there has always been a competition between the search engines and various others using search engine optimization to get to the top of the results. It isn't worth it for Alphabet or Microsoft to fix their search engines if they can push you to AI, so their non-AI results naturally degrade.
Search quality has been degrading for years, even before AI was involved.
But it feels it got much worse much faster over the last 2 years.
The one thing "AI" truly excels at.
Episode of the Better Offline podcast: The Man That Destroyed Google Search
Sheesh, the description sounds like a Behind the Bastards episode.
Can't wait to give it a listen. Thanks
Because data is never free. Google, duckduckgo, kagi, they all cost money one way or another whether that be a subscription service or by forcing websites to be shit so your only option is using their LLMs. Yes the reason why the internet has gone to shit is because of LLMs
I've went through hundreds of search results on duckduckgo, even when I make it intentionally broad and genetic that wikipedia would cover it, it still doesn't come as the first search result. Search engines have gone to shit because LLMs make the most money in terms of data surveillance. Meaning (take for example google) they have a profile of you, they have queries to train, they have data that can be sold to advertisers. The only reason LLMs are being pushed so much isn't because its intelligent, it's because it makes the most money.
There is a "search engine" that hasn't gone to shit, and that's called marginalia search, It's not exactly a search engine where you can type a url and it will bring you to a page, Instead it compliments your general search engine as a research aid and you can find some golden sites.
I've personally used it for data sheets on IC electronics and their implementations. Unfortunately I've found it insufferably difficult on duckduckgo (like multiple pages deep for a datasheet, something that should pop up on the first page), and I don't trust the output of LLMs (large lying models).
So yea, hopefully this helps.
Ironically I think I've gotten better results from AI-optimized search engines than from DuckDuckGo
Others have addressed what I'm about to say, but there is an answer older than LLMs: advertising.
Google made an amazing search product, one used by so so many people from all over. This made them a perfect platform for advertising, and as time passed and it became their primary focus, they realized they could make even more money people spent more time on their own site.
How do you do that, when the product people are there to use is aimed to redirect you elsewhere? Make it take longer to find the thing they are looking for. Search was incentived to work less well so that users saw more ads and for longer.
Integrating AI into search results introduced new ways to keep people on Google itself, including by presenting the answers themselves directly. But it started earlier.
I'll have to check that out. Thanks
SEO did this. It started before LLM and chatGPT. Blogger make money be generating trafic. So they wrote shits in a format that please search engine.
That's why if you search "search into file Linux", instead of just a grep command line, you end up on 2000 lines article with introduction, paragraph, explanation, blablabla. Because search engine boost this kind of content. Title, introduction paragraph that repeat the title, subsections, etc...
yeah, most slop content already existed before chatgpt and was written by humans. kind of like a student is asked to write "a minimum of 500 words for their homework assignment", too many bloggers copied that attitude to their blogs.
Sure, I'd say most topics already had slop content. However, LLMs can generate entire websites of slop content in minutes if not less. It's pointlessly inefficient to employ people to generate slop by hand. Entire news sites are now just LLM generated slop because it was determined to be cheaper than human generated slop. Most slop content is now, and will going forward be, LLM generated.
To my old ears, 'bloggers' sounds way too much like the self-employed people with a web-site about their hobby, that they are trying to make a living through.
But there were already content farms, that paid an army of underpaid people to write something about a subject they had no clue about and plaster it with SEO keywords in order to attract ad revenue.
Money.
Anytime you think it's not about the money, it's about the money.
I read an article about it somewhere, but, can't find it. Basically the guy in charge of search realized they were at the highest level of users they can get so the only way to increase search traffic is to make it just bad enough you have to search different phrasing a couple of times to get the result therefore increasing search traffic.
https://www.wheresyoured.at/the-men-who-killed-google/
Is this intentional ironic?
Lmao no it wasnt I did search for it too
I've read that, too. Probably that same article. I've read a few other explanations, about how SEO "ruined" the Internet. How they've deliberately handicapped search so you'd use AI, etc.
I feel like all of this is true but there is also more to all these stories than meets the eye.
To me it feels like back in the the day everyone just collectively allowed the tech giants to be lifeguards. But then we find out that they've been taking money on the side to let people shit in the pool.
Now the pool is completely filled with the commercial sectors turds and the "lifeguards" want to sell us a scooper to get our carefully curated results out, sans logs.
This one maybe ? https://www.wheresyoured.at/the-men-who-killed-google/
Yep
The thing that is eternally forgotten when people ask these sorts of questions, or perhaps it is just deliberately pushed to the back of their minds is that the corporations providing these services are not in the business of providing services, they are in the business of making profit, primarily by finding ways to push as much advertising as is possible. Remembering that is really the only true answer to these sorts of questions.
"Number go up" is all that counts.
Diabolical
I find it amusing and ironic that you couldn’t find it.
If I remember correctly, around 5 or 6 years ago, Google Search leadership changed such that the ads org started running it. The focus since then has been on advertisers rather than regular users.
This is literally the definition of enshittification: the company first focuses on gaining users, then switches the focus to business customers, then squeezes both of them.
In Cory Dowctorows latest book he describes that it was revealed that google not only neglected the search function, but deliberately made it worse. If you have to search several times to find a good result, they can show you several ads.
https://pluralistic.net/2024/04/24/naming-names/
Wow, what a great site. I like the cut of this guy's jib.
Doctorow is awesome. He's a great author too. I'd recommend https://en.wikipedia.org/wiki/Walkaway_(Doctorow_novel) and https://en.wikipedia.org/wiki/Little_Brother_(Doctorow_novel)
Thanks! This is what I was thinking of, but ironically I couldn't find the links/articles lol
the reason why google search results went to shit is because google did some basic maths.
99% of search queries actually ask for information that you can find on one of the top websites. only about 1% actually care about anything besides that.
so if you deliver results only due to a rather superficial algorithm, you will still retain about 99% of your userbase, but the computation cost for finding these results would go down dramatically. so you save a lot of money on electricity, while serving almost as many ads. it pays off for google.
SEO and decent search results were having a fist fight for years, then AI showed up and shot it in the face.
Because "Fuck you, pay me"
I've never seen any search engine officially confirm this, but search across all search engines became terrible just a few months after the release of ChatGPT.
Which is why my assumption is that LLMs are being used to generate heaps of credible-looking, but ultimately worthless webpages. And search engines have to try to filter those out or their results would be even worse.
But then there's no obvious distinction between a human-crafted blog and some spam webpage, which means the human-crafted blog is likely to be caught in the crossfire and get filtered as well.
Well, and on top of that, search engines presumably got hit by AI scrapers looking for training data, so that's why I would assume, they now limit the search results to one page...
Yeah. Noticed this too. Pages that look legit but are filled with useless word salad that doesn't go anywhere.
Gives you an impression that those pages didn't exist before your search. How it got to top results is beyond me.
Those sites train their models for SEO is how they get top results.
And I agree, somtimes even the domains look like they are generated on the fly to suit your search keywords.
On DDG (can't say for Google as I don't use it) search results include a date & I automatically disregard anything dated very recently - almost always slop
I've been using metasearch engines (engines that pass your query to multiple search engines so you can aggregate them easily) and the two I've been using were pretty good in my experience. 4get.ca and degoog.org.
I'll check them out, thanks
It's called enshitification, basically web services get worse so that you spend more time on them and see more adds. Switching from Google to Duck Duck Go has made things a little bit better for me.
Sometimes I want a technically savvy person to explain the inner workings and how it's changed, but lately I'm just so fed up with it I want to figuratively dump it all in the trash, cancel my data plan, and just live without any Internet at all.
It feels like we lived through a golden age and it'll never be that good ever again.
Yeah, that's why I'd just like to write now. I'm done with it all. I'll wrestle with a fucking word processor, and everyone else can eat a dick.
I do think that part of the problem is that so much up-to-date information is now trapped in discord, also that people are generally using forums and the like less to solve their technical problems because they're privately querying robots. This on top of enshittification... We did not know how good we had it 20 years ago
That and everything is a YouTube video now. Or at least that's what the search company that makes money off their video service wants you to think.
These points are valid in their own right but I don't think they explain how Google has become much worse at finding things that really are in plain text on a website somewhere. Things that could be found, that is now buried.
SEO got really big around 2016 and Google has done nothing to help users get around their algorithms being gamed. Instead, it appears they've leaned in to the fact that companies were ready to pay for better treatment by the Search.
You might try to give one of the alternative search providers a try. There's duck duck go for the USA, and Qwant for the EU.
Important to note that Qwant still uses the Bing search index everywhere except for France and Germany, where they use the Staan search index, which is being developed together with Ecosia.
I'm sure the more people use it, the faster they can expand, though.
Using it from Finland myself, quite satisfied with it even with this Bing nonsense. Only have to use Google for VERY RARE cases.
I know it's not the thing you likely want to hear, but... At this point, I'm glad I switched to a paid search engine (Kagi).
Theoretically, there's nothing stopping them from enshittifying down the road. But at least for now (and hopefully the future) they are a great search engine.
They have their own index (so no relying on google/bing), and it actually works as I remember Google working 7+ years ago. Minus the ads, and plus a ton of flexibility permanently boosting/removing sites from your searches, auto-grouping/ignoring listicle-type articles in results, flagging sites as AI slop and punishing them for it,...
They even have a fediverse search "lense" that actually works really well to search Lemmy/mastodon/...
I just realized I sound like a paid shill. Sorry about that. I'm not.
My understanding is that they use Google's index. It's just that unlike Google, they aren't motivated to give you shit results to keep you on their page longer.
...oh. My disappointment is immeasurable and my day is ruined.
Looks like only their "small web" initiative is 100% self-indexed, and everything else is indeed a mixture of sources.
Oof.
I guess tue point still stands: the results are a lot better than plain Google, all the bullshit is stripped out and you get to customize your results in actually useful ways.
Still sad though.
They can still re-rank Google's results to surface more relevant information.
I use Kagi too, it's great.
The other commenter corrected your index note.
What I found most illuminating was a comment by the CEO in an interview, perhaps The Verge? The TLDR of it was that Google makes money when you dwell on their site (so they want the highest ad serves per search), and a paid search engine loses money every page they have to serve you (so they want to give you your answer fast).
So, the user and Google are at odds, and the user and Kagi have the same goal.
It's simple. Cause Search engines transformed to ad engines.
I use kagi and metager. They've solved it for the most part for me.
As a kagi user I agree! It’s worth the paid service. My results are good, and their features to pin/demote sites make personalization actually personal and not some black box algorithm.
Kagi is definitely worth every penny. It feels like keyword searching in the 2010s, and that's a huge compliment. Their small web tool has led me to some fun blogs for my rss feed, but there are a lot of ai bro blogs on there too which is incredibly ironic.
How come you use both? Do they have different use cases or specialities?
Because I bought metager first and have a bunch of tokens left 😅
There was more money in showing you things than there was in helping you find things.
If you ever watched any of the saturday morning, over the air, wrestling "news" shows in the 80s and 90s, then that's all everything is now. There used to be a pretty good article about it.
You sort of said it yourself. The search used to be much better but it's so bad now you caved and used their ai. You wouldn't have done that if the quality of the search engine remained the same but they need us to use their ai to justify the cost of it and show investors how successful it is. And then the reduced traffic to the sites hosting content you're searching for hurts the sites' revenue, putting them at risk of shutting down And it hurts any still decent algorithms ability to find the site. The goal is to make us dependent on ai for eveything, control the information we have access to which makes it easier to manipulate and control us to squeeze us for more profits.
Ive been getting by with brave and duck duck go but ive still noticed a dip in quality of search results. There are lots of other engines out there to try though!
Thank you for coming to my tin foil Ted talk.
If you're not paying for it, then you're not the user, you're the product.
Open source: am I a joke to you?
If you're paying for it you're probably still the product.
If you're paying for it you're definitely either already the product, or will be eventually when the company sells out.
The only enshittification-resistant choice is copyleft Free Software.
A search service is different from the Linux kernel or a web server though. You can release the perfect open source search engine, but running it will be expensive. If we, as users, aren't paying for it, then someone is going to have to extract value from us to pay for it.
That's kinda true for any online-based service. Having the the software FOS is great, but someone somewhere has got to run the servers.
I wouldn't be able to write this, if there weren't people who set up the instances for my user account and this community here.
The difference with FOSS and the fediverse especially is, that any of us could be one of these people.
Sure, it is kind of true for any service. But the resources needed to scrape and index the entire web are so enormous that, AFAIK, only 2* companies are doing it. Every other search engine goes through them. Kagi only indexes the "small web". It's such a large difference in degree that it becomes a difference in kind.
*There was something about Qwant and Ecosia building a third. Haven't read up recently.
Because it makes AI much more appealing.
No more reading about the recipe author's grandma and how the food reminds them of home. AI will extract the useful info and ignore the SEO bullshit.
"AI" is why you now get thirty different pages talking around the topic you've searched for but not actually answering the question, reaffirming that they're talking about what you're searching for by repeating a useless bit of information about it, patting you on the back for thinking up such a nuanced topic and how only the smartest people think to ask this question, then three paragraphs of tangentially related information with an ad inbetween each one, before you find out that the page doesn't actually have the information you're after because the page was written by clankers.
Which is probably the intent of these cogsucking companies pushing this slop; flood the zone with useless trash, convince everyone that this sudden increase in quantity of websites necessitates another 900GW of data centre capacity right up in the ass of the nearest low income area.
Lol no
Or it might tell you to put glue in your pizza
The goal of a search engine is to make the company money, not to aid you in finding what you are looking for.
We're in between a rock and a hard place. The only way to shake the grip of enshitified companies is to either turn to government/tax backed public utility type incentives, which opens up a whole other can of worms, or lean on crowdsourcing, which is very hit or miss.
Because the AI hallucinations are pushed ahead of traditional search engine results?
This evening I was repairing something. My google search gave an AI summary that gave two important dimensions of a part I needed. I was like "OK let's go eat"
Two hours later, because I didn't write anything down, I entered the same search verbatim and got a different AI summary missing one of the dimensions.
We have made computing non-deterministic. We're busy giving sentience to refrigerators that can't keep ice cream frozen.
Check out SearxNG.
SearxNG is just a meta search engine that uses other search engines, so it'll have the same issues as the upstream search engines.
I tried using an instance on a public page and after about a month I gave up. Maybe it was configured poorly but it was real hit or miss. Sometimes it'd deliver but just as often the results were nonsensical.
There's probably some instance that's really dialed in but I quit looking after the first go around.
Google in particular getting worse to get you to click through more results pages or to run more queries so they can sell more ads on each of those is well attested. I used Bing for a while and didn't think it was better or worse than newly bad Google. So now I use an equally shitty non-US search engine (start page) and just give up sooner.
In their defence, I think the exponentially increasing content of the internet, and the not insignificant portion that just sits there dead or unmaintained, plus the bits and bytes that just disappear as subscriptions or support run out are difficult problems. It's getting ever harder to write an algorithm to sort through that as well. That plus greed equals shit.
Pre-Google we had very shitty search results. Than Google became the Kleenex, the word everybody used. Then Google got worse. While my faith in "American Intelligence" is somewhat limited, I think we will get back to better results soon. What Google had to do (build big server farms to look at all of the internet to index it) may become trivially easy for a sufficiently good LLM eventually. Although we're not there yet. Given that it's not just Microsoft or Google fighting in that market for dominance, I live in hope that competition from OpenAI etc. lights a fire under their arses.
Startpage is owned by an American advertising company.
Try Qwant or Ecosia. They're working together to develop an independent search index (Staan) and it's already in use in France and Germany.
Eff me, really? Can you back this up? I only see a company in The Netherlands.
Just check their Wikipedia page.
Good points.
Legacy stuff is definitely getting buried under the new wave. Exacerbated by the fact that "new" sells while old "costs somebody to maintain".
Speaking of costs, I guess the free lunch is about over. Besides the cost to access, most of the web was essentially free. A little bit of banner advertising paid for a simple website. But between then and now it's becoming clear that the continued expansion of increasingly invasive marketing and data collection to pay for larger and large sites, and their complexity, is not sustainable.
As others have said in the posts, you can no longer find information that was found many years ago. From my point of view, I think there are many causes. Some of them to copy from a movie would be "information is worth money". Another cause would be that the information you are looking for is no longer wanted to be searched. Because of this, I noticed that on the dark web sites there is still some information. In short, the information will be limited.
Copy - pass in your pc
Greed