Many consider AI bots as very bad because they ignore established guidelines when crawling a given site. There is a well known and respected way sites can tell good bots how to behave, called robots.txt. Any website can create one and expect good bots (Google, Bing, Facebook, etc) to honor it. For example https://lemmy.world/robots.txt tells bots what paths it should NOT crawl, and the maximum speed at which it should crawl the site.
Many AI bots have been seen completely disregarding this established standard.
It's a huge company, and I worked in a small subsection of it. For the most part, our team was fairly detatched from the main operations. It was good working there, the corporate BS didn't touch us too much and the people were great. I left for a job that was more Linux and networking focused. Not sure if that was a good move entirely, looking back. But if I stayed, I'd probably be getting bored if I didn't move into a different role
I’d consider search engine bots from the likes of Google, Microsoft, etc. to be benign. They provide a useful service to websites and honor robots.txt directives.
Bots that impersonate Google etc. and/or ignore robots.txt are considered bad bots.
1
Estimated share of global web traffic generated by humans and bots (2018-2025) | Spyke
/r/data-is-not-beautiful :(
/c/data-is-not-beautiful
Does AI scrapers and agents count as bae, or good bots ?
Many consider AI bots as very bad because they ignore established guidelines when crawling a given site. There is a well known and respected way sites can tell good bots how to behave, called robots.txt. Any website can create one and expect good bots (Google, Bing, Facebook, etc) to honor it. For example https://lemmy.world/robots.txt tells bots what paths it should NOT crawl, and the maximum speed at which it should crawl the site.
Many AI bots have been seen completely disregarding this established standard.
I used to work for Thales
How was work there? Why'd you leave?
It's a huge company, and I worked in a small subsection of it. For the most part, our team was fairly detatched from the main operations. It was good working there, the corporate BS didn't touch us too much and the people were great. I left for a job that was more Linux and networking focused. Not sure if that was a good move entirely, looking back. But if I stayed, I'd probably be getting bored if I didn't move into a different role
What are examples of benign bots?
Google's search indexer.
I’d consider search engine bots from the likes of Google, Microsoft, etc. to be benign. They provide a useful service to websites and honor robots.txt directives.
Bots that impersonate Google etc. and/or ignore robots.txt are considered bad bots.