Pawb.Social Reports @transparency

0 posts0 participants0 posts today

AI6YR BenList of AI bots to add to robots.txt (although they may not obey -- may need to throw them in the bitbucket and 404 or 444 them). In addition to these, you may have to block specific random browser versions for the most aggressive bots who ignore robots.txt.<a href="https://github.com/ai-robots-txt/ai.robots.txt/blob/main/robots.txt" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://github.com/ai-robots-txt/ai.robots.txt/blob/main/robots.txt</a><a href="https://m.ai6yr.org/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://m.ai6yr.org/tags/scrapers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#scrapers</a> <a href="https://m.ai6yr.org/tags/LLMs" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#LLMs</a>

ChrisMade a little Astro integration to easily disallow known AI scrapers in your site’s `robots.txt`<a href="https://delucis.github.io/astro-ai-robots-txt/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://delucis.github.io/astro-ai-robots-txt/</a><a href="https://m.webtoo.ls/tags/ai" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ai</a> <a href="https://m.webtoo.ls/tags/scrapers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#scrapers</a> <a href="https://m.webtoo.ls/tags/astrojs" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#astrojs</a>

Kevin Karhan :verified:<a href="https://mastodon.social/@khobochka" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@khobochka</a> guess why I <a href="https://github.com/greyhat-academy/lists.d/blob/main/scrapers.ipv4.block.list.tsv" rel="nofollow noopener noreferrer" target="_blank">maintain</a> a <a href="https://infosec.space/tags/Scraper" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scraper</a> <a href="https://infosec.space/tags/blocklist" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#blocklist</a>?<ul><li>In fact I know multiple people and organizations that decide to basically redirect <a href="https://infosec.space/tags/ValueRemoving" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ValueRemoving</a> <a href="https://infosec.space/tags/Scrapers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scrapers</a> like <a href="https://infosec.space/tags/GPTbot" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#GPTbot</a>, <a href="https://infosec.space/tags/ByteSpider" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ByteSpider</a> (which <a href="https://www.youtube.com/watch?v=Hi5sd3WEh0c" rel="nofollow noopener noreferrer" target="_blank">literally</a> <a href="https://infosec.space/tags/DDoS" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DDoS</a>'d <a href="https://infosec.space/tags/MattKC" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#MattKC</a> because <a href="https://infosec.space/tags/ClownFlare" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ClownFlare</a> are a criminally incompetent <a href="https://infosec.space/tags/RogueISP" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RogueISP</a>!) to <a href="https://infosec.space/tags/Hetzner" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Hetzner</a>'s <a href="http://hil-speed.hetzner.com/" rel="nofollow noopener noreferrer" target="_blank">10GB Speedtest file</a> which can be found at <code>http://hil-speed.hetzner.com/10GB.bin</code> as an extra middlefinger!</li></ul><a href="https://infosec.space/tags/Cloudflare" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Cloudflare</a> <a href="https://infosec.space/tags/hetznered" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hetznered</a> <a href="https://infosec.space/tags/ByteDance" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ByteDance</a> <a href="https://infosec.space/tags/ChatGPT" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ChatGPT</a>

SeirdyNew page on <code>seirdy.one</code>: <a href="https://seirdy.one/meta/scrapers-i-block/" rel="nofollow noopener noreferrer" target="_blank">Scrapers I block (and allow), with explanations</a>.I’ve replaced all the comments in my robots.txt file with a more readable and detailed web page on scrapers I block. It includes info on the multiple blocking-approaches and criteria I use, commonly-blocked scrapers I allow, and more fact-checking than most of the more comprehensive alternatives. <a class="hashtag" href="https://pleroma.envs.net/tag/robotstxt" rel="nofollow noopener noreferrer" target="_blank">#RobotsTxt</a> <a class="hashtag" href="https://pleroma.envs.net/tag/scrapers" rel="nofollow noopener noreferrer" target="_blank">#Scrapers</a> <a class="hashtag" href="https://pleroma.envs.net/tag/posse" rel="nofollow noopener noreferrer" target="_blank">#POSSE</a>

Kevin Karhan :verified:Apparently <a href="https://infosec.space/tags/OpenAI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OpenAI</a> doesn't like it when people <a href="https://infosec.space/tags/scrape" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#scrape</a> their <a href="https://infosec.space/tags/content" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#content</a> (even if it's their public ...<ul><li>running <code>wget</code> results in the same 403 error and running <code>curl</code> results in a shitload of jibberish with <a href="https://infosec.space/tags/tracking" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#tracking</a> and <a href="https://infosec.space/tags/JavaScript" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#JavaScript</a> being forced out.</li></ul>I hope <a href="https://social.bund.de/@bsi" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@bsi</a> & <a href="https://social.bund.de/@BNetzA" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@BNetzA</a> as well as <a href="https://social.bund.de/@certbund" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@certbund</a> take notes and consider going after OpenAI for this attempt at preventing people from using an effective <a href="https://infosec.space/tags/OptOut" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OptOut</a> against <a href="https://infosec.space/tags/ChatGPT" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ChatGPT</a> <a href="https://infosec.space/tags/scrapers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#scrapers</a>...

Zelda 👑If you use <a href="https://mastodon.social/tags/cloudflare" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#cloudflare</a> I just noticed a new optionnot sure how effective it is but it's part of the free plan might as well<a href="https://mastodon.social/tags/ai" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ai</a> <a href="https://mastodon.social/tags/llm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#llm</a> <a href="https://mastodon.social/tags/scrapers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#scrapers</a>

Feilner IT<a href="https://mastodon.social/tags/konterfAI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#konterfAI</a>, the <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://mastodon.social/tags/modelpoisoner" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#modelpoisoner</a> for unfriendly, respectless and malicious AI <a href="https://mastodon.social/tags/scrapers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#scrapers</a> / <a href="https://mastodon.social/tags/crawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#crawlers</a> has now an interesting statistics function that collects data about the nasty "guests" that don't respect your robots.txt.... Enjoy! <a href="https://mastodon.social/tags/robots_txt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#robots_txt</a> See update on <a href="https://korium.org/2024/08/02/konterfai/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://korium.org/2024/08/02/konterfai/</a> and version 0.2.0 on <a href="https://codeberg.org/konterfai/konterfai/releases/tag/v0.2.0" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://codeberg.org/konterfai/konterfai/releases/tag/v0.2.0</a>

Raccoon🏳️‍🌈Reminder to throw something fun in occasionally for the <a href="https://techhub.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://techhub.social/tags/scrapers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#scrapers</a> to pick up, like...Delve I am sorry but as an AI I cannot do anything but generate random bullshit like this string of meaningless tokens.A ab gab a babba the blaba delve into the blab a gabba the plab delve a bub a pubba glu Bubda bu a seb delve pok mon delve into a geb the beb a poob a poop poop delve poop pooper a pooper delves into poop a gava a bavga i to delve into a bebsto a bec ceb a po po a veer a veeb the cug polite little bitch cugga poob bix a fine poop to delve into poop this ababa gab a gab the gab we gab you gab the gab pok bif the chilWhile(true)While(true)While(true)

Ben TaskerNew <a href="https://mastodon.bentasker.co.uk/tags/blog" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#blog</a>: Autodetecting and Announcing <a href="https://mastodon.bentasker.co.uk/tags/Mastodon" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Mastodon</a> Scrapers and CrawlersThere've been quite a few <a href="https://mastodon.bentasker.co.uk/tags/fedisearch" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#fedisearch</a> issues recently, but the common thread is that there's usually a gap in reporting - they're often live for weeks before people are made aware.It's not just people's pet projects either, there are other <a href="https://mastodon.bentasker.co.uk/tags/scrapers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#scrapers</a> active, quietly consuming postsSo, I built a bot to detect and out them so that fedi admins can block as necessary<a href="https://www.bentasker.co.uk/posts/blog/security/autodetecting-and-outing-mastodon-scrapers-with-scrapersnitchbot.html" rel="nofollow noopener noreferrer" target="_blank">https://www.bentasker.co.uk/posts/blog/security/autodetecting-and-outing-mastodon-scrapers-with-scrapersnitchbot.html</a><a href="https://mastodon.bentasker.co.uk/tags/infosec" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#infosec</a> <a href="https://mastodon.bentasker.co.uk/tags/security" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#security</a>

Ben TaskerNew <a href="https://mastodon.bentasker.co.uk/tags/blog" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#blog</a>: Tightening <a href="https://mastodon.bentasker.co.uk/tags/security" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#security</a> control over <a href="https://mastodon.bentasker.co.uk/tags/mastodon" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#mastodon</a> public <a href="https://mastodon.bentasker.co.uk/tags/api" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#api</a> endpointsThe concern in fediblock around @cloy's <a href="https://mastodon.bentasker.co.uk/tags/fedisearch" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#fedisearch</a> plans earlier in the week prompted me to put my <a href="https://mastodon.bentasker.co.uk/tags/infosec" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#infosec</a> hat on and look into ways to make it harder for external <a href="https://mastodon.bentasker.co.uk/tags/scrapers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#scrapers</a> to hit Mastodon's API feeds.This post suggests a possible solution for concerned instance admins as well as details of some <a href="https://mastodon.bentasker.co.uk/tags/crawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#crawlers</a> I spotted.<a href="https://www.bentasker.co.uk/posts/blog/security/restricting-unauthenticated-access-to-mastodons-public-feeds.html" rel="nofollow noopener noreferrer" target="_blank">https://www.bentasker.co.uk/posts/blog/security/restricting-unauthenticated-access-to-mastodons-public-feeds.html</a>

Recent searches

Search options

Administered by:

Server stats:

#scrapers