Create accountLogin

Recent searches

No recent searches

Search options

Only available when logged in.
pawb.fun is one of the many independent Mastodon servers you can use to participate in the fediverse.
This instance aimed at any and all within the furry fandom, though anyone is welcome! We're friendly towards members of the LGBTQ+ community and aiming to offer a safe space for our users.

Administered by:

Server stats:

311
active users

pawb.fun: AboutStatusProfiles directoryPrivacy policy

Mastodon: AboutGet the appKeyboard shortcutsView source codev4.3.7+glitch

If you have a personal website like me, and are constantly fighting to keep web scrapers off your site, then you should add this User-Agent to your WAF rules:

CheckMarkNetwork/1.0 (+checkmarknetwork.com/spider.ht)

Just another company dedicated to snitching on you to copyright trolls by scraping your website without your permission.

www.checkmarknetwork.comCrawler Policy – CheckMark Network
#WebHost#SysAdmin#Copyright
Nov 28, 2024, 01:49 AM·Public·Web
6boosts·4favorites
Collin Atworth :unverified:
Public

Oh boy, here's another company scraping your website to benefit corporations.

User-Agent:
Mozilla/5.0 (compatible; VelenPublicWebCrawler/1.0; +velen.io)

a screenshot that reads as follows: "Velen is a crawler written in Go for Hunter. It analyzes millions of public internet pages every month. We only crawl pages that are publicly accessible. Any page behind a login or any other authentication won't be crawled. Our goal with this crawler is to build business datasets and machine learning models to better understand the web. This crawler follows robots.txt and meta instructions. We make sure our crawler doesn't add any noticeable load on websites we visit (usually, this means we'll never crawl more than one page at a time and one page every 2 seconds). If for whatever reason, you wish to ban this crawler in particular, you should target the User Agent VelenPublicWebCrawler in your robots.txt. We should pick up on this change within a few hours. If you notice bugs or want to send us any feedback regarding the crawler, contact us at: engineering@velen.io. "
Colin Cogle :verified:
Public

@collins_corner Thanks for the tip! I don’t mind bots visiting me, but these people seem like jerks.

lu 🍉
Quiet public

@collins_corner nothing stops them from using web browser user agents also

ExploreLive feeds
About