https://dotheneedful.online/forums/ Sun, 05 Apr 2026 15:01:03 +0000 MyBB https://dotheneedful.online/forums/showthread.php?tid=37 Thu, 20 Mar 2025 14:58:36 +0000 JollyRgrs]]> https://dotheneedful.online/forums/showthread.php?tid=37 https://thelibre.news/foss-infrastructur...companies/

Pretty good article and seems like an issue that will only be growing in problems in the future

tl;dr the scrapers are hammering websites (especially code repository ones) using different IPs and masking their UAs for each page request causing real performance issues (aside from the fact the data is being scraped against the robots.txt request)]]> https://thelibre.news/foss-infrastructur...companies/

Pretty good article and seems like an issue that will only be growing in problems in the future

tl;dr the scrapers are hammering websites (especially code repository ones) using different IPs and masking their UAs for each page request causing real performance issues (aside from the fact the data is being scraped against the robots.txt request)]]>