2025-08-05 Hacker News Top Articles and Its Summaries
1. Perplexity is using stealth, undeclared crawlers to evade no-crawl directives Total comment counts : 89 Summary Perplexity, an AI answer engine, has been observed engaging in stealth crawling by disguising its user agent and altering its IP addresses to bypass website protections and ignore robots.txt directives. Despite attempts to block this behavior, Perplexity continued to access restricted content from newly created domains, confirming concerns from affected website owners. Their actions, which violate web crawling norms, prompted the decision to de-list Perplexity as a verified bot and implement stricter blocking rules....