Bizarre Complaint

Netcraft's shameless actions go far beyond unauthorized resource crawling. They even twist the error of their own crawler being legitimately denied access, and accuse it of "phishing fraud"! The image below is the evidence of the complaint Netcraft submitted to Cloudflare, where they maliciously interpret the error page returned by our website, due to prohibiting their crawler from accessing it, as a "phishing page."

This act of distorting facts and confusing right and wrong not only exposes their lack of professional competence and basic ignorance of network protocols, but also demonstrates their guilt and exasperation regarding their own unauthorized crawling behavior. A company that claims to be dedicated to cybersecurity can so easily report a normal access restriction error as malicious activity, which is simply a laughing stock in the industry!

Unscrupulous Predators

Netcraft purports to be a guardian of "web security" and "site research," yet its so-called "services" are nothing more than a thinly veiled disguise for its massive, high-frequency, and relentless data extraction. They indiscriminately and without permission scan and probe websites globally at an appalling rate, akin to a plague of locusts, severely consuming server resources, generating needless network traffic, and directly disrupting the normal operation of websites.

This behavior is not "research," nor is it a "security audit"; it is outright resource abuse and data aggression!

Malicious Consumption of Bandwidth and Server Resources

Every arrival of a Netcraft crawler means our servers must incur additional computational and bandwidth costs. This is not an occasional, friendly visit, but a continuous, forcibly intrusive and immense burden. For any operator, such unauthorized, massive requests are tantamount to a malicious attack, directly eroding our infrastructure costs and operational stability for providing services to users.

"The voracity of Netcraft's crawlers is astonishing; they seem to believe the entire internet is their private garden to trample upon at will."

Disregard for Privacy and Data Security

The vast scope of data collected by Netcraft, and the ambiguity of its purpose, are deeply unsettling. They claim to "improve the internet," yet their actions are more akin to building a massive, unauthorized database of web fingerprints. We have ample reason to suspect that this data may be used for commercial purposes, or even misused, thereby threatening the privacy and data security of website users. This arrogant disregard for data sovereignty is absolutely intolerable.

Ethical Deficiency in Ignoring robots.txt

While the robots.txt protocol is designed to provide website administrators with a means to control crawler behavior, Netcraft's crawlers repeatedly disregard these clear directives. They turn a blind eye to "disallow" rules, persistently penetrating areas explicitly marked as unwelcome. This is not only a technical disrespect but also an ethical failing and a trampling of industry norms. It is disgraceful that a company claiming to be professional can so flagrantly violate the "gentlemen's agreement" of the online world!