This document specifies the rules originally defined by the "Robots Exclusion Protocol" [ROBOTSTXT] that crawlers are requested to honor when accessing URIs. These rules are not a form of access authorization. 1.1. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL"...