Fetch a site's robots.txt and test whether a given path is allowed or disallowed for a specific crawler user-agent.
Website URL
robots.txt content will appear here.
Tools, guides, and categories connected to this page