About Robots Exclusion Checker

The Story Behind the Extension

Sam created the Robots Exclusion Checker while working with a FTSE 100 ecommerce client who managed their faceted navigation through robots.txt exclusions, over 2000 of them! When visiting filtered URLs, it wasn't immediately clear if the robots.txt file was blocking the URL or not.

While traditional website crawlers could easily scan an entire site to determine which pages were crawlable, this approach wasn't convenient when actively browsing the site. Sam decided to create the extension to help his client and himself visually identify, in real-time, whether there was some kind of disallow directive on the page being viewed.

What began as a solution to a specific robots.txt challenge evolved as he incorporated additional crawlability and indexation signals into the extension. Today, it checks multiple technical factors across robots.txt files, meta robots tags, HTTP headers, and canonical tags to flag anything that might impact search visibility.

The beauty of the extension lies in its simplicity—a straightforward red-amber-green system gives users immediate insight, with detailed information available in the extension popup should they wish to investigate further. This approach has made technical SEO more accessible, even if one's job is not technical in nature, whether they're an ecommerce manager, category trader, or content writer. Anyone can now quickly understand page indexability without needing specialised technical knowledge.