Google's mobile-friendly test returns "The requested URL is disallowed by robots.txt."

If you perform Google's mobile-friendly test and you get the message "The requested URL is disallowed by robots.txt." then you should check the robots.txt file on your site.

To see your robots.txt file type in your website address into your browser followed by /robots.txt. The robots.txt file will be displayed. If you see the following this is telling bots not to crawl your site and thus your site will not appear in search results in sites such as Google.

User-agent: *
Disallow: /

If you'd like Google to index your site you should review your site and determine what you'd like the robots to crawl. In my case I know all the content in my sites and generally I'm happy for robots to crawl all the content on my sites. If there's content I don't want robots to crawl then I'd put that content in a password protected folder. However if you use one of the content management systems there may be a reason the robots.txt file has been set up to stop robots from crawling your site. In this case you should contact your support person and ask for their advice.

If you're in business and want people to find you in search sites such as Google, you definitely don't want your robots file set up to stop them crawling the pages you'd like people to find.

Kelvin Eldridge
www.OnlineConnections.com.au
Call 0415 910 703 for IT support.