What is a robots.txt file?
The robots.txt file is a simple text file that is found in the root directory of your website, or at least it should be. It is not a file that a typical user would see; it's not something you're going to link to from your main navigation. It's a file that instructs search engines like Google. Which areas of your website to include in their search rankings, and which areas to ignore.
You want your website to be found right? Silly question I know. If your robots.txt file is not configured correctly, you could be hurting your search engine rankings. The very first thing that search engines look at when they reach your site is the robots.txt file to find out what it can, and cannot, catalogue. An improperly configured robots.txt file can block essential parts of your pages from being indexed, or worse yet, your whole site.
One of the first things to remember, robots.txt is case sensitive. Don't name it robots.TXT or Robots.txt; it has to be all lowercase.
Next, you'll want to make sure it's set up to allow search engines to crawl your site. Pasting the example below into your robots.txt file would tell search engines it's okay to look at everything.
User-agent: *
Disallow:
If there are specific areas of your site that you don't want search engines to crawl, you can add those under the disallow portion. Let's say your website runs on Wordpress and you want to block search engines from scanning your admin and plugins folders. In this case, you would paste this into your file:
User-agent: *
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
And if you have a sitemap that you want to point search engines at, to make their job easier, you can add that in here as well:
User-agent: *
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Sitemap: https://www.mesltd.ca/sitemap_index.xml