Robots.txt - Big Importance in a Small File

What is a robots.txt file?

The robots.txt file is a simple text file that is found in the root directory of your website, or at least it should be. It is not a file that a typical user would see; it's not something you're going to link to from your main navigation. It's a file that instructs search engines like Google. Which areas of your website to include in their search rankings, and which areas to ignore.

Why should you care?

You want your website to be found right? Silly question I know. If your robots.txt file is not configured correctly, you could be hurting your search engine rankings. The very first thing that search engines look at when they reach your site is the robots.txt file to find out what it can, and cannot, catalogue. An improperly configured robots.txt file can block essential parts of your pages from being indexed, or worse yet, your whole site.

How to set up your robots.txt file

One of the first things to remember, robots.txt is case sensitive. Don't name it robots.TXT or Robots.txt; it has to be all lowercase.

Next, you'll want to make sure it's set up to allow search engines to crawl your site. Pasting the example below into your robots.txt file would tell search engines it's okay to look at everything.

User-agent: *
Disallow:

If there are specific areas of your site that you don't want search engines to crawl, you can add those under the disallow portion. Let's say your website runs on Wordpress and you want to block search engines from scanning your admin and plugins folders. In this case, you would paste this into your file:

User-agent: *
Disallow: /wp-content/plugins/
Disallow: /wp-admin/

And if you have a sitemap that you want to point search engines at, to make their job easier, you can add that in here as well:

User-agent: *
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Sitemap: https://www.mesltd.ca/sitemap_index.xml

It's undeniable that the demand for personalization is increasing. Our prospects and customers want solutions and information tailored to them, "one size fits all" is no longer acceptable.It's for...

Google Analytics (GA) is an excellent tool for analyzing your website traffic, performance, and other key metrics. However, it's not the most intuitive, or user-friendly software when you first...

So let's start off by defining what we mean by gating your content. Gated content refers to anything on your site that a visitor needs to complete a form to access. Basically, it is something of...