What is Robots.txt file? | Importance of Robots.txt in SEO

Are you in the Field of Blogging and SEO? then I am Sure you must have listened about Robots.txt, so in this Topic, we will discuss on Robots.txt File.

In this Blog-Post you will learn What is a Robots.txt File?, Importance of Robots.txt file in SEO and Blogging, How to Create a Robots.txt File? and How to Submit Robots.txt file on Blogger?

What is a Robots.txt File in SEO?

In simple meaning, we can understand Robots.txt as a text document  (not HTML) submitted to Google Webmaster to help Search Engine Robot to crawl and index a website and web pages.

Or we can describe Robots.txt file as some instruction given by a webmaster to Search Engine Bots to instruct a Robot How to Crawl and Index web pages of a website.

With Robots.txt file you can allow and disallow Search Engine like you can Disallow Bing and Allow Google. You can tell which page to be crawled and which not be crawled by Search Engine Crawler.

A simple Robots.txt File added on Blogger:

User-agent: Mediapartners-Google
Disallow: 

User-agent: *

Sitemap: https://bipinseo.com/sitemap.xml
Disallow: /search
Allow: /

MediaPartners-Google manages all kind of media file and Ads shown on your website.

User-agent : * is a common default robot for all blog and website, this leads a robot to manage all the URL of your website.  If we use only a specific URL instead of *(ALL) then the robot will manage only that specified URL or webpage.

>>Also Read: How to create and submit a sitemap?

How to add Robots.txt File on Blogspot Blogger?

First of all, Sign Up to your Blogger Account and follow the following steps to add Custom Robots.txt File on Blogger:

• Click on Setting Option of your Blogger Dashboard.
• Click on Search Preferences inside "Setting".
• Go to Crawlers and Indexing inside Search Preferences and Click on Custom robots.txt Option.
• Click on Edit Option and add Default Custom robots.txt file of your Blog here.

Importance and Advantages of a robots.txt file in SEO:

a. You can Disallow 404 error pages or invalid URL of your website.
b. Guide Search Engine Robot to crawl which page and which not to be crawled.
c. Help to solve different errors that may occur in a website.
d. You can hide duplicate content from Search Engine and readers.

If you can not handle and manage robots.txt file of your website then your Ranking can decrease very fast. So take a better advice from a webmaster to manage your robots.txt file and URL of your website.