How to Add a Custom Robots.txt in a Blogger Blog in 2023

Learn how to add a custom robots.txt file to your Blogger blog in 2023. A robots.txt file tells search engines which pages on your website they can and cannot crawl, and it can be a valuable tool for SEO.

 A robots.txt file is a text file that tells search engines which pages on your website they can and cannot crawl. This can be useful for a variety of reasons, such as preventing search engines from crawling pages that are not meant to be public, or for telling search engines how often to crawl your website.

To add a custom robots.txt file to your Blogger blog, follow these steps:

1. Go to your Blogger dashboard.


2. Click on the Settings tab.


3. Scroll down to the Crawlers and indexing section.


4. Enable the Custom robots.txt option.


5. Click on the Custom robots.txt link.


6. Enter your custom robots.txt code in the text box.


7. Click the Save button.

Here is an example of a simple custom robots.txt file:

User-agent: *
Disallow: /search
Disallow: /category
Disallow: /tag
Allow: /
Sitemap: https://[YOUR_DOMAIN]/sitemap.xml

This file will disallow search engines from crawling the search, category, and tag pages on your blog. It will also allow search engines to crawl the root of your blog and your sitemap.

You can customize your robots.txt file to meet your specific needs. For example, you may want to disallow search engines from crawling certain pages on your blog, such as login pages or admin pages. You can also use your robots.txt file to tell search engines how often to crawl your blog.

Here are some tips for creating a custom robots.txt file:

• Use the User-agent directive to specify which search engines the robots.txt file applies to. If you want the robots.txt file to apply to all search engines, use the * wildcard.


• Use the Disallow directive to block search engines from crawling specific pages on your website. The Disallow directive must be followed by a forward slash / and the path to the page that you want to block.


• Use the Allow directive to allow search engines to crawl specific pages on your website. The Allow directive must be followed by a forward slash / and the path to the page that you want to allow.


• Use the Sitemap directive to tell search engines where your sitemap is located. A sitemap is a file that lists all of the pages on your website. This can help search engines to crawl your website more efficiently.

Once you have added your custom robots.txt file, it is important to test it to make sure that it is working as expected. You can use the Google Search Console Robots.txt Tester tool to test your robots.txt file.

Please note that it may take some time for search engines to update their indexes after you have added a new robots.txt file.