T-Tools
← Back to Blog
2026-03-25|Webmaster

Robots.txt Complete Guide: Control Search Engine Crawling

What is robots.txt?

Think of robots.txt as a "please" sign for search engine crawlers. It sits at the root of your website and tells bots which parts of your site they should (or shouldn't) look at. It's the first thing Google's crawler reads when it visits your site.

I'll be honest — I used to ignore robots.txt entirely. Then I accidentally blocked Google from our entire staging environment... and somehow it leaked into production. That was a fun conversation with my manager.

Basic Syntax

User-agent: *
Disallow: /private/
Allow: /public/
Sitemap: https://example.com/sitemap.xml

Directives Explained

User-agent

Specifies which crawler the rules apply to:

User-agent: Googlebot        # Google's crawler
User-agent: Bingbot          # Bing's crawler
User-agent: *                # All crawlers

Disallow

Tells crawlers which URLs not to access:

Disallow: /admin/            # Block entire directory
Disallow: /page?sort=        # Block specific query patterns
Disallow:                    # Empty = allow everything

Allow

Specifically permits access to URLs that might be blocked by a Disallow:

Disallow: /blog/
Allow: /blog/public/

Sitemap

Points to your XML sitemap:

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-news.xml

Common Robots.txt Examples

Block All Crawlers

User-agent: *
Disallow: /

Allow All Crawlers

User-agent: *
Disallow:

Block Specific Bots

User-agent: AhrefsBot
Disallow: /

User-agent: MJ12bot
Disallow: /

Protect Admin Areas

User-agent: *
Disallow: /admin/
Disallow: /wp-admin/
Disallow: /wp-login.php

Important Notes

  1. robots.txt is a suggestion, not a requirement - Malicious bots may ignore it
  2. Block at the server level for truly sensitive content
  3. Don't block CSS/JS - Search engines need them to understand your pages
  4. Use noindex meta tags for pages you don't want in search results
  5. Test your robots.txt with Google Search Console

Testing Tools

  • Google Search Console: Robots.txt Tester
  • Screaming Frog: Crawls your site according to robots.txt
  • Online Validators: Check syntax and behavior

My Robots.txt Mistake

Early in my career, I made a critical mistake with robots.txt. I accidentally added Disallow: / while trying to block a specific admin path. Within hours, my entire site was deindexed from Google. The recovery took 2 weeks of monitoring and resubmitting the sitemap.

The lesson: always test your robots.txt changes on a staging environment first. I now use Google Search Console's robots.txt tester before deploying any changes. Also, remember that robots.txt is advisory — malicious bots will ignore it, so never rely on it for security.

For T-Tools, I keep the robots.txt simple: allow everything useful, block internal paths, and let the sitemap guide crawlers to our content.