What Is Robots.txt? Crawl Rules Explained

What robots.txt is and where it lives

Robots.txt is a plain text file at the root of a host, at example.com/robots.txt, that lists which paths crawlers may request. Search engines fetch it before crawling any other page on the site, and it manages crawling rather than indexing or removal.

It exists once per host. A subdomain such as blog.example.com needs its own copy, and the file at example.com does not apply to shop.example.com. It must sit at exactly that path as plain text; a robots.txt inside a subfolder is ignored.

A missing file means the site is treated as open. A server error is different: if the file itself fails to load, major crawlers slow down or pause crawling until it returns, so an outage on that one URL can affect the whole site.

A working robots.txt example

The file below is a realistic starting point for a WordPress site. Each line is a directive, and the lines under one User-agent apply until the next User-agent appears.

• User-agent: *
• Disallow: /wp-admin/
• Allow: /wp-admin/admin-ajax.php
• Disallow: /search
• Disallow: /*?replytocom=
• Sitemap: https://example.com/sitemap.xml

Line by line: every crawler is addressed, the admin area is closed, admin-ajax.php is opened again because the public site calls it, internal search and comment-reply URLs are closed, and the sitemap is named once at the end.

How User-agent, Allow and Disallow rules apply

User-agent names the crawler a group applies to, with * meaning all of them. Disallow lists paths that must not be fetched, and Allow carves an exception inside a blocked path.

When two rules match, the longest and most specific one wins, which is why the admin-ajax Allow line above works. An empty Disallow allows everything; Disallow: / blocks the entire site. Wildcards and the $ end anchor are understood by major crawlers, not by every bot.

A Sitemap line is not a rule. It is an address where crawlers can find your XML sitemap, it sits outside the User-agent group, and the file it points to is covered in what an XML sitemap does.

Where to edit robots.txt on WordPress or cPanel

In cPanel, open File Manager, go to public_html and edit robots.txt in place. In WordPress the file is usually virtual: the CMS prints it on request, so change it in the SEO plugin's robots.txt editor rather than uploading a second copy that will be ignored.

If a physical robots.txt already exists in public_html, check which version the browser actually returns before editing. Two files in two places is a common reason a change appears to do nothing.

Robots.txt versus meta noindex

Robots.txt controls crawling; noindex controls indexing. To remove a page from results, let crawlers fetch it and return a noindex meta tag or X-Robots-Tag header, because a URL blocked in robots.txt can never be recrawled to see that tag.

Removing a block is a two-step fix: open the path in robots.txt, wait for the recrawl, then confirm the noindex was read. If the URL already shows in results, use the removals tool while the tag takes effect.

The robots.txt mistakes that cost traffic

The expensive mistake is blocking CSS and JavaScript. Google renders the page to understand it, and a blocked stylesheet or script leaves the crawler guessing about layout and content.

• Shipping Disallow: / from a staging copy to production, which stops the whole site being crawled.
• Blocking a page to hide it, when noindex is the tool that removes it from results.
• Treating robots.txt as private: anyone can read it and learn the paths you wanted hidden.
• Forgetting that images, PDFs and video can stay indexed even when the page is blocked.
• Assuming every bot obeys, when scrapers and some AI crawlers ignore the file entirely.

Our SEO services team checks these five against Search Console before a site migration goes live.

What WordPress serves by default

WordPress publishes a virtual robots.txt that allows everything and includes no Sitemap line. Core sitemaps sit at /wp-sitemap.xml, and an SEO plugin usually replaces the virtual file with its own rules and sitemap reference.

After a plugin change, load yourdomain.com/robots.txt and read the actual output. The settings screen is not the file; only the served text is what crawlers see.

How to test a robots.txt change

Open yourdomain.com/robots.txt in a browser and confirm it returns your new lines with a 200 status. Then use the robots.txt report in Google Search Console, enter a URL and read which rule matched.

Test one page you meant to block and one you meant to keep. If the second is blocked, the rule is too broad — fix it before waiting days for a recrawl. The same care applies to canonical URLs, which also fail quietly when misconfigured.