Crawlers read robots.txt before anything else on your site, and one wrong line can hide pages you wanted indexed. This page breaks the file into its four directives, shows the disallow-versus-noindex distinction that traps most site owners, and ends with the deploy-and-verify loop in Search Console.
Build your robots.txt
Runs in your browserUpload the file to your site root so it answers at /robots.txt. Crawl rules are not de-indexing: to remove a page from results use noindex on the page itself, and Search Console verifies the outcome. Need the stack that runs it? See cPanel hosting.
What a robots.txt generator writes
Robots.txt directives are few and worth knowing precisely. User-agent: names the crawler the following rules apply to (* means all). Disallow: paths the crawler must not fetch — /admin/, /cart/, /wp-admin/. Allow: exceptions inside disallowed paths — Allow: /wp-admin/admin-ajax.php is the classic. Sitemap: the directive pointing crawlers at your XML sitemap — Sitemap: https://yourdomain.com/sitemap.xml — which belongs in every robots.txt because it hands crawlers the curated URL list alongside the rules.
| Directive | Meaning |
|---|---|
| User-agent: * | Rules for all crawlers |
| Disallow: /admin/ | Block that path |
| Allow: /wp-admin/admin-ajax.php | Exception inside a block |
| Sitemap: .../sitemap.xml | Declare the sitemap |
The file lives at the domain root — yourdomain.com/robots.txt — or it does not exist as far as crawlers care. Generate, deploy, then verify in Search Console; the XML sitemap guide covers the sitemap half and the knowledgebase carries the deployment steps.
How robots.txt disallow differs from noindex
Disallow versus noindex is the most consequential distinction in robots.txt work. Disallow stops crawling — the crawler never fetches the page — but it does not remove the page from search results; Google can still list the URL (without content) if links point at it. Noindex is the removal instruction — it must be crawled to be read, which is exactly what Disallow prevents. Block a page in robots.txt that carries a noindex, and you have guaranteed Google can never see the removal order.
The rule: pages you want out of search get meta noindex and stay crawlable; pages that merely waste crawl budget (carts, admin, internal search) get Disallow. The file you write carries the crawl rules; the noindex decision belongs in the page's meta settings, and Search Console verifies which state each URL is actually in.
How WebJi® helps with the robots.txt file
Put the file at the domain root and check it in Search Console. The definition sits on what robots.txt is. Email support@webji.services if a live site should change. WebJi® has hosted sites since 2005. Invoices are in INR and show 18% GST.
Frequently Asked Questions
How do I create a robots.txt file for my website?
Write User-agent, Disallow, Allow, and Sitemap lines, save the file as robots.txt, and put it at the domain root. Then check it in Search Console. The sitemap half is on the XML sitemap guide.
What should I disallow in robots.txt?
Disallow paths that waste crawl budget, such as /admin/, /cart/, and /wp-admin/. Allow an exception when a file inside that path must be fetched, such as /wp-admin/admin-ajax.php.
Where do I upload the robots.txt file?
The domain root, so the URL is yourdomain.com/robots.txt. A file in a subfolder is not the file crawlers request.
Is robots.txt disallow the same as noindex?
No. Disallow stops the fetch. It does not remove a URL that other pages link to. Noindex has to be crawled to be read, so disallowing that URL hides the removal instruction.
How do I block AI bots like GPTBot in robots.txt?
Add a User-agent line for that bot and a Disallow rule. This page does not publish a current list of bot names.
Can robots.txt remove pages from Google?
Disallow alone does not. A URL can still be listed without a snippet if other pages link to it. Use noindex on the page, and leave that URL crawlable.
Is WebJi® an Indian company?
Yes. WebJi® is Webji Hosting Pvt. Ltd., formerly B4UIndia, hosting since 2005. The brand has been WebJi® since 2018. The office is Shop #27, Ajmer Road, Tagore Nagar, Jaipur 302021.
Do you provide GST invoices?
Yes. Invoices are in INR and show 18% GST. Email support@webji.services is 24/7 in Hindi and English. Phone +91 141 4941811 is the Jaipur office line during business hours.