Your website contains a small text file that works as a set of instructions for search engine bots. This file is called robots.txt and it sits in the root directory of your site. The robots.txt file tells search engine bots which pages of your website they may crawl and index, and which ones they should avoid.
Every day, search engines send bots (crawlers) to browse websites and collect information. Without a robots.txt file, these bots can reach any page on your site. That means your test pages, duplicate content or internal search results could end up in Google’s results.
Understanding how the robots.txt file works matters for every website owner. In this article you will learn how to configure the file correctly, which commands to use and which mistakes to avoid. That way you can better control how search engines see your website.
Key takeaways
- The robots.txt file is a text file that controls search engine bots’ access to the pages of your website
- The file uses a simple syntax with commands such as „User-agent”, „Disallow” and „Allow” to tell bots what they may do
- Regularly testing and updating your robots.txt file helps you avoid indexing errors and improve SEO results
Why robots.txt matters and its main functions
The robots.txt file plays an essential role in managing how search engine bots browse your website. This simple text file directly influences which pages get indexed and shown in the SERPs.
Robots.txt and search engine bots
The robots.txt file is the first point of contact between your website and search bots. When a bot from Google or another search engine visits your site, it first looks for this file at www.yourwebsite.com/robots.txt.
The file sets clear rules about which pages of the site bots may browse. You can block access to admin pages, staging versions or duplicate content. You can also manage how quickly bots crawl so that they do not overload your server.
The main functions of robots.txt:
- Disallows or allows access to specific pages
- Points to the location of the XML sitemap
- Saves server resources by limiting bot activity
- Protects against unwanted content being indexed
It is worth knowing that well-behaved bots follow robots.txt instructions, while bad bots may ignore them.
Impact on SEO and site visibility
A correctly configured robots.txt file has a direct effect on your SEO results. By blocking unnecessary pages, you let search bots focus on the important content that should be visible in the SERPs.
The most common SEO mistakes happen when you accidentally block important pages. In that case Google cannot index them and they will not appear in search results. The Google Search Console tool helps you check whether your robots.txt file is working correctly.
A well-configured file improves your crawl budget: bots discover new pages faster and index them more efficiently. This is especially important for large websites with thousands of pages. By pointing to the sitemap location in robots.txt, you speed up the discovery and indexing of new content.
How the robots.txt file works
The robots.txt file is the first point of contact between your website and search engine bots. When crawlers visit your site, they automatically check whether this file exists and what instructions it contains.
How crawler bots behave
Search engine bots always start browsing a site by checking the robots.txt file. When Googlebot, Bingbot or other crawlers visit your site, they immediately request the www.yourwebsite.com/robots.txt address. If they find the file, the bot reads all of the rules and then begins browsing the site according to those instructions.
Bots look for their own specific User-agent instructions in the file. If they find a directive aimed at them, they follow only the rules in that section. When there is no specific directive, bots follow the User-agent: * section, which applies to all bots.
It is important to understand that the robots.txt file controls only crawling, not indexing. Even if you forbid a bot from accessing a certain page, that page can still appear in search results if other websites link to it.
The difference between polite and impolite bots
Not all web bots treat the robots.txt file the same way. The bots of the major search engines – Googlebot, Bingbot, Yandexbot – are considered polite bots because they respect and follow robots.txt rules. These crawlers read the file’s instructions and obey them without exception.
Impolite bots are a completely different matter. They are usually malware, spammer tools, content scrapers or other bad-faith bots. They simply ignore robots.txt instructions and try to reach every part of the site.
For this reason, the robots.txt file is not a security measure. You should not rely on robots.txt alone to protect private information or sensitive data. For such purposes you need password protection, IP blocking or other security mechanisms.
Where the robots.txt file lives
The robots.txt file must be placed in the root directory – the main folder of the website, reachable via the main domain. The file must be available at: https://www.yourwebsite.com/robots.txt. The name must be written in lowercase – Robots.txt or ROBOTS.TXT will not be recognised.
Each domain and subdomain requires its own robots.txt file. If you have blog.yourwebsite.com, it needs a separate file at blog.yourwebsite.com/robots.txt. The HTTP and HTTPS protocols are also treated as separate origins.
What happens when the file is unavailable
When bots cannot find a robots.txt file, or it returns an error, they interpret this as permission to crawl the whole site. Google and other search engines treat a missing robots.txt file like an empty Disallow: directive – meaning all pages are allowed to be crawled.
If the file returns a 404 error, bots behave the same way – they crawl the whole site without restrictions. However, if the file returns 5xx server errors, bots treat it as a temporary problem and may try again later.
You can check how your robots.txt file works using the Google Search Console robots.txt tester. This tool lets you not only see whether the file is reachable, but also test specific URLs to see whether they are blocked by the rules you have set.
Syntax: the main robots.txt commands
The robots.txt file follows a simple syntax made up of a few core commands. Each line in the file should consist of a field name, a colon and an instruction, e.g. user-agent: Googlebot.
User-agent and user agents
The User-agent command specifies which search engine bot the rules apply to. You can name specific bots or use the universal symbol user-agent: *, which applies to all bots.
The most common user agents:
- Googlebot – Google’s main search crawler
- Bingbot – Microsoft Bing’s search crawler
- Baiduspider – the crawler of Baidu (China’s search engine)
- Slurp – Yahoo’s search crawler
- DuckDuckBot – DuckDuckGo’s search crawler
- Googlebot-Image – Google’s image crawler
- Googlebot-Video – Google’s video crawler
You can create separate groups of rules for each bot. For example, if you want to let Googlebot reach a certain page but block it for other bots, you can create two separate groups with different commands.
Disallow, Allow and how to apply them
The Disallow command specifies URLs or directories that bots should not crawl. The Allow command lets bots access specific pages even if they are inside a blocked directory.
Disallow: / blocks the entire site for all bots. This is useful only when testing a site before launch.
Practical examples:
User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /private/public-page.html
This example blocks the /admin/ and /private/ directories but allows access to the specific page /private/public-page.html. If you do not specify any disallow rule, bots are free to crawl the whole site.
Adding the Sitemap command
The Sitemap command tells search engines where to find your site map. This helps bots find all of the site’s pages faster and more efficiently.
Example:
Sitemap: https://yourwebsite.com/sitemap.xml
Sitemap: https://yourwebsite.com/sitemap-images.xml
You can list several sitemap.xml files if your site has a complex structure. The sitemap should be a full URL including the protocol (https://).
This command does not need to be assigned to a specific user-agent and can be placed anywhere in the file.
Using wildcards and Crawl-delay
Wildcards let you filter URLs more flexibly. The * symbol matches any sequence of characters, and $ marks the end of a URL.
Examples:
Disallow: /*.pdf$
Disallow: /*?sessionid=
Disallow: /private-*.html
The first line blocks all PDF files. The second blocks URLs containing the sessionid parameter. The third blocks all HTML files whose name starts with private-.
The Crawl-delay command used to be used to limit how fast bots crawl, but Google does not support this command. Some other search engines (e.g. Bing) still understand it, but it is better to use the Google Search Console settings to manage crawl rate.
Practical robots.txt examples and use cases
The robots.txt file works through simple directives that tell bots what to do. The most common scenarios include protecting admin areas, blocking duplicate content and managing technical directories.
Protecting private and admin areas
Admin areas are the first pages worth protecting from search bots. Login pages, dashboards and user profiles should not be visible in search results.
Here is what a simple protection example looks like:
User-agent: *
Disallow: /admin/
Disallow: /login/
Disallow: /user/
User-agent: * means the rules apply to all bots. The Disallow directive specifies which directories not to crawl. For WordPress sites it is essential to protect the wp-admin directory, which contains the entire management system.
If you have public pages inside an admin directory, you can use the Allow directive:
User-agent: *
Disallow: /admin/
Allow: /admin/public-page/
Blocking duplicate or temporary content
Internal search results and parameterised URLs create a lot of duplicate content. Such pages have no value for visitors coming from search and can confuse Google.
Block parameters and temporary pages:
User-agent: *
Disallow: /search?
Disallow: /*?sort=
Disallow: /temp/
Disallow: /draft/
The asterisk lets you block any URL containing a certain parameter. This is useful for online shops, where filters and sorting options create many variations.
Blocking duplicate content helps Google focus on the most important pages. Remember, though, that blocking via robots.txt does not mean the page will definitely not appear in search.
Managing specialised resources (e.g. wp-admin)
WordPress and other platforms have technical directories that bots should skip. Wp-admin is the main WordPress management directory and must be blocked.
A complete WordPress robots.txt file looks like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Sitemap: https://yourwebsite.com/sitemap.xml
The exception for the admin-ajax.php file is important because many features rely on it to work. The Sitemap directive tells bots where to find the site structure with its important pages.
Never block CSS and JavaScript files. Google needs these resources to render your site correctly. Blocking js and css directories used to be common practice, but today it harms indexing.
Robots.txt mistakes, testing and updates
An incorrectly configured robots.txt file can block important pages or allow unnecessary content to be indexed. Mistakes are often noticed only once traffic is already falling, so regular testing and updating is essential.
The most common syntax and implementation mistakes
One of the most common mistakes is incorrect robots.txt syntax. If you skip spaces between directives or misspell commands, bots may misinterpret the rules. For example, Dissallow: / instead of Disallow: / will not work.
On WordPress sites, a frequent mistake is blocking the whole /wp-admin/ directory without adding Allow: /wp-admin/admin-ajax.php. This line is necessary because many themes use that file to load dynamic content. Without it, Google sees a broken site.
Another problem is blocking /wp-content/. This denies access to all images and files. Your product photos will not reach Google Images, which cuts traffic by 12–18 percent for online shops.
A staging-environment robots.txt file left in production blocks the entire site. The line Disallow: / prevents any page from being indexed, and recovery can take 2–6 weeks.
How mistakes affect indexing
Robots.txt controls crawling, not indexing. If a page is already in Google’s index and you block it via robots.txt, Googlebot can no longer visit it. The page stays in search results, but instead of a description it shows a message about being blocked by robots.txt.
To remove a page from the index you need to use a noindex meta tag or an X-Robots-Tag header. Robots.txt does not have this function. First add <meta name="robots" content="noindex"> to the page’s code. Then request reindexing via Google Search Console.
Only once the page has disappeared from the index can you add a Disallow rule. The reverse order does not work and leaves the page in the SERP without a description.
Tools for checking the robots.txt file
Google Search Console is the main tool for testing robots.txt. In the „URL Inspection” section, enter a page address and check whether robots.txt is blocking access. In „Indexing > Pages”, look for the status „Indexed, though blocked by robots.txt” – it signals a problem.
Other useful tools:
- Screaming Frog with the „Respect robots.txt” option shows how many URLs are blocked
- Server log analysis shows where Googlebot spends most of its time
- Online robots.txt testers check for syntax errors
Open yourwebsite.com/robots.txt in a browser. The file must be visible to all visitors. If you see errors or a lone Disallow: /, fix it immediately.
Latest changes and recommendations
The robots.txt standard was officially approved in September 2022 as RFC 9309. Until then it had been an informal protocol dating back to 1994. Since July 2019 Google no longer supports the crawl-delay directive, although Bing and Yandex still use it.
Since 2023, new user agents for artificial intelligence models have appeared. GPTBot controls OpenAI’s access, Google-Extended manages Gemini training, and ClaudeBot covers Anthropic’s models. These bots are separate from search crawlers.
If you want to block AI training but still allow indexing in Google Search, use:
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Add a robots.txt check to your website migration checklist. Monitoring tools such as ContentKing automatically report changed robots.txt rules. Check the file after every site update.
