robots.txt
A file at a site's root that tells crawlers which paths it prefers they avoid, a convention, not an enforced barrier.
robots.txt is a plain text file a site publishes at its root to signal how it would like automated visitors to behave, which paths to skip, sometimes how slowly to crawl. It is a long-standing convention that well-behaved crawlers follow, and search engines honour it strictly.
It is important to be clear about what it is not: an access control. robots.txt asks rather than enforces; nothing technical stops a request to a disallowed path. But ignoring it is both poor practice and a practical risk, because a site that sees traffic hitting paths it asked crawlers to avoid has an easy signal that the visitor is a bot behaving badly.
The sensible reading is that robots.txt marks the boundary of welcome automation. Staying inside it keeps a crawler in the category of tolerated traffic rather than flagged abuse.
Mechanically it is simple, which is part of why it has survived unchanged for decades. Groups begin with a User-agent line naming which crawler the rules apply to, followed by Disallow and Allow lines listing path prefixes. An asterisk in the User-agent line means everyone not addressed more specifically. Many sites also publish a Sitemap line, which is genuinely useful: it points at a structured list of the URLs the site actually wants found.
The trap it creates is the inverse of its purpose, and it catches people who think they are being clever. Read as a list of interesting places rather than places to avoid, a disallowed path looks like a hint that something valuable is hidden there. It is far more often the opposite: a path no legitimate crawler will ever request, which makes anything requesting it trivially identifiable. Treating the file as a map is how you walk into a honeypot deliberately.
It is also worth being clear that it is not security and was never intended as one. A disallowed path is still publicly reachable, so anyone can request it and nothing stops them. Publishing the location of something sensitive in robots.txt advertises it to everyone while asking politely that nobody look, which is the worst of both.
The practical position we take, and would suggest as a floor: read it, honour the disallow rules, honour any crawl delay it specifies, and use the sitemap it offers. Compliance costs almost nothing, keeps you in the category of traffic a site tolerates, and avoids the specific signal that most reliably marks an address as behaving badly.
Frequently asked questions
Is ignoring robots.txt illegal?
It is a convention rather than a law, and the legal position varies by jurisdiction and can interact with a site's terms of service, which makes it a question for a lawyer rather than a glossary. The practical answer is separate and clearer: ignoring it is one of the most reliable ways to have an address flagged, because disallowed paths are exactly what nothing legitimate requests.
Does robots.txt actually block anything?
No. It is a published request, not an access control. Every disallowed path remains publicly reachable and nothing technical prevents a request to one. That is also why listing something sensitive there is counterproductive: it advertises the location to everyone while relying on politeness to protect it.
Can robots.txt tell me where the good data is?
It can tell you where the traps are, which is the useful inversion. A disallowed path that nothing links to is a strong honeypot candidate, since only automation ignoring the file will ever appear there. The genuinely useful line is Sitemap, which points at a structured list of URLs the site actively wants discovered.
How do I read a robots.txt file?
It is grouped by crawler. A User-agent line names who the following rules apply to, with an asterisk meaning anyone not addressed specifically, and Disallow and Allow lines list path prefixes beneath it. Find the group that applies to you, which is usually the asterisk group, and follow those rules. A Sitemap line can appear anywhere and applies to everyone.
Back to the full glossary.