Glossary · Scraping & automation
robots.txt
A file at a site's root that tells crawlers which paths it prefers they avoid, a convention, not an enforced barrier.
robots.txt is a plain text file a site publishes at its root to signal how it would like automated visitors to behave, which paths to skip, sometimes how slowly to crawl. It is a long-standing convention that well-behaved crawlers follow, and search engines honour it strictly.
It is important to be clear about what it is not: an access control. robots.txt asks rather than enforces; nothing technical stops a request to a disallowed path. But ignoring it is both poor practice and a practical risk, because a site that sees traffic hitting paths it asked crawlers to avoid has an easy signal that the visitor is a bot behaving badly.
The sensible reading is that robots.txt marks the boundary of welcome automation. Staying inside it keeps a crawler in the category of tolerated traffic rather than flagged abuse.
Back to the full glossary.