Honeypot
A trap planted for automation, a link or field invisible to humans, that flags anything which interacts with it as a bot.
A honeypot is bait planted for automation. The site puts something on the page that a person cannot perceive and therefore will never touch, then treats anything that touches it as proof of a program. There is no ambiguity to argue with, which is what makes it such a clean signal: a human being simply does not fill in an invisible field.
The forms are mundane once you know them. A link styled off-screen or given zero dimensions, so it exists in the markup and not on the screen. A form input hidden with CSS, sometimes named to look tempting to a form-filling script. A URL that appears nowhere in the visible navigation. And the classic, a path listed as disallowed in robots.txt, which a well-behaved crawler will never request and which therefore identifies anything that does.
That last one is worth pausing on, because it inverts what people assume robots.txt is for. Treated as a list of interesting places rather than a list of places to leave alone, it becomes a map of exactly which requests will mark you. Reading it to find what a site is hiding is a reliable way to walk into the trap deliberately.
The elegance of the defence is that it turns thoroughness into the tell. The more mechanically complete your automation is, the more certain it is to trip something a person would never encounter, so the very quality that makes a crawler good at collecting data makes it easy to identify. Once tripped, the consequence usually lands on the address, which is why a single careless run can spoil an IP for everything that follows.
Avoiding them is a question of behaving selectively instead of exhaustively. Follow what is genuinely visible and navigable, respect robots.txt as an instruction rather than reading it as an index, do not submit fields a rendered page does not show, and be suspicious of any element that is present in the DOM but invisible on screen. A web crawler that mimics what a reader would plausibly do is not just politer, it is harder to catch.
Frequently asked questions
How do I detect a honeypot before tripping it?
Render the page rather than only parsing the markup, then ignore anything a person could not see or reach: elements with display none, visibility hidden, zero width or height, opacity zero, or positioned far off-screen. If you are parsing raw HTML without rendering, you are working from a document that contains traps the visual page does not, which is exactly the gap the technique exploits.
Does robots.txt tell me where the honeypots are?
It often lists them, and using it that way is how people get caught. A disallowed path that no visible link points at is a strong candidate for bait, because a compliant crawler will never request it and so only automation ignoring the file will appear there. Treat the file as an instruction about where not to go, not as a shortcut to interesting URLs.
What happens after I trip one?
That depends on the site, and it is frequently not an immediate block. Some mark the address and act later, some start serving degraded or fabricated content so the data you collect is quietly wrong, and some simply raise your score with the anti-bot system so every later request is treated with more suspicion. Nothing obvious happening is not evidence that nothing happened.
Will rotating my IP undo a honeypot flag?
It moves the flag off that specific address, which helps if the mark was recorded against the IP alone. It does not help if the site also fingerprinted the client, because the same fingerprint arriving on a new address is a stronger signal than staying put would have been. And it does nothing about the behaviour that tripped the trap, so the next address will trip it too.
Back to the full glossary.