Cloudflare
A CDN and security layer that fronts a large share of the web and includes some of the most common anti-bot challenges scrapers meet.
Cloudflare sits in front of a very large fraction of websites as a CDN and security proxy, which means its bot-defence is one of the most common obstacles automation runs into. When a request looks automated, Cloudflare can interpose a challenge, a JavaScript check, a managed challenge, sometimes a CAPTCHA, before the real site is reached.
Because Cloudflare is a reverse proxy for the site, what you are dealing with is its defence, not the origin's. That defence weighs the same signals every serious anti-bot system does: IP reputation, TLS fingerprint, browser behaviour. A datacenter address draws a challenge far more readily than a clean residential one.
The practical upshot is unchanged: a trusted address, a genuine browser fingerprint and unhurried behaviour get through where any one of them alone does not, and for the hardest configurations a purpose-built scraper endpoint is often the pragmatic route.
The structural point worth understanding is that the protection is configurable by each site, not a single fixed obstacle. Site operators choose their own security level, their own rules and their own thresholds, which is why one site fronted by it is entirely open while another challenges you immediately. Concluding that you cannot scrape anything behind it, or that you can, both generalise from a setting somebody chose rather than from the platform itself.
It also means the response you get is not binary. Depending on configuration a request can be served normally, given a silent JavaScript check that resolves without you noticing, shown an interactive challenge, rate limited, or refused outright. Reading which of those you actually received tells you far more about what to change than treating every non-200 as the same wall.
One genuinely useful consequence for anyone tracing an address: because it acts as a reverse proxy, the addresses you see belong to it rather than to the site's origin servers. A lookup on a site's IP frequently returns the CDN rather than the host, which is expected behaviour rather than evidence of anything.
Our honest position on the hardest configurations, since it is the question we are actually asked: a clean residential address raises your floor and does not by itself decide the outcome, because the checks above the network layer continue regardless. When a target is configured aggressively enough, the realistic options are a purpose-built scraping endpoint, an official API where one exists, or accepting that the site does not want automated access. Selling a more expensive proxy as the answer to that would not be true.
Frequently asked questions
Why does Cloudflare block my scraper but not my browser?
Because your browser and your script differ in everything except the address. The TLS fingerprint of an HTTP library is not a browser's, the header order differs, client hints are missing, and there is no human interaction at all. Those are read before your request reaches the site, which is why the same IP succeeds in one and fails in the other.
Do residential proxies get past Cloudflare?
They remove one of the strongest reasons to challenge you, and on lightly configured sites that is often enough on its own. On aggressively configured ones it is not, because the fingerprint and behaviour checks run regardless of how trusted your address is. It raises your floor rather than deciding the outcome.
Why do different sites behind Cloudflare behave so differently?
Because each operator configures their own security level, rules and thresholds. It is a platform with settings rather than a single fixed barrier, so one site can be completely open and another immediately hostile while both are using the same product. Test the specific site rather than generalising from the platform.
Is it Cloudflare or the website blocking me?
Usually the edge, acting on rules the site's operator chose. Because it is a reverse proxy, the decision is typically made before the site's own application sees anything, which is why a block can be instant and why the origin server may have no record of your request at all. It also explains why a lookup on the site's address returns the CDN rather than the host.
Back to the full glossary.