We Tested 52 US Retailers' Bot Defenses. Here's Who Runs What

We spend our days getting legitimate traffic past retail bot defenses, so we decided to measure the front doors directly. On August 24, 2026, we sent 52 major US retail and marketplace storefronts the same plain request, once from a residential home connection and once from a datacenter server, and wrote down exactly what came back: the status code, the cookies, the headers, and any block or challenge page. From those fingerprints we identified which bot-defense vendor guards each site.

This is a snapshot of the door, not a bypass guide. But the door tells you a lot: which vendor a retailer trusts, whether it rejects datacenter traffic on sight, and whether "the site loaded" actually means anything. The uncomfortable parts are left in, including the two sites that made all of our advice look unnecessary.

Finding 1: Akamai runs close to half of US retail

Of the 52 storefronts, 23 ran Akamai Bot Manager as their primary defense. That is close to half the sample, and it is not close: Cloudflare, the runner-up, guarded 12. If you are building anything that reads US retail at scale, you are mostly building against Akamai.

The Akamai group spans every category: home improvement (Home Depot, Lowe's), grocery (Kroger), department stores (Kohl's, Dillard's), beauty (Sephora, Ulta), apparel (American Eagle, Lululemon, Zara, Uniqlo), sporting goods and outdoor (Dick's, JD Sports, Academy, Bass Pro), pharmacy (Walgreens), pet (PetSmart), luxury (Farfetch), and the warehouse club (Costco). The Akamai mechanics, the _abck sensor cookie and the JA3/JA4 TLS read, are the same everywhere, which is why one Akamai-aware pipeline transfers across all of them. We wrote the vendor playbook in how to scrape past Akamai.

Finding 2: nine sites answered home and datacenter differently

The comparison that looked most useful was the split between our two vantage points. On nine sites, our request from a home IP loaded the storefront and our request from a datacenter IP was blocked or challenged.

Correction, October 22, 2026. The two requests also came from different clients, and a controlled re-test showed that this matters. With the same client (curl 8.5 on Linux), a home and a datacenter address got the same answer at 14 of 14 sites, while changing only the client's TLS stack (curl 8.16 on Windows) changed the answers of SSENSE and Whatnot. So this split does not prove that the address decided. The re-test has the details.

Vendors do score the address. Cloudflare, for one, documents a detection for residential proxy traffic and scraping patterns watched per network, so a job that sends all its traffic from one hosting provider stands out. What this snapshot cannot tell is how much the address weighed against the client on each site, which is why we now test the two separately. The general comparison is in datacenter vs residential proxies. One site, Instacart, ran the split in reverse: it geo-gated our home IP as outside its service area and served our US datacenter server, a reminder that some doors are about geography, not bots. The other sites were less tidy: many blocked both a residential and a datacenter bare client, because they objected to the missing browser more than the address, which brings us to the next finding.

Finding 3: seventeen retailers stack two or more defenses

Seventeen of the 52 ran more than one defense layer at once. A single vendor is common but far from universal at the top end of retail, and the hardest sites are the most layered.

The layering is not decorative. Chewy and Whatnot run vendors that each check something different, so a solver for one does nothing about the rest. Pokemon Center adds a Queue-it waiting room and challenge vendors on top of Imperva for its trading-card restocks. Costco, LEGO and Victoria's Secret arm queues for the products people fight over. The lesson is that "which vendor" is often the wrong question; the right one is "how many, and in what order."

Finding 4: "the site loaded" usually means nothing

Seventeen of the 52 returned a normal homepage to a bare client. It would be easy to call those the soft targets. That reading is wrong, and it is the most expensive mistake in retail scraping.

Most of those seventeen still gate the product data behind something the homepage does not need. Akamai sites like Ulta and Walgreens serve a cached homepage freely but require a valid _abck sensor cookie, which only a real browser produces, before they return a price. The F5 Shape sites, Nordstrom and Macy's, are the sharpest example: they deliberately admit the first request and then score the session's behavior over time, so a scraper gets a clean start and a wall ten requests later. Against those, rotating IPs faster makes things worse, because a mid-session IP swap is itself a bot signal.

Finding 5: the honest outliers, TCGplayer, DSW and Kroger

Three results cut against selling proxies, and leaving them out would make this study a brochure instead of a measurement.

TCGplayer, the large trading-card marketplace, and DSW, the shoe retailer, each served a plain client from both IPs with no challenge, no sensor cookie and no fingerprint wall. For light reads you barely need proxies on either; they earn their place only at the volume where a large platform starts throttling by IP. And Kroger, despite an Akamai-guarded website that tar-pitted our home-IP request and returned an instant 403 to our datacenter one, publishes a free API that returns store-level prices for 10,000 calls a day, which makes scraping the site the wrong tool for most of what people want from it.

Two more, Instacart and Whole Foods, gate by geography rather than a bot wall: their real challenge is reading the right region and, for Whole Foods, the Amazon Prime member price, not getting past a defense. The honest answer for all of them is to match the tool to the job and not buy infrastructure a light task does not need. Saying that out loud is what makes the other recommendations trustworthy.

What the data adds up to

Four things, from one day of measuring 52 front doors.

• Akamai is the house vendor of US retail. Build for it first; it guards close to half of these sites and the same pipeline transfers across them.
• A bare client fails almost everywhere. Most defended retailers refused it from any address. Nine sites answered our two vantage points differently, but the clients differed too, so the baseline is a real browser on an address in the site's own market.
• The browser is not optional. The sites that "load" still gate the catalog behind a sensor or a behavioral score, so a residential IP without a real browser stalls on the first price.
• Read the door before you build. A 403 on request one, a queue on the drop, a behavioral score that admits you and then bans you, a four-vendor stack on a live-auction site, and a marketplace with no wall at all are different problems that need different setups.

The full set of retailer-by-retailer writeups, each with the measured defense and the honest setup, starts from proxies for Amazon, the gold-standard ecommerce guide this study was built to support.

Method, and what we could not verify

We tested 52 US retail and marketplace storefronts on August 24, 2026. Each got one plain HTTP GET from a residential home connection and one from a datacenter server, both sending a current desktop-Chrome user-agent and no JavaScript. We recorded the HTTP status, the Set-Cookie and response headers, and any block or challenge body, and identified the vendor from documented fingerprints (Akamai's _abck and AkamaiGHost, Cloudflare's cf-ray and challenge platform, DataDome's datadome cookie and x-datadome headers, PerimeterX's _px cookies and are-you-human page, F5 Shape's telemetry cookie, Kasada's kpsdk markers, Imperva's X-Iinfo header and incap_ses cookies).

What this does not capture: it is a single snapshot, and bot defenses are tuned continuously, so a site's posture can change day to day and by geography. It reflects the front door, not a full bypass attempt, so a site that let our bare client load a homepage may still be very hard to scrape at depth, which Finding 4 is entirely about. Where a site layers vendors, we named the ones visible from the response and the page; there may be more we could not see from outside. And "primary vendor" on a layered site is our judgment call about which one makes the block decision, not a claim that it is the only one present. Three sites (Guitar Center, Tractor Supply, REI) reset or timed out our connection and were left out of the count rather than guessed at.

Sources

• HProxy retail bot-defense test, August 24, 2026: 52 US retail and marketplace storefronts, two vantage points each, fingerprints recorded from live responses.
• Vendor identification references: Akamai Bot Manager, Cloudflare bot management, DataDome, HUMAN Security (formerly PerimeterX), F5 Distributed Cloud Bot Defense, Kasada, Imperva.
• The per-retailer writeups linked throughout, each carrying its own measured evidence and sources.