WebSocket
A protocol that upgrades an HTTP request into a persistent two-way connection, used for live data feeds, and a special case for proxies to carry.
A WebSocket starts life disguised as an ordinary web request. The client sends what looks like a normal HTTP request carrying an Upgrade header; the server answers 101 Switching Protocols; and from that moment the connection stops being request-and-response at all. It becomes a persistent, two-way channel over the same TCP connection, where either side sends messages whenever it likes, for as long as the connection lives.
The disguise was the design goal: by beginning as HTTP on the standard web ports, 80 for ws and 443 for the TLS-wrapped wss, the protocol traverses infrastructure built for web traffic. What it buys over repeatedly asking a server whether anything changed is immediacy and economy, one standing connection instead of a drumbeat of polls, which is why it carries the web's live surfaces: price tickers, order books, chat, notifications, sports scores, collaborative editors, live dashboards.
That list explains why scrapers keep meeting it. On such pages the document you fetch is scaffolding, and the numbers you wanted arrive later, as messages down a socket the page opens after loading. Fetching the HTML gets you a shell with no data; even rendering the page only helps if you then listen. The direct route is the same one the page uses: open the socket endpoint yourself, speak whatever subscription message the site's own client sends, and read the feed, which is usually cleaner JSON than the page would ever have shown you.
Proxying it works, with asymmetries worth knowing. Encrypted wss rides through an HTTP proxy inside an ordinary CONNECT tunnel, the same blind relay HTTPS uses, so it generally just works. Plain ws asks the proxy to handle the Upgrade explicitly, which some HTTP proxies handle badly; in practice almost everything real is wss anyway. A SOCKS5 proxy, relaying TCP without interpreting it, carries either happily. The failure mode unique to sockets is duration: a connection held open for hours makes the exit's stability part of your architecture, and a rotating pool's exits are not built to persist. Long-lived feeds want a static address or a sticky session sized to the job.
One more property matters at volume: a socket is stateful in a way requests are not. Your subscription, your authentication and the server's idea of who you are all live on the one connection, so an exit dying mid-stream is not a retry, it is a reconnect and re-subscribe, and a well-built consumer treats that as an expected event rather than an error.
Frequently asked questions
Do WebSockets work through a proxy?
Generally yes. The encrypted form, wss, travels through HTTP proxies inside the same CONNECT tunnel as any HTTPS traffic, and SOCKS5 relays it as ordinary TCP. Unencrypted ws through an HTTP proxy is the fragile combination, since the proxy must relay the Upgrade handshake, and support there is uneven. Since real deployments are overwhelmingly wss, most setups simply work.
Why is the data missing when I scrape a live page?
Because the page is a shell and the content arrives over a socket after load. Fetching the document captures the scaffolding only. The efficient fix is to find the socket endpoint in the browser's network tab, connect to it directly, send the same subscription message the page sends, and consume the feed, which typically returns cleaner structured data than parsing the rendered page would.
Which proxy type should I use for WebSocket feeds?
One whose exit will still exist in an hour. A held-open connection makes exit stability part of the design, so rotating pools are the wrong shape: a static address, an ISP proxy, or a sticky session sized to the connection's expected life fit better. Beyond persistence, the usual network-trust considerations apply unchanged, since the site sees the socket's origin exactly as it would a request's.
Are WebSocket connections rate limited?
Differently from requests. Sites cap concurrent connections per address and per account, throttle message rates within a connection, and drop consumers that subscribe too broadly. Because state lives on the connection, enforcement often looks like a disconnect rather than an error code, so a resilient consumer expects to reconnect and re-subscribe as a matter of course.
Back to the full glossary.