Rate Limiting and Etiquette: Don't Take Down Someone's Site
Rotating IPs gets you around per-IP throttling, not around the target's capacity. Per-domain limits, exponential backoff with jitter, caching, and a Redis token bucket.
Most scraper failures
are actually proxy-layer failures
Field notes from production crawlers: which proxy to use, how to design a pool that does not hammer the extraction API, how to write a Scrapy middleware that really rotates IPs, and how to tell a proxy error from a target-site error. Every post ships runnable code, not just concepts.
Latest posts
5 posts · newest firstA Scrapy Proxy Middleware That Rotates IPs per Request
What each downloader-middleware hook is for, why retries keep reusing the dead proxy, and why adding proxies sometimes makes failures go up. Full working code.
Proxy Errors 407, 406, 403 and Timeouts: How to Tell Them Apart
These codes come from three different layers. One lookup table plus a per-code playbook, starting with the question that actually matters: who sent this response?
Four Design Decisions Behind a Crawler Proxy Pool (with Python)
Why you should never extract one IP per request, how to size the low-water mark, whether to trust the server's expiry field, and how much concurrency one IP can take.
How an HTTP Proxy Actually Works: A Request's Full Journey
Absolute-form URLs versus CONNECT tunnels, why HTTPS needs a handshake first, and when to use IP allowlisting instead of username/password.
About the code: every example is packaged as a runnable project covering Python, Node.js, TypeScript, Java, Kotlin, Scala, C#, VB.NET, Go, PHP, Ruby, Perl, Rust, Swift, Dart, C++, Shell and PowerShell — 18 languages in total, each demonstrating the same loop: extract an IP → fetch through it → swap IP and retry on failure. It lives in github-xydaili-examples.
Note on the Chinese posts: there is a full Chinese edition of this site with the same articles plus extra details specific to domestic Chinese proxy networks.