Rate Limiting and Etiquette: Don't Take Down Someone's Site

Years ago I built a small price-comparison crawler against a site that was not very large. I ran 200 concurrent requests. By the afternoon their site was returning 502s, and the next day my whole block of IPs was null-routed and the job was dead for three days.

That is when I stopped treating proxies as a substitute for restraint. A proxy answers "who am I". Rate limiting answers "how hard am I pushing". You need both.

Rotation bypasses per-IP limits, not capacity

There is a persistent illusion that a big enough IP pool means unlimited throughput. But the target is not only throttling per IP — it has database connections, bandwidth and backend services with their own ceilings. Spread the same request volume over 500 IPs and the origin still falls over; the only difference is that the logs no longer show one culprit address.

And the consequences of breaking someone's service are worse than a temporary block: legal letters, or an entire IP range blacklisted along with your other, unrelated projects.

Per-domain limits: pick numbers you will respect

My defaults:

• At most 10 concurrent requests and 5 requests per second to a single domain;
• For small sites (personal blogs, legacy systems) drop to 2–3 concurrent;
• Relax slightly overnight, tighten during their business hours.

Note that "concurrent" means aggregated across all proxies. After adding rotation, people often only watch per-IP concurrency: 50 IPs × 5 concurrent each is 250 concurrent requests to the target — which is not rate limiting at all.

A global token bucket in Redis

With multiple processes or machines, local counters add up to something you did not intend. Keep the limiter in Redis so every worker draws from one budget:

import time

import redis

TOKEN_BUCKET_LUA = """
local key = KEYS[1]
local rate = tonumber(ARGV[1])        -- tokens added per second
local burst = tonumber(ARGV[2])       -- bucket capacity
local now = tonumber(ARGV[3])

local data = redis.call('HMGET', key, 'tokens', 'ts')
local tokens = tonumber(data[1]) or burst
local ts = tonumber(data[2]) or now

tokens = math.min(burst, tokens + (now - ts) * rate)
if tokens < 1 then
    redis.call('HMSET', key, 'tokens', tokens, 'ts', now)
    return 0
end
tokens = tokens - 1
redis.call('HMSET', key, 'tokens', tokens, 'ts', now)
return 1
"""

client = redis.Redis(decode_responses=True)
script = client.register_script(TOKEN_BUCKET_LUA)


def allow(domain, rate=5, burst=10):
    """Ask for a token before hitting this domain."""
    ok = script(keys=["rl:%s" % domain],
                args=[rate, burst, time.time()])
    return bool(ok)

Usage is one line before each request: if not allow(domain): time.sleep(0.2). Key the bucket by domain, never by URL — per-URL buckets are equivalent to no limit at all.

Backoff: exponential, with jitter

Retrying immediately after being throttled is the worst possible response — it converts a temporary limit into a sustained ban. Use exponential backoff, and add randomness, or hundreds of queued tasks will retry in lockstep and create a fresh spike:

import random
import time


def backoff_sleep(attempt, base=1.0, cap=60.0):
    """After failure N: 1s, 2s, 4s ... capped at 60s, with jitter."""
    delay = min(cap, base * (2 ** attempt))
    delay = delay * (0.5 + random.random() * 0.5)   # spread the retries out
    time.sleep(delay)

Also branch on the error class: an explicit 403 or 429 deserves a long wait; a timeout may just be a flaky line and can be retried sooner. One uniform retry policy is always wrong for one of the two.

Caching and incremental crawls: the cheapest optimisation

Fetching less beats fetching faster. In order of payoff:

1. Incremental list pages — filter by the newest timestamp or ID you already have instead of paginating everything;
2. Response cache with a TTL — the same URL within the window is served locally;
3. Conditional requests — send If-Modified-Since or ETag; a 304 is nearly free for the origin.

Applied to a daily job of mine, these three cut request volume from roughly 400k to 60k per day, and the target's response times improved noticeably.

Where the ethical line sits

• robots.txt is a floor, not a ceiling. Nothing forbidding you does not mean you should saturate the site at peak hours.
• Do not collect personal data. Phone numbers, ID numbers, home addresses — even when they are on a public page, the exposure is not worth it.
• Do not bypass paywalls. Technically circumventing a paid access control is straightforwardly infringing.
• Identify yourself. Use a User-Agent that points to a contact; if something goes wrong, people prefer emailing you over blocking you.
• Prefer official APIs when they exist — they are usually far more stable than scraping HTML.

One note on where proxies fit: spreading requests over many egress IPs keeps each individual IP's request rate in the range a normal user produces. It is not a way to push more load onto someone else's servers. The lines I use are from 星月代理, rotating short-lived IPs, which sits nicely under the global limiter above. The pool implementation and examples in 18 languages are in github-xydaili-examples.