Proxy Errors 407, 406, 403 and Timeouts: How to Tell Them Apart

The expensive part of debugging a crawler is not changing code, it is working out which layer failed. The same 403 can mean "your IP is not on the allowlist" or "the site blocked you", and the correct reaction is opposite in each case. Here is the order I check things in.

Step one: who sent this response?

• Proxy layer: 407, 502, 504, plus the status field of the extraction API itself;
• Target site: 403, 404, 429, 503;
• Local network: connect timeouts, connection resets, DNS failures.

How to tell them apart: look at the body and headers. Proxy-generated errors are usually a short self-contained HTML or plain-text page with none of the origin's Server or Set-Cookie headers. Print the first 200 characters of the response and it is usually obvious.

Lookup table

Symptom	Meaning	What to do
407	The proxy wants credentials	That line does not match your auth mode — swap IP and retry
406 (from the API)	Extracting too often	Batch it: 10–20 IPs per call
403	Allowlist mismatch, or the target blocked you	Verify the egress IP first, then decide
502 / 504	Proxy node cannot reach the origin	Swap IP; if it persists, test another domain
Connect timeout	Cannot even reach the proxy port	Swap IP, and shorten the timeout
Connection reset	Middlebox or origin closed it	Lower concurrency, add headers, swap IP

407: the proxy wants proof

407 is Proxy Authentication Required. It comes from the proxy, never from the site you were fetching. Two causes:

1. You are using allowlist mode but the line expects username/password (or the reverse);
2. The credentials are wrong, or a password containing @ or : was not URL-encoded.

Treat it as a failure and rotate. In my examples that is exactly what happens, because batches occasionally contain a line that needs credentials.

code, body = request_via_proxy(target, proxy)
if code == 407:
    # this line wants credentials — swap it
    raise RuntimeError("%s returned 407" % proxy)

406: the extraction API telling you to slow down

Extraction endpoints usually carry their own status field, e.g. {"status":406,"msg":"too many requests"}. This is not a network fault — you are simply calling too often. Batching fixes it; see the pool design post.

403: check the egress IP before blaming the site

1. Fetch an IP echo endpoint (for example http://httpbin.org/ip) through the proxy and see what it reports;
2. If it echoes your own address, the proxy never applied — check the proxies argument, environment variables, and whether the extraction call was accidentally routed through a proxy too;
3. If it shows the proxy IP and you still get 403, read the body: allowlist errors are usually explicit text from the provider, whereas an origin 403 carries the site's own page and headers.

The remaining case is origin-side filtering on User-Agent, Referer or cookies. That is not a proxy problem, and no amount of IP rotation will fix it.

Timeouts: separate "cannot connect" from "connected but silent"

• Connect timeout: the TCP handshake never completed — the IP is almost certainly dead. Rotate.
• Read timeout: connected, but the remote is not sending data. Could be a slow origin or a poor line.

Configure both, and keep them short. In a crawler, waiting 60 seconds is almost never better than rotating and retrying.

r = requests.get(url,
                 proxies={"http": proxy, "https": proxy},
                 timeout=(6, 12))   # (connect timeout, read timeout)

Closing checklist

• Is the extraction request going direct? (Any HTTP_PROXY in the environment?)
• Does your retry logic actually rotate the IP? Many retry frameworks reuse the same proxy — that is not a retry.
• Is per-IP concurrency too high? Drop to 2–5 and compare.
• Are timeouts so long that the queue looks hung?
• Do you count proxy-layer errors separately from origin errors? Mixing them makes both invisible.

The lines I use come from 星月代理, whose error messages are refreshingly explicit — it will tell you the IP is not on the allowlist, and how many seconds until the next attempt is allowed. Whatever provider you use, split these error classes in your metrics; the next incident will diagnose itself.