Four Design Decisions Behind a Crawler Proxy Pool (with Python)

My first crawler broke like this: fetch one IP before every request, throw it away after. The moment the job got real traffic, the extraction API started answering status=406, rate limited and whole batches of requests began failing. That evening I rewrote it as "batch extraction plus a local pool". API calls dropped by an order of magnitude and the success rate went from about 70% back to 99%.

These are the four decisions I have re-used in every project since.

1. Never extract one IP per request

Extraction endpoints are rate limited for good reason: they hand out resources, they are not a per-request middleware. There is also a latency argument — a 10–25 ms API round trip in front of every fetch caps your throughput no matter how many threads you run.

What works: pull 10–20 IPs at a time into a local queue, and refill before the queue runs dry. Extraction frequency drops from "once per request" to "once per few dozen requests".

2. Size the low-water mark properly

Two thresholds do the job:

• High-water mark (say 30): how much you are willing to stockpile. Short-lived IPs expire while sitting idle, so hoarding is pointless.
• Low-water mark (say 5): refill as soon as the pool drops below it.

Do not set the low-water mark to 1. Refilling is a network call; by the time the pool is down to a single IP, that call leaves your workers starved and throwing exceptions. The low-water mark exists precisely to absorb that gap.

A rough rule: low-water ≈ worker count × 2. Ten workers, a low-water mark around 20 is comfortable.

3. Expiry: trust the server, but floor it locally

Short-lived IPs die in minutes, so stale entries are the difference between a working pool and a pool full of corpses. Both sources of truth are useful:

Source	Upside	Watch out for
The expire field from the API	Accurate, matches the service's own view	Time zones — it is usually local server time
A local timestamp	Simple, no parsing of a format that may change	Must be set shorter than the real lifetime

I use both and take the minimum: expire_ts = min(now + local_max_age, server_expire). A local cap of 170 seconds works well when IPs live for three minutes — it leaves a ten-second margin so you never hand a worker an IP that expires mid-request.

4. How much concurrency can one IP take?

This is the most commonly missed part. Rotating IPs does not grant unlimited concurrency; the target still rate limits per IP and will return 403 if you push too hard. My ranges:

• 2–5 concurrent requests per IP, and no more than ~3 requests per second;
• for more throughput, add IPs — do not squeeze a single one;
• the real ceiling for a given site is what the site can absorb, which is the subject of the rate-limiting post.

The implementation

This pool does three things: thread-safe checkout, automatic low-water refill, and expiry-based eviction.

import threading
import time

import requests

API = "http://api.xydaili.net:2022/tools/ip.ashx"


class ProxyPool(object):
    def __init__(self, order, qty=20, low_watermark=5,
                 area="", isp="", timeout=10, max_age=170):
        self.order = order
        self.qty = qty
        self.low_watermark = low_watermark
        self.area = area
        self.isp = isp
        self.timeout = timeout
        self.max_age = max_age

        self._items = []          # [(proxy_url, expire_ts)]
        self._lock = threading.Lock()

    def _fetch(self):
        params = {
            "action": "GetIP",
            "OrderNumber": self.order,
            "protocol": 1,
            "qty": self.qty,
            "split": "json",
        }
        if self.area:
            params["Area"] = self.area
        if self.isp:
            params["Isp"] = self.isp

        # Extraction must go direct: trust_env=False ignores env proxies
        session = requests.Session()
        session.trust_env = False
        try:
            resp = session.get(API, params=params, timeout=self.timeout)
        finally:
            session.close()

        data = resp.json()
        if data.get("status") != 200 or not data.get("data"):
            raise RuntimeError("extract failed: %s" % data.get("msg"))

        now = time.time()
        fresh = []
        for item in data["data"]:
            proxy_url = "http://%s:%s" % (item["ip"], item["port"])
            expire_ts = now + self.max_age
            try:  # API returns e.g. 2026-09-27 10:26:31
                text = str(item["expire"])[:19].replace("T", " ")
                expire_ts = min(expire_ts, time.mktime(
                    time.strptime(text, "%Y-%m-%d %H:%M:%S")))
            except Exception:
                pass
            fresh.append((proxy_url, expire_ts))
        return fresh

    def get(self):
        """Check out a live proxy, refilling when the pool runs low."""
        with self._lock:
            now = time.time()
            self._items = [it for it in self._items if it[1] > now]
            if len(self._items) < self.low_watermark:
                self._items.extend(self._fetch())
            return self._items.pop(0)[0]

    @property
    def size(self):
        with self._lock:
            now = time.time()
            return len([1 for _, exp in self._items if exp > now])

Usage:

pool = ProxyPool(order="YOUR-ORDER-ID", qty=20, low_watermark=5)

for url in urls:
    proxy = pool.get()
    try:
        r = requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=10)
    except requests.RequestException as exc:
        # Dead short-lived IPs are normal: log it, move on
        print("swapping IP:", exc.__class__.__name__)

Details that bite later

• Keep the extraction call off the proxy. If your container exports HTTP_PROXY, that call can end up going through a proxy — and when the proxy dies, the whole pipeline dies with it.
• Do not hot-loop on refill failures. When extraction returns an error (expired order, rate limit), retrying immediately makes it worse. Log it and back off.
• Track consumption. Most APIs return a usage counter — ship it to monitoring, or you will discover an exhausted quota the hard way.
• Multi-process needs a shared pool. The class above is per-process; move the queue into Redis when you scale out.

The code above is packaged as a runnable project alongside 17 other languages — Go, Node.js, TypeScript, Java, C#, PHP, Rust and more — in github-xydaili-examples. The lines I use come from 星月代理 (Xingyue Proxy), which covers 300+ Chinese cities with automatically rotated short-lived IPs; the field names above match its API.