How an HTTP Proxy Actually Works: A Request's Full Journey

When I started using proxies I assumed they were just "forwarders": put in an address, the request goes out. Then I spent an afternoon debugging a timeout against an HTTPS site, and a packet capture showed I had never understood CONNECT tunneling at all. This post is the explanation I needed back then — it makes every later debugging session shorter.

Where the proxy sits

Direct traffic goes client → origin server. With a proxy it becomes client → proxy → origin server, and the origin sees the proxy's egress IP instead of yours. That is the entire mechanism behind "IP rotation".

There is no magic involved: the proxy sends your request on your behalf and returns the response unchanged. It does not rewrite your payload, and it does not encrypt anything.

Two ways to forward a request

1. Absolute-form URL (plain HTTP only)

The request line carries the full URL instead of just a path:

GET http://example.com/index.html HTTP/1.1
Host: example.com

Seeing a host in the request target tells the proxy where to connect. This explains a classic bug: when you hand-roll an HTTP client, the path you pass must be the complete URL. Most standard libraries do that for you; hand-written sockets usually do not.

2. CONNECT tunnel (required for HTTPS)

HTTPS payloads are encrypted, so the proxy cannot read them — and does not need to. The client first asks the proxy to open a raw TCP channel:

CONNECT example.com:443 HTTP/1.1
Host: example.com:443

Once the proxy answers 200, the client performs its TLS handshake with the origin through that channel. From then on the proxy just moves bytes; certificates and encryption are end to end. That is why HTTPS through an HTTP proxy is slightly slower: one extra round trip to establish the tunnel.

If you wrote your own socket client and forgot CONNECT, the symptom is either an immediate 400 from the proxy or a request that hangs forever.

How a proxy decides you are allowed in

Proxy bandwidth costs money, so access is gated. Two common models:

Model	How it works	Good fit
IP allowlist	Your egress IP is registered; the proxy lets it through	Servers with a stable IP, scheduled jobs
Username / password	Credentials sent in the Proxy-Authorization header	Local development, IPs that change

Allowlisting looks simpler until your egress IP moves. Home connections change IP on reconnect, and some clouds float their elastic IPs. I have lost more than one afternoon to "everything worked yesterday and today every request returns 403" — always a changed egress IP.

As for 407, it is Proxy Authentication Required, returned by the proxy itself. If you are on an allowlist setup and still see 407, that particular line expects credentials. Treat it as a bad line, swap IPs and retry — do not go debugging the origin server.

Verify it yourself

curl is the fastest way to see the whole thing:

# With credentials
curl -x http://user:pass@proxy-ip:port https://httpbin.org/ip

# Allowlist mode needs no credentials
curl -x http://proxy-ip:port https://httpbin.org/ip

In Python, keep two things separate: fetching an IP must go direct, while the request through the proxy is the one that carries credentials.

import requests

session = requests.Session()
session.trust_env = False   # do not let env proxies hijack this call

resp = session.get("http://api.xydaili.net:2022/tools/ip.ashx",
                   params={"action": "GetIP", "OrderNumber": "YOUR-ORDER-ID",
                           "protocol": 1, "qty": 1, "split": "json"},
                   timeout=25)
item = resp.json()["data"][0]
proxy = "http://%s:%s" % (item["ip"], item["port"])

r = requests.get("http://example.com",
                 proxies={"http": proxy, "https": proxy},
                 timeout=15)
print(r.status_code)

Three myths worth dropping

• A proxy is not a VPN. A VPN captures all traffic at the network layer; a proxy only affects the client you configured.
• An HTTP proxy does not encrypt your traffic. Plain HTTP is readable at the proxy hop. HTTPS is safe because of TLS, not because of the proxy.
• A proxy is not anonymity. A high-anonymity proxy only hides your real IP from the origin. The origin still logs the egress IP.

I run my crawlers through 星月代理, which rotates short-lived IPs and supports both API extraction and allowlist modes — the checks above were run on its lines. If you are new to this, get a few trial IPs first and prove the path works end to end before optimising anything.