Cases / #53 · 2026-10-16 · Medium

Surviving a soft block: honour Retry-After, then move to another exit without losing the login

Two crawlers against the same rate-limited site. The one that retries a 429 straight away is challenged and gets 3 of 8 pages. The one that waits as told and moves the session to another proxy when its exit is blocked gets all 8 — same browser, same cookies.

Chromium

Run on production on 2026-10-16: ✓ Node.js ✓ Python ✓ Java ✓ C# ✓ Go

The problem

Sites rarely block outright on the first request. They slow you down first — a 429 Too Many Requests with a Retry-After — and only challenge or ban an address that ignores it. A crawler needs a policy: tell a rate limit from a block, wait when asked, and when one exit is burned, continue from another without starting over (and without logging in again). What does that look like on a remote browser, and what does ignoring the signal cost?

What we used, and why

What	Why
A stand-in rate limiter (`context.route`)	Real sites don't rate-limit on cue, so the site is simulated in front of real httpbin requests. Each request really goes out through the session's proxy (`route.fetch`), and the limiter decides by the IP httpbin saw: 3 pages in a burst, then one per 5 s (429 with `Retry-After`), at most 6 per IP. An IP that comes back early or exceeds the quota gets a challenge page from then on.
A response classifier	`429` is a rate limit (wait). `403`/`503` or challenge text means this IP is blocked (move). `200` is data. Anything else (a `502` from a flaky upstream) is transient: retry after a second. Those show up in the trail now and then.
`Retry-After`, with exponential backoff as a fallback	Wait exactly as long as the site asks; if it doesn't say, back off 1, 2, 4… seconds.
Live proxy swap (`POST /admin/session/proxy` on the session's router)	Moves the running session to another proxy — no relaunch.
A new context with `storageState`	New connections, so every request uses the new proxy (live proxy swap), while the cookies come along.
Sticky SOCKS5 proxies	One exit IP per proxy, so "this IP is blocked" means something.

How it works

Naive crawler: one session, sets a session cookie, fetches 8 pages; on a 429 it retries straight away (up to 3 times), on a block it skips the page.
Polite crawler: the same, but on a 429 it waits Retry-After seconds, and on a block it swaps the session's proxy, opens a new context with the old storageState and retries.
Both read the cookie back at the end and report, per page, the status trail and the exit IP that served it.

The code

The same program in five languages (also on GitHub, with the raw output). Set these environment variables first:

CDPFLEET_API_KEY — your API key (dashboard → API keys)
SOCKS_PROXIES — comma-separated socks5://user:pass@host:port proxies

// npm install [email protected]
// env: CDPFLEET_API_KEY, SOCKS_PROXIES (comma-separated sticky socks5:// URLs: one exit IP each)
import { chromium } from 'playwright';

const KEY = process.env.CDPFLEET_API_KEY;
const PROXIES = process.env.SOCKS_PROXIES.split(',');
const PAGES = Array.from({ length: 8 }, (_, i) => `https://httpbin.org/anything/catalogue/page-${i + 1}`);

// A stand-in for a rate-limited site, in front of real httpbin requests. Each request really goes
// out through the session's proxy; the limiter decides by the IP httpbin saw (its "origin"):
// 3 pages in a burst, then one page per 5 s (429 + Retry-After: 5), 6 pages per IP in total. An IP
// that comes back before its Retry-After, or is over its quota, gets a challenge page from then on.
function limiter() {
  const ips = new Map();
  return async (route) => {
    const now = Date.now(); // when the request arrived
    const real = await route.fetch({ timeout: 30000 }).catch(() => null);
    if (!real) return route.fulfill({ status: 502, contentType: 'text/plain', body: 'Bad Gateway' }); // the proxy hiccuped
    if (real.status() !== 200) return route.fulfill({ response: real }); // httpbin itself failed
    const ip = (await real.json()).origin;
    const s = ips.get(ip) ?? { served: 0, last: 0, retryAt: 0, challenged: false };
    ips.set(ip, s);
    if (now < s.retryAt || s.served >= 6) s.challenged = true;
    if (s.challenged) {
      return route.fulfill({ status: 403, contentType: 'text/html', body: '<title>Just a moment...</title><p>Checking your browser before accessing the site.</p>' });
    }
    if (s.served >= 3 && now - s.last < 5000) {
      s.retryAt = now + 5000;
      return route.fulfill({ status: 429, headers: { 'retry-after': '5' }, contentType: 'text/plain', body: 'Too Many Requests' });
    }
    s.served++;
    s.last = now;
    return route.fulfill({ response: real });
  };
}

// What the response means: the data, a rate limit (wait), or a block (this IP is done).
function classify(status, text) {
  if (status === 429) return 'rate_limited';
  if (status === 403 || status === 503 || /just a moment|captcha|access denied/i.test(text)) return 'blocked';
  return status === 200 ? 'ok' : 'error';
}

// 503 means the fleet is momentarily busy: wait Retry-After and try again.
async function launch(body) {
  for (let attempt = 1; ; attempt++) {
    const res = await fetch('https://starter.cdpfleet.com/chromium/session', {
      method: 'POST',
      headers: { 'x-api-key': KEY, 'content-type': 'application/json' },
      body: JSON.stringify(body),
    });
    if (res.ok) return res.json();
    if (res.status !== 503 || attempt === 5) throw new Error(`launch ${res.status} ${await res.text()}`);
    await new Promise((r) => setTimeout(r, (Number(res.headers.get('retry-after')) || 5) * 1000));
  }
}

async function crawl(polite) {
  const { sessionId, wsUrl } = await launch({ proxy: PROXIES[polite ? 1 : 0], headless: 'new', record: false });
  const browser = await chromium.connect(wsUrl, { headers: { 'x-api-key': KEY } });
  const site = limiter();
  // Live proxy swap on the session's router: new connections use the new proxy.
  const swapProxy = async (proxy) => {
    const r = await fetch(`https://${new URL(wsUrl).host}/admin/session/proxy`, {
      method: 'POST',
      headers: { 'x-api-key': KEY, 'content-type': 'application/json' },
      body: JSON.stringify({ session_id: sessionId, proxy }),
    });
    if (!r.ok) throw new Error(`swap ${r.status} ${await r.text()}`);
  };
  const open = async (storageState) => {
    const context = await browser.newContext({ storageState });
    await context.route('https://httpbin.org/anything/**', site);
    return { context, page: await context.newPage() };
  };
  const rows = [];
  let waited = 0;
  let swaps = 0;
  let next = 2;
  let cookieKept = false;
  try {
    let { context, page } = await open();
    // A logged-in state that must survive a proxy swap.
    await page.goto('https://httpbin.org/cookies/set?session=alice-42', { timeout: 60000 });
    for (const [i, url] of PAGES.entries()) {
      const trail = [];
      let row = { page: i + 1, outcome: 'gave up', exit_ip: null };
      for (let attempt = 1; attempt <= 6; attempt++) {
        const res = await page.goto(url, { timeout: 60000 });
        const text = await page.textContent('body');
        const kind = classify(res.status(), text);
        trail.push(res.status());
        if (kind === 'ok') {
          row = { page: i + 1, outcome: 'ok', exit_ip: JSON.parse(text).origin };
          break;
        }
        if (kind === 'rate_limited') {
          // Honour Retry-After; without one, back off exponentially (1, 2, 4… s, capped).
          if (!polite && attempt === 3) break;
          const seconds = polite ? Number(res.headers()['retry-after'] ?? Math.min(2 ** (attempt - 1), 30)) : 0;
          if (seconds) trail.push(`wait ${seconds}s`);
          waited += seconds;
          await new Promise((r) => setTimeout(r, seconds * 1000));
          continue;
        }
        if (kind === 'blocked' && polite && next < PROXIES.length) {
          // This exit is done: move the session to another proxy and continue in a new context
          // (its own connections) that keeps the cookies and storage.
          await swapProxy(PROXIES[next++]);
          const state = await context.storageState();
          await context.close();
          ({ context, page } = await open(state));
          swaps++;
          trail.push('swap proxy');
          continue;
        }
        if (kind === 'error') {
          await new Promise((r) => setTimeout(r, 1000)); // a transient failure: try again
          continue;
        }
        break;
      }
      rows.push({ ...row, trail: trail.join(' → ') });
    }
    await page.goto('https://httpbin.org/cookies', { timeout: 60000 });
    cookieKept = JSON.parse(await page.textContent('body')).cookies.session === 'alice-42';
  } finally {
    await browser.close();
  }
  return {
    client: polite ? 'polite' : 'naive',
    pages_fetched: rows.filter((r) => r.outcome === 'ok').length,
    exits: [...new Set(rows.map((r) => r.exit_ip).filter(Boolean))],
    waited_seconds: waited,
    proxy_swaps: swaps,
    session_cookie_kept: cookieKept,
    pages: rows,
  };
}

// Naive: retries a 429 straight away and has no plan for a block. Polite: waits as told and moves
// to another exit when one is blocked.
const naive = await crawl(false);
const polite = await crawl(true);
console.log(JSON.stringify({ naive, polite }, null, 2));

# pip install playwright==1.60.0 requests
# env: CDPFLEET_API_KEY, SOCKS_PROXIES (comma-separated sticky socks5:// URLs: one exit IP each)
import json
import os
import re
import time
from urllib.parse import urlparse

import requests
from playwright.sync_api import sync_playwright

KEY = os.environ["CDPFLEET_API_KEY"]
PROXIES = os.environ["SOCKS_PROXIES"].split(",")
PAGES = [f"https://httpbin.org/anything/catalogue/page-{i + 1}" for i in range(8)]
CHALLENGE = "<title>Just a moment...</title><p>Checking your browser before accessing the site.</p>"


# A stand-in for a rate-limited site, in front of real httpbin requests. Each request really goes
# out through the session's proxy; the limiter decides by the IP httpbin saw (its "origin"):
# 3 pages in a burst, then one page per 5 s (429 + Retry-After: 5), 6 pages per IP in total. An IP
# that comes back before its Retry-After, or is over its quota, gets a challenge page from then on.
def limiter():
    ips = {}

    def handle(route):
        now = time.time() * 1000  # when the request arrived
        try:
            real = route.fetch(timeout=30000)
        except Exception:
            return route.fulfill(status=502, content_type="text/plain", body="Bad Gateway")  # the proxy hiccuped
        if real.status != 200:
            return route.fulfill(response=real)  # httpbin itself failed
        s = ips.setdefault(real.json()["origin"], {"served": 0, "last": 0, "retry_at": 0, "challenged": False})
        if now < s["retry_at"] or s["served"] >= 6:
            s["challenged"] = True
        if s["challenged"]:
            return route.fulfill(status=403, content_type="text/html", body=CHALLENGE)
        if s["served"] >= 3 and now - s["last"] < 5000:
            s["retry_at"] = now + 5000
            return route.fulfill(status=429, headers={"retry-after": "5"}, content_type="text/plain", body="Too Many Requests")
        s["served"] += 1
        s["last"] = now
        return route.fulfill(response=real)

    return handle


# What the response means: the data, a rate limit (wait), or a block (this IP is done).
def classify(status, text):
    if status == 429:
        return "rate_limited"
    if status in (403, 503) or re.search(r"just a moment|captcha|access denied", text, re.I):
        return "blocked"
    return "ok" if status == 200 else "error"


# 503 means the fleet is momentarily busy: wait Retry-After and try again.
def launch(body):
    for attempt in range(1, 6):
        r = requests.post("https://starter.cdpfleet.com/chromium/session", headers={"x-api-key": KEY}, json=body, timeout=120)
        if r.ok:
            return r.json()
        if r.status_code != 503 or attempt == 5:
            raise RuntimeError(f"launch {r.status_code} {r.text}")
        time.sleep(int(r.headers.get("retry-after") or 5))


def crawl(p, polite):
    session = launch({"proxy": PROXIES[1 if polite else 0], "headless": "new", "record": False})
    browser = p.chromium.connect(session["wsUrl"], headers={"x-api-key": KEY})
    site = limiter()

    # Live proxy swap on the session's router: new connections use the new proxy.
    def swap_proxy(proxy):
        r = requests.post(f"https://{urlparse(session['wsUrl']).netloc}/admin/session/proxy", headers={"x-api-key": KEY},
                          json={"session_id": session["sessionId"], "proxy": proxy}, timeout=30)
        if not r.ok:
            raise RuntimeError(f"swap {r.status_code} {r.text}")

    def open_context(storage_state=None):
        context = browser.new_context(storage_state=storage_state)
        context.route("https://httpbin.org/anything/**", site)
        return context, context.new_page()

    rows, waited, swaps, nxt, cookie_kept = [], 0, 0, 2, False
    try:
        context, page = open_context()
        # A logged-in state that must survive a proxy swap.
        page.goto("https://httpbin.org/cookies/set?session=alice-42", timeout=60000)
        for i, url in enumerate(PAGES):
            trail = []
            row = {"page": i + 1, "outcome": "gave up", "exit_ip": None}
            for attempt in range(1, 7):
                res = page.goto(url, timeout=60000)
                text = page.text_content("body")
                kind = classify(res.status, text)
                trail.append(str(res.status))
                if kind == "ok":
                    row = {"page": i + 1, "outcome": "ok", "exit_ip": json.loads(text)["origin"]}
                    break
                if kind == "rate_limited":
                    if not polite and attempt == 3:
                        break
                    # Honour Retry-After; without one, back off exponentially (1, 2, 4… s, capped).
                    seconds = int(res.headers.get("retry-after") or min(2 ** (attempt - 1), 30)) if polite else 0
                    if seconds:
                        trail.append(f"wait {seconds}s")
                    waited += seconds
                    time.sleep(seconds)
                    continue
                if kind == "blocked" and polite and nxt < len(PROXIES):
                    # This exit is done: move the session to another proxy and continue in a new context
                    # (its own connections) that keeps the cookies and storage.
                    swap_proxy(PROXIES[nxt])
                    nxt += 1
                    state = context.storage_state()
                    context.close()
                    context, page = open_context(state)
                    swaps += 1
                    trail.append("swap proxy")
                    continue
                if kind == "error":
                    time.sleep(1)  # a transient failure: try again
                    continue
                break
            rows.append({**row, "trail": " → ".join(trail)})
        page.goto("https://httpbin.org/cookies", timeout=60000)
        cookie_kept = json.loads(page.text_content("body"))["cookies"].get("session") == "alice-42"
    finally:
        browser.close()
    return {
        "client": "polite" if polite else "naive",
        "pages_fetched": sum(1 for r in rows if r["outcome"] == "ok"),
        "exits": list(dict.fromkeys(r["exit_ip"] for r in rows if r["exit_ip"])),
        "waited_seconds": waited,
        "proxy_swaps": swaps,
        "session_cookie_kept": cookie_kept,
        "pages": rows,
    }


# Naive: retries a 429 straight away and has no plan for a block. Polite: waits as told and moves
# to another exit when one is blocked.
with sync_playwright() as p:
    naive = crawl(p, False)
    polite = crawl(p, True)
print(json.dumps({"naive": naive, "polite": polite}, indent=2, ensure_ascii=False))

// Maven: com.microsoft.playwright:playwright:1.60.0, com.google.code.gson:gson:2.11.0
// Run with PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1.
// env: CDPFLEET_API_KEY, SOCKS_PROXIES (comma-separated sticky socks5:// URLs: one exit IP each)
import com.google.gson.*;
import com.microsoft.playwright.*;
import java.net.URI;
import java.net.http.*;
import java.util.*;
import java.util.function.Consumer;
import java.util.regex.Pattern;

public class Main {
  static final String KEY = System.getenv("CDPFLEET_API_KEY");
  static final String[] PROXIES = System.getenv("SOCKS_PROXIES").split(",");
  static final Gson GSON = new GsonBuilder().setPrettyPrinting().disableHtmlEscaping().serializeNulls().create();
  static final HttpClient HTTP = HttpClient.newHttpClient();
  static final String CHALLENGE = "<title>Just a moment...</title><p>Checking your browser before accessing the site.</p>";
  static final Pattern CHALLENGE_TEXT = Pattern.compile("just a moment|captcha|access denied", Pattern.CASE_INSENSITIVE);

  static class IpState { int served; long last; long retryAt; boolean challenged; }

  // A stand-in for a rate-limited site, in front of real httpbin requests. Each request really goes
  // out through the session's proxy; the limiter decides by the IP httpbin saw (its "origin"):
  // 3 pages in a burst, then one page per 5 s (429 + Retry-After: 5), 6 pages per IP in total. An IP
  // that comes back before its Retry-After, or is over its quota, gets a challenge page from then on.
  static Consumer<Route> limiter() {
    Map<String, IpState> ips = new HashMap<>();
    return (route) -> {
      long now = System.currentTimeMillis(); // when the request arrived
      APIResponse real;
      try {
        real = route.fetch(new Route.FetchOptions().setTimeout(30000));
      } catch (PlaywrightException e) { // the proxy hiccuped
        route.fulfill(new Route.FulfillOptions().setStatus(502).setContentType("text/plain").setBody("Bad Gateway"));
        return;
      }
      if (real.status() != 200) { // httpbin itself failed
        route.fulfill(new Route.FulfillOptions().setResponse(real));
        return;
      }
      String ip = JsonParser.parseString(real.text()).getAsJsonObject().get("origin").getAsString();
      IpState s = ips.computeIfAbsent(ip, (k) -> new IpState());
      if (now < s.retryAt || s.served >= 6) s.challenged = true;
      if (s.challenged) {
        route.fulfill(new Route.FulfillOptions().setStatus(403).setContentType("text/html").setBody(CHALLENGE));
        return;
      }
      if (s.served >= 3 && now - s.last < 5000) {
        s.retryAt = now + 5000;
        route.fulfill(new Route.FulfillOptions().setStatus(429).setHeaders(Map.of("retry-after", "5")).setContentType("text/plain").setBody("Too Many Requests"));
        return;
      }
      s.served++;
      s.last = now;
      route.fulfill(new Route.FulfillOptions().setResponse(real));
    };
  }

  // What the response means: the data, a rate limit (wait), or a block (this IP is done).
  static String classify(int status, String text) {
    if (status == 429) return "rate_limited";
    if (status == 403 || status == 503 || CHALLENGE_TEXT.matcher(text).find()) return "blocked";
    return status == 200 ? "ok" : "error";
  }

  // 503 means the fleet is momentarily busy: wait Retry-After and try again.
  static JsonObject launch(JsonObject body) throws Exception {
    for (int attempt = 1; ; attempt++) {
      HttpResponse<String> res = HTTP.send(HttpRequest.newBuilder(URI.create("https://starter.cdpfleet.com/chromium/session"))
          .header("x-api-key", KEY).header("content-type", "application/json")
          .POST(HttpRequest.BodyPublishers.ofString(body.toString())).build(), HttpResponse.BodyHandlers.ofString());
      if (res.statusCode() == 200) return JsonParser.parseString(res.body()).getAsJsonObject();
      if (res.statusCode() != 503 || attempt == 5) throw new RuntimeException("launch " + res.statusCode() + " " + res.body());
      Thread.sleep(Long.parseLong(res.headers().firstValue("retry-after").orElse("5")) * 1000);
    }
  }

  // Live proxy swap on the session's router: new connections use the new proxy.
  static void swapProxy(String wsUrl, String sessionId, String proxy) throws Exception {
    JsonObject body = new JsonObject();
    body.addProperty("session_id", sessionId);
    body.addProperty("proxy", proxy);
    HttpResponse<String> res = HTTP.send(HttpRequest.newBuilder(URI.create("https://" + URI.create(wsUrl).getHost() + "/admin/session/proxy"))
        .header("x-api-key", KEY).header("content-type", "application/json")
        .POST(HttpRequest.BodyPublishers.ofString(body.toString())).build(), HttpResponse.BodyHandlers.ofString());
    if (res.statusCode() != 200) throw new RuntimeException("swap " + res.statusCode() + " " + res.body());
  }

  static JsonObject crawl(Playwright pw, boolean polite) throws Exception {
    JsonObject launchBody = new JsonObject();
    launchBody.addProperty("proxy", PROXIES[polite ? 1 : 0]);
    launchBody.addProperty("headless", "new");
    launchBody.addProperty("record", false);
    JsonObject session = launch(launchBody);
    String wsUrl = session.get("wsUrl").getAsString();
    String sessionId = session.get("sessionId").getAsString();
    Browser browser = pw.chromium().connect(wsUrl, new BrowserType.ConnectOptions().setHeaders(Map.of("x-api-key", KEY)));
    Consumer<Route> site = limiter();
    JsonArray rows = new JsonArray();
    List<String> exits = new ArrayList<>();
    int waited = 0, swaps = 0, next = 2, fetched = 0;
    boolean cookieKept;
    try {
      BrowserContext context = browser.newContext();
      context.route("https://httpbin.org/anything/**", site);
      Page page = context.newPage();
      // A logged-in state that must survive a proxy swap.
      page.navigate("https://httpbin.org/cookies/set?session=alice-42", new Page.NavigateOptions().setTimeout(60000));
      for (int i = 0; i < 8; i++) {
        List<String> trail = new ArrayList<>();
        String outcome = "gave up";
        String exitIp = null;
        for (int attempt = 1; attempt <= 6; attempt++) {
          Response res = page.navigate("https://httpbin.org/anything/catalogue/page-" + (i + 1), new Page.NavigateOptions().setTimeout(60000));
          String text = page.textContent("body");
          String kind = classify(res.status(), text);
          trail.add(String.valueOf(res.status()));
          if (kind.equals("ok")) {
            outcome = "ok";
            exitIp = JsonParser.parseString(text).getAsJsonObject().get("origin").getAsString();
            break;
          }
          if (kind.equals("rate_limited")) {
            if (!polite && attempt == 3) break;
            // Honour Retry-After; without one, back off exponentially (1, 2, 4… s, capped).
            String ra = res.headers().get("retry-after");
            int seconds = !polite ? 0 : ra != null ? Integer.parseInt(ra) : (int) Math.min(Math.pow(2, attempt - 1), 30);
            if (seconds > 0) trail.add("wait " + seconds + "s");
            waited += seconds;
            Thread.sleep(seconds * 1000L);
            continue;
          }
          if (kind.equals("blocked") && polite && next < PROXIES.length) {
            // This exit is done: move the session to another proxy and continue in a new context
            // (its own connections) that keeps the cookies and storage.
            swapProxy(wsUrl, sessionId, PROXIES[next++]);
            String state = context.storageState();
            context.close();
            context = browser.newContext(new Browser.NewContextOptions().setStorageState(state));
            context.route("https://httpbin.org/anything/**", site);
            page = context.newPage();
            swaps++;
            trail.add("swap proxy");
            continue;
          }
          if (kind.equals("error")) {
            Thread.sleep(1000); // a transient failure: try again
            continue;
          }
          break;
        }
        if (exitIp != null) {
          fetched++;
          if (!exits.contains(exitIp)) exits.add(exitIp);
        }
        JsonObject row = new JsonObject();
        row.addProperty("page", i + 1);
        row.addProperty("outcome", outcome);
        row.addProperty("exit_ip", exitIp);
        row.addProperty("trail", String.join(" → ", trail));
        rows.add(row);
      }
      page.navigate("https://httpbin.org/cookies", new Page.NavigateOptions().setTimeout(60000));
      JsonObject cookies = JsonParser.parseString(page.textContent("body")).getAsJsonObject().getAsJsonObject("cookies");
      cookieKept = cookies.has("session") && cookies.get("session").getAsString().equals("alice-42");
    } finally {
      browser.close();
    }
    JsonObject out = new JsonObject();
    out.addProperty("client", polite ? "polite" : "naive");
    out.addProperty("pages_fetched", fetched);
    JsonArray exitArray = new JsonArray();
    exits.forEach(exitArray::add);
    out.add("exits", exitArray);
    out.addProperty("waited_seconds", waited);
    out.addProperty("proxy_swaps", swaps);
    out.addProperty("session_cookie_kept", cookieKept);
    out.add("pages", rows);
    return out;
  }

  public static void main(String[] args) throws Exception {
    JsonObject out = new JsonObject();
    try (Playwright pw = Playwright.create()) {
      // Naive: retries a 429 straight away and has no plan for a block. Polite: waits as told and
      // moves to another exit when one is blocked.
      out.add("naive", crawl(pw, false));
      out.add("polite", crawl(pw, true));
    }
    System.out.println(GSON.toJson(out));
  }
}

// dotnet add package Microsoft.Playwright --version 1.60.0
// env: CDPFLEET_API_KEY, SOCKS_PROXIES (comma-separated sticky socks5:// URLs: one exit IP each)
using System.Net.Http.Json;
using System.Text;
using System.Text.Encodings.Web;
using System.Text.Json;
using System.Text.Json.Nodes;
using System.Text.RegularExpressions;
using Microsoft.Playwright;

var key = Environment.GetEnvironmentVariable("CDPFLEET_API_KEY")!;
var proxies = Environment.GetEnvironmentVariable("SOCKS_PROXIES")!.Split(',');
const string Challenge = "<title>Just a moment...</title><p>Checking your browser before accessing the site.</p>";

using var http = new HttpClient();
http.DefaultRequestHeaders.Add("x-api-key", key);
using var playwright = await Playwright.CreateAsync();

// A stand-in for a rate-limited site, in front of real httpbin requests. Each request really goes
// out through the session's proxy; the limiter decides by the IP httpbin saw (its "origin"):
// 3 pages in a burst, then one page per 5 s (429 + Retry-After: 5), 6 pages per IP in total. An IP
// that comes back before its Retry-After, or is over its quota, gets a challenge page from then on.
Func<IRoute, Task> Limiter()
{
    var ips = new Dictionary<string, IpState>();
    return async (route) =>
    {
        var now = DateTime.UtcNow; // when the request arrived
        IAPIResponse real;
        try { real = await route.FetchAsync(new() { Timeout = 30000 }); }
        catch (PlaywrightException) { await route.FulfillAsync(new() { Status = 502, ContentType = "text/plain", Body = "Bad Gateway" }); return; } // the proxy hiccuped
        if (real.Status != 200) { await route.FulfillAsync(new() { Response = real }); return; } // httpbin itself failed
        var ip = (await real.JsonAsync())!.Value.GetProperty("origin").GetString()!;
        if (!ips.TryGetValue(ip, out var s)) ips[ip] = s = new IpState();
        if (now < s.RetryAt || s.Served >= 6) s.Challenged = true;
        if (s.Challenged) { await route.FulfillAsync(new() { Status = 403, ContentType = "text/html", Body = Challenge }); return; }
        if (s.Served >= 3 && now - s.Last < TimeSpan.FromSeconds(5))
        {
            s.RetryAt = now.AddSeconds(5);
            await route.FulfillAsync(new() { Status = 429, Headers = new Dictionary<string, string> { ["retry-after"] = "5" }, ContentType = "text/plain", Body = "Too Many Requests" });
            return;
        }
        s.Served++;
        s.Last = now;
        await route.FulfillAsync(new() { Response = real });
    };
}

// What the response means: the data, a rate limit (wait), or a block (this IP is done).
string Classify(int status, string text) =>
    status == 429 ? "rate_limited"
    : status is 403 or 503 || Regex.IsMatch(text, "just a moment|captcha|access denied", RegexOptions.IgnoreCase) ? "blocked"
    : status == 200 ? "ok" : "error";

// 503 means the fleet is momentarily busy: wait Retry-After and try again.
async Task<JsonElement> Launch(JsonObject body)
{
    for (var attempt = 1; ; attempt++)
    {
        var res = await http.PostAsync("https://starter.cdpfleet.com/chromium/session", new StringContent(body.ToJsonString(), Encoding.UTF8, "application/json"));
        if (res.IsSuccessStatusCode) return await res.Content.ReadFromJsonAsync<JsonElement>();
        if ((int)res.StatusCode != 503 || attempt == 5) throw new Exception($"launch {(int)res.StatusCode} {await res.Content.ReadAsStringAsync()}");
        await Task.Delay((int)(res.Headers.RetryAfter?.Delta?.TotalMilliseconds ?? 5000));
    }
}

async Task<JsonObject> Crawl(bool polite)
{
    var session = await Launch(new JsonObject { ["proxy"] = proxies[polite ? 1 : 0], ["headless"] = "new", ["record"] = false });
    var wsUrl = session.GetProperty("wsUrl").GetString()!;
    var sessionId = session.GetProperty("sessionId").GetString()!;
    var browser = await playwright.Chromium.ConnectAsync(wsUrl, new() { Headers = new Dictionary<string, string> { ["x-api-key"] = key } });
    var site = Limiter();

    // Live proxy swap on the session's router: new connections use the new proxy.
    async Task SwapProxy(string proxy)
    {
        var r = await http.PostAsJsonAsync($"https://{new Uri(wsUrl).Host}/admin/session/proxy", new { session_id = sessionId, proxy });
        if (!r.IsSuccessStatusCode) throw new Exception($"swap {(int)r.StatusCode} {await r.Content.ReadAsStringAsync()}");
    }
    async Task<(IBrowserContext, IPage)> Open(string? storageState = null)
    {
        var context = await browser.NewContextAsync(new() { StorageState = storageState });
        await context.RouteAsync("https://httpbin.org/anything/**", site);
        return (context, await context.NewPageAsync());
    }

    var rows = new JsonArray();
    var exits = new List<string>();
    int waited = 0, swaps = 0, next = 2, fetched = 0;
    bool cookieKept;
    try
    {
        var (context, page) = await Open();
        // A logged-in state that must survive a proxy swap.
        await page.GotoAsync("https://httpbin.org/cookies/set?session=alice-42", new() { Timeout = 60000 });
        for (var i = 0; i < 8; i++)
        {
            var trail = new List<string>();
            string outcome = "gave up";
            string? exitIp = null;
            for (var attempt = 1; attempt <= 6; attempt++)
            {
                var res = (await page.GotoAsync($"https://httpbin.org/anything/catalogue/page-{i + 1}", new() { Timeout = 60000 }))!;
                var text = await page.TextContentAsync("body") ?? "";
                var kind = Classify(res.Status, text);
                trail.Add(res.Status.ToString());
                if (kind == "ok")
                {
                    outcome = "ok";
                    exitIp = JsonDocument.Parse(text).RootElement.GetProperty("origin").GetString();
                    break;
                }
                if (kind == "rate_limited")
                {
                    if (!polite && attempt == 3) break;
                    // Honour Retry-After; without one, back off exponentially (1, 2, 4… s, capped).
                    var seconds = !polite ? 0 : res.Headers.TryGetValue("retry-after", out var ra) ? int.Parse(ra) : (int)Math.Min(Math.Pow(2, attempt - 1), 30);
                    if (seconds > 0) trail.Add($"wait {seconds}s");
                    waited += seconds;
                    await Task.Delay(seconds * 1000);
                    continue;
                }
                if (kind == "blocked" && polite && next < proxies.Length)
                {
                    // This exit is done: move the session to another proxy and continue in a new context
                    // (its own connections) that keeps the cookies and storage.
                    await SwapProxy(proxies[next++]);
                    var state = await context.StorageStateAsync();
                    await context.CloseAsync();
                    (context, page) = await Open(state);
                    swaps++;
                    trail.Add("swap proxy");
                    continue;
                }
                if (kind == "error")
                {
                    await Task.Delay(1000); // a transient failure: try again
                    continue;
                }
                break;
            }
            if (exitIp != null)
            {
                fetched++;
                if (!exits.Contains(exitIp)) exits.Add(exitIp);
            }
            rows.Add(new JsonObject { ["page"] = i + 1, ["outcome"] = outcome, ["exit_ip"] = exitIp, ["trail"] = string.Join(" → ", trail) });
        }
        await page.GotoAsync("https://httpbin.org/cookies", new() { Timeout = 60000 });
        var cookies = JsonDocument.Parse(await page.TextContentAsync("body") ?? "{}").RootElement.GetProperty("cookies");
        cookieKept = cookies.TryGetProperty("session", out var c) && c.GetString() == "alice-42";
    }
    finally
    {
        await browser.CloseAsync();
    }
    return new JsonObject
    {
        ["client"] = polite ? "polite" : "naive",
        ["pages_fetched"] = fetched,
        ["exits"] = new JsonArray(exits.Select((e) => (JsonNode)JsonValue.Create(e)!).ToArray()),
        ["waited_seconds"] = waited,
        ["proxy_swaps"] = swaps,
        ["session_cookie_kept"] = cookieKept,
        ["pages"] = rows,
    };
}

// Naive: retries a 429 straight away and has no plan for a block. Polite: waits as told and moves
// to another exit when one is blocked.
var naive = await Crawl(false);
var polite = await Crawl(true);
Console.WriteLine(new JsonObject { ["naive"] = naive, ["polite"] = polite }.ToJsonString(new JsonSerializerOptions { WriteIndented = true, Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping }));

class IpState
{
    public int Served;
    public DateTime Last;
    public DateTime RetryAt;
    public bool Challenged;
}

// go get github.com/playwright-community/[email protected]
// Driver: build playwright-core 1.60.0 from npm and set PLAYWRIGHT_DRIVER_PATH (see /docs/quickstart).
// env: CDPFLEET_API_KEY, SOCKS_PROXIES (comma-separated sticky socks5:// URLs: one exit IP each)
package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io"
	"log"
	"math"
	"net/http"
	"net/url"
	"os"
	"regexp"
	"strconv"
	"strings"
	"time"

	"github.com/playwright-community/playwright-go"
)

var (
	key     = os.Getenv("CDPFLEET_API_KEY")
	proxies = strings.Split(os.Getenv("SOCKS_PROXIES"), ",")
)

const challenge = "<title>Just a moment...</title><p>Checking your browser before accessing the site.</p>"

type ipState struct {
	served     int
	last       time.Time
	retryAt    time.Time
	challenged bool
}

// A stand-in for a rate-limited site, in front of real httpbin requests. Each request really goes
// out through the session's proxy; the limiter decides by the IP httpbin saw (its "origin"):
// 3 pages in a burst, then one page per 5 s (429 + Retry-After: 5), 6 pages per IP in total. An IP
// that comes back before its Retry-After, or is over its quota, gets a challenge page from then on.
func limiter() func(playwright.Route) {
	ips := map[string]*ipState{}
	return func(route playwright.Route) {
		now := time.Now() // when the request arrived
		real, err := route.Fetch(playwright.RouteFetchOptions{Timeout: playwright.Float(30000)})
		if err != nil { // the proxy hiccuped
			route.Fulfill(playwright.RouteFulfillOptions{Status: playwright.Int(502), ContentType: playwright.String("text/plain"), Body: "Bad Gateway"})
			return
		}
		if real.Status() != 200 { // httpbin itself failed
			route.Fulfill(playwright.RouteFulfillOptions{Response: real})
			return
		}
		var body struct {
			Origin string `json:"origin"`
		}
		real.JSON(&body)
		s, ok := ips[body.Origin]
		if !ok {
			s = &ipState{}
			ips[body.Origin] = s
		}
		if now.Before(s.retryAt) || s.served >= 6 {
			s.challenged = true
		}
		if s.challenged {
			route.Fulfill(playwright.RouteFulfillOptions{Status: playwright.Int(403), ContentType: playwright.String("text/html"), Body: challenge})
			return
		}
		if s.served >= 3 && now.Sub(s.last) < 5*time.Second {
			s.retryAt = now.Add(5 * time.Second)
			route.Fulfill(playwright.RouteFulfillOptions{Status: playwright.Int(429), Headers: map[string]string{"retry-after": "5"}, ContentType: playwright.String("text/plain"), Body: "Too Many Requests"})
			return
		}
		s.served++
		s.last = now
		route.Fulfill(playwright.RouteFulfillOptions{Response: real})
	}
}

var challengeText = regexp.MustCompile(`(?i)just a moment|captcha|access denied`)

// What the response means: the data, a rate limit (wait), or a block (this IP is done).
func classify(status int, text string) string {
	if status == 429 {
		return "rate_limited"
	}
	if status == 403 || status == 503 || challengeText.MatchString(text) {
		return "blocked"
	}
	if status == 200 {
		return "ok"
	}
	return "error"
}

// 503 means the fleet is momentarily busy: wait Retry-After and try again.
func launch(opts map[string]any) (map[string]any, error) {
	body, _ := json.Marshal(opts)
	for attempt := 1; ; attempt++ {
		req, _ := http.NewRequest("POST", "https://starter.cdpfleet.com/chromium/session", bytes.NewReader(body))
		req.Header.Set("x-api-key", key)
		req.Header.Set("content-type", "application/json")
		res, err := http.DefaultClient.Do(req)
		if err != nil {
			return nil, err
		}
		b, _ := io.ReadAll(res.Body)
		res.Body.Close()
		if res.StatusCode == 200 {
			var session map[string]any
			return session, json.Unmarshal(b, &session)
		}
		if res.StatusCode != 503 || attempt == 5 {
			return nil, fmt.Errorf("launch %d %s", res.StatusCode, strings.TrimSpace(string(b)))
		}
		wait, err := strconv.Atoi(res.Header.Get("retry-after"))
		if err != nil {
			wait = 5
		}
		time.Sleep(time.Duration(wait) * time.Second)
	}
}

// Live proxy swap on the session's router: new connections use the new proxy.
func swapProxy(wsURL, sessionID, proxy string) error {
	u, _ := url.Parse(wsURL)
	body, _ := json.Marshal(map[string]string{"session_id": sessionID, "proxy": proxy})
	req, _ := http.NewRequest("POST", "https://"+u.Host+"/admin/session/proxy", bytes.NewReader(body))
	req.Header.Set("x-api-key", key)
	req.Header.Set("content-type", "application/json")
	res, err := http.DefaultClient.Do(req)
	if err != nil {
		return err
	}
	defer res.Body.Close()
	if res.StatusCode != 200 {
		b, _ := io.ReadAll(res.Body)
		return fmt.Errorf("swap %d %s", res.StatusCode, b)
	}
	return nil
}

type pageRow struct {
	Page    int     `json:"page"`
	Outcome string  `json:"outcome"`
	ExitIP  *string `json:"exit_ip"`
	Trail   string  `json:"trail"`
}

type crawlResult struct {
	Client            string    `json:"client"`
	PagesFetched      int       `json:"pages_fetched"`
	Exits             []string  `json:"exits"`
	WaitedSeconds     int       `json:"waited_seconds"`
	ProxySwaps        int       `json:"proxy_swaps"`
	SessionCookieKept bool      `json:"session_cookie_kept"`
	Pages             []pageRow `json:"pages"`
}

func crawl(pw *playwright.Playwright, polite bool) (crawlResult, error) {
	out := crawlResult{Client: "naive", Exits: []string{}, Pages: []pageRow{}}
	first := proxies[0]
	if polite {
		out.Client, first = "polite", proxies[1]
	}
	session, err := launch(map[string]any{"proxy": first, "headless": "new", "record": false})
	if err != nil {
		return out, err
	}
	wsURL, sessionID := session["wsUrl"].(string), session["sessionId"].(string)
	browser, err := pw.Chromium.Connect(wsURL, playwright.BrowserTypeConnectOptions{Headers: map[string]string{"x-api-key": key}})
	if err != nil {
		return out, err
	}
	defer browser.Close()
	site := limiter()
	open := func(state *playwright.StorageState) (playwright.BrowserContext, playwright.Page, error) {
		opts := playwright.BrowserNewContextOptions{}
		if state != nil {
			opts.StorageState = state.ToOptionalStorageState()
		}
		context, err := browser.NewContext(opts)
		if err != nil {
			return nil, nil, err
		}
		if err := context.Route("https://httpbin.org/anything/**", site); err != nil {
			return nil, nil, err
		}
		page, err := context.NewPage()
		return context, page, err
	}
	context, page, err := open(nil)
	if err != nil {
		return out, err
	}
	// A logged-in state that must survive a proxy swap.
	if _, err := page.Goto("https://httpbin.org/cookies/set?session=alice-42", playwright.PageGotoOptions{Timeout: playwright.Float(60000)}); err != nil {
		return out, err
	}
	next := 2
	seen := map[string]bool{}
	for i := 0; i < 8; i++ {
		pageURL := fmt.Sprintf("https://httpbin.org/anything/catalogue/page-%d", i+1)
		trail := []string{}
		row := pageRow{Page: i + 1, Outcome: "gave up"}
		for attempt := 1; attempt <= 6; attempt++ {
			res, err := page.Goto(pageURL, playwright.PageGotoOptions{Timeout: playwright.Float(60000)})
			if err != nil {
				return out, err
			}
			text, err := page.TextContent("body")
			if err != nil {
				return out, err
			}
			kind := classify(res.Status(), text)
			trail = append(trail, strconv.Itoa(res.Status()))
			if kind == "ok" {
				var body struct {
					Origin string `json:"origin"`
				}
				json.Unmarshal([]byte(text), &body)
				row.Outcome, row.ExitIP = "ok", &body.Origin
				break
			}
			if kind == "rate_limited" {
				if !polite && attempt == 3 {
					break
				}
				seconds := 0
				if polite {
					// Honour Retry-After; without one, back off exponentially (1, 2, 4… s, capped).
					if seconds, err = strconv.Atoi(res.Headers()["retry-after"]); err != nil {
						seconds = int(math.Min(math.Pow(2, float64(attempt-1)), 30))
					}
					trail = append(trail, fmt.Sprintf("wait %ds", seconds))
				}
				out.WaitedSeconds += seconds
				time.Sleep(time.Duration(seconds) * time.Second)
				continue
			}
			if kind == "blocked" && polite && next < len(proxies) {
				// This exit is done: move the session to another proxy and continue in a new context
				// (its own connections) that keeps the cookies and storage.
				if err := swapProxy(wsURL, sessionID, proxies[next]); err != nil {
					return out, err
				}
				next++
				state, err := context.StorageState()
				if err != nil {
					return out, err
				}
				context.Close()
				if context, page, err = open(state); err != nil {
					return out, err
				}
				out.ProxySwaps++
				trail = append(trail, "swap proxy")
				continue
			}
			if kind == "error" {
				time.Sleep(time.Second) // a transient failure: try again
				continue
			}
			break
		}
		row.Trail = strings.Join(trail, " → ")
		out.Pages = append(out.Pages, row)
		if row.ExitIP != nil {
			out.PagesFetched++
			if !seen[*row.ExitIP] {
				seen[*row.ExitIP] = true
				out.Exits = append(out.Exits, *row.ExitIP)
			}
		}
	}
	if _, err := page.Goto("https://httpbin.org/cookies", playwright.PageGotoOptions{Timeout: playwright.Float(60000)}); err != nil {
		return out, err
	}
	text, err := page.TextContent("body")
	if err != nil {
		return out, err
	}
	var cookies struct {
		Cookies map[string]string `json:"cookies"`
	}
	json.Unmarshal([]byte(text), &cookies)
	out.SessionCookieKept = cookies.Cookies["session"] == "alice-42"
	return out, nil
}

func run() error {
	pw, err := playwright.Run(&playwright.RunOptions{SkipInstallBrowsers: true})
	if err != nil {
		return err
	}
	defer pw.Stop()
	// Naive: retries a 429 straight away and has no plan for a block. Polite: waits as told and
	// moves to another exit when one is blocked.
	naive, err := crawl(pw, false)
	if err != nil {
		return fmt.Errorf("naive: %w", err)
	}
	polite, err := crawl(pw, true)
	if err != nil {
		return fmt.Errorf("polite: %w", err)
	}
	enc := json.NewEncoder(os.Stdout)
	enc.SetEscapeHTML(false)
	enc.SetIndent("", "  ")
	return enc.Encode(map[string]crawlResult{"naive": naive, "polite": polite})
}

func main() {
	if err := run(); err != nil {
		log.Fatal(err)
	}
}

What we got

Crawler	Page	What happened	Result	Served from
naive	1	200	ok	203.0.113.1
naive	2	200	ok	203.0.113.1
naive	3	200	ok	203.0.113.1
naive	4	429 → 403	gave up	—
naive	5	403	gave up	—
naive	6	403	gave up	—
naive	7	403	gave up	—
naive	8	403	gave up	—
polite	1	200	ok	203.0.113.2
polite	2	200	ok	203.0.113.2
polite	3	200	ok	203.0.113.2
polite	4	429 → wait 5s → 200	ok	203.0.113.2
polite	5	429 → wait 5s → 200	ok	203.0.113.2
polite	6	502 → 429 → wait 5s → 200	ok	203.0.113.2
polite	7	403 → swap proxy → 200	ok	203.0.113.3
polite	8	200	ok	203.0.113.3

From the Node.js run on 2026-10-16. IP addresses are replaced with placeholders (203.0.113.x); equal addresses stay equal. The other languages produced the same findings.

Raw output (Node.js)

{
  "naive": {
    "client": "naive",
    "pages_fetched": 3,
    "exits": [
      "203.0.113.1"
    ],
    "waited_seconds": 0,
    "proxy_swaps": 0,
    "session_cookie_kept": true,
    "pages": [
      {
        "page": 1,
        "outcome": "ok",
        "exit_ip": "203.0.113.1",
        "trail": "200"
      },
      {
        "page": 2,
        "outcome": "ok",
        "exit_ip": "203.0.113.1",
        "trail": "200"
      },
      {
        "page": 3,
        "outcome": "ok",
        "exit_ip": "203.0.113.1",
        "trail": "200"
      },
      {
        "page": 4,
        "outcome": "gave up",
        "exit_ip": null,
        "trail": "429 → 403"
      },
      {
        "page": 5,
        "outcome": "gave up",
        "exit_ip": null,
        "trail": "403"
      },
      {
        "page": 6,
        "outcome": "gave up",
        "exit_ip": null,
        "trail": "403"
      },
      {
        "page": 7,
        "outcome": "gave up",
        "exit_ip": null,
        "trail": "403"
      },
      {
        "page": 8,
        "outcome": "gave up",
        "exit_ip": null,
        "trail": "403"
      }
    ]
  },
  "polite": {
    "client": "polite",
    "pages_fetched": 8,
    "exits": [
      "203.0.113.2",
      "203.0.113.3"
    ],
    "waited_seconds": 15,
    "proxy_swaps": 1,
    "session_cookie_kept": true,
    "pages": [
      {
        "page": 1,
        "outcome": "ok",
        "exit_ip": "203.0.113.2",
        "trail": "200"
      },
      {
        "page": 2,
        "outcome": "ok",
        "exit_ip": "203.0.113.2",
        "trail": "200"
      },
      {
        "page": 3,
        "outcome": "ok",
        "exit_ip": "203.0.113.2",
        "trail": "200"
      },
      {
        "page": 4,
        "outcome": "ok",
        "exit_ip": "203.0.113.2",
        "trail": "429 → wait 5s → 200"
      },
      {
        "page": 5,
        "outcome": "ok",
        "exit_ip": "203.0.113.2",
        "trail": "429 → wait 5s → 200"
      },
      {
        "page": 6,
        "outcome": "ok",
        "exit_ip": "203.0.113.2",
        "trail": "502 → 429 → wait 5s → 200"
      },
      {
        "page": 7,
        "outcome": "ok",
        "exit_ip": "203.0.113.3",
        "trail": "403 → swap proxy → 200"
      },
      {
        "page": 8,
        "outcome": "ok",
        "exit_ip": "203.0.113.3",
        "trail": "200"
      }
    ]
  }
}

Takeaways

Retrying a 429 straight away turns a slowdown into a block. The naive crawler got its first burst of pages, then a 429; its immediate retry arrived before Retry-After and the IP was challenged for the rest of the run: 3 or 4 of 8 pages.
Waiting as told costs seconds, not pages. The polite crawler waited Retry-After after every 429, a few seconds each, and kept getting pages from the same exit until its quota ran out.
When an exit is burned, move — don't wait. At its quota the first IP was challenged. The crawler swapped the session to another proxy and opened a new context, and the blocked page came back 200 from a different exit IP: 8 of 8 pages, two exits, one proxy swap.
The login survived the move. The session cookie set before the crawl was still there after the swap, because the new context started from the old storageState. Same browser and same session, with no relaunch.
Tell the signals apart: a 429 asks you to slow down on this IP, while a challenge means this IP is done for now. Treating every non-200 as "retry" gets the first wrong, and treating every one as "swap" spends exits you didn't need to.