[Examples & benchmarks](https://cdpfleet.com/docs/cases) / #53 · 2026-10-16 · Example · Medium · for [web scraping](https://cdpfleet.com/web-scraping), [monitoring](https://cdpfleet.com/monitoring)

# Surviving a soft block: honour Retry-After, then move to another exit without losing the login

Two crawlers against the same rate-limited site. The one that retries a 429 straight away is challenged and gets 3 of 8 pages. The one that waits as told and moves the session to another proxy when its exit is blocked gets all 8 — same browser, same cookies.

[Chromium](https://cdpfleet.com/docs/browsers/chromium)

Run on production on 2026-10-16: ✓ Node.js ✓ Python ✓ Java ✓ C# ✓ Go

## The problem

Sites rarely block outright on the first request. They slow you down first — a `429 Too Many Requests` with a `Retry-After` — and only challenge or ban an address that ignores it. A crawler needs a policy: tell a rate limit from a block, wait when asked, and when one exit is burned, continue from another without starting over (and without logging in again). What does that look like on a remote browser, and what does ignoring the signal cost?

## What we used, and why

| What | Why |
| --- | --- |
| A stand-in rate limiter (`context.route`) | Real sites don't rate-limit on cue, so the site is simulated in front of real httpbin requests. Each request really goes out through the session's proxy (`route.fetch`), and the limiter decides by the IP httpbin saw: 3 pages in a burst, then one per 5 s (429 with `Retry-After`), at most 6 per IP. An IP that comes back early or exceeds the quota gets a challenge page from then on. |
| A response classifier | `429` is a rate limit (wait). `403`/`503` or challenge text means this IP is blocked (move). `200` is data. Anything else (a `502` from a flaky upstream) is transient: retry after a second. Those show up in the trail now and then. |
| `Retry-After`, with exponential backoff as a fallback | Wait exactly as long as the site asks; if it doesn't say, back off 1, 2, 4… seconds. |
| Live proxy swap (`POST /admin/session/proxy` on the session's router) | Moves the running session to another proxy — no relaunch. |
| A new context with `storageState` | New connections, so every request uses the new proxy ([live proxy swap](https://cdpfleet.com/docs/cases/live-proxy-swap)), while the cookies come along. |
| Sticky SOCKS5 proxies | One exit IP per proxy, so "this IP is blocked" means something. |

## How it works

1.  Naive crawler: one session, sets a session cookie, fetches 8 pages; on a 429 it retries straight away (up to 3 times), on a block it skips the page.
2.  Polite crawler: the same, but on a 429 it waits `Retry-After` seconds, and on a block it swaps the session's proxy, opens a new context with the old `storageState` and retries.
3.  Both read the cookie back at the end and report, per page, the status trail and the exit IP that served it.

## The code

The same program in five languages (also [on GitHub](https://github.com/cdpfleet/cdpfleet/tree/main/cases/53-soft-block-backoff), with the raw output). Set these environment variables first:

-   `CDPFLEET_API_KEY` — your API key (dashboard → API keys)
-   `SOCKS_PROXIES` — comma-separated `socks5://user:pass@host:port` proxies

 

```node
// npm install playwright@1.60.0
// env: CDPFLEET_API_KEY, SOCKS_PROXIES (comma-separated sticky socks5:// URLs: one exit IP each)
import { chromium } from 'playwright';

const KEY = process.env.CDPFLEET_API_KEY;
const PROXIES = process.env.SOCKS_PROXIES.split(',');
const PAGES = Array.from({ length: 8 }, (_, i) => `https://httpbin.org/anything/catalogue/page-${i + 1}`);

// A stand-in for a rate-limited site, in front of real httpbin requests. Each request really goes
// out through the session's proxy; the limiter decides by the IP httpbin saw (its "origin"):
// 3 pages in a burst, then one page per 5 s (429 + Retry-After: 5), 6 pages per IP in total. An IP
// that comes back before its Retry-After, or is over its quota, gets a challenge page from then on.
function limiter() {
  const ips = new Map();
  return async (route) => {
    const now = Date.now(); // when the request arrived
    const real = await route.fetch({ timeout: 30000 }).catch(() => null);
    if (!real) return route.fulfill({ status: 502, contentType: 'text/plain', body: 'Bad Gateway' }); // the proxy hiccuped
    if (real.status() !== 200) return route.fulfill({ response: real }); // httpbin itself failed
    const ip = (await real.json()).origin;
    const s = ips.get(ip) ?? { served: 0, last: 0, retryAt: 0, challenged: false };
    ips.set(ip, s);
    if (now < s.retryAt || s.served >= 6) s.challenged = true;
    if (s.challenged) {
      return route.fulfill({ status: 403, contentType: 'text/html', body: '<title>Just a moment...</title><p>Checking your browser before accessing the site.</p>' });
    }
    if (s.served >= 3 && now - s.last < 5000) {
      s.retryAt = now + 5000;
      return route.fulfill({ status: 429, headers: { 'retry-after': '5' }, contentType: 'text/plain', body: 'Too Many Requests' });
    }
    s.served++;
    s.last = now;
    return route.fulfill({ response: real });
  };
}

// What the response means: the data, a rate limit (wait), or a block (this IP is done).
function classify(status, text) {
  if (status === 429) return 'rate_limited';
  if (status === 403 || status === 503 || /just a moment|captcha|access denied/i.test(text)) return 'blocked';
  return status === 200 ? 'ok' : 'error';
}

// 503 means the fleet is momentarily busy: wait Retry-After and try again.
async function launch(body) {
  for (let attempt = 1; ; attempt++) {
    const res = await fetch('https://starter.cdpfleet.com/chromium/session', {
      method: 'POST',
      headers: { 'x-api-key': KEY, 'content-type': 'application/json' },
      body: JSON.stringify(body),
    });
    if (res.ok) return res.json();
    if (res.status !== 503 || attempt === 5) throw new Error(`launch ${res.status} ${await res.text()}`);
    await new Promise((r) => setTimeout(r, (Number(res.headers.get('retry-after')) || 5) * 1000));
  }
}

async function crawl(polite) {
  const { sessionId, wsUrl } = await launch({ proxy: PROXIES[polite ? 1 : 0], headless: 'new', record: false });
  const browser = await chromium.connect(wsUrl, { headers: { 'x-api-key': KEY } });
  const site = limiter();
  // Live proxy swap on the session's router: new connections use the new proxy.
  const swapProxy = async (proxy) => {
    const r = await fetch(`https://${new URL(wsUrl).host}/admin/session/proxy`, {
      method: 'POST',
      headers: { 'x-api-key': KEY, 'content-type': 'application/json' },
      body: JSON.stringify({ session_id: sessionId, proxy }),
    });
    if (!r.ok) throw new Error(`swap ${r.status} ${await r.text()}`);
  };
  const open = async (storageState) => {
    const context = await browser.newContext({ storageState });
    await context.route('https://httpbin.org/anything/**', site);
    return { context, page: await context.newPage() };
  };
  const rows = [];
  let waited = 0;
  let swaps = 0;
  let next = 2;
  let cookieKept = false;
  try {
    let { context, page } = await open();
    // A logged-in state that must survive a proxy swap.
    await page.goto('https://httpbin.org/cookies/set?session=alice-42', { timeout: 60000 });
    for (const [i, url] of PAGES.entries()) {
      const trail = [];
      let row = { page: i + 1, outcome: 'gave up', exit_ip: null };
      for (let attempt = 1; attempt <= 6; attempt++) {
        const res = await page.goto(url, { timeout: 60000 });
        const text = await page.textContent('body');
        const kind = classify(res.status(), text);
        trail.push(res.status());
        if (kind === 'ok') {
          row = { page: i + 1, outcome: 'ok', exit_ip: JSON.parse(text).origin };
          break;
        }
        if (kind === 'rate_limited') {
          // Honour Retry-After; without one, back off exponentially (1, 2, 4… s, capped).
          if (!polite && attempt === 3) break;
          const seconds = polite ? Number(res.headers()['retry-after'] ?? Math.min(2 ** (attempt - 1), 30)) : 0;
          if (seconds) trail.push(`wait ${seconds}s`);
          waited += seconds;
          await new Promise((r) => setTimeout(r, seconds * 1000));
          continue;
        }
        if (kind === 'blocked' && polite && next < PROXIES.length) {
          // This exit is done: move the session to another proxy and continue in a new context
          // (its own connections) that keeps the cookies and storage.
          await swapProxy(PROXIES[next++]);
          const state = await context.storageState();
          await context.close();
          ({ context, page } = await open(state));
          swaps++;
          trail.push('swap proxy');
          continue;
        }
        if (kind === 'error') {
          await new Promise((r) => setTimeout(r, 1000)); // a transient failure: try again
          continue;
        }
        break;
      }
      rows.push({ ...row, trail: trail.join(' → ') });
    }
    await page.goto('https://httpbin.org/cookies', { timeout: 60000 });
    cookieKept = JSON.parse(await page.textContent('body')).cookies.session === 'alice-42';
  } finally {
    await browser.close();
  }
  return {
    client: polite ? 'polite' : 'naive',
    pages_fetched: rows.filter((r) => r.outcome === 'ok').length,
    exits: [...new Set(rows.map((r) => r.exit_ip).filter(Boolean))],
    waited_seconds: waited,
    proxy_swaps: swaps,
    session_cookie_kept: cookieKept,
    pages: rows,
  };
}

// Naive: retries a 429 straight away and has no plan for a block. Polite: waits as told and moves
// to another exit when one is blocked.
const naive = await crawl(false);
const polite = await crawl(true);
console.log(JSON.stringify({ naive, polite }, null, 2));
```

```python
# pip install playwright==1.60.0 requests
# env: CDPFLEET_API_KEY, SOCKS_PROXIES (comma-separated sticky socks5:// URLs: one exit IP each)
import json
import os
import re
import time
from urllib.parse import urlparse

import requests
from playwright.sync_api import sync_playwright

KEY = os.environ["CDPFLEET_API_KEY"]
PROXIES = os.environ["SOCKS_PROXIES"].split(",")
PAGES = [f"https://httpbin.org/anything/catalogue/page-{i + 1}" for i in range(8)]
CHALLENGE = "<title>Just a moment...</title><p>Checking your browser before accessing the site.</p>"

# A stand-in for a rate-limited site, in front of real httpbin requests. Each request really goes
# out through the session's proxy; the limiter decides by the IP httpbin saw (its "origin"):
# 3 pages in a burst, then one page per 5 s (429 + Retry-After: 5), 6 pages per IP in total. An IP
# that comes back before its Retry-After, or is over its quota, gets a challenge page from then on.
def limiter():
    ips = {}

    def handle(route):
        now = time.time() * 1000  # when the request arrived
        try:
            real = route.fetch(timeout=30000)
        except Exception:
            return route.fulfill(status=502, content_type="text/plain", body="Bad Gateway")  # the proxy hiccuped
        if real.status != 200:
            return route.fulfill(response=real)  # httpbin itself failed
        s = ips.setdefault(real.json()["origin"], {"served": 0, "last": 0, "retry_at": 0, "challenged": False})
        if now < s["retry_at"] or s["served"] >= 6:
            s["challenged"] = True
        if s["challenged"]:
            return route.fulfill(status=403, content_type="text/html", body=CHALLENGE)
        if s["served"] >= 3 and now - s["last"] < 5000:
            s["retry_at"] = now + 5000
            return route.fulfill(status=429, headers={"retry-after": "5"}, content_type="text/plain", body="Too Many Requests")
        s["served"] += 1
        s["last"] = now
        return route.fulfill(response=real)

    return handle

# What the response means: the data, a rate limit (wait), or a block (this IP is done).
def classify(status, text):
    if status == 429:
        return "rate_limited"
    if status in (403, 503) or re.search(r"just a moment|captcha|access denied", text, re.I):
        return "blocked"
    return "ok" if status == 200 else "error"

# 503 means the fleet is momentarily busy: wait Retry-After and try again.
def launch(body):
    for attempt in range(1, 6):
        r = requests.post("https://starter.cdpfleet.com/chromium/session", headers={"x-api-key": KEY}, json=body, timeout=120)
        if r.ok:
            return r.json()
        if r.status_code != 503 or attempt == 5:
            raise RuntimeError(f"launch {r.status_code} {r.text}")
        time.sleep(int(r.headers.get("retry-after") or 5))

def crawl(p, polite):
    session = launch({"proxy": PROXIES[1 if polite else 0], "headless": "new", "record": False})
    browser = p.chromium.connect(session["wsUrl"], headers={"x-api-key": KEY})
    site = limiter()

    # Live proxy swap on the session's router: new connections use the new proxy.
    def swap_proxy(proxy):
        r = requests.post(f"https://{urlparse(session['wsUrl']).netloc}/admin/session/proxy", headers={"x-api-key": KEY},
                          json={"session_id": session["sessionId"], "proxy": proxy}, timeout=30)
        if not r.ok:
            raise RuntimeError(f"swap {r.status_code} {r.text}")

    def open_context(storage_state=None):
        context = browser.new_context(storage_state=storage_state)
        context.route("https://httpbin.org/anything/**", site)
        return context, context.new_page()

    rows, waited, swaps, nxt, cookie_kept = [], 0, 0, 2, False
    try:
        context, page = open_context()
        # A logged-in state that must survive a proxy swap.
        page.goto("https://httpbin.org/cookies/set?session=alice-42", timeout=60000)
        for i, url in enumerate(PAGES):
            trail = []
            row = {"page": i + 1, "outcome": "gave up", "exit_ip": None}
            for attempt in range(1, 7):
                res = page.goto(url, timeout=60000)
                text = page.text_content("body")
                kind = classify(res.status, text)
                trail.append(str(res.status))
                if kind == "ok":
                    row = {"page": i + 1, "outcome": "ok", "exit_ip": json.loads(text)["origin"]}
                    break
                if kind == "rate_limited":
                    if not polite and attempt == 3:
                        break
                    # Honour Retry-After; without one, back off exponentially (1, 2, 4… s, capped).
                    seconds = int(res.headers.get("retry-after") or min(2 ** (attempt - 1), 30)) if polite else 0
                    if seconds:
                        trail.append(f"wait {seconds}s")
                    waited += seconds
                    time.sleep(seconds)
                    continue
                if kind == "blocked" and polite and nxt < len(PROXIES):
                    # This exit is done: move the session to another proxy and continue in a new context
                    # (its own connections) that keeps the cookies and storage.
                    swap_proxy(PROXIES[nxt])
                    nxt += 1
                    state = context.storage_state()
                    context.close()
                    context, page = open_context(state)
                    swaps += 1
                    trail.append("swap proxy")
                    continue
                if kind == "error":
                    time.sleep(1)  # a transient failure: try again
                    continue
                break
            rows.append({**row, "trail": " → ".join(trail)})
        page.goto("https://httpbin.org/cookies", timeout=60000)
        cookie_kept = json.loads(page.text_content("body"))["cookies"].get("session") == "alice-42"
    finally:
        browser.close()
    return {
        "client": "polite" if polite else "naive",
        "pages_fetched": sum(1 for r in rows if r["outcome"] == "ok"),
        "exits": list(dict.fromkeys(r["exit_ip"] for r in rows if r["exit_ip"])),
        "waited_seconds": waited,
        "proxy_swaps": swaps,
        "session_cookie_kept": cookie_kept,
        "pages": rows,
    }

# Naive: retries a 429 straight away and has no plan for a block. Polite: waits as told and moves
# to another exit when one is blocked.
with sync_playwright() as p:
    naive = crawl(p, False)
    polite = crawl(p, True)
print(json.dumps({"naive": naive, "polite": polite}, indent=2, ensure_ascii=False))
```

```java
// Maven: com.microsoft.playwright:playwright:1.60.0, com.google.code.gson:gson:2.11.0
// Run with PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1.
// env: CDPFLEET_API_KEY, SOCKS_PROXIES (comma-separated sticky socks5:// URLs: one exit IP each)
import com.google.gson.*;
import com.microsoft.playwright.*;
import java.net.URI;
import java.net.http.*;
import java.util.*;
import java.util.function.Consumer;
import java.util.regex.Pattern;

public class Main {
  static final String KEY = System.getenv("CDPFLEET_API_KEY");
  static final String[] PROXIES = System.getenv("SOCKS_PROXIES").split(",");
  static final Gson GSON = new GsonBuilder().setPrettyPrinting().disableHtmlEscaping().serializeNulls().create();
  static final HttpClient HTTP = HttpClient.newHttpClient();
  static final String CHALLENGE = "<title>Just a moment...</title><p>Checking your browser before accessing the site.</p>";
  static final Pattern CHALLENGE_TEXT = Pattern.compile("just a moment|captcha|access denied", Pattern.CASE_INSENSITIVE);

  static class IpState { int served; long last; long retryAt; boolean challenged; }

  // A stand-in for a rate-limited site, in front of real httpbin requests. Each request really goes
  // out through the session's proxy; the limiter decides by the IP httpbin saw (its "origin"):
  // 3 pages in a burst, then one page per 5 s (429 + Retry-After: 5), 6 pages per IP in total. An IP
  // that comes back before its Retry-After, or is over its quota, gets a challenge page from then on.
  static Consumer<Route> limiter() {
    Map<String, IpState> ips = new HashMap<>();
    return (route) -> {
      long now = System.currentTimeMillis(); // when the request arrived
      APIResponse real;
      try {
        real = route.fetch(new Route.FetchOptions().setTimeout(30000));
      } catch (PlaywrightException e) { // the proxy hiccuped
        route.fulfill(new Route.FulfillOptions().setStatus(502).setContentType("text/plain").setBody("Bad Gateway"));
        return;
      }
      if (real.status() != 200) { // httpbin itself failed
        route.fulfill(new Route.FulfillOptions().setResponse(real));
        return;
      }
      String ip = JsonParser.parseString(real.text()).getAsJsonObject().get("origin").getAsString();
      IpState s = ips.computeIfAbsent(ip, (k) -> new IpState());
      if (now < s.retryAt || s.served >= 6) s.challenged = true;
      if (s.challenged) {
        route.fulfill(new Route.FulfillOptions().setStatus(403).setContentType("text/html").setBody(CHALLENGE));
        return;
      }
      if (s.served >= 3 && now - s.last < 5000) {
        s.retryAt = now + 5000;
        route.fulfill(new Route.FulfillOptions().setStatus(429).setHeaders(Map.of("retry-after", "5")).setContentType("text/plain").setBody("Too Many Requests"));
        return;
      }
      s.served++;
      s.last = now;
      route.fulfill(new Route.FulfillOptions().setResponse(real));
    };
  }

  // What the response means: the data, a rate limit (wait), or a block (this IP is done).
  static String classify(int status, String text) {
    if (status == 429) return "rate_limited";
    if (status == 403 || status == 503 || CHALLENGE_TEXT.matcher(text).find()) return "blocked";
    return status == 200 ? "ok" : "error";
  }

  // 503 means the fleet is momentarily busy: wait Retry-After and try again.
  static JsonObject launch(JsonObject body) throws Exception {
    for (int attempt = 1; ; attempt++) {
      HttpResponse<String> res = HTTP.send(HttpRequest.newBuilder(URI.create("https://starter.cdpfleet.com/chromium/session"))
          .header("x-api-key", KEY).header("content-type", "application/json")
          .POST(HttpRequest.BodyPublishers.ofString(body.toString())).build(), HttpResponse.BodyHandlers.ofString());
      if (res.statusCode() == 200) return JsonParser.parseString(res.body()).getAsJsonObject();
      if (res.statusCode() != 503 || attempt == 5) throw new RuntimeException("launch " + res.statusCode() + " " + res.body());
      Thread.sleep(Long.parseLong(res.headers().firstValue("retry-after").orElse("5")) * 1000);
    }
  }

  // Live proxy swap on the session's router: new connections use the new proxy.
  static void swapProxy(String wsUrl, String sessionId, String proxy) throws Exception {
    JsonObject body = new JsonObject();
    body.addProperty("session_id", sessionId);
    body.addProperty("proxy", proxy);
    HttpResponse<String> res = HTTP.send(HttpRequest.newBuilder(URI.create("https://" + URI.create(wsUrl).getHost() + "/admin/session/proxy"))
        .header("x-api-key", KEY).header("content-type", "application/json")
        .POST(HttpRequest.BodyPublishers.ofString(body.toString())).build(), HttpResponse.BodyHandlers.ofString());
    if (res.statusCode() != 200) throw new RuntimeException("swap " + res.statusCode() + " " + res.body());
  }

  static JsonObject crawl(Playwright pw, boolean polite) throws Exception {
    JsonObject launchBody = new JsonObject();
    launchBody.addProperty("proxy", PROXIES[polite ? 1 : 0]);
    launchBody.addProperty("headless", "new");
    launchBody.addProperty("record", false);
    JsonObject session = launch(launchBody);
    String wsUrl = session.get("wsUrl").getAsString();
    String sessionId = session.get("sessionId").getAsString();
    Browser browser = pw.chromium().connect(wsUrl, new BrowserType.ConnectOptions().setHeaders(Map.of("x-api-key", KEY)));
    Consumer<Route> site = limiter();
    JsonArray rows = new JsonArray();
    List<String> exits = new ArrayList<>();
    int waited = 0, swaps = 0, next = 2, fetched = 0;
    boolean cookieKept;
    try {
      BrowserContext context = browser.newContext();
      context.route("https://httpbin.org/anything/**", site);
      Page page = context.newPage();
      // A logged-in state that must survive a proxy swap.
      page.navigate("https://httpbin.org/cookies/set?session=alice-42", new Page.NavigateOptions().setTimeout(60000));
      for (int i = 0; i < 8; i++) {
        List<String> trail = new ArrayList<>();
        String outcome = "gave up";
        String exitIp = null;
        for (int attempt = 1; attempt <= 6; attempt++) {
          Response res = page.navigate("https://httpbin.org/anything/catalogue/page-" + (i + 1), new Page.NavigateOptions().setTimeout(60000));
          String text = page.textContent("body");
          String kind = classify(res.status(), text);
          trail.add(String.valueOf(res.status()));
          if (kind.equals("ok")) {
            outcome = "ok";
            exitIp = JsonParser.parseString(text).getAsJsonObject().get("origin").getAsString();
            break;
          }
          if (kind.equals("rate_limited")) {
            if (!polite && attempt == 3) break;
            // Honour Retry-After; without one, back off exponentially (1, 2, 4… s, capped).
            String ra = res.headers().get("retry-after");
            int seconds = !polite ? 0 : ra != null ? Integer.parseInt(ra) : (int) Math.min(Math.pow(2, attempt - 1), 30);
            if (seconds > 0) trail.add("wait " + seconds + "s");
            waited += seconds;
            Thread.sleep(seconds * 1000L);
            continue;
          }
          if (kind.equals("blocked") && polite && next < PROXIES.length) {
            // This exit is done: move the session to another proxy and continue in a new context
            // (its own connections) that keeps the cookies and storage.
            swapProxy(wsUrl, sessionId, PROXIES[next++]);
            String state = context.storageState();
            context.close();
            context = browser.newContext(new Browser.NewContextOptions().setStorageState(state));
            context.route("https://httpbin.org/anything/**", site);
            page = context.newPage();
            swaps++;
            trail.add("swap proxy");
            continue;
          }
          if (kind.equals("error")) {
            Thread.sleep(1000); // a transient failure: try again
            continue;
          }
          break;
        }
        if (exitIp != null) {
          fetched++;
          if (!exits.contains(exitIp)) exits.add(exitIp);
        }
        JsonObject row = new JsonObject();
        row.addProperty("page", i + 1);
        row.addProperty("outcome", outcome);
        row.addProperty("exit_ip", exitIp);
        row.addProperty("trail", String.join(" → ", trail));
        rows.add(row);
      }
      page.navigate("https://httpbin.org/cookies", new Page.NavigateOptions().setTimeout(60000));
      JsonObject cookies = JsonParser.parseString(page.textContent("body")).getAsJsonObject().getAsJsonObject("cookies");
      cookieKept = cookies.has("session") && cookies.get("session").getAsString().equals("alice-42");
    } finally {
      browser.close();
    }
    JsonObject out = new JsonObject();
    out.addProperty("client", polite ? "polite" : "naive");
    out.addProperty("pages_fetched", fetched);
    JsonArray exitArray = new JsonArray();
    exits.forEach(exitArray::add);
    out.add("exits", exitArray);
    out.addProperty("waited_seconds", waited);
    out.addProperty("proxy_swaps", swaps);
    out.addProperty("session_cookie_kept", cookieKept);
    out.add("pages", rows);
    return out;
  }

  public static void main(String[] args) throws Exception {
    JsonObject out = new JsonObject();
    try (Playwright pw = Playwright.create()) {
      // Naive: retries a 429 straight away and has no plan for a block. Polite: waits as told and
      // moves to another exit when one is blocked.
      out.add("naive", crawl(pw, false));
      out.add("polite", crawl(pw, true));
    }
    System.out.println(GSON.toJson(out));
  }
}
```

```csharp
// dotnet add package Microsoft.Playwright --version 1.60.0
// env: CDPFLEET_API_KEY, SOCKS_PROXIES (comma-separated sticky socks5:// URLs: one exit IP each)
using System.Net.Http.Json;
using System.Text;
using System.Text.Encodings.Web;
using System.Text.Json;
using System.Text.Json.Nodes;
using System.Text.RegularExpressions;
using Microsoft.Playwright;

var key = Environment.GetEnvironmentVariable("CDPFLEET_API_KEY")!;
var proxies = Environment.GetEnvironmentVariable("SOCKS_PROXIES")!.Split(',');
const string Challenge = "<title>Just a moment...</title><p>Checking your browser before accessing the site.</p>";

using var http = new HttpClient();
http.DefaultRequestHeaders.Add("x-api-key", key);
using var playwright = await Playwright.CreateAsync();

// A stand-in for a rate-limited site, in front of real httpbin requests. Each request really goes
// out through the session's proxy; the limiter decides by the IP httpbin saw (its "origin"):
// 3 pages in a burst, then one page per 5 s (429 + Retry-After: 5), 6 pages per IP in total. An IP
// that comes back before its Retry-After, or is over its quota, gets a challenge page from then on.
Func<IRoute, Task> Limiter()
{
    var ips = new Dictionary<string, IpState>();
    return async (route) =>
    {
        var now = DateTime.UtcNow; // when the request arrived
        IAPIResponse real;
        try { real = await route.FetchAsync(new() { Timeout = 30000 }); }
        catch (PlaywrightException) { await route.FulfillAsync(new() { Status = 502, ContentType = "text/plain", Body = "Bad Gateway" }); return; } // the proxy hiccuped
        if (real.Status != 200) { await route.FulfillAsync(new() { Response = real }); return; } // httpbin itself failed
        var ip = (await real.JsonAsync())!.Value.GetProperty("origin").GetString()!;
        if (!ips.TryGetValue(ip, out var s)) ips[ip] = s = new IpState();
        if (now < s.RetryAt || s.Served >= 6) s.Challenged = true;
        if (s.Challenged) { await route.FulfillAsync(new() { Status = 403, ContentType = "text/html", Body = Challenge }); return; }
        if (s.Served >= 3 && now - s.Last < TimeSpan.FromSeconds(5))
        {
            s.RetryAt = now.AddSeconds(5);
            await route.FulfillAsync(new() { Status = 429, Headers = new Dictionary<string, string> { ["retry-after"] = "5" }, ContentType = "text/plain", Body = "Too Many Requests" });
            return;
        }
        s.Served++;
        s.Last = now;
        await route.FulfillAsync(new() { Response = real });
    };
}

// What the response means: the data, a rate limit (wait), or a block (this IP is done).
string Classify(int status, string text) =>
    status == 429 ? "rate_limited"
    : status is 403 or 503 || Regex.IsMatch(text, "just a moment|captcha|access denied", RegexOptions.IgnoreCase) ? "blocked"
    : status == 200 ? "ok" : "error";

// 503 means the fleet is momentarily busy: wait Retry-After and try again.
async Task<JsonElement> Launch(JsonObject body)
{
    for (var attempt = 1; ; attempt++)
    {
        var res = await http.PostAsync("https://starter.cdpfleet.com/chromium/session", new StringContent(body.ToJsonString(), Encoding.UTF8, "application/json"));
        if (res.IsSuccessStatusCode) return await res.Content.ReadFromJsonAsync<JsonElement>();
        if ((int)res.StatusCode != 503 || attempt == 5) throw new Exception($"launch {(int)res.StatusCode} {await res.Content.ReadAsStringAsync()}");
        await Task.Delay((int)(res.Headers.RetryAfter?.Delta?.TotalMilliseconds ?? 5000));
    }
}

async Task<JsonObject> Crawl(bool polite)
{
    var session = await Launch(new JsonObject { ["proxy"] = proxies[polite ? 1 : 0], ["headless"] = "new", ["record"] = false });
    var wsUrl = session.GetProperty("wsUrl").GetString()!;
    var sessionId = session.GetProperty("sessionId").GetString()!;
    var browser = await playwright.Chromium.ConnectAsync(wsUrl, new() { Headers = new Dictionary<string, string> { ["x-api-key"] = key } });
    var site = Limiter();

    // Live proxy swap on the session's router: new connections use the new proxy.
    async Task SwapProxy(string proxy)
    {
        var r = await http.PostAsJsonAsync($"https://{new Uri(wsUrl).Host}/admin/session/proxy", new { session_id = sessionId, proxy });
        if (!r.IsSuccessStatusCode) throw new Exception($"swap {(int)r.StatusCode} {await r.Content.ReadAsStringAsync()}");
    }
    async Task<(IBrowserContext, IPage)> Open(string? storageState = null)
    {
        var context = await browser.NewContextAsync(new() { StorageState = storageState });
        await context.RouteAsync("https://httpbin.org/anything/**", site);
        return (context, await context.NewPageAsync());
    }

    var rows = new JsonArray();
    var exits = new List<string>();
    int waited = 0, swaps = 0, next = 2, fetched = 0;
    bool cookieKept;
    try
    {
        var (context, page) = await Open();
        // A logged-in state that must survive a proxy swap.
        await page.GotoAsync("https://httpbin.org/cookies/set?session=alice-42", new() { Timeout = 60000 });
        for (var i = 0; i < 8; i++)
        {
            var trail = new List<string>();
            string outcome = "gave up";
            string? exitIp = null;
            for (var attempt = 1; attempt <= 6; attempt++)
            {
                var res = (await page.GotoAsync($"https://httpbin.org/anything/catalogue/page-{i + 1}", new() { Timeout = 60000 }))!;
                var text = await page.TextContentAsync("body") ?? "";
                var kind = Classify(res.Status, text);
                trail.Add(res.Status.ToString());
                if (kind == "ok")
                {
                    outcome = "ok";
                    exitIp = JsonDocument.Parse(text).RootElement.GetProperty("origin").GetString();
                    break;
                }
                if (kind == "rate_limited")
                {
                    if (!polite && attempt == 3) break;
                    // Honour Retry-After; without one, back off exponentially (1, 2, 4… s, capped).
                    var seconds = !polite ? 0 : res.Headers.TryGetValue("retry-after", out var ra) ? int.Parse(ra) : (int)Math.Min(Math.Pow(2, attempt - 1), 30);
                    if (seconds > 0) trail.Add($"wait {seconds}s");
                    waited += seconds;
                    await Task.Delay(seconds * 1000);
                    continue;
                }
                if (kind == "blocked" && polite && next < proxies.Length)
                {
                    // This exit is done: move the session to another proxy and continue in a new context
                    // (its own connections) that keeps the cookies and storage.
                    await SwapProxy(proxies[next++]);
                    var state = await context.StorageStateAsync();
                    await context.CloseAsync();
                    (context, page) = await Open(state);
                    swaps++;
                    trail.Add("swap proxy");
                    continue;
                }
                if (kind == "error")
                {
                    await Task.Delay(1000); // a transient failure: try again
                    continue;
                }
                break;
            }
            if (exitIp != null)
            {
                fetched++;
                if (!exits.Contains(exitIp)) exits.Add(exitIp);
            }
            rows.Add(new JsonObject { ["page"] = i + 1, ["outcome"] = outcome, ["exit_ip"] = exitIp, ["trail"] = string.Join(" → ", trail) });
        }
        await page.GotoAsync("https://httpbin.org/cookies", new() { Timeout = 60000 });
        var cookies = JsonDocument.Parse(await page.TextContentAsync("body") ?? "{}").RootElement.GetProperty("cookies");
        cookieKept = cookies.TryGetProperty("session", out var c) && c.GetString() == "alice-42";
    }
    finally
    {
        await browser.CloseAsync();
    }
    return new JsonObject
    {
        ["client"] = polite ? "polite" : "naive",
        ["pages_fetched"] = fetched,
        ["exits"] = new JsonArray(exits.Select((e) => (JsonNode)JsonValue.Create(e)!).ToArray()),
        ["waited_seconds"] = waited,
        ["proxy_swaps"] = swaps,
        ["session_cookie_kept"] = cookieKept,
        ["pages"] = rows,
    };
}

// Naive: retries a 429 straight away and has no plan for a block. Polite: waits as told and moves
// to another exit when one is blocked.
var naive = await Crawl(false);
var polite = await Crawl(true);
Console.WriteLine(new JsonObject { ["naive"] = naive, ["polite"] = polite }.ToJsonString(new JsonSerializerOptions { WriteIndented = true, Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping }));

class IpState
{
    public int Served;
    public DateTime Last;
    public DateTime RetryAt;
    public bool Challenged;
}
```

```go
// go get github.com/playwright-community/playwright-go@v0.6000.0
// Driver: build playwright-core 1.60.0 from npm and set PLAYWRIGHT_DRIVER_PATH (see /docs/quickstart).
// env: CDPFLEET_API_KEY, SOCKS_PROXIES (comma-separated sticky socks5:// URLs: one exit IP each)
package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io"
	"log"
	"math"
	"net/http"
	"net/url"
	"os"
	"regexp"
	"strconv"
	"strings"
	"time"

	"github.com/playwright-community/playwright-go"
)

var (
	key     = os.Getenv("CDPFLEET_API_KEY")
	proxies = strings.Split(os.Getenv("SOCKS_PROXIES"), ",")
)

const challenge = "<title>Just a moment...</title><p>Checking your browser before accessing the site.</p>"

type ipState struct {
	served     int
	last       time.Time
	retryAt    time.Time
	challenged bool
}

// A stand-in for a rate-limited site, in front of real httpbin requests. Each request really goes
// out through the session's proxy; the limiter decides by the IP httpbin saw (its "origin"):
// 3 pages in a burst, then one page per 5 s (429 + Retry-After: 5), 6 pages per IP in total. An IP
// that comes back before its Retry-After, or is over its quota, gets a challenge page from then on.
func limiter() func(playwright.Route) {
	ips := map[string]*ipState{}
	return func(route playwright.Route) {
		now := time.Now() // when the request arrived
		real, err := route.Fetch(playwright.RouteFetchOptions{Timeout: playwright.Float(30000)})
		if err != nil { // the proxy hiccuped
			route.Fulfill(playwright.RouteFulfillOptions{Status: playwright.Int(502), ContentType: playwright.String("text/plain"), Body: "Bad Gateway"})
			return
		}
		if real.Status() != 200 { // httpbin itself failed
			route.Fulfill(playwright.RouteFulfillOptions{Response: real})
			return
		}
		var body struct {
			Origin string `json:"origin"`
		}
		real.JSON(&body)
		s, ok := ips[body.Origin]
		if !ok {
			s = &ipState{}
			ips[body.Origin] = s
		}
		if now.Before(s.retryAt) || s.served >= 6 {
			s.challenged = true
		}
		if s.challenged {
			route.Fulfill(playwright.RouteFulfillOptions{Status: playwright.Int(403), ContentType: playwright.String("text/html"), Body: challenge})
			return
		}
		if s.served >= 3 && now.Sub(s.last) < 5*time.Second {
			s.retryAt = now.Add(5 * time.Second)
			route.Fulfill(playwright.RouteFulfillOptions{Status: playwright.Int(429), Headers: map[string]string{"retry-after": "5"}, ContentType: playwright.String("text/plain"), Body: "Too Many Requests"})
			return
		}
		s.served++
		s.last = now
		route.Fulfill(playwright.RouteFulfillOptions{Response: real})
	}
}

var challengeText = regexp.MustCompile(`(?i)just a moment|captcha|access denied`)

// What the response means: the data, a rate limit (wait), or a block (this IP is done).
func classify(status int, text string) string {
	if status == 429 {
		return "rate_limited"
	}
	if status == 403 || status == 503 || challengeText.MatchString(text) {
		return "blocked"
	}
	if status == 200 {
		return "ok"
	}
	return "error"
}

// 503 means the fleet is momentarily busy: wait Retry-After and try again.
func launch(opts map[string]any) (map[string]any, error) {
	body, _ := json.Marshal(opts)
	for attempt := 1; ; attempt++ {
		req, _ := http.NewRequest("POST", "https://starter.cdpfleet.com/chromium/session", bytes.NewReader(body))
		req.Header.Set("x-api-key", key)
		req.Header.Set("content-type", "application/json")
		res, err := http.DefaultClient.Do(req)
		if err != nil {
			return nil, err
		}
		b, _ := io.ReadAll(res.Body)
		res.Body.Close()
		if res.StatusCode == 200 {
			var session map[string]any
			return session, json.Unmarshal(b, &session)
		}
		if res.StatusCode != 503 || attempt == 5 {
			return nil, fmt.Errorf("launch %d %s", res.StatusCode, strings.TrimSpace(string(b)))
		}
		wait, err := strconv.Atoi(res.Header.Get("retry-after"))
		if err != nil {
			wait = 5
		}
		time.Sleep(time.Duration(wait) * time.Second)
	}
}

// Live proxy swap on the session's router: new connections use the new proxy.
func swapProxy(wsURL, sessionID, proxy string) error {
	u, _ := url.Parse(wsURL)
	body, _ := json.Marshal(map[string]string{"session_id": sessionID, "proxy": proxy})
	req, _ := http.NewRequest("POST", "https://"+u.Host+"/admin/session/proxy", bytes.NewReader(body))
	req.Header.Set("x-api-key", key)
	req.Header.Set("content-type", "application/json")
	res, err := http.DefaultClient.Do(req)
	if err != nil {
		return err
	}
	defer res.Body.Close()
	if res.StatusCode != 200 {
		b, _ := io.ReadAll(res.Body)
		return fmt.Errorf("swap %d %s", res.StatusCode, b)
	}
	return nil
}

type pageRow struct {
	Page    int     `json:"page"`
	Outcome string  `json:"outcome"`
	ExitIP  *string `json:"exit_ip"`
	Trail   string  `json:"trail"`
}

type crawlResult struct {
	Client            string    `json:"client"`
	PagesFetched      int       `json:"pages_fetched"`
	Exits             []string  `json:"exits"`
	WaitedSeconds     int       `json:"waited_seconds"`
	ProxySwaps        int       `json:"proxy_swaps"`
	SessionCookieKept bool      `json:"session_cookie_kept"`
	Pages             []pageRow `json:"pages"`
}

func crawl(pw *playwright.Playwright, polite bool) (crawlResult, error) {
	out := crawlResult{Client: "naive", Exits: []string{}, Pages: []pageRow{}}
	first := proxies[0]
	if polite {
		out.Client, first = "polite", proxies[1]
	}
	session, err := launch(map[string]any{"proxy": first, "headless": "new", "record": false})
	if err != nil {
		return out, err
	}
	wsURL, sessionID := session["wsUrl"].(string), session["sessionId"].(string)
	browser, err := pw.Chromium.Connect(wsURL, playwright.BrowserTypeConnectOptions{Headers: map[string]string{"x-api-key": key}})
	if err != nil {
		return out, err
	}
	defer browser.Close()
	site := limiter()
	open := func(state *playwright.StorageState) (playwright.BrowserContext, playwright.Page, error) {
		opts := playwright.BrowserNewContextOptions{}
		if state != nil {
			opts.StorageState = state.ToOptionalStorageState()
		}
		context, err := browser.NewContext(opts)
		if err != nil {
			return nil, nil, err
		}
		if err := context.Route("https://httpbin.org/anything/**", site); err != nil {
			return nil, nil, err
		}
		page, err := context.NewPage()
		return context, page, err
	}
	context, page, err := open(nil)
	if err != nil {
		return out, err
	}
	// A logged-in state that must survive a proxy swap.
	if _, err := page.Goto("https://httpbin.org/cookies/set?session=alice-42", playwright.PageGotoOptions{Timeout: playwright.Float(60000)}); err != nil {
		return out, err
	}
	next := 2
	seen := map[string]bool{}
	for i := 0; i < 8; i++ {
		pageURL := fmt.Sprintf("https://httpbin.org/anything/catalogue/page-%d", i+1)
		trail := []string{}
		row := pageRow{Page: i + 1, Outcome: "gave up"}
		for attempt := 1; attempt <= 6; attempt++ {
			res, err := page.Goto(pageURL, playwright.PageGotoOptions{Timeout: playwright.Float(60000)})
			if err != nil {
				return out, err
			}
			text, err := page.TextContent("body")
			if err != nil {
				return out, err
			}
			kind := classify(res.Status(), text)
			trail = append(trail, strconv.Itoa(res.Status()))
			if kind == "ok" {
				var body struct {
					Origin string `json:"origin"`
				}
				json.Unmarshal([]byte(text), &body)
				row.Outcome, row.ExitIP = "ok", &body.Origin
				break
			}
			if kind == "rate_limited" {
				if !polite && attempt == 3 {
					break
				}
				seconds := 0
				if polite {
					// Honour Retry-After; without one, back off exponentially (1, 2, 4… s, capped).
					if seconds, err = strconv.Atoi(res.Headers()["retry-after"]); err != nil {
						seconds = int(math.Min(math.Pow(2, float64(attempt-1)), 30))
					}
					trail = append(trail, fmt.Sprintf("wait %ds", seconds))
				}
				out.WaitedSeconds += seconds
				time.Sleep(time.Duration(seconds) * time.Second)
				continue
			}
			if kind == "blocked" && polite && next < len(proxies) {
				// This exit is done: move the session to another proxy and continue in a new context
				// (its own connections) that keeps the cookies and storage.
				if err := swapProxy(wsURL, sessionID, proxies[next]); err != nil {
					return out, err
				}
				next++
				state, err := context.StorageState()
				if err != nil {
					return out, err
				}
				context.Close()
				if context, page, err = open(state); err != nil {
					return out, err
				}
				out.ProxySwaps++
				trail = append(trail, "swap proxy")
				continue
			}
			if kind == "error" {
				time.Sleep(time.Second) // a transient failure: try again
				continue
			}
			break
		}
		row.Trail = strings.Join(trail, " → ")
		out.Pages = append(out.Pages, row)
		if row.ExitIP != nil {
			out.PagesFetched++
			if !seen[*row.ExitIP] {
				seen[*row.ExitIP] = true
				out.Exits = append(out.Exits, *row.ExitIP)
			}
		}
	}
	if _, err := page.Goto("https://httpbin.org/cookies", playwright.PageGotoOptions{Timeout: playwright.Float(60000)}); err != nil {
		return out, err
	}
	text, err := page.TextContent("body")
	if err != nil {
		return out, err
	}
	var cookies struct {
		Cookies map[string]string `json:"cookies"`
	}
	json.Unmarshal([]byte(text), &cookies)
	out.SessionCookieKept = cookies.Cookies["session"] == "alice-42"
	return out, nil
}

func run() error {
	pw, err := playwright.Run(&playwright.RunOptions{SkipInstallBrowsers: true})
	if err != nil {
		return err
	}
	defer pw.Stop()
	// Naive: retries a 429 straight away and has no plan for a block. Polite: waits as told and
	// moves to another exit when one is blocked.
	naive, err := crawl(pw, false)
	if err != nil {
		return fmt.Errorf("naive: %w", err)
	}
	polite, err := crawl(pw, true)
	if err != nil {
		return fmt.Errorf("polite: %w", err)
	}
	enc := json.NewEncoder(os.Stdout)
	enc.SetEscapeHTML(false)
	enc.SetIndent("", "  ")
	return enc.Encode(map[string]crawlResult{"naive": naive, "polite": polite})
}

func main() {
	if err := run(); err != nil {
		log.Fatal(err)
	}
}
```

## What we got

| Crawler | Page | What happened | Result | Served from |
| --- | --- | --- | --- | --- |
| naive | 1 | 200 | ok | 203.0.113.1 |
| naive | 2 | 200 | ok | 203.0.113.1 |
| naive | 3 | 200 | ok | 203.0.113.1 |
| naive | 4 | 429 → 403 | gave up | — |
| naive | 5 | 403 | gave up | — |
| naive | 6 | 403 | gave up | — |
| naive | 7 | 403 | gave up | — |
| naive | 8 | 403 | gave up | — |
| polite | 1 | 200 | ok | 203.0.113.2 |
| polite | 2 | 200 | ok | 203.0.113.2 |
| polite | 3 | 200 | ok | 203.0.113.2 |
| polite | 4 | 429 → wait 5s → 200 | ok | 203.0.113.2 |
| polite | 5 | 429 → wait 5s → 200 | ok | 203.0.113.2 |
| polite | 6 | 502 → 429 → wait 5s → 200 | ok | 203.0.113.2 |
| polite | 7 | 403 → swap proxy → 200 | ok | 203.0.113.3 |
| polite | 8 | 200 | ok | 203.0.113.3 |

From the Node.js run on 2026-10-16. IP addresses are replaced with placeholders (203.0.113.x); equal addresses stay equal. The other languages produced the same findings.

Raw output (Node.js)

```
{
  "naive": {
    "client": "naive",
    "pages_fetched": 3,
    "exits": [
      "203.0.113.1"
    ],
    "waited_seconds": 0,
    "proxy_swaps": 0,
    "session_cookie_kept": true,
    "pages": [
      {
        "page": 1,
        "outcome": "ok",
        "exit_ip": "203.0.113.1",
        "trail": "200"
      },
      {
        "page": 2,
        "outcome": "ok",
        "exit_ip": "203.0.113.1",
        "trail": "200"
      },
      {
        "page": 3,
        "outcome": "ok",
        "exit_ip": "203.0.113.1",
        "trail": "200"
      },
      {
        "page": 4,
        "outcome": "gave up",
        "exit_ip": null,
        "trail": "429 → 403"
      },
      {
        "page": 5,
        "outcome": "gave up",
        "exit_ip": null,
        "trail": "403"
      },
      {
        "page": 6,
        "outcome": "gave up",
        "exit_ip": null,
        "trail": "403"
      },
      {
        "page": 7,
        "outcome": "gave up",
        "exit_ip": null,
        "trail": "403"
      },
      {
        "page": 8,
        "outcome": "gave up",
        "exit_ip": null,
        "trail": "403"
      }
    ]
  },
  "polite": {
    "client": "polite",
    "pages_fetched": 8,
    "exits": [
      "203.0.113.2",
      "203.0.113.3"
    ],
    "waited_seconds": 15,
    "proxy_swaps": 1,
    "session_cookie_kept": true,
    "pages": [
      {
        "page": 1,
        "outcome": "ok",
        "exit_ip": "203.0.113.2",
        "trail": "200"
      },
      {
        "page": 2,
        "outcome": "ok",
        "exit_ip": "203.0.113.2",
        "trail": "200"
      },
      {
        "page": 3,
        "outcome": "ok",
        "exit_ip": "203.0.113.2",
        "trail": "200"
      },
      {
        "page": 4,
        "outcome": "ok",
        "exit_ip": "203.0.113.2",
        "trail": "429 → wait 5s → 200"
      },
      {
        "page": 5,
        "outcome": "ok",
        "exit_ip": "203.0.113.2",
        "trail": "429 → wait 5s → 200"
      },
      {
        "page": 6,
        "outcome": "ok",
        "exit_ip": "203.0.113.2",
        "trail": "502 → 429 → wait 5s → 200"
      },
      {
        "page": 7,
        "outcome": "ok",
        "exit_ip": "203.0.113.3",
        "trail": "403 → swap proxy → 200"
      },
      {
        "page": 8,
        "outcome": "ok",
        "exit_ip": "203.0.113.3",
        "trail": "200"
      }
    ]
  }
}
```

## Takeaways

-   **Retrying a 429 straight away turns a slowdown into a block.** The naive crawler got its first burst of pages, then a 429; its immediate retry arrived before `Retry-After` and the IP was challenged for the rest of the run: **3 or 4 of 8 pages**.
-   **Waiting as told costs seconds, not pages.** The polite crawler waited `Retry-After` after every 429, a few seconds each, and kept getting pages from the same exit until its quota ran out.
-   **When an exit is burned, move — don't wait.** At its quota the first IP was challenged. The crawler swapped the session to another proxy and opened a new context, and the blocked page came back `200` from a different exit IP: **8 of 8 pages, two exits, one proxy swap**.
-   **The login survived the move.** The session cookie set before the crawl was still there after the swap, because the new context started from the old `storageState`. Same browser and same session, with no relaunch.
-   Tell the signals apart: a 429 asks you to slow down on this IP, while a challenge means this IP is done for now. Treating every non-200 as "retry" gets the first wrong, and treating every one as "swap" spends exits you didn't need to.

---
Source: https://cdpfleet.com/docs/cases/soft-block-backoff · Markdown: https://cdpfleet.com/docs/cases/soft-block-backoff.md · All docs for AI tools: https://cdpfleet.com/llms.txt