[Examples & benchmarks](https://cdpfleet.com/docs/cases) / #61 · 2026-10-18 · Example · Medium · for [web scraping](https://cdpfleet.com/web-scraping), [llm data](https://cdpfleet.com/llm-data)

# Borrow the page's token and read its paginated API directly

Log in once in a remote browser, find the bearer token the app sends to its own API, and fetch every page of the API from inside the page: the same records as clicking through every page, with 16 fields each instead of 5 columns, in about a second instead of many.

[Chromium](https://cdpfleet.com/docs/browsers/chromium)

Run on production on 2026-10-18: ✓ Node.js ✓ Python ✓ Java ✓ C# ✓ Go

## The problem

Many sites behind a login are single-page apps: the table you see is drawn from a JSON API, and the app proves who you are with a token on every call. Scraping the table means clicking through pages and parsing cells; the API has the same records with every field, but answers 401 without the token. The token is already in the browser after you log in — the question is where to find it, how long it lasts, and how to use it so the requests still come from the browser. The site here is Toolshop (practicesoftwaretesting.com), a public demo shop built for practising test automation, with its published demo account.

## What we used, and why

| What | Why |
| --- | --- |
| `chromium`, `headless: "new"` | One session. The login, the token and the API calls all live in it. |
| `page.setDefaultTimeout(90000)` | Through a rotating residential exit the app sometimes took 30 s to draw its login form — longer than Playwright's default 30 s for an action. |
| `localStorage.getItem('auth-token')` | Where this app keeps its token after login (a JWT). Other apps use a cookie, `sessionStorage` or memory only. |
| `page.waitForRequest(…)` + `allHeaders()` | Catches the app's own API call and reads its `Authorization` header — this works even when the token is never stored anywhere you can read. |
| `page.evaluate(() => fetch(url, { headers: { authorization } }))` | Calls the API from inside the page, so the requests come from the browser, through the session's proxy, with the site's own origin — not from a different HTTP stack ([four ways to make a request](https://cdpfleet.com/docs/cases/proxy-paths)). |

## How it works

1.  Open the login page and sign in with the site's demo account.
2.  Read the token from `localStorage`, open the invoices page and catch the app's first `/invoices` request to compare its `Authorization` header; decode the JWT's lifetime.
3.  Click Next through all pages, reading the table each time, and count the app's `/invoices` calls.
4.  Fetch every page of `/invoices?page=N` from inside the page with the token: one by one, then all at once; then once without the token.
5.  Compare the records both ways collected.

## The code

The same program in five languages (also [on GitHub](https://github.com/cdpfleet/cdpfleet/tree/main/cases/61-scrape-a-paginated-json-api-with-a-token), with the raw output). Set these environment variables first:

-   `CDPFLEET_API_KEY` — your API key (dashboard → API keys)
-   `PROXY_URL` — your proxy, e.g. `http://user:pass@proxy.example.com:8000`

 

```node
// npm install playwright@1.60.0
// env: CDPFLEET_API_KEY, PROXY_URL
import { chromium } from 'playwright';

const KEY = process.env.CDPFLEET_API_KEY;
const SITE = 'https://practicesoftwaretesting.com';  // Toolshop: a public demo shop built for test automation
const API = 'https://api.practicesoftwaretesting.com';
const LOGIN = { email: 'customer@practicesoftwaretesting.com', password: 'welcome01' }; // the site's published demo account

// 503 means the fleet is momentarily busy: wait Retry-After and try again.
async function launch(body) {
  for (let attempt = 1; ; attempt++) {
    const res = await fetch('https://starter.cdpfleet.com/chromium/session', {
      method: 'POST',
      headers: { 'x-api-key': KEY, 'content-type': 'application/json' },
      body: JSON.stringify(body),
    });
    if (res.ok) return res.json();
    if (res.status !== 503 || attempt === 5) throw new Error(`launch ${res.status} ${await res.text()}`);
    await new Promise((r) => setTimeout(r, (Number(res.headers.get('retry-after')) || 5) * 1000));
  }
}

const { wsUrl } = await launch({ proxy: process.env.PROXY_URL, headless: 'new', record: false });
const browser = await chromium.connect(wsUrl, { headers: { 'x-api-key': KEY } });
try {
  const page = await browser.newPage();
  // The app can take 30 s to render through a rotating residential exit: wait up to 90 s.
  page.setDefaultTimeout(90000);

  // 1. Log in like a person; the site stores its token and sends it with every API call.
  await page.goto(`${SITE}/auth/login`);
  await page.fill('#email', LOGIN.email);
  await page.fill('#password', LOGIN.password);
  await page.click('[type=submit]');
  await page.waitForURL('**/account');

  // 2. Find the token twice: in localStorage, and in the header of the page's own API request.
  const stored = await page.evaluate(() => localStorage.getItem('auth-token'));
  let uiCalls = 0;
  const countCall = (r) => { if (r.url().startsWith(`${API}/invoices`)) uiCalls++; };
  page.on('request', countCall);
  const [firstCall] = await Promise.all([
    page.waitForRequest((r) => r.url().startsWith(`${API}/invoices`)),
    page.goto(`${SITE}/account/invoices`),
  ]);
  const header = (await firstCall.allHeaders()).authorization || '';
  const claims = JSON.parse(Buffer.from(stored.split('.')[1], 'base64url').toString());

  // 3. The page's way: click through the pager, reading the table each time.
  await page.waitForSelector('table tbody tr');
  const t0 = Date.now();
  const viaUi = [];
  const readTable = () => page.$$eval('table tbody tr td:first-child', (tds) => tds.map((td) => td.textContent.trim()));
  viaUi.push(...await readTable());
  const lastPage = Number(await page.locator('button[aria-label^="Page-"]').last().innerText());
  for (let n = 2; n <= lastPage; n++) {
    await Promise.all([
      page.waitForResponse((r) => r.url() === `${API}/invoices?page=${n}`),
      page.click('[data-test="pagination-next"]'),
    ]);
    await page.waitForFunction((first) => document.querySelector('table tbody tr td')?.textContent.trim() !== first, viaUi[viaUi.length - 15]);
    viaUi.push(...await readTable());
  }
  const uiSeconds = (Date.now() - t0) / 1000;
  page.off('request', countCall);
  const columns = await page.locator('table thead th').count();

  // 4. The API's way: the same token on fetch() from inside the page — the browser makes the
  //    requests, through the session's proxy. Page 1 tells last_page; then all pages one by one,
  //    and all pages at once.
  const viaApi = await page.evaluate(async ({ api, token }) => {
    const get = (n) => fetch(`${api}/invoices?page=${n}`, { headers: { authorization: `Bearer ${token}` } }).then(async (r) => ({ status: r.status, text: await r.text() }));
    const meta = JSON.parse((await get(1)).text);
    const numbers = Array.from({ length: meta.last_page }, (_, i) => i + 1);
    let t = performance.now();
    const pages = [];
    for (const n of numbers) pages.push(await get(n));
    const sequential = (performance.now() - t) / 1000;
    t = performance.now();
    await Promise.all(numbers.map(get));
    const parallel = (performance.now() - t) / 1000;
    const noToken = await fetch(`${api}/invoices?page=1`).then((r) => r.status);
    const items = pages.flatMap((p) => JSON.parse(p.text).data);
    return {
      sequential, parallel, total: meta.total, perPage: meta.per_page, lastPage: meta.last_page,
      requests: pages.length, bytes: pages.reduce((s, p) => s + p.text.length, 0), statuses: [...new Set(pages.map((p) => p.status))],
      invoices: items.map((i) => i.invoice_number), fields: Object.keys(items[0]).length, noToken,
    };
  }, { api: API, token: stored });

  const sameSet = viaUi.length === viaApi.invoices.length && [...viaUi].sort().join() === [...viaApi.invoices].sort().join();
  console.log(JSON.stringify({
    token: {
      in_local_storage: stored ? 'auth-token' : null,
      header_matches: header === `Bearer ${stored}`,
      lifetime_seconds: claims.exp - claims.iat,
      seconds_left_at_end: claims.exp - Math.floor(Date.now() / 1000),
    },
    api: { total: viaApi.total, per_page: viaApi.perPage, last_page: viaApi.lastPage, without_token: viaApi.noToken },
    ui: { pages: lastPage, invoices: viaUi.length, invoice_calls: uiCalls, columns, seconds: uiSeconds },
    direct: { requests: viaApi.requests, statuses: viaApi.statuses, invoices: viaApi.invoices.length, fields_per_invoice: viaApi.fields, json_bytes: viaApi.bytes, seconds_one_by_one: viaApi.sequential, seconds_all_at_once: viaApi.parallel },
    same_invoices: sameSet,
    ui_missing_from_api: [...new Set(viaUi)].filter((n) => !viaApi.invoices.includes(n)).length,
  }));
} finally {
  await browser.close();
}
```

```python
# pip install playwright==1.60.0 requests
# env: CDPFLEET_API_KEY, PROXY_URL
import base64
import json
import os
import time

import requests
from playwright.sync_api import sync_playwright

KEY = os.environ["CDPFLEET_API_KEY"]
SITE = "https://practicesoftwaretesting.com"  # Toolshop: a public demo shop built for test automation
API = "https://api.practicesoftwaretesting.com"
LOGIN = {"email": "customer@practicesoftwaretesting.com", "password": "welcome01"}  # the site's published demo account

# Fetches every page of the API from inside the page with the token: one by one, then all at once.
FETCH_ALL = """async ({ api, token }) => {
  const get = (n) => fetch(`${api}/invoices?page=${n}`, { headers: { authorization: `Bearer ${token}` } }).then(async (r) => ({ status: r.status, text: await r.text() }));
  const meta = JSON.parse((await get(1)).text);
  const numbers = Array.from({ length: meta.last_page }, (_, i) => i + 1);
  let t = performance.now();
  const pages = [];
  for (const n of numbers) pages.push(await get(n));
  const sequential = (performance.now() - t) / 1000;
  t = performance.now();
  await Promise.all(numbers.map(get));
  const parallel = (performance.now() - t) / 1000;
  const noToken = await fetch(`${api}/invoices?page=1`).then((r) => r.status);
  const items = pages.flatMap((p) => JSON.parse(p.text).data);
  return {
    sequential, parallel, total: meta.total, perPage: meta.per_page, lastPage: meta.last_page,
    requests: pages.length, bytes: pages.reduce((s, p) => s + p.text.length, 0), statuses: [...new Set(pages.map((p) => p.status))],
    invoices: items.map((i) => i.invoice_number), fields: Object.keys(items[0]).length, noToken,
  };
}"""

def launch(body):
    # 503 means the fleet is momentarily busy: wait Retry-After and try again.
    for attempt in range(1, 6):
        res = requests.post("https://starter.cdpfleet.com/chromium/session", headers={"x-api-key": KEY}, json=body)
        if res.ok:
            return res.json()
        if res.status_code != 503 or attempt == 5:
            raise RuntimeError(f"launch {res.status_code} {res.text}")
        time.sleep(int(res.headers.get("retry-after") or 5))

session = launch({"proxy": os.environ["PROXY_URL"], "headless": "new", "record": False})
with sync_playwright() as p:
    browser = p.chromium.connect(session["wsUrl"], headers={"x-api-key": KEY})
    try:
        page = browser.new_page()
        # The app can take 30 s to render through a rotating residential exit: wait up to 90 s.
        page.set_default_timeout(90000)

        # 1. Log in like a person; the site stores its token and sends it with every API call.
        page.goto(f"{SITE}/auth/login")
        page.fill("#email", LOGIN["email"])
        page.fill("#password", LOGIN["password"])
        page.click("[type=submit]")
        page.wait_for_url("**/account")

        # 2. Find the token twice: in localStorage, and in the header of the page's own API request.
        stored = page.evaluate("() => localStorage.getItem('auth-token')")
        ui_calls = 0

        def count_call(r):
            global ui_calls
            if r.url.startswith(f"{API}/invoices"):
                ui_calls += 1

        page.on("request", count_call)
        with page.expect_request(lambda r: r.url.startswith(f"{API}/invoices")) as first_call:
            page.goto(f"{SITE}/account/invoices")
        header = first_call.value.all_headers().get("authorization", "")
        payload = stored.split(".")[1]
        claims = json.loads(base64.urlsafe_b64decode(payload + "=" * (-len(payload) % 4)))

        # 3. The page's way: click through the pager, reading the table each time.
        page.wait_for_selector("table tbody tr")
        t0 = time.time()
        read_table = lambda: page.eval_on_selector_all("table tbody tr td:first-child", "tds => tds.map((td) => td.textContent.trim())")
        via_ui = read_table()
        last_page = int(page.locator('button[aria-label^="Page-"]').last.inner_text())
        for n in range(2, last_page + 1):
            with page.expect_response(lambda r, n=n: r.url == f"{API}/invoices?page={n}"):
                page.click('[data-test="pagination-next"]')
            page.wait_for_function("first => document.querySelector('table tbody tr td')?.textContent.trim() !== first", arg=via_ui[-15])
            via_ui += read_table()
        ui_seconds = time.time() - t0
        page.remove_listener("request", count_call)
        columns = page.locator("table thead th").count()

        # 4. The API's way: the same token on fetch() from inside the page — the browser makes the
        #    requests, through the session's proxy.
        via_api = page.evaluate(FETCH_ALL, {"api": API, "token": stored})

        print(json.dumps({
            "token": {
                "in_local_storage": "auth-token" if stored else None,
                "header_matches": header == f"Bearer {stored}",
                "lifetime_seconds": claims["exp"] - claims["iat"],
                "seconds_left_at_end": claims["exp"] - int(time.time()),
            },
            "api": {"total": via_api["total"], "per_page": via_api["perPage"], "last_page": via_api["lastPage"], "without_token": via_api["noToken"]},
            "ui": {"pages": last_page, "invoices": len(via_ui), "invoice_calls": ui_calls, "columns": columns, "seconds": ui_seconds},
            "direct": {"requests": via_api["requests"], "statuses": via_api["statuses"], "invoices": len(via_api["invoices"]), "fields_per_invoice": via_api["fields"],
                       "json_bytes": via_api["bytes"], "seconds_one_by_one": via_api["sequential"], "seconds_all_at_once": via_api["parallel"]},
            "same_invoices": len(via_ui) == len(via_api["invoices"]) and sorted(via_ui) == sorted(via_api["invoices"]),
            "ui_missing_from_api": len(set(via_ui) - set(via_api["invoices"])),
        }))
    finally:
        browser.close()
```

```java
// Maven: com.microsoft.playwright:playwright:1.60.0, com.google.code.gson:gson:2.11.0
// Run with PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1.
// env: CDPFLEET_API_KEY, PROXY_URL
import com.google.gson.*;
import com.microsoft.playwright.*;
import java.net.URI;
import java.net.http.*;
import java.nio.charset.StandardCharsets;
import java.util.*;
import java.util.concurrent.atomic.AtomicInteger;

public class Main {
  static final String KEY = System.getenv("CDPFLEET_API_KEY");
  static final String SITE = "https://practicesoftwaretesting.com"; // Toolshop: a public demo shop built for test automation
  static final String API = "https://api.practicesoftwaretesting.com";
  static final String EMAIL = "customer@practicesoftwaretesting.com", PASSWORD = "welcome01"; // the site's published demo account

  // Fetches every page of the API from inside the page with the token: one by one, then all at once.
  static final String FETCH_ALL = String.join("\n",
      "async ({ api, token }) => {",
      "  const get = (n) => fetch(`${api}/invoices?page=${n}`, { headers: { authorization: `Bearer ${token}` } }).then(async (r) => ({ status: r.status, text: await r.text() }));",
      "  const meta = JSON.parse((await get(1)).text);",
      "  const numbers = Array.from({ length: meta.last_page }, (_, i) => i + 1);",
      "  let t = performance.now();",
      "  const pages = [];",
      "  for (const n of numbers) pages.push(await get(n));",
      "  const sequential = (performance.now() - t) / 1000;",
      "  t = performance.now();",
      "  await Promise.all(numbers.map(get));",
      "  const parallel = (performance.now() - t) / 1000;",
      "  const noToken = await fetch(`${api}/invoices?page=1`).then((r) => r.status);",
      "  const items = pages.flatMap((p) => JSON.parse(p.text).data);",
      "  return JSON.stringify({",
      "    sequential, parallel, total: meta.total, perPage: meta.per_page, lastPage: meta.last_page,",
      "    requests: pages.length, bytes: pages.reduce((s, p) => s + p.text.length, 0), statuses: [...new Set(pages.map((p) => p.status))],",
      "    invoices: items.map((i) => i.invoice_number), fields: Object.keys(items[0]).length, noToken,",
      "  });",
      "}");

  // 503 means the fleet is momentarily busy: wait Retry-After and try again.
  static JsonObject launch(JsonObject body) throws Exception {
    for (int attempt = 1; ; attempt++) {
      HttpResponse<String> res = HttpClient.newHttpClient().send(HttpRequest.newBuilder(URI.create("https://starter.cdpfleet.com/chromium/session"))
          .header("x-api-key", KEY).header("content-type", "application/json")
          .POST(HttpRequest.BodyPublishers.ofString(body.toString())).build(), HttpResponse.BodyHandlers.ofString());
      if (res.statusCode() == 200) return JsonParser.parseString(res.body()).getAsJsonObject();
      if (res.statusCode() != 503 || attempt == 5) throw new RuntimeException("launch " + res.statusCode() + " " + res.body());
      Thread.sleep(Long.parseLong(res.headers().firstValue("retry-after").orElse("5")) * 1000);
    }
  }

  @SuppressWarnings("unchecked")
  static List<String> readTable(Page page) {
    return (List<String>) page.evalOnSelectorAll("table tbody tr td:first-child", "tds => tds.map((td) => td.textContent.trim())");
  }

  public static void main(String[] args) throws Exception {
    JsonObject body = new JsonObject();
    body.addProperty("proxy", System.getenv("PROXY_URL"));
    body.addProperty("headless", "new");
    body.addProperty("record", false);
    String wsUrl = launch(body).get("wsUrl").getAsString();
    try (Playwright playwright = Playwright.create()) {
      Browser browser = playwright.chromium().connect(wsUrl, new BrowserType.ConnectOptions().setHeaders(Map.of("x-api-key", KEY)));
      try {
        Page page = browser.newPage();
        // The app can take 30 s to render through a rotating residential exit: wait up to 90 s.
        page.setDefaultTimeout(90000);

        // 1. Log in like a person; the site stores its token and sends it with every API call.
        page.navigate(SITE + "/auth/login");
        page.fill("#email", EMAIL);
        page.fill("#password", PASSWORD);
        page.click("[type=submit]");
        page.waitForURL("**/account");

        // 2. Find the token twice: in localStorage, and in the header of the page's own API request.
        String stored = (String) page.evaluate("() => localStorage.getItem('auth-token')");
        AtomicInteger uiCalls = new AtomicInteger();
        java.util.function.Consumer<Request> countCall = r -> { if (r.url().startsWith(API + "/invoices")) uiCalls.incrementAndGet(); };
        page.onRequest(countCall);
        Request firstCall = page.waitForRequest(r -> r.url().startsWith(API + "/invoices"), () -> page.navigate(SITE + "/account/invoices"));
        String header = firstCall.allHeaders().getOrDefault("authorization", "");
        JsonObject claims = JsonParser.parseString(new String(Base64.getUrlDecoder().decode(stored.split("\\.")[1]), StandardCharsets.UTF_8)).getAsJsonObject();

        // 3. The page's way: click through the pager, reading the table each time.
        page.waitForSelector("table tbody tr");
        long t0 = System.nanoTime();
        List<String> viaUi = new ArrayList<>(readTable(page));
        int lastPage = Integer.parseInt(page.locator("button[aria-label^=\"Page-\"]").last().innerText().trim());
        for (int n = 2; n <= lastPage; n++) {
          String want = API + "/invoices?page=" + n;
          page.waitForResponse(r -> r.url().equals(want), () -> page.click("[data-test=\"pagination-next\"]"));
          page.waitForFunction("first => document.querySelector('table tbody tr td')?.textContent.trim() !== first", viaUi.get(viaUi.size() - 15));
          viaUi.addAll(readTable(page));
        }
        double uiSeconds = (System.nanoTime() - t0) / 1e9;
        page.offRequest(countCall);
        int columns = page.locator("table thead th").count();

        // 4. The API's way: the same token on fetch() from inside the page — the browser makes the
        //    requests, through the session's proxy.
        JsonObject viaApi = JsonParser.parseString((String) page.evaluate(FETCH_ALL, Map.of("api", API, "token", stored))).getAsJsonObject();
        List<String> apiInvoices = new ArrayList<>();
        for (JsonElement e : viaApi.getAsJsonArray("invoices")) apiInvoices.add(e.getAsString());
        List<String> a = new ArrayList<>(viaUi), b = new ArrayList<>(apiInvoices);
        Collections.sort(a);
        Collections.sort(b);

        JsonObject token = new JsonObject();
        token.addProperty("in_local_storage", stored != null ? "auth-token" : null);
        token.addProperty("header_matches", header.equals("Bearer " + stored));
        token.addProperty("lifetime_seconds", claims.get("exp").getAsLong() - claims.get("iat").getAsLong());
        token.addProperty("seconds_left_at_end", claims.get("exp").getAsLong() - System.currentTimeMillis() / 1000);
        JsonObject api = new JsonObject();
        api.add("total", viaApi.get("total"));
        api.add("per_page", viaApi.get("perPage"));
        api.add("last_page", viaApi.get("lastPage"));
        api.add("without_token", viaApi.get("noToken"));
        JsonObject ui = new JsonObject();
        ui.addProperty("pages", lastPage);
        ui.addProperty("invoices", viaUi.size());
        ui.addProperty("invoice_calls", uiCalls.get());
        ui.addProperty("columns", columns);
        ui.addProperty("seconds", uiSeconds);
        JsonObject direct = new JsonObject();
        direct.add("requests", viaApi.get("requests"));
        direct.add("statuses", viaApi.get("statuses"));
        direct.addProperty("invoices", apiInvoices.size());
        direct.add("fields_per_invoice", viaApi.get("fields"));
        direct.add("json_bytes", viaApi.get("bytes"));
        direct.add("seconds_one_by_one", viaApi.get("sequential"));
        direct.add("seconds_all_at_once", viaApi.get("parallel"));
        JsonObject out = new JsonObject();
        out.add("token", token);
        out.add("api", api);
        out.add("ui", ui);
        out.add("direct", direct);
        out.addProperty("same_invoices", a.equals(b));
        Set<String> missing = new HashSet<>(viaUi);
        missing.removeAll(apiInvoices);
        out.addProperty("ui_missing_from_api", missing.size());
        System.out.println(out);
      } finally {
        browser.close();
      }
    }
  }
}
```

```csharp
// dotnet add package Microsoft.Playwright --version 1.60.0
// env: CDPFLEET_API_KEY, PROXY_URL
using System.Diagnostics;
using System.Net.Http.Json;
using System.Text;
using System.Text.Json;
using System.Text.Json.Nodes;
using Microsoft.Playwright;

var key = Environment.GetEnvironmentVariable("CDPFLEET_API_KEY")!;
const string Site = "https://practicesoftwaretesting.com"; // Toolshop: a public demo shop built for test automation
const string Api = "https://api.practicesoftwaretesting.com";
const string Email = "customer@practicesoftwaretesting.com", Password = "welcome01"; // the site's published demo account

// Fetches every page of the API from inside the page with the token: one by one, then all at once.
const string FetchAll = """
async ({ api, token }) => {
  const get = (n) => fetch(`${api}/invoices?page=${n}`, { headers: { authorization: `Bearer ${token}` } }).then(async (r) => ({ status: r.status, text: await r.text() }));
  const meta = JSON.parse((await get(1)).text);
  const numbers = Array.from({ length: meta.last_page }, (_, i) => i + 1);
  let t = performance.now();
  const pages = [];
  for (const n of numbers) pages.push(await get(n));
  const sequential = (performance.now() - t) / 1000;
  t = performance.now();
  await Promise.all(numbers.map(get));
  const parallel = (performance.now() - t) / 1000;
  const noToken = await fetch(`${api}/invoices?page=1`).then((r) => r.status);
  const items = pages.flatMap((p) => JSON.parse(p.text).data);
  return JSON.stringify({
    sequential, parallel, total: meta.total, perPage: meta.per_page, lastPage: meta.last_page,
    requests: pages.length, bytes: pages.reduce((s, p) => s + p.text.length, 0), statuses: [...new Set(pages.map((p) => p.status))],
    invoices: items.map((i) => i.invoice_number), fields: Object.keys(items[0]).length, noToken,
  });
}
""";

using var http = new HttpClient();
http.DefaultRequestHeaders.Add("x-api-key", key);

// 503 means the fleet is momentarily busy: wait Retry-After and try again.
async Task<JsonElement> Launch(JsonObject body)
{
    for (var attempt = 1; ; attempt++)
    {
        var res = await http.PostAsync("https://starter.cdpfleet.com/chromium/session",
            new StringContent(body.ToJsonString(), Encoding.UTF8, "application/json"));
        if (res.IsSuccessStatusCode) return await res.Content.ReadFromJsonAsync<JsonElement>();
        if ((int)res.StatusCode != 503 || attempt == 5) throw new Exception($"launch {(int)res.StatusCode} {await res.Content.ReadAsStringAsync()}");
        await Task.Delay((int)(res.Headers.RetryAfter?.Delta?.TotalMilliseconds ?? 5000));
    }
}

var session = await Launch(new JsonObject { ["proxy"] = Environment.GetEnvironmentVariable("PROXY_URL"), ["headless"] = "new", ["record"] = false });
using var playwright = await Playwright.CreateAsync();
var browser = await playwright.Chromium.ConnectAsync(session.GetProperty("wsUrl").GetString()!, new() { Headers = new Dictionary<string, string> { ["x-api-key"] = key } });
try
{
    var page = await browser.NewPageAsync();
    // The app can take 30 s to render through a rotating residential exit: wait up to 90 s.
    page.SetDefaultTimeout(90000);

    // 1. Log in like a person; the site stores its token and sends it with every API call.
    await page.GotoAsync($"{Site}/auth/login");
    await page.FillAsync("#email", Email);
    await page.FillAsync("#password", Password);
    await page.ClickAsync("[type=submit]");
    await page.WaitForURLAsync("**/account");

    // 2. Find the token twice: in localStorage, and in the header of the page's own API request.
    var stored = await page.EvaluateAsync<string>("() => localStorage.getItem('auth-token')");
    var uiCalls = 0;
    void CountCall(object? _, IRequest r) { if (r.Url.StartsWith($"{Api}/invoices")) uiCalls++; }
    page.Request += CountCall;
    var firstCall = await page.RunAndWaitForRequestAsync(() => page.GotoAsync($"{Site}/account/invoices"), r => r.Url.StartsWith($"{Api}/invoices"));
    var header = (await firstCall.AllHeadersAsync()).GetValueOrDefault("authorization", "");
    var payload = stored.Split('.')[1].Replace('-', '+').Replace('_', '/');
    payload = payload.PadRight(payload.Length + (4 - payload.Length % 4) % 4, '=');
    var claims = JsonDocument.Parse(Convert.FromBase64String(payload)).RootElement;

    // 3. The page's way: click through the pager, reading the table each time.
    await page.WaitForSelectorAsync("table tbody tr");
    var clock = Stopwatch.StartNew();
    async Task<string[]> ReadTable() => await page.EvalOnSelectorAllAsync<string[]>("table tbody tr td:first-child", "tds => tds.map((td) => td.textContent.trim())");
    var viaUi = new List<string>(await ReadTable());
    var lastPage = int.Parse((await page.Locator("button[aria-label^=\"Page-\"]").Last.InnerTextAsync()).Trim());
    for (var n = 2; n <= lastPage; n++)
    {
        var want = $"{Api}/invoices?page={n}";
        await page.RunAndWaitForResponseAsync(() => page.ClickAsync("[data-test=\"pagination-next\"]"), r => r.Url == want);
        await page.WaitForFunctionAsync("first => document.querySelector('table tbody tr td')?.textContent.trim() !== first", viaUi[^15]);
        viaUi.AddRange(await ReadTable());
    }
    var uiSeconds = clock.Elapsed.TotalSeconds;
    page.Request -= CountCall;
    var columns = await page.Locator("table thead th").CountAsync();

    // 4. The API's way: the same token on fetch() from inside the page — the browser makes the
    //    requests, through the session's proxy.
    var viaApi = JsonDocument.Parse(await page.EvaluateAsync<string>(FetchAll, new { api = Api, token = stored })).RootElement;
    var apiInvoices = viaApi.GetProperty("invoices").EnumerateArray().Select((e) => e.GetString()!).ToList();

    var output = new JsonObject
    {
        ["token"] = new JsonObject
        {
            ["in_local_storage"] = stored != null ? "auth-token" : null,
            ["header_matches"] = header == $"Bearer {stored}",
            ["lifetime_seconds"] = claims.GetProperty("exp").GetInt64() - claims.GetProperty("iat").GetInt64(),
            ["seconds_left_at_end"] = claims.GetProperty("exp").GetInt64() - DateTimeOffset.UtcNow.ToUnixTimeSeconds(),
        },
        ["api"] = new JsonObject
        {
            ["total"] = viaApi.GetProperty("total").GetInt32(), ["per_page"] = viaApi.GetProperty("perPage").GetInt32(),
            ["last_page"] = viaApi.GetProperty("lastPage").GetInt32(), ["without_token"] = viaApi.GetProperty("noToken").GetInt32(),
        },
        ["ui"] = new JsonObject { ["pages"] = lastPage, ["invoices"] = viaUi.Count, ["invoice_calls"] = uiCalls, ["columns"] = columns, ["seconds"] = uiSeconds },
        ["direct"] = new JsonObject
        {
            ["requests"] = viaApi.GetProperty("requests").GetInt32(),
            ["statuses"] = JsonNode.Parse(viaApi.GetProperty("statuses").GetRawText()),
            ["invoices"] = apiInvoices.Count,
            ["fields_per_invoice"] = viaApi.GetProperty("fields").GetInt32(),
            ["json_bytes"] = viaApi.GetProperty("bytes").GetInt64(),
            ["seconds_one_by_one"] = viaApi.GetProperty("sequential").GetDouble(),
            ["seconds_all_at_once"] = viaApi.GetProperty("parallel").GetDouble(),
        },
        ["same_invoices"] = viaUi.Count == apiInvoices.Count && viaUi.Order().SequenceEqual(apiInvoices.Order()),
        ["ui_missing_from_api"] = viaUi.Distinct().Except(apiInvoices).Count(),
    };
    Console.WriteLine(output.ToJsonString());
}
finally
{
    await browser.CloseAsync();
}
```

```go
// go get github.com/playwright-community/playwright-go@v0.6000.0
// Driver: build playwright-core 1.60.0 from npm and set PLAYWRIGHT_DRIVER_PATH (see /docs/quickstart).
// env: CDPFLEET_API_KEY, PROXY_URL
package main

import (
	"bytes"
	"encoding/base64"
	"encoding/json"
	"fmt"
	"io"
	"log"
	"net/http"
	"os"
	"regexp"
	"sort"
	"strconv"
	"strings"
	"sync/atomic"
	"time"

	"github.com/playwright-community/playwright-go"
)

var key = os.Getenv("CDPFLEET_API_KEY")

const (
	site     = "https://practicesoftwaretesting.com" // Toolshop: a public demo shop built for test automation
	api      = "https://api.practicesoftwaretesting.com"
	email    = "customer@practicesoftwaretesting.com" // the site's published demo account
	password = "welcome01"
)

// Fetches every page of the API from inside the page with the token: one by one, then all at once.
// (No JS template literals: Go raw strings can't hold backticks.)
const fetchAll = `async ({ api, token }) => {
  const get = (n) => fetch(api + '/invoices?page=' + n, { headers: { authorization: 'Bearer ' + token } }).then(async (r) => ({ status: r.status, text: await r.text() }));
  const meta = JSON.parse((await get(1)).text);
  const numbers = Array.from({ length: meta.last_page }, (_, i) => i + 1);
  let t = performance.now();
  const pages = [];
  for (const n of numbers) pages.push(await get(n));
  const sequential = (performance.now() - t) / 1000;
  t = performance.now();
  await Promise.all(numbers.map(get));
  const parallel = (performance.now() - t) / 1000;
  const noToken = await fetch(api + '/invoices?page=1').then((r) => r.status);
  const items = pages.flatMap((p) => JSON.parse(p.text).data);
  return JSON.stringify({
    sequential, parallel, total: meta.total, perPage: meta.per_page, lastPage: meta.last_page,
    requests: pages.length, bytes: pages.reduce((s, p) => s + p.text.length, 0), statuses: [...new Set(pages.map((p) => p.status))],
    invoices: items.map((i) => i.invoice_number), fields: Object.keys(items[0]).length, noToken,
  });
}`

// 503 means the fleet is momentarily busy: wait Retry-After and try again.
func launch(opts map[string]any) (map[string]any, error) {
	body, _ := json.Marshal(opts)
	for attempt := 1; ; attempt++ {
		req, _ := http.NewRequest("POST", "https://starter.cdpfleet.com/chromium/session", bytes.NewReader(body))
		req.Header.Set("x-api-key", key)
		req.Header.Set("content-type", "application/json")
		res, err := http.DefaultClient.Do(req)
		if err != nil {
			return nil, err
		}
		b, _ := io.ReadAll(res.Body)
		res.Body.Close()
		if res.StatusCode == 200 {
			var session map[string]any
			return session, json.Unmarshal(b, &session)
		}
		if res.StatusCode != 503 || attempt == 5 {
			return nil, fmt.Errorf("launch %d %s", res.StatusCode, strings.TrimSpace(string(b)))
		}
		wait, _ := strconv.Atoi(res.Header.Get("retry-after"))
		if wait == 0 {
			wait = 5
		}
		time.Sleep(time.Duration(wait) * time.Second)
	}
}

func must(err error) {
	if err != nil {
		log.Fatal(err)
	}
}

func readTable(page playwright.Page) []string {
	cells, err := page.EvalOnSelectorAll("table tbody tr td:first-child", "tds => tds.map((td) => td.textContent.trim())")
	must(err)
	var out []string
	for _, c := range cells.([]any) {
		out = append(out, c.(string))
	}
	return out
}

func main() {
	session, err := launch(map[string]any{"proxy": os.Getenv("PROXY_URL"), "headless": "new", "record": false})
	must(err)
	pw, err := playwright.Run(&playwright.RunOptions{SkipInstallBrowsers: true})
	must(err)
	defer pw.Stop()
	browser, err := pw.Chromium.Connect(session["wsUrl"].(string), playwright.BrowserTypeConnectOptions{Headers: map[string]string{"x-api-key": key}})
	must(err)
	defer browser.Close()

	page, err := browser.NewPage()
	must(err)
	// The app can take 30 s to render through a rotating residential exit: wait up to 90 s.
	page.SetDefaultTimeout(90000)

	// 1. Log in like a person; the site stores its token and sends it with every API call.
	_, err = page.Goto(site + "/auth/login")
	must(err)
	must(page.Fill("#email", email))
	must(page.Fill("#password", password))
	must(page.Click("[type=submit]"))
	must(page.WaitForURL("**/account"))

	// 2. Find the token twice: in localStorage, and in the header of the page's own API request.
	storedAny, err := page.Evaluate("() => localStorage.getItem('auth-token')")
	must(err)
	stored, _ := storedAny.(string)
	var uiCalls, counting int32 = 0, 1
	page.OnRequest(func(r playwright.Request) {
		if atomic.LoadInt32(&counting) == 1 && strings.HasPrefix(r.URL(), api+"/invoices") {
			atomic.AddInt32(&uiCalls, 1)
		}
	})
	firstCall, err := page.ExpectRequest(regexp.MustCompile(`^`+regexp.QuoteMeta(api+"/invoices")), func() error {
		_, err := page.Goto(site + "/account/invoices")
		return err
	})
	must(err)
	headers, err := firstCall.AllHeaders()
	must(err)
	raw, err := base64.RawURLEncoding.DecodeString(strings.Split(stored, ".")[1])
	must(err)
	var claims struct {
		Exp int64 `json:"exp"`
		Iat int64 `json:"iat"`
	}
	must(json.Unmarshal(raw, &claims))

	// 3. The page's way: click through the pager, reading the table each time.
	_, err = page.WaitForSelector("table tbody tr")
	must(err)
	t0 := time.Now()
	viaUi := readTable(page)
	lastText, err := page.Locator(`button[aria-label^="Page-"]`).Last().InnerText()
	must(err)
	lastPage, _ := strconv.Atoi(strings.TrimSpace(lastText))
	for n := 2; n <= lastPage; n++ {
		_, err := page.ExpectResponse(fmt.Sprintf("%s/invoices?page=%d", api, n), func() error {
			return page.Click(`[data-test="pagination-next"]`)
		})
		must(err)
		_, err = page.WaitForFunction("first => document.querySelector('table tbody tr td')?.textContent.trim() !== first", viaUi[len(viaUi)-15])
		must(err)
		viaUi = append(viaUi, readTable(page)...)
	}
	uiSeconds := time.Since(t0).Seconds()
	atomic.StoreInt32(&counting, 0)
	columns, err := page.Locator("table thead th").Count()
	must(err)

	// 4. The API's way: the same token on fetch() from inside the page — the browser makes the
	//    requests, through the session's proxy.
	res, err := page.Evaluate(fetchAll, map[string]any{"api": api, "token": stored})
	must(err)
	var viaApi struct {
		Sequential float64  `json:"sequential"`
		Parallel   float64  `json:"parallel"`
		Total      int      `json:"total"`
		PerPage    int      `json:"perPage"`
		LastPage   int      `json:"lastPage"`
		Requests   int      `json:"requests"`
		Bytes      int      `json:"bytes"`
		Statuses   []int    `json:"statuses"`
		Invoices   []string `json:"invoices"`
		Fields     int      `json:"fields"`
		NoToken    int      `json:"noToken"`
	}
	must(json.Unmarshal([]byte(res.(string)), &viaApi))
	a := append([]string{}, viaUi...)
	b := append([]string{}, viaApi.Invoices...)
	sort.Strings(a)
	sort.Strings(b)

	inApi := map[string]bool{}
	for _, n := range viaApi.Invoices {
		inApi[n] = true
	}
	seen := map[string]bool{}
	missing := 0
	for _, n := range viaUi {
		if !inApi[n] && !seen[n] {
			missing++
		}
		seen[n] = true
	}
	var inStorage any
	if stored != "" {
		inStorage = "auth-token"
	}
	out, _ := json.Marshal(map[string]any{
		"token": map[string]any{
			"in_local_storage":    inStorage,
			"header_matches":      headers["authorization"] == "Bearer "+stored,
			"lifetime_seconds":    claims.Exp - claims.Iat,
			"seconds_left_at_end": claims.Exp - time.Now().Unix(),
		},
		"api": map[string]any{"total": viaApi.Total, "per_page": viaApi.PerPage, "last_page": viaApi.LastPage, "without_token": viaApi.NoToken},
		"ui":  map[string]any{"pages": lastPage, "invoices": len(viaUi), "invoice_calls": atomic.LoadInt32(&uiCalls), "columns": columns, "seconds": uiSeconds},
		"direct": map[string]any{
			"requests": viaApi.Requests, "statuses": viaApi.Statuses, "invoices": len(viaApi.Invoices), "fields_per_invoice": viaApi.Fields,
			"json_bytes": viaApi.Bytes, "seconds_one_by_one": viaApi.Sequential, "seconds_all_at_once": viaApi.Parallel,
		},
		"same_invoices":       strings.Join(a, ",") == strings.Join(b, ","),
		"ui_missing_from_api": missing,
	})
	fmt.Println(string(out))
}
```

## What we got

| Way | Invoices | /invoices calls | Data per invoice | Time (s) |
| --- | --- | --- | --- | --- |
| Click Next through the table | 162 | 11 | 5 columns | 5.5 |
| API with the token, one by one | 162 | 11 | 16 fields | 4.2 |
| API with the token, all at once | 162 | 11 | 16 fields | 0.8 |
| API without the token | 0 | 1 | HTTP 401 |  |

From the Node.js run on 2026-10-18. IP addresses are replaced with placeholders (203.0.113.x); equal addresses stay equal. The other languages produced the same findings.

Raw output (Node.js)

```
{
  "token": {
    "in_local_storage": "auth-token",
    "header_matches": true,
    "lifetime_seconds": 300,
    "seconds_left_at_end": 287
  },
  "api": {
    "total": 162,
    "per_page": 15,
    "last_page": 11,
    "without_token": 401
  },
  "ui": {
    "pages": 11,
    "invoices": 162,
    "invoice_calls": 11,
    "columns": 5,
    "seconds": 5.479
  },
  "direct": {
    "requests": 11,
    "statuses": [
      200
    ],
    "invoices": 162,
    "fields_per_invoice": 16,
    "json_bytes": 406666,
    "seconds_one_by_one": 4.1995,
    "seconds_all_at_once": 0.7525
  },
  "same_invoices": true,
  "ui_missing_from_api": 0
}
```

## Takeaways

-   **The token is in the browser after login:** the app stored a JWT under `auth-token` in localStorage and sent exactly that as `Authorization: Bearer …` on its own API calls. Without it the API answered **401**.
-   **Same records, more data:** the API returned every invoice the table showed, about 160 of them. The page showed 5 columns; the API returned **16 fields** per invoice, including the invoice lines and payment — about 400 KB of JSON.
-   **Page numbers over a list that changes:** the demo account is shared with everyone practising on the site, and invoices were added while the programs ran: while building this case, one run counted 161 in the table and 162 from the API, another got 164 entries from 11 API pages for 163 invoices: a new invoice at the top pushed one onto the next page, so it was read twice. De-duplicate by id when you page by number, and expect to miss what moves the other way.
-   **The page size is the server's:** 15 per page — 11 or 12 pages as the list grew — and `per_page` or `limit` parameters were ignored, so the API needs as many requests as the clicks do: one `/invoices` call per page either way.
-   **The win is parallelism, not skipping the UI:** fetched one by one, the 11 API pages took about as long as clicking Next through the pages — several seconds, each request a round trip through the residential exit — but **all at once they took about a second**. Without a token you could only click.
-   **Tokens expire:** this one lived **300 seconds** (`exp − iat` in the JWT). Check `exp` before a long job and log in again — or read the fresh token the page keeps — when it runs out.
-   Compare with [infinite scroll, two ways](https://cdpfleet.com/docs/cases/infinite-scroll), where the page's API needs no token, and [fetch first](https://cdpfleet.com/docs/cases/fetch-first), which skips the browser when the HTML already has the data.

---
Source: https://cdpfleet.com/docs/cases/scrape-a-paginated-json-api-with-a-token · Markdown: https://cdpfleet.com/docs/cases/scrape-a-paginated-json-api-with-a-token.md · All docs for AI tools: https://cdpfleet.com/llms.txt