[Examples & benchmarks](https://cdpfleet.com/docs/cases) / #41 · 2026-10-13 · Example · Easy · for [web scraping](https://cdpfleet.com/web-scraping), [ai agents](https://cdpfleet.com/ai-agents)

# A scrape broke: find out why from its session recording

A scraper dies with a timeout. Its session was recorded anyway: fetch the video by session id and see the page it failed on, with no tracing set up in advance.

[Chromium](https://cdpfleet.com/docs/browsers/chromium)

Run on production on 2026-10-13: ✓ Node.js ✓ Python ✓ Java ✓ C# ✓ Go

## The problem

A scraper that ran fine for weeks fails with "Timeout 5000ms exceeded" and nothing else. Was the selector wrong, did the site change, was there a captcha, a cookie banner, an error page? Playwright tracing would tell you — if you had turned it on before the run. cdpfleet [records sessions](https://cdpfleet.com/docs/recordings) without any setup: headful sessions are recorded from their display by default, so the video of the failed run already exists. Here a scraper breaks on purpose, and we fetch, check and delete its recording through the API.

## What we used, and why

| What | Why |
| --- | --- |
| `"headless": false` | Headful, the default. Headful sessions are recorded from the browser's display; headless ones only with `cdp` or `live_view`. |
| `viewport: null` | The page fills the whole display instead of a 1280×720 window on it — so it fills the video too (Python `no_viewport=True`, Java `setViewportSize(null)`, C# `ViewportSize.NoViewport`, Go `NoViewport`). |
| [quotes.toscrape.com](https://quotes.toscrape.com/) | A public practice site with ten listing pages. The scraper starts on page 8 and assumes every page has a "Next" link — page 10, the last one, doesn't. |
| `GET /v1/me/recordings` | Lists the account's recordings with their `session_id`, duration, size and resolution. A recording appears once the node has uploaded it, after the session ended. |
| `GET /v1/me/recordings/{id}/url` | A short-lived private link to the MP4. `DELETE /v1/me/recordings/{id}` deletes it. |
| `"record": false` | A second session launched with recording off, to show it gets no video. |

## How it works

1.  Launch a session with `"record": false`, load the site, close it.
2.  Launch the scraper's session (headful, recorded) and keep its `sessionId`. From page 8, collect the quotes and follow the "Next" link — 5 seconds to find it, then load its page — until the link can't be found; note the time since launch, the URL and what the page has.
3.  Close the browser, which ends the session. Poll `GET /v1/me/recordings` until a recording with that `session_id` appears.
4.  Get its link, download the MP4, compare its duration with the moment of failure, then delete it.
5.  Wait another minute and check that the `record: false` session never got a recording.

## The code

The same program in five languages (also [on GitHub](https://github.com/cdpfleet/cdpfleet/tree/main/cases/41-replay-a-failed-run), with the raw output). Set these environment variables first:

-   `CDPFLEET_API_KEY` — your API key (dashboard → API keys)
-   `PROXY_URL` — your proxy, e.g. `http://user:pass@proxy.example.com:8000`

 

```node
// npm install playwright@1.60.0
// env: CDPFLEET_API_KEY, PROXY_URL
import { chromium } from 'playwright';

const KEY = process.env.CDPFLEET_API_KEY;
const SITE = 'https://quotes.toscrape.com';
const NEXT = 'li.next a'; // the scraper assumes every listing page has a "Next" link

async function api(method, path) {
  const res = await fetch(`https://cdpfleet.com${path}`, { method, headers: { 'x-api-key': KEY } });
  return { status: res.status, body: await res.json().catch(() => ({})) };
}

// Headful (the default) sessions are recorded from the browser's display.
async function launch(extra = {}) {
  const res = await fetch('https://starter.cdpfleet.com/chromium/session', {
    method: 'POST',
    headers: { 'x-api-key': KEY, 'content-type': 'application/json' },
    body: JSON.stringify({ proxy: process.env.PROXY_URL, headless: false, ...extra }),
  });
  if (!res.ok) throw new Error(`launch: ${res.status} ${await res.text()}`);
  const { sessionId, wsUrl } = await res.json();
  const started = Date.now();
  const browser = await chromium.connect(wsUrl, { headers: { 'x-api-key': KEY } });
  // viewport: null — the page fills the whole display, so it fills the recording too.
  const page = await (await browser.newContext({ viewport: null })).newPage();
  return { sessionId, started, browser, page };
}

// Proxies drop a tunnel now and then: up to two retries.
async function go(page, url) {
  for (let attempt = 1; ; attempt++) {
    try {
      return await page.goto(url, { timeout: 60000 });
    } catch (err) {
      if (attempt === 3) throw err;
      await new Promise((r) => setTimeout(r, 2000));
    }
  }
}

// The recording appears once the node has uploaded it, after the session ended.
async function waitForRecording(sessionId, maxSeconds) {
  const t = Date.now();
  while ((Date.now() - t) / 1000 < maxSeconds) {
    const rec = (await api('GET', '/v1/me/recordings')).body.recordings?.find((r) => r.session_id === sessionId);
    if (rec) return { rec, waited_s: Math.round((Date.now() - t) / 1000) };
    await new Promise((r) => setTimeout(r, 5000));
  }
  return { rec: null, waited_s: Math.round((Date.now() - t) / 1000) };
}

const out = { site: SITE, scrape: {}, recording: {}, unrecorded: {} };

// A session launched with record: false, for comparison: it must never get a recording.
const quiet = await launch({ record: false });
try {
  await go(quiet.page, SITE);
} finally {
  await quiet.browser.close();
}
out.unrecorded.session_id = quiet.sessionId;

// The scrape: pages 8, 9, 10 — follow "Next" on each. Page 10 is the last one and has none.
const s = await launch();
out.session_id = s.sessionId;
const quotes = [];
try {
  await go(s.page, `${SITE}/page/8/`);
  for (let pageNo = 8; ; pageNo++) {
    quotes.push(...await s.page.locator('.quote .text').allTextContents());
    out.scrape.last_page = pageNo;
    let href;
    try {
      href = await s.page.locator(NEXT).getAttribute('href', { timeout: 5000 }); // 5 s to find the link
    } catch (err) {
      out.scrape.failed_at_s = Math.round((Date.now() - s.started) / 100) / 10;
      out.scrape.error = `${err.name === 'TimeoutError' ? 'timeout' : err.name} waiting for ${NEXT}`;
      break;
    }
    await go(s.page, new URL(href, SITE).href); // page loads keep their own timeout and retries
  }
  out.scrape.quotes = quotes.length;
  // What the video shows at that moment: the last page — a "Previous" link and no "Next".
  out.scrape.url_at_failure = s.page.url();
  out.scrape.next_links = await s.page.locator('li.next').count();
  out.scrape.previous_links = await s.page.locator('li.previous').count();
} finally {
  await s.browser.close(); // ends the session; the node uploads the video afterwards
}
out.session_seconds = Math.round((Date.now() - s.started) / 100) / 10;

// Find the recording, check it covers the failure, download it, delete it.
const { rec, waited_s: waited } = await waitForRecording(s.sessionId, 600);
if (rec) {
  const link = (await api('GET', `/v1/me/recordings/${rec.id}/url`)).body;
  const video = await fetch(link.url);
  const bytes = Buffer.from(await video.arrayBuffer());
  out.recording = {
    appeared_after_s: waited,
    duration_seconds: rec.duration_seconds,
    width: rec.width,
    height: rec.height,
    bytes: rec.bytes,
    downloaded_bytes: bytes.length,
    content_type: video.headers.get('content-type'),
    mp4: bytes.subarray(4, 8).toString() === 'ftyp',
    link_expires_in_s: link.expires_in_s,
  };
  out.failure_within_recording = out.scrape.failed_at_s <= rec.duration_seconds + 1;
  out.recording.deleted = (await api('DELETE', `/v1/me/recordings/${rec.id}`)).status === 200;
}

// By now the scrape's video has been uploaded; give the unrecorded session another minute.
await new Promise((r) => setTimeout(r, 60000));
const list = (await api('GET', '/v1/me/recordings')).body.recordings || [];
out.unrecorded.recording_found = list.some((r) => r.session_id === quiet.sessionId);
console.log(JSON.stringify(out, null, 2));
```

```python
# pip install playwright==1.60.0 requests
# env: CDPFLEET_API_KEY, PROXY_URL
import json
import os
import time
from urllib.parse import urljoin

import requests
from playwright.sync_api import Error as PlaywrightError, TimeoutError as PlaywrightTimeout, sync_playwright

KEY = os.environ["CDPFLEET_API_KEY"]
SITE = "https://quotes.toscrape.com"
NEXT = "li.next a"  # the scraper assumes every listing page has a "Next" link

def api(method, path):
    res = requests.request(method, f"https://cdpfleet.com{path}", headers={"x-api-key": KEY}, timeout=30)
    try:
        body = res.json()
    except ValueError:
        body = {}
    return res.status_code, body

# Headful (the default) sessions are recorded from the browser's display.
def launch(p, **extra):
    res = requests.post(
        "https://starter.cdpfleet.com/chromium/session",
        headers={"x-api-key": KEY},
        json={"proxy": os.environ["PROXY_URL"], "headless": False, **extra},
        timeout=60,
    )
    if not res.ok:
        raise RuntimeError(f"launch: {res.status_code} {res.text}")
    body = res.json()
    started = time.time()
    browser = p.chromium.connect(body["wsUrl"], headers={"x-api-key": KEY})
    # no_viewport: the page fills the whole display, so it fills the recording too.
    page = browser.new_context(no_viewport=True).new_page()
    return body["sessionId"], started, browser, page

# Proxies drop a tunnel now and then: up to two retries.
def go(page, url):
    for attempt in range(1, 4):
        try:
            return page.goto(url, timeout=60000)
        except PlaywrightError:
            if attempt == 3:
                raise
            time.sleep(2)

# The recording appears once the node has uploaded it, after the session ended.
def wait_for_recording(session_id, max_seconds):
    t = time.time()
    while time.time() - t < max_seconds:
        recs = api("GET", "/v1/me/recordings")[1].get("recordings") or []
        rec = next((r for r in recs if r["session_id"] == session_id), None)
        if rec:
            return rec, round(time.time() - t)
        time.sleep(5)
    return None, round(time.time() - t)

out = {"site": SITE, "scrape": {}, "recording": {}, "unrecorded": {}}
with sync_playwright() as p:
    # A session launched with record: false, for comparison: it must never get a recording.
    quiet_id, _, quiet_browser, quiet_page = launch(p, record=False)
    try:
        go(quiet_page, SITE)
    finally:
        quiet_browser.close()
    out["unrecorded"]["session_id"] = quiet_id

    # The scrape: pages 8, 9, 10 — follow "Next" on each. Page 10 is the last one and has none.
    session_id, started, browser, page = launch(p)
    out["session_id"] = session_id
    quotes = []
    try:
        go(page, f"{SITE}/page/8/")
        page_no = 8
        while True:
            quotes += page.locator(".quote .text").all_text_contents()
            out["scrape"]["last_page"] = page_no
            try:
                href = page.locator(NEXT).get_attribute("href", timeout=5000)  # 5 s to find the link
            except PlaywrightError as err:
                out["scrape"]["failed_at_s"] = round(time.time() - started, 1)
                kind = "timeout" if isinstance(err, PlaywrightTimeout) else type(err).__name__
                out["scrape"]["error"] = f"{kind} waiting for {NEXT}"
                break
            go(page, urljoin(SITE, href))  # page loads keep their own timeout and retries
            page_no += 1
        out["scrape"]["quotes"] = len(quotes)
        # What the video shows at that moment: the last page — a "Previous" link and no "Next".
        out["scrape"]["url_at_failure"] = page.url
        out["scrape"]["next_links"] = page.locator("li.next").count()
        out["scrape"]["previous_links"] = page.locator("li.previous").count()
    finally:
        browser.close()  # ends the session; the node uploads the video afterwards
    out["session_seconds"] = round(time.time() - started, 1)

# Find the recording, check it covers the failure, download it, delete it.
rec, waited = wait_for_recording(session_id, 600)
if rec:
    link = api("GET", f"/v1/me/recordings/{rec['id']}/url")[1]
    video = requests.get(link["url"], timeout=60)
    out["recording"] = {
        "appeared_after_s": waited,
        "duration_seconds": rec["duration_seconds"],
        "width": rec["width"],
        "height": rec["height"],
        "bytes": rec["bytes"],
        "downloaded_bytes": len(video.content),
        "content_type": video.headers.get("content-type"),
        "mp4": video.content[4:8] == b"ftyp",
        "link_expires_in_s": link["expires_in_s"],
    }
    out["failure_within_recording"] = out["scrape"]["failed_at_s"] <= rec["duration_seconds"] + 1
    out["recording"]["deleted"] = api("DELETE", f"/v1/me/recordings/{rec['id']}")[0] == 200

# By now the scrape's video has been uploaded; give the unrecorded session another minute.
time.sleep(60)
recs = api("GET", "/v1/me/recordings")[1].get("recordings") or []
out["unrecorded"]["recording_found"] = any(r["session_id"] == quiet_id for r in recs)
print(json.dumps(out, indent=2))
```

```java
// Maven: com.microsoft.playwright:playwright:1.60.0, com.google.code.gson:gson:2.11.0
// Run with PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1. env: CDPFLEET_API_KEY, PROXY_URL
import com.google.gson.*;
import com.microsoft.playwright.*;
import java.net.URI;
import java.net.http.*;
import java.util.*;

public class Main {
  static final String KEY = System.getenv("CDPFLEET_API_KEY");
  static final String SITE = "https://quotes.toscrape.com";
  static final String NEXT = "li.next a"; // the scraper assumes every listing page has a "Next" link
  static final HttpClient HTTP = HttpClient.newHttpClient();

  static double since(long startedMs) { return Math.round((System.currentTimeMillis() - startedMs) / 100.0) / 10.0; }

  static HttpResponse<String> api(String method, String path) throws Exception {
    return HTTP.send(HttpRequest.newBuilder(URI.create("https://cdpfleet.com" + path)).header("x-api-key", KEY)
        .method(method, HttpRequest.BodyPublishers.noBody()).build(), HttpResponse.BodyHandlers.ofString());
  }

  static JsonArray recordings() throws Exception {
    JsonElement list = JsonParser.parseString(api("GET", "/v1/me/recordings").body()).getAsJsonObject().get("recordings");
    return list == null || list.isJsonNull() ? new JsonArray() : list.getAsJsonArray();
  }

  record Session(String id, long started, Browser browser, Page page) {}

  // Headful (the default) sessions are recorded from the browser's display.
  static Session launch(Playwright playwright, Boolean record) throws Exception {
    JsonObject req = new JsonObject();
    req.addProperty("proxy", System.getenv("PROXY_URL"));
    req.addProperty("headless", false);
    if (record != null) req.addProperty("record", record);
    HttpResponse<String> res = HTTP.send(HttpRequest.newBuilder(URI.create("https://starter.cdpfleet.com/chromium/session"))
        .header("x-api-key", KEY).header("content-type", "application/json")
        .POST(HttpRequest.BodyPublishers.ofString(req.toString())).build(), HttpResponse.BodyHandlers.ofString());
    if (res.statusCode() != 200) throw new RuntimeException("launch: " + res.statusCode() + " " + res.body());
    JsonObject body = JsonParser.parseString(res.body()).getAsJsonObject();
    long started = System.currentTimeMillis();
    Browser browser = playwright.chromium().connect(body.get("wsUrl").getAsString(),
        new BrowserType.ConnectOptions().setHeaders(Map.of("x-api-key", KEY)));
    // setViewportSize(null): the page fills the whole display, so it fills the recording too.
    Page page = browser.newContext(new Browser.NewContextOptions().setViewportSize(null)).newPage();
    return new Session(body.get("sessionId").getAsString(), started, browser, page);
  }

  // Proxies drop a tunnel now and then: up to two retries.
  static void go(Page page, String url) throws Exception {
    for (int attempt = 1; ; attempt++) {
      try {
        page.navigate(url, new Page.NavigateOptions().setTimeout(60000));
        return;
      } catch (PlaywrightException e) {
        if (attempt == 3) throw e;
        Thread.sleep(2000);
      }
    }
  }

  public static void main(String[] args) throws Exception {
    JsonObject out = new JsonObject();
    out.addProperty("site", SITE);
    JsonObject scrape = new JsonObject(), recording = new JsonObject(), unrecorded = new JsonObject();
    out.add("scrape", scrape);
    out.add("recording", recording);
    out.add("unrecorded", unrecorded);
    String sessionId;
    try (Playwright playwright = Playwright.create()) {
      // A session launched with record: false, for comparison: it must never get a recording.
      Session quiet = launch(playwright, false);
      try {
        go(quiet.page(), SITE);
      } finally {
        quiet.browser().close();
      }
      unrecorded.addProperty("session_id", quiet.id());

      // The scrape: pages 8, 9, 10 — follow "Next" on each. Page 10 is the last one and has none.
      Session s = launch(playwright, null);
      sessionId = s.id();
      out.addProperty("session_id", sessionId);
      List<String> quotes = new ArrayList<>();
      try {
        go(s.page(), SITE + "/page/8/");
        for (int pageNo = 8; ; pageNo++) {
          quotes.addAll(s.page().locator(".quote .text").allTextContents());
          scrape.addProperty("last_page", pageNo);
          String href;
          try {
            href = s.page().locator(NEXT).getAttribute("href", new Locator.GetAttributeOptions().setTimeout(5000)); // 5 s to find the link
          } catch (PlaywrightException e) {
            scrape.addProperty("failed_at_s", since(s.started()));
            scrape.addProperty("error", (e instanceof TimeoutError ? "timeout" : e.getClass().getSimpleName()) + " waiting for " + NEXT);
            break;
          }
          go(s.page(), URI.create(SITE).resolve(href).toString()); // page loads keep their own timeout and retries
        }
        scrape.addProperty("quotes", quotes.size());
        // What the video shows at that moment: the last page — a "Previous" link and no "Next".
        scrape.addProperty("url_at_failure", s.page().url());
        scrape.addProperty("next_links", s.page().locator("li.next").count());
        scrape.addProperty("previous_links", s.page().locator("li.previous").count());
      } finally {
        s.browser().close(); // ends the session; the node uploads the video afterwards
      }
      out.addProperty("session_seconds", since(s.started()));
    }

    // Find the recording, check it covers the failure, download it, delete it.
    long t = System.currentTimeMillis();
    JsonObject rec = null;
    while (rec == null && System.currentTimeMillis() - t < 600_000) {
      for (JsonElement r : recordings()) {
        if (r.getAsJsonObject().get("session_id").getAsString().equals(sessionId)) rec = r.getAsJsonObject();
      }
      if (rec == null) Thread.sleep(5000);
    }
    if (rec != null) {
      recording.addProperty("appeared_after_s", Math.round((System.currentTimeMillis() - t) / 1000.0));
      String id = rec.get("id").getAsString();
      JsonObject link = JsonParser.parseString(api("GET", "/v1/me/recordings/" + id + "/url").body()).getAsJsonObject();
      HttpResponse<byte[]> video = HTTP.send(HttpRequest.newBuilder(URI.create(link.get("url").getAsString())).build(),
          HttpResponse.BodyHandlers.ofByteArray());
      byte[] bytes = video.body();
      for (String k : List.of("duration_seconds", "width", "height", "bytes")) recording.add(k, rec.get(k));
      recording.addProperty("downloaded_bytes", bytes.length);
      recording.addProperty("content_type", video.headers().firstValue("content-type").orElse(null));
      recording.addProperty("mp4", bytes.length > 8 && new String(bytes, 4, 4).equals("ftyp"));
      recording.add("link_expires_in_s", link.get("expires_in_s"));
      out.addProperty("failure_within_recording", scrape.get("failed_at_s").getAsDouble() <= rec.get("duration_seconds").getAsDouble() + 1);
      recording.addProperty("deleted", api("DELETE", "/v1/me/recordings/" + id).statusCode() == 200);
    }

    // By now the scrape's video has been uploaded; give the unrecorded session another minute.
    Thread.sleep(60_000);
    boolean found = false;
    for (JsonElement r : recordings()) {
      if (r.getAsJsonObject().get("session_id").getAsString().equals(unrecorded.get("session_id").getAsString())) found = true;
    }
    unrecorded.addProperty("recording_found", found);
    System.out.println(new GsonBuilder().setPrettyPrinting().create().toJson(out));
  }
}
```

```csharp
// dotnet add package Microsoft.Playwright --version 1.60.0
// env: CDPFLEET_API_KEY, PROXY_URL
using System.Net.Http.Json;
using System.Text.Encodings.Web;
using System.Text.Json;
using System.Text.Json.Nodes;
using Microsoft.Playwright;

const string Site = "https://quotes.toscrape.com";
const string Next = "li.next a"; // the scraper assumes every listing page has a "Next" link

var key = Environment.GetEnvironmentVariable("CDPFLEET_API_KEY")!;
using var http = new HttpClient();
http.DefaultRequestHeaders.Add("x-api-key", key);

var output = new JsonObject { ["site"] = Site, ["scrape"] = new JsonObject(), ["recording"] = new JsonObject(), ["unrecorded"] = new JsonObject() };
var scrape = output["scrape"]!.AsObject();
var recording = output["recording"]!.AsObject();
var unrecorded = output["unrecorded"]!.AsObject();
string sessionId;
using (var playwright = await Playwright.CreateAsync())
{
    // A session launched with record: false, for comparison: it must never get a recording.
    var quiet = await Launch(playwright, false);
    try { await Go(quiet.Page, Site); }
    finally { await quiet.Browser.CloseAsync(); }
    unrecorded["session_id"] = quiet.Id;

    // The scrape: pages 8, 9, 10 — follow "Next" on each. Page 10 is the last one and has none.
    var s = await Launch(playwright, null);
    sessionId = s.Id;
    output["session_id"] = sessionId;
    var quotes = new List<string>();
    try
    {
        await Go(s.Page, $"{Site}/page/8/");
        for (var pageNo = 8; ; pageNo++)
        {
            quotes.AddRange(await s.Page.Locator(".quote .text").AllTextContentsAsync());
            scrape["last_page"] = pageNo;
            string? href;
            try
            {
                href = await s.Page.Locator(Next).GetAttributeAsync("href", new() { Timeout = 5000 }); // 5 s to find the link
            }
            catch (Exception e) when (e is PlaywrightException or TimeoutException)
            {
                scrape["failed_at_s"] = Since(s.Started);
                scrape["error"] = $"{(e.GetType().Name == "TimeoutException" ? "timeout" : e.GetType().Name)} waiting for {Next}";
                break;
            }
            await Go(s.Page, new Uri(new Uri(Site), href).ToString()); // page loads keep their own timeout and retries
        }
        scrape["quotes"] = quotes.Count;
        // What the video shows at that moment: the last page — a "Previous" link and no "Next".
        scrape["url_at_failure"] = s.Page.Url;
        scrape["next_links"] = await s.Page.Locator("li.next").CountAsync();
        scrape["previous_links"] = await s.Page.Locator("li.previous").CountAsync();
    }
    finally
    {
        await s.Browser.CloseAsync(); // ends the session; the node uploads the video afterwards
    }
    output["session_seconds"] = Since(s.Started);
}

// Find the recording, check it covers the failure, download it, delete it.
var t = DateTime.UtcNow;
JsonObject? rec = null;
while (rec is null && (DateTime.UtcNow - t).TotalSeconds < 600)
{
    rec = (await Recordings()).FirstOrDefault(r => (string?)r!["session_id"] == sessionId)?.AsObject();
    if (rec is null) await Task.Delay(5000);
}
if (rec is not null)
{
    recording["appeared_after_s"] = (long)Math.Round((DateTime.UtcNow - t).TotalSeconds);
    var id = (string)rec["id"]!;
    var link = (await http.GetFromJsonAsync<JsonObject>($"https://cdpfleet.com/v1/me/recordings/{id}/url"))!;
    using var plain = new HttpClient(); // the private link needs no API key, so none is sent
    using var video = await plain.GetAsync((string)link["url"]!);
    var bytes = await video.Content.ReadAsByteArrayAsync();
    foreach (var k in new[] { "duration_seconds", "width", "height", "bytes" }) recording[k] = rec[k]!.DeepClone();
    recording["downloaded_bytes"] = bytes.Length;
    recording["content_type"] = video.Content.Headers.ContentType?.MediaType;
    recording["mp4"] = bytes.Length > 8 && System.Text.Encoding.ASCII.GetString(bytes, 4, 4) == "ftyp";
    recording["link_expires_in_s"] = link["expires_in_s"]!.DeepClone();
    output["failure_within_recording"] = (double)scrape["failed_at_s"]! <= (double)rec["duration_seconds"]! + 1;
    recording["deleted"] = (await http.DeleteAsync($"https://cdpfleet.com/v1/me/recordings/{id}")).IsSuccessStatusCode;
}

// By now the scrape's video has been uploaded; give the unrecorded session another minute.
await Task.Delay(60000);
unrecorded["recording_found"] = (await Recordings()).Any(r => (string?)r!["session_id"] == (string)unrecorded["session_id"]!);
Console.WriteLine(output.ToJsonString(new JsonSerializerOptions { WriteIndented = true, Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping }));

async Task<JsonArray> Recordings() =>
    (await http.GetFromJsonAsync<JsonObject>("https://cdpfleet.com/v1/me/recordings"))?["recordings"]?.AsArray() ?? [];

// Headful (the default) sessions are recorded from the browser's display.
async Task<Session> Launch(IPlaywright playwright, bool? record)
{
    var body = new JsonObject { ["proxy"] = Environment.GetEnvironmentVariable("PROXY_URL"), ["headless"] = false };
    if (record is not null) body["record"] = record;
    var res = await http.PostAsJsonAsync("https://starter.cdpfleet.com/chromium/session", body);
    if (!res.IsSuccessStatusCode) throw new Exception($"launch: {(int)res.StatusCode} {await res.Content.ReadAsStringAsync()}");
    var launched = (await res.Content.ReadFromJsonAsync<JsonObject>())!;
    var started = DateTime.UtcNow;
    var browser = await playwright.Chromium.ConnectAsync((string)launched["wsUrl"]!, new() { Headers = new Dictionary<string, string> { ["x-api-key"] = key } });
    // NoViewport: the page fills the whole display, so it fills the recording too.
    var page = await (await browser.NewContextAsync(new() { ViewportSize = ViewportSize.NoViewport })).NewPageAsync();
    return new Session((string)launched["sessionId"]!, started, browser, page);
}

static double Since(DateTime started) => Math.Round((DateTime.UtcNow - started).TotalSeconds, 1);

// Proxies drop a tunnel now and then: up to two retries.
static async Task Go(IPage page, string url)
{
    for (var attempt = 1; ; attempt++)
    {
        try { await page.GotoAsync(url, new() { Timeout = 60000 }); return; }
        catch (Exception e) when (attempt < 3 && e is PlaywrightException or TimeoutException) { await Task.Delay(2000); }
    }
}

record Session(string Id, DateTime Started, IBrowser Browser, IPage Page);
```

```go
// go get github.com/playwright-community/playwright-go@v0.6000.0
// Driver: build playwright-core 1.60.0 from npm and set PLAYWRIGHT_DRIVER_PATH (see /docs/quickstart).
// env: CDPFLEET_API_KEY, PROXY_URL
package main

import (
	"bytes"
	"encoding/json"
	"errors"
	"fmt"
	"io"
	"log"
	"math"
	"net/http"
	"net/url"
	"os"
	"time"

	"github.com/playwright-community/playwright-go"
)

const site = "https://quotes.toscrape.com"
const next = "li.next a" // the scraper assumes every listing page has a "Next" link

var key = os.Getenv("CDPFLEET_API_KEY")

type Scrape struct {
	LastPage      int     `json:"last_page"`
	FailedAtS     float64 `json:"failed_at_s"`
	Error         string  `json:"error"`
	Quotes        int     `json:"quotes"`
	URLAtFailure  string  `json:"url_at_failure"`
	NextLinks     int     `json:"next_links"`
	PreviousLinks int     `json:"previous_links"`
}

type Recording struct {
	AppearedAfterS  int64    `json:"appeared_after_s"`
	DurationSeconds *float64 `json:"duration_seconds"`
	Width           *int     `json:"width"`
	Height          *int     `json:"height"`
	Bytes           *int64   `json:"bytes"`
	DownloadedBytes int      `json:"downloaded_bytes"`
	ContentType     string   `json:"content_type"`
	MP4             bool     `json:"mp4"`
	LinkExpiresInS  int      `json:"link_expires_in_s"`
	Deleted         bool     `json:"deleted"`
}

type Unrecorded struct {
	SessionID      string `json:"session_id"`
	RecordingFound bool   `json:"recording_found"`
}

type Output struct {
	Site                   string     `json:"site"`
	Scrape                 Scrape     `json:"scrape"`
	Recording              *Recording `json:"recording"`
	Unrecorded             Unrecorded `json:"unrecorded"`
	SessionID              string     `json:"session_id"`
	SessionSeconds         float64    `json:"session_seconds"`
	FailureWithinRecording *bool      `json:"failure_within_recording,omitempty"`
}

type apiRecording struct {
	ID              string   `json:"id"`
	SessionID       string   `json:"session_id"`
	DurationSeconds *float64 `json:"duration_seconds"`
	Width           *int     `json:"width"`
	Height          *int     `json:"height"`
	Bytes           *int64   `json:"bytes"`
}

func api(method, path string, into any) (int, error) {
	req, _ := http.NewRequest(method, "https://cdpfleet.com"+path, nil)
	req.Header.Set("x-api-key", key)
	res, err := http.DefaultClient.Do(req)
	if err != nil {
		return 0, err
	}
	defer res.Body.Close()
	if into != nil {
		_ = json.NewDecoder(res.Body).Decode(into)
	}
	return res.StatusCode, nil
}

func recordings() []apiRecording {
	var body struct {
		Recordings []apiRecording `json:"recordings"`
	}
	_, _ = api("GET", "/v1/me/recordings", &body)
	return body.Recordings
}

func since(t time.Time) float64 { return math.Round(time.Since(t).Seconds()*10) / 10 }

type session struct {
	id      string
	started time.Time
	browser playwright.Browser
	page    playwright.Page
}

// Headful (the default) sessions are recorded from the browser's display.
func launch(pw *playwright.Playwright, record *bool) (*session, error) {
	body := map[string]any{"proxy": os.Getenv("PROXY_URL"), "headless": false}
	if record != nil {
		body["record"] = *record
	}
	payload, _ := json.Marshal(body)
	req, _ := http.NewRequest("POST", "https://starter.cdpfleet.com/chromium/session", bytes.NewReader(payload))
	req.Header.Set("x-api-key", key)
	req.Header.Set("content-type", "application/json")
	res, err := http.DefaultClient.Do(req)
	if err != nil {
		return nil, err
	}
	defer res.Body.Close()
	if res.StatusCode != 200 {
		text, _ := io.ReadAll(res.Body)
		return nil, fmt.Errorf("launch: %d %s", res.StatusCode, text)
	}
	var launched struct {
		SessionID string `json:"sessionId"`
		WsURL     string `json:"wsUrl"`
	}
	if err := json.NewDecoder(res.Body).Decode(&launched); err != nil {
		return nil, err
	}
	started := time.Now()
	browser, err := pw.Chromium.Connect(launched.WsURL, playwright.BrowserTypeConnectOptions{Headers: map[string]string{"x-api-key": key}})
	if err != nil {
		return nil, err
	}
	// NoViewport: the page fills the whole display, so it fills the recording too.
	context, err := browser.NewContext(playwright.BrowserNewContextOptions{NoViewport: playwright.Bool(true)})
	if err != nil {
		browser.Close()
		return nil, err
	}
	page, err := context.NewPage()
	if err != nil {
		browser.Close()
		return nil, err
	}
	return &session{launched.SessionID, started, browser, page}, nil
}

// Proxies drop a tunnel now and then: up to two retries.
func goTo(page playwright.Page, url string) error {
	var err error
	for attempt := 1; attempt <= 3; attempt++ {
		if _, err = page.Goto(url, playwright.PageGotoOptions{Timeout: playwright.Float(60000)}); err == nil {
			return nil
		}
		time.Sleep(2 * time.Second)
	}
	return err
}

func scrapeUntilItBreaks(s *session, out *Output) error {
	defer s.browser.Close() // ends the session; the node uploads the video afterwards
	if err := goTo(s.page, site+"/page/8/"); err != nil {
		return err
	}
	quotes := 0
	for pageNo := 8; ; pageNo++ {
		texts, err := s.page.Locator(".quote .text").AllTextContents()
		if err != nil {
			return err
		}
		quotes += len(texts)
		out.Scrape.LastPage = pageNo
		href, err := s.page.Locator(next).GetAttribute("href", playwright.LocatorGetAttributeOptions{Timeout: playwright.Float(5000)}) // 5 s to find the link
		if err != nil {
			out.Scrape.FailedAtS = since(s.started)
			kind := "error"
			if errors.Is(err, playwright.ErrTimeout) {
				kind = "timeout"
			}
			out.Scrape.Error = fmt.Sprintf("%s waiting for %s", kind, next)
			break
		}
		base, _ := url.Parse(site)
		ref, err := url.Parse(href)
		if err != nil {
			return err
		}
		if err := goTo(s.page, base.ResolveReference(ref).String()); err != nil { // page loads keep their own timeout and retries
			return err
		}
	}
	out.Scrape.Quotes = quotes
	// What the video shows at that moment: the last page — a "Previous" link and no "Next".
	out.Scrape.URLAtFailure = s.page.URL()
	out.Scrape.NextLinks, _ = s.page.Locator("li.next").Count()
	out.Scrape.PreviousLinks, _ = s.page.Locator("li.previous").Count()
	return nil
}

func run(out *Output) error {
	pw, err := playwright.Run(&playwright.RunOptions{SkipInstallBrowsers: true})
	if err != nil {
		return err
	}
	defer pw.Stop()

	// A session launched with record: false, for comparison: it must never get a recording.
	quiet, err := launch(pw, playwright.Bool(false))
	if err != nil {
		return err
	}
	err = goTo(quiet.page, site)
	quiet.browser.Close()
	if err != nil {
		return err
	}
	out.Unrecorded.SessionID = quiet.id

	// The scrape: pages 8, 9, 10 — follow "Next" on each. Page 10 is the last one and has none.
	s, err := launch(pw, nil)
	if err != nil {
		return err
	}
	out.SessionID = s.id
	if err := scrapeUntilItBreaks(s, out); err != nil {
		return err
	}
	out.SessionSeconds = since(s.started)

	// Find the recording, check it covers the failure, download it, delete it.
	t := time.Now()
	var rec *apiRecording
	for rec == nil && time.Since(t) < 10*time.Minute {
		for _, r := range recordings() {
			if r.SessionID == s.id {
				rec = &r
				break
			}
		}
		if rec == nil {
			time.Sleep(5 * time.Second)
		}
	}
	if rec != nil {
		var link struct {
			URL        string `json:"url"`
			ExpiresInS int    `json:"expires_in_s"`
		}
		if _, err := api("GET", "/v1/me/recordings/"+rec.ID+"/url", &link); err != nil {
			return err
		}
		res, err := http.Get(link.URL)
		if err != nil {
			return err
		}
		video, _ := io.ReadAll(res.Body)
		res.Body.Close()
		out.Recording = &Recording{
			AppearedAfterS:  int64(math.Round(time.Since(t).Seconds())),
			DurationSeconds: rec.DurationSeconds,
			Width:           rec.Width,
			Height:          rec.Height,
			Bytes:           rec.Bytes,
			DownloadedBytes: len(video),
			ContentType:     res.Header.Get("content-type"),
			MP4:             len(video) > 8 && string(video[4:8]) == "ftyp",
			LinkExpiresInS:  link.ExpiresInS,
		}
		within := rec.DurationSeconds != nil && out.Scrape.FailedAtS <= *rec.DurationSeconds+1
		out.FailureWithinRecording = &within
		status, _ := api("DELETE", "/v1/me/recordings/"+rec.ID, nil)
		out.Recording.Deleted = status == 200
	}

	// By now the scrape's video has been uploaded; give the unrecorded session another minute.
	time.Sleep(60 * time.Second)
	for _, r := range recordings() {
		if r.SessionID == quiet.id {
			out.Unrecorded.RecordingFound = true
		}
	}
	return nil
}

func main() {
	out := Output{Site: site, Recording: &Recording{}}
	if err := run(&out); err != nil {
		log.Fatal(err)
	}
	b, _ := json.MarshalIndent(out, "", "  ")
	fmt.Println(string(b))
}
```

## What we got

|  | Measured |
| --- | --- |
| Scrape | 30 quotes from pages 8–10, then: timeout waiting for li.next a |
| Failed at | 10.4 s after launch, on https://quotes.toscrape.com/page/10/ (0 "Next" links, 1 "Previous") |
| Recording appeared | 5 s after the session ended |
| Video | 11 s, 854×480, 57443 bytes, video/mp4 |
| Failure inside the video | yes |
| record: false session | no recording |

From the Node.js run on 2026-10-13. IP addresses are replaced with placeholders (203.0.113.x); equal addresses stay equal. The other languages produced the same findings.

Raw output (Node.js)

```
{
  "site": "https://quotes.toscrape.com",
  "scrape": {
    "last_page": 10,
    "failed_at_s": 10.4,
    "error": "timeout waiting for li.next a",
    "quotes": 30,
    "url_at_failure": "https://quotes.toscrape.com/page/10/",
    "next_links": 0,
    "previous_links": 1
  },
  "recording": {
    "appeared_after_s": 5,
    "duration_seconds": 11,
    "width": 854,
    "height": 480,
    "bytes": 57443,
    "downloaded_bytes": 57443,
    "content_type": "video/mp4",
    "mp4": true,
    "link_expires_in_s": 900,
    "deleted": true
  },
  "unrecorded": {
    "session_id": "f3cdd8f0-40cf-4290-98ad-c3bb100b306d",
    "recording_found": false
  },
  "session_id": "392bb039-23c2-4509-81c9-1bccbde164ed",
  "session_seconds": 10.5,
  "failure_within_recording": true
}
```

## Takeaways

-   **The error alone said only "timeout waiting for `li.next a`".** At that moment the browser was on `/page/10/` with a "Previous" link and no "Next" — the last page, not a broken selector. The failure came 10.4–13.2 s after launch, the browser closed a moment later, and the videos were 11–14 s long — so their last seconds were recorded with that page on screen.
-   **No setup:** the scraper's code does nothing about recording. Its headful session was recorded by default, and in every run the video was listed under its `session_id` on the second check, 5 seconds after the session ended (the first, right after closing, found nothing yet).
-   **What you get:** an MP4 at 854×480 — the free tier's quality ([plans](https://cdpfleet.com/docs/recordings) add 720p and 1080p) — of 57–62 KB for these 11–14 s, behind a private link valid for 15 minutes. `DELETE /v1/me/recordings/{id}` deletes it (it answered 200 in every run).
-   **`"record": false` really is off:** that session never got a recording, checked a minute after the other video had arrived.
-   **Keep "find the element" and "load the page" on separate timeouts.** Reading the "Next" link with a 5-second timeout and loading its page with the normal one means the error names what was missing; a click that also waits for a slow page load can time out mid-navigation and send you looking at the wrong page.

---
Source: https://cdpfleet.com/docs/cases/replay-a-failed-run · Markdown: https://cdpfleet.com/docs/cases/replay-a-failed-run.md · All docs for AI tools: https://cdpfleet.com/llms.txt