[Examples & benchmarks](https://cdpfleet.com/docs/cases) / #28 · 2026-10-09 · Example · Easy · for [web scraping](https://cdpfleet.com/web-scraping), [llm data](https://cdpfleet.com/llm-data)

# Scrape a catalogue page by page: structured data from rendered HTML

Every book on two pages of a catalogue — title, price, star rating and availability — as clean JSON, parsed from the rendered DOM in one evaluate call per page.

[Chromium](https://cdpfleet.com/docs/browsers/chromium)

Run on production on 2026-10-09: ✓ Node.js ✓ Python ✓ Java ✓ C# ✓ Go

## The problem

A catalogue page shows 20 items with prices in formatted strings ("£51.77"), star ratings encoded in CSS class names ("star-rating Three") and availability wrapped in whitespace. You want records with real numbers, integers and booleans — and you want the next page too, the way a visitor gets there.

## What we used, and why

| What | Why |
| --- | --- |
| `chromium`, `headless: "new"` | One session, nothing browser-specific. |
| `page.evaluate` with `querySelectorAll` | All 20 books in one round trip, parsed inside the page. |
| `parseFloat(text.replace(/[^0-9.]/g, ""))` | Strips the currency symbol and returns a number. |
| The class list of `.star-rating` | The rating is a word (One … Five) in a class name; a lookup table turns it into 1–5. |
| Clicking `li.next a` | Follows pagination with the site's own link instead of building URLs. |

## How it works

1.  Open the first page of books.toscrape.com and extract all 20 books: title, price (number), rating (1–5), in stock (boolean).
2.  Click "next" and wait for page 2.
3.  Extract page 2 the same way and print one JSON document with the count, the records and the time taken.

## The code

The same program in five languages (also [on GitHub](https://github.com/cdpfleet/cdpfleet/tree/main/cases/28-structured-extraction), with the raw output). Set these environment variables first:

-   `CDPFLEET_API_KEY` — your API key (dashboard → API keys)
-   `PROXY_URL` — your proxy, e.g. `http://user:pass@proxy.example.com:8000`

 

```node
// npm install playwright@1.60.0
// env: CDPFLEET_API_KEY, PROXY_URL
import { chromium } from 'playwright';

const KEY = process.env.CDPFLEET_API_KEY;
const START = 'https://books.toscrape.com/';
const STARS = { One: 1, Two: 2, Three: 3, Four: 4, Five: 5 };

const EXTRACT_JS = `() => [...document.querySelectorAll('article.product_pod')].map(el => {
  const stars = { One: 1, Two: 2, Three: 3, Four: 4, Five: 5 };
  const ratingClass = [...el.querySelector('.star-rating').classList].find(c => c !== 'star-rating');
  return {
    title: el.querySelector('h3 a').getAttribute('title'),
    price: parseFloat(el.querySelector('.price_color').textContent.replace(/[^0-9.]/g, '')),
    rating: stars[ratingClass] || 0,
    in_stock: el.querySelector('.availability').textContent.trim().toLowerCase().includes('in stock'),
  };
})`;

const res = await fetch('https://starter.cdpfleet.com/chromium/session', {
  method: 'POST',
  headers: { 'x-api-key': KEY, 'content-type': 'application/json' },
  body: JSON.stringify({ proxy: process.env.PROXY_URL, headless: 'new' }),
});
if (!res.ok) throw new Error(`launch ${res.status} ${await res.text()}`);
const { wsUrl } = await res.json();
const browser = await chromium.connect(wsUrl, { headers: { 'x-api-key': KEY } });

try {
  const page = await browser.newPage();
  const t = Date.now();
  const books = [];

  await page.goto(START, { timeout: 60000 });
  books.push(...await page.evaluate(`(${EXTRACT_JS})()`));

  const next = page.locator('li.next a');
  if (await next.count() > 0) {
    await next.click();
    await page.waitForLoadState('domcontentloaded');
    books.push(...await page.evaluate(`(${EXTRACT_JS})()`));
  }

  console.log(JSON.stringify({
    pages_scraped: 2,
    total_books: books.length,
    books,
    seconds: Math.round((Date.now() - t) / 10) / 100,
  }, null, 2));
} finally {
  await browser.close();
}
```

```python
# pip install playwright==1.60.0 requests
# env: CDPFLEET_API_KEY, PROXY_URL
import json
import os
import time

import requests
from playwright.sync_api import sync_playwright

KEY = os.environ["CDPFLEET_API_KEY"]
START = "https://books.toscrape.com/"

EXTRACT_JS = """() => [...document.querySelectorAll('article.product_pod')].map(el => {
  const stars = { One: 1, Two: 2, Three: 3, Four: 4, Five: 5 };
  const ratingClass = [...el.querySelector('.star-rating').classList].find(c => c !== 'star-rating');
  return {
    title: el.querySelector('h3 a').getAttribute('title'),
    price: parseFloat(el.querySelector('.price_color').textContent.replace(/[^0-9.]/g, '')),
    rating: stars[ratingClass] || 0,
    in_stock: el.querySelector('.availability').textContent.trim().toLowerCase().includes('in stock'),
  };
})"""

res = requests.post("https://starter.cdpfleet.com/chromium/session",
    headers={"x-api-key": KEY},
    json={"proxy": os.environ["PROXY_URL"], "headless": "new"}, timeout=60)
if not res.ok:
    raise SystemExit(f"launch {res.status_code} {res.text}")

with sync_playwright() as p:
    browser = p.chromium.connect(res.json()["wsUrl"], headers={"x-api-key": KEY})
    try:
        page = browser.new_page()
        t = time.time()
        books = []

        page.goto(START, timeout=60000)
        books.extend(page.evaluate(EXTRACT_JS))

        nxt = page.locator("li.next a")
        if nxt.count() > 0:
            nxt.click()
            page.wait_for_load_state("domcontentloaded")
            books.extend(page.evaluate(EXTRACT_JS))

        print(json.dumps({
            "pages_scraped": 2,
            "total_books": len(books),
            "books": books,
            "seconds": round(time.time() - t, 2),
        }, indent=2, ensure_ascii=False))
    finally:
        browser.close()
```

```java
// Maven: com.microsoft.playwright:playwright:1.60.0, com.google.code.gson:gson:2.11.0
// Run with PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1. env: CDPFLEET_API_KEY, PROXY_URL
import com.google.gson.*;
import com.microsoft.playwright.*;
import java.net.URI;
import java.net.http.*;
import java.util.*;

public class Main {
  static final String KEY = System.getenv("CDPFLEET_API_KEY");
  static final String START = "https://books.toscrape.com/";
  static final Gson GSON = new GsonBuilder().serializeNulls().create();

  static final String EXTRACT_JS = """
      () => [...document.querySelectorAll('article.product_pod')].map(el => {
        const stars = { One: 1, Two: 2, Three: 3, Four: 4, Five: 5 };
        const ratingClass = [...el.querySelector('.star-rating').classList].find(c => c !== 'star-rating');
        return {
          title: el.querySelector('h3 a').getAttribute('title'),
          price: parseFloat(el.querySelector('.price_color').textContent.replace(/[^0-9.]/g, '')),
          rating: stars[ratingClass] || 0,
          in_stock: el.querySelector('.availability').textContent.trim().toLowerCase().includes('in stock'),
        };
      })""";

  public static void main(String[] args) throws Exception {
    String body = "{\"proxy\": " + GSON.toJson(System.getenv("PROXY_URL")) + ", \"headless\": \"new\"}";
    HttpResponse<String> res = HttpClient.newHttpClient().send(HttpRequest.newBuilder(URI.create("https://starter.cdpfleet.com/chromium/session"))
        .header("x-api-key", KEY).header("content-type", "application/json")
        .POST(HttpRequest.BodyPublishers.ofString(body)).build(), HttpResponse.BodyHandlers.ofString());
    if (res.statusCode() != 200) throw new RuntimeException("launch: " + res.statusCode() + " " + res.body());
    String wsUrl = JsonParser.parseString(res.body()).getAsJsonObject().get("wsUrl").getAsString();

    try (Playwright playwright = Playwright.create()) {
      Browser browser = playwright.chromium().connect(wsUrl, new BrowserType.ConnectOptions().setHeaders(Map.of("x-api-key", KEY)));
      try {
        Page page = browser.newPage();
        long t = System.currentTimeMillis();
        JsonArray books = new JsonArray();

        page.navigate(START, new Page.NavigateOptions().setTimeout(60000));
        for (JsonElement e : GSON.toJsonTree(page.evaluate(EXTRACT_JS)).getAsJsonArray()) books.add(e);

        Locator next = page.locator("li.next a");
        if (next.count() > 0) {
          next.click();
          page.waitForLoadState(com.microsoft.playwright.options.LoadState.DOMCONTENTLOADED);
          for (JsonElement e : GSON.toJsonTree(page.evaluate(EXTRACT_JS)).getAsJsonArray()) books.add(e);
        }

        JsonObject out = new JsonObject();
        out.addProperty("pages_scraped", 2);
        out.addProperty("total_books", books.size());
        out.add("books", books);
        out.addProperty("seconds", Math.round((System.currentTimeMillis() - t) / 10.0) / 100.0);

        System.out.println(new GsonBuilder().serializeNulls().setPrettyPrinting().disableHtmlEscaping().create().toJson(out));
      } finally {
        browser.close();
      }
    }
  }
}
```

```csharp
// dotnet add package Microsoft.Playwright --version 1.60.0
// env: CDPFLEET_API_KEY, PROXY_URL
using System.Net.Http.Json;
using System.Text.Encodings.Web;
using System.Text.Json;
using System.Text.Json.Nodes;
using Microsoft.Playwright;

const string Start = "https://books.toscrape.com/";
const string ExtractJs = """
    () => [...document.querySelectorAll('article.product_pod')].map(el => {
      const stars = { One: 1, Two: 2, Three: 3, Four: 4, Five: 5 };
      const ratingClass = [...el.querySelector('.star-rating').classList].find(c => c !== 'star-rating');
      return {
        title: el.querySelector('h3 a').getAttribute('title'),
        price: parseFloat(el.querySelector('.price_color').textContent.replace(/[^0-9.]/g, '')),
        rating: stars[ratingClass] || 0,
        in_stock: el.querySelector('.availability').textContent.trim().toLowerCase().includes('in stock'),
      };
    })
    """;

var key = Environment.GetEnvironmentVariable("CDPFLEET_API_KEY")!;
using var http = new HttpClient();
http.DefaultRequestHeaders.Add("x-api-key", key);
var res = await http.PostAsJsonAsync("https://starter.cdpfleet.com/chromium/session",
    new { proxy = Environment.GetEnvironmentVariable("PROXY_URL"), headless = "new" });
if (!res.IsSuccessStatusCode) throw new Exception($"launch: {(int)res.StatusCode} {await res.Content.ReadAsStringAsync()}");
var wsUrl = (await res.Content.ReadFromJsonAsync<JsonElement>()).GetProperty("wsUrl").GetString()!;

using var playwright = await Playwright.CreateAsync();
var browser = await playwright.Chromium.ConnectAsync(wsUrl, new() { Headers = new Dictionary<string, string> { ["x-api-key"] = key } });
try
{
    var page = await browser.NewPageAsync();
    var sw = System.Diagnostics.Stopwatch.StartNew();
    var books = new JsonArray();

    await page.GotoAsync(Start, new() { Timeout = 60000 });
    foreach (var el in (await page.EvaluateAsync<JsonElement>(ExtractJs)).EnumerateArray())
        books.Add(JsonNode.Parse(el.GetRawText()));

    var next = page.Locator("li.next a");
    if (await next.CountAsync() > 0)
    {
        await next.ClickAsync();
        await page.WaitForLoadStateAsync(LoadState.DOMContentLoaded);
        foreach (var el in (await page.EvaluateAsync<JsonElement>(ExtractJs)).EnumerateArray())
            books.Add(JsonNode.Parse(el.GetRawText()));
    }

    var result = new JsonObject
    {
        ["pages_scraped"] = 2,
        ["total_books"] = books.Count,
        ["books"] = books,
        ["seconds"] = Math.Round(sw.Elapsed.TotalSeconds, 2),
    };
    Console.WriteLine(result.ToJsonString(new JsonSerializerOptions { WriteIndented = true, Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping }));
}
finally
{
    await browser.CloseAsync();
}
```

```go
// go get github.com/playwright-community/playwright-go@v0.6000.0
// Driver: build playwright-core 1.60.0 from npm and set PLAYWRIGHT_DRIVER_PATH (see /docs/quickstart).
// env: CDPFLEET_API_KEY, PROXY_URL
package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io"
	"log"
	"math"
	"net/http"
	"os"
	"time"

	"github.com/playwright-community/playwright-go"
)

var key = os.Getenv("CDPFLEET_API_KEY")

const start = "https://books.toscrape.com/"
const extractJS = `() => [...document.querySelectorAll('article.product_pod')].map(el => {
  const stars = { One: 1, Two: 2, Three: 3, Four: 4, Five: 5 };
  const ratingClass = [...el.querySelector('.star-rating').classList].find(c => c !== 'star-rating');
  return {
    title: el.querySelector('h3 a').getAttribute('title'),
    price: parseFloat(el.querySelector('.price_color').textContent.replace(/[^0-9.]/g, '')),
    rating: stars[ratingClass] || 0,
    in_stock: el.querySelector('.availability').textContent.trim().toLowerCase().includes('in stock'),
  };
})`

func launch(name string, options map[string]any) (map[string]any, error) {
	body, _ := json.Marshal(options)
	req, _ := http.NewRequest("POST", "https://starter.cdpfleet.com/"+name+"/session", bytes.NewReader(body))
	req.Header.Set("x-api-key", key)
	req.Header.Set("content-type", "application/json")
	res, err := http.DefaultClient.Do(req)
	if err != nil {
		return nil, err
	}
	defer res.Body.Close()
	if res.StatusCode != http.StatusOK {
		msg, _ := io.ReadAll(res.Body)
		return nil, fmt.Errorf("launch %s: %s %s", name, res.Status, msg)
	}
	var session map[string]any
	return session, json.NewDecoder(res.Body).Decode(&session)
}

type book struct {
	Title   string  `json:"title"`
	Price   float64 `json:"price"`
	Rating  int     `json:"rating"`
	InStock bool    `json:"in_stock"`
}

type output struct {
	PagesScraped int    `json:"pages_scraped"`
	TotalBooks   int    `json:"total_books"`
	Books        []book `json:"books"`
	Seconds      float64 `json:"seconds"`
}

func extractBooks(page playwright.Page) []book {
	v, err := page.Evaluate(extractJS)
	if err != nil {
		log.Fatal(err)
	}
	var books []book
	for _, item := range v.([]any) {
		m := item.(map[string]any)
		b := book{Title: m["title"].(string), InStock: m["in_stock"].(bool)}
		switch p := m["price"].(type) {
		case float64:
			b.Price = p
		}
		switch r := m["rating"].(type) {
		case float64:
			b.Rating = int(r)
		case int:
			b.Rating = r
		}
		books = append(books, b)
	}
	return books
}

func main() {
	session, err := launch("chromium", map[string]any{"proxy": os.Getenv("PROXY_URL"), "headless": "new"})
	if err != nil {
		log.Fatal(err)
	}
	pw, err := playwright.Run(&playwright.RunOptions{SkipInstallBrowsers: true})
	if err != nil {
		log.Fatal(err)
	}
	defer pw.Stop()
	browser, err := pw.Chromium.Connect(session["wsUrl"].(string), playwright.BrowserTypeConnectOptions{Headers: map[string]string{"x-api-key": key}})
	if err != nil {
		log.Fatal(err)
	}
	defer browser.Close()

	page, err := browser.NewPage()
	if err != nil {
		log.Fatal(err)
	}
	t := time.Now()

	if _, err := page.Goto(start, playwright.PageGotoOptions{Timeout: playwright.Float(60000)}); err != nil {
		log.Fatal(err)
	}
	books := extractBooks(page)

	next := page.Locator("li.next a")
	count, err := next.Count()
	if err != nil {
		log.Fatal(err)
	}
	if count > 0 {
		if err := next.Click(); err != nil {
			log.Fatal(err)
		}
		if err := page.WaitForLoadState(playwright.PageWaitForLoadStateOptions{State: playwright.LoadStateDomcontentloaded}); err != nil {
			log.Fatal(err)
		}
		books = append(books, extractBooks(page)...)
	}

	out := output{
		PagesScraped: 2,
		TotalBooks:   len(books),
		Books:        books,
		Seconds:      math.Round(time.Since(t).Seconds()*100) / 100,
	}
	var buf bytes.Buffer
	enc := json.NewEncoder(&buf)
	enc.SetEscapeHTML(false)
	enc.SetIndent("", "  ")
	enc.Encode(out)
	fmt.Print(buf.String())
}
```

## What we got

| Pages | Books | Prices | Ratings | In stock | Seconds |
| --- | --- | --- | --- | --- | --- |
| 2 | 40 | £12.84 – £57.25 | 1, 2, 3, 4, 5 | 40/40 | 10.18 |

From the Node.js run on 2026-10-09. IP addresses are replaced with placeholders (203.0.113.x); equal addresses stay equal. The other languages produced the same findings.

Raw output (Node.js)

```
{
  "pages_scraped": 2,
  "total_books": 40,
  "books": [
    {
      "title": "A Light in the Attic",
      "price": 51.77,
      "rating": 3,
      "in_stock": true
    },
    {
      "title": "Tipping the Velvet",
      "price": 53.74,
      "rating": 1,
      "in_stock": true
    },
    {
      "title": "Soumission",
      "price": 50.1,
      "rating": 1,
      "in_stock": true
    },
    {
      "title": "Sharp Objects",
      "price": 47.82,
      "rating": 4,
      "in_stock": true
    },
    {
      "title": "Sapiens: A Brief History of Humankind",
      "price": 54.23,
      "rating": 5,
      "in_stock": true
    },
    {
      "title": "The Requiem Red",
      "price": 22.65,
      "rating": 1,
      "in_stock": true
    },
    {
      "title": "The Dirty Little Secrets of Getting Your Dream Job",
      "price": 33.34,
      "rating": 4,
      "in_stock": true
    },
    {
      "title": "The Coming Woman: A Novel Based on the Life of the Infamous Feminist, Victoria Woodhull",
      "price": 17.93,
      "rating": 3,
      "in_stock": true
    },
    {
      "title": "The Boys in the Boat: Nine Americans and Their Epic Quest for Gold at the 1936 Berlin Olympics",
      "price": 22.6,
      "rating": 4,
      "in_stock": true
    },
    {
      "title": "The Black Maria",
      "price": 52.15,
      "rating": 1,
      "in_stock": true
    },
    {
      "title": "Starving Hearts (Triangular Trade Trilogy, #1)",
      "price": 13.99,
      "rating": 2,
      "in_stock": true
    },
    {
      "title": "Shakespeare's Sonnets",
      "price": 20.66,
      "rating": 4,
      "in_stock": true
    },
    {
      "title": "Set Me Free",
      "price": 17.46,
      "rating": 5,
      "in_stock": true
    },
    {
      "title": "Scott Pilgrim's Precious Little Life (Scott Pilgrim #1)",
      "price": 52.29,
      "rating": 5,
      "in_stock": true
    },
    {
      "title": "Rip it Up and Start Again",
      "price": 35.02,
      "rating": 5,
      "in_stock": true
    },
    {
      "title": "Our Band Could Be Your Life: Scenes from the American Indie Underground, 1981-1991",
      "price": 57.25,
      "rating": 3,
      "in_stock": true
    },
    {
      "title": "Olio",
      "price": 23.88,
      "rating": 1,
      "in_stock": true
    },
    {
      "title": "Mesaerion: The Best Science Fiction Stories 1800-1849",
      "price": 37.59,
      "rating": 1,
      "in_stock": true
    },
    {
      "title": "Libertarianism for Beginners",
      "price": 51.33,
      "rating": 2,
      "in_stock": true
    },
    {
      "title": "It's Only the Himalayas",
      "price": 45.17,
      "rating": 2,
      "in_stock": true
    },
    {
      "title": "In Her Wake",
      "price": 12.84,
      "rating": 1,
      "in_stock": true
    },
    {
      "title": "How Music Works",
      "price": 37.32,
      "rating": 2,
      "in_stock": true
    },
    {
      "title": "Foolproof Preserving: A Guide to Small Batch Jams, Jellies, Pickles, Condiments, and More: A Foolproof Guide to Making Small Batch Jams, Jellies, Pickles, Condiments, and More",
      "price": 30.52,
      "rating": 3,
      "in_stock": true
    },
    {
      "title": "Chase Me (Paris Nights #2)",
      "price": 25.27,
      "rating": 5,
      "in_stock": true
    },
    {
      "title": "Black Dust",
      "price": 34.53,
      "rating": 5,
      "in_stock": true
    },
    {
      "title": "Birdsong: A Story in Pictures",
      "price": 54.64,
      "rating": 3,
      "in_stock": true
    },
    {
      "title": "America's Cradle of Quarterbacks: Western Pennsylvania's Football Factory from Johnny Unitas to Joe Montana",
      "price": 22.5,
      "rating": 3,
      "in_stock": true
    },
    {
      "title": "Aladdin and His Wonderful Lamp",
      "price": 53.13,
      "rating": 3,
      "in_stock": true
    },
    {
      "title": "Worlds Elsewhere: Journeys Around Shakespeare’s Globe",
      "price": 40.3,
      "rating": 5,
      "in_stock": true
    },
    {
      "title": "Wall and Piece",
      "price": 44.18,
      "rating": 4,
      "in_stock": true
    },
    {
      "title": "The Four Agreements: A Practical Guide to Personal Freedom",
      "price": 17.66,
      "rating": 5,
      "in_stock": true
    },
    {
      "title": "The Five Love Languages: How to Express Heartfelt Commitment to Your Mate",
      "price": 31.05,
      "rating": 3,
      "in_stock": true
    },
    {
      "title": "The Elephant Tree",
      "price": 23.82,
      "rating": 5,
      "in_stock": true
    },
    {
      "title": "The Bear and the Piano",
      "price": 36.89,
      "rating": 1,
      "in_stock": true
    },
    {
      "title": "Sophie's World",
      "price": 15.94,
      "rating": 5,
      "in_stock": true
    },
    {
      "title": "Penny Maybe",
      "price": 33.29,
      "rating": 3,
      "in_stock": true
    },
    {
      "title": "Maude (1883-1993):She Grew Up with the country",
      "price": 18.02,
      "rating": 2,
      "in_stock": true
    },
    {
      "title": "In a Dark, Dark Wood",
      "price": 19.63,
      "rating": 1,
      "in_stock": true
    },
    {
      "title": "Behind Closed Doors",
      "price": 52.22,
      "rating": 4,
      "in_stock": true
    },
    {
      "title": "You can't bury them all: Poems",
      "price": 33.63,
      "rating": 2,
      "in_stock": true
    }
  ],
  "seconds": 10.18
}
```

## Takeaways

-   **One `evaluate` per page extracts all 20 records** — parsing in the page avoids 20 round trips over the session's connection.
-   **Ratings in class names are a common pattern:** the word in `star-rating Three` is sturdier than counting star icons.
-   **Paginate with the site's own link:** clicking `li.next a` survives URL scheme changes and exercises the path a visitor takes.
-   **Validate types at the source:** every price parsed to a number and every rating to 1–5, or the run fails — catch layout changes on day one.

---
Source: https://cdpfleet.com/docs/cases/structured-extraction · Markdown: https://cdpfleet.com/docs/cases/structured-extraction.md · All docs for AI tools: https://cdpfleet.com/llms.txt