Cases / #37 · 2026-10-12 · Medium

Cookie banners by country: the same five sites from Germany and from the US

The Guardian, BBC, AP News, The Verge and HuffPost through a German and a US exit: which show a consent banner, which consent API they expose, what loads before anyone clicks, and what "reject" and "accept" change.

Chromium

Run on production on 2026-10-12: ✓ Node.js ✓ Python ✓ Java ✓ C# ✓ Go

The problem

Your scraper's exit country decides which version of a site it gets. From the EU most news and media sites put a consent banner over the page, run IAB Europe's TCF consent API and are supposed to hold back ad and analytics tags until you choose; from the US the same sites often show nothing and load everything. That changes what you see (an overlay hiding the content), how much bandwidth a page costs and which cookies your session carries afterwards. How big is the difference, and what do "reject" and "accept" actually do?

What we used, and why

What	Why
`PROXY_URL_DE` and `PROXY_URL_US`	One headful Chromium per country; every site is loaded in a fresh context from each.
A probe in `page.evaluate`	Detects the consent platform (OneTrust, Sourcepoint, Didomi, Cookiebot, Usercentrics), whether a OneTrust or Sourcepoint banner is visible, and the consent APIs the page exposes: `__tcfapi` (IAB Europe's TCF) and `__gpp` (IAB Tech Lab's GPP, used for US state privacy laws).
`locator.waitFor({ state: "visible" })`	Waits up to 20 s for the banner and records how long after `domcontentloaded` it appeared.
`page.on("request")`	Counts the hosts on other domains the page contacted — ad, analytics and CDN hosts — up to each moment.
`context.cookies()`	All cookies in the context, including third-party ones a page script can't see.
`frameLocator('iframe[id^="sp_message_iframe"]')`	Sourcepoint draws its banner in an iframe; OneTrust's buttons (`#onetrust-reject-all-handler`, `#onetrust-accept-btn-handler`) are in the page.

How it works

Launch Chromium through a US proxy and load each site in a fresh context; wait for a banner (up to 20 s, at least 6 s in all) and record the platform, banner, consent APIs, cookies and other-domain hosts. Then close that session and launch one through a German proxy (an idle session on shared threads ends after 60 s, so the countries run one after the other).
From Germany, load each site twice in fresh contexts: record the same before any choice, then click the first-layer "reject" in one and "accept" in the other — clicking once more if the banner is still up 5 s later — and measure again 6 s after that.
A free "reject" counts only if the first layer offers one — "Reject all and subscribe" isn't clicked. A page that answers with an error status is loaded once more (the rotating proxy gives the retry another IP).

The code

The same program in five languages (also on GitHub, with the raw output). Set these environment variables first:

CDPFLEET_API_KEY — your API key (dashboard → API keys)
PROXY_URL — your proxy, e.g. http://user:[email protected]:8000
PROXY_URL_DE — a proxy exit in Germany
PROXY_URL_US — a proxy exit in the United States

// npm install [email protected]
// env: CDPFLEET_API_KEY, PROXY_URL_DE (a German proxy), PROXY_URL_US (a US proxy)
import { chromium } from 'playwright';

const KEY = process.env.CDPFLEET_API_KEY;
const SITES = ['https://www.theguardian.com/international', 'https://www.bbc.com/', 'https://apnews.com/', 'https://www.theverge.com/', 'https://www.huffpost.com/'];
// First-layer buttons of the two consent platforms these sites use. OneTrust renders in the
// page; Sourcepoint in an iframe. Only a free "reject" counts: "Reject all and subscribe" doesn't.
const REJECT = { OneTrust: '#onetrust-reject-all-handler', Sourcepoint: 'button[title="Reject all"], button[title="I do not agree"]' };
const ACCEPT = { OneTrust: '#onetrust-accept-btn-handler', Sourcepoint: 'button[title="Accept all"], button[title="I agree"]' };
const SP_FRAME = 'iframe[id^="sp_message_iframe"]';
const BANNER = '#onetrust-banner-sdk, iframe[id^="sp_message_iframe"]';

// Which consent platform the page runs, whether its banner is showing, and which consent
// APIs it exposes (__tcfapi: IAB Europe's TCF; __gpp: IAB Tech Lab's GPP, used for US state
// privacy laws).
const PROBE = `() => {
  const shown = (s) => [...document.querySelectorAll(s)].some((e) => e.getClientRects().length && getComputedStyle(e).visibility !== 'hidden');
  const cmp = window.OneTrust || document.querySelector('#onetrust-consent-sdk') ? 'OneTrust'
    : window._sp_ || document.querySelector('iframe[id^="sp_message_iframe"]') ? 'Sourcepoint'
    : window.Didomi ? 'Didomi' : window.Cookiebot ? 'Cookiebot' : window.UC_UI ? 'Usercentrics' : null;
  return { cmp, banner: shown('#onetrust-banner-sdk, iframe[id^="sp_message_iframe"]'),
    tcf: typeof window.__tcfapi === 'function', gpp: typeof window.__gpp === 'function' };
}`;

async function launch(proxy) {
  const res = await fetch('https://starter.cdpfleet.com/chromium/session', {
    method: 'POST',
    headers: { 'x-api-key': KEY, 'content-type': 'application/json' },
    body: JSON.stringify({ proxy, headless: false }),
  });
  if (!res.ok) throw new Error(`launch ${res.status} ${await res.text()}`);
  const { wsUrl } = await res.json();
  return chromium.connect(wsUrl, { headers: { 'x-api-key': KEY } });
}

const onDomain = (host, domain) => host === domain || host.endsWith(`.${domain}`);

// Clicks a consent button; true if the click went through (one retry: a consent iframe that
// re-renders can swallow a click).
async function press(button) {
  for (let attempt = 1; attempt <= 2; attempt++) {
    try { await button.click({ timeout: 10000 }); return true; } catch { /* retry once */ }
  }
  return false;
}

// Load a site in a fresh context, then optionally click a consent button. Counts cookies and
// the hosts on other domains the page contacted (ads, analytics, CDNs) up to each moment.
async function visit(browser, url, choice) {
  const domain = new URL(url).hostname.split('.').slice(-2).join('.');
  const ctx = await browser.newContext({ viewport: { width: 1280, height: 900 } });
  const page = await ctx.newPage();
  const hosts = new Set();
  page.on('request', (r) => { const h = new URL(r.url()).hostname; if (h && !onDomain(h, domain)) hosts.add(h); });
  const measure = async () => {
    const cookies = await ctx.cookies();
    return { cookies: cookies.length, third_party_cookies: cookies.filter((c) => !onDomain(c.domain.replace(/^\./, ''), domain)).length, other_hosts: hosts.size };
  };
  try {
    // Proxies drop a connection now and then: one retry.
    const open = () => page.goto(url, { waitUntil: 'domcontentloaded', timeout: 60000 });
    const res = await open().catch(open);
    // Consent scripts load late on busy pages: wait up to 20 s for the banner, and give
    // every page at least 6 s before measuring.
    const t0 = Date.now();
    const seen = await page.locator(BANNER).first().waitFor({ state: 'visible', timeout: 20000 }).then(() => true, () => false);
    const bannerAfterMs = seen ? Date.now() - t0 : null;
    if (Date.now() - t0 < 6000) await page.waitForTimeout(6000 - (Date.now() - t0));
    const probe = await page.evaluate(`(${PROBE})()`);
    const before = { status: res?.status() ?? null, ...probe, banner_after_ms: bannerAfterMs, ...(await measure()) };
    if (!choice) return { before };
    const selector = (choice === 'reject' ? REJECT : ACCEPT)[probe.cmp];
    const scope = probe.cmp === 'Sourcepoint' ? page.frameLocator(SP_FRAME) : page;
    const button = selector && scope.locator(selector).first();
    if (!button || !(await button.waitFor({ state: 'visible', timeout: 10000 }).then(() => true, () => false))) {
      return { before, after: null, missing: probe.banner ? `no ${choice} button on the first layer` : 'no banner' };
    }
    const label = (await button.getAttribute('title')) || (await button.innerText()).trim();
    await page.waitForTimeout(2000); // let the banner finish animating in, or the click can be lost
    if (!(await press(button))) return { before, after: null, missing: 'click failed' };
    // If the banner is still up, the click was lost: click once more.
    let clicks = 1;
    if (!(await page.locator(BANNER).first().waitFor({ state: 'hidden', timeout: 5000 }).then(() => true, () => false))) {
      clicks = 2;
      await press(button);
    }
    await page.waitForTimeout(6000);
    return { before, after: { button: label, clicks, ...(await measure()) } };
  } finally {
    await ctx.close();
  }
}

// A site may refuse one exit IP (an error status): visit again in a fresh context, whose new
// proxy connection gets another IP from the rotating proxy.
async function visitOnce(browser, url, choice) {
  const r = await visit(browser, url, choice);
  return r.before.status >= 200 && r.before.status < 300 ? r : visit(browser, url, choice);
}

// Sessions on shared threads end after 60 s without traffic, so the countries run one after the other.
const fromUs = [];
const us = await launch(process.env.PROXY_URL_US);
try {
  for (const url of SITES) fromUs.push(await visitOnce(us, url));
} finally {
  await us.close();
}
const sites = [];
const de = await launch(process.env.PROXY_URL_DE);
try {
  for (const [i, url] of SITES.entries()) {
    const rejected = await visitOnce(de, url, 'reject');
    const accepted = await visitOnce(de, url, 'accept');
    sites.push({
      site: new URL(url).hostname.replace(/^www\./, ''), us: fromUs[i].before, de: rejected.before,
      de_reject: rejected.after, de_reject_missing: rejected.missing ?? null, de_accept: accepted.after, de_accept_missing: accepted.missing ?? null,
    });
  }
} finally {
  await de.close();
}
console.log(JSON.stringify({ sites }, null, 2));

# pip install playwright==1.60.0 requests
# env: CDPFLEET_API_KEY, PROXY_URL_DE (a German proxy), PROXY_URL_US (a US proxy)
import json
import os
import time
from urllib.parse import urlparse

import requests
from playwright.sync_api import sync_playwright

KEY = os.environ["CDPFLEET_API_KEY"]
SITES = ["https://www.theguardian.com/international", "https://www.bbc.com/", "https://apnews.com/", "https://www.theverge.com/", "https://www.huffpost.com/"]
# First-layer buttons of the two consent platforms these sites use. OneTrust renders in the
# page; Sourcepoint in an iframe. Only a free "reject" counts: "Reject all and subscribe" doesn't.
REJECT = {"OneTrust": "#onetrust-reject-all-handler", "Sourcepoint": 'button[title="Reject all"], button[title="I do not agree"]'}
ACCEPT = {"OneTrust": "#onetrust-accept-btn-handler", "Sourcepoint": 'button[title="Accept all"], button[title="I agree"]'}
SP_FRAME = 'iframe[id^="sp_message_iframe"]'
BANNER = '#onetrust-banner-sdk, iframe[id^="sp_message_iframe"]'

# Which consent platform the page runs, whether its banner is showing, and which consent
# APIs it exposes (__tcfapi: IAB Europe's TCF; __gpp: IAB Tech Lab's GPP, used for US state
# privacy laws).
PROBE = """() => {
  const shown = (s) => [...document.querySelectorAll(s)].some((e) => e.getClientRects().length && getComputedStyle(e).visibility !== 'hidden');
  const cmp = window.OneTrust || document.querySelector('#onetrust-consent-sdk') ? 'OneTrust'
    : window._sp_ || document.querySelector('iframe[id^="sp_message_iframe"]') ? 'Sourcepoint'
    : window.Didomi ? 'Didomi' : window.Cookiebot ? 'Cookiebot' : window.UC_UI ? 'Usercentrics' : null;
  return { cmp, banner: shown('#onetrust-banner-sdk, iframe[id^="sp_message_iframe"]'),
    tcf: typeof window.__tcfapi === 'function', gpp: typeof window.__gpp === 'function' };
}"""


def launch(p, proxy):
    res = requests.post("https://starter.cdpfleet.com/chromium/session",
        headers={"x-api-key": KEY}, json={"proxy": proxy, "headless": False}, timeout=60)
    if not res.ok:
        raise RuntimeError(f"launch {res.status_code} {res.text}")
    return p.chromium.connect(res.json()["wsUrl"], headers={"x-api-key": KEY})


def on_domain(host, domain):
    return host == domain or host.endswith("." + domain)


def press(button):
    """Clicks a consent button; True if the click went through (one retry: a consent iframe that
    re-renders can swallow a click)."""
    for _ in range(2):
        try:
            button.click(timeout=10000)
            return True
        except Exception:
            pass
    return False


def visit(browser, url, choice=None):
    """Load a site in a fresh context, then optionally click a consent button. Counts cookies and
    the hosts on other domains the page contacted (ads, analytics, CDNs) up to each moment."""
    domain = ".".join(urlparse(url).hostname.split(".")[-2:])
    ctx = browser.new_context(viewport={"width": 1280, "height": 900})
    page = ctx.new_page()
    hosts = set()

    def on_request(r):
        h = urlparse(r.url).hostname or ""
        if h and not on_domain(h, domain):
            hosts.add(h)
    page.on("request", on_request)

    def measure():
        cookies = ctx.cookies()
        return {"cookies": len(cookies), "third_party_cookies": len([c for c in cookies if not on_domain(c["domain"].lstrip("."), domain)]), "other_hosts": len(hosts)}

    def open_page():
        return page.goto(url, wait_until="domcontentloaded", timeout=60000)

    try:
        try:
            res = open_page()
        except Exception:  # proxies drop a connection now and then: one retry
            res = open_page()
        # Consent scripts load late on busy pages: wait up to 20 s for the banner, and give
        # every page at least 6 s before measuring.
        t0 = time.monotonic()
        try:
            page.locator(BANNER).first.wait_for(state="visible", timeout=20000)
            banner_after_ms = round((time.monotonic() - t0) * 1000)
        except Exception:
            banner_after_ms = None
        elapsed = (time.monotonic() - t0) * 1000
        if elapsed < 6000:
            page.wait_for_timeout(6000 - elapsed)
        probe = page.evaluate(f"({PROBE})()")
        before = {"status": res.status if res else None, **probe, "banner_after_ms": banner_after_ms, **measure()}
        if not choice:
            return {"before": before}
        missing = f"no {choice} button on the first layer" if probe["banner"] else "no banner"
        selector = (REJECT if choice == "reject" else ACCEPT).get(probe["cmp"])
        if not selector:
            return {"before": before, "after": None, "missing": missing}
        scope = page.frame_locator(SP_FRAME) if probe["cmp"] == "Sourcepoint" else page
        button = scope.locator(selector).first
        try:
            button.wait_for(state="visible", timeout=10000)
        except Exception:
            return {"before": before, "after": None, "missing": missing}
        label = button.get_attribute("title") or button.inner_text().strip()
        page.wait_for_timeout(2000)  # let the banner finish animating in, or the click can be lost
        if not press(button):
            return {"before": before, "after": None, "missing": "click failed"}
        clicks = 1
        try:  # if the banner is still up, the click was lost: click once more
            page.locator(BANNER).first.wait_for(state="hidden", timeout=5000)
        except Exception:
            clicks = 2
            press(button)
        page.wait_for_timeout(6000)
        return {"before": before, "after": {"button": label, "clicks": clicks, **measure()}}
    finally:
        ctx.close()


def visit_once(browser, url, choice=None):
    """A site may refuse one exit IP (an error status): visit again in a fresh context, whose new
    proxy connection gets another IP from the rotating proxy."""
    r = visit(browser, url, choice)
    status = r["before"]["status"]
    return r if status and 200 <= status < 300 else visit(browser, url, choice)


with sync_playwright() as p:
    # Sessions on shared threads end after 60 s without traffic, so the countries run one after the other.
    us = launch(p, os.environ["PROXY_URL_US"])
    try:
        from_us = [visit_once(us, url) for url in SITES]
    finally:
        us.close()
    sites = []
    de = launch(p, os.environ["PROXY_URL_DE"])
    try:
        for i, url in enumerate(SITES):
            rejected = visit_once(de, url, "reject")
            accepted = visit_once(de, url, "accept")
            sites.append({"site": urlparse(url).hostname.removeprefix("www."), "us": from_us[i]["before"], "de": rejected["before"],
                          "de_reject": rejected["after"], "de_reject_missing": rejected.get("missing"),
                          "de_accept": accepted["after"], "de_accept_missing": accepted.get("missing")})
    finally:
        de.close()
    print(json.dumps({"sites": sites}, indent=2))

// Maven: com.microsoft.playwright:playwright:1.60.0, com.google.code.gson:gson:2.11.0
// Run with PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1. env: CDPFLEET_API_KEY, PROXY_URL_DE (a German proxy), PROXY_URL_US (a US proxy)
import com.google.gson.*;
import com.microsoft.playwright.*;
import com.microsoft.playwright.options.WaitForSelectorState;
import com.microsoft.playwright.options.WaitUntilState;
import java.net.URI;
import java.net.http.*;
import java.util.*;

public class Main {
  static final String KEY = System.getenv("CDPFLEET_API_KEY");
  static final Gson GSON = new GsonBuilder().serializeNulls().create();
  static final List<String> SITES = List.of("https://www.theguardian.com/international", "https://www.bbc.com/", "https://apnews.com/", "https://www.theverge.com/", "https://www.huffpost.com/");
  // First-layer buttons of the two consent platforms these sites use. OneTrust renders in the
  // page; Sourcepoint in an iframe. Only a free "reject" counts: "Reject all and subscribe" doesn't.
  static final Map<String, String> REJECT = Map.of("OneTrust", "#onetrust-reject-all-handler", "Sourcepoint", "button[title=\"Reject all\"], button[title=\"I do not agree\"]");
  static final Map<String, String> ACCEPT = Map.of("OneTrust", "#onetrust-accept-btn-handler", "Sourcepoint", "button[title=\"Accept all\"], button[title=\"I agree\"]");
  static final String SP_FRAME = "iframe[id^=\"sp_message_iframe\"]";
  static final String BANNER = "#onetrust-banner-sdk, iframe[id^=\"sp_message_iframe\"]";

  // Which consent platform the page runs, whether its banner is showing, and which consent
  // APIs it exposes (__tcfapi: IAB Europe's TCF; __gpp: IAB Tech Lab's GPP, used for US state
  // privacy laws).
  static final String PROBE = """
      () => {
        const shown = (s) => [...document.querySelectorAll(s)].some((e) => e.getClientRects().length && getComputedStyle(e).visibility !== 'hidden');
        const cmp = window.OneTrust || document.querySelector('#onetrust-consent-sdk') ? 'OneTrust'
          : window._sp_ || document.querySelector('iframe[id^="sp_message_iframe"]') ? 'Sourcepoint'
          : window.Didomi ? 'Didomi' : window.Cookiebot ? 'Cookiebot' : window.UC_UI ? 'Usercentrics' : null;
        return { cmp, banner: shown('#onetrust-banner-sdk, iframe[id^="sp_message_iframe"]'),
          tcf: typeof window.__tcfapi === 'function', gpp: typeof window.__gpp === 'function' };
      }""";

  static Browser launch(Playwright playwright, String proxy) throws Exception {
    String body = "{\"proxy\": " + GSON.toJson(proxy) + ", \"headless\": false}";
    HttpResponse<String> res = HttpClient.newHttpClient().send(HttpRequest.newBuilder(URI.create("https://starter.cdpfleet.com/chromium/session"))
        .header("x-api-key", KEY).header("content-type", "application/json")
        .POST(HttpRequest.BodyPublishers.ofString(body)).build(), HttpResponse.BodyHandlers.ofString());
    if (res.statusCode() / 100 != 2) throw new RuntimeException("launch: " + res.statusCode() + " " + res.body());
    String wsUrl = JsonParser.parseString(res.body()).getAsJsonObject().get("wsUrl").getAsString();
    return playwright.chromium().connect(wsUrl, new BrowserType.ConnectOptions().setHeaders(Map.of("x-api-key", KEY)));
  }

  // Request URLs aren't always valid java.net.URI input (unescaped {} or "), so cut the host out.
  static String hostOf(String url) {
    return url.replaceFirst("^[a-zA-Z][a-zA-Z0-9+.-]*://", "").split("[/?#:]", 2)[0];
  }

  static String domainOf(String url) {
    String[] parts = hostOf(url).split("\\.");
    return parts[parts.length - 2] + "." + parts[parts.length - 1];
  }

  static boolean onDomain(String host, String domain) {
    return host.equals(domain) || host.endsWith("." + domain);
  }

  // Clicks a consent button; true if the click went through (one retry: a consent iframe that
  // re-renders can swallow a click).
  static boolean press(Locator button) {
    for (int attempt = 1; attempt <= 2; attempt++) {
      try {
        button.click(new Locator.ClickOptions().setTimeout(10000));
        return true;
      } catch (PlaywrightException e) {
        // retry once
      }
    }
    return false;
  }

  static JsonObject measure(BrowserContext ctx, Set<String> hosts, String domain) {
    List<com.microsoft.playwright.options.Cookie> cookies = ctx.cookies();
    JsonObject m = new JsonObject();
    m.addProperty("cookies", cookies.size());
    m.addProperty("third_party_cookies", cookies.stream().filter(c -> !onDomain(c.domain.replaceFirst("^\\.", ""), domain)).count());
    m.addProperty("other_hosts", hosts.size());
    return m;
  }

  // Load a site in a fresh context, then optionally click a consent button. Counts cookies and
  // the hosts on other domains the page contacted (ads, analytics, CDNs) up to each moment.
  // Returns [before, after, missing]; after is null (and missing says why) when there was
  // nothing to click.
  static JsonElement[] visit(Browser browser, String url, String choice) {
    String domain = domainOf(url);
    BrowserContext ctx = browser.newContext(new Browser.NewContextOptions().setViewportSize(1280, 900));
    Page page = ctx.newPage();
    Set<String> hosts = new HashSet<>();
    page.onRequest(r -> {
      String h = r.url().startsWith("http") ? hostOf(r.url()) : "";
      if (!h.isEmpty() && !onDomain(h, domain)) hosts.add(h);
    });
    try {
      Page.NavigateOptions load = new Page.NavigateOptions().setWaitUntil(WaitUntilState.DOMCONTENTLOADED).setTimeout(60000);
      Response res;
      try {
        res = page.navigate(url, load);
      } catch (PlaywrightException e) { // proxies drop a connection now and then: one retry
        res = page.navigate(url, load);
      }
      // Consent scripts load late on busy pages: wait up to 20 s for the banner, and give
      // every page at least 6 s before measuring.
      long t0 = System.nanoTime();
      Long bannerAfterMs = null;
      try {
        page.locator(BANNER).first().waitFor(new Locator.WaitForOptions().setState(WaitForSelectorState.VISIBLE).setTimeout(20000));
        bannerAfterMs = (System.nanoTime() - t0) / 1_000_000;
      } catch (PlaywrightException e) {
        // no banner
      }
      long elapsed = (System.nanoTime() - t0) / 1_000_000;
      if (elapsed < 6000) page.waitForTimeout(6000 - elapsed);
      JsonObject probe = GSON.toJsonTree(page.evaluate("(" + PROBE + ")()")).getAsJsonObject();
      JsonObject before = new JsonObject();
      if (res == null) before.add("status", JsonNull.INSTANCE); else before.addProperty("status", res.status());
      for (Map.Entry<String, JsonElement> e : probe.entrySet()) before.add(e.getKey(), e.getValue());
      before.addProperty("banner_after_ms", bannerAfterMs);
      for (Map.Entry<String, JsonElement> e : measure(ctx, hosts, domain).entrySet()) before.add(e.getKey(), e.getValue());
      if (choice == null) return new JsonElement[] { before, JsonNull.INSTANCE, JsonNull.INSTANCE };
      JsonPrimitive missing = new JsonPrimitive(probe.get("banner").getAsBoolean() ? "no " + choice + " button on the first layer" : "no banner");
      String cmp = probe.get("cmp").isJsonNull() ? null : probe.get("cmp").getAsString();
      String selector = cmp == null ? null : (choice.equals("reject") ? REJECT : ACCEPT).get(cmp);
      if (selector == null) return new JsonElement[] { before, JsonNull.INSTANCE, missing };
      Locator button = "Sourcepoint".equals(cmp) ? page.frameLocator(SP_FRAME).locator(selector).first() : page.locator(selector).first();
      try {
        button.waitFor(new Locator.WaitForOptions().setState(WaitForSelectorState.VISIBLE).setTimeout(10000));
      } catch (PlaywrightException e) {
        return new JsonElement[] { before, JsonNull.INSTANCE, missing };
      }
      String title = button.getAttribute("title");
      String label = title != null ? title : button.innerText().trim();
      page.waitForTimeout(2000); // let the banner finish animating in, or the click can be lost
      if (!press(button)) return new JsonElement[] { before, JsonNull.INSTANCE, new JsonPrimitive("click failed") };
      int clicks = 1;
      try { // if the banner is still up, the click was lost: click once more
        page.locator(BANNER).first().waitFor(new Locator.WaitForOptions().setState(WaitForSelectorState.HIDDEN).setTimeout(5000));
      } catch (PlaywrightException e) {
        clicks = 2;
        press(button);
      }
      page.waitForTimeout(6000);
      JsonObject after = new JsonObject();
      after.addProperty("button", label);
      after.addProperty("clicks", clicks);
      for (Map.Entry<String, JsonElement> e : measure(ctx, hosts, domain).entrySet()) after.add(e.getKey(), e.getValue());
      return new JsonElement[] { before, after, JsonNull.INSTANCE };
    } finally {
      ctx.close();
    }
  }

  // A site may refuse one exit IP (an error status): visit again in a fresh context, whose new
  // proxy connection gets another IP from the rotating proxy.
  static JsonElement[] visitOnce(Browser browser, String url, String choice) {
    JsonElement[] r = visit(browser, url, choice);
    JsonElement status = r[0].getAsJsonObject().get("status");
    return !status.isJsonNull() && status.getAsInt() / 100 == 2 ? r : visit(browser, url, choice);
  }

  public static void main(String[] args) throws Exception {
    try (Playwright playwright = Playwright.create()) {
      // Sessions on shared threads end after 60 s without traffic, so the countries run one after the other.
      List<JsonElement[]> fromUs = new ArrayList<>();
      Browser us = launch(playwright, System.getenv("PROXY_URL_US"));
      try {
        for (String url : SITES) fromUs.add(visitOnce(us, url, null));
      } finally {
        us.close();
      }
      Browser de = launch(playwright, System.getenv("PROXY_URL_DE"));
      try {
        JsonArray sites = new JsonArray();
        for (int i = 0; i < SITES.size(); i++) {
          String url = SITES.get(i);
          JsonElement[] rejected = visitOnce(de, url, "reject");
          JsonElement[] accepted = visitOnce(de, url, "accept");
          JsonObject site = new JsonObject();
          site.addProperty("site", hostOf(url).replaceFirst("^www\\.", ""));
          site.add("us", fromUs.get(i)[0]);
          site.add("de", rejected[0]);
          site.add("de_reject", rejected[1]);
          site.add("de_reject_missing", rejected[2]);
          site.add("de_accept", accepted[1]);
          site.add("de_accept_missing", accepted[2]);
          sites.add(site);
        }
        JsonObject out = new JsonObject();
        out.add("sites", sites);
        System.out.println(new GsonBuilder().serializeNulls().setPrettyPrinting().disableHtmlEscaping().create().toJson(out));
      } finally {
        de.close();
      }
    }
  }
}

// dotnet add package Microsoft.Playwright --version 1.60.0
// env: CDPFLEET_API_KEY, PROXY_URL_DE (a German proxy), PROXY_URL_US (a US proxy)
using System.Net.Http.Json;
using System.Text.Encodings.Web;
using System.Text.Json;
using System.Text.Json.Nodes;
using Microsoft.Playwright;

string[] sites = ["https://www.theguardian.com/international", "https://www.bbc.com/", "https://apnews.com/", "https://www.theverge.com/", "https://www.huffpost.com/"];
// First-layer buttons of the two consent platforms these sites use. OneTrust renders in the
// page; Sourcepoint in an iframe. Only a free "reject" counts: "Reject all and subscribe" doesn't.
var reject = new Dictionary<string, string> { ["OneTrust"] = "#onetrust-reject-all-handler", ["Sourcepoint"] = "button[title=\"Reject all\"], button[title=\"I do not agree\"]" };
var accept = new Dictionary<string, string> { ["OneTrust"] = "#onetrust-accept-btn-handler", ["Sourcepoint"] = "button[title=\"Accept all\"], button[title=\"I agree\"]" };
const string SpFrame = "iframe[id^=\"sp_message_iframe\"]";
const string Banner = "#onetrust-banner-sdk, iframe[id^=\"sp_message_iframe\"]";

// Which consent platform the page runs, whether its banner is showing, and which consent
// APIs it exposes (__tcfapi: IAB Europe's TCF; __gpp: IAB Tech Lab's GPP, used for US state
// privacy laws).
const string Probe = """
    () => {
      const shown = (s) => [...document.querySelectorAll(s)].some((e) => e.getClientRects().length && getComputedStyle(e).visibility !== 'hidden');
      const cmp = window.OneTrust || document.querySelector('#onetrust-consent-sdk') ? 'OneTrust'
        : window._sp_ || document.querySelector('iframe[id^="sp_message_iframe"]') ? 'Sourcepoint'
        : window.Didomi ? 'Didomi' : window.Cookiebot ? 'Cookiebot' : window.UC_UI ? 'Usercentrics' : null;
      return { cmp, banner: shown('#onetrust-banner-sdk, iframe[id^="sp_message_iframe"]'),
        tcf: typeof window.__tcfapi === 'function', gpp: typeof window.__gpp === 'function' };
    }
    """;

var key = Environment.GetEnvironmentVariable("CDPFLEET_API_KEY")!;
using var http = new HttpClient();
http.DefaultRequestHeaders.Add("x-api-key", key);
using var playwright = await Playwright.CreateAsync();

async Task<IBrowser> Launch(string proxy)
{
    var res = await http.PostAsJsonAsync("https://starter.cdpfleet.com/chromium/session", new { proxy, headless = false });
    if (!res.IsSuccessStatusCode) throw new Exception($"launch: {(int)res.StatusCode} {await res.Content.ReadAsStringAsync()}");
    var wsUrl = (await res.Content.ReadFromJsonAsync<JsonElement>()).GetProperty("wsUrl").GetString()!;
    return await playwright.Chromium.ConnectAsync(wsUrl, new() { Headers = new Dictionary<string, string> { ["x-api-key"] = key } });
}

static bool OnDomain(string host, string domain) => host == domain || host.EndsWith("." + domain, StringComparison.Ordinal);

// Clicks a consent button; true if the click went through (one retry: a consent iframe that
// re-renders can swallow a click).
static async Task<bool> Press(ILocator button)
{
    for (var attempt = 1; attempt <= 2; attempt++)
    {
        try { await button.ClickAsync(new() { Timeout = 10000 }); return true; }
        catch (Exception) { /* retry once */ }
    }
    return false;
}

// Load a site in a fresh context, then optionally click a consent button. Counts cookies and
// the hosts on other domains the page contacted (ads, analytics, CDNs) up to each moment.
// After is null (and Missing says why) when there was nothing to click.
async Task<(JsonObject Before, JsonObject? After, string? Missing)> Visit(IBrowser browser, string url, string? choice)
{
    var domain = string.Join('.', new Uri(url).Host.Split('.')[^2..]);
    var ctx = await browser.NewContextAsync(new() { ViewportSize = new() { Width = 1280, Height = 900 } });
    var page = await ctx.NewPageAsync();
    var hosts = new HashSet<string>();
    page.Request += (_, r) =>
    {
        var h = Uri.TryCreate(r.Url, UriKind.Absolute, out var u) ? u.Host : "";
        if (h != "" && !OnDomain(h, domain)) lock (hosts) hosts.Add(h);
    };
    async Task<JsonObject> Measure()
    {
        var cookies = await ctx.CookiesAsync();
        int count;
        lock (hosts) count = hosts.Count;
        return new JsonObject
        {
            ["cookies"] = cookies.Count,
            ["third_party_cookies"] = cookies.Count(c => !OnDomain(c.Domain.TrimStart('.'), domain)),
            ["other_hosts"] = count,
        };
    }
    var load = new PageGotoOptions { WaitUntil = WaitUntilState.DOMContentLoaded, Timeout = 60000 };
    try
    {
        IResponse? res;
        try { res = await page.GotoAsync(url, load); }
        catch (PlaywrightException) { res = await page.GotoAsync(url, load); } // proxies drop a connection now and then: one retry
        // Consent scripts load late on busy pages: wait up to 20 s for the banner, and give
        // every page at least 6 s before measuring.
        var clock = System.Diagnostics.Stopwatch.StartNew();
        long? bannerAfterMs = null;
        try
        {
            await page.Locator(Banner).First.WaitForAsync(new() { State = WaitForSelectorState.Visible, Timeout = 20000 });
            bannerAfterMs = clock.ElapsedMilliseconds;
        }
        catch (Exception) { /* no banner */ }
        if (clock.ElapsedMilliseconds < 6000) await page.WaitForTimeoutAsync(6000 - clock.ElapsedMilliseconds);
        var probe = (JsonObject)Plain(await page.EvaluateAsync<JsonElement>($"({Probe})()"))!;
        var before = new JsonObject { ["status"] = res?.Status };
        foreach (var prop in probe.ToList()) before[prop.Key] = prop.Value?.DeepClone();
        before["banner_after_ms"] = bannerAfterMs;
        foreach (var prop in (await Measure()).ToList()) before[prop.Key] = prop.Value?.DeepClone();
        if (choice is null) return (before, null, null);
        var missing = probe["banner"]!.GetValue<bool>() ? $"no {choice} button on the first layer" : "no banner";
        var cmp = probe["cmp"]?.GetValue<string>();
        if (cmp is null || !(choice == "reject" ? reject : accept).TryGetValue(cmp, out var selector)) return (before, null, missing);
        var button = cmp == "Sourcepoint" ? page.FrameLocator(SpFrame).Locator(selector).First : page.Locator(selector).First;
        try { await button.WaitForAsync(new() { State = WaitForSelectorState.Visible, Timeout = 10000 }); }
        catch (Exception) { return (before, null, missing); }
        var label = await button.GetAttributeAsync("title") ?? (await button.InnerTextAsync()).Trim();
        await page.WaitForTimeoutAsync(2000); // let the banner finish animating in, or the click can be lost
        if (!await Press(button)) return (before, null, "click failed");
        // If the banner is still up, the click was lost: click once more.
        var clicks = 1;
        try { await page.Locator(Banner).First.WaitForAsync(new() { State = WaitForSelectorState.Hidden, Timeout = 5000 }); }
        catch (Exception)
        {
            clicks = 2;
            await Press(button);
        }
        await page.WaitForTimeoutAsync(6000);
        var after = new JsonObject { ["button"] = label, ["clicks"] = clicks };
        foreach (var prop in (await Measure()).ToList()) after[prop.Key] = prop.Value?.DeepClone();
        return (before, after, null);
    }
    finally
    {
        await ctx.CloseAsync();
    }
}

// A site may refuse one exit IP (an error status): visit again in a fresh context, whose new
// proxy connection gets another IP from the rotating proxy.
async Task<(JsonObject Before, JsonObject? After, string? Missing)> VisitOnce(IBrowser browser, string url, string? choice)
{
    var r = await Visit(browser, url, choice);
    var status = r.Before["status"]?.GetValue<int>();
    return status is >= 200 and < 300 ? r : await Visit(browser, url, choice);
}

// Sessions on shared threads end after 60 s without traffic, so the countries run one after the other.
var fromUs = new List<(JsonObject Before, JsonObject? After, string? Missing)>();
var us = await Launch(Environment.GetEnvironmentVariable("PROXY_URL_US")!);
try
{
    foreach (var url in sites) fromUs.Add(await VisitOnce(us, url, null));
}
finally
{
    await us.CloseAsync();
}
var de = await Launch(Environment.GetEnvironmentVariable("PROXY_URL_DE")!);
try
{
    var results = new JsonArray();
    for (var i = 0; i < sites.Length; i++)
    {
        var url = sites[i];
        var rejected = await VisitOnce(de, url, "reject");
        var accepted = await VisitOnce(de, url, "accept");
        var host = new Uri(url).Host;
        results.Add(new JsonObject
        {
            ["site"] = host.StartsWith("www.") ? host[4..] : host,
            ["us"] = fromUs[i].Before,
            ["de"] = rejected.Before,
            ["de_reject"] = rejected.After,
            ["de_reject_missing"] = rejected.Missing,
            ["de_accept"] = accepted.After,
            ["de_accept_missing"] = accepted.Missing,
        });
    }
    var result = new JsonObject { ["sites"] = results };
    Console.WriteLine(result.ToJsonString(new JsonSerializerOptions { WriteIndented = true, Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping }));
}
finally
{
    await de.CloseAsync();
}

// EvaluateAsync<JsonElement> tags every object with a "$id" reference key; drop those.
static JsonNode? Plain(JsonElement e) => e.ValueKind switch
{
    JsonValueKind.Object => new JsonObject(e.EnumerateObject().Where(p => p.Name != "$id")
        .Select(p => KeyValuePair.Create(p.Name, Plain(p.Value)))),
    JsonValueKind.Array => new JsonArray(e.EnumerateArray().Select(Plain).ToArray()),
    _ => JsonNode.Parse(e.GetRawText()),
};

// go get github.com/playwright-community/[email protected]
// Driver: build playwright-core 1.60.0 from npm and set PLAYWRIGHT_DRIVER_PATH (see /docs/quickstart).
// env: CDPFLEET_API_KEY, PROXY_URL_DE (a German proxy), PROXY_URL_US (a US proxy)
package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io"
	"log"
	"net/http"
	"net/url"
	"os"
	"strings"
	"sync"
	"time"

	"github.com/playwright-community/playwright-go"
)

var key = os.Getenv("CDPFLEET_API_KEY")

var sites = []string{"https://www.theguardian.com/international", "https://www.bbc.com/", "https://apnews.com/", "https://www.theverge.com/", "https://www.huffpost.com/"}

// First-layer buttons of the two consent platforms these sites use. OneTrust renders in the
// page; Sourcepoint in an iframe. Only a free "reject" counts: "Reject all and subscribe" doesn't.
var reject = map[string]string{"OneTrust": "#onetrust-reject-all-handler", "Sourcepoint": `button[title="Reject all"], button[title="I do not agree"]`}
var accept = map[string]string{"OneTrust": "#onetrust-accept-btn-handler", "Sourcepoint": `button[title="Accept all"], button[title="I agree"]`}

const spFrame = `iframe[id^="sp_message_iframe"]`
const banner = `#onetrust-banner-sdk, iframe[id^="sp_message_iframe"]`

// Which consent platform the page runs, whether its banner is showing, and which consent
// APIs it exposes (__tcfapi: IAB Europe's TCF; __gpp: IAB Tech Lab's GPP, used for US state
// privacy laws).
const probe = `() => {
  const shown = (s) => [...document.querySelectorAll(s)].some((e) => e.getClientRects().length && getComputedStyle(e).visibility !== 'hidden');
  const cmp = window.OneTrust || document.querySelector('#onetrust-consent-sdk') ? 'OneTrust'
    : window._sp_ || document.querySelector('iframe[id^="sp_message_iframe"]') ? 'Sourcepoint'
    : window.Didomi ? 'Didomi' : window.Cookiebot ? 'Cookiebot' : window.UC_UI ? 'Usercentrics' : null;
  return { cmp, banner: shown('#onetrust-banner-sdk, iframe[id^="sp_message_iframe"]'),
    tcf: typeof window.__tcfapi === 'function', gpp: typeof window.__gpp === 'function' };
}`

type before struct {
	Status            *int   `json:"status"`
	CMP               any    `json:"cmp"`
	Banner            bool   `json:"banner"`
	TCF               bool   `json:"tcf"`
	GPP               bool   `json:"gpp"`
	BannerAfterMs     *int64 `json:"banner_after_ms"`
	Cookies           int    `json:"cookies"`
	ThirdPartyCookies int    `json:"third_party_cookies"`
	OtherHosts        int    `json:"other_hosts"`
}

type after struct {
	Button            string `json:"button"`
	Clicks            int    `json:"clicks"`
	Cookies           int    `json:"cookies"`
	ThirdPartyCookies int    `json:"third_party_cookies"`
	OtherHosts        int    `json:"other_hosts"`
}

type siteResult struct {
	Site            string  `json:"site"`
	US              *before `json:"us"`
	DE              *before `json:"de"`
	DEReject        *after  `json:"de_reject"`
	DERejectMissing *string `json:"de_reject_missing"`
	DEAccept        *after  `json:"de_accept"`
	DEAcceptMissing *string `json:"de_accept_missing"`
}

func launch(proxy string) (string, error) {
	body, _ := json.Marshal(map[string]any{"proxy": proxy, "headless": false})
	req, _ := http.NewRequest("POST", "https://starter.cdpfleet.com/chromium/session", bytes.NewReader(body))
	req.Header.Set("x-api-key", key)
	req.Header.Set("content-type", "application/json")
	res, err := http.DefaultClient.Do(req)
	if err != nil {
		return "", err
	}
	defer res.Body.Close()
	if res.StatusCode/100 != 2 {
		msg, _ := io.ReadAll(res.Body)
		return "", fmt.Errorf("launch: %s %s", res.Status, msg)
	}
	var session struct {
		WsURL string `json:"wsUrl"`
	}
	return session.WsURL, json.NewDecoder(res.Body).Decode(&session)
}

func domainOf(u string) string {
	parsed, _ := url.Parse(u)
	parts := strings.Split(parsed.Hostname(), ".")
	return strings.Join(parts[len(parts)-2:], ".")
}

func onDomain(host, domain string) bool {
	return host == domain || strings.HasSuffix(host, "."+domain)
}

// press clicks a consent button and reports whether the click went through (one retry: a
// consent iframe that re-renders can swallow a click).
func press(button playwright.Locator) bool {
	for attempt := 1; attempt <= 2; attempt++ {
		if button.Click(playwright.LocatorClickOptions{Timeout: playwright.Float(10000)}) == nil {
			return true
		}
	}
	return false
}

// visit loads a site in a fresh context, then optionally clicks a consent button. It counts
// cookies and the hosts on other domains the page contacted (ads, analytics, CDNs) up to each
// moment. after is nil (and missing says why) when there was nothing to click.
func visit(browser playwright.Browser, site, choice string) (*before, *after, *string, error) {
	domain := domainOf(site)
	ctx, err := browser.NewContext(playwright.BrowserNewContextOptions{Viewport: &playwright.Size{Width: 1280, Height: 900}})
	if err != nil {
		return nil, nil, nil, err
	}
	defer ctx.Close()
	page, err := ctx.NewPage()
	if err != nil {
		return nil, nil, nil, err
	}
	var mu sync.Mutex
	hosts := map[string]bool{}
	page.On("request", func(r playwright.Request) {
		u, err := url.Parse(r.URL())
		if err != nil || u.Hostname() == "" || onDomain(u.Hostname(), domain) {
			return
		}
		mu.Lock()
		hosts[u.Hostname()] = true
		mu.Unlock()
	})
	measure := func() (int, int, int, error) {
		cookies, err := ctx.Cookies()
		if err != nil {
			return 0, 0, 0, err
		}
		third := 0
		for _, c := range cookies {
			if !onDomain(strings.TrimPrefix(c.Domain, "."), domain) {
				third++
			}
		}
		mu.Lock()
		defer mu.Unlock()
		return len(cookies), third, len(hosts), nil
	}

	load := playwright.PageGotoOptions{WaitUntil: playwright.WaitUntilStateDomcontentloaded, Timeout: playwright.Float(60000)}
	res, err := page.Goto(site, load)
	if err != nil { // proxies drop a connection now and then: one retry
		if res, err = page.Goto(site, load); err != nil {
			return nil, nil, nil, err
		}
	}
	// Consent scripts load late on busy pages: wait up to 20 s for the banner, and give every
	// page at least 6 s before measuring.
	t0 := time.Now()
	var bannerAfterMs *int64
	if page.Locator(banner).First().WaitFor(playwright.LocatorWaitForOptions{State: playwright.WaitForSelectorStateVisible, Timeout: playwright.Float(20000)}) == nil {
		ms := time.Since(t0).Milliseconds()
		bannerAfterMs = &ms
	}
	if elapsed := time.Since(t0).Milliseconds(); elapsed < 6000 {
		page.WaitForTimeout(float64(6000 - elapsed))
	}
	v, err := page.Evaluate("(" + probe + ")()")
	if err != nil {
		return nil, nil, nil, err
	}
	m := v.(map[string]any)
	b := &before{CMP: m["cmp"], Banner: m["banner"] == true, TCF: m["tcf"] == true, GPP: m["gpp"] == true, BannerAfterMs: bannerAfterMs}
	if res != nil {
		status := res.Status()
		b.Status = &status
	}
	if b.Cookies, b.ThirdPartyCookies, b.OtherHosts, err = measure(); err != nil {
		return nil, nil, nil, err
	}
	if choice == "" {
		return b, nil, nil, nil
	}
	missing := "no banner"
	if b.Banner {
		missing = "no " + choice + " button on the first layer"
	}
	cmp, _ := m["cmp"].(string)
	selectors := accept
	if choice == "reject" {
		selectors = reject
	}
	selector, ok := selectors[cmp]
	if !ok {
		return b, nil, &missing, nil
	}
	var button playwright.Locator
	if cmp == "Sourcepoint" {
		button = page.FrameLocator(spFrame).Locator(selector).First()
	} else {
		button = page.Locator(selector).First()
	}
	if err := button.WaitFor(playwright.LocatorWaitForOptions{State: playwright.WaitForSelectorStateVisible, Timeout: playwright.Float(10000)}); err != nil {
		return b, nil, &missing, nil
	}
	label, err := button.GetAttribute("title")
	if err != nil || label == "" {
		text, err := button.InnerText()
		if err != nil {
			return nil, nil, nil, err
		}
		label = strings.TrimSpace(text)
	}
	page.WaitForTimeout(2000) // let the banner finish animating in, or the click can be lost
	if !press(button) {
		failed := "click failed"
		return b, nil, &failed, nil
	}
	// If the banner is still up, the click was lost: click once more.
	clicks := 1
	if err := page.Locator(banner).First().WaitFor(playwright.LocatorWaitForOptions{State: playwright.WaitForSelectorStateHidden, Timeout: playwright.Float(5000)}); err != nil {
		clicks = 2
		press(button)
	}
	page.WaitForTimeout(6000)
	a := &after{Button: label, Clicks: clicks}
	if a.Cookies, a.ThirdPartyCookies, a.OtherHosts, err = measure(); err != nil {
		return nil, nil, nil, err
	}
	return b, a, nil, nil
}

// visitOnce: a site may refuse one exit IP (an error status), so visit again in a fresh
// context, whose new proxy connection gets another IP from the rotating proxy.
func visitOnce(browser playwright.Browser, site, choice string) (*before, *after, *string, error) {
	b, a, missing, err := visit(browser, site, choice)
	if err != nil || (b.Status != nil && *b.Status/100 == 2) {
		return b, a, missing, err
	}
	return visit(browser, site, choice)
}

func runUS(us playwright.Browser) ([]*before, error) {
	results := []*before{}
	for _, site := range sites {
		b, _, _, err := visitOnce(us, site, "")
		if err != nil {
			return nil, err
		}
		results = append(results, b)
	}
	return results, nil
}

func runDE(de playwright.Browser, fromUs []*before) ([]siteResult, error) {
	results := []siteResult{}
	for i, site := range sites {
		deBefore, rejected, rejectMissing, err := visitOnce(de, site, "reject")
		if err != nil {
			return nil, err
		}
		_, accepted, acceptMissing, err := visitOnce(de, site, "accept")
		if err != nil {
			return nil, err
		}
		u, _ := url.Parse(site)
		results = append(results, siteResult{Site: strings.TrimPrefix(u.Hostname(), "www."), US: fromUs[i], DE: deBefore,
			DEReject: rejected, DERejectMissing: rejectMissing, DEAccept: accepted, DEAcceptMissing: acceptMissing})
	}
	return results, nil
}

func main() {
	pw, err := playwright.Run(&playwright.RunOptions{SkipInstallBrowsers: true})
	if err != nil {
		log.Fatal(err)
	}
	defer pw.Stop()
	fatal := func(err error) { // log.Fatal would skip the deferred pw.Stop()
		pw.Stop()
		log.Fatal(err)
	}
	connect := func(proxy string) playwright.Browser {
		wsURL, err := launch(proxy)
		if err != nil {
			fatal(err)
		}
		browser, err := pw.Chromium.Connect(wsURL, playwright.BrowserTypeConnectOptions{Headers: map[string]string{"x-api-key": key}})
		if err != nil {
			fatal(err)
		}
		return browser
	}
	// Sessions on shared threads end after 60 s without traffic, so the countries run one after the other.
	us := connect(os.Getenv("PROXY_URL_US"))
	fromUs, err := runUS(us)
	us.Close()
	if err != nil {
		fatal(err)
	}
	de := connect(os.Getenv("PROXY_URL_DE"))
	results, err := runDE(de, fromUs)
	de.Close()
	if err != nil {
		fatal(err)
	}
	var buf bytes.Buffer
	enc := json.NewEncoder(&buf)
	enc.SetEscapeHTML(false)
	enc.SetIndent("", "  ")
	enc.Encode(map[string]any{"sites": results})
	fmt.Print(buf.String())
}

What we got

Site	Exit / choice	Banner	Cookies	Third-party cookies	Other-domain hosts
theguardian.com	US · Sourcepoint, GPP	yes, after 9.3 s	33	16	66
theguardian.com	DE · before a choice · Sourcepoint, TCF	yes, after 4.1 s	7	0	5
theguardian.com	DE · reject: no reject button on the first layer		—	—	—
theguardian.com	DE · "Accept all"		113	90	148
bbc.com	US · Sourcepoint, GPP, TCF	no	51	17	69
bbc.com	DE · before a choice · Sourcepoint, TCF, GPP	yes, after 2.5 s	18	2	16
bbc.com	DE · "I do not agree"		30	4	33
bbc.com	DE · "I agree"		44	15	48
apnews.com	US · OneTrust, GPP	yes, after 9.0 s	111	75	195
apnews.com	DE · before a choice · OneTrust, TCF	yes, after 0.4 s	26	3	72
apnews.com	DE · "I Reject All (except Strictly Necessary)"		35	3	79
apnews.com	DE · "I Accept All"		174	142	254
theverge.com	US · OneTrust, GPP	no	247	223	219
theverge.com	DE · before a choice · OneTrust, TCF	yes, after 1.9 s	8	0	34
theverge.com	DE · "Reject Non-Essential"		12	0	40
theverge.com	DE · "Accept"		48	14	87
huffpost.com	US · no platform, GPP	no	228	218	294
huffpost.com	DE · before a choice · OneTrust, TCF	yes, after 0.7 s	8	0	62
huffpost.com	DE · "Reject All"		11	0	65
huffpost.com	DE · "I Accept"		186	154	252

From the Node.js run on 2026-10-12. IP addresses are replaced with placeholders (203.0.113.x); equal addresses stay equal. The other languages produced the same findings.

Raw output (Node.js)

{
  "sites": [
    {
      "site": "theguardian.com",
      "us": {
        "status": 200,
        "cmp": "Sourcepoint",
        "banner": true,
        "tcf": false,
        "gpp": true,
        "banner_after_ms": 9289,
        "cookies": 33,
        "third_party_cookies": 16,
        "other_hosts": 66
      },
      "de": {
        "status": 200,
        "cmp": "Sourcepoint",
        "banner": true,
        "tcf": true,
        "gpp": false,
        "banner_after_ms": 4100,
        "cookies": 7,
        "third_party_cookies": 0,
        "other_hosts": 5
      },
      "de_reject": null,
      "de_reject_missing": "no reject button on the first layer",
      "de_accept": {
        "button": "Accept all",
        "clicks": 1,
        "cookies": 113,
        "third_party_cookies": 90,
        "other_hosts": 148
      },
      "de_accept_missing": null
    },
    {
      "site": "bbc.com",
      "us": {
        "status": 200,
        "cmp": "Sourcepoint",
        "banner": false,
        "tcf": true,
        "gpp": true,
        "banner_after_ms": null,
        "cookies": 51,
        "third_party_cookies": 17,
        "other_hosts": 69
      },
      "de": {
        "status": 200,
        "cmp": "Sourcepoint",
        "banner": true,
        "tcf": true,
        "gpp": true,
        "banner_after_ms": 2527,
        "cookies": 18,
        "third_party_cookies": 2,
        "other_hosts": 16
      },
      "de_reject": {
        "button": "I do not agree",
        "clicks": 1,
        "cookies": 30,
        "third_party_cookies": 4,
        "other_hosts": 33
      },
      "de_reject_missing": null,
      "de_accept": {
        "button": "I agree",
        "clicks": 1,
        "cookies": 44,
        "third_party_cookies": 15,
        "other_hosts": 48
      },
      "de_accept_missing": null
    },
    {
      "site": "apnews.com",
      "us": {
        "status": 200,
        "cmp": "OneTrust",
        "banner": true,
        "tcf": false,
        "gpp": true,
        "banner_after_ms": 9002,
        "cookies": 111,
        "third_party_cookies": 75,
        "other_hosts": 195
      },
      "de": {
        "status": 200,
        "cmp": "OneTrust",
        "banner": true,
        "tcf": true,
        "gpp": false,
        "banner_after_ms": 373,
        "cookies": 26,
        "third_party_cookies": 3,
        "other_hosts": 72
      },
      "de_reject": {
        "button": "I Reject All (except Strictly Necessary)",
        "clicks": 1,
        "cookies": 35,
        "third_party_cookies": 3,
        "other_hosts": 79
      },
      "de_reject_missing": null,
      "de_accept": {
        "button": "I Accept All",
        "clicks": 1,
        "cookies": 174,
        "third_party_cookies": 142,
        "other_hosts": 254
      },
      "de_accept_missing": null
    },
    {
      "site": "theverge.com",
      "us": {
        "status": 200,
        "cmp": "OneTrust",
        "banner": false,
        "tcf": false,
        "gpp": true,
        "banner_after_ms": null,
        "cookies": 247,
        "third_party_cookies": 223,
        "other_hosts": 219
      },
      "de": {
        "status": 200,
        "cmp": "OneTrust",
        "banner": true,
        "tcf": true,
        "gpp": false,
        "banner_after_ms": 1903,
        "cookies": 8,
        "third_party_cookies": 0,
        "other_hosts": 34
      },
      "de_reject": {
        "button": "Reject Non-Essential",
        "clicks": 1,
        "cookies": 12,
        "third_party_cookies": 0,
        "other_hosts": 40
      },
      "de_reject_missing": null,
      "de_accept": {
        "button": "Accept",
        "clicks": 1,
        "cookies": 48,
        "third_party_cookies": 14,
        "other_hosts": 87
      },
      "de_accept_missing": null
    },
    {
      "site": "huffpost.com",
      "us": {
        "status": 200,
        "cmp": null,
        "banner": false,
        "tcf": false,
        "gpp": true,
        "banner_after_ms": null,
        "cookies": 228,
        "third_party_cookies": 218,
        "other_hosts": 294
      },
      "de": {
        "status": 200,
        "cmp": "OneTrust",
        "banner": true,
        "tcf": true,
        "gpp": false,
        "banner_after_ms": 703,
        "cookies": 8,
        "third_party_cookies": 0,
        "other_hosts": 62
      },
      "de_reject": {
        "button": "Reject All",
        "clicks": 1,
        "cookies": 11,
        "third_party_cookies": 0,
        "other_hosts": 65
      },
      "de_reject_missing": null,
      "de_accept": {
        "button": "I Accept",
        "clicks": 1,
        "cookies": 186,
        "third_party_cookies": 154,
        "other_hosts": 252
      },
      "de_accept_missing": null
    }
  ]
}

Takeaways

The exit country picks the site you get. From Germany all five sites showed a consent banner and exposed IAB Europe's TCF API in every run (five runs, one per language); the banner appeared 0.1–8.5 s after domcontentloaded. From the US, only the BBC exposed TCF (it runs both APIs everywhere); every site that loaded exposed IAB Tech Lab's GPP instead. US banners were the Guardian's privacy message (every run), HuffPost's (three of five runs) and AP News's (both runs where it loaded); the BBC and The Verge showed none.
Some sites refuse some exits. AP News answered 403 to the US exit in three of five runs, even after a second try from a fresh context with a new IP; from Germany it loaded every time. Pick the exit for the site, and check the status before you parse.
A banner doesn't mean nothing loads. Before any choice, the German visitor's page had already contacted 5 other-domain hosts on the Guardian, 15–16 on the BBC, 34–51 on The Verge, 48–63 on HuffPost and 65–96 on AP News — mostly with 0–2 third-party cookies, but AP set 3–9.
"Reject" changes little; "accept" changes a lot on three of the five. After a reject, cookies rose by 3–15 and third-party cookies stayed at 0–2 on The Verge and HuffPost, 4 on the BBC and 3–13 on AP News. After "accept", the Guardian carried 60–99 third-party cookies, AP News 109–167 and HuffPost 121–157, with 128–285 other-domain hosts. On the BBC (9–15 against 4) and The Verge (9–16 against 0–2) accepting added far less.
From the US nothing is held back. With no click at all, the US visitor carried up to 223 third-party cookies on The Verge and 218 on HuffPost in some runs — more than accepting everything from Germany gave on The Verge (9–16).
Not every banner has a free "no". The Guardian's first layer had an "Accept all" but no free reject in any run — its alternative is pay-or-consent.
For scraping: pick the exit country on purpose (a country-pinned residential token or your own proxy), expect an overlay from the EU, and click the first-layer reject — #onetrust-reject-all-handler in the page, Sourcepoint's buttons inside iframe[id^="sp_message_iframe"]. Wait for the banner rather than sleeping a fixed time: it took up to 9.3 s on the Guardian from the US and 18.9 s on AP News. Check it's gone after the click and click again if not (no second click was needed in these runs, but consent iframes that re-render can swallow one).