Cases / #8 · 2026-09-30 · Medium
A bandwidth diet for residential proxies: what blocking really saves
Count every byte a news homepage pulls through your proxy, then block images, media, fonts and third parties — and see where the money actually goes.
Run on production on 2026-09-30: ✓ Node.js ✓ Python ✓ Java ✓ C# ✓ Go
The problem
Residential proxies bill per gigabyte, so a scraper's cost is mostly bytes. The usual advice is "block images". How much does that save on a real, ad-heavy page — and what else is worth blocking?
What we used, and why
| What | Why |
|---|---|
context.route('**/*') + route.abort() | Drops requests before they leave the browser, so they never cost proxy bandwidth. |
request.resourceType() | Classifies requests: image, media, font, script, xhr, … |
| A first-party list | The site's own domains and CDNs (bbc.com, bbc.co.uk, bbci.co.uk); everything else is third-party (ads, analytics, trackers). |
request.sizes() on requestfinished | Exact bytes on the wire per request: request and response headers plus the encoded body. |
| A fresh context per variant | Empty cache each time, so every variant pays full price. |
How it works
- Load the page with nothing blocked; count requests and bytes for DOM-ready plus 5 seconds.
- Again, blocking images, media and fonts.
- Again, also blocking every third-party host.
- Convert to proxy cost per 100,000 page loads at $3/GB.
The code
The same program in five languages (also on GitHub, with the raw output). Set these environment variables first:
CDPFLEET_API_KEY— your API key (dashboard → API keys)PROXY_URL— your proxy, e.g.http://user:[email protected]:8000
// npm install [email protected]
// env: CDPFLEET_API_KEY, PROXY_URL
import { chromium } from 'playwright';
const KEY = process.env.CDPFLEET_API_KEY;
const PAGE = 'https://www.bbc.com/news';
const FIRST_PARTY = ['bbc.com', 'bbc.co.uk', 'bbci.co.uk']; // the site's own domains and CDNs
const PRICE_PER_GB = 3; // a typical residential proxy price, USD
const res = await fetch('https://starter.cdpfleet.com/chromium/session', {
method: 'POST',
headers: { 'x-api-key': KEY, 'content-type': 'application/json' },
body: JSON.stringify({ proxy: process.env.PROXY_URL, headless: true }),
});
if (!res.ok) throw new Error(`launch: ${res.status} ${await res.text()}`);
const { wsUrl } = await res.json();
const browser = await chromium.connect(wsUrl, { headers: { 'x-api-key': KEY } });
const HEAVY = new Set(['image', 'media', 'font']);
// Load the page in a fresh context (empty cache) and count every byte on the wire.
async function measure(label, block) {
const context = await browser.newContext();
if (block) {
await context.route('**/*', (route) => {
const r = route.request();
if (HEAVY.has(r.resourceType()) || (block === 'first-party' && !FIRST_PARTY.some((d) => new URL(r.url()).hostname.endsWith(d)))) {
return route.abort();
}
return route.continue();
});
}
const page = await context.newPage();
let bytes = 0;
let requests = 0;
let blocked = 0;
const count = async (req) => {
const s = await req.sizes();
bytes += s.requestHeadersSize + s.requestBodySize + s.responseHeadersSize + s.responseBodySize;
requests++;
};
page.on('requestfinished', count);
page.on('requestfailed', () => { blocked++; });
const t = Date.now();
// DOM ready, then a fixed 5 s for the rest to arrive: the same window for every variant
// ('load' can wait forever on a blocked video).
await page.goto(PAGE, { waitUntil: 'domcontentloaded', timeout: 90000 });
const readyMs = Date.now() - t;
await page.waitForTimeout(5000);
const title = await page.title();
page.off('requestfinished', count); // stop counting before closing
await context.close();
return { variant: label, title, requests, blocked, kilobytes: Math.round(bytes / 1024), dom_ready_ms: readyMs };
}
try {
const rows = [
await measure('everything', null),
await measure('no images, media or fonts', 'heavy'),
await measure('…and first-party only', 'first-party'),
];
const full = rows[0].kilobytes;
for (const r of rows) {
r.saved = `${Math.round((1 - r.kilobytes / full) * 100)}%`;
r.proxy_cost_per_100k_pages = `$${((r.kilobytes * 100000) / 1024 / 1024 * PRICE_PER_GB).toFixed(0)}`;
}
console.log(JSON.stringify(rows, null, 2));
} finally {
await browser.close();
}
# pip install playwright==1.60.0 requests
# env: CDPFLEET_API_KEY, PROXY_URL
import json
import os
import time
from urllib.parse import urlparse
import requests
from playwright.sync_api import sync_playwright
KEY = os.environ["CDPFLEET_API_KEY"]
PAGE = "https://www.bbc.com/news"
FIRST_PARTY = ("bbc.com", "bbc.co.uk", "bbci.co.uk") # the site's own domains and CDNs
PRICE_PER_GB = 3 # a typical residential proxy price, USD
HEAVY = {"image", "media", "font"}
res = requests.post("https://starter.cdpfleet.com/chromium/session", headers={"x-api-key": KEY},
json={"proxy": os.environ["PROXY_URL"], "headless": True}, timeout=60)
res.raise_for_status()
def measure(browser, label, block):
"""Load the page in a fresh context (empty cache) and count every byte on the wire."""
context = browser.new_context()
if block:
def handle(route):
r = route.request
third_party = not urlparse(r.url).hostname.endswith(FIRST_PARTY)
if r.resource_type in HEAVY or (block == "first-party" and third_party):
return route.abort()
return route.continue_()
context.route("**/*", handle)
page = context.new_page()
stats = {"bytes": 0, "requests": 0, "blocked": 0}
def finished(req):
s = req.sizes()
stats["bytes"] += s["requestHeadersSize"] + s["requestBodySize"] + s["responseHeadersSize"] + s["responseBodySize"]
stats["requests"] += 1
page.on("requestfinished", finished)
page.on("requestfailed", lambda req: stats.__setitem__("blocked", stats["blocked"] + 1))
t = time.time()
# DOM ready, then a fixed 5 s for the rest to arrive: the same window for every variant
# ('load' can wait forever on a blocked video).
page.goto(PAGE, wait_until="domcontentloaded", timeout=90000)
ready_ms = round((time.time() - t) * 1000)
page.wait_for_timeout(5000)
title = page.title()
page.remove_listener("requestfinished", finished) # stop counting before closing
context.close()
return {"variant": label, "title": title, "requests": stats["requests"], "blocked": stats["blocked"],
"kilobytes": round(stats["bytes"] / 1024), "dom_ready_ms": ready_ms}
with sync_playwright() as p:
browser = p.chromium.connect(res.json()["wsUrl"], headers={"x-api-key": KEY})
try:
rows = [
measure(browser, "everything", None),
measure(browser, "no images, media or fonts", "heavy"),
measure(browser, "…and first-party only", "first-party"),
]
full = rows[0]["kilobytes"]
for r in rows:
r["saved"] = f"{round((1 - r['kilobytes'] / full) * 100)}%"
r["proxy_cost_per_100k_pages"] = f"${r['kilobytes'] * 100000 / 1024 / 1024 * PRICE_PER_GB:.0f}"
print(json.dumps(rows, indent=2, ensure_ascii=False))
finally:
browser.close()
// Maven: com.microsoft.playwright:playwright:1.60.0, com.google.code.gson:gson:2.11.0
// Run with PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1. env: CDPFLEET_API_KEY, PROXY_URL
import com.google.gson.*;
import com.microsoft.playwright.*;
import com.microsoft.playwright.options.*;
import java.net.URI;
import java.net.http.*;
import java.util.*;
import java.util.function.Consumer;
public class Main {
static final String KEY = System.getenv("CDPFLEET_API_KEY");
static final String PAGE = "https://www.bbc.com/news";
static final List<String> FIRST_PARTY = List.of("bbc.com", "bbc.co.uk", "bbci.co.uk"); // the site's own domains and CDNs
static final Set<String> HEAVY = Set.of("image", "media", "font");
static final double PRICE_PER_GB = 3; // a typical residential proxy price, USD
// Load the page in a fresh context (empty cache) and count every byte on the wire.
static JsonObject measure(Browser browser, String label, String block) {
BrowserContext context = browser.newContext();
if (block != null) {
context.route("**/*", route -> {
Request r = route.request();
String host = URI.create(r.url()).getHost();
boolean thirdParty = FIRST_PARTY.stream().noneMatch(host::endsWith);
if (HEAVY.contains(r.resourceType()) || (block.equals("first-party") && thirdParty)) route.abort();
else route.resume();
});
}
Page page = context.newPage();
long[] bytes = {0};
int[] counts = {0, 0}; // requests, blocked
Consumer<Request> count = req -> {
Sizes s = req.sizes();
bytes[0] += s.requestHeadersSize + s.requestBodySize + s.responseHeadersSize + s.responseBodySize;
counts[0]++;
};
page.onRequestFinished(count);
page.onRequestFailed(req -> counts[1]++);
long t = System.currentTimeMillis();
// DOM ready, then a fixed 5 s for the rest to arrive: the same window for every variant
// ('load' can wait forever on a blocked video).
page.navigate(PAGE, new Page.NavigateOptions().setWaitUntil(WaitUntilState.DOMCONTENTLOADED).setTimeout(90000));
long readyMs = System.currentTimeMillis() - t;
page.waitForTimeout(5000);
String title = page.title();
page.offRequestFinished(count); // stop counting before closing
context.close();
JsonObject row = new JsonObject();
row.addProperty("variant", label);
row.addProperty("title", title);
row.addProperty("requests", counts[0]);
row.addProperty("blocked", counts[1]);
row.addProperty("kilobytes", Math.round(bytes[0] / 1024.0));
row.addProperty("dom_ready_ms", readyMs);
return row;
}
public static void main(String[] args) throws Exception {
String body = "{\"proxy\": " + new Gson().toJson(System.getenv("PROXY_URL")) + ", \"headless\": true}";
HttpResponse<String> res = HttpClient.newHttpClient().send(HttpRequest.newBuilder(URI.create("https://starter.cdpfleet.com/chromium/session"))
.header("x-api-key", KEY).header("content-type", "application/json")
.POST(HttpRequest.BodyPublishers.ofString(body)).build(), HttpResponse.BodyHandlers.ofString());
if (res.statusCode() != 200) throw new RuntimeException("launch: " + res.statusCode() + " " + res.body());
String wsUrl = JsonParser.parseString(res.body()).getAsJsonObject().get("wsUrl").getAsString();
try (Playwright playwright = Playwright.create()) {
Browser browser = playwright.chromium().connect(wsUrl, new BrowserType.ConnectOptions().setHeaders(Map.of("x-api-key", KEY)));
try {
List<JsonObject> rows = List.of(
measure(browser, "everything", null),
measure(browser, "no images, media or fonts", "heavy"),
measure(browser, "…and first-party only", "first-party"));
double full = rows.get(0).get("kilobytes").getAsDouble();
JsonArray out = new JsonArray();
for (JsonObject r : rows) {
double kb = r.get("kilobytes").getAsDouble();
r.addProperty("saved", Math.round((1 - kb / full) * 100) + "%");
r.addProperty("proxy_cost_per_100k_pages", "$" + Math.round(kb * 100000 / 1024 / 1024 * PRICE_PER_GB));
out.add(r);
}
System.out.println(new GsonBuilder().setPrettyPrinting().disableHtmlEscaping().create().toJson(out));
} finally {
browser.close();
}
}
}
}
// dotnet add package Microsoft.Playwright --version 1.60.0
// env: CDPFLEET_API_KEY, PROXY_URL
using System.Diagnostics;
using System.Net.Http.Json;
using System.Text.Encodings.Web;
using System.Text.Json;
using System.Text.Json.Nodes;
using Microsoft.Playwright;
const string PageUrl = "https://www.bbc.com/news";
string[] firstParty = { "bbc.com", "bbc.co.uk", "bbci.co.uk" }; // the site's own domains and CDNs
var heavy = new HashSet<string> { "image", "media", "font" };
const double PricePerGb = 3; // a typical residential proxy price, USD
var key = Environment.GetEnvironmentVariable("CDPFLEET_API_KEY")!;
using var http = new HttpClient();
http.DefaultRequestHeaders.Add("x-api-key", key);
var res = await http.PostAsJsonAsync("https://starter.cdpfleet.com/chromium/session",
new { proxy = Environment.GetEnvironmentVariable("PROXY_URL"), headless = true });
if (!res.IsSuccessStatusCode) throw new Exception($"launch: {(int)res.StatusCode} {await res.Content.ReadAsStringAsync()}");
var wsUrl = (await res.Content.ReadFromJsonAsync<JsonElement>()).GetProperty("wsUrl").GetString()!;
using var playwright = await Playwright.CreateAsync();
var browser = await playwright.Chromium.ConnectAsync(wsUrl, new() { Headers = new Dictionary<string, string> { ["x-api-key"] = key } });
// Load the page in a fresh context (empty cache) and count every byte on the wire.
async Task<JsonObject> Measure(string label, string? block)
{
var context = await browser.NewContextAsync();
if (block != null)
{
await context.RouteAsync("**/*", route =>
{
var r = route.Request;
var thirdParty = !firstParty.Any(d => new Uri(r.Url).Host.EndsWith(d));
return heavy.Contains(r.ResourceType) || (block == "first-party" && thirdParty) ? route.AbortAsync() : route.ContinueAsync();
});
}
var page = await context.NewPageAsync();
long bytes = 0;
int requests = 0, blocked = 0;
var pending = new List<Task>();
async Task Count(IRequest req)
{
var s = await req.SizesAsync();
Interlocked.Add(ref bytes, s.RequestHeadersSize + s.RequestBodySize + s.ResponseHeadersSize + s.ResponseBodySize);
Interlocked.Increment(ref requests);
}
EventHandler<IRequest> onFinished = (_, req) => { lock (pending) pending.Add(Count(req)); };
page.RequestFinished += onFinished;
page.RequestFailed += (_, _) => Interlocked.Increment(ref blocked);
var sw = Stopwatch.StartNew();
// DOM ready, then a fixed 5 s for the rest to arrive: the same window for every variant
// ('load' can wait forever on a blocked video).
await page.GotoAsync(PageUrl, new() { WaitUntil = WaitUntilState.DOMContentLoaded, Timeout = 90000 });
var readyMs = sw.ElapsedMilliseconds;
await page.WaitForTimeoutAsync(5000);
var title = await page.TitleAsync();
page.RequestFinished -= onFinished; // stop counting before closing
Task[] counting;
lock (pending) counting = pending.ToArray();
await Task.WhenAll(counting);
await context.CloseAsync();
return new JsonObject
{
["variant"] = label, ["title"] = title, ["requests"] = requests, ["blocked"] = blocked,
["kilobytes"] = (long)Math.Round(bytes / 1024.0), ["dom_ready_ms"] = readyMs,
};
}
try
{
var rows = new[]
{
await Measure("everything", null),
await Measure("no images, media or fonts", "heavy"),
await Measure("…and first-party only", "first-party"),
};
var full = rows[0]["kilobytes"]!.GetValue<long>();
foreach (var r in rows)
{
double kb = r["kilobytes"]!.GetValue<long>();
r["saved"] = $"{Math.Round((1 - kb / full) * 100)}%";
r["proxy_cost_per_100k_pages"] = $"${Math.Round(kb * 100000 / 1024 / 1024 * PricePerGb)}";
}
Console.WriteLine(new JsonArray(rows.ToArray<JsonNode?>()).ToJsonString(new JsonSerializerOptions { WriteIndented = true, Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping }));
}
finally
{
await browser.CloseAsync();
}
// go get github.com/playwright-community/[email protected]
// Driver: build playwright-core 1.60.0 from npm and set PLAYWRIGHT_DRIVER_PATH (see /docs/quickstart).
// env: CDPFLEET_API_KEY, PROXY_URL
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"log"
"math"
"net/http"
"net/url"
"os"
"strings"
"sync"
"time"
"github.com/playwright-community/playwright-go"
)
var key = os.Getenv("CDPFLEET_API_KEY")
const pageURL = "https://www.bbc.com/news"
const pricePerGB = 3.0 // a typical residential proxy price, USD
var firstParty = []string{"bbc.com", "bbc.co.uk", "bbci.co.uk"} // the site's own domains and CDNs
var heavy = map[string]bool{"image": true, "media": true, "font": true}
func launch(name string, options map[string]any) (map[string]any, error) {
body, _ := json.Marshal(options)
req, _ := http.NewRequest("POST", "https://starter.cdpfleet.com/"+name+"/session", bytes.NewReader(body))
req.Header.Set("x-api-key", key)
req.Header.Set("content-type", "application/json")
res, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
defer res.Body.Close()
if res.StatusCode != http.StatusOK {
msg, _ := io.ReadAll(res.Body)
return nil, fmt.Errorf("launch %s: %s %s", name, res.Status, msg)
}
var session map[string]any
return session, json.NewDecoder(res.Body).Decode(&session)
}
type row struct {
Variant string `json:"variant"`
Title string `json:"title"`
Requests int `json:"requests"`
Blocked int `json:"blocked"`
Kilobytes int64 `json:"kilobytes"`
DomReadyMs int64 `json:"dom_ready_ms"`
Saved string `json:"saved"`
ProxyCost string `json:"proxy_cost_per_100k_pages"`
}
// Load the page in a fresh context (empty cache) and count every byte on the wire.
func measure(browser playwright.Browser, label, block string) row {
context, err := browser.NewContext()
if err != nil {
log.Fatal(err)
}
defer context.Close()
if block != "" {
context.Route("**/*", func(route playwright.Route) {
r := route.Request()
u, _ := url.Parse(r.URL())
thirdParty := true
for _, d := range firstParty {
if strings.HasSuffix(u.Hostname(), d) {
thirdParty = false
}
}
if heavy[r.ResourceType()] || (block == "first-party" && thirdParty) {
route.Abort()
} else {
route.Continue()
}
})
}
page, _ := context.NewPage()
var mu sync.Mutex
var bytesSeen int64
requests, blocked := 0, 0
count := func(req playwright.Request) {
s, err := req.Sizes()
if err != nil {
return
}
mu.Lock()
bytesSeen += int64(s.RequestHeadersSize + s.RequestBodySize + s.ResponseHeadersSize + s.ResponseBodySize)
requests++
mu.Unlock()
}
page.OnRequestFinished(count)
page.OnRequestFailed(func(playwright.Request) { mu.Lock(); blocked++; mu.Unlock() })
t := time.Now()
// DOM ready, then a fixed 5 s for the rest to arrive: the same window for every variant
// ('load' can wait forever on a blocked video).
if _, err := page.Goto(pageURL, playwright.PageGotoOptions{WaitUntil: playwright.WaitUntilStateDomcontentloaded, Timeout: playwright.Float(90000)}); err != nil {
log.Fatal(err)
}
readyMs := time.Since(t).Milliseconds()
page.WaitForTimeout(5000)
title, _ := page.Title()
page.RemoveListener("requestfinished", count) // stop counting before closing
mu.Lock()
defer mu.Unlock()
return row{Variant: label, Title: title, Requests: requests, Blocked: blocked,
Kilobytes: int64(math.Round(float64(bytesSeen) / 1024)), DomReadyMs: readyMs}
}
func main() {
session, err := launch("chromium", map[string]any{"proxy": os.Getenv("PROXY_URL"), "headless": true})
if err != nil {
log.Fatal(err)
}
pw, err := playwright.Run(&playwright.RunOptions{SkipInstallBrowsers: true})
if err != nil {
log.Fatal(err)
}
defer pw.Stop()
browser, err := pw.Chromium.Connect(session["wsUrl"].(string), playwright.BrowserTypeConnectOptions{Headers: map[string]string{"x-api-key": key}})
if err != nil {
log.Fatal(err)
}
defer browser.Close()
rows := []row{
measure(browser, "everything", ""),
measure(browser, "no images, media or fonts", "heavy"),
measure(browser, "…and first-party only", "first-party"),
}
full := float64(rows[0].Kilobytes)
for i := range rows {
kb := float64(rows[i].Kilobytes)
rows[i].Saved = fmt.Sprintf("%.0f%%", (1-kb/full)*100)
rows[i].ProxyCost = fmt.Sprintf("$%.0f", kb*100000/1024/1024*pricePerGB)
}
var buf bytes.Buffer
enc := json.NewEncoder(&buf)
enc.SetEscapeHTML(false)
enc.SetIndent("", " ")
enc.Encode(rows)
fmt.Print(buf.String())
}
What we got
| Variant | Requests | Blocked | KB | Saved | Proxy cost / 100k pages | DOM ready (ms) |
|---|---|---|---|---|---|---|
| everything | 80 | 0 | 1662 | 0% | $476 | 4314 |
| no images, media or fonts | 84 | 28 | 1533 | 8% | $439 | 5030 |
| …and first-party only | 53 | 31 | 791 | 52% | $226 | 2688 |
From the Node.js run on 2026-09-30. IP addresses are replaced with placeholders (203.0.113.x); equal addresses stay equal. The other languages produced the same findings.
Raw output (Node.js)
[
{
"variant": "everything",
"title": "BBC News - Breaking news, video and the latest top stories from the U.S. and around the world",
"requests": 80,
"blocked": 0,
"kilobytes": 1662,
"dom_ready_ms": 4314,
"saved": "0%",
"proxy_cost_per_100k_pages": "$476"
},
{
"variant": "no images, media or fonts",
"title": "BBC News - Breaking news, video and the latest top stories from the U.S. and around the world",
"requests": 84,
"blocked": 28,
"kilobytes": 1533,
"dom_ready_ms": 5030,
"saved": "8%",
"proxy_cost_per_100k_pages": "$439"
},
{
"variant": "…and first-party only",
"title": "BBC News - Breaking news, video and the latest top stories from the U.S. and around the world",
"requests": 53,
"blocked": 31,
"kilobytes": 791,
"dom_ready_ms": 2688,
"saved": "52%",
"proxy_cost_per_100k_pages": "$226"
}
]Takeaways
- Images, media and fonts are a smaller share than you'd think on a script-heavy news page — at most about a fifth, and on a single load it can even look worse: the ad slots fill differently every time.
- Third parties are the bigger lever: ads, analytics and consent tooling pull scripts, pixels and beacons from dozens of hosts. Keeping only first-party requests cut the bytes much further.
- Blocking can also make pages faster — fewer requests competing for the proxy. But test it: some sites need a third-party script (consent, anti-bot) to render at all.
loadmay never fire when a video is blocked; measure withdomcontentloadedplus a fixed window, as here.- Numbers move between runs (ad auctions differ per load); the proportions hold.