cdpfleet Docs GitHub Dashboard

Cases / #31 · 2026-10-10 · Medium

Content you can't querySelector: iframes and shadow DOM

Same-origin and cross-origin iframes, open and nested shadow roots, and a closed one: what document.querySelector finds (nothing), what Playwright locators find (all but the closed root), and how.

Chromium

Run on production on 2026-10-10: ✓ Node.js ✓ Python ✓ Java ✓ C# ✓ Go

The problem

Modern pages hide content where a plain document.querySelector can't reach: widgets in iframes (often from another origin — payments, maps, chat), and design-system components that render inside shadow roots. A scraper that evaluates querySelector in the page comes back empty and looks broken. Which of these can you reach from Playwright, and with what?

What we used, and why

WhatWhy
page.setContent() with a test pageOne page with every case — a srcdoc iframe, a cross-origin iframe (httpbin.org), an open shadow root with a nested one, and a closed shadow root — so the result doesn't depend on a third-party layout.
document.querySelector in evaluateThe baseline: what page JavaScript sees from the top document.
page.frameLocator(sel).locator(…)Reaches into an iframe, same-origin or not — the remote browser runs out-of-process frames for you.
page.locator(css)Playwright's CSS engine pierces open shadow roots at any depth.
page.frames()Every frame as an object with its own URL and title.

How it works

  1. Launch Chromium and load the test page; wait until the cross-origin frame has rendered its heading.
  2. Look for each target with document.querySelector from the top document.
  3. Look for the same targets with Playwright locators and frame locators.
  4. Print what each approach found, and the frames the page has.

The code

The same program in five languages (also on GitHub, with the raw output). Set these environment variables first:

// npm install [email protected]
// env: CDPFLEET_API_KEY, PROXY_URL
import { chromium } from 'playwright';

const KEY = process.env.CDPFLEET_API_KEY;

// Content a plain document.querySelector can't see: a same-origin iframe, a cross-origin
// iframe, an open shadow root (with another one nested inside) and a closed shadow root.
const HTML = `<!doctype html><title>Hidden content</title>
<h1>Main document</h1>
<iframe id="same" srcdoc="<p id='inner'>same-origin iframe text</p>"></iframe>
<iframe id="cross" src="https://httpbin.org/html"></iframe>
<open-card></open-card>
<closed-card></closed-card>
<script>
customElements.define('open-card', class extends HTMLElement {
  connectedCallback() {
    const root = this.attachShadow({ mode: 'open' });
    root.innerHTML = '<p class="msg">open shadow text</p><nested-badge></nested-badge>';
  }
});
customElements.define('nested-badge', class extends HTMLElement {
  connectedCallback() { this.attachShadow({ mode: 'open' }).innerHTML = '<span class="badge">nested shadow text</span>'; }
});
customElements.define('closed-card', class extends HTMLElement {
  connectedCallback() { this.attachShadow({ mode: 'closed' }).innerHTML = '<p class="secret">closed shadow text</p>'; }
});
</script>`;

const res = await fetch('https://starter.cdpfleet.com/chromium/session', {
  method: 'POST',
  headers: { 'x-api-key': KEY, 'content-type': 'application/json' },
  body: JSON.stringify({ proxy: process.env.PROXY_URL, headless: 'new' }),
});
if (!res.ok) throw new Error(`launch ${res.status} ${await res.text()}`);
const { wsUrl } = await res.json();
const browser = await chromium.connect(wsUrl, { headers: { 'x-api-key': KEY } });

try {
  const page = await browser.newPage();
  await page.setContent(HTML);
  await page.frameLocator('#cross').locator('h1').waitFor({ timeout: 60000 });

  const qs = (sel) => page.evaluate((s) => document.querySelector(s)?.textContent ?? null, sel);
  const pw = async (loc) => (await loc.count()) ? (await loc.first().textContent()) : null;

  const rows = [
    { target: 'same-origin iframe', querySelector: await qs('#inner'), playwright: await pw(page.frameLocator('#same').locator('#inner')), how: "page.frameLocator('#same').locator('#inner')" },
    { target: 'cross-origin iframe', querySelector: await qs('h1 + div p'), playwright: await pw(page.frameLocator('#cross').locator('h1')), how: "page.frameLocator('#cross').locator('h1')" },
    { target: 'open shadow root', querySelector: await qs('.msg'), playwright: await pw(page.locator('.msg')), how: "page.locator('.msg') — CSS pierces open shadow roots" },
    { target: 'nested open shadow root', querySelector: await qs('.badge'), playwright: await pw(page.locator('.badge')), how: "page.locator('.badge') — any depth" },
    { target: 'closed shadow root', querySelector: await qs('.secret'), playwright: await pw(page.locator('.secret')), how: 'not reachable from page scripts or locators' },
  ];
  // The cross-origin frame is a separate document: its URL and title come from the frame object.
  const cross = page.frames().find((f) => f.url().startsWith('https://httpbin.org'));
  console.log(JSON.stringify({ frames: page.frames().length, cross_origin_frame: { url: cross?.url() ?? null, title: cross ? await cross.title() : null }, rows }, null, 2));
} finally {
  await browser.close();
}

What we got

Targetdocument.querySelectorPlaywrightHow
same-origin iframe— (not found)same-origin iframe textpage.frameLocator('#same').locator('#inner')
cross-origin iframe— (not found)Herman Melville - Moby-Dickpage.frameLocator('#cross').locator('h1')
open shadow root— (not found)open shadow textpage.locator('.msg') — CSS pierces open shadow roots
nested open shadow root— (not found)nested shadow textpage.locator('.badge') — any depth
closed shadow root— (not found)— (not found)not reachable from page scripts or locators

From the Node.js run on 2026-10-10. IP addresses are replaced with placeholders (203.0.113.x); equal addresses stay equal. The other languages produced the same findings.

Raw output (Node.js)
{
  "frames": 3,
  "cross_origin_frame": {
    "url": "https://httpbin.org/html",
    "title": ""
  },
  "rows": [
    {
      "target": "same-origin iframe",
      "querySelector": null,
      "playwright": "same-origin iframe text",
      "how": "page.frameLocator('#same').locator('#inner')"
    },
    {
      "target": "cross-origin iframe",
      "querySelector": null,
      "playwright": "Herman Melville - Moby-Dick",
      "how": "page.frameLocator('#cross').locator('h1')"
    },
    {
      "target": "open shadow root",
      "querySelector": null,
      "playwright": "open shadow text",
      "how": "page.locator('.msg') — CSS pierces open shadow roots"
    },
    {
      "target": "nested open shadow root",
      "querySelector": null,
      "playwright": "nested shadow text",
      "how": "page.locator('.badge') — any depth"
    },
    {
      "target": "closed shadow root",
      "querySelector": null,
      "playwright": null,
      "how": "not reachable from page scripts or locators"
    }
  ]
}

Takeaways