Function jobs

For work longer than one call: batch scrapes a list of URLs, crawl follows a site's links, map lists a whole site's URLs, watch reports live changes on a page, warmup keeps a browser browsing like a person. A job starts with one request, runs on your threads in the background for as long as it needs, and you follow it, read its results as they arrive — as JSON lines or one zip archive — stop it and resume it. Jobs come with every paid plan.

Start a job

POST https://cdpfleet.com/v1/functions/batch, /crawl, /watch, /warmup, or /map with "job": true, with your API key, returns 202 {"job_id", "status_url", "concurrency"} at once. Jobs have no size or time limit of their own: a run that reaches your plan's session limit (24 hours) continues in a new run by itself, so a job goes on until it's done or you stop it — it uses your threads and minutes all along.

curl -s https://cdpfleet.com/v1/functions/crawl -H "x-api-key: $CDPFLEET_API_KEY" -H "content-type: application/json" \
  -d '{"url":"https://books.toscrape.com/","max_depth":2,"max_pages":200,"formats":["markdown"],"concurrency":4}'
# {"job_id":"7c1e…","status_url":"https://cdpfleet.com/v1/jobs/7c1e…","concurrency":4}

batch

Field	Type	Default	What it does
`urls`	string[]	this or `urls_url`	Public `http`/`https` URLs, scraped in any order — as many as fit in the request (1 MB, about 10,000).
`urls_url`	string	this or `urls`	A public link to a text file with one URL per line, any length.

A batch from start to finish — start it, poll its status until it ends, then download every result as JSON lines:

# curl: the job id and status are read with grep
H="x-api-key: $CDPFLEET_API_KEY"
JOB=$(curl -s https://cdpfleet.com/v1/functions/batch -H "$H" -H "content-type: application/json" \
  -d '{"urls":["https://books.toscrape.com/catalogue/page-1.html","https://books.toscrape.com/catalogue/page-2.html"],"formats":["markdown"],"concurrency":2}' | grep -o '"job_id":"[^"]*"' | cut -d'"' -f4)
echo "job $JOB"
while sleep 5; do
  STATUS=$(curl -s https://cdpfleet.com/v1/jobs/$JOB -H "$H" | grep -o '"status":"[a-z]*"' | head -1 | cut -d'"' -f4)
  echo "$STATUS"
  case $STATUS in done|failed|stopped|interrupted) break ;; esac
done
curl -s https://cdpfleet.com/v1/jobs/$JOB/results.jsonl -H "$H" -o results.jsonl
wc -l results.jsonl

// Node.js 18+: save as batch.mjs, run node batch.mjs
import { writeFile } from 'node:fs/promises';

async function api(url, body) {
  const res = await fetch(url, {
    method: body ? 'POST' : 'GET',
    headers: { 'x-api-key': process.env.CDPFLEET_API_KEY, ...(body && { 'content-type': 'application/json' }) },
    body: body && JSON.stringify(body),
    signal: AbortSignal.timeout(30000),
  });
  if (!res.ok) throw new Error(`${res.status} ${await res.text()}`); // 429: wait Retry-After seconds
  return res;
}

// 1. Start the job: 202 {job_id, status_url, concurrency}
const { job_id } = await (await api('https://cdpfleet.com/v1/functions/batch', {
  urls: [
    'https://books.toscrape.com/catalogue/page-1.html',
    'https://books.toscrape.com/catalogue/page-2.html',
  ],
  formats: ['markdown'],
  concurrency: 2,
})).json();
console.log('job', job_id);

// 2. Poll its status until it ends
let job;
do {
  await new Promise((r) => setTimeout(r, 5000));
  job = await (await api(`https://cdpfleet.com/v1/jobs/${job_id}`)).json();
  console.log(job.status, JSON.stringify(job.progress));
} while (!['done', 'failed', 'stopped', 'interrupted'].includes(job.status));

// 3. Every result as JSON lines, one page per line
const jsonl = await (await api(`https://cdpfleet.com/v1/jobs/${job_id}/results.jsonl`)).text();
await writeFile('results.jsonl', jsonl);
console.log(`results.jsonl: ${jsonl.split('\n').filter(Boolean).length} pages`);

# Python 3 with requests (pip install requests): save as batch.py, run python batch.py
import os
import time
import requests

def api(method, url, **kw):
    r = requests.request(method, url, headers={"x-api-key": os.environ["CDPFLEET_API_KEY"]}, timeout=30, **kw)
    if not r.ok:
        raise SystemExit(f"{r.status_code} {r.text}")  # 429: wait r.headers["Retry-After"] seconds
    return r

# 1. Start the job: 202 {job_id, status_url, concurrency}
job_id = api("POST", "https://cdpfleet.com/v1/functions/batch", json={
    "urls": [
        "https://books.toscrape.com/catalogue/page-1.html",
        "https://books.toscrape.com/catalogue/page-2.html",
    ],
    "formats": ["markdown"],
    "concurrency": 2,
}).json()["job_id"]
print("job", job_id)

# 2. Poll its status until it ends
while True:
    time.sleep(5)
    job = api("GET", f"https://cdpfleet.com/v1/jobs/{job_id}").json()
    print(job["status"], job["progress"])
    if job["status"] in ("done", "failed", "stopped", "interrupted"):
        break

# 3. Every result as JSON lines, one page per line
jsonl = api("GET", f"https://cdpfleet.com/v1/jobs/{job_id}/results.jsonl").content
open("results.jsonl", "wb").write(jsonl)
print(f"results.jsonl: {len(jsonl.splitlines())} pages")

// Java 17+ and Gson (com.google.code.gson:gson): save as Main.java, run java -cp gson-2.11.0.jar Main.java
import com.google.gson.*;
import java.net.URI;
import java.net.http.*;
import java.nio.charset.StandardCharsets;
import java.nio.file.*;
import java.time.Duration;
import java.util.List;

public class Main {
  static final HttpClient HTTP = HttpClient.newHttpClient();

  static byte[] api(String url, String body) throws Exception {
    HttpRequest.Builder req = HttpRequest.newBuilder(URI.create(url))
        .header("x-api-key", System.getenv("CDPFLEET_API_KEY"))
        .timeout(Duration.ofSeconds(30));
    if (body != null) req.header("content-type", "application/json").POST(HttpRequest.BodyPublishers.ofString(body));
    HttpResponse<byte[]> res = HTTP.send(req.build(), HttpResponse.BodyHandlers.ofByteArray());
    if (res.statusCode() >= 300) { // 429: wait Retry-After seconds
      throw new RuntimeException(res.statusCode() + " " + new String(res.body(), StandardCharsets.UTF_8));
    }
    return res.body();
  }

  static JsonObject json(byte[] bytes) {
    return JsonParser.parseString(new String(bytes, StandardCharsets.UTF_8)).getAsJsonObject();
  }

  public static void main(String[] args) throws Exception {
    // 1. Start the job: 202 {job_id, status_url, concurrency}
    String body = """
        {
          "urls": [
            "https://books.toscrape.com/catalogue/page-1.html",
            "https://books.toscrape.com/catalogue/page-2.html"
          ],
          "formats": ["markdown"],
          "concurrency": 2
        }
        """;
    String jobId = json(api("https://cdpfleet.com/v1/functions/batch", body)).get("job_id").getAsString();
    System.out.println("job " + jobId);

    // 2. Poll its status until it ends
    String status;
    do {
      Thread.sleep(5000);
      JsonObject job = json(api("https://cdpfleet.com/v1/jobs/" + jobId, null));
      status = job.get("status").getAsString();
      System.out.println(status + " " + job.get("progress"));
    } while (!List.of("done", "failed", "stopped", "interrupted").contains(status));

    // 3. Every result as JSON lines, one page per line
    byte[] jsonl = api("https://cdpfleet.com/v1/jobs/" + jobId + "/results.jsonl", null);
    Files.write(Path.of("results.jsonl"), jsonl);
    System.out.println("results.jsonl: " + new String(jsonl, StandardCharsets.UTF_8).lines().count() + " pages");
  }
}

// .NET 8+: dotnet new console, replace Program.cs with this, run dotnet run
using System.Text;
using System.Text.Json;

using var http = new HttpClient { Timeout = TimeSpan.FromSeconds(30) };
http.DefaultRequestHeaders.Add("x-api-key", Environment.GetEnvironmentVariable("CDPFLEET_API_KEY"));

async Task<HttpContent> Api(string url, string? body = null)
{
    var res = body == null ? await http.GetAsync(url)
        : await http.PostAsync(url, new StringContent(body, Encoding.UTF8, "application/json"));
    if (!res.IsSuccessStatusCode) // 429: wait Retry-After seconds
        throw new Exception($"{(int)res.StatusCode} {await res.Content.ReadAsStringAsync()}");
    return res.Content;
}

// 1. Start the job: 202 {job_id, status_url, concurrency}
var body = """
    {
      "urls": [
        "https://books.toscrape.com/catalogue/page-1.html",
        "https://books.toscrape.com/catalogue/page-2.html"
      ],
      "formats": ["markdown"],
      "concurrency": 2
    }
    """;
var start = JsonDocument.Parse(await (await Api("https://cdpfleet.com/v1/functions/batch", body)).ReadAsStringAsync()).RootElement;
var jobId = start.GetProperty("job_id").GetString();
Console.WriteLine($"job {jobId}");

// 2. Poll its status until it ends
string? status;
do
{
    await Task.Delay(5000);
    var job = JsonDocument.Parse(await (await Api($"https://cdpfleet.com/v1/jobs/{jobId}")).ReadAsStringAsync()).RootElement;
    status = job.GetProperty("status").GetString();
    Console.WriteLine($"{status} {job.GetProperty("progress")}");
} while (status is not ("done" or "failed" or "stopped" or "interrupted"));

// 3. Every result as JSON lines, one page per line
var jsonl = await (await Api($"https://cdpfleet.com/v1/jobs/{jobId}/results.jsonl")).ReadAsByteArrayAsync();
await File.WriteAllBytesAsync("results.jsonl", jsonl);
Console.WriteLine($"results.jsonl: {Encoding.UTF8.GetString(jsonl).Split('\n', StringSplitOptions.RemoveEmptyEntries).Length} pages");

// Go 1.21+: save as main.go, run go run main.go
package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io"
	"log"
	"net/http"
	"os"
	"slices"
	"strings"
	"time"
)

func api(method, url, body string) []byte {
	req, _ := http.NewRequest(method, url, strings.NewReader(body))
	req.Header.Set("x-api-key", os.Getenv("CDPFLEET_API_KEY"))
	if body != "" {
		req.Header.Set("content-type", "application/json")
	}
	res, err := (&http.Client{Timeout: 30 * time.Second}).Do(req)
	if err != nil {
		log.Fatal(err)
	}
	defer res.Body.Close()
	data, err := io.ReadAll(res.Body)
	if err != nil {
		log.Fatal(err)
	}
	if res.StatusCode >= 300 { // 429: wait Retry-After seconds
		log.Fatalf("%s %s", res.Status, data)
	}
	return data
}

func main() {
	// 1. Start the job: 202 {job_id, status_url, concurrency}
	body := `{
	  "urls": [
	    "https://books.toscrape.com/catalogue/page-1.html",
	    "https://books.toscrape.com/catalogue/page-2.html"
	  ],
	  "formats": ["markdown"],
	  "concurrency": 2
	}`
	var start struct {
		JobID string `json:"job_id"`
	}
	if err := json.Unmarshal(api("POST", "https://cdpfleet.com/v1/functions/batch", body), &start); err != nil {
		log.Fatal(err)
	}
	fmt.Println("job", start.JobID)

	// 2. Poll its status until it ends
	for {
		time.Sleep(5 * time.Second)
		var job struct {
			Status   string         `json:"status"`
			Progress map[string]any `json:"progress"`
		}
		if err := json.Unmarshal(api("GET", "https://cdpfleet.com/v1/jobs/"+start.JobID, ""), &job); err != nil {
			log.Fatal(err)
		}
		fmt.Println(job.Status, job.Progress)
		if slices.Contains([]string{"done", "failed", "stopped", "interrupted"}, job.Status) {
			break
		}
	}

	// 3. Every result as JSON lines, one page per line
	jsonl := api("GET", "https://cdpfleet.com/v1/jobs/"+start.JobID+"/results.jsonl", "")
	if err := os.WriteFile("results.jsonl", jsonl, 0o644); err != nil {
		log.Fatal(err)
	}
	fmt.Printf("results.jsonl: %d pages\n", bytes.Count(jsonl, []byte("\n")))
}

crawl

Field	Type	Default	What it does
`url`	string	required	Where the crawl starts.
`max_depth`	number	`1`	How many links deep to follow from the start page; no upper limit.
`max_pages`	number	`100`	Stop after this many pages; no upper limit.
`include / exclude`	string[]	all / none	URL patterns as in map: `` within a path segment, across segments, a leading `/` matches the path, e.g. `["/blog/`*`"]`.
`same_site`	boolean	`true`	Only follow links on the start URL's site.
`delay_ms`	number	`0`	Pause between pages, per thread.
`robots`	boolean	`true`	Follow the site's robots.txt.

map

Field	Type	Default	What it does
`url`	string	required	The site to map.
`job`	boolean	required: `true`	Without it, map answers at once, up to 5,000 URLs.
`max_urls`	number	no limit	Stop after this many URLs.
`max_depth`	number	`1`	1 = the sitemaps and the start page's links; more also opens the pages found, level by level, for their links (no content is kept).
`include / exclude, same_site`	—	as in crawl	Which URLs to keep and follow.

Each result is {"url", "source": "sitemap" | "links", "lastmod", "depth"}.

Both

Field	Type	Default	What it does
`formats, wait_for, actions, extract, max_wait`	—	as in scrape	Applied to every page — see scrape.
`country, proxy`	—	random, the Community proxy	As for any function.
`concurrency`	number	`2`	Pages at once: the threads the job holds while it runs, up to your plan's threads (asking for more uses all of them; the 202 says how many).
`deliver`	object	none	`{"webhook": {"url": "https://…", "events": ["page", "done"]}, "email": true, "format": "archive"}` — `events` takes `page`, `done` (default), `event` (watch) and `archive`; see webhooks; `"format": "archive"` also builds the zip archive at the end. Results are always stored.

Follow it

GET /v1/jobs/{id} — status (running, stopping, done, failed, stopped, interrupted), progress {done, failed, queued}, runs, billed_seconds, results {parts, bytes} and expires_at.
GET /v1/jobs/{id}/events — a live stream (server-sent events): progress, part when new results are stored, and done at the end.
GET /v1/jobs — your jobs, newest first (?status=, ?type=).

curl -N https://cdpfleet.com/v1/jobs/$JOB/events -H "x-api-key: $CDPFLEET_API_KEY"
# event: progress
# data: {"status":"running","progress":{"done":40,"failed":1,"queued":112}}

Results

One JSON object per page — url, status, depth (crawl), partial, error, and the formats you asked for — stored in parts of up to 500 pages as the job runs.

GET /v1/jobs/{id}/results?cursor=0 — one part: {"results": [...], "next_cursor"}. Keep passing next_cursor; while the job runs it can return the same cursor with no results yet, and after the end it is null.
GET /v1/jobs/{id}/results.jsonl — everything as one JSON-lines download.
GET /v1/jobs/{id}/results/files — every stored part as a direct download link (valid an hour), for fetching a big job in parallel.
Kept 14 days after the job ends; then 410 results_expired.

// Node.js 18+: every result of a finished job
let cursor = 0;
while (cursor !== null) {
  const r = await fetch(`https://cdpfleet.com/v1/jobs/${job}/results?cursor=${cursor}`, { headers: { 'x-api-key': process.env.CDPFLEET_API_KEY } });
  const { results, next_cursor } = await r.json();
  for (const page of results) console.log(page.url, page.status);
  cursor = next_cursor;
}

Archive (zip)

Instead of JSON lines, get everything as one zip: POST /v1/jobs/{id}/archive once the job has ended (or start the job with "deliver": {"format": "archive"}), then GET /v1/jobs/{id}/archive → {"status": "building" | "ready" | "failed", "bytes", "url"}. The url downloads straight from storage for an hour (ask again for a fresh one), and a job.archive webhook says when it's ready. Archives of any size work (zip64); they're kept as long as the job's results.

index.jsonl                          one line per page: url, status, depth, error, and the files below
pages/<host>/<path>/index.html       html
pages/<host>/<path>/index.md         markdown
pages/<host>/<path>/text.txt         text
pages/<host>/<path>/screenshot.png   screenshot, as an image
pages/<host>/<path>/links.json · requests.json · cookies.json · console.json · extract.json
errors.jsonl                         pages that failed
job.json                             the job, as GET /v1/jobs/{id} shows it

Paths keep the URL's segments with unsafe characters replaced; a query string or a very long path adds a short hash, so two URLs never share a folder. A map job's archive has urls.txt (one per line) and index.jsonl.

Watch: live changes on a page

POST https://cdpfleet.com/v1/functions/watch opens the page in a browser of yours and stays on it, reporting what changes while it's open: the XHR and fetch calls it makes (with their responses), WebSocket messages, and changes to a part of the page. Use it for prices, scores, order books, chat — anything that updates without a reload.

Field	Type	Default	What it does
`url`	string	required	The page to watch.
`duration`	number	until stopped	Seconds to watch.
`filter`	object	everything	`{"url": ["api"], "selector": "#prices", "types": ["xhr", "fetch", "ws", "dom"]}` — URL patterns for network events, the element whose changes count (debounced 0.5 s).
`reload_every`	number	never	Reload the page every so many seconds.
`browser, launch, device`	—	Chrome	As in your own browser; `"launch": {"live_view": true}` lets you watch the tab (`GET /v1/jobs/{id}/live`). WebSocket messages need a Chromium-based browser.
`deliver`	object	none	Webhooks get `job.event` with a batch of events, at most every 2 s.

Events arrive at once on GET /v1/jobs/{id}/events as event: event with {"events": [...]}, and are stored with the job's results: {"type": "xhr" | "fetch", "url", "method", "status", "response_body"}, {"type": "ws", "url", "direction", "data"}, {"type": "dom", "selector", "text", "html"}, each with at; bodies are cut at 64 KB.

curl -s https://cdpfleet.com/v1/functions/watch -H "x-api-key: $CDPFLEET_API_KEY" -H "content-type: application/json"   -d '{"url":"https://quotes.toscrape.com/scroll","duration":300,"filter":{"url":["*api/quotes*"],"types":["xhr"]}}'
curl -N https://cdpfleet.com/v1/jobs/$JOB/events -H "x-api-key: $CDPFLEET_API_KEY"

Warmup: a browser that browses like a person

POST https://cdpfleet.com/v1/functions/warmup keeps a browser of yours browsing a site the way a visitor would — reading, scrolling, moving the mouse, following links, now and then searching — on one sticky IP, for as long as you like. Its cookies and storage build up in a persistent profile, and you can scrape pages in the same tab whenever you need, with that history behind you.

Field	Type	Default	What it does
`url`	string	required	Where to start.
`sites`	string[]	none	More start pages to wander between.
`duration`	number	until stopped	Seconds to run.
`profile`	string	none	Your persistent profile to keep it in (created if missing, saved when the warmup ends — stopped or at its `duration`); reuse it later in any session or function.
`proxy`	string	Community proxy	Best a sticky residential token (`cdpfleet-resi-…-session-name`) or your own; the same exit is used throughout.
`behavior`	object	all on	`{"scroll": true, "mouse": true, "typing": true, "dwell_s": [5, 40], "pages_per_site": [3, 12]}`.
`cookie_every`	number	`30`	Minutes between cookie deliveries (webhook `job.cookies`, results, and `GET /v1/jobs/{id}/cookies`).
`browser, launch, device, deliver`	—	Camoufox	As for watch, except that the default browser is Camoufox.

POST /v1/jobs/{id}/scrape with a scrape body — the warmup pauses, loads the page in its own tab (same cookies, same IP), answers like scrape, and carries on.
GET /v1/jobs/{id}/cookies — the cookies now, or after the end the last ones it delivered.
It never follows logout, delete, checkout or order links and never submits forms other than searches. It never solves a CAPTCHA or waits on one: on a challenge page it goes back or elsewhere on the site, counts it (progress.challenges) and leaves that site alone for a while.
Progress: {visits, sites, challenges, scrapes, last_url, cookies_at}; results also log each visit.

Watch and warmup run in your own browser session: they hold its threads (2 for a headful browser) and are billed as a session, and continue past the 24-hour session limit by themselves.

Stop and resume

DELETE /v1/jobs/{id} stops a job after the pages in flight (stopping, then stopped); its results stay. POST /v1/jobs/{id}/resume continues a stopped or interrupted job where it left off — a crawl keeps its queue and the pages it has seen, so nothing is fetched twice. A job is interrupted when its run can't continue: your minutes or threads running out, or a restart on our side.

Billing

A running job holds concurrency threads and uses browser-minutes per second for each, like that many sessions; billed_seconds is the total in thread-seconds. Starting a job needs that many free threads (429 threads_exceeded otherwise). The Community proxy and your own proxies cost nothing extra; residential traffic is billed per GB.

Webhooks and email

With "deliver": {"webhook": {"url": "https://…"}} we POST JSON to your URL. events picks what you get (default ["done"]):

"done" — job.done when the job ends, with its status: done, stopped (you stopped it), failed or interrupted.
"page" — job.page for each page of a batch or crawl (a page over 1 MB comes as result_url).
"event" — job.event batches of a watch's live events, at most one request per 2 s (the last batch, sent when the watch stops, can follow sooner).
"archive" — job.archive with the zip's url and bytes once it's ready (jobs with "format": "archive").

Each request carries x-cdpfleet-event, x-cdpfleet-delivery (a unique id; retries reuse it) and x-cdpfleet-signature: t=<unix time>,v1=<hex>. Answer with any 2xx; otherwise we retry after 10 s, 1 min, 5 min, 30 min and 2 h. The URL must be public https.

Your signing secret (whsec_…) is the same for all your jobs: GET /v1/me/webhook-secret, and POST /v1/me/webhook-secret/rotate to replace it. Check every request:

// Node.js: verify x-cdpfleet-signature on the raw request body
import crypto from 'node:crypto';
function verify(rawBody, header, secret, toleranceS = 300) {
  const { t, v1 } = Object.fromEntries(header.split(',').map((kv) => kv.split('=')));
  if (Math.abs(Date.now() / 1000 - Number(t)) > toleranceS) return false;
  const want = crypto.createHmac('sha256', secret).update(`${t}.${rawBody}`).digest('hex');
  return crypto.timingSafeEqual(Buffer.from(want), Buffer.from(v1));
}

The same check in Python, Java, C# and Go, de-duplicating retries and rotating the secret: Webhooks.

"email": true sends a short note to your account's address when the job ends.

Errors

Status	`error`	When
400	`invalid_option` / `invalid_url`	A field out of range, or a URL that isn't public http(s); the message names it.
403	`jobs_not_available`	Jobs come with paid plans.
404	`no such job`	Not one of your jobs.
409	`not_resumable`	Only `stopped` and `interrupted` jobs resume.
410	`results_expired` / `job_expired`	More than 14 days after the end.
429	`threads_exceeded`, `daily_quota_exhausted`	Not enough free threads or minutes to start; wait `Retry-After`.