[Functions](https://cdpfleet.com/docs/functions)

# Map

A site's URLs from its sitemaps and the start page's links, without loading every page — the quick first look before a crawl. `POST https://cdpfleet.com/v1/functions/map` → `application/json`.

 

```bash
curl -s https://cdpfleet.com/v1/functions/map \
  -H "x-api-key: $CDPFLEET_API_KEY" -H "content-type: application/json" \
  -d '{"url":"https://www.djangoproject.com/","max_urls":200,"include":["/weblog/**"]}' -o urls.json -D - | grep -i '^x-call-id'
```

```js
// Node.js 18+: save as map.mjs, run node map.mjs
const res = await fetch('https://cdpfleet.com/v1/functions/map', {
  method: 'POST',
  headers: { 'x-api-key': process.env.CDPFLEET_API_KEY, 'content-type': 'application/json' },
  body: JSON.stringify({
    url: 'https://www.djangoproject.com/',
    max_urls: 200,
    include: ['/weblog/**'],
  }),
  signal: AbortSignal.timeout(30000),
});
if (!res.ok) throw new Error(`${res.status} ${await res.text()}`); // 429: wait Retry-After seconds
console.log(await res.text());
```

```python
# Python 3 with requests (pip install requests): save as map.py, run python map.py
import os
import requests

r = requests.post(
    "https://cdpfleet.com/v1/functions/map",
    headers={"x-api-key": os.environ["CDPFLEET_API_KEY"]},
    json={
        "url": "https://www.djangoproject.com/",
        "max_urls": 200,
        "include": ["/weblog/**"],
    },
    timeout=30,
)
if not r.ok:
    raise SystemExit(f"{r.status_code} {r.text}")  # 429: wait r.headers["Retry-After"] seconds
print(r.text)
```

```java
// Java 17+: save as Main.java, run java Main.java
import java.net.URI;
import java.net.http.*;
import java.time.Duration;

public class Main {
  public static void main(String[] args) throws Exception {
    String body = """
        {
          "url": "https://www.djangoproject.com/",
          "max_urls": 200,
          "include": ["/weblog/**"]
        }
        """;
    HttpResponse<byte[]> res = HttpClient.newHttpClient().send(
        HttpRequest.newBuilder(URI.create("https://cdpfleet.com/v1/functions/map"))
            .header("x-api-key", System.getenv("CDPFLEET_API_KEY"))
            .header("content-type", "application/json")
            .timeout(Duration.ofSeconds(30))
            .POST(HttpRequest.BodyPublishers.ofString(body))
            .build(),
        HttpResponse.BodyHandlers.ofByteArray());
    if (res.statusCode() != 200) { // 429: wait Retry-After seconds
      throw new RuntimeException(res.statusCode() + " " + new String(res.body()));
    }
    System.out.writeBytes(res.body()); // UTF-8 as it came
    System.out.println();
  }
}
```

```csharp
// .NET 8+: dotnet new console, replace Program.cs with this, run dotnet run
using System.Text;

using var http = new HttpClient { Timeout = TimeSpan.FromSeconds(30) };
http.DefaultRequestHeaders.Add("x-api-key", Environment.GetEnvironmentVariable("CDPFLEET_API_KEY"));
var body = """
    {
      "url": "https://www.djangoproject.com/",
      "max_urls": 200,
      "include": ["/weblog/**"]
    }
    """;
var res = await http.PostAsync("https://cdpfleet.com/v1/functions/map",
    new StringContent(body, Encoding.UTF8, "application/json"));
if (!res.IsSuccessStatusCode) // 429: wait Retry-After seconds
    throw new Exception($"{(int)res.StatusCode} {await res.Content.ReadAsStringAsync()}");
Console.WriteLine(await res.Content.ReadAsStringAsync());
```

```go
// Go 1.21+: save as main.go, run go run main.go
package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
	"os"
	"strings"
	"time"
)

func main() {
	body := `{
	  "url": "https://www.djangoproject.com/",
	  "max_urls": 200,
	  "include": ["/weblog/**"]
	}`
	req, _ := http.NewRequest("POST", "https://cdpfleet.com/v1/functions/map", strings.NewReader(body))
	req.Header.Set("x-api-key", os.Getenv("CDPFLEET_API_KEY"))
	req.Header.Set("content-type", "application/json")
	res, err := (&http.Client{Timeout: 30 * time.Second}).Do(req)
	if err != nil {
		log.Fatal(err)
	}
	defer res.Body.Close()
	data, err := io.ReadAll(res.Body)
	if err != nil {
		log.Fatal(err)
	}
	if res.StatusCode != http.StatusOK { // 429: wait Retry-After seconds
		log.Fatalf("%s %s", res.Status, data)
	}
	fmt.Println(string(data))
}
```

## Fields

| Field | Type | Default | What it does |
| --- | --- | --- | --- |
| `url` | string | required | The page to load. A public `http` or `https` address: private, loopback, link-local, CGNAT and cloud-metadata addresses, `localhost`, `.local`/`.internal` names and names that resolve to them are refused (`400 invalid_url`). |
| `country` | string | `"random"` | Two-letter country for the exit IP and for the page's locale, timezone, geolocation and `Accept-Language` (see [countries](#countries)); `"random"` picks one. Unknown codes: `400 invalid_country`. |
| `max_wait` | number | `20` | Seconds to wait for the page to load, 1–60 (`400 invalid_option` otherwise). The whole call returns within about `max_wait` + 5 s, so a client timeout of `max_wait` + 10 s is safe. A page that is still loading then comes back partial (below); one that never started: `408 page_timeout`. |
| `proxy` | string or string\[\] | the Community proxy | Omit it to use the free Community proxy in `country`. Or: your own proxy URL (`http://`, `https://`, `socks5://`), a list (one is picked per call), a `cdpfleet-plan…` token, or a residential `cdpfleet-resi…` token (tier, country, device OS, sticky session) billed per GB to your residential balance ([proxies](https://cdpfleet.com/docs/proxies)). `proxy_rules` isn't supported yet (`400 invalid_option`). |
| `max_urls` | number | `500` | At most this many URLs, 1–5,000. |
| `include` | string\[\] | all | Keep only URLs matching one of these patterns. `*` matches within a path segment, **across segments; a pattern starting with `/` matches the path on any host, e.g. `"/blog/`**`"`. |
| `exclude` | string\[\] | none | Drop URLs matching one of these patterns (same syntax), e.g. `["/tag/**", "**.pdf"]`. |
| `same_site` | boolean | `true` | Only URLs on the start URL's site. |

## Good to know

-   Response: `urls[]` and `sources {sitemap, links}` — how many came from sitemaps (from robots.txt, including nested sitemap indexes) and how many from the start page's links.
-   For more than 5,000 URLs, or links followed deeper than the start page, add `"job": true` (paid plans): it runs as a [map job](https://cdpfleet.com/docs/jobs#map) with no limit.

Billing, errors and countries: [Functions](https://cdpfleet.com/docs/functions).

---
Source: https://cdpfleet.com/docs/functions/map · Markdown: https://cdpfleet.com/docs/functions/map.md · All docs for AI tools: https://cdpfleet.com/llms.txt