> ## Documentation Index
> Fetch the complete documentation index at: https://docs.beta.b3os.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Map, batch and crawl

List a site's URLs, scrape up to 1,000 known URLs, or crawl a site by following its links, as jobs you can wait on, stream or get a webhook for.

Three operations cover more than one page. Map is a single call; batch and crawl are [jobs](/concepts/jobs).

## Map a site

`POST /v1/vision/map` lists a site's URLs from its sitemaps and from the links on the start page, without scraping each page. Use it to decide what to scrape or crawl. It bills one `vision.page` per call.

**curl**

```bash
    curl -s https://beta-api.b3os.org/v1/vision/map \
      -H "Authorization: Bearer $B3OS_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"url": "https://docs.example.com", "search": "authentication", "limit": 100}'
    ```
  **TypeScript**

```ts
    import { B3OS } from "@b3os/sdk";

    const b3os = new B3OS();
    const map = await b3os.vision.map({ url: "https://docs.example.com", search: "authentication", limit: 100 });
    for (const link of map.links) console.log(link.url, link.title);
    ```
  **Python**

```python
    from b3os import B3OS

    b3os = B3OS()
    site = b3os.vision.map({"url": "https://docs.example.com", "search": "authentication", "limit": 100})
    for link in site["links"]:
        print(link["url"], link.get("title"))
    ```
  **MCP**

```text
    Map https://docs.example.com and list the pages about authentication.
    ```

    The agent calls the `vision_map` tool with `url` and `search`, and gets back URLs and titles.

| Field | Default | Meaning |
|---|---|---|
| `search` | none | Order results by relevance to this phrase |
| `sitemap` | `include` | Use sitemaps alongside links (`include`), not at all (`skip`), or only sitemaps (`only`) |
| `include_subdomains` | `true` | Include URLs on subdomains |
| `ignore_query_parameters` | `true` | Treat URLs that differ only in their query string as one |
| `limit` | 1000 | At most 5000 |

## Scrape a list of URLs

`POST /v1/vision/batches` scrapes up to 1,000 URLs with the same options, as one job. Each page bills one `vision.page`.

**curl**

```bash
    curl -s "https://beta-api.b3os.org/v1/vision/batches?wait=60" \
      -H "Authorization: Bearer $B3OS_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"urls": ["https://example.com/blog/a", "https://example.com/blog/b"], "scrape": {"formats": ["markdown"]}}'
    ```
  **TypeScript**

```ts
    // .wait() resolves with the finished job
    const batch = await b3os.vision
      .createBatch({
        urls: ["https://example.com/blog/a", "https://example.com/blog/b"],
        scrape: { formats: ["markdown"] },
      })
      .wait();
    // for await walks every page across result pages
    for await (const page of b3os.vision.listBatchPages(batch.id, { status: "succeeded" })) {
      console.log(page.url, page.scrape?.markdown?.length);
    }
    ```
  **Python**

```python
    batch = b3os.vision.create_batch(
        {"urls": ["https://example.com/blog/a", "https://example.com/blog/b"], "scrape": {"formats": ["markdown"]}},
        wait=True,  # returns the finished job
    )
    pages = b3os.vision.list_batch_pages(batch["id"], status="succeeded")
    for page in pages.auto_paging_iter():  # every page across result pages
        print(page["url"], len(page["scrape"]["markdown"]))
    ```
  **MCP**

```text
    Scrape these three blog posts and tell me which one mentions pricing: …
    ```

    The MCP server has no batch tool. For a handful of URLs the agent calls `vision_scrape` once per URL; for hundreds, call the batch endpoint from code.

- `scrape` takes the same options as a [single scrape](/vision/scrape) (formats, engine, actions, profile, headers) and applies them to every URL.
- `ignore_invalid_urls: true` skips URLs that do not parse and lists them in `result.invalid_urls`, instead of rejecting the request.
- `max_concurrency` limits how many pages run at once; your concurrency limit caps it either way.

The finished job's `result` counts `pages_succeeded` and `pages_failed`. The pages themselves, failures included, are at `GET /v1/vision/batches/{batch_id}/pages` in the order they finished, and they are listed while the batch is still running.

## Crawl a site

`POST /v1/vision/crawls` starts at `url`, discovers pages from links and sitemaps breadth first, and scrapes up to `limit` of them. One crawl runs at a time per organization. Each page bills one `vision.page`.

**curl**

```bash
    curl -s https://beta-api.b3os.org/v1/vision/crawls \
      -H "Authorization: Bearer $B3OS_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"url": "https://docs.example.com", "limit": 200, "include_paths": ["^/guides/"],
           "scrape": {"formats": ["markdown"]},
           "webhook": {"url": "https://example.com/hooks/b3os"}}'
    ```
  **TypeScript**

```ts
    // .events() asks the creating request for an event stream; it resumes after a dropped connection
    const events = b3os.vision
      .createCrawl({
        url: "https://docs.example.com",
        limit: 200,
        include_paths: ["^/guides/"],
        scrape: { formats: ["markdown"] },
      })
      .events();
    for await (const event of events) {
      if (event.type === "page.succeeded") console.log(event.page?.url);
    }
    ```
  **Python**

```python
    events = b3os.vision.create_crawl(
        {
            "url": "https://docs.example.com",
            "limit": 200,
            "include_paths": ["^/guides/"],
            "scrape": {"formats": ["markdown"]},
        },
        stream=True,  # the creating request answers with the job's event stream
    )
    for event in events:
        if event["type"] == "page.succeeded":
            print(event["page"]["url"])
    ```
  **MCP**

```text
    Crawl the guides on https://docs.example.com, at most 50 pages, and summarize what each guide covers.
    ```

    The agent calls `vision_crawl` with `url`, `limit` and `include_paths`. On the local MCP server the tool waits up to `wait_s` seconds and returns the pages as Markdown, or the crawl ID while it still runs, to pass back as `job_id`; on the remote server it takes the API body plus `wait` and returns the crawl job. For crawls of hundreds of pages, start them from code or a routine and let the agent read the pages afterwards.

| Field | Default | Meaning |
|---|---|---|
| `limit` | 100 | Most pages to scrape, at most 500 |
| `max_depth` | 5 | Most link hops from `url`, at most 20; sitemap URLs count as depth 0 |
| `include_paths`, `exclude_paths` | none | Regular expressions (RE2) a URL path must match, or must not match |
| `sitemap` | `include` | `include`, `skip` or `only` |
| `crawl_entire_domain` | `false` | Follow links to parent and sibling paths, not only below `url` |
| `allow_subdomains` | `false` | Follow links to subdomains |
| `allow_external_links` | `false` | Scrape pages on other sites the crawl links to, one hop only |
| `ignore_query_parameters` | `false` | Treat URLs that differ only in their query as one |
| `delay_ms` | none | Minimum gap between requests to the same host |
| `scrape` | `{"formats": ["markdown"]}` | Options for every page, as in a single scrape |

A crawl honors robots.txt: URLs it skipped for that reason are listed in `result.robots_blocked`. Pages, failures included, are at `GET /v1/vision/crawls/{crawl_id}/pages` with their `depth`, available while the crawl runs.

## Follow, wait or cancel

Batches and crawls follow the [job contract](/concepts/jobs):

- `?wait=<seconds>` (up to 60) on the create call returns the finished job if it ends in time, or `202` with the job and a `Location` to poll.
- `GET .../{id}/events` streams `job.*` lifecycle events plus `page.succeeded` and `page.failed` for each page. Reconnect with `Last-Event-ID` to resume.
- `webhook: {url, events}` on create posts the events you choose to your URL, signed.
- `POST .../{id}/cancel` stops new pages. Pages already scraped stay listed and billed.

Batch and crawl records, with their pages, are kept for 7 days after they finish.