> ## Documentation Index
> Fetch the complete documentation index at: https://docs.beta.b3os.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Vision settings

Every setting for scrapes, batches, crawls, maps and browser sessions, with its default, values and an example, and what is fixed in the beta.

Set these in the body of each request. Batches and crawls take the scrape settings under `scrape` and apply them to every page: `{"scrape": {"engine": "fetch"}}`. [Scrape a page](/vision/scrape) explains how the tiers and the cache work together.

| Setting | Default |
|---|---|
| [Engine](#engine): `engine` | `auto` |
| [Browser](#browser): `browser` | `headless-shell` (scrape), `chrome` (session) |
| [Formats](#formats): `formats` | `markdown` |
| [Main content only](#main-content-only): `only_main_content` | `true` |
| [Wait after load](#wait-after-load): `wait_for_ms` | 0 ms |
| [Scrape timeout](#scrape-timeout): `timeout_ms` | 30,000 ms |
| [Walls](#walls): `walls` | `free` |
| [Robots.txt](#robotstxt): `respect_robots` | `false` |
| [Locale](#locale): `locale` | `en-US` |
| [Timezone](#timezone): `timezone` | `America/Los_Angeles` |
| [Viewport](#viewport): `viewport` | 1280 by 800 CSS pixels, or 390 by 844 with `mobile` ([per field](#viewport)) |
| [Phone](#phone): `mobile` | `false` |
| [Ad blocking](#ad-blocking): `block_ads` | `true` (scrape), `false` (session) |
| [Resources not loaded](#resources-not-loaded): `block_resources` | `image`, `media`, `font`; none when a screenshot is asked for ([per field](#resources-not-loaded)) |
| [Screenshot](#screenshot): `screenshot.format` | `jpeg` ([per field](#screenshot)) |
| [Cache age](#cache-age): `max_age_ms` | 172,800,000 ms (2 days) |
| [Store in the cache](#store-in-the-cache): `store_in_cache` | `true` |
| [Cache only](#cache-only): `cache_only` | `false` |
| [Page limit](#page-limit): `limit` | 100 pages (crawl), 1,000 pages (map) |
| [Depth](#depth): `max_depth` | 5 |
| [Sitemaps](#sitemaps): `sitemap` | `include` |
| [Whole domain](#whole-domain): `crawl_entire_domain` | `false` |
| [Subdomains](#subdomains): `allow_subdomains` | `false` ([per field](#subdomains)) |
| [Other sites](#other-sites): `allow_external_links` | `false` |
| [Query strings](#query-strings): `ignore_query_parameters` | `false` (crawl), `true` (map) |
| [Spacing](#spacing): `delay_ms` | 250 ms, or robots.txt's Crawl-delay with `respect_robots` |
| [Pages at once](#pages-at-once): `max_concurrency` | your plan's scrapes at once (also the most) ([per field](#pages-at-once)) |
| [Invalid URLs](#invalid-urls): `ignore_invalid_urls` | `false` |
| [Session lifetime](#session-lifetime): `timeout_s` | 300 s |
| [Save the profile](#save-the-profile): `persist_profile` | `false` |
| [Timeline thumbnails](#timeline-thumbnails): `thumbnails` | `false` |

## How the page is read

### Engine

Which tier reads the page. `auto` tries the cache, then a plain HTTP fetch, and starts a browser only when the page needs one. `fetch` never starts a browser: faster and cheaper, but a page that builds its content with JavaScript comes back thin. `browser` always renders, and `actions`, `screenshot` and `profile` imply it.

- **Field:** `engine` on [scrape](/api/post-v1-vision-scrape)
- **Default:** `auto`
- **Values:** `auto`, `fetch` or `browser`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)
- **Guide:** [Choose the engine](/vision/scrape#choose-the-engine)

```json
{"url": "https://example.com", "engine": "fetch"}
```

```bash
b3os scrape https://example.com --engine fetch
```

### Browser

Which browser renders the page. A scrape's `headless-shell` page renders in the shared scrape pool. `chrome` is full Chrome, with `window.chrome`, plugins and WebGL; a scrape that asks for it renders in a browser of its own, which counts toward `vision.sessions.concurrent` while it runs. A session that asks for `headless-shell` starts its browser cold. Not allowed with `engine: fetch`.

- **Field:** `browser` on [scrape](/api/post-v1-vision-scrape) and [session](/api/post-v1-vision-sessions)
- **Default:** `headless-shell` (scrape), `chrome` (session)
- **Values:** `chrome` or `headless-shell`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)
- **Guide:** [Browsers](/vision#browsers)

```json
{"url": "https://example.com", "browser": "chrome"}
```

```bash
b3os scrape https://example.com --browser chrome
```

### Formats

What comes back. Each format is one more member of the result. `screenshot` needs a browser, and `json` asks the node's model to fill a JSON Schema from the page, which also needs the `infer:run` scope and bills its tokens.

- **Field:** `formats` on [scrape](/api/post-v1-vision-scrape)
- **Default:** `markdown`
- **Values:** any of `markdown`, `html`, `raw_html`, `text`, `links`, `metadata`, `screenshot` and `json`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)
- **Guide:** [Formats](/vision/scrape#formats)

```json
{"url": "https://example.com", "formats": ["markdown", "links"]}
```

```bash
b3os scrape https://example.com --format markdown --format links
```

### Main content only

Drops headers, navigation, footers and sidebars from the Markdown, HTML and text. Turn it off to keep the whole page.

- **Field:** `only_main_content` on [scrape](/api/post-v1-vision-scrape)
- **Default:** `true`
- **Values:** `true` or `false`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)
- **Guide:** [Main content only](/vision/scrape#main-content-only)

```json
{"url": "https://example.com", "only_main_content": false}
```

```bash
b3os scrape https://example.com --all-content
```

### Wait after load

Extra time after the page is ready and before vision reads it, for content that arrives late. It applies when a browser renders the page, and it counts inside `timeout_ms`.

- **Field:** `wait_for_ms` on [scrape](/api/post-v1-vision-scrape)
- **Default:** 0 ms
- **Values:** 0 to 30,000 ms
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)

```json
{"url": "https://example.com", "wait_for_ms": 2000}
```

```bash
b3os scrape https://example.com --wait-for-ms 2000
```

### Scrape timeout

How long the whole request may take, the wait after load included. A scrape that runs out answers `504 scrape_timeout` and bills nothing. With `engine: auto`, a page the fetch found thin is rendered in a browser only while at least 5 seconds remain.

- **Field:** `timeout_ms` on [scrape](/api/post-v1-vision-scrape) and [map](/api/post-v1-vision-map)
- **Default:** 30,000 ms
- **Values:** 1,000 to 60,000 ms
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)

```json
{"url": "https://example.com", "timeout_ms": 60000}
```

### Walls

What a wall costs: a refusal or challenge page, `page_state` `blocked` or `captcha`. `free` bills nothing and returns no content, only `page_state`, `status_code`, `final_url` and `content_withheld: true`. `bill` bills it as a page and returns its content. A scrape with `actions` bills either way.

- **Field:** `walls` on [scrape](/api/post-v1-vision-scrape)
- **Default:** `free`
- **Values:** `free` or `bill`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)
- **Guide:** [Walls are free](/vision/scrape#walls-are-free)

```json
{"url": "https://example.com", "walls": "bill"}
```

### Robots.txt

Checks the site's robots.txt (product token `B3OSBot`) and Content Signals before fetching. A URL they disallow answers `403 robots_disallowed`; a crawl skips it and uses the file's Crawl-delay. Off, vision does not read robots.txt.

- **Field:** `respect_robots` on [scrape](/api/post-v1-vision-scrape), [crawl](/api/post-v1-vision-crawls) and [map](/api/post-v1-vision-map)
- **Default:** `false`
- **Values:** `true` or `false`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)
- **Guide:** [Rules vision follows](/vision#rules-vision-follows)

```json
{"url": "https://example.com", "respect_robots": true}
```

```bash
b3os crawl https://example.com --respect-robots
```

## How the browser presents itself

These apply when a browser renders the page. A page read by the plain fetch sends only `locale`, as `Accept-Language`.

### Locale

The language the page sees: `Accept-Language`, `navigator.language` and the browser's own language. Set it for a site that picks its language from the request.

- **Field:** `locale` on [scrape](/api/post-v1-vision-scrape) and [session](/api/post-v1-vision-sessions)
- **Default:** `en-US`
- **Values:** a BCP 47 language tag, such as `de-DE` or `ja-JP`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)

```json
{"url": "https://example.com", "locale": "de-DE"}
```

### Timezone

The time zone the page's `Date` and `Intl` use. The default is where the node's traffic leaves from.

- **Field:** `timezone` on [scrape](/api/post-v1-vision-scrape) and [session](/api/post-v1-vision-sessions)
- **Default:** `America/Los_Angeles`
- **Values:** an IANA time zone name, such as `Europe/Berlin`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)

```json
{"url": "https://example.com", "timezone": "Europe/Berlin"}
```

### Viewport

The browser window's size. A page that lays itself out by width shows its desktop or narrow layout from it.

- **Field:** `viewport` on [scrape](/api/post-v1-vision-scrape) and [session](/api/post-v1-vision-sessions)
- **Default:** 1280 by 800 CSS pixels, or 390 by 844 with `mobile` (scrape), 1280 by 800 CSS pixels (session)
- **Values:** `width`: 320 to 3,840; `height`: 240 to 2,160
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)

```json
{"url": "https://example.com", "viewport": {"width": 1440, "height": 900}}
```

### Phone

Presents a phone: a narrow viewport, a phone's user agent and touch. Use it for a site that serves phones a different page.

- **Field:** `mobile` on [scrape](/api/post-v1-vision-scrape)
- **Default:** `false`
- **Values:** `true` or `false`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)

```json
{"url": "https://example.com", "mobile": true}
```

```bash
b3os scrape https://example.com --mobile
```

### Ad blocking

Blocks requests to ad and tracker hosts and hides cookie banners. Turn it off to read the page as a visitor with no blocker sees it.

- **Field:** `block_ads` on [scrape](/api/post-v1-vision-scrape) and [session](/api/post-v1-vision-sessions)
- **Default:** `true` (scrape), `false` (session)
- **Values:** `true` or `false`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)

```json
{"url": "https://example.com", "block_ads": false}
```

### Resources not loaded

Resource types the browser does not download. Fewer bytes make a faster page; a page whose layout needs its stylesheets or fonts reads worse. `[]` loads everything.

- **Field:** `block_resources` on [scrape](/api/post-v1-vision-scrape) and [session](/api/post-v1-vision-sessions)
- **Default:** `image`, `media`, `font`; none when a screenshot is asked for (scrape), none (session)
- **Values:** any of `image`, `media`, `font` and `stylesheet`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)

```json
{"url": "https://example.com", "block_resources": []}
```

### Screenshot

How the `screenshot` format encodes the page. These defaults are vision's and differ from render's screenshots, which default to WebP at 90. `full_page` captures the whole scroll height instead of the viewport.

- **Field:** `screenshot.format`, `screenshot.quality` and `screenshot.full_page` on [scrape](/api/post-v1-vision-scrape)
- **Default:** `jpeg` (`screenshot.format`), 80 (`screenshot.quality`), `false` (`screenshot.full_page`)
- **Values:** `jpeg`, `png` or `webp` (`screenshot.format`); 1 to 100 (`screenshot.quality`); `true` or `false` (`screenshot.full_page`)
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)

```json
{"url": "https://example.com", "formats": ["screenshot"], "screenshot": {"format": "webp", "quality": 90}}
```

## The cache

### Cache age

Accepts a cached copy up to this old. A cached page answers fastest and may be stale. `0` always fetches.

- **Field:** `max_age_ms` on [scrape](/api/post-v1-vision-scrape)
- **Default:** 172,800,000 ms (2 days)
- **Values:** 0 ms or more
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)
- **Guide:** [The cache](/vision/scrape#the-cache)

```json
{"url": "https://example.com", "max_age_ms": 0}
```

```bash
b3os scrape https://example.com --max-age-ms 0
```

### Store in the cache

Stores this result for later scrapes. A request with headers, a profile or actions is never shared with other organizations either way.

- **Field:** `store_in_cache` on [scrape](/api/post-v1-vision-scrape)
- **Default:** `true`
- **Values:** `true` or `false`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)

```json
{"url": "https://example.com", "store_in_cache": false}
```

### Cache only

Answers from the cache and never contacts the site: a cached copy, or `404 scrape_no_cached_data`, which is not billed. Cannot be combined with `headers`, `profile`, `actions` or a screenshot.

- **Field:** `cache_only` on [scrape](/api/post-v1-vision-scrape)
- **Default:** `false`
- **Values:** `true` or `false`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)
- **Guide:** [Cache only](/vision/scrape#cache-only)

```json
{"url": "https://example.com", "cache_only": true}
```

## Crawls, batches and maps

### Page limit

The most pages a crawl scrapes, or the most URLs a map returns. A crawl over your plan's `vision.crawl.pages_max` is refused before it starts.

- **Field:** `limit` on [crawl](/api/post-v1-vision-crawls) and [map](/api/post-v1-vision-map)
- **Default:** 100 pages (crawl), 1,000 pages (map)
- **Values:** 1 to 500 pages (crawl); 1 to 5,000 pages (map)
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request), [`422 limit_exceeded`](/errors#limit_exceeded)

```json
{"url": "https://example.com", "limit": 50}
```

```bash
b3os crawl https://example.com --limit 50
```

### Depth

The most link hops from the start URL. URLs from a sitemap count as depth 0.

- **Field:** `max_depth` on [crawl](/api/post-v1-vision-crawls)
- **Default:** 5
- **Values:** 0 to 20
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)

```json
{"url": "https://example.com", "max_depth": 2}
```

```bash
b3os crawl https://example.com --max-depth 2
```

### Sitemaps

Whether to use the site's sitemaps beside its links (`include`), ignore them (`skip`), or use only them (`only`).

- **Field:** `sitemap` on [crawl](/api/post-v1-vision-crawls) and [map](/api/post-v1-vision-map)
- **Default:** `include`
- **Values:** `include`, `skip` or `only`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)

```json
{"url": "https://example.com", "sitemap": "only"}
```

```bash
b3os map https://example.com --sitemap only
```

### Whole domain

Follows links to parent and sibling paths too, not only to paths below the start URL.

- **Field:** `crawl_entire_domain` on [crawl](/api/post-v1-vision-crawls)
- **Default:** `false`
- **Values:** `true` or `false`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)

```json
{"url": "https://example.com", "crawl_entire_domain": true}
```

### Subdomains

Whether a crawl follows, or a map lists, URLs on the site's subdomains. The two fields default differently: a crawl stays on its host, a map lists subdomains.

- **Field:** `allow_subdomains` and `include_subdomains` on [crawl](/api/post-v1-vision-crawls) and [map](/api/post-v1-vision-map)
- **Default:** `false` (`allow_subdomains`, crawl), `true` (`include_subdomains`, map)
- **Values:** `true` or `false`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)

```json
{"url": "https://example.com", "allow_subdomains": true}
```

```bash
b3os crawl https://example.com --include-subdomains
```

### Other sites

Scrapes the pages on other sites that the crawl links to, one hop. Their own links are not followed.

- **Field:** `allow_external_links` on [crawl](/api/post-v1-vision-crawls)
- **Default:** `false`
- **Values:** `true` or `false`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)

```json
{"url": "https://example.com", "allow_external_links": true}
```

### Query strings

Treats URLs that differ only in their query string as one page. A map does by default; a crawl does not.

- **Field:** `ignore_query_parameters` on [crawl](/api/post-v1-vision-crawls) and [map](/api/post-v1-vision-map)
- **Default:** `false` (crawl), `true` (map)
- **Values:** `true` or `false`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)

```json
{"url": "https://example.com", "ignore_query_parameters": true}
```

### Spacing

The least time between this crawl's requests to one host. Whatever you set, job pages to one host start at least 250 ms apart node-wide, at most 2 at a time for your organization, and a 429 or 503 from the host pauses them for its `Retry-After`.

- **Field:** `delay_ms` on [crawl](/api/post-v1-vision-crawls)
- **Default:** 250 ms, or robots.txt's Crawl-delay with `respect_robots`
- **Values:** 0 to 60,000 ms
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)

```json
{"url": "https://example.com", "delay_ms": 1000}
```

### Pages at once

How many pages of one batch or crawl are scraped at once. A batch takes at most your `vision.scrape.concurrent` limit. A higher value can finish a job sooner when its pages are on many hosts; pages to one host are spaced either way.

- **Field:** `max_concurrency` on [batch](/api/post-v1-vision-batches) and [crawl](/api/post-v1-vision-crawls)
- **Default:** your plan's scrapes at once (also the most) (batch), 8 (also the most) (crawl)
- **Values:** 1 or more
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)

```json
{"urls": ["https://example.com/a", "https://example.com/b"], "max_concurrency": 2}
```

```bash
b3os scrape https://example.com/a https://example.com/b --max-concurrency 2
```

### Invalid URLs

Skips URLs that do not parse, and lists them in `result.invalid_urls`, instead of refusing the whole batch.

- **Field:** `ignore_invalid_urls` on [batch](/api/post-v1-vision-batches)
- **Default:** `false`
- **Values:** `true` or `false`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)

```json
{"urls": ["https://example.com/a", "not a url"], "ignore_invalid_urls": true}
```

## Browser sessions

### Session lifetime

Closes the session this many seconds after it was created. A session bills browser time while it is open, so close it when you are done.

- **Field:** `timeout_s` on [session](/api/post-v1-vision-sessions)
- **Default:** 300 s
- **Values:** 10 to 1,800 s
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)
- **Guide:** [Create a session](/vision/sessions#create-a-session)

```json
{"timeout_s": 900}
```

```bash
b3os session create --timeout-s 900
```

### Save the profile

`persist_profile` saves the session's cookies and storage back to its `profile` when it closes; the profile is locked meanwhile. `overwrite_profile` saves even when another session saved a newer revision since this one loaded it.

- **Field:** `persist_profile` and `overwrite_profile` on [session](/api/post-v1-vision-sessions)
- **Default:** `false`
- **Values:** `true` or `false`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)
- **Guide:** [Profiles](/vision/profiles)

```json
{"profile": "github", "persist_profile": true}
```

```bash
b3os session create --profile github --persist-profile
```

### Timeline thumbnails

Adds a small JPEG of the viewport, taken after each act step, to that step's row in the session timeline. Each one is one more capture in the step's time.

- **Field:** `thumbnails` on [session](/api/post-v1-vision-sessions)
- **Default:** `false`
- **Values:** `true` or `false`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)

```json
{"thumbnails": true}
```

## Fixed in the beta

These work one way in the beta, and no request field changes them.

### TLS verification

**Always on.** Vision checks every site's certificate. A site with an expired, self-signed or mismatched certificate fails, and retrying does not help: the fetch answers `502 upstream_unreachable` with `reason: tls`, and the browser `502 navigation_failed` with Chrome's certificate error. The Firecrawl surface accepts `skipTlsVerification` and ignores it.

- **Error:** [`502 upstream_unreachable`](/errors#upstream_unreachable), [`502 navigation_failed`](/errors#navigation_failed)

### PDF files

**Not read.** A URL that serves a PDF answers `422 content_type_unsupported`, bills nothing, and is not retried in a browser. The Firecrawl surface accepts `parsePDF` and `parsers: ["pdf"]`, which change nothing.

- **Error:** [`422 content_type_unsupported`](/errors#content_type_unsupported)

### Challenge pages

**Not waited on.** Vision does not wait for a Cloudflare or Akamai challenge to pass, and never solves a CAPTCHA. A challenge comes back as `page_state` `captcha` or `blocked`, which is free under `walls: free`.

### Walled pages in full Chrome

**On.** A page from the scrape pool that comes back a wall is read once more in a full Chrome browser of your organization's own, and that answer is used when it is not a wall; a warning says so. It adds time to a walled page and takes none of your `vision.sessions.concurrent` places. It is skipped when the request names `browser` or runs `actions`.

### Model for structured data

**The node's default chat model.** The `json` format asks `b3os/default` (see [Infer](/infer#models)) for at most 2,048 tokens per call, at most two calls per page. When no answer is JSON, the page comes back without `json` and with a warning.

- **Guide:** [Structured data](/vision/scrape#structured-data)

### Spacing between job pages

**250 ms per host.** Pages of batches and crawls to one host start at least 250 ms apart across every organization on the node, with at most 2 in flight for one organization and 2 for all. A crawl's `delay_ms` can only add to it. Single scrapes are not spaced.

### Proxy

**None.** Every fetch and navigation leaves through the node's egress guard, from the node's own address. You cannot bring a proxy, and a session's CDP socket refuses a browser context with proxy settings of its own. The Firecrawl surface's `proxy: "enhanced"` and `"stealth"` answer `400 unsupported_option`.

- **Error:** [`400 unsupported_option`](/errors#unsupported_option)