Vision settings

Every setting for scrapes, batches, crawls, maps and browser sessions, with its default, values and an example, and what is fixed in the beta.

Set these in the body of each request. Batches and crawls take the scrape settings under scrape and apply them to every page: {"scrape": {"engine": "fetch"}}. Scrape a page explains how the tiers and the cache work together.

SettingDefault
Engine: engineauto
Browser: browserheadless-shell (scrape), chrome (session)
Formats: formatsmarkdown
Main content only: only_main_contenttrue
Wait after load: wait_for_ms0 ms
Scrape timeout: timeout_ms30,000 ms
Walls: wallsfree
Robots.txt: respect_robotsfalse
Locale: localeen-US
Timezone: timezoneAmerica/Los_Angeles
Viewport: viewport1280 by 800 CSS pixels, or 390 by 844 with mobile (per field)
Phone: mobilefalse
Ad blocking: block_adstrue (scrape), false (session)
Resources not loaded: block_resourcesimage, media, font; none when a screenshot is asked for (per field)
Screenshot: screenshot.formatjpeg (per field)
Cache age: max_age_ms172,800,000 ms (2 days)
Store in the cache: store_in_cachetrue
Cache only: cache_onlyfalse
Page limit: limit100 pages (crawl), 1,000 pages (map)
Depth: max_depth5
Sitemaps: sitemapinclude
Whole domain: crawl_entire_domainfalse
Subdomains: allow_subdomainsfalse (per field)
Other sites: allow_external_linksfalse
Query strings: ignore_query_parametersfalse (crawl), true (map)
Spacing: delay_ms250 ms, or robots.txt's Crawl-delay with respect_robots
Pages at once: max_concurrencyyour plan's scrapes at once (also the most) (per field)
Invalid URLs: ignore_invalid_urlsfalse
Session lifetime: timeout_s300 s
Save the profile: persist_profilefalse
Timeline thumbnails: thumbnailsfalse

How the page is read

Engine

Which tier reads the page. auto tries the cache, then a plain HTTP fetch, and starts a browser only when the page needs one. fetch never starts a browser: faster and cheaper, but a page that builds its content with JavaScript comes back thin. browser always renders, and actions, screenshot and profile imply it.

json
{"url": "https://example.com", "engine": "fetch"}
bash
b3os scrape https://example.com --engine fetch

Browser

Which browser renders the page. A scrape's headless-shell page renders in the shared scrape pool. chrome is full Chrome, with window.chrome, plugins and WebGL; a scrape that asks for it renders in a browser of its own, which counts toward vision.sessions.concurrent while it runs. A session that asks for headless-shell starts its browser cold. Not allowed with engine: fetch.

json
{"url": "https://example.com", "browser": "chrome"}
bash
b3os scrape https://example.com --browser chrome

Formats

What comes back. Each format is one more member of the result. screenshot needs a browser, and json asks the node's model to fill a JSON Schema from the page, which also needs the infer:run scope and bills its tokens.

  • Field: formats on scrape
  • Default: markdown
  • Values: any of markdown, html, raw_html, text, links, metadata, screenshot and json
  • Invalid value: 400 invalid_request
  • Guide: Formats
json
{"url": "https://example.com", "formats": ["markdown", "links"]}
bash
b3os scrape https://example.com --format markdown --format links

Main content only

Drops headers, navigation, footers and sidebars from the Markdown, HTML and text. Turn it off to keep the whole page.

json
{"url": "https://example.com", "only_main_content": false}
bash
b3os scrape https://example.com --all-content

Wait after load

Extra time after the page is ready and before vision reads it, for content that arrives late. It applies when a browser renders the page, and it counts inside timeout_ms.

json
{"url": "https://example.com", "wait_for_ms": 2000}
bash
b3os scrape https://example.com --wait-for-ms 2000

Scrape timeout

How long the whole request may take, the wait after load included. A scrape that runs out answers 504 scrape_timeout and bills nothing. With engine: auto, a page the fetch found thin is rendered in a browser only while at least 5 seconds remain.

json
{"url": "https://example.com", "timeout_ms": 60000}

Walls

What a wall costs: a refusal or challenge page, page_state blocked or captcha. free bills nothing and returns no content, only page_state, status_code, final_url and content_withheld: true. bill bills it as a page and returns its content. A scrape with actions bills either way.

json
{"url": "https://example.com", "walls": "bill"}

Robots.txt

Checks the site's robots.txt (product token B3OSBot) and Content Signals before fetching. A URL they disallow answers 403 robots_disallowed; a crawl skips it and uses the file's Crawl-delay. Off, vision does not read robots.txt.

json
{"url": "https://example.com", "respect_robots": true}
bash
b3os crawl https://example.com --respect-robots

How the browser presents itself

These apply when a browser renders the page. A page read by the plain fetch sends only locale, as Accept-Language.

Locale

The language the page sees: Accept-Language, navigator.language and the browser's own language. Set it for a site that picks its language from the request.

json
{"url": "https://example.com", "locale": "de-DE"}

Timezone

The time zone the page's Date and Intl use. The default is where the node's traffic leaves from.

json
{"url": "https://example.com", "timezone": "Europe/Berlin"}

Viewport

The browser window's size. A page that lays itself out by width shows its desktop or narrow layout from it.

  • Field: viewport on scrape and session
  • Default: 1280 by 800 CSS pixels, or 390 by 844 with mobile (scrape), 1280 by 800 CSS pixels (session)
  • Values: width: 320 to 3,840; height: 240 to 2,160
  • Invalid value: 400 invalid_request
json
{"url": "https://example.com", "viewport": {"width": 1440, "height": 900}}

Phone

Presents a phone: a narrow viewport, a phone's user agent and touch. Use it for a site that serves phones a different page.

json
{"url": "https://example.com", "mobile": true}
bash
b3os scrape https://example.com --mobile

Ad blocking

Blocks requests to ad and tracker hosts and hides cookie banners. Turn it off to read the page as a visitor with no blocker sees it.

json
{"url": "https://example.com", "block_ads": false}

Resources not loaded

Resource types the browser does not download. Fewer bytes make a faster page; a page whose layout needs its stylesheets or fonts reads worse. [] loads everything.

  • Field: block_resources on scrape and session
  • Default: image, media, font; none when a screenshot is asked for (scrape), none (session)
  • Values: any of image, media, font and stylesheet
  • Invalid value: 400 invalid_request
json
{"url": "https://example.com", "block_resources": []}

Screenshot

How the screenshot format encodes the page. These defaults are vision's and differ from render's screenshots, which default to WebP at 90. full_page captures the whole scroll height instead of the viewport.

  • Field: screenshot.format, screenshot.quality and screenshot.full_page on scrape
  • Default: jpeg (screenshot.format), 80 (screenshot.quality), false (screenshot.full_page)
  • Values: jpeg, png or webp (screenshot.format); 1 to 100 (screenshot.quality); true or false (screenshot.full_page)
  • Invalid value: 400 invalid_request
json
{"url": "https://example.com", "formats": ["screenshot"], "screenshot": {"format": "webp", "quality": 90}}

The cache

Cache age

Accepts a cached copy up to this old. A cached page answers fastest and may be stale. 0 always fetches.

json
{"url": "https://example.com", "max_age_ms": 0}
bash
b3os scrape https://example.com --max-age-ms 0

Store in the cache

Stores this result for later scrapes. A request with headers, a profile or actions is never shared with other organizations either way.

json
{"url": "https://example.com", "store_in_cache": false}

Cache only

Answers from the cache and never contacts the site: a cached copy, or 404 scrape_no_cached_data, which is not billed. Cannot be combined with headers, profile, actions or a screenshot.

json
{"url": "https://example.com", "cache_only": true}

Crawls, batches and maps

Page limit

The most pages a crawl scrapes, or the most URLs a map returns. A crawl over your plan's vision.crawl.pages_max is refused before it starts.

json
{"url": "https://example.com", "limit": 50}
bash
b3os crawl https://example.com --limit 50

Depth

The most link hops from the start URL. URLs from a sitemap count as depth 0.

json
{"url": "https://example.com", "max_depth": 2}
bash
b3os crawl https://example.com --max-depth 2

Sitemaps

Whether to use the site's sitemaps beside its links (include), ignore them (skip), or use only them (only).

json
{"url": "https://example.com", "sitemap": "only"}
bash
b3os map https://example.com --sitemap only

Whole domain

Follows links to parent and sibling paths too, not only to paths below the start URL.

json
{"url": "https://example.com", "crawl_entire_domain": true}

Subdomains

Whether a crawl follows, or a map lists, URLs on the site's subdomains. The two fields default differently: a crawl stays on its host, a map lists subdomains.

  • Field: allow_subdomains and include_subdomains on crawl and map
  • Default: false (allow_subdomains, crawl), true (include_subdomains, map)
  • Values: true or false
  • Invalid value: 400 invalid_request
json
{"url": "https://example.com", "allow_subdomains": true}
bash
b3os crawl https://example.com --include-subdomains

Other sites

Scrapes the pages on other sites that the crawl links to, one hop. Their own links are not followed.

json
{"url": "https://example.com", "allow_external_links": true}

Query strings

Treats URLs that differ only in their query string as one page. A map does by default; a crawl does not.

  • Field: ignore_query_parameters on crawl and map
  • Default: false (crawl), true (map)
  • Values: true or false
  • Invalid value: 400 invalid_request
json
{"url": "https://example.com", "ignore_query_parameters": true}

Spacing

The least time between this crawl's requests to one host. Whatever you set, job pages to one host start at least 250 ms apart node-wide, at most 2 at a time for your organization, and a 429 or 503 from the host pauses them for its Retry-After.

  • Field: delay_ms on crawl
  • Default: 250 ms, or robots.txt's Crawl-delay with respect_robots
  • Values: 0 to 60,000 ms
  • Invalid value: 400 invalid_request
json
{"url": "https://example.com", "delay_ms": 1000}

Pages at once

How many pages of one batch or crawl are scraped at once. A batch takes at most your vision.scrape.concurrent limit. A higher value can finish a job sooner when its pages are on many hosts; pages to one host are spaced either way.

  • Field: max_concurrency on batch and crawl
  • Default: your plan's scrapes at once (also the most) (batch), 8 (also the most) (crawl)
  • Values: 1 or more
  • Invalid value: 400 invalid_request
json
{"urls": ["https://example.com/a", "https://example.com/b"], "max_concurrency": 2}
bash
b3os scrape https://example.com/a https://example.com/b --max-concurrency 2

Invalid URLs

Skips URLs that do not parse, and lists them in result.invalid_urls, instead of refusing the whole batch.

json
{"urls": ["https://example.com/a", "not a url"], "ignore_invalid_urls": true}

Browser sessions

Session lifetime

Closes the session this many seconds after it was created. A session bills browser time while it is open, so close it when you are done.

json
{"timeout_s": 900}
bash
b3os session create --timeout-s 900

Save the profile

persist_profile saves the session's cookies and storage back to its profile when it closes; the profile is locked meanwhile. overwrite_profile saves even when another session saved a newer revision since this one loaded it.

json
{"profile": "github", "persist_profile": true}
bash
b3os session create --profile github --persist-profile

Timeline thumbnails

Adds a small JPEG of the viewport, taken after each act step, to that step's row in the session timeline. Each one is one more capture in the step's time.

json
{"thumbnails": true}

Fixed in the beta

These work one way in the beta, and no request field changes them.

TLS verification

Always on. Vision checks every site's certificate. A site with an expired, self-signed or mismatched certificate fails, and retrying does not help: the fetch answers 502 upstream_unreachable with reason: tls, and the browser 502 navigation_failed with Chrome's certificate error. The Firecrawl surface accepts skipTlsVerification and ignores it.

PDF files

Not read. A URL that serves a PDF answers 422 content_type_unsupported, bills nothing, and is not retried in a browser. The Firecrawl surface accepts parsePDF and parsers: ["pdf"], which change nothing.

Challenge pages

Not waited on. Vision does not wait for a Cloudflare or Akamai challenge to pass, and never solves a CAPTCHA. A challenge comes back as page_state captcha or blocked, which is free under walls: free.

Walled pages in full Chrome

On. A page from the scrape pool that comes back a wall is read once more in a full Chrome browser of your organization's own, and that answer is used when it is not a wall; a warning says so. It adds time to a walled page and takes none of your vision.sessions.concurrent places. It is skipped when the request names browser or runs actions.

Model for structured data

The node's default chat model. The json format asks b3os/default (see Infer) for at most 2,048 tokens per call, at most two calls per page. When no answer is JSON, the page comes back without json and with a warning.

Spacing between job pages

250 ms per host. Pages of batches and crawls to one host start at least 250 ms apart across every organization on the node, with at most 2 in flight for one organization and 2 for all. A crawl's delay_ms can only add to it. Single scrapes are not spaced.

Proxy

None. Every fetch and navigation leaves through the node's egress guard, from the node's own address. You cannot bring a proxy, and a session's CDP socket refuses a browser context with proxy settings of its own. The Firecrawl surface's proxy: "enhanced" and "stealth" answer 400 unsupported_option.

Previous Profiles
Next Limits and FAQ
Ask a question... ⌘I