Scrape a page

Fetches one URL and returns the formats you ask for. With `engine: "auto"` (the default) vision answers from the cache when a fresh copy exists (`max_age_ms`, default 2 days), otherwise fetches over HTTP and switches to a warm browser only when the page needs JavaScript. Custom `headers`, a `profile` or `actions` bypass the cache; a page that sets cookies (by a response or a script) is cached for your organization only. The response's `cache.state` and `tier` say what served it. Synchronous: the response arrives when the page is done, within `timeout_ms` (at most 60 seconds). For many pages use a batch scrape or a crawl.

POST /v1/vision/scrape
Fetches one URL and returns the formats you ask for. With `engine: "auto"` (the default) vision answers from the cache when a fresh copy exists (`max_age_ms`, default 2 days), otherwise fetches over HTTP and switches to a warm browser only when the page needs JavaScript. Custom `headers`, a `profile` or `actions` bypass the cache; a page that sets cookies (by a response or a script) is cached for your organization only. The response's `cache.state` and `tier` say what served it. Synchronous: the response arrives when the page is done, within `timeout_ms` (at most 60 seconds). For many pages use a batch scrape or a crawl.

Authentication

Bearer Token (b3os_sk_ API key or b3os_pat_ token) API Key (cookie: __Host-b3os_session)

Request Body required

application/json
url string (uri) REQUIRED
Absolute `http` or `https` URL.
formats string[]
Array of: string
only_main_content boolean
Drop headers, navigation, footers and sidebars.
include_tags string[]
CSS selectors to keep; everything else is dropped.
Array of: string
exclude_tags string[]
CSS selectors to drop.
Array of: string
engine string
Which tier may serve the page. `auto` (default): cache, then HTTP fetch, then a browser when the page needs JavaScript. `fetch`: never start a browser. `browser`: always render in Chrome (needed for `actions`, `screenshot` and profiles; implied by them).
Enum: auto, fetch, browser
wait_for_ms integer
Extra wait after the page is ready, before capture.
timeout_ms integer
max_age_ms integer
Accept a cached copy up to this old (default 2 days). `0` always fetches.
store_in_cache boolean
Store the result in the cache. Requests with headers, a profile or actions are never shared across organizations.
headers object
Extra request headers. A request with custom headers bypasses the shared cache. In a browser, a `Cookie` header becomes cookies for the page's host (path `/`), so it reaches the origin as on an HTTP fetch and is not sent to the other hosts the page loads from.
profile string
A profile ID or name to load cookies and storage from. Implies `engine: browser`. The page renders in a browser of its own (never one shared with other organizations), which counts toward `vision.sessions.concurrent` while it runs.
actions (object | object | object | object | object | object | object | object | object | object | object)[]
Browser actions to run before capture. Implies `engine: browser`.
Array of:
One of:
Option 1
kind string REQUIRED
Enum: navigate
url string (uri) REQUIRED
wait_until string
Enum: load, domcontentloaded, networkidle
timeout_ms integer
Option 2
kind string REQUIRED
Enum: click
target unknown | unknown | unknown REQUIRED
How an action finds its element (the same rules as render timelines, `timeline.schema.json#/$defs/target`): candidates match `selector` (CSS), `text` (visible text, case-insensitive substring unless `exact`) or `role` plus accessible `name`, limited to `within`; the best-scored candidate wins, or the `nth` match when set. An element that stays outside the viewport after scrolling it into view fails with `target_not_found`.
Any of:
Option 1unknown
Option 2unknown
Option 3unknown
button string
Enum: left, right, middle
count integer
modifiers string[]
Modifier keys held during the action.
Array of: string
timeout_ms integer
How long to wait for the target to appear.
Option 3
kind string REQUIRED
Enum: hover
target unknown | unknown | unknown REQUIRED
How an action finds its element (the same rules as render timelines, `timeline.schema.json#/$defs/target`): candidates match `selector` (CSS), `text` (visible text, case-insensitive substring unless `exact`) or `role` plus accessible `name`, limited to `within`; the best-scored candidate wins, or the `nth` match when set. An element that stays outside the viewport after scrolling it into view fails with `target_not_found`.
Any of:
Option 1unknown
Option 2unknown
Option 3unknown
timeout_ms integer
Option 4
kind string REQUIRED
Enum: type
target unknown | unknown | unknown
How an action finds its element (the same rules as render timelines, `timeline.schema.json#/$defs/target`): candidates match `selector` (CSS), `text` (visible text, case-insensitive substring unless `exact`) or `role` plus accessible `name`, limited to `within`; the best-scored candidate wins, or the `nth` match when set. An element that stays outside the viewport after scrolling it into view fails with `target_not_found`.
Any of:
Option 1unknown
Option 2unknown
Option 3unknown
text string REQUIRED
clear boolean
Clear the field before typing.
interval_ms integer
Delay between characters.
timeout_ms integer
Option 5
kind string REQUIRED
Enum: key
key string REQUIRED
`KeyboardEvent.key`, such as `Enter`, `Tab` or `a`.
modifiers string[]
Modifier keys held during the action.
Array of: string
Option 6
kind string REQUIRED
Enum: scroll
target unknown | unknown | unknown
How an action finds its element (the same rules as render timelines, `timeline.schema.json#/$defs/target`): candidates match `selector` (CSS), `text` (visible text, case-insensitive substring unless `exact`) or `role` plus accessible `name`, limited to `within`; the best-scored candidate wins, or the `nth` match when set. An element that stays outside the viewport after scrolling it into view fails with `target_not_found`.
Any of:
Option 1unknown
Option 2unknown
Option 3unknown
to object
A position in CSS pixels.
x number REQUIRED
y number REQUIRED
by object
A position in CSS pixels.
x number REQUIRED
y number REQUIRED
Option 7
kind string REQUIRED
Enum: select
target unknown | unknown | unknown REQUIRED
How an action finds its element (the same rules as render timelines, `timeline.schema.json#/$defs/target`): candidates match `selector` (CSS), `text` (visible text, case-insensitive substring unless `exact`) or `role` plus accessible `name`, limited to `within`; the best-scored candidate wins, or the `nth` match when set. An element that stays outside the viewport after scrolling it into view fails with `target_not_found`.
Any of:
Option 1unknown
Option 2unknown
Option 3unknown
value string REQUIRED
The option value to select.
Option 8
kind string REQUIRED
Enum: wait
ms integer
for object
Exactly one condition to wait for (the same as a timeline `wait` action's `for`).
selector string
An element matching this CSS selector is visible.
text string
This text is visible.
url string
The page URL contains this substring.
network_idle boolean
The network has been quiet.
Enum: true
timeout_ms integer
Option 9
kind string REQUIRED
Enum: evaluate
expression string REQUIRED
JavaScript evaluated in the page; a returned promise is awaited and the JSON-serializable result is returned.
Option 10
kind string REQUIRED
Enum: screenshot
target unknown | unknown | unknown
How an action finds its element (the same rules as render timelines, `timeline.schema.json#/$defs/target`): candidates match `selector` (CSS), `text` (visible text, case-insensitive substring unless `exact`) or `role` plus accessible `name`, limited to `within`; the best-scored candidate wins, or the `nth` match when set. An element that stays outside the viewport after scrolling it into view fails with `target_not_found`.
Any of:
Option 1unknown
Option 2unknown
Option 3unknown
full_page boolean
format string
Enum: jpeg, png, webp
quality integer
Option 11
kind string REQUIRED
Enum: markdown
only_main_content boolean
locale string
BCP 47 locale for `Accept-Language` and the browser.
timezone string
IANA time zone for the browser, such as `America/Los_Angeles`.
mobile boolean
Emulate a phone (viewport, user agent, touch).
viewport object
Browser viewport in CSS pixels.
width integer REQUIRED
height integer REQUIRED
block_ads boolean
Block ads, trackers and cookie banners.
block_resources string[]
Resource types the browser does not download. Defaults to images, media and fonts unless a screenshot is requested.
Array of: string
screenshot object
Options for the `screenshot` format.
full_page boolean
format string
Enum: jpeg, png, webp
quality integer
JPEG and WebP quality.
json object
Options for the `json` format. The node's model fills `schema` from the page.
schema object
A JSON Schema the result must match.
prompt string
Guidance for the model.

Responses

200 The scraped page.
application/json
object string REQUIRED
Enum: scrape
url string (uri) REQUIRED
The requested URL.
final_url string (uri) REQUIRED
The URL after redirects.
status_code integer REQUIRED
The HTTP status the site returned. A 404 page is still a successful scrape.
content_type string
tier string REQUIRED
What served the page.
Enum: cache, fetch, browser
cache object REQUIRED
state string REQUIRED
`hit` served from cache; `miss` fetched and stored; `bypass` fetched and not shared (custom headers, profile, actions).
Enum: hit, miss, bypass
age_s integer REQUIRED
Age of the served copy in seconds.
markdown string
html string
raw_html string
text string
links string[]
Array of: string
screenshot object
format string REQUIRED
Enum: jpeg, png, webp
width integer REQUIRED
height integer REQUIRED
data_base64 string (byte) REQUIRED
json unknown
Data matching `json.schema`, when the `json` format was requested.
metadata object
Metadata read from the page.
title string
description string
language string
canonical_url string (uri)
favicon string (uri)
og object
Open Graph properties, keyed without the `og:` prefix.
published_time string
modified_time string
robots string
The page's robots meta tag.
actions object[]
Results of `actions`, in order.
Array of:
kind string REQUIRED
Enum: navigate, click, hover, type, key, scroll, select, wait, evaluate, screenshot, markdown
status string REQUIRED
Enum: ok, failed, skipped
duration_ms number REQUIRED
url string
`navigate`: the URL after redirects.
status_code integer
`navigate`: the HTTP status.
value unknown
`evaluate`: the JSON result.
screenshot object
format string REQUIRED
Enum: jpeg, png, webp
width integer REQUIRED
height integer REQUIRED
data_base64 string (byte) REQUIRED
markdown string
`markdown`: the page as Markdown.
error object
An RFC 9457 problem. `code` is stable and listed in the errors catalog; `type` links to its entry. `detail` is human-readable and may change. Extension members appear only for the codes that document them; ignore members you do not recognize.
type string (uri) REQUIRED
`https://docs.beta.b3os.org/errors#<code>`.
title string REQUIRED
Short summary of the problem type.
status integer REQUIRED
The HTTP status code, repeated for convenience.
code string REQUIRED
Stable machine-readable error code from the errors catalog.
detail string REQUIRED
Explanation of this occurrence. Never contains secrets.
instance string
The request ID (`req_…`) of the request that failed. Include it when you contact support.
errors object[]
Validation problems, one per field (`invalid_request` and the other codes that list `errors`).
Array of:
pointer string REQUIRED
Where the problem is: a JSON pointer into the request body (`/url`, `/steps/0/uses`), or `#query/<name>`, `#path/<name>` or `#header/<name>` for parameters.
detail string REQUIRED
What is wrong there.
reason string
Finer-grained cause for codes that document one (`unauthorized`, `forbidden`, `egress_denied`, `upstream_unreachable`, …).
required_scope string
The scope that was missing (`forbidden` with `reason: scope`).
retry_after_s integer
Same value as the `Retry-After` header, for clients that only read the body.
limit object
The limit an error refers to. IDs and current values are listed by `GET /v1/limits`.
id string REQUIRED
Limit ID, such as `vision.scrape.rpm` or `sandboxes.concurrent`.
limit number REQUIRED
The limit's value.
unit string REQUIRED
What the limit counts, such as `requests`, `sandboxes`, `vcpu` or `MiB`.
window_s integer
Window length in seconds for rate limits and quotas.
current number
Current usage against the limit, when known.
current_status string
The object's current status, for state conflicts such as `sandbox_not_running`.
upstream_status integer
The HTTP status the upstream returned (`upstream_error`, `action_output_invalid`).
step string
The failed step, as a step ID (routines), an index (`act`) or a path such as `beats/2/do/0` (render).
pack_error object
The error a pack's error mapping chose (`upstream_error` from pack actions).
code string REQUIRED
message string REQUIRED
retryable boolean REQUIRED
warnings string[] REQUIRED
Non-fatal problems, such as a format that could not be produced.
Array of: string
timing object REQUIRED
Server-side phase durations in milliseconds with microsecond resolution: the same phases as the `Server-Timing` header, keyed `<phase>_ms`. `total_ms` is always present. Phase names are product-specific and documented per operation. Example values in this reference are illustrative, not measurements.
total_ms number REQUIRED
400 The request failed validation (`invalid_request`, `invalid_url`).
401 Missing, malformed, expired or revoked credentials (`unauthorized`).
402 The organization has no credits left, or the key reached its spend cap.
403 The egress guard refused the URL (`egress_denied`), or robots.txt disallows it (`robots_disallowed`).
404 No such profile in this project (`profile_not_found`).
413 The request body exceeds this operation's limit.
415 The Content-Type is not accepted by this operation.
422 The URL returned a content type vision cannot convert.
429 A rate, concurrency or quota limit applies (`rate_limited`, `concurrency_limited`, `quota_exceeded`).
500 Something failed inside B3OS (`internal`). Retry with backoff.
502 The page could not be fetched or loaded (`navigation_failed`, `upstream_unreachable`, `too_many_redirects`).
503 The node or the product is not available right now (`node_unavailable`, `product_unavailable`, `capacity_unavailable`).
504 The page did not finish within `timeout_ms` (`scrape_timeout`). 
curl -X POST 'https://beta-api.b3os.org/v1/vision/scrape' \  -H 'Authorization: Bearer YOUR_API_TOKEN' \  -H 'Content-Type: application/json' \  -d '{  "url": "https://example.com",  "formats": [    "markdown"  ],  "only_main_content": true,  "include_tags": [    "string"  ],  "exclude_tags": [    "string"  ],  "engine": "auto",  "wait_for_ms": 0,  "timeout_ms": 30000,  "max_age_ms": 172800000,  "store_in_cache": true,  "headers": {},  "profile": "string",  "actions": [    {      "kind": "navigate",      "url": "https://example.com",      "wait_until": "load",      "timeout_ms": 10000    }  ],  "locale": "en-US",  "timezone": "string",  "mobile": false,  "viewport": {    "width": 320,    "height": 240  },  "block_ads": true,  "block_resources": [    "image"  ],  "screenshot": {    "full_page": false,    "format": "jpeg",    "quality": 80  },  "json": {    "schema": {},    "prompt": "string"  }}'
const response = await fetch('https://beta-api.b3os.org/v1/vision/scrape', {  method: 'POST',  headers: {      "Authorization": "Bearer YOUR_API_TOKEN",      "Content-Type": "application/json"  },  body: JSON.stringify({    "url": "https://example.com",    "formats": [      "markdown"    ],    "only_main_content": true,    "include_tags": [      "string"    ],    "exclude_tags": [      "string"    ],    "engine": "auto",    "wait_for_ms": 0,    "timeout_ms": 30000,    "max_age_ms": 172800000,    "store_in_cache": true,    "headers": {},    "profile": "string",    "actions": [      {        "kind": "navigate",        "url": "https://example.com",        "wait_until": "load",        "timeout_ms": 10000      }    ],    "locale": "en-US",    "timezone": "string",    "mobile": false,    "viewport": {      "width": 320,      "height": 240    },    "block_ads": true,    "block_resources": [      "image"    ],    "screenshot": {      "full_page": false,      "format": "jpeg",      "quality": 80    },    "json": {      "schema": {},      "prompt": "string"    }  })});const data = await response.json();console.log(data);
import requestsheaders = {    'Authorization': 'Bearer YOUR_API_TOKEN'}response = requests.post('https://beta-api.b3os.org/v1/vision/scrape', headers=headers, json={  "url": "https://example.com",  "formats": [    "markdown"  ],  "only_main_content": true,  "include_tags": [    "string"  ],  "exclude_tags": [    "string"  ],  "engine": "auto",  "wait_for_ms": 0,  "timeout_ms": 30000,  "max_age_ms": 172800000,  "store_in_cache": true,  "headers": {},  "profile": "string",  "actions": [    {      "kind": "navigate",      "url": "https://example.com",      "wait_until": "load",      "timeout_ms": 10000    }  ],  "locale": "en-US",  "timezone": "string",  "mobile": false,  "viewport": {    "width": 320,    "height": 240  },  "block_ads": true,  "block_resources": [    "image"  ],  "screenshot": {    "full_page": false,    "format": "jpeg",    "quality": 80  },  "json": {    "schema": {},    "prompt": "string"  }})print(response.json())
200 Response 
{  "object": "scrape",  "url": "https://example.com/pricing",  "final_url": "https://example.com/pricing",  "status_code": 200,  "content_type": "text/html; charset=utf-8",  "tier": "fetch",  "cache": {    "state": "miss",    "age_s": 0  },  "markdown": "# Pricing\n\nStart free, then pay per page.\n",  "links": [    "https://example.com/",    "https://example.com/docs"  ],  "metadata": {    "title": "Pricing | Example",    "description": "Plans and prices for Example.",    "language": "en",    "canonical_url": "https://example.com/pricing"  },  "warnings": [],  "timing": {    "fetch_ms": 41.2,    "extract_ms": 3.9,    "total_ms": 45.8  }}
Previous Overview
Next Map a site's URLs
Ask a question... ⌘I