Crawl a site

Starts a job that discovers pages from `url` (links and sitemaps), breadth first, and scrapes up to `limit` pages (at most 500). robots.txt is honored and requests to one host are spaced politely. One crawl runs at a time per organization. Each page bills one `vision.page`.

POST /v1/vision/crawls
Starts a job that discovers pages from `url` (links and sitemaps), breadth first, and scrapes up to `limit` pages (at most 500). robots.txt is honored and requests to one host are spaced politely. One crawl runs at a time per organization. Each page bills one `vision.page`.

Authentication

Bearer Token (b3os_sk_ API key or b3os_pat_ token) API Key (cookie: __Host-b3os_session)

Query Parameters

wait integer optional query
Seconds to wait for the job to finish before responding, from 0 to 60. If it finishes in time the response is `200` with the finished job; otherwise `202` with the job as it is, a `Location` to poll and `Retry-After`. `0` (the default) returns immediately.

Header Parameters

Idempotency-Key string optional header
A unique value (for example a UUID) that makes this request safe to retry. Scoped to your project for 24 hours: a retry with the same key, method, path and body returns the stored response with `Idempotent-Replayed: true`, and only to the same API key. Reusing the key for a different request returns `409 idempotency_key_conflict`.

Request Body required

application/json
url string (uri) REQUIRED
Where to start.
limit integer
Most pages to scrape.
max_depth integer
Most link hops from `url` (sitemap URLs count as depth 0).
include_paths string[]
Regular expressions (RE2 syntax) a URL path must match to be crawled.
Array of: string
exclude_paths string[]
Regular expressions (RE2 syntax); matching paths are skipped.
Array of: string
sitemap string
Enum: include, skip, only
crawl_entire_domain boolean
Follow links to parent and sibling paths, not only below `url`.
allow_subdomains boolean
allow_external_links boolean
Scrape pages on other sites that the crawl links to (one hop; their links are not followed).
ignore_query_parameters boolean
delay_ms integer
Minimum gap between requests to the same host.
max_concurrency integer
scrape object
Scrape options applied to every page.
formats string[]
Array of: string
only_main_content boolean
Drop headers, navigation, footers and sidebars.
include_tags string[]
CSS selectors to keep; everything else is dropped.
Array of: string
exclude_tags string[]
CSS selectors to drop.
Array of: string
engine string
Which tier may serve the page. `auto` (default): cache, then HTTP fetch, then a browser when the page needs JavaScript. `fetch`: never start a browser. `browser`: always render in Chrome (needed for `actions`, `screenshot` and profiles; implied by them).
Enum: auto, fetch, browser
wait_for_ms integer
Extra wait after the page is ready, before capture.
timeout_ms integer
max_age_ms integer
Accept a cached copy up to this old (default 2 days). `0` always fetches.
store_in_cache boolean
Store the result in the cache. Requests with headers, a profile or actions are never shared across organizations.
headers object
Extra request headers. A request with custom headers bypasses the shared cache. In a browser, a `Cookie` header becomes cookies for the page's host (path `/`), so it reaches the origin as on an HTTP fetch and is not sent to the other hosts the page loads from.
profile string
A profile ID or name to load cookies and storage from. Implies `engine: browser`. The page renders in a browser of its own (never one shared with other organizations), which counts toward `vision.sessions.concurrent` while it runs.
actions (object | object | object | object | object | object | object | object | object | object | object)[]
Browser actions to run before capture. Implies `engine: browser`.
Array of:
One of:
Option 1
kind string REQUIRED
Enum: navigate
url string (uri) REQUIRED
wait_until string
Enum: load, domcontentloaded, networkidle
timeout_ms integer
Option 2
kind string REQUIRED
Enum: click
target unknown | unknown | unknown REQUIRED
How an action finds its element (the same rules as render timelines, `timeline.schema.json#/$defs/target`): candidates match `selector` (CSS), `text` (visible text, case-insensitive substring unless `exact`) or `role` plus accessible `name`, limited to `within`; the best-scored candidate wins, or the `nth` match when set. An element that stays outside the viewport after scrolling it into view fails with `target_not_found`.
Any of:
Option 1unknown
Option 2unknown
Option 3unknown
button string
Enum: left, right, middle
count integer
modifiers string[]
Modifier keys held during the action.
Array of: string
timeout_ms integer
How long to wait for the target to appear.
Option 3
kind string REQUIRED
Enum: hover
target unknown | unknown | unknown REQUIRED
How an action finds its element (the same rules as render timelines, `timeline.schema.json#/$defs/target`): candidates match `selector` (CSS), `text` (visible text, case-insensitive substring unless `exact`) or `role` plus accessible `name`, limited to `within`; the best-scored candidate wins, or the `nth` match when set. An element that stays outside the viewport after scrolling it into view fails with `target_not_found`.
Any of:
Option 1unknown
Option 2unknown
Option 3unknown
timeout_ms integer
Option 4
kind string REQUIRED
Enum: type
target unknown | unknown | unknown
How an action finds its element (the same rules as render timelines, `timeline.schema.json#/$defs/target`): candidates match `selector` (CSS), `text` (visible text, case-insensitive substring unless `exact`) or `role` plus accessible `name`, limited to `within`; the best-scored candidate wins, or the `nth` match when set. An element that stays outside the viewport after scrolling it into view fails with `target_not_found`.
Any of:
Option 1unknown
Option 2unknown
Option 3unknown
text string REQUIRED
clear boolean
Clear the field before typing.
interval_ms integer
Delay between characters.
timeout_ms integer
Option 5
kind string REQUIRED
Enum: key
key string REQUIRED
`KeyboardEvent.key`, such as `Enter`, `Tab` or `a`.
modifiers string[]
Modifier keys held during the action.
Array of: string
Option 6
kind string REQUIRED
Enum: scroll
target unknown | unknown | unknown
How an action finds its element (the same rules as render timelines, `timeline.schema.json#/$defs/target`): candidates match `selector` (CSS), `text` (visible text, case-insensitive substring unless `exact`) or `role` plus accessible `name`, limited to `within`; the best-scored candidate wins, or the `nth` match when set. An element that stays outside the viewport after scrolling it into view fails with `target_not_found`.
Any of:
Option 1unknown
Option 2unknown
Option 3unknown
to object
A position in CSS pixels.
x number REQUIRED
y number REQUIRED
by object
A position in CSS pixels.
x number REQUIRED
y number REQUIRED
Option 7
kind string REQUIRED
Enum: select
target unknown | unknown | unknown REQUIRED
How an action finds its element (the same rules as render timelines, `timeline.schema.json#/$defs/target`): candidates match `selector` (CSS), `text` (visible text, case-insensitive substring unless `exact`) or `role` plus accessible `name`, limited to `within`; the best-scored candidate wins, or the `nth` match when set. An element that stays outside the viewport after scrolling it into view fails with `target_not_found`.
Any of:
Option 1unknown
Option 2unknown
Option 3unknown
value string REQUIRED
The option value to select.
Option 8
kind string REQUIRED
Enum: wait
ms integer
for object
Exactly one condition to wait for (the same as a timeline `wait` action's `for`).
selector string
An element matching this CSS selector is visible.
text string
This text is visible.
url string
The page URL contains this substring.
network_idle boolean
The network has been quiet.
Enum: true
timeout_ms integer
Option 9
kind string REQUIRED
Enum: evaluate
expression string REQUIRED
JavaScript evaluated in the page; a returned promise is awaited and the JSON-serializable result is returned.
Option 10
kind string REQUIRED
Enum: screenshot
target unknown | unknown | unknown
How an action finds its element (the same rules as render timelines, `timeline.schema.json#/$defs/target`): candidates match `selector` (CSS), `text` (visible text, case-insensitive substring unless `exact`) or `role` plus accessible `name`, limited to `within`; the best-scored candidate wins, or the `nth` match when set. An element that stays outside the viewport after scrolling it into view fails with `target_not_found`.
Any of:
Option 1unknown
Option 2unknown
Option 3unknown
full_page boolean
format string
Enum: jpeg, png, webp
quality integer
Option 11
kind string REQUIRED
Enum: markdown
only_main_content boolean
locale string
BCP 47 locale for `Accept-Language` and the browser.
timezone string
IANA time zone for the browser, such as `America/Los_Angeles`.
mobile boolean
Emulate a phone (viewport, user agent, touch).
viewport object
Browser viewport in CSS pixels.
width integer REQUIRED
height integer REQUIRED
block_ads boolean
Block ads, trackers and cookie banners.
block_resources string[]
Resource types the browser does not download. Defaults to images, media and fonts unless a screenshot is requested.
Array of: string
screenshot object
Options for the `screenshot` format.
full_page boolean
format string
Enum: jpeg, png, webp
quality integer
JPEG and WebP quality.
json object
Options for the `json` format. The node's model fills `schema` from the page.
schema object
A JSON Schema the result must match.
prompt string
Guidance for the model.
webhook object
Where to deliver this job's events. Deliveries are `POST`s of a `WebhookEvent` signed with `B3OS-Signature: t=<unix seconds>,v1=<hex HMAC-SHA256 of "<t>.<raw body>">` using the endpoint's secret (`POST /v1/webhooks/secret`). Failed deliveries are retried with backoff for 24 hours. Delivery goes through the egress guard, so private and loopback addresses are refused.
url string (uri) REQUIRED
An `https://` URL.
events string[]
Event types to deliver. Defaults to the terminal events (`job.succeeded`, `job.failed`, `job.canceled`); `["*"]` delivers every event.
Array of: string
metadata object
Up to 16 string key-value pairs you attach to an object for your own use. Keys are 1 to 40 characters of `A-Z a-z 0-9 _ . : -`; values are at most 500 characters. B3OS never interprets them.

Responses

200 The crawl finished within `wait` seconds, or the event stream when `Accept` is `text/event-stream`.
application/json
id string REQUIRED
object string REQUIRED
Enum: crawl
status string REQUIRED
`queued` → `running` → one of the terminal statuses `succeeded`, `failed` or `canceled`. A job never leaves a terminal status.
Enum: queued, running, succeeded, failed, canceled
created_at string (date-time) REQUIRED
updated_at string (date-time) REQUIRED
started_at string (date-time) | null
finished_at string (date-time) | null
progress object
How far a job has got, in the job's own unit (pages, items, steps).
completed integer REQUIRED
Units finished, successfully or not.
failed integer
Units that failed (included in `completed`).
total integer | null
Units expected, when known.
unit string REQUIRED
What is counted, such as `pages`, `items` or `steps`.
result object
pages_succeeded integer REQUIRED
pages_failed integer REQUIRED
robots_blocked string[] REQUIRED
URLs skipped because robots.txt disallows them.
Array of: string
error object
An RFC 9457 problem. `code` is stable and listed in the errors catalog; `type` links to its entry. `detail` is human-readable and may change. Extension members appear only for the codes that document them; ignore members you do not recognize.
type string (uri) REQUIRED
`https://docs.beta.b3os.org/errors#<code>`.
title string REQUIRED
Short summary of the problem type.
status integer REQUIRED
The HTTP status code, repeated for convenience.
code string REQUIRED
Stable machine-readable error code from the errors catalog.
detail string REQUIRED
Explanation of this occurrence. Never contains secrets.
instance string
The request ID (`req_…`) of the request that failed. Include it when you contact support.
errors object[]
Validation problems, one per field (`invalid_request` and the other codes that list `errors`).
Array of:
pointer string REQUIRED
Where the problem is: a JSON pointer into the request body (`/url`, `/steps/0/uses`), or `#query/<name>`, `#path/<name>` or `#header/<name>` for parameters.
detail string REQUIRED
What is wrong there.
reason string
Finer-grained cause for codes that document one (`unauthorized`, `forbidden`, `egress_denied`, `upstream_unreachable`, …).
required_scope string
The scope that was missing (`forbidden` with `reason: scope`).
retry_after_s integer
Same value as the `Retry-After` header, for clients that only read the body.
limit object
The limit an error refers to. IDs and current values are listed by `GET /v1/limits`.
id string REQUIRED
Limit ID, such as `vision.scrape.rpm` or `sandboxes.concurrent`.
limit number REQUIRED
The limit's value.
unit string REQUIRED
What the limit counts, such as `requests`, `sandboxes`, `vcpu` or `MiB`.
window_s integer
Window length in seconds for rate limits and quotas.
current number
Current usage against the limit, when known.
current_status string
The object's current status, for state conflicts such as `sandbox_not_running`.
upstream_status integer
The HTTP status the upstream returned (`upstream_error`, `action_output_invalid`).
step string
The failed step, as a step ID (routines), an index (`act`) or a path such as `beats/2/do/0` (render).
pack_error object
The error a pack's error mapping chose (`upstream_error` from pack actions).
code string REQUIRED
message string REQUIRED
retryable boolean REQUIRED
timing object
Server-side phase durations in milliseconds with microsecond resolution: the same phases as the `Server-Timing` header, keyed `<phase>_ms`. `total_ms` is always present. Phase names are product-specific and documented per operation. Example values in this reference are illustrative, not measurements.
total_ms number REQUIRED
expires_at string (date-time) | null
When the job's results are deleted (retention); `null` while the job is not terminal.
metadata object
Up to 16 string key-value pairs you attach to an object for your own use. Keys are 1 to 40 characters of `A-Z a-z 0-9 _ . : -`; values are at most 500 characters. B3OS never interprets them.
webhook object
Where to deliver this job's events. Deliveries are `POST`s of a `WebhookEvent` signed with `B3OS-Signature: t=<unix seconds>,v1=<hex HMAC-SHA256 of "<t>.<raw body>">` using the endpoint's secret (`POST /v1/webhooks/secret`). Failed deliveries are retried with backoff for 24 hours. Delivery goes through the egress guard, so private and loopback addresses are refused.
url string (uri) REQUIRED
An `https://` URL.
events string[]
Event types to deliver. Defaults to the terminal events (`job.succeeded`, `job.failed`, `job.canceled`); `["*"]` delivers every event.
Array of: string
url string (uri) REQUIRED
The start URL.
202 The crawl was accepted and is still running.
400 The request failed validation (`invalid_request`, `invalid_url`).
401 Missing, malformed, expired or revoked credentials (`unauthorized`).
402 The organization has no credits left, or the key reached its spend cap.
403 The egress guard refused the URL (`egress_denied`), or robots.txt disallows it (`robots_disallowed`).
409 The object's state does not allow this operation, or the idempotency key conflicts.
413 The request body exceeds this operation's limit.
415 The Content-Type is not accepted by this operation.
422 A parameter exceeds a plan limit (`limit_exceeded`).
429 A rate, concurrency or quota limit applies (`rate_limited`, `concurrency_limited`, `quota_exceeded`).
500 Something failed inside B3OS (`internal`). Retry with backoff.
503 The node or the product is not available right now (`node_unavailable`, `product_unavailable`, `capacity_unavailable`). 
curl -X POST 'https://beta-api.b3os.org/v1/vision/crawls' \  -H 'Authorization: Bearer YOUR_API_TOKEN' \  -H 'Content-Type: application/json' \  -d '{  "url": "https://docs.example.com",  "limit": 200,  "include_paths": [    "^/guides/"  ],  "scrape": {    "formats": [      "markdown"    ]  }}'
const response = await fetch('https://beta-api.b3os.org/v1/vision/crawls', {  method: 'POST',  headers: {      "Authorization": "Bearer YOUR_API_TOKEN",      "Content-Type": "application/json"  },  body: JSON.stringify({    "url": "https://docs.example.com",    "limit": 200,    "include_paths": [      "^/guides/"    ],    "scrape": {      "formats": [        "markdown"      ]    }  })});const data = await response.json();console.log(data);
import requestsheaders = {    'Authorization': 'Bearer YOUR_API_TOKEN'}response = requests.post('https://beta-api.b3os.org/v1/vision/crawls', headers=headers, json={  "url": "https://docs.example.com",  "limit": 200,  "include_paths": [    "^/guides/"  ],  "scrape": {    "formats": [      "markdown"    ]  }})print(response.json())
200 Response 
{  "id": "crawl_01m5t4gh4qhgs70zwajezbe5p0",  "object": "crawl",  "status": "succeeded",  "url": "https://docs.example.com",  "created_at": "2026-10-25T18:10:02.114Z",  "updated_at": "2026-10-25T18:10:41.020Z",  "started_at": "2026-10-25T18:10:02.120Z",  "finished_at": "2026-10-25T18:10:41.020Z",  "progress": {    "completed": 37,    "failed": 1,    "total": 37,    "unit": "pages"  },  "result": {    "pages_succeeded": 36,    "pages_failed": 1,    "robots_blocked": [      "https://docs.example.com/private/"    ]  },  "timing": {    "discover_ms": 812.5,    "total_ms": 38900.3  },  "expires_at": "2026-11-01T18:10:41.020Z",  "metadata": {}}
Previous List crawls
Next Get a crawl
Ask a question... ⌘I