> ## Documentation Index > Fetch the complete documentation index at: https://docs.beta.b3os.org/llms.txt > Use this file to discover all available pages before exploring further. # Start a batch scrape (Firecrawl) Firecrawl's `POST /v1/batch/scrape`, served by a vision batch scrape (`batch_` job). The scrape options are at the top level of the body, as Firecrawl takes them. `webhook` deliveries use Firecrawl's payload and `X-Firecrawl-Signature`. Each page bills one `vision.page`. ` POST /compat/firecrawl/v1/batch/scrape ` Start a batch scrape (Firecrawl) Firecrawl's `POST /v1/batch/scrape`, served by a vision batch scrape (`batch_` job). The scrape options are at the top level of the body, as Firecrawl takes them. `webhook` deliveries use Firecrawl's payload and `X-Firecrawl-Signature`. Each page bills one `vision.page`. ## Servers - ` https://beta-api.b3os.org ` — API (API keys and CLI tokens) - ` https://beta.b3os.org ` — Dashboard (session cookie and CSRF header; same paths) ## Authentication The operation accepts these alternatives. Schemes within one alternative are required together. ### Alternative 1 - ` apiKey `: ` http ` - HTTP scheme: ` bearer ` (b3os_sk_ API key or b3os_pat_ token) - `Authorization: Bearer `. Two credential kinds are accepted: - **API keys** `b3os_sk_<32 base32>_`, created in the dashboard or with `POST /v1/keys`. A key belongs to one project, has scopes, an optional spend cap and expiry, and is shown only once. - **CLI tokens** `b3os_pat_<32 base32>_`, issued by the device login (`b3os login`). They act for one user in the organization and project chosen at approval. Keys are stored as SHA-256 hashes. The edge exchanges them for a 60-second node token before forwarding a request to a node, so nodes never see your key. ## Parameters ### ` x-idempotency-key ` (header, optional) Idempotency key as the Firecrawl SDKs send it; same semantics as `Idempotency-Key`. - ` value `: type ` string ` - minLength: ` 1 ` - maxLength: ` 255 ` ## Request body (required) Content type: ` application/json ` - ` value `: type ` object ` - Required fields: ` urls ` - ` urls `: type ` array ` - ` array item `: type ` string `; format ` uri ` - ` webhook `: type ` unknown ` — A webhook URL, or an object with the URL and options. - ` oneOf alternative 1 `: type ` string `; format ` uri ` - ` oneOf alternative 2 `: type ` object ` - Required fields: ` url ` - ` url `: type ` string `; format ` uri ` - ` headers `: type ` object ` - ` metadata `: type ` object ` - ` events `: type ` array ` - ` array item `: type ` string ` - enum: ` ["completed","page","failed","started"] ` - ` maxConcurrency `: type ` integer ` - ` ignoreInvalidURLs `: type ` boolean ` - default: ` false ` - ` appendToId `: type ` string ` — Not supported by B3OS: setting it returns `400` with code `unsupported_option`. - ` zeroDataRetention `: type ` boolean ` — Do not cache or store the result. - default: ` false ` - ` integration `: type ` string ` — Accepted and ignored (the Firecrawl SDKs send it). - ` origin `: type ` string ` — Accepted and ignored (the Firecrawl SDKs send it). - ` formats `: type ` array ` — `changeTracking` returns `unsupported_option`. - default: ` ["markdown"] ` - ` array item `: type ` string ` - enum: ` ["markdown","html","rawHtml","links","screenshot","screenshot@fullPage","json","changeTracking"] ` - ` onlyMainContent `: type ` boolean ` - default: ` true ` - ` includeTags `: type ` array ` - ` array item `: type ` string ` - ` excludeTags `: type ` array ` - ` array item `: type ` string ` - ` maxAge `: type ` integer ` — Accept a cached copy up to this old, in milliseconds. Default 0 (always fetch). - default: ` 0 ` - ` headers `: type ` object ` - ` waitFor `: type ` integer ` - default: ` 0 ` - ` mobile `: type ` boolean ` - default: ` false ` - ` skipTlsVerification `: type ` boolean ` - default: ` false ` - ` timeout `: type ` integer ` — Milliseconds; values above 60000 are treated as 60000. - default: ` 30000 ` - ` parsePDF `: type ` boolean ` — PDF parsing is not supported; PDFs return `content_type_unsupported`. - default: ` true ` - ` jsonOptions `: type ` object ` - ` schema `: type ` object ` - ` systemPrompt `: type ` string ` - ` prompt `: type ` string ` - ` checkPromptInjection `: type ` boolean ` — Not supported by B3OS: setting it returns `400` with code `unsupported_option`. - default: ` false ` - ` actions `: type ` array ` - ` array item `: type ` unknown ` — A browser action, run with vision browser actions. `pdf` is not supported. - ` oneOf alternative 1 `: type ` object ` - Title: Wait - Required fields: ` type ` - ` type `: type ` string ` - enum: ` ["wait"] ` - ` milliseconds `: type ` integer ` - minimum: ` 1 ` - ` selector `: type ` string ` - ` oneOf alternative 2 `: type ` object ` - Title: Screenshot - Required fields: ` type ` - ` type `: type ` string ` - enum: ` ["screenshot"] ` - ` fullPage `: type ` boolean ` - default: ` false ` - ` quality `: type ` integer ` - minimum: ` 1 ` - maximum: ` 100 ` - ` viewport `: type ` object ` - Required fields: ` width `, ` height ` - ` width `: type ` integer ` - ` height `: type ` integer ` - ` oneOf alternative 3 `: type ` object ` - Title: Click - Required fields: ` type `, ` selector ` - ` type `: type ` string ` - enum: ` ["click"] ` - ` selector `: type ` string ` - ` all `: type ` boolean ` - default: ` false ` - ` oneOf alternative 4 `: type ` object ` - Title: Write text - Required fields: ` type `, ` text ` - ` type `: type ` string ` - enum: ` ["write"] ` - ` text `: type ` string ` - ` oneOf alternative 5 `: type ` object ` - Title: Press a key - Required fields: ` type `, ` key ` - ` type `: type ` string ` - enum: ` ["press"] ` - ` key `: type ` string ` - ` oneOf alternative 6 `: type ` object ` - Title: Scroll - Required fields: ` type ` - ` type `: type ` string ` - enum: ` ["scroll"] ` - ` direction `: type ` string ` - enum: ` ["up","down"] ` - default: ` "down" ` - ` selector `: type ` string ` - ` oneOf alternative 7 `: type ` object ` - Title: Scrape - Required fields: ` type ` - ` type `: type ` string ` - enum: ` ["scrape"] ` - ` oneOf alternative 8 `: type ` object ` - Title: Execute JavaScript - Required fields: ` type `, ` script ` - ` type `: type ` string ` - enum: ` ["executeJavascript"] ` - ` script `: type ` string ` - ` oneOf alternative 9 `: type ` object ` — Not supported by B3OS: setting it returns `400` with code `unsupported_option`. - Title: Generate PDF - Required fields: ` type ` - ` type `: type ` string ` - enum: ` ["pdf"] ` - ` format `: type ` string ` - ` landscape `: type ` boolean ` - ` scale `: type ` number ` - ` location `: type ` object ` - ` country `: type ` string ` — ISO 3166-1 alpha-2 code. B3OS emulates the language and time zone; it has no per-country proxies. - default: ` "US" ` - pattern: ` "^[A-Z]{2}$" ` - ` languages `: type ` array ` - ` array item `: type ` string ` - ` removeBase64Images `: type ` boolean ` - default: ` true ` - ` blockAds `: type ` boolean ` - default: ` true ` - ` proxy `: type ` string ` — `enhanced` returns `unsupported_option`. - enum: ` ["basic","enhanced","auto"] ` - ` storeInCache `: type ` boolean ` - default: ` true ` - ` changeTrackingOptions `: type ` object ` — Not supported by B3OS: setting it returns `400` with code `unsupported_option`. - ` threatProtection `: type ` object ` — Not supported by B3OS: setting it returns `400` with code `unsupported_option`. Example: ```json { "urls": [ "https://example.com/blog/a", "https://example.com/blog/b" ], "formats": [ "markdown" ] } ``` ## Responses ### ` 200 ` — The batch was started. Content type: ` application/json ` - ` value `: type ` object ` - Required fields: ` success `, ` id `, ` url ` - ` success `: type ` boolean ` - enum: ` [true] ` - ` id `: type ` string ` - ` url `: type ` string `; format ` uri ` — The job's status URL. - ` invalidURLs `: type ` array `; nullable - ` array item `: type ` string ` Response header ` x-request-id ` — The request ID (`req_…`). It is the `instance` of any problem and the key for `GET /v1/requests/{request_id}`. - ` value `: type ` string ` - pattern: ` "^req_[0-9a-hjkmnp-tv-z]{26}$" ` Response header ` Server-Timing ` — Phase durations in milliseconds (W3C Server Timing); the handler's phases break the request down. For a response whose body size is known, `total` runs from reading the request head to writing the response head. Streaming responses (event streams, WebSockets, chunked bodies) carry no `total` here: they report their final phases in the terminal event (`job.succeeded`, `job.failed`, `job.canceled`) or the final job. The time from the first request byte to the last response byte is recorded in the node's metrics and request log (`GET /v1/requests/{request_id}`). The edge prepends `edge_auth` and `edge_upstream`. - ` value `: type ` string ` Example: ```json { "success": true, "id": "batch_01m5t3v176vrhf8a1mn570c5gd", "url": "https://beta-api.b3os.org/compat/firecrawl/v1/batch/scrape/batch_01m5t3v176vrhf8a1mn570c5gd", "invalidURLs": [] } ``` ### ` 400 ` — The body failed validation, uses an unsupported option, or a URL is invalid (`invalid_request`, `unsupported_option`, `invalid_url`). Content type: ` application/json ` - ` value `: type ` object ` — Firecrawl's error body. `code` and the status come from the B3OS code's `firecrawl` field in errors.yaml (the Firecrawl code where one exists, such as `SCRAPE_TIMEOUT`, otherwise the B3OS code); `x-request-id` carries the request ID. - Required fields: ` success `, ` error ` - ` success `: type ` boolean ` - enum: ` [false] ` - ` error `: type ` string ` — `: `, where the prefix is the B3OS code's `firecrawl.error` in errors.yaml and the detail is the B3OS problem's `detail`; the prefix alone when the problem has no detail. - ` code `: type ` string ` - ` details `: type ` unknown ` — Validation issues. A `400` `invalid_request` carries the B3OS problem's `errors[]` array here; unknown body fields are reported as Firecrawl reports them, as Zod-style `unrecognized_keys` issues (`{code, keys, path, message}`). Response header ` x-request-id ` — The request ID (`req_…`). It is the `instance` of any problem and the key for `GET /v1/requests/{request_id}`. - ` value `: type ` string ` - pattern: ` "^req_[0-9a-hjkmnp-tv-z]{26}$" ` Response header ` Server-Timing ` — Phase durations in milliseconds (W3C Server Timing); the handler's phases break the request down. For a response whose body size is known, `total` runs from reading the request head to writing the response head. Streaming responses (event streams, WebSockets, chunked bodies) carry no `total` here: they report their final phases in the terminal event (`job.succeeded`, `job.failed`, `job.canceled`) or the final job. The time from the first request byte to the last response byte is recorded in the node's metrics and request log (`GET /v1/requests/{request_id}`). The edge prepends `edge_auth` and `edge_upstream`. - ` value `: type ` string ` Example: ```json { "success": false, "error": "Bad Request", "code": "BAD_REQUEST", "details": [ { "code": "unrecognized_keys", "keys": [ "fromats" ], "path": [], "message": "Unrecognized key in body -- please review the v2 API documentation for request body changes" } ] } ``` ### ` 401 ` — Missing or invalid B3OS key (`unauthorized`). Firecrawl `fc-` keys are not accepted. Content type: ` application/json ` - ` value `: type ` object ` — Firecrawl's error body. `code` and the status come from the B3OS code's `firecrawl` field in errors.yaml (the Firecrawl code where one exists, such as `SCRAPE_TIMEOUT`, otherwise the B3OS code); `x-request-id` carries the request ID. - Required fields: ` success `, ` error ` - ` success `: type ` boolean ` - enum: ` [false] ` - ` error `: type ` string ` — `: `, where the prefix is the B3OS code's `firecrawl.error` in errors.yaml and the detail is the B3OS problem's `detail`; the prefix alone when the problem has no detail. - ` code `: type ` string ` - ` details `: type ` unknown ` — Validation issues. A `400` `invalid_request` carries the B3OS problem's `errors[]` array here; unknown body fields are reported as Firecrawl reports them, as Zod-style `unrecognized_keys` issues (`{code, keys, path, message}`). Response header ` x-request-id ` — The request ID (`req_…`). It is the `instance` of any problem and the key for `GET /v1/requests/{request_id}`. - ` value `: type ` string ` - pattern: ` "^req_[0-9a-hjkmnp-tv-z]{26}$" ` Response header ` Server-Timing ` — Phase durations in milliseconds (W3C Server Timing); the handler's phases break the request down. For a response whose body size is known, `total` runs from reading the request head to writing the response head. Streaming responses (event streams, WebSockets, chunked bodies) carry no `total` here: they report their final phases in the terminal event (`job.succeeded`, `job.failed`, `job.canceled`) or the final job. The time from the first request byte to the last response byte is recorded in the node's metrics and request log (`GET /v1/requests/{request_id}`). The edge prepends `edge_auth` and `edge_upstream`. - ` value `: type ` string ` Example: ```json { "success": false, "error": "Unauthorized: Invalid token", "code": "unauthorized" } ``` ### ` 402 ` — No credits left or the key's spend cap is reached (`credits_exhausted`, `spend_cap_exceeded`). Content type: ` application/json ` - ` value `: type ` object ` — Firecrawl's error body. `code` and the status come from the B3OS code's `firecrawl` field in errors.yaml (the Firecrawl code where one exists, such as `SCRAPE_TIMEOUT`, otherwise the B3OS code); `x-request-id` carries the request ID. - Required fields: ` success `, ` error ` - ` success `: type ` boolean ` - enum: ` [false] ` - ` error `: type ` string ` — `: `, where the prefix is the B3OS code's `firecrawl.error` in errors.yaml and the detail is the B3OS problem's `detail`; the prefix alone when the problem has no detail. - ` code `: type ` string ` - ` details `: type ` unknown ` — Validation issues. A `400` `invalid_request` carries the B3OS problem's `errors[]` array here; unknown body fields are reported as Firecrawl reports them, as Zod-style `unrecognized_keys` issues (`{code, keys, path, message}`). Response header ` x-request-id ` — The request ID (`req_…`). It is the `instance` of any problem and the key for `GET /v1/requests/{request_id}`. - ` value `: type ` string ` - pattern: ` "^req_[0-9a-hjkmnp-tv-z]{26}$" ` Response header ` Server-Timing ` — Phase durations in milliseconds (W3C Server Timing); the handler's phases break the request down. For a response whose body size is known, `total` runs from reading the request head to writing the response head. Streaming responses (event streams, WebSockets, chunked bodies) carry no `total` here: they report their final phases in the terminal event (`job.succeeded`, `job.failed`, `job.canceled`) or the final job. The time from the first request byte to the last response byte is recorded in the node's metrics and request log (`GET /v1/requests/{request_id}`). The edge prepends `edge_auth` and `edge_upstream`. - ` value `: type ` string ` Example: ```json { "success": false, "error": "Insufficient credits: Organization org_01m5t5hthd29mm5wpza5ndd6mm has no credits left", "code": "credits_exhausted" } ``` ### ` 403 ` — The URL is refused by the egress guard or robots.txt, or the key lacks a scope (`egress_denied`, `robots_disallowed`, `forbidden`). Content type: ` application/json ` - ` value `: type ` object ` — Firecrawl's error body. `code` and the status come from the B3OS code's `firecrawl` field in errors.yaml (the Firecrawl code where one exists, such as `SCRAPE_TIMEOUT`, otherwise the B3OS code); `x-request-id` carries the request ID. - Required fields: ` success `, ` error ` - ` success `: type ` boolean ` - enum: ` [false] ` - ` error `: type ` string ` — `: `, where the prefix is the B3OS code's `firecrawl.error` in errors.yaml and the detail is the B3OS problem's `detail`; the prefix alone when the problem has no detail. - ` code `: type ` string ` - ` details `: type ` unknown ` — Validation issues. A `400` `invalid_request` carries the B3OS problem's `errors[]` array here; unknown body fields are reported as Firecrawl reports them, as Zod-style `unrecognized_keys` issues (`{code, keys, path, message}`). Response header ` x-request-id ` — The request ID (`req_…`). It is the `instance` of any problem and the key for `GET /v1/requests/{request_id}`. - ` value `: type ` string ` - pattern: ` "^req_[0-9a-hjkmnp-tv-z]{26}$" ` Response header ` Server-Timing ` — Phase durations in milliseconds (W3C Server Timing); the handler's phases break the request down. For a response whose body size is known, `total` runs from reading the request head to writing the response head. Streaming responses (event streams, WebSockets, chunked bodies) carry no `total` here: they report their final phases in the terminal event (`job.succeeded`, `job.failed`, `job.canceled`) or the final job. The time from the first request byte to the last response byte is recorded in the node's metrics and request log (`GET /v1/requests/{request_id}`). The edge prepends `edge_auth` and `edge_upstream`. - ` value `: type ` string ` Example: ```json { "success": false, "error": "Destination not allowed: http://10.0.0.5/ resolves to a private address", "code": "egress_denied" } ``` ### ` 429 ` — A rate or concurrency limit applies (`rate_limited`, `concurrency_limited`). Content type: ` application/json ` - ` value `: type ` object ` — Firecrawl's error body. `code` and the status come from the B3OS code's `firecrawl` field in errors.yaml (the Firecrawl code where one exists, such as `SCRAPE_TIMEOUT`, otherwise the B3OS code); `x-request-id` carries the request ID. - Required fields: ` success `, ` error ` - ` success `: type ` boolean ` - enum: ` [false] ` - ` error `: type ` string ` — `: `, where the prefix is the B3OS code's `firecrawl.error` in errors.yaml and the detail is the B3OS problem's `detail`; the prefix alone when the problem has no detail. - ` code `: type ` string ` - ` details `: type ` unknown ` — Validation issues. A `400` `invalid_request` carries the B3OS problem's `errors[]` array here; unknown body fields are reported as Firecrawl reports them, as Zod-style `unrecognized_keys` issues (`{code, keys, path, message}`). Response header ` x-request-id ` — The request ID (`req_…`). It is the `instance` of any problem and the key for `GET /v1/requests/{request_id}`. - ` value `: type ` string ` - pattern: ` "^req_[0-9a-hjkmnp-tv-z]{26}$" ` Response header ` Server-Timing ` — Phase durations in milliseconds (W3C Server Timing); the handler's phases break the request down. For a response whose body size is known, `total` runs from reading the request head to writing the response head. Streaming responses (event streams, WebSockets, chunked bodies) carry no `total` here: they report their final phases in the terminal event (`job.succeeded`, `job.failed`, `job.canceled`) or the final job. The time from the first request byte to the last response byte is recorded in the node's metrics and request log (`GET /v1/requests/{request_id}`). The edge prepends `edge_auth` and `edge_upstream`. - ` value `: type ` string ` Response header ` Retry-After ` — Seconds to wait before retrying or polling again. - ` value `: type ` integer ` - minimum: ` 0 ` Example: ```json { "success": false, "error": "Rate limit exceeded", "code": "rate_limited" } ``` ### ` 500 ` — Something failed inside B3OS (`internal`). Content type: ` application/json ` - ` value `: type ` object ` — Firecrawl's error body. `code` and the status come from the B3OS code's `firecrawl` field in errors.yaml (the Firecrawl code where one exists, such as `SCRAPE_TIMEOUT`, otherwise the B3OS code); `x-request-id` carries the request ID. - Required fields: ` success `, ` error ` - ` success `: type ` boolean ` - enum: ` [false] ` - ` error `: type ` string ` — `: `, where the prefix is the B3OS code's `firecrawl.error` in errors.yaml and the detail is the B3OS problem's `detail`; the prefix alone when the problem has no detail. - ` code `: type ` string ` - ` details `: type ` unknown ` — Validation issues. A `400` `invalid_request` carries the B3OS problem's `errors[]` array here; unknown body fields are reported as Firecrawl reports them, as Zod-style `unrecognized_keys` issues (`{code, keys, path, message}`). Response header ` x-request-id ` — The request ID (`req_…`). It is the `instance` of any problem and the key for `GET /v1/requests/{request_id}`. - ` value `: type ` string ` - pattern: ` "^req_[0-9a-hjkmnp-tv-z]{26}$" ` Response header ` Server-Timing ` — Phase durations in milliseconds (W3C Server Timing); the handler's phases break the request down. For a response whose body size is known, `total` runs from reading the request head to writing the response head. Streaming responses (event streams, WebSockets, chunked bodies) carry no `total` here: they report their final phases in the terminal event (`job.succeeded`, `job.failed`, `job.canceled`) or the final job. The time from the first request byte to the last response byte is recorded in the node's metrics and request log (`GET /v1/requests/{request_id}`). The edge prepends `edge_auth` and `edge_upstream`. - ` value `: type ` string ` Example: ```json { "success": false, "error": "Internal Server Error", "code": "internal" } ``` ### ` 503 ` — The node or vision is unavailable (`node_unavailable`, `product_unavailable`). Content type: ` application/json ` - ` value `: type ` object ` — Firecrawl's error body. `code` and the status come from the B3OS code's `firecrawl` field in errors.yaml (the Firecrawl code where one exists, such as `SCRAPE_TIMEOUT`, otherwise the B3OS code); `x-request-id` carries the request ID. - Required fields: ` success `, ` error ` - ` success `: type ` boolean ` - enum: ` [false] ` - ` error `: type ` string ` — `: `, where the prefix is the B3OS code's `firecrawl.error` in errors.yaml and the detail is the B3OS problem's `detail`; the prefix alone when the problem has no detail. - ` code `: type ` string ` - ` details `: type ` unknown ` — Validation issues. A `400` `invalid_request` carries the B3OS problem's `errors[]` array here; unknown body fields are reported as Firecrawl reports them, as Zod-style `unrecognized_keys` issues (`{code, keys, path, message}`). Response header ` x-request-id ` — The request ID (`req_…`). It is the `instance` of any problem and the key for `GET /v1/requests/{request_id}`. - ` value `: type ` string ` - pattern: ` "^req_[0-9a-hjkmnp-tv-z]{26}$" ` Response header ` Server-Timing ` — Phase durations in milliseconds (W3C Server Timing); the handler's phases break the request down. For a response whose body size is known, `total` runs from reading the request head to writing the response head. Streaming responses (event streams, WebSockets, chunked bodies) carry no `total` here: they report their final phases in the terminal event (`job.succeeded`, `job.failed`, `job.canceled`) or the final job. The time from the first request byte to the last response byte is recorded in the node's metrics and request log (`GET /v1/requests/{request_id}`). The edge prepends `edge_auth` and `edge_upstream`. - ` value `: type ` string ` Response header ` Retry-After ` — Seconds to wait before retrying or polling again. - ` value `: type ` integer ` - minimum: ` 0 ` Example: ```json { "success": false, "error": "Service Unavailable", "code": "node_unavailable" } ``` ## Request examples ### cURL ```curl curl -X POST 'https://beta-api.b3os.org/compat/firecrawl/v1/batch/scrape' \ -H 'Authorization: Bearer YOUR_API_TOKEN' \ -H 'Content-Type: application/json' \ -d '{ "urls": [ "https://example.com/blog/a", "https://example.com/blog/b" ], "formats": [ "markdown" ] }' ``` ### JavaScript ```javascript const response = await fetch('https://beta-api.b3os.org/compat/firecrawl/v1/batch/scrape', { method: 'POST', headers: { "Authorization": "Bearer YOUR_API_TOKEN", "Content-Type": "application/json" }, body: JSON.stringify({ "urls": [ "https://example.com/blog/a", "https://example.com/blog/b" ], "formats": [ "markdown" ] }) }); const data = await response.json(); console.log(data); ``` ### Python ```python import requests headers = { 'Authorization': 'Bearer YOUR_API_TOKEN' } response = requests.post('https://beta-api.b3os.org/compat/firecrawl/v1/batch/scrape', headers=headers, json={ "urls": [ "https://example.com/blog/a", "https://example.com/blog/b" ], "formats": [ "markdown" ] }) print(response.json()) ```