> ## Documentation Index > Fetch the complete documentation index at: https://docs.beta.b3os.org/llms.txt > Use this file to discover all available pages before exploring further. # Get a batch scrape (Firecrawl) Status and documents of a batch scrape, 10 MB per page of results. `next` is an absolute URL under `https://beta-api.b3os.org/compat/firecrawl` while more documents exist; the SDKs follow it. Opened as a WebSocket instead (firecrawl-js `watcher()`), the same URL streams Firecrawl's watcher protocol: one `catchup` message with this body, a `document` message per page as it finishes, then a close with code 1000 and reason `{"type":"done"}`. ` GET /compat/firecrawl/v2/batch/scrape/{id} ` Get a batch scrape (Firecrawl) Status and documents of a batch scrape, 10 MB per page of results. `next` is an absolute URL under `https://beta-api.b3os.org/compat/firecrawl` while more documents exist; the SDKs follow it. Opened as a WebSocket instead (firecrawl-js `watcher()`), the same URL streams Firecrawl's watcher protocol: one `catchup` message with this body, a `document` message per page as it finishes, then a close with code 1000 and reason `{"type":"done"}`. ## Servers - ` https://beta-api.b3os.org ` — API (API keys and CLI tokens) - ` https://beta.b3os.org ` — Dashboard (session cookie and CSRF header; same paths) ## Authentication The operation accepts these alternatives. Schemes within one alternative are required together. ### Alternative 1 - ` apiKey `: ` http ` - HTTP scheme: ` bearer ` (b3os_sk_ API key or b3os_pat_ token) - `Authorization: Bearer `. Two credential kinds are accepted: - **API keys** `b3os_sk_<32 base32>_`, created in the dashboard or with `POST /v1/keys`. A key belongs to one project, has scopes, an optional spend cap and expiry, and is shown only once. - **CLI tokens** `b3os_pat_<32 base32>_`, issued by the device login (`b3os login`). They act for one user in the organization and project chosen at approval. Keys are stored as SHA-256 hashes. The edge exchanges them for a 60-second node token before forwarding a request to a node, so nodes never see your key. ## Parameters ### ` id ` (path, required) The job ID B3OS returned (a `crawl_`, `batch_` or `job_` ID; Firecrawl clients treat it as opaque). - ` value `: type ` string ` - maxLength: ` 64 ` ### ` skip ` (query, optional) Documents to skip; the `next` URL sets it for you. - ` value `: type ` integer ` - minimum: ` 0 ` ## Responses ### ` 200 ` — The batch status. Content type: ` application/json ` - ` value `: type ` object ` — Status of a crawl or batch scrape. B3OS always includes `success: true` (the SDKs require it). - Required fields: ` success `, ` status `, ` total `, ` completed `, ` creditsUsed `, ` expiresAt `, ` next `, ` data ` - ` success `: type ` boolean ` - enum: ` [true] ` - ` status `: type ` string ` - enum: ` ["scraping","completed","failed","cancelled"] ` - ` total `: type ` integer ` - ` completed `: type ` integer ` - ` creditsUsed `: type ` integer ` - ` expiresAt `: type ` string `; format ` date-time ` - ` createdAt `: type ` string `; format ` date-time ` - ` completedAt `: type ` string `; format ` date-time ` - ` duration `: type ` number ` — Seconds since start (or until completion). - ` next `: type ` string `; nullable — Absolute URL of the next page of documents, under `/compat/firecrawl`. - ` data `: type ` array ` - ` array item `: type ` object ` - ` markdown `: type ` string ` - ` summary `: type ` string `; nullable - ` html `: type ` string `; nullable - ` rawHtml `: type ` string `; nullable - ` links `: type ` array ` - ` array item `: type ` string ` - ` images `: type ` array ` - ` array item `: type ` string ` - ` screenshot `: type ` string `; nullable — A `data:image/...;base64,` URL (B3OS returns the image inline). - ` json `: type ` unknown ` — Data from the `json` format. - ` actions `: type ` object `; nullable - ` screenshots `: type ` array ` - ` array item `: type ` string ` - ` scrapes `: type ` array ` - ` array item `: type ` object ` - ` url `: type ` string ` - ` html `: type ` string ` - ` javascriptReturns `: type ` array ` - ` array item `: type ` object ` - ` type `: type ` string ` - ` value `: type ` unknown ` - ` pdfs `: type ` array ` - ` array item `: type ` string ` - ` metadata `: type ` object ` — Page metadata. Other `` tags appear under their own names. - ` title `: type ` unknown ` - ` oneOf alternative 1 `: type ` string ` - ` oneOf alternative 2 `: type ` array ` - ` array item `: type ` string ` - ` description `: type ` unknown ` - ` oneOf alternative 1 `: type ` string ` - ` oneOf alternative 2 `: type ` array ` - ` array item `: type ` string ` - ` language `: type ` unknown ` - ` oneOf alternative 1 `: type ` string `; nullable - ` oneOf alternative 2 `: type ` array ` - ` array item `: type ` string ` - ` sourceURL `: type ` string `; format ` uri ` — The requested URL. - ` url `: type ` string `; format ` uri ` — The final URL after redirects. - ` keywords `: type ` unknown ` - ` oneOf alternative 1 `: type ` string ` - ` oneOf alternative 2 `: type ` array ` - ` array item `: type ` string ` - ` ogLocaleAlternate `: type ` array ` - ` array item `: type ` string ` - ` statusCode `: type ` integer ` - ` contentType `: type ` string ` - ` error `: type ` string `; nullable - ` scrapeId `: type ` string ` — The B3OS request ID of the scrape. - ` cacheState `: type ` string ` - enum: ` ["hit","miss"] ` - ` cachedAt `: type ` string `; format ` date-time ` - ` creditsUsed `: type ` integer ` - ` proxyUsed `: type ` string ` - enum: ` ["basic","stealth"] ` - ` concurrencyLimited `: type ` boolean ` - ` concurrencyQueueDurationMs `: type ` number ` - ` b3os `: type ` object ` — B3OS extras (Firecrawl clients ignore them). - ` tier `: type ` string ` - enum: ` ["cache","fetch","browser"] ` - ` request_id `: type ` string ` - ` timing `: type ` object ` — Server-side phase durations in milliseconds with microsecond resolution: the same phases as the `Server-Timing` header, keyed `_ms`. `total_ms` is always present. Phase names are product-specific and documented per operation. Example values in this reference are illustrative, not measurements. - example: ` {"claim_ms":1.1,"navigate_ms":31,"extract_ms":4.2,"total_ms":36.6} ` - Required fields: ` total_ms ` - ` total_ms `: type ` number ` - minimum: ` 0 ` - ` warning `: type ` string `; nullable Response header ` x-request-id ` — The request ID (`req_…`). It is the `instance` of any problem and the key for `GET /v1/requests/{request_id}`. - ` value `: type ` string ` - pattern: ` "^req_[0-9a-hjkmnp-tv-z]{26}$" ` Response header ` Server-Timing ` — Phase durations in milliseconds (W3C Server Timing); the handler's phases break the request down. For a response whose body size is known, `total` runs from reading the request head to writing the response head. Streaming responses (event streams, WebSockets, chunked bodies) carry no `total` here: they report their final phases in the terminal event (`job.succeeded`, `job.failed`, `job.canceled`) or the final job. The time from the first request byte to the last response byte is recorded in the node's metrics and request log (`GET /v1/requests/{request_id}`). The edge prepends `edge_auth` and `edge_upstream`. - ` value `: type ` string ` Example: ```json { "success": true, "status": "scraping", "total": 37, "completed": 12, "creditsUsed": 12, "expiresAt": "2026-11-01T18:10:41.020Z", "createdAt": "2026-10-25T18:10:02.114Z", "duration": 18.3, "next": "https://beta-api.b3os.org/compat/firecrawl/v2/batch/scrape/batch_01m5t3v176vrhf8a1mn570c5gd?skip=10", "data": [ { "markdown": "# Pricing\n\nStart free, then pay per page.", "metadata": { "title": "Pricing | Example", "description": "Plans and prices for Example.", "language": "en", "sourceURL": "https://example.com/pricing", "url": "https://example.com/pricing", "statusCode": 200, "contentType": "text/html; charset=utf-8", "scrapeId": "req_01m5t4yg394wegr3j4y01qb363", "cacheState": "miss", "creditsUsed": 1, "proxyUsed": "basic", "b3os": { "tier": "fetch", "request_id": "req_01m5t4yg394wegr3j4y01qb363", "timing": { "fetch_ms": 41.2, "extract_ms": 3.9, "total_ms": 45.8 } } } } ] } ``` ### ` 400 ` — The body failed validation, uses an unsupported option, or a URL is invalid (`invalid_request`, `unsupported_option`, `invalid_url`). Content type: ` application/json ` - ` value `: type ` object ` — Firecrawl's error body. `code` and the status come from the B3OS code's `firecrawl` field in errors.yaml (the Firecrawl code where one exists, such as `SCRAPE_TIMEOUT`, otherwise the B3OS code); `x-request-id` carries the request ID. - Required fields: ` success `, ` error ` - ` success `: type ` boolean ` - enum: ` [false] ` - ` error `: type ` string ` — `: `, where the prefix is the B3OS code's `firecrawl.error` in errors.yaml and the detail is the B3OS problem's `detail`; the prefix alone when the problem has no detail. - ` code `: type ` string ` - ` details `: type ` unknown ` — Validation issues. A `400` `invalid_request` carries the B3OS problem's `errors[]` array here; unknown body fields are reported as Firecrawl reports them, as Zod-style `unrecognized_keys` issues (`{code, keys, path, message}`). Response header ` x-request-id ` — The request ID (`req_…`). It is the `instance` of any problem and the key for `GET /v1/requests/{request_id}`. - ` value `: type ` string ` - pattern: ` "^req_[0-9a-hjkmnp-tv-z]{26}$" ` Response header ` Server-Timing ` — Phase durations in milliseconds (W3C Server Timing); the handler's phases break the request down. For a response whose body size is known, `total` runs from reading the request head to writing the response head. Streaming responses (event streams, WebSockets, chunked bodies) carry no `total` here: they report their final phases in the terminal event (`job.succeeded`, `job.failed`, `job.canceled`) or the final job. The time from the first request byte to the last response byte is recorded in the node's metrics and request log (`GET /v1/requests/{request_id}`). The edge prepends `edge_auth` and `edge_upstream`. - ` value `: type ` string ` Example: ```json { "success": false, "error": "Bad Request", "code": "BAD_REQUEST", "details": [ { "code": "unrecognized_keys", "keys": [ "fromats" ], "path": [], "message": "Unrecognized key in body -- please review the v2 API documentation for request body changes" } ] } ``` ### ` 401 ` — Missing or invalid B3OS key (`unauthorized`). Firecrawl `fc-` keys are not accepted. Content type: ` application/json ` - ` value `: type ` object ` — Firecrawl's error body. `code` and the status come from the B3OS code's `firecrawl` field in errors.yaml (the Firecrawl code where one exists, such as `SCRAPE_TIMEOUT`, otherwise the B3OS code); `x-request-id` carries the request ID. - Required fields: ` success `, ` error ` - ` success `: type ` boolean ` - enum: ` [false] ` - ` error `: type ` string ` — `: `, where the prefix is the B3OS code's `firecrawl.error` in errors.yaml and the detail is the B3OS problem's `detail`; the prefix alone when the problem has no detail. - ` code `: type ` string ` - ` details `: type ` unknown ` — Validation issues. A `400` `invalid_request` carries the B3OS problem's `errors[]` array here; unknown body fields are reported as Firecrawl reports them, as Zod-style `unrecognized_keys` issues (`{code, keys, path, message}`). Response header ` x-request-id ` — The request ID (`req_…`). It is the `instance` of any problem and the key for `GET /v1/requests/{request_id}`. - ` value `: type ` string ` - pattern: ` "^req_[0-9a-hjkmnp-tv-z]{26}$" ` Response header ` Server-Timing ` — Phase durations in milliseconds (W3C Server Timing); the handler's phases break the request down. For a response whose body size is known, `total` runs from reading the request head to writing the response head. Streaming responses (event streams, WebSockets, chunked bodies) carry no `total` here: they report their final phases in the terminal event (`job.succeeded`, `job.failed`, `job.canceled`) or the final job. The time from the first request byte to the last response byte is recorded in the node's metrics and request log (`GET /v1/requests/{request_id}`). The edge prepends `edge_auth` and `edge_upstream`. - ` value `: type ` string ` Example: ```json { "success": false, "error": "Unauthorized: Invalid token", "code": "unauthorized" } ``` ### ` 404 ` — No such job in this project (`crawl_not_found`, `batch_not_found`, `job_not_found`). Content type: ` application/json ` - ` value `: type ` object ` — Firecrawl's error body. `code` and the status come from the B3OS code's `firecrawl` field in errors.yaml (the Firecrawl code where one exists, such as `SCRAPE_TIMEOUT`, otherwise the B3OS code); `x-request-id` carries the request ID. - Required fields: ` success `, ` error ` - ` success `: type ` boolean ` - enum: ` [false] ` - ` error `: type ` string ` — `: `, where the prefix is the B3OS code's `firecrawl.error` in errors.yaml and the detail is the B3OS problem's `detail`; the prefix alone when the problem has no detail. - ` code `: type ` string ` - ` details `: type ` unknown ` — Validation issues. A `400` `invalid_request` carries the B3OS problem's `errors[]` array here; unknown body fields are reported as Firecrawl reports them, as Zod-style `unrecognized_keys` issues (`{code, keys, path, message}`). Response header ` x-request-id ` — The request ID (`req_…`). It is the `instance` of any problem and the key for `GET /v1/requests/{request_id}`. - ` value `: type ` string ` - pattern: ` "^req_[0-9a-hjkmnp-tv-z]{26}$" ` Response header ` Server-Timing ` — Phase durations in milliseconds (W3C Server Timing); the handler's phases break the request down. For a response whose body size is known, `total` runs from reading the request head to writing the response head. Streaming responses (event streams, WebSockets, chunked bodies) carry no `total` here: they report their final phases in the terminal event (`job.succeeded`, `job.failed`, `job.canceled`) or the final job. The time from the first request byte to the last response byte is recorded in the node's metrics and request log (`GET /v1/requests/{request_id}`). The edge prepends `edge_auth` and `edge_upstream`. - ` value `: type ` string ` Example: ```json { "success": false, "error": "Crawl job not found: No crawl crawl_01m5t4gh4qhgs70zwajezbe5p0 in this project", "code": "crawl_not_found" } ``` ### ` 429 ` — A rate or concurrency limit applies (`rate_limited`, `concurrency_limited`). Content type: ` application/json ` - ` value `: type ` object ` — Firecrawl's error body. `code` and the status come from the B3OS code's `firecrawl` field in errors.yaml (the Firecrawl code where one exists, such as `SCRAPE_TIMEOUT`, otherwise the B3OS code); `x-request-id` carries the request ID. - Required fields: ` success `, ` error ` - ` success `: type ` boolean ` - enum: ` [false] ` - ` error `: type ` string ` — `: `, where the prefix is the B3OS code's `firecrawl.error` in errors.yaml and the detail is the B3OS problem's `detail`; the prefix alone when the problem has no detail. - ` code `: type ` string ` - ` details `: type ` unknown ` — Validation issues. A `400` `invalid_request` carries the B3OS problem's `errors[]` array here; unknown body fields are reported as Firecrawl reports them, as Zod-style `unrecognized_keys` issues (`{code, keys, path, message}`). Response header ` x-request-id ` — The request ID (`req_…`). It is the `instance` of any problem and the key for `GET /v1/requests/{request_id}`. - ` value `: type ` string ` - pattern: ` "^req_[0-9a-hjkmnp-tv-z]{26}$" ` Response header ` Server-Timing ` — Phase durations in milliseconds (W3C Server Timing); the handler's phases break the request down. For a response whose body size is known, `total` runs from reading the request head to writing the response head. Streaming responses (event streams, WebSockets, chunked bodies) carry no `total` here: they report their final phases in the terminal event (`job.succeeded`, `job.failed`, `job.canceled`) or the final job. The time from the first request byte to the last response byte is recorded in the node's metrics and request log (`GET /v1/requests/{request_id}`). The edge prepends `edge_auth` and `edge_upstream`. - ` value `: type ` string ` Response header ` Retry-After ` — Seconds to wait before retrying or polling again. - ` value `: type ` integer ` - minimum: ` 0 ` Example: ```json { "success": false, "error": "Rate limit exceeded", "code": "rate_limited" } ``` ### ` 500 ` — Something failed inside B3OS (`internal`). Content type: ` application/json ` - ` value `: type ` object ` — Firecrawl's error body. `code` and the status come from the B3OS code's `firecrawl` field in errors.yaml (the Firecrawl code where one exists, such as `SCRAPE_TIMEOUT`, otherwise the B3OS code); `x-request-id` carries the request ID. - Required fields: ` success `, ` error ` - ` success `: type ` boolean ` - enum: ` [false] ` - ` error `: type ` string ` — `: `, where the prefix is the B3OS code's `firecrawl.error` in errors.yaml and the detail is the B3OS problem's `detail`; the prefix alone when the problem has no detail. - ` code `: type ` string ` - ` details `: type ` unknown ` — Validation issues. A `400` `invalid_request` carries the B3OS problem's `errors[]` array here; unknown body fields are reported as Firecrawl reports them, as Zod-style `unrecognized_keys` issues (`{code, keys, path, message}`). Response header ` x-request-id ` — The request ID (`req_…`). It is the `instance` of any problem and the key for `GET /v1/requests/{request_id}`. - ` value `: type ` string ` - pattern: ` "^req_[0-9a-hjkmnp-tv-z]{26}$" ` Response header ` Server-Timing ` — Phase durations in milliseconds (W3C Server Timing); the handler's phases break the request down. For a response whose body size is known, `total` runs from reading the request head to writing the response head. Streaming responses (event streams, WebSockets, chunked bodies) carry no `total` here: they report their final phases in the terminal event (`job.succeeded`, `job.failed`, `job.canceled`) or the final job. The time from the first request byte to the last response byte is recorded in the node's metrics and request log (`GET /v1/requests/{request_id}`). The edge prepends `edge_auth` and `edge_upstream`. - ` value `: type ` string ` Example: ```json { "success": false, "error": "Internal Server Error", "code": "internal" } ``` ### ` 503 ` — The node or vision is unavailable (`node_unavailable`, `product_unavailable`). Content type: ` application/json ` - ` value `: type ` object ` — Firecrawl's error body. `code` and the status come from the B3OS code's `firecrawl` field in errors.yaml (the Firecrawl code where one exists, such as `SCRAPE_TIMEOUT`, otherwise the B3OS code); `x-request-id` carries the request ID. - Required fields: ` success `, ` error ` - ` success `: type ` boolean ` - enum: ` [false] ` - ` error `: type ` string ` — `: `, where the prefix is the B3OS code's `firecrawl.error` in errors.yaml and the detail is the B3OS problem's `detail`; the prefix alone when the problem has no detail. - ` code `: type ` string ` - ` details `: type ` unknown ` — Validation issues. A `400` `invalid_request` carries the B3OS problem's `errors[]` array here; unknown body fields are reported as Firecrawl reports them, as Zod-style `unrecognized_keys` issues (`{code, keys, path, message}`). Response header ` x-request-id ` — The request ID (`req_…`). It is the `instance` of any problem and the key for `GET /v1/requests/{request_id}`. - ` value `: type ` string ` - pattern: ` "^req_[0-9a-hjkmnp-tv-z]{26}$" ` Response header ` Server-Timing ` — Phase durations in milliseconds (W3C Server Timing); the handler's phases break the request down. For a response whose body size is known, `total` runs from reading the request head to writing the response head. Streaming responses (event streams, WebSockets, chunked bodies) carry no `total` here: they report their final phases in the terminal event (`job.succeeded`, `job.failed`, `job.canceled`) or the final job. The time from the first request byte to the last response byte is recorded in the node's metrics and request log (`GET /v1/requests/{request_id}`). The edge prepends `edge_auth` and `edge_upstream`. - ` value `: type ` string ` Response header ` Retry-After ` — Seconds to wait before retrying or polling again. - ` value `: type ` integer ` - minimum: ` 0 ` Example: ```json { "success": false, "error": "Service Unavailable", "code": "node_unavailable" } ``` ## Request examples ### cURL ```curl curl -X GET 'https://beta-api.b3os.org/compat/firecrawl/v2/batch/scrape/string' \ -H 'Authorization: Bearer YOUR_API_TOKEN' ``` ### JavaScript ```javascript const response = await fetch('https://beta-api.b3os.org/compat/firecrawl/v2/batch/scrape/string', { method: 'GET', headers: { "Authorization": "Bearer YOUR_API_TOKEN" } }); const data = await response.json(); console.log(data); ``` ### Python ```python import requests headers = { 'Authorization': 'Bearer YOUR_API_TOKEN' } response = requests.get('https://beta-api.b3os.org/compat/firecrawl/v2/batch/scrape/string', headers=headers) print(response.json()) ```