> ## Documentation Index > Fetch the complete documentation index at: https://docs.beta.b3os.org/llms.txt > Use this file to discover all available pages before exploring further. # Map a site (Firecrawl) Firecrawl's `POST /v1/map`, served by vision map. `links` are URL strings, as in Firecrawl v1. `limit` above 5000 is treated as 5000. Bills one `vision.page`. ` POST /compat/firecrawl/v1/map ` Map a site (Firecrawl) Firecrawl's `POST /v1/map`, served by vision map. `links` are URL strings, as in Firecrawl v1. `limit` above 5000 is treated as 5000. Bills one `vision.page`. ## Servers - ` https://beta-api.b3os.org ` — API (API keys and CLI tokens) - ` https://beta.b3os.org ` — Dashboard (session cookie and CSRF header; same paths) ## Authentication The operation accepts these alternatives. Schemes within one alternative are required together. ### Alternative 1 - ` apiKey `: ` http ` - HTTP scheme: ` bearer ` (b3os_sk_ API key or b3os_pat_ token) - `Authorization: Bearer `. Two credential kinds are accepted: - **API keys** `b3os_sk_<32 base32>_`, created in the dashboard or with `POST /v1/keys`. A key belongs to one project, has scopes, an optional spend cap and expiry, and is shown only once. - **CLI tokens** `b3os_pat_<32 base32>_`, issued by the device login (`b3os login`). They act for one user in the organization and project chosen at approval. Keys are stored as SHA-256 hashes. The edge exchanges them for a 60-second node token before forwarding a request to a node, so nodes never see your key. ## Request body (required) Content type: ` application/json ` - ` value `: type ` object ` - Required fields: ` url ` - ` url `: type ` string `; format ` uri ` - ` search `: type ` string ` - ` ignoreSitemap `: type ` boolean ` - default: ` true ` - ` sitemapOnly `: type ` boolean ` - default: ` false ` - ` includeSubdomains `: type ` boolean ` - default: ` true ` - ` limit `: type ` integer ` — Above 5000 it is treated as 5000. - default: ` 5000 ` - maximum: ` 30000 ` - ` timeout `: type ` integer ` - ` location `: type ` object ` - ` country `: type ` string ` — ISO 3166-1 alpha-2 code. B3OS emulates the language and time zone; it has no per-country proxies. - default: ` "US" ` - pattern: ` "^[A-Z]{2}$" ` - ` languages `: type ` array ` - ` array item `: type ` string ` - ` threatProtection `: type ` object ` — Not supported by B3OS: setting it returns `400` with code `unsupported_option`. - ` integration `: type ` string ` — Accepted and ignored (the Firecrawl SDKs send it). - ` origin `: type ` string ` — Accepted and ignored (the Firecrawl SDKs send it). Example: ```json { "url": "https://docs.example.com", "search": "authentication", "limit": 100 } ``` ## Responses ### ` 200 ` — The URLs found. Content type: ` application/json ` - ` value `: type ` object ` - Required fields: ` success `, ` links ` - ` success `: type ` boolean ` - enum: ` [true] ` - ` links `: type ` array ` - ` array item `: type ` string ` Response header ` x-request-id ` — The request ID (`req_…`). It is the `instance` of any problem and the key for `GET /v1/requests/{request_id}`. - ` value `: type ` string ` - pattern: ` "^req_[0-9a-hjkmnp-tv-z]{26}$" ` Response header ` Server-Timing ` — Phase durations in milliseconds (W3C Server Timing); the handler's phases break the request down. For a response whose body size is known, `total` runs from reading the request head to writing the response head. Streaming responses (event streams, WebSockets, chunked bodies) carry no `total` here: they report their final phases in the terminal event (`job.succeeded`, `job.failed`, `job.canceled`) or the final job. The time from the first request byte to the last response byte is recorded in the node's metrics and request log (`GET /v1/requests/{request_id}`). The edge prepends `edge_auth` and `edge_upstream`. - ` value `: type ` string ` Example: ```json { "success": true, "links": [ "https://docs.example.com/auth", "https://docs.example.com/auth/keys" ] } ``` ### ` 400 ` — The body failed validation, uses an unsupported option, or a URL is invalid (`invalid_request`, `unsupported_option`, `invalid_url`). Content type: ` application/json ` - ` value `: type ` object ` — Firecrawl's error body. `code` and the status come from the B3OS code's `firecrawl` field in errors.yaml (the Firecrawl code where one exists, such as `SCRAPE_TIMEOUT`, otherwise the B3OS code); `x-request-id` carries the request ID. - Required fields: ` success `, ` error ` - ` success `: type ` boolean ` - enum: ` [false] ` - ` error `: type ` string ` — `: `, where the prefix is the B3OS code's `firecrawl.error` in errors.yaml and the detail is the B3OS problem's `detail`; the prefix alone when the problem has no detail. - ` code `: type ` string ` - ` details `: type ` unknown ` — Validation issues. A `400` `invalid_request` carries the B3OS problem's `errors[]` array here; unknown body fields are reported as Firecrawl reports them, as Zod-style `unrecognized_keys` issues (`{code, keys, path, message}`). Response header ` x-request-id ` — The request ID (`req_…`). It is the `instance` of any problem and the key for `GET /v1/requests/{request_id}`. - ` value `: type ` string ` - pattern: ` "^req_[0-9a-hjkmnp-tv-z]{26}$" ` Response header ` Server-Timing ` — Phase durations in milliseconds (W3C Server Timing); the handler's phases break the request down. For a response whose body size is known, `total` runs from reading the request head to writing the response head. Streaming responses (event streams, WebSockets, chunked bodies) carry no `total` here: they report their final phases in the terminal event (`job.succeeded`, `job.failed`, `job.canceled`) or the final job. The time from the first request byte to the last response byte is recorded in the node's metrics and request log (`GET /v1/requests/{request_id}`). The edge prepends `edge_auth` and `edge_upstream`. - ` value `: type ` string ` Example: ```json { "success": false, "error": "Bad Request", "code": "BAD_REQUEST", "details": [ { "code": "unrecognized_keys", "keys": [ "fromats" ], "path": [], "message": "Unrecognized key in body -- please review the v2 API documentation for request body changes" } ] } ``` ### ` 401 ` — Missing or invalid B3OS key (`unauthorized`). Firecrawl `fc-` keys are not accepted. Content type: ` application/json ` - ` value `: type ` object ` — Firecrawl's error body. `code` and the status come from the B3OS code's `firecrawl` field in errors.yaml (the Firecrawl code where one exists, such as `SCRAPE_TIMEOUT`, otherwise the B3OS code); `x-request-id` carries the request ID. - Required fields: ` success `, ` error ` - ` success `: type ` boolean ` - enum: ` [false] ` - ` error `: type ` string ` — `: `, where the prefix is the B3OS code's `firecrawl.error` in errors.yaml and the detail is the B3OS problem's `detail`; the prefix alone when the problem has no detail. - ` code `: type ` string ` - ` details `: type ` unknown ` — Validation issues. A `400` `invalid_request` carries the B3OS problem's `errors[]` array here; unknown body fields are reported as Firecrawl reports them, as Zod-style `unrecognized_keys` issues (`{code, keys, path, message}`). Response header ` x-request-id ` — The request ID (`req_…`). It is the `instance` of any problem and the key for `GET /v1/requests/{request_id}`. - ` value `: type ` string ` - pattern: ` "^req_[0-9a-hjkmnp-tv-z]{26}$" ` Response header ` Server-Timing ` — Phase durations in milliseconds (W3C Server Timing); the handler's phases break the request down. For a response whose body size is known, `total` runs from reading the request head to writing the response head. Streaming responses (event streams, WebSockets, chunked bodies) carry no `total` here: they report their final phases in the terminal event (`job.succeeded`, `job.failed`, `job.canceled`) or the final job. The time from the first request byte to the last response byte is recorded in the node's metrics and request log (`GET /v1/requests/{request_id}`). The edge prepends `edge_auth` and `edge_upstream`. - ` value `: type ` string ` Example: ```json { "success": false, "error": "Unauthorized: Invalid token", "code": "unauthorized" } ``` ### ` 402 ` — No credits left or the key's spend cap is reached (`credits_exhausted`, `spend_cap_exceeded`). Content type: ` application/json ` - ` value `: type ` object ` — Firecrawl's error body. `code` and the status come from the B3OS code's `firecrawl` field in errors.yaml (the Firecrawl code where one exists, such as `SCRAPE_TIMEOUT`, otherwise the B3OS code); `x-request-id` carries the request ID. - Required fields: ` success `, ` error ` - ` success `: type ` boolean ` - enum: ` [false] ` - ` error `: type ` string ` — `: `, where the prefix is the B3OS code's `firecrawl.error` in errors.yaml and the detail is the B3OS problem's `detail`; the prefix alone when the problem has no detail. - ` code `: type ` string ` - ` details `: type ` unknown ` — Validation issues. A `400` `invalid_request` carries the B3OS problem's `errors[]` array here; unknown body fields are reported as Firecrawl reports them, as Zod-style `unrecognized_keys` issues (`{code, keys, path, message}`). Response header ` x-request-id ` — The request ID (`req_…`). It is the `instance` of any problem and the key for `GET /v1/requests/{request_id}`. - ` value `: type ` string ` - pattern: ` "^req_[0-9a-hjkmnp-tv-z]{26}$" ` Response header ` Server-Timing ` — Phase durations in milliseconds (W3C Server Timing); the handler's phases break the request down. For a response whose body size is known, `total` runs from reading the request head to writing the response head. Streaming responses (event streams, WebSockets, chunked bodies) carry no `total` here: they report their final phases in the terminal event (`job.succeeded`, `job.failed`, `job.canceled`) or the final job. The time from the first request byte to the last response byte is recorded in the node's metrics and request log (`GET /v1/requests/{request_id}`). The edge prepends `edge_auth` and `edge_upstream`. - ` value `: type ` string ` Example: ```json { "success": false, "error": "Insufficient credits: Organization org_01m5t5hthd29mm5wpza5ndd6mm has no credits left", "code": "credits_exhausted" } ``` ### ` 403 ` — The URL is refused by the egress guard or robots.txt, or the key lacks a scope (`egress_denied`, `robots_disallowed`, `forbidden`). Content type: ` application/json ` - ` value `: type ` object ` — Firecrawl's error body. `code` and the status come from the B3OS code's `firecrawl` field in errors.yaml (the Firecrawl code where one exists, such as `SCRAPE_TIMEOUT`, otherwise the B3OS code); `x-request-id` carries the request ID. - Required fields: ` success `, ` error ` - ` success `: type ` boolean ` - enum: ` [false] ` - ` error `: type ` string ` — `: `, where the prefix is the B3OS code's `firecrawl.error` in errors.yaml and the detail is the B3OS problem's `detail`; the prefix alone when the problem has no detail. - ` code `: type ` string ` - ` details `: type ` unknown ` — Validation issues. A `400` `invalid_request` carries the B3OS problem's `errors[]` array here; unknown body fields are reported as Firecrawl reports them, as Zod-style `unrecognized_keys` issues (`{code, keys, path, message}`). Response header ` x-request-id ` — The request ID (`req_…`). It is the `instance` of any problem and the key for `GET /v1/requests/{request_id}`. - ` value `: type ` string ` - pattern: ` "^req_[0-9a-hjkmnp-tv-z]{26}$" ` Response header ` Server-Timing ` — Phase durations in milliseconds (W3C Server Timing); the handler's phases break the request down. For a response whose body size is known, `total` runs from reading the request head to writing the response head. Streaming responses (event streams, WebSockets, chunked bodies) carry no `total` here: they report their final phases in the terminal event (`job.succeeded`, `job.failed`, `job.canceled`) or the final job. The time from the first request byte to the last response byte is recorded in the node's metrics and request log (`GET /v1/requests/{request_id}`). The edge prepends `edge_auth` and `edge_upstream`. - ` value `: type ` string ` Example: ```json { "success": false, "error": "Destination not allowed: http://10.0.0.5/ resolves to a private address", "code": "egress_denied" } ``` ### ` 429 ` — A rate or concurrency limit applies (`rate_limited`, `concurrency_limited`). Content type: ` application/json ` - ` value `: type ` object ` — Firecrawl's error body. `code` and the status come from the B3OS code's `firecrawl` field in errors.yaml (the Firecrawl code where one exists, such as `SCRAPE_TIMEOUT`, otherwise the B3OS code); `x-request-id` carries the request ID. - Required fields: ` success `, ` error ` - ` success `: type ` boolean ` - enum: ` [false] ` - ` error `: type ` string ` — `: `, where the prefix is the B3OS code's `firecrawl.error` in errors.yaml and the detail is the B3OS problem's `detail`; the prefix alone when the problem has no detail. - ` code `: type ` string ` - ` details `: type ` unknown ` — Validation issues. A `400` `invalid_request` carries the B3OS problem's `errors[]` array here; unknown body fields are reported as Firecrawl reports them, as Zod-style `unrecognized_keys` issues (`{code, keys, path, message}`). Response header ` x-request-id ` — The request ID (`req_…`). It is the `instance` of any problem and the key for `GET /v1/requests/{request_id}`. - ` value `: type ` string ` - pattern: ` "^req_[0-9a-hjkmnp-tv-z]{26}$" ` Response header ` Server-Timing ` — Phase durations in milliseconds (W3C Server Timing); the handler's phases break the request down. For a response whose body size is known, `total` runs from reading the request head to writing the response head. Streaming responses (event streams, WebSockets, chunked bodies) carry no `total` here: they report their final phases in the terminal event (`job.succeeded`, `job.failed`, `job.canceled`) or the final job. The time from the first request byte to the last response byte is recorded in the node's metrics and request log (`GET /v1/requests/{request_id}`). The edge prepends `edge_auth` and `edge_upstream`. - ` value `: type ` string ` Response header ` Retry-After ` — Seconds to wait before retrying or polling again. - ` value `: type ` integer ` - minimum: ` 0 ` Example: ```json { "success": false, "error": "Rate limit exceeded", "code": "rate_limited" } ``` ### ` 500 ` — Something failed inside B3OS (`internal`). Content type: ` application/json ` - ` value `: type ` object ` — Firecrawl's error body. `code` and the status come from the B3OS code's `firecrawl` field in errors.yaml (the Firecrawl code where one exists, such as `SCRAPE_TIMEOUT`, otherwise the B3OS code); `x-request-id` carries the request ID. - Required fields: ` success `, ` error ` - ` success `: type ` boolean ` - enum: ` [false] ` - ` error `: type ` string ` — `: `, where the prefix is the B3OS code's `firecrawl.error` in errors.yaml and the detail is the B3OS problem's `detail`; the prefix alone when the problem has no detail. - ` code `: type ` string ` - ` details `: type ` unknown ` — Validation issues. A `400` `invalid_request` carries the B3OS problem's `errors[]` array here; unknown body fields are reported as Firecrawl reports them, as Zod-style `unrecognized_keys` issues (`{code, keys, path, message}`). Response header ` x-request-id ` — The request ID (`req_…`). It is the `instance` of any problem and the key for `GET /v1/requests/{request_id}`. - ` value `: type ` string ` - pattern: ` "^req_[0-9a-hjkmnp-tv-z]{26}$" ` Response header ` Server-Timing ` — Phase durations in milliseconds (W3C Server Timing); the handler's phases break the request down. For a response whose body size is known, `total` runs from reading the request head to writing the response head. Streaming responses (event streams, WebSockets, chunked bodies) carry no `total` here: they report their final phases in the terminal event (`job.succeeded`, `job.failed`, `job.canceled`) or the final job. The time from the first request byte to the last response byte is recorded in the node's metrics and request log (`GET /v1/requests/{request_id}`). The edge prepends `edge_auth` and `edge_upstream`. - ` value `: type ` string ` Example: ```json { "success": false, "error": "Internal Server Error", "code": "internal" } ``` ### ` 502 ` — The site could not be reached or loaded (`upstream_unreachable`, `navigation_failed`). Content type: ` application/json ` - ` value `: type ` object ` — Firecrawl's error body. `code` and the status come from the B3OS code's `firecrawl` field in errors.yaml (the Firecrawl code where one exists, such as `SCRAPE_TIMEOUT`, otherwise the B3OS code); `x-request-id` carries the request ID. - Required fields: ` success `, ` error ` - ` success `: type ` boolean ` - enum: ` [false] ` - ` error `: type ` string ` — `: `, where the prefix is the B3OS code's `firecrawl.error` in errors.yaml and the detail is the B3OS problem's `detail`; the prefix alone when the problem has no detail. - ` code `: type ` string ` - ` details `: type ` unknown ` — Validation issues. A `400` `invalid_request` carries the B3OS problem's `errors[]` array here; unknown body fields are reported as Firecrawl reports them, as Zod-style `unrecognized_keys` issues (`{code, keys, path, message}`). Response header ` x-request-id ` — The request ID (`req_…`). It is the `instance` of any problem and the key for `GET /v1/requests/{request_id}`. - ` value `: type ` string ` - pattern: ` "^req_[0-9a-hjkmnp-tv-z]{26}$" ` Response header ` Server-Timing ` — Phase durations in milliseconds (W3C Server Timing); the handler's phases break the request down. For a response whose body size is known, `total` runs from reading the request head to writing the response head. Streaming responses (event streams, WebSockets, chunked bodies) carry no `total` here: they report their final phases in the terminal event (`job.succeeded`, `job.failed`, `job.canceled`) or the final job. The time from the first request byte to the last response byte is recorded in the node's metrics and request log (`GET /v1/requests/{request_id}`). The edge prepends `edge_auth` and `edge_upstream`. - ` value `: type ` string ` Example: ```json { "success": false, "error": "Bad Gateway: DNS lookup for shop.example failed", "code": "upstream_unreachable" } ``` ### ` 503 ` — The node or vision is unavailable (`node_unavailable`, `product_unavailable`). Content type: ` application/json ` - ` value `: type ` object ` — Firecrawl's error body. `code` and the status come from the B3OS code's `firecrawl` field in errors.yaml (the Firecrawl code where one exists, such as `SCRAPE_TIMEOUT`, otherwise the B3OS code); `x-request-id` carries the request ID. - Required fields: ` success `, ` error ` - ` success `: type ` boolean ` - enum: ` [false] ` - ` error `: type ` string ` — `: `, where the prefix is the B3OS code's `firecrawl.error` in errors.yaml and the detail is the B3OS problem's `detail`; the prefix alone when the problem has no detail. - ` code `: type ` string ` - ` details `: type ` unknown ` — Validation issues. A `400` `invalid_request` carries the B3OS problem's `errors[]` array here; unknown body fields are reported as Firecrawl reports them, as Zod-style `unrecognized_keys` issues (`{code, keys, path, message}`). Response header ` x-request-id ` — The request ID (`req_…`). It is the `instance` of any problem and the key for `GET /v1/requests/{request_id}`. - ` value `: type ` string ` - pattern: ` "^req_[0-9a-hjkmnp-tv-z]{26}$" ` Response header ` Server-Timing ` — Phase durations in milliseconds (W3C Server Timing); the handler's phases break the request down. For a response whose body size is known, `total` runs from reading the request head to writing the response head. Streaming responses (event streams, WebSockets, chunked bodies) carry no `total` here: they report their final phases in the terminal event (`job.succeeded`, `job.failed`, `job.canceled`) or the final job. The time from the first request byte to the last response byte is recorded in the node's metrics and request log (`GET /v1/requests/{request_id}`). The edge prepends `edge_auth` and `edge_upstream`. - ` value `: type ` string ` Response header ` Retry-After ` — Seconds to wait before retrying or polling again. - ` value `: type ` integer ` - minimum: ` 0 ` Example: ```json { "success": false, "error": "Service Unavailable", "code": "node_unavailable" } ``` ## Request examples ### cURL ```curl curl -X POST 'https://beta-api.b3os.org/compat/firecrawl/v1/map' \ -H 'Authorization: Bearer YOUR_API_TOKEN' \ -H 'Content-Type: application/json' \ -d '{ "url": "https://docs.example.com", "search": "authentication", "limit": 100 }' ``` ### JavaScript ```javascript const response = await fetch('https://beta-api.b3os.org/compat/firecrawl/v1/map', { method: 'POST', headers: { "Authorization": "Bearer YOUR_API_TOKEN", "Content-Type": "application/json" }, body: JSON.stringify({ "url": "https://docs.example.com", "search": "authentication", "limit": 100 }) }); const data = await response.json(); console.log(data); ``` ### Python ```python import requests headers = { 'Authorization': 'Bearer YOUR_API_TOKEN' } response = requests.post('https://beta-api.b3os.org/compat/firecrawl/v1/map', headers=headers, json={ "url": "https://docs.example.com", "search": "authentication", "limit": 100 }) print(response.json()) ```