Start a batch scrape (Firecrawl)

Firecrawl's `POST /v2/batch/scrape`, served by a vision batch scrape (`batch_` job). The scrape options are at the top level of the body, as Firecrawl takes them. `webhook` deliveries use Firecrawl's payload and `X-Firecrawl-Signature`. Each page bills one `vision.page`.

POST /compat/firecrawl/v2/batch/scrape

Authentication

Bearer Token (b3os_sk_ API key or b3os_pat_ token)

Header Parameters

x-idempotency-key string optional header

Idempotency key as the Firecrawl SDKs send it; same semantics as `Idempotency-Key`.

Request Body required

application/json

urls string (uri)[] REQUIRED

Array of: string (uri)

webhook string (uri) | object

A webhook URL, or an object with the URL and options.

One of:

Option 1string (uri)

Option 2

url string (uri) REQUIRED

headers object

metadata object

events string[]

Array of: string

maxConcurrency integer

ignoreInvalidURLs boolean

appendToId string

Not supported by B3OS: setting it returns `400` with code `unsupported_option`.

zeroDataRetention boolean

Do not cache or store the result.

integration string

Accepted and ignored (the Firecrawl SDKs send it).

origin string

Accepted and ignored (the Firecrawl SDKs send it).

Array of:

One of:

Option 1string

Option 2

type string REQUIRED

Enum: markdown, summary, html, rawHtml, rawBase64, links, images, branding, product, menu, audio, video

Option 3

type string REQUIRED

Enum: screenshot

fullPage boolean

quality integer

viewport object

width integer REQUIRED

height integer REQUIRED

Option 4

type string REQUIRED

Enum: json

schema object

prompt string

checkPromptInjection boolean

Deprecated in Firecrawl. Not supported by B3OS: setting it returns `400` with code `unsupported_option`.

Option 5

type string REQUIRED

Enum: changeTracking

modes string[]

Array of: string

schema object

prompt string

tag string | null

Option 6

type string REQUIRED

Enum: attributes

selectors object[] REQUIRED

Array of:

selector string REQUIRED

attribute string REQUIRED

Option 7

type string REQUIRED

Enum: question

question string REQUIRED

Option 8

type string REQUIRED

Enum: highlights

query string REQUIRED

onlyMainContent boolean

onlyCleanContent boolean

Not supported by B3OS: setting it returns `400` with code `unsupported_option`.

includeTags string[]

Array of: string

excludeTags string[]

Array of: string

maxAge integer

Accept a cached copy up to this old, in milliseconds. Default 2 days.

minAge integer

Only answer from the cache; a miss returns `404` with code `SCRAPE_NO_CACHED_DATA`.

headers object

waitFor integer

mobile boolean

skipTlsVerification boolean

timeout integer

Milliseconds; values above 60000 are treated as 60000.

parsers unknown[]

PDF and image parsing are not supported; a non-empty list returns `unsupported_option`.

Array of: unknown

Array of:

One of:

Option 1

type string REQUIRED

Enum: wait

milliseconds integer

selector string

Option 2

type string REQUIRED

Enum: screenshot

fullPage boolean

quality integer

viewport object

width integer REQUIRED

height integer REQUIRED

Option 3

type string REQUIRED

Enum: click

selector string REQUIRED

all boolean

Option 4

type string REQUIRED

Enum: write

text string REQUIRED

Option 5

type string REQUIRED

Enum: press

key string REQUIRED

Option 6

type string REQUIRED

Enum: scroll

direction string

Enum: up, down

selector string

Option 7

type string REQUIRED

Enum: scrape

Option 8

type string REQUIRED

Enum: executeJavascript

script string REQUIRED

Option 9

type string REQUIRED

Enum: pdf

format string

landscape boolean

scale number

location object

country string

ISO 3166-1 alpha-2 code. B3OS emulates the language and time zone; it has no per-country proxies.

languages string[]

Array of: string

removeBase64Images boolean

blockAds boolean

proxy string

`basic` and `auto` fetch directly from the node; `enhanced` returns `unsupported_option` (B3OS has no proxy network).

Enum: basic, enhanced, auto

storeInCache boolean

lockdown boolean

Not supported by B3OS: setting it returns `400` with code `unsupported_option`.

checkPromptInjection boolean

Not supported by B3OS: setting it returns `400` with code `unsupported_option`.

redactPII boolean | object

Not supported by B3OS: setting it returns `400` with code `unsupported_option`.

One of:

Option 1boolean

Option 2object

profile object

Load a vision profile by name; `saveChanges` saves cookies back.

name string REQUIRED

saveChanges boolean

threatProtection object

Not supported by B3OS: setting it returns `400` with code `unsupported_option`.

auditMetadata object

Accepted and ignored (the Firecrawl SDKs send it).

Responses

200 The batch was started.

application/json

success boolean REQUIRED

Enum: true

id string REQUIRED

url string (uri) REQUIRED

The job's status URL.

invalidURLs string[] | null

Array of: string

400 The body failed validation, uses an unsupported option, or a URL is invalid (`invalid_request`, `unsupported_option`, `invalid_url`).

401 Missing or invalid B3OS key (`unauthorized`). Firecrawl `fc-` keys are not accepted.

402 No credits left or the key's spend cap is reached (`credits_exhausted`, `spend_cap_exceeded`).

403 The URL is refused by the egress guard or robots.txt, or the key lacks a scope (`egress_denied`, `robots_disallowed`, `forbidden`).

429 A rate or concurrency limit applies (`rate_limited`, `concurrency_limited`).

500 Something failed inside B3OS (`internal`).

503 The node or vision is unavailable (`node_unavailable`, `product_unavailable`).

curl -X POST 'https://beta-api.b3os.org/compat/firecrawl/v2/batch/scrape' \  -H 'Authorization: Bearer YOUR_API_TOKEN' \  -H 'Content-Type: application/json' \  -d '{  "urls": [    "https://example.com/blog/a",    "https://example.com/blog/b"  ],  "formats": [    "markdown"  ]}'

const response = await fetch('https://beta-api.b3os.org/compat/firecrawl/v2/batch/scrape', {  method: 'POST',  headers: {      "Authorization": "Bearer YOUR_API_TOKEN",      "Content-Type": "application/json"  },  body: JSON.stringify({    "urls": [      "https://example.com/blog/a",      "https://example.com/blog/b"    ],    "formats": [      "markdown"    ]  })});const data = await response.json();console.log(data);

import requestsheaders = {    'Authorization': 'Bearer YOUR_API_TOKEN'}response = requests.post('https://beta-api.b3os.org/compat/firecrawl/v2/batch/scrape', headers=headers, json={  "urls": [    "https://example.com/blog/a",    "https://example.com/blog/b"  ],  "formats": [    "markdown"  ]})print(response.json())

200 Response

{  "success": true,  "id": "batch_01m5t3v176vrhf8a1mn570c5gd",  "url": "https://beta-api.b3os.org/compat/firecrawl/v2/batch/scrape/batch_01m5t3v176vrhf8a1mn570c5gd",  "invalidURLs": []}

400 Response

{  "success": false,  "error": "Bad Request",  "code": "BAD_REQUEST",  "details": [    {      "code": "unrecognized_keys",      "keys": [        "fromats"      ],      "path": [],      "message": "Unrecognized key in body -- please review the v2 API documentation for request body changes"    }  ]}

401 Response

{  "success": false,  "error": "Unauthorized: Invalid token",  "code": "unauthorized"}

402 Response

{  "success": false,  "error": "Insufficient credits: Organization org_01m5t5hthd29mm5wpza5ndd6mm has no credits left",  "code": "credits_exhausted"}

403 Response

{  "success": false,  "error": "Destination not allowed: http://10.0.0.5/ resolves to a private address",  "code": "egress_denied"}

429 Response

{  "success": false,  "error": "Rate limit exceeded",  "code": "rate_limited"}

500 Response

{  "success": false,  "error": "Internal Server Error",  "code": "internal"}

503 Response

{  "success": false,  "error": "Service Unavailable",  "code": "node_unavailable"}

POST /compat/firecrawl/v2/batch/scrape

Base URL

Bearer Token

Headers

x-idempotency-key string

Request Body required application/json

Response

Was this page helpful?

Previous Scrape a URL (Firecrawl)

Next Get a batch scrape (Firecrawl)