Start a crawl (Firecrawl)

Firecrawl's `POST /v1/crawl`, served by a vision crawl (`crawl_` job). robots.txt is always honored (`ignoreRobotsTxt: true` returns `unsupported_option`). `limit` above your plan's crawl limit (500 pages in the beta) is treated as that limit. Each page bills one `vision.page`.

POST /compat/firecrawl/v1/crawl
Firecrawl's `POST /v1/crawl`, served by a vision crawl (`crawl_` job). robots.txt is always honored (`ignoreRobotsTxt: true` returns `unsupported_option`). `limit` above your plan's crawl limit (500 pages in the beta) is treated as that limit. Each page bills one `vision.page`.

Authentication

Bearer Token (b3os_sk_ API key or b3os_pat_ token)

Header Parameters

x-idempotency-key string optional header
Idempotency key as the Firecrawl SDKs send it; same semantics as `Idempotency-Key`.

Request Body required

application/json
url string (uri) REQUIRED
Where to start.
excludePaths string[]
Array of: string
includePaths string[]
Array of: string
maxDepth integer
maxDiscoveryDepth integer
ignoreSitemap boolean
ignoreQueryParameters boolean
limit integer
Pages to crawl; above your plan's crawl limit it is treated as that limit.
allowBackwardLinks boolean
crawlEntireDomain boolean
allowExternalLinks boolean
allowSubdomains boolean
delay number
maxConcurrency integer
webhook string (uri) | object
A webhook URL, or an object with the URL and options.
One of:
Option 1string (uri)
Option 2
url string (uri) REQUIRED
headers object
metadata object
events string[]
Array of: string
scrapeOptions object
formats string[]
`changeTracking` returns `unsupported_option`.
Array of: string
onlyMainContent boolean
includeTags string[]
Array of: string
excludeTags string[]
Array of: string
maxAge integer
Accept a cached copy up to this old, in milliseconds. Default 0 (always fetch).
headers object
waitFor integer
mobile boolean
skipTlsVerification boolean
timeout integer
Milliseconds; values above 60000 are treated as 60000.
parsePDF boolean
PDF parsing is not supported; PDFs return `content_type_unsupported`.
jsonOptions object
schema object
systemPrompt string
prompt string
checkPromptInjection boolean
Not supported by B3OS: setting it returns `400` with code `unsupported_option`.
actions (object | object | object | object | object | object | object | object | object)[]
Array of:
One of:
Option 1
type string REQUIRED
Enum: wait
milliseconds integer
selector string
Option 2
type string REQUIRED
Enum: screenshot
fullPage boolean
quality integer
viewport object
width integer REQUIRED
height integer REQUIRED
Option 3
type string REQUIRED
Enum: click
selector string REQUIRED
all boolean
Option 4
type string REQUIRED
Enum: write
text string REQUIRED
Option 5
type string REQUIRED
Enum: press
key string REQUIRED
Option 6
type string REQUIRED
Enum: scroll
direction string
Enum: up, down
selector string
Option 7
type string REQUIRED
Enum: scrape
Option 8
type string REQUIRED
Enum: executeJavascript
script string REQUIRED
Option 9
type string REQUIRED
Enum: pdf
format string
landscape boolean
scale number
location object
country string
ISO 3166-1 alpha-2 code. B3OS emulates the language and time zone; it has no per-country proxies.
languages string[]
Array of: string
removeBase64Images boolean
blockAds boolean
proxy string
`enhanced` returns `unsupported_option`.
Enum: basic, enhanced, auto
storeInCache boolean
changeTrackingOptions object
Not supported by B3OS: setting it returns `400` with code `unsupported_option`.
zeroDataRetention boolean
Do not cache or store the result.
threatProtection object
Not supported by B3OS: setting it returns `400` with code `unsupported_option`.
integration string
Accepted and ignored (the Firecrawl SDKs send it).
origin string
Accepted and ignored (the Firecrawl SDKs send it).
regexOnFullURL boolean
zeroDataRetention boolean
integration string
Accepted and ignored (the Firecrawl SDKs send it).
origin string
Accepted and ignored (the Firecrawl SDKs send it).

Responses

200 The crawl was started.
application/json
success boolean REQUIRED
Enum: true
id string REQUIRED
url string (uri) REQUIRED
The job's status URL.
invalidURLs string[] | null
Array of: string
400 The body failed validation, uses an unsupported option, or a URL is invalid (`invalid_request`, `unsupported_option`, `invalid_url`).
401 Missing or invalid B3OS key (`unauthorized`). Firecrawl `fc-` keys are not accepted.
402 No credits left or the key's spend cap is reached (`credits_exhausted`, `spend_cap_exceeded`).
403 The URL is refused by the egress guard or robots.txt, or the key lacks a scope (`egress_denied`, `robots_disallowed`, `forbidden`).
429 A rate or concurrency limit applies (`rate_limited`, `concurrency_limited`).
500 Something failed inside B3OS (`internal`).
503 The node or vision is unavailable (`node_unavailable`, `product_unavailable`). 
curl -X POST 'https://beta-api.b3os.org/compat/firecrawl/v1/crawl' \  -H 'Authorization: Bearer YOUR_API_TOKEN' \  -H 'Content-Type: application/json' \  -d '{  "url": "https://docs.example.com",  "limit": 100,  "scrapeOptions": {    "formats": [      "markdown"    ]  }}'
const response = await fetch('https://beta-api.b3os.org/compat/firecrawl/v1/crawl', {  method: 'POST',  headers: {      "Authorization": "Bearer YOUR_API_TOKEN",      "Content-Type": "application/json"  },  body: JSON.stringify({    "url": "https://docs.example.com",    "limit": 100,    "scrapeOptions": {      "formats": [        "markdown"      ]    }  })});const data = await response.json();console.log(data);
import requestsheaders = {    'Authorization': 'Bearer YOUR_API_TOKEN'}response = requests.post('https://beta-api.b3os.org/compat/firecrawl/v1/crawl', headers=headers, json={  "url": "https://docs.example.com",  "limit": 100,  "scrapeOptions": {    "formats": [      "markdown"    ]  }})print(response.json())
200 Response 
{  "success": true,  "id": "crawl_01m5t4gh4qhgs70zwajezbe5p0",  "url": "https://beta-api.b3os.org/compat/firecrawl/v1/crawl/crawl_01m5t4gh4qhgs70zwajezbe5p0"}
Previous Get batch scrape errors (Firecrawl)
Next List active crawls (Firecrawl)
Ask a question... ⌘I