Start a crawl (Firecrawl)

Firecrawl's `POST /v2/crawl`, served by a vision crawl (`crawl_` job). robots.txt is always honored (`ignoreRobotsTxt: true` returns `unsupported_option`). `limit` above your plan's crawl limit (500 pages in the beta) is treated as that limit. Each page bills one `vision.page`.

POST /compat/firecrawl/v2/crawl
Firecrawl's `POST /v2/crawl`, served by a vision crawl (`crawl_` job). robots.txt is always honored (`ignoreRobotsTxt: true` returns `unsupported_option`). `limit` above your plan's crawl limit (500 pages in the beta) is treated as that limit. Each page bills one `vision.page`.

Authentication

Bearer Token (b3os_sk_ API key or b3os_pat_ token)

Header Parameters

x-idempotency-key string optional header
Idempotency key as the Firecrawl SDKs send it; same semantics as `Idempotency-Key`.

Request Body required

application/json
url string (uri) REQUIRED
Where to start.
prompt string
Not supported by B3OS: setting it returns `400` with code `unsupported_option`.
excludePaths string[]
Array of: string
includePaths string[]
Array of: string
maxDiscoveryDepth integer
sitemap string
Enum: skip, include, only
ignoreQueryParameters boolean
regexOnFullURL boolean
limit integer
Pages to crawl; above your plan's crawl limit it is treated as that limit.
crawlEntireDomain boolean
allowExternalLinks boolean
allowSubdomains boolean
ignoreRobotsTxt boolean
robots.txt is always honored; `true` returns `unsupported_option`.
robotsUserAgent string
Not supported by B3OS: setting it returns `400` with code `unsupported_option`.
deduplicateSimilarURLs boolean
delay number
Seconds between requests.
maxConcurrency integer
webhook string (uri) | object
A webhook URL, or an object with the URL and options.
One of:
Option 1string (uri)
Option 2
url string (uri) REQUIRED
headers object
metadata object
events string[]
Array of: string
scrapeOptions object
formats (string | object | object | object | object | object | object | object)[]
Array of:
One of:
Option 1string
Option 2
type string REQUIRED
Enum: markdown, summary, html, rawHtml, rawBase64, links, images, branding, product, menu, audio, video
Option 3
type string REQUIRED
Enum: screenshot
fullPage boolean
quality integer
viewport object
width integer REQUIRED
height integer REQUIRED
Option 4
type string REQUIRED
Enum: json
schema object
prompt string
checkPromptInjection boolean
Deprecated in Firecrawl. Not supported by B3OS: setting it returns `400` with code `unsupported_option`.
Option 5
type string REQUIRED
Enum: changeTracking
modes string[]
Array of: string
schema object
prompt string
tag string | null
Option 6
type string REQUIRED
Enum: attributes
selectors object[] REQUIRED
Array of:
selector string REQUIRED
attribute string REQUIRED
Option 7
type string REQUIRED
Enum: question
question string REQUIRED
Option 8
type string REQUIRED
Enum: highlights
query string REQUIRED
onlyMainContent boolean
onlyCleanContent boolean
Not supported by B3OS: setting it returns `400` with code `unsupported_option`.
includeTags string[]
Array of: string
excludeTags string[]
Array of: string
maxAge integer
Accept a cached copy up to this old, in milliseconds. Default 2 days.
minAge integer
Only answer from the cache; a miss returns `404` with code `SCRAPE_NO_CACHED_DATA`.
headers object
waitFor integer
mobile boolean
skipTlsVerification boolean
timeout integer
Milliseconds; values above 60000 are treated as 60000.
parsers unknown[]
PDF and image parsing are not supported; a non-empty list returns `unsupported_option`.
Array of: unknown
actions (object | object | object | object | object | object | object | object | object)[]
Array of:
One of:
Option 1
type string REQUIRED
Enum: wait
milliseconds integer
selector string
Option 2
type string REQUIRED
Enum: screenshot
fullPage boolean
quality integer
viewport object
width integer REQUIRED
height integer REQUIRED
Option 3
type string REQUIRED
Enum: click
selector string REQUIRED
all boolean
Option 4
type string REQUIRED
Enum: write
text string REQUIRED
Option 5
type string REQUIRED
Enum: press
key string REQUIRED
Option 6
type string REQUIRED
Enum: scroll
direction string
Enum: up, down
selector string
Option 7
type string REQUIRED
Enum: scrape
Option 8
type string REQUIRED
Enum: executeJavascript
script string REQUIRED
Option 9
type string REQUIRED
Enum: pdf
format string
landscape boolean
scale number
location object
country string
ISO 3166-1 alpha-2 code. B3OS emulates the language and time zone; it has no per-country proxies.
languages string[]
Array of: string
removeBase64Images boolean
blockAds boolean
proxy string
`basic` and `auto` fetch directly from the node; `enhanced` returns `unsupported_option` (B3OS has no proxy network).
Enum: basic, enhanced, auto
storeInCache boolean
lockdown boolean
Not supported by B3OS: setting it returns `400` with code `unsupported_option`.
checkPromptInjection boolean
Not supported by B3OS: setting it returns `400` with code `unsupported_option`.
redactPII boolean | object
Not supported by B3OS: setting it returns `400` with code `unsupported_option`.
One of:
Option 1boolean
Option 2object
profile object
Load a vision profile by name; `saveChanges` saves cookies back.
name string REQUIRED
saveChanges boolean
zeroDataRetention boolean
Do not cache or store the result.
threatProtection object
Not supported by B3OS: setting it returns `400` with code `unsupported_option`.
auditMetadata object
Accepted and ignored (the Firecrawl SDKs send it).
integration string
Accepted and ignored (the Firecrawl SDKs send it).
origin string
Accepted and ignored (the Firecrawl SDKs send it).
zeroDataRetention boolean
integration string
Accepted and ignored (the Firecrawl SDKs send it).
origin string
Accepted and ignored (the Firecrawl SDKs send it).

Responses

200 The crawl was started.
application/json
success boolean REQUIRED
Enum: true
id string REQUIRED
url string (uri) REQUIRED
The job's status URL.
invalidURLs string[] | null
Array of: string
400 The body failed validation, uses an unsupported option, or a URL is invalid (`invalid_request`, `unsupported_option`, `invalid_url`).
401 Missing or invalid B3OS key (`unauthorized`). Firecrawl `fc-` keys are not accepted.
402 No credits left or the key's spend cap is reached (`credits_exhausted`, `spend_cap_exceeded`).
403 The URL is refused by the egress guard or robots.txt, or the key lacks a scope (`egress_denied`, `robots_disallowed`, `forbidden`).
429 A rate or concurrency limit applies (`rate_limited`, `concurrency_limited`).
500 Something failed inside B3OS (`internal`).
503 The node or vision is unavailable (`node_unavailable`, `product_unavailable`). 
curl -X POST 'https://beta-api.b3os.org/compat/firecrawl/v2/crawl' \  -H 'Authorization: Bearer YOUR_API_TOKEN' \  -H 'Content-Type: application/json' \  -d '{  "url": "https://docs.example.com",  "limit": 100,  "includePaths": [    "^/guides/.*"  ],  "scrapeOptions": {    "formats": [      "markdown"    ]  }}'
const response = await fetch('https://beta-api.b3os.org/compat/firecrawl/v2/crawl', {  method: 'POST',  headers: {      "Authorization": "Bearer YOUR_API_TOKEN",      "Content-Type": "application/json"  },  body: JSON.stringify({    "url": "https://docs.example.com",    "limit": 100,    "includePaths": [      "^/guides/.*"    ],    "scrapeOptions": {      "formats": [        "markdown"      ]    }  })});const data = await response.json();console.log(data);
import requestsheaders = {    'Authorization': 'Bearer YOUR_API_TOKEN'}response = requests.post('https://beta-api.b3os.org/compat/firecrawl/v2/crawl', headers=headers, json={  "url": "https://docs.example.com",  "limit": 100,  "includePaths": [    "^/guides/.*"  ],  "scrapeOptions": {    "formats": [      "markdown"    ]  }})print(response.json())
200 Response 
{  "success": true,  "id": "crawl_01m5t4gh4qhgs70zwajezbe5p0",  "url": "https://beta-api.b3os.org/compat/firecrawl/v2/crawl/crawl_01m5t4gh4qhgs70zwajezbe5p0"}
Previous Get batch scrape errors (Firecrawl)
Next List active crawls (Firecrawl)
Ask a question... ⌘I