Scrape a URL (Firecrawl)

Firecrawl's `POST /v1/scrape`, served by vision. Same request and response fields as Firecrawl; B3OS extras are in `data.metadata.b3os` and the `Server-Timing` header. `timeout` above 60000 is treated as 60000 (no request is held longer). Options B3OS cannot honor return `400` with code `unsupported_option` instead of being ignored. Bills one `vision.page`.

POST /compat/firecrawl/v1/scrape
Firecrawl's `POST /v1/scrape`, served by vision. Same request and response fields as Firecrawl; B3OS extras are in `data.metadata.b3os` and the `Server-Timing` header. `timeout` above 60000 is treated as 60000 (no request is held longer). Options B3OS cannot honor return `400` with code `unsupported_option` instead of being ignored. Bills one `vision.page`.

Authentication

Bearer Token (b3os_sk_ API key or b3os_pat_ token)

Request Body required

application/json
url string (uri) REQUIRED
The URL to scrape.
formats string[]
`changeTracking` returns `unsupported_option`.
Array of: string
onlyMainContent boolean
includeTags string[]
Array of: string
excludeTags string[]
Array of: string
maxAge integer
Accept a cached copy up to this old, in milliseconds. Default 0 (always fetch).
headers object
waitFor integer
mobile boolean
skipTlsVerification boolean
timeout integer
Milliseconds; values above 60000 are treated as 60000.
parsePDF boolean
PDF parsing is not supported; PDFs return `content_type_unsupported`.
jsonOptions object
schema object
systemPrompt string
prompt string
checkPromptInjection boolean
Not supported by B3OS: setting it returns `400` with code `unsupported_option`.
actions (object | object | object | object | object | object | object | object | object)[]
Array of:
One of:
Option 1
type string REQUIRED
Enum: wait
milliseconds integer
selector string
Option 2
type string REQUIRED
Enum: screenshot
fullPage boolean
quality integer
viewport object
width integer REQUIRED
height integer REQUIRED
Option 3
type string REQUIRED
Enum: click
selector string REQUIRED
all boolean
Option 4
type string REQUIRED
Enum: write
text string REQUIRED
Option 5
type string REQUIRED
Enum: press
key string REQUIRED
Option 6
type string REQUIRED
Enum: scroll
direction string
Enum: up, down
selector string
Option 7
type string REQUIRED
Enum: scrape
Option 8
type string REQUIRED
Enum: executeJavascript
script string REQUIRED
Option 9
type string REQUIRED
Enum: pdf
format string
landscape boolean
scale number
location object
country string
ISO 3166-1 alpha-2 code. B3OS emulates the language and time zone; it has no per-country proxies.
languages string[]
Array of: string
removeBase64Images boolean
blockAds boolean
proxy string
`enhanced` returns `unsupported_option`.
Enum: basic, enhanced, auto
storeInCache boolean
changeTrackingOptions object
Not supported by B3OS: setting it returns `400` with code `unsupported_option`.
zeroDataRetention boolean
Do not cache or store the result.
threatProtection object
Not supported by B3OS: setting it returns `400` with code `unsupported_option`.
integration string
Accepted and ignored (the Firecrawl SDKs send it).
origin string
Accepted and ignored (the Firecrawl SDKs send it).

Responses

200 The scraped document.
application/json
success boolean REQUIRED
Enum: true
data object REQUIRED
markdown string
html string | null
rawHtml string | null
links string[]
Array of: string
screenshot string | null
A `data:image/...;base64,` URL.
json unknown
Data from the `json` format.
actions object | null
screenshots string[]
Array of: string
scrapes object[]
Array of:
url string
html string
javascriptReturns object[]
Array of:
type string
value unknown
pdfs string[]
Array of: string
metadata object
Page metadata. Other `<meta>` tags appear under their own names.
title string | string[]
One of:
Option 1string
Option 2
Array of: string
description string | string[]
One of:
Option 1string
Option 2
Array of: string
language string | null | string[]
One of:
Option 1string | null
Option 2
Array of: string
sourceURL string (uri)
The requested URL.
url string (uri)
The final URL after redirects.
keywords string | string[]
One of:
Option 1string
Option 2
Array of: string
ogLocaleAlternate string[]
Array of: string
statusCode integer
contentType string
error string | null
scrapeId string
The B3OS request ID of the scrape.
cacheState string
Enum: hit, miss
cachedAt string (date-time)
creditsUsed integer
proxyUsed string
Enum: basic, stealth
concurrencyLimited boolean
concurrencyQueueDurationMs number
b3os object
B3OS extras (Firecrawl clients ignore them).
tier string
Enum: cache, fetch, browser
request_id string
timing object
Server-side phase durations in milliseconds with microsecond resolution: the same phases as the `Server-Timing` header, keyed `<phase>_ms`. `total_ms` is always present. Phase names are product-specific and documented per operation. Example values in this reference are illustrative, not measurements.
total_ms number REQUIRED
llm_extraction object | null
Deprecated v0 field; always absent.
warning string | null
400 The body failed validation, uses an unsupported option, or a URL is invalid (`invalid_request`, `unsupported_option`, `invalid_url`).
401 Missing or invalid B3OS key (`unauthorized`). Firecrawl `fc-` keys are not accepted.
402 No credits left or the key's spend cap is reached (`credits_exhausted`, `spend_cap_exceeded`).
403 The URL is refused by the egress guard or robots.txt, or the key lacks a scope (`egress_denied`, `robots_disallowed`, `forbidden`).
404 No such job in this project (`crawl_not_found`, `batch_not_found`, `job_not_found`).
408 The page did not load within the timeout (`scrape_timeout`).
429 A rate or concurrency limit applies (`rate_limited`, `concurrency_limited`).
500 Something failed inside B3OS (`internal`).
502 The site could not be reached or loaded (`upstream_unreachable`, `navigation_failed`).
503 The node or vision is unavailable (`node_unavailable`, `product_unavailable`). 
curl -X POST 'https://beta-api.b3os.org/compat/firecrawl/v1/scrape' \  -H 'Authorization: Bearer YOUR_API_TOKEN' \  -H 'Content-Type: application/json' \  -d '{  "url": "https://example.com/pricing",  "formats": [    "markdown"  ],  "onlyMainContent": true}'
const response = await fetch('https://beta-api.b3os.org/compat/firecrawl/v1/scrape', {  method: 'POST',  headers: {      "Authorization": "Bearer YOUR_API_TOKEN",      "Content-Type": "application/json"  },  body: JSON.stringify({    "url": "https://example.com/pricing",    "formats": [      "markdown"    ],    "onlyMainContent": true  })});const data = await response.json();console.log(data);
import requestsheaders = {    'Authorization': 'Bearer YOUR_API_TOKEN'}response = requests.post('https://beta-api.b3os.org/compat/firecrawl/v1/scrape', headers=headers, json={  "url": "https://example.com/pricing",  "formats": [    "markdown"  ],  "onlyMainContent": true})print(response.json())
200 Response 
{  "success": true,  "data": {    "markdown": "# Pricing\n\nStart free, then pay per page.",    "metadata": {      "title": "Pricing | Example",      "description": "Plans and prices for Example.",      "language": "en",      "sourceURL": "https://example.com/pricing",      "statusCode": 200,      "b3os": {        "tier": "cache",        "request_id": "req_01m5t4yg394wegr3j4y01qb363"      }    }  }}
Previous Concurrency check (not supported)
Next Start a batch scrape (Firecrawl)
Ask a question... ⌘I