Scrape a URL (Firecrawl)

Firecrawl's `POST /v2/scrape`, served by vision. Same request and response fields as Firecrawl; B3OS extras are in `data.metadata.b3os` and the `Server-Timing` header. `timeout` above 60000 is treated as 60000 (no request is held longer). Options B3OS cannot honor return `400` with code `unsupported_option` instead of being ignored. Bills one `vision.page`.

POST /compat/firecrawl/v2/scrape

Authentication

Bearer Token (b3os_sk_ API key or b3os_pat_ token)

Request Body required

application/json

url string (uri) REQUIRED

The URL to scrape.

Array of:

One of:

Option 1string

Option 2

type string REQUIRED

Enum: markdown, summary, html, rawHtml, rawBase64, links, images, branding, product, menu, audio, video

Option 3

type string REQUIRED

Enum: screenshot

fullPage boolean

quality integer

viewport object

width integer REQUIRED

height integer REQUIRED

Option 4

type string REQUIRED

Enum: json

schema object

prompt string

checkPromptInjection boolean

Deprecated in Firecrawl. Not supported by B3OS: setting it returns `400` with code `unsupported_option`.

Option 5

type string REQUIRED

Enum: changeTracking

modes string[]

Array of: string

schema object

prompt string

tag string | null

Option 6

type string REQUIRED

Enum: attributes

selectors object[] REQUIRED

Array of:

selector string REQUIRED

attribute string REQUIRED

Option 7

type string REQUIRED

Enum: question

question string REQUIRED

Option 8

type string REQUIRED

Enum: highlights

query string REQUIRED

onlyMainContent boolean

onlyCleanContent boolean

Not supported by B3OS: setting it returns `400` with code `unsupported_option`.

includeTags string[]

Array of: string

excludeTags string[]

Array of: string

maxAge integer

Accept a cached copy up to this old, in milliseconds. Default 2 days.

minAge integer

Only answer from the cache; a miss returns `404` with code `SCRAPE_NO_CACHED_DATA`.

headers object

waitFor integer

mobile boolean

skipTlsVerification boolean

timeout integer

Milliseconds; values above 60000 are treated as 60000.

parsers unknown[]

PDF and image parsing are not supported; a non-empty list returns `unsupported_option`.

Array of: unknown

Array of:

One of:

Option 1

type string REQUIRED

Enum: wait

milliseconds integer

selector string

Option 2

type string REQUIRED

Enum: screenshot

fullPage boolean

quality integer

viewport object

width integer REQUIRED

height integer REQUIRED

Option 3

type string REQUIRED

Enum: click

selector string REQUIRED

all boolean

Option 4

type string REQUIRED

Enum: write

text string REQUIRED

Option 5

type string REQUIRED

Enum: press

key string REQUIRED

Option 6

type string REQUIRED

Enum: scroll

direction string

Enum: up, down

selector string

Option 7

type string REQUIRED

Enum: scrape

Option 8

type string REQUIRED

Enum: executeJavascript

script string REQUIRED

Option 9

type string REQUIRED

Enum: pdf

format string

landscape boolean

scale number

location object

country string

ISO 3166-1 alpha-2 code. B3OS emulates the language and time zone; it has no per-country proxies.

languages string[]

Array of: string

removeBase64Images boolean

blockAds boolean

proxy string

`basic` and `auto` fetch directly from the node; `enhanced` returns `unsupported_option` (B3OS has no proxy network).

Enum: basic, enhanced, auto

storeInCache boolean

lockdown boolean

Not supported by B3OS: setting it returns `400` with code `unsupported_option`.

checkPromptInjection boolean

Not supported by B3OS: setting it returns `400` with code `unsupported_option`.

redactPII boolean | object

Not supported by B3OS: setting it returns `400` with code `unsupported_option`.

One of:

Option 1boolean

Option 2object

profile object

Load a vision profile by name; `saveChanges` saves cookies back.

name string REQUIRED

saveChanges boolean

zeroDataRetention boolean

Do not cache or store the result.

threatProtection object

Not supported by B3OS: setting it returns `400` with code `unsupported_option`.

auditMetadata object

Accepted and ignored (the Firecrawl SDKs send it).

integration string

Accepted and ignored (the Firecrawl SDKs send it).

origin string

Accepted and ignored (the Firecrawl SDKs send it).

Responses

200 The scraped document.

application/json

success boolean REQUIRED

Enum: true

data object REQUIRED

markdown string

summary string | null

html string | null

rawHtml string | null

links string[]

Array of: string

images string[]

Array of: string

screenshot string | null

A `data:image/...;base64,` URL (B3OS returns the image inline).

json unknown

Data from the `json` format.

actions object | null

screenshots string[]

Array of: string

scrapes object[]

Array of:

url string

html string

javascriptReturns object[]

Array of:

type string

value unknown

pdfs string[]

Array of: string

metadata object

Page metadata. Other `<meta>` tags appear under their own names.

title string | string[]

One of:

Option 1string

Option 2

Array of: string

description string | string[]

One of:

Option 1string

Option 2

Array of: string

language string | null | string[]

One of:

Option 1string | null

Option 2

Array of: string

sourceURL string (uri)

The requested URL.

url string (uri)

The final URL after redirects.

keywords string | string[]

One of:

Option 1string

Option 2

Array of: string

ogLocaleAlternate string[]

Array of: string

statusCode integer

contentType string

error string | null

scrapeId string

The B3OS request ID of the scrape.

cacheState string

Enum: hit, miss

cachedAt string (date-time)

creditsUsed integer

proxyUsed string

Enum: basic, stealth

concurrencyLimited boolean

concurrencyQueueDurationMs number

b3os object

B3OS extras (Firecrawl clients ignore them).

tier string

Enum: cache, fetch, browser

request_id string

timing object

Server-side phase durations in milliseconds with microsecond resolution: the same phases as the `Server-Timing` header, keyed `<phase>_ms`. `total_ms` is always present. Phase names are product-specific and documented per operation. Example values in this reference are illustrative, not measurements.

total_ms number REQUIRED

warning string | null

400 The body failed validation, uses an unsupported option, or a URL is invalid (`invalid_request`, `unsupported_option`, `invalid_url`).

401 Missing or invalid B3OS key (`unauthorized`). Firecrawl `fc-` keys are not accepted.

402 No credits left or the key's spend cap is reached (`credits_exhausted`, `spend_cap_exceeded`).

403 The URL is refused by the egress guard or robots.txt, or the key lacks a scope (`egress_denied`, `robots_disallowed`, `forbidden`).

404 No such job in this project (`crawl_not_found`, `batch_not_found`, `job_not_found`).

408 The page did not load within the timeout (`scrape_timeout`).

429 A rate or concurrency limit applies (`rate_limited`, `concurrency_limited`).

500 Something failed inside B3OS (`internal`).

502 The site could not be reached or loaded (`upstream_unreachable`, `navigation_failed`).

503 The node or vision is unavailable (`node_unavailable`, `product_unavailable`).

curl -X POST 'https://beta-api.b3os.org/compat/firecrawl/v2/scrape' \  -H 'Authorization: Bearer YOUR_API_TOKEN' \  -H 'Content-Type: application/json' \  -d '{  "url": "https://example.com/pricing",  "formats": [    "markdown"  ]}'

const response = await fetch('https://beta-api.b3os.org/compat/firecrawl/v2/scrape', {  method: 'POST',  headers: {      "Authorization": "Bearer YOUR_API_TOKEN",      "Content-Type": "application/json"  },  body: JSON.stringify({    "url": "https://example.com/pricing",    "formats": [      "markdown"    ]  })});const data = await response.json();console.log(data);

import requestsheaders = {    'Authorization': 'Bearer YOUR_API_TOKEN'}response = requests.post('https://beta-api.b3os.org/compat/firecrawl/v2/scrape', headers=headers, json={  "url": "https://example.com/pricing",  "formats": [    "markdown"  ]})print(response.json())

200 Response

{  "success": true,  "data": {    "markdown": "# Pricing\n\nStart free, then pay per page.",    "metadata": {      "title": "Pricing | Example",      "description": "Plans and prices for Example.",      "language": "en",      "sourceURL": "https://example.com/pricing",      "url": "https://example.com/pricing",      "statusCode": 200,      "contentType": "text/html; charset=utf-8",      "scrapeId": "req_01m5t4yg394wegr3j4y01qb363",      "cacheState": "miss",      "creditsUsed": 1,      "proxyUsed": "basic",      "b3os": {        "tier": "fetch",        "request_id": "req_01m5t4yg394wegr3j4y01qb363",        "timing": {          "fetch_ms": 41.2,          "extract_ms": 3.9,          "total_ms": 45.8        }      }    }  }}

400 Response

{  "success": false,  "error": "Bad Request",  "code": "BAD_REQUEST",  "details": [    {      "code": "unrecognized_keys",      "keys": [        "fromats"      ],      "path": [],      "message": "Unrecognized key in body -- please review the v2 API documentation for request body changes"    }  ]}

401 Response

{  "success": false,  "error": "Unauthorized: Invalid token",  "code": "unauthorized"}

402 Response

{  "success": false,  "error": "Insufficient credits: Organization org_01m5t5hthd29mm5wpza5ndd6mm has no credits left",  "code": "credits_exhausted"}

403 Response

{  "success": false,  "error": "Destination not allowed: http://10.0.0.5/ resolves to a private address",  "code": "egress_denied"}

404 Response

{  "success": false,  "error": "Crawl job not found: No crawl crawl_01m5t4gh4qhgs70zwajezbe5p0 in this project",  "code": "crawl_not_found"}

408 Response

{  "success": false,  "error": "Request Timeout",  "code": "SCRAPE_TIMEOUT"}

429 Response

{  "success": false,  "error": "Rate limit exceeded",  "code": "rate_limited"}

500 Response

{  "success": false,  "error": "Internal Server Error",  "code": "internal"}

502 Response

{  "success": false,  "error": "Bad Gateway: DNS lookup for shop.example failed",  "code": "upstream_unreachable"}

503 Response

{  "success": false,  "error": "Service Unavailable",  "code": "node_unavailable"}

POST /compat/firecrawl/v2/scrape

Base URL

Bearer Token

Request Body required application/json

Response

Was this page helpful?

Previous Get a request

Next Start a batch scrape (Firecrawl)