Scrape a page

POST /v1/vision/scrape fetches one URL and returns the formats you ask for. It is synchronous: the response arrives when the page is done, within timeout_ms (30 seconds by default, 60 at most).

Scrape one page

bash
curl -s https://beta-api.b3os.org/v1/vision/scrape \  -H "Authorization: Bearer $B3OS_API_KEY" \  -H "Content-Type: application/json" \  -d '{"url": "https://example.com", "formats": ["markdown", "links"]}'

ts
import { B3OS } from "@b3os/sdk";const b3os = new B3OS(); // reads B3OS_API_KEYconst page = await b3os.vision.scrape({  url: "https://example.com",  formats: ["markdown", "links"],});console.log(page.tier, page.cache.state);console.log(page.markdown);

python
from b3os import B3OSb3os = B3OS()  # reads B3OS_API_KEYpage = b3os.vision.scrape({"url": "https://example.com", "formats": ["markdown", "links"]})print(page["tier"], page["cache"]["state"])print(page["markdown"])

text
Scrape https://example.com and give me the page as Markdown, plus its links.

The response holds each format you asked for, plus what served the page and how long it took:

Formats

Main content only

By default vision drops headers, navigation, footers and sidebars (only_main_content: true). Turn it off to keep the whole page, or steer it with CSS selectors:

json
{  "url": "https://docs.example.com/guides/keys",  "formats": ["markdown"],  "include_tags": ["main", "article"],  "exclude_tags": [".cookie-banner", "#feedback"]}

include_tags keeps only matching elements; exclude_tags drops matching ones. Up to 100 selectors each.

Choose the engine

With the browser tier you can also set wait_for_ms (extra settle time after the page is ready, up to 30 seconds), viewport, mobile: true to emulate a phone, locale (default en-US) and timezone. block_ads (on by default) blocks ads, trackers and cookie banners, and block_resources lists resource types not to download: images, media and fonts by default unless you ask for a screenshot.

Act before you capture

actions runs browser actions before the page is captured, up to 50 of them. Use them to dismiss a dialog, switch a tab or wait for content:

json
{  "url": "https://app.example.com/pricing",  "formats": ["markdown", "screenshot"],  "actions": [    { "kind": "click", "target": { "role": "button", "name": "Accept cookies" } },    { "kind": "click", "target": { "text": "Annual", "exact": true } },    { "kind": "wait", "for": { "selector": "[data-plan]" } }  ],  "screenshot": { "format": "jpeg", "quality": 80 }}

A target finds its element by CSS selector, visible text (a case-insensitive substring unless exact), or ARIA role plus accessible name, optionally within a scope and picking the nth match. If no element matches in time, the scrape fails with target_not_found. The action kinds are listed on browser sessions; scrapes accept all of them.

Use a profile

Pass a profile ID or name to load its cookies and storage before navigating, so the scrape sees the page as a signed-in user:

json
{ "url": "https://app.example.com/dashboard", "profile": "acme-staging", "formats": ["markdown"] }

A profile implies the browser tier and keeps the result out of the shared cache. See profiles.

The cache

Results of requests with custom headers, a profile or actions are never shared with other organizations.

Structured data

The json format asks the model on the node to fill a JSON Schema from the page:

json
{  "url": "https://example.com/pricing",  "formats": ["json"],  "json": {    "schema": {      "type": "object",      "properties": { "plans": { "type": "array", "items": { "type": "string" } } },      "required": ["plans"]    },    "prompt": "The name of each plan."  }}

The result is in json. To extract from up to 10 pages as one job, an extraction endpoint is planned: see the beta contract.

One-line reads

For a quick read from a shell or a prompt, put the URL after /v1/vision/r/, unencoded. The answer is the page itself, Markdown by default:

bash
curl -s "https://beta-api.b3os.org/v1/vision/r/https://example.com/pricing?plan=team" \  -H "Authorization: Bearer $B3OS_API_KEY" \  -H "X-B3OS-Engine: fetch"

The target keeps its own path and query (?plan=team here belongs to example.com). It returns text/markdown by default. Options go in headers so the target URL stays untouched: X-B3OS-Format (markdown, html, text or json), X-B3OS-Engine, X-B3OS-Timeout-Ms, X-B3OS-Max-Age-Ms, X-B3OS-Only-Main-Content and X-B3OS-Wait-For-Ms. It uses the same tiers, cache and billing as a scrape.

When it fails

The full list is on the errors page.