Scrape a page with curl

Send a URL and the formats you want back:

bash
curl -s https://beta-api.b3os.org/v1/vision/scrape \  -H "Authorization: Bearer $B3OS_API_KEY" \  -H "Content-Type: application/json" \  -d '{"url": "https://example.com", "formats": ["markdown", "links"]}'

The page comes back as Markdown, with what served it and how long each phase took. Abridged:

jsonc
{  "object": "scrape",  "url": "https://example.com",  "status_code": 200,               // the site's status; a 404 page is still a successful scrape  "tier": "fetch",                  // cache, fetch or browser  "cache": { "state": "miss", "age_s": 0 },  "markdown": "# Example Domain\n\n…",  "links": ["…"],  "warnings": [],  "timing": { "fetch_ms": …, "extract_ms": …, "total_ms": … }}

tier says what served the page: cache, fetch (plain HTTP) or browser (a warm Chrome, used when the page needs JavaScript). cache.state is hit, miss or bypass. A scrape of the same URL within two days can come from the cache, and timing holds the server-side phase durations in milliseconds.

Every response carries a Server-Timing header and an x-request-id. Print the headers instead of the body:

bash
curl -s -o /dev/null -D - https://beta-api.b3os.org/v1/vision/scrape \  -H "Authorization: Bearer $B3OS_API_KEY" \  -H "Content-Type: application/json" \  -d '{"url": "https://example.com"}' | grep -i -E 'server-timing|x-request-id'

The edge's phases (edge_auth, edge_upstream) come first, then the node's (fetch, extract, total). Copy the x-request-id value and ask the request log for the full record:

bash
export REQUEST_ID="req_..."   # the x-request-id value from the headerscurl -s "https://beta-api.b3os.org/v1/requests/$REQUEST_ID" \  -H "Authorization: Bearer $B3OS_API_KEY"

Timing explains each phase.

Put the target URL after /v1/vision/r/ and the page comes back as Markdown:

bash
curl -s https://beta-api.b3os.org/v1/vision/r/https://example.com \  -H "Authorization: Bearer $B3OS_API_KEY"

Options go in headers so the URL stays untouched: X-B3OS-Format picks html, text or json instead, and X-B3OS-Engine: browser forces a browser.