> ## Documentation Index
> Fetch the complete documentation index at: https://docs.beta.b3os.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Scrape a page

Fetch one URL as Markdown, HTML, text, links, metadata or a screenshot, with main-content extraction, browser actions, profiles and caching.

`POST /v1/vision/scrape` fetches one URL and returns the formats you ask for. It is synchronous: the response arrives when the page is done, within `timeout_ms` (30 seconds by default, 60 at most).

## Scrape one page

**curl**

```bash
    curl -s https://beta-api.b3os.org/v1/vision/scrape \
      -H "Authorization: Bearer $B3OS_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"url": "https://example.com", "formats": ["markdown", "links"]}'
    ```
  **TypeScript**

```ts
    import { B3OS } from "@b3os/sdk";

    const b3os = new B3OS(); // reads B3OS_API_KEY
    const page = await b3os.vision.scrape({
      url: "https://example.com",
      formats: ["markdown", "links"],
    });
    console.log(page.tier, page.cache.state);
    console.log(page.markdown);
    ```
  **Python**

```python
    from b3os import B3OS

    b3os = B3OS()  # reads B3OS_API_KEY
    page = b3os.vision.scrape({"url": "https://example.com", "formats": ["markdown", "links"]})
    print(page["tier"], page["cache"]["state"])
    print(page["markdown"])
    ```
  **MCP**

With the [B3OS MCP server](/sdks/mcp) connected, ask your agent:

    ```text
    Scrape https://example.com and give me the page as Markdown, plus its links.
    ```

    The agent calls the `vision_scrape` tool with `url` and `formats`, the same fields as the API body.

The response holds each format you asked for, plus what served the page and how long it took:

| Field | Meaning |
|---|---|
| `url`, `final_url` | The URL you asked for and the URL after redirects |
| `status_code` | The site's HTTP status. A 404 page is still a successful scrape. |
| `tier` | `cache`, `fetch` or `browser` |
| `cache` | `state` (`hit`, `miss` or `bypass`) and `age_s` |
| `markdown`, `html`, `raw_html`, `text`, `links`, `screenshot`, `json` | The formats you asked for |
| `metadata` | Title, description, language, canonical URL, favicon, Open Graph fields, dates and the robots meta tag; always included |
| `warnings` | Problems that did not stop the scrape, such as a format that could not be produced |
| `timing` | Server-side phase durations in milliseconds |

## Formats

| Format | What you get |
|---|---|
| `markdown` | The main content as Markdown (the default) |
| `html` | Cleaned HTML |
| `raw_html` | The HTML as the site sent it |
| `text` | Plain text |
| `links` | Every link on the page, as absolute URLs |
| `metadata` | Always included; listed for compatibility |
| `screenshot` | An image of the page; needs the browser tier |
| `json` | Data shaped by your JSON Schema in `json.schema`, filled in by the model on the node |

## Main content only

By default vision drops headers, navigation, footers and sidebars (`only_main_content: true`). Turn it off to keep the whole page, or steer it with CSS selectors:

```json
{
  "url": "https://docs.example.com/guides/keys",
  "formats": ["markdown"],
  "include_tags": ["main", "article"],
  "exclude_tags": [".cookie-banner", "#feedback"]
}
```

`include_tags` keeps only matching elements; `exclude_tags` drops matching ones. Up to 100 selectors each.

## Choose the engine

| `engine` | Behavior |
|---|---|
| `auto` (default) | Cache, then a plain fetch, then a browser only when the page needs JavaScript |
| `fetch` | Never starts a browser. Use it for pages rendered on the server. |
| `browser` | Always renders in Chrome. Implied by `actions`, `screenshot` and `profile`. |

With the browser tier you can also set `wait_for_ms` (extra settle time after the page is ready, up to 30 seconds), `viewport`, `mobile: true` to emulate a phone, `locale` (default `en-US`) and `timezone`. `block_ads` (on by default) blocks ads, trackers and cookie banners, and `block_resources` lists resource types not to download: images, media and fonts by default unless you ask for a screenshot.

## Act before you capture

`actions` runs browser actions before the page is captured, up to 50 of them. Use them to dismiss a dialog, switch a tab or wait for content:

```json
{
  "url": "https://app.example.com/pricing",
  "formats": ["markdown", "screenshot"],
  "actions": [
    { "kind": "click", "target": { "role": "button", "name": "Accept cookies" } },
    { "kind": "click", "target": { "text": "Annual", "exact": true } },
    { "kind": "wait", "for": { "selector": "[data-plan]" } }
  ],
  "screenshot": { "format": "jpeg", "quality": 80 }
}
```

A target finds its element by CSS `selector`, visible `text` (a case-insensitive substring unless `exact`), or ARIA `role` plus accessible `name`, optionally `within` a scope and picking the `nth` match. If no element matches in time, the scrape fails with [`target_not_found`](/errors#target_not_found). The action kinds are listed on [browser sessions](/vision/sessions#browser-actions); scrapes accept all of them.

## Use a profile

Pass a profile ID or name to load its cookies and storage before navigating, so the scrape sees the page as a signed-in user:

```json
{ "url": "https://app.example.com/dashboard", "profile": "acme-staging", "formats": ["markdown"] }
```

A profile implies the browser tier and keeps the result out of the shared cache. See [profiles](/vision/profiles).

## The cache

| Field | Default | Effect |
|---|---|---|
| `max_age_ms` | 172800000 (two days) | Accept a cached copy up to this old. `0` always fetches. |
| `store_in_cache` | `true` | Store this result for later scrapes |
| `headers` | none | Extra request headers. Any custom header keeps the result out of the shared cache. |

Results of requests with custom headers, a profile or actions are never shared with other organizations.

## Structured data

The `json` format asks the model on the node to fill a JSON Schema from the page:

```json
{
  "url": "https://example.com/pricing",
  "formats": ["json"],
  "json": {
    "schema": {
      "type": "object",
      "properties": { "plans": { "type": "array", "items": { "type": "string" } } },
      "required": ["plans"]
    },
    "prompt": "The name of each plan."
  }
}
```

The result is in `json`. To extract from up to 10 pages as one job, an extraction endpoint is planned: see the [beta contract](/concepts/beta#changes).

## One-line reads

For a quick read from a shell or a prompt, put the URL after `/v1/vision/r/`, unencoded. The answer is the page itself, Markdown by default:

```bash
curl -s "https://beta-api.b3os.org/v1/vision/r/https://example.com/pricing?plan=team" \
  -H "Authorization: Bearer $B3OS_API_KEY" \
  -H "X-B3OS-Engine: fetch"
```

The target keeps its own path and query (`?plan=team` here belongs to example.com). It returns `text/markdown` by default. Options go in headers so the target URL stays untouched: `X-B3OS-Format` (`markdown`, `html`, `text` or `json`), `X-B3OS-Engine`, `X-B3OS-Timeout-Ms`, `X-B3OS-Max-Age-Ms`, `X-B3OS-Only-Main-Content` and `X-B3OS-Wait-For-Ms`. It uses the same tiers, cache and billing as a scrape.

## When it fails

| Code | Status | Meaning |
|---|---|---|
| [`scrape_timeout`](/errors#scrape_timeout) | 504 | The page did not load and convert within `timeout_ms`. Raise it, or use `engine: "fetch"`. |
| [`navigation_failed`](/errors#navigation_failed) | 502 | The browser could not load the page; `reason` carries the browser's error. |
| [`robots_disallowed`](/errors#robots_disallowed) | 403 | robots.txt or Content Signals disallow the URL. |
| [`content_type_unsupported`](/errors#content_type_unsupported) | 422 | The URL returned something vision cannot convert, such as a video. |
| [`target_not_found`](/errors#target_not_found) | 422 | A browser action's target never appeared. |
| [`egress_denied`](/errors#egress_denied) | 403 | The URL resolves to an address the egress guard refuses. |
| [`rate_limited`](/errors#rate_limited) | 429 | More than your scrapes per minute. |

The full list is on the [errors page](/errors#vision).