Read a URL as Markdown

The shortest way to read a page: put the target URL after `/v1/vision/r/`, unencoded, for example `GET /v1/vision/r/https://example.com/pricing`. Returns `text/markdown` by default. Options go in `X-B3OS-*` headers so the target URL stays untouched. Same tiers, cache and billing as `scrape`.

GET /v1/vision/r/{url}
The shortest way to read a page: put the target URL after `/v1/vision/r/`, unencoded, for example `GET /v1/vision/r/https://example.com/pricing`. Returns `text/markdown` by default. Options go in `X-B3OS-*` headers so the target URL stays untouched. Same tiers, cache and billing as `scrape`.

Authentication

Bearer Token (b3os_sk_ API key or b3os_pat_ token) API Key (cookie: __Host-b3os_session)

Path Parameters

url string required path
The absolute `http` or `https` URL to read, including its own path and query, not percent-encoded.
Example: "https://example.com/pricing"

Header Parameters

X-B3OS-Format string optional header
Response format.
X-B3OS-Engine string optional header
Which tier may serve the page (see `engine` on scrape).
X-B3OS-Timeout-Ms integer optional header
Timeout in milliseconds, 1000 to 60000.
X-B3OS-Max-Age-Ms integer optional header
Accept a cached copy up to this old; `0` bypasses the cache.
X-B3OS-Only-Main-Content boolean optional header
`false` keeps headers, navigation and footers.
X-B3OS-Wait-For-Ms integer optional header
Extra wait after the page is ready, in milliseconds.

Responses

200 The page in the requested format. `X-B3OS-Cache` and `X-B3OS-Tier` report what served it.
application/json
object string REQUIRED
Enum: scrape
url string (uri) REQUIRED
The requested URL.
final_url string (uri) REQUIRED
The URL after redirects.
status_code integer REQUIRED
The HTTP status the site returned. A 404 page is still a successful scrape.
content_type string
tier string REQUIRED
What served the page.
Enum: cache, fetch, browser
cache object REQUIRED
state string REQUIRED
`hit` served from cache; `miss` fetched and stored; `bypass` fetched and not shared (custom headers, profile, actions).
Enum: hit, miss, bypass
age_s integer REQUIRED
Age of the served copy in seconds.
markdown string
html string
raw_html string
text string
links string[]
Array of: string
screenshot object
format string REQUIRED
Enum: jpeg, png, webp
width integer REQUIRED
height integer REQUIRED
data_base64 string (byte) REQUIRED
json unknown
Data matching `json.schema`, when the `json` format was requested.
metadata object
Metadata read from the page.
title string
description string
language string
canonical_url string (uri)
favicon string (uri)
og object
Open Graph properties, keyed without the `og:` prefix.
published_time string
modified_time string
robots string
The page's robots meta tag.
actions object[]
Results of `actions`, in order.
Array of:
kind string REQUIRED
Enum: navigate, click, hover, type, key, scroll, select, wait, evaluate, screenshot, markdown
status string REQUIRED
Enum: ok, failed, skipped
duration_ms number REQUIRED
url string
`navigate`: the URL after redirects.
status_code integer
`navigate`: the HTTP status.
value unknown
`evaluate`: the JSON result.
screenshot object
format string REQUIRED
Enum: jpeg, png, webp
width integer REQUIRED
height integer REQUIRED
data_base64 string (byte) REQUIRED
markdown string
`markdown`: the page as Markdown.
error object
An RFC 9457 problem. `code` is stable and listed in the errors catalog; `type` links to its entry. `detail` is human-readable and may change. Extension members appear only for the codes that document them; ignore members you do not recognize.
type string (uri) REQUIRED
`https://docs.beta.b3os.org/errors#<code>`.
title string REQUIRED
Short summary of the problem type.
status integer REQUIRED
The HTTP status code, repeated for convenience.
code string REQUIRED
Stable machine-readable error code from the errors catalog.
detail string REQUIRED
Explanation of this occurrence. Never contains secrets.
instance string
The request ID (`req_…`) of the request that failed. Include it when you contact support.
errors object[]
Validation problems, one per field (`invalid_request` and the other codes that list `errors`).
Array of:
pointer string REQUIRED
Where the problem is: a JSON pointer into the request body (`/url`, `/steps/0/uses`), or `#query/<name>`, `#path/<name>` or `#header/<name>` for parameters.
detail string REQUIRED
What is wrong there.
reason string
Finer-grained cause for codes that document one (`unauthorized`, `forbidden`, `egress_denied`, `upstream_unreachable`, …).
required_scope string
The scope that was missing (`forbidden` with `reason: scope`).
retry_after_s integer
Same value as the `Retry-After` header, for clients that only read the body.
limit object
The limit an error refers to. IDs and current values are listed by `GET /v1/limits`.
id string REQUIRED
Limit ID, such as `vision.scrape.rpm` or `sandboxes.concurrent`.
limit number REQUIRED
The limit's value.
unit string REQUIRED
What the limit counts, such as `requests`, `sandboxes`, `vcpu` or `MiB`.
window_s integer
Window length in seconds for rate limits and quotas.
current number
Current usage against the limit, when known.
current_status string
The object's current status, for state conflicts such as `sandbox_not_running`.
upstream_status integer
The HTTP status the upstream returned (`upstream_error`, `action_output_invalid`).
step string
The failed step, as a step ID (routines), an index (`act`) or a path such as `beats/2/do/0` (render).
pack_error object
The error a pack's error mapping chose (`upstream_error` from pack actions).
code string REQUIRED
message string REQUIRED
retryable boolean REQUIRED
warnings string[] REQUIRED
Non-fatal problems, such as a format that could not be produced.
Array of: string
timing object REQUIRED
Server-side phase durations in milliseconds with microsecond resolution: the same phases as the `Server-Timing` header, keyed `<phase>_ms`. `total_ms` is always present. Phase names are product-specific and documented per operation. Example values in this reference are illustrative, not measurements.
total_ms number REQUIRED
400 The request failed validation (`invalid_request`, `invalid_url`).
401 Missing, malformed, expired or revoked credentials (`unauthorized`).
402 The organization has no credits left, or the key reached its spend cap.
403 The egress guard refused the URL (`egress_denied`), or robots.txt disallows it (`robots_disallowed`).
404 The object does not exist or belongs to another project.
429 A rate, concurrency or quota limit applies (`rate_limited`, `concurrency_limited`, `quota_exceeded`).
500 Something failed inside B3OS (`internal`). Retry with backoff.
502 The page could not be fetched or loaded (`navigation_failed`, `upstream_unreachable`, `too_many_redirects`).
503 The node or the product is not available right now (`node_unavailable`, `product_unavailable`, `capacity_unavailable`).
504 The page did not finish within `timeout_ms` (`scrape_timeout`). 
curl -X GET 'https://beta-api.b3os.org/v1/vision/r/https://example.com/pricing' \  -H 'Authorization: Bearer YOUR_API_TOKEN'
const response = await fetch('https://beta-api.b3os.org/v1/vision/r/https://example.com/pricing', {  method: 'GET',  headers: {      "Authorization": "Bearer YOUR_API_TOKEN"  }});const data = await response.json();console.log(data);
import requestsheaders = {    'Authorization': 'Bearer YOUR_API_TOKEN'}response = requests.get('https://beta-api.b3os.org/v1/vision/r/https://example.com/pricing', headers=headers)print(response.json())
200 Response 
{  "object": "scrape",  "url": "https://example.com/pricing",  "final_url": "https://example.com/pricing",  "status_code": 200,  "content_type": "text/html; charset=utf-8",  "tier": "cache",  "cache": {    "state": "hit",    "age_s": 3605  },  "markdown": "# Pricing\n\nStart free, then pay per page.\n",  "warnings": [],  "timing": {    "cache_ms": 0.8,    "total_ms": 1.1  }}
Previous Map a site's URLs
Next Extract structured data
Ask a question... ⌘I