Prepare a coming scrape

Warms what a scrape of `url` will need, so it is off the scrape's critical path: the destination resolved and checked by the egress guard, and a keep-alive connection to the origin that the scrape's HTTP fetch reuses (opened with a `HEAD` of the URL, whose answer is not used, carrying the request headers the scrape's navigation sends: Chrome's for that URL). Routines call it for the next step while the current one runs. Bills nothing. With `respect_robots: true` it fetches and caches the origin's robots.txt instead of the `HEAD`, and `robots` says whether robots.txt allows the URL (`unreachable`: a scrape still proceeds, a crawl does not); otherwise `robots` is `not_checked`. It sends the target at most one `HEAD` per origin per 60 seconds for your organization: a prepare of an origin you prepared within the last 60 seconds finds the connection pooled and sends nothing (its `timing` then has no `connect_ms`). Each request a prepare does send (the `HEAD`, or a robots.txt this node does not hold yet) counts against `vision.scrape.rpm` like a scrape: `429 rate_limited` when it is used up.

POST /v1/vision/prepare
Warms what a scrape of `url` will need, so it is off the scrape's critical path: the destination resolved and checked by the egress guard, and a keep-alive connection to the origin that the scrape's HTTP fetch reuses (opened with a `HEAD` of the URL, whose answer is not used, carrying the request headers the scrape's navigation sends: Chrome's for that URL). Routines call it for the next step while the current one runs. Bills nothing. With `respect_robots: true` it fetches and caches the origin's robots.txt instead of the `HEAD`, and `robots` says whether robots.txt allows the URL (`unreachable`: a scrape still proceeds, a crawl does not); otherwise `robots` is `not_checked`. It sends the target at most one `HEAD` per origin per 60 seconds for your organization: a prepare of an origin you prepared within the last 60 seconds finds the connection pooled and sends nothing (its `timing` then has no `connect_ms`). Each request a prepare does send (the `HEAD`, or a robots.txt this node does not hold yet) counts against `vision.scrape.rpm` like a scrape: `429 rate_limited` when it is used up.

Authentication

Bearer Token (b3os_sk_ API key or b3os_pat_ token) API Key (cookie: __Host-b3os_session)

Request Body required

application/json
url string (uri) REQUIRED
The page a coming scrape will fetch.
respect_robots boolean
Fetch and cache the origin's robots.txt and say whether it allows `url` (the scrape itself checks only when it sets `respect_robots`).

Responses

200 What was prepared.
application/json
object string REQUIRED
Enum: prepare
url string (uri) REQUIRED
robots string REQUIRED
`not_checked` unless the request set `respect_robots`.
Enum: allowed, disallowed, unreachable, not_checked
timing object REQUIRED
Server-side phase durations in milliseconds with microsecond resolution: the same phases as the `Server-Timing` header, keyed `<phase>_ms`. `total_ms` is always present. Phase names are product-specific and documented per operation. Example values in this reference are illustrative, not measurements.
total_ms number REQUIRED
400 The request failed validation (`invalid_request`, `invalid_url`).
401 Missing, malformed, expired or revoked credentials (`unauthorized`).
403 The egress guard refused the URL (`egress_denied`), or, with `respect_robots`, robots.txt disallows it (`robots_disallowed`).
413 The request body exceeds this operation's limit.
415 The Content-Type is not accepted by this operation.
429 A rate, concurrency or quota limit applies (`rate_limited`, `concurrency_limited`, `quota_exceeded`).
500 Something failed inside B3OS (`internal`). Retry with backoff.
502 The destination's host does not resolve (`upstream_unreachable`, reason `dns`).
503 The node or the product is not available right now (`node_unavailable`, `product_unavailable`, `capacity_unavailable`). 
curl -X POST 'https://beta-api.b3os.org/v1/vision/prepare' \  -H 'Authorization: Bearer YOUR_API_TOKEN' \  -H 'Content-Type: application/json' \  -d '{  "url": "https://docs.example.com/guides/auth"}'
const response = await fetch('https://beta-api.b3os.org/v1/vision/prepare', {  method: 'POST',  headers: {      "Authorization": "Bearer YOUR_API_TOKEN",      "Content-Type": "application/json"  },  body: JSON.stringify({    "url": "https://docs.example.com/guides/auth"  })});const data = await response.json();console.log(data);
import requestsheaders = {    'Authorization': 'Bearer YOUR_API_TOKEN'}response = requests.post('https://beta-api.b3os.org/v1/vision/prepare', headers=headers, json={  "url": "https://docs.example.com/guides/auth"})print(response.json())
200 Response 
{  "object": "prepare",  "url": "https://docs.example.com/guides/auth",  "robots": "not_checked",  "timing": {    "admit_ms": 0.1,    "resolve_ms": 3.1,    "connect_ms": 41.7,    "total_ms": 44.9  }}
Previous Scrape a page
Next Map a site's URLs
Ask a question... ⌘I