Jobs, streams and webhooks

Crawls, batch scrapes, videos, routine runs and extractions take longer than a request should. They are jobs, and every job works the same way.

The job object

json
{  "id": "crawl_01m5t4gh4qhgs70zwajezbe5p0",  "object": "crawl",  "status": "running",  "created_at": "2026-10-25T18:10:02.114Z",  "updated_at": "2026-10-25T18:10:20.402Z",  "started_at": "2026-10-25T18:10:02.120Z",  "finished_at": null,  "progress": { "completed": 12, "failed": 0, "total": 37, "unit": "pages" },  "expires_at": null,  "metadata": {}}

status moves from queued to running to one of succeeded, failed or canceled, and never leaves a terminal status. progress counts in the job's own unit (pages, phases, steps). Once the job ends, result holds the output on success, error holds a problem on failure or cancel, timing has the final phase durations, and expires_at says when the results are deleted.

Three ways to get the result

Wait in the request. Add ?wait=<seconds> (up to 60) to the POST that creates the job. If it finishes in time you get 200 with the finished job. If not, you get 202 with the job as it is, a Location header to poll and a Retry-After header.

bash
curl -s "https://beta-api.b3os.org/v1/vision/crawls?wait=30" \  -H "Authorization: Bearer $B3OS_API_KEY" \  -H "Content-Type: application/json" \  -d '{"url": "https://docs.example.com", "limit": 20}'

Stream its events. Send Accept: text/event-stream on the creating POST, or open GET .../{id}/events later.

Get a webhook. Pass webhook: {url, events} when you create the job and B3OS posts the events to your URL. See webhooks below.

You can also poll GET on the job with its ID. To stop a job, POST .../{id}/cancel: it ends canceled with the code job_canceled, and work already done stays done and billed.

Event streams

Every stream in B3OS (job events, command output, sandbox logs) uses the same server-sent events framing:

text
id: 14event: page.succeededdata: {"type":"page.succeeded","ts":"2026-10-25T18:10:20.402Z","job_id":"crawl_01m5t4gh4qhgs70zwajezbe5p0","seq":14,"page":{"object":"page_result","url":"https://docs.example.com/guides/keys","status":"succeeded","depth":2}}

• seq goes up by one per event and equals the SSE id. If the connection drops, reconnect with Last-Event-ID: <seq> and the stream resumes after that event.
• A : ping comment arrives every 15 seconds, so idle proxies keep the connection open.
• Lifecycle events are job.queued, job.running, job.progress and the terminal job.succeeded, job.failed and job.canceled. The terminal event carries the final job, timing included, and ends the stream.
• Products add their own events, such as page.succeeded for crawls, step.waiting for routine runs and video.artifact for videos. Each operation in the API reference lists the events it sends.

bash
curl -N -s "https://beta-api.b3os.org/v1/vision/crawls/$CRAWL_ID/events" \  -H "Authorization: Bearer $B3OS_API_KEY"

Webhooks

Pass a webhook when you create a job:

json
{ "url": "https://docs.example.com", "webhook": { "url": "https://example.com/hooks/b3os", "events": ["job.succeeded", "job.failed"] } }

events defaults to the terminal events; ["*"] sends every event. B3OS posts each one as a JSON event (the job is abridged here):

json
{  "id": "evt_01m5t4p5ycda35cq27epnbfqhe",  "object": "event",  "type": "job.succeeded",  "created_at": "2026-10-25T18:10:41.030Z",  "job_id": "crawl_01m5t4gh4qhgs70zwajezbe5p0",  "data": { "job": { "id": "crawl_01m5t4gh4qhgs70zwajezbe5p0", "object": "crawl", "status": "succeeded" } }}

The event id stays the same across retries of one delivery, so use it to drop duplicates. Failed deliveries are retried with backoff for 24 hours. Deliveries leave from the node through the egress guard, so URLs on private or loopback addresses are refused.

Verify the signature

Every delivery carries B3OS-Signature: t=<unix seconds>,v1=<hex>. The signature is HMAC-SHA256 over <t>.<raw body>, keyed with the signing secret of your URL: the UTF-8 bytes of the whole whsec_... string, prefix included, never decoded. Each attempt is signed when it is sent, so a retry hours later carries a fresh t and still passes a five-minute check. Get the secret once, when you set up the receiver; the same URL always gets the same secret. The call needs a key with platform:write, which the first key you get with your invite does not have: create one in the dashboard or with POST /v1/keys first.

bash
curl -s https://beta-api.b3os.org/v1/webhooks/secret \  -H "Authorization: Bearer $B3OS_API_KEY" \  -H "Content-Type: application/json" \  -d '{"url": "https://example.com/hooks/b3os"}'# {"object": "webhook_secret", "url": "https://example.com/hooks/b3os", "secret": "whsec_...", ...}

Then check every delivery against the raw body, before parsing it. The SDKs do it in one call and throw WebhookVerificationError for a delivery that fails; the versions without the SDK return false for anything malformed instead of throwing:

ts
import { verifyWebhook, WebhookVerificationError } from "@b3os/sdk/webhooks";// rawBody is the request body exactly as received, before any JSON parsingtry {  const event = await verifyWebhook(rawBody, request.headers.get("b3os-signature"), process.env.B3OS_WEBHOOK_SECRET ?? "");  console.log(event.type, event.id);} catch (error) {  if (!(error instanceof WebhookVerificationError)) throw error;  // answer 400 and drop the delivery; error.reason says why}

python
import osfrom b3os import WebhookVerificationError, verify_webhook# raw_body is the request body exactly as received, before any JSON parsingtry:    event = verify_webhook(raw_body, headers.get("b3os-signature"), os.environ["B3OS_WEBHOOK_SECRET"])    print(event["type"], event["id"])except WebhookVerificationError as error:    print("rejected:", error.reason)  # answer 400 and drop the delivery

ts
import { createHmac, timingSafeEqual } from "node:crypto";// True only for a delivery signed with `secret` less than five minutes ago. Never throws.export function verifyB3osSignature(  rawBody: string | Buffer,  header: string | null | undefined,  secret: string,): boolean {  if (typeof header !== "string") return false;  let t: number | undefined;  const signatures: Buffer[] = [];  for (const part of header.split(",")) {    const eq = part.indexOf("=");    if (eq < 0) continue;    const key = part.slice(0, eq).trim();    const value = part.slice(eq + 1).trim();    if (key === "t" && /^[0-9]{1,12}$/.test(value)) t = Number(value);    else if (key === "v1" && /^[0-9a-fA-F]{64}$/.test(value)) signatures.push(Buffer.from(value, "hex"));  }  if (t === undefined || Math.abs(Date.now() / 1000 - t) > 300) return false;  const expected = createHmac("sha256", secret).update(`${t}.`).update(rawBody).digest();  return signatures.some((signature) => timingSafeEqual(signature, expected));}

python
import hashlibimport hmacimport reimport timedef verify_b3os_signature(raw_body: bytes, header: str | None, secret: str) -> bool:    """True only for a delivery signed with `secret` less than five minutes ago. Never raises."""    if not isinstance(header, str) or not isinstance(raw_body, bytes):        return False    t = None    signatures = []    for part in header.split(","):        key, sep, value = part.partition("=")        key, value = key.strip(), value.strip()        if not sep:            continue        if key == "t" and re.fullmatch(r"[0-9]{1,12}", value):            t = int(value)        elif key == "v1" and re.fullmatch(r"[0-9a-fA-F]{64}", value):            signatures.append(value.lower())    if t is None or abs(time.time() - t) > 300:        return False    expected = hmac.new(secret.encode(), f"{t}.".encode() + raw_body, hashlib.sha256).hexdigest()    return any(hmac.compare_digest(signature, expected) for signature in signatures)

All four reject a timestamp more than five minutes from your clock, which limits replays (the SDKs take a tolerance option). All four accept a delivery when any v1 value matches: while you rotate a secret with POST /v1/webhooks/secret/rotate, deliveries carry two, one per secret, for the grace period you choose (a day by default). POST /v1/webhooks/test sends a signed webhook.test event so you can check the receiver, and GET /v1/webhooks/deliveries lists recent deliveries with their attempts and last status. Webhooks covers both.

Retry writes safely

Operations that change something accept an Idempotency-Key header. Send a unique value, such as a UUID, and retry with the same key after a timeout or a dropped connection:

bash
curl -s https://beta-api.b3os.org/v1/sandboxes \  -H "Authorization: Bearer $B3OS_API_KEY" \  -H "Idempotency-Key: 6f1c2a0e-4b7d-4e55-9a51-0d8f2c8e7b21" \  -H "Content-Type: application/json" \  -d '{"template": "base"}'

• A retry with the same key, method, path and body within 24 hours returns the stored response with Idempotent-Replayed: true, and only to the same API key.
• The same key with a different request fails with idempotency_key_conflict; a retry while the first request is still running gets idempotency_request_in_progress and should wait Retry-After.
• Keys are scoped to your project. Responses of operations with external effects are stored durably. For creates without external effects, the stored response lives in memory and a node restart forgets it.

The API reference shows which operations take the header.