Traces, evals and alerts (design preview)

B3OS serves the model, the browsers, the sandboxes and the routines your agent calls, so it already records every call in the request log, with its timing and its trace id. A trace is every call that carried one W3C trace id. Make each turn of your agent one trace, and you can read the turn back: what it called, in what order, how long each call took, and what it used.

One trace per turn

Wrap a turn in traces.turn. Every B3OS call inside it, across await, joins one trace:

ts
await b3os.traces.turn(  "answer",  async (turn) => {    const page = await b3os.vision.scrape({ url: "https://example.com" });    console.log(turn.traceId, page.url);  },  { conversation: "thread_42", subject: "u_8812" },);

python
with b3os.traces.turn("answer", conversation="thread_42", subject="u_8812") as turn:    page = b3os.vision.scrape({"url": "https://example.com"})print(turn.trace_id)

Each call in the turn sends traceparent with the turn's trace id, tracestate: b3os=turn, and a baggage header with the turn's name, conversation, subject and metadata. A call outside any turn gets a trace of its own, marked tracestate: b3os=call, so B3OS never mistakes one SDK call for a turn. A traceparent you send yourself is kept as is and carries no mark: that trace is yours, and it counts as a turn too.

• subject is the end user the turn is for. Use an opaque id, never an email. At most 128 characters.
• metadata takes up to 16 keys of at most 64 characters, with values of at most 512.
• turn.headers() gives traceparent and tracestate for another client, such as an OpenAI SDK pointed at B3OS. turn.headers({ baggage: true }) adds the baggage; send that only to B3OS, since it holds your end user's id. b3os.traces.wrapFetch() adds the headers to every request a model SDK sends inside the turn, and adds the baggage only for B3OS hosts.
• turn.env() gives TRACEPARENT and TRACESTATE for a command you run in another sandbox (a sandbox your agent hands work to), so its calls join the turn.
• traces.span(name, fn) times a step of your own code inside the turn. The SDK keeps spans on the turn and passes each to the client's onSpan option (on_span in Python). It sends them nowhere yet.

SDK versions up to 0.1.3 send no mark. B3OS treats their traces, recognized by their User-Agent, as SDK traces rather than turns.

Read a turn back

GET /v1/traces/{trace_id} returns the trace's calls in start order, each with its status, timing phases and the usage it was charged, and the tags of the sandboxes that made them (b3os.app, b3os.deploy, b3os.item). It needs platform:read.

bash
curl -s "https://beta-api.b3os.org/v1/traces/$TRACE_ID" -H "Authorization: Bearer $B3OS_API_KEY"

b3os.traces.get(traceId) in TypeScript and b3os.traces.get(trace_id) in Python call the same route. harness_ms is the time inside the turn when no B3OS call was in flight: the time your own code took between calls.

In the console, an application's Traces tab lists its services' turns. Choosing one draws its calls as a waterfall, with the time no call was in flight shaded, and lights the application's canvas: each node the turn called shows its order and its time. Calls with no shared trace are on a timeline under it.

What the first version cannot do

• Calls are listed flat. A call B3OS made inside one of yours, such as a routine step's scrape, sits inside that call's time and has no row of its own.
• The request log keeps the trace id, not the tracestate mark, so a trace of one call shows as a single call, not a turn.
• A turn is read from the last 300 calls of its trace, and the request log keeps 7 days.
• The model and the pack a call used are not in the request log, so the canvas lights them only when the application has one model or one pack.
• Prompts and completions are never stored. Prompt logging comes later, off by default and opt-in per project.
• A turn's name, conversation and subject are sent but not yet kept or shown. The console names a turn by what it called, and you cannot filter turns by user yet.
• A turn has a link (?trace= on the Traces tab) only while it is among a service's last 50 calls. Read an older one with GET /v1/traces/{trace_id}.

Evals on every pull request

Declare a suite in b3os.yaml and keep its cases in the repository, one JSON object per line (the fields):

json
{"id": "paris", "input": {"question": "What is the capital of France?"}, "expected": {"answer": "Paris"}}

With the GitHub App linked, B3OS runs a suite when a pull request changes its cases, its scorer files or its service's folder, and runs every suite on each push to the linked branch. Each run:

1. Starts the service from the commit under test in a short-lived sandbox in a project of its own, <app>-evals. From the pull request it takes only how the service is built and run: build, command, cwd, env, readiness and the size. Everything else comes from the linked branch's b3os.yaml: the grant and its spend cap, the secrets and resources, the suite's connections, and the egress allowlist (the service's, its build's and the suite's hosts). It never gets a platform, settings, vault-write or prompt-reading scope, and it opens no port. A host the pull request adds is reachable once the change is merged, and the check says which hosts it asked for. The sandbox installs its dependencies itself and is deleted when the run ends.
2. Sends each case's input to the service, one case at a time, with a fresh traceparent.
3. Scores the answer: equals, contains, numeric and json_schema at B3OS; regex and code inside the sandbox; judge through the node's model. A json_schema scorer's schema may not use pattern, patternProperties or format: regex, and it checks an output of at most 64 KiB. Check a pattern with a regex scorer.
4. Counts each case's usage from what the sandbox's calls were charged while the case ran.
5. Compares with the base branch: a stored run of the same suite at the base commit when one exists, otherwise the base's code run beside it, case by case in turn.

The result is a b3os/evals (<app>/<suite>) check on the commit. Its title gives the cases passed of those run and the pass rate, the base's, a drop in points when there is one, and the change in model tokens per case (in cost once units have a price). Each failing case is marked at its line in the cases file with the scorer that failed and the trace id it ran under. A drop of more than gate.pass_rate_drop_max with at least gate.min_failed_cases more failures concludes neutral, which never blocks a merge; gate.block: true makes it failure, which blocks where the repository requires the check. On a public repository the check holds the counts only.

Every suite gets its check on every pull request, so you can make it required:

GitHub counts skipped and neutral as passing a required check. For the last two rows, B3OS names the suites from the linked branch's b3os.yaml and reads nothing of the pull request's code. You cannot start a run by hand yet: b3os evals run and POST /v1/evals/runs are not built.

B3OS keeps each run 90 days with its cases' scores, times, usage and trace ids. It never keeps the outputs, the inputs or a judge's explanation. The console has no page for eval runs yet, so read them from the API; the check's Details link comes here.

bash
curl -s "https://beta-api.b3os.org/v1/evals/runs?app=research-agent&pull_request=12" -H "Authorization: Bearer $B3OS_API_KEY"curl -s "https://beta-api.b3os.org/v1/evals/runs/$EVAL_RUN_ID" -H "Authorization: Bearer $B3OS_API_KEY"

A failing case's trace_id opens what the code did on that case with GET /v1/traces/{trace_id} for 7 days.

Alerts

B3OS raises an alert when a deploy from GitHub fails, when a deploy goes live and then goes wrong, when the lease renewal finds a service crashed or stopped and starts it again or cannot, when a renewal fails, and when an eval gate fails on the linked branch. Name where alerts go in b3os.yaml:

yaml
observe:  notify: [ops-slack]

Then store the destination once, in the project the application deploys to:

bash
curl -s -X PUT "https://beta-api.b3os.org/v1/alerts/destinations/ops-slack" \  -H "Authorization: Bearer $B3OS_API_KEY" -H "Content-Type: application/json" \  -d '{"kind": "slack", "url": "https://hooks.slack.com/services/..."}'

A slack destination takes an incoming webhook URL. A webhook destination takes any https URL, and its deliveries are signed as job webhooks are, with a secret B3OS returns once when you store it (rotate: true makes a new one). B3OS never returns a stored URL. POST /v1/alerts/destinations/{name}/test sends a test alert, at most 5 a minute per project, and GET /v1/alerts lists the alerts of the last 90 days with where each went. One kind of alert about one service is sent at most every 30 minutes.

The plan says when notify names a destination the project does not store ("ops-slack is not stored in this project, so alerts to it are dropped"), as a ? that does not stop the deploy. The application's Settings tab shows where alerts go, the last one sent, and a button that sends a test. Drop notify from the linked branch's file and alerts stop going anywhere.

Errors