A sandboxed agent drives a browser
Code in a sandbox opens a browser session and lets the node's model read and click a page until it answers, every turn timed next to the model.
An agent loop with the model, the browser and the agent on one node. Grant a sandbox infer:run, vision:write and
vision:run, and the code inside it can open a browser session and call the model with no key: the sandbox gets
OPENAI_BASE_URL, a placeholder OPENAI_API_KEY and B3OS_MODEL, and B3OS answers at http://127.0.0.1:7700. The
agent (agent.mjs in the recipe) opens the demo app's billing page and asks the model what the Business plan costs
when billed yearly. The page shows monthly prices first, so the model has to press Annual before it can answer.
It has three tools: read_page (the page's main content as Markdown), click (a control by its visible name) and
answer. Each turn is one model call and one tool call:
jsconst r = await ask(); // tools: read_page, click, answer; tool_choice "required"const call = r.json.choices[0].message.tool_calls[0];if (call.function.name === "click") { const { name } = JSON.parse(call.function.arguments); await post(`${API}/v1/vision/sessions/${sid}/act`, { actions: [{ kind: "click", target: { role: "button", name } }, { kind: "markdown", only_main_content: true }], });}
After the model answers, the agent's own code reads the page (which period is pressed, the Business price) so the answer is checked against the page and not against the model.
Verified 2026-11-02: this recipe's last run passed against https://beta-api.b3os.org, label live-recorded (recorded from production), from commit a5c2b1a7. The recipe harness runs it again, and a failed run takes this page out of the cookbook.
Run it
Each tab creates the sandbox with the grant, copies agent.mjs in, runs it and deletes the sandbox. The agent closes
its browser session itself, also when a turn fails.
bashSBX=$(curl -s https://beta-api.b3os.org/v1/sandboxes \ -H "Authorization: Bearer $B3OS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"template": "base", "timeout_s": 300, "api": {"scopes": ["infer:run", "vision:write", "vision:run"], "spend_cap_usd": 1}}' \ | jq -r .id)curl -s -X PUT "https://beta-api.b3os.org/v1/sandboxes/$SBX/files?path=/tmp/agent.mjs" \ -H "Authorization: Bearer $B3OS_API_KEY" \ -H "Content-Type: application/octet-stream" \ --data-binary @agent.mjscurl -s "https://beta-api.b3os.org/v1/sandboxes/$SBX/exec" \ -H "Authorization: Bearer $B3OS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"cmd": "node /tmp/agent.mjs", "timeout_ms": 60000}' | jq -r .stdoutcurl -s -X DELETE "https://beta-api.b3os.org/v1/sandboxes/$SBX" -H "Authorization: Bearer $B3OS_API_KEY"
tsimport { readFileSync } from "node:fs";const API = "https://beta-api.b3os.org";const auth = { authorization: `Bearer ${process.env.B3OS_API_KEY}` };const json = { ...auth, "content-type": "application/json" };const sbx: any = await ( await fetch(`${API}/v1/sandboxes`, { method: "POST", headers: json, body: JSON.stringify({ template: "base", timeout_s: 300, api: { scopes: ["infer:run", "vision:write", "vision:run"], spend_cap_usd: 1 }, }), })).json();try { await fetch(`${API}/v1/sandboxes/${sbx.id}/files?path=/tmp/agent.mjs`, { method: "PUT", headers: { ...auth, "content-type": "application/octet-stream" }, body: readFileSync("agent.mjs"), }); const exec: any = await ( await fetch(`${API}/v1/sandboxes/${sbx.id}/exec`, { method: "POST", headers: json, body: JSON.stringify({ cmd: "node /tmp/agent.mjs", timeout_ms: 60000 }), }) ).json(); console.log(exec.stdout);} finally { await fetch(`${API}/v1/sandboxes/${sbx.id}`, { method: "DELETE", headers: auth });}
pythonimport osimport requestsAPI = "https://beta-api.b3os.org"s = requests.Session()s.headers["Authorization"] = f"Bearer {os.environ['B3OS_API_KEY']}"sbx = s.post(f"{API}/v1/sandboxes", json={ "template": "base", "timeout_s": 300, "api": {"scopes": ["infer:run", "vision:write", "vision:run"], "spend_cap_usd": 1},}).json()try: with open("agent.mjs", "rb") as f: s.put(f"{API}/v1/sandboxes/{sbx['id']}/files", params={"path": "/tmp/agent.mjs"}, data=f.read(), headers={"Content-Type": "application/octet-stream"}) run = s.post(f"{API}/v1/sandboxes/{sbx['id']}/exec", json={"cmd": "node /tmp/agent.mjs", "timeout_ms": 60000}).json() print(run["stdout"])finally: s.delete(f"{API}/v1/sandboxes/{sbx['id']}")
On the remote B3OS MCP server, sandbox_create takes the same body as the API, api grant included.
Ask your agent:
textCreate a base sandbox that may call the model and use a browser session (infer:run, vision:write, vision:run, spendcap 1 USD), write agent.mjs into /tmp, run it with node, show me its output, then delete the sandbox.
The agent calls sandbox_create, sandbox_write_file, sandbox_exec and sandbox_delete.
model_ms and tool_ms below are each turn's model call and browser action, measured inside the sandbox, on the same
machine as the model and the browser. A buffered exec waits at most 60 seconds; a longer agent streams its output
or runs in the background. The demo app is a fictional workspace with made-up prices.
Owner: Docs. Products: Docs, Sandboxes, Infer, Vision.
Output of the last green run
bash$ node run.mjs# exit 0
json{ "model": "b3os/qwen3.5-9b", "page_url": "https://beta.b3os.org/demo/billing/", "question": "What does the Business plan cost per month when it is billed yearly? The page shows monthly prices first.", "turns": 3, "moves": [ "read_page", "click Annual", "answer" ], "answer_price": "$40 per month", "answer": "The Business plan costs $40 per month when billed yearly.", "page_period": "annual", "page_price": "$40", "read_sha256": "e7b7453fe0d6c482d439cf8ea23096a855f3d25c1c063c5c76f84182605afc4e", "model_ms": [ 193.6, 152.1, 287.2 ], "tool_ms": [ 8, 14.4, 0 ], "session_ms": 40.9, "agent_ms": 912.2, "session_closed": true, "create_ms": 127, "exec_ms": 1042.9, "deleted": true}
The outputs this recipe publishes (its recipe.json publish.outputs), with workspace identifiers redacted. The run's stdout, stderr and other outputs are not published.
Captured 2026-11-02 from commit
a5c2b1a7
, label
live-recorded
(recorded from production). Case
recipes/sandbox-agent-browser
.