Render a page, read it, act on what it read

Inside the sandbox (Node 24 in the base template):

ts
const API = "http://127.0.0.1:7700";const PAGE = "https://beta.b3os.org/demo/billing/";const post = async (url: string, body: unknown, auth?: string): Promise<any> =>  (    await fetch(url, {      method: "POST",      headers: { "content-type": "application/json", ...(auth ? { authorization: auth } : {}) },      body: JSON.stringify(body),    })  ).json();// 1. render: the page as an imageconst shot = await post(`${API}/v1/render/screenshots`, {  url: PAGE,  viewport: { width: 1280, height: 800 },  format: "jpeg",});// 2. infer: the model reads the imageconst read = await post(  `${process.env.OPENAI_BASE_URL}/chat/completions`,  {    model: process.env.B3OS_MODEL,    temperature: 0,    chat_template_kwargs: { enable_thinking: false },    response_format: {      type: "json_schema",      json_schema: {        name: "pricing_page",        schema: {          type: "object",          properties: { toggle: { type: "string" }, business_monthly: { type: "string" } },          required: ["toggle", "business_monthly"],        },      },    },    messages: [      {        role: "user",        content: [          {            type: "text",            text: "This is a screenshot of a pricing page. Give the visible text of the control that switches the prices to yearly billing, and the Business plan's price exactly as the image shows it.",          },          { type: "image_url", image_url: { url: shot.artifacts[0].url } },        ],      },    ],  },  `Bearer ${process.env.OPENAI_API_KEY}`,);const seen = JSON.parse(read.choices[0].message.content);await fetch(`${API}/v1/render/screenshots/${shot.id}`, { method: "DELETE" }); // read; no need to keep it 7 days// 3. vision: press what the model named, then read the pageconst session = await post(`${API}/v1/vision/sessions`, {});const done = await post(`${API}/v1/vision/sessions/${session.id}/act`, {  actions: [    { kind: "navigate", url: PAGE },    { kind: "click", target: { role: "button", name: seen.toggle } },    { kind: "markdown", only_main_content: true },  ],});await fetch(`${API}/v1/vision/sessions/${session.id}`, { method: "DELETE" });console.log(seen, done.results[2].markdown);

Create the sandbox with the grant, then run the script above in it as agent.mjs:

bash
API=https://beta-api.b3os.orgAUTH="Authorization: Bearer $B3OS_API_KEY"SBX=$(curl -s "$API/v1/sandboxes" -H "$AUTH" -H "Content-Type: application/json" -d '{  "template": "base", "timeout_s": 300,  "api": {"scopes": ["render:run", "render:write", "infer:run", "vision:write", "vision:run"], "spend_cap_usd": 1}}' | jq -r .id)curl -s -X PUT "$API/v1/sandboxes/$SBX/files?path=/workspace/agent.mjs" -H "$AUTH" \  -H "Content-Type: application/octet-stream" --data-binary @agent.mjs > /dev/nullcurl -s "$API/v1/sandboxes/$SBX/exec" -H "$AUTH" -H "Content-Type: application/json" \  -d '{"cmd": "node /workspace/agent.mjs", "timeout_ms": 60000}' | jq -r .stdoutcurl -s -X DELETE "$API/v1/sandboxes/$SBX" -H "$AUTH" | jq .status

The same three calls from outside the node, with your key. Each one then crosses your link and the edge:

python
import jsonimport osimport requestsAPI = "https://beta-api.b3os.org"PAGE = "https://beta.b3os.org/demo/billing/"s = requests.Session()s.headers["Authorization"] = f"Bearer {os.environ['B3OS_API_KEY']}"shot = s.post(f"{API}/v1/render/screenshots", json={    "url": PAGE, "viewport": {"width": 1280, "height": 800}, "format": "jpeg"}).json()read = s.post(f"{API}/v1/infer/chat/completions", json={    "model": "b3os/qwen3.5-9b", "temperature": 0, "chat_template_kwargs": {"enable_thinking": False},    "response_format": {"type": "json_schema", "json_schema": {"name": "pricing_page", "schema": {        "type": "object", "required": ["toggle", "business_monthly"],        "properties": {"toggle": {"type": "string"}, "business_monthly": {"type": "string"}}}}},    "messages": [{"role": "user", "content": [        {"type": "text", "text": "This is a screenshot of a pricing page. Give the visible text of the control that "                                 "switches the prices to yearly billing, and the Business plan's price exactly as "                                 "the image shows it."},        {"type": "image_url", "image_url": {"url": shot["artifacts"][0]["url"]}}]}],}).json()seen = json.loads(read["choices"][0]["message"]["content"])session = s.post(f"{API}/v1/vision/sessions", json={}).json()try:    done = s.post(f"{API}/v1/vision/sessions/{session['id']}/act", json={"actions": [        {"kind": "navigate", "url": PAGE},        {"kind": "click", "target": {"role": "button", "name": seen["toggle"]}},        {"kind": "markdown", "only_main_content": True}]}).json()    print(seen, done["results"][2]["markdown"][:400])finally:    s.delete(f"{API}/v1/vision/sessions/{session['id']}")

On the remote B3OS MCP server, ask your agent:

text
Take a 1280x800 screenshot of https://beta.b3os.org/demo/billing/. Look at the image and tell me which controlswitches the prices to yearly billing and what the Business plan costs as shown. Then open the page in a browsersession, press that control, and tell me the Business price it shows now. Close the session when you are done.

The agent calls render_screenshot, then reads the image itself, then vision_create_session, vision_act and vision_close_session.