A narrated app tour, scripted from what vision saw

bash
API=https://beta-api.b3os.orgAUTH="Authorization: Bearer $B3OS_API_KEY"PAGE=https://beta.b3os.org/demo/MAIN='{"kind": "snapshot", "mode": "full", "root": "main", "max_tokens": 800}'post() { curl -s "$API$1" -H "$AUTH" -H "Content-Type: application/json" -d @-; }act() { jq -n --argjson a "$1" '{actions: $a, continue_on_error: true}' | post "/v1/vision/sessions/$SID/act"; }lines() { sed -E 's/ \[ref=[^]]*\]//g; s/ \[selected\]//g; s/^ +//' | grep -v '^$'; }PASSWORD='{"kind": "evaluate", "expression": "document.querySelector(\"input[type=password]\") !== null"}'# 1. vision: the page's outline and content, then what each tab, button and link showsSID=$(echo '{}' | post /v1/vision/sessions | jq -r .id)act "[{\"kind\": \"navigate\", \"url\": \"$PAGE\"}, {\"kind\": \"snapshot\", \"mode\": \"skeleton\"}, $MAIN]" > first.jsonjq -r '.results[2].snapshot.text' first.json | lines > home.txtjq -r '.results[1].snapshot.text' first.json | sed -E 's/ \[ref=[^]]*\]//g' \| grep -oE -- '- (tab|button|link) "[^"]+"( \[(selected|disabled)\])?' | grep -vE '\[(selected|disabled)\]' \| sed -E 's/^- (tab|button|link) "(.*)"$/click \1 \2/' | awk '!seen[$0]++' > candidates.txt: > moves.txt: > views.txtwhile read -r kind role name; do  act "$(jq -n --arg url "$PAGE" --arg role "$role" --arg name "$name" --argjson pw "$PASSWORD" --argjson main "$MAIN" \    '[{kind: "navigate", url: $url}, {kind: "click", target: {role: $role, name: $name}},      {kind: "wait", for: {network_idle: true}}, $pw, $main]')" \  | jq -r 'select(.ok and .results[3].value == false) | .results[4].snapshot.text // empty' | lines > view.txt  if [ "$(grep -cvxFf home.txt view.txt)" -ge 2 ]; then   # two lines the page does not show, and no password field    echo "$kind $role $name" >> moves.txt    { echo; echo "Move \"$kind $role $name\" then shows:"; head -16 view.txt; } >> views.txt  fidone < candidates.txtcurl -s -X DELETE "$API/v1/vision/sessions/$SID" -H "$AUTH" > /dev/null# 2. infer: the script, its moves held to the qualifying ones{ echo "Page: $PAGE"; echo "What the page shows:"; head -20 home.txt; cat views.txt; } > read.txtjq -R . moves.txt | jq -s . > moves.jsonN=$(jq 'if length > 3 then 3 else length end' moves.json)jq -n --rawfile read read.txt --slurpfile moves moves.json --argjson n "$N" '{  model: "b3os/qwen3.5-9b", temperature: 0, max_tokens: 600, chat_template_kwargs: {enable_thinking: false},  response_format: {type: "json_schema", json_schema: {name: "tour", strict: true, schema: {    type: "object", additionalProperties: false, required: ["intro", "beats"], properties: {      intro: {type: "string", maxLength: 160},      beats: {type: "array", minItems: $n, maxItems: $n, items: {        type: "object", additionalProperties: false, required: ["title", "say", "move"], properties: {          title: {type: "string", maxLength: 30}, say: {type: "string", maxLength: 160},          move: {type: "string", enum: $moves[0]}}}}}}}},  messages: [    {role: "system", content: "You script a short narrated tour of a web app for a product video. Write an opening line (at most 16 words) about what the page shows, then \($n) beats. Each beat has a title of two or three words, one sentence of at most 18 words, and one move; use \($n) different moves, and put moves on this page (tabs, buttons) before links, which open other pages. A beat'"'"'s sentence describes what its move shows, using only the lines given for that move. Plain words, no exclamation marks."},    {role: "user", content: $read}]}' \| post /v1/infer/chat/completions \| jq '.choices[0].message.content | fromjson | .beats |= sort_by(.move | startswith("click link "))' > script.jsonjq -r '.intro, (.beats[] | "\(.move): \(.say)")' script.json# 3. render: the beats as they are, a voice and captionsjq --arg page "$PAGE" '{  source: {url: $page},  beats: ([{title: "Overview", say: .intro}] + [.beats[] | {title, say, do: [.move | split(" ") as $m    | {kind: "click", target: {role: $m[1], name: ($m[2:] | join(" "))}}]}]),  voice: {model: "kokoro-82m", voice: "af_heart"},  captions: {burn_in: "karaoke", sidecars: ["vtt"]},  style: {background: "midnight", frame: "browser-dark", cursor: "arrow", captions: "default", click_effect: "ripple"},  output: {width: 1280, height: 720, fps: 30}, formats: ["mp4", "poster"]}' script.json \| post "/v1/render/videos?wait=60" > video.jsonjq -r '.status, .result.duration_ms' video.jsonjq -r '.result.artifacts[] | select(.kind == "mp4") | .url' video.json | xargs curl -s -o tour.mp4

wait=60 holds the request until the video is done, up to 60 seconds; a longer one answers running, and GET /v1/render/videos/{id} reports it until it is succeeded. The recipe's run.mjs also plays the moves in order in vision before it records.

ts
const API = "https://beta-api.b3os.org";const PAGE = "https://beta.b3os.org/demo/";const MAIN = { kind: "snapshot", mode: "full", root: "main", max_tokens: 800 };const PASSWORD = { kind: "evaluate", expression: "document.querySelector('input[type=password]') !== null" };const headers = { authorization: `Bearer ${process.env.B3OS_API_KEY}`, "content-type": "application/json" };const call = async (method: string, path: string, body?: unknown): Promise<any> =>  (await fetch(`${API}${path}`, { method, headers, body: body === undefined ? undefined : JSON.stringify(body) })).json();const lines = (text: string) =>  text    .replace(/ \[ref=[^\]]*\]/g, "")    .split("\n")    .map((l) => l.replace(/ \[selected\]/g, "").trim())    .filter(Boolean);const target = (move: string) => {  const [, role, ...name] = move.split(" ");  return { role, name: name.join(" ") };};// 1. vision: the page's outline and content, then what each tab and link showsconst session = await call("POST", "/v1/vision/sessions", {});const act = (actions: unknown[]) =>  call("POST", `/v1/vision/sessions/${session.id}/act`, { actions, continue_on_error: true });const first = await act([{ kind: "navigate", url: PAGE }, { kind: "snapshot", mode: "skeleton" }, MAIN]);const home = lines(first.results[2].snapshot.text);const candidates = [  ...new Set(    [...first.results[1].snapshot.text.matchAll(/- (tab|button|link) "([^"]+)"(.*)/g)]      .filter((m: RegExpMatchArray) => !(m[1] === "tab" && (m[3] ?? "").includes("[selected]")))      .filter((m: RegExpMatchArray) => !(m[3] ?? "").includes("[disabled]"))      .map((m: RegExpMatchArray) => `click ${m[1]} ${m[2]}`),  ),] as string[];const moves: { move: string; view: string[] }[] = [];for (const move of candidates) {  const r = await act([    { kind: "navigate", url: PAGE },    { kind: "click", target: target(move) },    { kind: "wait", for: { network_idle: true } },    PASSWORD,    MAIN,  ]);  if (!r.ok || r.results[3].value !== false) continue; // never a page with a password field  const view = lines(r.results[4].snapshot.text);  if (view.filter((l) => !home.includes(l)).length >= 2) moves.push({ move, view: view.slice(0, 16) });}await call("DELETE", `/v1/vision/sessions/${session.id}`);const n = Math.min(3, moves.length);// 2. infer: the script, its moves held to the qualifying onesconst read = [  `Page: ${PAGE}`,  "What the page shows:",  ...home.slice(0, 20),  ...moves.flatMap((m) => ["", `Move "${m.move}" then shows:`, ...m.view]),].join("\n");const chat = await call("POST", "/v1/infer/chat/completions", {  model: "b3os/qwen3.5-9b",  temperature: 0,  max_tokens: 600,  chat_template_kwargs: { enable_thinking: false },  response_format: {    type: "json_schema",    json_schema: {      name: "tour",      strict: true,      schema: {        type: "object",        additionalProperties: false,        required: ["intro", "beats"],        properties: {          intro: { type: "string", maxLength: 160 },          beats: {            type: "array",            minItems: n,            maxItems: n,            items: {              type: "object",              additionalProperties: false,              required: ["title", "say", "move"],              properties: {                title: { type: "string", maxLength: 30 },                say: { type: "string", maxLength: 160 },                move: { type: "string", enum: moves.map((m) => m.move) },              },            },          },        },      },    },  },  messages: [    {      role: "system",      content:        `You script a short narrated tour of a web app for a product video. Write an opening line (at most 16 words) about what the page shows, then ${n} beats. Each beat has a title of two or three words, one sentence of at most 18 words, and one move; use ${n} different moves, and put moves on this page (tabs, buttons) before links, which open other pages. A beat's sentence describes what its move shows, using only the lines given for that move. Plain words, no exclamation marks.`,    },    { role: "user", content: read },  ],});const script: { intro: string; beats: { title: string; say: string; move: string }[] } = JSON.parse(  chat.choices[0].message.content,);script.beats.sort((a, b) => Number(a.move.startsWith("click link ")) - Number(b.move.startsWith("click link ")));// 3. render: the beats, a voice and captionsconst video = await call("POST", "/v1/render/videos?wait=60", {  source: { url: PAGE },  beats: [    { title: "Overview", say: script.intro },    ...script.beats.map((b) => ({ title: b.title, say: b.say, do: [{ kind: "click", target: target(b.move) }] })),  ],  voice: { model: "kokoro-82m", voice: "af_heart" },  captions: { burn_in: "karaoke", sidecars: ["vtt"] },  style: { background: "midnight", frame: "browser-dark", cursor: "arrow", captions: "default", click_effect: "ripple" },  output: { width: 1280, height: 720, fps: 30 },  formats: ["mp4", "poster"],});console.log(video.status, video.result?.artifacts.find((a: any) => a.kind === "mp4")?.url);

python
import jsonimport osimport reimport requestsAPI = "https://beta-api.b3os.org"PAGE = "https://beta.b3os.org/demo/"MAIN = {"kind": "snapshot", "mode": "full", "root": "main", "max_tokens": 800}PASSWORD = {"kind": "evaluate", "expression": "document.querySelector('input[type=password]') !== null"}s = requests.Session()s.headers["Authorization"] = f"Bearer {os.environ['B3OS_API_KEY']}"def lines(text):    text = re.sub(r" \[ref=[^\]]*\]", "", text)    return [l.replace(" [selected]", "").strip() for l in text.split("\n") if l.strip()]def target(move):    _, role, *name = move.split(" ")    return {"role": role, "name": " ".join(name)}# 1. vision: the page's outline and content, then what each tab, button and link showssid = s.post(f"{API}/v1/vision/sessions", json={}).json()["id"]act = lambda actions: s.post(f"{API}/v1/vision/sessions/{sid}/act",                             json={"actions": actions, "continue_on_error": True}).json()first = act([{"kind": "navigate", "url": PAGE}, {"kind": "snapshot", "mode": "skeleton"}, MAIN])home = lines(first["results"][2]["snapshot"]["text"])candidates = []for role, name, rest in re.findall(r'- (tab|button|link) "([^"]+)"(.*)', first["results"][1]["snapshot"]["text"]):    move = f"click {role} {name}"    skip = (role == "tab" and "[selected]" in rest) or "[disabled]" in rest    if not skip and move not in candidates:        candidates.append(move)moves = []for move in candidates:    r = act([{"kind": "navigate", "url": PAGE}, {"kind": "click", "target": target(move)},             {"kind": "wait", "for": {"network_idle": True}}, PASSWORD, MAIN])    if r.get("ok") and r["results"][3].get("value") is False:  # never a page with a password field        view = lines(r["results"][4]["snapshot"]["text"])        if len([l for l in view if l not in home]) >= 2:            moves.append({"move": move, "view": view[:16]})s.delete(f"{API}/v1/vision/sessions/{sid}")n = min(3, len(moves))# 2. infer: the script, its moves held to the qualifying onesread = "\n".join([f"Page: {PAGE}", "What the page shows:", *home[:20],                  *[x for m in moves for x in ["", f'Move "{m["move"]}" then shows:', *m["view"]]]])beat = {"type": "object", "additionalProperties": False, "required": ["title", "say", "move"],        "properties": {"title": {"type": "string", "maxLength": 30}, "say": {"type": "string", "maxLength": 160},                       "move": {"type": "string", "enum": [m["move"] for m in moves]}}}schema = {"type": "object", "additionalProperties": False, "required": ["intro", "beats"],          "properties": {"intro": {"type": "string", "maxLength": 160},                         "beats": {"type": "array", "minItems": n, "maxItems": n, "items": beat}}}chat = s.post(f"{API}/v1/infer/chat/completions", json={    "model": "b3os/qwen3.5-9b", "temperature": 0, "max_tokens": 600,    "chat_template_kwargs": {"enable_thinking": False},    "response_format": {"type": "json_schema", "json_schema": {"name": "tour", "strict": True, "schema": schema}},    "messages": [        {"role": "system", "content": "You script a short narrated tour of a web app for a product video. Write an "         f"opening line (at most 16 words) about what the page shows, then {n} beats. Each beat has a title of two "         f"or three words, one sentence of at most 18 words, and one move; use {n} different moves, and put moves "         "on this page (tabs, buttons) before links, which open other pages. A beat's sentence describes what its "         "move shows, using only the lines given for that move. Plain words, no exclamation marks."},        {"role": "user", "content": read},    ],}).json()script = json.loads(chat["choices"][0]["message"]["content"])script["beats"].sort(key=lambda b: b["move"].startswith("click link "))# 3. render: the beats, a voice and captionsvideo = s.post(f"{API}/v1/render/videos", params={"wait": 60}, json={    "source": {"url": PAGE},    "beats": [{"title": "Overview", "say": script["intro"]}]    + [{"title": b["title"], "say": b["say"], "do": [{"kind": "click", "target": target(b["move"])}]}       for b in script["beats"]],    "voice": {"model": "kokoro-82m", "voice": "af_heart"},    "captions": {"burn_in": "karaoke", "sidecars": ["vtt"]},    "style": {"background": "midnight", "frame": "browser-dark", "cursor": "arrow", "captions": "default",              "click_effect": "ripple"},    "output": {"width": 1280, "height": 720, "fps": 30},    "formats": ["mp4", "poster"],}).json()print(video["status"], next(a["url"] for a in video["result"]["artifacts"] if a["kind"] == "mp4"))

On the remote B3OS MCP server, ask your agent:

text
Open https://beta.b3os.org/demo/ in a vision session and take a skeleton snapshot and a full snapshot of <main>. Foreach tab that is not selected, each button and each link in the outline, go back to the page, click it and take afull snapshot of <main>; keep the clicks that show at least two things the page itself does not, and none that opensa sign-in form. Close the session. Script a narrated tour: an opening line about the page, then up to three beats,each a short title, one sentence about what its click shows (only from that snapshot) and the click, with clicksthat stay on the page before links. Record it with render_video: the page as the source, eachbeat's click as a role and name target, the kokoro-82m voice af_heart, karaoke captions, the midnight background andthe browser-dark frame. Give me the MP4 URL.

The agent calls vision_create_session, vision_act once per click and vision_close_session, writes the script itself and calls render_video. Nothing holds its script to the clicks it tried but its own reading; the recipe's schema enum is what keeps a scripted run from naming an element the page lacks.