Map, batch and crawl

bash
curl -s https://beta-api.b3os.org/v1/vision/map \  -H "Authorization: Bearer $B3OS_API_KEY" \  -H "Content-Type: application/json" \  -d '{"url": "https://docs.example.com", "search": "authentication", "limit": 100}'

ts
import { B3OS } from "@b3os/sdk";const b3os = new B3OS();const map = await b3os.vision.map({ url: "https://docs.example.com", search: "authentication", limit: 100 });for (const link of map.links) console.log(link.url, link.title);

python
from b3os import B3OSb3os = B3OS()site = b3os.vision.map({"url": "https://docs.example.com", "search": "authentication", "limit": 100})for link in site["links"]:    print(link["url"], link.get("title"))

text
Map https://docs.example.com and list the pages about authentication.

The agent calls the vision_map tool with url and search, and gets back URLs and titles.

bash
curl -s "https://beta-api.b3os.org/v1/vision/batches?wait=60" \  -H "Authorization: Bearer $B3OS_API_KEY" \  -H "Content-Type: application/json" \  -d '{"urls": ["https://example.com/blog/a", "https://example.com/blog/b"], "scrape": {"formats": ["markdown"]}}'

ts
// .wait() resolves with the finished jobconst batch = await b3os.vision  .createBatch({    urls: ["https://example.com/blog/a", "https://example.com/blog/b"],    scrape: { formats: ["markdown"] },  })  .wait();// for await walks every page across result pagesfor await (const page of b3os.vision.listBatchPages(batch.id, { status: "succeeded" })) {  console.log(page.url, page.scrape?.markdown?.length);}

python
batch = b3os.vision.create_batch(    {"urls": ["https://example.com/blog/a", "https://example.com/blog/b"], "scrape": {"formats": ["markdown"]}},    wait=True,  # returns the finished job)pages = b3os.vision.list_batch_pages(batch["id"], status="succeeded")for page in pages.auto_paging_iter():  # every page across result pages    print(page["url"], len(page["scrape"]["markdown"]))

text
Scrape these three blog posts and tell me which one mentions pricing: …

The MCP server has no batch tool. For a handful of URLs the agent calls vision_scrape once per URL; for hundreds, call the batch endpoint from code.

bash
curl -s https://beta-api.b3os.org/v1/vision/crawls \  -H "Authorization: Bearer $B3OS_API_KEY" \  -H "Content-Type: application/json" \  -d '{"url": "https://docs.example.com", "limit": 200, "include_paths": ["^/guides/"],       "scrape": {"formats": ["markdown"]},       "webhook": {"url": "https://example.com/hooks/b3os"}}'

ts
// .events() asks the creating request for an event stream; it resumes after a dropped connectionconst events = b3os.vision  .createCrawl({    url: "https://docs.example.com",    limit: 200,    include_paths: ["^/guides/"],    scrape: { formats: ["markdown"] },  })  .events();for await (const event of events) {  if (event.type === "page.succeeded") console.log(event.page?.url);}

python
events = b3os.vision.create_crawl(    {        "url": "https://docs.example.com",        "limit": 200,        "include_paths": ["^/guides/"],        "scrape": {"formats": ["markdown"]},    },    stream=True,  # the creating request answers with the job's event stream)for event in events:    if event["type"] == "page.succeeded":        print(event["page"]["url"])

text
Crawl the guides on https://docs.example.com, at most 50 pages, and summarize what each guide covers.

The agent calls vision_crawl with url, limit and include_paths. On the local MCP server the tool waits up to wait_s seconds and returns the pages as Markdown, or the crawl ID while it still runs, to pass back as job_id; on the remote server it takes the API body plus wait and returns the crawl job. For crawls of hundreds of pages, start them from code or a routine and let the agent read the pages afterwards.