Infer
OpenAI-compatible chat completions and speech from models served on the B3OS node, so an agent in a sandbox reaches its model without leaving the machine.
Infer serves models on the node, behind the OpenAI wire format. In the beta, B3OS runs its own small instruct model, an 8B-class model with tool calling and JSON Schema output, and a Kokoro text-to-speech model, on a capped slice of one GPU next to B3IQ's own inference. An agent in a sandbox calls them at http://127.0.0.1:7700/v1/infer and the request never leaves the machine.
Models
GET /v1/infer/models lists what the node serves. Each model has the OpenAI fields plus a b3os object:
json{ "id": "kokoro-82m", "object": "model", "owned_by": "b3os", "b3os": { "kind": "speech", "context_length": null, "quantization": null, "tools": false, "json_schema": false, "image_input": false, "max_request_bytes": 65536, "limits": { "tpm": null, "rpm": 300 } }}
The values are illustrative. kind is chat, speech or embedding; tools, json_schema and image_input say what a model supports; limits are your tokens and requests per minute on it. Use the IDs from this list in requests; examples in these docs use $B3OS_MODEL for the chat model's ID.
No silent fallback
A request for a model the node does not serve fails with model_not_available. B3OS never quietly sends it somewhere else. To use the B3IQ gateway for models beyond the node's, opt in per request with "route": "gateway". The Server-Timing header reports where each request went as infer_route, and responses carry b3os.route.
What to call
| You want | Call |
|---|---|
| Chat, tools, JSON output, streaming | POST /v1/infer/chat/completions |
| Speech, with optional word timings | POST /v1/infer/audio/speech |
| The voices of each speech model | GET /v1/infer/audio/voices |
| Embeddings | POST /v1/infer/embeddings, planned for embedding models the node serves |
Any OpenAI SDK works with the base URL set to https://beta-api.b3os.org/v1/infer; see OpenAI compatibility for the differences.
Billing and scopes
Chat bills infer.input_tokens and infer.output_tokens; speech bills infer.audio_chars. Inference is free in the beta, within each model's tokens-per-minute limit.
| Scope | For |
|---|---|
infer:run | Chat completions, speech and embeddings |
infer:read | List models and voices |