> ## Documentation Index
> Fetch the complete documentation index at: https://docs.beta.b3os.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Infer

OpenAI-compatible chat completions and speech from models served on the B3OS node, so an agent in a sandbox reaches its model without leaving the machine.

Infer serves models on the node, behind the OpenAI wire format. In the beta, B3OS runs its own small instruct model, an 8B-class model with tool calling and JSON Schema output, and a Kokoro text-to-speech model, on a capped slice of one GPU next to B3IQ's own inference. An agent in a [sandbox](/sandboxes/agents) calls them at `http://127.0.0.1:7700/v1/infer` and the request never leaves the machine.

## Models

`GET /v1/infer/models` lists what the node serves. Each model has the OpenAI fields plus a `b3os` object:

```json
{
  "id": "kokoro-82m",
  "object": "model",
  "owned_by": "b3os",
  "b3os": {
    "kind": "speech",
    "context_length": null,
    "quantization": null,
    "tools": false,
    "json_schema": false,
    "image_input": false,
    "max_request_bytes": 65536,
    "limits": { "tpm": null, "rpm": 300 }
  }
}
```

The values are illustrative. `kind` is `chat`, `speech` or `embedding`; `tools`, `json_schema` and `image_input` say what a model supports; `limits` are your tokens and requests per minute on it. Use the IDs from this list in requests; examples in these docs use `$B3OS_MODEL` for the chat model's ID.

## No silent fallback

A request for a model the node does not serve fails with [`model_not_available`](/errors#model_not_available). B3OS never quietly sends it somewhere else. To use the B3IQ gateway for models beyond the node's, opt in per request with `"route": "gateway"`. The `Server-Timing` header reports where each request went as `infer_route`, and responses carry `b3os.route`.

## What to call

| You want | Call |
|---|---|
| Chat, tools, JSON output, streaming | [`POST /v1/infer/chat/completions`](/infer/chat) |
| Speech, with optional word timings | [`POST /v1/infer/audio/speech`](/infer/speech) |
| The voices of each speech model | `GET /v1/infer/audio/voices` |
| Embeddings | `POST /v1/infer/embeddings`, planned for embedding models the node serves |

Any OpenAI SDK works with the base URL set to `https://beta-api.b3os.org/v1/infer`; see [OpenAI compatibility](/compat/openai) for the differences.

## Billing and scopes

Chat bills `infer.input_tokens` and `infer.output_tokens`; speech bills `infer.audio_chars`. Inference is free in the beta, within each model's tokens-per-minute limit.

| Scope | For |
|---|---|
| `infer:run` | Chat completions, speech and embeddings |
| `infer:read` | List models and voices |

**[Chat](/infer/chat)**

Tools, JSON Schema output, streaming.
  **[Speech](/infer/speech)**

Audio, with word timestamps for captions.
  **[Limits and FAQ](/infer/limits)**

Rates, sizes and common questions.