> ## Documentation Index
> Fetch the complete documentation index at: https://docs.beta.b3os.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Infer settings

Every infer setting: routing, output cap, reasoning, sampling and speech, with defaults, values and examples, and what is fixed in the beta.

These are fields of the chat completions, Responses and speech requests. Most follow OpenAI's API, so an OpenAI client sends them as it always does; `route` and the runtime extensions are B3OS's. [Chat completions](/infer/chat) explains each in context.

| Setting | Default |
|---|---|
| [Gateway route](#gateway-route): `route` | `local` |
| [Output cap](#output-cap): `max_tokens` | the model's `b3os.default_max_tokens`; `null` there means no cap |
| [Reasoning](#reasoning): `reasoning_effort` | `none` |
| [Sampling](#sampling): `temperature` | unset, so the runtime's own applies |
| [Audio format](#audio-format): `response_format` | `mp3` |
| [Speed](#speed): `speed` | 1 |
| [Word timings](#word-timings): `timestamps` | `none` |
| [Streaming](#streaming): `stream_format` | unset: the clip comes in one piece |

## Chat

### Gateway route

`local` answers only with a model this node serves; an unknown model is `404 model_not_available`. `gateway` sends a model the node does not serve to the B3IQ gateway. The beta node has no gateway route configured, so `gateway` answers the same 404 there. B3OS never sends a request elsewhere on its own.

- **Field:** `route` on [chat](/api/post-v1-infer-chat-completions)
- **Default:** `local`
- **Values:** `local` or `gateway`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)
- **Guide:** [No silent fallback](/infer#no-silent-fallback)

```json
{"model": "b3os/default", "messages": [{"role": "user", "content": "Name three rivers."}], "route": "gateway"}
```

### Output cap

The most tokens to generate. The answer stops there with `finish_reason: "length"`. With no cap from the request or the node, it can run to the end of the context. `max_completion_tokens` wins over `max_tokens`. `GET /v1/infer/models` shows each model's default cap.

- **Field:** `max_tokens`, `max_completion_tokens` and `max_output_tokens` on [chat](/api/post-v1-infer-chat-completions) and [responses](/api/post-v1-infer-responses)
- **Default:** the model's `b3os.default_max_tokens`; `null` there means no cap
- **Values:** 1 or more
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request), [`400 context_length_exceeded`](/errors#context_length_exceeded)
- **Guide:** [Models](/infer#models)

```json
{"model": "b3os/default", "messages": [{"role": "user", "content": "Name three rivers."}], "max_tokens": 512}
```

```bash
b3os infer chat "Name three rivers" --max-tokens 512
```

### Reasoning

How long the model may reason before it answers. `low`, `medium`, `high` and `xhigh` allow at most 256, 1,024, 4,096 and 8,192 reasoning tokens; at the budget the node closes the reasoning and the answer follows in the same request. Reasoning tokens count as output tokens.

- **Field:** `reasoning_effort` and `reasoning.effort` on [chat](/api/post-v1-infer-chat-completions) and [responses](/api/post-v1-infer-responses)
- **Default:** `none`
- **Values:** `none`, `minimal`, `low`, `medium`, `high` or `xhigh`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)
- **Guide:** [Reasoning](/infer/chat#reasoning)

```json
{"model": "b3os/default", "messages": [{"role": "user", "content": "Name three rivers."}], "reasoning_effort": "low"}
```

### Sampling

How the next token is picked. Lower `temperature` and `top_p` make answers more repeatable; the penalties discourage repeating tokens. `top_k`, `min_p` and `repetition_penalty` are runtime extensions that OpenAI clients pass through `extra_body`.

- **Field:** `temperature`, `top_p`, `top_k`, `min_p`, `repetition_penalty`, `presence_penalty` and `frequency_penalty` on [chat](/api/post-v1-infer-chat-completions) and [responses](/api/post-v1-infer-responses)
- **Default:** unset, so the runtime's own applies
- **Values:** 0 to 2 (`temperature`, chat); 0 to 1 (`top_p`, chat); -1 to 1,000,000 (`top_k`, chat); 0 to 1 (`min_p`, chat); 0 to 2 (`repetition_penalty`, chat); -2 to 2 (`presence_penalty`, chat); -2 to 2 (`frequency_penalty`, chat); 0 to 2 (`temperature`, responses); 0 to 1 (`top_p`, responses)
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)

```json
{"model": "b3os/default", "messages": [{"role": "user", "content": "Name three rivers."}], "temperature": 0.2, "top_p": 0.9}
```

## Speech

### Audio format

The encoding of the clip. `pcm` and `wav` start fastest to play; `mp3` and `opus` are smaller.

- **Field:** `response_format` on [speech](/api/post-v1-infer-audio-speech)
- **Default:** `mp3`
- **Values:** `mp3`, `opus`, `aac`, `flac`, `wav` or `pcm`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)
- **Guide:** [Speech](/infer/speech)

```json
{"model": "kokoro-82m", "input": "Hello.", "voice": "af_heart", "response_format": "wav"}
```

```bash
b3os infer speech "Hello" --out hello.wav --response-format wav
```

### Speed

How fast the voice speaks.

- **Field:** `speed` on [speech](/api/post-v1-infer-audio-speech)
- **Default:** 1
- **Values:** 0.25 to 4; `kokoro-82m` speaks at 0.5 to 2; outside a model's range is `400 unsupported_parameter`, never clamped
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request), [`400 unsupported_parameter`](/errors#unsupported_parameter)

```json
{"model": "kokoro-82m", "input": "Hello.", "voice": "af_heart", "speed": 1.2}
```

```bash
b3os infer speech "Hello" --out hello.mp3 --speed 1.2
```

### Word timings

`word` returns JSON with the audio as base64 and each word's start and end time, for captions or lip sync.

- **Field:** `timestamps` on [speech](/api/post-v1-infer-audio-speech)
- **Default:** `none`
- **Values:** `none` or `word`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)

```json
{"model": "kokoro-82m", "input": "Hello.", "voice": "af_heart", "timestamps": "word"}
```

### Streaming

`audio` streams the bytes of `response_format` as they are made; `sse` streams events with base64 audio. A stream starts playing sooner.

- **Field:** `stream_format` on [speech](/api/post-v1-infer-audio-speech)
- **Default:** unset: the clip comes in one piece
- **Values:** `audio` or `sse`
- **Invalid value:** [`400 invalid_request`](/errors#invalid_request)

```json
{"model": "kokoro-82m", "input": "Hello.", "voice": "af_heart", "stream_format": "audio", "response_format": "pcm"}
```

## Fixed in the beta

These work one way in the beta, and no request field changes them.

### Cached tokens and the token limit

**Counted in full.** Prompt tokens the runtime serves from its prefix cache count in full toward your tokens-per-minute limit (`tpm`), the same as any other prompt token. They are billed as `infer.input_tokens` like the rest.

### Precision

**FP8.** The chat model runs in an FP8 quantization. A different precision would be a different model ID; the model list shows each model's `quantization`.