Infer settings

These are fields of the chat completions, Responses and speech requests. Most follow OpenAI's API, so an OpenAI client sends them as it always does; route and the runtime extensions are B3OS's. Chat completions explains each in context.

Chat

Gateway route

local answers only with a model this node serves; an unknown model is 404 model_not_available. gateway sends a model the node does not serve to the B3IQ gateway. The beta node has no gateway route configured, so gateway answers the same 404 there. B3OS never sends a request elsewhere on its own.

• Field: route on chat
• Default: local
• Values: local or gateway
• Invalid value: 400 invalid_request
• Guide: No silent fallback

json
{"model": "b3os/default", "messages": [{"role": "user", "content": "Name three rivers."}], "route": "gateway"}

Output cap

The most tokens to generate. The answer stops there with finish_reason: "length". With no cap from the request or the node, it can run to the end of the context. max_completion_tokens wins over max_tokens. GET /v1/infer/models shows each model's default cap.

• Field: max_tokens, max_completion_tokens and max_output_tokens on chat and responses
• Default: the model's b3os.default_max_tokens; null there means no cap
• Values: 1 or more
• Invalid value: 400 invalid_request, 400 context_length_exceeded
• Guide: Models

json
{"model": "b3os/default", "messages": [{"role": "user", "content": "Name three rivers."}], "max_tokens": 512}

bash
b3os infer chat "Name three rivers" --max-tokens 512

Reasoning

How long the model may reason before it answers. low, medium, high and xhigh allow at most 256, 1,024, 4,096 and 8,192 reasoning tokens; at the budget the node closes the reasoning and the answer follows in the same request. Reasoning tokens count as output tokens.

• Field: reasoning_effort and reasoning.effort on chat and responses
• Default: none
• Values: none, minimal, low, medium, high or xhigh
• Invalid value: 400 invalid_request
• Guide: Reasoning

json
{"model": "b3os/default", "messages": [{"role": "user", "content": "Name three rivers."}], "reasoning_effort": "low"}

Sampling

How the next token is picked. Lower temperature and top_p make answers more repeatable; the penalties discourage repeating tokens. top_k, min_p and repetition_penalty are runtime extensions that OpenAI clients pass through extra_body.

• Field: temperature, top_p, top_k, min_p, repetition_penalty, presence_penalty and frequency_penalty on chat and responses
• Default: unset, so the runtime's own applies
• Values: 0 to 2 (temperature, chat); 0 to 1 (top_p, chat); -1 to 1,000,000 (top_k, chat); 0 to 1 (min_p, chat); 0 to 2 (repetition_penalty, chat); -2 to 2 (presence_penalty, chat); -2 to 2 (frequency_penalty, chat); 0 to 2 (temperature, responses); 0 to 1 (top_p, responses)
• Invalid value: 400 invalid_request

json
{"model": "b3os/default", "messages": [{"role": "user", "content": "Name three rivers."}], "temperature": 0.2, "top_p": 0.9}

Speech

Audio format

The encoding of the clip. pcm and wav start fastest to play; mp3 and opus are smaller.

• Field: response_format on speech
• Default: mp3
• Values: mp3, opus, aac, flac, wav or pcm
• Invalid value: 400 invalid_request
• Guide: Speech

json
{"model": "kokoro-82m", "input": "Hello.", "voice": "af_heart", "response_format": "wav"}

bash
b3os infer speech "Hello" --out hello.wav --response-format wav

Speed

How fast the voice speaks.

• Field: speed on speech
• Default: 1
• Values: 0.25 to 4; kokoro-82m speaks at 0.5 to 2; outside a model's range is 400 unsupported_parameter, never clamped
• Invalid value: 400 invalid_request, 400 unsupported_parameter

json
{"model": "kokoro-82m", "input": "Hello.", "voice": "af_heart", "speed": 1.2}

bash
b3os infer speech "Hello" --out hello.mp3 --speed 1.2

Word timings

word returns JSON with the audio as base64 and each word's start and end time, for captions or lip sync.

• Field: timestamps on speech
• Default: none
• Values: none or word
• Invalid value: 400 invalid_request

json
{"model": "kokoro-82m", "input": "Hello.", "voice": "af_heart", "timestamps": "word"}

Streaming

audio streams the bytes of response_format as they are made; sse streams events with base64 audio. A stream starts playing sooner.

• Field: stream_format on speech
• Default: unset: the clip comes in one piece
• Values: audio or sse
• Invalid value: 400 invalid_request

json
{"model": "kokoro-82m", "input": "Hello.", "voice": "af_heart", "stream_format": "audio", "response_format": "pcm"}

Fixed in the beta

These work one way in the beta, and no request field changes them.

Cached tokens and the token limit

Counted in full. Prompt tokens the runtime serves from its prefix cache count in full toward your tokens-per-minute limit (tpm), the same as any other prompt token. They are billed as infer.input_tokens like the rest.

Precision

FP8. The chat model runs in an FP8 quantization. A different precision would be a different model ID; the model list shows each model's quantization.