Infer limits and FAQ
Per-model token and request limits, request sizes and stream lengths for inference in the beta, and answers to common questions about the node's models.
Limits
| Limit | Value |
|---|---|
| Tokens and requests per minute | Per model: b3os.limits.tpm and b3os.limits.rpm in GET /v1/infer/models |
| Largest request | Per model: b3os.max_request_bytes; the chat endpoint accepts at most 10 MiB |
| Chat stream length | 10 minutes |
Choices per request (n) | 1 |
| Speech input | 4,096 characters per request; request body 64 KiB |
Going over a model's rate returns 429 rate_limited with Retry-After.
FAQ
GET /v1/infer/models says, with each model's context length, quantization and features. The beta runs one 8B-class instruct model with tool calling and JSON Schema output, and a Kokoro speech model. Write code against the IDs in that list rather than fixed names.
No. Tokens are metered so you can see your usage, and each model's tokens-per-minute limit applies.
Set "route": "gateway" to send a request for another model to the B3IQ gateway. B3OS never does this on its own: without the flag, an unknown model fails with model_not_available. You can also call any provider you like from your agent; B3OS tools work with any model.
Not yet. An Anthropic Messages endpoint is planned. Today the surface is OpenAI's.
The sandbox's grant lacks the scope. Create it with api.scopes including infer:run (and infer:read to list models).