OpenAI compatibility

/v1/infer speaks the OpenAI wire format, so the OpenAI SDKs work with the base URL changed:

Pick a chat model from the node's list (the list also holds the speech model, so filter on b3os.kind):

bash
export B3OS_MODEL=$(curl -s https://beta-api.b3os.org/v1/infer/models \  -H "Authorization: Bearer $B3OS_API_KEY" | jq -r '[.data[] | select(.b3os.kind == "chat")][0].id')

ts
import OpenAI from "openai";const client = new OpenAI({  apiKey: process.env.B3OS_API_KEY,  baseURL: "https://beta-api.b3os.org/v1/infer",});const completion = await client.chat.completions.create({  model: process.env.B3OS_MODEL!,  messages: [{ role: "user", content: "Name one river in Oregon." }],});console.log(completion.choices[0].message.content);

python
import osfrom openai import OpenAIclient = OpenAI(    api_key=os.environ["B3OS_API_KEY"],    base_url="https://beta-api.b3os.org/v1/infer",)completion = client.chat.completions.create(    model=os.environ["B3OS_MODEL"],    messages=[{"role": "user", "content": "Name one river in Oregon."}],)print(completion.choices[0].message.content)

bash
curl -s https://beta-api.b3os.org/v1/infer/chat/completions \  -H "Authorization: Bearer $B3OS_API_KEY" \  -H "Content-Type: application/json" \  -d "{\"model\": \"$B3OS_MODEL\", \"messages\": [{\"role\": \"user\", \"content\": \"Name one river in Oregon.\"}]}"

Infer describes the models the node serves.

Endpoints

GET /v1/infer/audio/voices, which lists the voices of each speech model, is a B3OS endpoint and uses B3OS lists and problem errors.

Differences from OpenAI

• No silent fallback. A model the node does not serve fails with model_not_available. To send it to the B3IQ gateway instead, set "route": "gateway" in the request body. Server-Timing reports the route the request took as infer_route.
• One choice. n must be 1.
• Unsupported parameters fail. A parameter the model's runtime does not support returns 400 with unsupported_parameter; other parameters the runtime accepts pass through.
• Extensions live in a b3os object. Responses carry b3os: {route, node}, and each model in the list carries b3os with its kind, context length, quantization, support for tools, JSON Schema output and image input, the largest request it accepts, and your token and request limits for it. OpenAI SDKs ignore the field.
• Word timestamps. POST /audio/speech with "timestamps": "word" returns JSON with base64 audio and the start and end of each word, instead of audio bytes. In v0.1 speech is produced as wav, pcm or mp3; other formats return unsupported_parameter until enabled.
• Errors use OpenAI's shape, {"error": {"message", "type", "code", "param"}}, with the B3OS code in error.code. See infer errors.

Tokens are metered as infer.input_tokens and infer.output_tokens (characters as infer.audio_chars for speech) and are free in the beta, within each model's tokens-per-minute limit.

Planned

An Anthropic Messages endpoint at /v1/infer/messages and E2B's sandbox API are planned for later releases.