Types Reference
Types Reference
Section titled “Types Reference”All types defined by the library, grouped by category. Types are shown using Rust as the canonical representation.
Result Types
Section titled “Result Types”ModerationResult
Section titled “ModerationResult”A single moderation classification result.
| Field | Type | Default | Description |
|---|---|---|---|
flagged |
bool |
— | True if any category was flagged. |
categories |
ModerationCategories |
— | Boolean flags for each moderation category. |
category_scores |
ModerationCategoryScores |
— | Confidence scores for each category. |
RerankResult
Section titled “RerankResult”A single reranked document with its relevance score.
| Field | Type | Default | Description |
|---|---|---|---|
index |
u32 |
— | Original document index in the input list. |
relevance_score |
f64 |
— | Relevance score. Higher indicates more relevant. Most providers use [0, 1]; NVIDIA NIM sends a raw logit. |
document |
Option<RerankResultDocument> |
/* serde(default) */ |
Original document content (if return_documents was true). |
RerankResultDocument
Section titled “RerankResultDocument”The text content of a reranked document, returned when return_documents is true.
| Field | Type | Default | Description |
|---|---|---|---|
text |
String |
— | Document text. |
SearchResult
Section titled “SearchResult”An individual search result.
| Field | Type | Default | Description |
|---|---|---|---|
title |
String |
— | Result title. |
url |
String |
— | Result URL. |
snippet |
String |
— | Text snippet or excerpt from the page. |
date |
Option<String> |
/* serde(default) */ |
Publication or last-updated date, if available. |
SingleflightResult
Section titled “SingleflightResult”The value broadcast from a singleflight leader to all followers.
The error value is shared so every follower receives the same upstream failure without cloning the underlying error.
Opaque type — fields are not directly accessible.
Configuration Types
Section titled “Configuration Types”See Configuration Reference for detailed defaults and language-specific representations.
ClientOptions
Section titled “ClientOptions”Binding-friendly options for constructing a client without an unbounded response body.
| Field | Type | Default | Description |
|---|---|---|---|
api_key |
String |
— | Provider API key; an empty value omits the authentication header. |
base_url |
Option<String> |
Default::default() |
Optional provider base URL override. |
timeout_secs |
Option<u64> |
Default::default() |
Optional request timeout in seconds. |
connect_timeout_secs |
Option<u64> |
Default::default() |
Optional connect timeout in seconds. A connect timeout is retried up to max_retries times. Zero is an error on every target. wasm32 ignores any other value. |
max_retries |
Option<u32> |
Default::default() |
Optional retry count for retryable provider failures. |
max_response_bytes |
Option<u64> |
Default::default() |
Optional maximum number of response bytes retained in memory. |
model_hint |
Option<String> |
Default::default() |
Optional model or provider hint used for initial provider selection. |
blocked_networks |
Vec<String> |
vec![] |
Optional CIDR networks the client never connects to, such as "169.254.0.0/16". A host name is refused when any address it resolves to is in a listed network. A bad entry is a BadRequest error that names the value. wasm32 refuses a list that is not empty with a BadRequest error. |
SystemMessage
Section titled “SystemMessage”System message guiding model behavior for the entire conversation.
| Field | Type | Default | Description |
|---|---|---|---|
content |
UserContent |
UserContent::Text |
Instructions or context that apply throughout the conversation. Accepts either a plain text string or an array of content parts, mirroring UserContent so that Message::system_with_parts works. |
name |
Option<String> |
Default::default() |
Optional name for the system message source. |
UserMessage
Section titled “UserMessage”User message in the conversation.
| Field | Type | Default | Description |
|---|---|---|---|
content |
UserContent |
UserContent::Text |
Message content as plain text or array of content parts (text, images, documents, audio). |
name |
Option<String> |
Default::default() |
Optional name for the user. |
ImageUrl
Section titled “ImageUrl”An image URL reference with optional detail level for processing.
No deny_unknown_fields: this type is shared with the response side
(see AssistantPart.OutputImage) where it is deserialized from
provider output, not just constructed as request input (see #51). A
provider adding a new field to its image-output object must not hard-fail
the whole response.
| Field | Type | Default | Description |
|---|---|---|---|
url |
String |
— | URL of the image (data URI or HTTP/HTTPS URL). |
detail |
Option<ImageDetail> |
Default::default() |
Detail level: low (512x512), high (2x2 tiles), or auto (model-selected). |
DocumentContent
Section titled “DocumentContent”PDF/document content part for vision-capable models.
| Field | Type | Default | Description |
|---|---|---|---|
data |
String |
— | Base64-encoded document data or URL. |
media_type |
String |
— | MIME type (e.g., “application/pdf”, “text/csv”). |
AudioContent
Section titled “AudioContent”Audio content part for speech-capable models.
No deny_unknown_fields: shared with the response side (see
AssistantPart.OutputAudio), same rationale as ImageUrl (#51).
| Field | Type | Default | Description |
|---|---|---|---|
data |
String |
— | Base64-encoded audio data. |
format |
String |
— | Audio format (e.g., “wav”, “mp3”, “ogg”). |
AssistantMessage
Section titled “AssistantMessage”Assistant’s response to a user message.
| Field | Type | Default | Description |
|---|---|---|---|
content |
Option<AssistantContent> |
Default::default() |
The assistant’s response: plain text, structured parts, or absent. None is valid when the model replies with tool calls only. |
name |
Option<String> |
Default::default() |
Optional name for the assistant. |
tool_calls |
Vec<ToolCall> |
vec![] |
Tool calls the model wants to execute, if any. |
refusal |
Option<String> |
Default::default() |
Refusal reason, if the model declined to respond per safety policies. Omitted when None, so assistant turns re-sent as request history do not carry a response-only key that strict providers (e.g. Groq) reject (#292). OpenAI’s response schema requires the key on choices[].message, so Choice serialization restores it as null. |
function_call |
Option<FunctionCall> |
Default::default() |
Deprecated legacy function_call field; retained for API compatibility. |
reasoning_content |
Option<String> |
Default::default() |
Reasoning/thinking tokens returned by the provider, if any (e.g. DeepSeek R1, Qwen reasoning_content, or Anthropic extended thinking). |
thinking_blocks |
Vec<ThinkingBlock> |
vec![] |
Reasoning blocks the provider requires back unchanged on the next turn: Anthropic thinking blocks with their signature, and redacted_thinking blocks. Leave them on the message when you send the turn back: Anthropic requires thinking blocks to be returned unchanged, for example alongside tool results. Each block goes back at its index in the turn. Other providers never receive them. |
ToolMessage
Section titled “ToolMessage”Tool execution result returned to the model.
| Field | Type | Default | Description |
|---|---|---|---|
content |
UserContent |
UserContent::Text |
Result of the tool execution as plain text or an array of content parts (text, images, documents, audio), mirroring UserMessage::content. #[serde(untagged)] on UserContent means a bare JSON string still deserialises into Text, so tool results persisted before this field carried structured content continue to round-trip. |
tool_call_id |
String |
— | ID of the tool call this result responds to. |
name |
Option<String> |
Default::default() |
Optional tool/function name. |
DeveloperMessage
Section titled “DeveloperMessage”Developer message (system-like message for Claude models).
| Field | Type | Default | Description |
|---|---|---|---|
content |
String |
— | Developer-specific instructions or context. |
name |
Option<String> |
Default::default() |
Optional name for the developer message source. |
FunctionMessage
Section titled “FunctionMessage”Deprecated legacy function-role message body.
| Field | Type | Default | Description |
|---|---|---|---|
content |
String |
— | The extracted text content |
name |
String |
— | The name |
ThinkingBlock
Section titled “ThinkingBlock”A reasoning block a provider requires back unchanged: an Anthropic thinking block
with its signature, or a redacted_thinking block with its encrypted data.
In a stream, each thinking_blocks entry is a fragment:
the first carries type (and all of a redacted_thinking block), later ones carry
thinking text to append and, once, the signature. In Rust,
ThinkingBlockAccumulator merges them.
| Field | Type | Default | Description |
|---|---|---|---|
block_type |
Option<String> |
Default::default() |
Block type: thinking or redacted_thinking. A stream sends it in the first fragment of a block only. |
thinking |
Option<String> |
Default::default() |
The reasoning text (a summary, or empty, on models that do not return it). |
signature |
Option<String> |
Default::default() |
Opaque token that verifies the block; required when the block is sent back. |
data |
Option<String> |
Default::default() |
Encrypted reasoning of a redacted_thinking block. |
index |
Option<u32> |
Default::default() |
Position of the block among the content blocks of the provider’s turn. |
text_offset |
Option<u32> |
Default::default() |
UTF-8 bytes of the turn’s text content that come before the block. With tool_call_offset, it puts the block back between the right text and tool calls. Replies without streaming set both, and so does the Rust ThinkingBlockAccumulator; stream fragments do not. Blocks without both offsets, or with offsets that do not fit the text, are put back in index order. |
tool_call_offset |
Option<u32> |
Default::default() |
Number of the turn’s tool_calls that come before the block (see text_offset). |
SpecificToolChoice
Section titled “SpecificToolChoice”Directive to call a specific tool.
| Field | Type | Default | Description |
|---|---|---|---|
choice_type |
ToolType |
ToolType::Function |
Tool type (always “function”). |
function |
SpecificFunction |
— | The specific function to invoke. |
SpecificFunction
Section titled “SpecificFunction”Name of the specific function to invoke.
| Field | Type | Default | Description |
|---|---|---|---|
name |
String |
— | Function name. |
JsonSchemaFormat
Section titled “JsonSchemaFormat”JSON Schema specification for constrained output.
| Field | Type | Default | Description |
|---|---|---|---|
name |
String |
— | Name of the schema (must be unique in the request). |
description |
Option<String> |
Default::default() |
Description of what the schema represents. |
schema |
serde_json::Value |
— | JSON Schema object defining the output structure. |
strict |
Option<bool> |
Default::default() |
If true, enforce strict schema validation. |
Token-usage accounting returned by the provider on each completion / embedding call.
| Field | Type | Default | Description |
|---|---|---|---|
prompt_tokens |
u64 |
— | Prompt tokens used. Defaults to 0 when absent (some providers omit this). |
completion_tokens |
u64 |
— | Completion tokens used. Defaults to 0 when absent (e.g. embedding responses). |
total_tokens |
u64 |
— | Total tokens used. Defaults to 0 when absent (some providers omit this). |
prompt_tokens_details |
Option<PromptTokensDetails> |
Default::default() |
Breakdown of tokens used in the prompt, including cached tokens served at the provider’s discounted cache-read rate. Absent when the provider does not return prompt-token details. |
completion_tokens_details |
Option<CompletionTokensDetails> |
Default::default() |
Breakdown of tokens used in the completion, including reasoning tokens. Absent when the provider does not return completion-token details. |
PromptTokensDetails
Section titled “PromptTokensDetails”Breakdown of tokens used in the prompt portion of a request.
cached_tokens is included in Usage.prompt_tokens — it is not an
additional charge on top of the prompt token count. When pricing supports
a cache_read_input_token_cost, the cached portion is billed at the
discounted rate and the remainder at the regular input rate.
| Field | Type | Default | Description |
|---|---|---|---|
cached_tokens |
u64 |
— | Cached tokens present in the prompt. Defaults to 0 when absent. |
cache_creation_tokens |
u64 |
— | Tokens written to the provider prompt cache. Defaults to 0 when absent. |
audio_tokens |
u64 |
— | Audio input tokens present in the prompt. Defaults to 0 when absent. |
CompletionTokensDetails
Section titled “CompletionTokensDetails”Breakdown of tokens used in the completion portion of a request.
reasoning_tokens is included in Usage.completion_tokens — it is not
additional usage on top of the completion token count.
| Field | Type | Default | Description |
|---|---|---|---|
reasoning_tokens |
u64 |
— | Reasoning or thinking tokens present in the completion. Defaults to 0 when absent. |
ChatCompletionRequest
Section titled “ChatCompletionRequest”Chat completion request (compatible with OpenAI and similar APIs).
| Field | Type | Default | Description |
|---|---|---|---|
model |
String |
— | Model ID (e.g., "gpt-4o-mini", "claude-3-5-sonnet"). |
messages |
Vec<Message> |
vec![] |
Conversation history from oldest to newest. |
temperature |
Option<f64> |
Default::default() |
Sampling temperature. Higher increases randomness, lower is more deterministic. Defaults to 1.0. The accepted range depends on the provider the request is routed to. OpenAI-compatible providers accept [0.0, 2.0]; Anthropic and Amazon Bedrock both cap it at 1.0, and for those two a value above the cap is rejected with a BadRequest error before the request is sent, rather than being silently clamped or left for the provider to reject. No range is enforced for providers whose own documentation does not state one — the value is forwarded and the provider decides. Consult the target provider’s reference rather than assuming [0.0, 2.0] is portable. |
top_p |
Option<f64> |
Default::default() |
Nucleus sampling parameter. Lower is more focused. Accepted ranges vary by provider (most document [0.0, 1.0], but this is not universal — check the target provider’s own documentation for its exact bounds). |
n |
Option<u32> |
Default::default() |
Number of chat completions to generate. Defaults to 1. |
stream |
Option<bool> |
Default::default() |
Whether to stream the response. Managed by the client layer — do not set directly. |
stop |
Option<StopSequence> |
Default::default() |
Stop sequence(s) that halt token generation. |
max_tokens |
Option<u64> |
Default::default() |
Max output tokens. Different from max_completion_tokens in some providers. |
presence_penalty |
Option<f64> |
Default::default() |
Presence penalty in [-2.0, 2.0]. Positive discourages repeated topics. |
frequency_penalty |
Option<f64> |
Default::default() |
Frequency penalty in [-2.0, 2.0]. Positive discourages repeated tokens. |
logit_bias |
HashMap<String, f64> |
HashMap::new() |
Token bias map. Uses BTreeMap (sorted keys) for deterministic serialization order — important when hashing or signing requests. |
user |
Option<String> |
Default::default() |
User identifier for request tracking and abuse detection. |
tools |
Vec<ChatCompletionTool> |
vec![] |
Tools the model can invoke. |
tool_choice |
Option<ToolChoice> |
Default::default() |
Tool usage mode (auto, required, none, or specific tool). |
parallel_tool_calls |
Option<bool> |
Default::default() |
Whether the model can call multiple tools in parallel. Defaults to true. |
response_format |
Option<ResponseFormat> |
Default::default() |
Output format constraint (text, JSON, JSON schema). |
stream_options |
Option<StreamOptions> |
Default::default() |
Streaming options (e.g., include_usage). |
seed |
Option<i64> |
Default::default() |
Random seed for reproducible outputs. Provider support varies. |
reasoning_effort |
Option<ReasoningEffort> |
Default::default() |
Reasoning effort level (minimal, low, medium, high, max) for extended-thinking models. |
modalities |
Vec<Modality> |
vec![] |
Output modalities to request from the model. For OpenAI audio models, pass ["text", "audio"]. Vertex AI / Gemini translates these to generationConfig.responseModalities (uppercase). |
logprobs |
Option<bool> |
Default::default() |
Whether to return log probabilities of the output tokens. |
top_logprobs |
Option<u32> |
Default::default() |
Number of most-likely tokens to return log probabilities for, 0..=20. Requires logprobs to be true. |
max_completion_tokens |
Option<u64> |
Default::default() |
Upper bound on generated tokens, including reasoning tokens. Supersedes max_tokens on OpenAI reasoning models, which reject max_tokens outright. |
service_tier |
Option<String> |
Default::default() |
Latency tier to process the request under (e.g. "auto", "default", "flex"). |
store |
Option<bool> |
Default::default() |
Whether to store the completion for later retrieval by the provider. |
metadata |
HashMap<String, String> |
HashMap::new() |
Developer-defined tags attached to the completion. |
prediction |
Option<serde_json::Value> |
Default::default() |
Predicted output, for latency reduction when much of the response is known ahead of time. Untyped: the shape is provider-defined and still evolving, and the value is forwarded verbatim. |
audio |
Option<serde_json::Value> |
Default::default() |
Audio output parameters, required when modalities includes audio. Untyped for the same reason as prediction. |
web_search_options |
Option<serde_json::Value> |
Default::default() |
Web-search tool configuration for search-enabled models. Untyped for the same reason as prediction. |
extra_body |
Option<serde_json::Value> |
Default::default() |
Provider-specific extra parameters merged into the request body. Use for guardrails, safety settings, grounding config, etc. |
StreamOptions
Section titled “StreamOptions”Options for streaming responses.
| Field | Type | Default | Description |
|---|---|---|---|
include_usage |
Option<bool> |
Default::default() |
If true, include token usage in the final stream chunk. |
ChatCompletionResponse
Section titled “ChatCompletionResponse”Chat completion response from the API.
| Field | Type | Default | Description |
|---|---|---|---|
id |
String |
— | Unique identifier for this response. |
object |
String |
— | Always "chat.completion" from OpenAI-compatible APIs. Stored as a plain String so non-standard provider values do not break deserialization. |
created |
u64 |
— | Unix timestamp of response creation. |
model |
String |
— | Provider-reported model identifier. For Gemini, this is modelVersion; when omitted, the client retains the requested model, which may be an alias and does not establish the resolved model version. |
choices |
Vec<Choice> |
vec![] |
List of completion choices. |
usage |
Option<Usage> |
Default::default() |
Token usage statistics. |
system_fingerprint |
Option<String> |
Default::default() |
Fingerprint of the system configuration (OpenAI-specific). |
service_tier |
Option<String> |
Default::default() |
Service tier used (OpenAI-specific). |
Choice
Section titled “Choice”A single completion choice.
| Field | Type | Default | Description |
|---|---|---|---|
index |
u32 |
— | Index of this choice in the choices array. |
message |
AssistantMessage |
— | The assistant’s message response. Serialized with an explicit role: "assistant". The field is not stored on AssistantMessage because Message is an internally-tagged enum keyed on role, so a stored field would emit the key twice inside a request. OpenAI’s response schema requires it here. |
finish_reason |
Option<FinishReason> |
Default::default() |
Why the model stopped generating (stop, length, tool_calls, content_filter, etc.). |
logprobs |
Option<serde_json::Value> |
Default::default() |
Per-token log probabilities, when the request asked for them. Required by OpenAI’s response schema as an always-present, nullable key, so this is deliberately not skip_serializing_if. |
ChatCompletionChunk
Section titled “ChatCompletionChunk”A streamed chunk of a chat completion response.
| Field | Type | Default | Description |
|---|---|---|---|
id |
String |
— | Unique identifier for this stream. |
object |
String |
— | Always "chat.completion.chunk" from OpenAI-compatible APIs. Stored as a plain String so non-standard provider values do not fail parsing. |
created |
u64 |
— | Unix timestamp of chunk creation. |
model |
String |
— | Provider-reported model identifier. Gemini chunks without modelVersion retain the last reported model, or the requested model if none was reported. A requested-model fallback does not establish the resolved model version. |
choices |
Vec<StreamChoice> |
vec![] |
Streaming choices (delta updates). |
usage |
Option<Usage> |
Default::default() |
Token usage (typically only in the final chunk). |
system_fingerprint |
Option<String> |
Default::default() |
Fingerprint of the system configuration (OpenAI-specific). |
service_tier |
Option<String> |
Default::default() |
Service tier used (OpenAI-specific). |
StreamChoice
Section titled “StreamChoice”A streaming choice with incremental delta.
| Field | Type | Default | Description |
|---|---|---|---|
index |
u32 |
— | Index of this choice in the choices array. |
delta |
StreamDelta |
— | Incremental update to the message (content, tool calls, etc.). |
finish_reason |
Option<FinishReason> |
Default::default() |
Why the stream ended (present only in final chunk). |
StreamDelta
Section titled “StreamDelta”Incremental delta in a stream chunk.
| Field | Type | Default | Description |
|---|---|---|---|
role |
Option<String> |
Default::default() |
Role (typically present only in the first chunk). |
content |
Option<String> |
Default::default() |
Partial content chunk (e.g., a few words of the response). |
tool_calls |
Vec<StreamToolCall> |
vec![] |
Partial tool calls being streamed. |
function_call |
Option<StreamFunctionCall> |
Default::default() |
Deprecated legacy function_call delta; retained for API compatibility. |
refusal |
Option<String> |
Default::default() |
Partial refusal message. |
reasoning_content |
Option<String> |
Default::default() |
Partial reasoning/thinking tokens (OpenAI-compatible extension used by DeepSeek R1, Qwen, etc.). |
thinking_blocks |
Vec<ThinkingBlock> |
vec![] |
Fragments of reasoning blocks the provider requires back unchanged (Anthropic thinking and redacted_thinking). Merge them by index as ThinkingBlock describes (in Rust, with ThinkingBlockAccumulator), and set the merged blocks as thinking_blocks of the turn. |
StreamToolCall
Section titled “StreamToolCall”A streaming tool call being built incrementally.
| Field | Type | Default | Description |
|---|---|---|---|
index |
u32 |
— | Index of this tool call in the tool_calls array. |
id |
Option<String> |
Default::default() |
Tool call ID (typically in the first chunk for this call). |
call_type |
Option<ToolType> |
Default::default() |
Tool type (typically “function”). |
function |
Option<StreamFunctionCall> |
Default::default() |
Partial function name and arguments. |
StreamFunctionCall
Section titled “StreamFunctionCall”Partial function call details in a stream.
| Field | Type | Default | Description |
|---|---|---|---|
name |
Option<String> |
Default::default() |
Function name (typically in the first chunk). |
arguments |
Option<String> |
Default::default() |
Partial JSON arguments chunk. |
EmbeddingRequest
Section titled “EmbeddingRequest”Embedding request.
| Field | Type | Default | Description |
|---|---|---|---|
model |
String |
— | Model ID (e.g., "text-embedding-3-small"). |
input |
EmbeddingInput |
EmbeddingInput::Single |
Text, texts, or multimodal content to embed. |
encoding_format |
Option<EmbeddingFormat> |
Default::default() |
Output format: float (native) or base64. |
dimensions |
Option<u32> |
Default::default() |
Requested embedding dimensions (if supported by the model). |
user |
Option<String> |
Default::default() |
User identifier for request tracking. |
extra_body |
Option<serde_json::Value> |
Default::default() |
Provider-specific parameters merged into the request body, as chat’s extra_body is. Keys here override identically named fields. |
CreateImageRequest
Section titled “CreateImageRequest”Request to create images from a text prompt.
| Field | Type | Default | Description |
|---|---|---|---|
prompt |
String |
— | Text description of the image to generate. |
model |
Option<String> |
Default::default() |
Model ID (e.g., "dall-e-3"). Optional; API may use default if unset. |
n |
Option<u32> |
Default::default() |
Number of images to generate. Defaults to 1. |
size |
Option<String> |
Default::default() |
Image size (e.g., "1024x1024", "1792x1024"). |
quality |
Option<String> |
Default::default() |
Image quality: "standard" or "hd". |
style |
Option<String> |
Default::default() |
Style: "natural" or "vivid" (DALL-E 3 only). |
response_format |
Option<String> |
Default::default() |
Response format: "url" or "b64_json". |
user |
Option<String> |
Default::default() |
User identifier for request tracking. |
extra_body |
Option<serde_json::Value> |
Default::default() |
Provider-specific parameters merged into the request body, as chat’s extra_body is. Keys here override identically named fields. |
ImagesResponse
Section titled “ImagesResponse”Response containing generated images.
| Field | Type | Default | Description |
|---|---|---|---|
created |
u64 |
— | Unix timestamp of image creation. |
data |
Vec<Image> |
vec![] |
List of generated images. |
A single generated image, returned as either a URL or base64 data.
| Field | Type | Default | Description |
|---|---|---|---|
url |
Option<String> |
Default::default() |
Image URL (if response_format was “url”). |
b64_json |
Option<String> |
Default::default() |
Base64-encoded image data (if response_format was “b64_json”). |
revised_prompt |
Option<String> |
Default::default() |
The final prompt used to generate the image (DALL-E 3). |
DecodedDataUrl
Section titled “DecodedDataUrl”Result of decoding a data: URL — MIME type and the decoded byte payload.
Named struct (rather than a tuple) so polyglot bindings can extract
decode_data_url with a typed return rather than a sanitized scalar.
| Field | Type | Default | Description |
|---|---|---|---|
mime |
String |
— | MIME type extracted from the URL prefix (verbatim, not normalised). |
data |
Vec<u8> |
— | Decoded base64 payload. |
CreateSpeechRequest
Section titled “CreateSpeechRequest”Request to generate speech audio from text.
| Field | Type | Default | Description |
|---|---|---|---|
model |
String |
— | Model ID (e.g., "tts-1", "tts-1-hd"). |
input |
String |
— | Text to synthesize into speech. |
voice |
String |
— | Voice name (e.g., "alloy", "echo", "fable", "onyx", "nova", "shimmer"). |
response_format |
Option<String> |
Default::default() |
Audio format (e.g., "mp3", "opus", "aac", "flac", "wav", "pcm"). |
speed |
Option<f64> |
Default::default() |
Playback speed in [0.25, 4.0]. Defaults to 1.0. |
extra_body |
Option<serde_json::Value> |
Default::default() |
Provider-specific parameters merged into the request body, as chat’s extra_body is. Keys here override identically named fields. |
CreateTranscriptionRequest
Section titled “CreateTranscriptionRequest”Request to transcribe audio into text.
| Field | Type | Default | Description |
|---|---|---|---|
model |
String |
— | Model ID (e.g., "whisper-1"). |
file |
String |
— | Base64-encoded audio file data. |
language |
Option<String> |
Default::default() |
Language ISO-639-1 code (e.g., "en", "fr", "de"). Optional; model auto-detects. |
prompt |
Option<String> |
Default::default() |
Optional text to guide the model (improves accuracy for domain-specific terms). |
response_format |
Option<String> |
Default::default() |
Output format (e.g., "json", "text", "vtt", "srt", "verbose_json"). |
temperature |
Option<f64> |
Default::default() |
Sampling temperature in [0.0, 1.0]. Higher increases variability. Defaults to 0. |
extra_body |
Option<serde_json::Value> |
Default::default() |
Provider-specific parameters merged into the request body, as chat’s extra_body is. Keys here override identically named fields. |
TranscriptionResponse
Section titled “TranscriptionResponse”Response from a transcription request.
| Field | Type | Default | Description |
|---|---|---|---|
text |
String |
— | The transcribed text. |
language |
Option<String> |
Default::default() |
Detected language (ISO-639-1 code). |
duration |
Option<f64> |
Default::default() |
Total audio duration in seconds. |
segments |
Vec<TranscriptionSegment> |
vec![] |
Detailed segment-level transcription (if response_format is “verbose_json”). |
TranscriptionSegment
Section titled “TranscriptionSegment”A segment of transcribed audio with timing information.
| Field | Type | Default | Description |
|---|---|---|---|
id |
u32 |
— | Segment index (0-based). |
start |
f64 |
— | Start time in seconds. |
end |
f64 |
— | End time in seconds. |
text |
String |
— | Transcribed text for this segment. |
ModerationRequest
Section titled “ModerationRequest”Request to classify content for policy violations.
| Field | Type | Default | Description |
|---|---|---|---|
input |
ModerationInput |
ModerationInput::Single |
Text or texts to check. |
model |
Option<String> |
Default::default() |
Model ID (e.g., "text-moderation-latest"). Optional; API uses default if unset. |
extra_body |
Option<serde_json::Value> |
Default::default() |
Provider-specific parameters merged into the request body, as chat’s extra_body is. Keys here override identically named fields. |
ModerationCategories
Section titled “ModerationCategories”Boolean flags for each moderation category.
| Field | Type | Default | Description |
|---|---|---|---|
sexual |
bool |
— | Sexual content. |
hate |
bool |
— | Hate speech. |
harassment |
bool |
— | Harassment. |
self_harm |
bool |
— | Self-harm content. |
sexual_minors |
bool |
— | Sexual content involving minors. |
hate_threatening |
bool |
— | Hate speech that threatens violence. |
violence_graphic |
bool |
— | Graphic violence. |
self_harm_intent |
bool |
— | Intent to self-harm. |
self_harm_instructions |
bool |
— | Instructions for self-harm. |
harassment_threatening |
bool |
— | Harassment that threatens violence. |
violence |
bool |
— | Non-graphic violence. |
ModerationCategoryScores
Section titled “ModerationCategoryScores”Confidence scores for each moderation category.
| Field | Type | Default | Description |
|---|---|---|---|
sexual |
f64 |
— | Sexual content score. |
hate |
f64 |
— | Hate speech score. |
harassment |
f64 |
— | Harassment score. |
self_harm |
f64 |
— | Self-harm content score. |
sexual_minors |
f64 |
— | Sexual content involving minors score. |
hate_threatening |
f64 |
— | Hate speech that threatens violence score. |
violence_graphic |
f64 |
— | Graphic violence score. |
self_harm_intent |
f64 |
— | Intent to self-harm score. |
self_harm_instructions |
f64 |
— | Instructions for self-harm score. |
harassment_threatening |
f64 |
— | Harassment that threatens violence score. |
violence |
f64 |
— | Non-graphic violence score. |
RerankRequest
Section titled “RerankRequest”Request to rerank documents by relevance to a query.
| Field | Type | Default | Description |
|---|---|---|---|
model |
String |
— | Model ID (e.g., "cohere/rerank-english-v3.0"). |
query |
String |
— | The search query. |
documents |
Vec<RerankDocument> |
vec![] |
Documents to rerank. |
top_n |
Option<u32> |
Default::default() |
Return only the top N results. Optional. |
return_documents |
Option<bool> |
Default::default() |
Include the document content in results. Defaults to false. |
extra_body |
Option<serde_json::Value> |
Default::default() |
Provider-specific parameters merged into the request body, as chat’s extra_body is. Keys here override identically named fields. |
SearchRequest
Section titled “SearchRequest”A search request.
| Field | Type | Default | Description |
|---|---|---|---|
model |
String |
— | The model/provider to use (e.g. "brave/web-search"). |
query |
String |
— | The search query string. |
max_results |
Option<u32> |
Default::default() |
Maximum number of results to return. |
search_domain_filter |
Vec<String> |
vec![] |
Domain filter — restrict results to specific domains. |
country |
Option<String> |
Default::default() |
Country code for localized results (ISO 3166-1 alpha-2, e.g., "US", "FR"). |
extra_body |
Option<serde_json::Value> |
Default::default() |
Provider-specific parameters merged into the request body, as chat’s extra_body is. Keys here override identically named fields. |
OcrRequest
Section titled “OcrRequest”An OCR request.
| Field | Type | Default | Description |
|---|---|---|---|
model |
String |
— | The model/provider to use (e.g. "mistral/mistral-ocr-latest"). |
document |
OcrDocument |
OcrDocument::Url |
The document to process (URL or base64). |
pages |
Vec<u32> |
vec![] |
Specific pages to process (1-indexed). None means all pages. |
include_image_base64 |
Option<bool> |
Default::default() |
Whether to include base64-encoded images of each processed page. |
extra_body |
Option<serde_json::Value> |
Default::default() |
Provider-specific parameters merged into the request body, as chat’s extra_body is. Keys here override identically named fields. |
ModelsListResponse
Section titled “ModelsListResponse”Response listing available models.
| Field | Type | Default | Description |
|---|---|---|---|
object |
String |
— | Always "list" from OpenAI-compatible APIs. Stored as a plain String so non-standard provider values do not break deserialization. Defaults to empty when a provider omits the field. |
data |
Vec<ModelObject> |
vec![] |
List of available models. |
ModelObject
Section titled “ModelObject”A model available from the API.
| Field | Type | Default | Description |
|---|---|---|---|
id |
String |
— | Model ID (e.g., "gpt-4o", "claude-3-5-sonnet"). |
object |
String |
— | Always "model" from OpenAI-compatible APIs. Stored as a plain String so non-standard provider values do not break deserialization. Defaults to empty when a provider omits the field. |
created |
u64 |
— | Unix timestamp of model creation (or release date). Defaults to 0 when a provider omits it — DeepSeek and some other OpenAI-compatible providers do not return created from /v1/models. |
owned_by |
String |
— | Organization or entity that owns the model. Defaults to empty when a provider omits the field. |
CreateFileRequest
Section titled “CreateFileRequest”Request to upload a file.
| Field | Type | Default | Description |
|---|---|---|---|
file |
String |
— | Base64-encoded file data. |
purpose |
FilePurpose |
FilePurpose::Assistants |
Purpose for the file. |
filename |
Option<String> |
Default::default() |
Optional filename to associate with the upload. |
FileObject
Section titled “FileObject”An uploaded file object.
| Field | Type | Default | Description |
|---|---|---|---|
id |
String |
— | Unique file ID. |
object |
String |
— | Object type (always "file"). |
bytes |
u64 |
— | File size in bytes. |
created_at |
u64 |
— | Unix timestamp of file creation. |
filename |
String |
— | Filename. |
purpose |
String |
— | File purpose. |
status |
Option<String> |
Default::default() |
Processing status (e.g., "uploaded", "processed"). |
FileListResponse
Section titled “FileListResponse”Response from listing files.
| Field | Type | Default | Description |
|---|---|---|---|
object |
String |
— | Object type (always "list"). |
data |
Vec<FileObject> |
vec![] |
List of file objects. |
has_more |
Option<bool> |
Default::default() |
Whether more results are available. |
FileListQuery
Section titled “FileListQuery”Query parameters for listing files.
| Field | Type | Default | Description |
|---|---|---|---|
purpose |
Option<String> |
Default::default() |
Filter by file purpose (e.g., "batch", "fine-tune"). |
limit |
Option<u32> |
Default::default() |
Maximum number of results to return. Defaults to 20. |
after |
Option<String> |
Default::default() |
Pagination cursor: return results after this file ID. |
DeleteResponse
Section titled “DeleteResponse”Response from a delete operation.
| Field | Type | Default | Description |
|---|---|---|---|
id |
String |
— | ID of the deleted resource. |
object |
String |
— | Object type. |
deleted |
bool |
— | Confirmation that the resource was deleted. |
CreateBatchRequest
Section titled “CreateBatchRequest”Request to create a batch job.
| Field | Type | Default | Description |
|---|---|---|---|
input_file_id |
String |
— | ID of the uploaded input file (JSONL format). |
endpoint |
String |
— | API endpoint (e.g., "/v1/chat/completions"). |
completion_window |
String |
— | Completion window (e.g., "24h"). |
metadata |
Option<serde_json::Value> |
Default::default() |
Optional metadata to attach to the batch. |
BatchObject
Section titled “BatchObject”A batch job object.
| Field | Type | Default | Description |
|---|---|---|---|
id |
String |
— | Unique batch ID. |
object |
String |
— | Object type (always "batch"). |
endpoint |
String |
— | API endpoint (e.g., "/v1/chat/completions"). |
input_file_id |
String |
— | ID of the input file. |
completion_window |
String |
— | Completion window (e.g., "24h"). |
status |
BatchStatus |
BatchStatus::Validating |
Current job status. |
output_file_id |
Option<String> |
Default::default() |
ID of the output file (present when completed). |
error_file_id |
Option<String> |
Default::default() |
ID of the error file (present if some requests failed). |
created_at |
u64 |
— | Unix timestamp of batch creation. |
completed_at |
Option<u64> |
Default::default() |
Unix timestamp of completion (if completed). |
failed_at |
Option<u64> |
Default::default() |
Unix timestamp of failure (if failed). |
expired_at |
Option<u64> |
Default::default() |
Unix timestamp of expiration (if expired). |
request_counts |
Option<BatchRequestCounts> |
Default::default() |
Request processing counts. |
metadata |
Option<serde_json::Value> |
Default::default() |
Metadata attached to the batch. |
BatchRequestCounts
Section titled “BatchRequestCounts”Request processing counts for a batch.
| Field | Type | Default | Description |
|---|---|---|---|
total |
u64 |
— | Total requests in the batch. |
completed |
u64 |
— | Completed requests. |
failed |
u64 |
— | Failed requests. |
BatchListResponse
Section titled “BatchListResponse”Response from listing batches.
| Field | Type | Default | Description |
|---|---|---|---|
object |
String |
— | Object type (always "list"). |
data |
Vec<BatchObject> |
vec![] |
List of batch objects. |
has_more |
Option<bool> |
Default::default() |
Whether more results are available. |
first_id |
Option<String> |
Default::default() |
First batch ID in the result set (for pagination). |
last_id |
Option<String> |
Default::default() |
Last batch ID in the result set (for pagination). |
BatchListQuery
Section titled “BatchListQuery”Query parameters for listing batches.
| Field | Type | Default | Description |
|---|---|---|---|
limit |
Option<u32> |
Default::default() |
Maximum number of results to return. Defaults to 20. |
after |
Option<String> |
Default::default() |
Pagination cursor: return results after this batch ID. |
CreateResponseRequest
Section titled “CreateResponseRequest”Request to create a response via the OpenAI Responses API (POST /responses).
Provider support
Section titled “Provider support”The Responses API path is OpenAI-only. This type models the OpenAI Responses
wire format, and unlike ChatCompletionRequest the body is sent to the provider
verbatim: neither Provider.transform_request nor Provider.transform_response
runs for Responses calls, and no provider in schemas/providers.json declares a
responses endpoint.
Pointing a Responses call at a provider that does not natively serve the OpenAI
/responses contract (Anthropic, Vertex, Bedrock, Cohere, Google AI, Azure) is not
supported: the request goes out unmodified, so the provider rejects it or returns a
body that cannot be deserialized into ResponseObject. Use
ChatCompletionRequest for cross-provider work — that path applies the
per-provider request and response normalization this one does not.
ChatCompletionRequest: crate.types.ChatCompletionRequest
| Field | Type | Default | Description |
|---|---|---|---|
model |
String |
— | Model ID, as named by the OpenAI Responses API (e.g. "gpt-5"). Sent to the wire exactly as given. The Responses path performs none of the chat path’s model handling: a provider/model routing prefix is not stripped and does not re-route the request, which stays pinned to the provider the client was constructed with. |
input |
serde_json::Value |
— | Input data to process (e.g., a document to extract from). |
instructions |
Option<String> |
Default::default() |
Instructions for processing the input. |
tools |
Vec<ResponseTool> |
vec![] |
Available tools the model can use. |
temperature |
Option<f64> |
Default::default() |
Sampling temperature in [0.0, 2.0]. Defaults to 1.0. |
max_output_tokens |
Option<u64> |
Default::default() |
Maximum output tokens. |
metadata |
Option<serde_json::Value> |
Default::default() |
Optional metadata. |
extra_body |
Option<serde_json::Value> |
Default::default() |
Extra top-level parameters shallow-merged into the request body, OpenAI-Python style ({**body, **extra_body}) — keys here override identically named fields above. Use it for OpenAI Responses fields this struct does not model directly, such as the top-level reasoning.effort. This is an OpenAI escape hatch, not a cross-provider one. On the chat path the providers that consume extra_body natively (Anthropic, Vertex, Bedrock) claim it inside their own transform_request; here no provider transform runs, so the merged keys always travel to the wire as literal OpenAI Responses fields. A non-object value cannot be merged into the body root and is dropped with a warning rather than sent. |
stream |
Option<bool> |
Default::default() |
Whether to stream the response. Managed by the client layer — do not set directly. |
ResponseTool
Section titled “ResponseTool”A tool available for the response request.
| Field | Type | Default | Description |
|---|---|---|---|
tool_type |
String |
— | Tool type (e.g., “extractor”, “search”). |
config |
serde_json::Value |
— | Tool configuration (flattened into the object). |
ResponseObject
Section titled “ResponseObject”Response from a structured response request.
| Field | Type | Default | Description |
|---|---|---|---|
id |
String |
— | Unique response ID. |
object |
String |
— | Object type (e.g., "response"). |
created_at |
u64 |
— | Unix timestamp of response creation. |
model |
String |
— | Model used to generate the response. |
status |
String |
— | Status (e.g., "succeeded", "failed"). |
output |
Vec<ResponseOutputItem> |
vec![] |
Output items from the response. |
usage |
Option<ResponseUsage> |
Default::default() |
Token usage. |
error |
Option<serde_json::Value> |
Default::default() |
Error details (if status is “failed”). |
ResponseOutputItem
Section titled “ResponseOutputItem”A single output item from the response.
| Field | Type | Default | Description |
|---|---|---|---|
item_type |
String |
— | Output type (e.g., "text", "object", "error"). |
content |
serde_json::Value |
— | Output content (flattened into the object). |
ResponseUsage
Section titled “ResponseUsage”Token usage for a response.
| Field | Type | Default | Description |
|---|---|---|---|
input_tokens |
u64 |
— | Input tokens used. |
output_tokens |
u64 |
— | Output tokens used. |
total_tokens |
u64 |
— | Total tokens used. |
LlmConfig
Section titled “LlmConfig”Canonical configuration for an LLM client.
All fields except model are optional so that partially-specified
configs (e.g. from environment-driven defaults) round-trip cleanly.
Convert to a runtime client configuration via
LlmConfig.into_client_builder.
temperature and max_tokens are request-time parameters rather than
client-level settings; they are carried on this struct for callers to
read when building individual requests, and are intentionally not
mapped by LlmConfig.into_client_builder.
Implements Debug manually (see below) so api_key and header values are
redacted rather than printed in full.
| Field | Type | Default | Description |
|---|---|---|---|
model |
String |
— | Model identifier (e.g. "gpt-4o", "bedrock/anthropic.claude-3-sonnet-20240229-v1:0"). |
api_key |
Option<String> |
Default::default() |
API key for authentication. |
base_url |
Option<String> |
Default::default() |
Override base URL. When set, all requests go here and provider auto-detection is skipped. The model is sent verbatim, except that a leading X/ is stripped when X is the provider named by model_hint ("openai" strips openai/gpt-4o-mini to gpt-4o-mini). |
timeout_secs |
Option<u64> |
Default::default() |
Request timeout, in seconds. |
connect_timeout_secs |
Option<u64> |
Default::default() |
Connect timeout, in seconds. A connect timeout is retried up to max_retries times. Zero is an error on every target. wasm32 ignores any other value. |
max_retries |
Option<u32> |
Default::default() |
Maximum number of retries on 429 / 5xx responses. |
temperature |
Option<f64> |
Default::default() |
Sampling temperature for requests built from this config. |
max_tokens |
Option<u64> |
Default::default() |
Maximum number of tokens to generate for requests built from this config. |
load_env |
Option<bool> |
Default::default() |
Automatically load the API key from the provider’s environment variable when no explicit key is provided (default: true). |
headers |
HashMap<String, String> |
HashMap::new() |
Extra headers sent on every request. |
providers |
Vec<LlmProviderConfig> |
vec![] |
Custom provider configurations, in addition to the built-in providers. |
cache |
Option<LlmCacheConfig> |
Default::default() |
Response cache configuration. |
budget |
Option<LlmBudgetConfig> |
Default::default() |
Budget enforcement configuration. |
rate_limit |
Option<LlmRateLimitConfig> |
Default::default() |
Per-model rate limiting configuration. |
in_flight_limit |
Option<LlmInFlightLimitConfig> |
Default::default() |
Global per-client in-flight provider request limit. |
cost_tracking |
Option<bool> |
Default::default() |
Enable per-request cost tracking. |
tracing |
Option<bool> |
Default::default() |
Enable OpenTelemetry-compatible tracing spans. |
cooldown_secs |
Option<u64> |
Default::default() |
Cooldown duration after transient errors, in seconds. |
health_check_secs |
Option<u64> |
Default::default() |
Background health check interval, in seconds. |
bedrock |
Option<BedrockConfig> |
Default::default() |
AWS Bedrock configuration (region, credentials, cross-region routing). |
LlmCacheConfig
Section titled “LlmCacheConfig”Response cache configuration.
| Field | Type | Default | Description |
|---|---|---|---|
max_entries |
Option<usize> |
Default::default() |
Maximum number of cached entries. |
ttl_seconds |
Option<u64> |
Default::default() |
Cache entry time-to-live, in seconds. |
backend |
Option<String> |
Default::default() |
Cache backend name (e.g. "memory", or an opendal scheme). |
backend_config |
HashMap<String, String> |
HashMap::new() |
Backend-specific configuration key/value pairs. |
LlmBudgetConfig
Section titled “LlmBudgetConfig”Budget enforcement configuration.
| Field | Type | Default | Description |
|---|---|---|---|
global_limit |
Option<f64> |
Default::default() |
Global spend limit in USD. |
model_limits |
HashMap<String, f64> |
HashMap::new() |
Per-model spend limits in USD, keyed by model name. |
enforcement |
Option<String> |
Default::default() |
Enforcement mode: "hard" (reject over-budget requests) or "soft" (log only). |
LlmRateLimitConfig
Section titled “LlmRateLimitConfig”Per-model rate limiting configuration.
| Field | Type | Default | Description |
|---|---|---|---|
rpm |
Option<u32> |
Default::default() |
Requests per minute limit. |
tpm |
Option<u64> |
Default::default() |
Tokens per minute limit. |
window_seconds |
Option<u64> |
Default::default() |
Rate limit window, in seconds. |
LlmInFlightLimitConfig
Section titled “LlmInFlightLimitConfig”Global per-client in-flight provider request limit.
| Field | Type | Default | Description |
|---|---|---|---|
max_in_flight |
Option<usize> |
Default::default() |
Maximum simultaneously outstanding provider requests. None means unlimited. |
LlmProviderConfig
Section titled “LlmProviderConfig”A custom provider configuration entry.
| Field | Type | Default | Description |
|---|---|---|---|
name |
String |
— | Provider name, used to key model prefix matching. |
base_url |
String |
— | Base URL for the provider’s OpenAI-compatible API. |
auth_header |
Option<String> |
Default::default() |
Header name used to carry the API key (defaults to Authorization when unset). |
model_prefixes |
Vec<String> |
vec![] |
Model name prefixes routed to this provider (e.g. ["my-provider/"]). |
BedrockConfig
Section titled “BedrockConfig”AWS Bedrock configuration.
All fields are optional; anything left unset falls back to the standard
AWS environment variables (AWS_DEFAULT_REGION / AWS_REGION,
AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN,
BEDROCK_CROSS_REGION).
A Bedrock API key in AWS_BEARER_TOKEN_BEDROCK authenticates on its own and
needs none of the credential fields below. It is read per request rather than
from this struct; in a build that can sign (the bedrock feature) a credential
set here takes precedence over it.
Implements Debug manually (see below) so the AWS credential fields are
redacted rather than printed in full.
| Field | Type | Default | Description |
|---|---|---|---|
region |
Option<String> |
Default::default() |
AWS region (e.g. "us-east-1"). |
cross_region_prefix |
Option<String> |
Default::default() |
Cross-region inference profile prefix (e.g. "us"). |
access_key_id |
Option<String> |
Default::default() |
Explicit AWS access key ID. |
secret_access_key |
Option<String> |
Default::default() |
Explicit AWS secret access key. |
session_token |
Option<String> |
Default::default() |
Explicit AWS session token (temporary credentials). |
WaitForBatchConfig
Section titled “WaitForBatchConfig”Configuration for polling a batch until terminal status.
All time values are in seconds as f64 so the struct bridges across FFI
boundaries without requiring a Duration shim.
| Field | Type | Default | Description |
|---|---|---|---|
initial_interval_secs |
f64 |
5 |
Initial interval between polls, in seconds. |
max_interval_secs |
f64 |
60 |
Maximum interval between polls (backoff plateau), in seconds. |
backoff_multiplier |
f32 |
1.5 |
Exponential backoff multiplier (e.g., 1.5 increases delay by 50% each poll). |
timeout_secs |
Option<f64> |
None |
Optional timeout in seconds — polling fails if this duration is exceeded. |
CustomProviderConfig
Section titled “CustomProviderConfig”Configuration for registering a custom LLM provider at runtime.
| Field | Type | Default | Description |
|---|---|---|---|
name |
String |
— | Unique name for this provider (e.g., “my-provider”). |
base_url |
String |
— | Base URL for the provider’s API (e.g., <https://api.my-provider.com/v1>). |
auth_header |
AuthHeaderFormat |
— | Authentication header format. |
model_prefixes |
Vec<String> |
— | Model name prefixes that route to this provider (e.g., ["my-"]). |
ProviderCapabilities
Section titled “ProviderCapabilities”Static capability flags for a provider.
Each flag indicates whether the provider’s models generally support that
feature. For providers that aggregate many underlying models (e.g. Bedrock,
OpenRouter, vLLM) the flags reflect the superset of available model
capabilities — a flag being True means at least one model supports the
feature, not every model.
All flags default to False so that newly added providers are safe.
Access via the crate-level capabilities function:
| Field | Type | Default | Description |
|---|---|---|---|
vision |
bool |
— | The provider accepts image input in chat messages. |
reasoning |
bool |
— | The provider supports extended-thinking / reasoning tokens. |
structured_output |
bool |
— | The provider supports JSON-mode or response_format structured output. |
function_calling |
bool |
— | The provider supports tool / function calling. |
audio_in |
bool |
— | The provider accepts audio as input. |
audio_out |
bool |
— | The provider can generate audio / TTS output. |
video_in |
bool |
— | The provider accepts video as input. |
ProviderConfig
Section titled “ProviderConfig”Static configuration for a single provider entry in providers.json.
This struct deliberately does not include capability flags or streaming
format, which are accessed via the capabilities function.
| Field | Type | Default | Description |
|---|---|---|---|
name |
String |
— | Provider identifier (matches the entry key in providers.json). |
display_name |
Option<String> |
None |
Human-readable provider name shown in UIs. |
base_url |
Option<String> |
None |
Base URL used as the default for this provider’s HTTP client. |
auth |
Option<AuthConfig> |
None |
Authentication scheme metadata (auth type + env var holding the key). |
endpoints |
Vec<String> |
None |
Supported endpoint kinds (e.g. chat, embeddings). |
model_prefixes |
Vec<String> |
None |
Model-name prefixes claimed by this provider (e.g. ["gpt-", "o1-"]). |
param_mappings |
HashMap<String, String> |
None |
Parameter key renaming for this provider. Each entry maps an OpenAI-spec field name (e.g. "max_completion_tokens") to the name this provider expects (e.g. "max_tokens"). Applied automatically by ConfigDrivenProvider::transform_request. |
AuthConfig
Section titled “AuthConfig”Auth configuration block.
| Field | Type | Default | Description |
|---|---|---|---|
auth_type |
AuthType |
— | Auth scheme classification. |
env_var |
Option<String> |
None |
Name of the environment variable that holds the API key (e.g. "OPENAI_API_KEY"). Holds the variable name, never the secret value. |
CatalogInfo
Section titled “CatalogInfo”Provenance for the active model catalog.
| Field | Type | Default | Description |
|---|---|---|---|
source |
String |
— | Upstream catalog source. |
source_sha256 |
String |
— | SHA-256 of the upstream source payload. |
fetched |
String |
— | Date the upstream source was fetched. |
library_version |
String |
— | Library version recorded when the catalog was generated. |
schema_version |
u64 |
— | Catalog schema version. |
origin |
String |
— | Active catalog origin: "embedded" or "overlay". |
ModelInfo
Section titled “ModelInfo”Public, FFI-friendly snapshot of a model’s pricing and capability
metadata, projected from ModelPricing.
Unlike ModelPricing (which is excluded from binding generation),
ModelInfo is an owned plain-data DTO safe to hand across the FFI
boundary — see model_info.
| Field | Type | Default | Description |
|---|---|---|---|
matched_key |
String |
— | Catalog key that supplied this metadata. |
exact |
bool |
— | Whether the requested model key matched the catalog exactly. |
price_known |
bool |
— | Whether the per-token price is known; false prevents cost estimation. |
input_cost_per_token |
f64 |
— | Cost in USD per input (prompt) token. |
output_cost_per_token |
f64 |
— | Cost in USD per output (completion) token. |
cache_read_input_token_cost |
Option<f64> |
Default::default() |
Cost in USD per cached input token (cache hit / read). |
cache_creation_input_token_cost |
Option<f64> |
Default::default() |
Cost in USD per token written to the prompt cache. |
input_cost_per_audio_token |
Option<f64> |
Default::default() |
Cost in USD per input audio token. |
output_cost_per_audio_token |
Option<f64> |
Default::default() |
Cost in USD per output audio token. |
output_cost_per_reasoning_token |
Option<f64> |
Default::default() |
Cost in USD per reasoning (extended-thinking) output token. |
max_tokens |
Option<u64> |
Default::default() |
Total context window size in tokens (input + output). |
max_input_tokens |
Option<u64> |
Default::default() |
Maximum input (prompt) tokens accepted. |
max_output_tokens |
Option<u64> |
Default::default() |
Maximum output (completion) tokens the model can generate. |
mode |
Option<String> |
Default::default() |
Best-effort operating mode, e.g. "chat", "embedding". |
supports_vision |
Option<bool> |
Default::default() |
The model accepts image input. |
supports_function_calling |
Option<bool> |
Default::default() |
The model supports tool / function calling. |
supports_reasoning |
Option<bool> |
Default::default() |
The model supports extended-thinking / reasoning tokens. |
supports_structured_output |
Option<bool> |
Default::default() |
The model supports JSON-mode or response_format structured output. |
supports_audio_input |
Option<bool> |
Default::default() |
The model accepts audio input. |
supports_audio_output |
Option<bool> |
Default::default() |
The model can generate audio output. |
supports_prompt_caching |
Option<bool> |
Default::default() |
The model supports prompt caching. |
tiers |
Vec<ModelTier> |
vec![] |
Context-tiered pricing overrides, sorted by ascending min_context_tokens. Empty when the model has flat pricing. |
ModelTier
Section titled “ModelTier”Public, FFI-friendly snapshot of a single context-window pricing tier,
projected from PricingTier.
| Field | Type | Default | Description |
|---|---|---|---|
min_context_tokens |
u64 |
— | The tier applies when the prompt/context token count is at least this value. |
input_cost_per_token |
f64 |
— | Cost in USD per input (prompt) token within this tier. |
output_cost_per_token |
f64 |
— | Cost in USD per output (completion) token within this tier. |
cache_read_input_token_cost |
Option<f64> |
Default::default() |
Cost in USD per cached input token within this tier. |
cache_creation_input_token_cost |
Option<f64> |
Default::default() |
Cost in USD per cache-write token within this tier. |
input_cost_per_audio_token |
Option<f64> |
Default::default() |
Cost in USD per input audio token within this tier. |
output_cost_per_audio_token |
Option<f64> |
Default::default() |
Cost in USD per output audio token within this tier. |
output_cost_per_reasoning_token |
Option<f64> |
Default::default() |
Cost in USD per reasoning output token within this tier. |
CatalogModel
Section titled “CatalogModel”One model of the built-in catalog, as returned by the catalog listing.
An owned plain-data DTO. It carries what a model picker needs: the name, the provider, the mode and the limits and abilities. The prices stay in the model lookup, which accepts the key of this entry unchanged.
| Field | Type | Default | Description |
|---|---|---|---|
key |
String |
— | Full catalog name, “provider/model”. The model lookup and the exact model lookup accept it unchanged. |
provider |
String |
— | Catalog provider id, such as “openai”. |
model |
String |
— | Model id within the provider, e.g. "gpt-4o". |
mode |
Option<String> |
Default::default() |
Best-effort operating mode, e.g. "chat", "embedding". |
max_tokens |
Option<u64> |
Default::default() |
Total context window size in tokens (input + output). |
max_input_tokens |
Option<u64> |
Default::default() |
Maximum input (prompt) tokens accepted. |
max_output_tokens |
Option<u64> |
Default::default() |
Maximum output (completion) tokens the model can generate. |
supports_vision |
Option<bool> |
Default::default() |
The model accepts image input. |
supports_function_calling |
Option<bool> |
Default::default() |
The model supports tool / function calling. |
supports_reasoning |
Option<bool> |
Default::default() |
The model supports extended-thinking / reasoning tokens. |
supports_structured_output |
Option<bool> |
Default::default() |
The model supports JSON-mode or response_format structured output. |
supports_audio_input |
Option<bool> |
Default::default() |
The model accepts audio input. |
supports_audio_output |
Option<bool> |
Default::default() |
The model can generate audio output. |
supports_prompt_caching |
Option<bool> |
Default::default() |
The model supports prompt caching. |
CatalogRefreshConfig
Section titled “CatalogRefreshConfig”Plain-data configuration for refresh_catalog.
Deliberately FFI/binding-friendly: no Duration or PathBuf, just
primitives that translate directly across language boundaries.
| Field | Type | Default | Description |
|---|---|---|---|
enabled |
bool |
false |
Runtime catalog refresh is entirely opt-in: when false, refresh_catalog is a no-op that returns Ok(RefreshOutcome::Disabled) without touching the network, the filesystem, or the overlay registry. |
source_url |
String |
"<https://github.com/xberg-io/liter-llm/releases/download/model-catalog/catalog.json>" |
Source URL to fetch catalog.json from. Must be https. Defaults to DEFAULT_CATALOG_URL; configurable so self-hosted mirrors work. |
ttl_seconds |
u64 |
86400 |
How long a cached catalog.json remains valid before a network refetch is attempted, in seconds. |
cache_path |
Option<String> |
None |
Filesystem path for the on-disk cache. None uses a default path under the current user’s platform cache directory. |
BudgetConfig
Section titled “BudgetConfig”Configuration for budget enforcement.
| Field | Type | Default | Description |
|---|---|---|---|
global_limit |
Option<f64> |
None |
Maximum total spend across all models, in USD. None means unlimited. |
model_limits |
HashMap<String, f64> |
HashMap::new() |
Per-model spending limits in USD. Models not listed here are only constrained by global_limit. |
enforcement |
Enforcement |
Enforcement::Hard |
Whether to reject requests or merely warn when a limit is exceeded. |
CacheConfig
Section titled “CacheConfig”Configuration for the response cache.
| Field | Type | Default | Description |
|---|---|---|---|
max_entries |
usize |
256 |
Maximum number of cached entries. |
ttl |
Duration |
300000ms |
Time-to-live for each cached entry. |
backend |
CacheBackend |
CacheBackend::Memory |
Storage backend to use. |
InFlightLimitConfig
Section titled “InFlightLimitConfig”Configuration for the global per-client in-flight request limit.
| Field | Type | Default | Description |
|---|---|---|---|
max_in_flight |
Option<usize> |
Default::default() |
Maximum simultaneously outstanding provider requests. None means unlimited. |
RateLimitConfig
Section titled “RateLimitConfig”Configuration for per-model rate limits.
| Field | Type | Default | Description |
|---|---|---|---|
rpm |
Option<u32> |
None |
Maximum requests per window. None means unlimited. |
tpm |
Option<u64> |
None |
Maximum tokens per window. None means unlimited. |
window |
Duration |
60000ms |
Fixed window duration (defaults to 60 s). |
Other Types
Section titled “Other Types”ChatCompletionTool
Section titled “ChatCompletionTool”A tool the model can invoke (currently, all tools are functions).
| Field | Type | Default | Description |
|---|---|---|---|
tool_type |
ToolType |
— | Tool type (always “function” in OpenAI spec). |
function |
FunctionDefinition |
— | Function definition with name, description, and JSON schema parameters. |
FunctionDefinition
Section titled “FunctionDefinition”Function definition exposed to the model.
| Field | Type | Default | Description |
|---|---|---|---|
name |
String |
— | Name of the function. Required and must be alphanumeric + underscores. |
description |
Option<String> |
/* serde(default) */ |
Human-readable description explaining what the function does. |
parameters |
Option<serde_json::Value> |
/* serde(default) */ |
JSON Schema defining the function’s parameters. |
strict |
Option<bool> |
/* serde(default) */ |
If true, enforce strict JSON schema validation for arguments. |
ToolCall
Section titled “ToolCall”A tool call the model wants to execute.
| Field | Type | Default | Description |
|---|---|---|---|
id |
String |
— | Unique ID for this call, used to reference in tool result messages. |
call_type |
ToolType |
— | Tool type (always “function”). |
function |
FunctionCall |
— | Function name and arguments. |
FunctionCall
Section titled “FunctionCall”Function call details.
| Field | Type | Default | Description |
|---|---|---|---|
name |
String |
— | Function name. |
arguments |
String |
— | Arguments as a JSON string (parse with serde_json::from_str). |
EmbeddingResponse
Section titled “EmbeddingResponse”Embedding response.
| Field | Type | Default | Description |
|---|---|---|---|
object |
String |
— | Always "list" from OpenAI-compatible APIs. Stored as a plain String so non-standard provider values do not break deserialization. |
data |
Vec<EmbeddingObject> |
— | List of embeddings. |
model |
String |
— | Model used to generate embeddings. |
usage |
Option<Usage> |
/* serde(default) */ |
Token usage (input tokens only; embeddings have zero output tokens). |
EmbeddingObject
Section titled “EmbeddingObject”A single embedding vector.
| Field | Type | Default | Description |
|---|---|---|---|
object |
String |
— | Always "embedding" from OpenAI-compatible APIs. Stored as a plain String so non-standard provider values do not break deserialization. |
embedding |
Vec<f32> |
— | The embedding vector. Providers may return this as a JSON float array or, when encoding_format: "base64" was requested, as a base64 string of little-endian f32 bytes. Base64 responses are decoded on read; this field always serializes back out as a JSON float array. |
index |
u32 |
— | Index in the batch (corresponds to input order). |
ModerationResponse
Section titled “ModerationResponse”Response from the moderation endpoint.
| Field | Type | Default | Description |
|---|---|---|---|
id |
String |
— | Unique identifier for this moderation request. |
model |
String |
— | Model used for classification. |
results |
Vec<ModerationResult> |
— | Results for each input string. |
RerankResponse
Section titled “RerankResponse”Response from the rerank endpoint.
| Field | Type | Default | Description |
|---|---|---|---|
id |
Option<String> |
None |
Unique identifier for this rerank request. |
results |
Vec<RerankResult> |
— | Reranked documents in order of relevance. |
meta |
Option<serde_json::Value> |
/* serde(default) */ |
Optional metadata about the reranking operation. |
SearchResponse
Section titled “SearchResponse”A search response.
| Field | Type | Default | Description |
|---|---|---|---|
results |
Vec<SearchResult> |
— | List of search results. |
model |
String |
— | Model/provider that performed the search. |
OcrResponse
Section titled “OcrResponse”An OCR response.
| Field | Type | Default | Description |
|---|---|---|---|
pages |
Vec<OcrPage> |
— | Extracted pages in order. |
model |
String |
— | Model/provider used for OCR. |
usage |
Option<Usage> |
/* serde(default) */ |
Token usage, if reported by the provider. |
OcrPage
Section titled “OcrPage”A single page of OCR output.
| Field | Type | Default | Description |
|---|---|---|---|
index |
u32 |
— | Page index (0-based). |
markdown |
String |
— | Extracted page content as Markdown. |
images |
Vec<OcrImage> |
/* serde(default) */ |
Embedded images extracted from the page (if include_image_base64 was true). |
dimensions |
Option<PageDimensions> |
/* serde(default) */ |
Page dimensions in pixels, if available. |
OcrImage
Section titled “OcrImage”An image extracted from an OCR page.
| Field | Type | Default | Description |
|---|---|---|---|
id |
String |
— | Unique image identifier within the document. |
image_base64 |
Option<String> |
/* serde(default) */ |
Base64-encoded image data (if include_image_base64 was true). |
PageDimensions
Section titled “PageDimensions”Page dimensions in pixels.
| Field | Type | Default | Description |
|---|---|---|---|
width |
u32 |
— | Width in pixels. |
height |
u32 |
— | Height in pixels. |
DefaultClient
Section titled “DefaultClient”Default client implementation backed by reqwest.
Sends requests to 174 LLM providers with automatic provider detection
and per-request routing. The provider is resolved at construction time
from model_hint (or defaults to OpenAI), but individual requests can
override the provider via model name prefix (e.g. "anthropic/claude-3-5-sonnet"
routes to Anthropic regardless of construction-time setting).
When the model prefix does not match any known provider, the construction-time provider is used as the fallback. This enables seamless migration between providers by changing only the model name.
The provider is stored behind an Arc so it can be shared cheaply into
async closures and streaming tasks. Pre-computed auth headers and extra
headers are cached at construction to avoid redundant encoding on every request.
Opaque type — fields are not directly accessible.
ChunkMiddleware
Section titled “ChunkMiddleware”A per-chunk transformation in the StreamPipeline.
Each middleware receives a typed chunk and returns Ok(Some(chunk))
to pass it through (optionally modified), Ok(None) to drop the chunk,
or Err(e) to propagate a stream error.
The trait is object-safe so multiple middleware implementations can be
chained inside StreamPipeline.
Opaque type — fields are not directly accessible.
HealthChecker
Section titled “HealthChecker”Abstraction over a health probe strategy.
Implementors issue a lightweight probe against upstream (typically a
provider base URL or named identifier) and report HealthStatus.
Opaque type — fields are not directly accessible.
IntentPrototype
Section titled “IntentPrototype”An intent prototype: (intent_name, prototype_embedding, target_model_id).
| Field | Type | Default | Description |
|---|---|---|---|
name |
String |
— | Human-readable name for the intent (used in logs/metrics). |
embedding |
Vec<f64> |
— | Pre-computed embedding vector for this intent. |
model |
String |
— | Model to route to when this intent is detected. |
AssistantContent
Section titled “AssistantContent”Content shape for assistant messages.
#[serde(untagged)] means providers returning a plain scalar string for the
content field still deserialise correctly into AssistantContent::Text(_).
Providers returning an array of typed parts (e.g. after an image-generation
or audio-synthesis request) deserialise into AssistantContent::Parts(_).
| Variant | Description |
|---|---|
Text |
Plain text response (the common case for text-only models). — Fields: _0: String |
Parts |
Structured parts — text, refusals, output images, output audio. — Fields: _0: Vec<AssistantPart> |
AssistantPart
Section titled “AssistantPart”One part of a structured assistant response.
#[serde(tag = "type", rename_all = "snake_case")] matches OpenAI’s
parts-spec discriminator ("type": "text", "type": "output_image", …).
| Variant | Wire value | Description |
|---|---|---|
Text |
text |
A text segment of the response. — Fields: text: String |
Refusal |
refusal |
A refusal — the model declined to respond. — Fields: refusal: String |
OutputImage |
output_image |
An image produced by the model (e.g. gpt-image-1, Gemini Imagen). — Fields: image_url: ImageUrl |
OutputAudio |
output_audio |
Audio produced by the model (e.g. gpt-4o-audio-preview). — Fields: audio: AudioContent |
AuthHeaderFormat
Section titled “AuthHeaderFormat”How the API key is sent in the HTTP request.
| Variant | Description |
|---|---|
Bearer |
Bearer token: Authorization: Bearer <key> |
ApiKey |
Custom header: e.g., X-Api-Key: <key> — Fields: _0: String |
None |
No authentication required. |
AuthType
Section titled “AuthType”Auth scheme used by a provider.
| Variant | Wire value | Description |
|---|---|---|
Bearer |
bearer |
Standard Authorization: Bearer <key> header. |
ApiKey |
api-key |
x-api-key: <key> header (also handles "header" and "x-api-key" aliases). |
None |
none |
No authentication header required. |
Unknown |
unknown |
Unrecognised auth scheme — falls back to bearer. |
BatchStatus
Section titled “BatchStatus”Status of a batch job.
| Variant | Wire value | Description |
|---|---|---|
Validating |
validating |
Validating the input file. |
Failed |
failed |
Job failed. |
InProgress |
in_progress |
Job is running. |
Finalizing |
finalizing |
Finalizing results. |
Completed |
completed |
Job completed successfully. |
Expired |
expired |
Job expired before completion. |
Cancelling |
cancelling |
Job is being cancelled. |
Cancelled |
cancelled |
Job has been cancelled. |
CacheBackend
Section titled “CacheBackend”Storage backend for the response cache.
| Variant | Wire value | Description |
|---|---|---|
Memory |
memory |
In-memory LRU cache (default). No external dependencies. |
OpenDal |
open_dal |
OpenDAL-backed storage. Supports 40+ backends (S3, Redis, GCS, local FS, etc.). — Fields: scheme: String, config: HashMap<String, String> |
CircuitState
Section titled “CircuitState”Observable state of a circuit breaker.
| Variant | Description |
|---|---|
Closed |
Requests flow through normally. |
Open |
All requests are rejected; the circuit is waiting for the backoff to elapse. |
HalfOpen |
One probe request is allowed through to test service health. |
ContentPart
Section titled “ContentPart”A single content part in a user message — text, image, document, or audio.
| Variant | Wire value | Description |
|---|---|---|
Text |
text |
Plain text. — Fields: text: String |
ImageUrl |
image_url |
Image identified by URL (with optional detail level). — Fields: image_url: ImageUrl |
Document |
document |
Document file (PDF, CSV, etc.) as base64 or URL. — Fields: document: DocumentContent |
InputAudio |
input_audio |
Audio input as base64. — Fields: input_audio: AudioContent |
EmbeddingContentPart
Section titled “EmbeddingContentPart”A content part in a multimodal embedding input.
| Variant | Wire value | Description |
|---|---|---|
Text |
text |
Plain text. — Fields: text: String |
ImageUrl |
image_url |
Image identified by a data URL or HTTP/HTTPS URL. — Fields: image_url: ImageUrl |
ImageBase64 |
image_base64 |
Image encoded as a complete data URL. — Fields: image_base64: String |
EmbeddingFormat
Section titled “EmbeddingFormat”The format in which the embedding vectors are returned.
| Variant | Wire value | Description |
|---|---|---|
Float |
float |
32-bit floating-point numbers (default). |
Base64 |
base64 |
Base64-encoded string representation of the floats. |
EmbeddingInput
Section titled “EmbeddingInput”Text, texts, or multimodal content to embed.
| Variant | Description |
|---|---|
Single |
Single text string. — Fields: _0: String |
Multiple |
Multiple text strings (batch embedding). — Fields: _0: Vec<String> |
Multimodal |
Text and image parts for a single multimodal embedding. — Fields: _0: Vec<EmbeddingContentPart> |
Enforcement
Section titled “Enforcement”How budget limits are enforced.
| Variant | Description |
|---|---|
Hard |
Reject requests that would exceed the budget with LiterLlmError::BudgetExceeded. |
Soft |
Allow requests through but emit a tracing::warn! when the budget is exceeded. |
FilePurpose
Section titled “FilePurpose”Purpose of an uploaded file.
| Variant | Wire value | Description |
|---|---|---|
Assistants |
assistants |
File for use with Assistants API. |
Batch |
batch |
File for batch processing. |
FineTune |
fine-tune |
File for fine-tuning. |
Vision |
vision |
File for vision/image tasks. |
FinishReason
Section titled “FinishReason”Why a choice stopped generating tokens.
| Variant | Wire value | Description |
|---|---|---|
Stop |
stop |
Stop |
Length |
length |
Length |
ToolCalls |
tool_calls |
Tool calls |
ContentFilter |
content_filter |
Content filter |
FunctionCall |
function_call |
Deprecated legacy finish reason; retained for API compatibility. |
Other |
other |
Catch-all for unknown finish reasons returned by non-OpenAI providers. Note: this intentionally does not carry the original string (e.g. Other(String)). Using #[serde(other)] requires a unit variant, and switching to #[serde(untagged)] would change deserialization semantics for all variants. The original value can be recovered by inspecting the raw JSON if needed. |
HealthStatus
Section titled “HealthStatus”The result of a single health probe.
| Variant | Description |
|---|---|
Healthy |
The probe succeeded; the upstream is reachable. |
Unhealthy |
The probe failed; the upstream may be down. |
ImageDetail
Section titled “ImageDetail”Image detail level controlling token cost and processing.
| Variant | Wire value | Description |
|---|---|---|
Low |
low |
Low detail: scales image to 512x512, uses fewer tokens. |
High |
high |
High detail: processes up to 2x2 grid of tiles, higher token cost. |
Auto |
auto |
Auto: model chooses low or high based on image dimensions. |
Message
Section titled “Message”A chat message in a conversation.
| Variant | Wire value | Description |
|---|---|---|
System |
system |
System — Fields (flattened into the tagged object): content: UserContent, name: String |
User |
user |
User — Fields (flattened into the tagged object): content: UserContent, name: String |
Assistant |
assistant |
Assistant — Fields (flattened into the tagged object): content: AssistantContent, name: String, tool_calls: Vec<ToolCall>, refusal: String, function_call: FunctionCall, reasoning_content: String, thinking_blocks: Vec<ThinkingBlock> |
Tool |
tool |
Tool — Fields (flattened into the tagged object): content: UserContent, tool_call_id: String, name: String |
Developer |
developer |
Developer — Fields (flattened into the tagged object): content: String, name: String |
Function |
function |
Deprecated legacy function-role message; retained for API compatibility. — Fields (flattened into the tagged object): content: String, name: String |
Modality
Section titled “Modality”Output modality requested from the model.
Passed as modalities: ["text", "audio"] (OpenAI) or translated to
generationConfig.responseModalities (Gemini / Vertex AI).
| Variant | Wire value | Description |
|---|---|---|
Text |
text |
Text output (the default for all providers). |
Audio |
audio |
Audio / speech output. |
Image |
image |
Image output (Gemini Imagen, gpt-image-1). |
ModerationInput
Section titled “ModerationInput”Input to the moderation endpoint — a single string or multiple strings.
| Variant | Description |
|---|---|
Single |
Single text string. — Fields: _0: String |
Multiple |
Multiple text strings (batch moderation). — Fields: _0: Vec<String> |
OcrDocument
Section titled “OcrDocument”Document input for OCR — either a URL or inline base64 data.
| Variant | Wire value | Description |
|---|---|---|
Url |
document_url |
A publicly accessible document URL. — Fields: url: String |
Base64 |
base64 |
Inline base64-encoded document data. — Fields: data: String, media_type: String |
ReasoningEffort
Section titled “ReasoningEffort”Controls how much reasoning effort the model should use.
| Variant | Wire value | Description |
|---|---|---|
Low |
low |
Low |
Medium |
medium |
Medium |
High |
high |
High |
Minimal |
minimal |
Minimal |
Max |
max |
Max |
None |
none |
Request that reasoning be turned off. Anthropic Opus 5.5 and Fable use low adaptive thinking. |
RefreshOutcome
Section titled “RefreshOutcome”Result of a refresh_catalog call.
| Variant | Description |
|---|---|
Disabled |
config.enabled was false; no network, filesystem, or overlay activity occurred. |
FromCache |
The on-disk cache was fresh (age < ttl_seconds); the overlay was installed from the cached file without a network request. |
Fetched |
The catalog was fetched over the network, the cache file was (best-effort) refreshed, and the overlay was installed from the fetched catalog. |
RerankDocument
Section titled “RerankDocument”A document to be reranked — either a plain string or an object with a text field.
| Variant | Description |
|---|---|
Text |
Plain text document content. — Fields: _0: String |
Object |
Document with explicit text field (may include metadata). — Fields: text: String |
ResponseFormat
Section titled “ResponseFormat”Wire format for the chat completions response_format field.
Provider mapping
Section titled “Provider mapping”-
OpenAI (and OpenAI-compatible providers): emitted verbatim as
{"type": "json_schema", "json_schema": {...}}per the chat-completions spec. -
Gemini / Vertex AI: translated to
generationConfig.responseMimeType = "application/json"andgenerationConfig.responseJsonSchema = <schema>. Thename,description, andstrictfields are dropped — Gemini’s structured-output API does not consume them. -
Anthropic: no native JSON mode. A system instruction is prepended asking the model to respond with valid JSON.
strictis advisory only; callers should still validate the returned JSON if the schema is load-bearing.
| Variant | Wire value | Description |
|---|---|---|
Text |
text |
Plain text output (default). |
JsonObject |
json_object |
Output must be valid JSON object (no schema validation). |
JsonSchema |
json_schema |
Output must conform to the specified JSON schema. — Fields: json_schema: JsonSchemaFormat |
StopSequence
Section titled “StopSequence”Stop sequence(s) that cause the model to stop generating.
| Variant | Description |
|---|---|
Single |
Single stop sequence. — Fields: _0: String |
Multiple |
Multiple stop sequences. — Fields: _0: Vec<String> |
StreamFormat
Section titled “StreamFormat”The streaming wire format a provider uses for its response stream.
Most providers use standard Server-Sent Events (SSE). AWS Bedrock uses a proprietary binary EventStream framing. Ollama’s native API streams newline-delimited JSON.
Deserialized from the streaming_format JSON field via serde.
| Variant | Wire value | Description |
|---|---|---|
Sse |
sse |
Standard Server-Sent Events (text/event-stream). |
AwsEventStream |
aws_event_stream |
AWS EventStream binary framing (application/vnd.amazon.eventstream). |
Ndjson |
ndjson |
Newline-delimited JSON (application/x-ndjson): one JSON object per line. |
ToolChoice
Section titled “ToolChoice”Tool usage mode or a specific tool to call.
| Variant | Description |
|---|---|
Mode |
Predefined mode: auto, required, or none. — Fields: _0: ToolChoiceMode |
Specific |
Force a specific tool to be called. — Fields: _0: SpecificToolChoice |
ToolChoiceMode
Section titled “ToolChoiceMode”Tool choice mode.
| Variant | Wire value | Description |
|---|---|---|
Auto |
auto |
Model may or may not call tools; default behavior. |
Required |
required |
Model must call at least one tool. |
None |
none |
Model must not call any tools. |
ToolType
Section titled “ToolType”The type discriminator for tool/tool-call objects.
Per the OpenAI spec this is always "function". Using an enum enforces
that constraint at the type level and rejects any other value on
deserialization.
| Variant | Wire value | Description |
|---|---|---|
Function |
function |
Function |
UserContent
Section titled “UserContent”User message content as either plain text or a list of multimodal parts.
| Variant | Description |
|---|---|
Text |
Plain text content. — Fields: _0: String |
Parts |
Array of content parts (text, images, documents, audio). — Fields: _0: Vec<ContentPart> |