Configuration Reference

Configuration Reference

This page documents all configuration types and their defaults across all languages.

SystemMessage

System message guiding model behavior for the entire conversation.

Field	Type	Default	Description
content	UserContent	UserContent.TEXT	Instructions or context that apply throughout the conversation. Accepts either a plain text string or an array of content parts, mirroring UserContent so that Message.system_with_parts works.
name	str | None	None	Optional name for the system message source.

---

UserMessage

User message in the conversation.

Field	Type	Default	Description
content	UserContent	UserContent.TEXT	Message content as plain text or array of content parts (text, images, documents, audio).
name	str | None	None	Optional name for the user.

---

ImageUrl

An image URL reference with optional detail level for processing.

Field	Type	Default	Description
url	str	—	URL of the image (data URI or HTTP/HTTPS URL).
detail	ImageDetail | None	None	Detail level: low (512x512), high (2x2 tiles), or auto (model-selected).

---

DocumentContent

PDF/document content part for vision-capable models.

Field	Type	Default	Description
data	str	—	Base64-encoded document data or URL.
media_type	str	—	MIME type (e.g., “application/pdf”, “text/csv”).

---

AudioContent

Audio content part for speech-capable models.

Field	Type	Default	Description
data	str	—	Base64-encoded audio data.
format	str	—	Audio format (e.g., “wav”, “mp3”, “ogg”).

---

AssistantMessage

Assistant’s response to a user message.

Field	Type	Default	Description
content	AssistantContent | None	None	The assistant’s response: plain text, structured parts, or absent. None is valid when the model replies with tool calls only.
name	str | None	None	Optional name for the assistant.
tool_calls	list\[ToolCall\] | None	\[\]	Tool calls the model wants to execute, if any.
refusal	str | None	None	Refusal reason, if the model declined to respond per safety policies.
function_call	FunctionCall | None	None	Deprecated legacy function_call field; retained for API compatibility.
reasoning_content	str | None	None	Reasoning/thinking tokens returned by the provider, if any (e.g. DeepSeek R1, Qwen reasoning_content, or Anthropic extended thinking).

---

ToolMessage

Tool execution result returned to the model.

Field	Type	Default	Description
content	UserContent	UserContent.TEXT	Result of the tool execution as plain text or an array of content parts (text, images, documents, audio), mirroring UserMessage.content. #\[serde(untagged)\] on UserContent means a bare JSON string still deserialises into Text, so tool results persisted before this field carried structured content continue to round-trip.
tool_call_id	str	—	ID of the tool call this result responds to.
name	str | None	None	Optional tool/function name.

---

DeveloperMessage

Developer message (system-like message for Claude models).

Field	Type	Default	Description
content	str	—	Developer-specific instructions or context.
name	str | None	None	Optional name for the developer message source.

---

FunctionMessage

Deprecated legacy function-role message body.

Field	Type	Default	Description
content	str	—	The extracted text content
name	str	—	The name

---

SpecificToolChoice

Directive to call a specific tool.

Field	Type	Default	Description
choice_type	ToolType	ToolType.FUNCTION	Tool type (always “function”).
function	SpecificFunction	—	The specific function to invoke.

---

SpecificFunction

Name of the specific function to invoke.

Field	Type	Default	Description
name	str	—	Function name.

---

JsonSchemaFormat

JSON Schema specification for constrained output.

Field	Type	Default	Description
name	str	—	Name of the schema (must be unique in the request).
description	str | None	None	Description of what the schema represents.
schema	dict\[str, Any\]	—	JSON Schema object defining the output structure.
strict	bool | None	None	If true, enforce strict schema validation.

---

Usage

Token-usage accounting returned by the provider on each completion / embedding call.

Field	Type	Default	Description
prompt_tokens	int	—	Prompt tokens used. Defaults to 0 when absent (some providers omit this).
completion_tokens	int	—	Completion tokens used. Defaults to 0 when absent (e.g. embedding responses).
total_tokens	int	—	Total tokens used. Defaults to 0 when absent (some providers omit this).
prompt_tokens_details	PromptTokensDetails | None	None	Breakdown of tokens used in the prompt, including cached tokens served at the provider’s discounted cache-read rate. Absent when the provider does not return prompt-token details.

---

PromptTokensDetails

Breakdown of tokens used in the prompt portion of a request.

cached_tokens is included in Usage.prompt_tokens — it is not an additional charge on top of the prompt token count. When pricing supports a cache_read_input_token_cost, the cached portion is billed at the discounted rate and the remainder at the regular input rate.

Field	Type	Default	Description
cached_tokens	int	—	Cached tokens present in the prompt. Defaults to 0 when absent.
audio_tokens	int	—	Audio input tokens present in the prompt. Defaults to 0 when absent.

---

ChatCompletionRequest

Chat completion request (compatible with OpenAI and similar APIs).

Field	Type	Default	Description
model	str	—	Model ID (e.g., "gpt-4o-mini", "claude-3-5-sonnet").
messages	list\[Message\]	\[\]	Conversation history from oldest to newest.
temperature	float | None	None	Sampling temperature in \[0.0, 2.0\]. Higher increases randomness. Defaults to 1.0.
top_p	float | None	None	Nucleus sampling parameter in \[0.0, 1.0\]. Lower is more focused.
n	int | None	None	Number of chat completions to generate. Defaults to 1.
stream	bool | None	None	Whether to stream the response. Managed by the client layer — do not set directly.
stop	StopSequence | None	None	Stop sequence(s) that halt token generation.
max_tokens	int | None	None	Max output tokens. Different from max_completion_tokens in some providers.
presence_penalty	float | None	None	Presence penalty in \[-2.0, 2.0\]. Positive discourages repeated topics.
frequency_penalty	float | None	None	Frequency penalty in \[-2.0, 2.0\]. Positive discourages repeated tokens.
logit_bias	dict\[str, float\] | None	{}	Token bias map. Uses BTreeMap (sorted keys) for deterministic serialization order — important when hashing or signing requests.
user	str | None	None	User identifier for request tracking and abuse detection.
tools	list\[ChatCompletionTool\] | None	\[\]	Tools the model can invoke.
tool_choice	ToolChoice | None	None	Tool usage mode (auto, required, none, or specific tool).
parallel_tool_calls	bool | None	None	Whether the model can call multiple tools in parallel. Defaults to true.
response_format	ResponseFormat | None	None	Output format constraint (text, JSON, JSON schema).
stream_options	StreamOptions | None	None	Streaming options (e.g., include_usage).
seed	int | None	None	Random seed for reproducible outputs. Provider support varies.
reasoning_effort	ReasoningEffort | None	None	Reasoning effort level (minimal, low, medium, high, max) for extended-thinking models.
modalities	list\[Modality\] | None	\[\]	Output modalities to request from the model. For OpenAI audio models, pass \["text", "audio"\]. Vertex AI / Gemini translates these to generationConfig.responseModalities (uppercase).
extra_body	dict\[str, Any\] | None	None	Provider-specific extra parameters merged into the request body. Use for guardrails, safety settings, grounding config, etc.

---

StreamOptions

Options for streaming responses.

Field	Type	Default	Description
include_usage	bool | None	None	If true, include token usage in the final stream chunk.

---

ChatCompletionResponse

Chat completion response from the API.

Field	Type	Default	Description
id	str	—	Unique identifier for this response.
object	str	—	Always "chat.completion" from OpenAI-compatible APIs. Stored as a plain String so non-standard provider values do not break deserialization.
created	int	—	Unix timestamp of response creation.
model	str	—	Model used to generate the response.
choices	list\[Choice\]	\[\]	List of completion choices.
usage	Usage | None	None	Token usage statistics.
system_fingerprint	str | None	None	Fingerprint of the system configuration (OpenAI-specific).
service_tier	str | None	None	Service tier used (OpenAI-specific).

---

Choice

A single completion choice.

Field	Type	Default	Description
index	int	—	Index of this choice in the choices array.
message	AssistantMessage	—	The assistant’s message response.
finish_reason	FinishReason | None	None	Why the model stopped generating (stop, length, tool_calls, content_filter, etc.).

---

ChatCompletionChunk

A streamed chunk of a chat completion response.

Field	Type	Default	Description
id	str	—	Unique identifier for this stream.
object	str	—	Always "chat.completion.chunk" from OpenAI-compatible APIs. Stored as a plain String so non-standard provider values do not fail parsing.
created	int	—	Unix timestamp of chunk creation.
model	str	—	Model used to generate the chunk.
choices	list\[StreamChoice\]	\[\]	Streaming choices (delta updates).
usage	Usage | None	None	Token usage (typically only in the final chunk).
system_fingerprint	str | None	None	Fingerprint of the system configuration (OpenAI-specific).
service_tier	str | None	None	Service tier used (OpenAI-specific).

---

StreamChoice

A streaming choice with incremental delta.

Field	Type	Default	Description
index	int	—	Index of this choice in the choices array.
delta	StreamDelta	—	Incremental update to the message (content, tool calls, etc.).
finish_reason	FinishReason | None	None	Why the stream ended (present only in final chunk).

---

StreamDelta

Incremental delta in a stream chunk.

Field	Type	Default	Description
role	str | None	None	Role (typically present only in the first chunk).
content	str | None	None	Partial content chunk (e.g., a few words of the response).
tool_calls	list\[StreamToolCall\] | None	\[\]	Partial tool calls being streamed.
function_call	StreamFunctionCall | None	None	Deprecated legacy function_call delta; retained for API compatibility.
refusal	str | None	None	Partial refusal message.
reasoning_content	str | None	None	Partial reasoning/thinking tokens (OpenAI-compatible extension used by DeepSeek R1, Qwen, etc.).

---

StreamToolCall

A streaming tool call being built incrementally.

Field	Type	Default	Description
index	int	—	Index of this tool call in the tool_calls array.
id	str | None	None	Tool call ID (typically in the first chunk for this call).
call_type	ToolType | None	None	Tool type (typically “function”).
function	StreamFunctionCall | None	None	Partial function name and arguments.

---

StreamFunctionCall

Partial function call details in a stream.

Field	Type	Default	Description
name	str | None	None	Function name (typically in the first chunk).
arguments	str | None	None	Partial JSON arguments chunk.

---

EmbeddingRequest

Embedding request.

Field	Type	Default	Description
model	str	—	Model ID (e.g., "text-embedding-3-small").
input	EmbeddingInput	EmbeddingInput.SINGLE	Text or texts to embed.
encoding_format	EmbeddingFormat | None	None	Output format: float (native) or base64.
dimensions	int | None	None	Requested embedding dimensions (if supported by the model).
user	str | None	None	User identifier for request tracking.

---

CreateImageRequest

Request to create images from a text prompt.

Field	Type	Default	Description
prompt	str	—	Text description of the image to generate.
model	str | None	None	Model ID (e.g., "dall-e-3"). Optional; API may use default if unset.
n	int | None	None	Number of images to generate. Defaults to 1.
size	str | None	None	Image size (e.g., "1024x1024", "1792x1024").
quality	str | None	None	Image quality: "standard" or "hd".
style	str | None	None	Style: "natural" or "vivid" (DALL-E 3 only).
response_format	str | None	None	Response format: "url" or "b64_json".
user	str | None	None	User identifier for request tracking.

---

ImagesResponse

Response containing generated images.

Field	Type	Default	Description
created	int	—	Unix timestamp of image creation.
data	list\[Image\]	\[\]	List of generated images.

---

Image

A single generated image, returned as either a URL or base64 data.

Field	Type	Default	Description
url	str | None	None	Image URL (if response_format was “url”).
b64_json	str | None	None	Base64-encoded image data (if response_format was “b64_json”).
revised_prompt	str | None	None	The final prompt used to generate the image (DALL-E 3).

---

DecodedDataUrl

Result of decoding a data: URL — MIME type and the decoded byte payload.

Named struct (rather than a tuple) so polyglot bindings can extract decode_data_url with a typed return rather than a sanitized scalar.

Field	Type	Default	Description
mime	str	—	MIME type extracted from the URL prefix (verbatim, not normalised).
data	bytes	—	Decoded base64 payload.

---

CreateSpeechRequest

Request to generate speech audio from text.

Field	Type	Default	Description
model	str	—	Model ID (e.g., "tts-1", "tts-1-hd").
input	str	—	Text to synthesize into speech.
voice	str	—	Voice name (e.g., "alloy", "echo", "fable", "onyx", "nova", "shimmer").
response_format	str | None	None	Audio format (e.g., "mp3", "opus", "aac", "flac", "wav", "pcm").
speed	float | None	None	Playback speed in \[0.25, 4.0\]. Defaults to 1.0.

---

CreateTranscriptionRequest

Request to transcribe audio into text.

Field	Type	Default	Description
model	str	—	Model ID (e.g., "whisper-1").
file	str	—	Base64-encoded audio file data.
language	str | None	None	Language ISO-639-1 code (e.g., "en", "fr", "de"). Optional; model auto-detects.
prompt	str | None	None	Optional text to guide the model (improves accuracy for domain-specific terms).
response_format	str | None	None	Output format (e.g., "json", "text", "vtt", "srt", "verbose_json").
temperature	float | None	None	Sampling temperature in \[0.0, 1.0\]. Higher increases variability. Defaults to 0.

---

TranscriptionResponse

Response from a transcription request.

Field	Type	Default	Description
text	str	—	The transcribed text.
language	str | None	None	Detected language (ISO-639-1 code).
duration	float | None	None	Total audio duration in seconds.
segments	list\[TranscriptionSegment\] | None	\[\]	Detailed segment-level transcription (if response_format is “verbose_json”).

---

TranscriptionSegment

A segment of transcribed audio with timing information.

Field	Type	Default	Description
id	int	—	Segment index (0-based).
start	float	—	Start time in seconds.
end	float	—	End time in seconds.
text	str	—	Transcribed text for this segment.

---

ModerationRequest

Request to classify content for policy violations.

Field	Type	Default	Description
input	ModerationInput	ModerationInput.SINGLE	Text or texts to check.
model	str | None	None	Model ID (e.g., "text-moderation-latest"). Optional; API uses default if unset.

---

ModerationCategories

Boolean flags for each moderation category.

Field	Type	Default	Description
sexual	bool	—	Sexual content.
hate	bool	—	Hate speech.
harassment	bool	—	Harassment.
self_harm	bool	—	Self-harm content.
sexual_minors	bool	—	Sexual content involving minors.
hate_threatening	bool	—	Hate speech that threatens violence.
violence_graphic	bool	—	Graphic violence.
self_harm_intent	bool	—	Intent to self-harm.
self_harm_instructions	bool	—	Instructions for self-harm.
harassment_threatening	bool	—	Harassment that threatens violence.
violence	bool	—	Non-graphic violence.

---

ModerationCategoryScores

Confidence scores for each moderation category.

Field	Type	Default	Description
sexual	float	—	Sexual content score.
hate	float	—	Hate speech score.
harassment	float	—	Harassment score.
self_harm	float	—	Self-harm content score.
sexual_minors	float	—	Sexual content involving minors score.
hate_threatening	float	—	Hate speech that threatens violence score.
violence_graphic	float	—	Graphic violence score.
self_harm_intent	float	—	Intent to self-harm score.
self_harm_instructions	float	—	Instructions for self-harm score.
harassment_threatening	float	—	Harassment that threatens violence score.
violence	float	—	Non-graphic violence score.

---

RerankRequest

Request to rerank documents by relevance to a query.

Field	Type	Default	Description
model	str	—	Model ID (e.g., "cohere/rerank-english-v3.0").
query	str	—	The search query.
documents	list\[RerankDocument\]	\[\]	Documents to rerank.
top_n	int | None	None	Return only the top N results. Optional.
return_documents	bool | None	None	Include the document content in results. Defaults to false.

---

SearchRequest

A search request.

Field	Type	Default	Description
model	str	—	The model/provider to use (e.g. "brave/web-search", "tavily/search").
query	str	—	The search query string.
max_results	int | None	None	Maximum number of results to return.
search_domain_filter	list\[str\] | None	\[\]	Domain filter — restrict results to specific domains.
country	str | None	None	Country code for localized results (ISO 3166-1 alpha-2, e.g., "US", "FR").

---

OcrRequest

An OCR request.

Field	Type	Default	Description
model	str	—	The model/provider to use (e.g. "mistral/mistral-ocr-latest").
document	OcrDocument	OcrDocument.URL	The document to process (URL or base64).
pages	list\[int\] | None	\[\]	Specific pages to process (1-indexed). None means all pages.
include_image_base64	bool | None	None	Whether to include base64-encoded images of each processed page.

---

ModelsListResponse

Response listing available models.

Field	Type	Default	Description
object	str	—	Always "list" from OpenAI-compatible APIs. Stored as a plain String so non-standard provider values do not break deserialization. Defaults to empty when a provider omits the field.
data	list\[ModelObject\]	\[\]	List of available models.

---

ModelObject

A model available from the API.

Field	Type	Default	Description
id	str	—	Model ID (e.g., "gpt-4o", "claude-3-5-sonnet").
object	str	—	Always "model" from OpenAI-compatible APIs. Stored as a plain String so non-standard provider values do not break deserialization. Defaults to empty when a provider omits the field.
created	int	—	Unix timestamp of model creation (or release date). Defaults to 0 when a provider omits it — DeepSeek and some other OpenAI-compatible providers do not return created from /v1/models.
owned_by	str	—	Organization or entity that owns the model. Defaults to empty when a provider omits the field.

---

CreateFileRequest

Request to upload a file.

Field	Type	Default	Description
file	str	—	Base64-encoded file data.
purpose	FilePurpose	FilePurpose.ASSISTANTS	Purpose for the file.
filename	str | None	None	Optional filename to associate with the upload.

---

FileObject

An uploaded file object.

Field	Type	Default	Description
id	str	—	Unique file ID.
object	str	—	Object type (always "file").
bytes	int	—	File size in bytes.
created_at	int	—	Unix timestamp of file creation.
filename	str	—	Filename.
purpose	str	—	File purpose.
status	str | None	None	Processing status (e.g., "uploaded", "processed").

---

FileListResponse

Response from listing files.

Field	Type	Default	Description
object	str	—	Object type (always "list").
data	list\[FileObject\]	\[\]	List of file objects.
has_more	bool | None	None	Whether more results are available.

---

FileListQuery

Query parameters for listing files.

Field	Type	Default	Description
purpose	str | None	None	Filter by file purpose (e.g., "batch", "fine-tune").
limit	int | None	None	Maximum number of results to return. Defaults to 20.
after	str | None	None	Pagination cursor: return results after this file ID.

---

DeleteResponse

Response from a delete operation.

Field	Type	Default	Description
id	str	—	ID of the deleted resource.
object	str	—	Object type.
deleted	bool	—	Confirmation that the resource was deleted.

---

CreateBatchRequest

Request to create a batch job.

Field	Type	Default	Description
input_file_id	str	—	ID of the uploaded input file (JSONL format).
endpoint	str	—	API endpoint (e.g., "/v1/chat/completions").
completion_window	str	—	Completion window (e.g., "24h").
metadata	dict\[str, Any\] | None	None	Optional metadata to attach to the batch.

---

BatchObject

A batch job object.

Field	Type	Default	Description
id	str	—	Unique batch ID.
object	str	—	Object type (always "batch").
endpoint	str	—	API endpoint (e.g., "/v1/chat/completions").
input_file_id	str	—	ID of the input file.
completion_window	str	—	Completion window (e.g., "24h").
status	BatchStatus	BatchStatus.VALIDATING	Current job status.
output_file_id	str | None	None	ID of the output file (present when completed).
error_file_id	str | None	None	ID of the error file (present if some requests failed).
created_at	int	—	Unix timestamp of batch creation.
completed_at	int | None	None	Unix timestamp of completion (if completed).
failed_at	int | None	None	Unix timestamp of failure (if failed).
expired_at	int | None	None	Unix timestamp of expiration (if expired).
request_counts	BatchRequestCounts | None	None	Request processing counts.
metadata	dict\[str, Any\] | None	None	Metadata attached to the batch.

---

BatchRequestCounts

Request processing counts for a batch.

Field	Type	Default	Description
total	int	—	Total requests in the batch.
completed	int	—	Completed requests.
failed	int	—	Failed requests.

---

BatchListResponse

Response from listing batches.

Field	Type	Default	Description
object	str	—	Object type (always "list").
data	list\[BatchObject\]	\[\]	List of batch objects.
has_more	bool | None	None	Whether more results are available.
first_id	str | None	None	First batch ID in the result set (for pagination).
last_id	str | None	None	Last batch ID in the result set (for pagination).

---

BatchListQuery

Query parameters for listing batches.

Field	Type	Default	Description
limit	int | None	None	Maximum number of results to return. Defaults to 20.
after	str | None	None	Pagination cursor: return results after this batch ID.

---

CreateResponseRequest

Request to create a structured response.

Field	Type	Default	Description
model	str	—	Model ID.
input	dict\[str, Any\]	—	Input data to process (e.g., a document to extract from).
instructions	str | None	None	Instructions for processing the input.
tools	list\[ResponseTool\] | None	\[\]	Available tools the model can use.
temperature	float | None	None	Sampling temperature in \[0.0, 2.0\]. Defaults to 1.0.
max_output_tokens	int | None	None	Maximum output tokens.
metadata	dict\[str, Any\] | None	None	Optional metadata.
stream	bool | None	None	Whether to stream the response. Managed by the client layer — do not set directly.

---

ResponseTool

A tool available for the response request.

Field	Type	Default	Description
tool_type	str	—	Tool type (e.g., “extractor”, “search”).
config	dict\[str, Any\]	—	Tool configuration (flattened into the object).

---

ResponseObject

Response from a structured response request.

Field	Type	Default	Description
id	str	—	Unique response ID.
object	str	—	Object type (e.g., "response").
created_at	int	—	Unix timestamp of response creation.
model	str	—	Model used to generate the response.
status	str	—	Status (e.g., "succeeded", "failed").
output	list\[ResponseOutputItem\]	\[\]	Output items from the response.
usage	ResponseUsage | None	None	Token usage.
error	dict\[str, Any\] | None	None	Error details (if status is “failed”).

---

ResponseOutputItem

A single output item from the response.

Field	Type	Default	Description
item_type	str	—	Output type (e.g., "text", "object", "error").
content	dict\[str, Any\]	—	Output content (flattened into the object).

---

ResponseUsage

Token usage for a response.

Field	Type	Default	Description
input_tokens	int	—	Input tokens used.
output_tokens	int	—	Output tokens used.
total_tokens	int	—	Total tokens used.

---

LlmConfig

Canonical configuration for an LLM client.

All fields except model are optional so that partially-specified configs (e.g. from environment-driven defaults) round-trip cleanly. Convert to a runtime client configuration via LlmConfig.into_client_builder.

temperature and max_tokens are request-time parameters rather than client-level settings; they are carried on this struct for callers to read when building individual requests, and are intentionally not mapped by LlmConfig.into_client_builder.

Field	Type	Default	Description
model	str	—	Model identifier (e.g. "gpt-4o", "bedrock/anthropic.claude-3-sonnet-20240229-v1:0").
api_key	str | None	None	API key for authentication.
base_url	str | None	None	Override base URL. When set, all requests go here and provider auto-detection is skipped.
timeout_secs	int | None	None	Request timeout, in seconds.
max_retries	int | None	None	Maximum number of retries on 429 / 5xx responses.
temperature	float | None	None	Sampling temperature for requests built from this config.
max_tokens	int | None	None	Maximum number of tokens to generate for requests built from this config.
load_env	bool | None	None	Automatically load the API key from the provider’s environment variable when no explicit key is provided (default: True).
headers	dict\[str, str\] | None	{}	Extra headers sent on every request.
providers	list\[LlmProviderConfig\] | None	\[\]	Custom provider configurations, in addition to the built-in providers.
cache	LlmCacheConfig | None	None	Response cache configuration.
budget	LlmBudgetConfig | None	None	Budget enforcement configuration.
rate_limit	LlmRateLimitConfig | None	None	Per-model rate limiting configuration.
cost_tracking	bool | None	None	Enable per-request cost tracking.
tracing	bool | None	None	Enable OpenTelemetry-compatible tracing spans.
cooldown_secs	int | None	None	Cooldown duration after transient errors, in seconds.
health_check_secs	int | None	None	Background health check interval, in seconds.
bedrock	BedrockConfig | None	None	AWS Bedrock configuration (region, credentials, cross-region routing).

---

LlmCacheConfig

Response cache configuration.

Field	Type	Default	Description
max_entries	int | None	None	Maximum number of cached entries.
ttl_seconds	int | None	None	Cache entry time-to-live, in seconds.
backend	str | None	None	Cache backend name (e.g. "memory", or an opendal scheme).
backend_config	dict\[str, str\] | None	{}	Backend-specific configuration key/value pairs.

---

LlmBudgetConfig

Budget enforcement configuration.

Field	Type	Default	Description
global_limit	float | None	None	Global spend limit in USD.
model_limits	dict\[str, float\] | None	{}	Per-model spend limits in USD, keyed by model name.
enforcement	str | None	None	Enforcement mode: "hard" (reject over-budget requests) or "soft" (log only).

---

LlmRateLimitConfig

Per-model rate limiting configuration.

Field	Type	Default	Description
rpm	int | None	None	Requests per minute limit.
tpm	int | None	None	Tokens per minute limit.
window_seconds	int | None	None	Rate limit window, in seconds.

---

LlmProviderConfig

A custom provider configuration entry.

Field	Type	Default	Description
name	str	—	Provider name, used to key model prefix matching.
base_url	str	—	Base URL for the provider’s OpenAI-compatible API.
auth_header	str | None	None	Header name used to carry the API key (defaults to Authorization when unset).
model_prefixes	list\[str\]	\[\]	Model name prefixes routed to this provider (e.g. \["my-provider/"\]).

---

BedrockConfig

AWS Bedrock configuration.

All fields are optional; anything left unset falls back to the standard AWS environment variables (AWS_DEFAULT_REGION / AWS_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, BEDROCK_CROSS_REGION).

Field	Type	Default	Description
region	str | None	None	AWS region (e.g. "us-east-1").
cross_region_prefix	str | None	None	Cross-region inference profile prefix (e.g. "us").
access_key_id	str | None	None	Explicit AWS access key ID.
secret_access_key	str | None	None	Explicit AWS secret access key.
session_token	str | None	None	Explicit AWS session token (temporary credentials).

---

WaitForBatchConfig

Configuration for polling a batch until terminal status.

All time values are in seconds as f64 so the struct bridges across FFI boundaries without requiring a Duration shim.

Field	Type	Default	Description
initial_interval_secs	float	5	Initial interval between polls, in seconds.
max_interval_secs	float	60	Maximum interval between polls (backoff plateau), in seconds.
backoff_multiplier	float	1.5	Exponential backoff multiplier (e.g., 1.5 increases delay by 50% each poll).
timeout_secs	float | None	None	Optional timeout in seconds — polling fails if this duration is exceeded.

---

CustomProviderConfig

Configuration for registering a custom LLM provider at runtime.

Field	Type	Default	Description
name	str	—	Unique name for this provider (e.g., “my-provider”).
base_url	str	—	Base URL for the provider’s API (e.g., <https://api.my-provider.com/v1>).
auth_header	AuthHeaderFormat	—	Authentication header format.
model_prefixes	list\[str\]	—	Model name prefixes that route to this provider (e.g., \["my-"\]).

---

ProviderCapabilities

Static capability flags for a provider.

Each flag indicates whether the provider’s models generally support that feature. For providers that aggregate many underlying models (e.g. Bedrock, OpenRouter, vLLM) the flags reflect the superset of available model capabilities — a flag being True means at least one model supports the feature, not every model.

All flags default to False so that newly added providers are safe.

Access via the crate-level capabilities function:

Field	Type	Default	Description
vision	bool	—	The provider accepts image input in chat messages.
reasoning	bool	—	The provider supports extended-thinking / reasoning tokens.
structured_output	bool	—	The provider supports JSON-mode or response_format structured output.
function_calling	bool	—	The provider supports tool / function calling.
audio_in	bool	—	The provider accepts audio as input.
audio_out	bool	—	The provider can generate audio / TTS output.
video_in	bool	—	The provider accepts video as input.

---

ProviderConfig

Static configuration for a single provider entry in providers.json.

This struct deliberately does not include capability flags or streaming format, which are accessed via the capabilities function.

Field	Type	Default	Description
name	str	—	Provider identifier (matches the entry key in providers.json).
display_name	str | None	None	Human-readable provider name shown in UIs.
base_url	str | None	None	Base URL used as the default for this provider’s HTTP client.
auth	AuthConfig | None	None	Authentication scheme metadata (auth type + env var holding the key).
endpoints	list\[str\] | None	None	Supported endpoint kinds (e.g. chat, embeddings).
model_prefixes	list\[str\] | None	None	Model-name prefixes claimed by this provider (e.g. \["gpt-", "o1-"\]).
param_mappings	dict\[str, str\] | None	None	Parameter key renaming for this provider. Each entry maps an OpenAI-spec field name (e.g. "max_completion_tokens") to the name this provider expects (e.g. "max_tokens"). Applied automatically by ConfigDrivenProvider.transform_request.

---

AuthConfig

Auth configuration block.

Field	Type	Default	Description
auth_type	AuthType	—	Auth scheme classification.
env_var	str | None	None	Name of the environment variable that holds the API key (e.g. "OPENAI_API_KEY"). Holds the variable name, never the secret value.

---

ModelInfo

Public, FFI-friendly snapshot of a model’s pricing and capability metadata, projected from ModelPricing.

Unlike ModelPricing (which is excluded from binding generation), ModelInfo is an owned plain-data DTO safe to hand across the FFI boundary — see model_info.

Field	Type	Default	Description
input_cost_per_token	float	—	Cost in USD per input (prompt) token.
output_cost_per_token	float	—	Cost in USD per output (completion) token.
cache_read_input_token_cost	float | None	None	Cost in USD per cached input token (cache hit / read).
cache_creation_input_token_cost	float | None	None	Cost in USD per token written to the prompt cache.
input_cost_per_audio_token	float | None	None	Cost in USD per input audio token.
output_cost_per_audio_token	float | None	None	Cost in USD per output audio token.
output_cost_per_reasoning_token	float | None	None	Cost in USD per reasoning (extended-thinking) output token.
max_tokens	int | None	None	Total context window size in tokens (input + output).
max_input_tokens	int | None	None	Maximum input (prompt) tokens accepted.
max_output_tokens	int | None	None	Maximum output (completion) tokens the model can generate.
mode	str | None	None	Best-effort operating mode, e.g. "chat", "embedding".
supports_vision	bool | None	None	The model accepts image input.
supports_function_calling	bool | None	None	The model supports tool / function calling.
supports_reasoning	bool | None	None	The model supports extended-thinking / reasoning tokens.
supports_structured_output	bool | None	None	The model supports JSON-mode or response_format structured output.
supports_audio_input	bool | None	None	The model accepts audio input.
supports_audio_output	bool | None	None	The model can generate audio output.
supports_prompt_caching	bool | None	None	The model supports prompt caching.
tiers	list\[ModelTier\]	\[\]	Context-tiered pricing overrides, sorted by ascending min_context_tokens. Empty when the model has flat pricing.

---

ModelTier

Public, FFI-friendly snapshot of a single context-window pricing tier, projected from PricingTier.

Field	Type	Default	Description
min_context_tokens	int	—	The tier applies when the prompt/context token count is at least this value.
input_cost_per_token	float	—	Cost in USD per input (prompt) token within this tier.
output_cost_per_token	float	—	Cost in USD per output (completion) token within this tier.
cache_read_input_token_cost	float | None	None	Cost in USD per cached input token within this tier.
cache_creation_input_token_cost	float | None	None	Cost in USD per cache-write token within this tier.
input_cost_per_audio_token	float | None	None	Cost in USD per input audio token within this tier.
output_cost_per_audio_token	float | None	None	Cost in USD per output audio token within this tier.
output_cost_per_reasoning_token	float | None	None	Cost in USD per reasoning output token within this tier.

---

BudgetConfig

Configuration for budget enforcement.

Field	Type	Default	Description
global_limit	float | None	None	Maximum total spend across all models, in USD. None means unlimited.
model_limits	dict\[str, float\]	{}	Per-model spending limits in USD. Models not listed here are only constrained by global_limit.
enforcement	Enforcement	Enforcement.HARD	Whether to reject requests or merely warn when a limit is exceeded.

---

CacheConfig

Configuration for the response cache.

Field	Type	Default	Description
max_entries	int	256	Maximum number of cached entries.
ttl	float	300000ms	Time-to-live for each cached entry.
backend	CacheBackend	CacheBackend.MEMORY	Storage backend to use.

---

RateLimitConfig

Configuration for per-model rate limits.

Field	Type	Default	Description
rpm	int | None	None	Maximum requests per window. None means unlimited.
tpm	int | None	None	Maximum tokens per window. None means unlimited.
window	float	60000ms	Fixed window duration (defaults to 60 s).

---

CatalogRefreshConfig

Plain-data configuration for refresh_catalog.

Deliberately FFI/binding-friendly: no Duration or PathBuf, just primitives that translate directly across language boundaries.

Field	Type	Default	Description
enabled	bool	False	Runtime catalog refresh is entirely opt-in: when False, refresh_catalog is a no-op that returns Ok(RefreshOutcome.Disabled) without touching the network, the filesystem, or the overlay registry.
source_url	str	—	Source URL to fetch catalog.json from. Must be https. Defaults to DEFAULT_CATALOG_URL; configurable so self-hosted mirrors work.
ttl_seconds	int	86400	How long a cached catalog.json remains valid before a network refetch is attempted, in seconds.
cache_path	str | None	None	Filesystem path for the on-disk cache. None uses a default path under std.env.temp_dir().

---

Enums

AssistantContent

Content shape for assistant messages.

#[serde(untagged)] means providers returning a plain scalar string for the content field still deserialise correctly into AssistantContent.Text(_). Providers returning an array of typed parts (e.g. after an image-generation or audio-synthesis request) deserialise into AssistantContent.Parts(_).

Variant	Description
Text	Plain text response (the common case for text-only models). — Fields: _0: String
Parts	Structured parts — text, refusals, output images, output audio. — Fields: _0: Vec<AssistantPart>

---

AuthHeaderFormat

How the API key is sent in the HTTP request.

Variant	Description
Bearer	Bearer token: Authorization: Bearer <key>
ApiKey	Custom header: e.g., X-Api-Key: <key> — Fields: _0: String
None	No authentication required.

---

AuthType

Auth scheme used by a provider.

Variant	Wire value	Description
Bearer	bearer	Standard Authorization: Bearer <key> header.
ApiKey	api-key	x-api-key: <key> header (also handles "header" and "x-api-key" aliases).
None	none	No authentication header required.
Unknown	unknown	Unrecognised auth scheme — falls back to bearer.

---

BatchStatus

Status of a batch job.

Variant	Wire value	Description
Validating	validating	Validating the input file.
Failed	failed	Job failed.
InProgress	in_progress	Job is running.
Finalizing	finalizing	Finalizing results.
Completed	completed	Job completed successfully.
Expired	expired	Job expired before completion.
Cancelling	cancelling	Job is being cancelled.
Cancelled	cancelled	Job has been cancelled.

---

CacheBackend

Storage backend for the response cache.

Variant	Wire value	Description
Memory	memory	In-memory LRU cache (default). No external dependencies.
OpenDal	open_dal	OpenDAL-backed storage. Supports 40+ backends (S3, Redis, GCS, local FS, etc.). — Fields: scheme: String, config: HashMap<String, String>

---

EmbeddingFormat

The format in which the embedding vectors are returned.

Variant	Wire value	Description
Float	float	32-bit floating-point numbers (default).
Base64	base64	Base64-encoded string representation of the floats.

---

EmbeddingInput

Text or texts to embed.

Variant	Description
Single	Single text string. — Fields: _0: String
Multiple	Multiple text strings (batch embedding). — Fields: _0: Vec<String>

---

Enforcement

How budget limits are enforced.

Variant	Description
Hard	Reject requests that would exceed the budget with LiterLlmError.BudgetExceeded.
Soft	Allow requests through but emit a tracing.warn! when the budget is exceeded.

---

FilePurpose

Purpose of an uploaded file.

Variant	Wire value	Description
Assistants	assistants	File for use with Assistants API.
Batch	batch	File for batch processing.
FineTune	fine-tune	File for fine-tuning.
Vision	vision	File for vision/image tasks.

---

FinishReason

Why a choice stopped generating tokens.

Variant	Wire value	Description
Stop	stop	Stop
Length	length	Length
ToolCalls	tool_calls	Tool calls
ContentFilter	content_filter	Content filter
FunctionCall	function_call	Deprecated legacy finish reason; retained for API compatibility.
Other	other	Catch-all for unknown finish reasons returned by non-OpenAI providers. Note: this intentionally does not carry the original string (e.g. Other(String)). Using #\[serde(other)\] requires a unit variant, and switching to #\[serde(untagged)\] would change deserialization semantics for all variants. The original value can be recovered by inspecting the raw JSON if needed.

---

ImageDetail

Image detail level controlling token cost and processing.

Variant	Wire value	Description
Low	low	Low detail: scales image to 512x512, uses fewer tokens.
High	high	High detail: processes up to 2x2 grid of tiles, higher token cost.
Auto	auto	Auto: model chooses low or high based on image dimensions.

---

Message

A chat message in a conversation.

Variant	Wire value	Description
System	system	System — Fields: _0: SystemMessage
User	user	User — Fields: _0: UserMessage
Assistant	assistant	Assistant — Fields: _0: AssistantMessage
Tool	tool	Tool — Fields: _0: ToolMessage
Developer	developer	Developer — Fields: _0: DeveloperMessage
Function	function	Deprecated legacy function-role message; retained for API compatibility. — Fields: _0: FunctionMessage

---

Modality

Output modality requested from the model.

Passed as modalities: ["text", "audio"] (OpenAI) or translated to generationConfig.responseModalities (Gemini / Vertex AI).

Variant	Wire value	Description
Text	text	Text output (the default for all providers).
Audio	audio	Audio / speech output.
Image	image	Image output (Gemini Imagen, gpt-image-1).

---

ModerationInput

Input to the moderation endpoint — a single string or multiple strings.

Variant	Description
Single	Single text string. — Fields: _0: String
Multiple	Multiple text strings (batch moderation). — Fields: _0: Vec<String>

---

OcrDocument

Document input for OCR — either a URL or inline base64 data.

Variant	Wire value	Description
Url	document_url	A publicly accessible document URL. — Fields: url: String
Base64	base64	Inline base64-encoded document data. — Fields: data: String, media_type: String

---

ReasoningEffort

Controls how much reasoning effort the model should use.

Variant	Wire value	Description
Low	low	Low
Medium	medium	Medium
High	high	High
Minimal	minimal	Minimal
Max	max	Max

---

RerankDocument

A document to be reranked — either a plain string or an object with a text field.

Variant	Description
Text	Plain text document content. — Fields: _0: String
Object	Document with explicit text field (may include metadata). — Fields: text: String

---

ResponseFormat

Wire format for the chat completions response_format field.

Provider mapping

• OpenAI (and OpenAI-compatible providers): emitted verbatim as {"type": "json_schema", "json_schema": {...}} per the chat-completions spec.

• Gemini / Vertex AI: translated to generationConfig.responseMimeType = "application/json" and generationConfig.responseSchema = <schema>. The name, description, and strict fields are dropped — Gemini’s structured-output API does not consume them.

• Anthropic: no native JSON mode. A system instruction is prepended asking the model to respond with valid JSON. strict is advisory only; callers should still validate the returned JSON if the schema is load-bearing.

Variant	Wire value	Description
Text	text	Plain text output (default).
JsonObject	json_object	Output must be valid JSON object (no schema validation).
JsonSchema	json_schema	Output must conform to the specified JSON schema. — Fields: json_schema: JsonSchemaFormat

---

StopSequence

Stop sequence(s) that cause the model to stop generating.

Variant	Description
Single	Single stop sequence. — Fields: _0: String
Multiple	Multiple stop sequences. — Fields: _0: Vec<String>

---

ToolChoice

Tool usage mode or a specific tool to call.

Variant	Description
Mode	Predefined mode: auto, required, or none. — Fields: _0: ToolChoiceMode
Specific	Force a specific tool to be called. — Fields: _0: SpecificToolChoice

---

ToolType

The type discriminator for tool/tool-call objects.

Per the OpenAI spec this is always "function". Using an enum enforces that constraint at the type level and rejects any other value on deserialization.

Variant	Wire value	Description
Function	function	Function

---

UserContent

User message content as either plain text or a list of multimodal parts.

Variant	Description
Text	Plain text content. — Fields: _0: String
Parts	Array of content parts (text, images, documents, audio). — Fields: _0: Vec<ContentPart>

---