Java API Reference

Java API Reference v2.1.2

Functions

createClient()

Create a new LLM client with simple scalar configuration.

This is the primary binding entry-point. All parameters except api_key are optional — omitting them uses the same defaults as ClientConfigBuilder.

Errors:

Returns LiterLlmError if the underlying HTTP client cannot be constructed, or if the resolved provider configuration is invalid.

Signature:

public static DefaultClient createClient(String apiKey, String baseUrl, long timeoutSecs, int maxRetries, String modelHint) throws LiterLlmRsException

Example:

var result = createClient("value", "value", 42, 42, "value");

Parameters:

Name	Type	Required	Description
apiKey	String	Yes	The api key
baseUrl	@Nullable String	No	The base url
timeoutSecs	@Nullable Long	No	The timeout secs
maxRetries	@Nullable Integer	No	The max retries
modelHint	@Nullable String	No	The model hint

Returns: DefaultClient

Errors: Throws LiterLlmRsException.

---

createClientFromJson()

Create a new LLM client from a JSON string.

The JSON object accepts the same fields as liter-llm.toml (snake_case).

Errors:

Returns LiterLlmError.BadRequest if json is not valid JSON or contains unknown fields.

Signature:

public static DefaultClient createClientFromJson(String json) throws LiterLlmRsException

Example:

var result = createClientFromJson("value");

Parameters:

Name	Type	Required	Description
json	String	Yes	The json

Returns: DefaultClient

Errors: Throws LiterLlmRsException.

---

encodeDataUrl()

Encode bytes as a base64 data URL: data:<mime>;base64,<b64>.

mime defaults to IMAGE_PNG when null.

Signature:

public static String encodeDataUrl(byte[] bytes, String mime) throws LiterLlmRsException

Example:

var result = encodeDataUrl("data".getBytes(), "value");

Parameters:

Name	Type	Required	Description
bytes	byte\[\]	Yes	The bytes
mime	@Nullable String	No	The mime

Returns: String

---

decodeDataUrl()

Decode a base64 data URL into DecodedDataUrl.

Returns null for:

• Non-data URLs (strings that do not start with "data:").
• Malformed prefixes (missing ";base64," marker).
• Invalid base64 payloads.

The returned MIME string is extracted verbatim from the URL prefix — it is not validated or normalised.

Signature:

public static @Nullable DecodedDataUrl decodeDataUrl(String url) throws LiterLlmRsException

Example:

var result = decodeDataUrl("value");

Parameters:

Name	Type	Required	Description
url	String	Yes	The URL to fetch

Returns: @Nullable DecodedDataUrl

---

registerCustomProvider()

Register a custom provider in the global runtime registry.

The provider will be checked before all built-in providers during model detection. If a provider with the same name already exists it is replaced.

Errors:

Returns an error if the config is invalid (empty name, empty base_url, or no model prefixes).

Signature:

public static void registerCustomProvider(CustomProviderConfig config) throws LiterLlmRsException

Example:

registerCustomProvider(new CustomProviderConfig());

Parameters:

Name	Type	Required	Description
config	CustomProviderConfig	Yes	The configuration options

Returns: No return value.

Errors: Throws LiterLlmRsException.

---

unregisterCustomProvider()

Remove a previously registered custom provider by name.

Returns true if a provider with the given name was found and removed, false if no such provider existed.

Errors:

Returns an error if the custom-provider registry cannot be updated.

Signature:

public static boolean unregisterCustomProvider(String name) throws LiterLlmRsException

Example:

var result = unregisterCustomProvider("value");

Parameters:

Name	Type	Required	Description
name	String	Yes	The name

Returns: boolean

Errors: Throws LiterLlmRsException.

---

capabilities()

Return the capability flags for a named provider.

Performs an O(n) linear scan over the embedded registry (165 entries). Returns an owned value so bindings can pass capability data without borrowing registry internals.

For unknown provider_name values the function returns an all-false sentinel so callers never need to handle Option.

Signature:

public static ProviderCapabilities capabilities(String providerName) throws LiterLlmRsException

Example:

var result = capabilities("value");

Parameters:

Name	Type	Required	Description
providerName	String	Yes	The provider name

Returns: ProviderCapabilities

---

allProviders()

Return all provider configs from the registry.

Useful for tooling, documentation generation, or runtime enumeration. Returns the public ProviderConfig slice (without capability flags). To query capability flags for a specific provider use capabilities.

Signature:

public static List<ProviderConfig> allProviders() throws LiterLlmRsException

Example:

var result = allProviders();

Returns: List<ProviderConfig>

Errors: Throws LiterLlmRsException.

---

complexProviderNames()

Return the set of complex provider names.

Complex providers require custom auth/routing logic beyond simple bearer tokens (e.g. AWS Bedrock SigV4, Vertex AI OAuth2).

The returned reference points into the static registry — no allocation.

Signature:

public static List<String> complexProviderNames() throws LiterLlmRsException

Example:

var result = complexProviderNames();

Returns: List<String>

Errors: Throws LiterLlmRsException.

---

completionCost()

Calculate the estimated cost of a completion given a model name and token counts.

Returns null if the model is not present in the embedded pricing registry. Returns Some(cost_usd) otherwise, where the value is in US dollars.

When an exact model name match is not found, progressively shorter prefixes are tried by stripping from the last - or . separator. For example, gpt-4-0613 will match gpt-4 if no gpt-4-0613 entry exists.

Signature:

public static @Nullable Double completionCost(String model, long promptTokens, long completionTokens) throws LiterLlmRsException

Example:

var result = completionCost("value", 42, 42);

Parameters:

Name	Type	Required	Description
model	String	Yes	The model
promptTokens	long	Yes	The prompt tokens
completionTokens	long	Yes	The completion tokens

Returns: @Nullable Double

---

completionCostWithCache()

Calculate the estimated cost of a completion, accounting for cached (cache-hit) prompt tokens billed at the provider’s discounted rate.

cached_tokens is the count of prompt tokens served from the provider’s prompt cache. It must be <= prompt_tokens (cached tokens are a subset of the prompt). The non-cached portion is billed at input_cost_per_token and the cached portion at cache_read_input_token_cost when the model has cache pricing; otherwise the entire prompt is billed at the regular input rate.

Returns null if the model is not present in the embedded pricing registry, mirroring completion_cost.

When the model has ModelPricing.tiers, the tier whose min_context_tokens is the highest value <= prompt_tokens supplies the input/output/cache rates for the whole call; models without tiers (or when prompt_tokens is below every tier threshold) use the base rates unchanged, matching the original flat-rate behaviour.

Signature:

public static @Nullable Double completionCostWithCache(String model, long promptTokens, long cachedTokens, long completionTokens) throws LiterLlmRsException

Example:

var result = completionCostWithCache("value", 42, 42, 42);

Parameters:

Name	Type	Required	Description
model	String	Yes	The model
promptTokens	long	Yes	The prompt tokens
cachedTokens	long	Yes	The cached tokens
completionTokens	long	Yes	The completion tokens

Returns: @Nullable Double

---

modelInfo()

Look up FFI-friendly pricing and capability metadata for a model.

Returns null if the model is not present in the active pricing registry. Uses the same exact-match-then-prefix-fallback resolution as model_pricing; unlike model_pricing, the result is an owned ModelInfo value safe to hand across the FFI boundary.

When a runtime catalog refresh has succeeded, this reflects the refreshed (overlay) catalog; otherwise it reflects the embedded catalog. See model_pricing for the embedded-only alternative.

Signature:

public static @Nullable ModelInfo modelInfo(String model) throws LiterLlmRsException

Example:

var result = modelInfo("value");

Parameters:

Name	Type	Required	Description
model	String	Yes	The model

Returns: @Nullable ModelInfo

---

installCatalogOverlayFromStr()

Install the overlay registry from a raw catalog JSON string, bypassing the network and disk cache entirely.

Parses and flattens catalog_json with the same registry_from_catalog_str logic used for the embedded catalog and the network refresh path, then atomically swaps it in as the active overlay. A parse failure returns CatalogRefreshError.Parse and leaves any existing overlay untouched.

This is primarily a testable seam: it lets tests exercise overlay installation and the embedded/overlay fallback behavior in completion_cost / model_info without a real network call.

Signature:

public static void installCatalogOverlayFromStr(String catalogJson) throws LiterLlmRsException

Example:

installCatalogOverlayFromStr("value");

Parameters:

Name	Type	Required	Description
catalogJson	String	Yes	The catalog json

Returns: No return value.

Errors: Throws LiterLlmRsException.

---

clearCatalogOverlay()

Clear the overlay registry, reverting completion_cost, completion_cost_with_cache, and model_info to the embedded catalog.

Primarily a test seam (see install_catalog_overlay_from_str); also usable by long-running processes that want to abandon a runtime refresh.

Signature:

public static void clearCatalogOverlay() throws LiterLlmRsException

Example:

clearCatalogOverlay();

Returns: No return value.

---

refreshCatalog()

Refresh the runtime catalog overlay per config.

• config.enabled == false: returns Ok(RefreshOutcome.Disabled) immediately. No network, filesystem, or overlay activity.

• A fresh on-disk cache (age < config.ttl_seconds) exists at the resolved cache path (config.cache_path, or a default under std.env.temp_dir()): read + flatten it and install the overlay, returning Ok(RefreshOutcome.FromCache). No network request is made.

• Otherwise: validate config.source_url uses https (CatalogRefreshError.InsecureUrl otherwise), fetch it, flatten it, install the overlay, best-effort write the raw JSON to the cache path (a cache write failure does not fail the refresh), and return Ok(RefreshOutcome.Fetched).

On any error return, the overlay is left untouched: the previously active registry (a prior successful overlay, or the embedded catalog if none was ever installed) remains in effect. This is what makes the feature air-gap-safe — an unreachable or invalid source_url never degrades completion_cost / model_info below embedded-catalog availability.

Signature:

public static RefreshOutcome refreshCatalog(CatalogRefreshConfig config) throws LiterLlmRsException

Example:

var result = refreshCatalog(new CatalogRefreshConfig());

Parameters:

Name	Type	Required	Description
config	CatalogRefreshConfig	Yes	The configuration options

Returns: RefreshOutcome

Errors: Throws LiterLlmRsException.

---

clear()

Remove all guardrails from the global registry.

Primarily useful in tests to reset state between test cases.

If the lock was poisoned by a panicking guardrail on a previous access, the poisoned state is recovered rather than propagating the panic.

Signature:

public static void clear() throws LiterLlmRsException

Example:

clear();

Returns: No return value.

---

countTokens()

Count tokens in a text string using the tokenizer for the given model.

The tokenizer is resolved from the model name prefix (e.g. "gpt-4o" maps to the Xenova/gpt-4o HuggingFace tokenizer). Tokenizers are cached after first load.

Errors:

Returns LiterLlmError.BadRequest if the tokenizer cannot be loaded (e.g. network failure on first use) or if tokenization itself fails.

Signature:

public static long countTokens(String model, String text) throws LiterLlmRsException

Example:

var result = countTokens("value", "value");

Parameters:

Name	Type	Required	Description
model	String	Yes	The model
text	String	Yes	The text

Returns: long

Errors: Throws LiterLlmRsException.

---

countRequestTokens()

Count tokens for a full ChatCompletionRequest.

Sums tokens across all message text contents plus a per-message overhead of ~4 tokens (for role, separators, and formatting metadata). Tool definitions and multimodal content parts (images, audio, documents) are not counted — only textual content contributes to the token total.

Errors:

Returns LiterLlmError.BadRequest if the tokenizer cannot be loaded or if tokenization fails for any message.

Signature:

public static long countRequestTokens(String model, ChatCompletionRequest req) throws LiterLlmRsException

Example:

var result = countRequestTokens("value", new ChatCompletionRequest());

Parameters:

Name	Type	Required	Description
model	String	Yes	The model
req	ChatCompletionRequest	Yes	The chat completion request

Returns: long

Errors: Throws LiterLlmRsException.

---

recordCostUsd()

Record the estimated USD cost of a completion.

Call from CostTrackingService once a completion’s cost has been computed. Emits gen_ai.client.cost.usd. If the meter has not been initialized, this call is a no-op.

Signature:

public static void recordCostUsd(String system, String model, String operation, double costUsd) throws LiterLlmRsException

Example:

recordCostUsd("value", "value", "value", 0.5);

Parameters:

Name	Type	Required	Description
system	String	Yes	The system
model	String	Yes	The model
operation	String	Yes	The operation
costUsd	double	Yes	The cost usd

Returns: No return value.

---

checkBound()

Assert that current_len + incoming does not exceed limit.

Call this before appending incoming bytes to any buffer that must stay below limit. Returns Err(LiterLlmError.Streaming) on overflow and emits a tracing.warn! with context.

Signature:

public static void checkBound(String context, long currentLen, long incoming, long limit) throws LiterLlmRsException

Example:

checkBound("value", 42, 42, 42);

Parameters:

Name	Type	Required	Description
context	String	Yes	The context
currentLen	long	Yes	The current len
incoming	long	Yes	The incoming
limit	long	Yes	The limit

Returns: No return value.

Errors: Throws LiterLlmRsException.

---

ensureCryptoProvider()

Install the ring crypto provider as the rustls process default, idempotently.

rustls 0.23+ removed the implicit default provider. This function installs ring once per process. Subsequent calls are no-ops. Calling it after another rustls crypto provider has already been installed is safe: the Err from install_default() is silently ignored.

Called automatically by every internal reqwest.Client constructor (auth providers, default HTTP client). Bindings and downstream consumers reach those constructors transitively, so no manual init is required.

WASM builds are exempt — the WASM target uses the browser/Node.js fetch API instead of rustls, so no crypto provider is needed.

Windows builds use native-tls (SChannel) via reqwest, so rustls is not present and no crypto provider installation is needed.

Signature:

public static void ensureCryptoProvider() throws LiterLlmRsException

Example:

ensureCryptoProvider();

Returns: No return value.

---

Types

AssistantMessage

Assistant’s response to a user message.

Field	Type	Default	Description
content	@Nullable AssistantContent	null	The assistant’s response: plain text, structured parts, or absent. null is valid when the model replies with tool calls only.
name	@Nullable String	null	Optional name for the assistant.
toolCalls	@Nullable List<ToolCall>	Collections.emptyList()	Tool calls the model wants to execute, if any.
refusal	@Nullable String	null	Refusal reason, if the model declined to respond per safety policies. OpenAI’s response schema requires this key to be present even when null, so it is deliberately not skip_serializing_if.
functionCall	@Nullable FunctionCall	null	Deprecated legacy function_call field; retained for API compatibility.
reasoningContent	@Nullable String	null	Reasoning/thinking tokens returned by the provider, if any (e.g. DeepSeek R1, Qwen reasoning_content, or Anthropic extended thinking).

---

AudioContent

Audio content part for speech-capable models.

No deny_unknown_fields: shared with the response side (see AssistantPart.OutputAudio), same rationale as ImageUrl (#51).

Field	Type	Default	Description
data	String	—	Base64-encoded audio data.
format	String	—	Audio format (e.g., “wav”, “mp3”, “ogg”).

---

AuthConfig

Auth configuration block.

Field	Type	Default	Description
authType	AuthType	—	Auth scheme classification.
envVar	@Nullable String	null	Name of the environment variable that holds the API key (e.g. "OPENAI_API_KEY"). Holds the variable name, never the secret value.

---

BatchListQuery

Query parameters for listing batches.

Field	Type	Default	Description
limit	@Nullable Integer	null	Maximum number of results to return. Defaults to 20.
after	@Nullable String	null	Pagination cursor: return results after this batch ID.

---

BatchListResponse

Response from listing batches.

Field	Type	Default	Description
object	String	—	Object type (always "list").
data	List<BatchObject>	Collections.emptyList()	List of batch objects.
hasMore	@Nullable Boolean	null	Whether more results are available.
firstId	@Nullable String	null	First batch ID in the result set (for pagination).
lastId	@Nullable String	null	Last batch ID in the result set (for pagination).

---

BatchObject

A batch job object.

Field	Type	Default	Description
id	String	—	Unique batch ID.
object	String	—	Object type (always "batch").
endpoint	String	—	API endpoint (e.g., "/v1/chat/completions").
inputFileId	String	—	ID of the input file.
completionWindow	String	—	Completion window (e.g., "24h").
status	BatchStatus	BatchStatus.VALIDATING	Current job status.
outputFileId	@Nullable String	null	ID of the output file (present when completed).
errorFileId	@Nullable String	null	ID of the error file (present if some requests failed).
createdAt	long	—	Unix timestamp of batch creation.
completedAt	@Nullable Long	null	Unix timestamp of completion (if completed).
failedAt	@Nullable Long	null	Unix timestamp of failure (if failed).
expiredAt	@Nullable Long	null	Unix timestamp of expiration (if expired).
requestCounts	@Nullable BatchRequestCounts	null	Request processing counts.
metadata	@Nullable Object	null	Metadata attached to the batch.

---

BatchRequestCounts

Request processing counts for a batch.

Field	Type	Default	Description
total	long	—	Total requests in the batch.
completed	long	—	Completed requests.
failed	long	—	Failed requests.

---

BedrockConfig

AWS Bedrock configuration.

All fields are optional; anything left unset falls back to the standard AWS environment variables (AWS_DEFAULT_REGION / AWS_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, BEDROCK_CROSS_REGION).

A Bedrock API key in AWS_BEARER_TOKEN_BEDROCK authenticates on its own and needs none of the credential fields below. It is read per request rather than from this struct; in a build that can sign (the bedrock feature) a credential set here takes precedence over it.

Implements Debug manually (see below) so the AWS credential fields are redacted rather than printed in full.

Field	Type	Default	Description
region	@Nullable String	null	AWS region (e.g. "us-east-1").
crossRegionPrefix	@Nullable String	null	Cross-region inference profile prefix (e.g. "us").
accessKeyId	@Nullable String	null	Explicit AWS access key ID.
secretAccessKey	@Nullable String	null	Explicit AWS secret access key.
sessionToken	@Nullable String	null	Explicit AWS session token (temporary credentials).

---

BudgetConfig

Configuration for budget enforcement.

Field	Type	Default	Description
globalLimit	@Nullable Double	null	Maximum total spend across all models, in USD. null means unlimited.
modelLimits	Map<String, Double>	Collections.emptyMap()	Per-model spending limits in USD. Models not listed here are only constrained by global_limit.
enforcement	Enforcement	Enforcement.HARD	Whether to reject requests or merely warn when a limit is exceeded.

---

CacheConfig

Configuration for the response cache.

Field	Type	Default	Description
maxEntries	long	256	Maximum number of cached entries.
ttl	Duration	300000ms	Time-to-live for each cached entry.
backend	CacheBackend	CacheBackend.MEMORY	Storage backend to use.

---

CatalogRefreshConfig

Plain-data configuration for refresh_catalog.

Deliberately FFI/binding-friendly: no Duration or PathBuf, just primitives that translate directly across language boundaries.

Field	Type	Default	Description
enabled	boolean	false	Runtime catalog refresh is entirely opt-in: when false, refresh_catalog is a no-op that returns Ok(RefreshOutcome.Disabled) without touching the network, the filesystem, or the overlay registry.
sourceUrl	String	"<https://github.com/xberg-io/liter-llm/releases/download/model-catalog/catalog.json>"	Source URL to fetch catalog.json from. Must be https. Defaults to DEFAULT_CATALOG_URL; configurable so self-hosted mirrors work.
ttlSeconds	long	86400	How long a cached catalog.json remains valid before a network refetch is attempted, in seconds.
cachePath	@Nullable String	null	Filesystem path for the on-disk cache. null uses a default path under std.env.temp_dir().

---

ChatCompletionChunk

A streamed chunk of a chat completion response.

Field	Type	Default	Description
id	String	—	Unique identifier for this stream.
object	String	—	Always "chat.completion.chunk" from OpenAI-compatible APIs. Stored as a plain String so non-standard provider values do not fail parsing.
created	long	—	Unix timestamp of chunk creation.
model	String	—	Model used to generate the chunk.
choices	List<StreamChoice>	Collections.emptyList()	Streaming choices (delta updates).
usage	@Nullable Usage	null	Token usage (typically only in the final chunk).
systemFingerprint	@Nullable String	null	Fingerprint of the system configuration (OpenAI-specific).
serviceTier	@Nullable String	null	Service tier used (OpenAI-specific).

---

ChatCompletionRequest

Chat completion request (compatible with OpenAI and similar APIs).

Field	Type	Default	Description
model	String	—	Model ID (e.g., "gpt-4o-mini", "claude-3-5-sonnet").
messages	List<Message>	Collections.emptyList()	Conversation history from oldest to newest.
temperature	@Nullable Double	null	Sampling temperature. Higher increases randomness, lower is more deterministic. Defaults to 1.0. The accepted range depends on the provider the request is routed to. OpenAI-compatible providers accept [0.0, 2.0]; Anthropic and Amazon Bedrock both cap it at 1.0, and for those two a value above the cap is rejected with a BadRequest error before the request is sent, rather than being silently clamped or left for the provider to reject. No range is enforced for providers whose own documentation does not state one — the value is forwarded and the provider decides. Consult the target provider’s reference rather than assuming [0.0, 2.0] is portable.
topP	@Nullable Double	null	Nucleus sampling parameter. Lower is more focused. Accepted ranges vary by provider (most document [0.0, 1.0], but this is not universal — check the target provider’s own documentation for its exact bounds).
n	@Nullable Integer	null	Number of chat completions to generate. Defaults to 1.
stream	@Nullable Boolean	null	Whether to stream the response. Managed by the client layer — do not set directly.
stop	@Nullable StopSequence	null	Stop sequence(s) that halt token generation.
maxTokens	@Nullable Long	null	Max output tokens. Different from max_completion_tokens in some providers.
presencePenalty	@Nullable Double	null	Presence penalty in [-2.0, 2.0]. Positive discourages repeated topics.
frequencyPenalty	@Nullable Double	null	Frequency penalty in [-2.0, 2.0]. Positive discourages repeated tokens.
logitBias	@Nullable Map<String, Double>	Collections.emptyMap()	Token bias map. Uses BTreeMap (sorted keys) for deterministic serialization order — important when hashing or signing requests.
user	@Nullable String	null	User identifier for request tracking and abuse detection.
tools	@Nullable List<ChatCompletionTool>	Collections.emptyList()	Tools the model can invoke.
toolChoice	@Nullable ToolChoice	null	Tool usage mode (auto, required, none, or specific tool).
parallelToolCalls	@Nullable Boolean	null	Whether the model can call multiple tools in parallel. Defaults to true.
responseFormat	@Nullable ResponseFormat	null	Output format constraint (text, JSON, JSON schema).
streamOptions	@Nullable StreamOptions	null	Streaming options (e.g., include_usage).
seed	@Nullable Long	null	Random seed for reproducible outputs. Provider support varies.
reasoningEffort	@Nullable ReasoningEffort	null	Reasoning effort level (minimal, low, medium, high, max) for extended-thinking models.
modalities	@Nullable List<Modality>	Collections.emptyList()	Output modalities to request from the model. For OpenAI audio models, pass ["text", "audio"]. Vertex AI / Gemini translates these to generationConfig.responseModalities (uppercase).
logprobs	@Nullable Boolean	null	Whether to return log probabilities of the output tokens.
topLogprobs	@Nullable Integer	null	Number of most-likely tokens to return log probabilities for, 0..=20. Requires logprobs to be true.
maxCompletionTokens	@Nullable Long	null	Upper bound on generated tokens, including reasoning tokens. Supersedes max_tokens on OpenAI reasoning models, which reject max_tokens outright.
serviceTier	@Nullable String	null	Latency tier to process the request under (e.g. "auto", "default", "flex").
store	@Nullable Boolean	null	Whether to store the completion for later retrieval by the provider.
metadata	@Nullable Map<String, String>	Collections.emptyMap()	Developer-defined tags attached to the completion.
prediction	@Nullable Object	null	Predicted output, for latency reduction when much of the response is known ahead of time. Untyped: the shape is provider-defined and still evolving, and the value is forwarded verbatim.
audio	@Nullable Object	null	Audio output parameters, required when modalities includes audio. Untyped for the same reason as prediction.
webSearchOptions	@Nullable Object	null	Web-search tool configuration for search-enabled models. Untyped for the same reason as prediction.
extraBody	@Nullable Object	null	Provider-specific extra parameters merged into the request body. Use for guardrails, safety settings, grounding config, etc.

---

ChatCompletionResponse

Chat completion response from the API.

Field	Type	Default	Description
id	String	—	Unique identifier for this response.
object	String	—	Always "chat.completion" from OpenAI-compatible APIs. Stored as a plain String so non-standard provider values do not break deserialization.
created	long	—	Unix timestamp of response creation.
model	String	—	Model used to generate the response.
choices	List<Choice>	Collections.emptyList()	List of completion choices.
usage	@Nullable Usage	null	Token usage statistics.
systemFingerprint	@Nullable String	null	Fingerprint of the system configuration (OpenAI-specific).
serviceTier	@Nullable String	null	Service tier used (OpenAI-specific).

---

ChatCompletionTool

A tool the model can invoke (currently, all tools are functions).

Field	Type	Default	Description
toolType	ToolType	—	Tool type (always “function” in OpenAI spec).
function	FunctionDefinition	—	Function definition with name, description, and JSON schema parameters.

---

Choice

A single completion choice.

Field	Type	Default	Description
index	int	—	Index of this choice in the choices array.
message	AssistantMessage	—	The assistant’s message response. Serialized with an explicit role: "assistant". The field is not stored on AssistantMessage because Message is an internally-tagged enum keyed on role, so a stored field would emit the key twice inside a request. OpenAI’s response schema requires it here.
finishReason	@Nullable FinishReason	null	Why the model stopped generating (stop, length, tool_calls, content_filter, etc.).
logprobs	@Nullable Object	null	Per-token log probabilities, when the request asked for them. Required by OpenAI’s response schema as an always-present, nullable key, so this is deliberately not skip_serializing_if.

---

ChunkMiddleware

A per-chunk transformation in the StreamPipeline.

Each middleware receives a typed chunk and returns Ok(Some(chunk)) to pass it through (optionally modified), Ok(None) to drop the chunk, or Err(e) to propagate a stream error.

The trait is object-safe so multiple middleware implementations can be chained inside StreamPipeline.

Methods

process()

Process a single chunk.

• Ok(Some(chunk)) — emit (possibly transformed) chunk.
• Ok(None) — drop this chunk silently.
• Err(e) — propagate as a stream error.

Signature:

public @Nullable ChatCompletionChunk process(ChatCompletionChunk chunk) throws LiterLlmRsException

Example:

var result = instance.process(new ChatCompletionChunk());

Parameters:

Name	Type	Required	Description
chunk	ChatCompletionChunk	Yes	The chat completion chunk

Returns: @Nullable ChatCompletionChunk

Errors: Throws LiterLlmRsException.

---

CreateBatchRequest

Request to create a batch job.

Field	Type	Default	Description
inputFileId	String	—	ID of the uploaded input file (JSONL format).
endpoint	String	—	API endpoint (e.g., "/v1/chat/completions").
completionWindow	String	—	Completion window (e.g., "24h").
metadata	@Nullable Object	null	Optional metadata to attach to the batch.

---

CreateFileRequest

Request to upload a file.

Field	Type	Default	Description
file	String	—	Base64-encoded file data.
purpose	FilePurpose	FilePurpose.ASSISTANTS	Purpose for the file.
filename	@Nullable String	null	Optional filename to associate with the upload.

---

CreateImageRequest

Request to create images from a text prompt.

Field	Type	Default	Description
prompt	String	—	Text description of the image to generate.
model	@Nullable String	null	Model ID (e.g., "dall-e-3"). Optional; API may use default if unset.
n	@Nullable Integer	null	Number of images to generate. Defaults to 1.
size	@Nullable String	null	Image size (e.g., "1024x1024", "1792x1024").
quality	@Nullable String	null	Image quality: "standard" or "hd".
style	@Nullable String	null	Style: "natural" or "vivid" (DALL-E 3 only).
responseFormat	@Nullable String	null	Response format: "url" or "b64_json".
user	@Nullable String	null	User identifier for request tracking.

---

CreateResponseRequest

Request to create a response via the OpenAI Responses API (POST /responses).

Provider support

The Responses API path is OpenAI-only. This type models the OpenAI Responses wire format, and unlike ChatCompletionRequest the body is sent to the provider verbatim: neither Provider.transform_request nor Provider.transform_response runs for Responses calls, and no provider in schemas/providers.json declares a responses endpoint.

Pointing a Responses call at a provider that does not natively serve the OpenAI /responses contract (Anthropic, Vertex, Bedrock, Cohere, Google AI, Azure) is not supported: the request goes out unmodified, so the provider rejects it or returns a body that cannot be deserialized into ResponseObject. Use ChatCompletionRequest for cross-provider work — that path applies the per-provider request and response normalization this one does not.

ChatCompletionRequest: crate.types.ChatCompletionRequest

Field	Type	Default	Description
model	String	—	Model ID, as named by the OpenAI Responses API (e.g. "gpt-5"). Sent to the wire exactly as given. The Responses path performs none of the chat path’s model handling: a provider/model routing prefix is not stripped and does not re-route the request, which stays pinned to the provider the client was constructed with.
input	Object	—	Input data to process (e.g., a document to extract from).
instructions	@Nullable String	null	Instructions for processing the input.
tools	@Nullable List<ResponseTool>	Collections.emptyList()	Available tools the model can use.
temperature	@Nullable Double	null	Sampling temperature in [0.0, 2.0]. Defaults to 1.0.
maxOutputTokens	@Nullable Long	null	Maximum output tokens.
metadata	@Nullable Object	null	Optional metadata.
extraBody	@Nullable Object	null	Extra top-level parameters shallow-merged into the request body, OpenAI-Python style ({**body, **extra_body}) — keys here override identically named fields above. Use it for OpenAI Responses fields this struct does not model directly, such as the top-level reasoning.effort. This is an OpenAI escape hatch, not a cross-provider one. On the chat path the providers that consume extra_body natively (Anthropic, Vertex, Bedrock) claim it inside their own transform_request; here no provider transform runs, so the merged keys always travel to the wire as literal OpenAI Responses fields. A non-object value cannot be merged into the body root and is dropped with a warning rather than sent.
stream	@Nullable Boolean	null	Whether to stream the response. Managed by the client layer — do not set directly.

---

CreateSpeechRequest

Request to generate speech audio from text.

Field	Type	Default	Description
model	String	—	Model ID (e.g., "tts-1", "tts-1-hd").
input	String	—	Text to synthesize into speech.
voice	String	—	Voice name (e.g., "alloy", "echo", "fable", "onyx", "nova", "shimmer").
responseFormat	@Nullable String	null	Audio format (e.g., "mp3", "opus", "aac", "flac", "wav", "pcm").
speed	@Nullable Double	null	Playback speed in [0.25, 4.0]. Defaults to 1.0.

---

CreateTranscriptionRequest

Request to transcribe audio into text.

Field	Type	Default	Description
model	String	—	Model ID (e.g., "whisper-1").
file	String	—	Base64-encoded audio file data.
language	@Nullable String	null	Language ISO-639-1 code (e.g., "en", "fr", "de"). Optional; model auto-detects.
prompt	@Nullable String	null	Optional text to guide the model (improves accuracy for domain-specific terms).
responseFormat	@Nullable String	null	Output format (e.g., "json", "text", "vtt", "srt", "verbose_json").
temperature	@Nullable Double	null	Sampling temperature in [0.0, 1.0]. Higher increases variability. Defaults to 0.

---

CustomProviderConfig

Configuration for registering a custom LLM provider at runtime.

Field	Type	Default	Description
name	String	—	Unique name for this provider (e.g., “my-provider”).
baseUrl	String	—	Base URL for the provider’s API (e.g., <https://api.my-provider.com/v1>).
authHeader	AuthHeaderFormat	—	Authentication header format.
modelPrefixes	List<String>	—	Model name prefixes that route to this provider (e.g., ["my-"]).

---

DecodedDataUrl

Result of decoding a data: URL — MIME type and the decoded byte payload.

Named struct (rather than a tuple) so polyglot bindings can extract decode_data_url with a typed return rather than a sanitized scalar.

Field	Type	Default	Description
mime	String	—	MIME type extracted from the URL prefix (verbatim, not normalised).
data	byte\[\]	—	Decoded base64 payload.

---

DefaultClient

Default client implementation backed by reqwest.

Sends requests to 165 LLM providers with automatic provider detection and per-request routing. The provider is resolved at construction time from model_hint (or defaults to OpenAI), but individual requests can override the provider via model name prefix (e.g. "anthropic/claude-3-5-sonnet" routes to Anthropic regardless of construction-time setting).

When the model prefix does not match any known provider, the construction-time provider is used as the fallback. This enables seamless migration between providers by changing only the model name.

The provider is stored behind an Arc so it can be shared cheaply into async closures and streaming tasks. Pre-computed auth headers and extra headers are cached at construction to avoid redundant encoding on every request.

Methods

fetchBatchForPolling()

Signature:

public BatchObject fetchBatchForPolling(String batchId) throws LiterLlmRsException

Example:

var result = instance.fetchBatchForPolling("value");

Parameters:

Name	Type	Required	Description
batchId	String	Yes	The batch id

Returns: BatchObject

Errors: Throws LiterLlmRsException.

waitForBatch()

Poll a batch until it reaches a terminal status (Completed, Failed, Expired, Cancelled).

Uses exponential backoff with configurable initial interval, maximum interval, and backoff multiplier. Optionally supports a timeout that aborts polling if exceeded.

Errors:

Returns BatchWaitError.Failed if the batch reaches a failure terminal status. Returns BatchWaitError.Timeout if the configured timeout is exceeded. Returns BatchWaitError.Client for underlying client errors.

Signature:

public BatchObject waitForBatch(String batchId, WaitForBatchConfig config) throws LiterLlmRsException

Example:

var result = instance.waitForBatch("value", new WaitForBatchConfig());

Parameters:

Name	Type	Required	Description
batchId	String	Yes	The batch id
config	WaitForBatchConfig	Yes	The configuration options

Returns: BatchObject

Errors: Throws LiterLlmRsException.

close()

Releases the native handle backing this instance. Implements AutoCloseable; safe to call more than once.

Signature:

public void close() throws LiterLlmRsException

Example:

instance.close();

Returns: No return value.

---

DeleteResponse

Response from a delete operation.

Field	Type	Default	Description
id	String	—	ID of the deleted resource.
object	String	—	Object type.
deleted	boolean	—	Confirmation that the resource was deleted.

---

DeveloperMessage

Developer message (system-like message for Claude models).

Field	Type	Default	Description
content	String	—	Developer-specific instructions or context.
name	@Nullable String	null	Optional name for the developer message source.

---

DocumentContent

PDF/document content part for vision-capable models.

Field	Type	Default	Description
data	String	—	Base64-encoded document data or URL.
mediaType	String	—	MIME type (e.g., “application/pdf”, “text/csv”).

---

EmbeddingObject

A single embedding vector.

Field	Type	Default	Description
object	String	—	Always "embedding" from OpenAI-compatible APIs. Stored as a plain String so non-standard provider values do not break deserialization.
embedding	List<Float>	—	The embedding vector. Providers may return this as a JSON float array or, when encoding_format: "base64" was requested, as a base64 string of little-endian f32 bytes. Base64 responses are decoded on read; this field always serializes back out as a JSON float array.
index	int	—	Index in the batch (corresponds to input order).

---

EmbeddingRequest

Embedding request.

Field	Type	Default	Description
model	String	—	Model ID (e.g., "text-embedding-3-small").
input	EmbeddingInput	EmbeddingInput.SINGLE	Text, texts, or multimodal content to embed.
encodingFormat	@Nullable EmbeddingFormat	null	Output format: float (native) or base64.
dimensions	@Nullable Integer	null	Requested embedding dimensions (if supported by the model).
user	@Nullable String	null	User identifier for request tracking.

---

EmbeddingResponse

Embedding response.

Field	Type	Default	Description
object	String	—	Always "list" from OpenAI-compatible APIs. Stored as a plain String so non-standard provider values do not break deserialization.
data	List<EmbeddingObject>	—	List of embeddings.
model	String	—	Model used to generate embeddings.
usage	@Nullable Usage	/* serde(default) */	Token usage (input tokens only; embeddings have zero output tokens).

---

FileListQuery

Query parameters for listing files.

Field	Type	Default	Description
purpose	@Nullable String	null	Filter by file purpose (e.g., "batch", "fine-tune").
limit	@Nullable Integer	null	Maximum number of results to return. Defaults to 20.
after	@Nullable String	null	Pagination cursor: return results after this file ID.

---

FileListResponse

Response from listing files.

Field	Type	Default	Description
object	String	—	Object type (always "list").
data	List<FileObject>	Collections.emptyList()	List of file objects.
hasMore	@Nullable Boolean	null	Whether more results are available.

---

FileObject

An uploaded file object.

Field	Type	Default	Description
id	String	—	Unique file ID.
object	String	—	Object type (always "file").
bytes	long	—	File size in bytes.
createdAt	long	—	Unix timestamp of file creation.
filename	String	—	Filename.
purpose	String	—	File purpose.
status	@Nullable String	null	Processing status (e.g., "uploaded", "processed").

---

FunctionCall

Function call details.

Field	Type	Default	Description
name	String	—	Function name.
arguments	String	—	Arguments as a JSON string (parse with serde_json.from_str).

---

FunctionDefinition

Function definition exposed to the model.

Field	Type	Default	Description
name	String	—	Name of the function. Required and must be alphanumeric + underscores.
description	@Nullable String	/* serde(default) */	Human-readable description explaining what the function does.
parameters	@Nullable Object	/* serde(default) */	JSON Schema defining the function’s parameters.
strict	@Nullable Boolean	/* serde(default) */	If true, enforce strict JSON schema validation for arguments.

---

FunctionMessage

Deprecated legacy function-role message body.

Field	Type	Default	Description
content	String	—	The extracted text content
name	String	—	The name

---

HealthChecker

Abstraction over a health probe strategy.

Implementors issue a lightweight probe against upstream (typically a provider base URL or named identifier) and report HealthStatus.

Methods

check()

Probe upstream and return its current HealthStatus.

The parameter is taken by value (String) so that implementations can move it into the returned future without a clone, making the 'static + Send bound on the future trivially satisfiable.

Signature:

public HealthStatus check(String upstream) throws LiterLlmRsException

Example:

var result = instance.check("value");

Parameters:

Name	Type	Required	Description
upstream	String	Yes	The upstream

Returns: HealthStatus

---

Image

A single generated image, returned as either a URL or base64 data.

Field	Type	Default	Description
url	@Nullable String	null	Image URL (if response_format was “url”).
b64Json	@Nullable String	null	Base64-encoded image data (if response_format was “b64_json”).
revisedPrompt	@Nullable String	null	The final prompt used to generate the image (DALL-E 3).

---

ImageUrl

An image URL reference with optional detail level for processing.

No deny_unknown_fields: this type is shared with the response side (see AssistantPart.OutputImage) where it is deserialized from provider output, not just constructed as request input (see #51). A provider adding a new field to its image-output object must not hard-fail the whole response.

Field	Type	Default	Description
url	String	—	URL of the image (data URI or HTTP/HTTPS URL).
detail	@Nullable ImageDetail	null	Detail level: low (512x512), high (2x2 tiles), or auto (model-selected).

---

ImagesResponse

Response containing generated images.

Field	Type	Default	Description
created	long	—	Unix timestamp of image creation.
data	List<Image>	Collections.emptyList()	List of generated images.

---

InFlightLimitConfig

Configuration for the global per-client in-flight request limit.

Field	Type	Default	Description
maxInFlight	@Nullable Long	null	Maximum simultaneously outstanding provider requests. null means unlimited.

---

IntentPrototype

An intent prototype: (intent_name, prototype_embedding, target_model_id).

Field	Type	Default	Description
name	String	—	Human-readable name for the intent (used in logs/metrics).
embedding	List<Double>	—	Pre-computed embedding vector for this intent.
model	String	—	Model to route to when this intent is detected.

---

JsonSchemaFormat

JSON Schema specification for constrained output.

Field	Type	Default	Description
name	String	—	Name of the schema (must be unique in the request).
description	@Nullable String	null	Description of what the schema represents.
schema	Object	—	JSON Schema object defining the output structure.
strict	@Nullable Boolean	null	If true, enforce strict schema validation.

---

LlmBudgetConfig

Budget enforcement configuration.

Field	Type	Default	Description
globalLimit	@Nullable Double	null	Global spend limit in USD.
modelLimits	@Nullable Map<String, Double>	Collections.emptyMap()	Per-model spend limits in USD, keyed by model name.
enforcement	@Nullable String	null	Enforcement mode: "hard" (reject over-budget requests) or "soft" (log only).

---

LlmCacheConfig

Response cache configuration.

Field	Type	Default	Description
maxEntries	@Nullable Long	null	Maximum number of cached entries.
ttlSeconds	@Nullable Long	null	Cache entry time-to-live, in seconds.
backend	@Nullable String	null	Cache backend name (e.g. "memory", or an opendal scheme).
backendConfig	@Nullable Map<String, String>	Collections.emptyMap()	Backend-specific configuration key/value pairs.

---

LlmConfig

Canonical configuration for an LLM client.

All fields except model are optional so that partially-specified configs (e.g. from environment-driven defaults) round-trip cleanly. Convert to a runtime client configuration via LlmConfig.into_client_builder.

temperature and max_tokens are request-time parameters rather than client-level settings; they are carried on this struct for callers to read when building individual requests, and are intentionally not mapped by LlmConfig.into_client_builder.

Implements Debug manually (see below) so api_key and header values are redacted rather than printed in full.

Field	Type	Default	Description
model	String	—	Model identifier (e.g. "gpt-4o", "bedrock/anthropic.claude-3-sonnet-20240229-v1:0").
apiKey	@Nullable String	null	API key for authentication.
baseUrl	@Nullable String	null	Override base URL. When set, all requests go here and provider auto-detection is skipped.
timeoutSecs	@Nullable Long	null	Request timeout, in seconds.
maxRetries	@Nullable Integer	null	Maximum number of retries on 429 / 5xx responses.
temperature	@Nullable Double	null	Sampling temperature for requests built from this config.
maxTokens	@Nullable Long	null	Maximum number of tokens to generate for requests built from this config.
loadEnv	@Nullable Boolean	null	Automatically load the API key from the provider’s environment variable when no explicit key is provided (default: true).
headers	@Nullable Map<String, String>	Collections.emptyMap()	Extra headers sent on every request.
providers	@Nullable List<LlmProviderConfig>	Collections.emptyList()	Custom provider configurations, in addition to the built-in providers.
cache	@Nullable LlmCacheConfig	null	Response cache configuration.
budget	@Nullable LlmBudgetConfig	null	Budget enforcement configuration.
rateLimit	@Nullable LlmRateLimitConfig	null	Per-model rate limiting configuration.
inFlightLimit	@Nullable LlmInFlightLimitConfig	null	Global per-client in-flight provider request limit.
costTracking	@Nullable Boolean	null	Enable per-request cost tracking.
tracing	@Nullable Boolean	null	Enable OpenTelemetry-compatible tracing spans.
cooldownSecs	@Nullable Long	null	Cooldown duration after transient errors, in seconds.
healthCheckSecs	@Nullable Long	null	Background health check interval, in seconds.
bedrock	@Nullable BedrockConfig	null	AWS Bedrock configuration (region, credentials, cross-region routing).

---

LlmInFlightLimitConfig

Global per-client in-flight provider request limit.

Field	Type	Default	Description
maxInFlight	@Nullable Long	null	Maximum simultaneously outstanding provider requests. null means unlimited.

---

LlmProviderConfig

A custom provider configuration entry.

Field	Type	Default	Description
name	String	—	Provider name, used to key model prefix matching.
baseUrl	String	—	Base URL for the provider’s OpenAI-compatible API.
authHeader	@Nullable String	null	Header name used to carry the API key (defaults to Authorization when unset).
modelPrefixes	List<String>	Collections.emptyList()	Model name prefixes routed to this provider (e.g. ["my-provider/"]).

---

LlmRateLimitConfig

Per-model rate limiting configuration.

Field	Type	Default	Description
rpm	@Nullable Integer	null	Requests per minute limit.
tpm	@Nullable Long	null	Tokens per minute limit.
windowSeconds	@Nullable Long	null	Rate limit window, in seconds.

---

ModelInfo

Public, FFI-friendly snapshot of a model’s pricing and capability metadata, projected from ModelPricing.

Unlike ModelPricing (which is excluded from binding generation), ModelInfo is an owned plain-data DTO safe to hand across the FFI boundary — see model_info.

Field	Type	Default	Description
inputCostPerToken	double	—	Cost in USD per input (prompt) token.
outputCostPerToken	double	—	Cost in USD per output (completion) token.
cacheReadInputTokenCost	@Nullable Double	null	Cost in USD per cached input token (cache hit / read).
cacheCreationInputTokenCost	@Nullable Double	null	Cost in USD per token written to the prompt cache.
inputCostPerAudioToken	@Nullable Double	null	Cost in USD per input audio token.
outputCostPerAudioToken	@Nullable Double	null	Cost in USD per output audio token.
outputCostPerReasoningToken	@Nullable Double	null	Cost in USD per reasoning (extended-thinking) output token.
maxTokens	@Nullable Long	null	Total context window size in tokens (input + output).
maxInputTokens	@Nullable Long	null	Maximum input (prompt) tokens accepted.
maxOutputTokens	@Nullable Long	null	Maximum output (completion) tokens the model can generate.
mode	@Nullable String	null	Best-effort operating mode, e.g. "chat", "embedding".
supportsVision	@Nullable Boolean	null	The model accepts image input.
supportsFunctionCalling	@Nullable Boolean	null	The model supports tool / function calling.
supportsReasoning	@Nullable Boolean	null	The model supports extended-thinking / reasoning tokens.
supportsStructuredOutput	@Nullable Boolean	null	The model supports JSON-mode or response_format structured output.
supportsAudioInput	@Nullable Boolean	null	The model accepts audio input.
supportsAudioOutput	@Nullable Boolean	null	The model can generate audio output.
supportsPromptCaching	@Nullable Boolean	null	The model supports prompt caching.
tiers	List<ModelTier>	Collections.emptyList()	Context-tiered pricing overrides, sorted by ascending min_context_tokens. Empty when the model has flat pricing.

---

ModelObject

A model available from the API.

Field	Type	Default	Description
id	String	—	Model ID (e.g., "gpt-4o", "claude-3-5-sonnet").
object	String	—	Always "model" from OpenAI-compatible APIs. Stored as a plain String so non-standard provider values do not break deserialization. Defaults to empty when a provider omits the field.
created	long	—	Unix timestamp of model creation (or release date). Defaults to 0 when a provider omits it — DeepSeek and some other OpenAI-compatible providers do not return created from /v1/models.
ownedBy	String	—	Organization or entity that owns the model. Defaults to empty when a provider omits the field.

---

ModelTier

Public, FFI-friendly snapshot of a single context-window pricing tier, projected from PricingTier.

Field	Type	Default	Description
minContextTokens	long	—	The tier applies when the prompt/context token count is at least this value.
inputCostPerToken	double	—	Cost in USD per input (prompt) token within this tier.
outputCostPerToken	double	—	Cost in USD per output (completion) token within this tier.
cacheReadInputTokenCost	@Nullable Double	null	Cost in USD per cached input token within this tier.
cacheCreationInputTokenCost	@Nullable Double	null	Cost in USD per cache-write token within this tier.
inputCostPerAudioToken	@Nullable Double	null	Cost in USD per input audio token within this tier.
outputCostPerAudioToken	@Nullable Double	null	Cost in USD per output audio token within this tier.
outputCostPerReasoningToken	@Nullable Double	null	Cost in USD per reasoning output token within this tier.

---

ModelsListResponse

Response listing available models.

Field	Type	Default	Description
object	String	—	Always "list" from OpenAI-compatible APIs. Stored as a plain String so non-standard provider values do not break deserialization. Defaults to empty when a provider omits the field.
data	List<ModelObject>	Collections.emptyList()	List of available models.

---

ModerationCategories

Boolean flags for each moderation category.

Field	Type	Default	Description
sexual	boolean	—	Sexual content.
hate	boolean	—	Hate speech.
harassment	boolean	—	Harassment.
selfHarm	boolean	—	Self-harm content.
sexualMinors	boolean	—	Sexual content involving minors.
hateThreatening	boolean	—	Hate speech that threatens violence.
violenceGraphic	boolean	—	Graphic violence.
selfHarmIntent	boolean	—	Intent to self-harm.
selfHarmInstructions	boolean	—	Instructions for self-harm.
harassmentThreatening	boolean	—	Harassment that threatens violence.
violence	boolean	—	Non-graphic violence.

---

ModerationCategoryScores

Confidence scores for each moderation category.

Field	Type	Default	Description
sexual	double	—	Sexual content score.
hate	double	—	Hate speech score.
harassment	double	—	Harassment score.
selfHarm	double	—	Self-harm content score.
sexualMinors	double	—	Sexual content involving minors score.
hateThreatening	double	—	Hate speech that threatens violence score.
violenceGraphic	double	—	Graphic violence score.
selfHarmIntent	double	—	Intent to self-harm score.
selfHarmInstructions	double	—	Instructions for self-harm score.
harassmentThreatening	double	—	Harassment that threatens violence score.
violence	double	—	Non-graphic violence score.

---

ModerationRequest

Request to classify content for policy violations.

Field	Type	Default	Description
input	ModerationInput	ModerationInput.SINGLE	Text or texts to check.
model	@Nullable String	null	Model ID (e.g., "text-moderation-latest"). Optional; API uses default if unset.

---

ModerationResponse

Response from the moderation endpoint.

Field	Type	Default	Description
id	String	—	Unique identifier for this moderation request.
model	String	—	Model used for classification.
results	List<ModerationResult>	—	Results for each input string.

---

ModerationResult

A single moderation classification result.

Field	Type	Default	Description
flagged	boolean	—	True if any category was flagged.
categories	ModerationCategories	—	Boolean flags for each moderation category.
categoryScores	ModerationCategoryScores	—	Confidence scores for each category.

---

OcrImage

An image extracted from an OCR page.

Field	Type	Default	Description
id	String	—	Unique image identifier within the document.
imageBase64	@Nullable String	/* serde(default) */	Base64-encoded image data (if include_image_base64 was true).

---

OcrPage

A single page of OCR output.

Field	Type	Default	Description
index	int	—	Page index (0-based).
markdown	String	—	Extracted page content as Markdown.
images	@Nullable List<OcrImage>	/* serde(default) */	Embedded images extracted from the page (if include_image_base64 was true).
dimensions	@Nullable PageDimensions	/* serde(default) */	Page dimensions in pixels, if available.

---

OcrRequest

An OCR request.

Field	Type	Default	Description
model	String	—	The model/provider to use (e.g. "mistral/mistral-ocr-latest").
document	OcrDocument	OcrDocument.URL	The document to process (URL or base64).
pages	@Nullable List<Integer>	Collections.emptyList()	Specific pages to process (1-indexed). null means all pages.
includeImageBase64	@Nullable Boolean	null	Whether to include base64-encoded images of each processed page.

---

OcrResponse

An OCR response.

Field	Type	Default	Description
pages	List<OcrPage>	—	Extracted pages in order.
model	String	—	Model/provider used for OCR.
usage	@Nullable Usage	/* serde(default) */	Token usage, if reported by the provider.

---

PageDimensions

Page dimensions in pixels.

Field	Type	Default	Description
width	int	—	Width in pixels.
height	int	—	Height in pixels.

---

PromptTokensDetails

Breakdown of tokens used in the prompt portion of a request.

cached_tokens is included in Usage.prompt_tokens — it is not an additional charge on top of the prompt token count. When pricing supports a cache_read_input_token_cost, the cached portion is billed at the discounted rate and the remainder at the regular input rate.

Field	Type	Default	Description
cachedTokens	long	—	Cached tokens present in the prompt. Defaults to 0 when absent.
audioTokens	long	—	Audio input tokens present in the prompt. Defaults to 0 when absent.

---

ProviderCapabilities

Static capability flags for a provider.

Each flag indicates whether the provider’s models generally support that feature. For providers that aggregate many underlying models (e.g. Bedrock, OpenRouter, vLLM) the flags reflect the superset of available model capabilities — a flag being true means at least one model supports the feature, not every model.

All flags default to false so that newly added providers are safe.

Access via the crate-level capabilities function:

Field	Type	Default	Description
vision	boolean	—	The provider accepts image input in chat messages.
reasoning	boolean	—	The provider supports extended-thinking / reasoning tokens.
structuredOutput	boolean	—	The provider supports JSON-mode or response_format structured output.
functionCalling	boolean	—	The provider supports tool / function calling.
audioIn	boolean	—	The provider accepts audio as input.
audioOut	boolean	—	The provider can generate audio / TTS output.
videoIn	boolean	—	The provider accepts video as input.

---

ProviderConfig

Static configuration for a single provider entry in providers.json.

This struct deliberately does not include capability flags or streaming format, which are accessed via the capabilities function.

Field	Type	Default	Description
name	String	—	Provider identifier (matches the entry key in providers.json).
displayName	@Nullable String	null	Human-readable provider name shown in UIs.
baseUrl	@Nullable String	null	Base URL used as the default for this provider’s HTTP client.
auth	@Nullable AuthConfig	null	Authentication scheme metadata (auth type + env var holding the key).
endpoints	@Nullable List<String>	null	Supported endpoint kinds (e.g. chat, embeddings).
modelPrefixes	@Nullable List<String>	null	Model-name prefixes claimed by this provider (e.g. ["gpt-", "o1-"]).
paramMappings	@Nullable Map<String, String>	null	Parameter key renaming for this provider. Each entry maps an OpenAI-spec field name (e.g. "max_completion_tokens") to the name this provider expects (e.g. "max_tokens"). Applied automatically by ConfigDrivenProvider.transform_request.

---

RateLimitConfig

Configuration for per-model rate limits.

Field	Type	Default	Description
rpm	@Nullable Integer	null	Maximum requests per window. null means unlimited.
tpm	@Nullable Long	null	Maximum tokens per window. null means unlimited.
window	Duration	60000ms	Fixed window duration (defaults to 60 s).

---

RerankRequest

Request to rerank documents by relevance to a query.

Field	Type	Default	Description
model	String	—	Model ID (e.g., "cohere/rerank-english-v3.0").
query	String	—	The search query.
documents	List<RerankDocument>	Collections.emptyList()	Documents to rerank.
topN	@Nullable Integer	null	Return only the top N results. Optional.
returnDocuments	@Nullable Boolean	null	Include the document content in results. Defaults to false.

---

RerankResponse

Response from the rerank endpoint.

Field	Type	Default	Description
id	@Nullable String	null	Unique identifier for this rerank request.
results	List<RerankResult>	—	Reranked documents in order of relevance.
meta	@Nullable Object	/* serde(default) */	Optional metadata about the reranking operation.

---

RerankResult

A single reranked document with its relevance score.

Field	Type	Default	Description
index	int	—	Original document index in the input list.
relevanceScore	double	—	Relevance score in [0, 1]. Higher indicates more relevant.
document	@Nullable RerankResultDocument	/* serde(default) */	Original document content (if return_documents was true).

---

RerankResultDocument

The text content of a reranked document, returned when return_documents is true.

Field	Type	Default	Description
text	String	—	Document text.

---

ResponseObject

Response from a structured response request.

Field	Type	Default	Description
id	String	—	Unique response ID.
object	String	—	Object type (e.g., "response").
createdAt	long	—	Unix timestamp of response creation.
model	String	—	Model used to generate the response.
status	String	—	Status (e.g., "succeeded", "failed").
output	List<ResponseOutputItem>	Collections.emptyList()	Output items from the response.
usage	@Nullable ResponseUsage	null	Token usage.
error	@Nullable Object	null	Error details (if status is “failed”).

---

ResponseOutputItem

A single output item from the response.

Field	Type	Default	Description
itemType	String	—	Output type (e.g., "text", "object", "error").
content	Object	—	Output content (flattened into the object).

---

ResponseTool

A tool available for the response request.

Field	Type	Default	Description
toolType	String	—	Tool type (e.g., “extractor”, “search”).
config	Object	—	Tool configuration (flattened into the object).

---

ResponseUsage

Token usage for a response.

Field	Type	Default	Description
inputTokens	long	—	Input tokens used.
outputTokens	long	—	Output tokens used.
totalTokens	long	—	Total tokens used.

---

SearchRequest

A search request.

Field	Type	Default	Description
model	String	—	The model/provider to use (e.g. "brave/web-search", "tavily/search").
query	String	—	The search query string.
maxResults	@Nullable Integer	null	Maximum number of results to return.
searchDomainFilter	@Nullable List<String>	Collections.emptyList()	Domain filter — restrict results to specific domains.
country	@Nullable String	null	Country code for localized results (ISO 3166-1 alpha-2, e.g., "US", "FR").

---

SearchResponse

A search response.

Field	Type	Default	Description
results	List<SearchResult>	—	List of search results.
model	String	—	Model/provider that performed the search.

---

SearchResult

An individual search result.

Field	Type	Default	Description
title	String	—	Result title.
url	String	—	Result URL.
snippet	String	—	Text snippet or excerpt from the page.
date	@Nullable String	/* serde(default) */	Publication or last-updated date, if available.

---

SingleflightResult

The value broadcast from a singleflight leader to all followers.

The error value is shared so every follower receives the same upstream failure without cloning the underlying error.

Methods

close()

Releases the native handle backing this instance. Implements AutoCloseable; safe to call more than once.

Signature:

public void close() throws LiterLlmRsException

Example:

instance.close();

Returns: No return value.

---

SpecificFunction

Name of the specific function to invoke.

Field	Type	Default	Description
name	String	—	Function name.

---

SpecificToolChoice

Directive to call a specific tool.

Field	Type	Default	Description
choiceType	ToolType	ToolType.FUNCTION	Tool type (always “function”).
function	SpecificFunction	—	The specific function to invoke.

---

StreamChoice

A streaming choice with incremental delta.

Field	Type	Default	Description
index	int	—	Index of this choice in the choices array.
delta	StreamDelta	—	Incremental update to the message (content, tool calls, etc.).
finishReason	@Nullable FinishReason	null	Why the stream ended (present only in final chunk).

---

StreamDelta

Incremental delta in a stream chunk.

Field	Type	Default	Description
role	@Nullable String	null	Role (typically present only in the first chunk).
content	@Nullable String	null	Partial content chunk (e.g., a few words of the response).
toolCalls	@Nullable List<StreamToolCall>	Collections.emptyList()	Partial tool calls being streamed.
functionCall	@Nullable StreamFunctionCall	null	Deprecated legacy function_call delta; retained for API compatibility.
refusal	@Nullable String	null	Partial refusal message.
reasoningContent	@Nullable String	null	Partial reasoning/thinking tokens (OpenAI-compatible extension used by DeepSeek R1, Qwen, etc.).

---

StreamFunctionCall

Partial function call details in a stream.

Field	Type	Default	Description
name	@Nullable String	null	Function name (typically in the first chunk).
arguments	@Nullable String	null	Partial JSON arguments chunk.

---

StreamOptions

Options for streaming responses.

Field	Type	Default	Description
includeUsage	@Nullable Boolean	null	If true, include token usage in the final stream chunk.

---

StreamToolCall

A streaming tool call being built incrementally.

Field	Type	Default	Description
index	int	—	Index of this tool call in the tool_calls array.
id	@Nullable String	null	Tool call ID (typically in the first chunk for this call).
callType	@Nullable ToolType	null	Tool type (typically “function”).
function	@Nullable StreamFunctionCall	null	Partial function name and arguments.

---

SystemMessage

System message guiding model behavior for the entire conversation.

Field	Type	Default	Description
content	UserContent	UserContent.TEXT	Instructions or context that apply throughout the conversation. Accepts either a plain text string or an array of content parts, mirroring UserContent so that Message.system_with_parts works.
name	@Nullable String	null	Optional name for the system message source.

---

ToolCall

A tool call the model wants to execute.

Field	Type	Default	Description
id	String	—	Unique ID for this call, used to reference in tool result messages.
callType	ToolType	—	Tool type (always “function”).
function	FunctionCall	—	Function name and arguments.

---

ToolMessage

Tool execution result returned to the model.

Field	Type	Default	Description
content	UserContent	UserContent.TEXT	Result of the tool execution as plain text or an array of content parts (text, images, documents, audio), mirroring UserMessage.content. #[serde(untagged)] on UserContent means a bare JSON string still deserialises into Text, so tool results persisted before this field carried structured content continue to round-trip.
toolCallId	String	—	ID of the tool call this result responds to.
name	@Nullable String	null	Optional tool/function name.

---

TranscriptionResponse

Response from a transcription request.

Field	Type	Default	Description
text	String	—	The transcribed text.
language	@Nullable String	null	Detected language (ISO-639-1 code).
duration	@Nullable Double	null	Total audio duration in seconds.
segments	@Nullable List<TranscriptionSegment>	Collections.emptyList()	Detailed segment-level transcription (if response_format is “verbose_json”).

---

TranscriptionSegment

A segment of transcribed audio with timing information.

Field	Type	Default	Description
id	int	—	Segment index (0-based).
start	double	—	Start time in seconds.
end	double	—	End time in seconds.
text	String	—	Transcribed text for this segment.

---

Usage

Token-usage accounting returned by the provider on each completion / embedding call.

Field	Type	Default	Description
promptTokens	long	—	Prompt tokens used. Defaults to 0 when absent (some providers omit this).
completionTokens	long	—	Completion tokens used. Defaults to 0 when absent (e.g. embedding responses).
totalTokens	long	—	Total tokens used. Defaults to 0 when absent (some providers omit this).
promptTokensDetails	@Nullable PromptTokensDetails	null	Breakdown of tokens used in the prompt, including cached tokens served at the provider’s discounted cache-read rate. Absent when the provider does not return prompt-token details.

---

UserMessage

User message in the conversation.

Field	Type	Default	Description
content	UserContent	UserContent.TEXT	Message content as plain text or array of content parts (text, images, documents, audio).
name	@Nullable String	null	Optional name for the user.

---

WaitForBatchConfig

Configuration for polling a batch until terminal status.

All time values are in seconds as f64 so the struct bridges across FFI boundaries without requiring a Duration shim.

Field	Type	Default	Description
initialIntervalSecs	double	5	Initial interval between polls, in seconds.
maxIntervalSecs	double	60	Maximum interval between polls (backoff plateau), in seconds.
backoffMultiplier	float	1.5	Exponential backoff multiplier (e.g., 1.5 increases delay by 50% each poll).
timeoutSecs	@Nullable Double	null	Optional timeout in seconds — polling fails if this duration is exceeded.

---

Enums

Message

A chat message in a conversation.

Value	Description
SYSTEM	System — Fields (flattened into the tagged object): content: UserContent, name: String
USER	User — Fields (flattened into the tagged object): content: UserContent, name: String
ASSISTANT	Assistant — Fields (flattened into the tagged object): content: AssistantContent, name: String, toolCalls: List<ToolCall>, refusal: String, functionCall: FunctionCall, reasoningContent: String
TOOL	Tool — Fields (flattened into the tagged object): content: UserContent, toolCallId: String, name: String
DEVELOPER	Developer — Fields (flattened into the tagged object): content: String, name: String
FUNCTION	Deprecated legacy function-role message; retained for API compatibility. — Fields (flattened into the tagged object): content: String, name: String

---

UserContent

User message content as either plain text or a list of multimodal parts.

Value	Description
TEXT	Plain text content. — Fields: 0: String
PARTS	Array of content parts (text, images, documents, audio). — Fields: 0: List<ContentPart>

---

ContentPart

A single content part in a user message — text, image, document, or audio.

Value	Description
TEXT	Plain text. — Fields: text: String
IMAGE_URL	Image identified by URL (with optional detail level). — Fields: imageUrl: ImageUrl
DOCUMENT	Document file (PDF, CSV, etc.) as base64 or URL. — Fields: document: DocumentContent
INPUT_AUDIO	Audio input as base64. — Fields: inputAudio: AudioContent

---

ImageDetail

Image detail level controlling token cost and processing.

Value	Description
LOW	Low detail: scales image to 512x512, uses fewer tokens.
HIGH	High detail: processes up to 2x2 grid of tiles, higher token cost.
AUTO	Auto: model chooses low or high based on image dimensions.

---

AssistantContent

Content shape for assistant messages.

#[serde(untagged)] means providers returning a plain scalar string for the content field still deserialise correctly into AssistantContent.Text(_). Providers returning an array of typed parts (e.g. after an image-generation or audio-synthesis request) deserialise into AssistantContent.Parts(_).

Value	Description
TEXT	Plain text response (the common case for text-only models). — Fields: 0: String
PARTS	Structured parts — text, refusals, output images, output audio. — Fields: 0: List<AssistantPart>

---

AssistantPart

One part of a structured assistant response.

#[serde(tag = "type", rename_all = "snake_case")] matches OpenAI’s parts-spec discriminator ("type": "text", "type": "output_image", …).

Value	Description
TEXT	A text segment of the response. — Fields: text: String
REFUSAL	A refusal — the model declined to respond. — Fields: refusal: String
OUTPUT_IMAGE	An image produced by the model (e.g. gpt-image-1, Gemini Imagen). — Fields: imageUrl: ImageUrl
OUTPUT_AUDIO	Audio produced by the model (e.g. gpt-4o-audio-preview). — Fields: audio: AudioContent

---

ToolType

The type discriminator for tool/tool-call objects.

Per the OpenAI spec this is always "function". Using an enum enforces that constraint at the type level and rejects any other value on deserialization.

Value	Description
FUNCTION	Function

---

ToolChoice

Tool usage mode or a specific tool to call.

Value	Description
MODE	Predefined mode: auto, required, or none. — Fields: 0: ToolChoiceMode
SPECIFIC	Force a specific tool to be called. — Fields: 0: SpecificToolChoice

---

ToolChoiceMode

Tool choice mode.

Value	Description
AUTO	Model may or may not call tools; default behavior.
REQUIRED	Model must call at least one tool.
NONE	Model must not call any tools.

---

ResponseFormat

Wire format for the chat completions response_format field.

Provider mapping

• OpenAI (and OpenAI-compatible providers): emitted verbatim as {"type": "json_schema", "json_schema": {...}} per the chat-completions spec.

• Gemini / Vertex AI: translated to generationConfig.responseMimeType = "application/json" and generationConfig.responseSchema = <schema>. The name, description, and strict fields are dropped — Gemini’s structured-output API does not consume them.

• Anthropic: no native JSON mode. A system instruction is prepended asking the model to respond with valid JSON. strict is advisory only; callers should still validate the returned JSON if the schema is load-bearing.

Value	Description
TEXT	Plain text output (default).
JSON_OBJECT	Output must be valid JSON object (no schema validation).
JSON_SCHEMA	Output must conform to the specified JSON schema. — Fields: jsonSchema: JsonSchemaFormat

---

StopSequence

Stop sequence(s) that cause the model to stop generating.

Value	Description
SINGLE	Single stop sequence. — Fields: 0: String
MULTIPLE	Multiple stop sequences. — Fields: 0: List<String>

---

Modality

Output modality requested from the model.

Passed as modalities: ["text", "audio"] (OpenAI) or translated to generationConfig.responseModalities (Gemini / Vertex AI).

Value	Description
TEXT	Text output (the default for all providers).
AUDIO	Audio / speech output.
IMAGE	Image output (Gemini Imagen, gpt-image-1).

---

FinishReason

Why a choice stopped generating tokens.

Value	Description
STOP	Stop
LENGTH	Length
TOOL_CALLS	Tool calls
CONTENT_FILTER	Content filter
FUNCTION_CALL	Deprecated legacy finish reason; retained for API compatibility.
OTHER	Catch-all for unknown finish reasons returned by non-OpenAI providers. Note: this intentionally does not carry the original string (e.g. Other(String)). Using #[serde(other)] requires a unit variant, and switching to #[serde(untagged)] would change deserialization semantics for all variants. The original value can be recovered by inspecting the raw JSON if needed.

---

ReasoningEffort

Controls how much reasoning effort the model should use.

Value	Description
LOW	Low
MEDIUM	Medium
HIGH	High
MINIMAL	Minimal
MAX	Max

---

EmbeddingFormat

The format in which the embedding vectors are returned.

Value	Description
FLOAT	32-bit floating-point numbers (default).
BASE64	Base64-encoded string representation of the floats.

---

EmbeddingInput

Text, texts, or multimodal content to embed.

Value	Description
SINGLE	Single text string. — Fields: 0: String
MULTIPLE	Multiple text strings (batch embedding). — Fields: 0: List<String>
MULTIMODAL	Text and image parts for a single multimodal embedding. — Fields: 0: List<EmbeddingContentPart>

---

EmbeddingContentPart

A content part in a multimodal embedding input.

Value	Description
TEXT	Plain text. — Fields: text: String
IMAGE_URL	Image identified by a data URL or HTTP/HTTPS URL. — Fields: imageUrl: ImageUrl
IMAGE_BASE64	Image encoded as a complete data URL. — Fields: imageBase64: String

---

ModerationInput

Input to the moderation endpoint — a single string or multiple strings.

Value	Description
SINGLE	Single text string. — Fields: 0: String
MULTIPLE	Multiple text strings (batch moderation). — Fields: 0: List<String>

---

RerankDocument

A document to be reranked — either a plain string or an object with a text field.

Value	Description
TEXT	Plain text document content. — Fields: 0: String
OBJECT	Document with explicit text field (may include metadata). — Fields: text: String

---

OcrDocument

Document input for OCR — either a URL or inline base64 data.

Value	Description
URL	A publicly accessible document URL. — Fields: url: String
BASE64	Inline base64-encoded document data. — Fields: data: String, mediaType: String

---

FilePurpose

Purpose of an uploaded file.

Value	Description
ASSISTANTS	File for use with Assistants API.
BATCH	File for batch processing.
FINE_TUNE	File for fine-tuning.
VISION	File for vision/image tasks.

---

BatchStatus

Status of a batch job.

Value	Description
VALIDATING	Validating the input file.
FAILED	Job failed.
IN_PROGRESS	Job is running.
FINALIZING	Finalizing results.
COMPLETED	Job completed successfully.
EXPIRED	Job expired before completion.
CANCELLING	Job is being cancelled.
CANCELLED	Job has been cancelled.

---

AuthHeaderFormat

How the API key is sent in the HTTP request.

Value	Description
BEARER	Bearer token: Authorization: Bearer <key>
API_KEY	Custom header: e.g., X-Api-Key: <key> — Fields: 0: String
NONE	No authentication required.

---

StreamFormat

The streaming wire format a provider uses for its response stream.

Most providers use standard Server-Sent Events (SSE). AWS Bedrock uses a proprietary binary EventStream framing.

Deserialized from the streaming_format JSON field via serde.

Value	Description
SSE	Standard Server-Sent Events (text/event-stream).
AWS_EVENT_STREAM	AWS EventStream binary framing (application/vnd.amazon.eventstream).

---

AuthType

Auth scheme used by a provider.

Value	Description
BEARER	Standard Authorization: Bearer <key> header.
API_KEY	x-api-key: <key> header (also handles "header" and "x-api-key" aliases).
NONE	No authentication header required.
UNKNOWN	Unrecognised auth scheme — falls back to bearer.

---

RefreshOutcome

Result of a refresh_catalog call.

Value	Description
DISABLED	config.enabled was false; no network, filesystem, or overlay activity occurred.
FROM_CACHE	The on-disk cache was fresh (age < ttl_seconds); the overlay was installed from the cached file without a network request.
FETCHED	The catalog was fetched over the network, the cache file was (best-effort) refreshed, and the overlay was installed from the fetched catalog.

---

Enforcement

How budget limits are enforced.

Value	Description
HARD	Reject requests that would exceed the budget with LiterLlmError.BudgetExceeded.
SOFT	Allow requests through but emit a tracing.warn! when the budget is exceeded.

---

CacheBackend

Storage backend for the response cache.

Value	Description
MEMORY	In-memory LRU cache (default). No external dependencies.
OPEN_DAL	OpenDAL-backed storage. Supports 40+ backends (S3, Redis, GCS, local FS, etc.). — Fields: scheme: String, config: Map<String, String>

---

CircuitState

Observable state of a circuit breaker.

Value	Description
CLOSED	Requests flow through normally.
OPEN	All requests are rejected; the circuit is waiting for the backoff to elapse.
HALF_OPEN	One probe request is allowed through to test service health.

---

HealthStatus

The result of a single health probe.

Value	Description
HEALTHY	The probe succeeded; the upstream is reachable.
UNHEALTHY	The probe failed; the upstream may be down.

---

Errors

LiterLlmError

All errors that can occur when using liter-llm.

Variant	Description
AuthenticationException	status preserves the exact HTTP status code received (401 or 403).
RateLimitedException	rate limited: {message}
BadRequestException	status preserves the exact HTTP status code received (400, 405, 413, 422, …).
ContextWindowExceededException	context window exceeded: {message}
ContentPolicyException	content policy violation: {message}
NotFoundException	not found: {message}
ServerErrorException	status preserves the exact HTTP status code received (500, or other 5xx not covered by ServiceUnavailable).
ServiceUnavailableException	status preserves the exact HTTP status code received (502, 503, or 504).
TimeoutException	request timeout
StreamingException	A catch-all for errors that occur during streaming response processing. This variant covers multiple sub-conditions including UTF-8 decoding failures, CRC/checksum mismatches (AWS EventStream), JSON parse errors in individual SSE chunks, and buffer overflow conditions. The message field contains a human-readable description of the specific failure.
EndpointNotSupportedException	provider {provider} does not support {endpoint}
InvalidHeaderException	invalid header {name:?}: {reason}
SerializationException	serialization error: {0}
BudgetExceededException	budget exceeded: {message}
HookRejectedException	hook rejected: {message}
InternalErrorException	An internal logic error (e.g. unexpected Tower response variant). This should never surface in normal operation — if it does, it indicates a bug in the library.
OutboundForbiddenException	An outbound request was blocked by the active OutboundPolicy. Returned when register_custom_provider is called with a base_url that violates the policy (e.g. a private-range IP under DenyPrivate), or when the per-connection DNS resolver detects a forbidden address at connect time.
IdempotencyConflictException	A different request body was submitted for an existing Idempotency-Key. Per the OpenAI Idempotency-Key convention, once a key is used with a particular request body, subsequent requests using the same key must carry an identical body. A body mismatch is a hard error (not retryable). HTTP equivalent: 409 Conflict.
IdempotencyInFlightException	The same Idempotency-Key is already in-flight (another request with the same key is currently being processed). The caller should wait briefly and retry. The response is not yet available, and this request has been short-circuited to avoid running the operation twice. HTTP equivalent: 409 Conflict (retryable after a brief delay).

---

CatalogRefreshError

Errors from refresh_catalog and install_catalog_overlay_from_str.

On every variant, the overlay registry is left untouched: a previously installed overlay (or the embedded catalog, if none was ever installed) remains active. This is the air-gap-safety contract — a failed refresh never degrades pricing/model-info availability.

Variant	Description
DisabledException	Runtime catalog refresh was not enabled. refresh_catalog itself never returns this — it returns Ok(RefreshOutcome.Disabled) instead — but the variant is part of the public error surface for callers that want to treat “disabled” as a hard error.
InsecureUrlException	source_url did not use the https scheme, or failed to parse as a URL at all. There is no host allowlist: the URL is user-configurable for self-hosted catalog mirrors, so only the scheme is enforced.
FetchException	The network fetch failed, timed out, or returned a non-success status.
ParseException	The fetched or cached catalog JSON failed to parse.
CacheException	A cache file read failed on an otherwise-fresh cache file. Cache write failures are best-effort and never surface as this error (see refresh_catalog).

---