Configuration

Snippets marked compile-only

Snippets with the <!-- snippet:compile-only --> HTML comment are extracted from CI test fixtures, kept in sync with the API at every build, and may include test scaffolding (assertions, fixture setup). Strip the scaffolding when copying into your application.

Liter-llm reads configuration from three sources, applied in priority order (lower numbers win):

1. Constructor arguments — passed directly to the client factory (create_client(...) / ClientConfigBuilder).
2. JSON string — create_client_from_json(json) for bindings without a builder.
3. TOML file — liter-llm.toml, auto-discovered by walking up from the current working directory.

The Rust core exposes the same surface to every language binding via the C FFI. Methods like chat, embed, etc. are called on the client instance returned by the factory; there is no top-level LlmClient class to subclass or pre-configure in the bindings.

TOML file

Place a liter-llm.toml file in your project directory. FileConfig::discover() (Rust) and the proxy server walk up the directory tree looking for it.

api_key = "sk-..."
base_url = "https://api.openai.com/v1"
model_hint = "openai"
timeout_secs = 120
max_retries = 5
cooldown_secs = 30
health_check_secs = 60
cost_tracking = true
tracing = true

[cache]
max_entries = 512
ttl_seconds = 600
backend = "memory"

[budget]
global_limit = 50.0
enforcement = "hard"

[budget.model_limits]
"openai/gpt-4o" = 25.0

[rate_limit]
rpm = 60
tpm = 100000
window_seconds = 60

[in_flight_limit]
max_in_flight = 8

[[providers]]
name = "my-provider"
base_url = "https://my-llm.example.com/v1"
auth_header = "Bearer"
model_prefixes = ["my-provider/"]

Top-level fields

Field	Type	Description
api_key	string	Provider API key. Wrapped in SecretString internally.
base_url	string	Override the provider’s base URL.
model_hint	string	Pre-resolve a provider (e.g. "openai"); skips prefix lookup.
timeout_secs	int	Per-request timeout in seconds (default: 60).
max_retries	int	Retries on 429/5xx with exponential backoff (default: 3).
cooldown_secs	int	Circuit-breaker cooldown after transient errors.
health_check_secs	int	Interval for background health checks.
cost_tracking	bool	Enable per-request cost calculation.
tracing	bool	Emit tracing spans for each request (see Observability).
extra_headers	map	Additional headers attached to every outgoing request.

[cache]

Field	Type	Description
max_entries	int	Maximum cached responses (default: 256).
ttl_seconds	int	Time-to-live for each entry in seconds (default: 300).
backend	string	"memory" (default, in-process LRU) or an OpenDAL scheme ("redis", "s3", "fs", "gcs", …). Prefix with opendal: to select any OpenDAL service explicitly, e.g. "opendal:memory".
backend_config	map	OpenDAL backend-specific key-value config.

The backend and backend_config fields are only honored when liter-llm is compiled with the opendal-cache feature.

[budget]

Field	Type	Description
global_limit	float	Maximum total spend in USD across all models.
model_limits	map	Per-model spend limits (model name → USD).
enforcement	string	"hard" (reject over-budget) or "soft" (warn only).

[rate_limit]

Field	Type	Description
rpm	int	Maximum requests per window.
tpm	int	Maximum tokens per window.
window_seconds	int	Window duration in seconds (default: 60).

[in_flight_limit]

Field	Type	Description
max_in_flight	int	Maximum outstanding provider requests. Must be greater than 0.

[[providers]]

Array of custom provider definitions. Each entry contains:

Field	Type	Description
name	string	Unique provider name.
base_url	string	Provider’s API base URL.
auth_header	string	"Bearer" (default), "none", or a header name.
model_prefixes	string[]	Model name prefixes that route to this provider.

Construction

The constructors below match the actual binding surface. Other bindings (Go, Java, Kotlin Android, C#, Ruby, PHP, Elixir, Dart, Swift, Zig, WebAssembly) expose the same scalar arguments through their generated wrappers; JVM Kotlin applications use the Java binding from Kotlin. Use the matching API page under Reference for exact language signatures.

Rust

use liter_llm::{ClientConfigBuilder, DefaultClient};
use std::time::Duration;

let config = ClientConfigBuilder::new("sk-...")
    .base_url("https://api.openai.com/v1")
    .timeout(Duration::from_secs(120))
    .max_retries(5)
    .build();

let client = DefaultClient::new(config, Some("openai"))?;

Load from a liter-llm.toml:

use liter_llm::{FileConfig, DefaultClient};

if let Some(file) = FileConfig::discover()? {
    let config = file.into_builder().build();
    let client = DefaultClient::new(config, None)?;
}

ManagedClient::new(config, model_hint) is available with the tower feature and wires the full middleware stack (cache, budget, rate limit, cooldown, health, hooks, tracing) from the same ClientConfig.

Python

from liter_llm import create_client

client = create_client(
    api_key="sk-...",
    base_url="https://api.openai.com/v1",
    timeout_secs=120,
    max_retries=5,
    model_hint="openai",
)

response = await client.chat(...)

For middleware and custom providers, use create_client_from_json (see JSON configuration):

from liter_llm import create_client_from_json
import json

client = create_client_from_json(json.dumps({
    "api_key": "sk-...",
    "timeout_secs": 120,
    "cache": {"max_entries": 256, "ttl_seconds": 300},
    "budget": {"global_limit": 50.0, "enforcement": "hard"},
}))

TypeScript

import { createClient } from "@xberg-io/liter-llm";

const client = createClient(
  process.env.OPENAI_API_KEY!,
  "https://api.openai.com/v1",
  120,    // timeoutSecs
  5,      // maxRetries
  "openai", // modelHint
);

const response = await client.chat({ /* ... */ });

For middleware and custom providers, pass a JSON config to createClientFromJson (see JSON configuration):

import { createClientFromJson } from "@xberg-io/liter-llm";

const client = createClientFromJson(JSON.stringify({
  api_key: process.env.OPENAI_API_KEY,
  timeout_secs: 120,
  cache: { max_entries: 256, ttl_seconds: 300 },
  budget: { global_limit: 50.0, enforcement: "hard" },
}));

JSON configuration

create_client_from_json (createClientFromJson in TypeScript) accepts the same snake_case keys as liter-llm.toml, as a JSON object. Unknown keys are rejected.

Key	Effect
api_key, base_url, model_hint, timeout_secs, max_retries, extra_headers	Client settings, applied to the underlying HTTP client.
cache, budget, cooldown_secs, rate_limit, in_flight_limit, health_check_secs, cost_tracking, tracing	Middleware. When any is set, chat, chat_stream, embed, list_models, image_generate, speech, transcribe, moderate, rerank, search and ocr run through the managed Tower stack. File, batch and response operations bypass it.
providers	Each entry is registered process-wide as a custom provider (auth_header: "Bearer"/"Authorization"/unset sends a bearer token, "none" sends no auth header, any other value names a header that carries the raw key).

Hooks are not expressible in JSON; build a ClientConfig in Rust to use them. A build compiled without the tower feature rejects the middleware keys with an error instead of ignoring them.

Per-language snippets

Python

Client configured with a custom base URL routes all requests to that endpoint

Python

import asyncio
import os
from liter_llm import create_client
from liter_llm._internal_bindings import ChatCompletionRequest

async def main() -> None:
    client = create_client(api_key=os.environ["API_KEY"])
    req = ChatCompletionRequest.from_json("{\"messages\":[{\"content\":\"Hello\",\"role\":\"user\"}],\"model\":\"local-model\"}")
    result = await client.chat(req)
    print(result.choices)
    print(result.choices[0].message.content)
    print(result.choices[0].finish_reason)
    print(result.model)

asyncio.run(main())

TypeScript

Client configured with a custom base URL routes all requests to that endpoint

TypeScript

import { createClient } from "@xberg-io/liter-llm";
async function main() {
  const client = createClient("your-api-key");
  const result = await client.chat({ messages: [{ content: "Hello", role: "user" }], model: "local-model" });
  console.log(result.choices);
  console.log(result.choices?.[0]?.message?.content);
  console.log(result.choices?.[0]?.finishReason);
  console.log(result.model);
}

void main();

Rust

Client configured with a custom base URL routes all requests to that endpoint

Rust

use liter_llm::BatchClient;
use liter_llm::FileClient;
use liter_llm::LlmClient;
use liter_llm::ResponseClient;

#[tokio::main]
async fn main() {
    let req_json: serde_json::Value = serde_json::from_str(r#"{"messages":[{"content":"Hello","role":"user"}],"model":"local-model"}"#).unwrap();
    let req = serde_json::from_value(req_json).unwrap();
    let client = liter_llm::create_client(std::env::var("API_KEY").expect("API_KEY must be set"), None, None, None, None).unwrap();
    let result = client.chat(req).await.expect("call failed");
    println!("{:?}", result.choices);
    println!("{:?}", result.choices[0].message.content);
    println!("{:?}", result.choices[0].finish_reason);
    println!("{:?}", result.model);
}

Go

Client configured with a custom base URL routes all requests to that endpoint

Go

package main

import (
  "fmt"
  pkg "github.com/xberg-io/liter-llm/packages/go/v2"
)

func main() {
  req := pkg.ChatCompletionRequest{
    Model: `local-model`,
    Messages: []pkg.Message{
      pkg.Message{Role: `user`, User: &pkg.UserMessage{
        Content: pkg.UserContent(`"Hello"`),
      }},
    },
  }
  client, clientErr := pkg.CreateClient("your-api-key", nil, nil, nil, nil)
  if clientErr != nil {
    panic(clientErr)
  }
  defer client.Free()
  result, err := client.Chat(req)
  if err != nil {
    panic(err)
  }
  fmt.Printf("%+v\n", result.Choices)
  fmt.Printf("%+v\n", result.Choices[0].Message.Content)
  fmt.Printf("%+v\n", result.Choices[0].FinishReason)
  fmt.Printf("%+v\n", result.Model)
}

Java

Client configured with a custom base URL routes all requests to that endpoint

Java

import io.xberg.literllm.*;

public final class Example {
    public static void main(String[] args) throws Exception {
        var reqJson = "{\"messages\":[{\"content\":\"Hello\",\"role\":\"user\"}],\"model\":\"local-model\"}";
        var req = JsonUtil.fromJson(reqJson, ChatCompletionRequest.class);
        var apiKey = System.getenv("API_KEY");
        if (apiKey == null || apiKey.isEmpty()) throw new IllegalStateException("API_KEY must be set");
        try (var client = LiterLlm.createClient(apiKey, null, null, null, null)) {
        var result = client.chat(req);
        System.out.println(result.choices());
        System.out.println(result.choices().get(0).message().content());
        System.out.println(result.choices().get(0).finishReason());
        System.out.println(result.model());
        }
    }
}

C#

Client configured with a custom base URL routes all requests to that endpoint

C#

using System;
using System.Text.Json;
using LiterLlm;

var ConfigOptions = new JsonSerializerOptions { PropertyNameCaseInsensitive = true };
var apiKey = Environment.GetEnvironmentVariable("API_KEY") ??
  throw new InvalidOperationException("API_KEY must be set");
using var client = LiterLlmConverter.CreateClient(apiKey, null, null, null, null);
var result = await client.ChatAsync(new ChatCompletionRequest { Messages = new List<Message>() { JsonSerializer.Deserialize<Message>("{\"content\":\"Hello\",\"role\":\"user\"}", ConfigOptions)! }, Model = "local-model" });
Console.WriteLine(result.Choices);
Console.WriteLine(result.Choices[0].Message.Content);
Console.WriteLine(result.Choices[0].FinishReason);
Console.WriteLine(result.Model);

Ruby

Client configured with a custom base URL routes all requests to that endpoint

Ruby

require "liter_llm"
api_key = ENV.fetch("API_KEY")
client = LiterLlm.create_client(api_key)
result = client.chat_async(LiterLlm::ChatCompletionRequest.new(messages: [{ 'content' => 'Hello', 'role' => 'user' }], model: 'local-model'))
puts result.choices.inspect
puts result.choices[0].message.content.inspect
puts result.choices[0].finish_reason.inspect
puts result.model.inspect

PHP

Client configured with a custom base URL routes all requests to that endpoint

PHP

<?php

declare(strict_types=1);

require_once __DIR__ . '/vendor/autoload.php';

use Liter\Llm\LiterLlm;
use Liter\Llm\ChatCompletionRequest;
$req = \Liter\Llm\ChatCompletionRequest::from_json(json_encode(["messages" => [["content" => "Hello", "role" => "user"]], "model" => "local-model"]));
$apiKey = getenv('API_KEY');
if ($apiKey === false || $apiKey === '') {
    throw new RuntimeException('API_KEY must be set');
}
$client = LiterLlm::createClient($apiKey);
$result = $client->chat($req);
var_dump($result->getChoices());
var_dump($result->getChoices()[0]->getMessage()->getContent());
var_dump($result->getChoices()[0]->finishReason);
var_dump($result->model);

Elixir

Client configured with a custom base URL routes all requests to that endpoint

Elixir

api_key = System.fetch_env!("API_KEY")
{:ok, client} = LiterLlm.create_client(api_key)
result = LiterLlm.defaultclient_chat_async(client, "{\"messages\":[{\"content\":\"Hello\",\"role\":\"user\"}],\"model\":\"local-model\"}")
IO.inspect(result.choices)
IO.inspect(Enum.at(result.choices, 0).message.content)
IO.inspect(Enum.at(result.choices, 0).finish_reason)
IO.inspect(result.model)

WASM

Client configured with a custom base URL routes all requests to that endpoint

WebAssembly

import { WasmChatCompletionRequest, createClient } from "@xberg-io/liter-llm-wasm";
async function main() {
  const req: WasmChatCompletionRequest = (() => { const _u0 = WasmChatCompletionRequest.default(); _u0.messages = [{ content: "Hello", role: "user" }]; _u0.model = "local-model"; return _u0; })();
  const client = createClient("your-api-key");
  try {
    const result = await client.chat(req);
    console.log(result.choices);
    console.log(result.choices[0].message.content);
    console.log(result.choices[0].finishReason);
    console.log(result.model);
  } finally {
    client.free();
  }
}

void main();

Scalar options

Option	Type	Default	Description
api_key	string	required	Provider API key. Wrapped in SecretString internally.
base_url	string	from registry	Override the provider’s base URL.
model_hint	string	none	Pre-resolve a provider at construction (e.g. "openai").
timeout_secs	int	60	Request timeout in seconds.
max_retries	int	3	Retries on 429/5xx responses with exponential backoff.

API key environment variables

API keys passed to the constructor are wrapped in secrecy::SecretString. They are never logged, serialized, or included in error messages. If the constructor is given an empty api_key, the builder reads the standard environment variable for the active provider:

Provider	Environment variable
OpenAI	OPENAI_API_KEY
Anthropic	ANTHROPIC_API_KEY
Google (Gemini)	GEMINI_API_KEY
Groq	GROQ_API_KEY
Mistral	MISTRAL_API_KEY
Cohere	CO_API_KEY

AWS Bedrock

Bedrock takes no API key here. Its credentials are read on every request instead: a Bedrock API key from AWS_BEARER_TOKEN_BEDROCK, or, with the bedrock feature, an access-key pair from AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY. See Authentication.

Custom base URLs

Override base_url to point at a local inference server or a corporate proxy:

# Ollama running locally
base_url = "http://localhost:11434/v1"

let config = ClientConfigBuilder::new("unused")
    .base_url("http://localhost:11434/v1")
    .build();

Cache

The response cache is wired through the Tower middleware stack. In Rust, attach CacheLayer directly; in other languages, configure it via TOML (and use ManagedClient indirectly through the bindings’ high-level clients).

TOML

[cache]
max_entries = 256
ttl_seconds = 300
backend = "memory"

Rust

use std::time::Duration;
use liter_llm::tower::cache::{CacheLayer, CacheConfig, CacheBackend};

let config = CacheConfig {
    max_entries: 256,
    ttl: Duration::from_secs(300),
    backend: CacheBackend::Memory,
};
let layer = CacheLayer::new(config);

OpenDAL-backed cache

With the opendal-cache feature, the cache backend can be any OpenDAL service — Redis, S3, GCS, Azure Blob, local filesystem, and more.

TOML

[cache]
ttl_seconds = 3600
backend = "redis"

[cache.backend_config]
endpoint = "redis://localhost:6379"

Rust

use std::collections::HashMap;
use std::time::Duration;
use liter_llm::tower::cache::{CacheConfig, CacheBackend};

let mut backend_config = HashMap::new();
backend_config.insert("endpoint".into(), "redis://localhost:6379".into());

let config = CacheConfig {
    max_entries: 1024,
    ttl: Duration::from_secs(3600),
    backend: CacheBackend::OpenDal {
        scheme: "redis".into(),
        config: backend_config,
    },
};

Budget

Track and enforce spending limits per model and globally. Costs are recomputed after every successful response using liter_llm::cost::completion_cost.

TOML

[budget]
global_limit = 10.0
enforcement = "hard"

[budget.model_limits]
"openai/gpt-4o" = 5.0

Rust

use std::collections::HashMap;
use std::sync::Arc;
use liter_llm::tower::budget::{BudgetLayer, BudgetConfig, BudgetState, Enforcement};

let mut model_limits = HashMap::new();
model_limits.insert("openai/gpt-4o".into(), 5.0);

let config = BudgetConfig {
    global_limit: Some(10.0),
    model_limits,
    enforcement: Enforcement::Hard,
};
let state = Arc::new(BudgetState::new());
let layer = BudgetLayer::new(config, state);

Enforcement::Hard rejects requests with LiterLlmError::BudgetExceeded once the limit is reached. Enforcement::Soft emits a tracing::warn! but allows the request through.

Rate limiting

Per-model RPM (requests per minute) and TPM (tokens per minute) limits using a fixed window.

TOML

[rate_limit]
rpm = 60
tpm = 100000
window_seconds = 60

Rust

use std::time::Duration;
use liter_llm::tower::rate_limit::{ModelRateLimitLayer, RateLimitConfig};

let config = RateLimitConfig {
    rpm: Some(60),
    tpm: Some(100_000),
    window: Duration::from_secs(60),
};
let layer = ModelRateLimitLayer::new(config);

In-flight concurrency limiting

Bound simultaneous provider requests across all operations that share a managed client. Requests wait for a permit; cached responses do not consume one.

TOML

[in_flight_limit]
max_in_flight = 8

Rust

use liter_llm::InFlightLimitConfig;

let config = InFlightLimitConfig {
    max_in_flight: Some(8),
};
let builder = builder.in_flight_limit(config);

Hooks

Hooks are implemented via the Rust LlmHook trait and are wired into the Tower stack through HooksLayer. They are not currently exposed through the language bindings.

use std::future::Future;
use std::pin::Pin;
use std::sync::Arc;
use liter_llm::error::{LiterLlmError, Result};
use liter_llm::tower::hooks::{HooksLayer, LlmHook};
use liter_llm::tower::types::{LlmRequest, LlmResponse};

struct LoggingHook;

impl LlmHook for LoggingHook {
    fn on_request(&self, _req: &LlmRequest)
        -> Pin<Box<dyn Future<Output = Result<()>> + Send + '_>>
    {
        Box::pin(async {
            tracing::info!("dispatching request");
            Ok(())
        })
    }

    fn on_response(&self, _req: &LlmRequest, _resp: &LlmResponse)
        -> Pin<Box<dyn Future<Output = ()> + Send + '_>>
    {
        Box::pin(async {
            tracing::info!("response received");
        })
    }

    fn on_error(&self, _req: &LlmRequest, err: &LiterLlmError)
        -> Pin<Box<dyn Future<Output = ()> + Send + '_>>
    {
        let message = err.to_string();
        Box::pin(async move {
            tracing::error!(error = %message, "request failed");
        })
    }
}

let hooks: Vec<Arc<dyn LlmHook>> = vec![Arc::new(LoggingHook)];
let layer = HooksLayer::new(hooks);

Returning Err from on_request short-circuits the service chain, so hooks can implement guardrails (content filtering, budget gating, etc.).

Custom providers

Custom providers are registered in a process-wide global registry. Once registered, any request whose model name matches one of the provider’s model_prefixes is routed there.

Rust

use liter_llm::{register_custom_provider, CustomProviderConfig, AuthHeaderFormat};

register_custom_provider(CustomProviderConfig {
    name: "my-provider".into(),
    base_url: "https://my-llm.example.com/v1".into(),
    auth_header: AuthHeaderFormat::Bearer,
    model_prefixes: vec!["my-provider/".into()],
})?;

Unregister with unregister_custom_provider("my-provider") (returns Result<bool>: Ok(true) if a provider with that name existed).

Python

from liter_llm import (
    register_custom_provider,
    unregister_custom_provider,
    CustomProviderConfig,
)
from liter_llm._internal_bindings import AuthHeaderFormat

register_custom_provider(CustomProviderConfig(
    name="my-provider",
    base_url="https://my-llm.example.com/v1",
    auth_header=AuthHeaderFormat.Bearer,
    model_prefixes=["my-provider/"],
))

unregister_custom_provider("my-provider")

TOML

liter-llm.toml can declare custom providers, but loading the file with FileConfig::into_builder() does not register them — it only maps client-level settings onto ClientConfigBuilder. (create_client_from_json does register its providers entries.) Read the entries back with FileConfig::providers() and register each one with register_custom_provider:

[[providers]]
name = "my-provider"
base_url = "https://my-llm.example.com/v1"
auth_header = "Bearer"
model_prefixes = ["my-provider/"]

use liter_llm::{register_custom_provider, FileConfig};

if let Some(file) = FileConfig::discover()? {
    for provider in file.providers() {
        register_custom_provider(provider.to_custom_provider_config())?;
    }
}

AuthHeaderFormat has three variants: Bearer (sends Authorization: Bearer <key>), ApiKey(header_name) (sends a custom header), and None (no auth header). FileProviderConfig::auth_header (from TOML or JSON) is a string; to_custom_provider_config() maps unset, "Bearer" and "Authorization" to Bearer, "none" to None, and any other value to ApiKey with that header name.

Tracing

The tracing reference has moved to Observability. That page covers span attributes, OTEL exporter setup, cost tracking, and Tower layer composition.