Proxy Configuration

The proxy loads a single TOML file named liter-llm-proxy.toml. Pass --config <path> to the liter-llm api command or place the file in the current working directory (or any parent) and the server will discover it automatically.

Every string value supports ${VAR_NAME} environment variable interpolation. Unknown fields are rejected at parse time, so typos fail fast instead of silently being ignored.

A minimal file that exposes one model:

[general]
master_key = "${LITER_LLM_MASTER_KEY}"

[[models]]
name = "gpt-4o"
provider_model = "openai/gpt-4o"
api_key = "${OPENAI_API_KEY}"

Discovery and overrides

The proxy resolves config in this order, later values winning:

1. Defaults from each struct’s Default impl.
2. Either auto-discovery of liter-llm-proxy.toml in the working directory walked upward to the filesystem root, or an explicit --config <path> file (mutually exclusive — --config disables auto-discovery).
3. CLI flags (--host, --port, --master-key; --watch, --etcd-endpoint, and --etcd-key select hot-reload mode).
4. Environment variables read during ${VAR} interpolation.

See Proxy Server > Command-line flags for the flag list.

[server]

HTTP listener settings.

Field	Type	Default	Description
host	string	"0.0.0.0"	Bind address.
port	u16	4000	Bind port.
request_timeout_secs	u64	600	Upper bound on request duration before the proxy returns 504.
body_limit_bytes	usize	10485760 (10 MiB)	Maximum request body size. Requests larger than this return 413.
cors_origins	list<string>	[]	Allowed CORS origins. Empty disables CORS. Set explicit origins for browser clients.

[server]
host = "0.0.0.0"
port = 4000
request_timeout_secs = 600
body_limit_bytes = 10_485_760
cors_origins = ["https://app.example.com", "https://admin.example.com"]

[general]

Proxy-wide behaviour.

Field	Type	Default	Description
master_key	string?	none	Superuser API key. Requests with this Bearer token bypass virtual-key restrictions. Can also be set via LITER_LLM_MASTER_KEY.
default_timeout_secs	u64	120	Default per-request timeout used when a model does not set its own.
max_retries	u32	3	Retry attempts per upstream request. Only 429 and 5xx trigger a retry.
enable_cost_tracking	bool	false	Record gen_ai.usage.cost on every response using embedded pricing data.
enable_tracing	bool	false	Emit OpenTelemetry spans with GenAI semantic conventions. Set to true when exporting to an OTEL collector.

[general]
master_key = "${LITER_LLM_MASTER_KEY}"
default_timeout_secs = 120
max_retries = 3
enable_cost_tracking = true
enable_tracing = true

[security]

Outbound policy for custom provider and model endpoint URLs. The proxy default is deny_private, which blocks private, loopback, link-local, CGNAT, and cloud-metadata targets. The policy is applied before custom providers are registered or requests are accepted.

Field	Type	Default	Description
outbound_policy	string	"deny_private"	One of deny_private, allowlist, or off. Use off only when every configured endpoint is trusted.
outbound_allowlist	list<string>	[]	Allowed origins when outbound_policy = "allowlist". Scheme, host, and port define the origin; paths are ignored.

liter-llm-proxy.toml

[security]
outbound_policy = "deny_private"

# Use allowlist mode to restrict outbound requests to specific provider origins.
# outbound_policy = "allowlist"
# outbound_allowlist = ["https://api.openai.com", "https://api.anthropic.com"]

[[models]]

Named model entries. A single model name may appear multiple times to define an active-active load-balanced pool.

Field	Type	Default	Description
name	string	required	Alias clients send in the model field.
provider_model	string	required	Fully-qualified provider model, like openai/gpt-4o.
api_key	string?	none	Provider API key. Falls back to the environment variable the provider crate expects.
base_url	string?	none	Override the provider endpoint.
timeout_secs	u64?	general.default_timeout_secs	Per-model timeout.
fallbacks	list<string>	[]	Named models to try in order when the primary returns a transient error.

[[models]]
name = "gpt-4o"
provider_model = "openai/gpt-4o"
api_key = "${OPENAI_API_KEY}"
timeout_secs = 60
fallbacks = ["claude-sonnet", "llama3-groq"]

[[models]]
name = "claude-sonnet"
provider_model = "anthropic/claude-sonnet-4-20250514"
api_key = "${ANTHROPIC_API_KEY}"

[[models]]
name = "llama3-groq"
provider_model = "groq/llama3-70b-8192"
api_key = "${GROQ_API_KEY}"

[[aliases]]

Glob-pattern credential overrides. Aliases apply to models that match the pattern and are not already defined as a [[models]] entry. Useful when you want a single Anthropic key to cover every Anthropic model without listing them individually.

Field	Type	Default	Description
pattern	string	required	Glob pattern such as anthropic/* or openai/gpt-4*.
api_key	string?	none	Credential override for matching models.
base_url	string?	none	Endpoint override for matching models.

# Any model matching "anthropic/*" uses the shared Anthropic key.
[[aliases]]
pattern = "anthropic/*"
api_key = "${ANTHROPIC_API_KEY}"

# Route all OpenAI models through an Azure deployment.
[[aliases]]
pattern = "openai/*"
api_key = "${AZURE_OPENAI_KEY}"
base_url = "https://my-azure.openai.azure.com"

[[keys]]

Virtual API keys. Each key is a Bearer token with its own model allowlist, rate limit, and spend cap. The master key bypasses all of these.

Field	Type	Default	Description
key	string	required	The Bearer token clients present.
description	string?	none	Free-text label surfaced in logs and the admin API.
models	list<string>	[]	Allowed model names. Empty means all models are allowed.
rpm	u32?	none	Not yet enforced. Parsed into VirtualKeyConfig but no per-key rate-limit layer reads it; only the global [rate_limit] table is wired to ModelRateLimitLayer. Setting this silently does nothing.
tpm	u64?	none	Not yet enforced. Same gap as rpm — no per-key layer consumes it.
budget_limit	f64?	none	Not yet enforced. Mapped into ResolvedKey.monthly_budget but nothing reads that field to reject requests. Only the global [budget] table is enforced via BudgetLayer. Requests are not rejected with 402 when a key’s lifetime spend exceeds this value.

[[keys]]
key = "vk-team-frontend"
description = "Frontend team, chat-only, capped spend"
models = ["gpt-4o", "claude-sonnet"]
rpm = 60
tpm = 200_000
budget_limit = 100.0

[[keys]]
key = "vk-batch-worker"
description = "Overnight batch jobs, unrestricted model access"
rpm = 10
budget_limit = 500.0

Provider credentials can also be scoped to a virtual key. The proxy rotates among a key’s [[keys.provider_credentials]] entries on 429 and 5xx responses.

[[keys]]
key = "vk-prod"
models = ["gpt-4o"]

[[keys.provider_credentials]]
provider = "openai"
id = "primary"
api_key = "${OPENAI_API_KEY_PRIMARY}"
model_allowlist = ["gpt-4o"]

[[keys.provider_credentials]]
provider = "openai"
id = "backup"
api_key = "${OPENAI_API_KEY_BACKUP}"
model_allowlist = ["gpt-4o"]

[mcp]

Authentication context for liter-llm mcp --transport stdio. HTTP MCP ignores this section because every /mcp request is authenticated with Authorization: Bearer <key>.

Field	Type	Default	Description
stdio_key_id	string?	none	Bind stdio MCP calls to an existing [[keys]].key virtual key.
stdio_trust_local	bool	false	Treat the local stdio process as master access. Use only for trusted local clients.

At least one stdio mode must be configured. Prefer stdio_key_id for policy enforcement; without stdio_key_id or stdio_trust_local = true, the stdio MCP server refuses to start.

[rate_limit]

Global request-per-minute and token-per-minute caps applied on top of per-key limits. Omit the table to disable global limiting.

Field	Type	Default	Description
rpm	u32?	none	Global requests-per-minute across all keys.
tpm	u64?	none	Global tokens-per-minute across all keys.

[rate_limit]
rpm = 600
tpm = 1_000_000

[budget]

Aggregate spend enforcement. When enforcement = "hard", requests that would cross the limit are rejected with 402. Under "soft", they are logged and passed through.

Field	Type	Default	Description
global_limit	f64?	none	Total lifetime spend cap in USD.
model_limits	map<string, f64>	{}	Per-model spend caps keyed by provider/model.
enforcement	"hard" or "soft"	"hard"	Whether to reject or log over-budget requests.

[budget]
global_limit = 1000.0
enforcement = "hard"  # or "soft" to log but allow through

[budget.model_limits]
"openai/gpt-4o" = 500.0
"anthropic/claude-opus-4-20250514" = 200.0

[cache]

Response cache for non-streaming completions and embeddings. Keys include the model name, request body, and any relevant headers.

Field	Type	Default	Description
max_entries	usize?	none	In-memory LRU capacity. Required for the memory backend.
ttl_seconds	u64?	none	Entry lifetime. Entries are evicted after this many seconds.
backend	string	"memory"	Backend identifier: memory, or any OpenDAL scheme (redis, s3, fs, gcs, azblob, and more).
backend_config	map<string, string>	{}	Backend-specific key/value options. See the OpenDAL docs for each scheme.

# In-memory cache (default).
[cache]
max_entries = 4096
ttl_seconds = 900
backend = "memory"

# Redis cache via OpenDAL.
[cache]
max_entries = 10_000
ttl_seconds = 3600
backend = "redis"

[cache.backend_config]
endpoint = "redis://cache.internal:6379"

[files]

Storage backend for the /v1/files endpoints.

Field	Type	Default	Description
backend	string	"memory"	Backend identifier. memory is volatile; use s3, gcs, azblob, or fs for persistence.
prefix	string	"liter-llm-files/"	Object key prefix under the backend’s root.
backend_config	map<string, string>	{}	Backend-specific options (bucket, region, credentials).

# In-memory (default). Files are lost on restart.
[files]
backend = "memory"

# S3-backed file store.
[files]
backend = "s3"
prefix = "liter-llm-files/"

[files.backend_config]
bucket = "my-llm-files"
region = "us-west-2"

[health]

Periodic upstream probes. When interval_secs is set, the proxy installs HealthCheckLayer on that interval to mark failing providers unhealthy so the fallback layer can skip them.

Field	Type	Default	Description
interval_secs	u64?	none	Probe interval. Disabled when omitted.
probe_model	string?	none	Not yet enforced. Parsed into HealthConfig but HealthCheckLayer::new only takes interval_secs — the configured model name is never read, so it does not select what gets probed.

[health]
interval_secs = 30
probe_model = "openai/gpt-4o-mini"

[routing]

Routing strategy applied to every multi-deployment [[models]] group — a model name declared by more than one [[models]] entry. Has no effect on single-deployment models or on fallbacks resolution (see Fallback and Routing). Omit the table to keep the default: round-robin.

Field	Type	Default	Description
strategy	"round_robin" | "fallback" | "latency_based" | "cost_based" | "weighted_random" | "semantic"	"round_robin" (table omitted)	Selects the RoutingStrategy. An unrecognised value is rejected at parse time.
weights	list<f64>	—	Required when strategy = "weighted_random". One entry per deployment, in [[models]] declaration order; a mismatched length is rejected when the router is built.
classifier	table	—	Required when strategy = "semantic". See below.

[routing]
strategy = "weighted_random"
weights = [3.0, 2.0, 1.0]

[routing.classifier]

Required when strategy = "semantic". kind selects which classifier drives RoutingStrategy::Semantic.

kind = "keyword" — regex-rule routing; the first matching rule wins.

Field	Type	Default	Description
rules	list<{ pattern: string, model: string }>	required	Evaluated in order.

[routing.classifier]
kind = "keyword"

[[routing.classifier.rules]]
pattern = "(?i)sql|database"
model = "gpt-4o"

kind = "embedding" — cosine-similarity routing against precomputed intent-prototype embeddings.

Field	Type	Default	Description
embedding_model	string	required	Model used to embed the live request prompt, e.g. "openai/text-embedding-3-small".
api_key	string?	none	Credential for the embedding call. Falls back to the provider’s environment variable.
base_url	string?	none	Endpoint override for the embedding call.
threshold	f64	required	Minimum cosine similarity in [0, 1] for the classifier to commit instead of deferring.
prototypes	list<{ name: string, model: string, embedding: list<f64> }>	required	Intent prototypes. embedding must be precomputed offline — the proxy does not embed prototype text for you.

[routing.classifier]
kind = "embedding"
embedding_model = "openai/text-embedding-3-small"
threshold = 0.75

[[routing.classifier.prototypes]]
name = "coding"
model = "gpt-4o"
# Precomputed offline; truncated here for brevity — paste the full vector.
embedding = [0.0123, -0.0456, 0.0789]

An invalid keyword regex, or an embedding classifier client that fails to construct, is rejected at proxy startup with a clear error — never a silent fallback to round-robin.

[cooldown]

Circuit-breaker duration. After a provider returns a transient error, the proxy refuses to send it traffic for duration_secs seconds and routes to fallbacks instead.

Field	Type	Default	Description
duration_secs	u64	required	Cooldown window in seconds.

[cooldown]
duration_secs = 60

Environment variable interpolation

Any ${VAR_NAME} pattern inside a string value is replaced with the environment variable’s value before parsing. Unknown variables expand to an empty string, which is usually what you want for Option<String> fields. The interpolation runs on the raw TOML source, so nested tables and array values are expanded uniformly.

# Any ${VAR} pattern in a string value is replaced with the env var at load time.
# Unknown variables expand to an empty string.
[general]
master_key = "${LITER_LLM_MASTER_KEY}"

[[models]]
name = "gpt-4o"
provider_model = "openai/gpt-4o"
api_key = "${OPENAI_API_KEY}"
base_url = "${OPENAI_BASE_URL}"  # empty if unset

Unclosed braces are treated as literals

If a ${ is missing its closing }, the proxy leaves the text as-is rather than silently truncating. That makes typos easy to spot in logs.

Hot reload

Use liter-llm api --config ./liter-llm-proxy.toml --watch to reload a local file after saves. Use liter-llm api --watch --etcd-endpoint http://127.0.0.1:2379 --etcd-key /liter-llm/config to watch distributed config from etcd.

Validation

The parser sets deny_unknown_fields on every struct. Any typo or unsupported field raises an invalid TOML config error with the line and column. Fix the typo and restart, or run with --watch so the proxy reloads the corrected file or etcd value.