Skip to content

Cost Estimation

Liter-llm embeds a pricing registry at compile time (crates/liter-llm/schemas/catalog.json) and exposes functions for estimating USD cost from token counts (completion_cost, completion_cost_with_cache) plus accessors for the underlying pricing (model_pricing) and per-model metadata (model_info). No network access or external service is required.

Pricing data is derived from models.dev (MIT License) and covers the most widely used models across major providers.

catalog.json is generated by the liter-llm-catalog-gen crate from the upstream models.dev unified model catalog (https://models.dev/api.json, MIT License, maintained by anomalyco/opencode). The generator:

  1. Fetches api.json over HTTPS, restricted to the models.dev host allowlist.
  2. Validates the payload against the models.dev schema, rejecting unknown fields so upstream schema drift fails loudly instead of silently corrupting pricing data.
  3. Transforms it into liter-llm’s provider -> model document shape (unpriced models simply omit their pricing sub-object).
  4. Dual-writes the result to schemas/catalog.json (canonical) and crates/liter-llm/schemas/catalog.json (the crate’s own copy, embedded via include_str! at compile time — see below).

Every generated catalog.json carries a $provenance block:

{
"$provenance": {
"source": "models.dev/api.json",
"source_sha256": "…",
"fetched": "2026-07-19",
"library_version": "…"
}
}

source_sha256 is the SHA-256 of the exact upstream bytes that produced the file, so any committed catalog.json traces back to a specific models.dev payload. The published release artifact (see Catalog Refresh) additionally carries generated_by_commit and release_tag, so a downloaded catalog also traces back to the exact liter-llm commit and release that published it.

The embedded catalog is compiled in — the crate build never touches the network. Regenerating it is a separate, opt-in step:

  • mise run generate:catalog — refetches api.json and rewrites both output files.
  • mise run generate:catalog:check — regenerates in memory and fails if the committed files have drifted from models.dev, without writing anything. Runs in CI on every PR that touches schemas/catalog.json, crates/liter-llm/schemas/catalog.json, or the generator crate.

A daily scheduled workflow (sync-catalog.yml) also runs mise run generate:catalog, opens a PR when the output changes, and enables auto-merge once required checks pass. A separate workflow (publish-catalog.yml) pushes every update to schemas/catalog.json on main to the rolling model-catalog GitHub release, injecting generated_by_commit and release_tag into $provenance before upload. This is the artifact the opt-in runtime refresh downloads from by default.

Calculate estimated USD cost given a model name and token counts.

use liter_llm::cost;
// Returns None for unknown models.
let unknown = cost::completion_cost("my-custom-model", 1000, 500);
assert!(unknown.is_none());
// Returns Some(usd) for known models.
// gpt-4o: input $2.50/1M tokens = 0.0000025/token
// output $10/1M tokens = 0.00001/token
let usd = cost::completion_cost("gpt-4o", 1_000, 500).unwrap();
// 1000 * 0.0000025 + 500 * 0.00001 = 0.0025 + 0.005 = 0.0075
assert!((usd - 0.0075).abs() < 1e-9);

Retrieve the per-token pricing struct directly.

use liter_llm::cost;
let p = cost::model_pricing("gpt-4o").unwrap();
println!("input: ${:.10}/token", p.input_cost_per_token);
println!("output: ${:.10}/token", p.output_cost_per_token);
pub struct ModelPricing {
pub input_cost_per_token: f64, // USD per prompt token
pub output_cost_per_token: f64, // USD per completion token; 0.0 for embedding models
// Plus optional fields, populated when the catalog provides them:
// cache read/write token costs, input/output audio token costs,
// reasoning token cost, and `tiers` for context-size-dependent pricing
// (e.g. a higher rate above a 200k-token context). See `model_info`
// and the reference docs for the full per-model metadata surface.
}

For models with context-tiered pricing, completion_cost automatically selects the rate for the highest tier whose minimum context is at or below the prompt token count, and adds any audio and reasoning token costs on top of the base input/output cost.

Retrieve per-model metadata — context window, max input/output tokens, mode, and capability flags (vision, reasoning, tool calling, structured output, audio) — as an owned, FFI-friendly ModelInfo value. Returns None for unknown models, using the same prefix-fallback lookup as model_pricing.

use liter_llm::cost;
if let Some(info) = cost::model_info("gpt-4o") {
println!("context window: {:?}", info.max_input_tokens);
println!("supports vision: {:?}", info.supports_vision);
}

List every model of the catalog without a network call or an API key, for example to fill a model picker. Each entry carries the key, the provider id, the model id, the mode, the token limits and the capability flags. model_info and exact_model_info accept the key unchanged. The list is sorted by key in byte order, so upper case sorts before lower case. An installed catalog refresh replaces the embedded entries of the same key.

The provider argument takes a catalog provider id: the part of a key before the first slash, such as openai in openai/gpt-4o. The catalog has its own set of provider ids. It is not the set of routing names that all_providers returns, and a routing name that is not a catalog id gives an empty list. The comparison is exact and case-sensitive. An absent or empty provider keeps every provider.

use liter_llm::cost;
for model in cost::list_catalog_models(Some("anthropic")) {
println!("{} reasoning: {:?}", model.key, model.supports_reasoning);
}

When an exact model name is not found, the registry strips from the last - or . separator and retries. This means versioned model names like gpt-4-0613 resolve to the gpt-4 entry automatically.

gpt-4-0613 → try "gpt-4-0613" → try "gpt-4" → found
claude-3-opus-20240229 → try exact → try "claude-3-opus" → found

Successful chat and embedding responses expose estimated_cost() directly on the response object, using the same registry lookup:

let resp = client.chat(req).await?;
if let Some(usd) = resp.estimated_cost() {
println!("cost: ${:.6}", usd);
}
let resp = client.embed(req).await?;
if let Some(usd) = resp.estimated_cost() {
println!("cost: ${:.6}", usd);
}

CostTrackingLayer records the estimated cost as gen_ai.usage.cost on the active OpenTelemetry span after each successful response. This requires the tower feature flag. See Observability for setup details.

The proxy server’s budget system ([budget] in the config file) uses the same pricing registry to track cumulative spend per virtual key and globally. Hard-budget mode rejects requests that would exceed the cap; soft-budget mode logs a warning but allows them through.

[budget]
global_limit = 50.0 # USD; rejects requests once exceeded in hard mode
enforcement = "hard" # "hard" or "soft"
[[keys]]
key = "vk-..."
budget_limit = 5.0 # per-key cap

See Proxy Configuration for the full [budget] and [[keys]] field reference.

The pricing registry covers all major OpenAI, Anthropic, Google, Mistral, Cohere, Meta, and Bedrock models. Run the following to inspect entries from the embedded registry at runtime:

// Check if a model has pricing before making a call.
if liter_llm::cost::model_pricing("anthropic/claude-3-5-sonnet-20241022").is_none() {
eprintln!("no pricing data for this model; cost tracking will be skipped");
}

Models not in the registry return None from both completion_cost and model_pricing. Cost tracking is silently skipped for those models. No error is raised.