Cost Estimation

Liter-llm embeds a pricing registry at compile time (crates/liter-llm/schemas/catalog.json) and exposes functions for estimating USD cost from token counts (completion_cost, completion_cost_with_cache) plus accessors for the underlying pricing (model_pricing) and per-model metadata (model_info). No network access or external service is required.

Pricing data is derived from models.dev (MIT License) and covers the most widely used models across major providers.

Data source & sync

catalog.json is generated by the liter-llm-catalog-gen crate from the upstream models.dev unified model catalog (https://models.dev/api.json, MIT License, maintained by anomalyco/opencode). The generator:

1. Fetches api.json over HTTPS, restricted to the models.dev host allowlist.
2. Validates the payload against the models.dev schema, rejecting unknown fields so upstream schema drift fails loudly instead of silently corrupting pricing data.
3. Transforms it into liter-llm’s provider -> model document shape (unpriced models simply omit their pricing sub-object).
4. Dual-writes the result to schemas/catalog.json (canonical) and crates/liter-llm/schemas/catalog.json (the crate’s own copy, embedded via include_str! at compile time — see below).

Every generated catalog.json carries a $provenance block:

{
  "$provenance": {
    "source": "models.dev/api.json",
    "source_sha256": "…",
    "fetched": "2026-07-19",
    "library_version": "…"
  }
}

source_sha256 is the SHA-256 of the exact upstream bytes that produced the file, so any committed catalog.json traces back to a specific models.dev payload. The published release artifact (see Catalog Refresh) additionally carries generated_by_commit and release_tag, so a downloaded catalog also traces back to the exact liter-llm commit and release that published it.

The embedded catalog is compiled in — the crate build never touches the network. Regenerating it is a separate, opt-in step:

• task generate:catalog — refetches api.json and rewrites both output files.
• task generate:catalog:check — regenerates in memory and fails if the committed files have drifted from models.dev, without writing anything. Runs in CI on every PR that touches schemas/catalog.json, crates/liter-llm/schemas/catalog.json, or the generator crate.

A daily scheduled workflow (sync-catalog.yml) also runs task generate:catalog, opens a PR when the output changes, and enables auto-merge once required checks pass. A separate workflow (publish-catalog.yml) pushes every update to schemas/catalog.json on main to the rolling model-catalog GitHub release, injecting generated_by_commit and release_tag into $provenance before upload. This is the artifact the opt-in runtime refresh downloads from by default.

API

cost::completion_cost

Calculate estimated USD cost given a model name and token counts.

use liter_llm::cost;

// Returns None for unknown models.
let unknown = cost::completion_cost("my-custom-model", 1000, 500);
assert!(unknown.is_none());

// Returns Some(usd) for known models.
// gpt-4o: input $2.50/1M tokens = 0.0000025/token
//         output $10/1M tokens  = 0.00001/token
let usd = cost::completion_cost("gpt-4o", 1_000, 500).unwrap();
// 1000 * 0.0000025 + 500 * 0.00001 = 0.0025 + 0.005 = 0.0075
assert!((usd - 0.0075).abs() < 1e-9);

cost::model_pricing

Retrieve the per-token pricing struct directly.

use liter_llm::cost;

let p = cost::model_pricing("gpt-4o").unwrap();
println!("input:  ${:.10}/token", p.input_cost_per_token);
println!("output: ${:.10}/token", p.output_cost_per_token);

pub struct ModelPricing {
    pub input_cost_per_token: f64,   // USD per prompt token
    pub output_cost_per_token: f64,  // USD per completion token; 0.0 for embedding models
    // Plus optional fields, populated when the catalog provides them:
    // cache read/write token costs, input/output audio token costs,
    // reasoning token cost, and `tiers` for context-size-dependent pricing
    // (e.g. a higher rate above a 200k-token context). See `model_info`
    // and the reference docs for the full per-model metadata surface.
}

For models with context-tiered pricing, completion_cost automatically selects the rate for the highest tier whose minimum context is at or below the prompt token count, and adds any audio and reasoning token costs on top of the base input/output cost.

cost::model_info

Retrieve per-model metadata — context window, max input/output tokens, mode, and capability flags (vision, reasoning, tool calling, structured output, audio) — as an owned, FFI-friendly ModelInfo value. Returns None for unknown models, using the same prefix-fallback lookup as model_pricing.

use liter_llm::cost;

if let Some(info) = cost::model_info("gpt-4o") {
    println!("context window: {:?}", info.max_input_tokens);
    println!("supports vision: {:?}", info.supports_vision);
}

Prefix fallback

When an exact model name is not found, the registry strips from the last - or . separator and retries. This means versioned model names like gpt-4-0613 resolve to the gpt-4 entry automatically.

gpt-4-0613  →  try "gpt-4-0613"  →  try "gpt-4"  →  found
claude-3-opus-20240229  →  try exact  →  try "claude-3-opus"  →  found

Response-level cost

Successful chat and embedding responses expose estimated_cost() directly on the response object, using the same registry lookup:

let resp = client.chat(req).await?;
if let Some(usd) = resp.estimated_cost() {
    println!("cost: ${:.6}", usd);
}

let resp = client.embed(req).await?;
if let Some(usd) = resp.estimated_cost() {
    println!("cost: ${:.6}", usd);
}

Tracing integration

CostTrackingLayer records the estimated cost as gen_ai.usage.cost on the active OpenTelemetry span after each successful response. This requires the tower feature flag. See Observability for setup details.

Proxy budget enforcement

The proxy server’s budget system ([budget] in the config file) uses the same pricing registry to track cumulative spend per virtual key and globally. Hard-budget mode rejects requests that would exceed the cap; soft-budget mode logs a warning but allows them through.

[budget]
global_limit = 50.0       # USD; rejects requests once exceeded in hard mode
enforcement = "hard"      # "hard" or "soft"

[[keys]]
key = "vk-..."
budget_limit = 5.0        # per-key cap

See Proxy Configuration for the full [budget] and [[keys]] field reference.

Supported models

The pricing registry covers all major OpenAI, Anthropic, Google, Mistral, Cohere, Meta, and Bedrock models. Run the following to inspect entries from the embedded registry at runtime:

// Check if a model has pricing before making a call.
if liter_llm::cost::model_pricing("anthropic/claude-3-5-sonnet-20241022").is_none() {
    eprintln!("no pricing data for this model; cost tracking will be skipped");
}

Models not in the registry return None from both completion_cost and model_pricing. Cost tracking is silently skipped for those models. No error is raised.