Local LLMs
Liter-llm routes to any local inference engine that exposes an OpenAI-compatible API. Run models on your own hardware with zero cloud dependencies and no API key.
Supported Providers
Section titled “Supported Providers”| Provider | Default URL | Prefix | Notes |
|---|---|---|---|
| Ollama | http://localhost:11434 |
ollama/ |
Most popular, easy setup. Uses the native Ollama API |
| LM Studio | http://localhost:1234/v1 |
lmstudio/ |
GUI-based, beginner-friendly |
| vLLM | http://localhost:8000/v1 |
vllm/ |
High-throughput serving |
| llamafile | http://localhost:8080/v1 |
llamafile/ |
Single-file executable |
All of these providers are registered in the provider registry. LocalAI and llama.cpp are also built in with the localai/ and llamacpp/ prefixes. For any other OpenAI-compatible server, use a custom provider — register the prefix and base URL once, then route to it like any other provider.
Ollama is the exception to the OpenAI-compatible rule: ollama/ calls its native API, /api/chat and /api/embed. See Ollama’s API. All other listed engines also support streaming via SSE and model listing via /v1/models. Tool calling, vision, and multimodal inputs work through the chat endpoint where the underlying model supports them.
Quick Start with Ollama
Section titled “Quick Start with Ollama”1. Install Ollama
Section titled “1. Install Ollama”# macOS / Linuxcurl -fsSL https://ollama.ai/install.sh | sh
# Or via Homebrewbrew install ollama2. Pull a Model
Section titled “2. Pull a Model”ollama pull qwen2:0.5b3. Use with liter-llm
Section titled “3. Use with liter-llm”Ollama local provider routes requests via ollama/ model prefix with no auth
import asyncioimport osfrom liter_llm import create_clientfrom liter_llm._internal_bindings import ChatCompletionRequest
async def main() -> None: client = create_client(api_key=os.environ["API_KEY"]) req = ChatCompletionRequest.from_json("{\"messages\":[{\"content\":\"Hello\",\"role\":\"user\"}],\"model\":\"ollama/qwen2:0.5b\"}") result = await client.chat(req) print(result.choices) print(result.choices[0].message.content) print(result.choices[0].finish_reason) print(result.model)
asyncio.run(main())Ollama local provider routes requests via ollama/ model prefix with no auth
import { createClient } from "@xberg-io/liter-llm";async function main() { const client = createClient("your-api-key"); const result = await client.chat({ messages: [{ content: "Hello", role: "user" }], model: "ollama/qwen2:0.5b" }); console.log(result.choices); console.log(result.choices?.[0]?.message?.content); console.log(result.choices?.[0]?.finishReason); console.log(result.model);}
void main();Ollama local provider routes requests via ollama/ model prefix with no auth
use liter_llm::LlmClient;
#[tokio::main]async fn main() { let req_json: serde_json::Value = serde_json::from_str(r#"{"messages":[{"content":"Hello","role":"user"}],"model":"ollama/qwen2:0.5b"}"#).unwrap(); let req = serde_json::from_value(req_json).unwrap(); let client = liter_llm::create_client(std::env::var("API_KEY").expect("API_KEY must be set"), None, None, None, None).unwrap(); let result = client.chat(req).await.expect("call failed"); println!("{:?}", result.choices); println!("{:?}", result.choices[0].message.content); println!("{:?}", result.choices[0].finish_reason); println!("{:?}", result.model);}Ollama local provider routes requests via ollama/ model prefix with no auth
package main
import ( "fmt" pkg "github.com/xberg-io/liter-llm/packages/go/v2")
func main() { req := pkg.ChatCompletionRequest{ Model: `ollama/qwen2:0.5b`, Messages: []pkg.Message{ pkg.Message{Role: `user`, User: &pkg.UserMessage{ Content: pkg.UserContent(`"Hello"`), }}, }, } client, clientErr := pkg.CreateClient("your-api-key", nil, nil, nil, nil) if clientErr != nil { panic(clientErr) } defer client.Free() result, err := client.Chat(req) if err != nil { panic(err) } fmt.Printf("%+v\n", result.Choices) fmt.Printf("%+v\n", result.Choices[0].Message.Content) fmt.Printf("%+v\n", result.Choices[0].FinishReason) fmt.Printf("%+v\n", result.Model)}Model Naming Convention
Section titled “Model Naming Convention”Liter-llm uses the standard provider/model-name prefix convention for local providers, just like cloud providers:
ollama/llama3.2 -> Ollama running Llama 3.2ollama/qwen2:0.5b -> Ollama running Qwen2 0.5Blmstudio/my-model -> LM Studiovllm/meta-llama/Llama-3 -> vLLMllamafile/my-model -> llamafileThe prefix determines which base URL and configuration to use. The model name after the / is forwarded to the local server as-is.
Streaming
Section titled “Streaming”All local providers support streaming responses via Server-Sent Events (SSE), identical to the cloud provider streaming interface:
async for chunk in client.chat_stream( model="ollama/qwen2:0.5b", messages=[{"role": "user", "content": "Hello!"}],): print(chunk.choices[0].delta.content, end="")use futures_util::StreamExt;
let mut stream = client.chat_stream(request).await?;while let Some(chunk) = stream.next().await { let chunk = chunk?; if let Some(content) = &chunk.choices[0].delta.content { print!("{content}"); }}Embeddings
Section titled “Embeddings”Several local providers support embedding models. Use the standard embeddings API:
response = await client.embed( model="ollama/all-minilm", input="The quick brown fox",)print(f"Dimensions: {len(response.data[0].embedding)}")let response = client.embed(EmbeddingRequest { model: "ollama/all-minilm".into(), input: EmbeddingInput::Single("The quick brown fox".into()), ..Default::default()}).await?;Popular local embedding models include all-minilm (384 dims), nomic-embed-text (768 dims), and mxbai-embed-large (1024 dims) on Ollama.
Provider Configuration
Section titled “Provider Configuration”Ollama
Section titled “Ollama”Ollama runs on port 11434 by default. The ollama/ prefix (and its alias ollama_chat/) needs no additional configuration. To use another host, set base_url to the server address. One trailing /v1 comes off before /api/... is appended, so an address written for the OpenAI-compatible endpoint works too:
api_key = ""base_url = "http://ollama.internal:11434"Ollama’s API
Section titled “Ollama’s API”The ollama/ prefix calls Ollama’s own API, /api/chat and /api/embed. Ollama reads the model settings num_ctx, top_k and repeat_penalty only on this API, so liter-llm puts every extra_body key except keep_alive, think, format, logprobs and top_logprobs into the options object of a chat request. An embed request keeps model, input, truncate, keep_alive and dimensions at the top level and puts every other extra_body key into options, think and format included. On an embed request, the keys of an explicit options object win over bare keys of the same name. Chat streams read newline-delimited JSON.
To reach Ollama’s OpenAI-compatible endpoint instead, use the openai/ prefix and set base_url to the Ollama address with /v1:
client = liter_llm.create_client_with_options( liter_llm.ClientOptions(api_key="", base_url="http://localhost:11434/v1"))# model="openai/qwen2:0.5b" calls /v1/chat/completionsOllama ignores num_ctx, top_k and repeat_penalty on that endpoint.
The routing rule is: an Ollama hint always means the native API. A client built with the hint ollama or ollama_chat, or with a hint that starts with ollama/ or ollama_chat/, sends every bare model such as qwen2:0.5b or hf.co/user/model to /api/chat and /api/embed, whatever base_url is. On such a client list_models reads /api/tags. A base_url that ends in /v1 loses that suffix first. A model with another provider’s prefix, such as openai/m or groq/m, goes to the OpenAI-compatible endpoint at the base_url as given, with the model name as given, as it did before. A provider prefix is the name of a built-in provider or of a custom provider registered with register_custom_provider; a prefix that names neither, such as acme/ with no acme provider, makes no provider prefix, and the model is bare. The OpenAI-compatible endpoint is reached only through a provider prefix, such as openai/, as the hint or the model, with the Ollama base_url.
Some request fields map to a native Ollama field:
| Request field | Ollama field |
|---|---|
reasoning_effort |
think (minimal becomes low, max becomes high, none becomes false). A think key in extra_body wins. |
logprobs, top_logprobs |
The same fields. A stream cannot carry them, so a stream request drops them with a warning. |
max_tokens, max_completion_tokens, stop, seed, temperature, top_p, presence_penalty, frequency_penalty |
options |
Ollama has no equivalent for these fields, so liter-llm drops them: n, user, logit_bias, parallel_tool_calls, service_tier, store, stream_options, prediction, and on embeddings user and encoding_format. liter-llm logs a warning for metadata, audio, web_search_options, a non-text modalities value, a tool_choice other than auto, and an encoding_format of base64.
An extra_body key with the value null, at the top level or inside options, is not sent.
liter-llm refuses some requests with a BadRequest error before it sends them, because Ollama’s API cannot take them:
| Request | What to send instead |
|---|---|
An image_url part with an http or https URL |
Download the image and send it as a base64 data URL, such as data:image/png;base64,.... A data URL without ;base64 is refused too. |
A message content part other than text and image_url, such as input_audio or file |
Send text and images only, or use a provider that takes the part. |
Tool-call arguments in an assistant message that are not valid JSON |
Send the arguments as a JSON object, as the model returned them. |
Embedding input that is not a string or a list of strings, such as token arrays or content parts |
Send the text as a string or a list of strings. |
LM Studio
Section titled “LM Studio”LM Studio runs on port 1234 by default. Load a model in the LM Studio GUI, then use it:
api_key = ""
[[providers]]name = "lmstudio"base_url = "http://localhost:1234/v1"model_prefixes = ["lmstudio/"]Start vLLM with the OpenAI-compatible server:
python -m vllm.entrypoints.openai.api_server \ --model meta-llama/Llama-3-8B \ --port 8000api_key = ""
[[providers]]name = "vllm"base_url = "http://localhost:8000/v1"model_prefixes = ["vllm/"]Llama.cpp
Section titled “Llama.cpp”Start the llama.cpp server:
./llama-server -m model.gguf --port 8080api_key = ""
[[providers]]name = "llamacpp"base_url = "http://localhost:8080/v1"model_prefixes = ["llamacpp/"]LocalAI
Section titled “LocalAI”docker run -p 8080:8080 localai/localai:latestapi_key = ""
[[providers]]name = "localai"base_url = "http://localhost:8080/v1"model_prefixes = ["localai/"]Llamafile
Section titled “Llamafile”Download and run a llamafile:
chmod +x llava-v1.5-7b-q4.llamafile./llava-v1.5-7b-q4.llamafile --server --port 8080api_key = ""
[[providers]]name = "llamafile"base_url = "http://localhost:8080/v1"model_prefixes = ["llamafile/"]Custom Base URL
Section titled “Custom Base URL”If your local provider runs on a non-default port or remote host, override the base URL when constructing the client:
from liter_llm import create_client
client = create_client(api_key="", base_url="http://192.168.1.100:9000/v1")import { createClient } from "@xberg-io/liter-llm";
const client = createClient("", "http://192.168.1.100:9000/v1");use liter_llm::{ClientConfigBuilder, DefaultClient};
let config = ClientConfigBuilder::new("") .base_url("http://192.168.1.100:9000/v1") .build();let client = DefaultClient::new(config, None)?;Or in liter-llm.toml:
api_key = ""base_url = "http://192.168.1.100:9000/v1"With a custom base_url, every request goes to that server and the model name is sent verbatim, so gateways that route on a prefix (OpenRouter’s openai/gpt-4o, a liter-llm or LiteLLM proxy’s anthropic/...) receive it unchanged. To have the client strip a provider prefix, pass that provider as model_hint (the fifth argument of create_client, or the second argument of DefaultClient::new): with model_hint="openai", openai/gpt-4o-mini is sent as gpt-4o-mini, while anthropic/claude-x and meta-llama/llama-3 are still sent as-is. A hint such as openai/gpt-4o-mini selects the same provider. A hint that names a built-in provider also makes the client use that provider’s request format and paths, and base_url changes only the host: with model_hint="cohere", the client sends the native Cohere request to your base_url. A hint that names no built-in provider, and no hint, give a plain OpenAI-compatible request. The openai hint keeps the plain OpenAI-compatible request. The custom/ prefix is stripped, except under a hint that names another built-in provider. A hint that selects a provider pins every request model without a provider/ prefix to that provider, so under a model-name hint such as gpt-4o, the client treats llama3 as an OpenAI model too. With a base_url and an empty api_key, a bare or model-name hint (cohere, anthropic, claude-sonnet-4, azure, gemini, vertex_ai, bedrock) reads no key from your environment, so no key from your environment goes to your server. Only a provider/ hint reads one: anthropic/claude-x, azure/..., gemini/... and google_ai/... read ANTHROPIC_API_KEY, AZURE_OPENAI_API_KEY and GEMINI_API_KEY, vertex_ai/... gets a Google access token, and bedrock/... uses AWS_BEARER_TOKEN_BEDROCK or signs each request with your AWS keys. A provider/ hint for any other provider, such as cohere/command-r-plus, reads no key variable.
Docker Compose
Section titled “Docker Compose”Run Ollama alongside the liter-llm proxy for a self-contained local setup:
services: ollama: image: ollama/ollama:latest ports: - "11434:11434" volumes: - ollama_data:/root/.ollama
liter-llm: image: ghcr.io/xberg-io/liter-llm:latest ports: - "4000:4000" environment: - LITER_LLM_API_KEY="" volumes: - ./liter-llm-proxy.toml:/etc/liter-llm/liter-llm-proxy.toml depends_on: - ollama
volumes: ollama_data:Example proxy config for local use:
[server]host = "0.0.0.0"port = 4000
[[providers]]name = "ollama"base_url = "http://ollama:11434"model_prefixes = ["ollama/"]Start the stack:
docker compose -f docker-compose.local.yaml up -d
# Pull a model into Ollamadocker exec -it $(docker compose -f docker-compose.local.yaml ps -q ollama) \ ollama pull qwen2:0.5b
# Chat via the proxycurl http://localhost:4000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "ollama/qwen2:0.5b", "messages": [{"role": "user", "content": "Hello!"}]}'Troubleshooting
Section titled “Troubleshooting”Connection Refused
Section titled “Connection Refused”Error: connection refused (os error 111)The local server is not running or is on a different port. Verify:
# Check if Ollama is runningcurl http://localhost:11434/v1/models
# Check if the port is in uselsof -i :11434Model Not Found
Section titled “Model Not Found”Error: model "llama3.2" not foundThe model is not downloaded. Pull it first:
# Ollamaollama pull llama3.2
# Check installed modelsollama listTimeout Errors
Section titled “Timeout Errors”Local models can be slow to load on first request (especially large models). Increase the timeout:
timeout_secs = 300 # 5 minutes for initial model loadDocker Networking
Section titled “Docker Networking”When running liter-llm in Docker and a local provider on the host:
- Linux: Use
http://host.docker.internal:11434orhttp://172.17.0.1:11434 - macOS/Windows: Use
http://host.docker.internal:11434
# liter-llm-proxy.toml (inside Docker)[[providers]]name = "ollama"base_url = "http://host.docker.internal:11434"model_prefixes = ["ollama/"]GPU / Performance
Section titled “GPU / Performance”- Ollama: Automatically uses GPU if available. Check with
ollama ps. - vLLM: Pass
--tensor-parallel-size Nfor multi-GPU. - llama.cpp: Use
-ngl Nto offload N layers to GPU. - LocalAI: Set
GPU_LAYERSenvironment variable.