Skip to content

Local LLMs

Liter-llm routes to any local inference engine that exposes an OpenAI-compatible API. Run models on your own hardware with zero cloud dependencies and no API key.

Provider Default URL Prefix Notes
Ollama http://localhost:11434 ollama/ Most popular, easy setup. Uses the native Ollama API
LM Studio http://localhost:1234/v1 lmstudio/ GUI-based, beginner-friendly
vLLM http://localhost:8000/v1 vllm/ High-throughput serving
llamafile http://localhost:8080/v1 llamafile/ Single-file executable

All of these providers are registered in the provider registry. LocalAI and llama.cpp are also built in with the localai/ and llamacpp/ prefixes. For any other OpenAI-compatible server, use a custom provider — register the prefix and base URL once, then route to it like any other provider.

Ollama is the exception to the OpenAI-compatible rule: ollama/ calls its native API, /api/chat and /api/embed. See Ollama’s API. All other listed engines also support streaming via SSE and model listing via /v1/models. Tool calling, vision, and multimodal inputs work through the chat endpoint where the underlying model supports them.

Terminal window
# macOS / Linux
curl -fsSL https://ollama.ai/install.sh | sh
# Or via Homebrew
brew install ollama
Terminal window
ollama pull qwen2:0.5b

Ollama local provider routes requests via ollama/ model prefix with no auth

Python
import asyncio
import os
from liter_llm import create_client
from liter_llm._internal_bindings import ChatCompletionRequest
async def main() -> None:
client = create_client(api_key=os.environ["API_KEY"])
req = ChatCompletionRequest.from_json("{\"messages\":[{\"content\":\"Hello\",\"role\":\"user\"}],\"model\":\"ollama/qwen2:0.5b\"}")
result = await client.chat(req)
print(result.choices)
print(result.choices[0].message.content)
print(result.choices[0].finish_reason)
print(result.model)
asyncio.run(main())

Liter-llm uses the standard provider/model-name prefix convention for local providers, just like cloud providers:

ollama/llama3.2 -> Ollama running Llama 3.2
ollama/qwen2:0.5b -> Ollama running Qwen2 0.5B
lmstudio/my-model -> LM Studio
vllm/meta-llama/Llama-3 -> vLLM
llamafile/my-model -> llamafile

The prefix determines which base URL and configuration to use. The model name after the / is forwarded to the local server as-is.

All local providers support streaming responses via Server-Sent Events (SSE), identical to the cloud provider streaming interface:

async for chunk in client.chat_stream(
model="ollama/qwen2:0.5b",
messages=[{"role": "user", "content": "Hello!"}],
):
print(chunk.choices[0].delta.content, end="")

Several local providers support embedding models. Use the standard embeddings API:

response = await client.embed(
model="ollama/all-minilm",
input="The quick brown fox",
)
print(f"Dimensions: {len(response.data[0].embedding)}")

Popular local embedding models include all-minilm (384 dims), nomic-embed-text (768 dims), and mxbai-embed-large (1024 dims) on Ollama.

Ollama runs on port 11434 by default. The ollama/ prefix (and its alias ollama_chat/) needs no additional configuration. To use another host, set base_url to the server address. One trailing /v1 comes off before /api/... is appended, so an address written for the OpenAI-compatible endpoint works too:

liter-llm.toml
api_key = ""
base_url = "http://ollama.internal:11434"

The ollama/ prefix calls Ollama’s own API, /api/chat and /api/embed. Ollama reads the model settings num_ctx, top_k and repeat_penalty only on this API, so liter-llm puts every extra_body key except keep_alive, think, format, logprobs and top_logprobs into the options object of a chat request. An embed request keeps model, input, truncate, keep_alive and dimensions at the top level and puts every other extra_body key into options, think and format included. On an embed request, the keys of an explicit options object win over bare keys of the same name. Chat streams read newline-delimited JSON.

To reach Ollama’s OpenAI-compatible endpoint instead, use the openai/ prefix and set base_url to the Ollama address with /v1:

client = liter_llm.create_client_with_options(
liter_llm.ClientOptions(api_key="", base_url="http://localhost:11434/v1")
)
# model="openai/qwen2:0.5b" calls /v1/chat/completions

Ollama ignores num_ctx, top_k and repeat_penalty on that endpoint.

The routing rule is: an Ollama hint always means the native API. A client built with the hint ollama or ollama_chat, or with a hint that starts with ollama/ or ollama_chat/, sends every bare model such as qwen2:0.5b or hf.co/user/model to /api/chat and /api/embed, whatever base_url is. On such a client list_models reads /api/tags. A base_url that ends in /v1 loses that suffix first. A model with another provider’s prefix, such as openai/m or groq/m, goes to the OpenAI-compatible endpoint at the base_url as given, with the model name as given, as it did before. A provider prefix is the name of a built-in provider or of a custom provider registered with register_custom_provider; a prefix that names neither, such as acme/ with no acme provider, makes no provider prefix, and the model is bare. The OpenAI-compatible endpoint is reached only through a provider prefix, such as openai/, as the hint or the model, with the Ollama base_url.

Some request fields map to a native Ollama field:

Request field Ollama field
reasoning_effort think (minimal becomes low, max becomes high, none becomes false). A think key in extra_body wins.
logprobs, top_logprobs The same fields. A stream cannot carry them, so a stream request drops them with a warning.
max_tokens, max_completion_tokens, stop, seed, temperature, top_p, presence_penalty, frequency_penalty options

Ollama has no equivalent for these fields, so liter-llm drops them: n, user, logit_bias, parallel_tool_calls, service_tier, store, stream_options, prediction, and on embeddings user and encoding_format. liter-llm logs a warning for metadata, audio, web_search_options, a non-text modalities value, a tool_choice other than auto, and an encoding_format of base64.

An extra_body key with the value null, at the top level or inside options, is not sent.

liter-llm refuses some requests with a BadRequest error before it sends them, because Ollama’s API cannot take them:

Request What to send instead
An image_url part with an http or https URL Download the image and send it as a base64 data URL, such as data:image/png;base64,.... A data URL without ;base64 is refused too.
A message content part other than text and image_url, such as input_audio or file Send text and images only, or use a provider that takes the part.
Tool-call arguments in an assistant message that are not valid JSON Send the arguments as a JSON object, as the model returned them.
Embedding input that is not a string or a list of strings, such as token arrays or content parts Send the text as a string or a list of strings.

LM Studio runs on port 1234 by default. Load a model in the LM Studio GUI, then use it:

liter-llm.toml
api_key = ""
[[providers]]
name = "lmstudio"
base_url = "http://localhost:1234/v1"
model_prefixes = ["lmstudio/"]

Start vLLM with the OpenAI-compatible server:

Terminal window
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3-8B \
--port 8000
liter-llm.toml
api_key = ""
[[providers]]
name = "vllm"
base_url = "http://localhost:8000/v1"
model_prefixes = ["vllm/"]

Start the llama.cpp server:

Terminal window
./llama-server -m model.gguf --port 8080
liter-llm.toml
api_key = ""
[[providers]]
name = "llamacpp"
base_url = "http://localhost:8080/v1"
model_prefixes = ["llamacpp/"]
Terminal window
docker run -p 8080:8080 localai/localai:latest
liter-llm.toml
api_key = ""
[[providers]]
name = "localai"
base_url = "http://localhost:8080/v1"
model_prefixes = ["localai/"]

Download and run a llamafile:

Terminal window
chmod +x llava-v1.5-7b-q4.llamafile
./llava-v1.5-7b-q4.llamafile --server --port 8080
liter-llm.toml
api_key = ""
[[providers]]
name = "llamafile"
base_url = "http://localhost:8080/v1"
model_prefixes = ["llamafile/"]

If your local provider runs on a non-default port or remote host, override the base URL when constructing the client:

from liter_llm import create_client
client = create_client(api_key="", base_url="http://192.168.1.100:9000/v1")

Or in liter-llm.toml:

api_key = ""
base_url = "http://192.168.1.100:9000/v1"

With a custom base_url, every request goes to that server and the model name is sent verbatim, so gateways that route on a prefix (OpenRouter’s openai/gpt-4o, a liter-llm or LiteLLM proxy’s anthropic/...) receive it unchanged. To have the client strip a provider prefix, pass that provider as model_hint (the fifth argument of create_client, or the second argument of DefaultClient::new): with model_hint="openai", openai/gpt-4o-mini is sent as gpt-4o-mini, while anthropic/claude-x and meta-llama/llama-3 are still sent as-is. A hint such as openai/gpt-4o-mini selects the same provider. A hint that names a built-in provider also makes the client use that provider’s request format and paths, and base_url changes only the host: with model_hint="cohere", the client sends the native Cohere request to your base_url. A hint that names no built-in provider, and no hint, give a plain OpenAI-compatible request. The openai hint keeps the plain OpenAI-compatible request. The custom/ prefix is stripped, except under a hint that names another built-in provider. A hint that selects a provider pins every request model without a provider/ prefix to that provider, so under a model-name hint such as gpt-4o, the client treats llama3 as an OpenAI model too. With a base_url and an empty api_key, a bare or model-name hint (cohere, anthropic, claude-sonnet-4, azure, gemini, vertex_ai, bedrock) reads no key from your environment, so no key from your environment goes to your server. Only a provider/ hint reads one: anthropic/claude-x, azure/..., gemini/... and google_ai/... read ANTHROPIC_API_KEY, AZURE_OPENAI_API_KEY and GEMINI_API_KEY, vertex_ai/... gets a Google access token, and bedrock/... uses AWS_BEARER_TOKEN_BEDROCK or signs each request with your AWS keys. A provider/ hint for any other provider, such as cohere/command-r-plus, reads no key variable.

Run Ollama alongside the liter-llm proxy for a self-contained local setup:

docker-compose.local.yaml
services:
ollama:
image: ollama/ollama:latest
ports:
- "11434:11434"
volumes:
- ollama_data:/root/.ollama
liter-llm:
image: ghcr.io/xberg-io/liter-llm:latest
ports:
- "4000:4000"
environment:
- LITER_LLM_API_KEY=""
volumes:
- ./liter-llm-proxy.toml:/etc/liter-llm/liter-llm-proxy.toml
depends_on:
- ollama
volumes:
ollama_data:

Example proxy config for local use:

liter-llm-proxy.toml
[server]
host = "0.0.0.0"
port = 4000
[[providers]]
name = "ollama"
base_url = "http://ollama:11434"
model_prefixes = ["ollama/"]

Start the stack:

Terminal window
docker compose -f docker-compose.local.yaml up -d
# Pull a model into Ollama
docker exec -it $(docker compose -f docker-compose.local.yaml ps -q ollama) \
ollama pull qwen2:0.5b
# Chat via the proxy
curl http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "ollama/qwen2:0.5b", "messages": [{"role": "user", "content": "Hello!"}]}'
Error: connection refused (os error 111)

The local server is not running or is on a different port. Verify:

Terminal window
# Check if Ollama is running
curl http://localhost:11434/v1/models
# Check if the port is in use
lsof -i :11434
Error: model "llama3.2" not found

The model is not downloaded. Pull it first:

Terminal window
# Ollama
ollama pull llama3.2
# Check installed models
ollama list

Local models can be slow to load on first request (especially large models). Increase the timeout:

liter-llm.toml
timeout_secs = 300 # 5 minutes for initial model load

When running liter-llm in Docker and a local provider on the host:

  • Linux: Use http://host.docker.internal:11434 or http://172.17.0.1:11434
  • macOS/Windows: Use http://host.docker.internal:11434
# liter-llm-proxy.toml (inside Docker)
[[providers]]
name = "ollama"
base_url = "http://host.docker.internal:11434"
model_prefixes = ["ollama/"]
  • Ollama: Automatically uses GPU if available. Check with ollama ps.
  • vLLM: Pass --tensor-parallel-size N for multi-GPU.
  • llama.cpp: Use -ngl N to offload N layers to GPU.
  • LocalAI: Set GPU_LAYERS environment variable.