| Name | lm-proxy JSON |
| Version |
2.0.0
JSON |
| download |
| home_page | None |
| Summary | "LM-Proxy" is OpenAI-compatible http proxy server for inferencing various LLMs capable of working with Google, Anthropic, OpenAI APIs, local PyTorch inference, etc. |
| upload_time | 2025-10-26 17:36:42 |
| maintainer | Vitalii Stepanenko |
| docs_url | None |
| author | Vitalii Stepanenko |
| requires_python | <4,>=3.11 |
| license | MIT License
Copyright (c) 2025 Vitalii Stepanenko
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE. |
| keywords |
llm
large language models
ai
gpt
openai
proxy
http
proxy-server
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<h1 align="center"><a href="#">LM-Proxy</a></h1>
<p align="center">
<b>Lightweight, OpenAI-compatible HTTP proxy server / gateway</b><br>unifying access to multiple <b>Large Language Model providers</b> and local inference <br>through a single, standardized API endpoint.
</p>
<p align="center">
<a href="https://pypi.org/project/lm-proxy/"><img src="https://img.shields.io/pypi/v/lm-proxy?color=blue" alt="PyPI"></a>
<a href="https://github.com/Nayjest/lm-proxy/actions/workflows/tests.yml"><img src="https://github.com/Nayjest/lm-proxy/actions/workflows/tests.yml/badge.svg" alt="Tests"></a>
<a href="https://github.com/Nayjest/lm-proxy/actions/workflows/code-style.yml"><img src="https://github.com/Nayjest/lm-proxy/actions/workflows/code-style.yml/badge.svg" alt="Code Style"></a>
<img src="https://raw.githubusercontent.com/Nayjest/lm-proxy/main/coverage.svg" alt="Code Coverage">
<a href="https://github.com/Nayjest/lm-proxy/blob/main/LICENSE"><img src="https://img.shields.io/github/license/Nayjest/lm-proxy?color=d08aff" alt="License"></a>
</p>
Built with Python, FastAPI and [MicroCore](https://github.com/Nayjest/ai-microcore), **LM-Proxy** seamlessly integrates cloud providers like Google, Anthropic, and OpenAI, as well as local PyTorch-based inference, while maintaining full compatibility with OpenAI's API format.
It works as a drop-in replacement for OpenAI's API, allowing you to switch between cloud providers and local models without modifying your existing client code.
**LM-Proxy** supports **real-time token streaming**, **secure Virtual API key management**, and can be used both as an importable Python library and as a standalone HTTP service. Whether you're building production applications or experimenting with different models, LM-Proxy eliminates integration complexity and keeps your codebase **provider-agnostic**.
## Table of Contents
- [Overview](#lm-proxy)
- [Features](#-features)
- [Getting Started](#-getting-started)
- [Installation](#installation)
- [Quick Start](#quick-start)
- [Configuration](#-configuration)
- [Basic Structure](#basic-structure)
- [Environment Variables](#environment-variables)
- [Proxy API Keys vs. Provider API Keys](#-proxy-api-keys-vs-provider-api-keys)
- [API Usage](#-api-usage)
- [Chat Completions Endpoint](#chat-completions-endpoint)
- [Models List Endpoint](#models-list-endpoint)
- [User Groups Configuration](#-user-groups-configuration)
- [Basic Group Definition](#basic-group-definition)
- [Group-based Access Control](#group-based-access-control)
- [Connection Restrictions](#connection-restrictions)
- [Virtual API Key Validation](#virtual-api-key-validation)
- [Advanced Usage](#%EF%B8%8F-advanced-usage)
- [Dynamic Model Routing](#dynamic-model-routing)
- [Load Balancing Example](#load-balancing-example)
- [Debugging](#-debugging)
- [Contributing](#-contributing)
- [License](#-license)
## ✨ Features
- **Provider Agnostic**: Connect to OpenAI, Anthropic, Google AI, local models, and more using a single API
- **Unified Interface**: Access all models through the standard OpenAI API format
- **Dynamic Routing**: Route requests to different LLM providers based on model name patterns
- **Stream Support**: Full streaming support for real-time responses
- **API Key Management**: Configurable API key validation and access control
- **Easy Configuration**: Simple TOML configuration files for setup
## 🚀 Getting Started
### Requirements
Python 3.11 | 3.12 | 3.13
### Installation
```bash
pip install lm-proxy
```
### Quick Start
#### 1. Create a `config.toml` file:
```toml
host = "0.0.0.0"
port = 8000
[connections]
[connections.openai]
api_type = "open_ai"
api_base = "https://api.openai.com/v1/"
api_key = "env:OPENAI_API_KEY"
[connections.anthropic]
api_type = "anthropic"
api_key = "env:ANTHROPIC_API_KEY"
[routing]
"gpt*" = "openai.*"
"claude*" = "anthropic.*"
"*" = "openai.gpt-3.5-turbo"
[groups.default]
api_keys = ["YOUR_API_KEY_HERE"]
```
> **Note**
> To enhance security, consider storing upstream API keys in operating system environment variables rather than embedding them directly in the configuration file. You can reference these variables in the configuration using the env:<VAR_NAME> syntax.
#### 2. Start the server:
```bash
lm-proxy
```
Alternatively, run it as a Python module:
```bash
python -m lm_proxy
```
#### 3. Use it with any OpenAI-compatible client:
```python
from openai import OpenAI
client = OpenAI(
api_key="YOUR_API_KEY_HERE",
base_url="http://localhost:8000/v1"
)
completion = client.chat.completions.create(
model="gpt-5", # This will be routed to OpenAI based on config
messages=[{"role": "user", "content": "Hello, world!"}]
)
print(completion.choices[0].message.content)
```
Or use the same endpoint with Claude models:
```python
completion = client.chat.completions.create(
model="claude-opus-4-1-20250805", # This will be routed to Anthropic based on config
messages=[{"role": "user", "content": "Hello, world!"}]
)
```
## 📝 Configuration
LM-Proxy is configured through a TOML file that specifies connections, routing rules, and access control.
### Basic Structure
```toml
host = "0.0.0.0" # Interface to bind to
port = 8000 # Port to listen on
dev_autoreload = false # Enable for development
# API key validation function (optional)
api_key_check = "lm_proxy.api_key_check.check_api_key_in_config"
# LLM Provider Connections
[connections]
[connections.openai]
api_type = "open_ai"
api_base = "https://api.openai.com/v1/"
api_key = "env:OPENAI_API_KEY"
[connections.google]
api_type = "google_ai_studio"
api_key = "env:GOOGLE_API_KEY"
[connections.anthropic]
api_type = "anthropic"
api_key = "env:ANTHROPIC_API_KEY"
# Routing rules (model_pattern = "connection.model")
[routing]
"gpt*" = "openai.*" # Route all GPT models to OpenAI
"claude*" = "anthropic.*" # Route all Claude models to Anthropic
"gemini*" = "google.*" # Route all Gemini models to Google
"*" = "openai.gpt-3.5-turbo" # Default fallback
# Access control groups
[groups.default]
api_keys = [
"KEY1",
"KEY2"
]
# optional
[[loggers]]
class = 'lm_proxy.loggers.BaseLogger'
[loggers.log_writer]
class = 'lm_proxy.loggers.log_writers.JsonLogWriter'
file_name = 'storage/json.log'
[loggers.entry_transformer]
class = 'lm_proxy.loggers.LogEntryTransformer'
completion_tokens = "response.usage.completion_tokens"
prompt_tokens = "response.usage.prompt_tokens"
prompt = "request.messages"
response = "response"
group = "group"
connection = "connection"
api_key_id = "api_key_id"
remote_addr = "remote_addr"
created_at = "created_at"
duration = "duration"
```
### Environment Variables
You can reference environment variables in your configuration file by prefixing values with `env:`.
For example:
```toml
[connections.openai]
api_key = "env:OPENAI_API_KEY"
```
At runtime, LM-Proxy automatically retrieves the value of the target variable
(OPENAI_API_KEY) from your operating system’s environment or from a .env file, if present.
### .env Files
By default, LM-Proxy looks for a `.env` file in the current working directory
and loads environment variables from it.
You can refer to the [.env.template](https://github.com/Nayjest/lm-proxy/blob/main/.env.template)
file for an example:
```dotenv
OPENAI_API_KEY=sk-u........
GOOGLE_API_KEY=AI........
ANTHROPIC_API_KEY=sk-ant-api03--vE........
# "1", "TRUE", "YES", "ON", "ENABLED", "Y", "+" are true, case-insensitive.
# See https://github.com/Nayjest/ai-microcore/blob/v4.4.3/microcore/configuration.py#L36
LM_PROXY_DEBUG=no
```
You can also control `.env` file usage with the `--env` command-line option:
```bash
# Use a custom .env file path
lm-proxy --env="path/to/your/.env"
# Disable .env loading
lm-proxy --env=""
```
## 🔑 Proxy API Keys vs. Provider API Keys
LM-Proxy utilizes two distinct types of API keys to facilitate secure and efficient request handling.
- **Proxy API Key (Virtual API Key, Client API Key):**
A unique key generated and managed within the LM-Proxy.
Clients use these keys to authenticate their requests to the proxy's API endpoints.
Each Client API Key is associated with a specific group, which defines the scope of access and permissions for the client's requests.
These keys allow users to securely interact with the proxy without direct access to external service credentials.
- **Provider API Key (Upstream API Key):**
A key provided by external LLM inference providers (e.g., OpenAI, Anthropic, Mistral, etc.) and configured within the LM-Proxy.
The proxy uses these keys to authenticate and forward validated client requests to the respective external services.
Provider API Keys remain hidden from end users, ensuring secure and transparent communication with provider APIs.
This distinction ensures a clear separation of concerns:
Virtual API Keys manage user authentication and access within the proxy,
while Upstream API Keys handle secure communication with external providers.
## 🔌 API Usage
LM-Proxy implements the OpenAI chat completions API endpoint. You can use any OpenAI-compatible client to interact with it.
### Chat Completions Endpoint
```http
POST /v1/chat/completions
```
#### Request Format
```json
{
"model": "gpt-3.5-turbo",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"}
],
"temperature": 0.7,
"stream": false
}
```
#### Response Format
```json
{
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "The capital of France is Paris."
},
"finish_reason": "stop"
}
]
}
```
### Models List Endpoint
List and describe all models available through the API.
```http
GET /v1/models
```
The **LM-Proxy** dynamically builds the models list based on routing rules defined in `config.routing`.
Routing keys can reference both **exact model names** and **model name patterns** (e.g., `"gpt*"`, `"claude*"`, etc.).
By default, wildcard patterns are displayed as-is in the models list (e.g., `"gpt*"`, `"claude*"`).
This behavior can be customized via the `model_listing_mode` configuration option:
```
model_listing_mode = "as_is" | "ignore_wildcards" | "expand_wildcards"
```
Available modes:
- **`as_is`** *(default)* — Lists all entries exactly as defined in the routing configuration, including wildcard patterns.
- **`ignore_wildcards`** — Excludes wildcard patterns, showing only explicitly defined model names.
- **`expand_wildcards`** — Expands wildcard patterns by querying each connected backend for available models *(feature not yet implemented)*.
To obtain a complete and accurate model list in the current implementation,
all supported models must be explicitly defined in the routing configuration, for example:
```toml
[routing]
"gpt-4" = "my_openai_connection.*"
"gpt-5" = "my_openai_connection.*"
"gpt-8"= "my_openai_connection.gpt-3.5-turbo"
"claude-4.5-sonnet" = "my_anthropic_connection.claude-sonnet-4-5-20250929"
"claude-4.1-opus" = "my_anthropic_connection.claude-opus-4-1-20250805"
[connections]
[connections.my_openai_connection]
api_type = "open_ai"
api_base = "https://api.openai.com/v1/"
api_key = "env:OPENAI_API_KEY"
[connections.my_anthropic_connection]
api_type = "anthropic"
api_key = "env:ANTHROPIC_API_KEY"
```
#### Response Format
```json
{
"object": "list",
"data": [
{
"id": "gpt-6",
"object": "model",
"created": 1686935002,
"owned_by": "organization-owner"
},
{
"id": "claude-5-sonnet",
"object": "model",
"created": 1686935002,
"owned_by": "organization-owner"
}
]
}
```
## 🔒 User Groups Configuration
The `[groups]` section in the configuration defines access control rules for different user groups.
Each group can have its own set of virtual API keys and permitted connections.
### Basic Group Definition
```toml
[groups.default]
api_keys = ["KEY1", "KEY2"]
allowed_connections = "*" # Allow access to all connections
```
### Group-based Access Control
You can create multiple groups to segment your users and control their access:
```toml
# Admin group with full access
[groups.admin]
api_keys = ["ADMIN_KEY_1", "ADMIN_KEY_2"]
allowed_connections = "*" # Access to all connections
# Regular users with limited access
[groups.users]
api_keys = ["USER_KEY_1", "USER_KEY_2"]
allowed_connections = "openai,anthropic" # Only allowed to use specific connections
# Free tier with minimal access
[groups.free]
api_keys = ["FREE_KEY_1", "FREE_KEY_2"]
allowed_connections = "openai" # Only allowed to use OpenAI connection
```
### Connection Restrictions
The `allowed_connections` parameter controls which upstream providers a group can access:
- `"*"` - Group can use all configured connections
- `"openai,anthropic"` - Comma-separated list of specific connections the group can use
This allows fine-grained control over which users can access which AI providers, enabling features like:
- Restricting expensive models to premium users
- Creating specialized access tiers for different user groups
- Implementing usage quotas per group
- Billing and cost allocation by user group
### Virtual API Key Validation
#### Overview
LM-proxy includes 2 built-in methods for validating Virtual API keys:
- `lm_proxy.api_key_check.check_api_key_in_config` - verifies API keys against those defined in the config file; used by default
- `lm_proxy.api_key_check.CheckAPIKeyWithRequest` - validates API keys via an external HTTP service
The API key check method can be configured using the `api_key_check` configuration key.
Its value can be either a reference to a Python function in the format `my_module.sub_module1.sub_module2.fn_name`,
or an object containing parameters for a class-based validator.
In the .py config representation, the validator function can be passed directly as a callable.
#### Example configuration for external API key validation using HTTP request to Keycloak / OpenID Connect
This example shows how to validate API keys against an external service (e.g., Keycloak):
```toml
[api_key_check]
class = "lm_proxy.api_key_check.CheckAPIKeyWithRequest"
method = "POST"
url = "http://keycloak:8080/realms/master/protocol/openid-connect/userinfo"
response_as_user_info = true # interpret response JSON as user info object for further processing / logging
use_cache = true # requires installing cachetools if True: pip install cachetools
cache_ttl = 60 # Cache duration in seconds
[api_key_check.headers]
Authorization = "Bearer {api_key}"
```
#### Custom API Key Validation / Extending functionality
For more advanced authentication needs,
you can implement a custom validator function:
```python
# my_validators.py
def validate_api_key(api_key: str) -> str | None:
"""
Validate an API key and return the group name if valid.
Args:
api_key: The API key to validate
Returns:
The name of the group if valid, None otherwise
"""
if api_key == "secret-key":
return "admin"
elif api_key.startswith("user-"):
return "users"
return None
```
Then reference it in your config:
```toml
api_key_check = "my_validators.validate_api_key"
```
> **NOTE**
> In this case, the `api_keys` lists in groups are ignored, and the custom function is responsible for all validation logic.
## 🛠️ Advanced Usage
### Dynamic Model Routing
The routing section allows flexible pattern matching with wildcards:
```toml
[routing]
"gpt-4*" = "openai.gpt-4" # Route gpt-4 requests to OpenAI GPT-4
"gpt-3.5*" = "openai.gpt-3.5-turbo" # Route gpt-3.5 requests to OpenAI
"claude*" = "anthropic.*" # Pass model name as-is to Anthropic
"gemini*" = "google.*" # Pass model name as-is to Google
"custom*" = "local.llama-7b" # Map any "custom*" to a specific local model
"*" = "openai.gpt-3.5-turbo" # Default fallback for unmatched models
```
Keys are model name patterns (with `*` wildcard support), and values are connection/model mappings.
Connection names reference those defined in the `[connections]` section.
### Load Balancing Example
- [Simple load-balancer configuration](https://github.com/Nayjest/lm-proxy/blob/main/examples/load_balancer_config.py)
This example demonstrates how to set up a load balancer that randomly
distributes requests across multiple language model servers using the lm_proxy.
## 🔍 Debugging
### Overview
When **debugging mode** is enabled,
LM-Proxy provides detailed logging information to help diagnose issues:
- Stack traces for exceptions are shown in the console
- Logging level is set to DEBUG instead of INFO
> **Warning** ⚠️
> Never enable debugging mode in production environments, as it may expose sensitive information to the application logs.
### Enabling Debugging Mode
To enable debugging, set the `LM_PROXY_DEBUG` environment variable to a truthy value (e.g., "1", "true", "yes").
> **Tip** 💡
> Environment variables can also be defined in a `.env` file.
Alternatively, you can enable or disable debugging via the command-line arguments:
- `--debug` to enable debugging
- `--no-debug` to disable debugging
> **Note** ℹ️
> CLI arguments override environment variable settings.
## 🤝 Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
1. Fork the repository
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add some amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request
## 📄 License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
© 2025 Vitalii Stepanenko
Raw data
{
"_id": null,
"home_page": null,
"name": "lm-proxy",
"maintainer": "Vitalii Stepanenko",
"docs_url": null,
"requires_python": "<4,>=3.11",
"maintainer_email": "mail@vitalii.in",
"keywords": "llm, large language models, ai, gpt, openai, proxy, http, proxy-server",
"author": "Vitalii Stepanenko",
"author_email": "mail@vitalii.in",
"download_url": "https://files.pythonhosted.org/packages/2a/60/a496d17c1d818d5bede9d8b1907247402b3fc58e6756a6c804a91f0672f7/lm_proxy-2.0.0.tar.gz",
"platform": null,
"description": "<h1 align=\"center\"><a href=\"#\">LM-Proxy</a></h1>\n<p align=\"center\">\n <b>Lightweight, OpenAI-compatible HTTP proxy server / gateway</b><br>unifying access to multiple <b>Large Language Model providers</b> and local inference <br>through a single, standardized API endpoint. \n</p>\n<p align=\"center\">\n <a href=\"https://pypi.org/project/lm-proxy/\"><img src=\"https://img.shields.io/pypi/v/lm-proxy?color=blue\" alt=\"PyPI\"></a>\n <a href=\"https://github.com/Nayjest/lm-proxy/actions/workflows/tests.yml\"><img src=\"https://github.com/Nayjest/lm-proxy/actions/workflows/tests.yml/badge.svg\" alt=\"Tests\"></a>\n <a href=\"https://github.com/Nayjest/lm-proxy/actions/workflows/code-style.yml\"><img src=\"https://github.com/Nayjest/lm-proxy/actions/workflows/code-style.yml/badge.svg\" alt=\"Code Style\"></a>\n <img src=\"https://raw.githubusercontent.com/Nayjest/lm-proxy/main/coverage.svg\" alt=\"Code Coverage\">\n <a href=\"https://github.com/Nayjest/lm-proxy/blob/main/LICENSE\"><img src=\"https://img.shields.io/github/license/Nayjest/lm-proxy?color=d08aff\" alt=\"License\"></a>\n</p>\n\nBuilt with Python, FastAPI and [MicroCore](https://github.com/Nayjest/ai-microcore), **LM-Proxy** seamlessly integrates cloud providers like Google, Anthropic, and OpenAI, as well as local PyTorch-based inference, while maintaining full compatibility with OpenAI's API format. \n\nIt works as a drop-in replacement for OpenAI's API, allowing you to switch between cloud providers and local models without modifying your existing client code. \n\n**LM-Proxy** supports **real-time token streaming**, **secure Virtual API key management**, and can be used both as an importable Python library and as a standalone HTTP service. Whether you're building production applications or experimenting with different models, LM-Proxy eliminates integration complexity and keeps your codebase **provider-agnostic**.\n\n\n## Table of Contents\n- [Overview](#lm-proxy)\n- [Features](#-features)\n- [Getting Started](#-getting-started)\n - [Installation](#installation)\n - [Quick Start](#quick-start)\n- [Configuration](#-configuration)\n - [Basic Structure](#basic-structure)\n - [Environment Variables](#environment-variables)\n- [Proxy API Keys vs. Provider API Keys](#-proxy-api-keys-vs-provider-api-keys)\n- [API Usage](#-api-usage)\n - [Chat Completions Endpoint](#chat-completions-endpoint)\n - [Models List Endpoint](#models-list-endpoint)\n- [User Groups Configuration](#-user-groups-configuration)\n - [Basic Group Definition](#basic-group-definition)\n - [Group-based Access Control](#group-based-access-control)\n - [Connection Restrictions](#connection-restrictions)\n - [Virtual API Key Validation](#virtual-api-key-validation)\n- [Advanced Usage](#%EF%B8%8F-advanced-usage)\n - [Dynamic Model Routing](#dynamic-model-routing)\n - [Load Balancing Example](#load-balancing-example)\n- [Debugging](#-debugging)\n- [Contributing](#-contributing)\n- [License](#-license)\n\n## \u2728 Features\n\n- **Provider Agnostic**: Connect to OpenAI, Anthropic, Google AI, local models, and more using a single API\n- **Unified Interface**: Access all models through the standard OpenAI API format\n- **Dynamic Routing**: Route requests to different LLM providers based on model name patterns\n- **Stream Support**: Full streaming support for real-time responses\n- **API Key Management**: Configurable API key validation and access control\n- **Easy Configuration**: Simple TOML configuration files for setup\n\n## \ud83d\ude80 Getting Started\n\n### Requirements\nPython 3.11 | 3.12 | 3.13\n\n### Installation\n\n```bash\npip install lm-proxy\n```\n\n### Quick Start\n\n#### 1. Create a `config.toml` file:\n\n```toml\nhost = \"0.0.0.0\"\nport = 8000\n\n[connections]\n[connections.openai]\napi_type = \"open_ai\"\napi_base = \"https://api.openai.com/v1/\"\napi_key = \"env:OPENAI_API_KEY\"\n\n[connections.anthropic]\napi_type = \"anthropic\"\napi_key = \"env:ANTHROPIC_API_KEY\"\n\n[routing]\n\"gpt*\" = \"openai.*\"\n\"claude*\" = \"anthropic.*\"\n\"*\" = \"openai.gpt-3.5-turbo\"\n\n[groups.default]\napi_keys = [\"YOUR_API_KEY_HERE\"]\n```\n> **Note**\n> To enhance security, consider storing upstream API keys in operating system environment variables rather than embedding them directly in the configuration file. You can reference these variables in the configuration using the env:<VAR_NAME> syntax.\n\n#### 2. Start the server:\n\n```bash\nlm-proxy\n```\nAlternatively, run it as a Python module:\n```bash\npython -m lm_proxy\n```\n\n#### 3. Use it with any OpenAI-compatible client:\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n api_key=\"YOUR_API_KEY_HERE\",\n base_url=\"http://localhost:8000/v1\"\n)\n\ncompletion = client.chat.completions.create(\n model=\"gpt-5\", # This will be routed to OpenAI based on config\n messages=[{\"role\": \"user\", \"content\": \"Hello, world!\"}]\n)\nprint(completion.choices[0].message.content)\n```\n\nOr use the same endpoint with Claude models:\n\n```python\ncompletion = client.chat.completions.create(\n model=\"claude-opus-4-1-20250805\", # This will be routed to Anthropic based on config\n messages=[{\"role\": \"user\", \"content\": \"Hello, world!\"}]\n)\n```\n\n## \ud83d\udcdd Configuration\n\nLM-Proxy is configured through a TOML file that specifies connections, routing rules, and access control.\n\n### Basic Structure\n\n```toml\nhost = \"0.0.0.0\" # Interface to bind to\nport = 8000 # Port to listen on\ndev_autoreload = false # Enable for development\n\n# API key validation function (optional)\napi_key_check = \"lm_proxy.api_key_check.check_api_key_in_config\"\n\n# LLM Provider Connections\n[connections]\n\n[connections.openai]\napi_type = \"open_ai\"\napi_base = \"https://api.openai.com/v1/\"\napi_key = \"env:OPENAI_API_KEY\"\n\n[connections.google]\napi_type = \"google_ai_studio\"\napi_key = \"env:GOOGLE_API_KEY\"\n\n[connections.anthropic]\napi_type = \"anthropic\"\napi_key = \"env:ANTHROPIC_API_KEY\"\n\n# Routing rules (model_pattern = \"connection.model\")\n[routing]\n\"gpt*\" = \"openai.*\" # Route all GPT models to OpenAI\n\"claude*\" = \"anthropic.*\" # Route all Claude models to Anthropic\n\"gemini*\" = \"google.*\" # Route all Gemini models to Google\n\"*\" = \"openai.gpt-3.5-turbo\" # Default fallback\n\n# Access control groups\n[groups.default]\napi_keys = [\n \"KEY1\",\n \"KEY2\"\n]\n\n# optional\n[[loggers]]\nclass = 'lm_proxy.loggers.BaseLogger'\n[loggers.log_writer]\nclass = 'lm_proxy.loggers.log_writers.JsonLogWriter'\nfile_name = 'storage/json.log'\n[loggers.entry_transformer]\nclass = 'lm_proxy.loggers.LogEntryTransformer'\ncompletion_tokens = \"response.usage.completion_tokens\"\nprompt_tokens = \"response.usage.prompt_tokens\"\nprompt = \"request.messages\"\nresponse = \"response\"\ngroup = \"group\"\nconnection = \"connection\"\napi_key_id = \"api_key_id\"\nremote_addr = \"remote_addr\"\ncreated_at = \"created_at\"\nduration = \"duration\"\n```\n\n### Environment Variables\n\nYou can reference environment variables in your configuration file by prefixing values with `env:`. \n\nFor example:\n\n```toml\n[connections.openai]\napi_key = \"env:OPENAI_API_KEY\"\n```\n\nAt runtime, LM-Proxy automatically retrieves the value of the target variable\n(OPENAI_API_KEY) from your operating system\u2019s environment or from a .env file, if present.\n\n### .env Files\n\nBy default, LM-Proxy looks for a `.env` file in the current working directory\nand loads environment variables from it.\n\nYou can refer to the [.env.template](https://github.com/Nayjest/lm-proxy/blob/main/.env.template)\n file for an example:\n```dotenv\nOPENAI_API_KEY=sk-u........\nGOOGLE_API_KEY=AI........\nANTHROPIC_API_KEY=sk-ant-api03--vE........\n\n# \"1\", \"TRUE\", \"YES\", \"ON\", \"ENABLED\", \"Y\", \"+\" are true, case-insensitive.\n# See https://github.com/Nayjest/ai-microcore/blob/v4.4.3/microcore/configuration.py#L36\nLM_PROXY_DEBUG=no\n```\n\nYou can also control `.env` file usage with the `--env` command-line option:\n\n```bash\n# Use a custom .env file path\nlm-proxy --env=\"path/to/your/.env\"\n# Disable .env loading\nlm-proxy --env=\"\"\n```\n\n## \ud83d\udd11 Proxy API Keys vs. Provider API Keys\n\nLM-Proxy utilizes two distinct types of API keys to facilitate secure and efficient request handling.\n\n- **Proxy API Key (Virtual API Key, Client API Key):** \nA unique key generated and managed within the LM-Proxy. \nClients use these keys to authenticate their requests to the proxy's API endpoints. \nEach Client API Key is associated with a specific group, which defines the scope of access and permissions for the client's requests. \nThese keys allow users to securely interact with the proxy without direct access to external service credentials.\n\n\n\n- **Provider API Key (Upstream API Key):**\nA key provided by external LLM inference providers (e.g., OpenAI, Anthropic, Mistral, etc.) and configured within the LM-Proxy. \nThe proxy uses these keys to authenticate and forward validated client requests to the respective external services. \nProvider API Keys remain hidden from end users, ensuring secure and transparent communication with provider APIs.\n\nThis distinction ensures a clear separation of concerns: \nVirtual API Keys manage user authentication and access within the proxy, \nwhile Upstream API Keys handle secure communication with external providers.\n\n## \ud83d\udd0c API Usage\n\nLM-Proxy implements the OpenAI chat completions API endpoint. You can use any OpenAI-compatible client to interact with it.\n\n### Chat Completions Endpoint\n\n```http\nPOST /v1/chat/completions\n```\n\n#### Request Format\n\n```json\n{\n \"model\": \"gpt-3.5-turbo\",\n \"messages\": [\n {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n {\"role\": \"user\", \"content\": \"What is the capital of France?\"}\n ],\n \"temperature\": 0.7,\n \"stream\": false\n}\n```\n\n#### Response Format\n\n```json\n{\n \"choices\": [\n {\n \"index\": 0,\n \"message\": {\n \"role\": \"assistant\",\n \"content\": \"The capital of France is Paris.\"\n },\n \"finish_reason\": \"stop\"\n }\n ]\n}\n```\n\n\n### Models List Endpoint\n\n\nList and describe all models available through the API.\n\n\n```http\nGET /v1/models\n```\n\nThe **LM-Proxy** dynamically builds the models list based on routing rules defined in `config.routing`. \nRouting keys can reference both **exact model names** and **model name patterns** (e.g., `\"gpt*\"`, `\"claude*\"`, etc.).\n\nBy default, wildcard patterns are displayed as-is in the models list (e.g., `\"gpt*\"`, `\"claude*\"`). \nThis behavior can be customized via the `model_listing_mode` configuration option:\n\n```\nmodel_listing_mode = \"as_is\" | \"ignore_wildcards\" | \"expand_wildcards\"\n```\n\nAvailable modes:\n\n- **`as_is`** *(default)* \u2014 Lists all entries exactly as defined in the routing configuration, including wildcard patterns. \n- **`ignore_wildcards`** \u2014 Excludes wildcard patterns, showing only explicitly defined model names. \n- **`expand_wildcards`** \u2014 Expands wildcard patterns by querying each connected backend for available models *(feature not yet implemented)*.\n\nTo obtain a complete and accurate model list in the current implementation,\nall supported models must be explicitly defined in the routing configuration, for example:\n```toml\n[routing]\n\"gpt-4\" = \"my_openai_connection.*\"\n\"gpt-5\" = \"my_openai_connection.*\"\n\"gpt-8\"= \"my_openai_connection.gpt-3.5-turbo\"\n\"claude-4.5-sonnet\" = \"my_anthropic_connection.claude-sonnet-4-5-20250929\"\n\"claude-4.1-opus\" = \"my_anthropic_connection.claude-opus-4-1-20250805\"\n[connections]\n[connections.my_openai_connection]\napi_type = \"open_ai\"\napi_base = \"https://api.openai.com/v1/\"\napi_key = \"env:OPENAI_API_KEY\"\n[connections.my_anthropic_connection]\napi_type = \"anthropic\"\napi_key = \"env:ANTHROPIC_API_KEY\"\n```\n\n\n\n#### Response Format\n\n```json\n{\n \"object\": \"list\",\n \"data\": [\n {\n \"id\": \"gpt-6\",\n \"object\": \"model\",\n \"created\": 1686935002,\n \"owned_by\": \"organization-owner\"\n },\n {\n \"id\": \"claude-5-sonnet\",\n \"object\": \"model\",\n \"created\": 1686935002,\n \"owned_by\": \"organization-owner\"\n }\n ]\n}\n```\n\n## \ud83d\udd12 User Groups Configuration\n\nThe `[groups]` section in the configuration defines access control rules for different user groups. \nEach group can have its own set of virtual API keys and permitted connections.\n\n### Basic Group Definition\n\n```toml\n[groups.default]\napi_keys = [\"KEY1\", \"KEY2\"]\nallowed_connections = \"*\" # Allow access to all connections\n```\n\n### Group-based Access Control\n\nYou can create multiple groups to segment your users and control their access:\n\n```toml\n# Admin group with full access\n[groups.admin]\napi_keys = [\"ADMIN_KEY_1\", \"ADMIN_KEY_2\"]\nallowed_connections = \"*\" # Access to all connections\n\n# Regular users with limited access\n[groups.users]\napi_keys = [\"USER_KEY_1\", \"USER_KEY_2\"]\nallowed_connections = \"openai,anthropic\" # Only allowed to use specific connections\n\n# Free tier with minimal access\n[groups.free]\napi_keys = [\"FREE_KEY_1\", \"FREE_KEY_2\"]\nallowed_connections = \"openai\" # Only allowed to use OpenAI connection\n```\n\n### Connection Restrictions\n\nThe `allowed_connections` parameter controls which upstream providers a group can access:\n\n- `\"*\"` - Group can use all configured connections\n- `\"openai,anthropic\"` - Comma-separated list of specific connections the group can use\n\nThis allows fine-grained control over which users can access which AI providers, enabling features like:\n\n- Restricting expensive models to premium users\n- Creating specialized access tiers for different user groups\n- Implementing usage quotas per group\n- Billing and cost allocation by user group\n\n### Virtual API Key Validation\n\n#### Overview\n\nLM-proxy includes 2 built-in methods for validating Virtual API keys:\n - `lm_proxy.api_key_check.check_api_key_in_config` - verifies API keys against those defined in the config file; used by default\n - `lm_proxy.api_key_check.CheckAPIKeyWithRequest` - validates API keys via an external HTTP service\n\nThe API key check method can be configured using the `api_key_check` configuration key. \nIts value can be either a reference to a Python function in the format `my_module.sub_module1.sub_module2.fn_name`,\nor an object containing parameters for a class-based validator. \n\nIn the .py config representation, the validator function can be passed directly as a callable.\n\n#### Example configuration for external API key validation using HTTP request to Keycloak / OpenID Connect\n\nThis example shows how to validate API keys against an external service (e.g., Keycloak):\n\n```toml\n[api_key_check]\nclass = \"lm_proxy.api_key_check.CheckAPIKeyWithRequest\"\nmethod = \"POST\"\nurl = \"http://keycloak:8080/realms/master/protocol/openid-connect/userinfo\"\nresponse_as_user_info = true # interpret response JSON as user info object for further processing / logging\nuse_cache = true # requires installing cachetools if True: pip install cachetools\ncache_ttl = 60 # Cache duration in seconds\n\n[api_key_check.headers]\nAuthorization = \"Bearer {api_key}\"\n```\n#### Custom API Key Validation / Extending functionality\n\nFor more advanced authentication needs,\nyou can implement a custom validator function:\n\n```python\n# my_validators.py\ndef validate_api_key(api_key: str) -> str | None:\n \"\"\"\n Validate an API key and return the group name if valid.\n \n Args:\n api_key: The API key to validate\n \n Returns:\n The name of the group if valid, None otherwise\n \"\"\"\n if api_key == \"secret-key\":\n return \"admin\"\n elif api_key.startswith(\"user-\"):\n return \"users\"\n return None\n```\n\nThen reference it in your config:\n\n```toml\napi_key_check = \"my_validators.validate_api_key\"\n```\n> **NOTE**\n> In this case, the `api_keys` lists in groups are ignored, and the custom function is responsible for all validation logic.\n\n\n## \ud83d\udee0\ufe0f Advanced Usage\n### Dynamic Model Routing\n\nThe routing section allows flexible pattern matching with wildcards:\n\n```toml\n[routing]\n\"gpt-4*\" = \"openai.gpt-4\" # Route gpt-4 requests to OpenAI GPT-4\n\"gpt-3.5*\" = \"openai.gpt-3.5-turbo\" # Route gpt-3.5 requests to OpenAI\n\"claude*\" = \"anthropic.*\" # Pass model name as-is to Anthropic\n\"gemini*\" = \"google.*\" # Pass model name as-is to Google\n\"custom*\" = \"local.llama-7b\" # Map any \"custom*\" to a specific local model\n\"*\" = \"openai.gpt-3.5-turbo\" # Default fallback for unmatched models\n```\nKeys are model name patterns (with `*` wildcard support), and values are connection/model mappings.\nConnection names reference those defined in the `[connections]` section.\n\n### Load Balancing Example\n\n- [Simple load-balancer configuration](https://github.com/Nayjest/lm-proxy/blob/main/examples/load_balancer_config.py) \n This example demonstrates how to set up a load balancer that randomly\ndistributes requests across multiple language model servers using the lm_proxy.\n\n## \ud83d\udd0d Debugging\n\n### Overview\nWhen **debugging mode** is enabled,\nLM-Proxy provides detailed logging information to help diagnose issues:\n- Stack traces for exceptions are shown in the console\n- Logging level is set to DEBUG instead of INFO\n\n> **Warning** \u26a0\ufe0f \n> Never enable debugging mode in production environments, as it may expose sensitive information to the application logs.\n\n### Enabling Debugging Mode\nTo enable debugging, set the `LM_PROXY_DEBUG` environment variable to a truthy value (e.g., \"1\", \"true\", \"yes\").\n> **Tip** \ud83d\udca1 \n> Environment variables can also be defined in a `.env` file.\n\nAlternatively, you can enable or disable debugging via the command-line arguments:\n- `--debug` to enable debugging\n- `--no-debug` to disable debugging\n\n> **Note** \u2139\ufe0f \n> CLI arguments override environment variable settings.\n\n\n## \ud83e\udd1d Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n1. Fork the repository\n2. Create your feature branch (`git checkout -b feature/amazing-feature`)\n3. Commit your changes (`git commit -m 'Add some amazing feature'`)\n4. Push to the branch (`git push origin feature/amazing-feature`)\n5. Open a Pull Request\n\n\n## \ud83d\udcc4 License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\u00a9 2025 Vitalii Stepanenko\n\n",
"bugtrack_url": null,
"license": "MIT License\n \n Copyright (c) 2025 Vitalii Stepanenko\n \n Permission is hereby granted, free of charge, to any person obtaining a copy\n of this software and associated documentation files (the \"Software\"), to deal\n in the Software without restriction, including without limitation the rights\n to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n copies of the Software, and to permit persons to whom the Software is\n furnished to do so, subject to the following conditions:\n \n The above copyright notice and this permission notice shall be included in all\n copies or substantial portions of the Software.\n \n THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n SOFTWARE.",
"summary": "\"LM-Proxy\" is OpenAI-compatible http proxy server for inferencing various LLMs capable of working with Google, Anthropic, OpenAI APIs, local PyTorch inference, etc.",
"version": "2.0.0",
"project_urls": {
"Source Code": "https://github.com/Nayjest/lm-proxy"
},
"split_keywords": [
"llm",
" large language models",
" ai",
" gpt",
" openai",
" proxy",
" http",
" proxy-server"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "4f63139a20e710e7f7d0b2f062d69aee06c77d354205f9af1457a882913a808a",
"md5": "f6fd6cf656c5f330bc17a0ad4d743f8c",
"sha256": "60be996becbf61a0afbafe9b81c705055007c591247fb81b96447770966aa918"
},
"downloads": -1,
"filename": "lm_proxy-2.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f6fd6cf656c5f330bc17a0ad4d743f8c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4,>=3.11",
"size": 25283,
"upload_time": "2025-10-26T17:36:28",
"upload_time_iso_8601": "2025-10-26T17:36:28.146810Z",
"url": "https://files.pythonhosted.org/packages/4f/63/139a20e710e7f7d0b2f062d69aee06c77d354205f9af1457a882913a808a/lm_proxy-2.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "2a60a496d17c1d818d5bede9d8b1907247402b3fc58e6756a6c804a91f0672f7",
"md5": "d210c7acbc97d23c6f464d3ccbe6de44",
"sha256": "152e1801accc2873eeccfecc3c4a19e738c2286c0bfa307287608e4fa9f3f30f"
},
"downloads": -1,
"filename": "lm_proxy-2.0.0.tar.gz",
"has_sig": false,
"md5_digest": "d210c7acbc97d23c6f464d3ccbe6de44",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4,>=3.11",
"size": 20458,
"upload_time": "2025-10-26T17:36:42",
"upload_time_iso_8601": "2025-10-26T17:36:42.115345Z",
"url": "https://files.pythonhosted.org/packages/2a/60/a496d17c1d818d5bede9d8b1907247402b3fc58e6756a6c804a91f0672f7/lm_proxy-2.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-10-26 17:36:42",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Nayjest",
"github_project": "lm-proxy",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "lm-proxy"
}
