# llm-litellm
[](https://pypi.org/project/llm-litellm/)
[](https://github.com/rajashekar/llm-litellm/releases)
[](https://github.com/rajashekar/llm-litellm/actions?query=workflow%3ATest)
[](https://github.com/rajashekar/llm-litellm/blob/main/LICENSE)
[LLM](https://llm.datasette.io/) plugin for models hosted by [LiteLLM](https://github.com/BerriAI/litellm) proxy server.
LiteLLM is a self-hosted proxy server that provides a unified interface to 100+ LLMs including OpenAI, Anthropic, Cohere, Replicate, PaLM, and more.
## Installation
First, [install the LLM command-line utility](https://llm.datasette.io/en/stable/setup.html).
Now install this plugin in the same environment as LLM:
```bash
llm install llm-litellm
```
### Updating to New Versions
To update the plugin to the latest version:
```bash
llm install --upgrade llm-litellm
```
Or using pip directly:
```bash
pip install --upgrade llm-litellm
```
### Installing from Source
For development or to install from source:
```bash
pip install -e .
```
## Configuration
### 1. Set up LiteLLM Server
First, you need to have a LiteLLM server running. You can set it up by:
```bash
pip install litellm[proxy]
litellm --model gpt-3.5-turbo
# This starts the server on http://localhost:4000
```
Or use Docker:
```bash
docker run -p 4000:4000 -e OPENAI_API_KEY=your-key ghcr.io/berriai/litellm:main-latest --model gpt-3.5-turbo
```
### 2. Set Environment Variable
Set the `LITELLM_URL` environment variable to point to your LiteLLM server:
```bash
export LITELLM_URL=http://localhost:4000
```
### 3. Set API Key (Optional)
If your LiteLLM server requires authentication, set the API key:
```bash
llm keys set litellm
# Enter your LiteLLM API key when prompted
```
## Usage
Once configured, you can use any model supported by your LiteLLM server.
### List Available Models
To see all available models:
```bash
llm models list
```
You should see models prefixed with `litellm:`:
```
litellm: gpt-3.5-turbo
litellm: gpt-4
litellm: claude-3-sonnet
...
```
You can also use the plugin-specific command:
```bash
llm litellm models
```
### Basic Usage
```bash
# Use a specific model
llm -m litellm/gpt-3.5-turbo "Hello, world!"
# Use with different models
llm -m litellm/claude-3-sonnet "Explain quantum computing"
llm -m litellm/gpt-4 "Write a short story"
```
### Advanced Usage
```bash
# Set temperature and other parameters
llm -m litellm/gpt-3.5-turbo -o temperature 0.9 -o max_tokens 500 "Be creative!"
# Use with streaming
llm -m litellm/gpt-4 "Write a long explanation" --stream
# Conversation mode
llm -m litellm/claude-3-sonnet -c "Let's discuss Python programming"
```
### Model Aliases
You can set shorter aliases for frequently used models:
```bash
llm aliases set gpt4 litellm/gpt-4
llm aliases set claude litellm/claude-3-sonnet
```
Now you can use:
```bash
llm -m gpt4 "Hello!"
llm -m claude "Explain this code" < script.py
```
## Plugin Commands
### Check Server Status
```bash
llm litellm status
```
This will check if your LiteLLM server is running and accessible.
### List Models
```bash
# Human-readable format
llm litellm models
# JSON format
llm litellm models --json
```
## Supported Models
The plugin supports any model that your LiteLLM server is configured to handle. Common models include:
- **OpenAI**: `gpt-4`, `gpt-3.5-turbo`, `gpt-4-turbo`
- **Anthropic**: `claude-3-opus`, `claude-3-sonnet`, `claude-3-haiku`
- **Google**: `gemini-pro`, `gemini-pro-vision`
- **Cohere**: `command-r`, `command-r-plus`
- **Meta**: `llama-2-70b`, `llama-2-13b`
- **Mistral**: `mistral-7b`, `mistral-medium`
- And many more...
Check your LiteLLM server configuration for the exact models available.
## Configuration Options
The plugin supports all standard LLM options:
- `temperature`: Controls randomness (0.0-2.0)
- `max_tokens`: Maximum tokens to generate
- `top_p`: Top-p sampling parameter
- `frequency_penalty`: Frequency penalty (-2.0 to 2.0)
- `presence_penalty`: Presence penalty (-2.0 to 2.0)
Example:
```bash
llm -m litellm/gpt-3.5-turbo \
-o temperature 0.7 \
-o max_tokens 1000 \
-o top_p 0.9 \
"Generate creative content"
```
## Troubleshooting
### Common Issues
1. **"LITELLM_URL environment variable is required"**
- Make sure you've set the `LITELLM_URL` environment variable
- Verify your LiteLLM server is running and accessible
2. **No models showing up**
- Check server status: `llm litellm status`
- Verify the URL is correct (should include protocol: http:// or https://)
- Test the server directly: `curl http://localhost:4000/health`
3. **Connection errors**
- Check that your LiteLLM server is running
- Verify firewall settings allow connections to the server
- Test with: `curl http://localhost:4000/v1/models`
4. **Authentication errors**
- If your LiteLLM server requires authentication, set the key: `llm keys set litellm`
- Check your LiteLLM server configuration for authentication requirements
5. **Model not found**
- Verify the model is configured in your LiteLLM server
- Check available models: `llm litellm models`
- Ensure the model name matches exactly
### Debug Mode
For debugging, you can check what models are available:
```bash
# Check server status
llm litellm status
# List all models
llm litellm models --json
# Test a simple query
llm -m litellm/gpt-3.5-turbo "test" -v
```
## Examples
### Setting up with Different Providers
#### OpenAI Models
```bash
export OPENAI_API_KEY=your-key
litellm --model gpt-3.5-turbo --model gpt-4
export LITELLM_URL=http://localhost:4000
llm -m litellm/gpt-3.5-turbo "Hello!"
```
#### Anthropic Models
```bash
export ANTHROPIC_API_KEY=your-key
litellm --model claude-3-sonnet --model claude-3-haiku
export LITELLM_URL=http://localhost:4000
llm -m litellm/claude-3-sonnet "Hello!"
```
#### Multiple Providers
```bash
export OPENAI_API_KEY=your-openai-key
export ANTHROPIC_API_KEY=your-anthropic-key
litellm --model gpt-4 --model claude-3-sonnet --model gemini-pro
export LITELLM_URL=http://localhost:4000
llm -m litellm/gpt-4 "Compare yourself to Claude"
```
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
```bash
cd llm-litellm
python3 -m venv venv
source venv/bin/activate
```
Install the plugin in development mode:
```bash
pip install -e '.[test]'
```
To run the tests:
```bash
pytest
```
## License
Apache License 2.0
Raw data
{
"_id": null,
"home_page": null,
"name": "llm-litellm",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "llm, ai, litellm, language-models, plugin",
"author": "Rajashekar Chintalapati",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/f2/85/91105ff2c1d4beb8ddc318425a0f88f8283606ddc93a783e5192f4e1051e/llm_litellm-0.2.0.tar.gz",
"platform": null,
"description": "# llm-litellm\n\n[](https://pypi.org/project/llm-litellm/)\n[](https://github.com/rajashekar/llm-litellm/releases)\n[](https://github.com/rajashekar/llm-litellm/actions?query=workflow%3ATest)\n[](https://github.com/rajashekar/llm-litellm/blob/main/LICENSE)\n\n[LLM](https://llm.datasette.io/) plugin for models hosted by [LiteLLM](https://github.com/BerriAI/litellm) proxy server.\n\nLiteLLM is a self-hosted proxy server that provides a unified interface to 100+ LLMs including OpenAI, Anthropic, Cohere, Replicate, PaLM, and more.\n\n## Installation\n\nFirst, [install the LLM command-line utility](https://llm.datasette.io/en/stable/setup.html).\n\nNow install this plugin in the same environment as LLM:\n```bash\nllm install llm-litellm\n```\n\n### Updating to New Versions\n\nTo update the plugin to the latest version:\n```bash\nllm install --upgrade llm-litellm\n```\n\nOr using pip directly:\n```bash\npip install --upgrade llm-litellm\n```\n\n### Installing from Source\n\nFor development or to install from source:\n```bash\npip install -e .\n```\n\n## Configuration\n\n### 1. Set up LiteLLM Server\n\nFirst, you need to have a LiteLLM server running. You can set it up by:\n\n```bash\npip install litellm[proxy]\nlitellm --model gpt-3.5-turbo\n# This starts the server on http://localhost:4000\n```\n\nOr use Docker:\n```bash\ndocker run -p 4000:4000 -e OPENAI_API_KEY=your-key ghcr.io/berriai/litellm:main-latest --model gpt-3.5-turbo\n```\n\n### 2. Set Environment Variable\n\nSet the `LITELLM_URL` environment variable to point to your LiteLLM server:\n\n```bash\nexport LITELLM_URL=http://localhost:4000\n```\n\n### 3. Set API Key (Optional)\n\nIf your LiteLLM server requires authentication, set the API key:\n\n```bash\nllm keys set litellm\n# Enter your LiteLLM API key when prompted\n```\n\n## Usage\n\nOnce configured, you can use any model supported by your LiteLLM server.\n\n### List Available Models\n\nTo see all available models:\n```bash\nllm models list\n```\n\nYou should see models prefixed with `litellm:`:\n```\nlitellm: gpt-3.5-turbo\nlitellm: gpt-4\nlitellm: claude-3-sonnet\n...\n```\n\nYou can also use the plugin-specific command:\n```bash\nllm litellm models\n```\n\n### Basic Usage\n\n```bash\n# Use a specific model\nllm -m litellm/gpt-3.5-turbo \"Hello, world!\"\n\n# Use with different models\nllm -m litellm/claude-3-sonnet \"Explain quantum computing\"\nllm -m litellm/gpt-4 \"Write a short story\"\n```\n\n### Advanced Usage\n\n```bash\n# Set temperature and other parameters\nllm -m litellm/gpt-3.5-turbo -o temperature 0.9 -o max_tokens 500 \"Be creative!\"\n\n# Use with streaming\nllm -m litellm/gpt-4 \"Write a long explanation\" --stream\n\n# Conversation mode\nllm -m litellm/claude-3-sonnet -c \"Let's discuss Python programming\"\n```\n\n### Model Aliases\n\nYou can set shorter aliases for frequently used models:\n\n```bash\nllm aliases set gpt4 litellm/gpt-4\nllm aliases set claude litellm/claude-3-sonnet\n```\n\nNow you can use:\n```bash\nllm -m gpt4 \"Hello!\"\nllm -m claude \"Explain this code\" < script.py\n```\n\n## Plugin Commands\n\n### Check Server Status\n\n```bash\nllm litellm status\n```\n\nThis will check if your LiteLLM server is running and accessible.\n\n### List Models\n\n```bash\n# Human-readable format\nllm litellm models\n\n# JSON format\nllm litellm models --json\n```\n\n## Supported Models\n\nThe plugin supports any model that your LiteLLM server is configured to handle. Common models include:\n\n- **OpenAI**: `gpt-4`, `gpt-3.5-turbo`, `gpt-4-turbo`\n- **Anthropic**: `claude-3-opus`, `claude-3-sonnet`, `claude-3-haiku`\n- **Google**: `gemini-pro`, `gemini-pro-vision`\n- **Cohere**: `command-r`, `command-r-plus`\n- **Meta**: `llama-2-70b`, `llama-2-13b`\n- **Mistral**: `mistral-7b`, `mistral-medium`\n- And many more...\n\nCheck your LiteLLM server configuration for the exact models available.\n\n## Configuration Options\n\nThe plugin supports all standard LLM options:\n\n- `temperature`: Controls randomness (0.0-2.0)\n- `max_tokens`: Maximum tokens to generate\n- `top_p`: Top-p sampling parameter\n- `frequency_penalty`: Frequency penalty (-2.0 to 2.0)\n- `presence_penalty`: Presence penalty (-2.0 to 2.0)\n\nExample:\n```bash\nllm -m litellm/gpt-3.5-turbo \\\n -o temperature 0.7 \\\n -o max_tokens 1000 \\\n -o top_p 0.9 \\\n \"Generate creative content\"\n```\n\n## Troubleshooting\n\n### Common Issues\n\n1. **\"LITELLM_URL environment variable is required\"**\n - Make sure you've set the `LITELLM_URL` environment variable\n - Verify your LiteLLM server is running and accessible\n\n2. **No models showing up**\n - Check server status: `llm litellm status`\n - Verify the URL is correct (should include protocol: http:// or https://)\n - Test the server directly: `curl http://localhost:4000/health`\n\n3. **Connection errors**\n - Check that your LiteLLM server is running\n - Verify firewall settings allow connections to the server\n - Test with: `curl http://localhost:4000/v1/models`\n\n4. **Authentication errors**\n - If your LiteLLM server requires authentication, set the key: `llm keys set litellm`\n - Check your LiteLLM server configuration for authentication requirements\n\n5. **Model not found**\n - Verify the model is configured in your LiteLLM server\n - Check available models: `llm litellm models`\n - Ensure the model name matches exactly\n\n### Debug Mode\n\nFor debugging, you can check what models are available:\n\n```bash\n# Check server status\nllm litellm status\n\n# List all models\nllm litellm models --json\n\n# Test a simple query\nllm -m litellm/gpt-3.5-turbo \"test\" -v\n```\n\n## Examples\n\n### Setting up with Different Providers\n\n#### OpenAI Models\n```bash\nexport OPENAI_API_KEY=your-key\nlitellm --model gpt-3.5-turbo --model gpt-4\nexport LITELLM_URL=http://localhost:4000\nllm -m litellm/gpt-3.5-turbo \"Hello!\"\n```\n\n#### Anthropic Models\n```bash\nexport ANTHROPIC_API_KEY=your-key\nlitellm --model claude-3-sonnet --model claude-3-haiku\nexport LITELLM_URL=http://localhost:4000\nllm -m litellm/claude-3-sonnet \"Hello!\"\n```\n\n#### Multiple Providers\n```bash\nexport OPENAI_API_KEY=your-openai-key\nexport ANTHROPIC_API_KEY=your-anthropic-key\nlitellm --model gpt-4 --model claude-3-sonnet --model gemini-pro\nexport LITELLM_URL=http://localhost:4000\nllm -m litellm/gpt-4 \"Compare yourself to Claude\"\n```\n\n## Development\n\nTo set up this plugin locally, first checkout the code. Then create a new virtual environment:\n\n```bash\ncd llm-litellm\npython3 -m venv venv\nsource venv/bin/activate\n```\n\nInstall the plugin in development mode:\n```bash\npip install -e '.[test]'\n```\n\nTo run the tests:\n```bash\npytest\n```\n\n## License\n\nApache License 2.0\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "LLM plugin for LiteLLM proxy server",
"version": "0.2.0",
"project_urls": {
"CI": "https://github.com/rajashekar/llm-litellm/actions",
"Changelog": "https://github.com/rajashekar/llm-litellm/releases",
"Homepage": "https://github.com/rajashekar/llm-litellm",
"Issues": "https://github.com/rajashekar/llm-litellm/issues"
},
"split_keywords": [
"llm",
" ai",
" litellm",
" language-models",
" plugin"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "01944c0777b629c2cdd83fb828b0eb825d51d904fb49e485f6d3998acc45045e",
"md5": "d67ac6c11d1c1ec688e181e6196e561a",
"sha256": "cd2f7b50b2c6008b3ba8326785badf52065b1807e5ebd3ae21497160f6850535"
},
"downloads": -1,
"filename": "llm_litellm-0.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d67ac6c11d1c1ec688e181e6196e561a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 10596,
"upload_time": "2025-07-15T03:43:45",
"upload_time_iso_8601": "2025-07-15T03:43:45.373092Z",
"url": "https://files.pythonhosted.org/packages/01/94/4c0777b629c2cdd83fb828b0eb825d51d904fb49e485f6d3998acc45045e/llm_litellm-0.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "f28591105ff2c1d4beb8ddc318425a0f88f8283606ddc93a783e5192f4e1051e",
"md5": "1909764e82e6b496c16b8e47ca39f94d",
"sha256": "5c0230850f57e50e376862a61568fe2be357eba627a846e12645279f32b608e6"
},
"downloads": -1,
"filename": "llm_litellm-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "1909764e82e6b496c16b8e47ca39f94d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 11566,
"upload_time": "2025-07-15T03:43:47",
"upload_time_iso_8601": "2025-07-15T03:43:47.106841Z",
"url": "https://files.pythonhosted.org/packages/f2/85/91105ff2c1d4beb8ddc318425a0f88f8283606ddc93a783e5192f4e1051e/llm_litellm-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-15 03:43:47",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "rajashekar",
"github_project": "llm-litellm",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "llm-litellm"
}
JSON
