# Token Vision
A fast, offline token calculator for images with various AI models (Claude, GPT-4o, Gemini). Calculate image tokens and costs without making API calls or hitting rate limits.
## Advantages
- **Offline Operation**: Calculate tokens without internet connectivity or API keys
- **No Rate Limits**: Process unlimited images without worrying about API quotas
- **Ultra Fast**: Get instant results without network latency
- **Cost Efficient**: Plan and estimate costs without spending API credits
- **Multi-Model Support**: One library for all major vision models
- **Accurate**: Uses the same tiling and scaling algorithms as the official implementations
- **Developer Friendly**: Clean API with type hints and comprehensive documentation
- **Extensible**: Support for custom models and pricing configurations
## Installation
```bash
pip install token-vision
```
## Quick Start
```python
from token_vision import ImageTokenCalculator
# Initialize calculator
calculator = ImageTokenCalculator()
# Calculate tokens for an image
tokens = calculator.calculate(
image_path="path/to/image.jpg",
model="gpt-4o"
)
# Get cost
cost = calculator.get_cost(tokens)
print(f"Tokens: {tokens:,}")
print(f"Cost: ${cost:.6f}")
```
## Features
- Support for multiple AI models:
- Claude (Sonnet, Haiku, Opus)
- GPT-4o (Latest, Nov 2024, Aug 2024, May 2024)
- Gemini (Pro, Flash)
- Accurate token calculation based on image dimensions
- Cost estimation for both input and output tokens
- Support for various image input formats
- Batch processing capabilities
- Memory efficient processing
- Custom model configuration support
## Documentation
### Supported Models
The library supports the following models with their respective token calculation methods:
#### Claude Models
- claude-3-sonnet
- claude-3-haiku
- claude-3-opus
#### OpenAI Models
- gpt-4o
- gpt-4o-2024-11-20
- gpt-4o-2024-08-06
- gpt-4o-2024-05-13
#### Google Models
- gemini-1.5-pro
- gemini-1.5-flash
### Custom Models
You can add your own models or override existing ones using a JSON configuration file:
```python
from token_vision import ImageTokenCalculator
from token_vision.models import load_custom_models
# Load custom models
load_custom_models("path/to/custom_models.json")
# Use custom model
calculator = ImageTokenCalculator()
tokens = calculator.calculate("image.jpg", model="custom-model-v1")
```
Example custom_models.json:
```json
{
"custom-provider": {
"name": "Custom Provider",
"max_images": 10,
"models": {
"custom-model-v1": {
"name": "Custom Model V1",
"input_rate": 0.001,
"output_rate": 0.005,
"batch_input_rate": 0.0005,
"batch_output_rate": 0.0025
}
}
}
}
```
### Advanced Usage
```python
# Custom configuration
calculator = ImageTokenCalculator(
default_model="gpt-4o",
detail_level="high"
)
# Process multiple images
results = calculator.calculate_batch([
"image1.jpg",
"image2.jpg"
])
# Using numpy arrays
import numpy as np
image_array = np.array(...) # Your image data
result = calculator.calculate(
image=image_array,
model="gpt-4o",
detail_level="low"
)
```
## Development
### Setup
1. Clone the repository:
```bash
git clone https://github.com/nerveband/token-vision.git
cd token-vision
```
2. Create a virtual environment:
```bash
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
```
3. Install development dependencies:
```bash
pip install -e ".[dev]"
```
### Running Tests
```bash
pytest
```
## License
This project is licensed under the MIT License - see the LICENSE file for details.
Raw data
{
"_id": null,
"home_page": null,
"name": "token-vision",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "AI, Claude, GPT-4, Gemini, image processing, tokenizer, tokens, vision",
"author": "nerveband",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/2b/bc/5e50d29e500af13de6b7d604cf1279fadbb78c148c3f7429bc1069f56e5a/token_vision-0.1.0.tar.gz",
"platform": null,
"description": "# Token Vision\n\nA fast, offline token calculator for images with various AI models (Claude, GPT-4o, Gemini). Calculate image tokens and costs without making API calls or hitting rate limits.\n\n## Advantages\n\n- **Offline Operation**: Calculate tokens without internet connectivity or API keys\n- **No Rate Limits**: Process unlimited images without worrying about API quotas\n- **Ultra Fast**: Get instant results without network latency\n- **Cost Efficient**: Plan and estimate costs without spending API credits\n- **Multi-Model Support**: One library for all major vision models\n- **Accurate**: Uses the same tiling and scaling algorithms as the official implementations\n- **Developer Friendly**: Clean API with type hints and comprehensive documentation\n- **Extensible**: Support for custom models and pricing configurations\n\n## Installation\n\n```bash\npip install token-vision\n```\n\n## Quick Start\n\n```python\nfrom token_vision import ImageTokenCalculator\n\n# Initialize calculator\ncalculator = ImageTokenCalculator()\n\n# Calculate tokens for an image\ntokens = calculator.calculate(\n image_path=\"path/to/image.jpg\",\n model=\"gpt-4o\"\n)\n\n# Get cost\ncost = calculator.get_cost(tokens)\nprint(f\"Tokens: {tokens:,}\")\nprint(f\"Cost: ${cost:.6f}\")\n```\n\n## Features\n\n- Support for multiple AI models:\n - Claude (Sonnet, Haiku, Opus)\n - GPT-4o (Latest, Nov 2024, Aug 2024, May 2024)\n - Gemini (Pro, Flash)\n- Accurate token calculation based on image dimensions\n- Cost estimation for both input and output tokens\n- Support for various image input formats\n- Batch processing capabilities\n- Memory efficient processing\n- Custom model configuration support\n\n## Documentation\n\n### Supported Models\n\nThe library supports the following models with their respective token calculation methods:\n\n#### Claude Models\n- claude-3-sonnet\n- claude-3-haiku\n- claude-3-opus\n\n#### OpenAI Models\n- gpt-4o\n- gpt-4o-2024-11-20\n- gpt-4o-2024-08-06\n- gpt-4o-2024-05-13\n\n#### Google Models\n- gemini-1.5-pro\n- gemini-1.5-flash\n\n### Custom Models\n\nYou can add your own models or override existing ones using a JSON configuration file:\n\n```python\nfrom token_vision import ImageTokenCalculator\nfrom token_vision.models import load_custom_models\n\n# Load custom models\nload_custom_models(\"path/to/custom_models.json\")\n\n# Use custom model\ncalculator = ImageTokenCalculator()\ntokens = calculator.calculate(\"image.jpg\", model=\"custom-model-v1\")\n```\n\nExample custom_models.json:\n```json\n{\n \"custom-provider\": {\n \"name\": \"Custom Provider\",\n \"max_images\": 10,\n \"models\": {\n \"custom-model-v1\": {\n \"name\": \"Custom Model V1\",\n \"input_rate\": 0.001,\n \"output_rate\": 0.005,\n \"batch_input_rate\": 0.0005,\n \"batch_output_rate\": 0.0025\n }\n }\n }\n}\n```\n\n### Advanced Usage\n\n```python\n# Custom configuration\ncalculator = ImageTokenCalculator(\n default_model=\"gpt-4o\",\n detail_level=\"high\"\n)\n\n# Process multiple images\nresults = calculator.calculate_batch([\n \"image1.jpg\",\n \"image2.jpg\"\n])\n\n# Using numpy arrays\nimport numpy as np\nimage_array = np.array(...) # Your image data\nresult = calculator.calculate(\n image=image_array,\n model=\"gpt-4o\",\n detail_level=\"low\"\n)\n```\n\n## Development\n\n### Setup\n\n1. Clone the repository:\n```bash\ngit clone https://github.com/nerveband/token-vision.git\ncd token-vision\n```\n\n2. Create a virtual environment:\n```bash\npython -m venv venv\nsource venv/bin/activate # On Windows: venv\\Scripts\\activate\n```\n\n3. Install development dependencies:\n```bash\npip install -e \".[dev]\"\n```\n\n### Running Tests\n\n```bash\npytest\n```\n\n## License\n\nThis project is licensed under the MIT License - see the LICENSE file for details. ",
"bugtrack_url": null,
"license": null,
"summary": "A fast, offline token calculator for images with various AI models (Claude, GPT-4V, Gemini)",
"version": "0.1.0",
"project_urls": {
"Bug Tracker": "https://github.com/nerveband/token-vision/issues",
"Homepage": "https://github.com/nerveband/token-vision",
"Repository": "https://github.com/nerveband/token-vision.git"
},
"split_keywords": [
"ai",
" claude",
" gpt-4",
" gemini",
" image processing",
" tokenizer",
" tokens",
" vision"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a3f34fd45ba6642f09394848303b9fdd50e5cc980bb06cd157c00bf1ddc08d69",
"md5": "0eb28bbcfde266fcc8f35e68cfca26e9",
"sha256": "6d3ba64624a1f0729c0a40d430fe60bf8cd589b65483130ed5cd3d1aa187ad9e"
},
"downloads": -1,
"filename": "token_vision-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0eb28bbcfde266fcc8f35e68cfca26e9",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 7433,
"upload_time": "2025-01-02T19:34:58",
"upload_time_iso_8601": "2025-01-02T19:34:58.907768Z",
"url": "https://files.pythonhosted.org/packages/a3/f3/4fd45ba6642f09394848303b9fdd50e5cc980bb06cd157c00bf1ddc08d69/token_vision-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "2bbc5e50d29e500af13de6b7d604cf1279fadbb78c148c3f7429bc1069f56e5a",
"md5": "26023fb845c1dfa5fdaa29ca6a6f4a5d",
"sha256": "1945e7731ba950ff77f18de88417bedb813530bc9f08535e985eaa8c581a4257"
},
"downloads": -1,
"filename": "token_vision-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "26023fb845c1dfa5fdaa29ca6a6f4a5d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 45045090,
"upload_time": "2025-01-02T19:35:06",
"upload_time_iso_8601": "2025-01-02T19:35:06.636301Z",
"url": "https://files.pythonhosted.org/packages/2b/bc/5e50d29e500af13de6b7d604cf1279fadbb78c148c3f7429bc1069f56e5a/token_vision-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-02 19:35:06",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "nerveband",
"github_project": "token-vision",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "numpy",
"specs": [
[
">=",
"1.24.0"
]
]
},
{
"name": "pillow",
"specs": [
[
">=",
"10.0.0"
]
]
}
],
"lcname": "token-vision"
}
JSON
