# HF VRAM Calculator
[](https://www.python.org/downloads/)
[](https://opensource.org/licenses/MIT)
[](https://github.com/astral-sh/uv)
A professional Python CLI tool for estimating GPU memory requirements for Hugging Face models with different data types and parallelization strategies.
> **⚡ Latest Features**: Smart dtype detection, MHA/MQA/GQA-aware KV cache, 12 quantization formats, 20+ GPU models, professional Rich UI
## Quick Demo
```bash
# Install and run
pip install hf-vram-calc
# Set up authentication (required for most models)
hf auth login --token yourtoken --add-to-git-credential
# Calculate memory requirements
hf-vram-calc microsoft/DialoGPT-medium
# Output: Beautiful tables showing 0.9GB inference, GPU compatibility, parallelization strategies
```
## Features
- 🔍 **Automatic Model Analysis**: Fetch configurations from Hugging Face Hub automatically
- 🧠 **Smart Data Type Detection**: Intelligent dtype recommendation from model names, config, or defaults
- 📊 **Comprehensive Data Type Support**: fp32, fp16, bf16, fp8, int8, int4, mxfp4, nvfp4, awq_int4, gptq_int4, nf4, fp4
- 🎯 **Multi-Scenario Memory Estimation**:
- **Inference**: Model weights + KV cache overhead (MHA/MQA/GQA-aware, ×1.2 factor)
- **Training**: Full Adam optimizer states (×4×1.3 factors)
- **LoRA Fine-tuning**: Low-rank adaptation with trainable parameter overhead
- ⚡ **Advanced Parallelization Analysis**:
- Tensor Parallelism (TP): 1, 2, 4, 8
- Pipeline Parallelism (PP): 1, 2, 4, 8
- Expert Parallelism (EP) for MoE models
- Data Parallelism (DP): 2, 4, 8
- Combined strategies (TP + PP combinations)
- 🎮 **GPU Compatibility Matrix**:
- 20+ GPU models (RTX 4090, A100, H100, L40S, etc.)
- Automatic compatibility checking for inference/training/LoRA
- Minimum GPU memory requirement calculations
- 📈 **Professional Rich UI**:
- 🎨 Beautiful color-coded tables and panels
- 📊 Real-time progress indicators
- 🚀 Modern CLI interface with emoji icons
- 💡 Smart recommendations and warnings
- 🔧 **Flexible Configuration**:
- Customizable LoRA rank, batch size, sequence length
- External JSON configuration files
- User-defined GPU models and data types
- 📋 **Parameter Display**: Raw count + human-readable format (e.g., "405,016,576 (405.0M)")
## Installation
### Quick Install (from PyPI)
```bash
pip install hf-vram-calc
```
### Build from Source
```bash
# Clone the repository
git clone <repository-url>
cd hf-vram-calc
# Build with uv (recommended)
uv build
uv pip install dist/hf_vram_calc-1.0.0-py3-none-any.whl
# Or install directly
uv pip install .
```
> **Dependencies**: `requests` (HTTP), `rich` (beautiful CLI), Python ≥3.8
For detailed build instructions, see: [BUILD.md](BUILD.md)
## Authentication Setup
Many models require a Hugging Face token. Get yours at https://huggingface.co/settings/tokens, then:
```bash
hf auth login --token yourtoken --add-to-git-credential
```
## Usage
### Basic Usage - Smart Dtype Detection
```bash
# Automatic dtype recommendation from model config/name
hf-vram-calc --model mistralai/Mistral-7B-v0.1
```
### Specify Data Type Override
```bash
# Override with specific data type
hf-vram-calc --model mistralai/Mistral-7B-v0.1 --dtype bf16
hf-vram-calc --model mistralai/Mistral-7B-v0.1 --dtype bf16,fp8
```
### Advanced Configuration
```bash
# Custom batch size and sequence length
hf-vram-calc --model mistralai/Mistral-7B-v0.1 --max_batch_size 4 --max_seq_len 4096
# Custom LoRA rank for fine-tuning estimation
hf-vram-calc --model mistralai/Mistral-7B-v0.1 --lora_rank 128
# Detailed analysis (disabled by default)
hf-vram-calc --model mistralai/Mistral-7B-v0.1 --log_level verbose
```
### YAML Configuration
```bash
# Use YAML configuration file (trtllm-bench compatible)
hf-vram-calc --extra_llm_api_options example_config.yaml
# Override YAML with command line arguments
hf-vram-calc --extra_llm_api_options example_config.yaml --max_batch_size 128
```
### JSON Output
```bash
# Save results to JSON file
hf-vram-calc --model mistralai/Mistral-7B-v0.1 --dtype bf16,fp8 --output_json results.json
```
### System Information
```bash
# List all available data types and GPU models
hf-vram-calc --list_types
# Use custom configuration directory
hf-vram-calc --config_dir ./my_config --model mistralai/Mistral-7B-v0.1
# Show help
hf-vram-calc --help
```
## Command Line Arguments
### Required
- `--model MODEL`: Hugging Face model name (e.g., `mistralai/Mistral-7B-v0.1`)
### Data Type Control
- `--dtype {fp32,fp16,bf16,fp8,int8,int4,mxfp4,nvfp4,awq_int4,fp4,nf4,gptq_int4}`: Override automatic dtype detection
- `--list_types`: List all available data types and GPU models
### Memory Estimation Parameters
- `--max_batch_size BATCH_SIZE`: Batch size for activation estimation (default: 1)
- `--max_seq_len SEQUENCE_LENGTH`: Sequence length for memory calculation (default: 2048)
- `--lora_rank LORA_RANK`: LoRA rank for fine-tuning estimation (default: 64)
### Parallelization Settings
- `--tp TP`: Tensor parallelism size (default: 1)
- `--pp PP`: Pipeline parallelism size (default: 1)
- `--ep EP`: Expert parallelism size (default: 1)
### Configuration & Output
- `--model_path MODEL_PATH`: Path to local model directory containing config.json
- `--extra_llm_api_options YAML_FILE`: Path to YAML configuration file (trtllm-bench compatible)
- `--output_json JSON_FILE`: Path to save results as JSON file
- `--log_level {info,verbose}`: Log level for output (default: info)
- `--config_dir CONFIG_DIR`: Custom configuration directory path
- `--help`: Show complete help message with examples
### Smart Behavior
- **No `--dtype`**: Uses intelligent priority (model name → config → fp16 default)
- **With `--dtype`**: Overrides automatic detection with specified type
- **YAML + CLI**: Command line arguments override YAML configuration
- **Invalid model**: Graceful error handling with helpful suggestions
## Quick Start Examples
```bash
# Set up authentication first time
hf auth login --token yourtoken --add-to-git-credential
# Estimate memory for different models
hf-vram-calc --model mistralai/Mistral-7B-v0.1 # → ~14GB inference (BF16)
hf-vram-calc --model mistralai/Mistral-7B-v0.1 --dtype fp16 # → ~14GB inference (FP16)
hf-vram-calc --model mistralai/Mistral-7B-v0.1 --dtype fp8 # → ~7GB inference (FP8)
# estimate size for specified quantization versions
hf-vram-calc --model mistralai/Mistral-7B-v0.1 --dtype fp16 # → ~14GB
hf-vram-calc --model mistralai/Mistral-7B-v0.1 --dtype int4 # → ~3.5GB
hf-vram-calc --model mistralai/Mistral-7B-v0.1 --dtype awq_int4 # → ~3.5GB
# for private access models, it is recommended to use --model_path
hf-vram-calc --model mistralai/Mistral-7B-v0.1 --model_path /llm_data/llm-models/Mistral-7B-v0.1
# Find optimal parallelization strategy
hf-vram-calc --model mistralai/Mistral-7B-v0.1 --log_level verbose # → TP/PP recommendations
# Save results to JSON
hf-vram-calc --model mistralai/Mistral-7B-v0.1 --output_json results.json
# Use YAML configuration (trtllm-bench compatible)
hf-vram-calc --extra_llm_api_options config.yaml
# Check what's available
hf-vram-calc --list_types # → All types & GPUs
```
## Data Type Priority & Detection
### Automatic Data Type Recommendation
The tool uses intelligent priority-based dtype selection:
1. **Model Name Detection** (Highest Priority)
- `model-fp16`, `model-bf16` → Extracts from model name
- `model-4bit`, `model-gptq`, `model-awq` → Detects quantization
2. **Config torch_dtype** (Medium Priority)
- Reads `torch_dtype` from model's `config.json`
- Maps `torch.float16` → `fp16`, `torch.bfloat16` → `bf16`, etc.
3. **Default Fallback** (Lowest Priority)
- Defaults to `fp16` when no dtype detected
### Supported Data Types
| Data Type | Bytes/Param | Description | Detection Patterns |
|-----------|-------------|-------------|--------------------|
| **fp32** | 4.0 | 32-bit floating point | `fp32`, `float32` |
| **fp16** | 2.0 | 16-bit floating point | `fp16`, `float16`, `half` |
| **bf16** | 2.0 | Brain Float 16 | `bf16`, `bfloat16` |
| **fp8** | 1.0 | 8-bit floating point | `fp8`, `float8` |
| **int8** | 1.0 | 8-bit integer | `int8`, `8bit` |
| **int4** | 0.5 | 4-bit integer | `int4`, `4bit` |
| **mxfp4** | 0.5 | Microsoft FP4 | `mxfp4` |
| **nvfp4** | 0.5 | NVIDIA FP4 | `nvfp4` |
| **awq_int4** | 0.5 | AWQ 4-bit quantization | `awq`, `awq-int4` |
| **gptq_int4** | 0.5 | GPTQ 4-bit quantization | `gptq`, `gptq-int4` |
| **nf4** | 0.5 | 4-bit NormalFloat | `nf4`, `bnb-4bit` |
| **fp4** | 0.5 | 4-bit floating point | `fp4` |
## YAML Configuration (trtllm-bench Compatible)
The `--extra_llm_api_options` argument allows you to use YAML configuration files with the same hierarchical structure as trtllm-bench:
```yaml
# config.yaml
model: "mistralai/Mistral-7B-v0.1"
kv_cache_config:
dtype: "fp8"
mamba_ssm_cache_dtype: "fp16"
enable_chunked_prefill: true
build_config:
max_batch_size: 64
max_num_tokens: 8192
max_seq_len: 4096
quant_config:
quant_algo: "fp8"
kv_cache_quant_algo: "fp8"
lora_config:
lora_dir: "/path/to/lora/weights"
max_lora_rank: 16
performance_options:
cuda_graphs: true
multi_block_mode: true
log_level: "verbose"
```
### YAML Section Mappings
- `build_config.max_batch_size` → `--max_batch_size`
- `build_config.max_seq_len` → `--max_seq_len`
- `lora_config.max_lora_rank` → `--lora_rank`
- `kv_cache_config.dtype` → `--dtype`
- `quant_config.quant_algo` → `--dtype` (with algorithm-to-dtype mapping)
## JSON Output
The `--output_json` argument saves calculation results in a simplified JSON format:
```json
{
"model": {
"name": "mistralai/Mistral-7B-v0.1",
"architecture": "mistral",
"parameters": 7241732096,
"parameters_formatted": "7.24B",
"original_torch_dtype": "torch.bfloat16",
"user_specified_dtype": "FP8,BF16"
},
"memory_requirements": [
{
"dtype": "FP8",
"batch_size": 1,
"sequence_length": 2048,
"lora_rank": 64,
"model_size_gib": 6.75,
"kv_cache_size_gib": 0.13,
"inference_total_gib": 8.10,
"training_gib": 35.07,
"lora_size_gib": 8.37
},
{
"dtype": "BF16",
"batch_size": 1,
"sequence_length": 2048,
"lora_rank": 64,
"model_size_gib": 13.49,
"kv_cache_size_gib": 0.25,
"inference_total_gib": 16.19,
"training_gib": 70.14,
"lora_size_gib": 16.73
}
]
}
```
## Parallelization Strategies
### Tensor Parallelism (TP)
Splits model weights by tensor dimensions across multiple GPUs.
### Pipeline Parallelism (PP)
Distributes different model layers to different GPUs.
### Expert Parallelism (EP)
For MoE (Mixture of Experts) models, distributes expert networks to different GPUs.
### Data Parallelism (DP)
Each GPU holds a complete model copy, only splitting data.
## Example Output
### Smart Dtype Detection Example
```bash
$ hf-vram-calc --model mistralai/Mistral-7B-v0.1 --log_level verbose
```
```
Using recommended data type: FP16
Use --dtype to specify different type, or see --list_types for all options
🔍 Fetching configuration for mistralai/Mistral-7B-v0.1...
Using recommended data type: FP16
Use --dtype to specify different type, or see --list_types for all options
📋 Parsing model configuration...
🧮 Calculating model parameters...
💾 Computing memory requirements...
╭─────── 🤖 Model Information ───────╮
│ │
│ Model: mistralai/Mistral-7B-v0.1 │
│ Architecture: mistral │
│ Parameters: 7,241,732,096 (7.24B) │
│ Recommended dtype: FP16 │
│ │
╰────────────────────────────────────╯
💾 Memory Requirements by Data Type and Scenario
╭──────────────┬──────────────┬─────────────────┬─────────────────┬─────────────────┬──────────────╮
│ │ Model Size │ KV Cache │ Inference │ Training │ LoRA │
│ Data Type │ (GB) │ (GB) │ Total (GB) │ (Adam) (GB) │ (GB) │
├──────────────┼──────────────┼─────────────────┼─────────────────┼─────────────────┼──────────────┤
│ FP16 │ 0.76 │ 0.19 │ 0.91 │ 3.94 │ 0.94 │
╰──────────────┴──────────────┴─────────────────┴─────────────────┴─────────────────┴──────────────╯
================================================================================
⚡ Parallelization Strategies (FP16 Inference)
╔════════════════════╤══════╤══════╤══════╤══════╤══════════════╤══════════════╗
║ │ │ │ │ │ Memory/GPU │ Min GPU ║
║ Strategy │ TP │ PP │ EP │ DP │ (GB) │ Required ║
╟────────────────────┼──────┼──────┼──────┼──────┼──────────────┼──────────────╢
║ Single GPU │ 1 │ 1 │ 1 │ 1 │ 0.91 │ 4GB+ ║
║ Tensor Parallel │ 2 │ 1 │ 1 │ 1 │ 0.45 │ 4GB+ ║
║ TP + PP │ 4 │ 4 │ 1 │ 1 │ 0.06 │ 4GB+ ║
╚════════════════════╧══════╧══════╧══════╧══════╧══════════════╧══════════════╝
🎮 GPU Compatibility Matrix
┏━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━┯━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━┓
┃ GPU Type │ Memory │ Inference │ Training │ LoRA ┃
┠─────────────────┼────────────┼──────────────┼──────────────┼──────────────┨
┃ RTX 4090 │ 24GB │ ✓ │ ✓ │ ✓ ┃
┃ A100 80GB │ 80GB │ ✓ │ ✓ │ ✓ ┃
┃ H100 80GB │ 80GB │ ✓ │ ✓ │ ✓ ┃
┗━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━┷━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━┛
╭─── 📋 Minimum GPU Requirements ────╮
│ │
│ Single GPU Inference: 0.9GB │
│ Single GPU Training: 3.9GB │
│ Single GPU LoRA: 0.9GB │
│ │
╰───────────────────────────────────╯
```
### Large Model with User Override
```bash
$ hf-vram-calc nvidia/DeepSeek-R1-0528-FP4 --dtype nvfp4
$ hf-vram-calc Qwen/Qwen-72B-Chat
```
```
╭──────── 🤖 Model Information ────────╮
│ │
│ Model: nvidia/DeepSeek-R1-0528-FP4 │
│ Architecture: deepseek_v3 │
│ Parameters: 30,510,606,336 (30.5B) │
│ Original torch_dtype: bfloat16 │
│ User specified dtype: NVFP4 │
│ │
╰──────────────────────────────────────╯
💾 Memory Requirements by Data Type and Scenario
╭──────────────┬──────────────┬──────────────┬─────────────────┬──────────────╮
│ │ Total Size │ Inference │ Training │ LoRA │
│ Data Type │ (GB) │ (GB) │ (Adam) (GB) │ (GB) │
├──────────────┼──────────────┼──────────────┼─────────────────┼──────────────┤
│ NVFP4 │ 14.21 │ 17.05 │ 73.88 │ 19.34 │
╰──────────────┴──────────────┴──────────────┴─────────────────┴──────────────╯
```
### List Available Types
```bash
$ hf-vram-calc --list_types
```
```
Available Data Types:
╭───────────┬─────────────┬────────────────────────╮
│ Data Type │ Bytes/Param │ Description │
├───────────┼─────────────┼────────────────────────┤
│ FP32 │ 4 │ 32-bit floating point │
│ FP16 │ 2 │ 16-bit floating point │
│ BF16 │ 2 │ Brain Float 16 │
│ NVFP4 │ 0.5 │ NVIDIA FP4 │
│ AWQ_INT4 │ 0.5 │ AWQ 4-bit quantization │
│ GPTQ_INT4 │ 0.5 │ GPTQ 4-bit quantization│
╰───────────┴─────────────┴────────────────────────╯
Available GPU Types:
╭───────────────────┬─────────────┬────────────┬──────────────╮
│ GPU Name │ Memory (GB) │ Category │ Architecture │
├───────────────────┼─────────────┼────────────┼──────────────┤
│ RTX 4090 │ 24 │ consumer │ Ada Lovelace │
│ A100 80GB │ 80 │ datacenter │ Ampere │
│ H100 80GB │ 80 │ datacenter │ Hopper │
╰───────────────────┴─────────────┴────────────┴──────────────╯
```
## Calculation Formulas
### Inference Memory
```
Inference Memory = Model Weights × 1.2
```
Includes model weights and KV cache overhead.
### KV Cache Memory
```
KV Cache (GB) = 2 × Batch_Size × Sequence_Length × Head_Dim × Num_KV_Heads × Num_Layers × Precision ÷ 1,073,741,824
```
- Head_Dim = hidden_size ÷ num_attention_heads
- Num_KV_Heads = config.num_key_value_heads (if present) else num_attention_heads
- Automatically supports MHA, MQA, and GQA via model config; KV cache uses FP16/BF16 for quantized models
### Training Memory (with Adam)
```
Training Memory = Model Weights × 4 × 1.3
```
- 4x factor: Model weights (1x) + Gradients (1x) + Adam optimizer states (2x)
- 1.3x factor: 30% additional overhead (activation caching, etc.)
### LoRA Fine-tuning Memory
```
LoRA Memory = (Model Weights + LoRA Parameter Overhead) × 1.2
```
LoRA parameter overhead calculated based on rank and target module ratio.
## Advanced Features
### Configuration System
External JSON configuration files for maximum flexibility:
- **`data_types.json`** - Add custom quantization formats
- **`gpu_types.json`** - Define new GPU models and specifications
- **`display_settings.json`** - Customize UI appearance and limits
```bash
# Use custom config directory
hf-vram-calc --config-dir ./custom_config model_name
# Add custom data type example (data_types.json)
{
"my_custom_int2": {
"bytes_per_param": 0.25,
"description": "Custom 2-bit quantization"
}
}
```
### Memory Calculation Details
| Scenario | Formula | Explanation |
|----------|---------|-------------|
| **Inference** | Model × 1.2 | Includes KV cache and activation overhead |
| **Training** | Model × 4 × 1.3 | Weights(1x) + Gradients(1x) + Adam(2x) + 30% overhead |
| **LoRA** | (Model + LoRA_params×4) × 1.2 | Base model + trainable parameters with optimizer |
### Parallelization Efficiency
- **TP (Tensor Parallel)**: Near-linear scaling, slight communication overhead
- **PP (Pipeline Parallel)**: Good efficiency, pipeline bubble ~10-15%
- **EP (Expert Parallel)**: MoE-specific, depends on expert routing efficiency
- **DP (Data Parallel)**: No memory reduction per GPU, full model replica
## Supported Architectures
### Fully Supported ✅
- **GPT Family**: GPT-2, GPT-3, GPT-4, GPT-NeoX, etc.
- **LLaMA Family**: LLaMA, LLaMA-2, Code Llama, Vicuna, etc.
- **Mistral Family**: Mistral 7B, Mixtral 8x7B (MoE), etc.
- **Other Transformers**: BERT, RoBERTa, T5, FLAN-T5, etc.
- **New Architectures**: DeepSeek, Qwen, ChatGLM, Baichuan, etc.
### Architecture Detection
- **Automatic field mapping** for different config.json formats
- **Fallback support** for uncommon architectures
- **MoE handling** for Mixture-of-Experts models
## Accuracy & Limitations
### ✅ Highly Accurate For:
- **Parameter counting** (exact calculation)
- **Memory estimation** (within 5-10% of actual)
- **Parallelization ratios** (theoretical maximum)
### ⚠️ Considerations:
- **Activation memory** varies with sequence length and optimization
- **Real-world efficiency** may differ due to framework overhead
- **Quantization accuracy** depends on specific implementation
- **MoE models** require expert routing consideration
## Build & Development
Built with modern Python tooling:
- **uv**: Fast Python package management and building
- **Rich**: Professional terminal interface
- **Requests**: HTTP client for model config fetching
- **JSON configuration**: Flexible external configuration system
For development setup, see: [BUILD.md](BUILD.md)
## Contributing
We welcome contributions! Areas for improvement:
- 🔧 **New quantization formats** (add to `data_types.json`)
- 🎮 **GPU models** (update `gpu_types.json`)
- 📊 **Architecture support** (enhance config parsing)
- 🚀 **Performance optimizations**
- 📚 **Documentation improvements**
- 🧪 **Test coverage expansion**
## See Also
- 📚 **[BUILD.md](BUILD.md)** - Complete build and installation guide
- ⚙️ **[CONFIG_GUIDE.md](CONFIG_GUIDE.md)** - Configuration customization details
- 📝 **Examples in help**: `hf-vram-calc --help` for usage examples
## Version History
- **v1.0.0**: Complete rewrite with uv build, smart dtype detection, professional UI
- **v0.x**: Legacy single-file version (deprecated)
## License
MIT License - see LICENSE file for details.
---
**Made with ❤️ for the ML community** | Built with [uv](https://github.com/astral-sh/uv) and [Rich](https://github.com/Textualize/rich)
Raw data
{
"_id": null,
"home_page": null,
"name": "hf-vram-calc",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "ai, calculator, gpu, huggingface, memory, ml, transformer, vram",
"author": null,
"author_email": "HF VRAM Calculator Contributors <hf-vram-calc@example.com>",
"download_url": "https://files.pythonhosted.org/packages/d8/49/de6c0aaa512ad3b5f021aafc83ccc60a9d8922336fc3776ce5a8cc714bbe/hf_vram_calc-1.0.8.tar.gz",
"platform": null,
"description": "# HF VRAM Calculator\n\n[](https://www.python.org/downloads/)\n[](https://opensource.org/licenses/MIT)\n[](https://github.com/astral-sh/uv)\n\nA professional Python CLI tool for estimating GPU memory requirements for Hugging Face models with different data types and parallelization strategies.\n\n> **\u26a1 Latest Features**: Smart dtype detection, MHA/MQA/GQA-aware KV cache, 12 quantization formats, 20+ GPU models, professional Rich UI\n\n## Quick Demo\n\n```bash\n# Install and run\npip install hf-vram-calc\n\n# Set up authentication (required for most models)\nhf auth login --token yourtoken --add-to-git-credential\n\n# Calculate memory requirements\nhf-vram-calc microsoft/DialoGPT-medium\n\n# Output: Beautiful tables showing 0.9GB inference, GPU compatibility, parallelization strategies\n```\n\n## Features\n\n- \ud83d\udd0d **Automatic Model Analysis**: Fetch configurations from Hugging Face Hub automatically\n- \ud83e\udde0 **Smart Data Type Detection**: Intelligent dtype recommendation from model names, config, or defaults\n- \ud83d\udcca **Comprehensive Data Type Support**: fp32, fp16, bf16, fp8, int8, int4, mxfp4, nvfp4, awq_int4, gptq_int4, nf4, fp4\n- \ud83c\udfaf **Multi-Scenario Memory Estimation**:\n - **Inference**: Model weights + KV cache overhead (MHA/MQA/GQA-aware, \u00d71.2 factor)\n - **Training**: Full Adam optimizer states (\u00d74\u00d71.3 factors)\n - **LoRA Fine-tuning**: Low-rank adaptation with trainable parameter overhead\n- \u26a1 **Advanced Parallelization Analysis**:\n - Tensor Parallelism (TP): 1, 2, 4, 8\n - Pipeline Parallelism (PP): 1, 2, 4, 8 \n - Expert Parallelism (EP) for MoE models\n - Data Parallelism (DP): 2, 4, 8\n - Combined strategies (TP + PP combinations)\n- \ud83c\udfae **GPU Compatibility Matrix**:\n - 20+ GPU models (RTX 4090, A100, H100, L40S, etc.)\n - Automatic compatibility checking for inference/training/LoRA\n - Minimum GPU memory requirement calculations\n- \ud83d\udcc8 **Professional Rich UI**:\n - \ud83c\udfa8 Beautiful color-coded tables and panels\n - \ud83d\udcca Real-time progress indicators\n - \ud83d\ude80 Modern CLI interface with emoji icons\n - \ud83d\udca1 Smart recommendations and warnings\n- \ud83d\udd27 **Flexible Configuration**:\n - Customizable LoRA rank, batch size, sequence length\n - External JSON configuration files\n - User-defined GPU models and data types\n- \ud83d\udccb **Parameter Display**: Raw count + human-readable format (e.g., \"405,016,576 (405.0M)\")\n\n## Installation\n\n### Quick Install (from PyPI)\n\n```bash\npip install hf-vram-calc\n```\n\n### Build from Source\n\n```bash\n# Clone the repository\ngit clone <repository-url>\ncd hf-vram-calc\n\n# Build with uv (recommended)\nuv build\nuv pip install dist/hf_vram_calc-1.0.0-py3-none-any.whl\n\n# Or install directly\nuv pip install .\n```\n\n> **Dependencies**: `requests` (HTTP), `rich` (beautiful CLI), Python \u22653.8\n\nFor detailed build instructions, see: [BUILD.md](BUILD.md)\n\n## Authentication Setup\n\nMany models require a Hugging Face token. Get yours at https://huggingface.co/settings/tokens, then:\n\n```bash\nhf auth login --token yourtoken --add-to-git-credential\n```\n\n## Usage\n\n### Basic Usage - Smart Dtype Detection\n\n```bash\n# Automatic dtype recommendation from model config/name\nhf-vram-calc --model mistralai/Mistral-7B-v0.1\n```\n\n### Specify Data Type Override\n\n```bash\n# Override with specific data type\nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --dtype bf16\nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --dtype bf16,fp8\n```\n\n### Advanced Configuration\n\n```bash\n# Custom batch size and sequence length\nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --max_batch_size 4 --max_seq_len 4096\n\n# Custom LoRA rank for fine-tuning estimation \nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --lora_rank 128\n\n# Detailed analysis (disabled by default)\nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --log_level verbose\n```\n\n### YAML Configuration\n\n```bash\n# Use YAML configuration file (trtllm-bench compatible)\nhf-vram-calc --extra_llm_api_options example_config.yaml\n\n# Override YAML with command line arguments\nhf-vram-calc --extra_llm_api_options example_config.yaml --max_batch_size 128\n```\n\n### JSON Output\n\n```bash\n# Save results to JSON file\nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --dtype bf16,fp8 --output_json results.json\n```\n\n### System Information\n\n```bash\n# List all available data types and GPU models\nhf-vram-calc --list_types\n\n# Use custom configuration directory\nhf-vram-calc --config_dir ./my_config --model mistralai/Mistral-7B-v0.1\n\n# Show help\nhf-vram-calc --help\n```\n\n## Command Line Arguments\n\n### Required\n- `--model MODEL`: Hugging Face model name (e.g., `mistralai/Mistral-7B-v0.1`)\n\n### Data Type Control \n- `--dtype {fp32,fp16,bf16,fp8,int8,int4,mxfp4,nvfp4,awq_int4,fp4,nf4,gptq_int4}`: Override automatic dtype detection\n- `--list_types`: List all available data types and GPU models\n\n### Memory Estimation Parameters\n- `--max_batch_size BATCH_SIZE`: Batch size for activation estimation (default: 1)\n- `--max_seq_len SEQUENCE_LENGTH`: Sequence length for memory calculation (default: 2048) \n- `--lora_rank LORA_RANK`: LoRA rank for fine-tuning estimation (default: 64)\n\n### Parallelization Settings\n- `--tp TP`: Tensor parallelism size (default: 1)\n- `--pp PP`: Pipeline parallelism size (default: 1)\n- `--ep EP`: Expert parallelism size (default: 1)\n\n### Configuration & Output\n- `--model_path MODEL_PATH`: Path to local model directory containing config.json\n- `--extra_llm_api_options YAML_FILE`: Path to YAML configuration file (trtllm-bench compatible)\n- `--output_json JSON_FILE`: Path to save results as JSON file\n- `--log_level {info,verbose}`: Log level for output (default: info)\n- `--config_dir CONFIG_DIR`: Custom configuration directory path\n- `--help`: Show complete help message with examples\n\n### Smart Behavior\n- **No `--dtype`**: Uses intelligent priority (model name \u2192 config \u2192 fp16 default)\n- **With `--dtype`**: Overrides automatic detection with specified type\n- **YAML + CLI**: Command line arguments override YAML configuration\n- **Invalid model**: Graceful error handling with helpful suggestions\n\n## Quick Start Examples\n\n```bash\n# Set up authentication first time\nhf auth login --token yourtoken --add-to-git-credential\n\n# Estimate memory for different models\nhf-vram-calc --model mistralai/Mistral-7B-v0.1 # \u2192 ~14GB inference (BF16)\nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --dtype fp16 # \u2192 ~14GB inference (FP16)\nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --dtype fp8 # \u2192 ~7GB inference (FP8)\n\n# estimate size for specified quantization versions\nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --dtype fp16 # \u2192 ~14GB\nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --dtype int4 # \u2192 ~3.5GB \nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --dtype awq_int4 # \u2192 ~3.5GB\n\n# for private access models, it is recommended to use --model_path\nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --model_path /llm_data/llm-models/Mistral-7B-v0.1\n\n# Find optimal parallelization strategy\nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --log_level verbose # \u2192 TP/PP recommendations\n\n# Save results to JSON\nhf-vram-calc --model mistralai/Mistral-7B-v0.1 --output_json results.json\n\n# Use YAML configuration (trtllm-bench compatible)\nhf-vram-calc --extra_llm_api_options config.yaml\n\n# Check what's available\nhf-vram-calc --list_types # \u2192 All types & GPUs\n```\n## Data Type Priority & Detection\n\n### Automatic Data Type Recommendation\n\nThe tool uses intelligent priority-based dtype selection:\n\n1. **Model Name Detection** (Highest Priority)\n - `model-fp16`, `model-bf16` \u2192 Extracts from model name \n - `model-4bit`, `model-gptq`, `model-awq` \u2192 Detects quantization\n \n2. **Config torch_dtype** (Medium Priority)\n - Reads `torch_dtype` from model's `config.json`\n - Maps `torch.float16` \u2192 `fp16`, `torch.bfloat16` \u2192 `bf16`, etc.\n\n3. **Default Fallback** (Lowest Priority)\n - Defaults to `fp16` when no dtype detected\n\n### Supported Data Types\n\n| Data Type | Bytes/Param | Description | Detection Patterns |\n|-----------|-------------|-------------|--------------------|\n| **fp32** | 4.0 | 32-bit floating point | `fp32`, `float32` |\n| **fp16** | 2.0 | 16-bit floating point | `fp16`, `float16`, `half` |\n| **bf16** | 2.0 | Brain Float 16 | `bf16`, `bfloat16` |\n| **fp8** | 1.0 | 8-bit floating point | `fp8`, `float8` |\n| **int8** | 1.0 | 8-bit integer | `int8`, `8bit` |\n| **int4** | 0.5 | 4-bit integer | `int4`, `4bit` |\n| **mxfp4** | 0.5 | Microsoft FP4 | `mxfp4` |\n| **nvfp4** | 0.5 | NVIDIA FP4 | `nvfp4` |\n| **awq_int4** | 0.5 | AWQ 4-bit quantization | `awq`, `awq-int4` |\n| **gptq_int4** | 0.5 | GPTQ 4-bit quantization | `gptq`, `gptq-int4` |\n| **nf4** | 0.5 | 4-bit NormalFloat | `nf4`, `bnb-4bit` |\n| **fp4** | 0.5 | 4-bit floating point | `fp4` |\n\n## YAML Configuration (trtllm-bench Compatible)\n\nThe `--extra_llm_api_options` argument allows you to use YAML configuration files with the same hierarchical structure as trtllm-bench:\n\n```yaml\n# config.yaml\nmodel: \"mistralai/Mistral-7B-v0.1\"\nkv_cache_config:\n dtype: \"fp8\"\n mamba_ssm_cache_dtype: \"fp16\"\nenable_chunked_prefill: true\nbuild_config:\n max_batch_size: 64\n max_num_tokens: 8192\n max_seq_len: 4096\nquant_config:\n quant_algo: \"fp8\"\n kv_cache_quant_algo: \"fp8\"\nlora_config:\n lora_dir: \"/path/to/lora/weights\"\n max_lora_rank: 16\nperformance_options:\n cuda_graphs: true\n multi_block_mode: true\nlog_level: \"verbose\"\n```\n\n### YAML Section Mappings\n\n- `build_config.max_batch_size` \u2192 `--max_batch_size`\n- `build_config.max_seq_len` \u2192 `--max_seq_len`\n- `lora_config.max_lora_rank` \u2192 `--lora_rank`\n- `kv_cache_config.dtype` \u2192 `--dtype`\n- `quant_config.quant_algo` \u2192 `--dtype` (with algorithm-to-dtype mapping)\n\n## JSON Output\n\nThe `--output_json` argument saves calculation results in a simplified JSON format:\n\n```json\n{\n \"model\": {\n \"name\": \"mistralai/Mistral-7B-v0.1\",\n \"architecture\": \"mistral\",\n \"parameters\": 7241732096,\n \"parameters_formatted\": \"7.24B\",\n \"original_torch_dtype\": \"torch.bfloat16\",\n \"user_specified_dtype\": \"FP8,BF16\"\n },\n \"memory_requirements\": [\n {\n \"dtype\": \"FP8\",\n \"batch_size\": 1,\n \"sequence_length\": 2048,\n \"lora_rank\": 64,\n \"model_size_gib\": 6.75,\n \"kv_cache_size_gib\": 0.13,\n \"inference_total_gib\": 8.10,\n \"training_gib\": 35.07,\n \"lora_size_gib\": 8.37\n },\n {\n \"dtype\": \"BF16\",\n \"batch_size\": 1,\n \"sequence_length\": 2048,\n \"lora_rank\": 64,\n \"model_size_gib\": 13.49,\n \"kv_cache_size_gib\": 0.25,\n \"inference_total_gib\": 16.19,\n \"training_gib\": 70.14,\n \"lora_size_gib\": 16.73\n }\n ]\n}\n```\n\n## Parallelization Strategies\n\n### Tensor Parallelism (TP)\nSplits model weights by tensor dimensions across multiple GPUs.\n\n### Pipeline Parallelism (PP)\nDistributes different model layers to different GPUs.\n\n### Expert Parallelism (EP)\nFor MoE (Mixture of Experts) models, distributes expert networks to different GPUs.\n\n### Data Parallelism (DP)\nEach GPU holds a complete model copy, only splitting data.\n\n## Example Output\n\n### Smart Dtype Detection Example\n\n```bash\n$ hf-vram-calc --model mistralai/Mistral-7B-v0.1 --log_level verbose\n```\n\n```\nUsing recommended data type: FP16\nUse --dtype to specify different type, or see --list_types for all options\n \ud83d\udd0d Fetching configuration for mistralai/Mistral-7B-v0.1...\nUsing recommended data type: FP16\nUse --dtype to specify different type, or see --list_types for all options\n \ud83d\udccb Parsing model configuration... \n \ud83e\uddee Calculating model parameters... \n \ud83d\udcbe Computing memory requirements... \n\n \u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500 \ud83e\udd16 Model Information \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n \u2502 \u2502\n \u2502 Model: mistralai/Mistral-7B-v0.1 \u2502\n \u2502 Architecture: mistral \u2502\n \u2502 Parameters: 7,241,732,096 (7.24B) \u2502\n \u2502 Recommended dtype: FP16 \u2502\n \u2502 \u2502\n \u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n\n \ud83d\udcbe Memory Requirements by Data Type and Scenario \n\u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 \u2502 Model Size \u2502 KV Cache \u2502 Inference \u2502 Training \u2502 LoRA \u2502\n\u2502 Data Type \u2502 (GB) \u2502 (GB) \u2502 Total (GB) \u2502 (Adam) (GB) \u2502 (GB) \u2502\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 FP16 \u2502 0.76 \u2502 0.19 \u2502 0.91 \u2502 3.94 \u2502 0.94 \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n================================================================================\n \u26a1 Parallelization Strategies (FP16 Inference) \n\u2554\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2564\u2550\u2550\u2550\u2550\u2550\u2550\u2564\u2550\u2550\u2550\u2550\u2550\u2550\u2564\u2550\u2550\u2550\u2550\u2550\u2550\u2564\u2550\u2550\u2550\u2550\u2550\u2550\u2564\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2564\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2557\n\u2551 \u2502 \u2502 \u2502 \u2502 \u2502 Memory/GPU \u2502 Min GPU \u2551\n\u2551 Strategy \u2502 TP \u2502 PP \u2502 EP \u2502 DP \u2502 (GB) \u2502 Required \u2551\n\u255f\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2562\n\u2551 Single GPU \u2502 1 \u2502 1 \u2502 1 \u2502 1 \u2502 0.91 \u2502 4GB+ \u2551\n\u2551 Tensor Parallel \u2502 2 \u2502 1 \u2502 1 \u2502 1 \u2502 0.45 \u2502 4GB+ \u2551\n\u2551 TP + PP \u2502 4 \u2502 4 \u2502 1 \u2502 1 \u2502 0.06 \u2502 4GB+ \u2551\n\u255a\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2567\u2550\u2550\u2550\u2550\u2550\u2550\u2567\u2550\u2550\u2550\u2550\u2550\u2550\u2567\u2550\u2550\u2550\u2550\u2550\u2550\u2567\u2550\u2550\u2550\u2550\u2550\u2550\u2567\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2567\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u255d\n\n \ud83c\udfae GPU Compatibility Matrix \n\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u252f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u252f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u252f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u252f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 GPU Type \u2502 Memory \u2502 Inference \u2502 Training \u2502 LoRA \u2503\n\u2520\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2528\n\u2503 RTX 4090 \u2502 24GB \u2502 \u2713 \u2502 \u2713 \u2502 \u2713 \u2503\n\u2503 A100 80GB \u2502 80GB \u2502 \u2713 \u2502 \u2713 \u2502 \u2713 \u2503\n\u2503 H100 80GB \u2502 80GB \u2502 \u2713 \u2502 \u2713 \u2502 \u2713 \u2503\n\u2517\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2537\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2537\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2537\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2537\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u251b\n\n\u256d\u2500\u2500\u2500 \ud83d\udccb Minimum GPU Requirements \u2500\u2500\u2500\u2500\u256e\n\u2502 \u2502\n\u2502 Single GPU Inference: 0.9GB \u2502\n\u2502 Single GPU Training: 3.9GB \u2502 \n\u2502 Single GPU LoRA: 0.9GB \u2502\n\u2502 \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n```\n\n### Large Model with User Override\n\n```bash\n$ hf-vram-calc nvidia/DeepSeek-R1-0528-FP4 --dtype nvfp4\n\n$ hf-vram-calc Qwen/Qwen-72B-Chat \n\n```\n\n```\n \u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 \ud83e\udd16 Model Information \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n \u2502 \u2502\n \u2502 Model: nvidia/DeepSeek-R1-0528-FP4 \u2502\n \u2502 Architecture: deepseek_v3 \u2502\n \u2502 Parameters: 30,510,606,336 (30.5B) \u2502\n \u2502 Original torch_dtype: bfloat16 \u2502\n \u2502 User specified dtype: NVFP4 \u2502\n \u2502 \u2502\n \u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n\n \ud83d\udcbe Memory Requirements by Data Type and Scenario \n\u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 \u2502 Total Size \u2502 Inference \u2502 Training \u2502 LoRA \u2502\n\u2502 Data Type \u2502 (GB) \u2502 (GB) \u2502 (Adam) (GB) \u2502 (GB) \u2502\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 NVFP4 \u2502 14.21 \u2502 17.05 \u2502 73.88 \u2502 19.34 \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n```\n\n### List Available Types\n\n```bash\n$ hf-vram-calc --list_types\n```\n\n```\nAvailable Data Types:\n\u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 Data Type \u2502 Bytes/Param \u2502 Description \u2502\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 FP32 \u2502 4 \u2502 32-bit floating point \u2502\n\u2502 FP16 \u2502 2 \u2502 16-bit floating point \u2502\n\u2502 BF16 \u2502 2 \u2502 Brain Float 16 \u2502\n\u2502 NVFP4 \u2502 0.5 \u2502 NVIDIA FP4 \u2502\n\u2502 AWQ_INT4 \u2502 0.5 \u2502 AWQ 4-bit quantization \u2502\n\u2502 GPTQ_INT4 \u2502 0.5 \u2502 GPTQ 4-bit quantization\u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n\nAvailable GPU Types:\n\u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 GPU Name \u2502 Memory (GB) \u2502 Category \u2502 Architecture \u2502\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 RTX 4090 \u2502 24 \u2502 consumer \u2502 Ada Lovelace \u2502\n\u2502 A100 80GB \u2502 80 \u2502 datacenter \u2502 Ampere \u2502\n\u2502 H100 80GB \u2502 80 \u2502 datacenter \u2502 Hopper \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n```\n\n## Calculation Formulas\n\n### Inference Memory\n```\nInference Memory = Model Weights \u00d7 1.2\n```\nIncludes model weights and KV cache overhead.\n\n### KV Cache Memory\n```\nKV Cache (GB) = 2 \u00d7 Batch_Size \u00d7 Sequence_Length \u00d7 Head_Dim \u00d7 Num_KV_Heads \u00d7 Num_Layers \u00d7 Precision \u00f7 1,073,741,824\n```\n- Head_Dim = hidden_size \u00f7 num_attention_heads\n- Num_KV_Heads = config.num_key_value_heads (if present) else num_attention_heads\n- Automatically supports MHA, MQA, and GQA via model config; KV cache uses FP16/BF16 for quantized models\n\n### Training Memory (with Adam)\n```\nTraining Memory = Model Weights \u00d7 4 \u00d7 1.3\n```\n- 4x factor: Model weights (1x) + Gradients (1x) + Adam optimizer states (2x)\n- 1.3x factor: 30% additional overhead (activation caching, etc.)\n\n### LoRA Fine-tuning Memory\n```\nLoRA Memory = (Model Weights + LoRA Parameter Overhead) \u00d7 1.2\n```\nLoRA parameter overhead calculated based on rank and target module ratio.\n\n## Advanced Features\n\n### Configuration System\n\nExternal JSON configuration files for maximum flexibility:\n\n- **`data_types.json`** - Add custom quantization formats\n- **`gpu_types.json`** - Define new GPU models and specifications \n- **`display_settings.json`** - Customize UI appearance and limits\n\n```bash\n# Use custom config directory\nhf-vram-calc --config-dir ./custom_config model_name\n\n# Add custom data type example (data_types.json)\n{\n \"my_custom_int2\": {\n \"bytes_per_param\": 0.25,\n \"description\": \"Custom 2-bit quantization\"\n }\n}\n```\n\n### Memory Calculation Details\n\n| Scenario | Formula | Explanation |\n|----------|---------|-------------|\n| **Inference** | Model \u00d7 1.2 | Includes KV cache and activation overhead |\n| **Training** | Model \u00d7 4 \u00d7 1.3 | Weights(1x) + Gradients(1x) + Adam(2x) + 30% overhead |\n| **LoRA** | (Model + LoRA_params\u00d74) \u00d7 1.2 | Base model + trainable parameters with optimizer |\n\n### Parallelization Efficiency\n\n- **TP (Tensor Parallel)**: Near-linear scaling, slight communication overhead\n- **PP (Pipeline Parallel)**: Good efficiency, pipeline bubble ~10-15% \n- **EP (Expert Parallel)**: MoE-specific, depends on expert routing efficiency\n- **DP (Data Parallel)**: No memory reduction per GPU, full model replica\n\n## Supported Architectures\n\n### Fully Supported \u2705\n- **GPT Family**: GPT-2, GPT-3, GPT-4, GPT-NeoX, etc.\n- **LLaMA Family**: LLaMA, LLaMA-2, Code Llama, Vicuna, etc.\n- **Mistral Family**: Mistral 7B, Mixtral 8x7B (MoE), etc.\n- **Other Transformers**: BERT, RoBERTa, T5, FLAN-T5, etc.\n- **New Architectures**: DeepSeek, Qwen, ChatGLM, Baichuan, etc.\n\n### Architecture Detection\n- **Automatic field mapping** for different config.json formats\n- **Fallback support** for uncommon architectures\n- **MoE handling** for Mixture-of-Experts models\n\n## Accuracy & Limitations\n\n### \u2705 Highly Accurate For:\n- **Parameter counting** (exact calculation)\n- **Memory estimation** (within 5-10% of actual)\n- **Parallelization ratios** (theoretical maximum)\n\n### \u26a0\ufe0f Considerations:\n- **Activation memory** varies with sequence length and optimization\n- **Real-world efficiency** may differ due to framework overhead \n- **Quantization accuracy** depends on specific implementation\n- **MoE models** require expert routing consideration\n\n## Build & Development\n\nBuilt with modern Python tooling:\n- **uv**: Fast Python package management and building\n- **Rich**: Professional terminal interface\n- **Requests**: HTTP client for model config fetching\n- **JSON configuration**: Flexible external configuration system\n\nFor development setup, see: [BUILD.md](BUILD.md)\n\n## Contributing\n\nWe welcome contributions! Areas for improvement:\n\n- \ud83d\udd27 **New quantization formats** (add to `data_types.json`)\n- \ud83c\udfae **GPU models** (update `gpu_types.json`) \n- \ud83d\udcca **Architecture support** (enhance config parsing)\n- \ud83d\ude80 **Performance optimizations**\n- \ud83d\udcda **Documentation improvements**\n- \ud83e\uddea **Test coverage expansion**\n\n## See Also\n\n- \ud83d\udcda **[BUILD.md](BUILD.md)** - Complete build and installation guide\n- \u2699\ufe0f **[CONFIG_GUIDE.md](CONFIG_GUIDE.md)** - Configuration customization details\n- \ud83d\udcdd **Examples in help**: `hf-vram-calc --help` for usage examples\n\n## Version History\n\n- **v1.0.0**: Complete rewrite with uv build, smart dtype detection, professional UI\n- **v0.x**: Legacy single-file version (deprecated)\n\n## License\n\nMIT License - see LICENSE file for details.\n\n---\n\n**Made with \u2764\ufe0f for the ML community** | Built with [uv](https://github.com/astral-sh/uv) and [Rich](https://github.com/Textualize/rich)\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "GPU memory calculator for Hugging Face models with different data types and parallelization strategies",
"version": "1.0.8",
"project_urls": {
"Homepage": "https://github.com/example/hf-vram-calc",
"Issues": "https://github.com/example/hf-vram-calc/issues",
"Repository": "https://github.com/example/hf-vram-calc"
},
"split_keywords": [
"ai",
" calculator",
" gpu",
" huggingface",
" memory",
" ml",
" transformer",
" vram"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "867a8bbbf1576c5a9ffa1123cbc9af10ffdef6c43b64fec8ab1f2e48c12e2631",
"md5": "5d7b5e32b31d6b412aeb1039c535ea4f",
"sha256": "9d040dafa01c7c4ee1bc4df7dc175b5916554ae265104f24f98fe0cd11768780"
},
"downloads": -1,
"filename": "hf_vram_calc-1.0.8-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5d7b5e32b31d6b412aeb1039c535ea4f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 30111,
"upload_time": "2025-10-22T01:35:33",
"upload_time_iso_8601": "2025-10-22T01:35:33.368109Z",
"url": "https://files.pythonhosted.org/packages/86/7a/8bbbf1576c5a9ffa1123cbc9af10ffdef6c43b64fec8ab1f2e48c12e2631/hf_vram_calc-1.0.8-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "d849de6c0aaa512ad3b5f021aafc83ccc60a9d8922336fc3776ce5a8cc714bbe",
"md5": "c41407be4d209079bab9fee50fd5b4ca",
"sha256": "7bc79e6174451fb589e14f6a03e9ff129677be92e568c51469008f383d6a8fe2"
},
"downloads": -1,
"filename": "hf_vram_calc-1.0.8.tar.gz",
"has_sig": false,
"md5_digest": "c41407be4d209079bab9fee50fd5b4ca",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 30804,
"upload_time": "2025-10-22T01:35:34",
"upload_time_iso_8601": "2025-10-22T01:35:34.432197Z",
"url": "https://files.pythonhosted.org/packages/d8/49/de6c0aaa512ad3b5f021aafc83ccc60a9d8922336fc3776ce5a8cc714bbe/hf_vram_calc-1.0.8.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-10-22 01:35:34",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "example",
"github_project": "hf-vram-calc",
"github_not_found": true,
"lcname": "hf-vram-calc"
}
JSON
