# ๐ฆ Vogel Model Trainer
**Languages:** [๐ฌ๐ง English](README.md) | [๐ฉ๐ช Deutsch](README.de.md) | [๐ฏ๐ต ๆฅๆฌ่ช](README.ja.md)
<p align="left">
<a href="https://pypi.org/project/vogel-model-trainer/"><img alt="PyPI version" src="https://img.shields.io/pypi/v/vogel-model-trainer.svg"></a>
<a href="https://pypi.org/project/vogel-model-trainer/"><img alt="Python Versions" src="https://img.shields.io/pypi/pyversions/vogel-model-trainer.svg"></a>
<a href="https://opensource.org/licenses/MIT"><img alt="License: MIT" src="https://img.shields.io/badge/License-MIT-yellow.svg"></a>
<a href="https://pypi.org/project/vogel-model-trainer/"><img alt="PyPI Status" src="https://img.shields.io/pypi/status/vogel-model-trainer.svg"></a>
<a href="https://pepy.tech/project/vogel-model-trainer"><img alt="Downloads" src="https://static.pepy.tech/badge/vogel-model-trainer"></a>
</p>
**Train custom bird species classifiers from your own video footage using YOLOv8 and EfficientNet.**
A specialized toolkit for creating high-accuracy bird species classifiers tailored to your specific monitoring setup. Extract training data from videos, organize datasets, and train custom models with >96% accuracy.
---
## โจ Features
- ๐ฏ **YOLO-based Bird Detection** - Automated bird cropping from videos using YOLOv8
- ๐ค **Three Extraction Modes** - Manual labeling, auto-sorting, or standard extraction
- ๐ **Wildcard Support** - Batch process multiple videos with glob patterns
- ๐ผ๏ธ **Flexible Image Sizes** - 224/384/448px or keep original size
- ๐ **Advanced Filtering** - Box size, blur detection, confidence thresholds
- ๐ **Duplicate Detection** - Perceptual hashing removes similar images
- โ
**Quality Check** - Find blurry, too-small, corrupted, or badly-exposed images
- ๐จ **AI Background Removal** - Remove backgrounds with gray default for optimal training
- ๐งน **Dataset Validation** - Clean transparent/gray datasets with automated checks
- ๐ง **EfficientNet-B0 Training** - Lightweight yet powerful classification model
- ๐จ **4-Level Data Augmentation** - None/light/medium/heavy intensity options
- โก **Mixed Precision Training** - FP16/BF16 support for faster GPU training
- ๐ **Advanced Training Options** - 13 configurable parameters for fine-tuning
- ๐ง **Dataset Deduplication** - Clean existing datasets with perceptual hashing
- โธ๏ธ **Graceful Shutdown** - Save model state on Ctrl+C interruption
- ๐ **Full i18n Support** - English, German, Japanese translations
- ๐ **Per-Species Metrics** - Detailed accuracy breakdown by species
## ๐ค Pre-trained Models
**German Garden Birds Classifier** - Ready to use!
We provide a pre-trained model on Hugging Face that can classify 8 common German garden bird species with 100% validation accuracy:
๐ **[kamera-linux/german-bird-classifier](https://huggingface.co/kamera-linux/german-bird-classifier)**
**Supported Species:**
- Blaumeise (Blue Tit)
- Grรผnling (European Greenfinch)
- Haussperling (House Sparrow)
- Kernbeiรer (Hawfinch)
- Kleiber (Eurasian Nuthatch)
- Kohlmeise (Great Tit)
- Rotkehlchen (European Robin)
- Sumpfmeise (Marsh Tit)
**Usage with extraction:**
```bash
vogel-trainer extract --folder ~/bird-data \
--species-model kamera-linux/german-bird-classifier \
--sample-rate 20 --skip-blurry --deduplicate \
video.mp4
```
The model will automatically classify detected birds during extraction!
---
## ๐ Quick Start
### Installation
#### Recommended: Using Virtual Environment
```bash
# Install venv if needed (Debian/Ubuntu)
sudo apt install python3-venv
# Create virtual environment
python3 -m venv ~/venv-vogel
# Activate it
source ~/venv-vogel/bin/activate # On Windows: ~/venv-vogel\Scripts\activate
# Install package
pip install vogel-model-trainer
```
#### Quick Install
```bash
# Install from PyPI
pip install vogel-model-trainer
# Or install from source
git clone https://github.com/kamera-linux/vogel-model-trainer.git
cd vogel-model-trainer
pip install -e .
```
### ๐ฅ Video Tutorials
Learn vogel-model-trainer with step-by-step video guides:
- **Getting Started** - Installation and first extraction (5 min)
- **Extracting Birds** - Quality filters, deduplication, species classification (10 min)
- **Organizing Datasets** - Train/val splits, class balance management (8 min) **NEW in v0.1.8**
- **Training Models** - Custom classifier training and parameters (12 min)
- **Testing & Evaluation** - Model testing and performance analysis (7 min)
๐บ *Video tutorials coming soon!*
### Basic Workflow
```bash
# 1. Extract bird images from videos
vogel-trainer extract video.mp4 --folder ~/training-data/ --bird kohlmeise
# 2. Organize into train/validation split
vogel-trainer organize ~/training-data/ -o ~/organized-data/
# 3. Train custom classifier
vogel-trainer train ~/organized-data/ -o ~/models/my-classifier/
# 4. Test the trained model
vogel-trainer test ~/models/my-classifier/ -d ~/organized-data/
```
---
## ๐ Usage Guide
### Using as a Library (New in v0.1.2)
All core functions can now be used programmatically in your Python code:
```python
from vogel_model_trainer.core import extractor, organizer, trainer, tester
# Extract birds from video
extractor.extract_birds_from_video(
video_path="video.mp4",
output_dir="output/",
bird_species="great-tit",
detection_model="yolov8n.pt",
species_model=None,
threshold=0.5,
sample_rate=3,
resize_to_target=True
)
# Organize into train/val splits
organizer.organize_dataset(
source_dir="output/",
output_dir="dataset/",
train_ratio=0.8
)
# Train model
trainer.train_model(
data_dir="dataset/",
output_dir="models/",
model_name="google/efficientnet-b0",
batch_size=16,
num_epochs=50,
learning_rate=3e-4
)
# Test model
results = tester.test_model(
model_path="models/bird_classifier/",
data_dir="dataset/"
)
print(f"Accuracy: {results['accuracy']:.2%}")
```
### 1. Extract Training Images
#### Manual Mode (Recommended for Initial Collection)
When you know the species in your video:
```bash
vogel-trainer extract ~/Videos/great-tit.mp4 \
--folder ~/training-data/ \
--bird great-tit \
--threshold 0.5 \
--sample-rate 3
```
#### Auto-Sort Mode (For Iterative Training)
Use an existing model to automatically classify and sort:
```bash
vogel-trainer extract ~/Videos/mixed.mp4 \
--folder ~/training-data/ \
--species-model ~/models/classifier/final/ \
--threshold 0.5
```
#### Batch Processing with Wildcards
```bash
# Process all videos in a directory
vogel-trainer extract "~/Videos/*.mp4" --folder ~/data/ --bird blue-tit
# Recursive directory search
vogel-trainer extract ~/Videos/ \
--folder ~/data/ \
--bird amsel \
--recursive
```
**Parameters:**
- `--folder`: Base directory for extracted images (required)
- `--bird`: Manual species label (creates subdirectory)
- `--species-model`: Path to trained model for auto-classification
- `--species-threshold`: Minimum confidence for species classification (e.g., 0.85 for 85%)
- `--threshold`: YOLO confidence threshold (default: 0.5)
- `--sample-rate`: Process every Nth frame (default: 3)
- `--detection-model`: YOLO model path (default: yolov8n.pt)
- `--image-size`: Target image size in pixels (default: 224, use 0 for original size)
- `--max-detections`: Maximum detections per frame (default: 10)
- `--min-box-size`: Minimum bounding box size in pixels (default: 50)
- `--max-box-size`: Maximum bounding box size in pixels (default: 800)
- `--quality`: JPEG quality 1-100 (default: 95)
- `--skip-blurry`: Skip blurry/out-of-focus images (experimental)
- `--deduplicate`: Skip duplicate/similar images using perceptual hashing
- `--similarity-threshold`: Similarity threshold for duplicates - Hamming distance 0-64 (default: 5)
- `--min-sharpness`: **NEW v0.1.9** - Minimum sharpness score (Laplacian variance, typical: 100-300)
- `--min-edge-quality`: **NEW v0.1.9** - Minimum edge quality (Sobel gradient, typical: 50-150)
- `--save-quality-report`: **NEW v0.1.9** - Generate detailed quality statistics report
- `--remove-background`: **๐งช EXPERIMENTAL v0.1.11** - Remove background using AI (rembg)
- `--bg-color [white|black|gray]`: **๐งช EXPERIMENTAL v0.1.11** - Background color (default: white)
- `--bg-model [u2net|u2netp|isnet-general-use]`: **๐งช EXPERIMENTAL v0.1.11** - AI model for background removal (default: u2net)
- `--recursive, -r`: Search directories recursively
- `--log`: Save console output to log file (`/var/log/vogel-kamera-linux/YYYY/KWXX/`)
**Advanced Filtering Examples:**
```bash
# High-quality extraction with all filters (v0.1.11)
vogel-trainer extract video.mp4 \
--folder data/ \
--bird rotkehlchen \
--threshold 0.6 \
--min-box-size 80 \
--max-box-size 600 \
--min-sharpness 150 \
--min-edge-quality 80 \
--skip-blurry \
--deduplicate \
--save-quality-report \
--remove-background \
--bg-color white \
--bg-model u2net
# Background removal with black background for contrast
vogel-trainer extract video.mp4 \
--folder data/ \
--bird blaumeise \
--remove-background \
--bg-color black \
--bg-model isnet-general-use
```
**๐งช Background Removal (EXPERIMENTAL v0.1.11+, Stable v0.1.14):**
The `--remove-background` feature uses AI-powered rembg library to automatically segment birds from backgrounds.
**NEW in v0.1.14:** Gray background is now the DEFAULT for optimal training! Smaller JPEG files, better compatibility.
- **Models:**
- `u2net` (default): Best overall quality, ~180MB download
- `u2netp`: Faster, smaller model for quick processing
- `isnet-general-use`: Best edge quality for detailed feathers
- **Background Colors (NEW DEFAULT v0.1.14):**
- `gray` (DEFAULT): Neutral gray background (#808080) - best for training
- `white`: Clean white background (#FFFFFF)
- `black`: High contrast black background (#000000)
- `green-screen`: Chroma key green (#00FF00)
- `blue-screen`: Chroma key blue (#0000FF)
- **Transparency Options:**
- `--bg-transparent`: Create PNG with alpha channel (flexible but larger files)
- `--no-bg-transparent` (DEFAULT): Use colored background (smaller JPEG files)
- `--bg-fill-black`: Makes black box areas transparent (requires --bg-transparent)
- `--no-bg-fill-black` (DEFAULT): Keep padding areas with background color
- **Features:**
- AI-based Uยฒ-Net segmentation for accurate bird isolation
- Alpha matting for smooth, professional edges
- Post-processing with morphological operations
- Handles complex backgrounds (branches, leaves, buildings)
- Works with varied bird plumage and fine feather details
- Automatically saves as PNG (transparent) or JPEG (opaque)
- **Note:** First use downloads ~180MB model (cached afterward), requires `rembg>=2.0.50` dependency
--remove-background \
--quality 98
# Extract with duplicate detection (prevents similar images)
vogel-trainer extract ~/Videos/*.mp4 \
--folder data/ \
--bird kohlmeise \
--deduplicate \
--similarity-threshold 3 # Stricter duplicate detection
# Large image size for high-detail training
vogel-trainer extract video.mp4 \
--folder data/ \
--bird amsel \
--image-size 384 # Larger images for better quality
# Auto-sort with confidence filter (only high-confidence classifications)
vogel-trainer extract video.mp4 \
--folder data/ \
--species-model ~/models/classifier/ \
--species-threshold 0.90 \
--deduplicate
```
**Logging Example:**
```bash
# Save output to log file
vogel-trainer extract ~/Videos/great-tit.mp4 \
--folder ~/data/ \
--bird great-tit \
--log
# Log file path: /var/log/vogel-kamera-linux/2025/KW45/20251109_160000_extract.log
```
### 1b. Clean Dataset Images (NEW v0.1.12+) ๐งน
**Clean Transparent Images** - For transparent PNG datasets:
```bash
# Safe mode: Report only (no files modified)
vogel-trainer clean-transparent ~/training-data/ --mode report
# Move invalid images to invalid_transparent/ folder
vogel-trainer clean-transparent ~/training-data/ --mode move --recursive
```
**Clean Gray Background Images (NEW v0.1.14)** - For gray background datasets:
```bash
# Check gray background ratio
vogel-trainer clean-gray ~/training-data/ --mode report
# Move images with wrong gray ratio to invalid_gray/
vogel-trainer clean-gray ~/training-data/ --mode move --recursive
# Custom thresholds
vogel-trainer clean-gray ~/training-data/ \
--min-gray 0.10 \
--max-gray 0.90 \
--gray-tolerance 30 \
--mode move
```
**Detection Criteria:**
*clean-transparent:*
- **Min Visible Pixels** (`--min-pixels`, default: 500): Minimum non-transparent pixels
- **Max Transparency** (`--max-transparency`, default: 0.95): Maximum 95% transparency allowed
- **Min Region Size** (`--min-region`, default: 100): Minimum size of largest connected object
*clean-gray:*
- **Min Gray Ratio** (`--min-gray`, default: 0.05): Minimum 5% gray background required
- **Max Gray Ratio** (`--max-gray`, default: 0.95): Maximum 95% gray allowed (need visible bird)
- **Gray Tolerance** (`--gray-tolerance`, default: 30): Tolerance for gray detection (RโGโBยฑ30)
**Use Cases:**
- Remove tiny fragments after background removal
- Clean up partial detections (bird flew out of frame)
- Eliminate images with too much/little background
- Find images where bird is barely visible or missing
### 2. Organize Dataset
```bash
# Basic organization (80/20 split)
vogel-trainer organize ~/training-data/ -o ~/organized-data/
# With class balance control (NEW in v0.1.8)
vogel-trainer organize ~/training-data/ -o ~/organized-data/ \
--max-images-per-class 100 \
--tolerance 15.0
```
**Class Balance Features:**
- `--max-images-per-class N`: Limit to N images per class, delete excess
- `--tolerance N`: Maximum allowed imbalance % (default: 15)
- < 10%: โ
Perfect
- 10-15%: โ ๏ธ Warning
- > 15%: โ Error with recommendations
Creates an 80/20 train/validation split:
```
organized/
โโโ train/
โ โโโ great-tit/
โ โโโ blue-tit/
โ โโโ robin/
โโโ val/
โโโ great-tit/
โโโ blue-tit/
โโโ robin/
```
### 3. Train Classifier
```bash
vogel-trainer train ~/organized-data/ -o ~/models/my-classifier/
```
**Training Parameters:**
- `--batch-size`: Training batch size (default: 16)
- `--epochs`: Number of training epochs (default: 50)
- `--learning-rate`: Learning rate (default: 2e-4)
- `--early-stopping-patience`: Early stopping patience in epochs (default: 5, 0 to disable)
- `--weight-decay`: Weight decay for regularization (default: 0.01)
- `--warmup-ratio`: Learning rate warmup ratio (default: 0.1)
- `--label-smoothing`: Label smoothing factor (default: 0.1, 0 to disable)
- `--save-total-limit`: Maximum checkpoints to keep (default: 3)
- `--augmentation-strength`: Data augmentation intensity: `none`, `light`, `medium` (default), `heavy`
- `--image-size`: Input image size in pixels (default: 224, supports 224/384/448)
- `--scheduler`: Learning rate scheduler: `cosine` (default), `linear`, `constant`
- `--seed`: Random seed for reproducibility (default: 42)
- `--resume-from-checkpoint`: Path to checkpoint to resume training
- `--gradient-accumulation-steps`: Gradient accumulation steps (default: 1)
- `--mixed-precision`: Mixed precision training: `no` (default), `fp16`, `bf16`
- `--push-to-hub`: Push model to HuggingFace Hub (default: False)
- `--log`: Save console output to log file
**Augmentation Strength Levels:**
- **none**: No augmentation (only normalization)
- **light**: Small rotations (ยฑ10ยฐ), minimal color jitter
- **medium** (default): Moderate rotations (ยฑ20ยฐ), affine transforms, color jitter, gaussian blur
- **heavy**: Strong rotations (ยฑ30ยฐ), aggressive transforms, strong color variations
**Advanced Training Examples:**
```bash
# High-accuracy training with large images and heavy augmentation
vogel-trainer train ~/organized-data/ \
-o ~/models/high-accuracy/ \
--image-size 384 \
--augmentation-strength heavy \
--epochs 100 \
--early-stopping-patience 10 \
--batch-size 8
# Fast training with mixed precision (requires GPU)
vogel-trainer train ~/organized-data/ \
-o ~/models/fast/ \
--mixed-precision fp16 \
--batch-size 32 \
--gradient-accumulation-steps 2
# Reproducible training with fixed seed
vogel-trainer train ~/organized-data/ \
-o ~/models/reproducible/ \
--seed 12345 \
--augmentation-strength light
# Resume interrupted training
vogel-trainer train ~/organized-data/ \
-o ~/models/continued/ \
--resume-from-checkpoint ~/models/my-classifier/checkpoints/checkpoint-1000
# Training with logging
vogel-trainer train ~/organized-data/ \
-o ~/models/logged/ \
--log
```
**Training Configuration:**
- Base Model: `google/efficientnet-b0` (8.5M parameters)
- Optimizer: AdamW with configurable LR schedule
- Augmentation: 4 intensity levels (none/light/medium/heavy)
- Regularization: Weight decay, label smoothing, early stopping
- Mixed Precision: FP16/BF16 support for faster training on GPU
**Output:**
```
~/models/my-classifier/
โโโ checkpoints/ # Intermediate checkpoints
โโโ logs/ # TensorBoard logs
โโโ final/ # Final trained model
โโโ config.json
โโโ model.safetensors
โโโ preprocessor_config.json
```
### 4. Deduplicate Dataset
Remove duplicate or very similar images from your dataset to improve training quality:
```bash
# Report duplicates without deleting
vogel-trainer deduplicate ~/training-data/ --recursive
# Delete duplicates (keep first occurrence)
vogel-trainer deduplicate ~/training-data/ \
--mode delete \
--recursive
# Move duplicates to separate folder
vogel-trainer deduplicate ~/training-data/ \
--mode move \
--recursive
# Stricter duplicate detection
vogel-trainer deduplicate ~/training-data/ \
--threshold 3 \
--recursive
# Keep largest file instead of first
vogel-trainer deduplicate ~/training-data/ \
--mode delete \
--keep largest \
--recursive
```
**Deduplication Parameters:**
- `--threshold`: Similarity threshold - Hamming distance 0-64, lower=stricter (default: 5)
- `--method`: Hash method: `phash` (default, recommended), `dhash`, `whash`, `average_hash`
- `--mode`: Action: `report` (show only, default), `delete` (remove), `move` (to duplicates/)
- `--keep`: Which duplicate to keep: `first` (chronological, default) or `largest` (file size)
- `--recursive, -r`: Search recursively through subdirectories
**How it works:**
- Uses perceptual hashing (pHash) to detect visually similar images
- Robust against resizing, cropping, and minor color changes
- Threshold of 5 = very similar, 10 = similar, >15 = different
- Safe default: `report` mode prevents accidental deletion
### 5. Quality Check Dataset (New!)
Check your dataset for low-quality images (blurry, too small, corrupted, brightness issues):
```bash
# Report quality issues without deleting
vogel-trainer quality-check ~/training-data/ --recursive
# Delete low-quality images
vogel-trainer quality-check ~/training-data/ \
--mode delete \
--recursive
# Move low-quality images to separate folder
vogel-trainer quality-check ~/training-data/ \
--mode move \
--recursive
# Stricter blur detection
vogel-trainer quality-check ~/training-data/ \
--blur-threshold 150.0 \
--recursive
# Check for brightness/contrast issues
vogel-trainer quality-check ~/training-data/ \
--check-brightness \
--recursive
# Comprehensive quality check with custom thresholds
vogel-trainer quality-check ~/training-data/ \
--blur-threshold 120.0 \
--min-resolution 100 \
--min-filesize 2048 \
--check-brightness \
--mode move \
--recursive
```
**Quality Check Parameters:**
- `--blur-threshold`: Minimum blur score (Laplacian variance), lower=more blurry (default: 100.0)
- `--min-resolution`: Minimum image width/height in pixels (default: 50)
- `--min-filesize`: Minimum file size in bytes (default: 1024)
- `--check-brightness`: Also check for brightness/contrast issues (too dark or overexposed)
- `--mode`: Action: `report` (show only, default), `delete` (remove), `move` (to low_quality/)
- `--recursive, -r`: Search recursively through subdirectories
**โ ๏ธ WARNING - Delete Mode:**
- The `--mode delete` option **permanently removes files** without backup
- **Always run `--mode report` first** to preview what will be deleted
- **Backup your dataset** before using delete mode
- Consider using `--mode move` instead (keeps files in `low_quality/` folder)
**What is checked:**
- โ
**Sharpness**: Detects blurry/out-of-focus images using Laplacian variance
- โ
**Resolution**: Filters out too-small images that hurt training
- โ
**File size**: Detects corrupted or empty files
- โ
**Readability**: Checks if images can be opened and processed
- โ
**Brightness** (optional): Detects too-dark or overexposed images
**Typical thresholds:**
- Blur: 100.0 (default) = moderate, 150.0 = stricter, 50.0 = more lenient
- Resolution: 50px (default) = very permissive, 100px = recommended, 224px = strict
- Filesize: 1024 bytes (default) = catches corrupted files
**Recommended workflow:**
```bash
# 1. Preview issues first (safe)
vogel-trainer quality-check ~/data/ --mode report --recursive
# 2. Move problematic images (reversible)
vogel-trainer quality-check ~/data/ --mode move --recursive
# 3. Review moved files in low_quality/ folder
# 4. Delete manually if satisfied: rm -rf ~/data/low_quality/
```
### 6. Test Model
```bash
# Test on validation dataset
vogel-trainer test ~/models/my-classifier/ -d ~/organized-data/
# Output:
# ๐งช Testing model on validation set...
# ๐ฆ Predicted: great-tit (98.5% confidence)
```
---
## ๐ Iterative Training Workflow
Improve your model accuracy through iterative refinement using auto-classification:
```mermaid
flowchart TD
Start([๐ Phase 1: Initial Model<br/>Manual Labeling]) --> Extract1[1๏ธโฃ Extract with manual labels<br/><code>vogel-trainer extract video.mp4<br/>--folder data/ --bird kohlmeise</code>]
Extract1 --> Organize1[2๏ธโฃ Organize dataset 80/20 split<br/><code>vogel-trainer organize data/<br/>-o organized/</code>]
Organize1 --> Train1[3๏ธโฃ Train initial model<br/><code>vogel-trainer train organized/<br/>-o models/v1/</code><br/>โ
<b>Result: 92% accuracy</b>]
Train1 --> Phase2([๐ Phase 2: Model Improvement<br/>Auto-Classification])
Phase2 --> Extract2[4๏ธโฃ Auto-extract with trained model<br/><code>vogel-trainer extract new-videos/<br/>--folder data-v2/<br/>--species-model models/v1/final/<br/>--species-threshold 0.85</code><br/>๐ฏ <b>Automatically sorted by species!</b>]
Extract2 --> Review[5๏ธโฃ Manual review & corrections<br/>โข Check auto-classifications<br/>โข Move misclassified images<br/>โข Merge with previous dataset]
Review --> Train2[6๏ธโฃ Retrain with expanded dataset<br/><code>vogel-trainer organize data-v2/<br/>-o organized-v2/<br/>vogel-trainer train organized-v2/<br/>-o models/v2/</code><br/>๐ <b>Result: 96% accuracy!</b>]
Train2 --> Repeat{โป๏ธ Continue<br/>improving?}
Repeat -->|Yes| Extract2
Repeat -->|No| End([โ
Final Model])
style Start fill:#e1f5ff,stroke:#0066cc,stroke-width:3px
style Phase2 fill:#e1f5ff,stroke:#0066cc,stroke-width:3px
style Train1 fill:#d4edda,stroke:#28a745,stroke-width:2px
style Train2 fill:#d4edda,stroke:#28a745,stroke-width:2px
style End fill:#d4edda,stroke:#28a745,stroke-width:3px
style Extract2 fill:#fff3cd,stroke:#ffc107,stroke-width:2px
style Review fill:#f8d7da,stroke:#dc3545,stroke-width:2px
```
**Key Benefits:**
- ๐ **Faster labeling**: Auto-classification saves manual work
- ๐ **Better accuracy**: More training data = better model
- ๐ฏ **Quality control**: `--species-threshold` filters uncertain predictions
- ๐ **Continuous improvement**: Each iteration improves the model
**Example Commands:**
```bash
# Phase 1: Manual training (initial dataset)
vogel-trainer extract ~/Videos/batch1/*.mp4 --folder ~/data/ --bird great-tit
vogel-trainer organize ~/data/ -o ~/data/organized/
vogel-trainer train ~/data/organized/ -o ~/models/v1/
# Phase 2: Auto-classification with trained model
vogel-trainer extract ~/Videos/batch2/*.mp4 \
--folder ~/data-v2/ \
--species-model ~/models/v1/final/ \
--species-threshold 0.85
# Review classifications in ~/data-v2/<species>/ folders
# Move any misclassified images to correct species folders
# Merge datasets and retrain
cp -r ~/data-v2/* ~/data/
vogel-trainer organize ~/data/ -o ~/data/organized-v2/
vogel-trainer train ~/data/organized-v2/ -o ~/models/v2/
```
---
## Performance & Best Practices
### Dataset Size Recommendations
| Quality | Images per Species | Expected Accuracy |
|---------|-------------------|-------------------|
| Minimum | 20-30 | ~85-90% |
| Good | 50-100 | ~92-96% |
| Optimal | 100+ | >96% |
### Tips for Better Results
1. **Dataset Diversity**
- Include various lighting conditions
- Capture different poses (side, front, back)
- Cover different seasons (plumage changes)
2. **Class Balance**
- Aim for similar image counts per species
- Avoid having one dominant class
3. **Quality Over Quantity**
- Use threshold 0.5-0.6 for clear detections
- Manual review of auto-sorted images improves quality
4. **Monitor Training**
- Check per-class accuracy for weak species
- Use confusion matrix to identify similar species
- Add more data for low-performing classes
---
## ๐ Integration with vogel-video-analyzer
Use your trained model for species identification:
```bash
vogel-analyze --identify-species \
--species-model ~/models/final/ \
--species-threshold 0.3 \
video.mp4
```
---
## ๐ ๏ธ Development
```bash
# Clone repository
git clone https://github.com/kamera-linux/vogel-model-trainer.git
cd vogel-model-trainer
# Install in development mode
pip install -e ".[dev]"
# Run tests
pytest tests/
```
---
## ๐ License
MIT License - see [LICENSE](LICENSE) for details.
---
## ๐ Credits
- **YOLO** by [Ultralytics](https://github.com/ultralytics/ultralytics)
- **EfficientNet** by [Google Research](https://github.com/google/automl)
- **Transformers** by [Hugging Face](https://huggingface.co/transformers)
---
## ๐ฎ Support & Contributing
- **Issues**: [GitHub Issues](https://github.com/kamera-linux/vogel-model-trainer/issues)
- **Discussions**: [GitHub Discussions](https://github.com/kamera-linux/vogel-model-trainer/discussions)
- **Pull Requests**: Contributions welcome!
---
Made with โค๏ธ for bird watching enthusiasts ๐ฆ
Raw data
{
"_id": null,
"home_page": null,
"name": "vogel-model-trainer",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "birds, computer-vision, machine-learning, object-detection, species-classification",
"author": null,
"author_email": "kamera-linux <kamera-linux@mailbox.org>",
"download_url": "https://files.pythonhosted.org/packages/fa/35/177e0f6cc045c69a6df193274cc1d31de16410a4d7d21e7b632f65380299/vogel_model_trainer-0.1.14.tar.gz",
"platform": null,
"description": "# \ud83d\udc26 Vogel Model Trainer\n\n**Languages:** [\ud83c\uddec\ud83c\udde7 English](README.md) | [\ud83c\udde9\ud83c\uddea Deutsch](README.de.md) | [\ud83c\uddef\ud83c\uddf5 \u65e5\u672c\u8a9e](README.ja.md)\n\n<p align=\"left\">\n <a href=\"https://pypi.org/project/vogel-model-trainer/\"><img alt=\"PyPI version\" src=\"https://img.shields.io/pypi/v/vogel-model-trainer.svg\"></a>\n <a href=\"https://pypi.org/project/vogel-model-trainer/\"><img alt=\"Python Versions\" src=\"https://img.shields.io/pypi/pyversions/vogel-model-trainer.svg\"></a>\n <a href=\"https://opensource.org/licenses/MIT\"><img alt=\"License: MIT\" src=\"https://img.shields.io/badge/License-MIT-yellow.svg\"></a>\n <a href=\"https://pypi.org/project/vogel-model-trainer/\"><img alt=\"PyPI Status\" src=\"https://img.shields.io/pypi/status/vogel-model-trainer.svg\"></a>\n <a href=\"https://pepy.tech/project/vogel-model-trainer\"><img alt=\"Downloads\" src=\"https://static.pepy.tech/badge/vogel-model-trainer\"></a>\n</p>\n\n**Train custom bird species classifiers from your own video footage using YOLOv8 and EfficientNet.**\n\nA specialized toolkit for creating high-accuracy bird species classifiers tailored to your specific monitoring setup. Extract training data from videos, organize datasets, and train custom models with >96% accuracy.\n\n---\n\n## \u2728 Features\n\n- \ud83c\udfaf **YOLO-based Bird Detection** - Automated bird cropping from videos using YOLOv8\n- \ud83e\udd16 **Three Extraction Modes** - Manual labeling, auto-sorting, or standard extraction\n- \ud83d\udcc1 **Wildcard Support** - Batch process multiple videos with glob patterns\n- \ud83d\uddbc\ufe0f **Flexible Image Sizes** - 224/384/448px or keep original size\n- \ud83d\udd0d **Advanced Filtering** - Box size, blur detection, confidence thresholds\n- \ud83d\udd04 **Duplicate Detection** - Perceptual hashing removes similar images\n- \u2705 **Quality Check** - Find blurry, too-small, corrupted, or badly-exposed images\n- \ud83c\udfa8 **AI Background Removal** - Remove backgrounds with gray default for optimal training\n- \ud83e\uddf9 **Dataset Validation** - Clean transparent/gray datasets with automated checks\n- \ud83e\udde0 **EfficientNet-B0 Training** - Lightweight yet powerful classification model\n- \ud83c\udfa8 **4-Level Data Augmentation** - None/light/medium/heavy intensity options\n- \u26a1 **Mixed Precision Training** - FP16/BF16 support for faster GPU training\n- \ud83d\udcca **Advanced Training Options** - 13 configurable parameters for fine-tuning\n- \ud83d\udd27 **Dataset Deduplication** - Clean existing datasets with perceptual hashing\n- \u23f8\ufe0f **Graceful Shutdown** - Save model state on Ctrl+C interruption\n- \ud83c\udf0d **Full i18n Support** - English, German, Japanese translations\n- \ud83d\udcc8 **Per-Species Metrics** - Detailed accuracy breakdown by species\n\n## \ud83e\udd16 Pre-trained Models\n\n**German Garden Birds Classifier** - Ready to use!\n\nWe provide a pre-trained model on Hugging Face that can classify 8 common German garden bird species with 100% validation accuracy:\n\n\ud83d\udd17 **[kamera-linux/german-bird-classifier](https://huggingface.co/kamera-linux/german-bird-classifier)**\n\n**Supported Species:**\n- Blaumeise (Blue Tit)\n- Gr\u00fcnling (European Greenfinch)\n- Haussperling (House Sparrow)\n- Kernbei\u00dfer (Hawfinch)\n- Kleiber (Eurasian Nuthatch)\n- Kohlmeise (Great Tit)\n- Rotkehlchen (European Robin)\n- Sumpfmeise (Marsh Tit)\n\n**Usage with extraction:**\n```bash\nvogel-trainer extract --folder ~/bird-data \\\n --species-model kamera-linux/german-bird-classifier \\\n --sample-rate 20 --skip-blurry --deduplicate \\\n video.mp4\n```\n\nThe model will automatically classify detected birds during extraction!\n\n---\n\n## \ud83d\ude80 Quick Start\n\n### Installation\n\n#### Recommended: Using Virtual Environment\n\n```bash\n# Install venv if needed (Debian/Ubuntu)\nsudo apt install python3-venv\n\n# Create virtual environment\npython3 -m venv ~/venv-vogel\n\n# Activate it\nsource ~/venv-vogel/bin/activate # On Windows: ~/venv-vogel\\Scripts\\activate\n\n# Install package\npip install vogel-model-trainer\n```\n\n#### Quick Install\n\n```bash\n# Install from PyPI\npip install vogel-model-trainer\n\n# Or install from source\ngit clone https://github.com/kamera-linux/vogel-model-trainer.git\ncd vogel-model-trainer\npip install -e .\n```\n\n### \ud83c\udfa5 Video Tutorials\n\nLearn vogel-model-trainer with step-by-step video guides:\n\n- **Getting Started** - Installation and first extraction (5 min)\n- **Extracting Birds** - Quality filters, deduplication, species classification (10 min)\n- **Organizing Datasets** - Train/val splits, class balance management (8 min) **NEW in v0.1.8**\n- **Training Models** - Custom classifier training and parameters (12 min)\n- **Testing & Evaluation** - Model testing and performance analysis (7 min)\n\n\ud83d\udcfa *Video tutorials coming soon!*\n\n### Basic Workflow\n\n```bash\n# 1. Extract bird images from videos\nvogel-trainer extract video.mp4 --folder ~/training-data/ --bird kohlmeise\n\n# 2. Organize into train/validation split\nvogel-trainer organize ~/training-data/ -o ~/organized-data/\n\n# 3. Train custom classifier\nvogel-trainer train ~/organized-data/ -o ~/models/my-classifier/\n\n# 4. Test the trained model\nvogel-trainer test ~/models/my-classifier/ -d ~/organized-data/\n```\n\n---\n\n## \ud83d\udcd6 Usage Guide\n\n### Using as a Library (New in v0.1.2)\n\nAll core functions can now be used programmatically in your Python code:\n\n```python\nfrom vogel_model_trainer.core import extractor, organizer, trainer, tester\n\n# Extract birds from video\nextractor.extract_birds_from_video(\n video_path=\"video.mp4\",\n output_dir=\"output/\",\n bird_species=\"great-tit\",\n detection_model=\"yolov8n.pt\",\n species_model=None,\n threshold=0.5,\n sample_rate=3,\n resize_to_target=True\n)\n\n# Organize into train/val splits\norganizer.organize_dataset(\n source_dir=\"output/\",\n output_dir=\"dataset/\",\n train_ratio=0.8\n)\n\n# Train model\ntrainer.train_model(\n data_dir=\"dataset/\",\n output_dir=\"models/\",\n model_name=\"google/efficientnet-b0\",\n batch_size=16,\n num_epochs=50,\n learning_rate=3e-4\n)\n\n# Test model\nresults = tester.test_model(\n model_path=\"models/bird_classifier/\",\n data_dir=\"dataset/\"\n)\nprint(f\"Accuracy: {results['accuracy']:.2%}\")\n```\n\n### 1. Extract Training Images\n\n#### Manual Mode (Recommended for Initial Collection)\n\nWhen you know the species in your video:\n\n```bash\nvogel-trainer extract ~/Videos/great-tit.mp4 \\\n --folder ~/training-data/ \\\n --bird great-tit \\\n --threshold 0.5 \\\n --sample-rate 3\n```\n\n#### Auto-Sort Mode (For Iterative Training)\n\nUse an existing model to automatically classify and sort:\n\n```bash\nvogel-trainer extract ~/Videos/mixed.mp4 \\\n --folder ~/training-data/ \\\n --species-model ~/models/classifier/final/ \\\n --threshold 0.5\n```\n\n#### Batch Processing with Wildcards\n\n```bash\n# Process all videos in a directory\nvogel-trainer extract \"~/Videos/*.mp4\" --folder ~/data/ --bird blue-tit\n\n# Recursive directory search\nvogel-trainer extract ~/Videos/ \\\n --folder ~/data/ \\\n --bird amsel \\\n --recursive\n```\n\n**Parameters:**\n- `--folder`: Base directory for extracted images (required)\n- `--bird`: Manual species label (creates subdirectory)\n- `--species-model`: Path to trained model for auto-classification\n- `--species-threshold`: Minimum confidence for species classification (e.g., 0.85 for 85%)\n- `--threshold`: YOLO confidence threshold (default: 0.5)\n- `--sample-rate`: Process every Nth frame (default: 3)\n- `--detection-model`: YOLO model path (default: yolov8n.pt)\n- `--image-size`: Target image size in pixels (default: 224, use 0 for original size)\n- `--max-detections`: Maximum detections per frame (default: 10)\n- `--min-box-size`: Minimum bounding box size in pixels (default: 50)\n- `--max-box-size`: Maximum bounding box size in pixels (default: 800)\n- `--quality`: JPEG quality 1-100 (default: 95)\n- `--skip-blurry`: Skip blurry/out-of-focus images (experimental)\n- `--deduplicate`: Skip duplicate/similar images using perceptual hashing\n- `--similarity-threshold`: Similarity threshold for duplicates - Hamming distance 0-64 (default: 5)\n- `--min-sharpness`: **NEW v0.1.9** - Minimum sharpness score (Laplacian variance, typical: 100-300)\n- `--min-edge-quality`: **NEW v0.1.9** - Minimum edge quality (Sobel gradient, typical: 50-150)\n- `--save-quality-report`: **NEW v0.1.9** - Generate detailed quality statistics report\n- `--remove-background`: **\ud83e\uddea EXPERIMENTAL v0.1.11** - Remove background using AI (rembg)\n- `--bg-color [white|black|gray]`: **\ud83e\uddea EXPERIMENTAL v0.1.11** - Background color (default: white)\n- `--bg-model [u2net|u2netp|isnet-general-use]`: **\ud83e\uddea EXPERIMENTAL v0.1.11** - AI model for background removal (default: u2net)\n- `--recursive, -r`: Search directories recursively\n- `--log`: Save console output to log file (`/var/log/vogel-kamera-linux/YYYY/KWXX/`)\n\n**Advanced Filtering Examples:**\n\n```bash\n# High-quality extraction with all filters (v0.1.11)\nvogel-trainer extract video.mp4 \\\n --folder data/ \\\n --bird rotkehlchen \\\n --threshold 0.6 \\\n --min-box-size 80 \\\n --max-box-size 600 \\\n --min-sharpness 150 \\\n --min-edge-quality 80 \\\n --skip-blurry \\\n --deduplicate \\\n --save-quality-report \\\n --remove-background \\\n --bg-color white \\\n --bg-model u2net\n\n# Background removal with black background for contrast\nvogel-trainer extract video.mp4 \\\n --folder data/ \\\n --bird blaumeise \\\n --remove-background \\\n --bg-color black \\\n --bg-model isnet-general-use\n```\n\n**\ud83e\uddea Background Removal (EXPERIMENTAL v0.1.11+, Stable v0.1.14):**\n\nThe `--remove-background` feature uses AI-powered rembg library to automatically segment birds from backgrounds.\n\n**NEW in v0.1.14:** Gray background is now the DEFAULT for optimal training! Smaller JPEG files, better compatibility.\n\n- **Models:**\n - `u2net` (default): Best overall quality, ~180MB download\n - `u2netp`: Faster, smaller model for quick processing\n - `isnet-general-use`: Best edge quality for detailed feathers\n\n- **Background Colors (NEW DEFAULT v0.1.14):**\n - `gray` (DEFAULT): Neutral gray background (#808080) - best for training\n - `white`: Clean white background (#FFFFFF)\n - `black`: High contrast black background (#000000)\n - `green-screen`: Chroma key green (#00FF00)\n - `blue-screen`: Chroma key blue (#0000FF)\n\n- **Transparency Options:**\n - `--bg-transparent`: Create PNG with alpha channel (flexible but larger files)\n - `--no-bg-transparent` (DEFAULT): Use colored background (smaller JPEG files)\n - `--bg-fill-black`: Makes black box areas transparent (requires --bg-transparent)\n - `--no-bg-fill-black` (DEFAULT): Keep padding areas with background color\n\n- **Features:**\n - AI-based U\u00b2-Net segmentation for accurate bird isolation\n - Alpha matting for smooth, professional edges\n - Post-processing with morphological operations\n - Handles complex backgrounds (branches, leaves, buildings)\n - Works with varied bird plumage and fine feather details\n - Automatically saves as PNG (transparent) or JPEG (opaque)\n\n- **Note:** First use downloads ~180MB model (cached afterward), requires `rembg>=2.0.50` dependency\n --remove-background \\\n --quality 98\n\n# Extract with duplicate detection (prevents similar images)\nvogel-trainer extract ~/Videos/*.mp4 \\\n --folder data/ \\\n --bird kohlmeise \\\n --deduplicate \\\n --similarity-threshold 3 # Stricter duplicate detection\n\n# Large image size for high-detail training\nvogel-trainer extract video.mp4 \\\n --folder data/ \\\n --bird amsel \\\n --image-size 384 # Larger images for better quality\n\n# Auto-sort with confidence filter (only high-confidence classifications)\nvogel-trainer extract video.mp4 \\\n --folder data/ \\\n --species-model ~/models/classifier/ \\\n --species-threshold 0.90 \\\n --deduplicate\n```\n\n**Logging Example:**\n\n```bash\n# Save output to log file\nvogel-trainer extract ~/Videos/great-tit.mp4 \\\n --folder ~/data/ \\\n --bird great-tit \\\n --log\n\n# Log file path: /var/log/vogel-kamera-linux/2025/KW45/20251109_160000_extract.log\n```\n\n### 1b. Clean Dataset Images (NEW v0.1.12+) \ud83e\uddf9\n\n**Clean Transparent Images** - For transparent PNG datasets:\n\n```bash\n# Safe mode: Report only (no files modified)\nvogel-trainer clean-transparent ~/training-data/ --mode report\n\n# Move invalid images to invalid_transparent/ folder\nvogel-trainer clean-transparent ~/training-data/ --mode move --recursive\n```\n\n**Clean Gray Background Images (NEW v0.1.14)** - For gray background datasets:\n\n```bash\n# Check gray background ratio\nvogel-trainer clean-gray ~/training-data/ --mode report\n\n# Move images with wrong gray ratio to invalid_gray/\nvogel-trainer clean-gray ~/training-data/ --mode move --recursive\n\n# Custom thresholds\nvogel-trainer clean-gray ~/training-data/ \\\n --min-gray 0.10 \\\n --max-gray 0.90 \\\n --gray-tolerance 30 \\\n --mode move\n```\n\n**Detection Criteria:**\n\n*clean-transparent:*\n- **Min Visible Pixels** (`--min-pixels`, default: 500): Minimum non-transparent pixels\n- **Max Transparency** (`--max-transparency`, default: 0.95): Maximum 95% transparency allowed\n- **Min Region Size** (`--min-region`, default: 100): Minimum size of largest connected object\n\n*clean-gray:*\n- **Min Gray Ratio** (`--min-gray`, default: 0.05): Minimum 5% gray background required\n- **Max Gray Ratio** (`--max-gray`, default: 0.95): Maximum 95% gray allowed (need visible bird)\n- **Gray Tolerance** (`--gray-tolerance`, default: 30): Tolerance for gray detection (R\u2248G\u2248B\u00b130)\n\n**Use Cases:**\n- Remove tiny fragments after background removal\n- Clean up partial detections (bird flew out of frame)\n- Eliminate images with too much/little background\n- Find images where bird is barely visible or missing\n\n### 2. Organize Dataset\n\n```bash\n# Basic organization (80/20 split)\nvogel-trainer organize ~/training-data/ -o ~/organized-data/\n\n# With class balance control (NEW in v0.1.8)\nvogel-trainer organize ~/training-data/ -o ~/organized-data/ \\\n --max-images-per-class 100 \\\n --tolerance 15.0\n```\n\n**Class Balance Features:**\n- `--max-images-per-class N`: Limit to N images per class, delete excess\n- `--tolerance N`: Maximum allowed imbalance % (default: 15)\n - < 10%: \u2705 Perfect\n - 10-15%: \u26a0\ufe0f Warning\n - > 15%: \u274c Error with recommendations\n\nCreates an 80/20 train/validation split:\n```\norganized/\n\u251c\u2500\u2500 train/\n\u2502 \u251c\u2500\u2500 great-tit/\n\u2502 \u251c\u2500\u2500 blue-tit/\n\u2502 \u2514\u2500\u2500 robin/\n\u2514\u2500\u2500 val/\n \u251c\u2500\u2500 great-tit/\n \u251c\u2500\u2500 blue-tit/\n \u2514\u2500\u2500 robin/\n```\n\n### 3. Train Classifier\n\n```bash\nvogel-trainer train ~/organized-data/ -o ~/models/my-classifier/\n```\n\n**Training Parameters:**\n- `--batch-size`: Training batch size (default: 16)\n- `--epochs`: Number of training epochs (default: 50)\n- `--learning-rate`: Learning rate (default: 2e-4)\n- `--early-stopping-patience`: Early stopping patience in epochs (default: 5, 0 to disable)\n- `--weight-decay`: Weight decay for regularization (default: 0.01)\n- `--warmup-ratio`: Learning rate warmup ratio (default: 0.1)\n- `--label-smoothing`: Label smoothing factor (default: 0.1, 0 to disable)\n- `--save-total-limit`: Maximum checkpoints to keep (default: 3)\n- `--augmentation-strength`: Data augmentation intensity: `none`, `light`, `medium` (default), `heavy`\n- `--image-size`: Input image size in pixels (default: 224, supports 224/384/448)\n- `--scheduler`: Learning rate scheduler: `cosine` (default), `linear`, `constant`\n- `--seed`: Random seed for reproducibility (default: 42)\n- `--resume-from-checkpoint`: Path to checkpoint to resume training\n- `--gradient-accumulation-steps`: Gradient accumulation steps (default: 1)\n- `--mixed-precision`: Mixed precision training: `no` (default), `fp16`, `bf16`\n- `--push-to-hub`: Push model to HuggingFace Hub (default: False)\n- `--log`: Save console output to log file\n\n**Augmentation Strength Levels:**\n\n- **none**: No augmentation (only normalization)\n- **light**: Small rotations (\u00b110\u00b0), minimal color jitter\n- **medium** (default): Moderate rotations (\u00b120\u00b0), affine transforms, color jitter, gaussian blur\n- **heavy**: Strong rotations (\u00b130\u00b0), aggressive transforms, strong color variations\n\n**Advanced Training Examples:**\n\n```bash\n# High-accuracy training with large images and heavy augmentation\nvogel-trainer train ~/organized-data/ \\\n -o ~/models/high-accuracy/ \\\n --image-size 384 \\\n --augmentation-strength heavy \\\n --epochs 100 \\\n --early-stopping-patience 10 \\\n --batch-size 8\n\n# Fast training with mixed precision (requires GPU)\nvogel-trainer train ~/organized-data/ \\\n -o ~/models/fast/ \\\n --mixed-precision fp16 \\\n --batch-size 32 \\\n --gradient-accumulation-steps 2\n\n# Reproducible training with fixed seed\nvogel-trainer train ~/organized-data/ \\\n -o ~/models/reproducible/ \\\n --seed 12345 \\\n --augmentation-strength light\n\n# Resume interrupted training\nvogel-trainer train ~/organized-data/ \\\n -o ~/models/continued/ \\\n --resume-from-checkpoint ~/models/my-classifier/checkpoints/checkpoint-1000\n\n# Training with logging\nvogel-trainer train ~/organized-data/ \\\n -o ~/models/logged/ \\\n --log\n```\n\n**Training Configuration:**\n- Base Model: `google/efficientnet-b0` (8.5M parameters)\n- Optimizer: AdamW with configurable LR schedule\n- Augmentation: 4 intensity levels (none/light/medium/heavy)\n- Regularization: Weight decay, label smoothing, early stopping\n- Mixed Precision: FP16/BF16 support for faster training on GPU\n\n**Output:**\n```\n~/models/my-classifier/\n\u251c\u2500\u2500 checkpoints/ # Intermediate checkpoints\n\u251c\u2500\u2500 logs/ # TensorBoard logs\n\u2514\u2500\u2500 final/ # Final trained model\n \u251c\u2500\u2500 config.json\n \u251c\u2500\u2500 model.safetensors\n \u2514\u2500\u2500 preprocessor_config.json\n```\n\n### 4. Deduplicate Dataset\n\nRemove duplicate or very similar images from your dataset to improve training quality:\n\n```bash\n# Report duplicates without deleting\nvogel-trainer deduplicate ~/training-data/ --recursive\n\n# Delete duplicates (keep first occurrence)\nvogel-trainer deduplicate ~/training-data/ \\\n --mode delete \\\n --recursive\n\n# Move duplicates to separate folder\nvogel-trainer deduplicate ~/training-data/ \\\n --mode move \\\n --recursive\n\n# Stricter duplicate detection\nvogel-trainer deduplicate ~/training-data/ \\\n --threshold 3 \\\n --recursive\n\n# Keep largest file instead of first\nvogel-trainer deduplicate ~/training-data/ \\\n --mode delete \\\n --keep largest \\\n --recursive\n```\n\n**Deduplication Parameters:**\n- `--threshold`: Similarity threshold - Hamming distance 0-64, lower=stricter (default: 5)\n- `--method`: Hash method: `phash` (default, recommended), `dhash`, `whash`, `average_hash`\n- `--mode`: Action: `report` (show only, default), `delete` (remove), `move` (to duplicates/)\n- `--keep`: Which duplicate to keep: `first` (chronological, default) or `largest` (file size)\n- `--recursive, -r`: Search recursively through subdirectories\n\n**How it works:**\n- Uses perceptual hashing (pHash) to detect visually similar images\n- Robust against resizing, cropping, and minor color changes\n- Threshold of 5 = very similar, 10 = similar, >15 = different\n- Safe default: `report` mode prevents accidental deletion\n\n### 5. Quality Check Dataset (New!)\n\nCheck your dataset for low-quality images (blurry, too small, corrupted, brightness issues):\n\n```bash\n# Report quality issues without deleting\nvogel-trainer quality-check ~/training-data/ --recursive\n\n# Delete low-quality images\nvogel-trainer quality-check ~/training-data/ \\\n --mode delete \\\n --recursive\n\n# Move low-quality images to separate folder\nvogel-trainer quality-check ~/training-data/ \\\n --mode move \\\n --recursive\n\n# Stricter blur detection\nvogel-trainer quality-check ~/training-data/ \\\n --blur-threshold 150.0 \\\n --recursive\n\n# Check for brightness/contrast issues\nvogel-trainer quality-check ~/training-data/ \\\n --check-brightness \\\n --recursive\n\n# Comprehensive quality check with custom thresholds\nvogel-trainer quality-check ~/training-data/ \\\n --blur-threshold 120.0 \\\n --min-resolution 100 \\\n --min-filesize 2048 \\\n --check-brightness \\\n --mode move \\\n --recursive\n```\n\n**Quality Check Parameters:**\n- `--blur-threshold`: Minimum blur score (Laplacian variance), lower=more blurry (default: 100.0)\n- `--min-resolution`: Minimum image width/height in pixels (default: 50)\n- `--min-filesize`: Minimum file size in bytes (default: 1024)\n- `--check-brightness`: Also check for brightness/contrast issues (too dark or overexposed)\n- `--mode`: Action: `report` (show only, default), `delete` (remove), `move` (to low_quality/)\n- `--recursive, -r`: Search recursively through subdirectories\n\n**\u26a0\ufe0f WARNING - Delete Mode:**\n- The `--mode delete` option **permanently removes files** without backup\n- **Always run `--mode report` first** to preview what will be deleted\n- **Backup your dataset** before using delete mode\n- Consider using `--mode move` instead (keeps files in `low_quality/` folder)\n\n**What is checked:**\n- \u2705 **Sharpness**: Detects blurry/out-of-focus images using Laplacian variance\n- \u2705 **Resolution**: Filters out too-small images that hurt training\n- \u2705 **File size**: Detects corrupted or empty files\n- \u2705 **Readability**: Checks if images can be opened and processed\n- \u2705 **Brightness** (optional): Detects too-dark or overexposed images\n\n**Typical thresholds:**\n- Blur: 100.0 (default) = moderate, 150.0 = stricter, 50.0 = more lenient\n- Resolution: 50px (default) = very permissive, 100px = recommended, 224px = strict\n- Filesize: 1024 bytes (default) = catches corrupted files\n\n**Recommended workflow:**\n```bash\n# 1. Preview issues first (safe)\nvogel-trainer quality-check ~/data/ --mode report --recursive\n\n# 2. Move problematic images (reversible)\nvogel-trainer quality-check ~/data/ --mode move --recursive\n\n# 3. Review moved files in low_quality/ folder\n# 4. Delete manually if satisfied: rm -rf ~/data/low_quality/\n```\n\n### 6. Test Model\n\n```bash\n# Test on validation dataset\nvogel-trainer test ~/models/my-classifier/ -d ~/organized-data/\n\n# Output:\n# \ud83e\uddea Testing model on validation set...\n# \ud83d\udc26 Predicted: great-tit (98.5% confidence)\n```\n\n---\n\n## \ud83d\udd04 Iterative Training Workflow\nImprove your model accuracy through iterative refinement using auto-classification:\n\n```mermaid\nflowchart TD\n Start([\ud83d\udccb Phase 1: Initial Model<br/>Manual Labeling]) --> Extract1[1\ufe0f\u20e3 Extract with manual labels<br/><code>vogel-trainer extract video.mp4<br/>--folder data/ --bird kohlmeise</code>]\n \n Extract1 --> Organize1[2\ufe0f\u20e3 Organize dataset 80/20 split<br/><code>vogel-trainer organize data/<br/>-o organized/</code>]\n \n Organize1 --> Train1[3\ufe0f\u20e3 Train initial model<br/><code>vogel-trainer train organized/<br/>-o models/v1/</code><br/>\u2705 <b>Result: 92% accuracy</b>]\n \n Train1 --> Phase2([\ud83d\udd04 Phase 2: Model Improvement<br/>Auto-Classification])\n \n Phase2 --> Extract2[4\ufe0f\u20e3 Auto-extract with trained model<br/><code>vogel-trainer extract new-videos/<br/>--folder data-v2/<br/>--species-model models/v1/final/<br/>--species-threshold 0.85</code><br/>\ud83c\udfaf <b>Automatically sorted by species!</b>]\n \n Extract2 --> Review[5\ufe0f\u20e3 Manual review & corrections<br/>\u2022 Check auto-classifications<br/>\u2022 Move misclassified images<br/>\u2022 Merge with previous dataset]\n \n Review --> Train2[6\ufe0f\u20e3 Retrain with expanded dataset<br/><code>vogel-trainer organize data-v2/<br/>-o organized-v2/<br/>vogel-trainer train organized-v2/<br/>-o models/v2/</code><br/>\ud83c\udf89 <b>Result: 96% accuracy!</b>]\n \n Train2 --> Repeat{\u267b\ufe0f Continue<br/>improving?}\n Repeat -->|Yes| Extract2\n Repeat -->|No| End([\u2705 Final Model])\n \n style Start fill:#e1f5ff,stroke:#0066cc,stroke-width:3px\n style Phase2 fill:#e1f5ff,stroke:#0066cc,stroke-width:3px\n style Train1 fill:#d4edda,stroke:#28a745,stroke-width:2px\n style Train2 fill:#d4edda,stroke:#28a745,stroke-width:2px\n style End fill:#d4edda,stroke:#28a745,stroke-width:3px\n style Extract2 fill:#fff3cd,stroke:#ffc107,stroke-width:2px\n style Review fill:#f8d7da,stroke:#dc3545,stroke-width:2px\n```\n\n**Key Benefits:**\n- \ud83d\ude80 **Faster labeling**: Auto-classification saves manual work\n- \ud83d\udcc8 **Better accuracy**: More training data = better model\n- \ud83c\udfaf **Quality control**: `--species-threshold` filters uncertain predictions\n- \ud83d\udd04 **Continuous improvement**: Each iteration improves the model\n\n**Example Commands:**\n\n```bash\n# Phase 1: Manual training (initial dataset)\nvogel-trainer extract ~/Videos/batch1/*.mp4 --folder ~/data/ --bird great-tit\nvogel-trainer organize ~/data/ -o ~/data/organized/\nvogel-trainer train ~/data/organized/ -o ~/models/v1/\n\n# Phase 2: Auto-classification with trained model\nvogel-trainer extract ~/Videos/batch2/*.mp4 \\\n --folder ~/data-v2/ \\\n --species-model ~/models/v1/final/ \\\n --species-threshold 0.85\n\n# Review classifications in ~/data-v2/<species>/ folders\n# Move any misclassified images to correct species folders\n\n# Merge datasets and retrain\ncp -r ~/data-v2/* ~/data/\nvogel-trainer organize ~/data/ -o ~/data/organized-v2/\nvogel-trainer train ~/data/organized-v2/ -o ~/models/v2/\n```\n\n---\n\n## Performance & Best Practices\n\n### Dataset Size Recommendations\n\n| Quality | Images per Species | Expected Accuracy |\n|---------|-------------------|-------------------|\n| Minimum | 20-30 | ~85-90% |\n| Good | 50-100 | ~92-96% |\n| Optimal | 100+ | >96% |\n\n### Tips for Better Results\n\n1. **Dataset Diversity**\n - Include various lighting conditions\n - Capture different poses (side, front, back)\n - Cover different seasons (plumage changes)\n\n2. **Class Balance**\n - Aim for similar image counts per species\n - Avoid having one dominant class\n\n3. **Quality Over Quantity**\n - Use threshold 0.5-0.6 for clear detections\n - Manual review of auto-sorted images improves quality\n\n4. **Monitor Training**\n - Check per-class accuracy for weak species\n - Use confusion matrix to identify similar species\n - Add more data for low-performing classes\n\n---\n\n## \ud83d\udd17 Integration with vogel-video-analyzer\n\nUse your trained model for species identification:\n\n```bash\nvogel-analyze --identify-species \\\n --species-model ~/models/final/ \\\n --species-threshold 0.3 \\\n video.mp4\n```\n\n---\n\n## \ud83d\udee0\ufe0f Development\n\n```bash\n# Clone repository\ngit clone https://github.com/kamera-linux/vogel-model-trainer.git\ncd vogel-model-trainer\n\n# Install in development mode\npip install -e \".[dev]\"\n\n# Run tests\npytest tests/\n```\n\n---\n\n## \ud83d\udcdd License\n\nMIT License - see [LICENSE](LICENSE) for details.\n\n---\n\n## \ud83d\ude4f Credits\n\n- **YOLO** by [Ultralytics](https://github.com/ultralytics/ultralytics)\n- **EfficientNet** by [Google Research](https://github.com/google/automl)\n- **Transformers** by [Hugging Face](https://huggingface.co/transformers)\n\n---\n\n## \ud83d\udcee Support & Contributing\n\n- **Issues**: [GitHub Issues](https://github.com/kamera-linux/vogel-model-trainer/issues)\n- **Discussions**: [GitHub Discussions](https://github.com/kamera-linux/vogel-model-trainer/discussions)\n- **Pull Requests**: Contributions welcome!\n\n---\n\nMade with \u2764\ufe0f for bird watching enthusiasts \ud83d\udc26\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A tool to extract bird images from videos and train custom species classifiers",
"version": "0.1.14",
"project_urls": {
"Homepage": "https://github.com/kamera-linux/vogel-model-trainer",
"Issues": "https://github.com/kamera-linux/vogel-model-trainer/issues",
"Repository": "https://github.com/kamera-linux/vogel-model-trainer"
},
"split_keywords": [
"birds",
" computer-vision",
" machine-learning",
" object-detection",
" species-classification"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "e598c3f53d74dcc65fa1eee946d3f85ad501103ef6c6edfae621063eb21d3da5",
"md5": "0b34e86117f0077c2ac5fdd2eeb96a93",
"sha256": "ad0daed2b0dfdcee32fb3af739879439444464944521d75d42b5f79b039964ed"
},
"downloads": -1,
"filename": "vogel_model_trainer-0.1.14-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0b34e86117f0077c2ac5fdd2eeb96a93",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 60130,
"upload_time": "2025-11-16T11:47:34",
"upload_time_iso_8601": "2025-11-16T11:47:34.922082Z",
"url": "https://files.pythonhosted.org/packages/e5/98/c3f53d74dcc65fa1eee946d3f85ad501103ef6c6edfae621063eb21d3da5/vogel_model_trainer-0.1.14-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "fa35177e0f6cc045c69a6df193274cc1d31de16410a4d7d21e7b632f65380299",
"md5": "5eb0ae22f3da3fb6778b0e5b22017eb2",
"sha256": "2f195a0986827bff63ee2a159327ab0dd8dab9ef4fe42339178f5f486f6b77ab"
},
"downloads": -1,
"filename": "vogel_model_trainer-0.1.14.tar.gz",
"has_sig": false,
"md5_digest": "5eb0ae22f3da3fb6778b0e5b22017eb2",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 129975,
"upload_time": "2025-11-16T11:47:36",
"upload_time_iso_8601": "2025-11-16T11:47:36.256698Z",
"url": "https://files.pythonhosted.org/packages/fa/35/177e0f6cc045c69a6df193274cc1d31de16410a4d7d21e7b632f65380299/vogel_model_trainer-0.1.14.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-11-16 11:47:36",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "kamera-linux",
"github_project": "vogel-model-trainer",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "vogel-model-trainer"
}
JSON
