# DeepFV - Fisher Vectors with Deep Learning
A TensorFlow-based implementation of Improved Fisher Vectors as described in [1]. This package provides a modern, scalable approach to computing Fisher Vectors using deep learning techniques. For a concise description of Fisher Vectors see [2].
## Features
- **Full & Diagonal Covariance Support**: Model complex elliptical clusters with full covariance matrices, or use diagonal covariance for faster training
- **Mini-batch Training**: Scalable to large datasets with mini-batch gradient descent
- **Variable-length Bags**: Native support for Multiple Instance Learning with variable instances per bag
- **BIC-based Model Selection**: Automatically determine optimal number of GMM components
- **GPU Acceleration**: Built on TensorFlow 2.x for fast training on GPUs
- **MiniBatchKMeans Initialization**: Smart initialization using scikit-learn's MiniBatchKMeans
- **Memory-efficient Batch Processing**: Handle millions of samples with configurable batch sizes
- **Save/Load Models**: Persist trained models for reuse
- **Normalized Fisher Vectors**: Implements improved Fisher Vector normalization
## Installation
Install from PyPI:
```bash
pip install DeepFV
```
Or install from source:
```bash
git clone https://github.com/sidhomj/DeepFV.git
cd DeepFV
pip install -r requirements.txt
pip install -e .
```
## Quick Start
### 1. Prepare your data
```python
import numpy as np
# Example: SIFT descriptors from images
shape = [300, 20, 32] # (n_samples, n_descriptors_per_sample, feature_dim)
sample_data = np.concatenate([
np.random.normal(-np.ones(30), size=shape),
np.random.normal(np.ones(30), size=shape)
], axis=0)
```
### 2. Train with mini-batch gradient descent
```python
from DeepFV import FisherVectorDL
# Create model with FULL covariance support
fv_dl = FisherVectorDL(
n_kernels=10,
feature_dim=32,
covariance_type='full' # or 'diag' for diagonal covariance
)
# Fit with mini-batch training
fv_dl.fit_minibatch(
sample_data,
epochs=100,
batch_size=1024*6,
learning_rate=0.001,
verbose=True
)
```
### 3. BIC-based model selection
```python
# Automatically select optimal number of components
fv_dl = FisherVectorDL(feature_dim=32, covariance_type='full')
fv_dl.fit_by_bic(
sample_data,
choices_n_kernels=[2, 5, 10, 20],
epochs=80,
batch_size=1024,
verbose=True
)
print(f"Selected {fv_dl.n_kernels} components")
```
### 4. Compute Fisher Vectors
For data with multiple descriptors per sample (3D):
```python
# Compute normalized Fisher Vectors
sample_data_test = sample_data[:20]
fisher_vectors = fv_dl.predict_fisher_vector(sample_data_test, normalized=True)
# Output shape: (n_samples, 2*n_kernels, feature_dim)
print(f"Fisher vector shape: {fisher_vectors.shape}")
```
For simple 2D data (each sample is a single feature vector):
```python
# 2D input: (n_samples, feature_dim)
simple_data = np.random.randn(100, 32)
fisher_vectors_2d = fv_dl.predict_fisher_vector(simple_data, normalized=True)
# Output shape: (n_samples, 2*n_kernels, feature_dim)
print(f"Fisher vector shape: {fisher_vectors_2d.shape}")
```
### 5. Variable-length bags (Multiple Instance Learning) - OPTIMIZED!
For datasets where each bag contains a variable number of instances. **Uses vectorized computation for 10-100x speedup!**
```python
# Example: 3 images with different numbers of SIFT descriptors
X = np.random.randn(245, 128) # 245 total descriptors, 128-dim features
# bag_ids maps each instance to its bag
# Image 0 has 50 descriptors, Image 1 has 120, Image 2 has 75
bag_ids = np.array([0]*50 + [1]*120 + [2]*75)
# Train on all instances (ignoring bag structure)
fv_dl = FisherVectorDL(n_kernels=10, feature_dim=128)
fv_dl.fit_minibatch(X, epochs=100, verbose=True)
# Compute Fisher Vectors per bag (FAST - vectorized!)
fisher_vectors, unique_bag_ids = fv_dl.predict_fisher_vector_bags(
X,
bag_ids,
normalized=True,
verbose=True
)
print(f"Fisher vectors shape: {fisher_vectors.shape}") # (3, 20, 128)
print(f"Bag IDs: {unique_bag_ids}") # [0, 1, 2]
```
**Use cases for bag-level Fisher Vectors:**
- **Image retrieval**: Variable number of SIFT/SURF descriptors per image
- **Document classification**: Variable number of word embeddings per document
- **Multiple Instance Learning (MIL)**: Variable instances per bag in medical imaging, etc.
- **Time series**: Variable-length sequences aggregated into fixed representations
**Get instance-level Fisher Vectors too:**
```python
# Optionally return both bag-level AND instance-level Fisher Vectors
fisher_vectors, unique_bag_ids, instance_fvs = fv_dl.predict_fisher_vector_bags(
X,
bag_ids,
return_instance_level=True, # Also return per-instance FVs
verbose=True
)
print(f"Bag-level FVs: {fisher_vectors.shape}") # (3, 20, 128) - 3 bags
print(f"Instance-level FVs: {instance_fvs.shape}") # (245, 20, 128) - 245 instances
```
**Performance:**
- **1M instances, 10K bags**: ~0.5-2 seconds (vs ~60 seconds with old approach)
- **Fully vectorized**: Single computation for all instances
- **Scales to millions**: Can handle massive datasets efficiently
### 6. Save and load models
```python
# Save trained model
fv_dl.save_model('my_model.pkl')
# Load model later
from DeepFV import FisherVectorDL
fv_dl_loaded = FisherVectorDL.load_model('my_model.pkl')
```
## Why FisherVectorDL?
### Advantages over traditional GMM implementations:
1. **Full Covariance Support**: Model rotated/tilted elliptical clusters, not just axis-aligned ones
2. **Scalability**: Mini-batch training handles datasets too large to fit in memory
3. **Speed**: GPU acceleration via TensorFlow for faster training
4. **Flexibility**: Customizable learning rate, batch size, and number of epochs
5. **Modern Stack**: Built on TensorFlow 2.x with eager execution
6. **Smart Initialization**: Uses MiniBatchKMeans for better starting parameters
## Testing
Run the test script to see a 2D visualization:
```bash
python test_fishervector_dl.py
```
This will:
- Generate 3 elliptical Gaussian clusters
- Train a GMM with full covariance
- Use BIC to select optimal number of components
- Compute and visualize Fisher Vectors
- Save visualizations as PNG files
### Example Results
**GMM Clustering with BIC Selection:**
The plot shows how full covariance GMMs can model rotated elliptical clusters. The BIC criterion automatically selects the optimal number of components.
**Fisher Vector Visualization:**
Left: Original 2D data colored by true cluster labels. Right: Fisher Vectors projected back to 2D using PCA, showing how the representation captures cluster structure.
## Contributors
* **John-William Sidhom** (https://github.com/sidhomj/) - Main contributor, TensorFlow implementation with full covariance support
### Original Contributors:
* Jonas Rothfuss (https://github.com/jonasrothfuss/) - Original implementation
* Fabio Ferreira (https://github.com/ferreirafabio/) - Original implementation
## References
- [1] Perronnin, F., Sánchez, J., & Mensink, T. (2010). Improving the fisher kernel for large-scale image classification. In European conference on computer vision (pp. 143-156). Springer, Berlin, Heidelberg. https://www.robots.ox.ac.uk/~vgg/rg/papers/peronnin_etal_ECCV10.pdf
- [2] Fisher Vector Fundamentals - VLFeat Documentation: http://www.vlfeat.org/api/fisher-fundamentals.html
## License
MIT License - see LICENSE file for details
Raw data
{
"_id": null,
"home_page": "https://github.com/sidhomj/DeepFV",
"name": "DeepFV",
"maintainer": "John-William Sidhom",
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": null,
"keywords": "fisher-vectors, gaussian-mixture-model, deep-learning, tensorflow, computer-vision",
"author": "John-William Sidhom",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/65/50/c0d22fa01099f2fd4e5e4d8358e4e13725343abff3cae1de32974a24c952/deepfv-0.2.10.tar.gz",
"platform": null,
"description": "\n# DeepFV - Fisher Vectors with Deep Learning\n\nA TensorFlow-based implementation of Improved Fisher Vectors as described in [1]. This package provides a modern, scalable approach to computing Fisher Vectors using deep learning techniques. For a concise description of Fisher Vectors see [2].\n\n## Features\n\n- **Full & Diagonal Covariance Support**: Model complex elliptical clusters with full covariance matrices, or use diagonal covariance for faster training\n- **Mini-batch Training**: Scalable to large datasets with mini-batch gradient descent\n- **Variable-length Bags**: Native support for Multiple Instance Learning with variable instances per bag\n- **BIC-based Model Selection**: Automatically determine optimal number of GMM components\n- **GPU Acceleration**: Built on TensorFlow 2.x for fast training on GPUs\n- **MiniBatchKMeans Initialization**: Smart initialization using scikit-learn's MiniBatchKMeans\n- **Memory-efficient Batch Processing**: Handle millions of samples with configurable batch sizes\n- **Save/Load Models**: Persist trained models for reuse\n- **Normalized Fisher Vectors**: Implements improved Fisher Vector normalization\n\n## Installation\n\nInstall from PyPI:\n```bash\npip install DeepFV\n```\n\nOr install from source:\n```bash\ngit clone https://github.com/sidhomj/DeepFV.git\ncd DeepFV\npip install -r requirements.txt\npip install -e .\n```\n\n## Quick Start\n\n### 1. Prepare your data\n```python\nimport numpy as np\n\n# Example: SIFT descriptors from images\nshape = [300, 20, 32] # (n_samples, n_descriptors_per_sample, feature_dim)\nsample_data = np.concatenate([\n np.random.normal(-np.ones(30), size=shape),\n np.random.normal(np.ones(30), size=shape)\n], axis=0)\n```\n\n### 2. Train with mini-batch gradient descent\n```python\nfrom DeepFV import FisherVectorDL\n\n# Create model with FULL covariance support\nfv_dl = FisherVectorDL(\n n_kernels=10,\n feature_dim=32,\n covariance_type='full' # or 'diag' for diagonal covariance\n)\n\n# Fit with mini-batch training\nfv_dl.fit_minibatch(\n sample_data,\n epochs=100,\n batch_size=1024*6,\n learning_rate=0.001,\n verbose=True\n)\n```\n\n### 3. BIC-based model selection\n```python\n# Automatically select optimal number of components\nfv_dl = FisherVectorDL(feature_dim=32, covariance_type='full')\nfv_dl.fit_by_bic(\n sample_data,\n choices_n_kernels=[2, 5, 10, 20],\n epochs=80,\n batch_size=1024,\n verbose=True\n)\n\nprint(f\"Selected {fv_dl.n_kernels} components\")\n```\n\n### 4. Compute Fisher Vectors\n\nFor data with multiple descriptors per sample (3D):\n```python\n# Compute normalized Fisher Vectors\nsample_data_test = sample_data[:20]\nfisher_vectors = fv_dl.predict_fisher_vector(sample_data_test, normalized=True)\n\n# Output shape: (n_samples, 2*n_kernels, feature_dim)\nprint(f\"Fisher vector shape: {fisher_vectors.shape}\")\n```\n\nFor simple 2D data (each sample is a single feature vector):\n```python\n# 2D input: (n_samples, feature_dim)\nsimple_data = np.random.randn(100, 32)\nfisher_vectors_2d = fv_dl.predict_fisher_vector(simple_data, normalized=True)\n\n# Output shape: (n_samples, 2*n_kernels, feature_dim)\nprint(f\"Fisher vector shape: {fisher_vectors_2d.shape}\")\n```\n\n### 5. Variable-length bags (Multiple Instance Learning) - OPTIMIZED!\n\nFor datasets where each bag contains a variable number of instances. **Uses vectorized computation for 10-100x speedup!**\n\n```python\n# Example: 3 images with different numbers of SIFT descriptors\nX = np.random.randn(245, 128) # 245 total descriptors, 128-dim features\n\n# bag_ids maps each instance to its bag\n# Image 0 has 50 descriptors, Image 1 has 120, Image 2 has 75\nbag_ids = np.array([0]*50 + [1]*120 + [2]*75)\n\n# Train on all instances (ignoring bag structure)\nfv_dl = FisherVectorDL(n_kernels=10, feature_dim=128)\nfv_dl.fit_minibatch(X, epochs=100, verbose=True)\n\n# Compute Fisher Vectors per bag (FAST - vectorized!)\nfisher_vectors, unique_bag_ids = fv_dl.predict_fisher_vector_bags(\n X,\n bag_ids,\n normalized=True,\n verbose=True\n)\n\nprint(f\"Fisher vectors shape: {fisher_vectors.shape}\") # (3, 20, 128)\nprint(f\"Bag IDs: {unique_bag_ids}\") # [0, 1, 2]\n```\n\n**Use cases for bag-level Fisher Vectors:**\n- **Image retrieval**: Variable number of SIFT/SURF descriptors per image\n- **Document classification**: Variable number of word embeddings per document\n- **Multiple Instance Learning (MIL)**: Variable instances per bag in medical imaging, etc.\n- **Time series**: Variable-length sequences aggregated into fixed representations\n\n**Get instance-level Fisher Vectors too:**\n```python\n# Optionally return both bag-level AND instance-level Fisher Vectors\nfisher_vectors, unique_bag_ids, instance_fvs = fv_dl.predict_fisher_vector_bags(\n X,\n bag_ids,\n return_instance_level=True, # Also return per-instance FVs\n verbose=True\n)\n\nprint(f\"Bag-level FVs: {fisher_vectors.shape}\") # (3, 20, 128) - 3 bags\nprint(f\"Instance-level FVs: {instance_fvs.shape}\") # (245, 20, 128) - 245 instances\n```\n\n**Performance:**\n- **1M instances, 10K bags**: ~0.5-2 seconds (vs ~60 seconds with old approach)\n- **Fully vectorized**: Single computation for all instances\n- **Scales to millions**: Can handle massive datasets efficiently\n\n### 6. Save and load models\n```python\n# Save trained model\nfv_dl.save_model('my_model.pkl')\n\n# Load model later\nfrom DeepFV import FisherVectorDL\nfv_dl_loaded = FisherVectorDL.load_model('my_model.pkl')\n```\n\n## Why FisherVectorDL?\n\n### Advantages over traditional GMM implementations:\n\n1. **Full Covariance Support**: Model rotated/tilted elliptical clusters, not just axis-aligned ones\n2. **Scalability**: Mini-batch training handles datasets too large to fit in memory\n3. **Speed**: GPU acceleration via TensorFlow for faster training\n4. **Flexibility**: Customizable learning rate, batch size, and number of epochs\n5. **Modern Stack**: Built on TensorFlow 2.x with eager execution\n6. **Smart Initialization**: Uses MiniBatchKMeans for better starting parameters\n\n## Testing\n\nRun the test script to see a 2D visualization:\n```bash\npython test_fishervector_dl.py\n```\n\nThis will:\n- Generate 3 elliptical Gaussian clusters\n- Train a GMM with full covariance\n- Use BIC to select optimal number of components\n- Compute and visualize Fisher Vectors\n- Save visualizations as PNG files\n\n### Example Results\n\n**GMM Clustering with BIC Selection:**\n\n\n\nThe plot shows how full covariance GMMs can model rotated elliptical clusters. The BIC criterion automatically selects the optimal number of components.\n\n**Fisher Vector Visualization:**\n\n\n\nLeft: Original 2D data colored by true cluster labels. Right: Fisher Vectors projected back to 2D using PCA, showing how the representation captures cluster structure.\n\n## Contributors\n\n* **John-William Sidhom** (https://github.com/sidhomj/) - Main contributor, TensorFlow implementation with full covariance support\n\n### Original Contributors:\n* Jonas Rothfuss (https://github.com/jonasrothfuss/) - Original implementation\n* Fabio Ferreira (https://github.com/ferreirafabio/) - Original implementation\n\n## References\n\n- [1] Perronnin, F., S\u00e1nchez, J., & Mensink, T. (2010). Improving the fisher kernel for large-scale image classification. In European conference on computer vision (pp. 143-156). Springer, Berlin, Heidelberg. https://www.robots.ox.ac.uk/~vgg/rg/papers/peronnin_etal_ECCV10.pdf\n- [2] Fisher Vector Fundamentals - VLFeat Documentation: http://www.vlfeat.org/api/fisher-fundamentals.html\n\n## License\n\nMIT License - see LICENSE file for details\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Fisher Vectors based on Gaussian Mixture Model with TensorFlow deep learning support",
"version": "0.2.10",
"project_urls": {
"Homepage": "https://github.com/sidhomj/DeepFV"
},
"split_keywords": [
"fisher-vectors",
" gaussian-mixture-model",
" deep-learning",
" tensorflow",
" computer-vision"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "7c544465dfe73726dd8d0cac62a6406ad5efd30d08f7ad20bb76ca9b138f2b42",
"md5": "368632bd044cce6425aa4cc89aa85429",
"sha256": "bad702515b4f5c0ad4dc5ea888cf7375913d4904fb97abe2ed17b0cf464a2db5"
},
"downloads": -1,
"filename": "deepfv-0.2.10-py3-none-any.whl",
"has_sig": false,
"md5_digest": "368632bd044cce6425aa4cc89aa85429",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 15644,
"upload_time": "2025-10-07T17:40:00",
"upload_time_iso_8601": "2025-10-07T17:40:00.478259Z",
"url": "https://files.pythonhosted.org/packages/7c/54/4465dfe73726dd8d0cac62a6406ad5efd30d08f7ad20bb76ca9b138f2b42/deepfv-0.2.10-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "6550c0d22fa01099f2fd4e5e4d8358e4e13725343abff3cae1de32974a24c952",
"md5": "b83afac8dee498db3ca1aa8614fe20da",
"sha256": "bb121216c788aa5ff9b5f8743b9903a9f89e81799f95c405375e678e039dbba6"
},
"downloads": -1,
"filename": "deepfv-0.2.10.tar.gz",
"has_sig": false,
"md5_digest": "b83afac8dee498db3ca1aa8614fe20da",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 15602,
"upload_time": "2025-10-07T17:40:01",
"upload_time_iso_8601": "2025-10-07T17:40:01.647224Z",
"url": "https://files.pythonhosted.org/packages/65/50/c0d22fa01099f2fd4e5e4d8358e4e13725343abff3cae1de32974a24c952/deepfv-0.2.10.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-10-07 17:40:01",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "sidhomj",
"github_project": "DeepFV",
"travis_ci": true,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "numpy",
"specs": [
[
"==",
"1.13.3"
]
]
},
{
"name": "scipy",
"specs": []
},
{
"name": "tensorflow",
"specs": [
[
">=",
"2.0.0"
]
]
},
{
"name": "scikit-learn",
"specs": [
[
">=",
"0.24.0"
]
]
},
{
"name": "matplotlib",
"specs": [
[
">=",
"3.0.0"
]
]
}
],
"lcname": "deepfv"
}
JSON
