# Welford
Python (Pytorch) implementation calculating Standard Deviation, Variance,
Covariance Matrix, and Whitening Matrix online and in parallel. This makes it
more memory efficient than computing in the standard way.
This implementation uses Welford's algorithm for variance, and standard
deviation. Online Covariance calculation uses a generalization shown on Wikipedia.
The algorithm is described in the followings,
* [Wikipedia:Welford Online Algorithm](https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Online_algorithm)
* [Wikipedia:Welford Parallel Algorithm](https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Parallel_algorithm)
* [Wikipedia:Covariance Online Algorithm](https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Online)
Welford's method is more numerically stable than the standard method. The theoretical background of Welford's method is mentioned in detail on the following blog articles. Please refer them if you are interested in.
* http://www.johndcook.com/blog/standard_deviation
* https://jonisalonen.com/2013/deriving-welfords-method-for-computing-variance/
This library is a fork of the `welford` library implemented in Numpy ( https://github.com/a-mitani/welford ).
I later added the covariance calculation inspired by the implementation by Carsten Schelp (
https://carstenschelp.github.io/2019/05/12/Online_Covariance_Algorithm_002.html
). The `OnlineCovariance` class has feature parity with the normal `Welford` class,
but takes more memory and compute.
## Install
Download package via [PyPI repository](https://pypi.org/project/welford-torch/)
```
$ pip install welford-torch
```
## Example (OnlineCovariance)
Example showing how to use `OnlineCovariance` to compute the covariance and
other quantities. If only computing mean, standard deviation and variance, use
the `Welford` class as it is faster and uses less memory. Otherwise, both
classes have the same functionality
(i.e: `.add()`, `.add_all()` and `.merge()`. See below for examples).
```python
import torch
from welford_torch import OnlineCovariance
# Initialize Welford object, and add samples
w = OnlineCovariance()
dataset = torch.tensor([[-1, 90], [0, 100], [1, 110], [-1, 100], [1, 100]])
for datapoint in dataset:
w.add(datapoint)
# output
print(w.mean) # Mean --> [ 0., 100.]
print(w.var_s) # Sample variance --> [ 1., 50.]
print(w.var_p) # Population variance --> [ 0.8000, 40.0000]
print(w.cov) # Covariance matrix --> [[ 0.8000, 4.0000], [ 4.0000, 40.0000]]
print(w.corrcoef) # Pearson correlation coefficient --> [[1.0, 0.7071], [0.7071, 1.0000]]
print(w.eig_val) # Eigenvalues (ascending) --> [ 0.3960, 40.4040]
print(w.eig_vec) # Eigenvectors --> [[-0.9949, 0.1005], [ 0.1005, 0.9949]]
print(w.whit) # Whitening Matrix --> [[ 1.5746, -0.1431], [-0.1431, 0.1718]]
print(w.whit_inv) # Whitening Matrix Inverse --> [[0.6871, 0.5726], [0.5726, 6.2986]]
# Whitened dataset
print( (dataset.to(torch.float32) - w.mean) @ w.whit.T )
# --> [[-0.1431, -1.5746],
# [ 0.0000, 0.0000],
# [ 0.1431, 1.5746],
# [-1.5746, 0.1431],
# [ 1.5746, -0.1431]]
```
## Example (Welford)
### For Online Calculation
```python
import numpy as torch
from welford_torch import Welford
# Initialize Welford object
w = Welford()
# Input data samples sequentialy
w.add(torch.tensor([0, 100]))
w.add(torch.tensor([1, 110]))
w.add(torch.tensor([2, 120]))
# output
print(w.mean) # mean --> [ 1. 110.]
print(w.var_s) # sample variance --> [1, 100]
print(w.var_p) # population variance --> [ 0.6666 66.66]
# You can add other samples after calculating variances.
w.add(torch.tensor([3, 130]))
w.add(torch.tensor([4, 140]))
# output with added samples
print(w.mean) # mean --> [ 2. 120.]
print(w.var_s) # sample variance --> [ 2.5 250. ]
print(w.var_p) # population variance --> [ 2. 200.]
```
Welford object supports initialization with data samples and batch addition of samples.
```python
# Initialize Welford object with samples
ini = torch.tensor([[0, 100],
[1, 110],
[2, 120]])
w = Welford(ini)
# output
print(w.mean) # mean --> [ 1. 110.]
print(w.var_s) # sample variance --> [1, 100]
print(w.var_p) # population variance --> [ 0.66666667 66.66666667]
# add other samples through batch method
other_samples = torch.tensor([[3, 130],
[4, 140]])
w.add_all(other_samples)
# output with added samples
print(w.mean) # mean --> [ 2. 120.]
print(w.var_s) # sample variance --> [ 2.5 250. ]
print(w.var_p) # population variance --> [ 2. 200.]
```
### For Parallel Calculation
Welford also offers parallel calculation method for variance.
```python
import numpy as torch
from welford_torch import Welford
# Initialize two Welford objects
w_1 = Welford()
w_2 = Welford()
# Each object will calculate variance of each samples in parallel.
# On w_1
w_1.add(torch.tensor([0, 100]))
w_1.add(torch.tensor([1, 110]))
w_1.add(torch.tensor([2, 120]))
print(w_1.var_s) # sample variance -->[ 1. 100.]
print(w_1.var_p) # population variance -->[ 0.66666667 66.66666667]
# On w_2
w_2.add(torch.tensor([3, 130]))
w_2.add(torch.tensor([4, 140]))
print(w_2.var_s) # sample variance -->[ 0.5 50. ]
print(w_2.var_p) # sample variance -->[ 0.25 25. ]
# You can Merge objects to get variance of WHOLE samples
w_1.merge(w_2)
print(w.var_s) # sample variance --> [ 2.5 250. ]
print(w_1.var_p) # sample variance -->[ 2. 200.]
```
Raw data
{
"_id": null,
"home_page": "https://github.com/pesvut/welford-torch",
"name": "welford-torch",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "statistics,online,welford,torch,covariance,correlation",
"author": "Nicky Pochinkov",
"author_email": "work@nicky.pro",
"download_url": "https://files.pythonhosted.org/packages/da/de/f11d8ed8edbb8684a56739ef65d5023aba01c2b0b9c40ed13d8990e07cd1/welford-torch-0.2.3.tar.gz",
"platform": null,
"description": "# Welford\nPython (Pytorch) implementation calculating Standard Deviation, Variance,\nCovariance Matrix, and Whitening Matrix online and in parallel. This makes it\nmore memory efficient than computing in the standard way.\n\nThis implementation uses Welford's algorithm for variance, and standard\ndeviation. Online Covariance calculation uses a generalization shown on Wikipedia.\n\nThe algorithm is described in the followings,\n\n* [Wikipedia:Welford Online Algorithm](https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Online_algorithm)\n* [Wikipedia:Welford Parallel Algorithm](https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Parallel_algorithm)\n* [Wikipedia:Covariance Online Algorithm](https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Online)\n\nWelford's method is more numerically stable than the standard method. The theoretical background of Welford's method is mentioned in detail on the following blog articles. Please refer them if you are interested in.\n\n* http://www.johndcook.com/blog/standard_deviation\n* https://jonisalonen.com/2013/deriving-welfords-method-for-computing-variance/\n\nThis library is a fork of the `welford` library implemented in Numpy ( https://github.com/a-mitani/welford ).\n\nI later added the covariance calculation inspired by the implementation by Carsten Schelp (\nhttps://carstenschelp.github.io/2019/05/12/Online_Covariance_Algorithm_002.html\n). The `OnlineCovariance` class has feature parity with the normal `Welford` class,\nbut takes more memory and compute.\n\n## Install\nDownload package via [PyPI repository](https://pypi.org/project/welford-torch/)\n```\n$ pip install welford-torch\n```\n\n## Example (OnlineCovariance)\n\nExample showing how to use `OnlineCovariance` to compute the covariance and\nother quantities. If only computing mean, standard deviation and variance, use\nthe `Welford` class as it is faster and uses less memory. Otherwise, both\nclasses have the same functionality\n(i.e: `.add()`, `.add_all()` and `.merge()`. See below for examples).\n\n```python\nimport torch\nfrom welford_torch import OnlineCovariance\n\n# Initialize Welford object, and add samples\nw = OnlineCovariance()\n\ndataset = torch.tensor([[-1, 90], [0, 100], [1, 110], [-1, 100], [1, 100]])\nfor datapoint in dataset:\n w.add(datapoint)\n\n# output\nprint(w.mean) # Mean --> [ 0., 100.]\nprint(w.var_s) # Sample variance --> [ 1., 50.]\nprint(w.var_p) # Population variance --> [ 0.8000, 40.0000]\nprint(w.cov) # Covariance matrix --> [[ 0.8000, 4.0000], [ 4.0000, 40.0000]]\nprint(w.corrcoef) # Pearson correlation coefficient --> [[1.0, 0.7071], [0.7071, 1.0000]]\nprint(w.eig_val) # Eigenvalues (ascending) --> [ 0.3960, 40.4040]\nprint(w.eig_vec) # Eigenvectors --> [[-0.9949, 0.1005], [ 0.1005, 0.9949]]\nprint(w.whit) # Whitening Matrix --> [[ 1.5746, -0.1431], [-0.1431, 0.1718]]\nprint(w.whit_inv) # Whitening Matrix Inverse --> [[0.6871, 0.5726], [0.5726, 6.2986]]\n\n# Whitened dataset\nprint( (dataset.to(torch.float32) - w.mean) @ w.whit.T )\n# --> [[-0.1431, -1.5746],\n# [ 0.0000, 0.0000],\n# [ 0.1431, 1.5746],\n# [-1.5746, 0.1431],\n# [ 1.5746, -0.1431]]\n```\n\n## Example (Welford)\n### For Online Calculation\n```python\nimport numpy as torch\nfrom welford_torch import Welford\n\n# Initialize Welford object\nw = Welford()\n\n# Input data samples sequentialy\nw.add(torch.tensor([0, 100]))\nw.add(torch.tensor([1, 110]))\nw.add(torch.tensor([2, 120]))\n\n# output\nprint(w.mean) # mean --> [ 1. 110.]\nprint(w.var_s) # sample variance --> [1, 100]\nprint(w.var_p) # population variance --> [ 0.6666 66.66]\n\n# You can add other samples after calculating variances.\nw.add(torch.tensor([3, 130]))\nw.add(torch.tensor([4, 140]))\n\n# output with added samples\nprint(w.mean) # mean --> [ 2. 120.]\nprint(w.var_s) # sample variance --> [ 2.5 250. ]\nprint(w.var_p) # population variance --> [ 2. 200.]\n```\n\nWelford object supports initialization with data samples and batch addition of samples.\n```python\n# Initialize Welford object with samples\nini = torch.tensor([[0, 100],\n [1, 110],\n [2, 120]])\nw = Welford(ini)\n\n# output\nprint(w.mean) # mean --> [ 1. 110.]\nprint(w.var_s) # sample variance --> [1, 100]\nprint(w.var_p) # population variance --> [ 0.66666667 66.66666667]\n\n# add other samples through batch method\nother_samples = torch.tensor([[3, 130],\n [4, 140]])\nw.add_all(other_samples)\n\n# output with added samples\nprint(w.mean) # mean --> [ 2. 120.]\nprint(w.var_s) # sample variance --> [ 2.5 250. ]\nprint(w.var_p) # population variance --> [ 2. 200.]\n```\n\n### For Parallel Calculation\nWelford also offers parallel calculation method for variance.\n```python\nimport numpy as torch\nfrom welford_torch import Welford\n\n# Initialize two Welford objects\nw_1 = Welford()\nw_2 = Welford()\n\n# Each object will calculate variance of each samples in parallel.\n# On w_1\nw_1.add(torch.tensor([0, 100]))\nw_1.add(torch.tensor([1, 110]))\nw_1.add(torch.tensor([2, 120]))\nprint(w_1.var_s) # sample variance -->[ 1. 100.]\nprint(w_1.var_p) # population variance -->[ 0.66666667 66.66666667]\n\n# On w_2\nw_2.add(torch.tensor([3, 130]))\nw_2.add(torch.tensor([4, 140]))\nprint(w_2.var_s) # sample variance -->[ 0.5 50. ]\nprint(w_2.var_p) # sample variance -->[ 0.25 25. ]\n\n# You can Merge objects to get variance of WHOLE samples\nw_1.merge(w_2)\nprint(w.var_s) # sample variance --> [ 2.5 250. ]\nprint(w_1.var_p) # sample variance -->[ 2. 200.]\n\n```\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Online Pytorch implementation to get Standard Deviation, Covariance, Correlation and Whitening.",
"version": "0.2.3",
"project_urls": {
"Homepage": "https://github.com/pesvut/welford-torch"
},
"split_keywords": [
"statistics",
"online",
"welford",
"torch",
"covariance",
"correlation"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "94cbb2e3b838af95501c2e418d47eda531d33c3d61da370c9ef1cd3f03b497a2",
"md5": "db2d1a2792842f2e850ad7498f1031d8",
"sha256": "2e5b2fe5e38bedecd833479fcad5c31914b786a50fbdb964fa90215726ffb54b"
},
"downloads": -1,
"filename": "welford_torch-0.2.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "db2d1a2792842f2e850ad7498f1031d8",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 12335,
"upload_time": "2024-02-27T15:26:59",
"upload_time_iso_8601": "2024-02-27T15:26:59.771047Z",
"url": "https://files.pythonhosted.org/packages/94/cb/b2e3b838af95501c2e418d47eda531d33c3d61da370c9ef1cd3f03b497a2/welford_torch-0.2.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "dadef11d8ed8edbb8684a56739ef65d5023aba01c2b0b9c40ed13d8990e07cd1",
"md5": "18b516faef4e9b0b09d3eac1852a4499",
"sha256": "191636f70f482f4d4478c3027a33dddef52da3cae23360803c05d80799f4cd11"
},
"downloads": -1,
"filename": "welford-torch-0.2.3.tar.gz",
"has_sig": false,
"md5_digest": "18b516faef4e9b0b09d3eac1852a4499",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 11661,
"upload_time": "2024-02-27T15:27:01",
"upload_time_iso_8601": "2024-02-27T15:27:01.450761Z",
"url": "https://files.pythonhosted.org/packages/da/de/f11d8ed8edbb8684a56739ef65d5023aba01c2b0b9c40ed13d8990e07cd1/welford-torch-0.2.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-27 15:27:01",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "pesvut",
"github_project": "welford-torch",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "welford-torch"
}
JSON
