| Name | timefiller JSON |
| Version |
1.0.6
JSON |
| download |
| home_page | None |
| Summary | A package for imputing missing data in time series and forecasting in missing-data contexts |
| upload_time | 2025-01-12 13:25:40 |
| maintainer | None |
| docs_url | None |
| author | Cyril Joly |
| requires_python | >=3.8 |
| license | None |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<div align="center">
[](https://pypi.org/project/timefiller/)
[](https://anaconda.org/conda-forge/timefiller)
[](https://timefiller.readthedocs.io/en/latest/?badge=latest)
[](https://github.com/CyrilJl/timefiller/actions/workflows/pytest.yml)
[](https://app.codacy.com/gh/CyrilJl/TimeFiller/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade)
<img src="https://raw.githubusercontent.com/CyrilJl/timefiller/main/_static/logo_timefiller.svg" alt="Logo BatchStats" width="200" height="200">
# timefiller
</div>
`timefiller` is a Python package designed for time series imputation and forecasting. When applied to a set of correlated time series, each series is processed individually, utilizing both its own auto-regressive patterns and correlations with the other series. The package is user-friendly, making it accessible even to non-experts.
Originally developed for imputation, it also proves useful for forecasting, particularly when covariates contain missing values.
## Why this package?
This package is an excellent choice for tasks involving missing data. It is lightweight and offers a straightforward, user-friendly API. Since its initial release, numerous enhancements have been made to accelerate data imputation, with further improvements planned for the future.
This package is ideal if:
- You have a collection of correlated time series with missing data and need to impute the missing values in one, several, or all of them
- You need to perform forecasting in scenarios with missing data, especially when dealing with unpredictable or irregular patterns, and require an algorithm capable of handling them dynamically
## Installation
You can get ``timefiller`` from PyPi:
```bash
pip install timefiller
```
But also from conda-forge:
```bash
conda install -c conda-forge timefiller
```
```bash
mamba install timefiller
```
## TimeSeriesImputer
The core of `timefiller` is the `TimeSeriesImputer` class is designed for the imputation of multivariate time series data. It extends the capabilities of `ImputeMultiVariate` by accounting for autoregressive and multivariate lags, as well as preprocessing.
### Key Features
- **Autoregressive Lags**: The imputer can handle autoregressive lags, which are specified using the `ar_lags` parameter. This allows the model to use past values of the series to predict missing values.
- **Multivariate Lags**: The imputer can also handle multivariate lags, specified using the `multivariate_lags` parameter. This allows the model to use lagged values of other series to predict missing values.
- **Preprocessing**: The imputer supports preprocessing steps, such as scaling or normalization, which can be specified using the `preprocessing` parameter. This ensures that the data is properly prepared before imputation.
- **Custom Estimators**: The imputer can use any scikit-learn compatible estimator for the imputation process. This allows for flexibility in choosing the best model for the data.
- **Handling Missing Values**: The imputer can handle missing values in both the target series and the covariates. It uses the `optimask` library to create NaN-free train and predict matrices for the estimator.
- **Uncertainty Estimation**: If the `alpha` parameter is specified, the imputer can provide uncertainty estimates for the imputed values, by wrapping the chosen estimator with [MAPIE](https://mapie.readthedocs.io/en/latest/).
### Basic Usage
The simplest usage example:
```python
from timefiller import TimeSeriesImputer
df = load_your_dataset()
tsi = TimeSeriesImputer()
df_imputed = tsi(X=df)
```
### Advanced Usage
```python
from sklearn.linear_model import Lasso
from timefiller import PositiveOutput, TimeSeriesImputer
df = load_your_dataset()
tsi = TimeSeriesImputer(estimator=Lasso(positive=True),
ar_lags=(1, 2, 3, 6, 24),
multivariate_lags=48,
negative_ar=True,
preprocessing=PositiveOutput())
df_imputed = tsi(X=df,
subset_cols=['col_1', 'col_17'],
after='2024-06-14',
n_nearest_features=75)
```
Check out the [documentation](https://timefiller.readthedocs.io/en/latest/index.html) for details on available options to customize your imputation.
### Real data example
Let's evaluate how ``timefiller`` performs on a real-world dataset, the [PeMS-Bay traffic data](https://zenodo.org/records/5724362). A sensor ID is selected for the experiment, and a contiguous block of missing values is introduced. To increase the complexity, additional Missing At Random (MAR) data is simulated, representing 1% of the entire dataset:
```python
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from timefiller import PositiveOutput, TimeSeriesImputer
from timefiller.utils import add_mar_nan, fetch_pems_bay
# Fetch the time series dataset (e.g., PeMS-Bay traffic data)
df = fetch_pems_bay()
df.shape, df.index.freq
>>> ((52128, 325), <5 * Minutes>)
```
```python
dfm = df.copy() # Create a copy to introduce missing values later
# Randomly select one column (sensor ID) to introduce missing values
k = np.random.randint(df.shape[1])
col = df.columns[k]
i, j = 20_000, 22_500 # Define a range in the dataset to set as NaN (missing values)
dfm.iloc[i:j, k] = np.nan # Introduce missing values in this range for the selected column
# Add more missing values randomly across the dataset (1% of the data)
dfm = add_mar_nan(dfm, ratio=0.01)
# Initialize the TimeSeriesImputer with AR lags and multivariate lags
tsi = TimeSeriesImputer(ar_lags=(1, 2, 3, 4, 5, 10, 15, 25, 50), preprocessing=PositiveOutput())
# Apply the imputation method on the modified dataframe
%time df_imputed = tsi(dfm, subset_cols=col)
>>> CPU times: total: 7.91 s
>>> Wall time: 2.93 s
```
```python
# Plot the imputed data alongside the data with missing values
df_imputed[col].rename('imputation').plot(figsize=(10, 3), lw=0.8, c='C0')
dfm[col].rename('data to impute').plot(ax=plt.gca(), lw=0.8, c='C1')
plt.title(f'sensor_id {col}')
plt.legend()
plt.show()
# Plot the imputed data vs the original complete data for comparison
df_imputed[col].rename('imputation').plot(figsize=(10, 3), lw=0.8, c='C0')
df[col].rename('complete data').plot(ax=plt.gca(), lw=0.8, c='C2')
plt.xlim(dfm.index[i], dfm.index[j]) # Focus on the region where data was missing
plt.legend()
plt.show()
```
<img src="https://raw.githubusercontent.com/CyrilJl/timefiller/main/_static/result_imputation.png" width="750" alt="result">
## Algorithmic Approach
`timefiller` relies heavily on [scikit-learn](https://scikit-learn.org/stable/) for the learning process and uses [optimask](https://optimask.readthedocs.io/en/latest/index.html) to create NaN-free train and
predict matrices for the estimator.
For each column requiring imputation, the algorithm differentiates between rows with valid data and those with missing values. For rows with missing data, it identifies the available sets of other columns (features).
For each set, OptiMask is called to train the chosen sklearn estimator on the largest possible submatrix without any NaNs. This process can become computationally expensive if the available sets of features vary
greatly or occur infrequently. In such cases, multiple calls to OptiMask and repeated fitting and predicting using the estimator may be necessary.
One important point to keep in mind is that within a single column, two different rows (timestamps) may be imputed using different estimators (regressors), each trained on distinct sets of columns (covariate features)
and samples (rows/timestamps).
Raw data
{
"_id": null,
"home_page": null,
"name": "timefiller",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": null,
"author": "Cyril Joly",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/1b/10/caa248e12dee1fc0793bc7bc247e2b4e5e3d869e4ebb2bd63e3c39da08e7/timefiller-1.0.6.tar.gz",
"platform": null,
"description": "<div align=\"center\">\n\n[](https://pypi.org/project/timefiller/)\n[](https://anaconda.org/conda-forge/timefiller)\n[](https://timefiller.readthedocs.io/en/latest/?badge=latest)\n[](https://github.com/CyrilJl/timefiller/actions/workflows/pytest.yml)\n[](https://app.codacy.com/gh/CyrilJl/TimeFiller/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade)\n\n\n<img src=\"https://raw.githubusercontent.com/CyrilJl/timefiller/main/_static/logo_timefiller.svg\" alt=\"Logo BatchStats\" width=\"200\" height=\"200\">\n\n# timefiller\n\n</div>\n\n`timefiller` is a Python package designed for time series imputation and forecasting. When applied to a set of correlated time series, each series is processed individually, utilizing both its own auto-regressive patterns and correlations with the other series. The package is user-friendly, making it accessible even to non-experts.\n\nOriginally developed for imputation, it also proves useful for forecasting, particularly when covariates contain missing values.\n\n## Why this package?\n\nThis package is an excellent choice for tasks involving missing data. It is lightweight and offers a straightforward, user-friendly API. Since its initial release, numerous enhancements have been made to accelerate data imputation, with further improvements planned for the future.\n\nThis package is ideal if:\n\n- You have a collection of correlated time series with missing data and need to impute the missing values in one, several, or all of them\n- You need to perform forecasting in scenarios with missing data, especially when dealing with unpredictable or irregular patterns, and require an algorithm capable of handling them dynamically\n\n## Installation\n\nYou can get ``timefiller`` from PyPi:\n\n```bash\npip install timefiller\n```\n\nBut also from conda-forge:\n\n```bash\nconda install -c conda-forge timefiller\n```\n\n```bash\nmamba install timefiller\n```\n\n## TimeSeriesImputer\n\nThe core of `timefiller` is the `TimeSeriesImputer` class is designed for the imputation of multivariate time series data. It extends the capabilities of `ImputeMultiVariate` by accounting for autoregressive and multivariate lags, as well as preprocessing.\n\n### Key Features\n\n- **Autoregressive Lags**: The imputer can handle autoregressive lags, which are specified using the `ar_lags` parameter. This allows the model to use past values of the series to predict missing values.\n- **Multivariate Lags**: The imputer can also handle multivariate lags, specified using the `multivariate_lags` parameter. This allows the model to use lagged values of other series to predict missing values.\n- **Preprocessing**: The imputer supports preprocessing steps, such as scaling or normalization, which can be specified using the `preprocessing` parameter. This ensures that the data is properly prepared before imputation.\n- **Custom Estimators**: The imputer can use any scikit-learn compatible estimator for the imputation process. This allows for flexibility in choosing the best model for the data.\n- **Handling Missing Values**: The imputer can handle missing values in both the target series and the covariates. It uses the `optimask` library to create NaN-free train and predict matrices for the estimator.\n- **Uncertainty Estimation**: If the `alpha` parameter is specified, the imputer can provide uncertainty estimates for the imputed values, by wrapping the chosen estimator with [MAPIE](https://mapie.readthedocs.io/en/latest/).\n\n### Basic Usage\n\nThe simplest usage example:\n\n```python\nfrom timefiller import TimeSeriesImputer\n\ndf = load_your_dataset()\ntsi = TimeSeriesImputer()\ndf_imputed = tsi(X=df)\n```\n\n### Advanced Usage\n\n```python\nfrom sklearn.linear_model import Lasso\nfrom timefiller import PositiveOutput, TimeSeriesImputer\n\ndf = load_your_dataset()\ntsi = TimeSeriesImputer(estimator=Lasso(positive=True),\n ar_lags=(1, 2, 3, 6, 24),\n multivariate_lags=48,\n negative_ar=True,\n preprocessing=PositiveOutput())\ndf_imputed = tsi(X=df,\n subset_cols=['col_1', 'col_17'],\n after='2024-06-14',\n n_nearest_features=75)\n```\n\nCheck out the [documentation](https://timefiller.readthedocs.io/en/latest/index.html) for details on available options to customize your imputation.\n\n### Real data example\n\nLet's evaluate how ``timefiller`` performs on a real-world dataset, the [PeMS-Bay traffic data](https://zenodo.org/records/5724362). A sensor ID is selected for the experiment, and a contiguous block of missing values is introduced. To increase the complexity, additional Missing At Random (MAR) data is simulated, representing 1% of the entire dataset:\n\n```python\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nfrom timefiller import PositiveOutput, TimeSeriesImputer\nfrom timefiller.utils import add_mar_nan, fetch_pems_bay\n\n# Fetch the time series dataset (e.g., PeMS-Bay traffic data)\ndf = fetch_pems_bay()\ndf.shape, df.index.freq\n>>> ((52128, 325), <5 * Minutes>)\n```\n\n```python\ndfm = df.copy() # Create a copy to introduce missing values later\n\n# Randomly select one column (sensor ID) to introduce missing values\nk = np.random.randint(df.shape[1])\ncol = df.columns[k]\ni, j = 20_000, 22_500 # Define a range in the dataset to set as NaN (missing values)\ndfm.iloc[i:j, k] = np.nan # Introduce missing values in this range for the selected column\n\n# Add more missing values randomly across the dataset (1% of the data)\ndfm = add_mar_nan(dfm, ratio=0.01)\n\n# Initialize the TimeSeriesImputer with AR lags and multivariate lags\ntsi = TimeSeriesImputer(ar_lags=(1, 2, 3, 4, 5, 10, 15, 25, 50), preprocessing=PositiveOutput())\n\n# Apply the imputation method on the modified dataframe\n%time df_imputed = tsi(dfm, subset_cols=col)\n>>> CPU times: total: 7.91 s\n>>> Wall time: 2.93 s\n```\n\n```python\n# Plot the imputed data alongside the data with missing values\ndf_imputed[col].rename('imputation').plot(figsize=(10, 3), lw=0.8, c='C0')\ndfm[col].rename('data to impute').plot(ax=plt.gca(), lw=0.8, c='C1')\nplt.title(f'sensor_id {col}')\nplt.legend()\nplt.show()\n\n# Plot the imputed data vs the original complete data for comparison\ndf_imputed[col].rename('imputation').plot(figsize=(10, 3), lw=0.8, c='C0')\ndf[col].rename('complete data').plot(ax=plt.gca(), lw=0.8, c='C2')\nplt.xlim(dfm.index[i], dfm.index[j]) # Focus on the region where data was missing\nplt.legend()\nplt.show()\n```\n\n<img src=\"https://raw.githubusercontent.com/CyrilJl/timefiller/main/_static/result_imputation.png\" width=\"750\" alt=\"result\">\n\n## Algorithmic Approach\n\n`timefiller` relies heavily on [scikit-learn](https://scikit-learn.org/stable/) for the learning process and uses [optimask](https://optimask.readthedocs.io/en/latest/index.html) to create NaN-free train and\npredict matrices for the estimator.\n\nFor each column requiring imputation, the algorithm differentiates between rows with valid data and those with missing values. For rows with missing data, it identifies the available sets of other columns (features).\nFor each set, OptiMask is called to train the chosen sklearn estimator on the largest possible submatrix without any NaNs. This process can become computationally expensive if the available sets of features vary\ngreatly or occur infrequently. In such cases, multiple calls to OptiMask and repeated fitting and predicting using the estimator may be necessary.\n\nOne important point to keep in mind is that within a single column, two different rows (timestamps) may be imputed using different estimators (regressors), each trained on distinct sets of columns (covariate features)\nand samples (rows/timestamps).\n",
"bugtrack_url": null,
"license": null,
"summary": "A package for imputing missing data in time series and forecasting in missing-data contexts",
"version": "1.0.6",
"project_urls": {
"Homepage": "https://github.com/CyrilJl/TimeFiller"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1b45ef412406dbc5978f6bc3197c51a9ec304089a83141a00026a1b7dabbb950",
"md5": "7dd3d2d2b8d88151d7922d0081d29218",
"sha256": "98c74630dc180c184d5feb8c03a376cf52af383b73311582f2bb36dc35f6ba07"
},
"downloads": -1,
"filename": "timefiller-1.0.6-py3-none-any.whl",
"has_sig": false,
"md5_digest": "7dd3d2d2b8d88151d7922d0081d29218",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 20616,
"upload_time": "2025-01-12T13:25:39",
"upload_time_iso_8601": "2025-01-12T13:25:39.633859Z",
"url": "https://files.pythonhosted.org/packages/1b/45/ef412406dbc5978f6bc3197c51a9ec304089a83141a00026a1b7dabbb950/timefiller-1.0.6-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1b10caa248e12dee1fc0793bc7bc247e2b4e5e3d869e4ebb2bd63e3c39da08e7",
"md5": "8e28ea5abdf9f229b8de8ca3f6b46f68",
"sha256": "23d74fb6a8927732a74b0122cc70f617ea4a3d5a98f56e10e82d83e92fb48520"
},
"downloads": -1,
"filename": "timefiller-1.0.6.tar.gz",
"has_sig": false,
"md5_digest": "8e28ea5abdf9f229b8de8ca3f6b46f68",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 23262,
"upload_time": "2025-01-12T13:25:40",
"upload_time_iso_8601": "2025-01-12T13:25:40.747050Z",
"url": "https://files.pythonhosted.org/packages/1b/10/caa248e12dee1fc0793bc7bc247e2b4e5e3d869e4ebb2bd63e3c39da08e7/timefiller-1.0.6.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-12 13:25:40",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "CyrilJl",
"github_project": "TimeFiller",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "timefiller"
}
