# Popyrous
(Pouya's Python Routines) A collection of useful and frequently encountered Python routines for (data) science, research, development, and everyday life.
Author: Pouya P. Niaz (<pniaz20@ku.edu.tr> , <pouya.p.niaz@gmail.com>)
Version: 0.0.12
Last Update: February 15, 2024
This is a collection of Python routines for the following purposes:
- Checking for packages and installing missing ones iwithin scripts without the need for Jupyter and symbols like "!" and "%".
- Reading and writing `.mat` files coming to/from MathWorks MATLAB software.
- Building and manipulating time series data using sliding windows, low-pass filtering, etc.
- Building flexible and easy-to-use datasets for data analysis or machine learning out of structured time series experiments (multiple subjects, conditions, repetitions, etc.).
- Downloading data/files from the internet and Google Drive, using simple functions.
- Compressing or extracting Zip files with LZMA, etc., using simple functions.
Install with:
```bash
pip install popyrous
```
-------------------------------------------------------------
## 1- Intro
This package is a collection of routines I have widely used in my scientific, academic and engineering life.
It holds functionality for data and file manipulation, some tools for manipulating time series data,
some tools for extracting machine-learning-ready time series datasets from tabular timeseries data of structured experiments,
i.e., experiments performed with multiple subjects, under multiple conditions, with many repetitions, and so forth.
The contents and applications of this package are described briefly below. However, extensive documentation is provided in the docstrings of
all functions and classes in the code, which is where you should look for further information.
-------------------------------------------------------------
## 2- Contents and Submodules
### 2-1- matlab
This submodule contains functions for reading and writing data to and from `.mat` files.
- `type_compatible(typ)`: Determining whether or not a Python data type is compatible for writing into `.mat` files.
- `save_workspace(filename, masterdict)`: Save dictionary holding variables and data into `.mat` file.
- `load_workspace(filename, dictname)`: Load contents of `.mat` file into an (existing or new) dictionary.
### 2-2- packages
This submodule contains functions for checking which packages are installed in the environment without having to be in a notebook and running commands with `!` or `%`.
Also, you can check for a list of required packages (with or without required versions) and install missing packages, or wrong-versioned packages at the same time.
- `get_package_list()`: Get list (dictionary with keys being packages and values being versions) of packages in the (conda) environment.
- `check_packages(pkglst, install_missing, **kwargs)`: Get a list of required packages and see if they are all installed, installing the missing ones in the process.
Example:
```python
from popyrous.packages import check_packages
check_packages(["numpy","scipy","pandas==1.5.2"], install_missing=True, reinstall_wrong_versions=True)
```
### 2-3- timeseries
This submodule contains some classes and functions for working easily and efficiently with time series data.
You can filter data, pass it through sliding window and extract data for machine/deep learning, etc.
Also, given the dataframe of a tructured time series experiment where multiple subjects repeated an experiment
multiple times under various conditions, you can get their data, preprocess, post-process, filter, extract sliding window, etc.
and then keep some subjects, conditions, or trials for training and the rest for testing (for data analysis or machine learning), and so forth.
#### 2-3-1- sliding_window
The `sliding_window` function gets tabular timeseries data, extracts sliding windows from it, then downsamples or inverts them, etc. then returns them.
Sliding windows of time series data is used for time series modeling, prediction, classification, regression and forecasting problems.
#### 2-3-2- datasets
- `TabularDataset`: A class for reading time series data from an array, downsampling, preprocessing, and extracting sliding windows from it.
- `make_squeezed_dataset(hparams, inputs, outputs, **kwargs)`: Gets inputs/outputs, returns squeezed (2D) sliding window dataset ready to be fed to, e.g., an ANN model.
- `make_unsqueezed_dataset(hparams, inputs, outputs, **kwargs)`: Gets inputs/outputs, returns unsqueezed (3D) sliding window dataset ready to be fed to, e.g., an LSTM model.
#### 2-3-3- experiment
- `TimeseriesExperiment`: A class that gets a single dataframe containing the time series data of a series of structured experiments where
there are multiple subjects, repetitions and trials. The data can then be processed such that data of each trial is separated and processed individually,
some subjects, conditions or trials are kept for training/testing, there is preprocessing before extracting sliding windows, and postprocessing after it,
and so on. This class comes in handy when the data of such a structured series of experiments needs to be processed and fed to a machine learning model, for instance.
- `generate_cell_array`: A function, which is a more concise version of the above class, doing everything in one shot and returning everything together.
#### 2-3-4- filt
Some functions for low-pass filtering time series data.
- `butter_lowpass_filter_forward` filters input data with a digital Butterworth low-pass filter gvien sampling and cutoff frequncies, and filter order.
This filter is causal, and only goes forward in time. It does not see its future. It is used for real-time implementations.
Because this filter is causal, it induces a phase shift, so the filtered signal will have a delay relative to the real signal.
The lower the cutoff frequency, the longer the delay. This function in turn uses the `sosfiltfilt` utility of SciPy.
- `butter_lowpass_filter_back_to_back` filters input data similarly, but uses `sosfiltfilt` to go back to back, so it looks both to past and future.
It can only smooth the data offline, since it has access to the future as well. Unlike the previous causal filter, it has no phase shift.
- `butter_highpass_filter_forward` and `butter_highpass_filter_back_to_back` can be used for similar purposes.
- The new `butter_filter` function encompasses virtually any kind of digital Butterworth filter, including all of the above.
#### 2-3-5- metrics
Some metrics used for time-series classification, etc.
- `tsc_metrics`: Time-series classification metrics, including accuracy, f1 score, concurrency (transitioning on time) and consistency (not changing prediction in consistent non-transitioning portions of the data)
#### 2-3-6- cwt
Continmuous Wavelet Transform
- `cwt_for_batch`: gets a numpy array of shape, e.g., (batchsize, channels, seqlen) [could be any shape, as long as time is the last dimension]
and returns an array of its CWT coefficients.
Additionally, it can downsample it and remove the last row and column. Returns a (batchsize, channels, coefs, seqlen) dataset of 2D images.
- `cwt_for_tensor`: gets a data tensor of any shape and simply performs CWT on it. Takes the last dimension as time, and adds a dimension
to the beginning, containing coefficients.
### 2-4- web
This submodule contains some web-related functions for downloading files from the internet or Google Drive, storing them, reading their contents, etc.
- `download_google_drive_file(shareable_link, output_file)`: Gets shareable link of a Google Drive file, and downlaods it.
- `download(url, filename, **kwargs)`: Downlaods material from the internet, and reads its content or stores in a file.
### 2-5- zipfiles
This submodule contains some functions for compressing/decompressing zip files.
- `extract_files(fileName)`: Extracts everything in the zip file.
- `compress_files(file_name, **kwargs)`: Compresses files into a zip file. Options for compression method, etc. are provided.
### 2-6- ml
This submodule contains some machine-learning-related code. For now, it just contains a function for pretty plotting confusion matrices (see credits).
- `make_confusion_matrix` gets a confusion matrix and some parameters, and pretty plots it.
-------------------------------------------------------------
## 3- License
This package is built with MIT license.
-------------------------------------------------------------
## 4- Credits
Pretty plotting confusion matrix:
Dennis T
<https://github.com/DTrimarchi10/confusion_matrix>
<https://medium.com/@dtuk81/confusion-matrix-visualization-fc31e3f30fea>
Raw data
{
"_id": null,
"home_page": "https://github.com/pniaz20/popyrous",
"name": "popyrous",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7, <4",
"maintainer_email": "",
"keywords": "python,routines,matlab,zipfile,packages,time series,filtering,download,web",
"author": "Pouya P. Niaz",
"author_email": "<pniaz20@ku.edu.tr>",
"download_url": "https://files.pythonhosted.org/packages/05/b4/9e7a87397c37808fb36143c27250bbaca24a9ea95e67dc228c8801a5ef64/popyrous-0.0.12.tar.gz",
"platform": null,
"description": "# Popyrous\r\n\r\n(Pouya's Python Routines) A collection of useful and frequently encountered Python routines for (data) science, research, development, and everyday life.\r\n\r\nAuthor: Pouya P. Niaz (<pniaz20@ku.edu.tr> , <pouya.p.niaz@gmail.com>) \r\nVersion: 0.0.12 \r\nLast Update: February 15, 2024\r\n\r\nThis is a collection of Python routines for the following purposes:\r\n\r\n- Checking for packages and installing missing ones iwithin scripts without the need for Jupyter and symbols like \"!\" and \"%\".\r\n- Reading and writing `.mat` files coming to/from MathWorks MATLAB software.\r\n- Building and manipulating time series data using sliding windows, low-pass filtering, etc.\r\n- Building flexible and easy-to-use datasets for data analysis or machine learning out of structured time series experiments (multiple subjects, conditions, repetitions, etc.).\r\n- Downloading data/files from the internet and Google Drive, using simple functions.\r\n- Compressing or extracting Zip files with LZMA, etc., using simple functions.\r\n\r\nInstall with:\r\n\r\n```bash\r\npip install popyrous\r\n```\r\n\r\n-------------------------------------------------------------\r\n\r\n## 1- Intro\r\n\r\nThis package is a collection of routines I have widely used in my scientific, academic and engineering life.\r\nIt holds functionality for data and file manipulation, some tools for manipulating time series data,\r\nsome tools for extracting machine-learning-ready time series datasets from tabular timeseries data of structured experiments,\r\ni.e., experiments performed with multiple subjects, under multiple conditions, with many repetitions, and so forth.\r\n\r\nThe contents and applications of this package are described briefly below. However, extensive documentation is provided in the docstrings of\r\nall functions and classes in the code, which is where you should look for further information.\r\n\r\n-------------------------------------------------------------\r\n\r\n## 2- Contents and Submodules\r\n\r\n### 2-1- matlab\r\n\r\nThis submodule contains functions for reading and writing data to and from `.mat` files.\r\n\r\n- `type_compatible(typ)`: Determining whether or not a Python data type is compatible for writing into `.mat` files.\r\n- `save_workspace(filename, masterdict)`: Save dictionary holding variables and data into `.mat` file.\r\n- `load_workspace(filename, dictname)`: Load contents of `.mat` file into an (existing or new) dictionary.\r\n\r\n### 2-2- packages\r\n\r\nThis submodule contains functions for checking which packages are installed in the environment without having to be in a notebook and running commands with `!` or `%`.\r\nAlso, you can check for a list of required packages (with or without required versions) and install missing packages, or wrong-versioned packages at the same time.\r\n\r\n- `get_package_list()`: Get list (dictionary with keys being packages and values being versions) of packages in the (conda) environment.\r\n- `check_packages(pkglst, install_missing, **kwargs)`: Get a list of required packages and see if they are all installed, installing the missing ones in the process.\r\n\r\nExample:\r\n\r\n```python\r\nfrom popyrous.packages import check_packages\r\ncheck_packages([\"numpy\",\"scipy\",\"pandas==1.5.2\"], install_missing=True, reinstall_wrong_versions=True)\r\n```\r\n\r\n### 2-3- timeseries\r\n\r\nThis submodule contains some classes and functions for working easily and efficiently with time series data.\r\nYou can filter data, pass it through sliding window and extract data for machine/deep learning, etc.\r\nAlso, given the dataframe of a tructured time series experiment where multiple subjects repeated an experiment\r\nmultiple times under various conditions, you can get their data, preprocess, post-process, filter, extract sliding window, etc.\r\nand then keep some subjects, conditions, or trials for training and the rest for testing (for data analysis or machine learning), and so forth.\r\n\r\n#### 2-3-1- sliding_window\r\n\r\nThe `sliding_window` function gets tabular timeseries data, extracts sliding windows from it, then downsamples or inverts them, etc. then returns them.\r\nSliding windows of time series data is used for time series modeling, prediction, classification, regression and forecasting problems.\r\n\r\n#### 2-3-2- datasets\r\n\r\n- `TabularDataset`: A class for reading time series data from an array, downsampling, preprocessing, and extracting sliding windows from it.\r\n- `make_squeezed_dataset(hparams, inputs, outputs, **kwargs)`: Gets inputs/outputs, returns squeezed (2D) sliding window dataset ready to be fed to, e.g., an ANN model.\r\n- `make_unsqueezed_dataset(hparams, inputs, outputs, **kwargs)`: Gets inputs/outputs, returns unsqueezed (3D) sliding window dataset ready to be fed to, e.g., an LSTM model.\r\n\r\n#### 2-3-3- experiment\r\n\r\n- `TimeseriesExperiment`: A class that gets a single dataframe containing the time series data of a series of structured experiments where\r\n there are multiple subjects, repetitions and trials. The data can then be processed such that data of each trial is separated and processed individually,\r\n some subjects, conditions or trials are kept for training/testing, there is preprocessing before extracting sliding windows, and postprocessing after it,\r\n and so on. This class comes in handy when the data of such a structured series of experiments needs to be processed and fed to a machine learning model, for instance.\r\n- `generate_cell_array`: A function, which is a more concise version of the above class, doing everything in one shot and returning everything together.\r\n\r\n#### 2-3-4- filt\r\n\r\nSome functions for low-pass filtering time series data.\r\n\r\n- `butter_lowpass_filter_forward` filters input data with a digital Butterworth low-pass filter gvien sampling and cutoff frequncies, and filter order.\r\n This filter is causal, and only goes forward in time. It does not see its future. It is used for real-time implementations.\r\n Because this filter is causal, it induces a phase shift, so the filtered signal will have a delay relative to the real signal.\r\n The lower the cutoff frequency, the longer the delay. This function in turn uses the `sosfiltfilt` utility of SciPy.\r\n- `butter_lowpass_filter_back_to_back` filters input data similarly, but uses `sosfiltfilt` to go back to back, so it looks both to past and future.\r\n It can only smooth the data offline, since it has access to the future as well. Unlike the previous causal filter, it has no phase shift.\r\n- `butter_highpass_filter_forward` and `butter_highpass_filter_back_to_back` can be used for similar purposes.\r\n- The new `butter_filter` function encompasses virtually any kind of digital Butterworth filter, including all of the above.\r\n\r\n#### 2-3-5- metrics\r\n\r\nSome metrics used for time-series classification, etc.\r\n\r\n- `tsc_metrics`: Time-series classification metrics, including accuracy, f1 score, concurrency (transitioning on time) and consistency (not changing prediction in consistent non-transitioning portions of the data)\r\n\r\n#### 2-3-6- cwt\r\n\r\nContinmuous Wavelet Transform\r\n\r\n- `cwt_for_batch`: gets a numpy array of shape, e.g., (batchsize, channels, seqlen) [could be any shape, as long as time is the last dimension]\r\n and returns an array of its CWT coefficients.\r\n Additionally, it can downsample it and remove the last row and column. Returns a (batchsize, channels, coefs, seqlen) dataset of 2D images.\r\n- `cwt_for_tensor`: gets a data tensor of any shape and simply performs CWT on it. Takes the last dimension as time, and adds a dimension\r\n to the beginning, containing coefficients.\r\n\r\n### 2-4- web\r\n\r\nThis submodule contains some web-related functions for downloading files from the internet or Google Drive, storing them, reading their contents, etc.\r\n\r\n- `download_google_drive_file(shareable_link, output_file)`: Gets shareable link of a Google Drive file, and downlaods it.\r\n- `download(url, filename, **kwargs)`: Downlaods material from the internet, and reads its content or stores in a file.\r\n\r\n### 2-5- zipfiles\r\n\r\nThis submodule contains some functions for compressing/decompressing zip files.\r\n\r\n- `extract_files(fileName)`: Extracts everything in the zip file.\r\n- `compress_files(file_name, **kwargs)`: Compresses files into a zip file. Options for compression method, etc. are provided.\r\n\r\n### 2-6- ml\r\n\r\nThis submodule contains some machine-learning-related code. For now, it just contains a function for pretty plotting confusion matrices (see credits).\r\n\r\n- `make_confusion_matrix` gets a confusion matrix and some parameters, and pretty plots it.\r\n\r\n-------------------------------------------------------------\r\n\r\n## 3- License\r\n\r\nThis package is built with MIT license.\r\n\r\n-------------------------------------------------------------\r\n\r\n## 4- Credits\r\n\r\nPretty plotting confusion matrix: \r\nDennis T \r\n<https://github.com/DTrimarchi10/confusion_matrix> \r\n<https://medium.com/@dtuk81/confusion-matrix-visualization-fc31e3f30fea>\r\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Pouya's Python routines. A collection of useful Python routines for everyday and professional life.",
"version": "0.0.12",
"project_urls": {
"Homepage": "https://github.com/pniaz20/popyrous"
},
"split_keywords": [
"python",
"routines",
"matlab",
"zipfile",
"packages",
"time series",
"filtering",
"download",
"web"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "72565f18871f0fa0cc5f6db146a3447e07ce0400921bf4e66522b0e6f73cd82f",
"md5": "a1c10e0d697c9337be0aa522895a547b",
"sha256": "23db3618f5eea5ce0b24660e4f7deb5e14e24c73503b25cc927fdb06a3422483"
},
"downloads": -1,
"filename": "popyrous-0.0.12-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a1c10e0d697c9337be0aa522895a547b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7, <4",
"size": 46171,
"upload_time": "2024-02-15T10:09:16",
"upload_time_iso_8601": "2024-02-15T10:09:16.580959Z",
"url": "https://files.pythonhosted.org/packages/72/56/5f18871f0fa0cc5f6db146a3447e07ce0400921bf4e66522b0e6f73cd82f/popyrous-0.0.12-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "05b49e7a87397c37808fb36143c27250bbaca24a9ea95e67dc228c8801a5ef64",
"md5": "98743eba41087e38d7a8594693640763",
"sha256": "d565373db7f5ffbe18a2b8cea71bb1ba6db5c4a6319e7e40168f48a71d331d01"
},
"downloads": -1,
"filename": "popyrous-0.0.12.tar.gz",
"has_sig": false,
"md5_digest": "98743eba41087e38d7a8594693640763",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7, <4",
"size": 44683,
"upload_time": "2024-02-15T10:09:18",
"upload_time_iso_8601": "2024-02-15T10:09:18.819178Z",
"url": "https://files.pythonhosted.org/packages/05/b4/9e7a87397c37808fb36143c27250bbaca24a9ea95e67dc228c8801a5ef64/popyrous-0.0.12.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-15 10:09:18",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "pniaz20",
"github_project": "popyrous",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "popyrous"
}
JSON
