quick_parallel
==============
This library is used to run custom functions using threads or process pool executor.
Requirements
************
- `tqdm` (this will be installed automatically if missing)
- Python 3.4 or higher
Installation
************
Install using pip:
.. code:: shell
pip install quick_parallel
Main Methods
************
`quick_parallel`: A function that splits multiple processes by either running them in parallel processes or using threads.
Usage
*****
.. code:: python
from quick_parallel.process import quick_parallel
def worker(x):
"""
Function to compute x^3 * 2.
"""
return x ** 3 * 2
def worker_with_arguments(x, p):
"""
Function that raises a number `x` to the power of `p`.
:param x: Base number
:param p: Power to which `x` is raised
:return: Result of x raised to the power of p
"""
return x ** p
if __name__ == "__main__":
# Example: Simple worker without arguments
gen_d = range(5) # Replace with your actual generator or iterable
# Running in a parallel process with 2 workers and disabling threading
lm = quick_parallel(worker, gen_d, use_thread=False, n_workers=2,
progress_message="Running in simple worker:")
# Collect and print the results
res = [i for i in lm]
print(res)
# Output: [0, 2, 16, 54, 128]
# Example: Worker with an extra positional argument
arg = 9 # Example positional argument (power)
ext_arg = quick_parallel(worker_with_arguments, gen_d, arg, use_thread=False, n_workers=2,
progress_message="Running worker_with_arguments function:")
print(list(ext_arg))
# Output: [0, 1, 512, 19683, 262144]
# Change the argument and check the updated results
arg = 10 # Example positional argument (power)
ext_arg = quick_parallel(worker_with_arguments, gen_d, arg, use_thread=False, n_workers=2,
progress_message="Running worker_with_arguments function:")
print(list(ext_arg))
# Output: [0, 1, 1024, 59049, 1048576]
Notes
*****
- When using threads (`use_threads=True`), the function utilizes `ThreadPoolExecutor` for parallel processing.
- If `n_workers` is not specified, the function defaults to using 40% of available CPU cores.
- Progress information is displayed during execution.
- Always run the code under `if __name__ == '__main__':` to prevent multiprocessing issues.
- Define worker functions in a separate script and import them into the processing script for better modularity.
- Pass a generator as the iterator (instead of a list, tuple, or numpy array) to save memory.
- If the function returns large datasets (e.g., DataFrames), store the data in a file (e.g., SQL database) to save memory and avoid performance slowdowns.
Raw data
{
"_id": null,
"home_page": "https://github.com/MAGALA-RICHARD/quick_parallel",
"name": "quick-parallel",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.4",
"maintainer_email": null,
"keywords": "python, quick parallel, worker, cores, threading, parallel processing",
"author": "Richard Magala",
"author_email": "magalarich20@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/11/2f/52c2dbdace5ffa6e8d74b9a73560f8ed63429701b50128d28387e669c77f/quick_parallel-0.0.1.3.tar.gz",
"platform": null,
"description": "quick_parallel\r\n==============\r\n\r\nThis library is used to run custom functions using threads or process pool executor.\r\n\r\nRequirements\r\n************\r\n\r\n- `tqdm` (this will be installed automatically if missing)\r\n- Python 3.4 or higher\r\n\r\nInstallation\r\n************\r\n\r\nInstall using pip:\r\n\r\n.. code:: shell\r\n\r\n pip install quick_parallel\r\n\r\nMain Methods\r\n************\r\n\r\n`quick_parallel`: A function that splits multiple processes by either running them in parallel processes or using threads.\r\n\r\nUsage\r\n*****\r\n\r\n.. code:: python\r\n\r\n from quick_parallel.process import quick_parallel\r\n\r\n def worker(x):\r\n \"\"\"\r\n Function to compute x^3 * 2.\r\n \"\"\"\r\n return x ** 3 * 2\r\n\r\n def worker_with_arguments(x, p):\r\n \"\"\"\r\n Function that raises a number `x` to the power of `p`.\r\n\r\n :param x: Base number\r\n :param p: Power to which `x` is raised\r\n :return: Result of x raised to the power of p\r\n \"\"\"\r\n return x ** p\r\n\r\n if __name__ == \"__main__\":\r\n # Example: Simple worker without arguments\r\n gen_d = range(5) # Replace with your actual generator or iterable\r\n\r\n # Running in a parallel process with 2 workers and disabling threading\r\n lm = quick_parallel(worker, gen_d, use_thread=False, n_workers=2,\r\n progress_message=\"Running in simple worker:\")\r\n\r\n # Collect and print the results\r\n res = [i for i in lm]\r\n print(res)\r\n # Output: [0, 2, 16, 54, 128]\r\n\r\n # Example: Worker with an extra positional argument\r\n arg = 9 # Example positional argument (power)\r\n ext_arg = quick_parallel(worker_with_arguments, gen_d, arg, use_thread=False, n_workers=2,\r\n progress_message=\"Running worker_with_arguments function:\")\r\n print(list(ext_arg))\r\n # Output: [0, 1, 512, 19683, 262144]\r\n\r\n # Change the argument and check the updated results\r\n arg = 10 # Example positional argument (power)\r\n ext_arg = quick_parallel(worker_with_arguments, gen_d, arg, use_thread=False, n_workers=2,\r\n progress_message=\"Running worker_with_arguments function:\")\r\n print(list(ext_arg))\r\n # Output: [0, 1, 1024, 59049, 1048576]\r\n\r\nNotes\r\n*****\r\n\r\n- When using threads (`use_threads=True`), the function utilizes `ThreadPoolExecutor` for parallel processing.\r\n- If `n_workers` is not specified, the function defaults to using 40% of available CPU cores.\r\n- Progress information is displayed during execution.\r\n- Always run the code under `if __name__ == '__main__':` to prevent multiprocessing issues.\r\n- Define worker functions in a separate script and import them into the processing script for better modularity.\r\n- Pass a generator as the iterator (instead of a list, tuple, or numpy array) to save memory.\r\n- If the function returns large datasets (e.g., DataFrames), store the data in a file (e.g., SQL database) to save memory and avoid performance slowdowns.\r\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Quick parallel processing library",
"version": "0.0.1.3",
"project_urls": {
"Homepage": "https://github.com/MAGALA-RICHARD/quick_parallel"
},
"split_keywords": [
"python",
" quick parallel",
" worker",
" cores",
" threading",
" parallel processing"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4f7315915dc4c5e4a4edc7f6cb6ea2e7682e96eef5878ff85ce1dc1aa6afe7de",
"md5": "ea3931e2221e052cc13b6499a7d35bdc",
"sha256": "abb369d1acdfd4c8510f8e37aa1324a8ffccdde04802907f2d1be5af1c91a7cd"
},
"downloads": -1,
"filename": "quick_parallel-0.0.1.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "ea3931e2221e052cc13b6499a7d35bdc",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.4",
"size": 6804,
"upload_time": "2024-11-19T16:43:52",
"upload_time_iso_8601": "2024-11-19T16:43:52.153552Z",
"url": "https://files.pythonhosted.org/packages/4f/73/15915dc4c5e4a4edc7f6cb6ea2e7682e96eef5878ff85ce1dc1aa6afe7de/quick_parallel-0.0.1.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "112f52c2dbdace5ffa6e8d74b9a73560f8ed63429701b50128d28387e669c77f",
"md5": "d716a357c9f4fd15590f04b0f8f793e9",
"sha256": "7f33708953b262a79ea0d7dab5c1aeec0d423ba27e808bd8e0f8a371c5da56bd"
},
"downloads": -1,
"filename": "quick_parallel-0.0.1.3.tar.gz",
"has_sig": false,
"md5_digest": "d716a357c9f4fd15590f04b0f8f793e9",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.4",
"size": 5911,
"upload_time": "2024-11-19T16:43:53",
"upload_time_iso_8601": "2024-11-19T16:43:53.013773Z",
"url": "https://files.pythonhosted.org/packages/11/2f/52c2dbdace5ffa6e8d74b9a73560f8ed63429701b50128d28387e669c77f/quick_parallel-0.0.1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-19 16:43:53",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "MAGALA-RICHARD",
"github_project": "quick_parallel",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "quick-parallel"
}
JSON
