| Name | twat-mp JSON |
| Version |
1.8.1
JSON |
| download |
| home_page | None |
| Summary | Parallel processing utilities using Pathos mpprocessing library |
| upload_time | 2025-02-15 05:50:19 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.10 |
| license | None |
| keywords |
map
mpprocessing
parallel
pathos
pool
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# twat-mp
(work in progress)
Parallel processing utilities using the Pathos multiprocessing library. This package provides convenient context managers and decorators for parallel processing, with both process-based and thread-based pools.
## Features
* Context managers for both process and thread pools:
+ `ProcessPool`: For CPU-intensive parallel processing
+ `ThreadPool`: For I/O-bound parallel processing
* Decorators for common parallel mapping operations:
+ `amap`: Asynchronous parallel map with automatic result retrieval
+ `imap`: Lazy parallel map returning an iterator
+ `pmap`: Standard parallel map (eager evaluation)
* Automatic CPU core detection for optimal pool sizing
* Clean resource management with context managers
* Full type hints and modern Python features
* Flexible pool configuration with customizable worker count
## Installation
```bash
pip install twat-mp
```
## Usage
### Using Process and Thread Pools
The package provides dedicated context managers for both process and thread pools:
```python
from twat_mp import ProcessPool, ThreadPool
# For CPU-intensive operations
with ProcessPool() as pool:
results = pool.map(lambda x: x * x, range(10))
print(list(results)) # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
# For I/O-bound operations
with ThreadPool() as pool:
results = pool.map(lambda x: x * 2, range(10))
print(list(results)) # [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
# Custom number of workers
with ProcessPool(nodes=4) as pool:
results = pool.map(lambda x: x * x, range(10))
```
### Using Map Decorators
The package provides three decorators for different mapping strategies:
```python
from twat_mp import amap, imap, pmap
# Standard parallel map (eager evaluation)
@pmap
def square(x: int) -> int:
return x * x
results = list(square(range(10)))
print(results) # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
# Lazy parallel map (returns iterator)
@imap
def cube(x: int) -> int:
return x * x * x
for result in cube(range(5)):
print(result) # Prints results as they become available
# Asynchronous parallel map with automatic result retrieval
@amap
def double(x: int) -> int:
return x * 2
results = list(double(range(10)))
print(results) # [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
```
### Function Composition
Decorators can be composed for complex parallel operations:
```python
from twat_mp import amap
@amap
def compute_intensive(x: int) -> int:
result = x
for _ in range(1000): # Simulate CPU-intensive work
result = (result * x + x) % 10000
return result
@amap
def io_intensive(x: int) -> int:
import time
time.sleep(0.001) # Simulate I/O wait
return x * 2
# Chain parallel operations
results = list(io_intensive(compute_intensive(range(100))))
```
## Dependencies
* `pathos`: For parallel processing functionality
## Development
To set up the development environment:
```bash
# Install in development mode with test dependencies
uv pip install -e ".[test]"
# Run tests
python -m pytest tests/
# Run benchmarks
python -m pytest tests/test_benchmark.py
```
## License
MIT License
.
Raw data
{
"_id": null,
"home_page": null,
"name": "twat-mp",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "map, mpprocessing, parallel, pathos, pool",
"author": null,
"author_email": "Adam Twardoch <adam+github@twardoch.com>",
"download_url": "https://files.pythonhosted.org/packages/13/76/fbde9a373b61f783769bb33f36a180527fae01f37f9efbf99a55b6d37861/twat_mp-1.8.1.tar.gz",
"platform": null,
"description": "# twat-mp\n\n(work in progress)\n\nParallel processing utilities using the Pathos multiprocessing library. This package provides convenient context managers and decorators for parallel processing, with both process-based and thread-based pools.\n\n## Features\n\n* Context managers for both process and thread pools:\n + `ProcessPool`: For CPU-intensive parallel processing\n + `ThreadPool`: For I/O-bound parallel processing\n* Decorators for common parallel mapping operations:\n + `amap`: Asynchronous parallel map with automatic result retrieval\n + `imap`: Lazy parallel map returning an iterator\n + `pmap`: Standard parallel map (eager evaluation)\n* Automatic CPU core detection for optimal pool sizing\n* Clean resource management with context managers\n* Full type hints and modern Python features\n* Flexible pool configuration with customizable worker count\n\n## Installation\n\n```bash\npip install twat-mp\n```\n\n## Usage\n\n### Using Process and Thread Pools\n\nThe package provides dedicated context managers for both process and thread pools:\n\n```python\nfrom twat_mp import ProcessPool, ThreadPool\n\n# For CPU-intensive operations\nwith ProcessPool() as pool:\n results = pool.map(lambda x: x * x, range(10))\n print(list(results)) # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]\n\n# For I/O-bound operations\nwith ThreadPool() as pool:\n results = pool.map(lambda x: x * 2, range(10))\n print(list(results)) # [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]\n\n# Custom number of workers\nwith ProcessPool(nodes=4) as pool:\n results = pool.map(lambda x: x * x, range(10))\n```\n\n### Using Map Decorators\n\nThe package provides three decorators for different mapping strategies:\n\n```python\nfrom twat_mp import amap, imap, pmap\n\n# Standard parallel map (eager evaluation)\n@pmap\ndef square(x: int) -> int:\n return x * x\n\nresults = list(square(range(10)))\nprint(results) # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]\n\n# Lazy parallel map (returns iterator)\n@imap\ndef cube(x: int) -> int:\n return x * x * x\n\nfor result in cube(range(5)):\n print(result) # Prints results as they become available\n\n# Asynchronous parallel map with automatic result retrieval\n@amap\ndef double(x: int) -> int:\n return x * 2\n\nresults = list(double(range(10)))\nprint(results) # [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]\n```\n\n### Function Composition\n\nDecorators can be composed for complex parallel operations:\n\n```python\nfrom twat_mp import amap\n\n@amap\ndef compute_intensive(x: int) -> int:\n result = x\n for _ in range(1000): # Simulate CPU-intensive work\n result = (result * x + x) % 10000\n return result\n\n@amap\ndef io_intensive(x: int) -> int:\n import time\n time.sleep(0.001) # Simulate I/O wait\n return x * 2\n\n# Chain parallel operations\nresults = list(io_intensive(compute_intensive(range(100))))\n```\n\n## Dependencies\n\n* `pathos`: For parallel processing functionality\n\n## Development\n\nTo set up the development environment:\n\n```bash\n# Install in development mode with test dependencies\nuv pip install -e \".[test]\"\n\n# Run tests\npython -m pytest tests/\n\n# Run benchmarks\npython -m pytest tests/test_benchmark.py\n```\n\n## License\n\nMIT License\n.\n",
"bugtrack_url": null,
"license": null,
"summary": "Parallel processing utilities using Pathos mpprocessing library",
"version": "1.8.1",
"project_urls": {
"Documentation": "https://github.com/twardoch/twat-mp#readme",
"Issues": "https://github.com/twardoch/twat-mp/issues",
"Source": "https://github.com/twardoch/twat-mp"
},
"split_keywords": [
"map",
" mpprocessing",
" parallel",
" pathos",
" pool"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "a2dff496254a052be6c4f546f3b4c1ba7fc34476349b81d90139b70b079c7f16",
"md5": "949bcd0c86185088ad0e6e81dd336946",
"sha256": "89c0b9213bc99dc19c714559d73c1cb20f394801d88fa7c40f9aae8f8bde561d"
},
"downloads": -1,
"filename": "twat_mp-1.8.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "949bcd0c86185088ad0e6e81dd336946",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 6597,
"upload_time": "2025-02-15T05:50:16",
"upload_time_iso_8601": "2025-02-15T05:50:16.949438Z",
"url": "https://files.pythonhosted.org/packages/a2/df/f496254a052be6c4f546f3b4c1ba7fc34476349b81d90139b70b079c7f16/twat_mp-1.8.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "1376fbde9a373b61f783769bb33f36a180527fae01f37f9efbf99a55b6d37861",
"md5": "0cb0f7edb9908c0f3f23954e03f83e55",
"sha256": "5463da76ec4570426cd4364683c7f89cf728aa27e7157fe2a52174c167eaf8e2"
},
"downloads": -1,
"filename": "twat_mp-1.8.1.tar.gz",
"has_sig": false,
"md5_digest": "0cb0f7edb9908c0f3f23954e03f83e55",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 12576,
"upload_time": "2025-02-15T05:50:19",
"upload_time_iso_8601": "2025-02-15T05:50:19.262994Z",
"url": "https://files.pythonhosted.org/packages/13/76/fbde9a373b61f783769bb33f36a180527fae01f37f9efbf99a55b6d37861/twat_mp-1.8.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-02-15 05:50:19",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "twardoch",
"github_project": "twat-mp#readme",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "twat-mp"
}
