# mpmsub - Memory-Aware Multiprocessing Subprocess
A Python library for running subprocess commands with intelligent memory-aware scheduling and resource management.
## Features
- Memory-aware scheduling with automatic resource management
- Pipeline support for chaining commands
- Multiple job interfaces (dictionary, object-oriented, convenience functions)
- Output redirection and progress tracking
- Job profiling for memory optimization
- Flexible API with multiple parameter names
## Installation
```bash
pip install mpmsub
```
## Quick Start
```python
import mpmsub
# Create cluster and add jobs
p = mpmsub.cluster(cpu=4, memory="8G")
# Dictionary interface
p.jobs.append({"cmd": ["echo", "hello"], "p": 1, "m": "1G"})
# Object interface
p.jobs.append(mpmsub.Job(["python", "script.py"]).cpu(2).memory("2G"))
# Pipeline interface
p.jobs.append(mpmsub.pipeline([
["cat", "data.txt"],
["grep", "pattern"]
], cpu=1, memory="500M"))
# Run and analyze
results = p.run()
print(f"Completed: {results['jobs']['completed']}/{results['jobs']['total']}")
```
## Performance
Run benchmarks to see performance benefits:
```bash
python examples/benchmark_demo.py
python benchmarks/run_all_benchmarks.py
```
Benefits include 1.2-2x speedup in memory-constrained scenarios and better system stability. See [`benchmarks/README.md`](benchmarks/README.md) for details.
## Documentation
**Complete documentation:** [https://nextgenusfs.github.io/mpmsub/](https://nextgenusfs.github.io/mpmsub/)
Includes tutorials, API reference, examples, and performance tips.
## Key Features
### Multiple Job Interfaces
```python
# Dictionary interface
p.jobs.append({"cmd": ["echo", "hello"], "p": 1, "m": "1G"})
# Object interface with builder pattern
job = mpmsub.Job(["python", "script.py"]) \
.cpu(2).memory("4G") \
.stdout_to("output.txt")
# Pipeline interface
pipeline = mpmsub.pipeline([
["cat", "data.txt"],
["grep", "pattern"],
["sort"]
], cpu=1, memory="500M")
```
### Job Profiling
```python
# Measure actual memory usage
profile_results = p.profile()
# Use recommendations for optimized scheduling
p.jobs.append({"cmd": ["my_command"], "p": 1, "m": "150M"})
```
### Memory Formats
- `"1G"` - Gigabytes
- `"1024M"` - Megabytes
- `1024` - MB (integer)
## Examples
See the `examples/` directory for usage demonstrations and the documentation for comprehensive tutorials.
## Development
```bash
# Clone and install in development mode
git clone https://github.com/nextgenusfs/mpmsub.git
cd mpmsub
pip install -e .[dev]
# Install pre-commit hooks (optional)
pre-commit install
```
### CI/CD
The project uses GitHub Actions for:
- **Tests**: Run on Python 3.8-3.12, Ubuntu/macOS
- **Code Quality**: Linting, formatting, type checking
- **Publishing**: Manual workflow for TestPyPI/PyPI, automatic PyPI on releases
- **Documentation**: Auto-deploy to GitHub Pages
## Requirements
- Python 3.8+
- psutil >= 5.8.0
## License
MIT License
Raw data
{
"_id": null,
"home_page": null,
"name": "mpmsub",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "multiprocessing, memory, subprocess, parallel, scheduling",
"author": null,
"author_email": "Jon Palmer <nextgenusfs@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/cf/0f/48ea1d3c220c50543bcfbc63d6d55c1eab8efb410d99702530eebe95930e/mpmsub-2025.10.16.tar.gz",
"platform": null,
"description": "# mpmsub - Memory-Aware Multiprocessing Subprocess\n\nA Python library for running subprocess commands with intelligent memory-aware scheduling and resource management.\n\n## Features\n\n- Memory-aware scheduling with automatic resource management\n- Pipeline support for chaining commands\n- Multiple job interfaces (dictionary, object-oriented, convenience functions)\n- Output redirection and progress tracking\n- Job profiling for memory optimization\n- Flexible API with multiple parameter names\n\n## Installation\n\n```bash\npip install mpmsub\n```\n\n## Quick Start\n\n```python\nimport mpmsub\n\n# Create cluster and add jobs\np = mpmsub.cluster(cpu=4, memory=\"8G\")\n\n# Dictionary interface\np.jobs.append({\"cmd\": [\"echo\", \"hello\"], \"p\": 1, \"m\": \"1G\"})\n\n# Object interface\np.jobs.append(mpmsub.Job([\"python\", \"script.py\"]).cpu(2).memory(\"2G\"))\n\n# Pipeline interface\np.jobs.append(mpmsub.pipeline([\n [\"cat\", \"data.txt\"],\n [\"grep\", \"pattern\"]\n], cpu=1, memory=\"500M\"))\n\n# Run and analyze\nresults = p.run()\nprint(f\"Completed: {results['jobs']['completed']}/{results['jobs']['total']}\")\n```\n\n## Performance\n\nRun benchmarks to see performance benefits:\n\n```bash\npython examples/benchmark_demo.py\npython benchmarks/run_all_benchmarks.py\n```\n\nBenefits include 1.2-2x speedup in memory-constrained scenarios and better system stability. See [`benchmarks/README.md`](benchmarks/README.md) for details.\n\n## Documentation\n\n**Complete documentation:** [https://nextgenusfs.github.io/mpmsub/](https://nextgenusfs.github.io/mpmsub/)\n\nIncludes tutorials, API reference, examples, and performance tips.\n\n## Key Features\n\n### Multiple Job Interfaces\n\n```python\n# Dictionary interface\np.jobs.append({\"cmd\": [\"echo\", \"hello\"], \"p\": 1, \"m\": \"1G\"})\n\n# Object interface with builder pattern\njob = mpmsub.Job([\"python\", \"script.py\"]) \\\n .cpu(2).memory(\"4G\") \\\n .stdout_to(\"output.txt\")\n\n# Pipeline interface\npipeline = mpmsub.pipeline([\n [\"cat\", \"data.txt\"],\n [\"grep\", \"pattern\"],\n [\"sort\"]\n], cpu=1, memory=\"500M\")\n```\n\n### Job Profiling\n\n```python\n# Measure actual memory usage\nprofile_results = p.profile()\n\n# Use recommendations for optimized scheduling\np.jobs.append({\"cmd\": [\"my_command\"], \"p\": 1, \"m\": \"150M\"})\n```\n\n### Memory Formats\n\n- `\"1G\"` - Gigabytes\n- `\"1024M\"` - Megabytes\n- `1024` - MB (integer)\n\n## Examples\n\nSee the `examples/` directory for usage demonstrations and the documentation for comprehensive tutorials.\n\n## Development\n\n```bash\n# Clone and install in development mode\ngit clone https://github.com/nextgenusfs/mpmsub.git\ncd mpmsub\npip install -e .[dev]\n\n# Install pre-commit hooks (optional)\npre-commit install\n```\n\n### CI/CD\n\nThe project uses GitHub Actions for:\n- **Tests**: Run on Python 3.8-3.12, Ubuntu/macOS\n- **Code Quality**: Linting, formatting, type checking\n- **Publishing**: Manual workflow for TestPyPI/PyPI, automatic PyPI on releases\n- **Documentation**: Auto-deploy to GitHub Pages\n\n## Requirements\n\n- Python 3.8+\n- psutil >= 5.8.0\n\n## License\n\nMIT License\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Memory-aware multiprocessing subprocess execution library",
"version": "2025.10.16",
"project_urls": {
"Homepage": "https://github.com/nextgenusfs/mpmsub",
"Issues": "https://github.com/nextgenusfs/mpmsub/issues",
"Repository": "https://github.com/nextgenusfs/mpmsub"
},
"split_keywords": [
"multiprocessing",
" memory",
" subprocess",
" parallel",
" scheduling"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "d953cc1d935f6e8ae87488ba78c72d0ee0de237897ba0fdd4700d907eb41e7dc",
"md5": "909265de490c7255248d87c41e552371",
"sha256": "835cc362b03cfb5798f44e5d13f7b1abd195a1963c46d3509c54ff517c804f37"
},
"downloads": -1,
"filename": "mpmsub-2025.10.16-py3-none-any.whl",
"has_sig": false,
"md5_digest": "909265de490c7255248d87c41e552371",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 17550,
"upload_time": "2025-10-16T22:34:43",
"upload_time_iso_8601": "2025-10-16T22:34:43.245024Z",
"url": "https://files.pythonhosted.org/packages/d9/53/cc1d935f6e8ae87488ba78c72d0ee0de237897ba0fdd4700d907eb41e7dc/mpmsub-2025.10.16-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "cf0f48ea1d3c220c50543bcfbc63d6d55c1eab8efb410d99702530eebe95930e",
"md5": "390258067ef35e1f64cf71c9e1692244",
"sha256": "cb72dc2b96ec16cce1ab38b68db13ed7cd15f68e9ac6ab840c109e4d93129a30"
},
"downloads": -1,
"filename": "mpmsub-2025.10.16.tar.gz",
"has_sig": false,
"md5_digest": "390258067ef35e1f64cf71c9e1692244",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 20001,
"upload_time": "2025-10-16T22:34:44",
"upload_time_iso_8601": "2025-10-16T22:34:44.633052Z",
"url": "https://files.pythonhosted.org/packages/cf/0f/48ea1d3c220c50543bcfbc63d6d55c1eab8efb410d99702530eebe95930e/mpmsub-2025.10.16.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-10-16 22:34:44",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "nextgenusfs",
"github_project": "mpmsub",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "mpmsub"
}
JSON
