# shared_allocator
High-performance shared memory allocation for Python multiprocessing.
Provides multiple allocator strategies optimized for different use cases:
- **BumpAllocator**: Ultra-fast (2-3x faster), reset-based allocation
- **FreeListAllocator**: General-purpose with individual deallocation support
- **SlabAllocator**: Fixed-size slot allocation with bitmap tracking
## Features
- 🚀 **High Performance**: 2-7 million operations per second
- 🔒 **Process-Safe**: Safe concurrent access from multiple processes
- 💾 **Zero-Copy**: Uses POSIX shared memory for efficient IPC
- 🎯 **Multiple Strategies**: Choose the right allocator for your use case
## Installation
```bash
uv pip install -e .
```
## Quick Start
### FreeListAllocator (Recommended for general use)
```python
from shared_allocator import FreeListAllocator
# Create allocator
allocator = FreeListAllocator(name="my_allocator", capacity=10*1024*1024, create=True)
# Allocate and write
offset = allocator.allocate(256)
allocator.write(offset, b"Hello from shared memory!")
# Read back
data = allocator.read(offset, 25)
print(data) # b"Hello from shared memory!"
# Free when done
allocator.free(offset)
# Cleanup
allocator.close()
allocator.unlink()
```
### BumpAllocator (Fastest for temporary allocations)
```python
from shared_allocator import BumpAllocator
# Create allocator
allocator = BumpAllocator(name="fast_allocator", capacity=10*1024*1024, create=True)
# Allocate lots of temporary data
for i in range(1000):
offset = allocator.allocate(128)
allocator.write(offset, f"Record {i}".encode())
# Reset all at once (much faster than individual frees)
allocator.reset()
# Memory is available again
print(f"Available: {allocator.available_bytes()} bytes")
allocator.close()
allocator.unlink()
```
## Performance Benchmarks
Based on pytest-benchmark results:
| Operation | BumpAllocator | FreeListAllocator | Winner |
|-----------|---------------|-------------------|--------|
| Write | 7,122 Kops/s | 4,216 Kops/s | Bump (1.7x) |
| Read | 5,958 Kops/s | 5,016 Kops/s | Bump (1.2x) |
| Sequential Alloc | 2,927 Kops/s | 1,040 Kops/s | Bump (2.8x) |
| Alloc+Free | N/A | 1,227 Kops/s | FreeList |
| Reuse Pattern | N/A | 596 Kops/s | FreeList |
**Recommendation**: Use `BumpAllocator` for 2-3x better performance when you can bulk-reset. Use `FreeListAllocator` when you need individual deallocation.
## Use Cases
### BumpAllocator
- Request-scoped allocations (allocate during request, reset after)
- Temporary buffers in data processing pipelines
- Phase-based computations
- Scenarios where all allocations have similar lifetimes
### FreeListAllocator
- Long-lived objects with mixed allocation/deallocation
- Dynamic data structures
- When you need fine-grained memory management
- General-purpose shared memory allocation
## Development
```bash
# Install dependencies
uv sync
# Run tests
pytest
# Run benchmarks
pytest tests/test_benchmarks.py --benchmark-only
# Run tests with coverage
pytest --cov=shared_allocator
# Run examples
python examples/basic_usage.py
```
## Architecture
All allocators use POSIX shared memory with the following structure:
```
[METADATA] [ALLOCATOR-SPECIFIC DATA] [DATA POOL]
```
- **BumpAllocator**: Simple offset counter, O(1) allocation
- **FreeListAllocator**: Stack of free slot indices, O(1) alloc/free
- **SlabAllocator**: Bitmap for slot tracking, O(n) allocation
Learned from [`shared_hashmap`](https://github.com/user/shared_hashmap):
- 8-byte alignment for atomic operations
- Lazy atomic view loading for performance
- Clean separation of metadata and data regions
## License
MIT
Raw data
{
"_id": null,
"home_page": null,
"name": "shared_allocator",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.13.7",
"maintainer_email": null,
"keywords": "shared-memory, allocator, allocation, multiprocessing, ipc",
"author": null,
"author_email": "Raymond Chastain <RaymondLC92@protonmail.com>",
"download_url": "https://files.pythonhosted.org/packages/bf/af/23388b127aac50a26ffa556e86e06ca7d06e8f547dc53bc16c74a8fd9a8e/shared_allocator-0.1.0.tar.gz",
"platform": null,
"description": "# shared_allocator\n\nHigh-performance shared memory allocation for Python multiprocessing.\n\nProvides multiple allocator strategies optimized for different use cases:\n\n- **BumpAllocator**: Ultra-fast (2-3x faster), reset-based allocation\n- **FreeListAllocator**: General-purpose with individual deallocation support\n- **SlabAllocator**: Fixed-size slot allocation with bitmap tracking\n\n## Features\n\n- \ud83d\ude80 **High Performance**: 2-7 million operations per second\n- \ud83d\udd12 **Process-Safe**: Safe concurrent access from multiple processes\n- \ud83d\udcbe **Zero-Copy**: Uses POSIX shared memory for efficient IPC\n- \ud83c\udfaf **Multiple Strategies**: Choose the right allocator for your use case\n\n## Installation\n\n```bash\nuv pip install -e .\n```\n\n## Quick Start\n\n### FreeListAllocator (Recommended for general use)\n\n```python\nfrom shared_allocator import FreeListAllocator\n\n# Create allocator\nallocator = FreeListAllocator(name=\"my_allocator\", capacity=10*1024*1024, create=True)\n\n# Allocate and write\noffset = allocator.allocate(256)\nallocator.write(offset, b\"Hello from shared memory!\")\n\n# Read back\ndata = allocator.read(offset, 25)\nprint(data) # b\"Hello from shared memory!\"\n\n# Free when done\nallocator.free(offset)\n\n# Cleanup\nallocator.close()\nallocator.unlink()\n```\n\n### BumpAllocator (Fastest for temporary allocations)\n\n```python\nfrom shared_allocator import BumpAllocator\n\n# Create allocator\nallocator = BumpAllocator(name=\"fast_allocator\", capacity=10*1024*1024, create=True)\n\n# Allocate lots of temporary data\nfor i in range(1000):\n offset = allocator.allocate(128)\n allocator.write(offset, f\"Record {i}\".encode())\n\n# Reset all at once (much faster than individual frees)\nallocator.reset()\n\n# Memory is available again\nprint(f\"Available: {allocator.available_bytes()} bytes\")\n\nallocator.close()\nallocator.unlink()\n```\n\n## Performance Benchmarks\n\nBased on pytest-benchmark results:\n\n| Operation | BumpAllocator | FreeListAllocator | Winner |\n|-----------|---------------|-------------------|--------|\n| Write | 7,122 Kops/s | 4,216 Kops/s | Bump (1.7x) |\n| Read | 5,958 Kops/s | 5,016 Kops/s | Bump (1.2x) |\n| Sequential Alloc | 2,927 Kops/s | 1,040 Kops/s | Bump (2.8x) |\n| Alloc+Free | N/A | 1,227 Kops/s | FreeList |\n| Reuse Pattern | N/A | 596 Kops/s | FreeList |\n\n**Recommendation**: Use `BumpAllocator` for 2-3x better performance when you can bulk-reset. Use `FreeListAllocator` when you need individual deallocation.\n\n## Use Cases\n\n### BumpAllocator\n- Request-scoped allocations (allocate during request, reset after)\n- Temporary buffers in data processing pipelines\n- Phase-based computations\n- Scenarios where all allocations have similar lifetimes\n\n### FreeListAllocator\n- Long-lived objects with mixed allocation/deallocation\n- Dynamic data structures\n- When you need fine-grained memory management\n- General-purpose shared memory allocation\n\n## Development\n\n```bash\n# Install dependencies\nuv sync\n\n# Run tests\npytest\n\n# Run benchmarks\npytest tests/test_benchmarks.py --benchmark-only\n\n# Run tests with coverage\npytest --cov=shared_allocator\n\n# Run examples\npython examples/basic_usage.py\n```\n\n## Architecture\n\nAll allocators use POSIX shared memory with the following structure:\n\n```\n[METADATA] [ALLOCATOR-SPECIFIC DATA] [DATA POOL]\n```\n\n- **BumpAllocator**: Simple offset counter, O(1) allocation\n- **FreeListAllocator**: Stack of free slot indices, O(1) alloc/free\n- **SlabAllocator**: Bitmap for slot tracking, O(n) allocation\n\nLearned from [`shared_hashmap`](https://github.com/user/shared_hashmap):\n- 8-byte alignment for atomic operations\n- Lazy atomic view loading for performance\n- Clean separation of metadata and data regions\n\n## License\n\nMIT\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "High-performance shared memory allocation for Python",
"version": "0.1.0",
"project_urls": null,
"split_keywords": [
"shared-memory",
" allocator",
" allocation",
" multiprocessing",
" ipc"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "36251899aa6fc994c85d5255b7986c4d00ce8125491026d0c6d1f1dc799b4ff1",
"md5": "c39ca90a95ce058ddeb0251b0b80aae4",
"sha256": "4407821fc038155aec1edaa082d7fb78aefd2c67e698917df6b4ea77625540fb"
},
"downloads": -1,
"filename": "shared_allocator-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c39ca90a95ce058ddeb0251b0b80aae4",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.13.7",
"size": 11017,
"upload_time": "2025-10-23T23:17:56",
"upload_time_iso_8601": "2025-10-23T23:17:56.208514Z",
"url": "https://files.pythonhosted.org/packages/36/25/1899aa6fc994c85d5255b7986c4d00ce8125491026d0c6d1f1dc799b4ff1/shared_allocator-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "bfaf23388b127aac50a26ffa556e86e06ca7d06e8f547dc53bc16c74a8fd9a8e",
"md5": "69cad7fff0c987d82c064f0691c9859b",
"sha256": "72c9eadf2596827c33397ae71a2aa794a0bc6574605e098f6022ea074073c9bf"
},
"downloads": -1,
"filename": "shared_allocator-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "69cad7fff0c987d82c064f0691c9859b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.13.7",
"size": 11060,
"upload_time": "2025-10-23T23:17:57",
"upload_time_iso_8601": "2025-10-23T23:17:57.507534Z",
"url": "https://files.pythonhosted.org/packages/bf/af/23388b127aac50a26ffa556e86e06ca7d06e8f547dc53bc16c74a8fd9a8e/shared_allocator-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-10-23 23:17:57",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "shared_allocator"
}
JSON