# ZeroBuffer Python Implementation
Native Python implementation of the ZeroBuffer protocol for high-performance zero-copy inter-process communication.
## Features
- **True Zero-Copy**: Uses memoryview and buffer protocol to avoid data copies
- **Cross-Platform**: Supports Linux, Windows, and macOS
- **Protocol Compatible**: Binary compatible with C++ and C# implementations
- **Pythonic API**: Clean, idiomatic Python interface
- **Type Safe**: Full type hints for better IDE support
- **Thread Safe**: Built-in synchronization for multi-threaded applications
## Requirements
- Python 3.8 or later
- `posix-ipc` (Linux/macOS only): `pip install posix-ipc`
- `pywin32` (Windows only): `pip install pywin32`
## Installation
```bash
pip install -e .
```
## Quick Start
### Reader Example
```python
from zerobuffer import Reader, BufferConfig
# Create a buffer
config = BufferConfig(metadata_size=1024, payload_size=1024*1024)
with Reader("my-buffer", config) as reader:
# Wait for writer to connect
print("Waiting for writer...")
# Read frames
while True:
frame = reader.read_frame(timeout=5.0)
if frame:
# Access data without copying
data = frame.data # This is a memoryview
print(f"Received frame {frame.sequence}: {len(data)} bytes")
# Process the frame...
# Must release frame to free buffer space
reader.release_frame(frame)
```
### Writer Example
```python
from zerobuffer import Writer
# Connect to existing buffer
with Writer("my-buffer") as writer:
# Write some metadata
metadata = b"{'format': 'raw', 'version': 1}"
writer.set_metadata(metadata)
# Write frames
for i in range(100):
data = f"Frame {i}".encode()
writer.write_frame(data)
print(f"Sent frame {i}")
```
### Zero-Copy Advanced Usage
```python
# True zero-copy writing with memoryview
import numpy as np
# Create numpy array
arr = np.arange(1000, dtype=np.float32)
# Get memoryview of array (no copy)
view = memoryview(arr)
# Write with zero-copy
writer.write_frame_zero_copy(view)
# Or use direct buffer access
buffer = writer.get_frame_buffer(size=4000)
# Write directly into shared memory
buffer[:] = arr.tobytes()
writer.commit_frame()
```
## API Reference
### Reader Class
```python
Reader(name: str, config: Optional[BufferConfig] = None)
```
Creates a new buffer and prepares for reading.
**Methods:**
- `get_metadata() -> Optional[memoryview]`: Get metadata as zero-copy memoryview
- `read_frame(timeout: Optional[float] = 5.0) -> Optional[Frame]`: Read next frame
- `release_frame(frame: Frame) -> None`: Release frame and free buffer space
- `is_writer_connected() -> bool`: Check if writer is connected
- `close() -> None`: Close reader and clean up resources
### Writer Class
```python
Writer(name: str)
```
Connects to an existing buffer for writing.
**Methods:**
- `set_metadata(data: Union[bytes, bytearray, memoryview]) -> None`: Write metadata (once only)
- `write_frame(data: Union[bytes, bytearray, memoryview]) -> None`: Write a frame
- `write_frame_zero_copy(data: memoryview) -> None`: Write frame with zero-copy
- `get_frame_buffer(size: int) -> memoryview`: Get buffer for direct writing
- `commit_frame() -> None`: Commit frame after direct writing
- `is_reader_connected() -> bool`: Check if reader is connected
- `close() -> None`: Close writer
### Frame Class
Represents a zero-copy reference to frame data.
**Properties:**
- `data -> memoryview`: Zero-copy view of frame data
- `size -> int`: Size of frame data
- `sequence -> int`: Sequence number
### BufferConfig Class
```python
BufferConfig(metadata_size: int = 1024, payload_size: int = 1024*1024)
```
Configuration for creating a buffer.
## Zero-Copy Guarantees
The Python implementation provides true zero-copy access through:
1. **memoryview objects**: No data copying when accessing frame data
2. **Buffer protocol**: Direct memory access for compatible objects
3. **Shared memory**: Direct mapping of shared memory into process space
### When Copies Occur
- Converting memoryview to bytes: `bytes(frame.data)`
- Using non-buffer protocol objects with `write_frame()`
- String encoding: `"text".encode()`
### Avoiding Copies
- Use `memoryview` objects whenever possible
- Use `write_frame_zero_copy()` for memoryview data
- Use numpy arrays or other buffer protocol objects
- Access frame data directly via `frame.data` memoryview
## Performance Considerations
1. **Pre-allocate buffers**: Reuse buffers instead of creating new ones
2. **Batch operations**: Process multiple frames before releasing
3. **Use appropriate buffer sizes**: See capacity planning in main README
4. **Monitor buffer utilization**: Avoid buffer full conditions
## Error Handling
All operations may raise exceptions from `zerobuffer.exceptions`:
- `WriterDeadException`: Writer process died
- `ReaderDeadException`: Reader process died
- `BufferFullException`: Buffer is full
- `FrameTooLargeException`: Frame exceeds buffer capacity
- `SequenceError`: Frame sequence validation failed
## Thread Safety
The Reader and Writer classes are thread-safe. Multiple threads can:
- Call methods on the same Reader/Writer instance
- Read frames concurrently (with proper frame release)
- Write frames concurrently
However, only one Reader and one Writer can connect to a buffer at a time.
## Platform Notes
### Linux
- Uses POSIX shared memory (`/dev/shm`)
- Requires `posix-ipc` package
- File locks in `/tmp/zerobuffer/`
### Windows
- Uses Windows named shared memory
- Requires `pywin32` package
- File locks in temp directory
### macOS
- Similar to Linux with BSD-specific handling
- Requires `posix-ipc` package
## Testing Utilities
The Python implementation includes testing utilities for cross-platform compatibility with C# and C++ implementations:
### BufferNamingService
Ensures unique buffer names across test runs to prevent conflicts:
```python
from zerobuffer_serve.services import BufferNamingService
# Creates unique buffer names for test isolation
naming_service = BufferNamingService(logger)
actual_name = naming_service.get_buffer_name("test-buffer")
# Returns: "test-buffer_<pid>_<timestamp>" or uses Harmony environment variables
```
### TestDataPatterns
Provides consistent test data generation across all language implementations:
```python
from zerobuffer_serve.test_data_patterns import TestDataPatterns
# Generate deterministic frame data
data = TestDataPatterns.generate_frame_data(size=1024, sequence=1)
# Generate simple pattern data
simple_data = TestDataPatterns.generate_simple_frame_data(size=1024)
# Verify data matches pattern
is_valid = TestDataPatterns.verify_simple_frame_data(data)
# Generate test metadata
metadata = TestDataPatterns.generate_metadata(size=256)
```
These utilities ensure that Python, C#, and C++ implementations can exchange data correctly in cross-platform tests.
## Duplex Channel Support
The Python implementation includes full support for duplex channels (bidirectional request-response communication):
```python
from zerobuffer.duplex import DuplexChannelFactory, ProcessingMode
from zerobuffer import BufferConfig
# Server side
factory = DuplexChannelFactory()
server = factory.create_immutable_server("my-service", BufferConfig(4096, 10*1024*1024))
def process_request(frame):
"""Process request and return response"""
# Frame is automatically disposed via RAII
data = bytes(frame.data)
result = process_data(data)
return result
server.start(process_request, ProcessingMode.SINGLE_THREAD)
# Client side
client = factory.create_client("my-service")
sequence = client.send_request(b"request data")
response = client.receive_response(timeout_ms=5000)
if response.is_valid and response.sequence == sequence:
with response: # Context manager for RAII
print(f"Response: {bytes(response.data)}")
```
See [DUPLEX_CHANNEL.md](DUPLEX_CHANNEL.md) for detailed documentation.
Raw data
{
"_id": null,
"home_page": "https://github.com/zerobuffer/zerobuffer-python",
"name": "zerobuffer-ipc",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "ipc, shared-memory, zero-copy, streaming, video",
"author": "ZeroBuffer Contributors",
"author_email": "ModelingEvolution <info@modelingevolution.com>",
"download_url": "https://files.pythonhosted.org/packages/1b/8a/7c5e30f58a52331d075888cbcfce8ad6d35b8af1d86f7cb29b7b84a9449d/zerobuffer_ipc-1.1.0.tar.gz",
"platform": null,
"description": "# ZeroBuffer Python Implementation\n\nNative Python implementation of the ZeroBuffer protocol for high-performance zero-copy inter-process communication.\n\n## Features\n\n- **True Zero-Copy**: Uses memoryview and buffer protocol to avoid data copies\n- **Cross-Platform**: Supports Linux, Windows, and macOS\n- **Protocol Compatible**: Binary compatible with C++ and C# implementations\n- **Pythonic API**: Clean, idiomatic Python interface\n- **Type Safe**: Full type hints for better IDE support\n- **Thread Safe**: Built-in synchronization for multi-threaded applications\n\n## Requirements\n\n- Python 3.8 or later\n- `posix-ipc` (Linux/macOS only): `pip install posix-ipc`\n- `pywin32` (Windows only): `pip install pywin32`\n\n## Installation\n\n```bash\npip install -e .\n```\n\n## Quick Start\n\n### Reader Example\n\n```python\nfrom zerobuffer import Reader, BufferConfig\n\n# Create a buffer\nconfig = BufferConfig(metadata_size=1024, payload_size=1024*1024)\nwith Reader(\"my-buffer\", config) as reader:\n # Wait for writer to connect\n print(\"Waiting for writer...\")\n \n # Read frames\n while True:\n frame = reader.read_frame(timeout=5.0)\n if frame:\n # Access data without copying\n data = frame.data # This is a memoryview\n print(f\"Received frame {frame.sequence}: {len(data)} bytes\")\n \n # Process the frame...\n \n # Must release frame to free buffer space\n reader.release_frame(frame)\n```\n\n### Writer Example\n\n```python\nfrom zerobuffer import Writer\n\n# Connect to existing buffer\nwith Writer(\"my-buffer\") as writer:\n # Write some metadata\n metadata = b\"{'format': 'raw', 'version': 1}\"\n writer.set_metadata(metadata)\n \n # Write frames\n for i in range(100):\n data = f\"Frame {i}\".encode()\n writer.write_frame(data)\n print(f\"Sent frame {i}\")\n```\n\n### Zero-Copy Advanced Usage\n\n```python\n# True zero-copy writing with memoryview\nimport numpy as np\n\n# Create numpy array\narr = np.arange(1000, dtype=np.float32)\n\n# Get memoryview of array (no copy)\nview = memoryview(arr)\n\n# Write with zero-copy\nwriter.write_frame_zero_copy(view)\n\n# Or use direct buffer access\nbuffer = writer.get_frame_buffer(size=4000)\n# Write directly into shared memory\nbuffer[:] = arr.tobytes()\nwriter.commit_frame()\n```\n\n## API Reference\n\n### Reader Class\n\n```python\nReader(name: str, config: Optional[BufferConfig] = None)\n```\n\nCreates a new buffer and prepares for reading.\n\n**Methods:**\n- `get_metadata() -> Optional[memoryview]`: Get metadata as zero-copy memoryview\n- `read_frame(timeout: Optional[float] = 5.0) -> Optional[Frame]`: Read next frame\n- `release_frame(frame: Frame) -> None`: Release frame and free buffer space\n- `is_writer_connected() -> bool`: Check if writer is connected\n- `close() -> None`: Close reader and clean up resources\n\n### Writer Class\n\n```python\nWriter(name: str)\n```\n\nConnects to an existing buffer for writing.\n\n**Methods:**\n- `set_metadata(data: Union[bytes, bytearray, memoryview]) -> None`: Write metadata (once only)\n- `write_frame(data: Union[bytes, bytearray, memoryview]) -> None`: Write a frame\n- `write_frame_zero_copy(data: memoryview) -> None`: Write frame with zero-copy\n- `get_frame_buffer(size: int) -> memoryview`: Get buffer for direct writing\n- `commit_frame() -> None`: Commit frame after direct writing\n- `is_reader_connected() -> bool`: Check if reader is connected\n- `close() -> None`: Close writer\n\n### Frame Class\n\nRepresents a zero-copy reference to frame data.\n\n**Properties:**\n- `data -> memoryview`: Zero-copy view of frame data\n- `size -> int`: Size of frame data\n- `sequence -> int`: Sequence number\n\n### BufferConfig Class\n\n```python\nBufferConfig(metadata_size: int = 1024, payload_size: int = 1024*1024)\n```\n\nConfiguration for creating a buffer.\n\n## Zero-Copy Guarantees\n\nThe Python implementation provides true zero-copy access through:\n\n1. **memoryview objects**: No data copying when accessing frame data\n2. **Buffer protocol**: Direct memory access for compatible objects\n3. **Shared memory**: Direct mapping of shared memory into process space\n\n### When Copies Occur\n\n- Converting memoryview to bytes: `bytes(frame.data)`\n- Using non-buffer protocol objects with `write_frame()`\n- String encoding: `\"text\".encode()`\n\n### Avoiding Copies\n\n- Use `memoryview` objects whenever possible\n- Use `write_frame_zero_copy()` for memoryview data\n- Use numpy arrays or other buffer protocol objects\n- Access frame data directly via `frame.data` memoryview\n\n## Performance Considerations\n\n1. **Pre-allocate buffers**: Reuse buffers instead of creating new ones\n2. **Batch operations**: Process multiple frames before releasing\n3. **Use appropriate buffer sizes**: See capacity planning in main README\n4. **Monitor buffer utilization**: Avoid buffer full conditions\n\n## Error Handling\n\nAll operations may raise exceptions from `zerobuffer.exceptions`:\n\n- `WriterDeadException`: Writer process died\n- `ReaderDeadException`: Reader process died \n- `BufferFullException`: Buffer is full\n- `FrameTooLargeException`: Frame exceeds buffer capacity\n- `SequenceError`: Frame sequence validation failed\n\n## Thread Safety\n\nThe Reader and Writer classes are thread-safe. Multiple threads can:\n- Call methods on the same Reader/Writer instance\n- Read frames concurrently (with proper frame release)\n- Write frames concurrently\n\nHowever, only one Reader and one Writer can connect to a buffer at a time.\n\n## Platform Notes\n\n### Linux\n- Uses POSIX shared memory (`/dev/shm`)\n- Requires `posix-ipc` package\n- File locks in `/tmp/zerobuffer/`\n\n### Windows\n- Uses Windows named shared memory\n- Requires `pywin32` package\n- File locks in temp directory\n\n### macOS\n- Similar to Linux with BSD-specific handling\n- Requires `posix-ipc` package\n\n## Testing Utilities\n\nThe Python implementation includes testing utilities for cross-platform compatibility with C# and C++ implementations:\n\n### BufferNamingService\n\nEnsures unique buffer names across test runs to prevent conflicts:\n\n```python\nfrom zerobuffer_serve.services import BufferNamingService\n\n# Creates unique buffer names for test isolation\nnaming_service = BufferNamingService(logger)\nactual_name = naming_service.get_buffer_name(\"test-buffer\")\n# Returns: \"test-buffer_<pid>_<timestamp>\" or uses Harmony environment variables\n```\n\n### TestDataPatterns\n\nProvides consistent test data generation across all language implementations:\n\n```python\nfrom zerobuffer_serve.test_data_patterns import TestDataPatterns\n\n# Generate deterministic frame data\ndata = TestDataPatterns.generate_frame_data(size=1024, sequence=1)\n\n# Generate simple pattern data\nsimple_data = TestDataPatterns.generate_simple_frame_data(size=1024)\n\n# Verify data matches pattern\nis_valid = TestDataPatterns.verify_simple_frame_data(data)\n\n# Generate test metadata\nmetadata = TestDataPatterns.generate_metadata(size=256)\n```\n\nThese utilities ensure that Python, C#, and C++ implementations can exchange data correctly in cross-platform tests.\n\n## Duplex Channel Support\n\nThe Python implementation includes full support for duplex channels (bidirectional request-response communication):\n\n```python\nfrom zerobuffer.duplex import DuplexChannelFactory, ProcessingMode\nfrom zerobuffer import BufferConfig\n\n# Server side\nfactory = DuplexChannelFactory()\nserver = factory.create_immutable_server(\"my-service\", BufferConfig(4096, 10*1024*1024))\n\ndef process_request(frame):\n \"\"\"Process request and return response\"\"\"\n # Frame is automatically disposed via RAII\n data = bytes(frame.data)\n result = process_data(data)\n return result\n\nserver.start(process_request, ProcessingMode.SINGLE_THREAD)\n\n# Client side\nclient = factory.create_client(\"my-service\")\nsequence = client.send_request(b\"request data\")\nresponse = client.receive_response(timeout_ms=5000)\n\nif response.is_valid and response.sequence == sequence:\n with response: # Context manager for RAII\n print(f\"Response: {bytes(response.data)}\")\n```\n\nSee [DUPLEX_CHANNEL.md](DUPLEX_CHANNEL.md) for detailed documentation.\n",
"bugtrack_url": null,
"license": null,
"summary": "High-performance zero-copy inter-process communication library for video streaming",
"version": "1.1.0",
"project_urls": {
"Documentation": "https://github.com/modelingevolution/zerobuffer#readme",
"Homepage": "https://github.com/modelingevolution/zerobuffer",
"Issues": "https://github.com/modelingevolution/zerobuffer/issues",
"Repository": "https://github.com/modelingevolution/zerobuffer"
},
"split_keywords": [
"ipc",
" shared-memory",
" zero-copy",
" streaming",
" video"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "2b9a06605854ed0d5e96bc98cacd942ffb90d7cf32b259256acf66ed071764cc",
"md5": "f08691c0e1fa98285eccc8afbb9ede19",
"sha256": "1ee0c25cfa5b906695e94974c28634382ac797fd4241c50c8cf5ddafb1ade540"
},
"downloads": -1,
"filename": "zerobuffer_ipc-1.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f08691c0e1fa98285eccc8afbb9ede19",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 100055,
"upload_time": "2025-08-15T17:11:23",
"upload_time_iso_8601": "2025-08-15T17:11:23.068848Z",
"url": "https://files.pythonhosted.org/packages/2b/9a/06605854ed0d5e96bc98cacd942ffb90d7cf32b259256acf66ed071764cc/zerobuffer_ipc-1.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "1b8a7c5e30f58a52331d075888cbcfce8ad6d35b8af1d86f7cb29b7b84a9449d",
"md5": "54ff5b6ea5cee28faaea5c472a29bfa0",
"sha256": "246e8d06240c5819de28d34a238b8f24e9e63cb1434eaf3278a78859d5ad02fa"
},
"downloads": -1,
"filename": "zerobuffer_ipc-1.1.0.tar.gz",
"has_sig": false,
"md5_digest": "54ff5b6ea5cee28faaea5c472a29bfa0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 98967,
"upload_time": "2025-08-15T17:11:24",
"upload_time_iso_8601": "2025-08-15T17:11:24.150313Z",
"url": "https://files.pythonhosted.org/packages/1b/8a/7c5e30f58a52331d075888cbcfce8ad6d35b8af1d86f7cb29b7b84a9449d/zerobuffer_ipc-1.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-15 17:11:24",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "zerobuffer",
"github_project": "zerobuffer-python",
"github_not_found": true,
"lcname": "zerobuffer-ipc"
}
JSON
