| Name | prefect-tool-state JSON |
| Version |
0.0.5
JSON |
| download |
| home_page | None |
| Summary | A high-throughput progress publisher for Prefect tools with optimized batching |
| upload_time | 2025-07-29 10:42:08 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.8 |
| license | None |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Prefect Tool State
A high-throughput tool state publisher for Prefect tools with optimized batching capabilities.
## Features
- **High-frequency publishing**: Optimized for scenarios where tool state updates are sent after each file is processed
- **Batch optimization**: Collects messages over 1-second intervals and publishes them as a single JSON array
- **Background worker**: Non-blocking interface with async background processing
- **Thread-safe**: Safe to use from multiple threads
- **Automatic resource management**: Handles Redis connection lifecycle automatically
## Installation
```bash
pip install prefect-tool-state
```
## Quick Start
```python
# Quick Start updated for state publisher
from prefect_tool_state import get_tool_state_publisher, ToolStatePublisher, ToolState
# Get the global state publisher instance
publisher = get_tool_state_publisher()
# Publish tool state updates (non-blocking)
state_publisher = get_tool_state_publisher()
tool_state = ToolState(
tool_run_id="tool-123",
status="processing",
current_file="example.txt",
# ... other fields
)
state_publisher.publish_data(tool_state)
```
## Configuration
The publisher requires Redis messaging environment variables:
- `PREFECT_REDIS_MESSAGING_HOST`: Redis host
- `PREFECT_REDIS_MESSAGING_PASSWORD`: Redis password
## How It Works
1. **Queue-based**: Your calls to `publish_data()` are queued in memory (non-blocking)
2. **Batched publishing**: A background worker wakes up every second to process all queued messages
3. **Optimized payload**: All messages are serialized into a single JSON array and published as one Redis message
4. **Consumer compatibility**: The consumer must handle the batch format (array of tool states)
## Performance
- **Before**: 1000 file updates = 1000 Redis operations
- **After**: 1000 file updates = 1 Redis operation per second
- **Latency**: Sub-millisecond for `publish_data()` calls
- **Throughput**: Handles thousands of updates per second
## License
MIT License
Raw data
{
"_id": null,
"home_page": null,
"name": "prefect-tool-state",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": null,
"author": null,
"author_email": "Team <team@example.com>",
"download_url": "https://files.pythonhosted.org/packages/e8/c1/88f4d81670d32052a94ac8be5a0b77b7a8f8860560f626065dd48be7f5f1/prefect_tool_state-0.0.5.tar.gz",
"platform": null,
"description": "# Prefect Tool State\n\nA high-throughput tool state publisher for Prefect tools with optimized batching capabilities.\n\n## Features\n\n- **High-frequency publishing**: Optimized for scenarios where tool state updates are sent after each file is processed\n- **Batch optimization**: Collects messages over 1-second intervals and publishes them as a single JSON array\n- **Background worker**: Non-blocking interface with async background processing\n- **Thread-safe**: Safe to use from multiple threads\n- **Automatic resource management**: Handles Redis connection lifecycle automatically\n\n## Installation\n\n```bash\npip install prefect-tool-state\n```\n\n## Quick Start\n\n```python\n# Quick Start updated for state publisher\nfrom prefect_tool_state import get_tool_state_publisher, ToolStatePublisher, ToolState\n\n# Get the global state publisher instance\npublisher = get_tool_state_publisher()\n\n# Publish tool state updates (non-blocking)\nstate_publisher = get_tool_state_publisher()\ntool_state = ToolState(\n tool_run_id=\"tool-123\",\n status=\"processing\",\n current_file=\"example.txt\",\n # ... other fields\n)\nstate_publisher.publish_data(tool_state)\n```\n\n## Configuration\n\nThe publisher requires Redis messaging environment variables:\n- `PREFECT_REDIS_MESSAGING_HOST`: Redis host\n- `PREFECT_REDIS_MESSAGING_PASSWORD`: Redis password\n\n## How It Works\n\n1. **Queue-based**: Your calls to `publish_data()` are queued in memory (non-blocking)\n2. **Batched publishing**: A background worker wakes up every second to process all queued messages\n3. **Optimized payload**: All messages are serialized into a single JSON array and published as one Redis message\n4. **Consumer compatibility**: The consumer must handle the batch format (array of tool states)\n\n## Performance\n\n- **Before**: 1000 file updates = 1000 Redis operations\n- **After**: 1000 file updates = 1 Redis operation per second\n- **Latency**: Sub-millisecond for `publish_data()` calls\n- **Throughput**: Handles thousands of updates per second\n\n## License\n\nMIT License\n",
"bugtrack_url": null,
"license": null,
"summary": "A high-throughput progress publisher for Prefect tools with optimized batching",
"version": "0.0.5",
"project_urls": {
"Bug Tracker": "https://github.com/your-org/prefect-tool-state/issues",
"Documentation": "https://github.com/your-org/prefect-tool-state#readme",
"Homepage": "https://github.com/your-org/prefect-tool-state"
},
"split_keywords": [],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "f06bc4154e8028530a3fe6b9ffa508194c62b34c4467164ffa2421062033f3fe",
"md5": "59807d541ca861673481d22b3729c103",
"sha256": "fd7e780d5399df361de22b79f2fca901c6a6c175ac47c6621258fcf34058eb9e"
},
"downloads": -1,
"filename": "prefect_tool_state-0.0.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "59807d541ca861673481d22b3729c103",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 6370,
"upload_time": "2025-07-29T10:42:07",
"upload_time_iso_8601": "2025-07-29T10:42:07.559695Z",
"url": "https://files.pythonhosted.org/packages/f0/6b/c4154e8028530a3fe6b9ffa508194c62b34c4467164ffa2421062033f3fe/prefect_tool_state-0.0.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "e8c188f4d81670d32052a94ac8be5a0b77b7a8f8860560f626065dd48be7f5f1",
"md5": "d5602d1f77b1a7fa8d08602235c7a87b",
"sha256": "3a45bbcb086e9c67e25bb32ccd4202d26b6c3d644b190ad4933b230d7badf8df"
},
"downloads": -1,
"filename": "prefect_tool_state-0.0.5.tar.gz",
"has_sig": false,
"md5_digest": "d5602d1f77b1a7fa8d08602235c7a87b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 5491,
"upload_time": "2025-07-29T10:42:08",
"upload_time_iso_8601": "2025-07-29T10:42:08.821295Z",
"url": "https://files.pythonhosted.org/packages/e8/c1/88f4d81670d32052a94ac8be5a0b77b7a8f8860560f626065dd48be7f5f1/prefect_tool_state-0.0.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-29 10:42:08",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "your-org",
"github_project": "prefect-tool-state",
"github_not_found": true,
"lcname": "prefect-tool-state"
}
