| Name | meesee JSON |
| Version |
1.6.0
JSON |
| download |
| home_page | https://github.com/Attumm/meesee |
| Summary | Task queue, Long lived workers process parallelization, with Redis as backend |
| upload_time | 2024-08-16 07:25:50 |
| maintainer | None |
| docs_url | None |
| author | Melvin Bijman |
| requires_python | >3.5 |
| license | MIT |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
|
| coveralls test coverage |
No coveralls.
|
# Meesee
[](https://github.com/Attumm/meesee/actions/workflows/ci.yml)
[](https://codecov.io/gh/Attumm/meesee)
[](https://pepy.tech/project/meesee)
Meesee is an task queue system featuring long-lived worker process parallelization through multiprocessing, with Redis as its backend. Engineered for reliability, efficiency, and ease of use, Meesee is tailored for distributed computing environments, particularly in big data and mission-critical software applications. By leveraging individual processes for each worker, Meesee circumvents Python's Global Interpreter Lock (GIL), making it ideal for compute intensive tasks.
## Production Proven
- In active production since 2018, powering the backends of at least three known companies
- Instances have demonstrated exceptional uptime, running for years without requiring maintenance
- The only identified scenario necessitating a restart is during network interface changes, which can leave workers connected to a non-functional network—an infrequent occurrence, but noteworthy for systems with multi-year uptime expectations
- Meesee workers are designed to restart without data loss, ensuring continuity even during rare restart events
## Core Design Principles
Meesee was specifically developed to address the following critical challenges in distributed computing:
1. **Long-Term Stability**: Ability to run for extended periods without maintenance or restarts
2. **Zero Message Loss**: Ensuring no messages are lost during service restarts for maintenance or deployments
3. **Optimized Performance**: Achieving surprising speed with minimal memory overhead for both client and Redis instances
4. **Deployment Flexibility**: Capability to schedule messages even when workers are offline during deployment
5. **Message Integrity Under Load**: Preventing message skips even during high-load scenarios
6. **Simplicity in Complexity**: Providing an intuitive interface to minimize the learning curve, acknowledging that distributed computing is challenging enough on its own
## Examples
How to [Examples](https://github.com/Attumm/meesee/tree/main/examples).
Create my_func that will
1. print starting message.
2. Sleep 1 second.
3. print a ending message.
Let's start 10 of those.
```python
import time
from meesee import startapp
def my_func(item, worker_id):
print("hello, look at me")
time.sleep(1)
print('finished item', locals())
startapp(my_func, workers=10)
```
Open another terminal, Let's produce some tasks
```python
from meesee import RedisQueue, config
def produce(items):
r = RedisQueue(**config)
for i in range(items):
r.send(i)
produce(10)
```
Great, the placement of both scripts can be on any machine with connectivity to the redis instance.
## Install
```
$ pip install meesee
```
## Example Usage
Let's use Python to make writing workers and producers more fun.
Here's a simple [example](https://github.com/Attumm/meesee/tree/main/examples/example_decorator_magic_simple.py) demonstrating how to use Meesee the pythonic way.
```python
from meesee import Meesee
box = Meesee()
@box.worker()
def foobar(item, worker_id):
print('func: foobar, worker_id: {}, item: {}'.format(worker_id, item))
@box.produce()
def produce_to_foobar(items):
return items
if __name__ == '__main__':
items = [{"name": f"name{i}"} for i in range(10)]
produce_to_foobar(items)
box.push_button(workers=5, wait=1)
```
This example demonstrates:
1. Creating a Meesee instance
2. Defining a worker function using the `@box.worker()` decorator
3. Defining a producer function using the `@box.produce()` decorator
4. Producing items to the queue
5. Starting workers to process the items
Example output
```bash
worker 1 started. foobar listening to foobar
worker 2 started. foobar listening to foobar
worker 3 started. foobar listening to foobar
worker 4 started. foobar listening to foobar
func: foobar, worker_id: 1, item: {"name": "name0"}
func: foobar, worker_id: 1, item: {"name": "name1"}
worker 5 started. foobar listening to foobar
func: foobar, worker_id: 2, item: {"name": "name4"}
func: foobar, worker_id: 3, item: {"name": "name2"}
func: foobar, worker_id: 4, item: {"name": "name3"}
func: foobar, worker_id: 1, item: {"name": "name5"}
func: foobar, worker_id: 1, item: {"name": "name6"}
func: foobar, worker_id: 3, item: {"name": "name7"}
func: foobar, worker_id: 4, item: {"name": "name8"}
func: foobar, worker_id: 2, item: {"name": "name9"}
timeout reached worker 5 stopped
timeout reached worker 2 stopped
timeout reached worker 1 stopped
timeout reached worker 4 stopped
timeout reached worker 3 stopped
Clean shut down
```
This output shows:
- Workers starting and listening to the 'foobar' queue
- Items being processed by different workers
- Workers shutting down after the timeout is reached
## Usage explained
Producers produce to workers, hence the name. They can either pass iterable values or iter themselves. For instance:
```python
@box.produce()
def produce():
return [1, 2, 3]
# or
@box.produce()
def produce_yield():
yield from [1, 2, 3]
```
We can control which queue they will message to in two ways:
1. Specify the queue in the decorator:
```python
@box.produce(queue="foobar")
def produce_yield():
yield from [1, 2, 3]
```
This will produce to the "foobar" queue.
2. Use magic naming:
```python
@box.produce()
def produce_to_foobar():
yield from [1, 2, 3]
```
By naming the function `produce_to_foobar`, the function will also send the data to the "foobar" queue.
For workers, they are special in that they will start during multiprocessing. Here's an example to start 5 workers. Since we only set up one worker, all workers will be of that type:
```python
box.push_button(workers=5, wait=1)
```
This will start 5 worker processes, each listening to the queue specified in the worker function.
### Prerequisites
#### Redis instance
For Docker
```
$ docker run --name some-redis -d redis
```
## Support and Resources
- For feature requests, additional information or to report issues use github issues.
- Explore our comprehensive [examples](https://github.com/Attumm/meesee/tree/main/examples) for in-depth usage scenarios and best practices.
Raw data
{
"_id": null,
"home_page": "https://github.com/Attumm/meesee",
"name": "meesee",
"maintainer": null,
"docs_url": null,
"requires_python": ">3.5",
"maintainer_email": null,
"keywords": null,
"author": "Melvin Bijman",
"author_email": "bijman.m.m@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/d5/2f/5bb00851b23b1b10d0c67b8a8a575eb778d0af10e28700ff5b16e66d6fba/meesee-1.6.0.tar.gz",
"platform": null,
"description": "# Meesee\n[](https://github.com/Attumm/meesee/actions/workflows/ci.yml)\n[](https://codecov.io/gh/Attumm/meesee)\n[](https://pepy.tech/project/meesee)\n\nMeesee is an task queue system featuring long-lived worker process parallelization through multiprocessing, with Redis as its backend. Engineered for reliability, efficiency, and ease of use, Meesee is tailored for distributed computing environments, particularly in big data and mission-critical software applications. By leveraging individual processes for each worker, Meesee circumvents Python's Global Interpreter Lock (GIL), making it ideal for compute intensive tasks.\n\n## Production Proven\n\n- In active production since 2018, powering the backends of at least three known companies\n- Instances have demonstrated exceptional uptime, running for years without requiring maintenance\n- The only identified scenario necessitating a restart is during network interface changes, which can leave workers connected to a non-functional network\u2014an infrequent occurrence, but noteworthy for systems with multi-year uptime expectations\n- Meesee workers are designed to restart without data loss, ensuring continuity even during rare restart events\n\n\n## Core Design Principles\n\nMeesee was specifically developed to address the following critical challenges in distributed computing:\n\n1. **Long-Term Stability**: Ability to run for extended periods without maintenance or restarts\n2. **Zero Message Loss**: Ensuring no messages are lost during service restarts for maintenance or deployments\n3. **Optimized Performance**: Achieving surprising speed with minimal memory overhead for both client and Redis instances\n4. **Deployment Flexibility**: Capability to schedule messages even when workers are offline during deployment\n5. **Message Integrity Under Load**: Preventing message skips even during high-load scenarios\n6. **Simplicity in Complexity**: Providing an intuitive interface to minimize the learning curve, acknowledging that distributed computing is challenging enough on its own\n\n\n## Examples\nHow to [Examples](https://github.com/Attumm/meesee/tree/main/examples).\n\nCreate my_func that will \n1. print starting message.\n2. Sleep 1 second.\n3. print a ending message.\n\nLet's start 10 of those.\n\n\n```python\nimport time\nfrom meesee import startapp\n\ndef my_func(item, worker_id):\n print(\"hello, look at me\")\n time.sleep(1)\n print('finished item', locals())\n\n\nstartapp(my_func, workers=10)\n```\n\nOpen another terminal, Let's produce some tasks\n```python\nfrom meesee import RedisQueue, config\n\ndef produce(items):\n r = RedisQueue(**config)\n for i in range(items):\n r.send(i)\n\nproduce(10)\n\n```\n\nGreat, the placement of both scripts can be on any machine with connectivity to the redis instance.\n\n## Install\n\n```\n$ pip install meesee\n```\n\n## Example Usage\n\nLet's use Python to make writing workers and producers more fun.\nHere's a simple [example](https://github.com/Attumm/meesee/tree/main/examples/example_decorator_magic_simple.py) demonstrating how to use Meesee the pythonic way.\n\n```python\nfrom meesee import Meesee \n\nbox = Meesee()\n\n@box.worker()\ndef foobar(item, worker_id):\n print('func: foobar, worker_id: {}, item: {}'.format(worker_id, item))\n\n@box.produce()\ndef produce_to_foobar(items):\n return items\n\nif __name__ == '__main__':\n items = [{\"name\": f\"name{i}\"} for i in range(10)]\n produce_to_foobar(items)\n box.push_button(workers=5, wait=1)\n```\n\nThis example demonstrates:\n1. Creating a Meesee instance\n2. Defining a worker function using the `@box.worker()` decorator\n3. Defining a producer function using the `@box.produce()` decorator\n4. Producing items to the queue\n5. Starting workers to process the items\n\n\n\nExample output\n```bash\nworker 1 started. foobar listening to foobar\nworker 2 started. foobar listening to foobar\nworker 3 started. foobar listening to foobar\nworker 4 started. foobar listening to foobar\nfunc: foobar, worker_id: 1, item: {\"name\": \"name0\"}\nfunc: foobar, worker_id: 1, item: {\"name\": \"name1\"}\nworker 5 started. foobar listening to foobar\nfunc: foobar, worker_id: 2, item: {\"name\": \"name4\"}\nfunc: foobar, worker_id: 3, item: {\"name\": \"name2\"}\nfunc: foobar, worker_id: 4, item: {\"name\": \"name3\"}\nfunc: foobar, worker_id: 1, item: {\"name\": \"name5\"}\nfunc: foobar, worker_id: 1, item: {\"name\": \"name6\"}\nfunc: foobar, worker_id: 3, item: {\"name\": \"name7\"}\nfunc: foobar, worker_id: 4, item: {\"name\": \"name8\"}\nfunc: foobar, worker_id: 2, item: {\"name\": \"name9\"}\ntimeout reached worker 5 stopped\ntimeout reached worker 2 stopped\ntimeout reached worker 1 stopped\ntimeout reached worker 4 stopped\ntimeout reached worker 3 stopped\nClean shut down\n```\n\nThis output shows:\n- Workers starting and listening to the 'foobar' queue\n- Items being processed by different workers\n- Workers shutting down after the timeout is reached\n\n## Usage explained\n\n\nProducers produce to workers, hence the name. They can either pass iterable values or iter themselves. For instance:\n\n```python\n@box.produce()\ndef produce():\n return [1, 2, 3]\n\n# or \n\n@box.produce()\ndef produce_yield():\n yield from [1, 2, 3]\n```\n\nWe can control which queue they will message to in two ways:\n\n1. Specify the queue in the decorator:\n```python\n@box.produce(queue=\"foobar\")\ndef produce_yield():\n yield from [1, 2, 3]\n```\nThis will produce to the \"foobar\" queue.\n\n2. Use magic naming:\n```python\n@box.produce()\ndef produce_to_foobar():\n yield from [1, 2, 3]\n```\nBy naming the function `produce_to_foobar`, the function will also send the data to the \"foobar\" queue.\n\nFor workers, they are special in that they will start during multiprocessing. Here's an example to start 5 workers. Since we only set up one worker, all workers will be of that type:\n\n```python\nbox.push_button(workers=5, wait=1)\n```\n\nThis will start 5 worker processes, each listening to the queue specified in the worker function.\n\n### Prerequisites\n\n#### Redis instance\n\nFor Docker\n```\n$ docker run --name some-redis -d redis\n```\n\n## Support and Resources\n\n- For feature requests, additional information or to report issues use github issues.\n- Explore our comprehensive [examples](https://github.com/Attumm/meesee/tree/main/examples) for in-depth usage scenarios and best practices.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Task queue, Long lived workers process parallelization, with Redis as backend",
"version": "1.6.0",
"project_urls": {
"Homepage": "https://github.com/Attumm/meesee"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "16bb3154bb0b6401b6640480ed2d6b192574d0a84911909ad10eb5f409084bbb",
"md5": "39fff66c54bda79c03310afc32dc9c75",
"sha256": "8d67b46d9bbd412c2725b670ff54a2ab9d973b971a9abb475697222355e4c6ea"
},
"downloads": -1,
"filename": "meesee-1.6.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "39fff66c54bda79c03310afc32dc9c75",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">3.5",
"size": 7425,
"upload_time": "2024-08-16T07:25:48",
"upload_time_iso_8601": "2024-08-16T07:25:48.815144Z",
"url": "https://files.pythonhosted.org/packages/16/bb/3154bb0b6401b6640480ed2d6b192574d0a84911909ad10eb5f409084bbb/meesee-1.6.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d52f5bb00851b23b1b10d0c67b8a8a575eb778d0af10e28700ff5b16e66d6fba",
"md5": "cc99906f1461fe6990ed6cc419844ded",
"sha256": "f51e735d8b53fe5a52bc0d90aebeb03ce7baa02a5f8c10d1fba8e6b6f578176a"
},
"downloads": -1,
"filename": "meesee-1.6.0.tar.gz",
"has_sig": false,
"md5_digest": "cc99906f1461fe6990ed6cc419844ded",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">3.5",
"size": 8423,
"upload_time": "2024-08-16T07:25:50",
"upload_time_iso_8601": "2024-08-16T07:25:50.183782Z",
"url": "https://files.pythonhosted.org/packages/d5/2f/5bb00851b23b1b10d0c67b8a8a575eb778d0af10e28700ff5b16e66d6fba/meesee-1.6.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-16 07:25:50",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Attumm",
"github_project": "meesee",
"travis_ci": true,
"coveralls": false,
"github_actions": true,
"requirements": [],
"tox": true,
"lcname": "meesee"
}
