| Name | azqueuemanager JSON |
| Version |
0.0.6
JSON |
| download |
| home_page | |
| Summary | Storage Queue Input-Output Manager for Azure Storage Queue |
| upload_time | 2022-12-16 00:11:35 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.10 |
| license | MIT |
| keywords |
azqueuemanager
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# AZ Queue Manager
AZ Queue Manager is a tool to manage messages in Azure Storage Queue.
It is a command line tool that can be used to send and receive messages from a queue. It can also be used to create and delete queues.
## Getting Started
### Install AZQueueManager and Extensions
```bash
# Install the base package
pip install azqueuemanager
# Install the csv-input extension
pip install azqeueuemanager-csv # or some other extension
```
## Setup your script
If this is your input file `data.json`
```json
# data.json
[
{"id": "msg_1", "text": "hello world"},
{"id": "msg_2": "text": "hello universe"}
]
```
Create your script.
```python
# test.py
from azqueuemanager import QueueManager,
from azqueuemanager-json import JSONTransformer
```
We'll come back to the extension. For now, let's define our queue_client. This is the storage queue itself.
```python
# test.py
CONNECTION_STRING="DefaultEndpointsProtocol=https;AccountName=MYACCOUNT;AccountKey=MYKEY;EndpointSuffix=core.windows.net" #You'll need your own connection string
queue_name="MYQUEUENAME"
queue = StorageQueue(
connection_string,
queue_name,
)
```
Next we'll need to setup our extension. Every extension has a transformer. That transformer has modules for passing data into and out of the queue. The input transformer will take data and create one or more messages. The output transformer will take a message and create output for each record.
These inputs and outputs are pretty dynamic so you will need to look at the documentation for each extension to see what the input and output looks like.
The json extension the input transformer takes `json_in_file` (or `json_data`) and (optionally) a `json_out_file`. The `transform_in` will add messages from the json content. The `transform_out` will take the messages and create a json file with all the records returned.
Let's create an instance of the JSONTransformer and pass it to the QueueManager along with our queue.
```python
# test.py
json_transform = JSONTransformer(
json_in_file='data.json',
)
queue_manager = QueueManager(
queue=queue
input_transformer=json_transform,
output_transformer=json_transform,
)
```
Now we can add or pull data to/from our Queue
### Loading Data
We pass data in using the `queue_messages` method. This will use the data processed from the `input_transformer` and add it to the storage queue.
If there is no `input_transformer` then the `queue_messages` method will take a list of messages (as `messages`) and add them to the queue.
You can look at all the messages loaded into the queue using the `list_messages` method.
```python
# test.py
# load data into the queue
queue_manager.queue_messages()
# list the messages in the queue without popping them from the stack
queue_manager.list_messages()
# >>> [
# {id: 1234567890, content:"{'id': 'msg_1', 'text': 'hello world'}"},
# ...
# ]
```
### Previewing Data
You can also preview the next record. The `preview_message` method will return the next message in the queue _transformed_ without removing it from the queue.
This is designed to ensure that you're data is correct before pushing it to another service.
```python
# test.py
queue_manager.preview_message()
# >>> PREVIEW: message={...} -> transformed_message={'id': 'msg_1', 'text': 'hello world'}
```
### Retrieving The Data
Use `next_messages(count=n)` to retrieve `n` messages from the queue and process them with the `output_transformer`. By default messages are not deleted from the queue, but you can delete them by setting `delete_after=True` in the method.
If you don't add a count, then it will return all the messages in the queue.
You can also process the next message ONLY by using `next_message(delete_after=True)`. This may be helpful if you want to process the messages one at a time (if the messages have different key:values that may make data more compicated).
```python
# test.py
# get messages from the queue
QueueManager.next_messages()
# >>>
# [
# {"id": "msg_1", "text": "hello world"},
# {"id": "msg_2": "text": "hello universe"}
# ]
```
Raw data
{
"_id": null,
"home_page": "",
"name": "azqueuemanager",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": "",
"keywords": "azqueuemanager",
"author": "",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/2a/cd/486d0e167713dace436d1d2e13460d5f11ab011453368b3d9e326f63b448/azqueuemanager-0.0.6.tar.gz",
"platform": null,
"description": "# AZ Queue Manager\n\nAZ Queue Manager is a tool to manage messages in Azure Storage Queue.\n\nIt is a command line tool that can be used to send and receive messages from a queue. It can also be used to create and delete queues.\n\n\n## Getting Started\n### Install AZQueueManager and Extensions\n```bash\n# Install the base package\npip install azqueuemanager\n\n# Install the csv-input extension\npip install azqeueuemanager-csv # or some other extension\n```\n\n## Setup your script\n\nIf this is your input file `data.json`\n```json\n# data.json\n[\n {\"id\": \"msg_1\", \"text\": \"hello world\"},\n {\"id\": \"msg_2\": \"text\": \"hello universe\"}\n]\n```\nCreate your script.\n\n```python\n# test.py\nfrom azqueuemanager import QueueManager, \nfrom azqueuemanager-json import JSONTransformer\n```\nWe'll come back to the extension. For now, let's define our queue_client. This is the storage queue itself.\n\n```python\n# test.py\nCONNECTION_STRING=\"DefaultEndpointsProtocol=https;AccountName=MYACCOUNT;AccountKey=MYKEY;EndpointSuffix=core.windows.net\" #You'll need your own connection string\nqueue_name=\"MYQUEUENAME\"\n\nqueue = StorageQueue(\n connection_string,\n queue_name,\n)\n```\nNext we'll need to setup our extension. Every extension has a transformer. That transformer has modules for passing data into and out of the queue. The input transformer will take data and create one or more messages. The output transformer will take a message and create output for each record.\n\n These inputs and outputs are pretty dynamic so you will need to look at the documentation for each extension to see what the input and output looks like.\n\nThe json extension the input transformer takes `json_in_file` (or `json_data`) and (optionally) a `json_out_file`. The `transform_in` will add messages from the json content. The `transform_out` will take the messages and create a json file with all the records returned.\n\nLet's create an instance of the JSONTransformer and pass it to the QueueManager along with our queue.\n\n```python\n# test.py\n\njson_transform = JSONTransformer(\n json_in_file='data.json',\n)\n\nqueue_manager = QueueManager(\n queue=queue\n input_transformer=json_transform,\n output_transformer=json_transform,\n)\n```\n\nNow we can add or pull data to/from our Queue\n\n### Loading Data\n\nWe pass data in using the `queue_messages` method. This will use the data processed from the `input_transformer` and add it to the storage queue.\n\nIf there is no `input_transformer` then the `queue_messages` method will take a list of messages (as `messages`) and add them to the queue.\n\nYou can look at all the messages loaded into the queue using the `list_messages` method.\n\n```python\n# test.py\n# load data into the queue\nqueue_manager.queue_messages() \n\n\n# list the messages in the queue without popping them from the stack\nqueue_manager.list_messages() \n\n# >>> [\n# {id: 1234567890, content:\"{'id': 'msg_1', 'text': 'hello world'}\"},\n# ...\n# ]\n```\n\n### Previewing Data\nYou can also preview the next record. The `preview_message` method will return the next message in the queue _transformed_ without removing it from the queue.\n\nThis is designed to ensure that you're data is correct before pushing it to another service.\n\n```python \n# test.py \n\nqueue_manager.preview_message()\n\n# >>> PREVIEW: message={...} -> transformed_message={'id': 'msg_1', 'text': 'hello world'}\n```\n\n### Retrieving The Data\n\nUse `next_messages(count=n)` to retrieve `n` messages from the queue and process them with the `output_transformer`. By default messages are not deleted from the queue, but you can delete them by setting `delete_after=True` in the method. \n\nIf you don't add a count, then it will return all the messages in the queue.\n\nYou can also process the next message ONLY by using `next_message(delete_after=True)`. This may be helpful if you want to process the messages one at a time (if the messages have different key:values that may make data more compicated).\n\n```python\n# test.py\n\n# get messages from the queue\nQueueManager.next_messages() \n\n# >>>\n#\t[\n#\t\t{\"id\": \"msg_1\", \"text\": \"hello world\"},\n#\t\t{\"id\": \"msg_2\": \"text\": \"hello universe\"}\n#\t]\n\n```\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Storage Queue Input-Output Manager for Azure Storage Queue",
"version": "0.0.6",
"split_keywords": [
"azqueuemanager"
],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "205d173d5d6b94bcd44cc04ea11ddda4",
"sha256": "4d01153b82e6477d8366b2734d890be95c551703c4cbf53ada3077213615cbf6"
},
"downloads": -1,
"filename": "azqueuemanager-0.0.6-py3-none-any.whl",
"has_sig": false,
"md5_digest": "205d173d5d6b94bcd44cc04ea11ddda4",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 5338,
"upload_time": "2022-12-16T00:11:33",
"upload_time_iso_8601": "2022-12-16T00:11:33.948952Z",
"url": "https://files.pythonhosted.org/packages/bb/8a/deeb81c4ab0565cf1c952bfc0aeaf795a9eaceec5b486cfe01c5a2df04fd/azqueuemanager-0.0.6-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"md5": "ad8c4920f364eaf2e7b2f5be40a3b8b9",
"sha256": "e7c82578c0278f8fba15a2286909cd6d5b6eccf01a2668436bb04a02be9ee12e"
},
"downloads": -1,
"filename": "azqueuemanager-0.0.6.tar.gz",
"has_sig": false,
"md5_digest": "ad8c4920f364eaf2e7b2f5be40a3b8b9",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 4700,
"upload_time": "2022-12-16T00:11:35",
"upload_time_iso_8601": "2022-12-16T00:11:35.511675Z",
"url": "https://files.pythonhosted.org/packages/2a/cd/486d0e167713dace436d1d2e13460d5f11ab011453368b3d9e326f63b448/azqueuemanager-0.0.6.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2022-12-16 00:11:35",
"github": false,
"gitlab": false,
"bitbucket": false,
"lcname": "azqueuemanager"
}
