| Name | atk-training-rin-q-basic JSON |
| Version |
0.1.0
JSON |
| download |
| home_page | |
| Summary | |
| upload_time | 2023-07-21 08:00:20 |
| maintainer | |
| docs_url | None |
| author | Your Name |
| requires_python | >=3.10,<4.0 |
| license | |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<h1>Assignment</h1>
Design a system with producer and a consumer.<br>
Producer produces some entities independent of consumer<br>
Consumer consumes the entities, producing new entities.<br>
Click <a href="#usage">here</a> for usage/commands.
<hr>
<h1>Solution</h1>
Here, let the job be a string of length 10.<br>
Using async and await, functions can be turned into coroutines to be passed to a ```execute_function``` command.
```mermaid
graph TD;
A["PersistentQInterface"]-- child class ---B["Producer(s)"];
A-- child class ---C["Consumer(s)"]
B-- submit jobs -->D["PersistentQSQlite"]
D-- process jobs -->C
```
<h3>System</h3>
<p>
Consists of abstract class (<a href="https://docs.python.org/3/library/abc.html">ABC</a>) <b>PersistentQInterface</b>, children <b>Producer</b>, <b>Consumer</b> and <b>Cleaner</b>.<br>
Also consists of class <b>PersistentQSQLite</b> that performs SQL queries on a database from Python.<br>
<br>
On being called, <b>Producer</b> generates a file name (and file), and submits this file name as a job. <b>PersistentQSQLite</b> adds this name to a queue/SQL database. Occurs every <b>5</b> seconds<br>
<b>Consumer</b> reads this queue and processes them, generating a new file with the processed content. Occurs every <b>7-15</b> seconds.<br>
If after <i>n</i> tries a file is not able to be processed by <b>Consumer</b>, it gets renamed with a <i>.failed</i> extension. <b> Cleaner</b> checks for .failed files every <b>30</b> seconds and deletes them.<br>
Wait times for Producers, Consumers and Cleaners are independent amongst themselves and each other.
</p>
<hr>
<h1 id="usage">Usage</h1>
```bash
atk-training-rin-q-basic --help
```
To run the web app, try
```bash
atk-training-rin-q-basic web-app
```
The program will open in ```localhost:8000```. <i>producer</i> and <i>consumer</i> will be running in the background -- any logs/output produced (primarily by consumer) will be visible in the terminal. <br>
<b>NOTE: All files produced will be in the CURRENT DIRECTORY BY DEFAULT. To set your custom path, modify the path in config.yaml.</b>
<h3> Setting up your own config</h3>
Parameters can be customised by giving the command path to a config.yaml file.<br>
The parameters along with their default values are as follows. If any parameter is missing from the file, the default value will be used.
```yaml
consumer:
n_time: 3600
n_tries: 3
general:
primary_path: current/directory
n_prod: 1
n_cons: 1
```
```n_time``` : Seconds a consumer is allowed to spend on one job before skipping
```n_tries``` : Number of time a consumer is allowed to try a job (in case of errors) before skipping
```primary_path```: Path where files are created
```n_prod``` : Number of producers
```n_cons``` : Number of consumers
Raw data
{
"_id": null,
"home_page": "",
"name": "atk-training-rin-q-basic",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.10,<4.0",
"maintainer_email": "",
"keywords": "",
"author": "Your Name",
"author_email": "you@example.com",
"download_url": "https://files.pythonhosted.org/packages/96/00/f7cb0dbbe3c7bccdf8d3ce9861501f34fb30ebd8cc5130cd690f5831b948/atk_training_rin_q_basic-0.1.0.tar.gz",
"platform": null,
"description": "<h1>Assignment</h1>\nDesign a system with producer and a consumer.<br>\nProducer produces some entities independent of consumer<br>\nConsumer consumes the entities, producing new entities.<br>\nClick <a href=\"#usage\">here</a> for usage/commands.\n<hr>\n<h1>Solution</h1>\nHere, let the job be a string of length 10.<br>\n\nUsing async and await, functions can be turned into coroutines to be passed to a ```execute_function``` command.\n\n```mermaid\ngraph TD;\n A[\"PersistentQInterface\"]-- child class ---B[\"Producer(s)\"];\n A-- child class ---C[\"Consumer(s)\"]\n B-- submit jobs -->D[\"PersistentQSQlite\"]\n D-- process jobs -->C\n```\n\n\n<h3>System</h3>\n<p>\nConsists of abstract class (<a href=\"https://docs.python.org/3/library/abc.html\">ABC</a>) <b>PersistentQInterface</b>, children <b>Producer</b>, <b>Consumer</b> and <b>Cleaner</b>.<br>\nAlso consists of class <b>PersistentQSQLite</b> that performs SQL queries on a database from Python.<br>\n<br>\nOn being called, <b>Producer</b> generates a file name (and file), and submits this file name as a job. <b>PersistentQSQLite</b> adds this name to a queue/SQL database. Occurs every <b>5</b> seconds<br>\n<b>Consumer</b> reads this queue and processes them, generating a new file with the processed content. Occurs every <b>7-15</b> seconds.<br>\nIf after <i>n</i> tries a file is not able to be processed by <b>Consumer</b>, it gets renamed with a <i>.failed</i> extension. <b> Cleaner</b> checks for .failed files every <b>30</b> seconds and deletes them.<br>\nWait times for Producers, Consumers and Cleaners are independent amongst themselves and each other.\n</p>\n<hr>\n<h1 id=\"usage\">Usage</h1>\n\n```bash\natk-training-rin-q-basic --help\n```\nTo run the web app, try\n```bash\natk-training-rin-q-basic web-app\n```\nThe program will open in ```localhost:8000```. <i>producer</i> and <i>consumer</i> will be running in the background -- any logs/output produced (primarily by consumer) will be visible in the terminal. <br>\n<b>NOTE: All files produced will be in the CURRENT DIRECTORY BY DEFAULT. To set your custom path, modify the path in config.yaml.</b>\n\n<h3> Setting up your own config</h3>\nParameters can be customised by giving the command path to a config.yaml file.<br>\nThe parameters along with their default values are as follows. If any parameter is missing from the file, the default value will be used.\n\n```yaml\nconsumer:\n n_time: 3600\n n_tries: 3\ngeneral:\n primary_path: current/directory\n n_prod: 1\n n_cons: 1\n```\n```n_time``` : Seconds a consumer is allowed to spend on one job before skipping\n```n_tries``` : Number of time a consumer is allowed to try a job (in case of errors) before skipping\n```primary_path```: Path where files are created\n```n_prod``` : Number of producers\n```n_cons``` : Number of consumers\n",
"bugtrack_url": null,
"license": "",
"summary": "",
"version": "0.1.0",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "cd0cd2faf3bda892171ae60ae523d837415fd36aa418a1685dbf4c356c5e36fe",
"md5": "2f6dba671020b8bb720dc37e29d5d20b",
"sha256": "fadd24a1de23e5c6ccc68e5f79112851c90f12cae571bf7c99cbc63caf90a902"
},
"downloads": -1,
"filename": "atk_training_rin_q_basic-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "2f6dba671020b8bb720dc37e29d5d20b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10,<4.0",
"size": 7567,
"upload_time": "2023-07-21T08:00:19",
"upload_time_iso_8601": "2023-07-21T08:00:19.469020Z",
"url": "https://files.pythonhosted.org/packages/cd/0c/d2faf3bda892171ae60ae523d837415fd36aa418a1685dbf4c356c5e36fe/atk_training_rin_q_basic-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9600f7cb0dbbe3c7bccdf8d3ce9861501f34fb30ebd8cc5130cd690f5831b948",
"md5": "3dcb774f347f6379326b2e0011f415ab",
"sha256": "956d0b12a37ce0686de00234ff5599d29b6781bb815c85925f715b71b64bbb65"
},
"downloads": -1,
"filename": "atk_training_rin_q_basic-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "3dcb774f347f6379326b2e0011f415ab",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10,<4.0",
"size": 5313,
"upload_time": "2023-07-21T08:00:20",
"upload_time_iso_8601": "2023-07-21T08:00:20.821356Z",
"url": "https://files.pythonhosted.org/packages/96/00/f7cb0dbbe3c7bccdf8d3ce9861501f34fb30ebd8cc5130cd690f5831b948/atk_training_rin_q_basic-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-21 08:00:20",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "atk-training-rin-q-basic"
}
