| Name | omnipy-examples JSON |
| Version |
0.7.2
JSON |
| download |
| home_page | None |
| Summary | None |
| upload_time | 2024-09-16 22:35:20 |
| maintainer | None |
| docs_url | None |
| author | Sveinung Gundersen |
| requires_python | <3.13,>=3.10 |
| license | None |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# omnipy-examples
Example projects that that makes use of the [omnipy](https://pypi.org/project/omnipy/) package for
type-driven, scalable and interoperable data wrangling!
## Main installation instructions
- Install:
- `pip install omnipy-examples`
- Run example scripts:
- Example: `omnipy-examples isajson`
- For help on the command line interface: `omnipy-examples --help`
- For help on a particular example: `omnipy-examples isajson --help`
### Output of flow runs
The output will by default appear in the `data` directory, with a timestamp.
- It is recommended to install a file viewer that are capable of browsing tar.gz files.
For instance, the "File Expander" plugin in PyCharm is excellent for this.
- To unpack the compressed files of a run on the command line
(just make sure to replace the datetime string from this example):
```
for f in $(ls data/2023_02_03-12_51_51/*.tar.gz); do mkdir ${f%.tar.gz}; tar xfzv $f -C ${f%.tar.gz}; done
```
### Run with the Prefect engine
Omnipy is integrated with the powerful [Prefect](https://prefect.io) data flow orchestration library.
- To run an local example using the `prefect` engine, e.g.:
- `omnipy-examples --engine prefect isajson`
- After completion of some runs, you can check the flow logs and orchestration options in the Prefect UI:
- `prefect server start`
To set up a kubernetes-based deployment on our NIRD test setup, run e.g.:
- `prefect config set PREFECT_API_URL=https://prefect.fairtracks.sigma2.no/api`
- `prefect deploy -n isajson`
The configuration of this job is found in the file `prefect.yaml`.
More info on Prefect configuration will come soon...
## Development setup
- Install Poetry:
- `curl -sSL https://install.python-poetry.org | python3 -`
- Install dependencies:
- `poetry install --with dev`
- Update all dependencies:
- `poetry update`
- Update single dependency, e.g.:
- `poetry update omnipy`
- If a dependency is not updated to the latest version available on Pypi, you might need to clear
the pip cache of poetry:
- `poetry cache clear pypi --all`
### For mypy support in PyCharm
- In PyCharm, install "Mypy" plugin (not "Mypy (Official)")
- `which mypy` to get path to mypy binary
- In the PyCharm settings for the mypy plugin:
- Select the mypy binary
- Select `pyproject.toml` as the mypy config file
### For automatic formatting and linting
I have added my typical setup for automatic formatting and linting. The main alternative is to use black, which is easier to set up, but it does
not have as many options. I am not fully happy with my config, but I at least like it better than black.
- In PyCharm -> File Watchers:
- Click arrow down icon
- Select `pycharm-file-watchers.xml`
```
[OMNIPY] Tue Aug 27 15:39:30 2024 - INFO: Finished running "task-transpose-dicts-2-lists-annoying-wombat"! [omnipy.log.registry.RunStateRegistry]
[OMNIPY] Tue Aug 27 15:39:30 2024 - INFO: Writing dataset as a gzipped tarpack to "/Users/sveinugu/PycharmProjects/omnipy_examples/outputs/2024_08_27-15_39_30/02_task_transpose_dicts_2_lists.tar.gz" [omnipy.compute.task.TaskWithMixins]
[OMNIPY] Tue Aug 27 15:39:30 2024 - INFO: Finished running "func-flow-transpose-dicts-of-lists-of-dicts-2-lists-of-dicts-banana-antelope"! [omnipy.log.registry.RunStateRegistry]
[OMNIPY] Tue Aug 27 15:39:30 2024 - INFO: Writing dataset as a gzipped tarpack to "/Users/sveinugu/PycharmProjects/omnipy_examples/outputs/2024_08_27-15_39_30/03_func_flow_transpose_dicts_of_lists_of_dicts_2_lists_of_dicts.tar.gz" [omnipy.compute.flow.FuncFlowWithMixins]
[OMNIPY] Tue Aug 27 15:39:30 2024 - INFO: Initialized "func-flow-flatten-nested-json-incredible-bobcat" [omnipy.log.registry.RunStateRegistry]
[OMNIPY] Tue Aug 27 15:39:30 2024 - INFO: Started running "func-flow-flatten-nested-json-incredible-bobcat"... [omnipy.log.registry.RunStateRegistry]
[OMNIPY] Tue Aug 27 15:39:30 2024 - INFO: Initialized "task-flatten-outer-level-of-all-data-files-axiomatic-ape" [omnipy.log.registry.RunStateRegistry]
[OMNIPY] Tue Aug 27 15:39:30 2024 - INFO: Started running "task-flatten-outer-level-of-all-data-files-axiomatic-ape"... [omnipy.log.registry.RunStateRegistry]
[OMNIPY] Tue Aug 27 15:39:30 2024 - INFO: Finished running "task-flatten-outer-level-of-all-data-files-axiomatic-ape"! [omnipy.log.registry.RunStateRegistry]
```
Raw data
{
"_id": null,
"home_page": null,
"name": "omnipy-examples",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.13,>=3.10",
"maintainer_email": null,
"keywords": null,
"author": "Sveinung Gundersen",
"author_email": "sveinugu@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/e5/7a/0136351b38386bfde9078f5a2b5b6241f7ac0da2354288517ebcded0791a/omnipy_examples-0.7.2.tar.gz",
"platform": null,
"description": "# omnipy-examples\n\nExample projects that that makes use of the [omnipy](https://pypi.org/project/omnipy/) package for \ntype-driven, scalable and interoperable data wrangling!\n\n## Main installation instructions\n\n- Install:\n - `pip install omnipy-examples`\n- Run example scripts:\n - Example: `omnipy-examples isajson`\n - For help on the command line interface: `omnipy-examples --help`\n - For help on a particular example: `omnipy-examples isajson --help`\n\n### Output of flow runs\n\nThe output will by default appear in the `data` directory, with a timestamp. \n\n - It is recommended to install a file viewer that are capable of browsing tar.gz files. \n For instance, the \"File Expander\" plugin in PyCharm is excellent for this.\n - To unpack the compressed files of a run on the command line \n (just make sure to replace the datetime string from this example): \n\n```\nfor f in $(ls data/2023_02_03-12_51_51/*.tar.gz); do mkdir ${f%.tar.gz}; tar xfzv $f -C ${f%.tar.gz}; done\n```\n \n### Run with the Prefect engine\n\nOmnipy is integrated with the powerful [Prefect](https://prefect.io) data flow orchestration library.\n\n- To run an local example using the `prefect` engine, e.g.:\n - `omnipy-examples --engine prefect isajson`\n- After completion of some runs, you can check the flow logs and orchestration options in the Prefect UI:\n - `prefect server start`\n\nTo set up a kubernetes-based deployment on our NIRD test setup, run e.g.:\n\n- `prefect config set PREFECT_API_URL=https://prefect.fairtracks.sigma2.no/api`\n- `prefect deploy -n isajson`\n\nThe configuration of this job is found in the file `prefect.yaml`. \n\nMore info on Prefect configuration will come soon...\n\n## Development setup\n\n- Install Poetry:\n - `curl -sSL https://install.python-poetry.org | python3 -`\n\n- Install dependencies:\n - `poetry install --with dev`\n\n- Update all dependencies:\n - `poetry update`\n\n- Update single dependency, e.g.:\n - `poetry update omnipy`\n\n- If a dependency is not updated to the latest version available on Pypi, you might need to clear\n the pip cache of poetry:\n - `poetry cache clear pypi --all`\n\n### For mypy support in PyCharm\n\n- In PyCharm, install \"Mypy\" plugin (not \"Mypy (Official)\")\n - `which mypy` to get path to mypy binary\n - In the PyCharm settings for the mypy plugin:\n - Select the mypy binary \n - Select `pyproject.toml` as the mypy config file\n\n### For automatic formatting and linting\n\nI have added my typical setup for automatic formatting and linting. The main alternative is to use black, which is easier to set up, but it does \nnot have as many options. I am not fully happy with my config, but I at least like it better than black. \n\n- In PyCharm -> File Watchers:\n - Click arrow down icon\n - Select `pycharm-file-watchers.xml`\n\n\n```\n[OMNIPY] Tue Aug 27 15:39:30 2024 - INFO: Finished running \"task-transpose-dicts-2-lists-annoying-wombat\"! [omnipy.log.registry.RunStateRegistry]\n[OMNIPY] Tue Aug 27 15:39:30 2024 - INFO: Writing dataset as a gzipped tarpack to \"/Users/sveinugu/PycharmProjects/omnipy_examples/outputs/2024_08_27-15_39_30/02_task_transpose_dicts_2_lists.tar.gz\" [omnipy.compute.task.TaskWithMixins]\n[OMNIPY] Tue Aug 27 15:39:30 2024 - INFO: Finished running \"func-flow-transpose-dicts-of-lists-of-dicts-2-lists-of-dicts-banana-antelope\"! [omnipy.log.registry.RunStateRegistry]\n[OMNIPY] Tue Aug 27 15:39:30 2024 - INFO: Writing dataset as a gzipped tarpack to \"/Users/sveinugu/PycharmProjects/omnipy_examples/outputs/2024_08_27-15_39_30/03_func_flow_transpose_dicts_of_lists_of_dicts_2_lists_of_dicts.tar.gz\" [omnipy.compute.flow.FuncFlowWithMixins]\n[OMNIPY] Tue Aug 27 15:39:30 2024 - INFO: Initialized \"func-flow-flatten-nested-json-incredible-bobcat\" [omnipy.log.registry.RunStateRegistry]\n[OMNIPY] Tue Aug 27 15:39:30 2024 - INFO: Started running \"func-flow-flatten-nested-json-incredible-bobcat\"... [omnipy.log.registry.RunStateRegistry]\n[OMNIPY] Tue Aug 27 15:39:30 2024 - INFO: Initialized \"task-flatten-outer-level-of-all-data-files-axiomatic-ape\" [omnipy.log.registry.RunStateRegistry]\n[OMNIPY] Tue Aug 27 15:39:30 2024 - INFO: Started running \"task-flatten-outer-level-of-all-data-files-axiomatic-ape\"... [omnipy.log.registry.RunStateRegistry]\n[OMNIPY] Tue Aug 27 15:39:30 2024 - INFO: Finished running \"task-flatten-outer-level-of-all-data-files-axiomatic-ape\"! [omnipy.log.registry.RunStateRegistry]\n```",
"bugtrack_url": null,
"license": null,
"summary": null,
"version": "0.7.2",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "6624442853966d04ea503fcd30d532736287d8ad6ceb6bc8e9ea67024ea41f29",
"md5": "73b843f41e52b9358c7f3012c42a21e9",
"sha256": "78f747febb8de97cb22dcee2ec167a0a479a9d0000b3c948258dd57b6e1fddde"
},
"downloads": -1,
"filename": "omnipy_examples-0.7.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "73b843f41e52b9358c7f3012c42a21e9",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.13,>=3.10",
"size": 10234723,
"upload_time": "2024-09-16T22:34:59",
"upload_time_iso_8601": "2024-09-16T22:34:59.817420Z",
"url": "https://files.pythonhosted.org/packages/66/24/442853966d04ea503fcd30d532736287d8ad6ceb6bc8e9ea67024ea41f29/omnipy_examples-0.7.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e57a0136351b38386bfde9078f5a2b5b6241f7ac0da2354288517ebcded0791a",
"md5": "bc58f96bca45c0678c4400a843e22e47",
"sha256": "ef7e4f640d0cfc78885a9a0ef9c4593a2777102620625e7e22a70aba32383186"
},
"downloads": -1,
"filename": "omnipy_examples-0.7.2.tar.gz",
"has_sig": false,
"md5_digest": "bc58f96bca45c0678c4400a843e22e47",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.13,>=3.10",
"size": 10201024,
"upload_time": "2024-09-16T22:35:20",
"upload_time_iso_8601": "2024-09-16T22:35:20.409945Z",
"url": "https://files.pythonhosted.org/packages/e5/7a/0136351b38386bfde9078f5a2b5b6241f7ac0da2354288517ebcded0791a/omnipy_examples-0.7.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-16 22:35:20",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "omnipy-examples"
}
