# About
This is a compilation of my data pipeline scripts written in Python.
### Conventions
Each pipeline function is an executable python file that accepts flags to modify the specific configurations of the pipeline (i.e. MSSQL DB Name, GCS Bucket Name).
When loading data from a third-party source you can set the temporary destination of the data to the `data/` folder. After the data has been successfully ingested remove the file from the data folder.
### Folder Structure
- config/
- contains any specific configurations that need to be modified within the Docker container
- data/
- a temporary landing zone for any data that is ingested from a third-party source
- functions/
- contains all pipeline functions
- functions/utils/
- contains all reusable code and can be organized futher as either a Source (where data is pulled from), or a Target (where data is placed)
# Setup
[//]: # (TODO: Review environment and dependency management best practices when using Hatch)
1. Create and activate a python virtual environment.
```
python3 -m venv venv
```
```
source /venv/bin/activate
```
2. Install python dependencies
```
pip install -r requirements.txt
```
Raw data
{
"_id": null,
"home_page": null,
"name": "pypeline-functions",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "ELT, data pipelines, dlt, pypeline",
"author": null,
"author_email": "Miguel Habana <mighabana@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/20/4f/40020cb662201e3335e3d9b5c8c6efd059b4099b865634fbee600bd5021b/pypeline_functions-0.1.1.tar.gz",
"platform": null,
"description": "# About\n\nThis is a compilation of my data pipeline scripts written in Python.\n\n### Conventions\n\nEach pipeline function is an executable python file that accepts flags to modify the specific configurations of the pipeline (i.e. MSSQL DB Name, GCS Bucket Name).\n\nWhen loading data from a third-party source you can set the temporary destination of the data to the `data/` folder. After the data has been successfully ingested remove the file from the data folder.\n\n### Folder Structure\n\n- config/\n - contains any specific configurations that need to be modified within the Docker container\n- data/\n - a temporary landing zone for any data that is ingested from a third-party source\n- functions/\n - contains all pipeline functions\n- functions/utils/\n - contains all reusable code and can be organized futher as either a Source (where data is pulled from), or a Target (where data is placed)\n\n# Setup\n\n[//]: # (TODO: Review environment and dependency management best practices when using Hatch)\n\n1. Create and activate a python virtual environment.\n\n```\npython3 -m venv venv\n```\n\n```\nsource /venv/bin/activate\n```\n\n\n2. Install python dependencies\n\n```\npip install -r requirements.txt\n```",
"bugtrack_url": null,
"license": "MIT",
"summary": "A compiliation of data pipeline scripts in Python",
"version": "0.1.1",
"project_urls": null,
"split_keywords": [
"elt",
" data pipelines",
" dlt",
" pypeline"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "47c39dafe3f3fca37db0fcf3d946acb7745835908412f66bef8fd4e6df29b4d9",
"md5": "e0f6e97af0a2bf40cd0d3d0c992a3787",
"sha256": "192aa326d1bd3b0611045fee630b237b072ee744ef1f91dcab6d26a3918d772c"
},
"downloads": -1,
"filename": "pypeline_functions-0.1.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e0f6e97af0a2bf40cd0d3d0c992a3787",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 17924,
"upload_time": "2024-11-29T23:51:07",
"upload_time_iso_8601": "2024-11-29T23:51:07.693207Z",
"url": "https://files.pythonhosted.org/packages/47/c3/9dafe3f3fca37db0fcf3d946acb7745835908412f66bef8fd4e6df29b4d9/pypeline_functions-0.1.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "204f40020cb662201e3335e3d9b5c8c6efd059b4099b865634fbee600bd5021b",
"md5": "b0c4734818905e2abe8d84a4be6e639b",
"sha256": "10c66cc8f01d84e19cf9bf76c7d4495c4c0912460742701ebcaf994572caacec"
},
"downloads": -1,
"filename": "pypeline_functions-0.1.1.tar.gz",
"has_sig": false,
"md5_digest": "b0c4734818905e2abe8d84a4be6e639b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 12416,
"upload_time": "2024-11-29T23:51:26",
"upload_time_iso_8601": "2024-11-29T23:51:26.553624Z",
"url": "https://files.pythonhosted.org/packages/20/4f/40020cb662201e3335e3d9b5c8c6efd059b4099b865634fbee600bd5021b/pypeline_functions-0.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-29 23:51:26",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "pypeline-functions"
}
JSON