| Name | dometl JSON |
| Version |
0.0.1
JSON |
| download |
| home_page | |
| Summary | Simple python ETL tool. |
| upload_time | 2022-12-08 01:52:49 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.9 |
| license | MIT License Copyright (c) 2022 Dominik Zulovec Sajovic Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
etl
csv
postgres
python
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# DOMETL (Python ETL Tool)
Dometl is a Python ETL package.
# Process
1. Init - Initializes the database
```
dometl -t init
```
2. Stage - Moves files into staging tables
```
dometl -t stage
```
3. Live - Runs transformations to move data from staging to live tables
```
dometl -t live
```
4. Test - Runs very simple tests on the data
```
dometl -t test
```
# How to Install & Run the Package?
Run the initialization step
```
dometl -t init -cp dometl_config
# if you don't install the package
# python -c "from dometl import run_dometl; run_dometl()" -t init -cp dometl_config
```
Run the staging step
```
dometl -t stage -ep datasets\\game_data\\daily\\20221105_g.csv -tb ST_GAME -cp dometl_config
# if you don't install the package
# python -c "from dometl import run_dometl; run_dometl()" -t stage -ep datasets\\game_data\\daily\\20221105_g.csv -tb st_game -cp dometl_config
# python -c "from dometl import run_dometl; run_dometl()" -t stage -ep datasets\\game_data\\seasons -tb st_game -cp dometl_config
```
Run the live step
```
dometl -t live -tb game -cp dometl_config
# if you don't install the package
# python -c "from dometl import run_dometl; run_dometl()" -t live -tb game -cp dometl_config
```
Run the test step
```
dometl -t test -tb game -cp dometl_config
# if you don't install the package
# python -c "from dometl import run_dometl; run_dometl()" -t test -tb game -cp dometl_config
```
The simple testing is made up of testing queries which are placed into the
config.yaml folder like below
```
tests:
table_name: ["some_test.sql", "other_test.sql"]
```
Each table can have a set of test queries.
The queries need to be written in a way that they return 0 rows when the
test passes. If the query returns more than 0 rows the test will fail.
As a suggestion the rows that are returned should help find the root
cause of the failure.
## Configuration Folder
```
\folder
config.yaml # structure defined below
db_create.sql # custom file which creates and initializes the db
file1.sql # custom SQL file
file2.sql # custom SQL file
file3.sql # custom SQL file
file4.sql # custom SQL file
file5.sql # custom SQL file
```
Structure for config.yaml
```
credentials_path: "path/to/creds.yaml"
init_order: [
"db_create.sql",
"file1.sql",
"file2.sql",
]
etl:
table_name_1: "file3.sql"
table_name_2: "file4.sql"
table_name_3: "file5.sql"
```
Structure for the creds.yaml
```
db_credentials:
username: ""
password: ""
hostname: ""
port: ""
db_name: ""
```
## Bonus
Run a script with psql
```
psql -U postgres -h 127.0.0.1 -d DBNAME -f path\path\file_name.sql
```
Copy CSV into a table
```
psql -U postgres -h 127.0.0.1 -d DBNAME -c "COPY table_name FROM '/'some_name.csv' WITH (FORMAT csv)"
```
Raw data
{
"_id": null,
"home_page": "",
"name": "dometl",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": "",
"keywords": "etl,csv,postgres,python",
"author": "",
"author_email": "Dominik Zulovec Sajovic <dominik.zsajovic@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/e0/ee/8bae49a6d479626364e90ebb9e54207d38593bbaf60401b5bb549581fc70/dometl-0.0.1.tar.gz",
"platform": null,
"description": "# DOMETL (Python ETL Tool)\r\nDometl is a Python ETL package.\r\n\r\n# Process\r\n\r\n1. Init - Initializes the database\r\n```\r\ndometl -t init\r\n```\r\n2. Stage - Moves files into staging tables\r\n```\r\ndometl -t stage\r\n```\r\n3. Live - Runs transformations to move data from staging to live tables\r\n```\r\ndometl -t live\r\n```\r\n4. Test - Runs very simple tests on the data\r\n```\r\ndometl -t test\r\n```\r\n\r\n# How to Install & Run the Package?\r\n\r\nRun the initialization step\r\n```\r\ndometl -t init -cp dometl_config\r\n# if you don't install the package\r\n# python -c \"from dometl import run_dometl; run_dometl()\" -t init -cp dometl_config\r\n```\r\n\r\nRun the staging step\r\n```\r\ndometl -t stage -ep datasets\\\\game_data\\\\daily\\\\20221105_g.csv -tb ST_GAME -cp dometl_config\r\n# if you don't install the package\r\n# python -c \"from dometl import run_dometl; run_dometl()\" -t stage -ep datasets\\\\game_data\\\\daily\\\\20221105_g.csv -tb st_game -cp dometl_config\r\n# python -c \"from dometl import run_dometl; run_dometl()\" -t stage -ep datasets\\\\game_data\\\\seasons -tb st_game -cp dometl_config\r\n```\r\n\r\nRun the live step\r\n```\r\ndometl -t live -tb game -cp dometl_config\r\n# if you don't install the package\r\n# python -c \"from dometl import run_dometl; run_dometl()\" -t live -tb game -cp dometl_config\r\n```\r\n\r\nRun the test step\r\n```\r\ndometl -t test -tb game -cp dometl_config\r\n# if you don't install the package\r\n# python -c \"from dometl import run_dometl; run_dometl()\" -t test -tb game -cp dometl_config\r\n```\r\nThe simple testing is made up of testing queries which are placed into the\r\nconfig.yaml folder like below\r\n```\r\ntests:\r\n table_name: [\"some_test.sql\", \"other_test.sql\"]\r\n```\r\nEach table can have a set of test queries.\r\nThe queries need to be written in a way that they return 0 rows when the\r\ntest passes. If the query returns more than 0 rows the test will fail.\r\nAs a suggestion the rows that are returned should help find the root\r\ncause of the failure.\r\n\r\n\r\n## Configuration Folder\r\n\r\n```\r\n\\folder\r\n config.yaml # structure defined below\r\n db_create.sql # custom file which creates and initializes the db\r\n file1.sql # custom SQL file\r\n file2.sql # custom SQL file\r\n file3.sql # custom SQL file\r\n file4.sql # custom SQL file\r\n file5.sql # custom SQL file\r\n```\r\n\r\nStructure for config.yaml\r\n```\r\ncredentials_path: \"path/to/creds.yaml\"\r\n\r\ninit_order: [\r\n \"db_create.sql\",\r\n \"file1.sql\",\r\n \"file2.sql\",\r\n]\r\n\r\netl:\r\n table_name_1: \"file3.sql\" \r\n table_name_2: \"file4.sql\" \r\n table_name_3: \"file5.sql\" \r\n```\r\n\r\nStructure for the creds.yaml\r\n```\r\ndb_credentials:\r\n username: \"\"\r\n password: \"\"\r\n hostname: \"\"\r\n port: \"\"\r\n db_name: \"\"\r\n```\r\n\r\n## Bonus\r\n\r\nRun a script with psql\r\n```\r\npsql -U postgres -h 127.0.0.1 -d DBNAME -f path\\path\\file_name.sql\r\n```\r\n\r\nCopy CSV into a table\r\n```\r\npsql -U postgres -h 127.0.0.1 -d DBNAME -c \"COPY table_name FROM '/'some_name.csv' WITH (FORMAT csv)\"\r\n```\r\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2022 Dominik Zulovec Sajovic Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
"summary": "Simple python ETL tool.",
"version": "0.0.1",
"split_keywords": [
"etl",
"csv",
"postgres",
"python"
],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "421a808d055e217068bf9414cc173bc5",
"sha256": "5254b02ef484d9bd4072348b93c0c14091128e16753298b497510a92dc809d05"
},
"downloads": -1,
"filename": "dometl-0.0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "421a808d055e217068bf9414cc173bc5",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 8958,
"upload_time": "2022-12-08T01:52:47",
"upload_time_iso_8601": "2022-12-08T01:52:47.262004Z",
"url": "https://files.pythonhosted.org/packages/48/92/002c5b8dcdd78818d4076effa622042cebf84dcbf514982db461cda35f09/dometl-0.0.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"md5": "be19f4987a2242b4c8621b5113a4e1b4",
"sha256": "4ab406e5e46a53a2b93890bb9252e810093663ba070ad237a13172582ae4dbc5"
},
"downloads": -1,
"filename": "dometl-0.0.1.tar.gz",
"has_sig": false,
"md5_digest": "be19f4987a2242b4c8621b5113a4e1b4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 8062,
"upload_time": "2022-12-08T01:52:49",
"upload_time_iso_8601": "2022-12-08T01:52:49.566353Z",
"url": "https://files.pythonhosted.org/packages/e0/ee/8bae49a6d479626364e90ebb9e54207d38593bbaf60401b5bb549581fc70/dometl-0.0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2022-12-08 01:52:49",
"github": false,
"gitlab": false,
"bitbucket": false,
"lcname": "dometl"
}
