| Name | etl-utilities JSON |
| Version |
0.9.9
JSON |
| download |
| home_page | None |
| Summary | This repository provides a collection of utility functions and classes for data cleaning, SQL query generation, and data analysis. The code is written in Python and uses libraries such as `pandas`, `numpy`, and `dateutil`. |
| upload_time | 2024-12-11 17:38:55 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.8 |
| license | MIT |
| keywords |
cleaning
database
dataframe
etl
etl-utilities
pandas
utilities
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Project Documentation
## Table of Contents
1. [Overview](#overview)
2. [Classes](#classes)
- [Connector](#connector)
- [Loader](#loader)
- [MySqlLoader](#mysqlloader)
- [MsSqlLoader](#mssqlloader)
- [Parser](#parser)
- [Cleaner](#cleaner)
- [Creator](#creator)
- [Analyzer](#analyzer)
- [Validator](#validator)
- [MsSqlUpdater](#mssqlupdater)
3. [Logging](#logging)
4. [Additional Utilities](#additional-utilities)
## Overview
This project provides a comprehensive Data ETL \(Extract, Transform, Load\) and data manipulation framework using Python. It integrates with databases using SQLAlchemy and provides tools for data parsing, cleaning, loading, validating, and more. The project is structured with classes that encapsulate different functionalities.
## Classes
### Connector
The `Connector` class handles creating connections to various types of databases \(MSSQL, PostgreSQL, MySQL\) using SQLAlchemy. It provides static methods for obtaining both trusted and user connections.
**Key Methods:**
- `get_mssql_trusted_connection`
- `get_mssql_user_connection`
- `get_postgres_user_connection`
- `get_mysql_user_connection`
- Instance methods for returning database connections based on stored configuration.
### Loader
The `Loader` class is responsible for loading data from a Pandas DataFrame into a database. It manages the insertion process, ensuring data is inserted efficiently and effectively with the use of SQLAlchemy and custom logging.
### MySqlLoader
A slight extension of the `Loader` class specifically for MySQL databases. It provides overrides to manage MySQL-specific data types and query formatting.
### MsSqlLoader
A specialized loader for loading data into MSSQL databases with additional functionalities like fast insertions using bulk methods.
### Parser
The `Parser` class consists of a series of static methods dedicated to parsing various data types—boolean, float, date, and integer. These methods are essential for data type conversion and consistency across the application.
### Cleaner
The `Cleaner` class provides methods for sanitizing and formatting data in a DataFrame. It includes functions for setting column name casing conventions, cleaning various types of data, and preparing data for reliable analysis and insertion.
### Creator
This class deals with generating SQL `CREATE TABLE` statements for different databases like MSSQL and MariaDB. The query generation considers data types deduced from DataFrame content.
### Analyzer
The `Analyzer` class assesses DataFrame characteristics and helps identify unique columns, column pairs, empty columns, and more. It aids in generating metadata for data types, which is crucial for creating or validating schemas.
### Validator
The `Validator` class ensures DataFrame compatibility with the target database table structure by checking for extra columns, validating data types, and ensuring that no data truncation will occur during upload.
### MsSqlUpdater
A class designed for constructing SQL statements for operations like mergers, updates, inserts, and appends to manage data transitions between tables efficiently.
## Logging
The project uses a singleton `Logger` class with colored output format for console logging. This helps in debugging and understanding the flow by logging messages at various severity levels.
## Additional Utilities
- **Parsing and Cleaning Functions:** Utility functions for parsing and cleaning various data types.
- **Standardization:** A set of utility functions to standardize and clean DataFrame column names and content.
Raw data
{
"_id": null,
"home_page": null,
"name": "etl-utilities",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "cleaning, database, dataframe, etl, etl-utilities, pandas, utilities",
"author": null,
"author_email": "Jesse <magicjedi90@aim.com>",
"download_url": "https://files.pythonhosted.org/packages/47/5d/be6c3f87da4613b931b8aca5e56c4e14883cd461dca6bfdbaf437476d611/etl_utilities-0.9.9.tar.gz",
"platform": null,
"description": "# Project Documentation\n\n## Table of Contents\n\n1. [Overview](#overview)\n2. [Classes](#classes)\n - [Connector](#connector)\n - [Loader](#loader)\n - [MySqlLoader](#mysqlloader)\n - [MsSqlLoader](#mssqlloader)\n - [Parser](#parser)\n - [Cleaner](#cleaner)\n - [Creator](#creator)\n - [Analyzer](#analyzer)\n - [Validator](#validator)\n - [MsSqlUpdater](#mssqlupdater)\n3. [Logging](#logging)\n4. [Additional Utilities](#additional-utilities)\n\n## Overview\n\nThis project provides a comprehensive Data ETL \\(Extract, Transform, Load\\) and data manipulation framework using Python. It integrates with databases using SQLAlchemy and provides tools for data parsing, cleaning, loading, validating, and more. The project is structured with classes that encapsulate different functionalities.\n\n## Classes\n\n### Connector\n\nThe `Connector` class handles creating connections to various types of databases \\(MSSQL, PostgreSQL, MySQL\\) using SQLAlchemy. It provides static methods for obtaining both trusted and user connections.\n\n**Key Methods:**\n- `get_mssql_trusted_connection`\n- `get_mssql_user_connection`\n- `get_postgres_user_connection`\n- `get_mysql_user_connection`\n- Instance methods for returning database connections based on stored configuration.\n\n### Loader\n\nThe `Loader` class is responsible for loading data from a Pandas DataFrame into a database. It manages the insertion process, ensuring data is inserted efficiently and effectively with the use of SQLAlchemy and custom logging.\n\n### MySqlLoader\n\nA slight extension of the `Loader` class specifically for MySQL databases. It provides overrides to manage MySQL-specific data types and query formatting.\n\n### MsSqlLoader\n\nA specialized loader for loading data into MSSQL databases with additional functionalities like fast insertions using bulk methods.\n\n### Parser\n\nThe `Parser` class consists of a series of static methods dedicated to parsing various data types\u2014boolean, float, date, and integer. These methods are essential for data type conversion and consistency across the application.\n\n### Cleaner\n\nThe `Cleaner` class provides methods for sanitizing and formatting data in a DataFrame. It includes functions for setting column name casing conventions, cleaning various types of data, and preparing data for reliable analysis and insertion.\n\n### Creator\n\nThis class deals with generating SQL `CREATE TABLE` statements for different databases like MSSQL and MariaDB. The query generation considers data types deduced from DataFrame content.\n\n### Analyzer\n\nThe `Analyzer` class assesses DataFrame characteristics and helps identify unique columns, column pairs, empty columns, and more. It aids in generating metadata for data types, which is crucial for creating or validating schemas.\n\n### Validator\n\nThe `Validator` class ensures DataFrame compatibility with the target database table structure by checking for extra columns, validating data types, and ensuring that no data truncation will occur during upload.\n\n### MsSqlUpdater\n\nA class designed for constructing SQL statements for operations like mergers, updates, inserts, and appends to manage data transitions between tables efficiently.\n\n## Logging\n\nThe project uses a singleton `Logger` class with colored output format for console logging. This helps in debugging and understanding the flow by logging messages at various severity levels.\n\n## Additional Utilities\n\n- **Parsing and Cleaning Functions:** Utility functions for parsing and cleaning various data types.\n- **Standardization:** A set of utility functions to standardize and clean DataFrame column names and content.",
"bugtrack_url": null,
"license": "MIT",
"summary": "This repository provides a collection of utility functions and classes for data cleaning, SQL query generation, and data analysis. The code is written in Python and uses libraries such as `pandas`, `numpy`, and `dateutil`.",
"version": "0.9.9",
"project_urls": {
"Documentation": "https://github.com/magicjedi90/etl_utilities#readme",
"Issues": "https://github.com/magicjedi90/etl_utilities/issues",
"Source": "https://github.com/magicjedi90/etl_utilities"
},
"split_keywords": [
"cleaning",
" database",
" dataframe",
" etl",
" etl-utilities",
" pandas",
" utilities"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "74b7cf802f07a3a10e8bb67d7aede22ac49a6a5250aefd60db110b6fc2ec208e",
"md5": "3afc5c2a333635fb5eadb2b9a634a164",
"sha256": "cc1a4061772bddedef5669352c936061aab371d2e912e9796a2bea72d5a3b825"
},
"downloads": -1,
"filename": "etl_utilities-0.9.9-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3afc5c2a333635fb5eadb2b9a634a164",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 22649,
"upload_time": "2024-12-11T17:38:54",
"upload_time_iso_8601": "2024-12-11T17:38:54.042967Z",
"url": "https://files.pythonhosted.org/packages/74/b7/cf802f07a3a10e8bb67d7aede22ac49a6a5250aefd60db110b6fc2ec208e/etl_utilities-0.9.9-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "475dbe6c3f87da4613b931b8aca5e56c4e14883cd461dca6bfdbaf437476d611",
"md5": "0661a5736b2da68c52ed12532b4dbbb1",
"sha256": "f9676e59b0ffc099f7d20e9228bac7db373f34ec9a57306fc2c649516d6136f5"
},
"downloads": -1,
"filename": "etl_utilities-0.9.9.tar.gz",
"has_sig": false,
"md5_digest": "0661a5736b2da68c52ed12532b4dbbb1",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 20361,
"upload_time": "2024-12-11T17:38:55",
"upload_time_iso_8601": "2024-12-11T17:38:55.617061Z",
"url": "https://files.pythonhosted.org/packages/47/5d/be6c3f87da4613b931b8aca5e56c4e14883cd461dca6bfdbaf437476d611/etl_utilities-0.9.9.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-11 17:38:55",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "magicjedi90",
"github_project": "etl_utilities#readme",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "etl-utilities"
}
