# connector-factory
[](https://opensource.org/licenses/MIT)
[](https://pypi.org/project/connector-factory)
Connector factory is used to manage/create connection with execute queries using the connection.
The concept of having single source to connect various sources and perform operations on top of it.
User need not to worry on the crafting the connection string and to identify the methods for the basic operations.
Connector factory supports DML / DDL executions and have support of Pandas DataFrame to create or replace existing tables.
Connector factory is wrapper on various sources and for database source it uses sqlalchemy for crafting the connection and supports below databases:
```bash
* Sqlite3
* PostgreSQl
* BigQuery (Upcomming)
* Snowflake
* MariaDB
* MySQL
* Redshift
* Salesforce
* S3Select
```
* Note: Only select operations are supported for Salesforce and S3Select.
Connector factory can be enhanced for all the sqlalchemy supported database.
Connector factory also provide the facility to create the connections for Salesforce
using simple-saleforce python package. Only Select query is supported in Salesforce. For advance usage it exposes the Salesforce object (simple-saleforce python package) which can be used as per the requirements. Refer [simple-saleforce](https://pypi.org/project/simple-salesforce/) for more detail.
## Getting Started
```bash
pip install connector-factory
```
By default connector-factory is installed with only SQlite3 support only. For other support it can be installed using combinations of below flags
* postgres
* redshift
* snowflake
* mysql
* mariadb
* salesforce
* s3select
* bigquery (upcomming)
* all (For all supported types)
### Using connector-factory
-----
```python
from connector_factory import ConnectorFactory
import tempfile
temp_dir = tempfile.gettempdir()
config = {
"path": temp_dir,
"database": "site.db"
}
db = ConnectorFactory(connector_type="sqlite", config=config)
db.create_session()
db.execute_sql(sql="create table test (id int PRIMARY KEY)")
db.execute_sql(sql="insert into test values (1)")
db.execute_sql(sql="insert into test values (2)")
rows = db.execute_sql(sql="select * from test")
if rows:
print(rows)
df = db.get_df(sql="select * from test")
print(df)
db.execute_df(panda_df=df, table_name="copy_test", exist_action="replace")
# db.execute_df(panda_df=df, table_name=copy_test, exist_action="replace", chunk_size=100)
db.execute_sql(sql="insert into copy_test values (3)")
rows_copy = db.execute_sql(sql="select * from copy_test")
if rows_copy:
print(rows_copy)
```
## Appendix
### Supported database type:
----
```
* sqlite (default)
* postgres
* mysql
* mariadb
* snowflake
* redshift
* salesforce
* s3select
```
### Configuration parameters for sqlite:
-----
-----
```python
* connector_type: sqlite
* config = {
"path": "<folder_path_where_sqlite_database_file_exist>",
"database": "<name_of_sqlite_database_file>"
}
```
**Details:**
* path: (Optional)=> Default user home directory. Path to folder where flat sqlite database file is present.
* database (Required)=> .db is optional, if not present .db will attach to look for database flat file. If file not present will create the file.
-----
### Connection parameters for postgres:
-----
-----
```python
* connector_type: postgres
* config = {
"username": "<postgres_user>",
"password": "<user_password>",
"host": "<host_of_postgres_service>",
"port": "<port_of_postgres_service>",
"database": "<name_of_database>"
}
```
**Details:**
* username: (Required)=> Name of user for connection.
* password (Required)=> Password of user.
* host: (Required)=> Host of PostgreSql service.
* port: (Optional)=> Default 5432. Port of PostgreSql service.
* database: (Optional)=> If provided will be used as default database in connection. If not then query should comply with fully qualified path to table.
-----
### Connection parameters for mysql:
-----
-----
```python
* connector_type: mysql
* config = {
"username": "<mysql_user>",
"password": "<user_password>",
"host": "<host_of_mysql_service>",
"port": "<port_of_mysql_service>",
"database": "<name_of_database>"
}
```
**Details:**
* username: (Required)=> Name of user for connection.
* password (Required)=> Password of user.
* host: (Required)=> Host of MySQL service.
* port: (Optional)=> Default 3306. Port of MySQL service.
* database: (Optional)=> If provided will be used as default database in connection. If not then query should comply with fully qualified path to table.
-----
### Connection parameters for mariadb:
-----
-----
```python
* connector_type: mariadb
* config = {
"username": "<mariadb_user>",
"password": "<user_password>",
"host": "<host_of_mariadb_service>",
"port": "<port_of_mariadb_service>",
"database": "<name_of_database>"
}
```
**Details:**
* username: (Required)=> Name of user for connection.
* password (Required)=> Password of user.
* host: (Required)=> Host of MariaDB service.
* port: (Optional)=> Default 3306. Port of MariaDB service.
* database: (Optional)=> If provided will be used as default database in connection. If not then query should comply with fully qualified path to table.
-----
### Connection parameters for snowflake:
-----
-----
```python
* connector_type: snowflake
* config = {
"username": "<snowflake_user>",
"password": "<user_password>",
"role": "<snowflake_role>",
"account": "<snowflake_account>",
"warehouse": "<snowflake_warehouse>",
"schema": "<snowflake_schema>",
"database": "<name_of_database>",
"key": "<private_key_path>"
}
```
**Details:**
* username: (Required)=> Name of user for connection.
* account: (Required)=> Snowflake account name.
* password (Optional)=> Password of user/private key. Required if private key is not provided.
* role: (Optional)=> If not provided user default role will be used. Consider USE statement to switch from default role.
* database: (Optional)=> If provided will be used as default database in connection. If not then query should comply with fully qualified path to table.
* warehouse: (Optional)=> If not provided user default warehouse will be used. Consider USE statement to switch from default warehouse.
* schema: (Optional)=> If not provided default public schema will be used. Consider USE statement to switch from default schema or fully qualified path to table. Ignored if database is not proivided.
* key: (Optional)=> Either Key or Password is required. If key is present and password is also present then password will be used to decrypt the key. If password is not given then consider the unencrypted key. We strongly recommended to use only encrypted key.
-----
### Connection parameters for redshift:
-----
-----
```python
* connector_type: redshift
* config = {
"username": "<redshift_user>",
"password": "<user_password>",
"host": "<host_of_redshift_service>",
"port": "<port_of_redshift_service>",
"database": "<name_of_database>"
}
```
**Details:**
* username: (Required)=> Name of user for connection.
* password (Required)=> Password of user.
* host: (Required)=> Host of Redshift service.
* port: (Optional)=> Default 5469. Port of Redshift service.
* database: (Required)=> Name of database for connection.
-----
### Connection parameters for Salesforce:
-----
-----
```python
* connector_type: salesforce
* config = {
"username": "<salesforce_user>",
"password": "<user_password>",
"domain": "<domain_of_salesforce_service>",
"token": "<user_token_for_salesforce_service>"
}
```
**Details:**
* username: (Required)=> Name of user for connection.
* password (Required)=> Password of user.
* domain: (Required)=> Host of Salesforce service.
* token: (Required)=> User token for Salesforce service.
-----
### Connection parameters for S3select:
-----
-----
```python
* connector_type: s3select
* config = {
"bucket": "<AWS_S3_bucket>",
"file": "<Path_of_file_in_bucket>",
"type": "<file_type>",
"limit": "<limit_of_record_to_fetch>",
"compression": "<compression_type>"
}
```
**Details:**
* bucket: (Required)=> Name of AWS S3 bucket to look for files for queries.
* type: (Required)=> Supported types is one of csv, json or parquet.
* file (Required)=> Full file path without bucket or prefix key to serach for similar files as per the type of files provided. First line is considered as header of file for csv. Example: if only prefix is provided like mypath/mydatafile and type is csv then all files named as mypath/mydatafile*.csv will be read.
* limit: (Optional)=> Commulative number of records to read from the file (multiple files).
* compression: (Optional)=> Supported types is one of GZIP, BZIP2 or NONE.
-----
### Development Setup
#### Using virtualenv
```bash
python3 -m venv venv
source env/bin/activate
pip install .[all]
```
### Contributing
1. Fork repo- https://github.com/shrivastava-v-ankit/connector-factory.git
2. Create your feature branch - `git checkout -b feature/name`
3. Install Python packages under all virtual environments of Python 3.8, 3.9, 3.10, 3.11
* pip install .[all]
* pip install coverage==7.2.3
* pip install exceptiongroup==1.1.1
* pip install pluggy==1.0.0
* pip install pytest==7.3.0
* pip install pytest-cov==4.0.0
* pip install tomli==2.0.1
4. Run Python test (pytest)
* For Python 3.8:
```python
pytest -v --cov --cov-report=html:test-results/connection_factory_test/htmlcov.3.8 --cov-report=xml:test-results/connection_factory_test/coverage.3.8.xml --junitxml=test-results/connection_factory_test/results.3.8.xml
```
* For Python 3.9:
```python
pytest -v --cov --cov-report=html:test-results/connection_factory_test/htmlcov.3.9 --cov-report=xml:test-results/connection_factory_test/coverage.3.9.xml --junitxml=test-results/connection_factory_test/results.3.9.xml
```
* For Python 3.10:
```python
pytest -v --cov --cov-report=html:test-results/connection_factory_test/htmlcov.3.10 --cov-report=xml:test-results/connection_factory_test/coverage.3.10.xml --junitxml=test-results/connection_factory_test/results.3.10.xml
```
* For Python 3.11:
```python
pytest -v --cov --cov-report=html:test-results/connection_factory_test/htmlcov.3.11 --cov-report=xml:test-results/connection_factory_test/coverage.3.11.xml --junitxml=test-results/connection_factory_test/results.3.11.xml
```
5. Add Python test (pytest) and covrage report for new/changed feature generated under test-results/connection_factory_test/
6. Commit your changes - `git commit -am "Added name"`
7. Push to the branch - `git push origin feature/name`
8. Create a new pull request
Raw data
{
"_id": null,
"home_page": "https://github.com/shrivastava-v-ankit/connector-factory",
"name": "connector-factory",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8, <3.12",
"maintainer_email": "",
"keywords": "python,os independent,database,sqlalchemy,sqlite3,sqlite,postgres,mysql,maridb,snowflake,bigquery,gcp,aws,s3select,salesforce",
"author": "Ankit Shrivastava",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/fb/a7/a8085126a5ae439c960d1fde24175181743b4a25a9b0160e4c5f47aaf63c/connector-factory-0.0.2.tar.gz",
"platform": "any",
"description": "# connector-factory\n\n[](https://opensource.org/licenses/MIT)\n[](https://pypi.org/project/connector-factory)\n\n\nConnector factory is used to manage/create connection with execute queries using the connection.\nThe concept of having single source to connect various sources and perform operations on top of it.\n\n\nUser need not to worry on the crafting the connection string and to identify the methods for the basic operations.\nConnector factory supports DML / DDL executions and have support of Pandas DataFrame to create or replace existing tables.\n\nConnector factory is wrapper on various sources and for database source it uses sqlalchemy for crafting the connection and supports below databases:\n\n```bash\n* Sqlite3\n* PostgreSQl\n* BigQuery (Upcomming)\n* Snowflake\n* MariaDB\n* MySQL\n* Redshift\n* Salesforce\n* S3Select\n```\n\n* Note: Only select operations are supported for Salesforce and S3Select.\nConnector factory can be enhanced for all the sqlalchemy supported database.\n\nConnector factory also provide the facility to create the connections for Salesforce\nusing simple-saleforce python package. Only Select query is supported in Salesforce. For advance usage it exposes the Salesforce object (simple-saleforce python package) which can be used as per the requirements. Refer [simple-saleforce](https://pypi.org/project/simple-salesforce/) for more detail.\n\n## Getting Started\n\n```bash\npip install connector-factory\n```\n\nBy default connector-factory is installed with only SQlite3 support only. For other support it can be installed using combinations of below flags\n* postgres\n* redshift\n* snowflake\n* mysql\n* mariadb\n* salesforce\n* s3select\n* bigquery (upcomming)\n* all (For all supported types)\n\n\n### Using connector-factory\n-----\n```python\nfrom connector_factory import ConnectorFactory\nimport tempfile\n\ntemp_dir = tempfile.gettempdir()\nconfig = {\n \"path\": temp_dir,\n \"database\": \"site.db\"\n }\ndb = ConnectorFactory(connector_type=\"sqlite\", config=config)\ndb.create_session()\n\ndb.execute_sql(sql=\"create table test (id int PRIMARY KEY)\")\ndb.execute_sql(sql=\"insert into test values (1)\")\ndb.execute_sql(sql=\"insert into test values (2)\")\n\nrows = db.execute_sql(sql=\"select * from test\")\nif rows:\n print(rows)\n\n\ndf = db.get_df(sql=\"select * from test\")\nprint(df)\n\ndb.execute_df(panda_df=df, table_name=\"copy_test\", exist_action=\"replace\")\n# db.execute_df(panda_df=df, table_name=copy_test, exist_action=\"replace\", chunk_size=100)\ndb.execute_sql(sql=\"insert into copy_test values (3)\")\nrows_copy = db.execute_sql(sql=\"select * from copy_test\")\nif rows_copy:\n print(rows_copy)\n```\n\n## Appendix\n### Supported database type:\n----\n```\n* sqlite (default)\n* postgres\n* mysql\n* mariadb\n* snowflake\n* redshift\n* salesforce\n* s3select\n```\n\n### Configuration parameters for sqlite:\n-----\n-----\n```python\n* connector_type: sqlite\n* config = {\n \"path\": \"<folder_path_where_sqlite_database_file_exist>\",\n \"database\": \"<name_of_sqlite_database_file>\"\n}\n```\n**Details:**\n* path: (Optional)=> Default user home directory. Path to folder where flat sqlite database file is present.\n* database (Required)=> .db is optional, if not present .db will attach to look for database flat file. If file not present will create the file.\n-----\n\n### Connection parameters for postgres:\n-----\n-----\n```python\n* connector_type: postgres\n* config = {\n \"username\": \"<postgres_user>\",\n \"password\": \"<user_password>\",\n \"host\": \"<host_of_postgres_service>\",\n \"port\": \"<port_of_postgres_service>\",\n \"database\": \"<name_of_database>\"\n}\n```\n\n**Details:**\n* username: (Required)=> Name of user for connection.\n* password (Required)=> Password of user.\n* host: (Required)=> Host of PostgreSql service.\n* port: (Optional)=> Default 5432. Port of PostgreSql service.\n* database: (Optional)=> If provided will be used as default database in connection. If not then query should comply with fully qualified path to table.\n-----\n\n### Connection parameters for mysql:\n-----\n-----\n```python\n* connector_type: mysql\n* config = {\n \"username\": \"<mysql_user>\",\n \"password\": \"<user_password>\",\n \"host\": \"<host_of_mysql_service>\",\n \"port\": \"<port_of_mysql_service>\",\n \"database\": \"<name_of_database>\"\n}\n```\n\n**Details:**\n* username: (Required)=> Name of user for connection.\n* password (Required)=> Password of user.\n* host: (Required)=> Host of MySQL service.\n* port: (Optional)=> Default 3306. Port of MySQL service.\n* database: (Optional)=> If provided will be used as default database in connection. If not then query should comply with fully qualified path to table.\n-----\n\n### Connection parameters for mariadb:\n-----\n-----\n```python\n* connector_type: mariadb\n* config = {\n \"username\": \"<mariadb_user>\",\n \"password\": \"<user_password>\",\n \"host\": \"<host_of_mariadb_service>\",\n \"port\": \"<port_of_mariadb_service>\",\n \"database\": \"<name_of_database>\"\n}\n```\n\n**Details:**\n* username: (Required)=> Name of user for connection.\n* password (Required)=> Password of user.\n* host: (Required)=> Host of MariaDB service.\n* port: (Optional)=> Default 3306. Port of MariaDB service.\n* database: (Optional)=> If provided will be used as default database in connection. If not then query should comply with fully qualified path to table.\n-----\n\n### Connection parameters for snowflake:\n-----\n-----\n```python\n* connector_type: snowflake\n* config = {\n \"username\": \"<snowflake_user>\",\n \"password\": \"<user_password>\",\n \"role\": \"<snowflake_role>\",\n \"account\": \"<snowflake_account>\",\n \"warehouse\": \"<snowflake_warehouse>\",\n \"schema\": \"<snowflake_schema>\",\n \"database\": \"<name_of_database>\",\n \"key\": \"<private_key_path>\"\n}\n```\n\n**Details:**\n* username: (Required)=> Name of user for connection.\n* account: (Required)=> Snowflake account name.\n* password (Optional)=> Password of user/private key. Required if private key is not provided.\n* role: (Optional)=> If not provided user default role will be used. Consider USE statement to switch from default role.\n* database: (Optional)=> If provided will be used as default database in connection. If not then query should comply with fully qualified path to table.\n* warehouse: (Optional)=> If not provided user default warehouse will be used. Consider USE statement to switch from default warehouse.\n* schema: (Optional)=> If not provided default public schema will be used. Consider USE statement to switch from default schema or fully qualified path to table. Ignored if database is not proivided.\n* key: (Optional)=> Either Key or Password is required. If key is present and password is also present then password will be used to decrypt the key. If password is not given then consider the unencrypted key. We strongly recommended to use only encrypted key.\n-----\n\n### Connection parameters for redshift:\n-----\n-----\n```python\n* connector_type: redshift\n* config = {\n \"username\": \"<redshift_user>\",\n \"password\": \"<user_password>\",\n \"host\": \"<host_of_redshift_service>\",\n \"port\": \"<port_of_redshift_service>\",\n \"database\": \"<name_of_database>\"\n}\n```\n\n**Details:**\n* username: (Required)=> Name of user for connection.\n* password (Required)=> Password of user.\n* host: (Required)=> Host of Redshift service.\n* port: (Optional)=> Default 5469. Port of Redshift service.\n* database: (Required)=> Name of database for connection.\n-----\n\n### Connection parameters for Salesforce:\n-----\n-----\n```python\n* connector_type: salesforce\n* config = {\n \"username\": \"<salesforce_user>\",\n \"password\": \"<user_password>\",\n \"domain\": \"<domain_of_salesforce_service>\",\n \"token\": \"<user_token_for_salesforce_service>\"\n}\n```\n\n**Details:**\n* username: (Required)=> Name of user for connection.\n* password (Required)=> Password of user.\n* domain: (Required)=> Host of Salesforce service.\n* token: (Required)=> User token for Salesforce service.\n-----\n\n\n### Connection parameters for S3select:\n-----\n-----\n```python\n* connector_type: s3select\n* config = {\n \"bucket\": \"<AWS_S3_bucket>\",\n \"file\": \"<Path_of_file_in_bucket>\",\n \"type\": \"<file_type>\",\n \"limit\": \"<limit_of_record_to_fetch>\",\n \"compression\": \"<compression_type>\"\n}\n```\n\n**Details:**\n* bucket: (Required)=> Name of AWS S3 bucket to look for files for queries.\n* type: (Required)=> Supported types is one of csv, json or parquet.\n* file (Required)=> Full file path without bucket or prefix key to serach for similar files as per the type of files provided. First line is considered as header of file for csv. Example: if only prefix is provided like mypath/mydatafile and type is csv then all files named as mypath/mydatafile*.csv will be read.\n* limit: (Optional)=> Commulative number of records to read from the file (multiple files).\n* compression: (Optional)=> Supported types is one of GZIP, BZIP2 or NONE.\n-----\n\n\n\n\n### Development Setup\n\n#### Using virtualenv\n\n```bash\npython3 -m venv venv\nsource env/bin/activate\npip install .[all]\n```\n\n### Contributing\n\n1. Fork repo- https://github.com/shrivastava-v-ankit/connector-factory.git\n2. Create your feature branch - `git checkout -b feature/name`\n3. Install Python packages under all virtual environments of Python 3.8, 3.9, 3.10, 3.11\n * pip install .[all]\n * pip install coverage==7.2.3\n * pip install exceptiongroup==1.1.1\n * pip install pluggy==1.0.0\n * pip install pytest==7.3.0\n * pip install pytest-cov==4.0.0\n * pip install tomli==2.0.1\n4. Run Python test (pytest)\n * For Python 3.8:\n ```python\n pytest -v --cov --cov-report=html:test-results/connection_factory_test/htmlcov.3.8 --cov-report=xml:test-results/connection_factory_test/coverage.3.8.xml --junitxml=test-results/connection_factory_test/results.3.8.xml\n ```\n * For Python 3.9:\n ```python\n pytest -v --cov --cov-report=html:test-results/connection_factory_test/htmlcov.3.9 --cov-report=xml:test-results/connection_factory_test/coverage.3.9.xml --junitxml=test-results/connection_factory_test/results.3.9.xml\n ```\n * For Python 3.10:\n ```python\n pytest -v --cov --cov-report=html:test-results/connection_factory_test/htmlcov.3.10 --cov-report=xml:test-results/connection_factory_test/coverage.3.10.xml --junitxml=test-results/connection_factory_test/results.3.10.xml\n ```\n * For Python 3.11:\n ```python\n pytest -v --cov --cov-report=html:test-results/connection_factory_test/htmlcov.3.11 --cov-report=xml:test-results/connection_factory_test/coverage.3.11.xml --junitxml=test-results/connection_factory_test/results.3.11.xml\n ```\n5. Add Python test (pytest) and covrage report for new/changed feature generated under test-results/connection_factory_test/\n6. Commit your changes - `git commit -am \"Added name\"`\n7. Push to the branch - `git push origin feature/name`\n8. Create a new pull request\n\n\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Connector Factory;",
"version": "0.0.2",
"project_urls": {
"Homepage": "https://github.com/shrivastava-v-ankit/connector-factory",
"Source": "https://github.com/shrivastava-v-ankit/connector-factory/",
"Tracker": "https://github.com/shrivastava-v-ankit/connector-factory/issues"
},
"split_keywords": [
"python",
"os independent",
"database",
"sqlalchemy",
"sqlite3",
"sqlite",
"postgres",
"mysql",
"maridb",
"snowflake",
"bigquery",
"gcp",
"aws",
"s3select",
"salesforce"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "55ef9a42339678081a7620be6275f01b391d71e4d4774dca64fe74073291da67",
"md5": "bc79232debfc93d51116c775b22e406a",
"sha256": "2e67862b5714d4cf1381be2c78a25d4527c2cedb4b891f9aae1c34a390c117ba"
},
"downloads": -1,
"filename": "connector_factory-0.0.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "bc79232debfc93d51116c775b22e406a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8, <3.12",
"size": 24861,
"upload_time": "2023-12-15T11:41:28",
"upload_time_iso_8601": "2023-12-15T11:41:28.417451Z",
"url": "https://files.pythonhosted.org/packages/55/ef/9a42339678081a7620be6275f01b391d71e4d4774dca64fe74073291da67/connector_factory-0.0.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "fba7a8085126a5ae439c960d1fde24175181743b4a25a9b0160e4c5f47aaf63c",
"md5": "13374f8d5e10c739009304eff87a2015",
"sha256": "10181d9d9e96df804be88533469247d1e89737a106f1419fb6acf5a4eb1020a5"
},
"downloads": -1,
"filename": "connector-factory-0.0.2.tar.gz",
"has_sig": false,
"md5_digest": "13374f8d5e10c739009304eff87a2015",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8, <3.12",
"size": 22198,
"upload_time": "2023-12-15T11:41:30",
"upload_time_iso_8601": "2023-12-15T11:41:30.907853Z",
"url": "https://files.pythonhosted.org/packages/fb/a7/a8085126a5ae439c960d1fde24175181743b4a25a9b0160e4c5f47aaf63c/connector-factory-0.0.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-12-15 11:41:30",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "shrivastava-v-ankit",
"github_project": "connector-factory",
"travis_ci": false,
"coveralls": true,
"github_actions": false,
"lcname": "connector-factory"
}
JSON
