| Name | datacx JSON |
| Version |
0.4.0
JSON |
| download |
| home_page | |
| Summary | Data Connectors for all data sources |
| upload_time | 2023-04-07 08:38:40 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.7 |
| license | MIT License Copyright (c) 2023 Vinish M Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
connectors
datalake reader
datawarehouse reader
dataconnector
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<div align="center">
<img src="docs/images/LOGO.png" alt="drawing" width="200"/>
A Data Connector for all the data sources
<div align="left">
This library helps to read and write data from most of the data sources. It accelerate the ML and ETL process without worrying about the multiple data connectors.
## Installation
```bash
pip install -U datacx
```
**Install from sources**
Alternatively, you can also clone the latest version from the [repository](https://github.com/VinishUchiha/datacx) and install it directly from the source code:
```bash
pip install -e .
```
## Quick tour
```python
>>> from datacx import DataCX
>>> from transformers import pipeline
>>> dcx = DataCX('./dcx_config.yaml') # Check the sample_dcx_config.yaml for reference
>>> print(dcx.get_supported_data_sources_list())
['s3', 'gcs', 'azureblob', 'bigquery', 'snowflake', 'redshift', 'starrocks', 'postgresql', 'mysql', 'oracle', 'mssql', 'mariadb', 'sqlite', 'elasticsearch', 'mongodb']
>>> mongodb = dcx.connect('mongodb')
>>> df = mongodb.read_as_dataframe(database='reviewdb',collection='reviews')
>>> df.head()
_id Review
0 64272bb06a14f52787e0a09e good and interesting
1 64272bb06a14f52787e0a09f This class is very helpful to me. Currently, I...
2 64272bb06a14f52787e0a0a0 like!Prof and TAs are helpful and the discussi...
3 64272bb06a14f52787e0a0a1 Easy to follow and includes a lot basic and im...
4 64272bb06a14f52787e0a0a2 Really nice teacher!I could got the point eazl...
>>> classifier = pipeline("sentiment-analysis")
>>> reviews = df.Review.tolist()
>>> results = classifier(reviews,truncation=True)
>>> for result in results:
>>> print(f"label: {result['label']}, with score: {round(result['score'], 4)}")
label: POSITIVE, with score: 0.9999
label: POSITIVE, with score: 0.9997
label: POSITIVE, with score: 0.9999
label: POSITIVE, with score: 0.999
label: POSITIVE, with score: 0.9967
>>> df['predicted_label'] = [result['label'] for result in results]
>>> df['predicted_score'] = [round(result['score'], 4) for result in results]
# Write the results to the MongoDB
>>> mongodb.write_dataframe(df,'reviewdb','review_sentiments')
```
## Supported Connectors
|Data Sources| Type | Read | Write |
|------------|------| ---- | -----|
|S3|datalake| ☑ | ☑ |
|GCS|datalake| ☑ | ☑ |
|Azure Blob Stoarge| datalake| ☑ | ☑ |
|Snowflake| datawarehouse | ☑ | ☑ |
|BigQuery| datawarehouse | ☑ | ☑ |
|StarRocks| datawarehouse | ☑ | ☑ |
|Redshift| datawarehouse | ☑ | ☑ |
|PostgreSQL| database | ☑ | ☑ |
|MySQL| database | ☑ | ☑ |
|MariaDB| database | ☑ | ☑ |
|MsSQL| database | ☑ | ☑ |
|Oracle| database | ☑ | ☑ |
|SQLite| database | ☑ | ☑ |
|MongoDB| nosql | ☑ | ☑ |
|ElasticSearch| nosql | ☑ | ☑ |
## Acknowledgement
Some functionalities of DataCX are inspired by the following packages.
- [ConnectorX](https://github.com/sfu-db/connector-x)
DataCX used Connectorx to read data from most of the RDBMS databases to utilize the performance benefits and inspired the return_type parameter from it
- [GeneratorREX](https://generatorrex.fandom.com/wiki/Generator_Rex_Wiki)
DataCX logo inspired by the American animated science fiction television series and created by my graphic designer friend [Belgin David](https://www.linkedin.com/in/belgin-david-4b699a1b8)
Raw data
{
"_id": null,
"home_page": "",
"name": "datacx",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "connectors,datalake reader,datawarehouse reader,dataconnector",
"author": "",
"author_email": "Vinish M <vinishuchiha@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/fa/56/d687448cf924ee601a457c227ff3a396fa4ec5db95157945aac53c06aee9/datacx-0.4.0.tar.gz",
"platform": null,
"description": "<div align=\"center\">\n\n<img src=\"docs/images/LOGO.png\" alt=\"drawing\" width=\"200\"/>\n\nA Data Connector for all the data sources\n\n<div align=\"left\">\n\nThis library helps to read and write data from most of the data sources. It accelerate the ML and ETL process without worrying about the multiple data connectors.\n\n## Installation\n```bash\npip install -U datacx\n```\n**Install from sources**\n\nAlternatively, you can also clone the latest version from the [repository](https://github.com/VinishUchiha/datacx) and install it directly from the source code:\n\n```bash\npip install -e .\n```\n\n## Quick tour\n```python\n>>> from datacx import DataCX\n>>> from transformers import pipeline\n\n>>> dcx = DataCX('./dcx_config.yaml') # Check the sample_dcx_config.yaml for reference\n>>> print(dcx.get_supported_data_sources_list())\n['s3', 'gcs', 'azureblob', 'bigquery', 'snowflake', 'redshift', 'starrocks', 'postgresql', 'mysql', 'oracle', 'mssql', 'mariadb', 'sqlite', 'elasticsearch', 'mongodb']\n\n>>> mongodb = dcx.connect('mongodb')\n>>> df = mongodb.read_as_dataframe(database='reviewdb',collection='reviews')\n>>> df.head()\n _id\t Review\n0\t64272bb06a14f52787e0a09e\tgood and interesting\n1\t64272bb06a14f52787e0a09f\tThis class is very helpful to me. Currently, I...\n2\t64272bb06a14f52787e0a0a0\tlike!Prof and TAs are helpful and the discussi...\n3\t64272bb06a14f52787e0a0a1\tEasy to follow and includes a lot basic and im...\n4\t64272bb06a14f52787e0a0a2\tReally nice teacher!I could got the point eazl...\n\n>>> classifier = pipeline(\"sentiment-analysis\")\n>>> reviews = df.Review.tolist()\n>>> results = classifier(reviews,truncation=True)\n>>> for result in results:\n>>> print(f\"label: {result['label']}, with score: {round(result['score'], 4)}\")\nlabel: POSITIVE, with score: 0.9999\nlabel: POSITIVE, with score: 0.9997\nlabel: POSITIVE, with score: 0.9999\nlabel: POSITIVE, with score: 0.999\nlabel: POSITIVE, with score: 0.9967\n\n>>> df['predicted_label'] = [result['label'] for result in results]\n>>> df['predicted_score'] = [round(result['score'], 4) for result in results]\n\n# Write the results to the MongoDB\n>>> mongodb.write_dataframe(df,'reviewdb','review_sentiments')\n```\n## Supported Connectors\n \n|Data Sources| Type | Read | Write |\n|------------|------| ---- | -----|\n|S3|datalake| ☑ | ☑ |\n|GCS|datalake| ☑ | ☑ |\n|Azure Blob Stoarge| datalake| ☑ | ☑ |\n|Snowflake| datawarehouse | ☑ | ☑ |\n|BigQuery| datawarehouse | ☑ | ☑ |\n|StarRocks| datawarehouse | ☑ | ☑ |\n|Redshift| datawarehouse | ☑ | ☑ |\n|PostgreSQL| database | ☑ | ☑ |\n|MySQL| database | ☑ | ☑ |\n|MariaDB| database | ☑ | ☑ |\n|MsSQL| database | ☑ | ☑ |\n|Oracle| database | ☑ | ☑ |\n|SQLite| database | ☑ | ☑ |\n|MongoDB| nosql | ☑ | ☑ |\n|ElasticSearch| nosql | ☑ | ☑ |\n\n## Acknowledgement\n\nSome functionalities of DataCX are inspired by the following packages.\n\n- [ConnectorX](https://github.com/sfu-db/connector-x)\n \n DataCX used Connectorx to read data from most of the RDBMS databases to utilize the performance benefits and inspired the return_type parameter from it\n \n- [GeneratorREX](https://generatorrex.fandom.com/wiki/Generator_Rex_Wiki)\n \n DataCX logo inspired by the American animated science fiction television series and created by my graphic designer friend [Belgin David](https://www.linkedin.com/in/belgin-david-4b699a1b8)\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2023 Vinish M Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
"summary": "Data Connectors for all data sources",
"version": "0.4.0",
"split_keywords": [
"connectors",
"datalake reader",
"datawarehouse reader",
"dataconnector"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4dc2a349fb6ad55eed20de3bf11969b3323782c47b73bf8025ad89e6d0b2893d",
"md5": "571d876fb0a6c906efbf4ac0698f4ac1",
"sha256": "1f6d054399df15e7226e3ce2787d4d50e956e9fdbbd82ce391a478b63b00dd14"
},
"downloads": -1,
"filename": "datacx-0.4.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "571d876fb0a6c906efbf4ac0698f4ac1",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 17783,
"upload_time": "2023-04-07T08:38:38",
"upload_time_iso_8601": "2023-04-07T08:38:38.472725Z",
"url": "https://files.pythonhosted.org/packages/4d/c2/a349fb6ad55eed20de3bf11969b3323782c47b73bf8025ad89e6d0b2893d/datacx-0.4.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "fa56d687448cf924ee601a457c227ff3a396fa4ec5db95157945aac53c06aee9",
"md5": "ab7713e5c8d089e21ed7389d7cfb1f44",
"sha256": "7fec31d8e00691edb3edf4b3e18c9df4b3ebfa20abb0bfb0010db3948f710186"
},
"downloads": -1,
"filename": "datacx-0.4.0.tar.gz",
"has_sig": false,
"md5_digest": "ab7713e5c8d089e21ed7389d7cfb1f44",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 15281,
"upload_time": "2023-04-07T08:38:40",
"upload_time_iso_8601": "2023-04-07T08:38:40.697760Z",
"url": "https://files.pythonhosted.org/packages/fa/56/d687448cf924ee601a457c227ff3a396fa4ec5db95157945aac53c06aee9/datacx-0.4.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-04-07 08:38:40",
"github": false,
"gitlab": false,
"bitbucket": false,
"lcname": "datacx"
}
