| Name | datamation JSON |
| Version |
0.1.3
JSON |
| download |
| home_page | None |
| Summary | Data information synchronization |
| upload_time | 2024-11-25 07:47:32 |
| maintainer | None |
| docs_url | None |
| author | lidaoran |
| requires_python | <4.0,>=3.7 |
| license | BSD |
| keywords |
etl
synchronization
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Datamation
#### 特性
- 不同数据库中大批量数据的导入、导出，ETL数据同步
- 关系型数据库的数据导入到ElsticSearch
- 数据库表结构差异比对、整表数据差异比对，定位到行列级别
- 消息通知，支持邮件、钉钉、企业微信
## 安装
1. 拉取git项目到本地
2. pip install datamation
#### 基本使用
初始化连接
```python
import pymysql
import datamation as dm
# 创建数据源连接
src_conn = lambda:pymysql.connect(host='127.0.0.1',user='root',passwd='xxx',database='demo',port=13307,write_timeout=50000,connect_timeout=20000)
# 创建目标数据库连接
tag_conn = lambda:pymysql.connect(host='127.0.0.1',user='root',passwd='xxx',database='demo',port=13306,read_timeout=50000,connect_timeout=10000)
```
示例1
```python
dm.sqlsync(
src_conn = src_conn,
src_table='table1',
tag_conn = tag_conn,
tag_table='table2',
batch_size=20000, # 批量写入行数
truncate=True # 同步开始时使用truncate命令清空目标数据表
)
```
示例2
```python
src = dm.source_sql(src_conn,'table1')
tag = dm.table_basic(tag_conn,'table1',columns =['col1','col2','col3'],batch_size=20000)
for row in src:
tag.insert(row)
tag.endload()
```
##### 数据库差异检测
校验两个数据表数据值是否一致，返回差异数据
1 当主键为数值类型时，使用主键进行比对
```python
comp = dm.get_sqldata_diff2(src_conn,tag_conn,'tb_data','tb_data_copy1',compare_field='id')
print(comp)
```
2. 当主键为uuid类型时
```python
comp = dm.get_sqldata_diff1(src_conn,tag_conn,'tb_data','tb_data_copy1',compare_field='id',)
print(comp)
```
表结构差异比对 基于sqlalchemy
```python
comp = dm.get_sqldb_diff(src_conn,tag_conn,'tb_data','tb_data_copy1')
print(comp)
```
##### 消息通知
集成有邮件、钉钉、企业微信的消息推送
邮件消息发送
示例1
```python
# 示例1
import datamation as dm
tm = dm.to_mail(user,passwd,host)
tm.name('hello word',to = ['xxx@xx.com','xxx@xxx.com'],
cc= ['xxx@xx.com','xxx@xxx.com'],
bcc= ['xxx@xx.com','xxx@xxx.com'],
showname = 'datamation')
tm.add_text('hello word')
tm.add_html('<p> hello word</p> <img src=cid:image001.jpg style="height:71px; width:116px" />')
tm.add_related({'image001.jpg':'data/image001.jpg'}) # 添加在html中引用显示的图片内容
tm.add_attachment({'data.xlsx':'/data/data.xlsx'}) # 添加附件
tm.send()
```
示例2
```python
# 示例2
import datamation as dm
tm = dm.to_mail(user,passwd,host)
tm.send('hello word',
to = ['xxx@xx.com'],
cc= [''],
bcc= [''],
showname = 'datamation',
related = {'image001.jpg':'data/image001.jpg'},
attachment = {'data.xlsx':'/data/data.xlsx'})
```
Raw data
{
"_id": null,
"home_page": null,
"name": "datamation",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.7",
"maintainer_email": null,
"keywords": "etl, synchronization",
"author": "lidaoran",
"author_email": "qianxuanyon@hotmail.com",
"download_url": "https://files.pythonhosted.org/packages/f1/68/e74dd15ab0d807b762c1aebda7ac6558e696ed144c3e03447e4097a607e2/datamation-0.1.3.tar.gz",
"platform": null,
"description": "# Datamation\n\n\n\n#### \u7279\u6027\n\n- \u4e0d\u540c\u6570\u636e\u5e93\u4e2d\u5927\u6279\u91cf\u6570\u636e\u7684\u5bfc\u5165\u3001\u5bfc\u51fa\uff0cETL\u6570\u636e\u540c\u6b65\n- \u5173\u7cfb\u578b\u6570\u636e\u5e93\u7684\u6570\u636e\u5bfc\u5165\u5230ElsticSearch\n- \u6570\u636e\u5e93\u8868\u7ed3\u6784\u5dee\u5f02\u6bd4\u5bf9\u3001\u6574\u8868\u6570\u636e\u5dee\u5f02\u6bd4\u5bf9\uff0c\u5b9a\u4f4d\u5230\u884c\u5217\u7ea7\u522b\n- \u6d88\u606f\u901a\u77e5\uff0c\u652f\u6301\u90ae\u4ef6\u3001\u9489\u9489\u3001\u4f01\u4e1a\u5fae\u4fe1\n\n## \u5b89\u88c5\n\n1. \u62c9\u53d6git\u9879\u76ee\u5230\u672c\u5730\n2. pip install datamation\n\n#### \u57fa\u672c\u4f7f\u7528\n\n\u521d\u59cb\u5316\u8fde\u63a5\n```python\nimport pymysql \nimport datamation as dm\n\n# \u521b\u5efa\u6570\u636e\u6e90\u8fde\u63a5\nsrc_conn = lambda:pymysql.connect(host='127.0.0.1',user='root',passwd='xxx',database='demo',port=13307,write_timeout=50000,connect_timeout=20000)\n\n# \u521b\u5efa\u76ee\u6807\u6570\u636e\u5e93\u8fde\u63a5\ntag_conn = lambda:pymysql.connect(host='127.0.0.1',user='root',passwd='xxx',database='demo',port=13306,read_timeout=50000,connect_timeout=10000)\n\n```\n\n\u793a\u4f8b1\n```python\ndm.sqlsync(\n src_conn = src_conn,\n src_table='table1',\n tag_conn = tag_conn,\n tag_table='table2',\n batch_size=20000, # \u6279\u91cf\u5199\u5165\u884c\u6570\n truncate=True # \u540c\u6b65\u5f00\u59cb\u65f6\u4f7f\u7528truncate\u547d\u4ee4\u6e05\u7a7a\u76ee\u6807\u6570\u636e\u8868\n ) \n\n```\n\n\u793a\u4f8b2\n```python\n\nsrc = dm.source_sql(src_conn,'table1')\ntag = dm.table_basic(tag_conn,'table1',columns =['col1','col2','col3'],batch_size=20000)\nfor row in src:\n tag.insert(row)\ntag.endload()\n```\n\n##### \u6570\u636e\u5e93\u5dee\u5f02\u68c0\u6d4b\n\n\u6821\u9a8c\u4e24\u4e2a\u6570\u636e\u8868\u6570\u636e\u503c\u662f\u5426\u4e00\u81f4\uff0c\u8fd4\u56de\u5dee\u5f02\u6570\u636e\n1 \u5f53\u4e3b\u952e\u4e3a\u6570\u503c\u7c7b\u578b\u65f6\uff0c\u4f7f\u7528\u4e3b\u952e\u8fdb\u884c\u6bd4\u5bf9\n```python\ncomp = dm.get_sqldata_diff2(src_conn,tag_conn,'tb_data','tb_data_copy1',compare_field='id')\nprint(comp)\n\n```\n2. \u5f53\u4e3b\u952e\u4e3auuid\u7c7b\u578b\u65f6\n\n```python\ncomp = dm.get_sqldata_diff1(src_conn,tag_conn,'tb_data','tb_data_copy1',compare_field='id',)\nprint(comp)\n\n```\n\u8868\u7ed3\u6784\u5dee\u5f02\u6bd4\u5bf9 \u57fa\u4e8esqlalchemy\n```python\ncomp = dm.get_sqldb_diff(src_conn,tag_conn,'tb_data','tb_data_copy1')\nprint(comp)\n\n```\n\n##### \u6d88\u606f\u901a\u77e5\n\u96c6\u6210\u6709\u90ae\u4ef6\u3001\u9489\u9489\u3001\u4f01\u4e1a\u5fae\u4fe1\u7684\u6d88\u606f\u63a8\u9001\n\n\u90ae\u4ef6\u6d88\u606f\u53d1\u9001\n\u793a\u4f8b1 \n```python\n# \u793a\u4f8b1\nimport datamation as dm\ntm = dm.to_mail(user,passwd,host)\ntm.name('hello word',to = ['xxx@xx.com','xxx@xxx.com'],\n cc= ['xxx@xx.com','xxx@xxx.com'],\n bcc= ['xxx@xx.com','xxx@xxx.com'],\n showname = 'datamation')\ntm.add_text('hello word')\ntm.add_html('<p> hello word</p> <img src=cid:image001.jpg style=\"height:71px; width:116px\" />')\ntm.add_related({'image001.jpg':'data/image001.jpg'}) # \u6dfb\u52a0\u5728html\u4e2d\u5f15\u7528\u663e\u793a\u7684\u56fe\u7247\u5185\u5bb9\ntm.add_attachment({'data.xlsx':'/data/data.xlsx'}) # \u6dfb\u52a0\u9644\u4ef6\ntm.send()\n\n```\n\u793a\u4f8b2\n\n```python\n# \u793a\u4f8b2\nimport datamation as dm\ntm = dm.to_mail(user,passwd,host)\ntm.send('hello word',\n to = ['xxx@xx.com'],\n cc= [''],\n bcc= [''],\n showname = 'datamation',\n related = {'image001.jpg':'data/image001.jpg'},\n attachment = {'data.xlsx':'/data/data.xlsx'})\n```\n\n",
"bugtrack_url": null,
"license": "BSD",
"summary": "Data information synchronization",
"version": "0.1.3",
"project_urls": null,
"split_keywords": [
"etl",
" synchronization"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "6b0da62a6ee66dd1b5765c73d0c0e6524b22423800c34842e4bea2733ddf3c14",
"md5": "00ccec47dc0e587223b87da98fc996f0",
"sha256": "4e2bb56a2b475c36947d0304a130a86cbbd2b05ec979ce39b03751cc97d9d241"
},
"downloads": -1,
"filename": "datamation-0.1.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "00ccec47dc0e587223b87da98fc996f0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.7",
"size": 81477,
"upload_time": "2024-11-25T07:47:30",
"upload_time_iso_8601": "2024-11-25T07:47:30.633976Z",
"url": "https://files.pythonhosted.org/packages/6b/0d/a62a6ee66dd1b5765c73d0c0e6524b22423800c34842e4bea2733ddf3c14/datamation-0.1.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f168e74dd15ab0d807b762c1aebda7ac6558e696ed144c3e03447e4097a607e2",
"md5": "599204440a8716134939c6293fd90c10",
"sha256": "fafb8239c4c0f1cf785ea596a65dc40e7c89da77312ca374e9ecc6c38ed9fde2"
},
"downloads": -1,
"filename": "datamation-0.1.3.tar.gz",
"has_sig": false,
"md5_digest": "599204440a8716134939c6293fd90c10",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.7",
"size": 68270,
"upload_time": "2024-11-25T07:47:32",
"upload_time_iso_8601": "2024-11-25T07:47:32.380883Z",
"url": "https://files.pythonhosted.org/packages/f1/68/e74dd15ab0d807b762c1aebda7ac6558e696ed144c3e03447e4097a607e2/datamation-0.1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-25 07:47:32",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "datamation"
}
