# Python-SDK
## 能力
- 授权模式（token 管理）
- [x] client_credentials 模式
- [ ] password 模式
- [ ] authorization code 模式
- 接口调用
- [x] GET
- [ ] POST
- [ ] PUT
- [ ] DELETE
- 数据同步（接口必须支持翻页）
- 全量同步
- [x] 同步为 csv 格式
- [ ] 同步为 xls/xlsx 格式
- [x] 同步到数据库
- [x] 同步到模型
- 增量同步（接口必须支持ts增量参数）
- [x] 同步到数据库
- [x] 同步到模型
## 依赖
- python 3.10 +
- [requirements.txt](requirements.txt)
## 相关资料
- [oauth2.0](https://oauth.net/2/)
- [sqlalchemy](https://www.sqlalchemy.org/)
## 支持的数据库
理论上只要 sqlalchemy支持的数据库驱动都可以支持，以下是测试的情况
如果sqlalchemy无法直接支持，可以先同步到模型，然后自行处理数据入库的逻辑。
| 数据库 | 驱动 | 测试情况 | upsert 支持 |
|------------|----------------------------| --- | --- |
| MySQL | pymysql | 测试通过 | todo |
| SQLite | null | 测试通过 | todo |
| PostgreSQL | psycopg2 | 测试通过 | todo |
| SQL Server | pymssql | 测试通过 | todo |
| Oracle | cx_oracle | 测试通过 | todo |
## 示例
详见以下示例代码，和示例代码中的相关注释
- [CallAPI](tests/example.py)
- [SyncToCSV](tests/example_csv.py)
- [SyncToModel](tests/example_model.py)
- [SyncToDB](tests/example_db.py)
## 性能
性能与 ORM 的实现方式（特别是对 upsert 的实现方式），数据库的实现方式，以及网络环境有关，不一定适用于所有情况。
当同步到数据库时，SDK 会采用分批读取/写入的方式，以减少内存的占用。
当同步到模型时，则会将所有数据写入到一个数组中，可能会占用较大的内存。
以下是测试环境
### 同步程序运行环境
- i5 cpu
- 32G 内存
- windows10 X64
- WD 固态硬盘
- dotnet 4.8
### 测试接口信息
- /api/v1/sync/fake
- 使用 pageSize=2000 仅限同步
- 接口请求耗时约 0.2 - 0.3 秒
- 接口数据示例
```json
{
"errCode": 0,
"errMsg": "success",
"requestId": "73a60094-c0f1-4daf-bc58-4626fbef7a2b",
"data": {
"pageSize": 2000,
"pageNum": 1,
"totalNum": 10000,
"rows": [{
"id": 1,
"colString1": "Oxqmn5MWCt",
"colString2": "mzavQncWeNlOlFgUW7HC",
"colString3": "mvy6K1HU7rdCicPbvvA3rNZcDWPhvV",
"colString4": "XGsK5NVQHOu4JrmHZ9ZL1iLf0UYpdIvNIzswULzb",
"colInt1": 3931594532918648027,
"colInt2": 337586114254574578,
"colInt3": 2291922259603323213,
"colInt4": 3000562485500051124,
"colFloat1": 0.46541339000557547,
"colFloat2": 0.6307996439929248,
"colFloat3": 0.9278393850101392,
"colFloat4": 0.7286866920659677,
"colSqlTime1": "2023-10-20 22:02:07",
"colSqlTime2": "2023-10-20 22:02:07",
"colSqlTime3": "2023-10-20 22:02:07",
"colSqlTime4": "2023-10-20 22:02:07"
}]
}
}
```
### 测试结果
- 数据库：本地mysql:5.7 (docker环境)
- pageSize = 2000
- batchSize = 100
| 数据量(`totalNum`) | Time Spent on `Direct CallAPI` | Time Spent on `Model Sync ` | Time Spent on `DB Sync ` |
| ------------------ | ------------------------------ | --------------------------- | ------------------------ |
| `1w` | 0.324615 | 1.192462 | 6.551163 |
| `10w` | 0.280315 | 7.836467 | 30.583456 |
| `100w` | 0.28313 | 78.528211 | 263.68381 |
| 数据量(`totalNum`) | Memory Spent on `Model Sync ` | Memory Spent on `DB Sync ` |
| ------------------ | ----------------------------- | -------------------------- |
| `1w` | 3.07421875 | 3.12890625 |
| `10w` | 107.265625 | 15.22265625 |
| `100w` | 695.171875 | 12.56640625 |
## 注意事项
### 1 `OAuth2`身份认证+鉴权
`initOauth2ClientCredentials`方法可以实现对token的自动续约与管理，指定client_id与client_secret即可
### 2 同步到`csv`文件`SyncToCSV`
使用案例在`exampleSyncToCSV()`方法中，开发者仅需在`example_csv.py`文件中进行配置即可。仅需配置`api`参数，包括`url`、`pageSize`等，还需配置写入的文件名。
### 3 同步到数据模型`SyncToModel`
因为写入到`csv`文件中的数据都是字符串类型，为满足开发者需求，提供同步数据到Model的功能。使用案例在`exampleSyncToModel()`方法中，开发者仅需在`example_model.py`文件中进行配置即可。配置：`Model`类型，`api`，指定`Model`对象
### 4 同步到数据库`SyncToDB`
使用案例在`exampleSyncToDB()`方法中，开发者仅需在`example_db.py`文件中进行配置即可。`ORM：sqlalchemy`
该`sdk`提供全量与增量刷新数据到DB中，支持数据库`MySQL`（建议）、`Oracle`、`PostgrelSQL`、`SQLServer`
目前暂时不支持 `upsert`，全量刷新时建议删除原本存在的表结构，增量同步时则不需要。配置如下：数据库、`api`、映射类，注意若接口返回的数据存在id，一般无需指定自增
Raw data
{
"_id": null,
"home_page": "https://github.com/ECNU/ecnu-openapi-sdk-python",
"name": "ecnu-openapi-sdk-python",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "",
"author": "ECNU",
"author_email": "dataservice@ecnu.edu.cn",
"download_url": "https://files.pythonhosted.org/packages/6e/e2/e11304ce88e8dbb4aec9e22c8ae9f38f4503db6c1697a78a5f829a4db68d/ecnu-openapi-sdk-python-1.0.0.tar.gz",
"platform": null,
"description": "# Python-SDK\r\n\r\n## \u80fd\u529b\r\n- \u6388\u6743\u6a21\u5f0f\uff08token \u7ba1\u7406\uff09\r\n - [x] client_credentials \u6a21\u5f0f\r\n - [ ] password \u6a21\u5f0f\r\n - [ ] authorization code \u6a21\u5f0f\r\n- \u63a5\u53e3\u8c03\u7528\r\n - [x] GET\r\n - [ ] POST\r\n - [ ] PUT\r\n - [ ] DELETE\r\n- \u6570\u636e\u540c\u6b65\uff08\u63a5\u53e3\u5fc5\u987b\u652f\u6301\u7ffb\u9875\uff09\r\n - \u5168\u91cf\u540c\u6b65\r\n - [x] \u540c\u6b65\u4e3a csv \u683c\u5f0f\r\n - [ ] \u540c\u6b65\u4e3a xls/xlsx \u683c\u5f0f\r\n - [x] \u540c\u6b65\u5230\u6570\u636e\u5e93\r\n - [x] \u540c\u6b65\u5230\u6a21\u578b\r\n - \u589e\u91cf\u540c\u6b65\uff08\u63a5\u53e3\u5fc5\u987b\u652f\u6301ts\u589e\u91cf\u53c2\u6570\uff09\r\n - [x] \u540c\u6b65\u5230\u6570\u636e\u5e93\r\n - [x] \u540c\u6b65\u5230\u6a21\u578b\r\n\r\n## \u4f9d\u8d56\r\n- python 3.10 +\r\n- [requirements.txt](requirements.txt)\r\n\r\n## \u76f8\u5173\u8d44\u6599\r\n- [oauth2.0](https://oauth.net/2/)\r\n- [sqlalchemy](https://www.sqlalchemy.org/)\r\n## \u652f\u6301\u7684\u6570\u636e\u5e93\r\n\u7406\u8bba\u4e0a\u53ea\u8981 sqlalchemy\u652f\u6301\u7684\u6570\u636e\u5e93\u9a71\u52a8\u90fd\u53ef\u4ee5\u652f\u6301\uff0c\u4ee5\u4e0b\u662f\u6d4b\u8bd5\u7684\u60c5\u51b5\r\n\r\n\u5982\u679csqlalchemy\u65e0\u6cd5\u76f4\u63a5\u652f\u6301\uff0c\u53ef\u4ee5\u5148\u540c\u6b65\u5230\u6a21\u578b\uff0c\u7136\u540e\u81ea\u884c\u5904\u7406\u6570\u636e\u5165\u5e93\u7684\u903b\u8f91\u3002\r\n\r\n| \u6570\u636e\u5e93 | \u9a71\u52a8 | \u6d4b\u8bd5\u60c5\u51b5 | upsert \u652f\u6301 |\r\n|------------|----------------------------| --- | --- |\r\n| MySQL | pymysql | \u6d4b\u8bd5\u901a\u8fc7 | todo |\r\n| SQLite | null | \u6d4b\u8bd5\u901a\u8fc7 | todo |\r\n| PostgreSQL | psycopg2 | \u6d4b\u8bd5\u901a\u8fc7 | todo |\r\n| SQL Server | pymssql | \u6d4b\u8bd5\u901a\u8fc7 | todo |\r\n| Oracle | cx_oracle | \u6d4b\u8bd5\u901a\u8fc7 | todo |\r\n\r\n## \u793a\u4f8b\r\n\r\n\u8be6\u89c1\u4ee5\u4e0b\u793a\u4f8b\u4ee3\u7801\uff0c\u548c\u793a\u4f8b\u4ee3\u7801\u4e2d\u7684\u76f8\u5173\u6ce8\u91ca\r\n\r\n- [CallAPI](tests/example.py)\r\n- [SyncToCSV](tests/example_csv.py)\r\n- [SyncToModel](tests/example_model.py)\r\n- [SyncToDB](tests/example_db.py)\r\n\r\n\r\n## \u6027\u80fd\r\n\r\n\u6027\u80fd\u4e0e ORM \u7684\u5b9e\u73b0\u65b9\u5f0f\uff08\u7279\u522b\u662f\u5bf9 upsert \u7684\u5b9e\u73b0\u65b9\u5f0f\uff09\uff0c\u6570\u636e\u5e93\u7684\u5b9e\u73b0\u65b9\u5f0f\uff0c\u4ee5\u53ca\u7f51\u7edc\u73af\u5883\u6709\u5173\uff0c\u4e0d\u4e00\u5b9a\u9002\u7528\u4e8e\u6240\u6709\u60c5\u51b5\u3002\r\n\r\n\u5f53\u540c\u6b65\u5230\u6570\u636e\u5e93\u65f6\uff0cSDK \u4f1a\u91c7\u7528\u5206\u6279\u8bfb\u53d6/\u5199\u5165\u7684\u65b9\u5f0f\uff0c\u4ee5\u51cf\u5c11\u5185\u5b58\u7684\u5360\u7528\u3002\r\n\r\n\u5f53\u540c\u6b65\u5230\u6a21\u578b\u65f6\uff0c\u5219\u4f1a\u5c06\u6240\u6709\u6570\u636e\u5199\u5165\u5230\u4e00\u4e2a\u6570\u7ec4\u4e2d\uff0c\u53ef\u80fd\u4f1a\u5360\u7528\u8f83\u5927\u7684\u5185\u5b58\u3002\r\n\r\n\u4ee5\u4e0b\u662f\u6d4b\u8bd5\u73af\u5883\r\n\r\n### \u540c\u6b65\u7a0b\u5e8f\u8fd0\u884c\u73af\u5883\r\n- i5 cpu\r\n- 32G \u5185\u5b58\r\n- windows10 X64\r\n- WD \u56fa\u6001\u786c\u76d8\r\n- dotnet 4.8\r\n\r\n### \u6d4b\u8bd5\u63a5\u53e3\u4fe1\u606f\r\n- /api/v1/sync/fake\r\n- \u4f7f\u7528 pageSize=2000 \u4ec5\u9650\u540c\u6b65\r\n- \u63a5\u53e3\u8bf7\u6c42\u8017\u65f6\u7ea6 0.2 - 0.3 \u79d2\r\n- \u63a5\u53e3\u6570\u636e\u793a\u4f8b\r\n\r\n```json\r\n{\r\n\t\"errCode\": 0,\r\n\t\"errMsg\": \"success\",\r\n\t\"requestId\": \"73a60094-c0f1-4daf-bc58-4626fbef7a2b\",\r\n\t\"data\": {\r\n\t\t\"pageSize\": 2000,\r\n\t\t\"pageNum\": 1,\r\n\t\t\"totalNum\": 10000,\r\n\t\t\"rows\": [{\r\n\t\t\t\"id\": 1,\r\n\t\t\t\"colString1\": \"Oxqmn5MWCt\",\r\n\t\t\t\"colString2\": \"mzavQncWeNlOlFgUW7HC\",\r\n\t\t\t\"colString3\": \"mvy6K1HU7rdCicPbvvA3rNZcDWPhvV\",\r\n\t\t\t\"colString4\": \"XGsK5NVQHOu4JrmHZ9ZL1iLf0UYpdIvNIzswULzb\",\r\n\t\t\t\"colInt1\": 3931594532918648027,\r\n\t\t\t\"colInt2\": 337586114254574578,\r\n\t\t\t\"colInt3\": 2291922259603323213,\r\n\t\t\t\"colInt4\": 3000562485500051124,\r\n\t\t\t\"colFloat1\": 0.46541339000557547,\r\n\t\t\t\"colFloat2\": 0.6307996439929248,\r\n\t\t\t\"colFloat3\": 0.9278393850101392,\r\n\t\t\t\"colFloat4\": 0.7286866920659677,\r\n\t\t\t\"colSqlTime1\": \"2023-10-20 22:02:07\",\r\n\t\t\t\"colSqlTime2\": \"2023-10-20 22:02:07\",\r\n\t\t\t\"colSqlTime3\": \"2023-10-20 22:02:07\",\r\n\t\t\t\"colSqlTime4\": \"2023-10-20 22:02:07\"\r\n\t\t}]\r\n\t}\r\n}\r\n```\r\n\r\n### \u6d4b\u8bd5\u7ed3\u679c\r\n- \u6570\u636e\u5e93\uff1a\u672c\u5730mysql:5.7 (docker\u73af\u5883)\r\n- pageSize = 2000\r\n- batchSize = 100\r\n\r\n| \u6570\u636e\u91cf(`totalNum`) | Time Spent on `Direct CallAPI` | Time Spent on `Model Sync ` | Time Spent on `DB Sync ` |\r\n| ------------------ | ------------------------------ | --------------------------- | ------------------------ |\r\n| `1w` | 0.324615 | 1.192462 | 6.551163 |\r\n| `10w` | 0.280315 | 7.836467 | 30.583456 |\r\n| `100w` | 0.28313 | 78.528211 | 263.68381 |\r\n\r\n| \u6570\u636e\u91cf(`totalNum`) | Memory Spent on `Model Sync ` | Memory Spent on `DB Sync ` |\r\n| ------------------ | ----------------------------- | -------------------------- |\r\n| `1w` | 3.07421875 | 3.12890625 |\r\n| `10w` | 107.265625 | 15.22265625 |\r\n| `100w` | 695.171875 | 12.56640625 |\r\n\r\n\r\n## \u6ce8\u610f\u4e8b\u9879\r\n\r\n### 1 `OAuth2`\u8eab\u4efd\u8ba4\u8bc1+\u9274\u6743\r\n\r\n`initOauth2ClientCredentials`\u65b9\u6cd5\u53ef\u4ee5\u5b9e\u73b0\u5bf9token\u7684\u81ea\u52a8\u7eed\u7ea6\u4e0e\u7ba1\u7406\uff0c\u6307\u5b9aclient_id\u4e0eclient_secret\u5373\u53ef\r\n\r\n### 2 \u540c\u6b65\u5230`csv`\u6587\u4ef6`SyncToCSV`\r\n\r\n\u4f7f\u7528\u6848\u4f8b\u5728`exampleSyncToCSV()`\u65b9\u6cd5\u4e2d\uff0c\u5f00\u53d1\u8005\u4ec5\u9700\u5728`example_csv.py`\u6587\u4ef6\u4e2d\u8fdb\u884c\u914d\u7f6e\u5373\u53ef\u3002\u4ec5\u9700\u914d\u7f6e`api`\u53c2\u6570\uff0c\u5305\u62ec`url`\u3001`pageSize`\u7b49\uff0c\u8fd8\u9700\u914d\u7f6e\u5199\u5165\u7684\u6587\u4ef6\u540d\u3002\r\n\r\n### 3 \u540c\u6b65\u5230\u6570\u636e\u6a21\u578b`SyncToModel`\r\n\r\n\u56e0\u4e3a\u5199\u5165\u5230`csv`\u6587\u4ef6\u4e2d\u7684\u6570\u636e\u90fd\u662f\u5b57\u7b26\u4e32\u7c7b\u578b\uff0c\u4e3a\u6ee1\u8db3\u5f00\u53d1\u8005\u9700\u6c42\uff0c\u63d0\u4f9b\u540c\u6b65\u6570\u636e\u5230Model\u7684\u529f\u80fd\u3002\u4f7f\u7528\u6848\u4f8b\u5728`exampleSyncToModel()`\u65b9\u6cd5\u4e2d\uff0c\u5f00\u53d1\u8005\u4ec5\u9700\u5728`example_model.py`\u6587\u4ef6\u4e2d\u8fdb\u884c\u914d\u7f6e\u5373\u53ef\u3002\u914d\u7f6e\uff1a`Model`\u7c7b\u578b\uff0c`api`\uff0c\u6307\u5b9a`Model`\u5bf9\u8c61\r\n\r\n### 4 \u540c\u6b65\u5230\u6570\u636e\u5e93`SyncToDB`\r\n\r\n\u4f7f\u7528\u6848\u4f8b\u5728`exampleSyncToDB()`\u65b9\u6cd5\u4e2d\uff0c\u5f00\u53d1\u8005\u4ec5\u9700\u5728`example_db.py`\u6587\u4ef6\u4e2d\u8fdb\u884c\u914d\u7f6e\u5373\u53ef\u3002`ORM\uff1asqlalchemy`\r\n\r\n\u8be5`sdk`\u63d0\u4f9b\u5168\u91cf\u4e0e\u589e\u91cf\u5237\u65b0\u6570\u636e\u5230DB\u4e2d\uff0c\u652f\u6301\u6570\u636e\u5e93`MySQL`\uff08\u5efa\u8bae\uff09\u3001`Oracle`\u3001`PostgrelSQL`\u3001`SQLServer`\r\n\r\n\u76ee\u524d\u6682\u65f6\u4e0d\u652f\u6301 `upsert`\uff0c\u5168\u91cf\u5237\u65b0\u65f6\u5efa\u8bae\u5220\u9664\u539f\u672c\u5b58\u5728\u7684\u8868\u7ed3\u6784\uff0c\u589e\u91cf\u540c\u6b65\u65f6\u5219\u4e0d\u9700\u8981\u3002\u914d\u7f6e\u5982\u4e0b\uff1a\u6570\u636e\u5e93\u3001`api`\u3001\u6620\u5c04\u7c7b\uff0c\u6ce8\u610f\u82e5\u63a5\u53e3\u8fd4\u56de\u7684\u6570\u636e\u5b58\u5728id\uff0c\u4e00\u822c\u65e0\u9700\u6307\u5b9a\u81ea\u589e\r\n\r\n\r\n\r\n",
"bugtrack_url": null,
"license": "",
"summary": "ecnu-openapi-sdk-python",
"version": "1.0.0",
"project_urls": {
"Homepage": "https://github.com/ECNU/ecnu-openapi-sdk-python"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "952b56eecbc80f525722b7af58e629bb78e85bf42a11dc707bf10d426c14fb92",
"md5": "242475954cc15fc39521019e8ebe82d5",
"sha256": "fb0892656f4292c8ea5577bc337047c846f7abb2b5dde5bf258955cd9b149fd8"
},
"downloads": -1,
"filename": "ecnu_openapi_sdk_python-1.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "242475954cc15fc39521019e8ebe82d5",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 13675,
"upload_time": "2023-11-24T04:57:35",
"upload_time_iso_8601": "2023-11-24T04:57:35.094577Z",
"url": "https://files.pythonhosted.org/packages/95/2b/56eecbc80f525722b7af58e629bb78e85bf42a11dc707bf10d426c14fb92/ecnu_openapi_sdk_python-1.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6ee2e11304ce88e8dbb4aec9e22c8ae9f38f4503db6c1697a78a5f829a4db68d",
"md5": "a90d633959bf06ca765dae1e3e97dfc1",
"sha256": "3066c59c7ce62d8c51ab028d4a056db569ad86424af17b5703df4b2f9e4da32c"
},
"downloads": -1,
"filename": "ecnu-openapi-sdk-python-1.0.0.tar.gz",
"has_sig": false,
"md5_digest": "a90d633959bf06ca765dae1e3e97dfc1",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 12747,
"upload_time": "2023-11-24T04:57:38",
"upload_time_iso_8601": "2023-11-24T04:57:38.393111Z",
"url": "https://files.pythonhosted.org/packages/6e/e2/e11304ce88e8dbb4aec9e22c8ae9f38f4503db6c1697a78a5f829a4db68d/ecnu-openapi-sdk-python-1.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-24 04:57:38",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ECNU",
"github_project": "ecnu-openapi-sdk-python",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "oauthlib",
"specs": [
[
"==",
"3.2.0"
]
]
},
{
"name": "psutil",
"specs": [
[
"==",
"5.9.4"
]
]
},
{
"name": "Requests",
"specs": [
[
"==",
"2.31.0"
]
]
},
{
"name": "requests_oauthlib",
"specs": [
[
"==",
"1.3.1"
]
]
},
{
"name": "SQLAlchemy",
"specs": [
[
"==",
"2.0.22"
]
]
}
],
"lcname": "ecnu-openapi-sdk-python"
}
JSON
