| Name | awsglue-rclone JSON |
| Version |
0.0.2
JSON |
| download |
| home_page | |
| Summary | RClone Wrapper for AWS Glue Python Shell |
| upload_time | 2024-02-10 14:09:04 |
| maintainer | |
| docs_url | None |
| author | Wang Zhiwei |
| requires_python | >=3.7 |
| license | |
| keywords |
awsglue
rclone
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# awsglue-rclone
awsglue-rclone is a wrapper for RClone designed for use with AWS Glue Python Shell, rclone everything within Glue!
## Installation
Install awsglue-rclone from source:
```shell
$ git clone https://github.com/zhiweio/awsglue-rclone.git && cd awsglue-rclone/
$ python3 setup.py install
```
## Usage
Install `rclone` on local machine (Glue shell)
```python
from awsglue_rclone import RClone
rclone = RClone()
```
Build AWS S3 remote config
```python
from awsglue_rclone.config import S3Config
cn_s3_conf = S3Config(
alias="cn",
properties={
"region": "cn-north-1",
"access_key_id": "xxx",
"secret_access_key": "xxx",
},
)
```
Create `rclone.config` file
```python
rclone.create_remote(cn_s3_conf)
```
Build from Secrets Manager
secrets: `rclone/aws_s3/us_west`
```json
{
"region": "us-west-1",
"access_key_id": "xxx",
"secret_access_key": "xxx",
}
```
```python
us_s3_conf = S3Config(
alias="us",
secret="rclone/aws_s3/us_west",
)
rclone.create_remote(us_s3_conf, append=True)
```
List remote files
```python
ret = rclone.ls("cn:bucket/yourfolder")
{
"code": 0,
"out": " 2079 README.md\n",
"error": ""
}
```
Print verbose
```python
ret = rclone.ls("cn:bucket/yourfolder", debug=True)
{
"code": 0,
"out": " 2079 README.md\n",
"error": "2024/02/08 17:00:09 DEBUG : rclone: Version \"v1.64.2\" starting with parameters [\"rclone\" \"--config=/tmp/opt/.config/rclone.conf\" \"-vv\" \"ls\" \"cn:bucket/yourfolder/\"]\n2024/02/08 17:00:09 DEBUG : Creating backend with remote \"cn:bucket/yourfolder/\"\n2024/02/08 17:00:09 DEBUG : Using config file from \"/tmp/opt/.config/rclone.conf\"\n2024/02/08 17:00:09 DEBUG : fs cache: renaming cache item \"cn:bucket/yourfolder/\" to be canonical \"cn:bucket/yourfolder\"\n2024/02/08 17:00:09 DEBUG : 7 go routines active\n"
}
```
Copy files between two remotes, set `nio=True` to save command output (stdout/stderr) into temporary files
```python
rclone.copy("cn:bucket/yourfolder", "us:bucket/yourfolder", nio=True)
{
"code": 0,
"out": "/var/folders/bj/h5g9fnl93dg6tllvsp4hr73r0000gn/T/tmpuhom0swf.stdout",
"error": "/var/folders/bj/h5g9fnl93dg6tllvsp4hr73r0000gn/T/tmp4jf_9nok.stderr"
}
```
Build other remote Config providers
```python
from awsglue_rclone.config import Config, RemoteTypes
# provider for onedrive storage
class OneDriveConfig(Config):
def create(self):
token = {
"access_token": self.properties["access_token"],
"token_type": self.properties.get("token_type", "Bear"),
"refresh_token": self.properties["refresh_token"],
"expiry": "2018-08-26T22:39:52.486512262+08:00",
}
token = json.dumps(token)
drive_id = self.properties["drive_id"]
drive_type = "business"
return textwrap.dedent(
f"""
[{self.alias}]
type = {RemoteTypes.onedrive}
token = {token}
drive_id = {drive_id}
drive_type = {drive_type}
"""
)
```
## Authors
- [@zhiweio](https://www.github.com/zhiweio)
## License
[GPL-3.0](https://choosealicense.com/licenses/gpl-3.0/)
Raw data
{
"_id": null,
"home_page": "",
"name": "awsglue-rclone",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "awsglue rclone",
"author": "Wang Zhiwei",
"author_email": "noparking188@gmail.com",
"download_url": "",
"platform": null,
"description": "# awsglue-rclone\n\nawsglue-rclone is a wrapper for RClone designed for use with AWS Glue Python Shell, rclone everything within Glue!\n\n## Installation\n\nInstall awsglue-rclone from source:\n\n```shell\n$ git clone https://github.com/zhiweio/awsglue-rclone.git && cd awsglue-rclone/\n$ python3 setup.py install\n```\n\n\n## Usage\n\nInstall `rclone` on local machine (Glue shell)\n\n```python\nfrom awsglue_rclone import RClone\n\nrclone = RClone()\n\n```\n\nBuild AWS S3 remote config\n\n```python\nfrom awsglue_rclone.config import S3Config\n\ncn_s3_conf = S3Config(\n alias=\"cn\",\n properties={\n \"region\": \"cn-north-1\",\n \"access_key_id\": \"xxx\",\n \"secret_access_key\": \"xxx\",\n },\n)\n```\n\nCreate `rclone.config` file\n\n```python\nrclone.create_remote(cn_s3_conf)\n```\n\nBuild from Secrets Manager\n\nsecrets: `rclone/aws_s3/us_west`\n```json\n{\n \"region\": \"us-west-1\",\n \"access_key_id\": \"xxx\",\n \"secret_access_key\": \"xxx\",\n}\n```\n\n```python\nus_s3_conf = S3Config(\n alias=\"us\",\n secret=\"rclone/aws_s3/us_west\",\n)\nrclone.create_remote(us_s3_conf, append=True)\n```\n\nList remote files\n\n```python\nret = rclone.ls(\"cn:bucket/yourfolder\")\n\n{\n \"code\": 0,\n \"out\": \" 2079 README.md\\n\",\n \"error\": \"\"\n}\n```\n\nPrint verbose\n\n```python\nret = rclone.ls(\"cn:bucket/yourfolder\", debug=True)\n\n{\n \"code\": 0,\n \"out\": \" 2079 README.md\\n\",\n \"error\": \"2024/02/08 17:00:09 DEBUG : rclone: Version \\\"v1.64.2\\\" starting with parameters [\\\"rclone\\\" \\\"--config=/tmp/opt/.config/rclone.conf\\\" \\\"-vv\\\" \\\"ls\\\" \\\"cn:bucket/yourfolder/\\\"]\\n2024/02/08 17:00:09 DEBUG : Creating backend with remote \\\"cn:bucket/yourfolder/\\\"\\n2024/02/08 17:00:09 DEBUG : Using config file from \\\"/tmp/opt/.config/rclone.conf\\\"\\n2024/02/08 17:00:09 DEBUG : fs cache: renaming cache item \\\"cn:bucket/yourfolder/\\\" to be canonical \\\"cn:bucket/yourfolder\\\"\\n2024/02/08 17:00:09 DEBUG : 7 go routines active\\n\"\n}\n```\n\nCopy files between two remotes, set `nio=True` to save command output (stdout/stderr) into temporary files\n```python\nrclone.copy(\"cn:bucket/yourfolder\", \"us:bucket/yourfolder\", nio=True)\n\n{\n \"code\": 0,\n \"out\": \"/var/folders/bj/h5g9fnl93dg6tllvsp4hr73r0000gn/T/tmpuhom0swf.stdout\",\n \"error\": \"/var/folders/bj/h5g9fnl93dg6tllvsp4hr73r0000gn/T/tmp4jf_9nok.stderr\"\n}\n```\n\nBuild other remote Config providers\n\n```python\nfrom awsglue_rclone.config import Config, RemoteTypes\n\n# provider for onedrive storage\nclass OneDriveConfig(Config):\n def create(self):\n token = {\n \"access_token\": self.properties[\"access_token\"],\n \"token_type\": self.properties.get(\"token_type\", \"Bear\"),\n \"refresh_token\": self.properties[\"refresh_token\"],\n \"expiry\": \"2018-08-26T22:39:52.486512262+08:00\",\n }\n token = json.dumps(token)\n drive_id = self.properties[\"drive_id\"]\n drive_type = \"business\"\n return textwrap.dedent(\n f\"\"\"\n[{self.alias}]\ntype = {RemoteTypes.onedrive}\ntoken = {token}\ndrive_id = {drive_id}\ndrive_type = {drive_type}\n \"\"\"\n )\n```\n\n## Authors\n\n- [@zhiweio](https://www.github.com/zhiweio)\n\n\n## License\n\n[GPL-3.0](https://choosealicense.com/licenses/gpl-3.0/)\n\n\n\n",
"bugtrack_url": null,
"license": "",
"summary": "RClone Wrapper for AWS Glue Python Shell",
"version": "0.0.2",
"project_urls": null,
"split_keywords": [
"awsglue",
"rclone"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "67c150750befa237737d18be2f001beb0953990c890dd155c249b1f589485198",
"md5": "1b67f2914f5950a122c1c3670ea489f0",
"sha256": "fa787a3d4e1d48de1dc049248968c6589ef810c634ee7cd40ca047a46d102edc"
},
"downloads": -1,
"filename": "awsglue_rclone-0.0.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "1b67f2914f5950a122c1c3670ea489f0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 19948,
"upload_time": "2024-02-10T14:09:04",
"upload_time_iso_8601": "2024-02-10T14:09:04.281316Z",
"url": "https://files.pythonhosted.org/packages/67/c1/50750befa237737d18be2f001beb0953990c890dd155c249b1f589485198/awsglue_rclone-0.0.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-10 14:09:04",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "awsglue-rclone"
}
