## import 방법
- from echoss_s3handler import s3_handler
custom yolov7과 image_utils의 함수 사용시 S3에 있는 데이터를 사용하고자 하여 만든 Class
크게 두가지의 class가 존재함
1. client를 활용한 clas
2. resource를 활용한 class
# Client Class
내부 주요 함수
```
# s3 config 파일 읽기
with open("../s3 config file path/ai_solution_dataset_test.yaml") as f:
data_dict = yaml.load(f, Loader=yaml.SafeLoader)
# class 할당
s3_client = S3ClientHandler(data_dict)
s3_client.get_bucket_list()
- 현재 config 파일에 해당하는 접속 계정에 있는 bucket들의 정보를 출력해준다.(버킷명, 버킷 생성날짜 등)
s3_client.get_object_info()
- parameter로 입력된 object의 마지막 수정일시(YYYY-MM-DD HH:MM:SS 꼴)와 s3link를 출력해준다.
s3_client.get_object_list()
- parameter로 입력된 prefix와 after_ts(기준이되는 시간 입렵 포맷:"2023-02-28 00:00:00")를 확인하여 object들의 path를 list로 반환한다.
```
# Resource Class
내부 주요 함수
```
# s3 config 파일 읽기
with open("../s3 config file path/ai_solution_dataset_test.yaml") as f:
data_dict = yaml.load(f, Loader=yaml.SafeLoader)
# class 할당
s3_resource = S3ResourceHandler(data_dict)
s3_resource.get_bucket_list()
- 현재 config 파일에 해당하는 접속 계정에 있는 bucket들의 정보를 출력해준다.(버킷명, 버킷 생성날짜 등)
- data_info를 True로 하게 되면 출력이 버킷명이 key값, 마지막 생성일자를 vlaue로 하는 Dict형태로 출력되며,
false로 하면 버킷의 목록만 list형태로 출력.
s3_resource.get_object_info()
- parameter로 입력된 object의 마지막 수정일시(YYYY-MM-DD HH:MM:SS 꼴)와 s3link를 출력해준다.
s3_resource.get_object_list()
- parameter로 입력된 prefix와 after_ts(기준이되는 시간 입렵 포맷:"2023-02-28 00:00:00")를 확인하여 object들의 path를 list로 반환한다.
s3_resource.download_object()
- S3에 있는 object를 local에 다운로드하기 위한 함수
s3_resource.upload_object()
- locla에 있는 데이터를 S3에 업로드하기 위한 함수
s3_resource.put_object()
- 메모리상으로 저장되어있는 json혹은 txt 데이터를 S3 object로 업로드하는데 사용하는 함수
- 이미지도 업로드할 수 있으나 이때에는 아래와 같이 이미지를 PIL, io라이브러리를 이용하여 가공한 뒤 업로드를 해야합니다.
img0 = Image.fromarray(im0, mode='RGB')
out_img = io.BytesIO()
img0.save(out_img, format='png')
out_img.seek(0)
s3_resource.s3tos3_put_object()
- 서로 다른 S3계정간의 데이터를 local에 따로 다운로드 한 뒤 업로드 하는 것이 아닌 메모리상으로 불러 업로드하는 함수
s3_resource.read_file()
- S3에 있는 파일을 메모리로 읽기 위한 함수
s3_resource.read_image()
- S3에 있는 이미지 파일을 메모리로 읽는 함수
- 이후 사용되는 이미지 형식에 따라 옵션에 "cv2", "PIL"을 선택하여 사용 가능
```
```python
```
Raw data
{
"_id": null,
"home_page": "",
"name": "echoss-s3handler",
"maintainer": "",
"docs_url": null,
"requires_python": ">3.7",
"maintainer_email": "",
"keywords": "echoss,echoss_s3handler,s3handler,s3_handler",
"author": "incheolshin",
"author_email": "incheolshin@12cm.co.kr",
"download_url": "https://files.pythonhosted.org/packages/22/25/4023e7b8a891a697a64330563eabde2012eab87fca18ed4a1d8fa6b95e58/echoss_s3handler-0.0.5.tar.gz",
"platform": null,
"description": "## import \ubc29\ubc95\n - from echoss_s3handler import s3_handler\n\ncustom yolov7\uacfc image_utils\uc758 \ud568\uc218 \uc0ac\uc6a9\uc2dc S3\uc5d0 \uc788\ub294 \ub370\uc774\ud130\ub97c \uc0ac\uc6a9\ud558\uace0\uc790 \ud558\uc5ec \ub9cc\ub4e0 Class\n\n\ud06c\uac8c \ub450\uac00\uc9c0\uc758 class\uac00 \uc874\uc7ac\ud568\n1. client\ub97c \ud65c\uc6a9\ud55c clas\n2. resource\ub97c \ud65c\uc6a9\ud55c class\n\n# Client Class\n\ub0b4\ubd80 \uc8fc\uc694 \ud568\uc218\n```\n # s3 config \ud30c\uc77c \uc77d\uae30\n with open(\"../s3 config file path/ai_solution_dataset_test.yaml\") as f:\n data_dict = yaml.load(f, Loader=yaml.SafeLoader)\n # class \ud560\ub2f9\n s3_client = S3ClientHandler(data_dict)\n \n \n s3_client.get_bucket_list()\n - \ud604\uc7ac config \ud30c\uc77c\uc5d0 \ud574\ub2f9\ud558\ub294 \uc811\uc18d \uacc4\uc815\uc5d0 \uc788\ub294 bucket\ub4e4\uc758 \uc815\ubcf4\ub97c \ucd9c\ub825\ud574\uc900\ub2e4.(\ubc84\ud0b7\uba85, \ubc84\ud0b7 \uc0dd\uc131\ub0a0\uc9dc \ub4f1)\n \n s3_client.get_object_info()\n - parameter\ub85c \uc785\ub825\ub41c object\uc758 \ub9c8\uc9c0\ub9c9 \uc218\uc815\uc77c\uc2dc(YYYY-MM-DD HH:MM:SS \uaf34)\uc640 s3link\ub97c \ucd9c\ub825\ud574\uc900\ub2e4.\n \n s3_client.get_object_list()\n - parameter\ub85c \uc785\ub825\ub41c prefix\uc640 after_ts(\uae30\uc900\uc774\ub418\ub294 \uc2dc\uac04 \uc785\ub835 \ud3ec\ub9f7:\"2023-02-28 00:00:00\")\ub97c \ud655\uc778\ud558\uc5ec object\ub4e4\uc758 path\ub97c list\ub85c \ubc18\ud658\ud55c\ub2e4.\n```\n\n# Resource Class\n\ub0b4\ubd80 \uc8fc\uc694 \ud568\uc218\n```\n # s3 config \ud30c\uc77c \uc77d\uae30\n with open(\"../s3 config file path/ai_solution_dataset_test.yaml\") as f:\n data_dict = yaml.load(f, Loader=yaml.SafeLoader)\n # class \ud560\ub2f9\n s3_resource = S3ResourceHandler(data_dict)\n \n \n s3_resource.get_bucket_list()\n - \ud604\uc7ac config \ud30c\uc77c\uc5d0 \ud574\ub2f9\ud558\ub294 \uc811\uc18d \uacc4\uc815\uc5d0 \uc788\ub294 bucket\ub4e4\uc758 \uc815\ubcf4\ub97c \ucd9c\ub825\ud574\uc900\ub2e4.(\ubc84\ud0b7\uba85, \ubc84\ud0b7 \uc0dd\uc131\ub0a0\uc9dc \ub4f1)\n - data_info\ub97c True\ub85c \ud558\uac8c \ub418\uba74 \ucd9c\ub825\uc774 \ubc84\ud0b7\uba85\uc774 key\uac12, \ub9c8\uc9c0\ub9c9 \uc0dd\uc131\uc77c\uc790\ub97c vlaue\ub85c \ud558\ub294 Dict\ud615\ud0dc\ub85c \ucd9c\ub825\ub418\uba70,\n false\ub85c \ud558\uba74 \ubc84\ud0b7\uc758 \ubaa9\ub85d\ub9cc list\ud615\ud0dc\ub85c \ucd9c\ub825.\n \n s3_resource.get_object_info()\n - parameter\ub85c \uc785\ub825\ub41c object\uc758 \ub9c8\uc9c0\ub9c9 \uc218\uc815\uc77c\uc2dc(YYYY-MM-DD HH:MM:SS \uaf34)\uc640 s3link\ub97c \ucd9c\ub825\ud574\uc900\ub2e4.\n \n s3_resource.get_object_list()\n - parameter\ub85c \uc785\ub825\ub41c prefix\uc640 after_ts(\uae30\uc900\uc774\ub418\ub294 \uc2dc\uac04 \uc785\ub835 \ud3ec\ub9f7:\"2023-02-28 00:00:00\")\ub97c \ud655\uc778\ud558\uc5ec object\ub4e4\uc758 path\ub97c list\ub85c \ubc18\ud658\ud55c\ub2e4.\n \n s3_resource.download_object()\n - S3\uc5d0 \uc788\ub294 object\ub97c local\uc5d0 \ub2e4\uc6b4\ub85c\ub4dc\ud558\uae30 \uc704\ud55c \ud568\uc218\n \n s3_resource.upload_object()\n - locla\uc5d0 \uc788\ub294 \ub370\uc774\ud130\ub97c S3\uc5d0 \uc5c5\ub85c\ub4dc\ud558\uae30 \uc704\ud55c \ud568\uc218\n \n s3_resource.put_object()\n - \uba54\ubaa8\ub9ac\uc0c1\uc73c\ub85c \uc800\uc7a5\ub418\uc5b4\uc788\ub294 json\ud639\uc740 txt \ub370\uc774\ud130\ub97c S3 object\ub85c \uc5c5\ub85c\ub4dc\ud558\ub294\ub370 \uc0ac\uc6a9\ud558\ub294 \ud568\uc218\n - \uc774\ubbf8\uc9c0\ub3c4 \uc5c5\ub85c\ub4dc\ud560 \uc218 \uc788\uc73c\ub098 \uc774\ub54c\uc5d0\ub294 \uc544\ub798\uc640 \uac19\uc774 \uc774\ubbf8\uc9c0\ub97c PIL, io\ub77c\uc774\ube0c\ub7ec\ub9ac\ub97c \uc774\uc6a9\ud558\uc5ec \uac00\uacf5\ud55c \ub4a4 \uc5c5\ub85c\ub4dc\ub97c \ud574\uc57c\ud569\ub2c8\ub2e4.\n img0 = Image.fromarray(im0, mode='RGB')\n out_img = io.BytesIO()\n img0.save(out_img, format='png')\n out_img.seek(0)\n \n s3_resource.s3tos3_put_object()\n - \uc11c\ub85c \ub2e4\ub978 S3\uacc4\uc815\uac04\uc758 \ub370\uc774\ud130\ub97c local\uc5d0 \ub530\ub85c \ub2e4\uc6b4\ub85c\ub4dc \ud55c \ub4a4 \uc5c5\ub85c\ub4dc \ud558\ub294 \uac83\uc774 \uc544\ub2cc \uba54\ubaa8\ub9ac\uc0c1\uc73c\ub85c \ubd88\ub7ec \uc5c5\ub85c\ub4dc\ud558\ub294 \ud568\uc218\n \n s3_resource.read_file()\n - S3\uc5d0 \uc788\ub294 \ud30c\uc77c\uc744 \uba54\ubaa8\ub9ac\ub85c \uc77d\uae30 \uc704\ud55c \ud568\uc218\n \n s3_resource.read_image()\n - S3\uc5d0 \uc788\ub294 \uc774\ubbf8\uc9c0 \ud30c\uc77c\uc744 \uba54\ubaa8\ub9ac\ub85c \uc77d\ub294 \ud568\uc218\n - \uc774\ud6c4 \uc0ac\uc6a9\ub418\ub294 \uc774\ubbf8\uc9c0 \ud615\uc2dd\uc5d0 \ub530\ub77c \uc635\uc158\uc5d0 \"cv2\", \"PIL\"\uc744 \uc120\ud0dd\ud558\uc5ec \uc0ac\uc6a9 \uac00\ub2a5\n \n```\n\n```python\n\n```\n",
"bugtrack_url": null,
"license": "",
"summary": "echoss AI Bigdata Solution - S3 handler",
"version": "0.0.5",
"project_urls": null,
"split_keywords": [
"echoss",
"echoss_s3handler",
"s3handler",
"s3_handler"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "64f92350ed1b211a35b3bb2f388cdf275deae725eb22d722ce84a1ace0953a05",
"md5": "8630034b5d063e9c9a97d444f7acc31b",
"sha256": "dee0ef0a970526c7c672e3cb08accc2287ffc877e095f373fb85fd7bee8d89bb"
},
"downloads": -1,
"filename": "echoss_s3handler-0.0.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8630034b5d063e9c9a97d444f7acc31b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">3.7",
"size": 7980,
"upload_time": "2024-01-08T07:00:20",
"upload_time_iso_8601": "2024-01-08T07:00:20.052046Z",
"url": "https://files.pythonhosted.org/packages/64/f9/2350ed1b211a35b3bb2f388cdf275deae725eb22d722ce84a1ace0953a05/echoss_s3handler-0.0.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "22254023e7b8a891a697a64330563eabde2012eab87fca18ed4a1d8fa6b95e58",
"md5": "804c94000859249ccb6b6873a40161cb",
"sha256": "3f69a7a1eed5a03cf6398256c10a4ba108aebc65f979a67ec45ab515ee51c720"
},
"downloads": -1,
"filename": "echoss_s3handler-0.0.5.tar.gz",
"has_sig": false,
"md5_digest": "804c94000859249ccb6b6873a40161cb",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">3.7",
"size": 7496,
"upload_time": "2024-01-08T07:00:21",
"upload_time_iso_8601": "2024-01-08T07:00:21.992163Z",
"url": "https://files.pythonhosted.org/packages/22/25/4023e7b8a891a697a64330563eabde2012eab87fca18ed4a1d8fa6b95e58/echoss_s3handler-0.0.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-08 07:00:21",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "echoss-s3handler"
}
JSON