[](https://github.com/softwrdai/softwrdwrangler/actions/workflows/publish_2_pypi.yml)
# softwrdwrangler
`softwrdwrangler` is a Python package designed to simplify and streamline operations on various AWS resources, primarily focusing on S3. It offers utilities for reading and writing different file formats and managing S3 objects efficiently.
## Pre-requisites
You need to have AWS CLI installed and configured with the necessary permissions to interact with AWS services.
### Install AWS CLI
Follow the installation guide here: [AWS CLI Installation](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)
Once installed, configure it by running:
```bash
aws configure
```
## Installation
Install the package via pip:
```bash
pip install softwrdwrangler
```
## Usage
### S3 Operations
#### S3 Read and Write Pickle
```python
import softwrdwrangler as swr
# Reading a pickle file from S3
data = swr.read_pickle('s3://bucket-name/path/to/file.pkl')
# Writing a DataFrame to a pickle file in S3
swr.write_pickle(data, 's3://bucket-name/path/to/file.pkl')
```
#### S3 Read and Write JSON
```python
import softwrdwrangler as swr
# Writing a dictionary to a JSON file in S3
data = {'a': 1, 'b': 2}
swr.write_json(data, 's3://bucket-name/path/to/file.json')
# Reading a JSON file from S3
print(swr.read_json('s3://bucket-name/path/to/file.json'))
```
#### S3 Read and Write CSV
```python
import softwrdwrangler as swr
# Reading a CSV file from S3
data = swr.read_csv('s3://bucket-name/path/to/file.csv')
# Writing a DataFrame to a CSV file in S3
swr.write_csv(data, 's3://bucket-name/path/to/file.csv')
```
#### S3 Read and Write GeoJSON
```python
import softwrdwrangler as swr
# Reading a GeoJSON file from S3
data = swr.read_geojson('s3://bucket-name/path/to/file.geojson')
# Writing a dictionary to a GeoJSON file in S3
swr.write_geojson(data, 's3://bucket-name/path/to/file.geojson')
```
### Additional S3 Utilities
#### List Files
List all files in a specified S3 bucket and prefix.
```python
import softwrdwrangler as swr
files = swr.list_files('s3://bucket-name/path/to/folder/')
print(files)
```
#### Delete File
Delete a specific file from S3.
```python
import softwrdwrangler as swr
swr.delete_file('s3://bucket-name/path/to/file')
```
#### Upload Local File to S3
Upload a file from local storage to S3.
```python
import softwrdwrangler as swr
swr.upload_file('/local/path/to/file', 's3://bucket-name/path/to/file')
```
#### Download S3 File to Local Storage
Download a file from S3 to local storage.
```python
import softwrdwrangler as swr
swr.download_file('s3://bucket-name/path/to/file', '/local/path/to/file')
```
#### Copy File within S3
Copy a file from one S3 location to another.
```python
import softwrdwrangler as swr
swr.copy_file('s3://source-bucket/path/to/file', 's3://destination-bucket/path/to/file')
```
#### Generate Pre-signed URL
Generate a pre-signed URL to grant temporary access to an S3 object.
```python
import softwrdwrangler as swr
url = swr.generate_presigned_url('s3://bucket-name/path/to/file', expiration=3600)
print(url)
```
### Advanced Operations
#### Get Latest Date-stamped Folder (ds_nodash)
Retrieve the latest date-stamped folder (e.g., `20221201`) from a given S3 prefix, with an optional skip count.
```python
import softwrdwrangler as swr
latest_date = swr.get_latest_ds_nodash('s3://bucket-name/path', skip=1)
if latest_date:
print(f"Latest ds_nodash after skipping 1: {latest_date}")
else:
print("No valid date folders found or error occurred.")
```
## License
`softwrdwrangler` is licensed under the Apache Software License.
```
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
```
---
Raw data
{
"_id": null,
"home_page": "https://github.com/softwrdai/softwrdwrangler",
"name": "softwrdwrangler",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "aws, s3, pandas, geojson, pickle, data wrangling, secrets manager",
"author": "Md Robiuddin",
"author_email": "mrrobi040@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/0f/52/51dab9deba78b6b25464fb5e0c79994aa07a41b4e3b291cc0bcbcfdf0484/softwrdwrangler-0.1.3.tar.gz",
"platform": null,
"description": "[](https://github.com/softwrdai/softwrdwrangler/actions/workflows/publish_2_pypi.yml)\n\n# softwrdwrangler\n\n`softwrdwrangler` is a Python package designed to simplify and streamline operations on various AWS resources, primarily focusing on S3. It offers utilities for reading and writing different file formats and managing S3 objects efficiently.\n\n## Pre-requisites\n\nYou need to have AWS CLI installed and configured with the necessary permissions to interact with AWS services.\n\n### Install AWS CLI\nFollow the installation guide here: [AWS CLI Installation](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)\n\nOnce installed, configure it by running:\n```bash\naws configure\n```\n\n## Installation\n\nInstall the package via pip:\n```bash\npip install softwrdwrangler\n```\n\n## Usage\n\n### S3 Operations\n\n#### S3 Read and Write Pickle\n```python\nimport softwrdwrangler as swr\n\n# Reading a pickle file from S3\ndata = swr.read_pickle('s3://bucket-name/path/to/file.pkl')\n\n# Writing a DataFrame to a pickle file in S3\nswr.write_pickle(data, 's3://bucket-name/path/to/file.pkl')\n```\n\n#### S3 Read and Write JSON\n```python\nimport softwrdwrangler as swr\n\n# Writing a dictionary to a JSON file in S3\ndata = {'a': 1, 'b': 2}\nswr.write_json(data, 's3://bucket-name/path/to/file.json')\n\n# Reading a JSON file from S3\nprint(swr.read_json('s3://bucket-name/path/to/file.json'))\n```\n\n#### S3 Read and Write CSV\n```python\nimport softwrdwrangler as swr\n\n# Reading a CSV file from S3\ndata = swr.read_csv('s3://bucket-name/path/to/file.csv')\n\n# Writing a DataFrame to a CSV file in S3\nswr.write_csv(data, 's3://bucket-name/path/to/file.csv')\n```\n\n#### S3 Read and Write GeoJSON\n```python\nimport softwrdwrangler as swr\n\n# Reading a GeoJSON file from S3\ndata = swr.read_geojson('s3://bucket-name/path/to/file.geojson')\n\n# Writing a dictionary to a GeoJSON file in S3\nswr.write_geojson(data, 's3://bucket-name/path/to/file.geojson')\n```\n\n### Additional S3 Utilities\n\n#### List Files\nList all files in a specified S3 bucket and prefix.\n```python\nimport softwrdwrangler as swr\n\nfiles = swr.list_files('s3://bucket-name/path/to/folder/')\nprint(files)\n```\n\n#### Delete File\nDelete a specific file from S3.\n```python\nimport softwrdwrangler as swr\n\nswr.delete_file('s3://bucket-name/path/to/file')\n```\n\n#### Upload Local File to S3\nUpload a file from local storage to S3.\n```python\nimport softwrdwrangler as swr\n\nswr.upload_file('/local/path/to/file', 's3://bucket-name/path/to/file')\n```\n\n#### Download S3 File to Local Storage\nDownload a file from S3 to local storage.\n```python\nimport softwrdwrangler as swr\n\nswr.download_file('s3://bucket-name/path/to/file', '/local/path/to/file')\n```\n\n#### Copy File within S3\nCopy a file from one S3 location to another.\n```python\nimport softwrdwrangler as swr\n\nswr.copy_file('s3://source-bucket/path/to/file', 's3://destination-bucket/path/to/file')\n```\n\n#### Generate Pre-signed URL\nGenerate a pre-signed URL to grant temporary access to an S3 object.\n```python\nimport softwrdwrangler as swr\n\nurl = swr.generate_presigned_url('s3://bucket-name/path/to/file', expiration=3600)\nprint(url)\n```\n\n### Advanced Operations\n\n#### Get Latest Date-stamped Folder (ds_nodash)\nRetrieve the latest date-stamped folder (e.g., `20221201`) from a given S3 prefix, with an optional skip count.\n```python\nimport softwrdwrangler as swr\n\nlatest_date = swr.get_latest_ds_nodash('s3://bucket-name/path', skip=1)\nif latest_date:\n print(f\"Latest ds_nodash after skipping 1: {latest_date}\")\nelse:\n print(\"No valid date folders found or error occurred.\")\n```\n\n## License\n\n`softwrdwrangler` is licensed under the Apache Software License.\n\n```\nApache License\nVersion 2.0, January 2004\nhttp://www.apache.org/licenses/\n```\n\n---\n",
"bugtrack_url": null,
"license": null,
"summary": "A package that provides utilities for data wrangling with AWS S3, pandas, and geojson.",
"version": "0.1.3",
"project_urls": {
"Homepage": "https://github.com/softwrdai/softwrdwrangler"
},
"split_keywords": [
"aws",
" s3",
" pandas",
" geojson",
" pickle",
" data wrangling",
" secrets manager"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "085797239a3c02d056e517a3f62f657461290f7a7bcb983390e02a7b23d403a7",
"md5": "dab2d7e8ffab96cc761dbad534f9b11c",
"sha256": "57dbba0bf20a71d324fea3ea777f47a1198cab8696937a8c9f0b72774218d44b"
},
"downloads": -1,
"filename": "softwrdwrangler-0.1.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "dab2d7e8ffab96cc761dbad534f9b11c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 9766,
"upload_time": "2024-11-04T09:02:30",
"upload_time_iso_8601": "2024-11-04T09:02:30.046084Z",
"url": "https://files.pythonhosted.org/packages/08/57/97239a3c02d056e517a3f62f657461290f7a7bcb983390e02a7b23d403a7/softwrdwrangler-0.1.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "0f5251dab9deba78b6b25464fb5e0c79994aa07a41b4e3b291cc0bcbcfdf0484",
"md5": "311234959c188ccab0d97df675c502c0",
"sha256": "995b0226ce7af94a01e676a135cc9860d43dcbcac41cb5eed5d8b7eff562f1cf"
},
"downloads": -1,
"filename": "softwrdwrangler-0.1.3.tar.gz",
"has_sig": false,
"md5_digest": "311234959c188ccab0d97df675c502c0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 7682,
"upload_time": "2024-11-04T09:02:31",
"upload_time_iso_8601": "2024-11-04T09:02:31.539853Z",
"url": "https://files.pythonhosted.org/packages/0f/52/51dab9deba78b6b25464fb5e0c79994aa07a41b4e3b291cc0bcbcfdf0484/softwrdwrangler-0.1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-04 09:02:31",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "softwrdai",
"github_project": "softwrdwrangler",
"github_not_found": true,
"lcname": "softwrdwrangler"
}
JSON
