david-aws-util


Namedavid-aws-util JSON
Version 1.0.9 PyPI version JSON
download
home_pagehttps://github.com/simon-asis/aws_util
Summarydavid's aws util
upload_time2025-01-13 09:20:51
maintainerNone
docs_urlNone
authordavid
requires_python<3.11,>=3.8
licenseMIT License
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls. 
            # DAVID L. Python Package

[![Workflow : Publish to PyPI](https://github.com/simon-asis/aws_util/actions/workflows/PyPI.yml/badge.svg)](https://github.com/simon-asis/aws_util/actions/workflows/PyPI.yml)

[comment]: <> (This is highly site dependent package. Resources are abstracted into package structure.)

## Usage
Get AWS Session Credentials by sts:assume_role

```python
from aws_util.aws import get_sts_assume_role

sts_credentials = get_sts_assume_role(aws_access_key, aws_secret_key, role_arn, role_session_name='aws_session'):
```
It return above dict.

```python
type: dict
result = 
  {'AccessKeyId': 'ASI...', 
   'SecretAccessKey': 'o8Y...', 
   'SessionToken': 'Fwo...', 
   'Expiration': datetime.datetime(2099, 12, 31, 00, 00, 00, tzinfo=tzutc())}
```


[comment]: <> (Get pandas dataframe from parquet file in hdfs)

[comment]: <> (```python)

[comment]: <> (from pydatafabric.ye import parquet_to_pandas)

[comment]: <> (pandas_df = parquet_to_pandas&#40;hdfs_path&#41;)

[comment]: <> (```)

[comment]: <> (Save pandas dataframe as parquet in hdfs)

[comment]: <> (```python)

[comment]: <> (from pydatafabric.ye import get_spark)

[comment]: <> (from pydatafabric.ye import pandas_to_parquet)

[comment]: <> (spark = get_spark&#40;&#41;)

[comment]: <> (pandas_to_parquet&#40;pandas_df, hdfs_path, spark&#41;  # we need spark for this operation)

[comment]: <> (spark.stop&#40;&#41;)

[comment]: <> (```)

[comment]: <> (Work with spark)

[comment]: <> (```python)

[comment]: <> (from pydatafabric.ye import get_spark)

[comment]: <> (spark = get_spark&#40;&#41;)

[comment]: <> (# do with spark session)

[comment]: <> (spark.stop&#40;&#41;)

[comment]: <> (```)

[comment]: <> (Work with spark-bigquery-connector)

[comment]: <> (```python)

[comment]: <> (# SELECT)

[comment]: <> (from pydatafabric.gcp import bq_table_to_pandas)

[comment]: <> (pandas_df = bq_table_to_pandas&#40;"dataset", "table_name", ["col_1", "col_2"], "2020-01-01", "cust_id is not null"&#41;)

[comment]: <> (# INSERT )

[comment]: <> (from pydatafabric.gcp import pandas_to_bq_table)

[comment]: <> (pandas_to_bq_table&#40;pandas_df, "dataset", "table_name", "2022-02-22"&#41;)

[comment]: <> (```)

[comment]: <> (Send slack message)

[comment]: <> (```python)

[comment]: <> (from pydatafabric.ye import slack_send)

[comment]: <> (text = 'Hello')

[comment]: <> (username = 'airflow')

[comment]: <> (channel = '#leavemealone')

[comment]: <> (slack_send&#40;text=text, username=username, channel=channel&#41;)

[comment]: <> (# Send dataframe as text)

[comment]: <> (df = pd.DataFrame&#40;data={'col1': [1, 2], 'col2': [3, 4]}&#41;)

[comment]: <> (slack_send&#40;text=df, username=username, channel=channel, dataframe=True&#41;)

[comment]: <> (```)

[comment]: <> (Get bigquery client)

[comment]: <> (```python)

[comment]: <> (from pydatafabric.gcp import get_bigquery_client)

[comment]: <> (bq = get_bigquery_client&#40;project="prj"&#41;)

[comment]: <> (bq.query&#40;query&#41;)

[comment]: <> (```)

[comment]: <> (IPython BigQuery Magic)

[comment]: <> (```python)

[comment]: <> (from pydatafabric.gcp import import_bigquery_ipython_magic)

[comment]: <> (import_bigquery_ipython_magic&#40;&#41;)

[comment]: <> (query_params = {)

[comment]: <> (    "p_1": "v_1",)

[comment]: <> (    "dataset": "common_dev",)

[comment]: <> (})

[comment]: <> (```)

[comment]: <> (```python)

[comment]: <> (%% bq --params $query_params)

[comment]: <> (SELECT c_1 )

[comment]: <> (FROM {dataset}.user_logs)

[comment]: <> (WHERE c_1 = @p_1)

[comment]: <> (```)

[comment]: <> (Use NES CLI)

[comment]: <> (```bas)

[comment]: <> (nes input_notebook_url -p k1 v1 -p k2 v2 -p k3 v3)

[comment]: <> (```)

[comment]: <> (Use github util)

[comment]: <> (```python)

[comment]: <> (from pydatafabric.ye import get_github_util)

[comment]: <> (g = get_github_util)

[comment]: <> (# query graphql)

[comment]: <> (res = g.query_gql&#40;graph_ql&#41;)

[comment]: <> (# get file in github repository)

[comment]: <> (byte_object = g.download_from_git&#40;github_url_path&#41;)

[comment]: <> (```)

## Installation

```sh
$ pip install david_aws_util --upgrade
```

If you would like to install submodules for Individual.

```sh
$ pip install david_aws_util[extra] --upgrade
```

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/simon-asis/aws_util",
    "name": "david-aws-util",
    "maintainer": null,
    "docs_url": null,
    "requires_python": "<3.11,>=3.8",
    "maintainer_email": null,
    "keywords": null,
    "author": "david",
    "author_email": "tingyoon@gmail.com",
    "download_url": "https://files.pythonhosted.org/packages/76/c8/1afcf1e00abe8df4413e7df9fc43ffe941122f8686685f9b5f1abcb9930e/david_aws_util-1.0.9.tar.gz",
    "platform": null,
    "description": "# DAVID L. Python Package\n\n[![Workflow : Publish to PyPI](https://github.com/simon-asis/aws_util/actions/workflows/PyPI.yml/badge.svg)](https://github.com/simon-asis/aws_util/actions/workflows/PyPI.yml)\n\n[comment]: <> (This is highly site dependent package. Resources are abstracted into package structure.)\n\n## Usage\nGet AWS Session Credentials by sts:assume_role\n\n```python\nfrom aws_util.aws import get_sts_assume_role\n\nsts_credentials = get_sts_assume_role(aws_access_key, aws_secret_key, role_arn, role_session_name='aws_session'):\n```\nIt return above dict.\n\n```python\ntype: dict\nresult = \n  {'AccessKeyId': 'ASI...', \n   'SecretAccessKey': 'o8Y...', \n   'SessionToken': 'Fwo...', \n   'Expiration': datetime.datetime(2099, 12, 31, 00, 00, 00, tzinfo=tzutc())}\n```\n\n\n[comment]: <> (Get pandas dataframe from parquet file in hdfs)\n\n[comment]: <> (```python)\n\n[comment]: <> (from pydatafabric.ye import parquet_to_pandas)\n\n[comment]: <> (pandas_df = parquet_to_pandas&#40;hdfs_path&#41;)\n\n[comment]: <> (```)\n\n[comment]: <> (Save pandas dataframe as parquet in hdfs)\n\n[comment]: <> (```python)\n\n[comment]: <> (from pydatafabric.ye import get_spark)\n\n[comment]: <> (from pydatafabric.ye import pandas_to_parquet)\n\n[comment]: <> (spark = get_spark&#40;&#41;)\n\n[comment]: <> (pandas_to_parquet&#40;pandas_df, hdfs_path, spark&#41;  # we need spark for this operation)\n\n[comment]: <> (spark.stop&#40;&#41;)\n\n[comment]: <> (```)\n\n[comment]: <> (Work with spark)\n\n[comment]: <> (```python)\n\n[comment]: <> (from pydatafabric.ye import get_spark)\n\n[comment]: <> (spark = get_spark&#40;&#41;)\n\n[comment]: <> (# do with spark session)\n\n[comment]: <> (spark.stop&#40;&#41;)\n\n[comment]: <> (```)\n\n[comment]: <> (Work with spark-bigquery-connector)\n\n[comment]: <> (```python)\n\n[comment]: <> (# SELECT)\n\n[comment]: <> (from pydatafabric.gcp import bq_table_to_pandas)\n\n[comment]: <> (pandas_df = bq_table_to_pandas&#40;\"dataset\", \"table_name\", [\"col_1\", \"col_2\"], \"2020-01-01\", \"cust_id is not null\"&#41;)\n\n[comment]: <> (# INSERT )\n\n[comment]: <> (from pydatafabric.gcp import pandas_to_bq_table)\n\n[comment]: <> (pandas_to_bq_table&#40;pandas_df, \"dataset\", \"table_name\", \"2022-02-22\"&#41;)\n\n[comment]: <> (```)\n\n[comment]: <> (Send slack message)\n\n[comment]: <> (```python)\n\n[comment]: <> (from pydatafabric.ye import slack_send)\n\n[comment]: <> (text = 'Hello')\n\n[comment]: <> (username = 'airflow')\n\n[comment]: <> (channel = '#leavemealone')\n\n[comment]: <> (slack_send&#40;text=text, username=username, channel=channel&#41;)\n\n[comment]: <> (# Send dataframe as text)\n\n[comment]: <> (df = pd.DataFrame&#40;data={'col1': [1, 2], 'col2': [3, 4]}&#41;)\n\n[comment]: <> (slack_send&#40;text=df, username=username, channel=channel, dataframe=True&#41;)\n\n[comment]: <> (```)\n\n[comment]: <> (Get bigquery client)\n\n[comment]: <> (```python)\n\n[comment]: <> (from pydatafabric.gcp import get_bigquery_client)\n\n[comment]: <> (bq = get_bigquery_client&#40;project=\"prj\"&#41;)\n\n[comment]: <> (bq.query&#40;query&#41;)\n\n[comment]: <> (```)\n\n[comment]: <> (IPython BigQuery Magic)\n\n[comment]: <> (```python)\n\n[comment]: <> (from pydatafabric.gcp import import_bigquery_ipython_magic)\n\n[comment]: <> (import_bigquery_ipython_magic&#40;&#41;)\n\n[comment]: <> (query_params = {)\n\n[comment]: <> (    \"p_1\": \"v_1\",)\n\n[comment]: <> (    \"dataset\": \"common_dev\",)\n\n[comment]: <> (})\n\n[comment]: <> (```)\n\n[comment]: <> (```python)\n\n[comment]: <> (%% bq --params $query_params)\n\n[comment]: <> (SELECT c_1 )\n\n[comment]: <> (FROM {dataset}.user_logs)\n\n[comment]: <> (WHERE c_1 = @p_1)\n\n[comment]: <> (```)\n\n[comment]: <> (Use NES CLI)\n\n[comment]: <> (```bas)\n\n[comment]: <> (nes input_notebook_url -p k1 v1 -p k2 v2 -p k3 v3)\n\n[comment]: <> (```)\n\n[comment]: <> (Use github util)\n\n[comment]: <> (```python)\n\n[comment]: <> (from pydatafabric.ye import get_github_util)\n\n[comment]: <> (g = get_github_util)\n\n[comment]: <> (# query graphql)\n\n[comment]: <> (res = g.query_gql&#40;graph_ql&#41;)\n\n[comment]: <> (# get file in github repository)\n\n[comment]: <> (byte_object = g.download_from_git&#40;github_url_path&#41;)\n\n[comment]: <> (```)\n\n## Installation\n\n```sh\n$ pip install david_aws_util --upgrade\n```\n\nIf you would like to install submodules for Individual.\n\n```sh\n$ pip install david_aws_util[extra] --upgrade\n```\n",
    "bugtrack_url": null,
    "license": "MIT License",
    "summary": "david's aws util",
    "version": "1.0.9",
    "project_urls": {
        "Homepage": "https://github.com/simon-asis/aws_util"
    },
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "5c566c024b75f9760653b36adb4b34b8fb052b5857f92d9a2871289dcf41cdf4",
                "md5": "33a57f929ca867a0b3d49abafd197b52",
                "sha256": "41317e1a7258b6bc3463be0efc3259529675c2f800d038da9060eebcf6657f6d"
            },
            "downloads": -1,
            "filename": "david_aws_util-1.0.9-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "33a57f929ca867a0b3d49abafd197b52",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": "<3.11,>=3.8",
            "size": 3579,
            "upload_time": "2025-01-13T09:20:49",
            "upload_time_iso_8601": "2025-01-13T09:20:49.475082Z",
            "url": "https://files.pythonhosted.org/packages/5c/56/6c024b75f9760653b36adb4b34b8fb052b5857f92d9a2871289dcf41cdf4/david_aws_util-1.0.9-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "76c81afcf1e00abe8df4413e7df9fc43ffe941122f8686685f9b5f1abcb9930e",
                "md5": "48f069b6d544f283c62278f6e2bd2eec",
                "sha256": "aceec2b2339b539919843477c646e9347db00f1b358cb9ef1572bed74a277277"
            },
            "downloads": -1,
            "filename": "david_aws_util-1.0.9.tar.gz",
            "has_sig": false,
            "md5_digest": "48f069b6d544f283c62278f6e2bd2eec",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": "<3.11,>=3.8",
            "size": 4150,
            "upload_time": "2025-01-13T09:20:51",
            "upload_time_iso_8601": "2025-01-13T09:20:51.643802Z",
            "url": "https://files.pythonhosted.org/packages/76/c8/1afcf1e00abe8df4413e7df9fc43ffe941122f8686685f9b5f1abcb9930e/david_aws_util-1.0.9.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2025-01-13 09:20:51",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "simon-asis",
    "github_project": "aws_util",
    "github_not_found": true,
    "lcname": "david-aws-util"
}
david
Elapsed time: 0.49415s