# Klondike
<img src="https://upload.wikimedia.org/wikipedia/en/d/d5/Klondike_logo.svg">
Klondike offers a lightweight API to read and write data to Google BigQuery using Polars DataFrames.
## Installation
### Installing Klondike
Install at the command line
```
pip install klondike
```
### Installing Rust
Since Polars leverages Rust speedups, you need to have Rust installed in your environment as well. See the Rust installation guide [here](https://www.rust-lang.org/tools/install).
## Usage
In this demo we'll connect to Google BigQuery, read data, transform it, and write it back to the data warehouse.
First, connect to the BigQuery warehouse by supplying the `BigQueryConnector()` object with the relative path to your service account credentials.
```
from klondike import BigQueryConnector
# Connect Klondike to Google BigQuery
bq = BigQueryConnector(
app_creds="dev/nba-player-service.json"
)
```
Next, supply the object with a SQL query in the `read_dataframe()` function to redner a `DataFrame` object:
```
# Write some valid SQL
sql = """
SELECT
*
FROM nba_dbt.cln_player__averages
ORDER BY avg_points DESC
"""
# Pull BigQuery data into a Polars DataFrame
nyk = bq.read_dataframe(sql=sql)
```
Now that your data is pulled into a local instance, you can clean and transform it using standard Polars functionality - [see the docs](https://docs.pola.rs/py-polars/html/reference/dataframe/index.html) for more information.
```
# Perform some transformations
key_metrics = [
"avg_offensive_rating",
"avg_defensive_rating",
"avg_effective_field_goal_percentage"
]
summary_stats = nyk[key_metrics].describe()
```
Finally, push your transformed data back to the BigQuery warehouse using the `write_dataframe()` function:
```
# Write back to BigQuery
bq.write_dataframe(
df=summary_stats,
table_name="nba_dbt.summary_statistics",
if_exists="truncate"
)
```
Raw data
{
"_id": null,
"home_page": "https://github.com/IanRFerguson/klondike",
"name": "klondike",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.11.0,>=3.8.0",
"maintainer_email": null,
"keywords": "API, ETL, BIGQUERY, SNOWFLAKE",
"author": "Ian Richard Ferguson",
"author_email": "IRF229@nyu.edu",
"download_url": "https://files.pythonhosted.org/packages/c6/4d/80f303fe7649ce46b5322d7c61742a106363863fd4d17dd1125e08d36b99/klondike-0.2.1.tar.gz",
"platform": null,
"description": "# Klondike\n\n<img src=\"https://upload.wikimedia.org/wikipedia/en/d/d5/Klondike_logo.svg\">\n\nKlondike offers a lightweight API to read and write data to Google BigQuery using Polars DataFrames.\n\n## Installation\n\n### Installing Klondike\nInstall at the command line\n\n```\npip install klondike\n```\n\n### Installing Rust\nSince Polars leverages Rust speedups, you need to have Rust installed in your environment as well. See the Rust installation guide [here](https://www.rust-lang.org/tools/install).\n\n\n## Usage\n\nIn this demo we'll connect to Google BigQuery, read data, transform it, and write it back to the data warehouse.\n\nFirst, connect to the BigQuery warehouse by supplying the `BigQueryConnector()` object with the relative path to your service account credentials.\n\n```\nfrom klondike import BigQueryConnector\n\n# Connect Klondike to Google BigQuery\nbq = BigQueryConnector(\n app_creds=\"dev/nba-player-service.json\"\n)\n```\n\nNext, supply the object with a SQL query in the `read_dataframe()` function to redner a `DataFrame` object:\n\n```\n# Write some valid SQL\nsql = \"\"\"\nSELECT\n *\nFROM nba_dbt.cln_player__averages\nORDER BY avg_points DESC\n\"\"\"\n\n\n# Pull BigQuery data into a Polars DataFrame\nnyk = bq.read_dataframe(sql=sql)\n```\n\nNow that your data is pulled into a local instance, you can clean and transform it using standard Polars functionality - [see the docs](https://docs.pola.rs/py-polars/html/reference/dataframe/index.html) for more information.\n\n```\n# Perform some transformations\nkey_metrics = [\n \"avg_offensive_rating\",\n \"avg_defensive_rating\",\n \"avg_effective_field_goal_percentage\"\n]\n\nsummary_stats = nyk[key_metrics].describe()\n```\n\nFinally, push your transformed data back to the BigQuery warehouse using the `write_dataframe()` function:\n\n```\n# Write back to BigQuery\nbq.write_dataframe(\n df=summary_stats,\n table_name=\"nba_dbt.summary_statistics\",\n if_exists=\"truncate\"\n)\n```\n",
"bugtrack_url": null,
"license": null,
"summary": null,
"version": "0.2.1",
"project_urls": {
"Homepage": "https://github.com/IanRFerguson/klondike"
},
"split_keywords": [
"api",
" etl",
" bigquery",
" snowflake"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f02a9994d1a0b69f07d5b82def5202f644ac0df91f22388bdb61ce077702f30c",
"md5": "1fefe7321fd4ae0e9e1a540b507e92ad",
"sha256": "066105fd1bb5b63faae2a49ddb164f50aa7ba6296ac151186f1b87d0b12f709b"
},
"downloads": -1,
"filename": "klondike-0.2.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "1fefe7321fd4ae0e9e1a540b507e92ad",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.11.0,>=3.8.0",
"size": 10787,
"upload_time": "2024-09-07T02:43:18",
"upload_time_iso_8601": "2024-09-07T02:43:18.541382Z",
"url": "https://files.pythonhosted.org/packages/f0/2a/9994d1a0b69f07d5b82def5202f644ac0df91f22388bdb61ce077702f30c/klondike-0.2.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c64d80f303fe7649ce46b5322d7c61742a106363863fd4d17dd1125e08d36b99",
"md5": "7c13f072e65966c8f0b1faeb14bbf26a",
"sha256": "3d13d7166b80527691bd6581f0cecb6c04962c36e0823d236e1bade34ee3e3e5"
},
"downloads": -1,
"filename": "klondike-0.2.1.tar.gz",
"has_sig": false,
"md5_digest": "7c13f072e65966c8f0b1faeb14bbf26a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.11.0,>=3.8.0",
"size": 10172,
"upload_time": "2024-09-07T02:43:19",
"upload_time_iso_8601": "2024-09-07T02:43:19.868746Z",
"url": "https://files.pythonhosted.org/packages/c6/4d/80f303fe7649ce46b5322d7c61742a106363863fd4d17dd1125e08d36b99/klondike-0.2.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-07 02:43:19",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "IanRFerguson",
"github_project": "klondike",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "klondike"
}
JSON
