# Klondike
Klondike offers a lightweight API to read and write data to Google BigQuery using Polars DataFrames.
## Installation
### Installing Klondike
Install at the command line
```
pip install klondike
```
### Installing Rust
Since Polars leverages Rust speedups, you need to have Rust installed in your environment as well. See the Rust installation guide [here](https://www.rust-lang.org/tools/install).
## Usage
In this demo we'll connect to Google BigQuery, read data, transform it, and write it back to the data warehouse.
First, connect to the BigQuery warehouse by supplying the `BigQueryConnector()` object with the relative path to your service account credentials.
```
from klondike import BigQueryConnector
# Connect Klondike to Google BigQuery
bq = BigQueryConnector(
app_creds="dev/nba-player-service.json"
)
```
Next, supply the object with a SQL query in the `read_dataframe()` function to redner a `DataFrame` object:
```
# Write some valid SQL
sql = """
SELECT
*
FROM nba_dbt.cln_player__averages
ORDER BY avg_points DESC
"""
# Pull BigQuery data into a Polars DataFrame
nyk = bq.read_dataframe(sql=sql)
```
Now that your data is pulled into a local instance, you can clean and transform it using standard Polars functionality - [see the docs](https://docs.pola.rs/py-polars/html/reference/dataframe/index.html) for more information.
```
# Perform some transformations
key_metrics = [
"avg_offensive_rating",
"avg_defensive_rating",
"avg_effective_field_goal_percentage"
]
summary_stats = nyk[key_metrics].describe()
```
Finally, push your transformed data back to the BigQuery warehouse using the `write_dataframe()` function:
```
# Write back to BigQuery
bq.write_dataframe(
df=summary_stats,
table_name="nba_dbt.summary_statistics",
if_exists="truncate"
)
```
Raw data
{
"_id": null,
"home_page": "https://github.com/IanRFerguson/klondike",
"name": "klondike",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.8",
"maintainer_email": null,
"keywords": "polars, bigquery, snowflake, redshift",
"author": "Ian Ferguson",
"author_email": "IANFERGUSONRVA@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/38/1c/0519efffa5965c9cb864e0fa4705b04c75fda6988762392864fe7d059e94/klondike-0.2.3.tar.gz",
"platform": null,
"description": "# Klondike\n\nKlondike offers a lightweight API to read and write data to Google BigQuery using Polars DataFrames.\n\n## Installation\n\n### Installing Klondike\nInstall at the command line\n\n```\npip install klondike\n```\n\n### Installing Rust\nSince Polars leverages Rust speedups, you need to have Rust installed in your environment as well. See the Rust installation guide [here](https://www.rust-lang.org/tools/install).\n\n\n## Usage\n\nIn this demo we'll connect to Google BigQuery, read data, transform it, and write it back to the data warehouse.\n\nFirst, connect to the BigQuery warehouse by supplying the `BigQueryConnector()` object with the relative path to your service account credentials.\n\n```\nfrom klondike import BigQueryConnector\n\n# Connect Klondike to Google BigQuery\nbq = BigQueryConnector(\n app_creds=\"dev/nba-player-service.json\"\n)\n```\n\nNext, supply the object with a SQL query in the `read_dataframe()` function to redner a `DataFrame` object:\n\n```\n# Write some valid SQL\nsql = \"\"\"\nSELECT\n *\nFROM nba_dbt.cln_player__averages\nORDER BY avg_points DESC\n\"\"\"\n\n\n# Pull BigQuery data into a Polars DataFrame\nnyk = bq.read_dataframe(sql=sql)\n```\n\nNow that your data is pulled into a local instance, you can clean and transform it using standard Polars functionality - [see the docs](https://docs.pola.rs/py-polars/html/reference/dataframe/index.html) for more information.\n\n```\n# Perform some transformations\nkey_metrics = [\n \"avg_offensive_rating\",\n \"avg_defensive_rating\",\n \"avg_effective_field_goal_percentage\"\n]\n\nsummary_stats = nyk[key_metrics].describe()\n```\n\nFinally, push your transformed data back to the BigQuery warehouse using the `write_dataframe()` function:\n\n```\n# Write back to BigQuery\nbq.write_dataframe(\n df=summary_stats,\n table_name=\"nba_dbt.summary_statistics\",\n if_exists=\"truncate\"\n)\n```\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Klondike is a suite of database connectors, powered by Polars",
"version": "0.2.3",
"project_urls": {
"Homepage": "https://github.com/IanRFerguson/klondike",
"Repository": "https://github.com/IanRFerguson/klondike"
},
"split_keywords": [
"polars",
" bigquery",
" snowflake",
" redshift"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "14170b1e7146189f2470b1908fd2e640d9d8344b25cb71f66fb8942fb5545637",
"md5": "ea45dee12f90f75cfd6a36e02c3c3b13",
"sha256": "193a8f5fcd14d52b54b9320957438e6cea23d88985f17a4c771943a9742a6916"
},
"downloads": -1,
"filename": "klondike-0.2.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "ea45dee12f90f75cfd6a36e02c3c3b13",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.8",
"size": 12651,
"upload_time": "2025-01-19T21:11:35",
"upload_time_iso_8601": "2025-01-19T21:11:35.391970Z",
"url": "https://files.pythonhosted.org/packages/14/17/0b1e7146189f2470b1908fd2e640d9d8344b25cb71f66fb8942fb5545637/klondike-0.2.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "381c0519efffa5965c9cb864e0fa4705b04c75fda6988762392864fe7d059e94",
"md5": "ca6480417293a069b795589519501b39",
"sha256": "11426682ff1fd0beb309d64151bc8cc46ce7597459074610ab493e5703d2a475"
},
"downloads": -1,
"filename": "klondike-0.2.3.tar.gz",
"has_sig": false,
"md5_digest": "ca6480417293a069b795589519501b39",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.8",
"size": 9499,
"upload_time": "2025-01-19T21:11:37",
"upload_time_iso_8601": "2025-01-19T21:11:37.011086Z",
"url": "https://files.pythonhosted.org/packages/38/1c/0519efffa5965c9cb864e0fa4705b04c75fda6988762392864fe7d059e94/klondike-0.2.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-19 21:11:37",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "IanRFerguson",
"github_project": "klondike",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "klondike"
}
JSON
