# Welcome To Datagrunt
Datagrunt is a Python library designed to simplify the way you work with CSV files. It provides a streamlined approach to reading, processing, and transforming your data into various formats, making data manipulation efficient and intuitive.
## Why Datagrunt?
Born out of real-world frustration, Datagrunt eliminates the need For repetitive coding when handling CSV files. Whether you're a data analyst, data engineer, or data scientist, Datagrunt empowers you to focus on insights, not tedious data wrangling.
## Key Features
- **Intelligent Delimiter Inference:** Datagrunt automatically detects and applies the correct delimiter for your csv files.
- **Seamless Data Processing:** Leverage the robust capabilities of [DuckDB](https://duckdb.org) and [Polars](https://pola.rs) to perform advanced data processing tasks directly on your CSV data.
- **Flexible Transformation:** Easily convert your processed CSV data into various formats to suit your needs.
- **Pythonic API:** Enjoy a clean and intuitive API that integrates seamlessly into your existing Python workflows.
## Installation
Get started with Datagrunt in seconds using pip:
```bash
pip install datagrunt
```
## Getting Started
```python
from datagrunt import CSVReader
# Load your CSV file
csv_file = 'electric_vehicle_population_data.csv'
engine = 'duckdb'
# Set duckdb as the processing engine. Engine set to 'polars' by default
dg = CSVReader(csv_file, engine=engine)
# return sample of the data to get a peek at the schema
dg.get_sample()
┌────────────┬───────────┬──────────────┬───┬──────────────────────┬──────────────────────┬───────────────────┐
│ VIN (1-10) │ County │ City │ … │ Vehicle Location │ Electric Utility │ 2020 Census Tract │
│ varchar │ varchar │ varchar │ │ varchar │ varchar │ varchar │
├────────────┼───────────┼──────────────┼───┼──────────────────────┼──────────────────────┼───────────────────┤
│ 5YJSA1E28K │ Snohomish │ Mukilteo │ … │ POINT (-122.29943 … │ PUGET SOUND ENERGY… │ 53061042001 │
│ 1C4JJXP68P │ Yakima │ Yakima │ … │ POINT (-120.468875… │ PACIFICORP │ 53077001601 │
│ WBY8P6C05L │ Kitsap │ Kingston │ … │ POINT (-122.517835… │ PUGET SOUND ENERGY… │ 53035090102 │
│ JTDKARFP1J │ Kitsap │ Port Orchard │ … │ POINT (-122.653005… │ PUGET SOUND ENERGY… │ 53035092802 │
│ 5UXTA6C09N │ Snohomish │ Everett │ … │ POINT (-122.203234… │ PUGET SOUND ENERGY… │ 53061041605 │
│ 5YJYGDEF8L │ King │ Seattle │ … │ POINT (-122.378886… │ CITY OF SEATTLE - … │ 53033004703 │
│ JTMAB3FV7P │ Thurston │ Rainier │ … │ POINT (-122.677141… │ PUGET SOUND ENERGY… │ 53067012530 │
│ JN1AZ0CPXC │ King │ Kirkland │ … │ POINT (-122.192596… │ PUGET SOUND ENERGY… │ 53033022402 │
│ JN1AZ0CP7B │ King │ Kirkland │ … │ POINT (-122.192596… │ PUGET SOUND ENERGY… │ 53033022603 │
│ 1N4AZ0CP0F │ Thurston │ Olympia │ … │ POINT (-122.86491 … │ PUGET SOUND ENERGY… │ 53067010300 │
│ · │ · │ · │ · │ · │ · │ · │
│ · │ · │ · │ · │ · │ · │ · │
│ · │ · │ · │ · │ · │ · │ · │
│ 5YJYGDEE7M │ Clark │ Vancouver │ … │ POINT (-122.515805… │ BONNEVILLE POWER A… │ 53011041310 │
│ 7SAYGAEE0P │ Snohomish │ Monroe │ … │ POINT (-121.968385… │ PUGET SOUND ENERGY… │ 53061052203 │
│ 2C4RC1N75P │ King │ Burien │ … │ POINT (-122.347227… │ CITY OF SEATTLE - … │ 53033027600 │
│ 1FTVW1EVXP │ King │ Kirkland │ … │ POINT (-122.202653… │ PUGET SOUND ENERGY… │ 53033022300 │
│ 4JGGM1CB2P │ King │ Seattle │ … │ POINT (-122.2453 4… │ CITY OF SEATTLE - … │ 53033011700 │
│ 1N4BZ0CP0G │ King │ Seattle │ … │ POINT (-122.334079… │ CITY OF SEATTLE - … │ 53033008300 │
│ 7SAYGDEF2N │ King │ Bellevue │ … │ POINT (-122.144149… │ PUGET SOUND ENERGY… │ 53033024704 │
│ 1N4BZ1DP7L │ King │ Bellevue │ … │ POINT (-122.144149… │ PUGET SOUND ENERGY… │ 53033024902 │
...
├────────────┴───────────┴──────────────┴───┴──────────────────────┴──────────────────────┴───────────────────┤
│ ? rows (>9999 rows, 20 shown) 17 columns (6 shown) │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
```
## DuckDB Integration for Performant SQL Queries
```python
from datagrunt import CSVReader
csv_file = 'electric_vehicle_population_data.csv'
engine = 'duckdb'
dg = CSVReader(csv_file, engine=engine)
# Construct your SQL query
query = f"""
WITH core AS (
SELECT
City AS city,
"VIN (1-10)" AS vin
FROM {dg.db_table}
)
SELECT
city,
COUNT(vin) AS vehicle_count
FROM core
GROUP BY 1
ORDER BY 2 DESC
"""
# Execute the query and get results as a Polars DataFrame
df = dg.query_data(query).pl()
print(df)
┌────────────────┬───────────────┐
│ city ┆ vehicle_count │
│ --- ┆ --- │
│ str ┆ i64 │
╞════════════════╪═══════════════╡
│ Seattle ┆ 32602 │
│ Bellevue ┆ 9960 │
│ Redmond ┆ 7165 │
│ Vancouver ┆ 7081 │
│ Bothell ┆ 6602 │
│ … ┆ … │
│ Glenwood ┆ 1 │
│ Walla Walla Co ┆ 1 │
│ Pittsburg ┆ 1 │
│ Decatur ┆ 1 │
│ Redwood City ┆ 1 │
└────────────────┴───────────────┘
```
## License
This project is licensed under the [MIT License](https://opensource.org/license/mit)
## Acknowledgements
A HUGE thank you to the open source community and the creators of [DuckDB](https://duckdb.org) and [Polars](https://pola.rs) for their fantastic libraries that power Datagrunt.
Raw data
{
"_id": null,
"home_page": null,
"name": "datagrunt",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "csv, data, duckdb, polars, pyarrow, xlsx, delimiter",
"author": null,
"author_email": "Martin Graham <datagrunt@datagrunt.io>",
"download_url": "https://files.pythonhosted.org/packages/81/49/2f85f495a1c57ec7a17620394ed3bc9d11cded41f9ac9c86cb6f9005748d/datagrunt-0.0.2.tar.gz",
"platform": null,
"description": "# Welcome To Datagrunt\n\nDatagrunt is a Python library designed to simplify the way you work with CSV files. It provides a streamlined approach to reading, processing, and transforming your data into various formats, making data manipulation efficient and intuitive.\n\n## Why Datagrunt?\n\nBorn out of real-world frustration, Datagrunt eliminates the need For repetitive coding when handling CSV files. Whether you're a data analyst, data engineer, or data scientist, Datagrunt empowers you to focus on insights, not tedious data wrangling.\n\n## Key Features\n\n- **Intelligent Delimiter Inference:** Datagrunt automatically detects and applies the correct delimiter for your csv files.\n- **Seamless Data Processing:** Leverage the robust capabilities of [DuckDB](https://duckdb.org) and [Polars](https://pola.rs) to perform advanced data processing tasks directly on your CSV data.\n- **Flexible Transformation:** Easily convert your processed CSV data into various formats to suit your needs.\n- **Pythonic API:** Enjoy a clean and intuitive API that integrates seamlessly into your existing Python workflows.\n\n## Installation\n\nGet started with Datagrunt in seconds using pip:\n\n```bash\npip install datagrunt\n```\n\n## Getting Started\n\n```python\nfrom datagrunt import CSVReader\n\n# Load your CSV file\ncsv_file = 'electric_vehicle_population_data.csv'\nengine = 'duckdb'\n\n# Set duckdb as the processing engine. Engine set to 'polars' by default\ndg = CSVReader(csv_file, engine=engine)\n\n# return sample of the data to get a peek at the schema\ndg.get_sample()\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 VIN (1-10) \u2502 County \u2502 City \u2502 \u2026 \u2502 Vehicle Location \u2502 Electric Utility \u2502 2020 Census Tract \u2502\n\u2502 varchar \u2502 varchar \u2502 varchar \u2502 \u2502 varchar \u2502 varchar \u2502 varchar \u2502\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 5YJSA1E28K \u2502 Snohomish \u2502 Mukilteo \u2502 \u2026 \u2502 POINT (-122.29943 \u2026 \u2502 PUGET SOUND ENERGY\u2026 \u2502 53061042001 \u2502\n\u2502 1C4JJXP68P \u2502 Yakima \u2502 Yakima \u2502 \u2026 \u2502 POINT (-120.468875\u2026 \u2502 PACIFICORP \u2502 53077001601 \u2502\n\u2502 WBY8P6C05L \u2502 Kitsap \u2502 Kingston \u2502 \u2026 \u2502 POINT (-122.517835\u2026 \u2502 PUGET SOUND ENERGY\u2026 \u2502 53035090102 \u2502\n\u2502 JTDKARFP1J \u2502 Kitsap \u2502 Port Orchard \u2502 \u2026 \u2502 POINT (-122.653005\u2026 \u2502 PUGET SOUND ENERGY\u2026 \u2502 53035092802 \u2502\n\u2502 5UXTA6C09N \u2502 Snohomish \u2502 Everett \u2502 \u2026 \u2502 POINT (-122.203234\u2026 \u2502 PUGET SOUND ENERGY\u2026 \u2502 53061041605 \u2502\n\u2502 5YJYGDEF8L \u2502 King \u2502 Seattle \u2502 \u2026 \u2502 POINT (-122.378886\u2026 \u2502 CITY OF SEATTLE - \u2026 \u2502 53033004703 \u2502\n\u2502 JTMAB3FV7P \u2502 Thurston \u2502 Rainier \u2502 \u2026 \u2502 POINT (-122.677141\u2026 \u2502 PUGET SOUND ENERGY\u2026 \u2502 53067012530 \u2502\n\u2502 JN1AZ0CPXC \u2502 King \u2502 Kirkland \u2502 \u2026 \u2502 POINT (-122.192596\u2026 \u2502 PUGET SOUND ENERGY\u2026 \u2502 53033022402 \u2502\n\u2502 JN1AZ0CP7B \u2502 King \u2502 Kirkland \u2502 \u2026 \u2502 POINT (-122.192596\u2026 \u2502 PUGET SOUND ENERGY\u2026 \u2502 53033022603 \u2502\n\u2502 1N4AZ0CP0F \u2502 Thurston \u2502 Olympia \u2502 \u2026 \u2502 POINT (-122.86491 \u2026 \u2502 PUGET SOUND ENERGY\u2026 \u2502 53067010300 \u2502\n\u2502 \u00b7 \u2502 \u00b7 \u2502 \u00b7 \u2502 \u00b7 \u2502 \u00b7 \u2502 \u00b7 \u2502 \u00b7 \u2502\n\u2502 \u00b7 \u2502 \u00b7 \u2502 \u00b7 \u2502 \u00b7 \u2502 \u00b7 \u2502 \u00b7 \u2502 \u00b7 \u2502\n\u2502 \u00b7 \u2502 \u00b7 \u2502 \u00b7 \u2502 \u00b7 \u2502 \u00b7 \u2502 \u00b7 \u2502 \u00b7 \u2502\n\u2502 5YJYGDEE7M \u2502 Clark \u2502 Vancouver \u2502 \u2026 \u2502 POINT (-122.515805\u2026 \u2502 BONNEVILLE POWER A\u2026 \u2502 53011041310 \u2502\n\u2502 7SAYGAEE0P \u2502 Snohomish \u2502 Monroe \u2502 \u2026 \u2502 POINT (-121.968385\u2026 \u2502 PUGET SOUND ENERGY\u2026 \u2502 53061052203 \u2502\n\u2502 2C4RC1N75P \u2502 King \u2502 Burien \u2502 \u2026 \u2502 POINT (-122.347227\u2026 \u2502 CITY OF SEATTLE - \u2026 \u2502 53033027600 \u2502\n\u2502 1FTVW1EVXP \u2502 King \u2502 Kirkland \u2502 \u2026 \u2502 POINT (-122.202653\u2026 \u2502 PUGET SOUND ENERGY\u2026 \u2502 53033022300 \u2502\n\u2502 4JGGM1CB2P \u2502 King \u2502 Seattle \u2502 \u2026 \u2502 POINT (-122.2453 4\u2026 \u2502 CITY OF SEATTLE - \u2026 \u2502 53033011700 \u2502\n\u2502 1N4BZ0CP0G \u2502 King \u2502 Seattle \u2502 \u2026 \u2502 POINT (-122.334079\u2026 \u2502 CITY OF SEATTLE - \u2026 \u2502 53033008300 \u2502\n\u2502 7SAYGDEF2N \u2502 King \u2502 Bellevue \u2502 \u2026 \u2502 POINT (-122.144149\u2026 \u2502 PUGET SOUND ENERGY\u2026 \u2502 53033024704 \u2502\n\u2502 1N4BZ1DP7L \u2502 King \u2502 Bellevue \u2502 \u2026 \u2502 POINT (-122.144149\u2026 \u2502 PUGET SOUND ENERGY\u2026 \u2502 53033024902 \u2502\n...\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 ? rows (>9999 rows, 20 shown) 17 columns (6 shown) \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n```\n\n## DuckDB Integration for Performant SQL Queries\n```python\nfrom datagrunt import CSVReader\n\ncsv_file = 'electric_vehicle_population_data.csv'\nengine = 'duckdb'\n\ndg = CSVReader(csv_file, engine=engine)\n\n# Construct your SQL query\nquery = f\"\"\"\nWITH core AS (\n SELECT\n City AS city,\n \"VIN (1-10)\" AS vin\n FROM {dg.db_table}\n)\nSELECT\n city,\n COUNT(vin) AS vehicle_count\nFROM core\nGROUP BY 1\nORDER BY 2 DESC\n\"\"\"\n\n# Execute the query and get results as a Polars DataFrame\ndf = dg.query_data(query).pl()\nprint(df)\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 city \u2506 vehicle_count \u2502\n\u2502 --- \u2506 --- \u2502\n\u2502 str \u2506 i64 \u2502\n\u255e\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u256a\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2561\n\u2502 Seattle \u2506 32602 \u2502\n\u2502 Bellevue \u2506 9960 \u2502\n\u2502 Redmond \u2506 7165 \u2502\n\u2502 Vancouver \u2506 7081 \u2502\n\u2502 Bothell \u2506 6602 \u2502\n\u2502 \u2026 \u2506 \u2026 \u2502\n\u2502 Glenwood \u2506 1 \u2502\n\u2502 Walla Walla Co \u2506 1 \u2502\n\u2502 Pittsburg \u2506 1 \u2502\n\u2502 Decatur \u2506 1 \u2502\n\u2502 Redwood City \u2506 1 \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n```\n## License\nThis project is licensed under the [MIT License](https://opensource.org/license/mit)\n\n## Acknowledgements\nA HUGE thank you to the open source community and the creators of [DuckDB](https://duckdb.org) and [Polars](https://pola.rs) for their fantastic libraries that power Datagrunt.\n",
"bugtrack_url": null,
"license": "MIT License",
"summary": "Read CSV files and convert to other file formats easily",
"version": "0.0.2",
"project_urls": {
"Bug Tracker": "https://github.com/pmgraham/datagrunt/issues",
"Documentation": "https://pmgraham.github.io/datagrunt-docs",
"Homepage": "https://pmgraham.github.io/datagrunt-docs",
"Source Code": "https://github.com/pmgraham/datagrunt"
},
"split_keywords": [
"csv",
" data",
" duckdb",
" polars",
" pyarrow",
" xlsx",
" delimiter"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "45744b46e8feee8e07c9d4c1bbf45081640e356d4edc09cb5d16b240795ba713",
"md5": "cdc9ac3dbd45b828a05d19205d6a28f6",
"sha256": "97a57cc2d1867304cc5dc92c020abd3cd72edde01ed78a68e086eeca4753f73e"
},
"downloads": -1,
"filename": "datagrunt-0.0.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "cdc9ac3dbd45b828a05d19205d6a28f6",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 13383,
"upload_time": "2024-10-01T19:28:16",
"upload_time_iso_8601": "2024-10-01T19:28:16.685296Z",
"url": "https://files.pythonhosted.org/packages/45/74/4b46e8feee8e07c9d4c1bbf45081640e356d4edc09cb5d16b240795ba713/datagrunt-0.0.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "81492f85f495a1c57ec7a17620394ed3bc9d11cded41f9ac9c86cb6f9005748d",
"md5": "f18653092e00ece091295e4b4fddb2e0",
"sha256": "7ed26e11f91feb61bea775b89db0d5bc1d290e0265a08df91b0b0e48e4613322"
},
"downloads": -1,
"filename": "datagrunt-0.0.2.tar.gz",
"has_sig": false,
"md5_digest": "f18653092e00ece091295e4b4fddb2e0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 18312,
"upload_time": "2024-10-01T19:28:17",
"upload_time_iso_8601": "2024-10-01T19:28:17.784805Z",
"url": "https://files.pythonhosted.org/packages/81/49/2f85f495a1c57ec7a17620394ed3bc9d11cded41f9ac9c86cb6f9005748d/datagrunt-0.0.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-01 19:28:17",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "pmgraham",
"github_project": "datagrunt",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "datagrunt"
}
JSON
