| Name | levi JSON |
| Version |
0.3.0
JSON |
| download |
| home_page | |
| Summary | Delta Lake helper methods |
| upload_time | 2023-06-14 17:47:26 |
| maintainer | |
| docs_url | None |
| author | Matthew Powers |
| requires_python | >=3.9,<4.0 |
| license | |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Levi
Delta Lake helper methods. No Spark dependency.
## Installation
Install the latest version with `pip install levi`.
## Delta File Stats
The `delta_file_stats` function provides information on the number of bytes in files of a Delta table. Example usage:
```python
import levi
from deltalake import DeltaTable
dt = DeltaTable("some_folder/some_table")
levi.delta_file_sizes(dt)
# return value
{
'num_files_<1mb': 345,
'num_files_1mb-500mb': 588,
'num_files_500mb-1gb': 960,
'num_files_1gb-2gb': 0,
'num_files_>2gb': 5
}
```
This output shows that there are 345 small files with less than 1mb of data and 5 huge files with more than 2gb of data. It'd be a good idea to compact the small files and split up the large files to make queries on this Delta table run faster.
You can also specify the boundaries when you invoke the function to get a custom result:
```python
levi.delta_file_sizes(dt, ["<1mb", "1mb-200mb", "200mb-800mb", "800mb-2gb", ">2gb"])
```
## Skipped stats
Provides information on the number of files and number of bytes that are skipped for a given set of predicates.
```python
import levi
dt = DeltaTable("some_folder/some_table")
levi.skipped_stats(dt, filters=[('a_float', '=', 4.5)])
# return value
{
'num_files': 2,
'num_files_skipped': 1,
'num_bytes_skipped': 996
}
```
This predicate will skip one file and 996 bytes of data.
You can use `skipped_stats` to figure out the percentage of files that get skipped. You can also use this information to see if you should Z ORDER your data or otherwise rearrange it to allow for better file skipping.
## Get Latest Delta Table Version
The `latest_version` function gets the most current Delta Table version number and returns it.
```python
import levi
from deltalake import DeltaTable
dt = DeltaTable("some_folder/some_table")
levi.latest_version(dt)
# return value
2
```
Raw data
{
"_id": null,
"home_page": "",
"name": "levi",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.9,<4.0",
"maintainer_email": "",
"keywords": "",
"author": "Matthew Powers",
"author_email": "matthewkevinpowers@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/f6/7d/80de65f3f6d438fcae989731e27ebbf38b8e6d65898ad01203d49de43217/levi-0.3.0.tar.gz",
"platform": null,
"description": "# Levi\n\nDelta Lake helper methods. No Spark dependency.\n\n## Installation\n\nInstall the latest version with `pip install levi`.\n\n## Delta File Stats\n\nThe `delta_file_stats` function provides information on the number of bytes in files of a Delta table. Example usage:\n\n```python\nimport levi\nfrom deltalake import DeltaTable\n\ndt = DeltaTable(\"some_folder/some_table\")\nlevi.delta_file_sizes(dt)\n\n# return value\n{\n 'num_files_<1mb': 345, \n 'num_files_1mb-500mb': 588,\n 'num_files_500mb-1gb': 960,\n 'num_files_1gb-2gb': 0, \n 'num_files_>2gb': 5\n}\n```\n\nThis output shows that there are 345 small files with less than 1mb of data and 5 huge files with more than 2gb of data. It'd be a good idea to compact the small files and split up the large files to make queries on this Delta table run faster.\n\nYou can also specify the boundaries when you invoke the function to get a custom result:\n\n```python\nlevi.delta_file_sizes(dt, [\"<1mb\", \"1mb-200mb\", \"200mb-800mb\", \"800mb-2gb\", \">2gb\"])\n```\n\n## Skipped stats\n\nProvides information on the number of files and number of bytes that are skipped for a given set of predicates.\n\n```python\nimport levi\n\ndt = DeltaTable(\"some_folder/some_table\")\nlevi.skipped_stats(dt, filters=[('a_float', '=', 4.5)])\n\n# return value\n{\n 'num_files': 2,\n 'num_files_skipped': 1,\n 'num_bytes_skipped': 996\n}\n```\n\nThis predicate will skip one file and 996 bytes of data.\n\nYou can use `skipped_stats` to figure out the percentage of files that get skipped. You can also use this information to see if you should Z ORDER your data or otherwise rearrange it to allow for better file skipping. \n\n## Get Latest Delta Table Version\n\nThe `latest_version` function gets the most current Delta Table version number and returns it.\n\n```python\nimport levi\nfrom deltalake import DeltaTable\n\ndt = DeltaTable(\"some_folder/some_table\")\nlevi.latest_version(dt)\n\n# return value\n2\n```\n\n",
"bugtrack_url": null,
"license": "",
"summary": "Delta Lake helper methods",
"version": "0.3.0",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "98ded4712434eede14d5be79cd67f8316fd88b15a2b2ea7f6a2fdade3c07cc00",
"md5": "317cdb8404145d4c4aced751a9372704",
"sha256": "383e5d7f34a1dfb39b209e2aa64c5880c7970c9a25e6eb1c364877ce374d30d6"
},
"downloads": -1,
"filename": "levi-0.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "317cdb8404145d4c4aced751a9372704",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9,<4.0",
"size": 3678,
"upload_time": "2023-06-14T17:47:25",
"upload_time_iso_8601": "2023-06-14T17:47:25.457014Z",
"url": "https://files.pythonhosted.org/packages/98/de/d4712434eede14d5be79cd67f8316fd88b15a2b2ea7f6a2fdade3c07cc00/levi-0.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f67d80de65f3f6d438fcae989731e27ebbf38b8e6d65898ad01203d49de43217",
"md5": "71fdd6cac9cf5e0aa42ddb811cf8d16c",
"sha256": "1e3166637baa3e080415ab052ef91524b35600432e198e62802955472d48ba96"
},
"downloads": -1,
"filename": "levi-0.3.0.tar.gz",
"has_sig": false,
"md5_digest": "71fdd6cac9cf5e0aa42ddb811cf8d16c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9,<4.0",
"size": 3178,
"upload_time": "2023-06-14T17:47:26",
"upload_time_iso_8601": "2023-06-14T17:47:26.964437Z",
"url": "https://files.pythonhosted.org/packages/f6/7d/80de65f3f6d438fcae989731e27ebbf38b8e6d65898ad01203d49de43217/levi-0.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-06-14 17:47:26",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "levi"
}
