# wkbparse
`wkbparse` is a Python module written in Rust for parsing [EWKB](https://postgis.net/docs/using_postgis_dbmanagement.html#EWKB_EWKT) and [TWKB](https://github.com/TWKB/Specification/blob/master/twkb.md) geometries into GeoJSON strings or GeoJSON-like Python dictionaries in a performant fashion.
Original EWKB/TWKB encoding code forked from [https://github.com/Mortal/rust-ewkb](Mortal/rust-ewkb) which in turn originates from [https://github.com/andelf/rust-postgis](andelf/rust-postgis).
`wkbparse` is developed mainly for usage with PostGIS and has been tested with geometries generated with one. However, it has no explicit dependencies towards PostgreSQL or PostGIS and can be used with EWKB/TWKB geometries originating from any system. In such case it is advisable to validate results carefully before using for anything serious.
It supports reading and writing ZM geometries as well, even though GeoJSON specification doesn't really recognize the M coordinate. The M coordinate is simply output as the fourth coordinate in a vertex. Respectively, input GeoJSON dictionaries with four coordinates in a vertex are treated as ZM geometries.
## Motivation
The main rationale behind this library is to offload compute related to geometry encoding from the database to the application and to minimize data transfer between them. This can be achieved by favoring native EWKB geometries or better-yet the transfer-optimized TWKB-geometries instead of making the database encode the data in some text-based format such as WKT or GeoJSON and sending that over the wire.
The benefits may be especially noticeable when dealing with large geometries with lots of vertices. E.g. the size of a 300 000 vertex multipolygon as EWKB is ~10 MB while as TWKB (1 cm precision) it is ~2 MB. Letting the database encode such geometry as GeoJSON and transferring it over the wire takes a long time (anecdotally way longer than a typical API timeout). Deserializing such MultiPolygon using `wkbparse` takes ~150 ms on an AMD Ryzen 4900 HS laptop and the transfer of TWKB is much quicker than of the other formats.
## Installation
Pre-built wheels are available for the following platforms and python versions:
Python versions: `[3.8, 3.9, 3.10, 3.11, 3.12]`
Platforms: Linux `[x86_64, x86, aarch64, armv7, s390x, ppc64le]`, Windows: `[x64, x86]`, MacOS: `[x86_64, aarch64]`
Install by saying `pip install wkbparse`.
Supported python version is >=3.8.
Tested on Python versions 3.8, 3.9, 3.10, 3.11 on Linux x86_64.
## Reprojection
Install `wkbparse-proj` instead of `wkbparse` to enable coordinate reprojection using the [Proj](https://github.com/OSGeo/PROJ) project.
Using the reprojection-enabled package allows one to pass in `from_srid` and `to_srid` as integers corresponding to EPSG-codes to many of the functions. The `from_srid` argument may be omitted if the source EWKB bytes or GeoJSON-dictionary data already contains the SRID. TWKB data never contains the SRID.
Pre-installed proj library must be present on the system for this feature to work. See [Proj installation](https://proj.org/en/stable/install.html).
Using `wkbparse-proj` bumps up the package size from ~250 kilobytes to ~10 megabytes due to rather large size of the Proj C++-dependency.
NOTE: Separate package is used instead of python "extras" within a single package due to extras not interacting nicely with wheels built with different Rust feature flags. This approach allows us to have a single codebase and a surefire way of selecting the appropriate package to avoid downloading redundant large dependencies.
`wbkparse-proj` pre-built wheels are only shipped for major Linux x86_64 platforms and MacOS.
## Usage
This module implements the following functionalities:
- TWKB to GeoJSON dictionary: `twkb_to_geojson`
- TWKB to EWKB: `twkb_to_ewkb`
- EWKB to GeoJSON dictionary: `ewkb_to_geojson`
- GeoJSON dictionary to EWKB: `geojson_to_ewkb`
- Reproject geojson `reproject_geojson` (only with `wkbparse-proj`)
The following is not currently implemented:
- Support for GeometryCollection types
- Encoding any data in TWKB
Example:
```python
import wkbparse
twkb_bytes = bytes.fromhex("610805d00fa01f50")
geometry = wkbparse.twkb_to_geojson(twkb_bytes)
print(geometry)
```
The result dict has the following shape:
```
{
type: str # GeoJSON geometry type
crs: Optional[int] # Spatial reference system identifier
coordinates: list[float] # nesting depth depending on geometry type
}
```
E.g.
```python
{'type': 'Point', 'crs': None, 'coordinates': [1.0, 2.0, 4.0]}
```
To reproject data when using `wkbparse-proj` we can additionally pass in `from_srid` and `to_srid`
```python
import wkbparse
twkb_bytes = bytes.fromhex("610805d00fa01f50")
geometry = wkbparse.twkb_to_geojson(twkb_bytes, from_srid=4326, to_srid=3857)
print(geometry)
```
```python
{'type': 'Point', 'crs': 3857, 'coordinates': [111319.49079327357, 222684.20850554405, 4.0]}
```
If we already have a dictionary as above, we can reproject it with `reproject_geojson`:
```python
import wkbparse
d = {"type": "Point", "crs": 3857, "coordinates": [111319.49079327357, 222684.20850554405, 4.0]}
reprojected = wkbparse.reproject_geojson(d, to_srid=4326)
print(reprojected)
```
```python
{'type': 'Point', 'crs': 4326, 'coordinates': [0.9999999999999998, 1.9999999999999996, 4.0]}
```
Note that `from_srid` was omitted in this case as the input geometry already had the `crs` field. One may provide it anyway to override the crs.
Raw data
{
"_id": null,
"home_page": "https://github.com/arjuote/wkbparse",
"name": "wkbparse-proj",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "PostgreSQL, PostGIS, GIS, GEO, WKB, TWKB, GeoJSON",
"author": "Arsi Juote <arsi.juote@gmail.com>",
"author_email": "Arsi Juote <arsi.juote@gmail.com>",
"download_url": null,
"platform": null,
"description": "# wkbparse\n\n`wkbparse` is a Python module written in Rust for parsing [EWKB](https://postgis.net/docs/using_postgis_dbmanagement.html#EWKB_EWKT) and [TWKB](https://github.com/TWKB/Specification/blob/master/twkb.md) geometries into GeoJSON strings or GeoJSON-like Python dictionaries in a performant fashion.\n\nOriginal EWKB/TWKB encoding code forked from [https://github.com/Mortal/rust-ewkb](Mortal/rust-ewkb) which in turn originates from [https://github.com/andelf/rust-postgis](andelf/rust-postgis).\n\n`wkbparse` is developed mainly for usage with PostGIS and has been tested with geometries generated with one. However, it has no explicit dependencies towards PostgreSQL or PostGIS and can be used with EWKB/TWKB geometries originating from any system. In such case it is advisable to validate results carefully before using for anything serious.\n\nIt supports reading and writing ZM geometries as well, even though GeoJSON specification doesn't really recognize the M coordinate. The M coordinate is simply output as the fourth coordinate in a vertex. Respectively, input GeoJSON dictionaries with four coordinates in a vertex are treated as ZM geometries.\n\n## Motivation\n\nThe main rationale behind this library is to offload compute related to geometry encoding from the database to the application and to minimize data transfer between them. This can be achieved by favoring native EWKB geometries or better-yet the transfer-optimized TWKB-geometries instead of making the database encode the data in some text-based format such as WKT or GeoJSON and sending that over the wire.\n\nThe benefits may be especially noticeable when dealing with large geometries with lots of vertices. E.g. the size of a 300 000 vertex multipolygon as EWKB is ~10 MB while as TWKB (1 cm precision) it is ~2 MB. Letting the database encode such geometry as GeoJSON and transferring it over the wire takes a long time (anecdotally way longer than a typical API timeout). Deserializing such MultiPolygon using `wkbparse` takes ~150 ms on an AMD Ryzen 4900 HS laptop and the transfer of TWKB is much quicker than of the other formats.\n\n## Installation\n\nPre-built wheels are available for the following platforms and python versions:\n\nPython versions: `[3.8, 3.9, 3.10, 3.11, 3.12]`\n\nPlatforms: Linux `[x86_64, x86, aarch64, armv7, s390x, ppc64le]`, Windows: `[x64, x86]`, MacOS: `[x86_64, aarch64]`\n\nInstall by saying `pip install wkbparse`.\n\nSupported python version is >=3.8.\n\nTested on Python versions 3.8, 3.9, 3.10, 3.11 on Linux x86_64.\n\n## Reprojection\n\nInstall `wkbparse-proj` instead of `wkbparse` to enable coordinate reprojection using the [Proj](https://github.com/OSGeo/PROJ) project.\n\nUsing the reprojection-enabled package allows one to pass in `from_srid` and `to_srid` as integers corresponding to EPSG-codes to many of the functions. The `from_srid` argument may be omitted if the source EWKB bytes or GeoJSON-dictionary data already contains the SRID. TWKB data never contains the SRID.\n\nPre-installed proj library must be present on the system for this feature to work. See [Proj installation](https://proj.org/en/stable/install.html).\n\nUsing `wkbparse-proj` bumps up the package size from ~250 kilobytes to ~10 megabytes due to rather large size of the Proj C++-dependency.\n\nNOTE: Separate package is used instead of python \"extras\" within a single package due to extras not interacting nicely with wheels built with different Rust feature flags. This approach allows us to have a single codebase and a surefire way of selecting the appropriate package to avoid downloading redundant large dependencies.\n\n`wbkparse-proj` pre-built wheels are only shipped for major Linux x86_64 platforms and MacOS.\n\n## Usage\n\nThis module implements the following functionalities:\n\n- TWKB to GeoJSON dictionary: `twkb_to_geojson`\n- TWKB to EWKB: `twkb_to_ewkb`\n- EWKB to GeoJSON dictionary: `ewkb_to_geojson`\n- GeoJSON dictionary to EWKB: `geojson_to_ewkb`\n- Reproject geojson `reproject_geojson` (only with `wkbparse-proj`)\n\nThe following is not currently implemented:\n\n- Support for GeometryCollection types\n- Encoding any data in TWKB\n\nExample:\n\n```python\nimport wkbparse\n\ntwkb_bytes = bytes.fromhex(\"610805d00fa01f50\")\ngeometry = wkbparse.twkb_to_geojson(twkb_bytes)\nprint(geometry)\n```\n\nThe result dict has the following shape:\n\n```\n{\n type: str # GeoJSON geometry type\n crs: Optional[int] # Spatial reference system identifier\n coordinates: list[float] # nesting depth depending on geometry type\n}\n```\n\nE.g.\n\n```python\n{'type': 'Point', 'crs': None, 'coordinates': [1.0, 2.0, 4.0]}\n```\n\nTo reproject data when using `wkbparse-proj` we can additionally pass in `from_srid` and `to_srid`\n\n```python\nimport wkbparse\n\ntwkb_bytes = bytes.fromhex(\"610805d00fa01f50\")\ngeometry = wkbparse.twkb_to_geojson(twkb_bytes, from_srid=4326, to_srid=3857)\nprint(geometry)\n```\n\n```python\n{'type': 'Point', 'crs': 3857, 'coordinates': [111319.49079327357, 222684.20850554405, 4.0]}\n```\n\nIf we already have a dictionary as above, we can reproject it with `reproject_geojson`:\n\n```python\nimport wkbparse\n\nd = {\"type\": \"Point\", \"crs\": 3857, \"coordinates\": [111319.49079327357, 222684.20850554405, 4.0]}\n\nreprojected = wkbparse.reproject_geojson(d, to_srid=4326)\nprint(reprojected)\n```\n\n```python\n{'type': 'Point', 'crs': 4326, 'coordinates': [0.9999999999999998, 1.9999999999999996, 4.0]}\n```\n\nNote that `from_srid` was omitted in this case as the input geometry already had the `crs` field. One may provide it anyway to override the crs.\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "EWKB and TWKB parsing and conversion to GeoJSON",
"version": "0.2.0",
"project_urls": {
"Homepage": "https://github.com/arjuote/wkbparse",
"Source": "https://github.com/arjuote/wkbparse"
},
"split_keywords": [
"postgresql",
" postgis",
" gis",
" geo",
" wkb",
" twkb",
" geojson"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "d45a489db4a7f1ed245fad819318e5882efa75fd22b69bfb3888c5778e4979bd",
"md5": "43e8ac04ab5aa86674a920b634bf9a5d",
"sha256": "1853fe2971d92ed040d831b8e65d299e2ca75cee7ffa384517b8293e6990556f"
},
"downloads": -1,
"filename": "wkbparse_proj-0.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "43e8ac04ab5aa86674a920b634bf9a5d",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.8",
"size": 2554515,
"upload_time": "2025-01-26T15:34:48",
"upload_time_iso_8601": "2025-01-26T15:34:48.764727Z",
"url": "https://files.pythonhosted.org/packages/d4/5a/489db4a7f1ed245fad819318e5882efa75fd22b69bfb3888c5778e4979bd/wkbparse_proj-0.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "266c1c7fcccd5316461e7faaf27899ad8680120b871a702172d3967e3fd4b2ec",
"md5": "18a8d1f7a37f9f64aa536b6750b9f432",
"sha256": "43060e51850021258c5c2914450a3316d3069b71bec493994c9932f572f510f8"
},
"downloads": -1,
"filename": "wkbparse_proj-0.2.0-cp311-cp311-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "18a8d1f7a37f9f64aa536b6750b9f432",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.8",
"size": 1455851,
"upload_time": "2025-01-26T15:35:01",
"upload_time_iso_8601": "2025-01-26T15:35:01.399014Z",
"url": "https://files.pythonhosted.org/packages/26/6c/1c7fcccd5316461e7faaf27899ad8680120b871a702172d3967e3fd4b2ec/wkbparse_proj-0.2.0-cp311-cp311-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "537dd9ee9cc507cb74aa7249c61af4d53c58762f73944b0f7bf80536539803a8",
"md5": "61d4d103e457854838c249277a48e6e0",
"sha256": "749338a1cc8ffe02b23583f33d3a855c9428165ea860e4f9d4e0257d8b5052ad"
},
"downloads": -1,
"filename": "wkbparse_proj-0.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "61d4d103e457854838c249277a48e6e0",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.8",
"size": 2554513,
"upload_time": "2025-01-26T15:34:50",
"upload_time_iso_8601": "2025-01-26T15:34:50.854722Z",
"url": "https://files.pythonhosted.org/packages/53/7d/d9ee9cc507cb74aa7249c61af4d53c58762f73944b0f7bf80536539803a8/wkbparse_proj-0.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "217ed62aee6334d3cbd172e487c5da53487eeb38b47da6c47a9c54f9635fca22",
"md5": "43945904b4b528c469b75f58e8f361c2",
"sha256": "fa64c1451ee027fb18ed6d3178ebaadabc9c7b68b0da9620556ce8fe7affe6fa"
},
"downloads": -1,
"filename": "wkbparse_proj-0.2.0-cp312-cp312-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "43945904b4b528c469b75f58e8f361c2",
"packagetype": "bdist_wheel",
"python_version": "cp312",
"requires_python": ">=3.8",
"size": 1454959,
"upload_time": "2025-01-26T15:35:03",
"upload_time_iso_8601": "2025-01-26T15:35:03.432475Z",
"url": "https://files.pythonhosted.org/packages/21/7e/d62aee6334d3cbd172e487c5da53487eeb38b47da6c47a9c54f9635fca22/wkbparse_proj-0.2.0-cp312-cp312-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "6a3e2f6d80d931184a2b67e26f1f77fac562304fe1806421eb47a8d9643f7c31",
"md5": "096d630ec61744dc51c82fd944339914",
"sha256": "a44a6a2e75a77a7fc5570b6a96155ef86f16c7ecafba7dde7d3c5e85151b6de4"
},
"downloads": -1,
"filename": "wkbparse_proj-0.2.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "096d630ec61744dc51c82fd944339914",
"packagetype": "bdist_wheel",
"python_version": "cp312",
"requires_python": ">=3.8",
"size": 2553908,
"upload_time": "2025-01-26T15:34:52",
"upload_time_iso_8601": "2025-01-26T15:34:52.479987Z",
"url": "https://files.pythonhosted.org/packages/6a/3e/2f6d80d931184a2b67e26f1f77fac562304fe1806421eb47a8d9643f7c31/wkbparse_proj-0.2.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "285b4a9867f7064023bbb8eae3f64754719eef8bea6ff1a7f9e9e26e499088fc",
"md5": "082452718a703c943ef462cccb76af89",
"sha256": "6f47fb684a88b80c9250a9dbaf641f1901d48401698483bad21db9d28a85bf73"
},
"downloads": -1,
"filename": "wkbparse_proj-0.2.0-cp313-cp313-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "082452718a703c943ef462cccb76af89",
"packagetype": "bdist_wheel",
"python_version": "cp313",
"requires_python": ">=3.8",
"size": 1454959,
"upload_time": "2025-01-26T15:35:05",
"upload_time_iso_8601": "2025-01-26T15:35:05.688016Z",
"url": "https://files.pythonhosted.org/packages/28/5b/4a9867f7064023bbb8eae3f64754719eef8bea6ff1a7f9e9e26e499088fc/wkbparse_proj-0.2.0-cp313-cp313-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "200bf56becbc7ee50dc410051f1d0aa428767f91a2e59f2710e89909deb7092a",
"md5": "83f93f2be9b449d79425467798181957",
"sha256": "fa6a9421c973bbd690e786bce701ca4e7ef64a8eaba994ae4ea10c017b7bcc8a"
},
"downloads": -1,
"filename": "wkbparse_proj-0.2.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "83f93f2be9b449d79425467798181957",
"packagetype": "bdist_wheel",
"python_version": "cp313",
"requires_python": ">=3.8",
"size": 2553908,
"upload_time": "2025-01-26T15:34:54",
"upload_time_iso_8601": "2025-01-26T15:34:54.174774Z",
"url": "https://files.pythonhosted.org/packages/20/0b/f56becbc7ee50dc410051f1d0aa428767f91a2e59f2710e89909deb7092a/wkbparse_proj-0.2.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "f288a533ae18abcd7a2194b681bee46d1233a6ae97f46090f314f14ff040c80f",
"md5": "32b81410ed5546ec50ddd2ee2c64f8ba",
"sha256": "36a80142fc7d32b40d3c533ce169fc7eebcdfd14f1ac36eae19ecc5fbe1a0b96"
},
"downloads": -1,
"filename": "wkbparse_proj-0.2.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "32b81410ed5546ec50ddd2ee2c64f8ba",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.8",
"size": 2554176,
"upload_time": "2025-01-26T15:34:55",
"upload_time_iso_8601": "2025-01-26T15:34:55.796665Z",
"url": "https://files.pythonhosted.org/packages/f2/88/a533ae18abcd7a2194b681bee46d1233a6ae97f46090f314f14ff040c80f/wkbparse_proj-0.2.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "c9af1b5442a71a4d35007b514816a9614e4355da668974f7a46f573a01a35345",
"md5": "21dbe920ac370e923837827acf922278",
"sha256": "d77f41ae23b92454fd3ddbf5dccc27038d5ba7098fd10885299a4d2d2a2914ab"
},
"downloads": -1,
"filename": "wkbparse_proj-0.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "21dbe920ac370e923837827acf922278",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.8",
"size": 2554409,
"upload_time": "2025-01-26T15:34:58",
"upload_time_iso_8601": "2025-01-26T15:34:58.181220Z",
"url": "https://files.pythonhosted.org/packages/c9/af/1b5442a71a4d35007b514816a9614e4355da668974f7a46f573a01a35345/wkbparse_proj-0.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "00acc09b18cc08f89d1fd7967e9dbbeca8f0c61dbf6b833f0941e21fde00e486",
"md5": "01839319054b2874a06d1d8b35b093f4",
"sha256": "0f26c0ef9c8d006cba5db3dd03b4e3937abbf52daad11c411ecac7026d4da7d8"
},
"downloads": -1,
"filename": "wkbparse_proj-0.2.0-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "01839319054b2874a06d1d8b35b093f4",
"packagetype": "bdist_wheel",
"python_version": "pp310",
"requires_python": ">=3.8",
"size": 2554519,
"upload_time": "2025-01-26T15:34:59",
"upload_time_iso_8601": "2025-01-26T15:34:59.966434Z",
"url": "https://files.pythonhosted.org/packages/00/ac/c09b18cc08f89d1fd7967e9dbbeca8f0c61dbf6b833f0941e21fde00e486/wkbparse_proj-0.2.0-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-26 15:34:48",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "arjuote",
"github_project": "wkbparse",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "wkbparse-proj"
}
JSON
