| Name | cog-worker JSON |
| Version |
0.3.0
JSON |
| download |
| home_page | None |
| Summary | Scalable geospatial analysis on Cloud Optimized GeoTIFFs. |
| upload_time | 2024-07-22 17:27:33 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.9 |
| license | MIT License Copyright (c) 2021 Simbiotica. S.L. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
cog
geotiff
raster
gdal
rasterio
dask
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Cog Worker
Scalable geospatial analysis on Cloud Optimized GeoTIFFs.
- **Documentation**: https://vizzuality.github.io/cog_worker
- **PyPI**: https://pypi.org/project/cog-worker
cog_worker is a simple library to help write scripts to conduct scaleable
analysis of gridded data. It's intended to be useful for moderate- to large-scale
GIS, remote sensing, and machine learning applications.
## Installation
```
pip install cog_worker
```
## Examples
See `docs/examples` for Jupyter notebook examples
## Quick start
0. A simple cog_worker script
```python
from rasterio.plot import show
from cog_worker import Manager
def my_analysis(worker):
arr = worker.read('roads_cog.tif')
return arr
manager = Manager(proj='wgs84', scale=0.083333)
arr, bbox = manager.preview(my_analysis)
show(arr)
```
1. Define an analysis function that recieves a cog_worker.Worker as the first parameter.
```python
from cog_worker import Worker, Manager
import numpy as np
# Define an analysis function to read and process COG data sources
def MyAnalysis(worker: Worker) -> np.ndarray:
# 1. Read a COG (reprojecting, resampling and clipping as necessary)
array: np.ndarray = worker.read('roads_cog.tif')
# 2. Work on the array
# ...
# 3. Return (or post to blob storage etc.)
return array
```
2. Run your analysis in different scales and projections
```python
import rasterio as rio
# Run your analysis using a cog_worker.Manager which handles chunking
manager = Manager(
proj = 'wgs84', # any pyproj string
scale = 0.083333, # in projection units (degrees or meters)
bounds = (-180, -90, 180, 90),
buffer = 128 # buffer pixels when chunking analysis
)
# preview analysis
arr, bbox = manager.preview(MyAnalysis, max_size=1024)
rio.plot.show(arr)
# preview analysis chunks
for bbox in manager.chunks(chunksize=1500):
print(bbox)
# execute analysis chunks sequentially
for arr, bbox in manager.chunk_execute(MyAnalysis, chunksize=1500):
rio.plot.show(arr)
# generate job execution parameters
for params in manager.chunk_params(chunksize=1500):
print(params)
```
3. Write scale-dependent functions¶
```python
import scipy
def focal_mean(
worker: Worker,
kernel_radius: float = 1000 # radius in projection units (meters)
) -> np.ndarray:
array: np.ndarray = worker.read('sample-geotiff.tif')
# Access the pixel size at worker.scale
kernel_size = kernel_radius * 2 / worker.scale
array = scipy.ndimage.uniform_filter(array, kernel_size)
return array
```
4. Chunk your analysis and run it in a dask cluster
```python
from cog_worker.distributed import DaskManager
from dask.distributed import LocalCluster, Client
# Set up a Manager with that connects to a Dask cluster
cluster = LocalCluster()
client = Client(cluster)
distributed_manager = DaskManager(
client,
proj = 'wgs84',
scale = 0.083333,
bounds = (-180, -90, 180, 90),
buffer = 128
)
# Execute in worker pool and save chunks to disk as they complete.
distributed_manager.chunk_save('output.tif', MyAnalysis, chunksize=2048)
```
Raw data
{
"_id": null,
"home_page": null,
"name": "cog-worker",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "cog, geotiff, raster, gdal, rasterio, dask",
"author": null,
"author_email": "Francis Gassert <francis.gassert@vizzuality.com>",
"download_url": "https://files.pythonhosted.org/packages/48/ec/90cf862f7d2532c28328c5584760d36ae717c5b4aeeba6b4f19c74a00532/cog_worker-0.3.0.tar.gz",
"platform": null,
"description": "# Cog Worker\n\nScalable geospatial analysis on Cloud Optimized GeoTIFFs.\n\n - **Documentation**: https://vizzuality.github.io/cog_worker\n - **PyPI**: https://pypi.org/project/cog-worker\n\ncog_worker is a simple library to help write scripts to conduct scaleable\nanalysis of gridded data. It's intended to be useful for moderate- to large-scale\nGIS, remote sensing, and machine learning applications.\n\n## Installation\n\n```\npip install cog_worker\n```\n\n## Examples\n\nSee `docs/examples` for Jupyter notebook examples\n\n## Quick start\n\n0. A simple cog_worker script\n\n```python\nfrom rasterio.plot import show\nfrom cog_worker import Manager\n\ndef my_analysis(worker):\n arr = worker.read('roads_cog.tif')\n return arr\n\nmanager = Manager(proj='wgs84', scale=0.083333)\narr, bbox = manager.preview(my_analysis)\nshow(arr)\n```\n\n1. Define an analysis function that recieves a cog_worker.Worker as the first parameter.\n\n```python\nfrom cog_worker import Worker, Manager\nimport numpy as np\n\n# Define an analysis function to read and process COG data sources\ndef MyAnalysis(worker: Worker) -> np.ndarray:\n\n # 1. Read a COG (reprojecting, resampling and clipping as necessary)\n array: np.ndarray = worker.read('roads_cog.tif')\n\n # 2. Work on the array\n # ...\n\n # 3. Return (or post to blob storage etc.)\n return array\n```\n\n2. Run your analysis in different scales and projections\n\n```python\nimport rasterio as rio\n\n# Run your analysis using a cog_worker.Manager which handles chunking\nmanager = Manager(\n proj = 'wgs84', # any pyproj string\n scale = 0.083333, # in projection units (degrees or meters)\n bounds = (-180, -90, 180, 90),\n buffer = 128 # buffer pixels when chunking analysis\n)\n\n# preview analysis\narr, bbox = manager.preview(MyAnalysis, max_size=1024)\nrio.plot.show(arr)\n\n# preview analysis chunks\nfor bbox in manager.chunks(chunksize=1500):\n print(bbox)\n\n# execute analysis chunks sequentially\nfor arr, bbox in manager.chunk_execute(MyAnalysis, chunksize=1500):\n rio.plot.show(arr)\n\n# generate job execution parameters\nfor params in manager.chunk_params(chunksize=1500):\n print(params)\n```\n\n3. Write scale-dependent functions\u00b6\n\n```python\nimport scipy\n\ndef focal_mean(\n worker: Worker,\n kernel_radius: float = 1000 # radius in projection units (meters)\n) -> np.ndarray:\n\n array: np.ndarray = worker.read('sample-geotiff.tif')\n\n # Access the pixel size at worker.scale\n kernel_size = kernel_radius * 2 / worker.scale\n array = scipy.ndimage.uniform_filter(array, kernel_size)\n\n return array\n```\n\n4. Chunk your analysis and run it in a dask cluster\n\n```python\nfrom cog_worker.distributed import DaskManager\nfrom dask.distributed import LocalCluster, Client\n\n# Set up a Manager with that connects to a Dask cluster\ncluster = LocalCluster()\nclient = Client(cluster)\ndistributed_manager = DaskManager(\n client,\n proj = 'wgs84',\n scale = 0.083333,\n bounds = (-180, -90, 180, 90),\n buffer = 128\n)\n\n# Execute in worker pool and save chunks to disk as they complete.\ndistributed_manager.chunk_save('output.tif', MyAnalysis, chunksize=2048)\n```\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2021 Simbiotica. S.L. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
"summary": "Scalable geospatial analysis on Cloud Optimized GeoTIFFs.",
"version": "0.3.0",
"project_urls": {
"Homepage": "https://github.com/vizzuality/cog_worker",
"Issues": "https://github.com/vizzuality/cog_worker/issues"
},
"split_keywords": [
"cog",
" geotiff",
" raster",
" gdal",
" rasterio",
" dask"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c4172567c14d882aaa92d06cf55a887948f002f7fb76eb3d933a0019cf8c85c1",
"md5": "757417aea9496ae65161e9e2805ed6dc",
"sha256": "eff959ec4f0d19a9a0bffc48b12c868e9d675768c70495b6e4a445e959f7c69f"
},
"downloads": -1,
"filename": "cog_worker-0.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "757417aea9496ae65161e9e2805ed6dc",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 17066,
"upload_time": "2024-07-22T17:27:05",
"upload_time_iso_8601": "2024-07-22T17:27:05.252745Z",
"url": "https://files.pythonhosted.org/packages/c4/17/2567c14d882aaa92d06cf55a887948f002f7fb76eb3d933a0019cf8c85c1/cog_worker-0.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "48ec90cf862f7d2532c28328c5584760d36ae717c5b4aeeba6b4f19c74a00532",
"md5": "7ea5d8a463ada5ab2637b333d74f0fa0",
"sha256": "897b06c8cae9fcb38e67775ad33d3a500b0fc8ad321687dd47fd5ed0407ab9cb"
},
"downloads": -1,
"filename": "cog_worker-0.3.0.tar.gz",
"has_sig": false,
"md5_digest": "7ea5d8a463ada5ab2637b333d74f0fa0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 17500,
"upload_time": "2024-07-22T17:27:33",
"upload_time_iso_8601": "2024-07-22T17:27:33.402794Z",
"url": "https://files.pythonhosted.org/packages/48/ec/90cf862f7d2532c28328c5584760d36ae717c5b4aeeba6b4f19c74a00532/cog_worker-0.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-07-22 17:27:33",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "vizzuality",
"github_project": "cog_worker",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"tox": true,
"lcname": "cog-worker"
}
