About hrds
===========
hrds is a python package for obtaining points from a set of rasters at
different resolutions.
You can request a point and hrds will return a value based on
the highest resolution dataset (as defined by the user) available at that point, blending
datasets in a buffer region to ensure consistency.
[](https://github.com/EnvModellingGroup/hrds/actions)
Current release:
[](https://zenodo.org/badge/latestdoi/155502078)
Paper: [](https://doi.org/10.21105/joss.01112)
Prerequisites
---------------
* python 3+
* numpy
* scipy
* osgeo.gdal (pygdal) to read and write raster data
hrds is available on conda-forge, so you can install easily using:
```bash
conda config --add channels conda-forge
conda install hrds
```
It is possible to list all of the versions of hrds available on your platform with:
```bash
conda search hrds --channel conda-forge
```
On Debian-based Linux you can also install manually. To install pygdal,
first install the libgdal-dev packages and binaries:
```bash
sudo apt-get install libgdal-dev gdal-bin
```
To install pygdal, first check which version of gdal is installed:
```bash
gdal-config --version
```
pygdal can be installed using pip, specifying the version obtained from the command above.
Note that you may need to increase the minor version number,
e.g. from 2.1.3 to 2.1.3.3.
```bash
pip install pygdal==2.1.3.3
```
Replace 2.1.3.3 with the output from the ``gdal-config`` command.
You can then install hrds from source using the standard:
```bash
python setup.py install
```
or from PyPi:
```bash
pip install hrds
```
Functionality
-------------
* Create buffer zones as a preprocessing step if needed
* Obtain value at a point based on user-defined priority of rasters
The software assumes all rasters are already in the same projection space and using the same datum.
This example loads in an XYZ file and obtains data at each point,
replacing the Z value with that from hrds.
```python
from hrds import HRDS
points = []
with open("test_mesh.csv",'r') as f:
for line in f:
row = line.split(",")
# grab X and Y
points.append([float(row[0]), float(row[1])])
bathy = HRDS("gebco_uk.tif",
rasters=("emod_utm.tif",
"inspire_data.tif"),
distances=(700, 200))
bathy.set_bands()
print(len(points))
with open("output.xyz","w") as f:
for p in points:
f.write(str(p[0])+"\t"+str(p[1])+"\t"+str(bathy.get_val(p))+"\n")
```
This will turn this:
```bash
$ head test_mesh.csv
805390.592314,5864132.9269,0
805658.162910036,5862180.30440542,0
805925.733505999,5860227.68191137,0
806193.304101986,5858275.05941714,0
806460.874698054,5856322.43692232,0
806728.445294035,5854369.81442814,0
806996.015889997,5852417.19193409,0
807263.586486046,5850464.56943942,0
807531.157082069,5848511.94694493,0
807798.727678031,5846559.32445088,0
```
into this:
```bash
$ head output.xyz
805390.592314 5864132.9269 -10.821567728305235
805658.16291 5862180.30441 2.721575532084955
805925.733506 5860227.68191 2.528217188012767
806193.304102 5858275.05942 3.1063558741547865
806460.874698 5856322.43692 5.470234157891056
806728.445294 5854369.81443 1.382685066254607
806996.01589 5852417.19193 1.8997482922322515
807263.586486 5850464.56944 4.0836843606647335
807531.157082 5848511.94694 -2.39508079759155
807798.727678 5846559.32445 -2.401006071401176
```
An example of use via [thetis](http://thetisproject.org/):
```python
import firedrake
import thetis
from hrds import HRDS
mesh2d = firedrake.Mesh('test_mesh.msh') # mesh file
P1_2d = firedrake.FunctionSpace(mesh2d, 'CG', 1)
bathymetry2d = firedrake.Function(P1_2d, name="bathymetry")
bvector = bathymetry2d.dat.data
bathy = HRDS("gebco_uk.tif",
rasters=("emod_utm.tif",
"inspire_data.tif"),
distances=(700, 200))
bathy.set_bands()
for i, (xy) in enumerate(mesh2d.coordinates.dat.data):
bvector[i] = bathy.get_val(xy)
thetis.File('bathy.pvd').write(bathymetry2d)
# rest of thetis code
```
These images show the original data in QGIS in the top right, with each data set using a different colour scheme (GEBCO - green-blue; EMOD - grey; UK Gov - plasma - highlighted by the black rectangle).The red line is the boundary of the mesh used (see figure below). Both the EMOD and UK Gov data has NODATA areas, which are shown as transparent here, hence the curved left edge of the EMOD data. The figure also shows the buffer regions created around the two higher resolution datasets (top left), with black showing that data isn't used to white where it is 100% used. The effect of NODATA is clear here. The bottom panel shows a close-up of the UK Gov data with the buffer overlayed as a transparancy from white (not used) to black (100% UK Gov). The coloured polygon is the area of the high resolution mesh (see below).
After running the code above, we produce this blended dataset. Note the coarse mesh used here - it's not realistic for a model simulation!
If we then zoom-in to the high resolution area we can see the high resolution UK Gov data being used and with no obvious lines between datasets.
Community
-----------
We welcome suggestions for future improvements, bug reports and other issues via the issue tracker. Anyone wishing to contribute code should contact Jon Hill (jon.hill@york.ac.uk) to discuss.
Raw data
{
"_id": null,
"home_page": "https://github.com/EnvModellingGroup/hrds",
"name": "HRDS",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "raster,gis,modelling",
"author": "Jon Hill",
"author_email": "Jon Hill <jon.hill@york.ac.uk>",
"download_url": "https://files.pythonhosted.org/packages/4f/ab/4590532ef298c230703be4f34408aca6e700e4798adbcd88b1a6c4a8dc72/HRDS-0.1.4.tar.gz",
"platform": null,
"description": "\n\n\n\nAbout hrds\n===========\nhrds is a python package for obtaining points from a set of rasters at\ndifferent resolutions.\nYou can request a point and hrds will return a value based on\nthe highest resolution dataset (as defined by the user) available at that point, blending\ndatasets in a buffer region to ensure consistency.\n\n[](https://github.com/EnvModellingGroup/hrds/actions)\n\nCurrent release:\n[](https://zenodo.org/badge/latestdoi/155502078)\n\nPaper: [](https://doi.org/10.21105/joss.01112)\n\nPrerequisites\n---------------\n* python 3+\n* numpy\n* scipy\n* osgeo.gdal (pygdal) to read and write raster data\n\nhrds is available on conda-forge, so you can install easily using:\n\n```bash\nconda config --add channels conda-forge\nconda install hrds\n```\nIt is possible to list all of the versions of hrds available on your platform with:\n\n```bash\nconda search hrds --channel conda-forge\n```\n\nOn Debian-based Linux you can also install manually. To install pygdal,\nfirst install the libgdal-dev packages and binaries:\n\n```bash\nsudo apt-get install libgdal-dev gdal-bin\n```\n\nTo install pygdal, first check which version of gdal is installed:\n```bash\ngdal-config --version\n```\n\npygdal can be installed using pip, specifying the version obtained from the command above.\nNote that you may need to increase the minor version number,\ne.g. from 2.1.3 to 2.1.3.3.\n\n```bash\npip install pygdal==2.1.3.3\n```\nReplace 2.1.3.3 with the output from the ``gdal-config`` command.\n\nYou can then install hrds from source using the standard:\n\n```bash\npython setup.py install\n```\nor from PyPi:\n\n```bash\npip install hrds\n```\n\nFunctionality\n-------------\n\n* Create buffer zones as a preprocessing step if needed\n\n* Obtain value at a point based on user-defined priority of rasters\n\nThe software assumes all rasters are already in the same projection space and using the same datum.\n\n\nThis example loads in an XYZ file and obtains data at each point,\nreplacing the Z value with that from hrds.\n\n```python\n\nfrom hrds import HRDS\n\npoints = []\nwith open(\"test_mesh.csv\",'r') as f:\n for line in f:\n row = line.split(\",\")\n # grab X and Y\n points.append([float(row[0]), float(row[1])])\n\nbathy = HRDS(\"gebco_uk.tif\",\n rasters=(\"emod_utm.tif\",\n \"inspire_data.tif\"),\n distances=(700, 200))\nbathy.set_bands()\n\nprint(len(points))\n\nwith open(\"output.xyz\",\"w\") as f:\n for p in points:\n f.write(str(p[0])+\"\\t\"+str(p[1])+\"\\t\"+str(bathy.get_val(p))+\"\\n\")\n```\n\n\nThis will turn this:\n```bash\n$ head test_mesh.csv\n805390.592314,5864132.9269,0\n805658.162910036,5862180.30440542,0\n805925.733505999,5860227.68191137,0\n806193.304101986,5858275.05941714,0\n806460.874698054,5856322.43692232,0\n806728.445294035,5854369.81442814,0\n806996.015889997,5852417.19193409,0\n807263.586486046,5850464.56943942,0\n807531.157082069,5848511.94694493,0\n807798.727678031,5846559.32445088,0\n```\n\ninto this:\n\n```bash\n$ head output.xyz\n805390.592314\t5864132.9269\t-10.821567728305235\n805658.16291\t5862180.30441\t2.721575532084955\n805925.733506\t5860227.68191\t2.528217188012767\n806193.304102\t5858275.05942\t3.1063558741547865\n806460.874698\t5856322.43692\t5.470234157891056\n806728.445294\t5854369.81443\t1.382685066254607\n806996.01589\t5852417.19193\t1.8997482922322515\n807263.586486\t5850464.56944\t4.0836843606647335\n807531.157082\t5848511.94694\t-2.39508079759155\n807798.727678\t5846559.32445\t-2.401006071401176\n```\n\n\nAn example of use via [thetis](http://thetisproject.org/):\n\n```python\nimport firedrake\nimport thetis\nfrom hrds import HRDS\n\nmesh2d = firedrake.Mesh('test_mesh.msh') # mesh file\n\nP1_2d = firedrake.FunctionSpace(mesh2d, 'CG', 1)\nbathymetry2d = firedrake.Function(P1_2d, name=\"bathymetry\")\nbvector = bathymetry2d.dat.data\nbathy = HRDS(\"gebco_uk.tif\", \n rasters=(\"emod_utm.tif\", \n \"inspire_data.tif\"), \n distances=(700, 200))\nbathy.set_bands()\nfor i, (xy) in enumerate(mesh2d.coordinates.dat.data):\n bvector[i] = bathy.get_val(xy)\nthetis.File('bathy.pvd').write(bathymetry2d)\n\n# rest of thetis code\n\n```\n\nThese images show the original data in QGIS in the top right, with each data set using a different colour scheme (GEBCO - green-blue; EMOD - grey; UK Gov - plasma - highlighted by the black rectangle).The red line is the boundary of the mesh used (see figure below). Both the EMOD and UK Gov data has NODATA areas, which are shown as transparent here, hence the curved left edge of the EMOD data. The figure also shows the buffer regions created around the two higher resolution datasets (top left), with black showing that data isn't used to white where it is 100% used. The effect of NODATA is clear here. The bottom panel shows a close-up of the UK Gov data with the buffer overlayed as a transparancy from white (not used) to black (100% UK Gov). The coloured polygon is the area of the high resolution mesh (see below).\n\n\n\nAfter running the code above, we produce this blended dataset. Note the coarse mesh used here - it's not realistic for a model simulation!\n\n\n\nIf we then zoom-in to the high resolution area we can see the high resolution UK Gov data being used and with no obvious lines between datasets.\n\n\n\nCommunity\n-----------\n\nWe welcome suggestions for future improvements, bug reports and other issues via the issue tracker. Anyone wishing to contribute code should contact Jon Hill (jon.hill@york.ac.uk) to discuss.\n\n",
"bugtrack_url": null,
"license": "",
"summary": "Read in multiple raster data files in a hierarchical fashion",
"version": "0.1.4",
"project_urls": {
"Homepage": "https://github.com/EnvModellingGroup/hrds"
},
"split_keywords": [
"raster",
"gis",
"modelling"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "b8eb1ffacf8576dfd6c2a1a6b66664f9955e5393792886bbc61a63edfbf25dd6",
"md5": "8adc96aad7d5a3d2a9e2308c4740c0a2",
"sha256": "99970b2815cde91d5648fe3de486502ff5f077dc1b387d35bf3ff6e1e5151d23"
},
"downloads": -1,
"filename": "HRDS-0.1.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8adc96aad7d5a3d2a9e2308c4740c0a2",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 26279,
"upload_time": "2023-08-29T10:33:40",
"upload_time_iso_8601": "2023-08-29T10:33:40.479582Z",
"url": "https://files.pythonhosted.org/packages/b8/eb/1ffacf8576dfd6c2a1a6b66664f9955e5393792886bbc61a63edfbf25dd6/HRDS-0.1.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "4fab4590532ef298c230703be4f34408aca6e700e4798adbcd88b1a6c4a8dc72",
"md5": "dc54ec6f6f498606cb9fd0b355c98390",
"sha256": "d9409bb00e3ee26465445fc8eb2592ceaa804094ff404d8eb1aa84b843ecabdd"
},
"downloads": -1,
"filename": "HRDS-0.1.4.tar.gz",
"has_sig": false,
"md5_digest": "dc54ec6f6f498606cb9fd0b355c98390",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 29946,
"upload_time": "2023-08-29T10:33:42",
"upload_time_iso_8601": "2023-08-29T10:33:42.127750Z",
"url": "https://files.pythonhosted.org/packages/4f/ab/4590532ef298c230703be4f34408aca6e700e4798adbcd88b1a6c4a8dc72/HRDS-0.1.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-29 10:33:42",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "EnvModellingGroup",
"github_project": "hrds",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "hrds"
}
JSON
