# stactools-noaa-hrrr
[](https://pypi.org/project/stactools-noaa-hrrr/)
- Name: noaa-hrrr
- Package: `stactools.noaa_hrrr`
- [stactools-noaa-hrrr on PyPI](https://pypi.org/project/stactools-noaa-hrrr/)
- Owner: @hrodmn
- [Dataset homepage](https://rapidrefresh.noaa.gov/hrrr/)
- STAC extensions used:
- [forecast](https://github.com/stac-extensions/forecast)
- [item-assets](https://github.com/stac-extensions/item-assets)
- [datacube](https://github.com/stac-extensions/datacube)
- Extra fields:
- `noaa-hrrr:forecast_cycle_type`: either standard (18-hour) or extended (48-hour)
- `noaa-hrrr:region`: either `conus` or `alaska`
- [Browse the example in human-readable form](https://radiantearth.github.io/stac-browser/#/external/raw.githubusercontent.com/stactools-packages/noaa-hrrr/main/examples/collection.json)
- [Browse a notebook demonstrating the example item and collection](https://github.com/stactools-packages/noaa-hrrr/tree/main/docs/example.ipynb)
This package can be used to generate STAC metadata for the NOAA High Resolution Rapid
Refresh (HRRR) atmospheric forecast dataset.
The data are uploaded to cloud storage in AWS, Azure, and Google so you can pick
which cloud provider you want to use for the `grib` and `index` hrefs using the
`cloud_provider` argument to the functions in `stactools.noaa_hrrr.stac`.
## Background
The [NOAA HRRR dataset](https://www.nco.ncep.noaa.gov/pmb/products/hrrr/#CO)
is a continuously updated atmospheric forecast data product.
### Data structure
- There are two regions: CONUS and Alaska
- Every hour, new hourly forecasts are generated for many atmospheric attributes
for each region
- All hours (`00-23`) get an 18-hour forecast in the `conus` region
- Forecasts are generated every three hours (`00`, `03`, `06`, etc) in the
`alaska` region
- On hours `00`, `06`, `12`, `18` a 48-hour forecast is generated
- One of the products (`subh`) gets 15 minute forecasts (four per hour per
attribute), but the sub-hourly forecasts are stored as layers within a
single GRIB2 file for the forecast hour rather than in separate files.
- The forecasts are broken up into 4 products (`sfc`, `prs`, `nat`, `subh`),
- Each GRIB2 file has hundreds to thousands of variables
- Each .grib2 file is accompanied by a .grib2.idx which has variable-level
metadata including the starting byte for the data in that variable (useful
for making range requests instead of reading the entire file) and some
other descriptive metadata
### Summary of Considerations for Organizing STAC Metadata
After extensive discussions, we decided to organize the STAC metadata with
the following structure:
1. **Collections**: Separate collections for each region-product combination
- regions: `conus` and `alaska`
- products: `sfc`, `prs`, `nat`, and `subh`
2. **Items**: Each GRIB file in the archive is represented as an item with two assets:
- `"grib"`: Contains the actual data.
- `"index"`: The .grib2.idx sidecar file.
Each GRIB file contains the forecasts for all of a product's variables for a
particular forecast hour from a reference time, so you need to combine data
from multiple items to construct a time series for a forecast.
3. **`grib:layers`**: Within each `"grib"` asset, a `grib:layers` property details
each layer's information, including description, units, and byte ranges.
This enables applications to access specific parts of the GRIB2 files without
downloading the entire file.
- We intend to propose a `GRIB` STAC extension with the `grib:layers` property
for storing byte-ranges after testing this specification out on other GRIB2
datasets.
- The layer-level metadata is worth storing in STAC because you can construct
URIs for specific layers that GDAL can read using either `/vsisubfile` or
`vrt://`:
- `/vsisubfile/{start_byte}_{byte_size},/vsicurl/{grib_href}`
- `vrt:///vsicurl/{grib_href}?bands={grib_message}`, where `grib_message` is
the index of the layer within the GRIB2 file.
- under the hood, GDAL's `vrt` driver is reading the sidecar .grib2.idx file
and translating it into a `/vsisubfile` URI.
### Advantages
- Applications can use `grib:layers` to create layer-specific data sets, facilitating
efficient data handling.
- Splitting by region and product allows defining coherent collection-level datacube
metadata, enhancing accessibility.
### Disadvantages
- Storing layer-level metadata like byte ranges in the STAC metadata bloats the STAC
items because there are hundreds to thousands of layers in each GRIB2 file.
For more details, please refer to the related [issue discussion](https://github.com/developmentseed/noaa-hrrr/issues/1)
and pull requests [#3](https://github.com/developmentseed/noaa-hrrr/pull/3) and
[#6](https://github.com/developmentseed/noaa-hrrr/pull/6).
## STAC examples
- [Collection](examples/collection.json)
- [Item](examples/hrrr-conus-sfc-2024-05-10T12-FH0/hrrr-conus-sfc-2024-05-10T12-FH0.json)
## Python usage example
- Check out the [example notebook](./docs/example.ipynb) for examples of how to
create STAC metadata and how to use STAC items with `grib:layers` metadata to
load the data into xarray.
## Installation
Install `stactools-noaa-hrrr` with pip:
```shell
pip install stactools-noaa-hrrr
```
## Command-line usage
To create a collection object:
```shell
stac noaahrrr create-collection {region} {product} {cloud_provider} {destination_file}
```
e.g.
```shell
stac noaahrrr create-collection conus sfc azure example-collection.json
```
To create an item:
```shell
stac noaahrrr create-item \
{region} \
{product} \
{cloud_provider} \
{reference_datetime} \
{forecast_hour} \
{destination_file}
```
e.g.
```shell
stac noaahrrr create-item conus sfc azure 2024-05-01T12 10 example-item.json
```
To create all items for a date range:
```shell
stac noaahrrr create-item-collection \
{region} \
{product} \
{cloud_provider} \
{start_date} \
{end_date} \
{destination_folder}
```
e.g.
```shell
stac noaahrrr create-item-collection conus sfc azure 2024-05-01 2024-05-31 /tmp/items
```
### Docker
You can launch a jupyterhub server in a docker container with all of the
dependencies installed using these commands:
```shell
docker/build
docker/jupyter
```
Use `stac noaahrrr --help` to see all subcommands and options.
## Contributing
We use [pre-commit](https://pre-commit.com/) to check any changes.
To set up your development environment:
```shell
pip install -e '.[dev]'
pre-commit install
```
To check all files:
```shell
pre-commit run --all-files
```
To run the tests:
```shell
pytest -vv
```
If you've updated the STAC metadata output, update the examples:
```shell
scripts/update-examples
```
Raw data
{
"_id": null,
"home_page": null,
"name": "stactools-noaa-hrrr",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "Henry Rodman <henry@developmentseed.org>",
"keywords": "stactools, pystac, catalog, STAC",
"author": null,
"author_email": "Henry Rodman <henry@developmentseed.org>",
"download_url": "https://files.pythonhosted.org/packages/35/b7/3708fd713836dbbeb587eb62c4006611396e20b070e9a03619edf2445e32/stactools_noaa_hrrr-0.1.2.tar.gz",
"platform": null,
"description": "# stactools-noaa-hrrr\n\n[](https://pypi.org/project/stactools-noaa-hrrr/)\n\n\n- Name: noaa-hrrr\n- Package: `stactools.noaa_hrrr`\n- [stactools-noaa-hrrr on PyPI](https://pypi.org/project/stactools-noaa-hrrr/)\n- Owner: @hrodmn\n- [Dataset homepage](https://rapidrefresh.noaa.gov/hrrr/)\n- STAC extensions used:\n - [forecast](https://github.com/stac-extensions/forecast)\n - [item-assets](https://github.com/stac-extensions/item-assets)\n - [datacube](https://github.com/stac-extensions/datacube)\n- Extra fields:\n - `noaa-hrrr:forecast_cycle_type`: either standard (18-hour) or extended (48-hour)\n - `noaa-hrrr:region`: either `conus` or `alaska`\n- [Browse the example in human-readable form](https://radiantearth.github.io/stac-browser/#/external/raw.githubusercontent.com/stactools-packages/noaa-hrrr/main/examples/collection.json)\n- [Browse a notebook demonstrating the example item and collection](https://github.com/stactools-packages/noaa-hrrr/tree/main/docs/example.ipynb)\n\n\n\nThis package can be used to generate STAC metadata for the NOAA High Resolution Rapid\nRefresh (HRRR) atmospheric forecast dataset.\n\nThe data are uploaded to cloud storage in AWS, Azure, and Google so you can pick\nwhich cloud provider you want to use for the `grib` and `index` hrefs using the\n`cloud_provider` argument to the functions in `stactools.noaa_hrrr.stac`.\n\n## Background\n\nThe [NOAA HRRR dataset](https://www.nco.ncep.noaa.gov/pmb/products/hrrr/#CO)\nis a continuously updated atmospheric forecast data product.\n\n### Data structure\n\n- There are two regions: CONUS and Alaska\n- Every hour, new hourly forecasts are generated for many atmospheric attributes\n for each region\n - All hours (`00-23`) get an 18-hour forecast in the `conus` region\n - Forecasts are generated every three hours (`00`, `03`, `06`, etc) in the\n `alaska` region\n - On hours `00`, `06`, `12`, `18` a 48-hour forecast is generated\n - One of the products (`subh`) gets 15 minute forecasts (four per hour per\n attribute), but the sub-hourly forecasts are stored as layers within a\n single GRIB2 file for the forecast hour rather than in separate files.\n- The forecasts are broken up into 4 products (`sfc`, `prs`, `nat`, `subh`), \n- Each GRIB2 file has hundreds to thousands of variables\n- Each .grib2 file is accompanied by a .grib2.idx which has variable-level\n metadata including the starting byte for the data in that variable (useful\n for making range requests instead of reading the entire file) and some\n other descriptive metadata\n\n### Summary of Considerations for Organizing STAC Metadata\n\nAfter extensive discussions, we decided to organize the STAC metadata with\nthe following structure:\n\n1. **Collections**: Separate collections for each region-product combination\n - regions: `conus` and `alaska`\n - products: `sfc`, `prs`, `nat`, and `subh`\n\n2. **Items**: Each GRIB file in the archive is represented as an item with two assets:\n - `\"grib\"`: Contains the actual data.\n - `\"index\"`: The .grib2.idx sidecar file.\n\n Each GRIB file contains the forecasts for all of a product's variables for a\n particular forecast hour from a reference time, so you need to combine data\n from multiple items to construct a time series for a forecast.\n\n3. **`grib:layers`**: Within each `\"grib\"` asset, a `grib:layers` property details\n each layer's information, including description, units, and byte ranges.\n This enables applications to access specific parts of the GRIB2 files without\n downloading the entire file.\n\n - We intend to propose a `GRIB` STAC extension with the `grib:layers` property\n for storing byte-ranges after testing this specification out on other GRIB2\n datasets.\n - The layer-level metadata is worth storing in STAC because you can construct\n URIs for specific layers that GDAL can read using either `/vsisubfile` or\n `vrt://`:\n - `/vsisubfile/{start_byte}_{byte_size},/vsicurl/{grib_href}`\n - `vrt:///vsicurl/{grib_href}?bands={grib_message}`, where `grib_message` is\n the index of the layer within the GRIB2 file.\n - under the hood, GDAL's `vrt` driver is reading the sidecar .grib2.idx file\n and translating it into a `/vsisubfile` URI.\n\n### Advantages\n\n- Applications can use `grib:layers` to create layer-specific data sets, facilitating\nefficient data handling.\n- Splitting by region and product allows defining coherent collection-level datacube\nmetadata, enhancing accessibility.\n\n### Disadvantages\n\n- Storing layer-level metadata like byte ranges in the STAC metadata bloats the STAC\n items because there are hundreds to thousands of layers in each GRIB2 file.\n\nFor more details, please refer to the related [issue discussion](https://github.com/developmentseed/noaa-hrrr/issues/1)\nand pull requests [#3](https://github.com/developmentseed/noaa-hrrr/pull/3) and\n[#6](https://github.com/developmentseed/noaa-hrrr/pull/6).\n\n## STAC examples\n\n- [Collection](examples/collection.json)\n- [Item](examples/hrrr-conus-sfc-2024-05-10T12-FH0/hrrr-conus-sfc-2024-05-10T12-FH0.json)\n\n## Python usage example\n\n- Check out the [example notebook](./docs/example.ipynb) for examples of how to\n create STAC metadata and how to use STAC items with `grib:layers` metadata to\n load the data into xarray.\n\n## Installation\n\nInstall `stactools-noaa-hrrr` with pip:\n\n```shell\npip install stactools-noaa-hrrr\n```\n\n## Command-line usage\n\nTo create a collection object:\n\n```shell\nstac noaahrrr create-collection {region} {product} {cloud_provider} {destination_file}\n```\n\ne.g.\n\n```shell\nstac noaahrrr create-collection conus sfc azure example-collection.json\n```\n\nTo create an item:\n\n```shell\nstac noaahrrr create-item \\\n {region} \\\n {product} \\\n {cloud_provider} \\\n {reference_datetime} \\\n {forecast_hour} \\\n {destination_file}\n```\n\ne.g.\n\n```shell\nstac noaahrrr create-item conus sfc azure 2024-05-01T12 10 example-item.json\n```\n\nTo create all items for a date range:\n\n```shell\nstac noaahrrr create-item-collection \\\n {region} \\\n {product} \\\n {cloud_provider} \\\n {start_date} \\\n {end_date} \\\n {destination_folder}\n```\n\ne.g.\n\n```shell\nstac noaahrrr create-item-collection conus sfc azure 2024-05-01 2024-05-31 /tmp/items\n```\n\n### Docker\n\nYou can launch a jupyterhub server in a docker container with all of the\ndependencies installed using these commands:\n\n```shell\ndocker/build\ndocker/jupyter\n```\n\nUse `stac noaahrrr --help` to see all subcommands and options.\n\n## Contributing\n\nWe use [pre-commit](https://pre-commit.com/) to check any changes.\nTo set up your development environment:\n\n```shell\npip install -e '.[dev]'\npre-commit install\n```\n\nTo check all files:\n\n```shell\npre-commit run --all-files\n```\n\nTo run the tests:\n\n```shell\npytest -vv\n```\n\nIf you've updated the STAC metadata output, update the examples:\n\n```shell\nscripts/update-examples\n```\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "NOAA High-Resolution Rapid Refresh (HRRR) stactools package",
"version": "0.1.2",
"project_urls": {
"Github": "https://github.com/stactools-packages/noaa-hrrr",
"Issues": "https://github.com/stactools-packages/noaa-hrrr/issues"
},
"split_keywords": [
"stactools",
" pystac",
" catalog",
" stac"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "5c3fc5f990d02b9fa5e71354a37a1ea72c2e1b30fa9ebbba1036c1762f374165",
"md5": "4490815d255627ccf6cc6732e5101ecd",
"sha256": "07f87845d300f85d56e092fb2b054103ed8fa690866da75205effc70f77ea3b5"
},
"downloads": -1,
"filename": "stactools_noaa_hrrr-0.1.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "4490815d255627ccf6cc6732e5101ecd",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 1063633,
"upload_time": "2024-08-16T20:53:45",
"upload_time_iso_8601": "2024-08-16T20:53:45.953978Z",
"url": "https://files.pythonhosted.org/packages/5c/3f/c5f990d02b9fa5e71354a37a1ea72c2e1b30fa9ebbba1036c1762f374165/stactools_noaa_hrrr-0.1.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "35b73708fd713836dbbeb587eb62c4006611396e20b070e9a03619edf2445e32",
"md5": "3eb0e6e44f68ba3d07e0293b8f343d95",
"sha256": "5f952669b49c8417926805ac1777eb4f03c1deb6ab5939952d7373553eb6eb56"
},
"downloads": -1,
"filename": "stactools_noaa_hrrr-0.1.2.tar.gz",
"has_sig": false,
"md5_digest": "3eb0e6e44f68ba3d07e0293b8f343d95",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 1068430,
"upload_time": "2024-08-16T20:53:48",
"upload_time_iso_8601": "2024-08-16T20:53:48.311039Z",
"url": "https://files.pythonhosted.org/packages/35/b7/3708fd713836dbbeb587eb62c4006611396e20b070e9a03619edf2445e32/stactools_noaa_hrrr-0.1.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-16 20:53:48",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "stactools-packages",
"github_project": "noaa-hrrr",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "stactools-noaa-hrrr"
}
JSON
