| Name | goes-dl JSON |
| Version |
0.1rc2
JSON |
| download |
| home_page | None |
| Summary | Satellite imagery downloader for GOES and GridSat datasets |
| upload_time | 2024-10-29 15:10:51 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.8 |
| license | MIT License Copyright (c) 2024 Waldemar Villamayor-Venialbo Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
goes
goes-r
gridsat
gridsat-b1
gridsat-goes/conus
atmospheric science
dataset
downloader
environmental data
meteorology
multi-source
satellite imagery
severe weather
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
boto3
mypy-boto3-s3
requests
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# GOES-DL
*Since 1975, Geostationary Operational Environmental Satellites (GOES) have
provided continuous imagery and data on atmospheric conditions and solar
activity (space weather). They have even aided in search and rescue of people
in distress. GOES data products have led to more accurate and timely weather
forecasts and better understanding of long-term climate conditions. The
National Aeronautics and Space Administration (NASA) builds and launches the
GOES, and the National Oceanic and Atmospheric Administration (NOAA) operates
them [[6](#hist)].*
**GOES-DL** is an open-source Python package that simplifies the process of
downloading satellite imagery datasets from various NOAA archives. The package
supports both second and third-generation GOES satellite data
[[4](#goesi),[7](#goesr)], as well as the Gridded Satellite B1
(GridSat-B1) Climate Data Record [[3](#gridb1)]. GOES-DL provides an
easy-to-use interface to access data for scientific analysis, research, and
other applications.
## Key Features
- **Real-time GOES 3rd Generation Satellite Data (GOES Series R)**: Access
real-time and archived data from NOAA's Amazon Web Services (AWS) cloud
archive.
- **Gridded Satellite Data from GOES 2nd Generation (GridSat-GOES)**: Download
data from NOAA's National Centers for Environmental Information (NCEI)
archive.
- **Gridded Satellite Data from ISCCP B1 (GridSat-B1)**: Fetch historical
climate data from both NOAA's AWS archive and the NCEI archive.
- Seamless integration of different data sources into a unified download
process.
- High-level API that abstracts away the complexity of data access from NOAA
archives.
## Supported Datasets
1. **GOES 2nd Generation (GOES-8 to GOES-15)**: Also known as the I to P
Series, these datasets provide environmental monitoring and meteorological
data for the Western Hemisphere [[4](#goesi)].
2. **GOES 3rd Generation (GOES-16 to GOES-18)**: Also known as the R to U
Series, these satellites offer advanced imagery and atmospheric measurements
with better spatial, spectral, and temporal resolution [[7](#goesr)].
3. **GridSat-B1 Climate Data Record (Version 2)**: Gridded satellite imagery
for climate research, containing global infrared window, visible, and water
vapor data over long time periods [[3](#gridb1)].
Refer to [Gridded Satellite GOES (GridSat-GOES) East and West Full Disk and
CONUS Coverage, Version 1][1] and [NOAA Climate Data Record (CDR) of Gridded
Satellite Data from ISCCP B1 (GridSat-B1) Infrared Channel Brightness
Temperature, Version 2][5] for more information on the data format and details
of the content.
See [NOAA Geostationary Operational Environmental Satellites (GOES) 16, 17 &
18][2] and [NOAA GOES on AWS (CICS)][3] for information on the GOES-R Series
data available from NOAA on AWS. You can find much more detailed information
about GOES-R Series data from NOAA's [Geostationary Operational Environmental
Satellites - R Series][4].
## Installation
To install **GOES-DL**, use `pip`:
```bash
pip install goes-dl
```
## Usage
Below are examples of how to use the GOES-DL package to download data from each
of the supported sources.
### 1. Download GOES 2nd Generation Data (from NOAA's NCEI archive)
```python
# Import the locator and datasource according to your desired product
from GOES_DL.dataset.gridsat import GridSatProductLocatorGC
from GOES_DL.datasource import DatasourceHTTP
from GOES_DL.downloader import Downloader
# Initialize the downloader for GridSat-GOES (GOES-12, Full Disk)
locator = GridSatProductLocatorGC("F", "G12")
datasource = DatasourceHTTP(locator, repository="./my_data/gridsat-gc")
downloader = Downloader(
datasource=datasource,
locator=locator,
date_format="%Y-%m-%dT%H:%M%z", # use a custom short date format
)
# Set your desired date...
files1 = downloader.get_files(start="2012-08-23T00:00Z")
# ...or your desired date range
files2 = downloader.get_files(
start="2012-08-23T00:00-0004",
end="2012-08-24T00:00-0004",
)
# `files1` and files2` are lists of tuple[str, bytes] with file path and
# file content, respectively. The file path is relative to the base URL.
```
### 2. Download GOES 3rd Generation Data (from NOAA's AWS archive)
```python
# Import the locator and datasource according to your desired product
from GOES_DL.dataset.goes import GOESProductLocatorABIPP
from GOES_DL.datasource import DatasourceAWS
from GOES_DL.downloader import Downloader
# Initialize the downloader for GOES-R Series (set your desired product)
locator = GOESProductLocatorABIPP("CMIP", "F", ["C02", "C08", "C13"], "G16")
# GOES-16 data is updated every 10 minutes. If you are downloading
# old data, you may leave the cache refresh rate as default (+inf).
datasource = DatasourceAWS(locator, repository="./my_data/goes-r", cache=600)
downloader = Downloader(
datasource=datasource,
locator=locator,
)
# Set your desired date...
files1 = downloader.get_files(start="2024-08-23T00:00:00Z")
# ...or your desired date range
files2 = downloader.get_files(
start="2024-08-23T00:00:00-0004", # use the default date format
end="2024-08-24T00:00:00-0004",
)
# `files1` and files2` are lists of tuple[str, bytes] with file path and
# file content, respectively. The file path is relative to the base URL.
```
### 3. Download GridSat-B1 Data (from NOAA's AWS archive)
```python
# Import the locator and datasource according to your desired product
from GOES_DL.dataset.gridsat import GridSatProductLocatorB1
from GOES_DL.datasource import DatasourceAWS
from GOES_DL.downloader import Downloader
# Initialize the downloader for GridSat-B1
locator = GridSatProductLocatorB1()
# Also available in HTTP from NCEI's archive, see next example
datasource = DatasourceAWS(locator, repository="./my_data/gridsat-b1")
downloader = Downloader(
datasource=datasource,
locator=locator,
date_format="%Y-%m-%dT%H:%M%z",
)
# Set your desired date...
files1 = downloader.get_files(start="1984-08-23T00:00Z")
# ...or your desired date range
files2 = downloader.get_files(
start="1984-08-23T00:00-0004",
end="1984-08-24T00:00-0004",
)
# `files1` and files2` are lists of tuple[str, bytes] with file path and
# file content, respectively. The file path is relative to the base URL.
```
### 4. Download GridSat-B1 Data (from NOAA's NCEI archive)
```python
# Import the locator and datasource according to your desired product
from GOES_DL.dataset.gridsat import GridSatProductLocatorB1
from GOES_DL.datasource import DatasourceHTTP
from GOES_DL.downloader import Downloader
# Initialize the downloader for GridSat-B1
locator = GridSatProductLocatorB1()
# NCEI archive has the same folder structure as AWS, so, if you have
# downloaded data from AWS, you can use the same locator and change the
# datasource to HTTP. If a file is not found in the local repository, it
# will be downloaded from the remote datasource. In all previous examples,
# if a file was already downloaded, it will not be downloaded again.
datasource = DatasourceHTTP(locator, repository="./my_data/gridsat-b1")
downloader = Downloader(
datasource=datasource,
locator=locator,
date_format="%Y-%m-%dT%H:%M%z",
)
# Set your desired date...
files1 = downloader.get_files(start="1984-08-23T00:00Z")
# ...or your desired date range
files2 = downloader.get_files(
start="1984-08-23T00:00-0004",
end="1984-08-24T00:00-0004",
)
# `files1` and files2` are lists of tuple[str, bytes] with file path and
# file content, respectively. The file path is relative to the base URL.
```
## Pipeline and parameters
The general workflow for downloading data using **GOES-DL** is as follows:
1. **Initialize the locator**: Import the appropriate locator class for the
desired product and satellite and initialize a locator object. The product
locator provides the necessary information to find the data files in the
dataset repository. This is the only step that is specific to the dataset
being downloaded.
2. **Initialize the datasource**: Import the appropriate datasource class for
the desired dataset and instantiate a datasource object. The datasource
provides the necessary functionality to access the data files from the
repository, it abstracts the complexity of accessing data from different
sources. This step is common to all datasets. The datasource is Initialized
with the locator object.
3. **Initialize the downloader**: Import the downloader class and initialize a
downloader object. The downloader is the main interface for downloading data.
It is initialized with the datasource and locator objects, as well as the
date format to be used in the download process.
The Downloader.get_files method accepts the following parameters:
- **start_time**: A string specifying the starting date for the dataset to be
downloaded.
- **end_time**: A string specifying the ending date for the dataset to be
downloaded. If not provided, only the data for the start_time will be
downloaded.
The default date format is the ISO 8601 format with timezone information
(`"%Y-%m-%dT%H:%M%z"`). The date format can be changed by passing the desired
format to the downloader object during initialization.
## Data Sources
1. **NOAA NCEI Archive**: GridSat-B1 Climate Data Record and GOES-8 to GOES-15
data is available through NOAA’s National Centers for Environmental
Information.
2. **NOAA AWS Cloud Archive**: GOES-16 to GOES-18 data and GridSat-B1 Climate
Data Record are accessible via the NOAA archive hosted on AWS.
## Contributing
Contributions to **GOES-DL** are welcome! If you'd like to contribute:
1. Fork the repository.
2. Create a new branch for your feature or bugfix.
3. Open a pull request with a description of your changes.
Please make sure to include tests for any new functionality.
## Requirements
- Python 3.8+
- [requests](https://pypi.org/project/requests): A simple, yet elegant, HTTP
library for Python.
- [boto3](https://pypi.org/project/boto3): AWS SDK for Python.
## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
for details.
## Acknowledgments
This package relies on data provided by NOAA’s NCEI and NOAA’s archive on AWS.
## Credits
When using **GOES-DL** in any research, publication or website, please cite this
package as:
> Villamayor-Venialbo, W. (2024): *GOES-DL: A Python package for downloading
> GOES and GridSat-B1 satellite data (Version 0.1.1)* [Software]. GitHub.
> [git:wvenialbo/GOES-DL](https://github.com/wvenialbo/GOES-DL), *[indicate
> access date]*.
### Credits for GridSat-GOES/CONUS
**Dataset Citation:**
> Knapp, K. R. (2017): Gridded Satellite GOES Coverage Data (GridSat-GOES),
> *[indicate subset used]*. *NOAA National Centers for Environmental
> Information* [doi:10.7289/V5HM56GM](https://doi.org/10.7289/V5HM56GM),
> *[indicate access date]*.
Please cite the following article when using GridSat-GOES/CONUS data in any
publication:
> Knapp, K. R. and Wilkins, S. L.: Gridded Satellite (GridSat) GOES and CONUS
> data, *Earth System Science Data*, **10(3)**, 1417–1425,
> [doi:10.5194/essd-10-1417-2018](https://doi.org/10.5194/essd-10-1417-2018),
> 2018.
### Credits for GridSat-B1
**Dataset Citation:**
> Knapp, K. R.; NOAA CDR Program; (2014): NOAA Climate Data Record (CDR) of
> Gridded Satellite Data from ISCCP B1 (GridSat-B1) Infrared Channel Brightness
> Temperature, Version 2, *[indicate subset used]*. *NOAA National Centers for
> Environmental Information*.
> [doi:10.7289/V59P2ZKR](https://doi.org/10.7289/V59P2ZKR), *[indicate access
> date]*.
Please cite the following article when using GridSat-B1 data in any publication:
> Knapp, K. R., Ansari S.; Bain, C. L.; Bourassa, M. A.; Dickinson, M. J.; Funk,
> C.; Helms, C. N.; Hennon, C. C.; Holmes, C. D.; Huffman, G. J.; Kossin, J. P.;
> Lee, H.-T.; Loew, A.; and Magnusdottir, G.: Globally gridded satellite
> (GridSat) observations for climate studies. *Bulletin of the American
> Meteorological Society*, **92(7)**, 893-907,
> [doi:10.1175/2011BAMS3039.1](https://doi.org/10.1175/2011BAMS3039.1), 2011.
When possible, please cite the following article when using the ISCCP-B1 data or
other ISCCP-B1 imagery or GIBBS imagery in a publication or website:
> Knapp, K. R.: Scientific data stewardship of International Satellite Cloud
> Climatology Project B1 global geostationary observations. *Journal of Applied
> Remote Sensing*, **2(1)**, 023548,
> [doi:10.1117/1.3043461](https://doi.org/10.1117/1.3043461), 2008.
## Contact and Support
For issues, questions, or requests, feel free to open an issue on this
repository or contact the author, [wvenialbo at
gmail.com](mailto:wvenialbo@gmail.com).
---
## Similar Projects
- [Brian Blaylock's goes2go](https://github.com/blaylockbk/goes2go): Download
and process GOES-16 and GOES-17 data from NOAA's archive on AWS using Python.
([readthedocs](https://goes2go.readthedocs.io/))
- [Joao Henry's GOES](https://github.com/joaohenry23/GOES): Python package to
download and manipulate GOES-16/17/18 data.
## References
1. Knapp, K. R. (2008): Scientific data stewardship of International Satellite
Cloud Climatology Project B1 global geostationary observations. *Journal of
Applied Remote Sensing*, **2(1)**, 023548,
[doi:10.1117/1.3043461](https://doi.org/10.1117/1.3043461).
2. Knapp, K. R., Ansari S.; Bain, C. L.; Bourassa, M. A.; Dickinson, M. J.;
Funk, C.; Helms, C. N.; Hennon, C. C.; Holmes, C. D.; Huffman, G. J.; Kossin,
J. P.; Lee, H.-T.; Loew, A.; and Magnusdottir, G.; (2011): Globally gridded
satellite (GridSat) observations for climate studies. *Bulletin of the
American Meteorological Society*, **92(7)**, 893-907,
[doi:10.1175/2011BAMS3039.1](https://doi.org/10.1175/2011BAMS3039.1).
3. Knapp, K. R; NOAA CDR Program; (2014): NOAA Climate Data Record (CDR) of
Gridded Satellite Data from ISCCP B1 (GridSat-B1) Infrared Channel Brightness
Temperature, Version 2. *NOAA National Centers for Environmental
Information*, [doi:10.7289/V59P2ZKR](https://doi.org/10.7289/V59P2ZKR).<a
name="gridb1"></a>
4. Knapp, K. R; (2017): Gridded Satellite GOES Coverage Data (GridSat-GOES).
*NOAA National Centers for Environmental Information*.
[doi:10.7289/V5HM56GM](https://doi.org/10.7289/V5HM56GM).<a name="goesi"></a>
5. Knapp, K. R. and Wilkins, S. L.; (2018): Gridded Satellite (GridSat) GOES and
CONUS data, *Earth System Science Data*, 10(3), 1417–1425,
[doi:10.5194/essd-10-1417-2018](https://doi.org/10.5194/essd-10-1417-2018).
6. GOES History. *GOES-R Website*, https://www.goes-r.gov/mission/history.html,
retrieved on 2024.<a name="hist"></a>
7. GOES-R Series Data Products. *GOES-R Website*,
https://www.goes-r.gov/products/overview.html, retrieved on 2024.<a
name="goesr"></a>
8. NOAA Big Data Program, *NOAA Open Data Dissemination Program*,
https://github.com/NOAA-Big-Data-Program/bdp-data-docs, retrieved on 2024.
9. Beginner’s Guide to GOES-R Series Data: How to acquire, analyze, and
visualize GOES-R Series data, *Resources compiled by GOES-R Product Readiness
and Operations*, Satellite Products and Services Division, National Oceanic
and Atmospheric Administration.
[PDF](https://www.goes-r.gov/downloads/resources/documents/Beginners_Guide_to_GOES-R_Series_Data.pdf)
Last Updated on May 23, 2024, retrieved on 2024.
10. GOES-R Series Data Book, *GOES-R Series Program Office*, Goddard Space
Flight Center, National Aeronautics and Space Administration.
[PDF](https://www.goes-r.gov/downloads/resources/documents/GOES-RSeriesDataBook.pdf),
retrieved on 2024.
[0]: hidden_references:
[1]: https://www.ncei.noaa.gov/access/metadata/landing-page/bin/iso?id=gov.noaa.ncdc:C00993
[2]: https://registry.opendata.aws/noaa-goes/
[3]: https://docs.opendata.aws/noaa-goes16/cics-readme.html
[4]: https://www.goes-r.gov/
[5]: https://www.ncei.noaa.gov/access/metadata/landing-page/bin/iso?id=gov.noaa.ncdc:C00829
Raw data
{
"_id": null,
"home_page": null,
"name": "goes-dl",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "GOES, GOES-R, GridSat, GridSat-B1, GridSat-GOES/CONUS, atmospheric science, dataset, downloader, environmental data, meteorology, multi-source, satellite imagery, severe weather",
"author": null,
"author_email": "Waldemar Villamayor-Venialbo <wvenialbo@fpuna.edu.py>",
"download_url": "https://files.pythonhosted.org/packages/2d/bb/d78e887d14fad56ee563dc9ec64d17cfb19027f3ed1d6de25833fffc95da/goes_dl-0.1rc2.tar.gz",
"platform": null,
"description": "# GOES-DL\r\n\r\n*Since 1975, Geostationary Operational Environmental Satellites (GOES) have\r\nprovided continuous imagery and data on atmospheric conditions and solar\r\nactivity (space weather). They have even aided in search and rescue of people\r\nin distress. GOES data products have led to more accurate and timely weather\r\nforecasts and better understanding of long-term climate conditions. The\r\nNational Aeronautics and Space Administration (NASA) builds and launches the\r\nGOES, and the National Oceanic and Atmospheric Administration (NOAA) operates\r\nthem [[6](#hist)].*\r\n\r\n**GOES-DL** is an open-source Python package that simplifies the process of\r\ndownloading satellite imagery datasets from various NOAA archives. The package\r\nsupports both second and third-generation GOES satellite data\r\n[[4](#goesi),[7](#goesr)], as well as the Gridded Satellite B1\r\n(GridSat-B1) Climate Data Record [[3](#gridb1)]. GOES-DL provides an\r\neasy-to-use interface to access data for scientific analysis, research, and\r\nother applications.\r\n\r\n## Key Features\r\n\r\n- **Real-time GOES 3rd Generation Satellite Data (GOES Series R)**: Access\r\n real-time and archived data from NOAA's Amazon Web Services (AWS) cloud\r\n archive.\r\n\r\n- **Gridded Satellite Data from GOES 2nd Generation (GridSat-GOES)**: Download\r\n data from NOAA's National Centers for Environmental Information (NCEI)\r\n archive.\r\n\r\n- **Gridded Satellite Data from ISCCP B1 (GridSat-B1)**: Fetch historical\r\n climate data from both NOAA's AWS archive and the NCEI archive.\r\n\r\n- Seamless integration of different data sources into a unified download\r\n process.\r\n\r\n- High-level API that abstracts away the complexity of data access from NOAA\r\n archives.\r\n\r\n## Supported Datasets\r\n\r\n1. **GOES 2nd Generation (GOES-8 to GOES-15)**: Also known as the I to P\r\n Series, these datasets provide environmental monitoring and meteorological\r\n data for the Western Hemisphere [[4](#goesi)].\r\n\r\n2. **GOES 3rd Generation (GOES-16 to GOES-18)**: Also known as the R to U\r\n Series, these satellites offer advanced imagery and atmospheric measurements\r\n with better spatial, spectral, and temporal resolution [[7](#goesr)].\r\n\r\n3. **GridSat-B1 Climate Data Record (Version 2)**: Gridded satellite imagery\r\n for climate research, containing global infrared window, visible, and water\r\n vapor data over long time periods [[3](#gridb1)].\r\n\r\nRefer to [Gridded Satellite GOES (GridSat-GOES) East and West Full Disk and\r\nCONUS Coverage, Version 1][1] and [NOAA Climate Data Record (CDR) of Gridded\r\nSatellite Data from ISCCP B1 (GridSat-B1) Infrared Channel Brightness\r\nTemperature, Version 2][5] for more information on the data format and details\r\nof the content.\r\n\r\nSee [NOAA Geostationary Operational Environmental Satellites (GOES) 16, 17 &\r\n18][2] and [NOAA GOES on AWS (CICS)][3] for information on the GOES-R Series\r\ndata available from NOAA on AWS. You can find much more detailed information\r\nabout GOES-R Series data from NOAA's [Geostationary Operational Environmental\r\nSatellites - R Series][4].\r\n\r\n## Installation\r\n\r\nTo install **GOES-DL**, use `pip`:\r\n\r\n```bash\r\npip install goes-dl\r\n```\r\n\r\n## Usage\r\n\r\nBelow are examples of how to use the GOES-DL package to download data from each\r\nof the supported sources.\r\n\r\n### 1. Download GOES 2nd Generation Data (from NOAA's NCEI archive)\r\n\r\n```python\r\n# Import the locator and datasource according to your desired product\r\nfrom GOES_DL.dataset.gridsat import GridSatProductLocatorGC\r\nfrom GOES_DL.datasource import DatasourceHTTP\r\nfrom GOES_DL.downloader import Downloader\r\n\r\n# Initialize the downloader for GridSat-GOES (GOES-12, Full Disk)\r\nlocator = GridSatProductLocatorGC(\"F\", \"G12\")\r\n\r\ndatasource = DatasourceHTTP(locator, repository=\"./my_data/gridsat-gc\")\r\n\r\ndownloader = Downloader(\r\n datasource=datasource,\r\n locator=locator,\r\n date_format=\"%Y-%m-%dT%H:%M%z\", # use a custom short date format\r\n)\r\n\r\n# Set your desired date...\r\nfiles1 = downloader.get_files(start=\"2012-08-23T00:00Z\")\r\n\r\n# ...or your desired date range\r\nfiles2 = downloader.get_files(\r\n start=\"2012-08-23T00:00-0004\",\r\n end=\"2012-08-24T00:00-0004\",\r\n)\r\n\r\n# `files1` and files2` are lists of tuple[str, bytes] with file path and\r\n# file content, respectively. The file path is relative to the base URL.\r\n```\r\n\r\n### 2. Download GOES 3rd Generation Data (from NOAA's AWS archive)\r\n\r\n```python\r\n# Import the locator and datasource according to your desired product\r\nfrom GOES_DL.dataset.goes import GOESProductLocatorABIPP\r\nfrom GOES_DL.datasource import DatasourceAWS\r\nfrom GOES_DL.downloader import Downloader\r\n\r\n# Initialize the downloader for GOES-R Series (set your desired product)\r\nlocator = GOESProductLocatorABIPP(\"CMIP\", \"F\", [\"C02\", \"C08\", \"C13\"], \"G16\")\r\n\r\n# GOES-16 data is updated every 10 minutes. If you are downloading\r\n# old data, you may leave the cache refresh rate as default (+inf).\r\ndatasource = DatasourceAWS(locator, repository=\"./my_data/goes-r\", cache=600)\r\n\r\ndownloader = Downloader(\r\n datasource=datasource,\r\n locator=locator,\r\n)\r\n\r\n# Set your desired date...\r\nfiles1 = downloader.get_files(start=\"2024-08-23T00:00:00Z\")\r\n\r\n# ...or your desired date range\r\nfiles2 = downloader.get_files(\r\n start=\"2024-08-23T00:00:00-0004\", # use the default date format\r\n end=\"2024-08-24T00:00:00-0004\",\r\n)\r\n\r\n# `files1` and files2` are lists of tuple[str, bytes] with file path and\r\n# file content, respectively. The file path is relative to the base URL.\r\n```\r\n\r\n### 3. Download GridSat-B1 Data (from NOAA's AWS archive)\r\n\r\n```python\r\n# Import the locator and datasource according to your desired product\r\nfrom GOES_DL.dataset.gridsat import GridSatProductLocatorB1\r\nfrom GOES_DL.datasource import DatasourceAWS\r\nfrom GOES_DL.downloader import Downloader\r\n\r\n# Initialize the downloader for GridSat-B1\r\nlocator = GridSatProductLocatorB1()\r\n\r\n# Also available in HTTP from NCEI's archive, see next example\r\ndatasource = DatasourceAWS(locator, repository=\"./my_data/gridsat-b1\")\r\n\r\ndownloader = Downloader(\r\n datasource=datasource,\r\n locator=locator,\r\n date_format=\"%Y-%m-%dT%H:%M%z\",\r\n)\r\n\r\n# Set your desired date...\r\nfiles1 = downloader.get_files(start=\"1984-08-23T00:00Z\")\r\n\r\n# ...or your desired date range\r\nfiles2 = downloader.get_files(\r\n start=\"1984-08-23T00:00-0004\",\r\n end=\"1984-08-24T00:00-0004\",\r\n)\r\n\r\n# `files1` and files2` are lists of tuple[str, bytes] with file path and\r\n# file content, respectively. The file path is relative to the base URL.\r\n```\r\n\r\n### 4. Download GridSat-B1 Data (from NOAA's NCEI archive)\r\n\r\n```python\r\n# Import the locator and datasource according to your desired product\r\nfrom GOES_DL.dataset.gridsat import GridSatProductLocatorB1\r\nfrom GOES_DL.datasource import DatasourceHTTP\r\nfrom GOES_DL.downloader import Downloader\r\n\r\n# Initialize the downloader for GridSat-B1\r\nlocator = GridSatProductLocatorB1()\r\n\r\n# NCEI archive has the same folder structure as AWS, so, if you have\r\n# downloaded data from AWS, you can use the same locator and change the\r\n# datasource to HTTP. If a file is not found in the local repository, it\r\n# will be downloaded from the remote datasource. In all previous examples,\r\n# if a file was already downloaded, it will not be downloaded again.\r\ndatasource = DatasourceHTTP(locator, repository=\"./my_data/gridsat-b1\")\r\n\r\ndownloader = Downloader(\r\n datasource=datasource,\r\n locator=locator,\r\n date_format=\"%Y-%m-%dT%H:%M%z\",\r\n)\r\n\r\n# Set your desired date...\r\nfiles1 = downloader.get_files(start=\"1984-08-23T00:00Z\")\r\n\r\n# ...or your desired date range\r\nfiles2 = downloader.get_files(\r\n start=\"1984-08-23T00:00-0004\",\r\n end=\"1984-08-24T00:00-0004\",\r\n)\r\n\r\n# `files1` and files2` are lists of tuple[str, bytes] with file path and\r\n# file content, respectively. The file path is relative to the base URL.\r\n```\r\n\r\n## Pipeline and parameters\r\n\r\nThe general workflow for downloading data using **GOES-DL** is as follows:\r\n\r\n1. **Initialize the locator**: Import the appropriate locator class for the\r\n desired product and satellite and initialize a locator object. The product\r\n locator provides the necessary information to find the data files in the\r\n dataset repository. This is the only step that is specific to the dataset\r\n being downloaded.\r\n2. **Initialize the datasource**: Import the appropriate datasource class for\r\n the desired dataset and instantiate a datasource object. The datasource\r\n provides the necessary functionality to access the data files from the\r\n repository, it abstracts the complexity of accessing data from different\r\n sources. This step is common to all datasets. The datasource is Initialized\r\n with the locator object.\r\n3. **Initialize the downloader**: Import the downloader class and initialize a\r\n downloader object. The downloader is the main interface for downloading data.\r\n It is initialized with the datasource and locator objects, as well as the\r\n date format to be used in the download process.\r\n\r\nThe Downloader.get_files method accepts the following parameters:\r\n\r\n- **start_time**: A string specifying the starting date for the dataset to be\r\n downloaded.\r\n- **end_time**: A string specifying the ending date for the dataset to be\r\n downloaded. If not provided, only the data for the start_time will be\r\n downloaded.\r\n\r\nThe default date format is the ISO 8601 format with timezone information\r\n(`\"%Y-%m-%dT%H:%M%z\"`). The date format can be changed by passing the desired\r\nformat to the downloader object during initialization.\r\n\r\n## Data Sources\r\n\r\n1. **NOAA NCEI Archive**: GridSat-B1 Climate Data Record and GOES-8 to GOES-15\r\n data is available through NOAA\u2019s National Centers for Environmental\r\n Information.\r\n2. **NOAA AWS Cloud Archive**: GOES-16 to GOES-18 data and GridSat-B1 Climate\r\n Data Record are accessible via the NOAA archive hosted on AWS.\r\n\r\n## Contributing\r\n\r\nContributions to **GOES-DL** are welcome! If you'd like to contribute:\r\n\r\n1. Fork the repository.\r\n2. Create a new branch for your feature or bugfix.\r\n3. Open a pull request with a description of your changes.\r\n\r\nPlease make sure to include tests for any new functionality.\r\n\r\n## Requirements\r\n\r\n- Python 3.8+\r\n- [requests](https://pypi.org/project/requests): A simple, yet elegant, HTTP\r\n library for Python.\r\n- [boto3](https://pypi.org/project/boto3): AWS SDK for Python.\r\n\r\n## License\r\n\r\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file\r\nfor details.\r\n\r\n## Acknowledgments\r\n\r\nThis package relies on data provided by NOAA\u2019s NCEI and NOAA\u2019s archive on AWS.\r\n\r\n## Credits\r\n\r\nWhen using **GOES-DL** in any research, publication or website, please cite this\r\npackage as:\r\n\r\n> Villamayor-Venialbo, W. (2024): *GOES-DL: A Python package for downloading\r\n> GOES and GridSat-B1 satellite data (Version 0.1.1)* [Software]. GitHub.\r\n> [git:wvenialbo/GOES-DL](https://github.com/wvenialbo/GOES-DL), *[indicate\r\n> access date]*.\r\n\r\n### Credits for GridSat-GOES/CONUS\r\n\r\n**Dataset Citation:**\r\n\r\n> Knapp, K. R. (2017): Gridded Satellite GOES Coverage Data (GridSat-GOES),\r\n> *[indicate subset used]*. *NOAA National Centers for Environmental\r\n> Information* [doi:10.7289/V5HM56GM](https://doi.org/10.7289/V5HM56GM),\r\n> *[indicate access date]*.\r\n\r\nPlease cite the following article when using GridSat-GOES/CONUS data in any\r\npublication:\r\n\r\n> Knapp, K. R. and Wilkins, S. L.: Gridded Satellite (GridSat) GOES and CONUS\r\n> data, *Earth System Science Data*, **10(3)**, 1417\u20131425,\r\n> [doi:10.5194/essd-10-1417-2018](https://doi.org/10.5194/essd-10-1417-2018),\r\n> 2018.\r\n\r\n### Credits for GridSat-B1\r\n\r\n**Dataset Citation:**\r\n\r\n> Knapp, K. R.; NOAA CDR Program; (2014): NOAA Climate Data Record (CDR) of\r\n> Gridded Satellite Data from ISCCP B1 (GridSat-B1) Infrared Channel Brightness\r\n> Temperature, Version 2, *[indicate subset used]*. *NOAA National Centers for\r\n> Environmental Information*.\r\n> [doi:10.7289/V59P2ZKR](https://doi.org/10.7289/V59P2ZKR), *[indicate access\r\n> date]*.\r\n\r\nPlease cite the following article when using GridSat-B1 data in any publication:\r\n\r\n> Knapp, K. R., Ansari S.; Bain, C. L.; Bourassa, M. A.; Dickinson, M. J.; Funk,\r\n> C.; Helms, C. N.; Hennon, C. C.; Holmes, C. D.; Huffman, G. J.; Kossin, J. P.;\r\n> Lee, H.-T.; Loew, A.; and Magnusdottir, G.: Globally gridded satellite\r\n> (GridSat) observations for climate studies. *Bulletin of the American\r\n> Meteorological Society*, **92(7)**, 893-907,\r\n> [doi:10.1175/2011BAMS3039.1](https://doi.org/10.1175/2011BAMS3039.1), 2011.\r\n\r\nWhen possible, please cite the following article when using the ISCCP-B1 data or\r\nother ISCCP-B1 imagery or GIBBS imagery in a publication or website:\r\n\r\n> Knapp, K. R.: Scientific data stewardship of International Satellite Cloud\r\n> Climatology Project B1 global geostationary observations. *Journal of Applied\r\n> Remote Sensing*, **2(1)**, 023548,\r\n> [doi:10.1117/1.3043461](https://doi.org/10.1117/1.3043461), 2008.\r\n\r\n## Contact and Support\r\n\r\nFor issues, questions, or requests, feel free to open an issue on this\r\nrepository or contact the author, [wvenialbo at\r\ngmail.com](mailto:wvenialbo@gmail.com).\r\n\r\n---\r\n\r\n## Similar Projects\r\n\r\n- [Brian Blaylock's goes2go](https://github.com/blaylockbk/goes2go): Download\r\n and process GOES-16 and GOES-17 data from NOAA's archive on AWS using Python.\r\n ([readthedocs](https://goes2go.readthedocs.io/))\r\n- [Joao Henry's GOES](https://github.com/joaohenry23/GOES): Python package to\r\n download and manipulate GOES-16/17/18 data.\r\n\r\n## References\r\n\r\n1. Knapp, K. R. (2008): Scientific data stewardship of International Satellite\r\n Cloud Climatology Project B1 global geostationary observations. *Journal of\r\n Applied Remote Sensing*, **2(1)**, 023548,\r\n [doi:10.1117/1.3043461](https://doi.org/10.1117/1.3043461).\r\n2. Knapp, K. R., Ansari S.; Bain, C. L.; Bourassa, M. A.; Dickinson, M. J.;\r\n Funk, C.; Helms, C. N.; Hennon, C. C.; Holmes, C. D.; Huffman, G. J.; Kossin,\r\n J. P.; Lee, H.-T.; Loew, A.; and Magnusdottir, G.; (2011): Globally gridded\r\n satellite (GridSat) observations for climate studies. *Bulletin of the\r\n American Meteorological Society*, **92(7)**, 893-907,\r\n [doi:10.1175/2011BAMS3039.1](https://doi.org/10.1175/2011BAMS3039.1).\r\n3. Knapp, K. R; NOAA CDR Program; (2014): NOAA Climate Data Record (CDR) of\r\n Gridded Satellite Data from ISCCP B1 (GridSat-B1) Infrared Channel Brightness\r\n Temperature, Version 2. *NOAA National Centers for Environmental\r\n Information*, [doi:10.7289/V59P2ZKR](https://doi.org/10.7289/V59P2ZKR).<a\r\n name=\"gridb1\"></a>\r\n4. Knapp, K. R; (2017): Gridded Satellite GOES Coverage Data (GridSat-GOES).\r\n *NOAA National Centers for Environmental Information*.\r\n [doi:10.7289/V5HM56GM](https://doi.org/10.7289/V5HM56GM).<a name=\"goesi\"></a>\r\n5. Knapp, K. R. and Wilkins, S. L.; (2018): Gridded Satellite (GridSat) GOES and\r\n CONUS data, *Earth System Science Data*, 10(3), 1417\u20131425,\r\n [doi:10.5194/essd-10-1417-2018](https://doi.org/10.5194/essd-10-1417-2018).\r\n6. GOES History. *GOES-R Website*, https://www.goes-r.gov/mission/history.html,\r\n retrieved on 2024.<a name=\"hist\"></a>\r\n7. GOES-R Series Data Products. *GOES-R Website*,\r\n https://www.goes-r.gov/products/overview.html, retrieved on 2024.<a\r\n name=\"goesr\"></a>\r\n8. NOAA Big Data Program, *NOAA Open Data Dissemination Program*,\r\n https://github.com/NOAA-Big-Data-Program/bdp-data-docs, retrieved on 2024.\r\n9. Beginner\u2019s Guide to GOES-R Series Data: How to acquire, analyze, and\r\n visualize GOES-R Series data, *Resources compiled by GOES-R Product Readiness\r\n and Operations*, Satellite Products and Services Division, National Oceanic\r\n and Atmospheric Administration.\r\n [PDF](https://www.goes-r.gov/downloads/resources/documents/Beginners_Guide_to_GOES-R_Series_Data.pdf)\r\n Last Updated on May 23, 2024, retrieved on 2024.\r\n10. GOES-R Series Data Book, *GOES-R Series Program Office*, Goddard Space\r\n Flight Center, National Aeronautics and Space Administration.\r\n [PDF](https://www.goes-r.gov/downloads/resources/documents/GOES-RSeriesDataBook.pdf),\r\n retrieved on 2024.\r\n\r\n[0]: hidden_references:\r\n[1]: https://www.ncei.noaa.gov/access/metadata/landing-page/bin/iso?id=gov.noaa.ncdc:C00993\r\n[2]: https://registry.opendata.aws/noaa-goes/\r\n[3]: https://docs.opendata.aws/noaa-goes16/cics-readme.html\r\n[4]: https://www.goes-r.gov/\r\n[5]: https://www.ncei.noaa.gov/access/metadata/landing-page/bin/iso?id=gov.noaa.ncdc:C00829\r\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2024 Waldemar Villamayor-Venialbo Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
"summary": "Satellite imagery downloader for GOES and GridSat datasets",
"version": "0.1rc2",
"project_urls": {
"Issues": "https://github.com/wvenialbo/GOES-DL/issues",
"Repository": "https://github.com/wvenialbo/GOES-DL"
},
"split_keywords": [
"goes",
" goes-r",
" gridsat",
" gridsat-b1",
" gridsat-goes/conus",
" atmospheric science",
" dataset",
" downloader",
" environmental data",
" meteorology",
" multi-source",
" satellite imagery",
" severe weather"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8a73c9b5af686bf3a801e41f0e41d05b2049d57d9a7d03ef222566a44325df0b",
"md5": "dd49087dc61674379cd876b959ae5bdb",
"sha256": "11fadaa5418b59c9edf5e89e94743a16d26c906c5265bbb8206d5d16f93a00ba"
},
"downloads": -1,
"filename": "goes_dl-0.1rc2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "dd49087dc61674379cd876b959ae5bdb",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 57796,
"upload_time": "2024-10-29T15:10:49",
"upload_time_iso_8601": "2024-10-29T15:10:49.748709Z",
"url": "https://files.pythonhosted.org/packages/8a/73/c9b5af686bf3a801e41f0e41d05b2049d57d9a7d03ef222566a44325df0b/goes_dl-0.1rc2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "2dbbd78e887d14fad56ee563dc9ec64d17cfb19027f3ed1d6de25833fffc95da",
"md5": "830c851b56214c3613660d3d2f4e0d92",
"sha256": "86ca17a3a16b7a11ed5039a136cc7134de11c0c2c1b2c32d4a0ab0acf9bce901"
},
"downloads": -1,
"filename": "goes_dl-0.1rc2.tar.gz",
"has_sig": false,
"md5_digest": "830c851b56214c3613660d3d2f4e0d92",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 78972,
"upload_time": "2024-10-29T15:10:51",
"upload_time_iso_8601": "2024-10-29T15:10:51.595306Z",
"url": "https://files.pythonhosted.org/packages/2d/bb/d78e887d14fad56ee563dc9ec64d17cfb19027f3ed1d6de25833fffc95da/goes_dl-0.1rc2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-29 15:10:51",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "wvenialbo",
"github_project": "GOES-DL",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "boto3",
"specs": [
[
"~=",
"1.35.47"
]
]
},
{
"name": "mypy-boto3-s3",
"specs": [
[
"~=",
"1.35.46"
]
]
},
{
"name": "requests",
"specs": [
[
"~=",
"2.32.3"
]
]
}
],
"lcname": "goes-dl"
}
