| Name | rio-tiler-pds JSON |
| Version |
0.10.1
JSON |
| download |
| home_page | None |
| Summary | Get mercator tile from cloud hosted dataset such as CBERS-4, Sentinel-2, Sentinel-1 and Landsat-8 AWS PDS. |
| upload_time | 2023-08-21 10:04:15 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.8 |
| license | None |
| keywords |
cogeo
cloud optimized geotiff
aws pds
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Rio-Tiler-PDS: A rio-tiler plugin for Public Datasets
<p align="center">
<img src="https://user-images.githubusercontent.com/10407788/91102350-ffa75400-e636-11ea-8374-3450a72745c9.png" style="max-width: 800px;" alt="rio-tiler-pds"></a>
</p>
<p align="center">
<em>A rio-tiler plugin to read from publicly-available datasets.</em>
</p>
<p align="center">
<a href="https://github.com/cogeotiff/rio-tiler-pds/actions?query=workflow%3ACI" target="_blank">
<img src="https://github.com/cogeotiff/rio-tiler-pds/workflows/CI/badge.svg" alt="Test">
</a>
<a href="https://codecov.io/gh/cogeotiff/rio-tiler-pds" target="_blank">
<img src="https://codecov.io/gh/cogeotiff/rio-tiler-pds/branch/main/graph/badge.svg" alt="Coverage">
</a>
<a href="https://pypi.org/project/rio-tiler-pds" target="_blank">
<img src="https://img.shields.io/pypi/v/rio-tiler-pds?color=%2334D058&label=pypi%20package" alt="Package version">
</a>
<a href="https://pypistats.org/packages/rio-tiler-pds" target="_blank">
<img src="https://img.shields.io/pypi/dm/rio-tiler-pds.svg" alt="Downloads">
</a>
<a href="https://github.com/cogeotiff/rio-tiler-pds/blob/main/LICENSE.txt" target="_blank">
<img src="https://img.shields.io/github/license/cogeotiff/rio-tiler-pds.svg" alt="Lincense">
</a>
</p>
**Important** This is the new module for rio-tiler missions specific (ref: https://github.com/cogeotiff/rio-tiler/issues/195)
---
**Documentation**: <a href="https://cogeotiff.github.io/rio-tiler-pds/" target="_blank">https://cogeotiff.github.io/rio-tiler-pds/</a>
**Source Code**: <a href="https://github.com/cogeotiff/rio-tiler-pds" target="_blank">https://github.com/cogeotiff/rio-tiler-pds</a>
---
## Installation
You can install rio-tiler-pds using pip
```bash
$ pip install -U pip
$ pip install rio-tiler-pds
```
or install from source:
```bash
$ pip install -U pip
$ pip install git+https://github.com/cogeotiff/rio-tiler-pds.git
```
## Datasets
| Data | Level/Product | Format | Owner | Region | Bucket Type |
| ----------------------------------------- | ------------------------------------------- | -------------------------- | -------------------------- | ------------ | ------------------ |
| [Sentinel 2][s2_l1c_jp2] | L1C | JPEG2000 | Sinergise / AWS | eu-central-1 | **Requester-pays** |
| [Sentinel 2][s2_l2a_jp2] | L2A | JPEG2000 | Sinergise / AWS | eu-central-1 | **Requester-pays** |
| [Sentinel 2][s2_l2a_cog] | L2A | COG | Digital Earth Africa / AWS | us-west-2 | Public |
| [Sentinel 1][s1_l1c_cog] | L1C GRD (IW, EW, S1-6) | COG (Internal GCPS) | Sinergise / AWS | eu-central-1 | **Requester-pays** |
| [Landsat Collection 2][landsat_c2_cog] | L1,L2 | COG | USGS / AWS | us-west-2 | **Requester-pays** |
| [CBERS 4/4A][cbers_cog] | L2/L4 | COG | AMS Kepler / AWS | us-east-1 | **Requester-pays** |
| [MODIS (modis-pds)][modis_pds] | MCD43A4, MOD09GQ, MYD09GQ, MOD09GA, MYD09GA | GTiff (External Overviews) | - | us-west-2 | Public |
| [MODIS (astraea-opendata)][modis_astraea] | MCD43A4, MOD11A1, MOD13A1, MYD11A1 MYD13A1 | COG | Astraea / AWS | us-west-2 | **Requester-pays** |
| [Copernicus Digital Elevation Model][copernicus_dem] | GLO-30, GLO-90 | COG | Sinergise / AWS | eu-central-1 | Public |
[s2_l1c_jp2]: https://registry.opendata.aws/sentinel-2/
[s2_l2a_jp2]: https://registry.opendata.aws/sentinel-2/
[s2_l2a_cog]: https://registry.opendata.aws/sentinel-2-l2a-cogs/
[s1_l1c_cog]: https://registry.opendata.aws/sentinel-1/
[landsat_c2_cog]: https://www.usgs.gov/core-science-systems/nli/landsat/landsat-commercial-cloud-data-access
[cbers_cog]: https://registry.opendata.aws/cbers/
[modis_pds]: https://docs.opendata.aws/modis-pds/readme.html
[modis_astraea]: https://registry.opendata.aws/modis-astraea/
[copernicus_dem]: https://registry.opendata.aws/copernicus-dem/
**Adding more dataset**:
If you know of another publicly-available dataset that can easily be described
with a "scene id", please feel free to [open an
issue](https://github.com/cogeotiff/rio-tiler-pds/issues/new).
## Warnings
#### Requester-pays Buckets
On AWS, `sentinel2`, `sentinel1`, `cbers` and `modis` (in astraea-opendata) datasets are stored in [_requester
pays_](https://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html)
buckets. This means that the cost of GET and LIST requests and egress fees for
downloading files outside the AWS region will be charged to the _accessing
users_, not the organization hosting the bucket. For `rio-tiler` and
`rio-tiler-pds` to work with such buckets, you'll need to set
`AWS_REQUEST_PAYER="requester"` in your shell environment.
#### Partial reading on Cloud hosted dataset
When reading data, `rio-tiler-pds` performs _partial_ reads when possible. Hence
performance will be best on data stored as [Cloud Optimized GeoTIFF
(COG)](http://cogeo.org). It's important to note that **Sentinel-2 scenes hosted
on AWS are not in Cloud Optimized format but in JPEG2000**. Partial reads from
JPEG2000 files are inefficient, and GDAL (the library underlying `rio-tiler-pds`
and `rasterio`) will need to make **many GET requests** and transfer a lot of
data. This will be both slow and expensive, since AWS's JPEG2000 collection of
Sentinel 2 data is stored in a requester pays bucket.
Ref: [Do you really want people using your data](https://medium.com/@_VincentS_/do-you-really-want-people-using-your-data-ec94cd94dc3f) blog post.
## Overview
### Readers
Each dataset has its own submodule (e.g sentinel2: `rio_tiler_pds.sentinel.aws`)
```python
from rio_tiler_pds.landsat.aws import LandsatC2Reader
from rio_tiler_pds.sentinel.aws import S1L1CReader
from rio_tiler_pds.sentinel.aws import (
S2JP2Reader, # JPEG2000
S2COGReader, # COG
)
from rio_tiler_pds.cbers.aws import CBERSReader
from rio_tiler_pds.modis.aws import MODISPDSReader, MODISASTRAEAReader
from rio_tiler_pds.copernicus.aws import Dem30Reader, Dem90Reader
```
All Readers are subclass of [`rio_tiler.io.BaseReader`](https://github.com/cogeotiff/rio-tiler/blob/f917d0eaf27f8644f3bb18856a63fe45eeb4a2ef/rio_tiler/io/base.py#L17) and inherit its properties/methods.
#### Properties
- **bounds**: Scene bounding box
- **crs**: CRS of the bounding box
- **geographic_bounds**: bounding box in geographic projection (e.g WGS84)
- **minzoom**: WebMercator MinZoom (e.g 7 for Landsat 8)
- **maxzoom**: WebMercator MaxZoom (e.g 12 for Landsat 8)
#### Methods
- **info**: Returns band's simple info (e.g nodata, band_descriptions, ....)
- **statistics**: Returns band's statistics (percentile, histogram, ...)
- **tile**: Read web mercator map tile from bands
- **part**: Extract part of bands
- **preview**: Returns a low resolution preview from bands
- **point**: Returns band's pixel value for a given lon,lat
- **feature**: Extract part of bands
#### Other
- **bands** (property): List of available bands for each dataset
### Scene ID
All readers take scene id as main input. The **scene id** is used internaly by the reader to derive the full path of the data.
e.g: Landsat on AWS
Because the Landsat AWS PDS follows a regular schema to store the data (`s3://{bucket}/c1/L8/{path}/{row}/{scene}/{scene}_{band}.TIF"`), we can easily reconstruct the full band's path by parsing the scene id.
```python
from rio_tiler_pds.landsat.aws import LandsatC2Reader
from rio_tiler_pds.landsat.utils import sceneid_parser
sceneid_parser("LC08_L2SP_001062_20201031_20201106_02_T2")
> {'sensor': 'C',
'satellite': '08',
'processingCorrectionLevel': 'L2SP',
'path': '001',
'row': '062',
'acquisitionYear': '2020',
'acquisitionMonth': '10',
'acquisitionDay': '31',
'processingYear': '2020',
'processingMonth': '11',
'processingDay': '06',
'collectionNumber': '02',
'collectionCategory': 'T2',
'scene': 'LC08_L2SP_001062_20201031_20201106_02_T2',
'date': '2020-10-31',
'_processingLevelNum': '2',
'category': 'standard',
'sensor_name': 'oli-tirs',
'_sensor_s3_prefix': 'oli-tirs',
'bands': ('QA_PIXEL',
'QA_RADSAT',
'SR_B1',
'SR_B2',
'SR_B3',
'SR_B4',
'SR_B5',
'SR_B6',
'SR_B7',
'SR_QA_AEROSOL',
'ST_ATRAN',
'ST_B10',
'ST_CDIST',
'ST_DRAD',
'ST_EMIS',
'ST_EMSD',
'ST_QA',
'ST_TRAD',
'ST_URAD')}
with LandsatC2Reader("LC08_L2SP_001062_20201031_20201106_02_T2") as landsat:
print(landsat._get_band_url("SR_B2"))
> s3://usgs-landsat/collection02/level-2/standard/oli-tirs/2020/001/062/LC08_L2SP_001062_20201031_20201106_02_T2/LC08_L2SP_001062_20201031_20201106_02_T2_SR_B2.TIF
```
Each dataset has a specific scene id format:
!!! note "Scene ID formats"
- Landsat
- link: [rio_tiler_pds.landsat.utils.sceneid_parser](https://github.com/cogeotiff/rio-tiler-pds/blob/e4421d3cf7c23b7b3552b8bb16ee5913a5483caf/rio_tiler_pds/landsat/utils.py#L35-L56)
- regex: `^L[COTEM]0[0-9]_L\d{1}[A-Z]{2}_\d{6}_\d{8}_\d{8}_\d{2}_(T1|T2|RT)$`
- example: `LC08_L1TP_016037_20170813_20170814_01_RT`
- Sentinel 1 L1C
- link: [rio_tiler_pds.sentinel.utils.s1_sceneid_parser](https://github.com/cogeotiff/rio-tiler-pds/blob/e4421d3cf7c23b7b3552b8bb16ee5913a5483caf/rio_tiler_pds/sentinel/utils.py#L98-L121)
- regex: `^S1[AB]_(IW|EW)_[A-Z]{3}[FHM]_[0-9][SA][A-Z]{2}_[0-9]{8}T[0-9]{6}_[0-9]{8}T[0-9]{6}_[0-9A-Z]{6}_[0-9A-Z]{6}_[0-9A-Z]{4}$`
- example: `S1A_IW_GRDH_1SDV_20180716T004042_20180716T004107_022812_02792A_FD5B`
- Sentinel 2 JPEG2000 and Sentinel 2 COG
- link: [rio_tiler_pds.sentinel.utils.s2_sceneid_parser](https://github.com/cogeotiff/rio-tiler-pds/blob/e4421d3cf7c23b7b3552b8bb16ee5913a5483caf/rio_tiler_pds/sentinel/utils.py#L25-L60)
- regex: `^S2[AB]_[0-9]{2}[A-Z]{3}_[0-9]{8}_[0-9]_L[0-2][A-C]$` or `^S2[AB]_L[0-2][A-C]_[0-9]{8}_[0-9]{2}[A-Z]{3}_[0-9]$`
- example: `S2A_29RKH_20200219_0_L2A`, `S2A_L1C_20170729_19UDP_0`, `S2A_L2A_20170729_19UDP_0`
- CBERS
- link: [rio_tiler_pds.cbers.utils.sceneid_parser](https://github.com/cogeotiff/rio-tiler-pds/blob/e4421d3cf7c23b7b3552b8bb16ee5913a5483caf/rio_tiler_pds/cbers/utils.py#L28-L43)
- regex: `^CBERS_(4|4A)_\w+_[0-9]{8}_[0-9]{3}_[0-9]{3}_L\w+$`
- example: `CBERS_4_MUX_20171121_057_094_L2`, `CBERS_4_AWFI_20170420_146_129_L2`, `CBERS_4_PAN10M_20170427_161_109_L4`, `CBERS_4_PAN5M_20170425_153_114_L4`, `CBERS_4A_WPM_20200730_209_139_L4`
- MODIS (PDS and Astraea)
- link: [rio_tiler_pds.modis.utils.sceneid_parser](https://github.com/cogeotiff/rio-tiler-pds/blob/c533d38330f46738c46cb9927dbe91b299dc643d/rio_tiler_pds/modis/utils.py#L29-L42)
- regex: `^M[COY]D[0-9]{2}[A-Z0-9]{2}\.A[0-9]{4}[0-9]{3}\.h[0-9]{2}v[0-9]{2}\.[0-9]{3}\.[0-9]{13}$`
- example: `MCD43A4.A2017006.h21v11.006.2017018074804`
### Band Per Asset/File
`rio-tiler-pds` Readers assume that bands (e.g eo:band in STAC) are stored in separate files.
```bash
$ aws s3 ls s3://usgs-landsat/collection02/level-2/standard/oli-tirs/2020/001/062/LC08_L2SP_001062_20201031_20201106_02_T2/ --request-payer
LC08_L2SP_001062_20201031_20201106_02_T2_ANG.txt
LC08_L2SP_001062_20201031_20201106_02_T2_MTL.json
LC08_L2SP_001062_20201031_20201106_02_T2_MTL.txt
LC08_L2SP_001062_20201031_20201106_02_T2_MTL.xml
LC08_L2SP_001062_20201031_20201106_02_T2_QA_PIXEL.TIF
LC08_L2SP_001062_20201031_20201106_02_T2_QA_RADSAT.TIF
LC08_L2SP_001062_20201031_20201106_02_T2_SR_B1.TIF
LC08_L2SP_001062_20201031_20201106_02_T2_SR_B2.TIF
LC08_L2SP_001062_20201031_20201106_02_T2_SR_B3.TIF
LC08_L2SP_001062_20201031_20201106_02_T2_SR_B4.TIF
LC08_L2SP_001062_20201031_20201106_02_T2_SR_B5.TIF
LC08_L2SP_001062_20201031_20201106_02_T2_SR_B6.TIF
LC08_L2SP_001062_20201031_20201106_02_T2_SR_B7.TIF
LC08_L2SP_001062_20201031_20201106_02_T2_SR_QA_AEROSOL.TIF
LC08_L2SP_001062_20201031_20201106_02_T2_SR_stac.json
LC08_L2SP_001062_20201031_20201106_02_T2_ST_ATRAN.TIF
LC08_L2SP_001062_20201031_20201106_02_T2_ST_B10.TIF
LC08_L2SP_001062_20201031_20201106_02_T2_ST_CDIST.TIF
LC08_L2SP_001062_20201031_20201106_02_T2_ST_DRAD.TIF
LC08_L2SP_001062_20201031_20201106_02_T2_ST_EMIS.TIF
LC08_L2SP_001062_20201031_20201106_02_T2_ST_EMSD.TIF
LC08_L2SP_001062_20201031_20201106_02_T2_ST_QA.TIF
LC08_L2SP_001062_20201031_20201106_02_T2_ST_TRAD.TIF
LC08_L2SP_001062_20201031_20201106_02_T2_ST_URAD.TIF
LC08_L2SP_001062_20201031_20201106_02_T2_ST_stac.json
LC08_L2SP_001062_20201031_20201106_02_T2_thumb_large.jpeg
LC08_L2SP_001062_20201031_20201106_02_T2_thumb_small.jpeg
```
When reading data or metadata, readers will merge them.
e.g
```python
with S2COGReader("S2A_L2A_20170729_19UDP_0") as sentinel:
img = sentinel.tile(78, 89, 8, bands=("B01", "B02"))
assert img.data.shape == (2, 256, 256)
stats = sentinel.statistics(bands=("B01", "B02"))
print(stats)
>> {
'B01': BandStatistics(
min=2.0,
max=17132.0,
mean=2183.7570706659685,
count=651247.0,
sum=1422165241.0,
std=3474.123975478363,
median=370.0,
majority=238.0,
minority=2.0,
unique=15112.0,
histogram=[
[476342.0, 35760.0, 27525.0, 24852.0, 24379.0, 23792.0, 20891.0, 13602.0, 3891.0, 213.0],
[2.0, 1715.0, 3428.0, 5141.0, 6854.0, 8567.0, 10280.0, 11993.0, 13706.0, 15419.0, 17132.0]
],
valid_percent=62.11,
masked_pixels=397329.0,
valid_pixels=651247.0,
percentile_2=179.0,
percentile_98=12465.0
),
'B02': BandStatistics(
min=1.0,
max=15749.0,
mean=1941.2052554560712,
count=651247.0,
sum=1264204099.0,
std=3130.545395156859,
median=329.0,
majority=206.0,
minority=11946.0,
unique=13904.0,
histogram=[
[479174.0, 34919.0, 27649.0, 25126.0, 24913.0, 24119.0, 20223.0, 12097.0, 2872.0, 155.0],
[1.0, 1575.8, 3150.6, 4725.4, 6300.2, 7875.0, 9449.8, 11024.6, 12599.4, 14174.199999999999, 15749.0]
],
valid_percent=62.11,
masked_pixels=397329.0,
valid_pixels=651247.0,
percentile_2=134.0,
percentile_98=11227.079999999958
)}
print(stats["B01"].min)
>> 2.0
```
### Mosaic Reader: Copernicus DEM
The Copernicus DEM GLO-30 and GLO-90 readers are not **per scene** but **mosaic** readers. This is possible because the dataset is a global dataset with file names having the `geo-location` of the COG, meaning we can easily contruct a filepath from a coordinate.
```python
from rio_tiler_pds.copernicus.aws import Dem30Reader
with Dem30Reader() as dem:
print(dem.assets_for_point(-57.2, -11.2))
>> ['s3://copernicus-dem-30m/Copernicus_DSM_COG_10_S12_00_W058_00_DEM/Copernicus_DSM_COG_10_S12_00_W058_00_DEM.tif']
```
## Changes
See [CHANGES.md](https://github.com/cogeotiff/rio-tiler-pds/blob/main/CHANGES.md).
## Contribution & Development
See [CONTRIBUTING.md](https://github.com/cogeotiff/rio-tiler/blob/main/CONTRIBUTING.md)
## License
See [LICENSE.txt](https://github.com/cogeotiff/rio-tiler-pds/blob/main/LICENSE.txt)
## Authors
The rio-tiler project was begun at Mapbox and has been transferred in January 2019.
See [AUTHORS.txt](https://github.com/cogeotiff/rio-tiler-pds/blob/main/AUTHORS.txt) for a listing of individual contributors.
Raw data
{
"_id": null,
"home_page": null,
"name": "rio-tiler-pds",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "COGEO,Cloud Optimized Geotiff,AWS PDS",
"author": null,
"author_email": "Vincent Sarago <vincent@developmentseed.com>",
"download_url": "https://files.pythonhosted.org/packages/c2/79/f266c9405dffd191bc7f2edf2bc0cf2121280c6ab1a4e77a59a22820390d/rio_tiler_pds-0.10.1.tar.gz",
"platform": null,
"description": "# Rio-Tiler-PDS: A rio-tiler plugin for Public Datasets\n\n<p align=\"center\">\n <img src=\"https://user-images.githubusercontent.com/10407788/91102350-ffa75400-e636-11ea-8374-3450a72745c9.png\" style=\"max-width: 800px;\" alt=\"rio-tiler-pds\"></a>\n</p>\n<p align=\"center\">\n <em>A rio-tiler plugin to read from publicly-available datasets.</em>\n</p>\n<p align=\"center\">\n <a href=\"https://github.com/cogeotiff/rio-tiler-pds/actions?query=workflow%3ACI\" target=\"_blank\">\n <img src=\"https://github.com/cogeotiff/rio-tiler-pds/workflows/CI/badge.svg\" alt=\"Test\">\n </a>\n <a href=\"https://codecov.io/gh/cogeotiff/rio-tiler-pds\" target=\"_blank\">\n <img src=\"https://codecov.io/gh/cogeotiff/rio-tiler-pds/branch/main/graph/badge.svg\" alt=\"Coverage\">\n </a>\n <a href=\"https://pypi.org/project/rio-tiler-pds\" target=\"_blank\">\n <img src=\"https://img.shields.io/pypi/v/rio-tiler-pds?color=%2334D058&label=pypi%20package\" alt=\"Package version\">\n </a>\n <a href=\"https://pypistats.org/packages/rio-tiler-pds\" target=\"_blank\">\n <img src=\"https://img.shields.io/pypi/dm/rio-tiler-pds.svg\" alt=\"Downloads\">\n </a>\n <a href=\"https://github.com/cogeotiff/rio-tiler-pds/blob/main/LICENSE.txt\" target=\"_blank\">\n <img src=\"https://img.shields.io/github/license/cogeotiff/rio-tiler-pds.svg\" alt=\"Lincense\">\n </a>\n</p>\n\n**Important** This is the new module for rio-tiler missions specific (ref: https://github.com/cogeotiff/rio-tiler/issues/195)\n\n---\n\n**Documentation**: <a href=\"https://cogeotiff.github.io/rio-tiler-pds/\" target=\"_blank\">https://cogeotiff.github.io/rio-tiler-pds/</a>\n\n**Source Code**: <a href=\"https://github.com/cogeotiff/rio-tiler-pds\" target=\"_blank\">https://github.com/cogeotiff/rio-tiler-pds</a>\n\n---\n\n## Installation\n\nYou can install rio-tiler-pds using pip\n\n```bash\n$ pip install -U pip\n$ pip install rio-tiler-pds\n```\n\nor install from source:\n\n```bash\n$ pip install -U pip\n$ pip install git+https://github.com/cogeotiff/rio-tiler-pds.git\n```\n\n## Datasets\n\n| Data | Level/Product | Format | Owner | Region | Bucket Type |\n| ----------------------------------------- | ------------------------------------------- | -------------------------- | -------------------------- | ------------ | ------------------ |\n| [Sentinel 2][s2_l1c_jp2] | L1C | JPEG2000 | Sinergise / AWS | eu-central-1 | **Requester-pays** |\n| [Sentinel 2][s2_l2a_jp2] | L2A | JPEG2000 | Sinergise / AWS | eu-central-1 | **Requester-pays** |\n| [Sentinel 2][s2_l2a_cog] | L2A | COG | Digital Earth Africa / AWS | us-west-2 | Public |\n| [Sentinel 1][s1_l1c_cog] | L1C GRD (IW, EW, S1-6) | COG (Internal GCPS) | Sinergise / AWS | eu-central-1 | **Requester-pays** |\n| [Landsat Collection 2][landsat_c2_cog] | L1,L2 | COG | USGS / AWS | us-west-2 | **Requester-pays** |\n| [CBERS 4/4A][cbers_cog] | L2/L4 | COG | AMS Kepler / AWS | us-east-1 | **Requester-pays** |\n| [MODIS (modis-pds)][modis_pds] | MCD43A4, MOD09GQ, MYD09GQ, MOD09GA, MYD09GA | GTiff (External Overviews) | - | us-west-2 | Public |\n| [MODIS (astraea-opendata)][modis_astraea] | MCD43A4, MOD11A1, MOD13A1, MYD11A1 MYD13A1 | COG | Astraea / AWS | us-west-2 | **Requester-pays** |\n| [Copernicus Digital Elevation Model][copernicus_dem] | GLO-30, GLO-90 | COG | Sinergise / AWS | eu-central-1 | Public |\n\n[s2_l1c_jp2]: https://registry.opendata.aws/sentinel-2/\n[s2_l2a_jp2]: https://registry.opendata.aws/sentinel-2/\n[s2_l2a_cog]: https://registry.opendata.aws/sentinel-2-l2a-cogs/\n[s1_l1c_cog]: https://registry.opendata.aws/sentinel-1/\n[landsat_c2_cog]: https://www.usgs.gov/core-science-systems/nli/landsat/landsat-commercial-cloud-data-access\n[cbers_cog]: https://registry.opendata.aws/cbers/\n[modis_pds]: https://docs.opendata.aws/modis-pds/readme.html\n[modis_astraea]: https://registry.opendata.aws/modis-astraea/\n[copernicus_dem]: https://registry.opendata.aws/copernicus-dem/\n\n**Adding more dataset**:\n\nIf you know of another publicly-available dataset that can easily be described\nwith a \"scene id\", please feel free to [open an\nissue](https://github.com/cogeotiff/rio-tiler-pds/issues/new).\n\n## Warnings\n\n#### Requester-pays Buckets\n\nOn AWS, `sentinel2`, `sentinel1`, `cbers` and `modis` (in astraea-opendata) datasets are stored in [_requester\npays_](https://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html)\nbuckets. This means that the cost of GET and LIST requests and egress fees for\ndownloading files outside the AWS region will be charged to the _accessing\nusers_, not the organization hosting the bucket. For `rio-tiler` and\n`rio-tiler-pds` to work with such buckets, you'll need to set\n`AWS_REQUEST_PAYER=\"requester\"` in your shell environment.\n\n#### Partial reading on Cloud hosted dataset\n\nWhen reading data, `rio-tiler-pds` performs _partial_ reads when possible. Hence\nperformance will be best on data stored as [Cloud Optimized GeoTIFF\n(COG)](http://cogeo.org). It's important to note that **Sentinel-2 scenes hosted\non AWS are not in Cloud Optimized format but in JPEG2000**. Partial reads from\nJPEG2000 files are inefficient, and GDAL (the library underlying `rio-tiler-pds`\nand `rasterio`) will need to make **many GET requests** and transfer a lot of\ndata. This will be both slow and expensive, since AWS's JPEG2000 collection of\nSentinel 2 data is stored in a requester pays bucket.\n\nRef: [Do you really want people using your data](https://medium.com/@_VincentS_/do-you-really-want-people-using-your-data-ec94cd94dc3f) blog post.\n\n## Overview\n\n### Readers\n\nEach dataset has its own submodule (e.g sentinel2: `rio_tiler_pds.sentinel.aws`)\n\n```python\nfrom rio_tiler_pds.landsat.aws import LandsatC2Reader\nfrom rio_tiler_pds.sentinel.aws import S1L1CReader\nfrom rio_tiler_pds.sentinel.aws import (\n S2JP2Reader, # JPEG2000\n S2COGReader, # COG\n)\n\nfrom rio_tiler_pds.cbers.aws import CBERSReader\nfrom rio_tiler_pds.modis.aws import MODISPDSReader, MODISASTRAEAReader\nfrom rio_tiler_pds.copernicus.aws import Dem30Reader, Dem90Reader\n```\n\nAll Readers are subclass of [`rio_tiler.io.BaseReader`](https://github.com/cogeotiff/rio-tiler/blob/f917d0eaf27f8644f3bb18856a63fe45eeb4a2ef/rio_tiler/io/base.py#L17) and inherit its properties/methods.\n\n#### Properties\n- **bounds**: Scene bounding box\n- **crs**: CRS of the bounding box\n- **geographic_bounds**: bounding box in geographic projection (e.g WGS84)\n- **minzoom**: WebMercator MinZoom (e.g 7 for Landsat 8)\n- **maxzoom**: WebMercator MaxZoom (e.g 12 for Landsat 8)\n\n#### Methods\n\n- **info**: Returns band's simple info (e.g nodata, band_descriptions, ....)\n- **statistics**: Returns band's statistics (percentile, histogram, ...)\n- **tile**: Read web mercator map tile from bands\n- **part**: Extract part of bands\n- **preview**: Returns a low resolution preview from bands\n- **point**: Returns band's pixel value for a given lon,lat\n- **feature**: Extract part of bands\n\n#### Other\n- **bands** (property): List of available bands for each dataset\n\n### Scene ID\n\nAll readers take scene id as main input. The **scene id** is used internaly by the reader to derive the full path of the data.\n\ne.g: Landsat on AWS\n\nBecause the Landsat AWS PDS follows a regular schema to store the data (`s3://{bucket}/c1/L8/{path}/{row}/{scene}/{scene}_{band}.TIF\"`), we can easily reconstruct the full band's path by parsing the scene id.\n\n```python\nfrom rio_tiler_pds.landsat.aws import LandsatC2Reader\nfrom rio_tiler_pds.landsat.utils import sceneid_parser\n\nsceneid_parser(\"LC08_L2SP_001062_20201031_20201106_02_T2\")\n\n> {'sensor': 'C',\n 'satellite': '08',\n 'processingCorrectionLevel': 'L2SP',\n 'path': '001',\n 'row': '062',\n 'acquisitionYear': '2020',\n 'acquisitionMonth': '10',\n 'acquisitionDay': '31',\n 'processingYear': '2020',\n 'processingMonth': '11',\n 'processingDay': '06',\n 'collectionNumber': '02',\n 'collectionCategory': 'T2',\n 'scene': 'LC08_L2SP_001062_20201031_20201106_02_T2',\n 'date': '2020-10-31',\n '_processingLevelNum': '2',\n 'category': 'standard',\n 'sensor_name': 'oli-tirs',\n '_sensor_s3_prefix': 'oli-tirs',\n 'bands': ('QA_PIXEL',\n 'QA_RADSAT',\n 'SR_B1',\n 'SR_B2',\n 'SR_B3',\n 'SR_B4',\n 'SR_B5',\n 'SR_B6',\n 'SR_B7',\n 'SR_QA_AEROSOL',\n 'ST_ATRAN',\n 'ST_B10',\n 'ST_CDIST',\n 'ST_DRAD',\n 'ST_EMIS',\n 'ST_EMSD',\n 'ST_QA',\n 'ST_TRAD',\n 'ST_URAD')}\n\nwith LandsatC2Reader(\"LC08_L2SP_001062_20201031_20201106_02_T2\") as landsat:\n print(landsat._get_band_url(\"SR_B2\"))\n\n> s3://usgs-landsat/collection02/level-2/standard/oli-tirs/2020/001/062/LC08_L2SP_001062_20201031_20201106_02_T2/LC08_L2SP_001062_20201031_20201106_02_T2_SR_B2.TIF\n```\n\nEach dataset has a specific scene id format:\n\n!!! note \"Scene ID formats\"\n\n - Landsat\n - link: [rio_tiler_pds.landsat.utils.sceneid_parser](https://github.com/cogeotiff/rio-tiler-pds/blob/e4421d3cf7c23b7b3552b8bb16ee5913a5483caf/rio_tiler_pds/landsat/utils.py#L35-L56)\n - regex: `^L[COTEM]0[0-9]_L\\d{1}[A-Z]{2}_\\d{6}_\\d{8}_\\d{8}_\\d{2}_(T1|T2|RT)$`\n - example: `LC08_L1TP_016037_20170813_20170814_01_RT`\n\n - Sentinel 1 L1C\n - link: [rio_tiler_pds.sentinel.utils.s1_sceneid_parser](https://github.com/cogeotiff/rio-tiler-pds/blob/e4421d3cf7c23b7b3552b8bb16ee5913a5483caf/rio_tiler_pds/sentinel/utils.py#L98-L121)\n - regex: `^S1[AB]_(IW|EW)_[A-Z]{3}[FHM]_[0-9][SA][A-Z]{2}_[0-9]{8}T[0-9]{6}_[0-9]{8}T[0-9]{6}_[0-9A-Z]{6}_[0-9A-Z]{6}_[0-9A-Z]{4}$`\n - example: `S1A_IW_GRDH_1SDV_20180716T004042_20180716T004107_022812_02792A_FD5B`\n\n - Sentinel 2 JPEG2000 and Sentinel 2 COG\n - link: [rio_tiler_pds.sentinel.utils.s2_sceneid_parser](https://github.com/cogeotiff/rio-tiler-pds/blob/e4421d3cf7c23b7b3552b8bb16ee5913a5483caf/rio_tiler_pds/sentinel/utils.py#L25-L60)\n - regex: `^S2[AB]_[0-9]{2}[A-Z]{3}_[0-9]{8}_[0-9]_L[0-2][A-C]$` or `^S2[AB]_L[0-2][A-C]_[0-9]{8}_[0-9]{2}[A-Z]{3}_[0-9]$`\n - example: `S2A_29RKH_20200219_0_L2A`, `S2A_L1C_20170729_19UDP_0`, `S2A_L2A_20170729_19UDP_0`\n\n - CBERS\n - link: [rio_tiler_pds.cbers.utils.sceneid_parser](https://github.com/cogeotiff/rio-tiler-pds/blob/e4421d3cf7c23b7b3552b8bb16ee5913a5483caf/rio_tiler_pds/cbers/utils.py#L28-L43)\n - regex: `^CBERS_(4|4A)_\\w+_[0-9]{8}_[0-9]{3}_[0-9]{3}_L\\w+$`\n - example: `CBERS_4_MUX_20171121_057_094_L2`, `CBERS_4_AWFI_20170420_146_129_L2`, `CBERS_4_PAN10M_20170427_161_109_L4`, `CBERS_4_PAN5M_20170425_153_114_L4`, `CBERS_4A_WPM_20200730_209_139_L4`\n\n - MODIS (PDS and Astraea)\n - link: [rio_tiler_pds.modis.utils.sceneid_parser](https://github.com/cogeotiff/rio-tiler-pds/blob/c533d38330f46738c46cb9927dbe91b299dc643d/rio_tiler_pds/modis/utils.py#L29-L42)\n - regex: `^M[COY]D[0-9]{2}[A-Z0-9]{2}\\.A[0-9]{4}[0-9]{3}\\.h[0-9]{2}v[0-9]{2}\\.[0-9]{3}\\.[0-9]{13}$`\n - example: `MCD43A4.A2017006.h21v11.006.2017018074804`\n\n### Band Per Asset/File\n\n`rio-tiler-pds` Readers assume that bands (e.g eo:band in STAC) are stored in separate files.\n\n```bash\n$ aws s3 ls s3://usgs-landsat/collection02/level-2/standard/oli-tirs/2020/001/062/LC08_L2SP_001062_20201031_20201106_02_T2/ --request-payer\nLC08_L2SP_001062_20201031_20201106_02_T2_ANG.txt\nLC08_L2SP_001062_20201031_20201106_02_T2_MTL.json\nLC08_L2SP_001062_20201031_20201106_02_T2_MTL.txt\nLC08_L2SP_001062_20201031_20201106_02_T2_MTL.xml\nLC08_L2SP_001062_20201031_20201106_02_T2_QA_PIXEL.TIF\nLC08_L2SP_001062_20201031_20201106_02_T2_QA_RADSAT.TIF\nLC08_L2SP_001062_20201031_20201106_02_T2_SR_B1.TIF\nLC08_L2SP_001062_20201031_20201106_02_T2_SR_B2.TIF\nLC08_L2SP_001062_20201031_20201106_02_T2_SR_B3.TIF\nLC08_L2SP_001062_20201031_20201106_02_T2_SR_B4.TIF\nLC08_L2SP_001062_20201031_20201106_02_T2_SR_B5.TIF\nLC08_L2SP_001062_20201031_20201106_02_T2_SR_B6.TIF\nLC08_L2SP_001062_20201031_20201106_02_T2_SR_B7.TIF\nLC08_L2SP_001062_20201031_20201106_02_T2_SR_QA_AEROSOL.TIF\nLC08_L2SP_001062_20201031_20201106_02_T2_SR_stac.json\nLC08_L2SP_001062_20201031_20201106_02_T2_ST_ATRAN.TIF\nLC08_L2SP_001062_20201031_20201106_02_T2_ST_B10.TIF\nLC08_L2SP_001062_20201031_20201106_02_T2_ST_CDIST.TIF\nLC08_L2SP_001062_20201031_20201106_02_T2_ST_DRAD.TIF\nLC08_L2SP_001062_20201031_20201106_02_T2_ST_EMIS.TIF\nLC08_L2SP_001062_20201031_20201106_02_T2_ST_EMSD.TIF\nLC08_L2SP_001062_20201031_20201106_02_T2_ST_QA.TIF\nLC08_L2SP_001062_20201031_20201106_02_T2_ST_TRAD.TIF\nLC08_L2SP_001062_20201031_20201106_02_T2_ST_URAD.TIF\nLC08_L2SP_001062_20201031_20201106_02_T2_ST_stac.json\nLC08_L2SP_001062_20201031_20201106_02_T2_thumb_large.jpeg\nLC08_L2SP_001062_20201031_20201106_02_T2_thumb_small.jpeg\n```\n\nWhen reading data or metadata, readers will merge them.\n\ne.g\n```python\nwith S2COGReader(\"S2A_L2A_20170729_19UDP_0\") as sentinel:\n img = sentinel.tile(78, 89, 8, bands=(\"B01\", \"B02\"))\n assert img.data.shape == (2, 256, 256)\n\n stats = sentinel.statistics(bands=(\"B01\", \"B02\"))\n print(stats)\n >> {\n 'B01': BandStatistics(\n min=2.0,\n max=17132.0,\n mean=2183.7570706659685,\n count=651247.0,\n sum=1422165241.0,\n std=3474.123975478363,\n median=370.0,\n majority=238.0,\n minority=2.0,\n unique=15112.0,\n histogram=[\n [476342.0, 35760.0, 27525.0, 24852.0, 24379.0, 23792.0, 20891.0, 13602.0, 3891.0, 213.0],\n [2.0, 1715.0, 3428.0, 5141.0, 6854.0, 8567.0, 10280.0, 11993.0, 13706.0, 15419.0, 17132.0]\n ],\n valid_percent=62.11,\n masked_pixels=397329.0,\n valid_pixels=651247.0,\n percentile_2=179.0,\n percentile_98=12465.0\n ),\n 'B02': BandStatistics(\n min=1.0,\n max=15749.0,\n mean=1941.2052554560712,\n count=651247.0,\n sum=1264204099.0,\n std=3130.545395156859,\n median=329.0,\n majority=206.0,\n minority=11946.0,\n unique=13904.0,\n histogram=[\n [479174.0, 34919.0, 27649.0, 25126.0, 24913.0, 24119.0, 20223.0, 12097.0, 2872.0, 155.0],\n [1.0, 1575.8, 3150.6, 4725.4, 6300.2, 7875.0, 9449.8, 11024.6, 12599.4, 14174.199999999999, 15749.0]\n ],\n valid_percent=62.11,\n masked_pixels=397329.0,\n valid_pixels=651247.0,\n percentile_2=134.0,\n percentile_98=11227.079999999958\n )}\n\n print(stats[\"B01\"].min)\n >> 2.0\n```\n\n### Mosaic Reader: Copernicus DEM\n\nThe Copernicus DEM GLO-30 and GLO-90 readers are not **per scene** but **mosaic** readers. This is possible because the dataset is a global dataset with file names having the `geo-location` of the COG, meaning we can easily contruct a filepath from a coordinate.\n\n```python\nfrom rio_tiler_pds.copernicus.aws import Dem30Reader\n\nwith Dem30Reader() as dem:\n print(dem.assets_for_point(-57.2, -11.2))\n\n>> ['s3://copernicus-dem-30m/Copernicus_DSM_COG_10_S12_00_W058_00_DEM/Copernicus_DSM_COG_10_S12_00_W058_00_DEM.tif']\n```\n\n## Changes\n\nSee [CHANGES.md](https://github.com/cogeotiff/rio-tiler-pds/blob/main/CHANGES.md).\n\n## Contribution & Development\n\nSee [CONTRIBUTING.md](https://github.com/cogeotiff/rio-tiler/blob/main/CONTRIBUTING.md)\n\n## License\n\nSee [LICENSE.txt](https://github.com/cogeotiff/rio-tiler-pds/blob/main/LICENSE.txt)\n\n## Authors\n\nThe rio-tiler project was begun at Mapbox and has been transferred in January 2019.\n\nSee [AUTHORS.txt](https://github.com/cogeotiff/rio-tiler-pds/blob/main/AUTHORS.txt) for a listing of individual contributors.\n",
"bugtrack_url": null,
"license": null,
"summary": "Get mercator tile from cloud hosted dataset such as CBERS-4, Sentinel-2, Sentinel-1 and Landsat-8 AWS PDS.",
"version": "0.10.1",
"project_urls": {
"Documentation": "https://cogeotiff.github.io/rio-tiler-pds/",
"Source": "https://github.com/cogeotiff/rio-tiler-pds"
},
"split_keywords": [
"cogeo",
"cloud optimized geotiff",
"aws pds"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "566887af1e44fdce4a21e1fe6d8fb1fa06d789dde42661cf914dc2dbe486b532",
"md5": "7fc85adc736ccaf4287224a4c88e3cef",
"sha256": "1884632a9535cfbbc3aff5e6cbe084a1285ab5f197f4b06bc9b0c639a2ce8be5"
},
"downloads": -1,
"filename": "rio_tiler_pds-0.10.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "7fc85adc736ccaf4287224a4c88e3cef",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 34757,
"upload_time": "2023-08-21T10:04:13",
"upload_time_iso_8601": "2023-08-21T10:04:13.645580Z",
"url": "https://files.pythonhosted.org/packages/56/68/87af1e44fdce4a21e1fe6d8fb1fa06d789dde42661cf914dc2dbe486b532/rio_tiler_pds-0.10.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "c279f266c9405dffd191bc7f2edf2bc0cf2121280c6ab1a4e77a59a22820390d",
"md5": "2f217f49c885971a0cfd136db85a871c",
"sha256": "215bc99728181d8eb4a87e5ed1ae2d83e856f8f9a1e3a894e0d258f61da0f89d"
},
"downloads": -1,
"filename": "rio_tiler_pds-0.10.1.tar.gz",
"has_sig": false,
"md5_digest": "2f217f49c885971a0cfd136db85a871c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 33678,
"upload_time": "2023-08-21T10:04:15",
"upload_time_iso_8601": "2023-08-21T10:04:15.634841Z",
"url": "https://files.pythonhosted.org/packages/c2/79/f266c9405dffd191bc7f2edf2bc0cf2121280c6ab1a4e77a59a22820390d/rio_tiler_pds-0.10.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-21 10:04:15",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "cogeotiff",
"github_project": "rio-tiler-pds",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "rio-tiler-pds"
}
