# ODC Statistician
Statistician is a framework of tools for generating statistical summaries of
large collections of Earth Observation Imagery managed in an Open
Datacube Instance. It is a spiritual successor to [datacube-stats](https://github.com/opendatacube/datacube-stats/),
but intended to run in a cloud environment rather than on a HPC.
It has already run at continental scale to produce annual geomedian summaries of all of Africa based on Sentinel-2 data.
It is still under development, including adding support for processing sibling products, eg. Water Observations together with Surface Reflectance Quality classifications.
## Installation
```
pip install odc-stats
```
## Usage
Statistician offers a set of tools to generate clear pixel count and geometric median on Sentinel-2 and Landsat satellite images.
Running statistician requires a few steps, to prepare and cache the input data, set up the configuration file
### Steps to run statistician
### 1 - Save tasks
From your Sandbox (or a machine that has access to your ODC database), run:
```
odc-stats save-tasks --frequency annual --grid au-30 ga_ls8c_ard_3
```
Running this command generates a set of files:
1. a CSV file listing all tasks for all the years in the database
2. a dataset cache file used by statistician when running jobs
3. A GeoJSON file per year, for visualising the prospective run
```
ga_ls8c_ard_3_all-2013--P1Y.geojson
ga_ls8c_ard_3_all-2014--P1Y.geojson
ga_ls8c_ard_3_all-2015--P1Y.geojson
ga_ls8c_ard_3_all-2016--P1Y.geojson
ga_ls8c_ard_3_all-2017--P1Y.geojson
ga_ls8c_ard_3_all-2018--P1Y.geojson
ga_ls8c_ard_3_all-2019--P1Y.geojson
ga_ls8c_ard_3_all-2020--P1Y.geojson
ga_ls8c_ard_3_all-2021--P1Y.geojson
ga_ls8c_ard_3_all.csv
ga_ls8c_ard_3_all.db
```
The CSV file contains the list of tasks for all the years and consists of x, y coordinates of each destination tile, as well as the counts of datasets and satellite observations.
GeoJSON files are useful when selecting test regions as well as for debugging specific tasks - see the example below from Landsat-8, 2015. This example shows a tile from an Australian coastal region near Sydney.
<img src="docs/auxiliary/screenshot-L8-2015.png" alt="Screenshot GeoJSON" width="1000"/>
#### Dataset Fusing
Some summary products require data from bands stored in multiple products. Products can be fused to use bands from both products in the derivative products, this creates a virtual product that contains the bands from both products. Note that for datasets to be fused they must have the same `center_time` and `region_code`. This process find the matching dataset from each product that are in the same time and place and fuses them into one product.
An example of this is `fc-percentiles`, which uses the fractional covers bands in `ga_ls_fc_3` to calculate the percentiles, and uses the `ga_ls_wo_3` band to mask out bad data.
Two products can be fused using the following syntax:
```
odc-stats save-tasks --frequency annual --grid au-30 --year 2017 --input-products ga_ls_fc_3+ga_ls_wo_3
```
#### Caching datasets from multiple products
Some summary products require data from multiple satellites, e.g. `tcw_pc` uses scenes from `ga_ls5t_ard_3`, `ga_ls7e_ard_3`, and `ga_ls8c_ard_3`. Note that this is different from fusing products, the scenes do not have to be in the same place or time and this simply caches the datasets from each product as is. To save datasets from multiple products:
```
odc-stats save-tasks --frequency annual --grid au-30 --year 2017 --input-products ga_ls5t_ard_3-ga_ls7e_ard_3-ga_ls8c_ard_3
```
### 2 - Run Statistician
To actually run Statistician to produce some output data, it requires:
1. A **task cache/database**, as generated in the previous step.
2. A **YAML configuration file** describing the calculations to perform,
where to output data, what metadata to attach, etc.
Statistician has multiple different execution modes:
1. Execute a single task. Useful when testing.
2. Execute a set of tasks from the task database.
3. Pull tasks continuously from an SQS queue and exit when the queue is empty.
See [Orchestration](#orchestration) below for running large jobs.
A sample command to run a single task:
```
odc-stats run ga_ls8c_ard_3_all.db \
2015--P1Y/41/13 \
--config cfg.yaml \
--resolution=30 \
--location file:///localpath/ \
--threads=16 \
--memory-limit=60Gi
```
```
Usage: odc-stats run [OPTIONS] FILEDB [TASKS]...
Run Stats.
Task could be one of the 3 things
1. Comma-separated triplet: period,x,y or 'x[+-]<int>/y[+-]<int>/period
2019--P1Y,+003,-004
2019--P1Y/3/-4 `/` is also accepted
x+003/y-004/2019--P1Y is accepted as well
2. A zero based index
3. A slice following python convention <start>:<stop>[:<step]
::10 -- every tenth task: 0,10,20,..
1::10 -- every tenth but skip first one 1, 11, 21 ..
:100 -- first 100 tasks
If no tasks are supplied and --from-sqs is not used, the whole file will
be processed.
```
Where `cfg.yaml` contains the following configuration (a sample from a geomedian run):
```yaml
plugin: gm-ls
plugin_config:
resampling: bilinear
bands: ["blue", "red", "green", "nir", "swir1", "swir2"]
rgb_bands: ["red", "green", "blue"]
mask_band: fmask
cloud_classes: ["shadow", "cloud"]
filters: [0, 0]
aux_names:
smad: sdev
emad: edev
bcmad: bcdev
count: count
product:
name: ga_ls8c_nbart_gm_cyear_3
short_name: ga_ls8c_nbart_gm_cyear_3
version: 3.0.0
product_family: geomedian
max_processing_time: 3600
job_queue_max_lease: 300
renew_safety_margin: 60
future_poll_interval: 2
s3_acl: public-read
# Generic product attributes
cog_opts:
zlevel: 9
overrides:
# If you're making an RGBA image, compress better
rgba:
compress: JPEG
jpeg_quality: 90
# If you're writing a categorical raster, do overview resampling with nearest or mode
qa:
overview_resampling: nearest
```
Note that configurations will vary between different products. See this [sample configuration for Australian Landsat-8](https://bitbucket.org/geoscienceaustralia/datakube-apps/src/develop/workspaces/dea-dev/processing/06_stats.yaml).
If no cog_opts:overrides:rgba in configurations or relative plug-in rgba method return None. There is no rgba.tif in output files.
It is also possible to define custom plugins outside of `odc.stats.*`. To do that, define class deriving from `odc.stats.model.StatsPluginInterface` and implement `.input_data` and `.reduce` methods. You can then specify fully qualified name of your custom plugin in `cfg.yaml`.
```yaml
plugin: mycustomlib.SomePluginOfMine
plugin_config:
param1: 100
param2: "string param"
# ...
```
## Orchestration
> :warning: **This documentation section references some private configuration**
>
> Also, Statistician Orchestration is still under development. As it stabilising look here
> for more information.
To run Statistician on a large volume of data, use the Statistician orchestration tool which relies on SQS queues and Kubernetes jobs.
Before running a job, you will need to create the necessary infrastructures by running the step 1 described below.
### With SQS queues
### 1. Create queues and a user
Each user will need to create a user and a queue to publish their tasks to.
Note that each queue will have a corresponding dead letter queue.
The user and the queue will be created using a Terraform module which contains code similar to the following:
```terraform
module "odc_stats_geomedian" {
source = "../../../modules/statistician"
stats_process = "geomedian"
destination_bucket = "destination-bucket-name"
destination_path = "gm_s2_annual"
test_bucket = "test-bucket-name"
cluster_id = local.cluster_id
owner = local.owner
}
```
The statistician process name is used to name the user and the queue, and the destination and test buckets specifies the output locations in which this user has access to.
For an example, check the following file:
```
https://bitbucket.org/geoscienceaustralia/datakube/src/master/workspaces/deafrica-dev/03_odc_k8s_apps/stats_procesing.tf
```
Once you created a file similar to the above file, run the datakube pipeline:
<img src="docs/auxiliary/screenshot-pipeline.png" alt="Screenshot of Pipeline" width="1000"/>
### 2. Publish tasks
```
odc-stats publish-tasks s3-bucket/s3-prefix/ga_ls8c_nbart_gm_cyear_3.db stats-queue [task-id/index] --dry-run
```
This function populates ```stats-queue``` from the cache file. Each message contains a tile id or an index. An example of a tile id is shown below:
```x+003/y+004/2019--P1Y```
```publish-tasks``` publishes all the messages in the cache file by default.
However, the command offers an option to publish a subset of tasks based on their indexes from the CSV file.
For example, if your CSV file contains tasks across multiple years, you can just publish tasks belonging to a particular year by providing the range:
```
tasks=$(shuf -i 1000-2000)
```
Where 1000 to 2000 covers the indexes for a single year.
Always run with the dryrun option and check tasks against the CSV file to ensure you're publishing the correct set of indexes.
### 3. Run tasks
To run a job, create a job template.
The job template is a YAML file which uses the AWS user and the queue that you created in step 1 and specifies your run configurations.
This includes, location of the cache file containing the status of the dataset, the queue containing the list of tasks to process, AWS node group types to be deployed, resource requests and limits, plugin and product configs.
An example of the job definition for running geometric median on Landsat-8 can be found in the following location.
<span style="color:red">WARNING: Do not run the following configurations directly as it will overwrite existing data. Always change the output directory, user and the queue to point to the infrastructure which you created in step 1.</span>
```
https://bitbucket.org/geoscienceaustralia/datakube-apps/src/develop/workspaces/dea-dev/processing/06_stats.yaml
```
The following command will start your job. This will create n pods where n is the number of replicas specified in your template.
```
kp apply -f path-to-job-template
```
When you run statistician using jobs, tasks that are tried three times and have failed, will end up in the dead letter queue.
The user can then find the logs corresponding to these tasks and identify why they failed.
If the failure is due to resource limits, a rerun will usually pass.
However, if failure is due to issues such as bad datasets, you need to archive the bad datasets and regenerate your cache file before restarting the job.
### Issues
During the run, monitor the number of inflight messages.
This can be done from the cli or the dashboard.
Counts of inflight messages should be the same as (or close to) the number of pod/replicas.
Currently when the jobs complete, pods do not shutdown cleanly due to an error in the dask library.
In order to ensure assigned EC2 instances are released, you need to delete the job manually by running the following command:
```
kubectl pod delete -f path-to-job-template.yaml
```
Monitor the AWS Autoscaling Groups and ensure they are reduced to 0 once the job has completed.
There is a delay of about 10 minutes before ASGs are released.
### 4. Redrive queue
For tasks that have failed and need to be retried, they can be redirected to the main statistician queue using the following command:
```
redrive-queue stats-dead-letter-queue-name dead-letter-queue-name
```
### With Argo
The mass production of the summary products is ochastreted with Argo. HowTo [GA private](https://bitbucket.org/geoscienceaustralia/datakube-apps/src/develop/workspaces/dea-dev/processing/argo/)
### Monitoring
Grafana is a powerful tool for monitoring statistician jobs and pods.
You can query all the logs for a single job, to detect errors, or drill down into logs of a particular pod to identify specific issues for tasks.
To use this, log into the grafana which corresponds to the cluster that you're running the job in, from the left panel, select ```Explore```, change the explore method to ```Loki```, from log label, select ```job_name``` and then select your job name for example, ```stats-geomedian```.
You can now run a query on the log of the selected job:
<img src="docs/auxiliary/screenshot-grafana.png" alt="Screenshot Grafana" width="1000"/>
## Mosaics
To quickly review products, stats can generate a low resolution continental mosaic of a summary product. To run the command:
```
odc-stats generate-mosaic product_definition.yaml 's3://input-data-bucket/folder/*/*/metadata-file.stac-item.json' s3://output-bucket/folder/subfolder --bands nbart_red,nbart_blue
```
The arguments are:
- an ODC product definition `yaml` file defining the input summary product
- an `s3://` glob that specifies the location of the `metadata` for the datasets to be included
- the destination bucket and folder
- and the bands to be included in the mosaic. If no bands are specified all bands are used
This generates a COG per band with 7 overview layers into the destination `s3://` location.
This is fully parallelised, run time should scale almost linearly with number of
CPU cores. Memory usage should be something like `2048 * 2048 * bands *
dtype_size * num_cores`, so for 64 cores maybe something like ~1GB ish. Each
core gets a 2048 * 2048 chunk of the whole continent.
## Docker image
The dockerfile for the docker image used in the ochastration, which can be used in local environment for test/development as well, is located under [docker](./docker).
## Integration test
The integration test for the summary products is located under [tests](./tests). Currently the test is performed on all the official summary products published by DEA. The "golden files" to test against are stored on the public accessible [S3 bucket](s3://dea-public-data-dev/stats-golden-files). The "golder files" should be achived but not deleted in the case that we decide to amend or upgrade any product. It will help with tracking the changes that we intend and alerting those that we do not.
The test is meant to be regressive, i.e., the new version of `odc-stats` need to pass the test on the last version of docker image. The new version of docker image needs to pass the test on the current released version of `odc-stats`.
Raw data
{
"_id": null,
"home_page": "https://github.com/opendatacube/odc-stats/",
"name": "odc-stats",
"maintainer": "Open Data Cube",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": null,
"author": "Open Data Cube",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/56/36/3836136f43c268e64c2c502ca007eb213c9eed3282280ce68909434c0ecd/odc_stats-1.0.43.tar.gz",
"platform": "any",
"description": "# ODC Statistician\n\nStatistician is a framework of tools for generating statistical summaries of\nlarge collections of Earth Observation Imagery managed in an Open\nDatacube Instance. It is a spiritual successor to [datacube-stats](https://github.com/opendatacube/datacube-stats/),\nbut intended to run in a cloud environment rather than on a HPC.\n\nIt has already run at continental scale to produce annual geomedian summaries of all of Africa based on Sentinel-2 data.\n\nIt is still under development, including adding support for processing sibling products, eg. Water Observations together with Surface Reflectance Quality classifications.\n\n\n## Installation\n\n```\npip install odc-stats\n```\n\n## Usage\n\nStatistician offers a set of tools to generate clear pixel count and geometric median on Sentinel-2 and Landsat satellite images.\nRunning statistician requires a few steps, to prepare and cache the input data, set up the configuration file\n\n\n\n### Steps to run statistician\n\n### 1 - Save tasks\n\nFrom your Sandbox (or a machine that has access to your ODC database), run:\n\n```\nodc-stats save-tasks --frequency annual --grid au-30 ga_ls8c_ard_3\n```\n\nRunning this command generates a set of files:\n\n1. a CSV file listing all tasks for all the years in the database\n2. a dataset cache file used by statistician when running jobs\n3. A GeoJSON file per year, for visualising the prospective run\n\n```\nga_ls8c_ard_3_all-2013--P1Y.geojson\nga_ls8c_ard_3_all-2014--P1Y.geojson\nga_ls8c_ard_3_all-2015--P1Y.geojson\nga_ls8c_ard_3_all-2016--P1Y.geojson\nga_ls8c_ard_3_all-2017--P1Y.geojson\nga_ls8c_ard_3_all-2018--P1Y.geojson\nga_ls8c_ard_3_all-2019--P1Y.geojson\nga_ls8c_ard_3_all-2020--P1Y.geojson\nga_ls8c_ard_3_all-2021--P1Y.geojson\nga_ls8c_ard_3_all.csv\nga_ls8c_ard_3_all.db\n```\n\nThe CSV file contains the list of tasks for all the years and consists of x, y coordinates of each destination tile, as well as the counts of datasets and satellite observations.\n\nGeoJSON files are useful when selecting test regions as well as for debugging specific tasks - see the example below from Landsat-8, 2015. This example shows a tile from an Australian coastal region near Sydney.\n\n<img src=\"docs/auxiliary/screenshot-L8-2015.png\" alt=\"Screenshot GeoJSON\" width=\"1000\"/>\n\n#### Dataset Fusing\n\nSome summary products require data from bands stored in multiple products. Products can be fused to use bands from both products in the derivative products, this creates a virtual product that contains the bands from both products. Note that for datasets to be fused they must have the same `center_time` and `region_code`. This process find the matching dataset from each product that are in the same time and place and fuses them into one product.\n\nAn example of this is `fc-percentiles`, which uses the fractional covers bands in `ga_ls_fc_3` to calculate the percentiles, and uses the `ga_ls_wo_3` band to mask out bad data. \n\nTwo products can be fused using the following syntax:\n\n```\nodc-stats save-tasks --frequency annual --grid au-30 --year 2017 --input-products ga_ls_fc_3+ga_ls_wo_3\n```\n\n\n#### Caching datasets from multiple products\n\nSome summary products require data from multiple satellites, e.g. `tcw_pc` uses scenes from `ga_ls5t_ard_3`, `ga_ls7e_ard_3`, and `ga_ls8c_ard_3`. Note that this is different from fusing products, the scenes do not have to be in the same place or time and this simply caches the datasets from each product as is. To save datasets from multiple products:\n\n```\nodc-stats save-tasks --frequency annual --grid au-30 --year 2017 --input-products ga_ls5t_ard_3-ga_ls7e_ard_3-ga_ls8c_ard_3\n```\n\n### 2 - Run Statistician\n\nTo actually run Statistician to produce some output data, it requires:\n\n1. A **task cache/database**, as generated in the previous step.\n2. A **YAML configuration file** describing the calculations to perform, \n where to output data, what metadata to attach, etc. \n \nStatistician has multiple different execution modes:\n\n1. Execute a single task. Useful when testing.\n2. Execute a set of tasks from the task database.\n3. Pull tasks continuously from an SQS queue and exit when the queue is empty.\n See [Orchestration](#orchestration) below for running large jobs.\n\nA sample command to run a single task:\n\n```\nodc-stats run ga_ls8c_ard_3_all.db \\\n 2015--P1Y/41/13 \\\n --config cfg.yaml \\\n --resolution=30 \\\n --location file:///localpath/ \\\n --threads=16 \\\n --memory-limit=60Gi\n```\n\n```\nUsage: odc-stats run [OPTIONS] FILEDB [TASKS]...\n\n Run Stats.\n\n Task could be one of the 3 things\n\n 1. Comma-separated triplet: period,x,y or 'x[+-]<int>/y[+-]<int>/period\n 2019--P1Y,+003,-004\n 2019--P1Y/3/-4 `/` is also accepted\n x+003/y-004/2019--P1Y is accepted as well\n 2. A zero based index\n 3. A slice following python convention <start>:<stop>[:<step]\n ::10 -- every tenth task: 0,10,20,..\n 1::10 -- every tenth but skip first one 1, 11, 21 ..\n :100 -- first 100 tasks\n\n If no tasks are supplied and --from-sqs is not used, the whole file will\n be processed.\n```\n\nWhere `cfg.yaml` contains the following configuration (a sample from a geomedian run):\n```yaml\nplugin: gm-ls\nplugin_config:\n resampling: bilinear\n bands: [\"blue\", \"red\", \"green\", \"nir\", \"swir1\", \"swir2\"]\n rgb_bands: [\"red\", \"green\", \"blue\"]\n mask_band: fmask\n cloud_classes: [\"shadow\", \"cloud\"]\n filters: [0, 0] \n aux_names:\n smad: sdev\n emad: edev\n bcmad: bcdev\n count: count\nproduct:\n name: ga_ls8c_nbart_gm_cyear_3\n short_name: ga_ls8c_nbart_gm_cyear_3 \n version: 3.0.0\n product_family: geomedian\nmax_processing_time: 3600\njob_queue_max_lease: 300\nrenew_safety_margin: 60\nfuture_poll_interval: 2\ns3_acl: public-read\n# Generic product attributes\ncog_opts:\n zlevel: 9\n overrides:\n # If you're making an RGBA image, compress better\n rgba:\n compress: JPEG\n jpeg_quality: 90\n # If you're writing a categorical raster, do overview resampling with nearest or mode\n qa:\n overview_resampling: nearest\n```\n\nNote that configurations will vary between different products. See this [sample configuration for Australian Landsat-8](https://bitbucket.org/geoscienceaustralia/datakube-apps/src/develop/workspaces/dea-dev/processing/06_stats.yaml).\n\nIf no cog_opts:overrides:rgba in configurations or relative plug-in rgba method return None. There is no rgba.tif in output files.\n\nIt is also possible to define custom plugins outside of `odc.stats.*`. To do that, define class deriving from `odc.stats.model.StatsPluginInterface` and implement `.input_data` and `.reduce` methods. You can then specify fully qualified name of your custom plugin in `cfg.yaml`.\n\n```yaml\nplugin: mycustomlib.SomePluginOfMine\nplugin_config:\n param1: 100\n param2: \"string param\"\n # ...\n```\n\n## Orchestration\n\n> :warning: **This documentation section references some private configuration**\n>\n> Also, Statistician Orchestration is still under development. As it stabilising look here\n> for more information.\n\nTo run Statistician on a large volume of data, use the Statistician orchestration tool which relies on SQS queues and Kubernetes jobs.\nBefore running a job, you will need to create the necessary infrastructures by running the step 1 described below.\n\n### With SQS queues\n\n### 1. Create queues and a user \nEach user will need to create a user and a queue to publish their tasks to.\nNote that each queue will have a corresponding dead letter queue. \n\nThe user and the queue will be created using a Terraform module which contains code similar to the following:\n\n```terraform\nmodule \"odc_stats_geomedian\" {\n\n source = \"../../../modules/statistician\"\n\n stats_process = \"geomedian\"\n\n destination_bucket = \"destination-bucket-name\"\n destination_path = \"gm_s2_annual\"\n\n test_bucket = \"test-bucket-name\"\n\n cluster_id = local.cluster_id\n owner = local.owner\n}\n```\nThe statistician process name is used to name the user and the queue, and the destination and test buckets specifies the output locations in which this user has access to.\n\nFor an example, check the following file:\n```\nhttps://bitbucket.org/geoscienceaustralia/datakube/src/master/workspaces/deafrica-dev/03_odc_k8s_apps/stats_procesing.tf\n```\nOnce you created a file similar to the above file, run the datakube pipeline:\n\n<img src=\"docs/auxiliary/screenshot-pipeline.png\" alt=\"Screenshot of Pipeline\" width=\"1000\"/>\n\n\n### 2. Publish tasks \n\n```\nodc-stats publish-tasks s3-bucket/s3-prefix/ga_ls8c_nbart_gm_cyear_3.db stats-queue [task-id/index] --dry-run\n```\nThis function populates ```stats-queue``` from the cache file. Each message contains a tile id or an index. An example of a tile id is shown below:\n\n```x+003/y+004/2019--P1Y```\n\n```publish-tasks``` publishes all the messages in the cache file by default.\nHowever, the command offers an option to publish a subset of tasks based on their indexes from the CSV file.\nFor example, if your CSV file contains tasks across multiple years, you can just publish tasks belonging to a particular year by providing the range:\n\n```\ntasks=$(shuf -i 1000-2000)\n```\n\nWhere 1000 to 2000 covers the indexes for a single year.\nAlways run with the dryrun option and check tasks against the CSV file to ensure you're publishing the correct set of indexes.\n\n### 3. Run tasks \n\nTo run a job, create a job template.\nThe job template is a YAML file which uses the AWS user and the queue that you created in step 1 and specifies your run configurations.\nThis includes, location of the cache file containing the status of the dataset, the queue containing the list of tasks to process, AWS node group types to be deployed, resource requests and limits, plugin and product configs.\nAn example of the job definition for running geometric median on Landsat-8 can be found in the following location.\n\n\n<span style=\"color:red\">WARNING: Do not run the following configurations directly as it will overwrite existing data. Always change the output directory, user and the queue to point to the infrastructure which you created in step 1.</span>\n```\nhttps://bitbucket.org/geoscienceaustralia/datakube-apps/src/develop/workspaces/dea-dev/processing/06_stats.yaml\n```\n\nThe following command will start your job. This will create n pods where n is the number of replicas specified in your template.\n```\nkp apply -f path-to-job-template\n```\n\nWhen you run statistician using jobs, tasks that are tried three times and have failed, will end up in the dead letter queue.\nThe user can then find the logs corresponding to these tasks and identify why they failed.\nIf the failure is due to resource limits, a rerun will usually pass.\nHowever, if failure is due to issues such as bad datasets, you need to archive the bad datasets and regenerate your cache file before restarting the job.\n\n### Issues \n\nDuring the run, monitor the number of inflight messages.\nThis can be done from the cli or the dashboard.\nCounts of inflight messages should be the same as (or close to) the number of pod/replicas.\n\nCurrently when the jobs complete, pods do not shutdown cleanly due to an error in the dask library.\nIn order to ensure assigned EC2 instances are released, you need to delete the job manually by running the following command:\n\n``` \nkubectl pod delete -f path-to-job-template.yaml\n```\n\nMonitor the AWS Autoscaling Groups and ensure they are reduced to 0 once the job has completed.\nThere is a delay of about 10 minutes before ASGs are released. \n\n### 4. Redrive queue\nFor tasks that have failed and need to be retried, they can be redirected to the main statistician queue using the following command:\n\n```\nredrive-queue stats-dead-letter-queue-name dead-letter-queue-name \n```\n\n### With Argo\n\nThe mass production of the summary products is ochastreted with Argo. HowTo [GA private](https://bitbucket.org/geoscienceaustralia/datakube-apps/src/develop/workspaces/dea-dev/processing/argo/)\n\n### Monitoring\n\nGrafana is a powerful tool for monitoring statistician jobs and pods.\nYou can query all the logs for a single job, to detect errors, or drill down into logs of a particular pod to identify specific issues for tasks.\nTo use this, log into the grafana which corresponds to the cluster that you're running the job in, from the left panel, select ```Explore```, change the explore method to ```Loki```, from log label, select ```job_name``` and then select your job name for example, ```stats-geomedian```.\nYou can now run a query on the log of the selected job:\n\n<img src=\"docs/auxiliary/screenshot-grafana.png\" alt=\"Screenshot Grafana\" width=\"1000\"/>\n\n\n## Mosaics\n\nTo quickly review products, stats can generate a low resolution continental mosaic of a summary product. To run the command:\n\n```\nodc-stats generate-mosaic product_definition.yaml 's3://input-data-bucket/folder/*/*/metadata-file.stac-item.json' s3://output-bucket/folder/subfolder --bands nbart_red,nbart_blue\n```\n\nThe arguments are:\n\n- an ODC product definition `yaml` file defining the input summary product\n- an `s3://` glob that specifies the location of the `metadata` for the datasets to be included\n- the destination bucket and folder\n- and the bands to be included in the mosaic. If no bands are specified all bands are used\n\n\nThis generates a COG per band with 7 overview layers into the destination `s3://` location.\n\nThis is fully parallelised, run time should scale almost linearly with number of\nCPU cores. Memory usage should be something like `2048 * 2048 * bands *\ndtype_size * num_cores`, so for 64 cores maybe something like ~1GB ish. Each\ncore gets a 2048 * 2048 chunk of the whole continent.\n\n## Docker image\n\nThe dockerfile for the docker image used in the ochastration, which can be used in local environment for test/development as well, is located under [docker](./docker).\n\n## Integration test\n\nThe integration test for the summary products is located under [tests](./tests). Currently the test is performed on all the official summary products published by DEA. The \"golden files\" to test against are stored on the public accessible [S3 bucket](s3://dea-public-data-dev/stats-golden-files). The \"golder files\" should be achived but not deleted in the case that we decide to amend or upgrade any product. It will help with tracking the changes that we intend and alerting those that we do not.\n\nThe test is meant to be regressive, i.e., the new version of `odc-stats` need to pass the test on the last version of docker image. The new version of docker image needs to pass the test on the current released version of `odc-stats`.\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "Statistical Product Generation Framework",
"version": "1.0.43",
"project_urls": {
"Homepage": "https://github.com/opendatacube/odc-stats/"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e38306f6b255e7fe069f2e3c0bdfba313be0a436e5cc6c1c7171688b15a3ff8a",
"md5": "27d2191c05c8c59665cad6d4c79ab85c",
"sha256": "dc166033a5c2e2581491f96b0d6102c26af3ce686eb34288d21856c861984cf9"
},
"downloads": -1,
"filename": "odc_stats-1.0.43-py3-none-any.whl",
"has_sig": false,
"md5_digest": "27d2191c05c8c59665cad6d4c79ab85c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 72683,
"upload_time": "2024-04-19T01:39:46",
"upload_time_iso_8601": "2024-04-19T01:39:46.881107Z",
"url": "https://files.pythonhosted.org/packages/e3/83/06f6b255e7fe069f2e3c0bdfba313be0a436e5cc6c1c7171688b15a3ff8a/odc_stats-1.0.43-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "56363836136f43c268e64c2c502ca007eb213c9eed3282280ce68909434c0ecd",
"md5": "2e81a72dd10a01463f629697e49029e0",
"sha256": "e89b480607f305ce2a0056b2c83539060d6e9bb7fb7326dfb42e8347b8397f52"
},
"downloads": -1,
"filename": "odc_stats-1.0.43.tar.gz",
"has_sig": false,
"md5_digest": "2e81a72dd10a01463f629697e49029e0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 65705,
"upload_time": "2024-04-19T01:39:48",
"upload_time_iso_8601": "2024-04-19T01:39:48.842136Z",
"url": "https://files.pythonhosted.org/packages/56/36/3836136f43c268e64c2c502ca007eb213c9eed3282280ce68909434c0ecd/odc_stats-1.0.43.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-19 01:39:48",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "opendatacube",
"github_project": "odc-stats",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"lcname": "odc-stats"
}
JSON
