***
nrt
***
*Python package for near real time detection of change in spatio-temporal datasets*
.. image:: https://badge.fury.io/py/nrt.svg
:target: https://badge.fury.io/py/nrt
.. image:: https://readthedocs.org/projects/nrt/badge/?version=latest
:target: https://nrt.readthedocs.io/en/latest/?badge=latest
:alt: Documentation Status
.. image:: https://github.com/ec-jrc/nrt/actions/workflows/build_and_test.yml/badge.svg
:target: https://github.com/ec-jrc/nrt/actions/workflows/build_and_test.yml
:alt: Build status
.. image:: https://joss.theoj.org/papers/10.21105/joss.06815/status.svg
:target: https://doi.org/10.21105/joss.06815
``nrt`` provides a standardized interface for Near Real Time monitoring of disturbances on satellite image time-series.
The package is optimized for fast computation and suitable for operational deployment at scale.
A typical operational use case of such package would be a system constantly receiving new satellite based acquisitions and generating alerts when an anomaly is detected.
Five monitoring frameworks from scientific literature on change detection are implemented and exposed via a common API.
All five monitoring framework share a common general approach which consists in modelling the "normal" behavior of the variable through time by fitting a linear model on a user defined stable history period and monitoring until a "break" is detected.
Monitoring starts right after the stable history period, and for each new incoming observation the observed value is compared to the predicted "normal" behavior.
When observations and predictions diverge, a "break" is detected.
A confirmed "break" typically requires several successive diverging observations, this sensitivity or rapid detection capacity depending on many variables such as the algorithm, its fitting and monitoring parameters, the noise level of the history period or the magnitude of the divergence.
The five monitoring frameworks implemented are:
- Exponentially Weighted Moving Average (EWMA_) (Brooks et al., 2013)
- Cumulative Sum of Residual (CuSum_) (Verbesselt et al., 2012; Zeileis et al., 2005). CuSum is one of the monitoring option of the ``bfastmonitor`` function available in the R package bfast_.
- Moving Sum of Residuals (MoSum_) (Verbesselt et al., 2012; Zeileis et al., 2005). MoSum is one of the monitoring option of the ``bfastmonitor`` function available in the R package bfast_.
- Continuous Change Detection and Classification of land cover (CCDC_, CMFDA_) (Zhu et al., 2012, 2014) - Partial implementation only of the original published method.
- InterQuantile Range (IQR) - Simple, unpublished outlier identification strategy described on stackexchange_.
Parts of this package are derived from Chris Holden's pybreakpoints_ and yatsm_ packages. Please see the copyright statements in the respective modules.
.. _EWMA: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=6573358
.. _CMFDA: https://www.sciencedirect.com/science/article/pii/S0034425712000387
.. _CCDC: https://www.sciencedirect.com/science/article/pii/S0034425714000248#bbb0350
.. _CuSum: https://www.sciencedirect.com/science/article/pii/S0034425712001150
.. _MoSum: https://www.sciencedirect.com/science/article/pii/S0034425712001150
.. _stackexchange: https://stats.stackexchange.com/a/1153
.. _bfast: https://bfast.r-forge.r-project.org/
.. _pybreakpoints: https://github.com/ceholden/pybreakpoints
.. _yatsm: https://github.com/ceholden/yatsm
Documentation
=============
Learn more about nrt in its official documentation at https://nrt.readthedocs.io/en/latest/
Installation
============
.. code-block:: bash
pip install nrt
The main dependencies, which should be automatically resolved by ``pip``, are:
- `numpy <https://pypi.org/project/numpy/>`_
- `scipy <https://pypi.org/project/scipy/>`_
- `xarray <https://pypi.org/project/xarray/>`_
- `numba <https://pypi.org/project/numba/>`_
- `rasterio <https://pypi.org/project/rasterio/>`_
- `netCDF4 <https://pypi.org/project/netCDF4/>`_
Example usage
=============
The snippet below presents a near real time monitoring simulation. The input data is split in stable history and monitoring period; the monitoring class is instantiated (EWMA algorithm), a simple harmonic model is fitted on the history period, and new acquisition are passed to the monitor method one at the time. Note that in a real operational scenario where new observations come at a less frequent interval (e.g. every 5 or 8 days which coorespond to the revisit frequency of sentinel 2 and Landsat constellations respectively), the monitoring state can be saved on disk and reloaded when required.
.. code-block:: python
import datetime
from nrt.monitor.ewma import EWMA
from nrt import data
# Forest/non-forest mask
mask = (data.romania_forest_cover_percentage() > 30).astype('int')
# NDVI training and monitoring periods
s2_cube = data.romania_20m()
s2_cube['ndvi'] = (s2_cube.B8A - s2_cube.B04) / (s2_cube.B8A + s2_cube.B04)
s2_cube = s2_cube.where(s2_cube.SCL.isin([4,5,7]))
ndvi_history = s2_cube.ndvi.sel(time=slice('2015-01-01', '2018-12-31'))
ndvi_monitoring = s2_cube.ndvi.sel(time=slice('2019-01-01', '2021-12-31'))
# Instantiate monitoring class and fit stable history
EwmaMonitor = EWMA(trend=False, mask=mask)
EwmaMonitor.fit(dataarray=ndvi_history)
# Monitor new observations
for array, date in zip(ndvi_monitoring.values,
ndvi_monitoring.time.values.astype('M8[s]').astype(datetime.datetime)):
EwmaMonitor.monitor(array=array, date=date)
# At any time a monitoring report can be produced with EwmaMonitor.report(filename)
# and state of the monitoring instance can be saved as netcdf with
# EwmaMonitor.to_netcdf(filename)
Contributing
============
Any type of contribution is welcome. Please see the contributing guidelines at `CONTRIBUTING.md <CONTRIBUTING.md>`_.
Citing nrt
==========
If you use nrt in your research or project, please consider citing it using the following BibTeX entry.
.. code-block:: bibtex
@article{dutrieux2024nrt,
year = {2024},
publisher = {The Open Journal},
volume = {9},
number = {100},
pages = {6815},
author = {Lo\"{i}c Dutrieux and Jonas Viehweger},
title = {nrt: operational monitoring of satellite image time-series in Python},
journal = {Journal of Open Source Software},
doi = {10.21105/joss.06815},
}
About the authors
=================
Loïc Dutrieux works as a remote sensing researcher at the Joint Research Center (JRC) in Ispra, Italy. His work focuses on forest disturbances mapping and characterization from satellite image time-series.
Jonas Viehweger is a young researcher with a MSc in remote sensing from the university of Marburg, Germany. He developped a large part of the nrt package during his traineeship period at the Joint Research Center (JRC) in Ispra, Italy.
Chris Holden implemented many time-series change detection algorithms in python during his PhD at Boston university.
References
==========
Brooks, E.B., Wynne, R.H., Thomas, V.A., Blinn, C.E. and Coulston, J.W., 2013. On-the-fly massively multitemporal change detection using statistical quality control charts and Landsat data. IEEE Transactions on Geoscience and Remote Sensing, 52(6), pp.3316-3332.
https://doi.org/10.1109/TGRS.2013.2272545
Verbesselt, J., Zeileis, A. and Herold, M., 2012. Near real-time disturbance detection using satellite image time series. Remote Sensing of Environment, 123, pp.98-108.
https://doi.org/10.1016/j.rse.2012.02.022
Zeileis, A., Leisch, F., Kleiber, C. and Hornik, K., 2005. Monitoring structural change in dynamic econometric models. Journal of Applied Econometrics, 20(1), pp.99-121.
https://doi.org/10.1002/jae.776
Zhu, Z., Woodcock, C.E. and Olofsson, P., 2012. Continuous monitoring of forest disturbance using all available Landsat imagery. Remote sensing of environment, 122, pp.75-91.
https://doi.org/10.1016/j.rse.2011.10.030
Zhu, Z. and Woodcock, C.E., 2014. Continuous change detection and classification of land cover using all available Landsat data. Remote sensing of Environment, 144, pp.152-171.
https://doi.org/10.1016/j.rse.2014.01.011
Raw data
{
"_id": null,
"home_page": null,
"name": "nrt",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "sentinel2, xarray, datacube, monitoring, change",
"author": "Jonas Viehweger, Chris Holden",
"author_email": "Loic Dutrieux <loic.dutrieux@ec.europa.eu>",
"download_url": "https://files.pythonhosted.org/packages/6f/58/a3b647e28ce82c4af28fbdc8ed93369ce39b48c3f6538c68fe27d982f7e8/nrt-0.3.0.tar.gz",
"platform": null,
"description": "***\nnrt\n***\n\n*Python package for near real time detection of change in spatio-temporal datasets*\n\n.. image:: https://badge.fury.io/py/nrt.svg\n :target: https://badge.fury.io/py/nrt\n\n.. image:: https://readthedocs.org/projects/nrt/badge/?version=latest\n :target: https://nrt.readthedocs.io/en/latest/?badge=latest\n :alt: Documentation Status\n\n.. image:: https://github.com/ec-jrc/nrt/actions/workflows/build_and_test.yml/badge.svg\n :target: https://github.com/ec-jrc/nrt/actions/workflows/build_and_test.yml\n :alt: Build status\n\n.. image:: https://joss.theoj.org/papers/10.21105/joss.06815/status.svg\n :target: https://doi.org/10.21105/joss.06815\n\n\n``nrt`` provides a standardized interface for Near Real Time monitoring of disturbances on satellite image time-series.\nThe package is optimized for fast computation and suitable for operational deployment at scale.\nA typical operational use case of such package would be a system constantly receiving new satellite based acquisitions and generating alerts when an anomaly is detected.\nFive monitoring frameworks from scientific literature on change detection are implemented and exposed via a common API.\nAll five monitoring framework share a common general approach which consists in modelling the \"normal\" behavior of the variable through time by fitting a linear model on a user defined stable history period and monitoring until a \"break\" is detected.\nMonitoring starts right after the stable history period, and for each new incoming observation the observed value is compared to the predicted \"normal\" behavior.\nWhen observations and predictions diverge, a \"break\" is detected.\nA confirmed \"break\" typically requires several successive diverging observations, this sensitivity or rapid detection capacity depending on many variables such as the algorithm, its fitting and monitoring parameters, the noise level of the history period or the magnitude of the divergence. \nThe five monitoring frameworks implemented are:\n\n- Exponentially Weighted Moving Average (EWMA_) (Brooks et al., 2013) \n- Cumulative Sum of Residual (CuSum_) (Verbesselt et al., 2012; Zeileis et al., 2005). CuSum is one of the monitoring option of the ``bfastmonitor`` function available in the R package bfast_.\n- Moving Sum of Residuals (MoSum_) (Verbesselt et al., 2012; Zeileis et al., 2005). MoSum is one of the monitoring option of the ``bfastmonitor`` function available in the R package bfast_.\n- Continuous Change Detection and Classification of land cover (CCDC_, CMFDA_) (Zhu et al., 2012, 2014) - Partial implementation only of the original published method.\n- InterQuantile Range (IQR) - Simple, unpublished outlier identification strategy described on stackexchange_.\n\n\nParts of this package are derived from Chris Holden's pybreakpoints_ and yatsm_ packages. Please see the copyright statements in the respective modules.\n\n.. _EWMA: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=6573358\n.. _CMFDA: https://www.sciencedirect.com/science/article/pii/S0034425712000387\n.. _CCDC: https://www.sciencedirect.com/science/article/pii/S0034425714000248#bbb0350\n.. _CuSum: https://www.sciencedirect.com/science/article/pii/S0034425712001150\n.. _MoSum: https://www.sciencedirect.com/science/article/pii/S0034425712001150\n.. _stackexchange: https://stats.stackexchange.com/a/1153\n.. _bfast: https://bfast.r-forge.r-project.org/\n.. _pybreakpoints: https://github.com/ceholden/pybreakpoints\n.. _yatsm: https://github.com/ceholden/yatsm\n\n\n\nDocumentation\n=============\n\nLearn more about nrt in its official documentation at https://nrt.readthedocs.io/en/latest/\n\n \nInstallation\n============\n\n.. code-block:: bash\n\n pip install nrt\n\n\nThe main dependencies, which should be automatically resolved by ``pip``, are:\n\n- `numpy <https://pypi.org/project/numpy/>`_\n- `scipy <https://pypi.org/project/scipy/>`_\n- `xarray <https://pypi.org/project/xarray/>`_\n- `numba <https://pypi.org/project/numba/>`_\n- `rasterio <https://pypi.org/project/rasterio/>`_\n- `netCDF4 <https://pypi.org/project/netCDF4/>`_\n\n\nExample usage\n=============\n\nThe snippet below presents a near real time monitoring simulation. The input data is split in stable history and monitoring period; the monitoring class is instantiated (EWMA algorithm), a simple harmonic model is fitted on the history period, and new acquisition are passed to the monitor method one at the time. Note that in a real operational scenario where new observations come at a less frequent interval (e.g. every 5 or 8 days which coorespond to the revisit frequency of sentinel 2 and Landsat constellations respectively), the monitoring state can be saved on disk and reloaded when required.\n\n.. code-block:: python\n\n import datetime\n\n from nrt.monitor.ewma import EWMA\n from nrt import data\n\n # Forest/non-forest mask\n mask = (data.romania_forest_cover_percentage() > 30).astype('int')\n\n # NDVI training and monitoring periods\n s2_cube = data.romania_20m()\n s2_cube['ndvi'] = (s2_cube.B8A - s2_cube.B04) / (s2_cube.B8A + s2_cube.B04)\n s2_cube = s2_cube.where(s2_cube.SCL.isin([4,5,7]))\n ndvi_history = s2_cube.ndvi.sel(time=slice('2015-01-01', '2018-12-31'))\n ndvi_monitoring = s2_cube.ndvi.sel(time=slice('2019-01-01', '2021-12-31'))\n\n # Instantiate monitoring class and fit stable history\n EwmaMonitor = EWMA(trend=False, mask=mask)\n EwmaMonitor.fit(dataarray=ndvi_history)\n\n # Monitor new observations\n for array, date in zip(ndvi_monitoring.values,\n ndvi_monitoring.time.values.astype('M8[s]').astype(datetime.datetime)):\n EwmaMonitor.monitor(array=array, date=date)\n\n # At any time a monitoring report can be produced with EwmaMonitor.report(filename)\n # and state of the monitoring instance can be saved as netcdf with\n # EwmaMonitor.to_netcdf(filename)\n\n\nContributing\n============\n\nAny type of contribution is welcome. Please see the contributing guidelines at `CONTRIBUTING.md <CONTRIBUTING.md>`_.\n\n\nCiting nrt\n==========\n\nIf you use nrt in your research or project, please consider citing it using the following BibTeX entry.\n\n.. code-block:: bibtex\n\n @article{dutrieux2024nrt,\n year = {2024},\n publisher = {The Open Journal},\n volume = {9},\n number = {100},\n pages = {6815},\n author = {Lo\\\"{i}c Dutrieux and Jonas Viehweger},\n title = {nrt: operational monitoring of satellite image time-series in Python},\n journal = {Journal of Open Source Software},\n doi = {10.21105/joss.06815},\n }\n\n\nAbout the authors\n=================\n\nLo\u00efc Dutrieux works as a remote sensing researcher at the Joint Research Center (JRC) in Ispra, Italy. His work focuses on forest disturbances mapping and characterization from satellite image time-series.\n\nJonas Viehweger is a young researcher with a MSc in remote sensing from the university of Marburg, Germany. He developped a large part of the nrt package during his traineeship period at the Joint Research Center (JRC) in Ispra, Italy.\n\nChris Holden implemented many time-series change detection algorithms in python during his PhD at Boston university.\n\n\nReferences\n==========\n\nBrooks, E.B., Wynne, R.H., Thomas, V.A., Blinn, C.E. and Coulston, J.W., 2013. On-the-fly massively multitemporal change detection using statistical quality control charts and Landsat data. IEEE Transactions on Geoscience and Remote Sensing, 52(6), pp.3316-3332.\nhttps://doi.org/10.1109/TGRS.2013.2272545\n\nVerbesselt, J., Zeileis, A. and Herold, M., 2012. Near real-time disturbance detection using satellite image time series. Remote Sensing of Environment, 123, pp.98-108.\nhttps://doi.org/10.1016/j.rse.2012.02.022\n\nZeileis, A., Leisch, F., Kleiber, C. and Hornik, K., 2005. Monitoring structural change in dynamic econometric models. Journal of Applied Econometrics, 20(1), pp.99-121.\nhttps://doi.org/10.1002/jae.776\n\nZhu, Z., Woodcock, C.E. and Olofsson, P., 2012. Continuous monitoring of forest disturbance using all available Landsat imagery. Remote sensing of environment, 122, pp.75-91.\nhttps://doi.org/10.1016/j.rse.2011.10.030\n\nZhu, Z. and Woodcock, C.E., 2014. Continuous change detection and classification of land cover using all available Landsat data. Remote sensing of Environment, 144, pp.152-171.\nhttps://doi.org/10.1016/j.rse.2014.01.011\n",
"bugtrack_url": null,
"license": "EUPL-1.2",
"summary": "Online monitoring with xarray",
"version": "0.3.0",
"project_urls": {
"Homepage": "https://github.com/ec-jrc/nrt.git"
},
"split_keywords": [
"sentinel2",
" xarray",
" datacube",
" monitoring",
" change"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e873189ebe7f260b92c90fb4ac268ee25598267f7791b434117f6b4b834dc61a",
"md5": "5ead4c4246a511f985757094e28cf852",
"sha256": "431dc765e64cc5ea58cec6d01fbfd3e0f540e80eed406caf33121916a34e9c8f"
},
"downloads": -1,
"filename": "nrt-0.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5ead4c4246a511f985757094e28cf852",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 1262925,
"upload_time": "2024-08-29T13:17:30",
"upload_time_iso_8601": "2024-08-29T13:17:30.657654Z",
"url": "https://files.pythonhosted.org/packages/e8/73/189ebe7f260b92c90fb4ac268ee25598267f7791b434117f6b4b834dc61a/nrt-0.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6f58a3b647e28ce82c4af28fbdc8ed93369ce39b48c3f6538c68fe27d982f7e8",
"md5": "491b9a9d1541cf606c51ee63bb074b70",
"sha256": "9f926e029dd12c20128f62d074156cb4e05586b630f46743dbc7b0d3ccd89231"
},
"downloads": -1,
"filename": "nrt-0.3.0.tar.gz",
"has_sig": false,
"md5_digest": "491b9a9d1541cf606c51ee63bb074b70",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 1250084,
"upload_time": "2024-08-29T13:17:32",
"upload_time_iso_8601": "2024-08-29T13:17:32.146719Z",
"url": "https://files.pythonhosted.org/packages/6f/58/a3b647e28ce82c4af28fbdc8ed93369ce39b48c3f6538c68fe27d982f7e8/nrt-0.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-29 13:17:32",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ec-jrc",
"github_project": "nrt",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "nrt"
}
JSON
