| Name | pyemu JSON |
| Version |
1.3.4
JSON |
| download |
| home_page | None |
| Summary | pyEMU is a set of python modules for interfacing with PEST and PEST++ |
| upload_time | 2024-03-26 22:34:55 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.8 |
| license | BSD 3-Clause |
| keywords |
pest
pestpp
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
pyEMU
=====
python modules for model-independent FOSM (first-order, second-moment) (a.k.a linear-based, a.k.a. Bayes linear)
uncertainty analyses and data-worth analyses, non-linear uncertainty analyses and interfacing with PEST and PEST++.
pyEMU also has a pure python (pandas and numpy) implementation of ordinary kriging for geostatistical interpolation and
support for generating high-dimensional PEST(++) model interfaces, including support for (very) high-dimensional
ensemble generation and handling
Main branch:
[](https://github.com/pypest/pyemu/actions)
[](https://coveralls.io/github/pypest/pyemu?branch=main)
Develop branch:
[](https://github.com/pypest/pyemu/actions)
[](https://coveralls.io/github/pypest/pyemu?branch=develop)
Documentation
=============
Complete user's guide:
[https://pyemu.readthedocs.io/en/latest/](https://pyemu.readthedocs.io/en/latest/)
The pyEMU documentation is being treated as a first-class citizen! Also see the example notebooks in the repo.
What is pyEMU?
================
pyEMU is a set of python modules for model-independent, user-friendly, computer model uncertainty analysis. pyEMU is tightly coupled to the open-source suite PEST (Doherty 2010a and 2010b, and Doherty and other, 2010) and PEST++ (Welter and others, 2015, Welter and other, 2012), which are tools for model-independent parameter estimation. However, pyEMU can be used with generic array objects, such as numpy ndarrays.
Several equations are implemented, including Schur's complement for conditional uncertainty propagation (a.k.a. Bayes Linear estimation) (the foundation of the PREDUNC suite from PEST) and error variance analysis (the foundation of the PREDVAR suite of PEST). pyEMU has easy-to-use routines for parameter and data worth analyses, which estimate how increased parameter knowledge and/or additional data effect forecast uncertainty in linear, Bayesian framework. Support is also provided for high-dimensional Monte Carlo analyses via `ObservationEnsemble` and `ParameterEnsemble` class, including the null-space monte carlo approach of Tonkin and Doherty (2009); these ensemble classes also play nicely with PESTPP-IES.
pyEMU also includes lots of functionality for dealing with PEST(++) datasets, such as:
* manipulation of PEST control files, including the use of pandas for sophisticated editing of the parameter data and observation data sections
* creation of PEST control files from instruction and template files
* going between site sample files and pandas dataframes - really cool for observation processing
* easy-to-use observation (re)weighting via residuals or user-defined functions
* handling Jacobian and covariance matrices, including functionality to go between binary and ASCII matrices, reading and writing PEST uncertainty files. Covariance matrices can be instantiated from relevant control file sections, such as parameter bounds or observation weights. The base Matrix class overloads most common linear algebra operators so that operations are automatically aligned by row and column name. Builtin SVD is also included in all Matrix instances.
* geostatistics including geostatistical structure support, reading and writing PEST structure files and creating covariance matrices implied by nested geostatistical structures, and ordinary kriging (in the utils.geostats.OrdinaryKrige object), which replicates the functionality of pest utility ``ppk2fac``.
* composite scaled sensitivity calculations
* calculation of correlation coefficient matrix from a given covariance matrix
* Karhunen-Loeve-based parameterization as an alternative to pilot points for spatially-distributed parameter fields
* a helper functions to start a group of tcp/ip workers on a local machine for parallel PEST++/BeoPEST runs
* full support for prior information equations in control files
* preferred differencing prior information equations where the weights are based on the Pearson correlation coefficient
* verification-based tests based on results from several PEST utilities
Version => 1.1 includes the `PstFrom` setup class to support generating PEST(++) interfaces in the 100,000 to 1,000,000 parameter range with all the bells and whistles. A publication documenting the `PstFrom` class can be found here:
[https://doi.org/10.1016/j.envsoft.2021.105022](https://doi.org/10.1016/j.envsoft.2021.105022)
A publication documenting pyEMU and an example application can be found here:
[http://dx.doi.org/10.1016/j.envsoft.2016.08.017](http://dx.doi.org/10.1016/j.envsoft.2016.08.017)
Funding
=======
pyEMU was originally developed with support from the U.S. Geological Survey. The New Zealand Strategic Science Investment Fund as part of GNS Science’s (https://www.gns.cri.nz/) Groundwater Research Programme has also funded contributions 2018-present. Intera, Inc. has also provided funding for pyEMU development and support
Examples
========
Several example ipython notebooks are provided to demonstrate typical workflows for FOSM parameter and forecast uncertainty analysis as well as techniques to investigate parameter contributions to forecast uncertainty and observation data worth. Example models include the Henry saltwater intrusion problem (Henry 1964) and the model of Freyberg (1988)
There is a whole world of detailed learning material for script-based approaches to parameter estimation and uncertainty quantification using PEST(++) at https://github.com/gmdsi/GMDSI_notebooks. These are and excellent resource for people picking up Pyemu for the first time and for those needing to revisit elements.
Related Links
=============
[PEST++ on GitHub](https://github.com/usgs/pestpp)
[PEST](http://www.pesthomepage.org/)
[Groundwater Modelling Decision Support Initiative](https://gmdsi.org)
How to get started with pyEMU
=============================
pyEMU is available through pyPI and conda. To install pyEMU type:
>>>conda install -c conda-forge pyemu
or
>>>pip install pyemu
pyEMU needs `numpy` and `pandas`. For plotting, `matplotloib`, `pyshp`, and `flopy` to take advantage of the auto interface construction
After pyEMU is installed, the PEST++ software suite can be installed for your operating system using the command:
get-pestpp :pyemu
See [documentation](get_pestpp.md) for more information.
Found a bug? Got a smart idea? Contributions welcome.
====================================================
Feel free to raise and issue or submit a pull request.
pyEMU CI testing, using GitHub actions, has recently been switched over to run with [pytest](https://docs.pytest.org/).
We make use of [pytest-xdist](https://pytest-xdist.readthedocs.io/en/latest/) for parallel execution.
Some notes that might be helpful for building your PR and testing:
* Test files are in [./autotest](https://github.com/pypest/pyemu/tree/develop/autotest)
* Pytest settings are in [./autotest/conftest.py](./autotest/conftest.py) and [./autotest/pytest.ini](./autotest/pytest.ini)
* Currently, files ending `_tests.py` or `_tests_2.py` are collected
* Functions starting `test_` or ending `_test` are collected
* ipython notebooks in [.examples](./examples) are also run
* As tests are run in parallel, where tests require read/write access to files it is safest to sandbox runs.
Pytest has a built-in fixture `tmp_path` that can help with this.
Setting optional argument `--basetemp` can be helpful for accessing the locally run files.
## Running test locally
To be able to make clean use of pytests fixture decorators etc.,
it is recommended to run local tests through `pytest` (rather than use from script execution and commenting in
__main__ block). For e.g.:
### Run all tests:
> pytest --basetemp=runner autotest
with pytest-xdist, local runs can be parallelized:
> pytest --basetemp=runner -n auto autotest
### Run all tests in a file:
> pytest --basetemp=runner -n auto autotest/testfile_tests.py
### Run a specific test [`this_test()`]:
> pytest --basetemp=runner autotest/testfile_tests.py::this_test
### Using an IDE:
Most modern, feature-rich editors and IDEs support launching pytest within debug or run consoles.
Some might need "encouraging" to recognise the non-standard test tags used in this library.
For example, in pycharm, to support click-and-run testing, the
[pytest-imp](https://plugins.jetbrains.com/plugin/14202-pytest-imp) plugin is required to
pickup test functions that end with `_test` (a nosetest hangover in pyEMU).
Raw data
{
"_id": null,
"home_page": null,
"name": "pyemu",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "pest, pestpp",
"author": null,
"author_email": "Jeremy White <jtwhite1000@gmail.com>, Mike Fienen <mnfienen@usgs.gov>, Brioch Hemmings <b.hemmings@gns.cri.nz>",
"download_url": "https://files.pythonhosted.org/packages/b3/ef/cb1ce4564c99c28973c6493d46577bf72985b7fe36714df329a5a52bb356/pyemu-1.3.4.tar.gz",
"platform": null,
"description": "pyEMU\n=====\n\npython modules for model-independent FOSM (first-order, second-moment) (a.k.a linear-based, a.k.a. Bayes linear) \nuncertainty analyses and data-worth analyses, non-linear uncertainty analyses and interfacing with PEST and PEST++. \npyEMU also has a pure python (pandas and numpy) implementation of ordinary kriging for geostatistical interpolation and \nsupport for generating high-dimensional PEST(++) model interfaces, including support for (very) high-dimensional \nensemble generation and handling \n\nMain branch:\n[](https://github.com/pypest/pyemu/actions)\n[](https://coveralls.io/github/pypest/pyemu?branch=main)\n\nDevelop branch:\n[](https://github.com/pypest/pyemu/actions)\n[](https://coveralls.io/github/pypest/pyemu?branch=develop)\n\nDocumentation\n=============\n\nComplete user's guide:\n\n[https://pyemu.readthedocs.io/en/latest/](https://pyemu.readthedocs.io/en/latest/)\n\nThe pyEMU documentation is being treated as a first-class citizen! Also see the example notebooks in the repo.\n\nWhat is pyEMU?\n================\n\npyEMU is a set of python modules for model-independent, user-friendly, computer model uncertainty analysis. pyEMU is tightly coupled to the open-source suite PEST (Doherty 2010a and 2010b, and Doherty and other, 2010) and PEST++ (Welter and others, 2015, Welter and other, 2012), which are tools for model-independent parameter estimation. However, pyEMU can be used with generic array objects, such as numpy ndarrays.\n\nSeveral equations are implemented, including Schur's complement for conditional uncertainty propagation (a.k.a. Bayes Linear estimation) (the foundation of the PREDUNC suite from PEST) and error variance analysis (the foundation of the PREDVAR suite of PEST). pyEMU has easy-to-use routines for parameter and data worth analyses, which estimate how increased parameter knowledge and/or additional data effect forecast uncertainty in linear, Bayesian framework. Support is also provided for high-dimensional Monte Carlo analyses via `ObservationEnsemble` and `ParameterEnsemble` class, including the null-space monte carlo approach of Tonkin and Doherty (2009); these ensemble classes also play nicely with PESTPP-IES.\n\npyEMU also includes lots of functionality for dealing with PEST(++) datasets, such as:\n* manipulation of PEST control files, including the use of pandas for sophisticated editing of the parameter data and observation data sections\n* creation of PEST control files from instruction and template files\n* going between site sample files and pandas dataframes - really cool for observation processing\n* easy-to-use observation (re)weighting via residuals or user-defined functions\n* handling Jacobian and covariance matrices, including functionality to go between binary and ASCII matrices, reading and writing PEST uncertainty files. Covariance matrices can be instantiated from relevant control file sections, such as parameter bounds or observation weights. The base Matrix class overloads most common linear algebra operators so that operations are automatically aligned by row and column name. Builtin SVD is also included in all Matrix instances.\n* geostatistics including geostatistical structure support, reading and writing PEST structure files and creating covariance matrices implied by nested geostatistical structures, and ordinary kriging (in the utils.geostats.OrdinaryKrige object), which replicates the functionality of pest utility ``ppk2fac``. \n* composite scaled sensitivity calculations\n* calculation of correlation coefficient matrix from a given covariance matrix\n* Karhunen-Loeve-based parameterization as an alternative to pilot points for spatially-distributed parameter fields\n* a helper functions to start a group of tcp/ip workers on a local machine for parallel PEST++/BeoPEST runs\n* full support for prior information equations in control files\n* preferred differencing prior information equations where the weights are based on the Pearson correlation coefficient\n* verification-based tests based on results from several PEST utilities\n\nVersion => 1.1 includes the `PstFrom` setup class to support generating PEST(++) interfaces in the 100,000 to 1,000,000 parameter range with all the bells and whistles. A publication documenting the `PstFrom` class can be found here:\n\n[https://doi.org/10.1016/j.envsoft.2021.105022](https://doi.org/10.1016/j.envsoft.2021.105022)\n\nA publication documenting pyEMU and an example application can be found here:\n\n[http://dx.doi.org/10.1016/j.envsoft.2016.08.017](http://dx.doi.org/10.1016/j.envsoft.2016.08.017)\n\n\nFunding\n=======\n\npyEMU was originally developed with support from the U.S. Geological Survey. The New Zealand Strategic Science Investment Fund as part of GNS Science\u2019s (https://www.gns.cri.nz/) Groundwater Research Programme has also funded contributions 2018-present. Intera, Inc. has also provided funding for pyEMU development and support\n\nExamples\n========\n\nSeveral example ipython notebooks are provided to demonstrate typical workflows for FOSM parameter and forecast uncertainty analysis as well as techniques to investigate parameter contributions to forecast uncertainty and observation data worth. Example models include the Henry saltwater intrusion problem (Henry 1964) and the model of Freyberg (1988)\n\nThere is a whole world of detailed learning material for script-based approaches to parameter estimation and uncertainty quantification using PEST(++) at https://github.com/gmdsi/GMDSI_notebooks. These are and excellent resource for people picking up Pyemu for the first time and for those needing to revisit elements.\n\nRelated Links\n=============\n\n[PEST++ on GitHub](https://github.com/usgs/pestpp)\n\n[PEST](http://www.pesthomepage.org/)\n\n[Groundwater Modelling Decision Support Initiative](https://gmdsi.org)\n\n\nHow to get started with pyEMU\n=============================\n\npyEMU is available through pyPI and conda. To install pyEMU type:\n\n >>>conda install -c conda-forge pyemu\n\nor\n\n >>>pip install pyemu\n\npyEMU needs `numpy` and `pandas`. For plotting, `matplotloib`, `pyshp`, and `flopy` to take advantage of the auto interface construction\n\nAfter pyEMU is installed, the PEST++ software suite can be installed for your operating system using the command:\n\n get-pestpp :pyemu\n\nSee [documentation](get_pestpp.md) for more information.\n\nFound a bug? Got a smart idea? Contributions welcome.\n====================================================\nFeel free to raise and issue or submit a pull request.\n\npyEMU CI testing, using GitHub actions, has recently been switched over to run with [pytest](https://docs.pytest.org/).\nWe make use of [pytest-xdist](https://pytest-xdist.readthedocs.io/en/latest/) for parallel execution. \nSome notes that might be helpful for building your PR and testing:\n* Test files are in [./autotest](https://github.com/pypest/pyemu/tree/develop/autotest)\n* Pytest settings are in [./autotest/conftest.py](./autotest/conftest.py) and [./autotest/pytest.ini](./autotest/pytest.ini)\n* Currently, files ending `_tests.py` or `_tests_2.py` are collected\n* Functions starting `test_` or ending `_test` are collected\n* ipython notebooks in [.examples](./examples) are also run\n* As tests are run in parallel, where tests require read/write access to files it is safest to sandbox runs. \nPytest has a built-in fixture `tmp_path` that can help with this. \nSetting optional argument `--basetemp` can be helpful for accessing the locally run files. \n## Running test locally\nTo be able to make clean use of pytests fixture decorators etc., \nit is recommended to run local tests through `pytest` (rather than use from script execution and commenting in \n__main__ block). For e.g.:\n### Run all tests: \n> pytest --basetemp=runner autotest\n\nwith pytest-xdist, local runs can be parallelized:\n> pytest --basetemp=runner -n auto autotest\n\n### Run all tests in a file: \n> pytest --basetemp=runner -n auto autotest/testfile_tests.py\n\n### Run a specific test [`this_test()`]:\n> pytest --basetemp=runner autotest/testfile_tests.py::this_test\n\n### Using an IDE:\nMost modern, feature-rich editors and IDEs support launching pytest within debug or run consoles. \nSome might need \"encouraging\" to recognise the non-standard test tags used in this library. \nFor example, in pycharm, to support click-and-run testing, the\n[pytest-imp](https://plugins.jetbrains.com/plugin/14202-pytest-imp) plugin is required to \npickup test functions that end with `_test` (a nosetest hangover in pyEMU).\n\n",
"bugtrack_url": null,
"license": "BSD 3-Clause",
"summary": "pyEMU is a set of python modules for interfacing with PEST and PEST++",
"version": "1.3.4",
"project_urls": {
"documentation": "https://pyemu.readthedocs.io/",
"repository": "https://github.com/pypest/pyemu"
},
"split_keywords": [
"pest",
" pestpp"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "d0aa7d49339d96a19106f806b5c5bf41534d8309e57e35588f2e2fb286138488",
"md5": "d30f64d859eb4fe041203d1673f65b3f",
"sha256": "9598aa4d3a8e16d5138af195a9543aa4adafe49270599cee5e294ab15c086c18"
},
"downloads": -1,
"filename": "pyemu-1.3.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d30f64d859eb4fe041203d1673f65b3f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 322890,
"upload_time": "2024-03-26T22:34:53",
"upload_time_iso_8601": "2024-03-26T22:34:53.888550Z",
"url": "https://files.pythonhosted.org/packages/d0/aa/7d49339d96a19106f806b5c5bf41534d8309e57e35588f2e2fb286138488/pyemu-1.3.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b3efcb1ce4564c99c28973c6493d46577bf72985b7fe36714df329a5a52bb356",
"md5": "a64a63891869ebf9e097ab8055ce5a8f",
"sha256": "080dfc264836c083e1722bfd0e9f63c9fc65844ffe40e0274f01e738f2f6cffd"
},
"downloads": -1,
"filename": "pyemu-1.3.4.tar.gz",
"has_sig": false,
"md5_digest": "a64a63891869ebf9e097ab8055ce5a8f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 332094,
"upload_time": "2024-03-26T22:34:55",
"upload_time_iso_8601": "2024-03-26T22:34:55.925316Z",
"url": "https://files.pythonhosted.org/packages/b3/ef/cb1ce4564c99c28973c6493d46577bf72985b7fe36714df329a5a52bb356/pyemu-1.3.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-26 22:34:55",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "pypest",
"github_project": "pyemu",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pyemu"
}
