# Parallel.GAMIT
# Author: Demián D. Gómez
A Python wrapper to parallelize GAMIT executions.
Parallel.GAMIT is a Python software solution for parallel GPS processing of large regional or global networks. It also incorporates a metadata and RINEX data management tool that guarantees a consistent archive. It relies on Postgres SQL (https://www.postgresql.org/) to store station metadata and the GPSPACE Precise-Point-Positioning (PPP) software (not included in this repository, available here: https://github.com/CGS-GIS/GPSPACE) to obtain reliable daily a-priori coordinates for GAMIT.
The software is divided into two modules: Parallel.GAMIT (PG) and Parallel.PPP (PP). PG requires all GAMIT-GLOBK (http://www-gpsg.mit.edu/~simon/gtgk/) dependencies installed in the processing nodes. PP requires GPSPACE PPP and several other dependencies detailed later in this document. Although PP was designed to use GPSPACE PPP, it can be easily changed to use any other open source PPP software such as RTKLIB (http://www.rtklib.com/), although this has not been tested.
PG uses dispy (https://github.com/pgiri/dispy) to create Python pickles that are sent to local or remote nodes for execution. PG has the ability to split a network of GPS stations into subnetworks for processing in GAMIT (when the network is larger than 50 stations, depending on PG's configuration). The parallel execution is performed per day-subnetwork. In other words, a GAMIT pickle is built for each subnetwork-day being processed and sent to the available nodes. At the end of each PG run, the subnetworks are combined with GLOBK and inserted as records in the Postgres database for later use. Some routines (such as the SINEX parser) are modified versions of the code from @softwarespartan (https://github.com/softwarespartan).
PP is a Python wrapper for the PGSPACE PPP which uses the same Postgres SQL database to store the daily PPP solutions and medatadata of all station-days in the GPS archive. Some of the abilities of PP are:
- Scan a directory structure containing RINEX files and add them to the Postgres database (DB).
- Manage station metadata in GAMIT's station info format with consistency check of the records.
- Add new RINEX data to the database by geolocation, i.e. the data is incorporated not by station name but by running PPP and finding the corresponding station in the DB. This avoids problems with duplicate station codes and misidentified RINEX files.
- Handle ocean loading coefficients to correct the PPP coordinates and produce consistent time series before running GAMIT. This allows to find problems in the metadata BEFORE executing a long GAMIT run.
- Plot PPP time series using Bevis and Brown's (2014) extended trajectory model.
- Merge stations with different names that in reality are the same station (but renamed or moved a couple of meters), if desired.
- Merge, delete and add metadata directly from GAMIT station info files or using UNAVCO's GSAC (https://www.unavco.org/software/data-management/gsac/user-info/user-info.html).
- Parse all ZTD results and store them in the database.
- Stack the GAMIT solution to produce regional or global reference frames following Bevis and Brown's (2014).
- Both PP and PG tolerate station name duplicates by using a three-letter network code. Although this is not supported by GAMIT, PG converts duplicate station codes (stored in different networks) to unique IDs that are used during processing, which are later converted back to the original names after the GLOBK combination of the subnetworks.
- Because all the information is stored in a relational database, PP and PG can handle very large datasets very easily (it has been tested with ~ 5,600,000 station-days but Postgres can easily handle more than 10 million records in a regular computer). Also, the relational database guarantees then consistency of the data and does not allow accidental duplicates in metadata.
PG and PP require the following dependencies:
- Python version > 3
- GAMIT-GLOBK: although PP does not use GAMIT to process data, it relies on grdtab, otl.grid and sh_rx2apr to obtain the ocean loading coefficients and station coordinates (when PPP fails to process a station-day). Bare in mind that sh_rx2apr needs the following dependencies to run in a computer without GAMIT installed: svdiff, svpos, tform, sh_rx2apr, doy
- gfzrnx: RINEX quality check and conversion tool which supports RINEX 3.
- pygressql: Python interface to connect to Postgres
- tqdm: a Python progress bar to show the processing progress
- rnx2crx: RINEX to CRINEX
- crx2rnx: CRINEX to RINEX
- crz2rnx: this is a script modified by me which is based on the the scripts found in http://terras.gsi.go.jp/ja/crx2rnx.html with a few minor tweaks to handle the most common problems found in CRINEZ files.
- rnx2crz: the regular C-shell script
- compress/gzip
- dispy to schedule parallel jobs
- matplotlib
- numpy
- scandir
- Neicio: the USGS NEIC Python interface and its dependencies found in https://github.com/usgs/neicio
BSD 3-Clause License
Copyright (c) 2024, Demian Gomez
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Raw data
{
"_id": null,
"home_page": "https://github.com/demiangomez/Parallel.GAMIT",
"name": "pgamit",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "GAMIT",
"author": "Demi\u00e1n D. G\u00f3mez",
"author_email": "demiang@gmail.com",
"download_url": null,
"platform": null,
"description": "# Parallel.GAMIT\n# Author: Demi\u00e1n D. G\u00f3mez\nA Python wrapper to parallelize GAMIT executions.\n\nParallel.GAMIT is a Python software solution for parallel GPS processing of large regional or global networks. It also incorporates a metadata and RINEX data management tool that guarantees a consistent archive. It relies on Postgres SQL (https://www.postgresql.org/) to store station metadata and the GPSPACE Precise-Point-Positioning (PPP) software (not included in this repository, available here: https://github.com/CGS-GIS/GPSPACE) to obtain reliable daily a-priori coordinates for GAMIT.\n\nThe software is divided into two modules: Parallel.GAMIT (PG) and Parallel.PPP (PP). PG requires all GAMIT-GLOBK (http://www-gpsg.mit.edu/~simon/gtgk/) dependencies installed in the processing nodes. PP requires GPSPACE PPP and several other dependencies detailed later in this document. Although PP was designed to use GPSPACE PPP, it can be easily changed to use any other open source PPP software such as RTKLIB (http://www.rtklib.com/), although this has not been tested.\n\nPG uses dispy (https://github.com/pgiri/dispy) to create Python pickles that are sent to local or remote nodes for execution. PG has the ability to split a network of GPS stations into subnetworks for processing in GAMIT (when the network is larger than 50 stations, depending on PG's configuration). The parallel execution is performed per day-subnetwork. In other words, a GAMIT pickle is built for each subnetwork-day being processed and sent to the available nodes. At the end of each PG run, the subnetworks are combined with GLOBK and inserted as records in the Postgres database for later use. Some routines (such as the SINEX parser) are modified versions of the code from @softwarespartan (https://github.com/softwarespartan).\n\nPP is a Python wrapper for the PGSPACE PPP which uses the same Postgres SQL database to store the daily PPP solutions and medatadata of all station-days in the GPS archive. Some of the abilities of PP are:\n\n- Scan a directory structure containing RINEX files and add them to the Postgres database (DB).\n- Manage station metadata in GAMIT's station info format with consistency check of the records.\n- Add new RINEX data to the database by geolocation, i.e. the data is incorporated not by station name but by running PPP and finding the corresponding station in the DB. This avoids problems with duplicate station codes and misidentified RINEX files.\n- Handle ocean loading coefficients to correct the PPP coordinates and produce consistent time series before running GAMIT. This allows to find problems in the metadata BEFORE executing a long GAMIT run.\n- Plot PPP time series using Bevis and Brown's (2014) extended trajectory model.\n- Merge stations with different names that in reality are the same station (but renamed or moved a couple of meters), if desired.\n- Merge, delete and add metadata directly from GAMIT station info files or using UNAVCO's GSAC (https://www.unavco.org/software/data-management/gsac/user-info/user-info.html).\n- Parse all ZTD results and store them in the database.\n- Stack the GAMIT solution to produce regional or global reference frames following Bevis and Brown's (2014).\n- Both PP and PG tolerate station name duplicates by using a three-letter network code. Although this is not supported by GAMIT, PG converts duplicate station codes (stored in different networks) to unique IDs that are used during processing, which are later converted back to the original names after the GLOBK combination of the subnetworks.\n- Because all the information is stored in a relational database, PP and PG can handle very large datasets very easily (it has been tested with ~ 5,600,000 station-days but Postgres can easily handle more than 10 million records in a regular computer). Also, the relational database guarantees then consistency of the data and does not allow accidental duplicates in metadata.\n\nPG and PP require the following dependencies:\n\n- Python version > 3\n- GAMIT-GLOBK: although PP does not use GAMIT to process data, it relies on grdtab, otl.grid and sh_rx2apr to obtain the ocean loading coefficients and station coordinates (when PPP fails to process a station-day). Bare in mind that sh_rx2apr needs the following dependencies to run in a computer without GAMIT installed: svdiff, svpos, tform, sh_rx2apr, doy\n- gfzrnx: RINEX quality check and conversion tool which supports RINEX 3.\n- pygressql: Python interface to connect to Postgres\n- tqdm: a Python progress bar to show the processing progress\n- rnx2crx: RINEX to CRINEX\n- crx2rnx: CRINEX to RINEX\n- crz2rnx: this is a script modified by me which is based on the the scripts found in http://terras.gsi.go.jp/ja/crx2rnx.html with a few minor tweaks to handle the most common problems found in CRINEZ files.\n- rnx2crz: the regular C-shell script\n- compress/gzip\n- dispy to schedule parallel jobs\n- matplotlib\n- numpy\n- scandir\n- Neicio: the USGS NEIC Python interface and its dependencies found in https://github.com/usgs/neicio\n\n\nBSD 3-Clause License\n\nCopyright (c) 2024, Demian Gomez\n\nRedistribution and use in source and binary forms, with or without\nmodification, are permitted provided that the following conditions are met:\n\n1. Redistributions of source code must retain the above copyright notice, this\n list of conditions and the following disclaimer.\n\n2. Redistributions in binary form must reproduce the above copyright notice,\n this list of conditions and the following disclaimer in the documentation\n and/or other materials provided with the distribution.\n\n3. Neither the name of the copyright holder nor the names of its\n contributors may be used to endorse or promote products derived from\n this software without specific prior written permission.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\nAND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\nIMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\nDISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE\nFOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL\nDAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR\nSERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\nCAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,\nOR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\nOF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n",
"bugtrack_url": null,
"license": "BSD 3-Clause License",
"summary": "A Python wrapper to parallelize GAMIT executions",
"version": "1.2.1",
"project_urls": {
"Homepage": "https://github.com/demiangomez/Parallel.GAMIT"
},
"split_keywords": [
"gamit"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "7b79489a48a0bc07b74124411566d646c7328ac7699332f734a600de3de448b3",
"md5": "043b9763460f6a57133d6792ca879d4d",
"sha256": "87aaa8fcc2313fac44d396a4f30834fef909bffafb8bce39dc06e6aa554db2d3"
},
"downloads": -1,
"filename": "pgamit-1.2.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "043b9763460f6a57133d6792ca879d4d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 363223,
"upload_time": "2024-10-03T12:50:10",
"upload_time_iso_8601": "2024-10-03T12:50:10.661129Z",
"url": "https://files.pythonhosted.org/packages/7b/79/489a48a0bc07b74124411566d646c7328ac7699332f734a600de3de448b3/pgamit-1.2.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-03 12:50:10",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "demiangomez",
"github_project": "Parallel.GAMIT",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "pgamit"
}
JSON
