# GN2PG Client
This project provides an import data from [GeoNature] instances to a PostgreSQL database (client side).
Widely inspired from [ClientApiVN](https://framagit.org/lpo/Client_API_VN/)
> [!WARNING]
> The minimum version of the source GeoNature instance required for the incremental update must be a version 2.12.0
## Project Setup
GN2PG Client can be installed by running `pip`. It requires Python 3.7.4 or above to run.
```bash
pip install gn2pg-client
```
## Issues
Please report any bugs or requests that you have using the [GitHub issue tracker](https://github.com/lpoaura/gn2pg_client/issues)!
## HowTo
### Help
```bash
gn2pg_cli --help
```
### Init config file
This command init a TOML config file within `~/.gn2pg` hidden directory (in user `HOME` directory), named as you want. PLEASE DO NOT SPECIFY PATH!
```bash
gn2pg_cli --init <myconfigfile>
```
Config file is structured as this. `[[source]]` block can be duplicate as many as needed (one block for each source).
The `data_type` value on each source is used to characterize the type of data. This value is used to identify which triggers to be triggered when inserting, updating or deleting data.
Current provided trigger configs are:
- `synthese_with_cd_nomenclature` which provide triggers to insert basically data on synthese and generate basic metadatas (acquisition framework and datasets). Source query sample is provided in file [geonature_export_sinp_with_cd_nomenclature.sql](https://github.com/lpoaura/GN2PG/tree/main/data/source_samples/geonature_export_sinp_with_cd_nomenclature.sql)
- `synthese_with_metadata` which provide triggers to insert data in synthese and populate most of the metadata data (acquisition frameworks, datasets, actors such as organisms and roles, territories, etc.). Source query sample is provided in file [geonature_export_sinp_with_metadata.sql](https://github.com/lpoaura/GN2PG/tree/main/data/source_samples/geonature_export_sinp_with_metadata.sql)
```TOML
# GN2PG configuration file
# Local db configuration
[db]
db_host = "localhost"
db_port = 5432
db_user = "<dbUser>"
db_password = "<dbPassword>"
db_name = "<dbName>"
db_schema_import = "schema"
# Additional connection options (optional)
[db.db_querystring]
sslmode = "prefer"
# Source configuration,
# Ducplicate this block for each source (1 source = 1 export)
[[source]]
# Source name, will be use to tag stored data in import table
name = "Source1"
# GeoNature source login
user_name = "<monuser>"
# GeoNature source password
user_password = "<monPwd>"
# GeoNature source URL
url = "<http://geonature1/>"
# GeoNature source Export id
export_id = 1
# Data type is facultative. By default the value is 'synthese'. Therefore, triggers from to_gnsynthese.sql are not activated.
# If you want to insert your data into a GeoNature database please choose either 'synthese_with_cd_nomenclature' or 'synthese_with_metadata'.
# If not, delete the line.
data_type = "synthese_with_cd_nomenclature"
[[source]]
# Source configuration
name = "Source2"
user_name = "<monuser>"
user_password = "<monPwd>"
url = "<http://geonature2/>"
export_id = 1
data_type = "synthese_with_cd_nomenclature"
```
> [!TIP]
> You can add variable in source block `enable = false` to disable a source
### InitDB Schema and tables
To create json tables where datas will be stored, run :
```bash
gn2pg_cli --json-tables-create <myconfigfile>
```
### Full download
To download all datas from API, run :
```bash
gn2pg_cli --full <myconfigfile>
```
### Incremental download
To update data since last download, run :
```bash
gn2pg_cli --update <myconfigfile>
```
To automate the launching of updates, you can write the cron task using the following command, for example every 30 minutes.
```cron
*/30 * * * * /usr/bin/env bash -c "source <path to python environment>/bin/activate && gn2pg_cli --update <myconfigfile>" > /dev/null 2>&1
```
### Debug mode
Debug mode can be activated using `--verbose` CLI argument
### Logs
Log files are stored in `$HOME/.gn2pg/log` directory.
### Import datas into GeoNature database
Default script to auto populate GeoNature is called "to_gnsynthese".
```bash
gn2pg_cli --custom-script to_gnsynthese <myconfigfile>
```
> [!TIP]
>You can also replacing synthese script by your own scripts, using file path instead of `to_gnsynthese`.
## Dashboard
A simple web dashboard can be run following [dashboard docs](./docs/dashboard.rst).
## Contributing
All devs must be done in forks (see [GitHub doc](https://docs.github.com/en/get-started/quickstart/fork-a-repo)).
Pull requests must be pulled to `dev` branch.
Install project and development requirements (require [poetry](https://python-poetry.org/)):
```bash
poetry install --with=docs --all-extras
poetry run pre-commit install
```
Make your devs and pull requests.
Test `gn2pg_cli` in dev mode by running this command:
```bash
poetry run gn2pg_cli <options>
```
## Licence
[GNU AGPLv3](https://www.gnu.org/licenses/gpl.html)
## Team
- [@lpofredc](https://github.com/lpofredc/) ([LPO Auvergne-Rhône-Alpes](https://github.com/lpoaura/)), main developer
- @ophdlv (Natural Solution), contributor
- @mvergez (Natural Solution), contributor
- @andriacap (Natural Solution), contributor
- @Adrien-Pajot (Natural Solution), contributor
______________________________________________________________________
With the financial support of the [DREAL Auvergne-Rhône-Alpes](http://www.auvergne-rhone-alpes.developpement-durable.gouv.fr/) and the [Office français de la biodiversité](https://www.ofb.gouv.fr/).
<img height="100px" src="https://data.lpo-aura.org/web/images/blocmarque_pref_region_auvergne_rhone_alpes_rvb_web.png" title="DREAL AuRA"> <img height="100px" src="https://www.ofb.gouv.fr/sites/default/files/logo-ofb.png" title="OFB" style="padding-left:10px;">
[geonature]: https://geonature.fr
Raw data
{
"_id": null,
"home_page": "https://github.com/lpoaura/gn2gn_client/",
"name": "gn2pg_client",
"maintainer": "lpofredc",
"docs_url": null,
"requires_python": "<4.0,>=3.8.1",
"maintainer_email": "frederic.cloitre@lpo.fr",
"keywords": "GeoNature, Export, SINP, opendata, biodiversity",
"author": "lpofredc",
"author_email": "frederic.cloitre@lpo.fr",
"download_url": "https://files.pythonhosted.org/packages/04/5a/f36953a5273b1773ab31d354a927a9daf7b1c25b8b7bec891111cab5ee39/gn2pg_client-1.6.9.tar.gz",
"platform": null,
"description": "# GN2PG Client\n\n\n\n\n\n\n\nThis project provides an import data from [GeoNature] instances to a PostgreSQL database (client side).\nWidely inspired from [ClientApiVN](https://framagit.org/lpo/Client_API_VN/)\n\n> [!WARNING]\n> The minimum version of the source GeoNature instance required for the incremental update must be a version 2.12.0\n\n\n\n## Project Setup\n\nGN2PG Client can be installed by running `pip`. It requires Python 3.7.4 or above to run.\n\n```bash\npip install gn2pg-client\n```\n\n## Issues\n\nPlease report any bugs or requests that you have using the [GitHub issue tracker](https://github.com/lpoaura/gn2pg_client/issues)!\n\n## HowTo\n\n### Help\n\n```bash\ngn2pg_cli --help\n```\n\n### Init config file\n\nThis command init a TOML config file within `~/.gn2pg` hidden directory (in user `HOME` directory), named as you want. PLEASE DO NOT SPECIFY PATH!\n\n```bash\ngn2pg_cli --init <myconfigfile>\n```\n\nConfig file is structured as this. `[[source]]` block can be duplicate as many as needed (one block for each source).\n\nThe `data_type` value on each source is used to characterize the type of data. This value is used to identify which triggers to be triggered when inserting, updating or deleting data.\nCurrent provided trigger configs are:\n\n- `synthese_with_cd_nomenclature` which provide triggers to insert basically data on synthese and generate basic metadatas (acquisition framework and datasets). Source query sample is provided in file [geonature_export_sinp_with_cd_nomenclature.sql](https://github.com/lpoaura/GN2PG/tree/main/data/source_samples/geonature_export_sinp_with_cd_nomenclature.sql)\n- `synthese_with_metadata` which provide triggers to insert data in synthese and populate most of the metadata data (acquisition frameworks, datasets, actors such as organisms and roles, territories, etc.). Source query sample is provided in file [geonature_export_sinp_with_metadata.sql](https://github.com/lpoaura/GN2PG/tree/main/data/source_samples/geonature_export_sinp_with_metadata.sql)\n\n```TOML\n# GN2PG configuration file\n\n# Local db configuration\n[db]\ndb_host = \"localhost\"\ndb_port = 5432\ndb_user = \"<dbUser>\"\ndb_password = \"<dbPassword>\"\ndb_name = \"<dbName>\"\ndb_schema_import = \"schema\"\n # Additional connection options (optional)\n [db.db_querystring]\n sslmode = \"prefer\"\n\n\n# Source configuration,\n# Ducplicate this block for each source (1 source = 1 export)\n[[source]]\n# Source name, will be use to tag stored data in import table\nname = \"Source1\"\n# GeoNature source login\nuser_name = \"<monuser>\"\n# GeoNature source password\nuser_password = \"<monPwd>\"\n# GeoNature source URL\nurl = \"<http://geonature1/>\"\n# GeoNature source Export id\nexport_id = 1\n# Data type is facultative. By default the value is 'synthese'. Therefore, triggers from to_gnsynthese.sql are not activated.\n# If you want to insert your data into a GeoNature database please choose either 'synthese_with_cd_nomenclature' or 'synthese_with_metadata'.\n# If not, delete the line.\ndata_type = \"synthese_with_cd_nomenclature\"\n\n\n[[source]]\n# Source configuration\nname = \"Source2\"\nuser_name = \"<monuser>\"\nuser_password = \"<monPwd>\"\nurl = \"<http://geonature2/>\"\nexport_id = 1\ndata_type = \"synthese_with_cd_nomenclature\"\n```\n\n> [!TIP]\n> You can add variable in source block `enable = false` to disable a source\n\n### InitDB Schema and tables\n\nTo create json tables where datas will be stored, run :\n\n```bash\ngn2pg_cli --json-tables-create <myconfigfile>\n```\n\n### Full download\n\nTo download all datas from API, run :\n\n```bash\ngn2pg_cli --full <myconfigfile>\n```\n\n### Incremental download\n\nTo update data since last download, run :\n\n```bash\ngn2pg_cli --update <myconfigfile>\n```\n\nTo automate the launching of updates, you can write the cron task using the following command, for example every 30 minutes.\n\n```cron\n*/30 * * * * /usr/bin/env bash -c \"source <path to python environment>/bin/activate && gn2pg_cli --update <myconfigfile>\" > /dev/null 2>&1\n```\n\n### Debug mode\n\nDebug mode can be activated using `--verbose` CLI argument\n\n### Logs\n\nLog files are stored in `$HOME/.gn2pg/log` directory.\n\n### Import datas into GeoNature database\n\nDefault script to auto populate GeoNature is called \"to_gnsynthese\".\n\n```bash\ngn2pg_cli --custom-script to_gnsynthese <myconfigfile>\n```\n\n> [!TIP]\n>You can also replacing synthese script by your own scripts, using file path instead of `to_gnsynthese`.\n\n## Dashboard\n\nA simple web dashboard can be run following [dashboard docs](./docs/dashboard.rst).\n\n\n\n\n\n## Contributing\n\nAll devs must be done in forks (see [GitHub doc](https://docs.github.com/en/get-started/quickstart/fork-a-repo)).\n\nPull requests must be pulled to `dev` branch.\n\nInstall project and development requirements (require [poetry](https://python-poetry.org/)):\n\n```bash\npoetry install --with=docs --all-extras\npoetry run pre-commit install\n```\n\nMake your devs and pull requests.\n\nTest `gn2pg_cli` in dev mode by running this command:\n\n```bash\npoetry run gn2pg_cli <options>\n```\n\n## Licence\n\n[GNU AGPLv3](https://www.gnu.org/licenses/gpl.html)\n\n## Team\n\n- [@lpofredc](https://github.com/lpofredc/) ([LPO Auvergne-Rh\u00f4ne-Alpes](https://github.com/lpoaura/)), main developer\n\n\n\n- @ophdlv (Natural Solution), contributor\n- @mvergez (Natural Solution), contributor\n- @andriacap (Natural Solution), contributor\n- @Adrien-Pajot (Natural Solution), contributor\n\n______________________________________________________________________\n\nWith the financial support of the [DREAL Auvergne-Rh\u00f4ne-Alpes](http://www.auvergne-rhone-alpes.developpement-durable.gouv.fr/) and the [Office fran\u00e7ais de la biodiversit\u00e9](https://www.ofb.gouv.fr/).\n\n\n<img height=\"100px\" src=\"https://data.lpo-aura.org/web/images/blocmarque_pref_region_auvergne_rhone_alpes_rvb_web.png\" title=\"DREAL AuRA\"> <img height=\"100px\" src=\"https://www.ofb.gouv.fr/sites/default/files/logo-ofb.png\" title=\"OFB\" style=\"padding-left:10px;\">\n\n[geonature]: https://geonature.fr\n",
"bugtrack_url": null,
"license": "AGPL-3.0-or-later",
"summary": "Import tool from GeoNature to a PostgreSQL database through Export module API (client side)",
"version": "1.6.9",
"project_urls": {
"Homepage": "https://github.com/lpoaura/gn2gn_client/"
},
"split_keywords": [
"geonature",
" export",
" sinp",
" opendata",
" biodiversity"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "27ffb5f18a18c04cbcdfc40a275ca2de288781d3d0e63a16949beb1d72de39a3",
"md5": "99642a2a23c6a977cd8201bbc159a1e5",
"sha256": "2e031f4c792f06f9f7b25e220b96d10219524f68a267c8f9404f12fa0a45713a"
},
"downloads": -1,
"filename": "gn2pg_client-1.6.9-py3-none-any.whl",
"has_sig": false,
"md5_digest": "99642a2a23c6a977cd8201bbc159a1e5",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.8.1",
"size": 86708,
"upload_time": "2024-10-08T08:57:32",
"upload_time_iso_8601": "2024-10-08T08:57:32.661077Z",
"url": "https://files.pythonhosted.org/packages/27/ff/b5f18a18c04cbcdfc40a275ca2de288781d3d0e63a16949beb1d72de39a3/gn2pg_client-1.6.9-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "045af36953a5273b1773ab31d354a927a9daf7b1c25b8b7bec891111cab5ee39",
"md5": "95fb2a21cf33afd0f71b3f4b4ee6ca16",
"sha256": "83cc8d1f9cf4c8be3ba82d65a359c1010e6ea198308d48093b6e15ecad40f607"
},
"downloads": -1,
"filename": "gn2pg_client-1.6.9.tar.gz",
"has_sig": false,
"md5_digest": "95fb2a21cf33afd0f71b3f4b4ee6ca16",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.8.1",
"size": 81098,
"upload_time": "2024-10-08T08:57:34",
"upload_time_iso_8601": "2024-10-08T08:57:34.553947Z",
"url": "https://files.pythonhosted.org/packages/04/5a/f36953a5273b1773ab31d354a927a9daf7b1c25b8b7bec891111cab5ee39/gn2pg_client-1.6.9.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-08 08:57:34",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "lpoaura",
"github_project": "gn2gn_client",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "gn2pg_client"
}
JSON
