# pgmerge - a PostgreSQL data import and merge utility
[](https://github.com/samuller/pgmerge/actions)
[](https://badge.fury.io/py/pgmerge)
[](https://github.com/samuller/pgmerge/actions)
[](http://mypy-lang.org/)
This utility's main purpose is to manage a set of CSV files that correspond with tables in a PostgreSQL database:
* Each of these CSV file's data can then be *merged* into their corresponding table (see section below).
* The database schema is analysed to determine dependencies among tables (due to foreign keys), and CSV files are then imported in the correct order such that the data/tables that they might depend on have been imported first.
* `pgmerge` can then also export data in the same format expected for import.
These features allow you to move data between databases with the same schema to keep them up to date and in sync, although it does not cover handling deleted data.
### Import merging
Merging CSVs into a table means that the following process will occur (also called an *upsert* operation):
* Rows whose primary key don't yet exist in the table will be imported.
* When the primary key already exists, row values will be updated.
* Rows that are missing or unchanged will be ignored.
## CLI arguments
$ pgmerge --help
Usage: pgmerge [OPTIONS] COMMAND [ARGS]...
Merges data in CSV files into a Postgresql database.
Options:
--version Show the version and exit.
--help Show this message and exit.
Commands:
export Export each table to a CSV file.
import Import/merge each CSV file into a table.
inspect Inspect database schema in various ways.
### Import
$ pgmerge import --help
Usage: pgmerge import [OPTIONS] DIRECTORY [TABLES]...
Import/merge each CSV file into a table.
All CSV files need the same name as their matching table and have to be located
in the given directory. If one or more tables are specified then only they will
be used, otherwise all tables found will be selected.
Options:
-d, --dbname TEXT Database name to connect to. [required]
-h, --host TEXT Database server host or socket directory.
[default: localhost]
-p, --port TEXT Database server port. [default: 5432]
-U, --username TEXT Database user name. [default: postgres]
-s, --schema TEXT Database schema to use. [default: public]
-w, --no-password Never prompt for password (e.g. peer
authentication).
-W, --password TEXT Database password (default is to prompt for
password or read config).
-L, --uri TEXT Connection URI can be used instead of specifying
parameters separately (also sets --no-password).
-f, --ignore-cycles Don't stop import when cycles are detected in
schema (will still fail if there are cycles in
data)
-F, --disable-foreign-keys Disable foreign key constraint checking during
import (necessary if you have cycles, but requires
superuser rights).
-c, --config PATH Config file for customizing how tables are
imported/exported.
-i, --include-dependent-tables When selecting specific tables, also include all
tables on which they depend due to foreign key
constraints.
--help Show this message and exit.
## Installation
> WARNING: the reliability of this utility is not guaranteed and loss or corruption of data is always a possibility.
### Install from PyPI
With `Python 3` installed on your system, you can run:
pip install pgmerge
To test that installation worked, run:
pgmerge --help
and you can uninstall at any time with:
pip uninstall pgmerge
### Install from Github
To install the newest code directly from Github:
pip install git+https://github.com/samuller/pgmerge
### Issues
If you have trouble installing and you're running a Debian-based Linux that uses `Python 2` as its system default, then you might need to run:
sudo apt install libpq-dev python3-pip python3-setuptools
sudo -H pip3 install pgmerge
Raw data
{
"_id": null,
"home_page": "https://github.com/samuller/pgmerge",
"name": "pgmerge",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.7",
"maintainer_email": null,
"keywords": "postgresql, csv, import, export",
"author": "Simon Muller",
"author_email": "samullers@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/aa/7f/cb7623b7981b0229572baeaa7c93a50ed0695883fabfd944d008302bbb75/pgmerge-1.13.0.tar.gz",
"platform": null,
"description": "# pgmerge - a PostgreSQL data import and merge utility\n\n[](https://github.com/samuller/pgmerge/actions)\n[](https://badge.fury.io/py/pgmerge)\n[](https://github.com/samuller/pgmerge/actions)\n[](http://mypy-lang.org/)\n\nThis utility's main purpose is to manage a set of CSV files that correspond with tables in a PostgreSQL database:\n\n* Each of these CSV file's data can then be *merged* into their corresponding table (see section below).\n* The database schema is analysed to determine dependencies among tables (due to foreign keys), and CSV files are then imported in the correct order such that the data/tables that they might depend on have been imported first.\n* `pgmerge` can then also export data in the same format expected for import.\n\nThese features allow you to move data between databases with the same schema to keep them up to date and in sync, although it does not cover handling deleted data.\n\n### Import merging\n\nMerging CSVs into a table means that the following process will occur (also called an *upsert* operation):\n\n* Rows whose primary key don't yet exist in the table will be imported.\n* When the primary key already exists, row values will be updated.\n* Rows that are missing or unchanged will be ignored.\n\n## CLI arguments\n\n $ pgmerge --help\n Usage: pgmerge [OPTIONS] COMMAND [ARGS]...\n\n Merges data in CSV files into a Postgresql database.\n\n Options:\n --version Show the version and exit.\n --help Show this message and exit.\n\n Commands:\n export Export each table to a CSV file.\n import Import/merge each CSV file into a table.\n inspect Inspect database schema in various ways.\n\n### Import\n\n $ pgmerge import --help\n Usage: pgmerge import [OPTIONS] DIRECTORY [TABLES]...\n\n Import/merge each CSV file into a table.\n\n All CSV files need the same name as their matching table and have to be located\n in the given directory. If one or more tables are specified then only they will\n be used, otherwise all tables found will be selected.\n\n Options:\n -d, --dbname TEXT Database name to connect to. [required]\n -h, --host TEXT Database server host or socket directory.\n [default: localhost]\n -p, --port TEXT Database server port. [default: 5432]\n -U, --username TEXT Database user name. [default: postgres]\n -s, --schema TEXT Database schema to use. [default: public]\n -w, --no-password Never prompt for password (e.g. peer\n authentication).\n -W, --password TEXT Database password (default is to prompt for\n password or read config).\n -L, --uri TEXT Connection URI can be used instead of specifying\n parameters separately (also sets --no-password).\n -f, --ignore-cycles Don't stop import when cycles are detected in\n schema (will still fail if there are cycles in\n data)\n -F, --disable-foreign-keys Disable foreign key constraint checking during\n import (necessary if you have cycles, but requires\n superuser rights).\n -c, --config PATH Config file for customizing how tables are\n imported/exported.\n -i, --include-dependent-tables When selecting specific tables, also include all\n tables on which they depend due to foreign key\n constraints.\n --help Show this message and exit.\n\n## Installation\n\n> WARNING: the reliability of this utility is not guaranteed and loss or corruption of data is always a possibility.\n\n### Install from PyPI\n\nWith `Python 3` installed on your system, you can run:\n\n pip install pgmerge\n\nTo test that installation worked, run:\n\n pgmerge --help\n\nand you can uninstall at any time with:\n\n pip uninstall pgmerge\n\n### Install from Github\n\nTo install the newest code directly from Github:\n\n pip install git+https://github.com/samuller/pgmerge\n\n### Issues\n\nIf you have trouble installing and you're running a Debian-based Linux that uses `Python 2` as its system default, then you might need to run:\n\n sudo apt install libpq-dev python3-pip python3-setuptools\n sudo -H pip3 install pgmerge\n",
"bugtrack_url": null,
"license": null,
"summary": "PostgreSQL data import/export utility",
"version": "1.13.0",
"project_urls": {
"Homepage": "https://github.com/samuller/pgmerge",
"Repository": "https://github.com/samuller/pgmerge"
},
"split_keywords": [
"postgresql",
" csv",
" import",
" export"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8fe67c74c03b4b8945bb9bf0da3280bb36a92ea9828dfa1e25eb4dc01eadb70b",
"md5": "1857ce5d120f6453648a4dea6288f254",
"sha256": "4eac4005f3da07f7b860ad41588edc70d8770e6ed78fc10b5ae44787d5e6035a"
},
"downloads": -1,
"filename": "pgmerge-1.13.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "1857ce5d120f6453648a4dea6288f254",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.7",
"size": 35998,
"upload_time": "2024-06-08T14:48:27",
"upload_time_iso_8601": "2024-06-08T14:48:27.970515Z",
"url": "https://files.pythonhosted.org/packages/8f/e6/7c74c03b4b8945bb9bf0da3280bb36a92ea9828dfa1e25eb4dc01eadb70b/pgmerge-1.13.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "aa7fcb7623b7981b0229572baeaa7c93a50ed0695883fabfd944d008302bbb75",
"md5": "195b53832605249d75ebad991c66c418",
"sha256": "1d8d8253f9f26d64dde23ff68661b435d840c4e8ee36f7a019cb1daf7ad0e463"
},
"downloads": -1,
"filename": "pgmerge-1.13.0.tar.gz",
"has_sig": false,
"md5_digest": "195b53832605249d75ebad991c66c418",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.7",
"size": 31901,
"upload_time": "2024-06-08T14:48:30",
"upload_time_iso_8601": "2024-06-08T14:48:30.326813Z",
"url": "https://files.pythonhosted.org/packages/aa/7f/cb7623b7981b0229572baeaa7c93a50ed0695883fabfd944d008302bbb75/pgmerge-1.13.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-06-08 14:48:30",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "samuller",
"github_project": "pgmerge",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pgmerge"
}
JSON
