pgsync


Namepgsync JSON
Version 3.1.0 PyPI version JSON
download
home_pagehttps://github.com/toluaina/pgsync
SummaryPostgres to Elasticsearch/OpenSearch sync
upload_time2024-01-03 13:43:54
maintainerTolu Aina
docs_urlNone
authorTolu Aina
requires_python>=3.8.0
licenseMIT
keywords pgsync elasticsearch opensearch postgres change data capture
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage
            # PostgreSQL to Elasticsearch/OpenSearch sync


- [PGSync](https://pgsync.com) is a middleware for syncing data from [Postgres](https://www.postgresql.org) to [Elasticsearch](https://www.elastic.co/products/elastic-stack)/[OpenSearch](https://opensearch.org/) or [OpenSearch](https://opensearch.org/).
- It allows you to keep [Postgres](https://www.postgresql.org) as your source of truth data source and
expose structured denormalized documents in [Elasticsearch](https://www.elastic.co/products/elastic-stack)/[OpenSearch](https://opensearch.org/).


### Requirements

- [Python](https://www.python.org) 3.8+
- [Postgres](https://www.postgresql.org) 9.6+
- [Redis](https://redis.io) 3.1.0
- [Elasticsearch](https://www.elastic.co/products/elastic-stack) 6.3.1+ or [OpenSearch](https://opensearch.org/) 1.3.7+
- [SQlAlchemy](https://www.sqlalchemy.org) 1.3.4+

### Postgres setup
  
  Enable [logical decoding](https://www.postgresql.org/docs/current/logicaldecoding.html) in your 
  Postgres setting.

  - You also need to set up two parameters in your Postgres config postgresql.conf

    ```wal_level = logical```

    ```max_replication_slots = 1```

### Installation

You can install PGSync from [PyPI](https://pypi.org):

    $ pip install pgsync

### Config

Create a schema for the application named e.g **schema.json**

[Example schema](https://github.com/toluaina/pgsync/blob/main/examples/airbnb/schema.json)

Example spec

.. code-block::

    [
        {
            "database": "[database name]",
            "index": "[Elasticsearch or OpenSearch index]",
            "nodes": {
                "table": "[table A]",
                "schema": "[table A schema]",
                "columns": [
                    "column 1 from table A",
                    "column 2 from table A",
                    ... additional columns
                ],
                "children": [
                    {
                        "table": "[table B with relationship to table A]",
                        "schema": "[table B schema]",
                        "columns": [
                          "column 1 from table B",
                          "column 2 from table B",
                          ... additional columns
                        ],
                        "relationship": {
                            "variant": "object",
                            "type": "one_to_many"
                        },
                        ...
                    },
                    {
                        ... additional children
                    }
                ]
            }
        }
    ]

### Environment variables 

Setup environment variables required for the application

    SCHEMA='/path/to/schema.json'

    ELASTICSEARCH_HOST=localhost
    ELASTICSEARCH_PORT=9200

    PG_HOST=localhost
    PG_USER=i-am-root # this must be a postgres superuser or replication user
    PG_PORT=5432
    PG_PASSWORD=*****

    REDIS_HOST=redis
    REDIS_PORT=6379
    REDIS_DB=0
    REDIS_AUTH=*****


### Running

Bootstrap the database (one time only)
  - $ bootstrap --config schema.json

Run pgsync as a daemon
  - $ pgsync --config schema.json --daemon

            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/toluaina/pgsync",
    "name": "pgsync",
    "maintainer": "Tolu Aina",
    "docs_url": null,
    "requires_python": ">=3.8.0",
    "maintainer_email": "tolu@pgsync.com",
    "keywords": "pgsync,elasticsearch,opensearch,postgres,change data capture",
    "author": "Tolu Aina",
    "author_email": "tolu@pgsync.com",
    "download_url": "https://files.pythonhosted.org/packages/67/da/ab3b9c118da84dcd0b67a7dc1b3aac7a51bbb5c7d2be58db709ea32b3d3e/pgsync-3.1.0.tar.gz",
    "platform": null,
    "description": "# PostgreSQL to Elasticsearch/OpenSearch sync\n\n\n- [PGSync](https://pgsync.com) is a middleware for syncing data from [Postgres](https://www.postgresql.org) to [Elasticsearch](https://www.elastic.co/products/elastic-stack)/[OpenSearch](https://opensearch.org/) or [OpenSearch](https://opensearch.org/).\n- It allows you to keep [Postgres](https://www.postgresql.org) as your source of truth data source and\nexpose structured denormalized documents in [Elasticsearch](https://www.elastic.co/products/elastic-stack)/[OpenSearch](https://opensearch.org/).\n\n\n### Requirements\n\n- [Python](https://www.python.org) 3.8+\n- [Postgres](https://www.postgresql.org) 9.6+\n- [Redis](https://redis.io) 3.1.0\n- [Elasticsearch](https://www.elastic.co/products/elastic-stack) 6.3.1+ or [OpenSearch](https://opensearch.org/) 1.3.7+\n- [SQlAlchemy](https://www.sqlalchemy.org) 1.3.4+\n\n### Postgres setup\n  \n  Enable [logical decoding](https://www.postgresql.org/docs/current/logicaldecoding.html) in your \n  Postgres setting.\n\n  - You also need to set up two parameters in your Postgres config postgresql.conf\n\n    ```wal_level = logical```\n\n    ```max_replication_slots = 1```\n\n### Installation\n\nYou can install PGSync from [PyPI](https://pypi.org):\n\n    $ pip install pgsync\n\n### Config\n\nCreate a schema for the application named e.g **schema.json**\n\n[Example schema](https://github.com/toluaina/pgsync/blob/main/examples/airbnb/schema.json)\n\nExample spec\n\n.. code-block::\n\n    [\n        {\n            \"database\": \"[database name]\",\n            \"index\": \"[Elasticsearch or OpenSearch index]\",\n            \"nodes\": {\n                \"table\": \"[table A]\",\n                \"schema\": \"[table A schema]\",\n                \"columns\": [\n                    \"column 1 from table A\",\n                    \"column 2 from table A\",\n                    ... additional columns\n                ],\n                \"children\": [\n                    {\n                        \"table\": \"[table B with relationship to table A]\",\n                        \"schema\": \"[table B schema]\",\n                        \"columns\": [\n                          \"column 1 from table B\",\n                          \"column 2 from table B\",\n                          ... additional columns\n                        ],\n                        \"relationship\": {\n                            \"variant\": \"object\",\n                            \"type\": \"one_to_many\"\n                        },\n                        ...\n                    },\n                    {\n                        ... additional children\n                    }\n                ]\n            }\n        }\n    ]\n\n### Environment variables \n\nSetup environment variables required for the application\n\n    SCHEMA='/path/to/schema.json'\n\n    ELASTICSEARCH_HOST=localhost\n    ELASTICSEARCH_PORT=9200\n\n    PG_HOST=localhost\n    PG_USER=i-am-root # this must be a postgres superuser or replication user\n    PG_PORT=5432\n    PG_PASSWORD=*****\n\n    REDIS_HOST=redis\n    REDIS_PORT=6379\n    REDIS_DB=0\n    REDIS_AUTH=*****\n\n\n### Running\n\nBootstrap the database (one time only)\n  - $ bootstrap --config schema.json\n\nRun pgsync as a daemon\n  - $ pgsync --config schema.json --daemon\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Postgres to Elasticsearch/OpenSearch sync",
    "version": "3.1.0",
    "project_urls": {
        "Bug Reports": "https://github.com/toluaina/pgsync/issues",
        "Documentation": "https://pgsync.com",
        "Funding": "https://github.com/sponsors/toluaina",
        "Homepage": "https://github.com/toluaina/pgsync",
        "Source": "https://github.com/toluaina/pgsync",
        "Web": "https://pgsync.com"
    },
    "split_keywords": [
        "pgsync",
        "elasticsearch",
        "opensearch",
        "postgres",
        "change data capture"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "0006d7f1d995aaa18f5f8abbf246932e7a70fd84a673c48b017e9e76e5484b41",
                "md5": "c27adedf0b3ed3722a4376f51bcff448",
                "sha256": "f6183d51adeb5ac0dc3ab9620d1c21f32afc5cc38b4baf013fa61b157f55976b"
            },
            "downloads": -1,
            "filename": "pgsync-3.1.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "c27adedf0b3ed3722a4376f51bcff448",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.8.0",
            "size": 60272,
            "upload_time": "2024-01-03T13:43:52",
            "upload_time_iso_8601": "2024-01-03T13:43:52.590318Z",
            "url": "https://files.pythonhosted.org/packages/00/06/d7f1d995aaa18f5f8abbf246932e7a70fd84a673c48b017e9e76e5484b41/pgsync-3.1.0-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "67daab3b9c118da84dcd0b67a7dc1b3aac7a51bbb5c7d2be58db709ea32b3d3e",
                "md5": "dae019e447e16eb2a0115a178de02bec",
                "sha256": "50b90bc61cd37f1ff9d45ea5f5b74b64f409b2d9725c1e6a05ea0b71220b40b5"
            },
            "downloads": -1,
            "filename": "pgsync-3.1.0.tar.gz",
            "has_sig": false,
            "md5_digest": "dae019e447e16eb2a0115a178de02bec",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.8.0",
            "size": 112672,
            "upload_time": "2024-01-03T13:43:54",
            "upload_time_iso_8601": "2024-01-03T13:43:54.188149Z",
            "url": "https://files.pythonhosted.org/packages/67/da/ab3b9c118da84dcd0b67a7dc1b3aac7a51bbb5c7d2be58db709ea32b3d3e/pgsync-3.1.0.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-01-03 13:43:54",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "toluaina",
    "github_project": "pgsync",
    "travis_ci": false,
    "coveralls": true,
    "github_actions": true,
    "lcname": "pgsync"
}
        
Elapsed time: 0.17102s