# ⚡ pg2elastic
Enhanced PostgreSQL to Elasticsearch Data Synchronization
### 📚 Description
Welcome to pg2elastic, a fork of the official [pgsync](https://pgsync.com/) package, designed to provide seamless and
efficient data synchronization between PostgreSQL databases and Elasticsearch clusters.
Building upon the solid foundation of pgsync, pg2elastic inherits all of its powerful capabilities and takes them a step
further.
#### Key Features:
* High-Performance Sync: pg2elastic inherits the robust data synchronization engine from pgsync, ensuring lightning-fast
and reliable transfers.
* Real-time Indexing: Seamlessly mirror your PostgreSQL data into Elasticsearch indices, keeping them in sync in
real-time.
* Schema Mapping: Easily define and customize the mapping of PostgreSQL schemas to Elasticsearch indexes, giving you
full control over the data structure.
* Efficient Data Types Handling: pg2elastic effortlessly handles data type conversions, ensuring accurate representation
across platforms.
* Continuous Enhancements: We are committed to actively maintaining and enhancing pg2elastic, incorporating the latest
advancements in both PostgreSQL and Elasticsearch technologies.
* Whether you're working on a data-driven application or performing complex data analysis, pg2elastic empowers you with
a streamlined and feature-rich solution for harmonizing your PostgreSQL and Elasticsearch ecosystems.
---
### 🛠️ Prerequisites
[PGSync Requirements](https://github.com/toluaina/pgsync#requirements)
---
### ✨ Key Enhancements
* [Loguru, Better Logging Module](https://github.com/Delgan/loguru)
---
#### Fixes
The actual numerical value of 1000000 becomes 1e when attempted to be converted to a float, leading to a crash during the conversion process.
* A fix was implemented to change 1e to 1e6, preventing the conversion to float from causing the process to crash.
The process crashes when custom field types are use in the database, and the returned field type is in the format abc.xyz.
* A fix was implemented, and the regular expression for LOGICAL_SLOT_SUFFIX was modified.
The process crashes when a partition notification is received
* A fix was implemented, and partitions are tracked in lib's materialized, lib's trigger is updated to include partition's parent table
Child records that are inserted are not updating the parent document
* A fix was implemented, which synchronizes main document in case a child record is created
---
#### Environment Variables
`PG_SCHEMA`
* Environment variable to enhance performance by eliminating the need to scan all schemas
`REDIS_USERNAME`
* Environment variable to specify redis username
`REDIS_PASSWORD`
* Environment variable to specify redis password
`REDIS_ENDPOINT`
* Environment variable to specify redis connection endpoint, defaults to `localhost`
`REDIS_SSL`
* Environment variable to specify if redis connection should use ssl, defaults to `true`
`REDIS_CLUSTER`
* Environment variable to specify if redis connection is clustered, defaults to `true`
`REDIS_CHECKPOINT`
* Environment variable to specify if redis will be used to save restore checkpoints, defaults to `true`
`SKIP_BOOTSTRAP`
* Environment variable to specify if boostrap command should be skipped, defaults to `true`.
* Use this env variable if bootstrap command was already run, and you have your bootstrap command stuck in a shell
script.
* Set to false in cause there are new indexes or schema changes
---
### 🚀 Deployment
#### Manual Deployment
How to run `pg2elastic` and initialize it.
- Create a .env file using the `cp .env.sample .env` command and replace the existing environment variables with
personal configuration settings.
- Download dependencies using `python setup.py develop`
- Start the app by using `pg2elastic` file command from bin folder, using `python3 pg2elastic --schema yourschema.json`
If you do not run the full setup, you will get errors when running this package.
---
### ✅ Testing
```bash
$ export PG_SCHEMA=
$ flake8 pg2elastic tests
$ python setup.py test
```
---
### 🔊 Logs
This project comes with a [loguru](https://github.com/Delgan/loguru) module for logging, the configurations
for loguru can be found in `pg2elastic` file from bin folder.
---
### 🚚 Deployment
```bash
$ python setup.py sdist bdist_wheel
$ twine upload dist/*
```
---
Raw data
{
"_id": null,
"home_page": "https://github.com/robert-walters/pg2elastic",
"name": "pg2elastic",
"maintainer": "Dorin Musteata RW",
"docs_url": null,
"requires_python": ">=3.7.0",
"maintainer_email": "dorin.musteata-contractor@robertwalters.com",
"keywords": "pg2elastic,elasticsearch,postgres,change data capture",
"author": "Dorin Musteata RW",
"author_email": "dorin.musteata-contractor@robertwalters.com",
"download_url": "https://files.pythonhosted.org/packages/ba/56/3844d15e257ffece2d9d4590c4af9665165bb52040a5977400290fd407d2/pg2elastic-0.3.5.tar.gz",
"platform": null,
"description": "# \u26a1 pg2elastic\n\nEnhanced PostgreSQL to Elasticsearch Data Synchronization\n\n### \ud83d\udcda Description\n\nWelcome to pg2elastic, a fork of the official [pgsync](https://pgsync.com/) package, designed to provide seamless and\nefficient data synchronization between PostgreSQL databases and Elasticsearch clusters.\nBuilding upon the solid foundation of pgsync, pg2elastic inherits all of its powerful capabilities and takes them a step\nfurther.\n\n#### Key Features:\n\n* High-Performance Sync: pg2elastic inherits the robust data synchronization engine from pgsync, ensuring lightning-fast\n and reliable transfers.\n* Real-time Indexing: Seamlessly mirror your PostgreSQL data into Elasticsearch indices, keeping them in sync in\n real-time.\n* Schema Mapping: Easily define and customize the mapping of PostgreSQL schemas to Elasticsearch indexes, giving you\n full control over the data structure.\n* Efficient Data Types Handling: pg2elastic effortlessly handles data type conversions, ensuring accurate representation\n across platforms.\n* Continuous Enhancements: We are committed to actively maintaining and enhancing pg2elastic, incorporating the latest\n advancements in both PostgreSQL and Elasticsearch technologies.\n* Whether you're working on a data-driven application or performing complex data analysis, pg2elastic empowers you with\n a streamlined and feature-rich solution for harmonizing your PostgreSQL and Elasticsearch ecosystems.\n\n---\n\n### \ud83d\udee0\ufe0f Prerequisites\n\n[PGSync Requirements](https://github.com/toluaina/pgsync#requirements)\n\n---\n\n### \u2728 Key Enhancements\n\n* [Loguru, Better Logging Module](https://github.com/Delgan/loguru)\n\n---\n\n#### Fixes\n\nThe actual numerical value of 1000000 becomes 1e when attempted to be converted to a float, leading to a crash during the conversion process.\n* A fix was implemented to change 1e to 1e6, preventing the conversion to float from causing the process to crash.\n\nThe process crashes when custom field types are use in the database, and the returned field type is in the format abc.xyz.\n* A fix was implemented, and the regular expression for LOGICAL_SLOT_SUFFIX was modified.\n\nThe process crashes when a partition notification is received\n* A fix was implemented, and partitions are tracked in lib's materialized, lib's trigger is updated to include partition's parent table\n\nChild records that are inserted are not updating the parent document\n* A fix was implemented, which synchronizes main document in case a child record is created\n\n---\n\n#### Environment Variables\n\n`PG_SCHEMA`\n\n* Environment variable to enhance performance by eliminating the need to scan all schemas\n\n`REDIS_USERNAME`\n\n* Environment variable to specify redis username\n\n`REDIS_PASSWORD`\n\n* Environment variable to specify redis password\n\n`REDIS_ENDPOINT`\n\n* Environment variable to specify redis connection endpoint, defaults to `localhost`\n\n`REDIS_SSL`\n\n* Environment variable to specify if redis connection should use ssl, defaults to `true`\n\n`REDIS_CLUSTER`\n\n* Environment variable to specify if redis connection is clustered, defaults to `true`\n\n`REDIS_CHECKPOINT`\n\n* Environment variable to specify if redis will be used to save restore checkpoints, defaults to `true`\n\n`SKIP_BOOTSTRAP`\n\n* Environment variable to specify if boostrap command should be skipped, defaults to `true`.\n* Use this env variable if bootstrap command was already run, and you have your bootstrap command stuck in a shell\n script.\n* Set to false in cause there are new indexes or schema changes\n\n---\n\n### \ud83d\ude80 Deployment\n\n#### Manual Deployment\n\nHow to run `pg2elastic` and initialize it.\n\n- Create a .env file using the `cp .env.sample .env` command and replace the existing environment variables with\n personal configuration settings.\n\n- Download dependencies using `python setup.py develop`\n\n- Start the app by using `pg2elastic` file command from bin folder, using `python3 pg2elastic --schema yourschema.json`\n\nIf you do not run the full setup, you will get errors when running this package.\n\n---\n\n### \u2705 Testing\n\n```bash\n$ export PG_SCHEMA=\n$ flake8 pg2elastic tests\n$ python setup.py test\n```\n\n---\n\n### \ud83d\udd0a Logs\n\nThis project comes with a [loguru](https://github.com/Delgan/loguru) module for logging, the configurations\nfor loguru can be found in `pg2elastic` file from bin folder.\n\n---\n\n### \ud83d\ude9a Deployment\n\n```bash\n$ python setup.py sdist bdist_wheel\n$ twine upload dist/*\n```\n\n---\n",
"bugtrack_url": null,
"license": "LGPLv3",
"summary": "Enhanced PostgreSQL to Elasticsearch Data Synchronization",
"version": "0.3.5",
"project_urls": {
"Bug Reports": "https://github.com/robert-walters/pg2elastic/issues",
"Homepage": "https://github.com/robert-walters/pg2elastic",
"Source": "https://github.com/robert-walters/pg2elastic"
},
"split_keywords": [
"pg2elastic",
"elasticsearch",
"postgres",
"change data capture"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "9a41563b3ca4648c983d631ff82202107f2b7a1f76072b63d5246d1ee2c26006",
"md5": "15ed54bdc7bd41c21ea9489df2f5c684",
"sha256": "c5691c88aab58a3e8bcb7fc9c516f402a5fac7427bb02aef065ba06973bd1ca9"
},
"downloads": -1,
"filename": "pg2elastic-0.3.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "15ed54bdc7bd41c21ea9489df2f5c684",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7.0",
"size": 58551,
"upload_time": "2023-09-20T16:10:08",
"upload_time_iso_8601": "2023-09-20T16:10:08.073772Z",
"url": "https://files.pythonhosted.org/packages/9a/41/563b3ca4648c983d631ff82202107f2b7a1f76072b63d5246d1ee2c26006/pg2elastic-0.3.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ba563844d15e257ffece2d9d4590c4af9665165bb52040a5977400290fd407d2",
"md5": "ca0648dff925ee3e4381259e94012155",
"sha256": "f7980a1b1372095c93f2d2d388ba4d02b91c4f7b923f4aab92634df9e53a34ba"
},
"downloads": -1,
"filename": "pg2elastic-0.3.5.tar.gz",
"has_sig": false,
"md5_digest": "ca0648dff925ee3e4381259e94012155",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7.0",
"size": 105550,
"upload_time": "2023-09-20T16:10:09",
"upload_time_iso_8601": "2023-09-20T16:10:09.951661Z",
"url": "https://files.pythonhosted.org/packages/ba/56/3844d15e257ffece2d9d4590c4af9665165bb52040a5977400290fd407d2/pg2elastic-0.3.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-09-20 16:10:09",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "robert-walters",
"github_project": "pg2elastic",
"github_not_found": true,
"lcname": "pg2elastic"
}
JSON
