# Beam - Postgres connector
[][pypi-project]
[][pypi-project]
[](https://opensource.org/licenses/MIT)
Beam - Postgres Connector provides an io connector for PostgreSQL read/write in [Apache Beam](https://beam.apache.org/) pipelines.
## Installation
```bash
pip install beam-postgres-connector
```
## Usage
- Read From PostgreSQL
```Python
import apache_beam as beam
from beam_postgres import splitters
from beam_postgres.io import ReadFromPostgres
with beam.Pipeline(options=options) as p:
read_from_postgres = ReadFromPostgres(
query="SELECT * FROM test_db.test.test;",
host="localhost",
database="test_db",
user="test",
password="test",
port=5432,
splitter=splitters.NoSplitter() # you can select how to split query for performance
)
(
p
| "ReadFromPostgres" >> read_from_postgres
| "WriteToStdout" >> beam.Map(print)
)
```
- Write To Postgres
```Python
import apache_beam as beam
from beam_postgres.io import WriteToPostgres
with beam.Pipeline(options=options) as p:
write_to_postgres = WriteToPostgres(
host="localhost",
database="test_db",
table="test.test",
user="test",
password="test",
port=5432,
batch_size=1000,
)
(
p
| "ReadFromInMemory"
>> beam.Create(
[
{
"name": "test data",
}
]
)
| "WriteToPostgres" >> write_to_postgres
)
```
See [here][examples] for more examples.
## splitters
- NoSplitter
Do not split the query
- QuerySplitter
Split the query by a specified separator string and distribute it for parallel processing across multiple nodes. Specify non-overlapping ranges for each query in the WHERE clause. The processing results will be combined using UNION ALL.
## License
MIT License.
[pypi-project]: https://pypi.org/project/beam-postgres-connector
[examples]: https://github.com/satokiyo/beam-postgres-connector/tree/main/examples
Raw data
{
"_id": null,
"home_page": "https://github.com/satokiyo/beam-postgres-connector",
"name": "beam-postgres-connector",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8.1,<4.0.0",
"maintainer_email": "",
"keywords": "apache beam,beam,postgres,postgresql",
"author": "satokiyo",
"author_email": "satokiyo@loop8.biz",
"download_url": "https://files.pythonhosted.org/packages/a2/15/7cc53f2b81025cda0e4e6de9e0b657164dc7a1ebec5360ec0af86938fe9c/beam_postgres_connector-0.1.2.tar.gz",
"platform": null,
"description": "# Beam - Postgres connector\n\n[][pypi-project]\n[][pypi-project]\n[](https://opensource.org/licenses/MIT)\n\nBeam - Postgres Connector provides an io connector for PostgreSQL read/write in [Apache Beam](https://beam.apache.org/) pipelines.\n\n## Installation\n\n```bash\npip install beam-postgres-connector\n```\n\n## Usage\n\n- Read From PostgreSQL\n\n```Python\nimport apache_beam as beam\nfrom beam_postgres import splitters\nfrom beam_postgres.io import ReadFromPostgres\n\nwith beam.Pipeline(options=options) as p:\n read_from_postgres = ReadFromPostgres(\n query=\"SELECT * FROM test_db.test.test;\",\n host=\"localhost\",\n database=\"test_db\",\n user=\"test\",\n password=\"test\",\n port=5432,\n splitter=splitters.NoSplitter() # you can select how to split query for performance\n )\n\n\n (\n p\n | \"ReadFromPostgres\" >> read_from_postgres\n | \"WriteToStdout\" >> beam.Map(print)\n )\n```\n\n- Write To Postgres\n\n```Python\nimport apache_beam as beam\nfrom beam_postgres.io import WriteToPostgres\n\n\nwith beam.Pipeline(options=options) as p:\n write_to_postgres = WriteToPostgres(\n host=\"localhost\",\n database=\"test_db\",\n table=\"test.test\",\n user=\"test\",\n password=\"test\",\n port=5432,\n batch_size=1000,\n )\n\n (\n p\n | \"ReadFromInMemory\"\n >> beam.Create(\n [\n {\n \"name\": \"test data\",\n }\n ]\n )\n | \"WriteToPostgres\" >> write_to_postgres\n )\n```\n\nSee [here][examples] for more examples.\n\n## splitters\n\n- NoSplitter\n\n Do not split the query\n\n- QuerySplitter\n\n Split the query by a specified separator string and distribute it for parallel processing across multiple nodes. Specify non-overlapping ranges for each query in the WHERE clause. The processing results will be combined using UNION ALL.\n\n## License\n\nMIT License.\n\n[pypi-project]: https://pypi.org/project/beam-postgres-connector\n[examples]: https://github.com/satokiyo/beam-postgres-connector/tree/main/examples\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "An io connector for PostgreSQL read/write in Apache Beam pipelines.",
"version": "0.1.2",
"project_urls": {
"Homepage": "https://github.com/satokiyo/beam-postgres-connector",
"Repository": "https://github.com/satokiyo/beam-postgres-connector"
},
"split_keywords": [
"apache beam",
"beam",
"postgres",
"postgresql"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "74752a5721a0eac0026a7c6f652c169f8e8fad4fb69d5331f679ab6fe1ba3bc0",
"md5": "d111f0a8cd37a7db70904e3156e6361a",
"sha256": "9342ba1c648934721adf0456bb1c79dc327581901e1ab33cd5e060f979f02344"
},
"downloads": -1,
"filename": "beam_postgres_connector-0.1.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d111f0a8cd37a7db70904e3156e6361a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8.1,<4.0.0",
"size": 8453,
"upload_time": "2023-07-01T02:24:30",
"upload_time_iso_8601": "2023-07-01T02:24:30.312308Z",
"url": "https://files.pythonhosted.org/packages/74/75/2a5721a0eac0026a7c6f652c169f8e8fad4fb69d5331f679ab6fe1ba3bc0/beam_postgres_connector-0.1.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a2157cc53f2b81025cda0e4e6de9e0b657164dc7a1ebec5360ec0af86938fe9c",
"md5": "b9ea4f4abdde7d35f6bc737c30f92d4c",
"sha256": "2356ca6ae2d60770d611d855f84554959dc0948584958b71c289b51bebb3f3a1"
},
"downloads": -1,
"filename": "beam_postgres_connector-0.1.2.tar.gz",
"has_sig": false,
"md5_digest": "b9ea4f4abdde7d35f6bc737c30f92d4c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8.1,<4.0.0",
"size": 7764,
"upload_time": "2023-07-01T02:24:31",
"upload_time_iso_8601": "2023-07-01T02:24:31.915692Z",
"url": "https://files.pythonhosted.org/packages/a2/15/7cc53f2b81025cda0e4e6de9e0b657164dc7a1ebec5360ec0af86938fe9c/beam_postgres_connector-0.1.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-01 02:24:31",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "satokiyo",
"github_project": "beam-postgres-connector",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "beam-postgres-connector"
}
JSON
