# Datahub Postgres View Lineage
A ingestion source to generate lineage for views in a Postgres database.
## Quick Start
First install [Poetry](https://python-poetry.org/docs/#installation) and [task](https://taskfile.dev) and initialize the project
```sh
task setup
```
Now, start a database
```sh
task start wait sample-view
```
Now run the ingestion to the console
```sh
task run
```
When it is successful, the output should include
```sh
Source (datahub_postgres_lineage.ingestion.PostgresLineageSource) report:
{'events_produced': '1',
'events_produced_per_sec': '26',
'event_ids': ['urn:li:dataset:(urn:li:dataPlatform:postgres,cool_db.public.emails,PROD)-upstreamLineage'],
'warnings': {},
'failures': {},
'filtered': ['public.names'],
'start_time': '2022-12-20 16:09:46.105046 (now).',
'running_time': '0.04 seconds'}
```
## Configuration
| Key | Description | Default |
| --- | --- | --- |
| `username` | The username to connect to the database | '' |
| `password` | The password to connect to the database | '' |
| `host_port` | The host and port to connect to the database | '' |
| `database` | The database to connect to | '' |
| `database_alias` | Alias to apply to database when ingesting. | '' |
| `sqlalchemy_uri` | SQLAlchemy URI to connect to the database | '' |
| `scheme` | The SQLAlchemy scheme to use | `postgressql+psycopg2` |
| `schema_pattern` | | |
| `schema_pattern.allow`| Regexp pattern to match schemas to include | `.*` |
| `schema_pattern.deny` | Regexp pattern to match schemas to exclude, 'information_schema' and 'pg_catalog' are already excluded | '' |
| `view_pattern` | | |
| `view_pattern.allow` | Regexp pattern to match view names to include | `.*` |
| `view_pattern.deny` | Regexp pattern to match view names to exclude | '' |
Raw data
{
"_id": null,
"home_page": "https://github.com/contiamo/datahub-postgres-lineage",
"name": "datahub-postgres-lineage",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8.1,<3.11",
"maintainer_email": "",
"keywords": "datahub,postgres,sql lineage",
"author": "Contiamo",
"author_email": "developers@contiamo.com",
"download_url": "https://files.pythonhosted.org/packages/9c/1a/229cf3c1875fc5c769dae087c0c0d0760405770e702826a8223a5a7b888c/datahub_postgres_lineage-0.1.3.tar.gz",
"platform": null,
"description": "# Datahub Postgres View Lineage\n\n\nA ingestion source to generate lineage for views in a Postgres database.\n\n\n## Quick Start\n\nFirst install [Poetry](https://python-poetry.org/docs/#installation) and [task](https://taskfile.dev) and initialize the project\n\n```sh\ntask setup\n```\n\nNow, start a database\n\n```sh\ntask start wait sample-view\n```\n\nNow run the ingestion to the console\n```sh\ntask run\n```\n\nWhen it is successful, the output should include\n\n```sh\nSource (datahub_postgres_lineage.ingestion.PostgresLineageSource) report:\n{'events_produced': '1',\n 'events_produced_per_sec': '26',\n 'event_ids': ['urn:li:dataset:(urn:li:dataPlatform:postgres,cool_db.public.emails,PROD)-upstreamLineage'],\n 'warnings': {},\n 'failures': {},\n 'filtered': ['public.names'],\n 'start_time': '2022-12-20 16:09:46.105046 (now).',\n 'running_time': '0.04 seconds'}\n```\n\n## Configuration\n\n| Key | Description | Default |\n| --- | --- | --- |\n| `username` | The username to connect to the database | '' |\n| `password` | The password to connect to the database | '' |\n| `host_port` | The host and port to connect to the database | '' |\n| `database` | The database to connect to | '' |\n| `database_alias` | Alias to apply to database when ingesting. | '' |\n| `sqlalchemy_uri` | SQLAlchemy URI to connect to the database | '' |\n| `scheme` | The SQLAlchemy scheme to use | `postgressql+psycopg2` |\n| `schema_pattern` | | |\n| `schema_pattern.allow`| Regexp pattern to match schemas to include | `.*` |\n| `schema_pattern.deny` | Regexp pattern to match schemas to exclude, 'information_schema' and 'pg_catalog' are already excluded | '' |\n| `view_pattern` | | |\n| `view_pattern.allow` | Regexp pattern to match view names to include | `.*` |\n| `view_pattern.deny` | Regexp pattern to match view names to exclude | '' |",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Extract table lineage from Postgres views",
"version": "0.1.3",
"split_keywords": [
"datahub",
"postgres",
"sql lineage"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "6b4ab04026375ab3ceee56a285cdc485126682c63c5d14c52b0e8f0f283a0f1e",
"md5": "4ecbeea2c6ed2ca178c8d8bc94398f49",
"sha256": "204828f768b4394ef16149916a70ddd08dbc852e6b69cc640e908c90d7b6f1b1"
},
"downloads": -1,
"filename": "datahub_postgres_lineage-0.1.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "4ecbeea2c6ed2ca178c8d8bc94398f49",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8.1,<3.11",
"size": 9610,
"upload_time": "2023-01-04T11:59:23",
"upload_time_iso_8601": "2023-01-04T11:59:23.455989Z",
"url": "https://files.pythonhosted.org/packages/6b/4a/b04026375ab3ceee56a285cdc485126682c63c5d14c52b0e8f0f283a0f1e/datahub_postgres_lineage-0.1.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9c1a229cf3c1875fc5c769dae087c0c0d0760405770e702826a8223a5a7b888c",
"md5": "ab320ba3587db56a63f4d6aeedc01123",
"sha256": "e5ee5b31db630d0502a1e3a166eb126306e321b3fc37272dccb1aba485260564"
},
"downloads": -1,
"filename": "datahub_postgres_lineage-0.1.3.tar.gz",
"has_sig": false,
"md5_digest": "ab320ba3587db56a63f4d6aeedc01123",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8.1,<3.11",
"size": 9360,
"upload_time": "2023-01-04T11:59:24",
"upload_time_iso_8601": "2023-01-04T11:59:24.746900Z",
"url": "https://files.pythonhosted.org/packages/9c/1a/229cf3c1875fc5c769dae087c0c0d0760405770e702826a8223a5a7b888c/datahub_postgres_lineage-0.1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-01-04 11:59:24",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "contiamo",
"github_project": "datahub-postgres-lineage",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "datahub-postgres-lineage"
}
JSON
