| Name | shaped-target-clickhouse JSON |
| Version |
0.2.5
JSON |
| download |
| home_page | None |
| Summary | `target-clickhouse` is a Singer target for clickhouse, built with the Meltano Singer SDK. |
| upload_time | 2024-03-25 14:45:08 |
| maintainer | None |
| docs_url | None |
| author | Ben Theunissen |
| requires_python | <3.12,>=3.8.0 |
| license | Apache-2.0 |
| keywords |
elt
clickhouse
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# target-clickhouse
`target-clickhouse` is a Singer target for clickhouse.
Build with the [Meltano Target SDK](https://sdk.meltano.com).
<!--
Developer TODO: Update the below as needed to correctly describe the install procedure. For instance, if you do not have a PyPi repo, or if you want users to directly install from your git repo, you can modify this step as appropriate.
## Installation
Install from PyPi:
```bash
pipx install target-clickhouse
```
Install from GitHub:
```bash
pipx install git+https://github.com/ORG_NAME/target-clickhouse.git@main
```
-->
## Configuration
### Accepted Config Options
<!--
Developer TODO: Provide a list of config options accepted by the target.
This section can be created by copy-pasting the CLI output from:
```
target-clickhouse --about --format=markdown
```
-->
| Setting | Required | Default | Description |
|:---------------------|:--------:|:-------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| sqlalchemy_url | False | None | The SQLAlchemy connection string for the ClickHouse database. Used if set, otherwise separate settings are used |
| driver | False | http | Driver type |
| username | False | default | Database user |
| password | False | None | Username password |
| host | False | localhost | Database host |
| port | False | 8123 | Database connection port |
| database | False | default | Database name |
| secure | False | 0 | Should the connection be secure |
| verify | False | 1 | Should secure connection need to verify SSL/TLS |
| engine_type | False | None | The engine type to use for the table. |
| table_name | False | None | The name of the table to write to. Defaults to stream name. |
| table_path | False | None | The table path for replicated tables. This is required when using any of the replication engines. Check out the [documentation](https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/replication#replicatedmergetree-parameters) for more information. Use `$table_name` to substitute the table name. |
| replica_name | False | None | The `replica_name` for replicated tables. This is required when using any of the replication engines. |
| cluster_name | False | None | The cluster to create tables in. This is passed as the `clickhouse_cluster` argument when creating a table. [Documentation](https://clickhouse.com/docs/en/sql-reference/distributed-ddl) can be found here. |
| default_target_schema| False | None | The default target database schema name to use for all streams. |
| optimize_after | False | 0 | Run 'OPTIMIZE TABLE' after data insert |
| optimize_after | False | 0 | Run 'OPTIMIZE TABLE' after data insert. Useful whentable engine removes duplicate rows. |
| load_method | False | TargetLoadMethods.APPEND_ONLY | The method to use when loading data into the destination. `append-only` will always write all input records whether that records already exists or not. `upsert` will update existing records and insert new records. `overwrite` will delete all existing records and insert all input records. |
| stream_maps | False | None | Config object for stream maps capability. For more information check out [Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html). |
| stream_map_config | False | None | User-defined config values to be used within map expressions. |
| flattening_enabled | False | None | 'True' to enable schema flattening and automatically expand nested properties. |
| flattening_max_depth | False | None | The max depth to flatten schemas. |
A full list of supported settings and capabilities is available by running: `target-clickhouse --about`
### Configure using environment variables
This Singer target will automatically import any environment variables within the working directory's
`.env` if the `--config=ENV` is provided, such that config values will be considered if a matching
environment variable is set either in the terminal context or in the `.env` file.
### Source Authentication and Authorization
<!--
Developer TODO: If your target requires special access on the destination system, or any special authentication requirements, provide those here.
-->
## Usage
You can easily run `target-clickhouse` by itself or in a pipeline using [Meltano](https://meltano.com/).
### Executing the Target Directly
```bash
target-clickhouse --version
target-clickhouse --help
# Test using the "Carbon Intensity" sample:
tap-carbon-intensity | target-clickhouse --config /path/to/target-clickhouse-config.json
```
## Developer Resources
Follow these instructions to contribute to this project.
### Initialize your Development Environment
```bash
pipx install poetry
poetry install
```
### Start the Clickhouse container
In order to run the tests locally, you must have a Docker daemon running on your host machine.
You can start the Clickhouse container by running:
```
./docker_run_clickhouse.sh
```
### Create and Run Tests
Create tests within the `tests` subfolder and
then run:
```bash
poetry run pytest
```
You can also test the `target-clickhouse` CLI interface directly using `poetry run`:
```bash
poetry run target-clickhouse --help
```
### Testing with [Meltano](https://meltano.com/)
_**Note:** This target will work in any Singer environment and does not require Meltano.
Examples here are for convenience and to streamline end-to-end orchestration scenarios._
<!--
Developer TODO:
Your project comes with a custom `meltano.yml` project file already created. Open the `meltano.yml` and follow any "TODO" items listed in
the file.
-->
Next, install Meltano (if you haven't already) and any needed plugins:
```bash
# Install meltano
pipx install meltano
# Initialize meltano within this directory
cd target-clickhouse
meltano install
```
Now you can test and orchestrate using Meltano:
```bash
# Test invocation:
meltano invoke target-clickhouse --version
# OR run a test `elt` pipeline with the Carbon Intensity sample tap:
meltano elt tap-carbon-intensity target-clickhouse
```
### SDK Dev Guide
See the [dev guide](https://sdk.meltano.com/en/latest/dev_guide.html) for more instructions on how to use the Meltano Singer SDK to
develop your own Singer taps and targets.
Raw data
{
"_id": null,
"home_page": null,
"name": "shaped-target-clickhouse",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.12,>=3.8.0",
"maintainer_email": null,
"keywords": "ELT, clickhouse",
"author": "Ben Theunissen",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/62/e3/c8d8366a259a6f91f731ca4d108aa895d2e716418fae9c66b48eea00b96d/shaped_target_clickhouse-0.2.5.tar.gz",
"platform": null,
"description": "# target-clickhouse\n\n`target-clickhouse` is a Singer target for clickhouse.\n\nBuild with the [Meltano Target SDK](https://sdk.meltano.com).\n\n<!--\n\nDeveloper TODO: Update the below as needed to correctly describe the install procedure. For instance, if you do not have a PyPi repo, or if you want users to directly install from your git repo, you can modify this step as appropriate.\n\n## Installation\n\nInstall from PyPi:\n\n```bash\npipx install target-clickhouse\n```\n\nInstall from GitHub:\n\n```bash\npipx install git+https://github.com/ORG_NAME/target-clickhouse.git@main\n```\n\n-->\n\n## Configuration\n\n### Accepted Config Options\n\n<!--\nDeveloper TODO: Provide a list of config options accepted by the target.\n\nThis section can be created by copy-pasting the CLI output from:\n\n```\ntarget-clickhouse --about --format=markdown\n```\n-->\n\n| Setting | Required | Default | Description |\n|:---------------------|:--------:|:-------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| sqlalchemy_url | False | None | The SQLAlchemy connection string for the ClickHouse database. Used if set, otherwise separate settings are used |\n| driver | False | http | Driver type |\n| username | False | default | Database user |\n| password | False | None | Username password |\n| host | False | localhost | Database host |\n| port | False | 8123 | Database connection port |\n| database | False | default | Database name |\n| secure | False | 0 | Should the connection be secure |\n| verify | False | 1 | Should secure connection need to verify SSL/TLS |\n| engine_type | False | None | The engine type to use for the table. |\n| table_name | False | None | The name of the table to write to. Defaults to stream name. |\n| table_path | False | None | The table path for replicated tables. This is required when using any of the replication engines. Check out the [documentation](https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/replication#replicatedmergetree-parameters) for more information. Use `$table_name` to substitute the table name. |\n| replica_name | False | None | The `replica_name` for replicated tables. This is required when using any of the replication engines. |\n| cluster_name | False | None | The cluster to create tables in. This is passed as the `clickhouse_cluster` argument when creating a table. [Documentation](https://clickhouse.com/docs/en/sql-reference/distributed-ddl) can be found here. |\n| default_target_schema| False | None | The default target database schema name to use for all streams. |\n| optimize_after | False | 0 | Run 'OPTIMIZE TABLE' after data insert |\n| optimize_after | False | 0 | Run 'OPTIMIZE TABLE' after data insert. Useful whentable engine removes duplicate rows. |\n| load_method | False | TargetLoadMethods.APPEND_ONLY | The method to use when loading data into the destination. `append-only` will always write all input records whether that records already exists or not. `upsert` will update existing records and insert new records. `overwrite` will delete all existing records and insert all input records. |\n| stream_maps | False | None | Config object for stream maps capability. For more information check out [Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html). |\n| stream_map_config | False | None | User-defined config values to be used within map expressions. |\n| flattening_enabled | False | None | 'True' to enable schema flattening and automatically expand nested properties. |\n| flattening_max_depth | False | None | The max depth to flatten schemas. |\n\nA full list of supported settings and capabilities is available by running: `target-clickhouse --about`\n\n### Configure using environment variables\n\nThis Singer target will automatically import any environment variables within the working directory's\n`.env` if the `--config=ENV` is provided, such that config values will be considered if a matching\nenvironment variable is set either in the terminal context or in the `.env` file.\n\n### Source Authentication and Authorization\n\n<!--\nDeveloper TODO: If your target requires special access on the destination system, or any special authentication requirements, provide those here.\n-->\n\n## Usage\n\nYou can easily run `target-clickhouse` by itself or in a pipeline using [Meltano](https://meltano.com/).\n\n### Executing the Target Directly\n\n```bash\ntarget-clickhouse --version\ntarget-clickhouse --help\n# Test using the \"Carbon Intensity\" sample:\ntap-carbon-intensity | target-clickhouse --config /path/to/target-clickhouse-config.json\n```\n\n## Developer Resources\n\nFollow these instructions to contribute to this project.\n\n### Initialize your Development Environment\n\n```bash\npipx install poetry\npoetry install\n```\n\n### Start the Clickhouse container\n\nIn order to run the tests locally, you must have a Docker daemon running on your host machine.\n\nYou can start the Clickhouse container by running:\n```\n./docker_run_clickhouse.sh\n```\n\n### Create and Run Tests\n\nCreate tests within the `tests` subfolder and\n then run:\n\n```bash\npoetry run pytest\n```\n\nYou can also test the `target-clickhouse` CLI interface directly using `poetry run`:\n\n```bash\npoetry run target-clickhouse --help\n```\n\n### Testing with [Meltano](https://meltano.com/)\n\n_**Note:** This target will work in any Singer environment and does not require Meltano.\nExamples here are for convenience and to streamline end-to-end orchestration scenarios._\n\n<!--\nDeveloper TODO:\nYour project comes with a custom `meltano.yml` project file already created. Open the `meltano.yml` and follow any \"TODO\" items listed in\nthe file.\n-->\n\nNext, install Meltano (if you haven't already) and any needed plugins:\n\n```bash\n# Install meltano\npipx install meltano\n# Initialize meltano within this directory\ncd target-clickhouse\nmeltano install\n```\n\nNow you can test and orchestrate using Meltano:\n\n```bash\n# Test invocation:\nmeltano invoke target-clickhouse --version\n# OR run a test `elt` pipeline with the Carbon Intensity sample tap:\nmeltano elt tap-carbon-intensity target-clickhouse\n```\n\n### SDK Dev Guide\n\nSee the [dev guide](https://sdk.meltano.com/en/latest/dev_guide.html) for more instructions on how to use the Meltano Singer SDK to\ndevelop your own Singer taps and targets.\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "`target-clickhouse` is a Singer target for clickhouse, built with the Meltano Singer SDK.",
"version": "0.2.5",
"project_urls": null,
"split_keywords": [
"elt",
" clickhouse"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "b89f332326fd901b8786247d3427a10d5a4ccf6e01aa7cd4c7103d000504f36d",
"md5": "0e4ba685012324b00d78bb53c12b4573",
"sha256": "7a86f9da406125ad8df4141f8bf672798d4ad635a0d9ff20a47d1e0d7b653715"
},
"downloads": -1,
"filename": "shaped_target_clickhouse-0.2.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0e4ba685012324b00d78bb53c12b4573",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.12,>=3.8.0",
"size": 16983,
"upload_time": "2024-03-25T14:45:06",
"upload_time_iso_8601": "2024-03-25T14:45:06.110321Z",
"url": "https://files.pythonhosted.org/packages/b8/9f/332326fd901b8786247d3427a10d5a4ccf6e01aa7cd4c7103d000504f36d/shaped_target_clickhouse-0.2.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "62e3c8d8366a259a6f91f731ca4d108aa895d2e716418fae9c66b48eea00b96d",
"md5": "2736fecf2def0bdb0c4fa10bc938554e",
"sha256": "0445e6951f86775536f425f8ec81553bf38fde2f43ba74d4c86b1819299135d1"
},
"downloads": -1,
"filename": "shaped_target_clickhouse-0.2.5.tar.gz",
"has_sig": false,
"md5_digest": "2736fecf2def0bdb0c4fa10bc938554e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.12,>=3.8.0",
"size": 16939,
"upload_time": "2024-03-25T14:45:08",
"upload_time_iso_8601": "2024-03-25T14:45:08.451211Z",
"url": "https://files.pythonhosted.org/packages/62/e3/c8d8366a259a6f91f731ca4d108aa895d2e716418fae9c66b48eea00b96d/shaped_target_clickhouse-0.2.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-25 14:45:08",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "shaped-target-clickhouse"
}
