# dbt-spark-livy
The `dbt-spark-livy` adapter allows you to use [dbt](https://www.getdbt.com/) along with [Apache spark-livy](https://spark.apache.org/) and [Cloudera Data Platform](https://cloudera.com) with Livy server support. This code bases use the dbt-spark project (https://github.com/dbt-labs/dbt-spark), and provides a Livy connectivity support over it.
## Getting started
- [Install dbt](https://docs.getdbt.com/docs/installation)
- Read the [introduction](https://docs.getdbt.com/docs/introduction/) and [viewpoint](https://docs.getdbt.com/docs/about/viewpoint/)
## Running locally
A `docker-compose` environment starts a Spark Thrift server and a Postgres database as a Hive Metastore backend.
Note: dbt-spark now supports Spark 3.1.1 (formerly on Spark 2.x).
Python >= 3.8
dbt-core ~= 1.3.0
pyspark
sqlparams
requests_kerberos
requests-toolbelt
python-decouple
### Installing dbt-spark-livy
`pip install dbt-spark-livy`
### Profile Setup
```
demo_project:
target: dev
outputs:
dev:
type: spark_livy
method: livy
schema: my_db
host: https://spark-livy-gateway.my.org.com/dbt-spark/cdp-proxy-api/livy_for_spark3/
user: my_user
password: my_pass
```
### Caveats
- While using livy , in the Livy UI if you notice sessions change state to dead from starting instead of idle, make sure there is a proper mapping for the user in the IDBroker mapping section
- Actions > Manage Access > IDBroker Mappings . [Reference](https://docs.cloudera.com/cdf-datahub/7.2.15/flink-analyzing-data/topics/cdf-datahub-sa-create-idbroker-mapping.html)
- Also make sure the workload password is set either through UI or CLI. [Reference](https://docs.cloudera.com/management-console/cloud/user-management/topics/mc-setting-the-ipa-password.html)
## Supported features
Please see the original adapter documentation: https://github.com/dbt-labs/dbt-spark and https://docs.getdbt.com/reference/warehouse-profiles/spark-profile
Raw data
{
"_id": null,
"home_page": "https://github.com/cloudera/dbt-spark-livy",
"name": "dbt-spark-livy",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "",
"author": "Cloudera",
"author_email": "innovation-feedback@cloudera.com",
"download_url": "https://files.pythonhosted.org/packages/55/f8/0e3e8dd35e8b81a08e54bdb3220e1bc7a18a5fea84b7301e06f3b502901d/dbt-spark-livy-1.3.1.tar.gz",
"platform": null,
"description": "# dbt-spark-livy\n\nThe `dbt-spark-livy` adapter allows you to use [dbt](https://www.getdbt.com/) along with [Apache spark-livy](https://spark.apache.org/) and [Cloudera Data Platform](https://cloudera.com) with Livy server support. This code bases use the dbt-spark project (https://github.com/dbt-labs/dbt-spark), and provides a Livy connectivity support over it. \n\n## Getting started\n\n- [Install dbt](https://docs.getdbt.com/docs/installation)\n- Read the [introduction](https://docs.getdbt.com/docs/introduction/) and [viewpoint](https://docs.getdbt.com/docs/about/viewpoint/)\n\n## Running locally\nA `docker-compose` environment starts a Spark Thrift server and a Postgres database as a Hive Metastore backend.\nNote: dbt-spark now supports Spark 3.1.1 (formerly on Spark 2.x).\n\nPython >= 3.8\n\ndbt-core ~= 1.3.0\n\npyspark\n\nsqlparams\n\nrequests_kerberos\n\nrequests-toolbelt\n\npython-decouple\n\n\n### Installing dbt-spark-livy\n\n`pip install dbt-spark-livy`\n\n### Profile Setup\n\n```\ndemo_project:\n target: dev\n outputs:\n dev:\n type: spark_livy\n method: livy\n schema: my_db\n host: https://spark-livy-gateway.my.org.com/dbt-spark/cdp-proxy-api/livy_for_spark3/\n user: my_user\n password: my_pass\n```\n\n### Caveats\n- While using livy , in the Livy UI if you notice sessions change state to dead from starting instead of idle, make sure there is a proper mapping for the user in the IDBroker mapping section \n- Actions > Manage Access > IDBroker Mappings . [Reference](https://docs.cloudera.com/cdf-datahub/7.2.15/flink-analyzing-data/topics/cdf-datahub-sa-create-idbroker-mapping.html)\n- Also make sure the workload password is set either through UI or CLI. [Reference](https://docs.cloudera.com/management-console/cloud/user-management/topics/mc-setting-the-ipa-password.html)\n\n## Supported features\nPlease see the original adapter documentation: https://github.com/dbt-labs/dbt-spark and https://docs.getdbt.com/reference/warehouse-profiles/spark-profile\n",
"bugtrack_url": null,
"license": "",
"summary": "The dbt-spark-livy adapter plugin for Spark in Cloudera DataHub with Livy interface",
"version": "1.3.1",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "998fa9273171e157bcbc79df7cfa70d8",
"sha256": "9b9fe3063361f1f2b0f50b5d9e59b30dc0a1323691ef5f1f402a9ecbb482295a"
},
"downloads": -1,
"filename": "dbt_spark_livy-1.3.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "998fa9273171e157bcbc79df7cfa70d8",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 49765,
"upload_time": "2022-12-05T05:14:03",
"upload_time_iso_8601": "2022-12-05T05:14:03.661375Z",
"url": "https://files.pythonhosted.org/packages/bb/ed/c78fbef98b728e10b0cc47148cd2743d28b8f0b532fa9db447d9f50710e9/dbt_spark_livy-1.3.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"md5": "ec166ab98352e8e5d6b44f59aa79fa22",
"sha256": "094cd6a4c001905c8757cd74df61ab21bd84ff6c1eb219f13ea83afbb3a2154e"
},
"downloads": -1,
"filename": "dbt-spark-livy-1.3.1.tar.gz",
"has_sig": false,
"md5_digest": "ec166ab98352e8e5d6b44f59aa79fa22",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 38612,
"upload_time": "2022-12-05T05:14:12",
"upload_time_iso_8601": "2022-12-05T05:14:12.831407Z",
"url": "https://files.pythonhosted.org/packages/55/f8/0e3e8dd35e8b81a08e54bdb3220e1bc7a18a5fea84b7301e06f3b502901d/dbt-spark-livy-1.3.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2022-12-05 05:14:12",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "cloudera",
"github_project": "dbt-spark-livy",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"tox": true,
"lcname": "dbt-spark-livy"
}
JSON
