<p align="center">
<img src="https://github.com/TobikoData/sqlmesh/blob/main/docs/readme/sqlmesh.png?raw=true" alt="SQLMesh logo" width="50%" height="50%">
</p>
SQLMesh is a next-generation data transformation framework designed to ship data quickly, efficiently, and without error. Data teams can efficiently run and deploy data transformations written in SQL or Python with visibility and control at any size.
It is more than just a [dbt alternative](https://tobikodata.com/reduce_costs_with_cron_and_partitions.html).
<p align="center">
<img src="https://github.com/TobikoData/sqlmesh/blob/main/docs/readme/architecture_diagram.png?raw=true" alt="Architecture Diagram" width="100%" height="100%">
</p>
## Core Features
<img src="https://github.com/TobikoData/sqlmesh-public-assets/blob/main/sqlmesh_plan_mode.gif?raw=true" alt="SQLMesh Plan Mode">
> Get instant SQL impact analysis of your changes, whether in the CLI or in [SQLMesh Plan Mode](https://sqlmesh.readthedocs.io/en/stable/guides/ui/?h=modes#working-with-an-ide)
<details>
<summary><b>Virtual Data Environments</b></summary>
* See a full diagram of how [Virtual Data Environments](https://whimsical.com/virtual-data-environments-MCT8ngSxFHict4wiL48ymz) work
* [Watch this video to learn more](https://www.youtube.com/watch?v=weJH3eM0rzc)
</details>
* Create isolated development environments without data warehouse costs
* Plan / Apply workflow like [Terraform](https://www.terraform.io/) to understand potential impact of changes
* Easy to use [CI/CD bot](https://sqlmesh.readthedocs.io/en/stable/integrations/github/) for true blue-green deployments
<details>
<summary><b>Efficiency and Testing</b></summary>
Running this command will generate a unit test file in the `tests/` folder: `test_stg_payments.yaml`
Runs a live query to generate the expected output of the model
```bash
sqlmesh create_test tcloud_demo.stg_payments --query tcloud_demo.seed_raw_payments "select * from tcloud_demo.seed_raw_payments limit 5"
# run the unit test
sqlmesh test
```
```sql
MODEL (
name tcloud_demo.stg_payments,
cron '@daily',
grain payment_id,
audits (UNIQUE_VALUES(columns = (
payment_id
)), NOT_NULL(columns = (
payment_id
)))
);
SELECT
id AS payment_id,
order_id,
payment_method,
amount / 100 AS amount, /* `amount` is currently stored in cents, so we convert it to dollars */
'new_column' AS new_column, /* non-breaking change example */
FROM tcloud_demo.seed_raw_payments
```
```yaml
test_stg_payments:
model: tcloud_demo.stg_payments
inputs:
tcloud_demo.seed_raw_payments:
- id: 66
order_id: 58
payment_method: coupon
amount: 1800
- id: 27
order_id: 24
payment_method: coupon
amount: 2600
- id: 30
order_id: 25
payment_method: coupon
amount: 1600
- id: 109
order_id: 95
payment_method: coupon
amount: 2400
- id: 3
order_id: 3
payment_method: coupon
amount: 100
outputs:
query:
- payment_id: 66
order_id: 58
payment_method: coupon
amount: 18.0
new_column: new_column
- payment_id: 27
order_id: 24
payment_method: coupon
amount: 26.0
new_column: new_column
- payment_id: 30
order_id: 25
payment_method: coupon
amount: 16.0
new_column: new_column
- payment_id: 109
order_id: 95
payment_method: coupon
amount: 24.0
new_column: new_column
- payment_id: 3
order_id: 3
payment_method: coupon
amount: 1.0
new_column: new_column
```
</details>
* Never build a table [more than once](https://tobikodata.com/simplicity-or-efficiency-how-dbt-makes-you-choose.html)
* Track what data’s been modified and run only the necessary transformations for [incremental models](https://tobikodata.com/correctly-loading-incremental-data-at-scale.html)
* Run [unit tests](https://tobikodata.com/we-need-even-greater-expectations.html) for free and configure automated audits
<details>
<summary><b>Level Up Your SQL</b></summary>
Write SQL in any dialect and SQLMesh will transpile it to your target SQL dialect on the fly before sending it to the warehouse.
<img src="https://github.com/TobikoData/sqlmesh/blob/main/docs/readme/transpile_example.png?raw=true" alt="Transpile Example">
</details>
* Debug transformation errors *before* you run them in your warehouse in [10+ different SQL dialects](https://sqlmesh.readthedocs.io/en/stable/integrations/overview/#execution-engines)
* Definitions using [simply SQL](https://sqlmesh.readthedocs.io/en/stable/concepts/models/sql_models/#sql-based-definition) (no need for redundant and confusing `Jinja` + `YAML`)
* See impact of changes before you run them in your warehouse with column-level lineage
For more information, check out the [website](https://sqlmesh.com) and [documentation](https://sqlmesh.readthedocs.io/en/stable/).
## Getting Started
Install SQLMesh through [pypi](https://pypi.org/project/sqlmesh/) by running:
```bash
mkdir sqlmesh-example
cd sqlmesh-example
python -m venv .env
source .env/bin/activate
pip install sqlmesh
sqlmesh init duckdb # get started right away with a local duckdb instance
```
Follow the [quickstart guide](https://sqlmesh.readthedocs.io/en/stable/quickstart/cli/#1-create-the-sqlmesh-project) to learn how to use SQLMesh. You already have a head start!
Follow this [example](https://sqlmesh.readthedocs.io/en/stable/examples/incremental_time_full_walkthrough/) to learn how to use SQLMesh in a full walkthrough.
## Join Our Community
Together, we want to build data transformation without the waste. Connect with us in the following ways:
* Join the [Tobiko Slack Community](https://tobikodata.com/slack) to ask questions, or just to say hi!
* File an issue on our [GitHub](https://github.com/TobikoData/sqlmesh/issues/new)
* Send us an email at [hello@tobikodata.com](mailto:hello@tobikodata.com) with your questions or feedback
* Read our [blog](https://tobikodata.com/blog)
## Contribution
Contributions in the form of issues or pull requests (from fork) are greatly appreciated.
[Read more](https://sqlmesh.readthedocs.io/en/stable/development/) on how to contribute to SQLMesh open source.
[Watch this video walkthrough](https://www.loom.com/share/2abd0d661c12459693fa155490633126?sid=b65c1c0f-8ef7-4036-ad19-3f85a3b87ff2) to see how our team contributes a feature to SQLMesh.
Raw data
{
"_id": null,
"home_page": "https://github.com/TobikoData/sqlmesh",
"name": "sqlmesh",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": null,
"author": "TobikoData Inc.",
"author_email": "engineering@tobikodata.com",
"download_url": "https://files.pythonhosted.org/packages/dd/06/03590a53218e06b23ca143dcdac045c22f8c48369d1f09993e6b3c22129a/sqlmesh-0.141.1.tar.gz",
"platform": null,
"description": "<p align=\"center\">\n <img src=\"https://github.com/TobikoData/sqlmesh/blob/main/docs/readme/sqlmesh.png?raw=true\" alt=\"SQLMesh logo\" width=\"50%\" height=\"50%\">\n</p>\n\nSQLMesh is a next-generation data transformation framework designed to ship data quickly, efficiently, and without error. Data teams can efficiently run and deploy data transformations written in SQL or Python with visibility and control at any size.\n\nIt is more than just a [dbt alternative](https://tobikodata.com/reduce_costs_with_cron_and_partitions.html).\n\n<p align=\"center\">\n <img src=\"https://github.com/TobikoData/sqlmesh/blob/main/docs/readme/architecture_diagram.png?raw=true\" alt=\"Architecture Diagram\" width=\"100%\" height=\"100%\">\n</p>\n\n## Core Features\n<img src=\"https://github.com/TobikoData/sqlmesh-public-assets/blob/main/sqlmesh_plan_mode.gif?raw=true\" alt=\"SQLMesh Plan Mode\">\n\n> Get instant SQL impact analysis of your changes, whether in the CLI or in [SQLMesh Plan Mode](https://sqlmesh.readthedocs.io/en/stable/guides/ui/?h=modes#working-with-an-ide)\n\n <details>\n <summary><b>Virtual Data Environments</b></summary>\n\n * See a full diagram of how [Virtual Data Environments](https://whimsical.com/virtual-data-environments-MCT8ngSxFHict4wiL48ymz) work\n * [Watch this video to learn more](https://www.youtube.com/watch?v=weJH3eM0rzc)\n\n </details>\n\n * Create isolated development environments without data warehouse costs\n * Plan / Apply workflow like [Terraform](https://www.terraform.io/) to understand potential impact of changes\n * Easy to use [CI/CD bot](https://sqlmesh.readthedocs.io/en/stable/integrations/github/) for true blue-green deployments\n\n<details>\n<summary><b>Efficiency and Testing</b></summary>\n\nRunning this command will generate a unit test file in the `tests/` folder: `test_stg_payments.yaml`\n\nRuns a live query to generate the expected output of the model\n\n```bash\nsqlmesh create_test tcloud_demo.stg_payments --query tcloud_demo.seed_raw_payments \"select * from tcloud_demo.seed_raw_payments limit 5\"\n\n# run the unit test\nsqlmesh test\n```\n\n```sql\nMODEL (\n name tcloud_demo.stg_payments,\n cron '@daily',\n grain payment_id,\n audits (UNIQUE_VALUES(columns = (\n payment_id\n )), NOT_NULL(columns = (\n payment_id\n )))\n);\n\nSELECT\n id AS payment_id,\n order_id,\n payment_method,\n amount / 100 AS amount, /* `amount` is currently stored in cents, so we convert it to dollars */\n 'new_column' AS new_column, /* non-breaking change example */\nFROM tcloud_demo.seed_raw_payments\n```\n\n```yaml\ntest_stg_payments:\nmodel: tcloud_demo.stg_payments\ninputs:\n tcloud_demo.seed_raw_payments:\n - id: 66\n order_id: 58\n payment_method: coupon\n amount: 1800\n - id: 27\n order_id: 24\n payment_method: coupon\n amount: 2600\n - id: 30\n order_id: 25\n payment_method: coupon\n amount: 1600\n - id: 109\n order_id: 95\n payment_method: coupon\n amount: 2400\n - id: 3\n order_id: 3\n payment_method: coupon\n amount: 100\noutputs:\n query:\n - payment_id: 66\n order_id: 58\n payment_method: coupon\n amount: 18.0\n new_column: new_column\n - payment_id: 27\n order_id: 24\n payment_method: coupon\n amount: 26.0\n new_column: new_column\n - payment_id: 30\n order_id: 25\n payment_method: coupon\n amount: 16.0\n new_column: new_column\n - payment_id: 109\n order_id: 95\n payment_method: coupon\n amount: 24.0\n new_column: new_column\n - payment_id: 3\n order_id: 3\n payment_method: coupon\n amount: 1.0\n new_column: new_column\n```\n</details>\n\n* Never build a table [more than once](https://tobikodata.com/simplicity-or-efficiency-how-dbt-makes-you-choose.html)\n* Track what data\u2019s been modified and run only the necessary transformations for [incremental models](https://tobikodata.com/correctly-loading-incremental-data-at-scale.html)\n* Run [unit tests](https://tobikodata.com/we-need-even-greater-expectations.html) for free and configure automated audits\n\n<details>\n<summary><b>Level Up Your SQL</b></summary>\nWrite SQL in any dialect and SQLMesh will transpile it to your target SQL dialect on the fly before sending it to the warehouse.\n<img src=\"https://github.com/TobikoData/sqlmesh/blob/main/docs/readme/transpile_example.png?raw=true\" alt=\"Transpile Example\">\n</details>\n\n* Debug transformation errors *before* you run them in your warehouse in [10+ different SQL dialects](https://sqlmesh.readthedocs.io/en/stable/integrations/overview/#execution-engines)\n* Definitions using [simply SQL](https://sqlmesh.readthedocs.io/en/stable/concepts/models/sql_models/#sql-based-definition) (no need for redundant and confusing `Jinja` + `YAML`)\n* See impact of changes before you run them in your warehouse with column-level lineage\n\nFor more information, check out the [website](https://sqlmesh.com) and [documentation](https://sqlmesh.readthedocs.io/en/stable/).\n\n## Getting Started\nInstall SQLMesh through [pypi](https://pypi.org/project/sqlmesh/) by running:\n\n```bash\nmkdir sqlmesh-example\ncd sqlmesh-example\npython -m venv .env\nsource .env/bin/activate\npip install sqlmesh\nsqlmesh init duckdb # get started right away with a local duckdb instance\n```\n\nFollow the [quickstart guide](https://sqlmesh.readthedocs.io/en/stable/quickstart/cli/#1-create-the-sqlmesh-project) to learn how to use SQLMesh. You already have a head start!\n\nFollow this [example](https://sqlmesh.readthedocs.io/en/stable/examples/incremental_time_full_walkthrough/) to learn how to use SQLMesh in a full walkthrough.\n\n## Join Our Community\nTogether, we want to build data transformation without the waste. Connect with us in the following ways:\n\n* Join the [Tobiko Slack Community](https://tobikodata.com/slack) to ask questions, or just to say hi!\n* File an issue on our [GitHub](https://github.com/TobikoData/sqlmesh/issues/new)\n* Send us an email at [hello@tobikodata.com](mailto:hello@tobikodata.com) with your questions or feedback\n* Read our [blog](https://tobikodata.com/blog)\n\n## Contribution\nContributions in the form of issues or pull requests (from fork) are greatly appreciated. \n\n[Read more](https://sqlmesh.readthedocs.io/en/stable/development/) on how to contribute to SQLMesh open source.\n\n[Watch this video walkthrough](https://www.loom.com/share/2abd0d661c12459693fa155490633126?sid=b65c1c0f-8ef7-4036-ad19-3f85a3b87ff2) to see how our team contributes a feature to SQLMesh.\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": null,
"version": "0.141.1",
"project_urls": {
"Homepage": "https://github.com/TobikoData/sqlmesh"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4a100cd915e7d1ce029820361fd01e833543aca36fe68f33d653d201a7e3ace5",
"md5": "33385f3e68a8fbc1bf4d37dab99abaa8",
"sha256": "8fdcf3d0a871dd0862d1ac253f68c471b44d5eeef90dc3a28f77a7844e8b7737"
},
"downloads": -1,
"filename": "sqlmesh-0.141.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "33385f3e68a8fbc1bf4d37dab99abaa8",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 1890950,
"upload_time": "2024-12-20T23:14:07",
"upload_time_iso_8601": "2024-12-20T23:14:07.647534Z",
"url": "https://files.pythonhosted.org/packages/4a/10/0cd915e7d1ce029820361fd01e833543aca36fe68f33d653d201a7e3ace5/sqlmesh-0.141.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "dd0603590a53218e06b23ca143dcdac045c22f8c48369d1f09993e6b3c22129a",
"md5": "4b6a783821f8f62fc90f1390bf4e5b35",
"sha256": "b1b789d5f97301c381b1c92b1298e99e862172ca87c905ab39c569986b614717"
},
"downloads": -1,
"filename": "sqlmesh-0.141.1.tar.gz",
"has_sig": false,
"md5_digest": "4b6a783821f8f62fc90f1390bf4e5b35",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 29170385,
"upload_time": "2024-12-20T23:14:12",
"upload_time_iso_8601": "2024-12-20T23:14:12.672549Z",
"url": "https://files.pythonhosted.org/packages/dd/06/03590a53218e06b23ca143dcdac045c22f8c48369d1f09993e6b3c22129a/sqlmesh-0.141.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-20 23:14:12",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "TobikoData",
"github_project": "sqlmesh",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"circle": true,
"lcname": "sqlmesh"
}
JSON
