# Fabric BQ (BigQuery) Sync
This project is provided as an accelerator to help synchronize or migrate data from Google BigQuery to Fabric. The primary use cases for this accelerator are:
- BigQuery customers who wish to continue to leverage their existing investments and data estate while optimizing their PowerBI experience and reducing overall analytics TCO
- BigQuery customers who wish to migrate all or part of their data estate to Microsoft Fabric
# Getting Started
The accelerator includes an automated installer that can set up your Fabric workspace and install all required dependencies automatically. To use the installer:
1. Download the Installer Notebook
2. Import the installer into your Fabric Workspace
3. Attach the installer to a Lakehouse within the Workspace
4. Upload your GCP Service Account credential json file to OneLake
5. Update the configuration parameters:
- <code>loader_name</code> – custom name for the sync operation used in dashboards/reports (ex: HR Data Sync, BQ Sales Transaction Mirror)
- <code>metadata_lakehouse</code> - name of the lakehouse used to drive the BQ Sync process
- <code>target_lakehouse</code> - name of the lakehouse where the BQ Sync data will be landed
- <code>gcp_project_id</code> - the GCP billing project id that contains the in-scope dataset
- <code>gcp_dataset_id</code> - the target BQ dataset name/id
- <code>gcp_credential_path</code> - the File API Path to your JSON credential file (Example: /lakehouse/default/Files/my-credential-file.json")
6. Run the installer notebook
The installer performs the following actions:
- Create the required Lakehouses, if they do not exists
- Creates the metadata tables and required metadata
- Downloads the correct version of your BQ Spark connector based on your configured spark runtime
- Downloads the current BQ Sync python package
- Creates an initial default user configuration file based on your config parameters
- Installs a fully configured and ready to run BQ-Sync-Notebook into your workspace
# Project Overview
For many of our customers, the native mirroring capabilities in Fabric are one of the most exciting features of the platform. While Fabric currently supports a growing number of different mirroring sources, BigQuery is not yet supported. This current gap in capabilities is the foundation of this accelerator.
The goal of this accelerator is to simplify the process of synchronizing data from Google BigQuery to Microsoft Fabric with an emphasis on reliability, performance, and cost optimization. The accelerator is implemented using Spark (PySpark) using many concepts common to an ETL framework. The accelerator is more than just an ETL framework however in that it uses BigQuery metadata to solve for the most optimal way to synchronize data between the two platforms.
# Features & Capabilities
Within the accelerator there is an ever-growing set of capabilities that either offer feature parity or enhance & optimize the overall synchronization process. Below is an overview of some of the core capabilities:
- Multi-Project/Multi-Dataset sync support
- Table & Partition expiration based on BigQuery configuration
- Synching support for Views & Materialized Views
- Support for handling tables with required partition filters
- BigQuery connector configuration for alternative billing and materialization targets
- Ability to rename BigQuery tables and map to specific Lakehouse
- Complex-type (STRUCT/ARRAY) handling/flattening
- Support for Delta schema evolution for evolving BigQuery table/view schemas
- Support for Delta table options for Lakehouse performance/optimization
- Automatic Lakehouse table maintenance on synced tables
- Detailed process telemetry that tracks data movement and pairs with native Delta Time Travel capabilities
# Contributing
This project welcomes contributions and suggestions. Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
# Trademarks
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
trademarks or logos is subject to and must follow
[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
Any use of third-party trademarks or logos are subject to those third-party's policies.
Raw data
{
"_id": null,
"home_page": null,
"name": "FabricSync",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "Fabric, BigQuery, Data Sync, Mirror, ETL",
"author": null,
"author_email": "Microsoft GBBs North America <chriprice@microsoft.com>",
"download_url": null,
"platform": null,
"description": "# Fabric BQ (BigQuery) Sync\r\n\r\nThis project is provided as an accelerator to help synchronize or migrate data from Google BigQuery to Fabric. The primary use cases for this accelerator are:\r\n - BigQuery customers who wish to continue to leverage their existing investments and data estate while optimizing their PowerBI experience and reducing overall analytics TCO\r\n - BigQuery customers who wish to migrate all or part of their data estate to Microsoft Fabric\r\n\r\n# Getting Started\r\n\r\nThe accelerator includes an automated installer that can set up your Fabric workspace and install all required dependencies automatically. To use the installer:\r\n1. Download the Installer Notebook\r\n2. Import the installer into your Fabric Workspace\r\n3. Attach the installer to a Lakehouse within the Workspace\r\n4. Upload your GCP Service Account credential json file to OneLake\r\n5. Update the configuration parameters:\r\n\t- <code>loader_name</code> \u2013 custom name for the sync operation used in dashboards/reports (ex: HR Data Sync, BQ Sales Transaction Mirror)\r\n\t- <code>metadata_lakehouse</code> - name of the lakehouse used to drive the BQ Sync process\r\n\t- <code>target_lakehouse</code> - name of the lakehouse where the BQ Sync data will be landed\r\n\t- <code>gcp_project_id</code> - the GCP billing project id that contains the in-scope dataset\r\n\t- <code>gcp_dataset_id</code> - the target BQ dataset name/id\r\n\t- <code>gcp_credential_path</code> - the File API Path to your JSON credential file (Example: /lakehouse/default/Files/my-credential-file.json\")\r\n6. Run the installer notebook\r\n\r\nThe installer performs the following actions:\r\n- Create the required Lakehouses, if they do not exists\r\n- Creates the metadata tables and required metadata\r\n- Downloads the correct version of your BQ Spark connector based on your configured spark runtime\r\n- Downloads the current BQ Sync python package\r\n- Creates an initial default user configuration file based on your config parameters\r\n- Installs a fully configured and ready to run BQ-Sync-Notebook into your workspace\r\n\r\n# Project Overview\r\n\r\nFor many of our customers, the native mirroring capabilities in Fabric are one of the most exciting features of the platform. While Fabric currently supports a growing number of different mirroring sources, BigQuery is not yet supported. This current gap in capabilities is the foundation of this accelerator.\r\n\r\nThe goal of this accelerator is to simplify the process of synchronizing data from Google BigQuery to Microsoft Fabric with an emphasis on reliability, performance, and cost optimization. The accelerator is implemented using Spark (PySpark) using many concepts common to an ETL framework. The accelerator is more than just an ETL framework however in that it uses BigQuery metadata to solve for the most optimal way to synchronize data between the two platforms.\r\n\r\n# Features & Capabilities\r\n\r\nWithin the accelerator there is an ever-growing set of capabilities that either offer feature parity or enhance & optimize the overall synchronization process. Below is an overview of some of the core capabilities:\r\n- Multi-Project/Multi-Dataset sync support\r\n- Table & Partition expiration based on BigQuery configuration\r\n- Synching support for Views & Materialized Views\r\n- Support for handling tables with required partition filters\r\n- BigQuery connector configuration for alternative billing and materialization targets\r\n- Ability to rename BigQuery tables and map to specific Lakehouse\r\n- Complex-type (STRUCT/ARRAY) handling/flattening\r\n- Support for Delta schema evolution for evolving BigQuery table/view schemas\r\n- Support for Delta table options for Lakehouse performance/optimization\r\n- Automatic Lakehouse table maintenance on synced tables\r\n- Detailed process telemetry that tracks data movement and pairs with native Delta Time Travel capabilities\r\n\r\n# Contributing\r\n\r\nThis project welcomes contributions and suggestions. Most contributions require you to agree to a\r\nContributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us\r\nthe rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.\r\n\r\nWhen you submit a pull request, a CLA bot will automatically determine whether you need to provide\r\na CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions\r\nprovided by the bot. You will only need to do this once across all repos using our CLA.\r\n\r\nThis project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).\r\nFor more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or\r\ncontact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.\r\n\r\n# Trademarks\r\n\r\nThis project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft \r\ntrademarks or logos is subject to and must follow \r\n[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).\r\nUse of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.\r\nAny use of third-party trademarks or logos are subject to those third-party's policies.\r\n",
"bugtrack_url": null,
"license": "MIT License",
"summary": "Fabric BigQuery Data Sync Utility",
"version": "2.0.8",
"project_urls": {
"Documentation": "https://github.com/microsoft/FabricBQSync/blob/main/Docs/Home.md",
"Homepage": "https://github.com/microsoft/FabricBQSync",
"Issues": "https://github.com/microsoft/FabricBQSync/issues",
"Repository": "https://github.com/microsoft/FabricBQSync.git"
},
"split_keywords": [
"fabric",
" bigquery",
" data sync",
" mirror",
" etl"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "9a656c4275275d781621e041d9db4991dc9b10bf265407bbb99c9a9364a13f44",
"md5": "86a329baaba18d09217fd1032c4769b0",
"sha256": "eb0ce2f9b28248d2de02b4926b6c1277cb6a44e90bf98e629a550ede3171a462"
},
"downloads": -1,
"filename": "FabricSync-2.0.8-py3-none-any.whl",
"has_sig": false,
"md5_digest": "86a329baaba18d09217fd1032c4769b0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 284170,
"upload_time": "2025-01-10T17:33:46",
"upload_time_iso_8601": "2025-01-10T17:33:46.749111Z",
"url": "https://files.pythonhosted.org/packages/9a/65/6c4275275d781621e041d9db4991dc9b10bf265407bbb99c9a9364a13f44/FabricSync-2.0.8-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-10 17:33:46",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "microsoft",
"github_project": "FabricBQSync",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "pydantic",
"specs": []
},
{
"name": "mlflow-skinny",
"specs": []
},
{
"name": "google-cloud-bigquery",
"specs": []
},
{
"name": "PyGithub",
"specs": []
},
{
"name": "db-dtypes",
"specs": []
}
],
"lcname": "fabricsync"
}
JSON
