# raster-loader
[](https://badge.fury.io/py/raster-loader)
[](https://pypistats.org/packages/raster-loader)
[](https://github.com/cartodb/raster-loader/actions)
[](https://raster-loader.readthedocs.io/en/latest/?badge=latest)
Python library for loading GIS raster data to standard cloud-based data warehouses that
don't natively support raster data.
Raster Loader is currently tested on Python 3.9, 3.10, 3.11 and 3.12.
## Documentation
The Raster Loader documentation is available at [raster-loader.readthedocs.io](https://raster-loader.readthedocs.io).
## Install
```bash
pip install -U raster-loader
pip install -U raster-loader"[bigquery]"
pip install -U raster-loader"[snowflake]"
```
### Installing from source
```bash
git clone https://github.com/cartodb/raster-loader
cd raster-loader
pip install .
```
## Usage
There are two ways you can use Raster Loader:
* Using the CLI by running `carto` in your terminal
* Using Raster Loader as a Python library (`import raster_loader`)
### CLI
After installing Raster Loader, you can run the CLI by typing `carto` in your terminal.
Currently, Raster Loader supports uploading raster data to [BigQuery](https://cloud.google.com/bigquery).
Accessing BigQuery with Raster Loader requires the
`GOOGLE_APPLICATION_CREDENTIALS` environment variable to be set to the path of a JSON
file containing your BigQuery credentials. See the
[GCP documentation](https://cloud.google.com/docs/authentication/provide-credentials-adc#local-key)
for more information.
Two commands are available:
#### Uploading to BigQuery
`carto bigquery upload` loads raster data from a local file to a BigQuery table.
At a minimum, the `carto bigquery upload` command requires a `file_path` to a local
raster file that can be [read by GDAL](https://gdal.org/drivers/raster/index.html) and processed with [rasterio](https://rasterio.readthedocs.io/en/latest/). It also requires
the `project` (the [GCP project name](https://cloud.google.com/resource-manager/docs/creating-managing-projects))
and `dataset` (the [BigQuery dataset name](https://cloud.google.com/bigquery/docs/datasets-intro))
parameters. There are also additional parameters, such as `table` ([BigQuery table
name](https://cloud.google.com/bigquery/docs/tables-intro)) and `overwrite` (to
overwrite existing data).
For example:
``` bash
carto bigquery upload \
--file_path /path/to/my/raster/file.tif \
--project my-gcp-project \
--dataset my-bigquery-dataset \
--table my-bigquery-table \
--overwrite
```
This command uploads the TIFF file from `/path/to/my/raster/file.tif` to a BigQuery
project named `my-gcp-project`, a dataset named `my-bigquery-dataset`, and a table
named `my-bigquery-table`. If the table already contains data, this data will be
overwritten because the `--overwrite` flag is set.
#### Inspecting a raster file on BigQuery
Use the `carto bigquery describe` command to retrieve information about a raster file
stored in a BigQuery table.
At a minimum, this command requires a
[GCP project name](https://cloud.google.com/resource-manager/docs/creating-managing-projects),
a [BigQuery dataset name](https://cloud.google.com/bigquery/docs/datasets-intro), and a
[BigQuery table name](https://cloud.google.com/bigquery/docs/tables-intro).
For example:
``` bash
carto bigquery describe \
--project my-gcp-project \
--dataset my-bigquery-dataset \
--table my-bigquery-table
```
### Using Raster Loader as a Python library
After installing Raster Loader, you can import the package into your Python project. For
example:
``` python
from raster_loader import rasterio_to_bigquery, bigquery_to_records
```
Currently, Raster Loader supports uploading raster data to [BigQuery](https://cloud.google.com/bigquery). Accessing BigQuery with Raster Loader requires the
`GOOGLE_APPLICATION_CREDENTIALS` environment variable to be set to the path of a JSON
file containing your BigQuery credentials. See the
[GCP documentation](https://cloud.google.com/docs/authentication/provide-credentials-adc#local-key)
for more information.
You can use Raster Loader to upload a local raster file to an existing
BigQuery table using the `rasterio_to_bigquery()` function:
``` python
rasterio_to_bigquery(
file_path = 'path/to/raster.tif',
project_id = 'my-project',
dataset_id = 'my_dataset',
table_id = 'my_table',
)
```
This function returns `True` if the upload was successful.
You can also access and inspect a raster file from a BigQuery table using the
`bigquery_to_records()` function:
``` python
records_df = bigquery_to_records(
project_id = 'my-project',
dataset_id = 'my_dataset',
table_id = 'my_table',
)
```
This function returns a DataFrame with some samples from the raster table on BigQuery
(10 rows by default).
## Development
See [CONTRIBUTING.md](CONTRIBUTING.md) for information on how to contribute to this
project.
[ROADMAP.md](ROADMAP.md) contains a list of features and improvements planned for future
versions of Raster Loader.
## Releasing
### 1. Create and merge a release PR updating the CHANGELOG
- Branch: `release/X.Y.Z`
- Title: `Release vX.Y.Z`
- Description: CHANGELOG release notes
Example:
```
## [0.7.0] - 2024-06-02
### Added
- Support raster overviews (#140)
### Enhancements
- increase chunk-size to 10000 (#142)
### Bug Fixes
- fix: make the gdalwarp examples consistent (#143)
```
### 2. Create and push a tag `vX.Y.Z`
This will trigger an automatic workflow that will publish the package at https://pypi.org/project/raster-loader.
### 3. Create the GitHub release
Go to the tags page (https://github.com/CartoDB/raster-loader/tags), select the release tag and click on "Create a new release"
- Title: `vX.Y.Z`
- Description: CHANGELOG release notes
Example:
```
### Added
- Support raster overviews (#140)
### Enhancements
- increase chunk-size to 10000 (#142)
### Bug Fixes
- fix: make the gdalwarp examples consistent (#143)
```
Raw data
{
"_id": null,
"home_page": "https://github.com/cartodb/raster-loader",
"name": "raster-loader",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "carto, raster, gis, data warehouse, bigquery, snowflake",
"author": "CARTO",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/05/a8/2df25414931b2d1f2583cb18eb61421c4b3f2066422581eb81fb1fd1d627/raster_loader-0.9.0.tar.gz",
"platform": null,
"description": "# raster-loader\n\n[](https://badge.fury.io/py/raster-loader)\n[](https://pypistats.org/packages/raster-loader)\n[](https://github.com/cartodb/raster-loader/actions)\n[](https://raster-loader.readthedocs.io/en/latest/?badge=latest)\n\nPython library for loading GIS raster data to standard cloud-based data warehouses that\ndon't natively support raster data.\n\nRaster Loader is currently tested on Python 3.9, 3.10, 3.11 and 3.12.\n\n## Documentation\n\nThe Raster Loader documentation is available at [raster-loader.readthedocs.io](https://raster-loader.readthedocs.io).\n\n## Install\n\n```bash\npip install -U raster-loader\n\npip install -U raster-loader\"[bigquery]\"\npip install -U raster-loader\"[snowflake]\"\n```\n\n### Installing from source\n\n```bash\ngit clone https://github.com/cartodb/raster-loader\ncd raster-loader\npip install .\n```\n\n## Usage\n\nThere are two ways you can use Raster Loader:\n\n* Using the CLI by running `carto` in your terminal\n* Using Raster Loader as a Python library (`import raster_loader`)\n\n### CLI\n\nAfter installing Raster Loader, you can run the CLI by typing `carto` in your terminal.\n\nCurrently, Raster Loader supports uploading raster data to [BigQuery](https://cloud.google.com/bigquery).\nAccessing BigQuery with Raster Loader requires the\n`GOOGLE_APPLICATION_CREDENTIALS` environment variable to be set to the path of a JSON\nfile containing your BigQuery credentials. See the\n[GCP documentation](https://cloud.google.com/docs/authentication/provide-credentials-adc#local-key)\nfor more information.\n\nTwo commands are available:\n\n#### Uploading to BigQuery\n\n`carto bigquery upload` loads raster data from a local file to a BigQuery table.\nAt a minimum, the `carto bigquery upload` command requires a `file_path` to a local\nraster file that can be [read by GDAL](https://gdal.org/drivers/raster/index.html) and processed with [rasterio](https://rasterio.readthedocs.io/en/latest/). It also requires\nthe `project` (the [GCP project name](https://cloud.google.com/resource-manager/docs/creating-managing-projects))\nand `dataset` (the [BigQuery dataset name](https://cloud.google.com/bigquery/docs/datasets-intro))\nparameters. There are also additional parameters, such as `table` ([BigQuery table\nname](https://cloud.google.com/bigquery/docs/tables-intro)) and `overwrite` (to\noverwrite existing data).\n\nFor example:\n\n``` bash\n\ncarto bigquery upload \\\n --file_path /path/to/my/raster/file.tif \\\n --project my-gcp-project \\\n --dataset my-bigquery-dataset \\\n --table my-bigquery-table \\\n --overwrite\n\n```\n\nThis command uploads the TIFF file from `/path/to/my/raster/file.tif` to a BigQuery\nproject named `my-gcp-project`, a dataset named `my-bigquery-dataset`, and a table\nnamed `my-bigquery-table`. If the table already contains data, this data will be\noverwritten because the `--overwrite` flag is set.\n\n#### Inspecting a raster file on BigQuery\n\nUse the `carto bigquery describe` command to retrieve information about a raster file\nstored in a BigQuery table.\n\nAt a minimum, this command requires a\n[GCP project name](https://cloud.google.com/resource-manager/docs/creating-managing-projects),\na [BigQuery dataset name](https://cloud.google.com/bigquery/docs/datasets-intro), and a\n[BigQuery table name](https://cloud.google.com/bigquery/docs/tables-intro).\n\nFor example:\n\n``` bash\ncarto bigquery describe \\\n --project my-gcp-project \\\n --dataset my-bigquery-dataset \\\n --table my-bigquery-table\n```\n\n### Using Raster Loader as a Python library\n\nAfter installing Raster Loader, you can import the package into your Python project. For\nexample:\n\n``` python\nfrom raster_loader import rasterio_to_bigquery, bigquery_to_records\n```\n\nCurrently, Raster Loader supports uploading raster data to [BigQuery](https://cloud.google.com/bigquery). Accessing BigQuery with Raster Loader requires the\n`GOOGLE_APPLICATION_CREDENTIALS` environment variable to be set to the path of a JSON\nfile containing your BigQuery credentials. See the\n[GCP documentation](https://cloud.google.com/docs/authentication/provide-credentials-adc#local-key)\nfor more information.\n\nYou can use Raster Loader to upload a local raster file to an existing\nBigQuery table using the `rasterio_to_bigquery()` function:\n\n``` python\nrasterio_to_bigquery(\n file_path = 'path/to/raster.tif',\n project_id = 'my-project',\n dataset_id = 'my_dataset',\n table_id = 'my_table',\n)\n```\n\nThis function returns `True` if the upload was successful.\n\nYou can also access and inspect a raster file from a BigQuery table using the\n`bigquery_to_records()` function:\n\n``` python\nrecords_df = bigquery_to_records(\n project_id = 'my-project',\n dataset_id = 'my_dataset',\n table_id = 'my_table',\n)\n```\n\nThis function returns a DataFrame with some samples from the raster table on BigQuery\n(10 rows by default).\n\n## Development\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for information on how to contribute to this\nproject.\n\n[ROADMAP.md](ROADMAP.md) contains a list of features and improvements planned for future\nversions of Raster Loader.\n\n## Releasing\n\n### 1. Create and merge a release PR updating the CHANGELOG\n\n- Branch: `release/X.Y.Z`\n- Title: `Release vX.Y.Z`\n- Description: CHANGELOG release notes\n\nExample:\n```\n## [0.7.0] - 2024-06-02\n\n### Added\n- Support raster overviews (#140)\n\n### Enhancements\n- increase chunk-size to 10000 (#142)\n\n### Bug Fixes\n- fix: make the gdalwarp examples consistent (#143)\n```\n\n### 2. Create and push a tag `vX.Y.Z`\n\nThis will trigger an automatic workflow that will publish the package at https://pypi.org/project/raster-loader.\n\n### 3. Create the GitHub release\n\nGo to the tags page (https://github.com/CartoDB/raster-loader/tags), select the release tag and click on \"Create a new release\"\n\n- Title: `vX.Y.Z`\n- Description: CHANGELOG release notes\n\nExample:\n```\n### Added\n- Support raster overviews (#140)\n\n### Enhancements\n- increase chunk-size to 10000 (#142)\n\n### Bug Fixes\n- fix: make the gdalwarp examples consistent (#143)\n```\n",
"bugtrack_url": null,
"license": "BSD 3-Clause",
"summary": "Python library for loading GIS raster data to standard cloud-based data warehouses that don't natively support raster data.",
"version": "0.9.0",
"project_urls": {
"Homepage": "https://github.com/cartodb/raster-loader"
},
"split_keywords": [
"carto",
" raster",
" gis",
" data warehouse",
" bigquery",
" snowflake"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "abba1d3d4d227ed76e72f360ee7de8426e3e4aa6537204daf6aecca21bd31a22",
"md5": "795af1e4d9b99548183523483812c4fa",
"sha256": "083aaf761f85847fa45d4139fbbcd51661d3d8c0d60cadf09fec7cfd05437f09"
},
"downloads": -1,
"filename": "raster_loader-0.9.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "795af1e4d9b99548183523483812c4fa",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 38810,
"upload_time": "2024-11-04T07:08:10",
"upload_time_iso_8601": "2024-11-04T07:08:10.260016Z",
"url": "https://files.pythonhosted.org/packages/ab/ba/1d3d4d227ed76e72f360ee7de8426e3e4aa6537204daf6aecca21bd31a22/raster_loader-0.9.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "05a82df25414931b2d1f2583cb18eb61421c4b3f2066422581eb81fb1fd1d627",
"md5": "3559b8fc9f72dd470031bd4cf11f82bc",
"sha256": "86fee432aef758e3c10cbf9a80a6cfb7ca4c75be84d9b43f450707f5228dcc30"
},
"downloads": -1,
"filename": "raster_loader-0.9.0.tar.gz",
"has_sig": false,
"md5_digest": "3559b8fc9f72dd470031bd4cf11f82bc",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 2136824,
"upload_time": "2024-11-04T07:08:11",
"upload_time_iso_8601": "2024-11-04T07:08:11.292871Z",
"url": "https://files.pythonhosted.org/packages/05/a8/2df25414931b2d1f2583cb18eb61421c4b3f2066422581eb81fb1fd1d627/raster_loader-0.9.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-04 07:08:11",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "cartodb",
"github_project": "raster-loader",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"requirements": [],
"lcname": "raster-loader"
}
JSON
