# raster-loader
[](https://badge.fury.io/py/raster-loader)
[](https://pypistats.org/packages/raster-loader)
[](https://github.com/cartodb/raster-loader/actions)
[](https://raster-loader.readthedocs.io/en/latest/?badge=latest)
Python library for loading GIS raster data to standard cloud-based data warehouses that
don't natively support raster data.
Raster Loader is currently tested on Python 3.8, 3.9, 3.10, and 3.11.
## Documentation
The Raster Loader documentation is available at [raster-loader.readthedocs.io](https://raster-loader.readthedocs.io).
## Install
```bash
pip install -U raster-loader
pip install -U raster-loader"[bigquery]"
pip install -U raster-loader"[snowflake]"
```
### Installing from source
```bash
git clone https://github.com/cartodb/raster-loader
cd raster-loader
pip install .
```
## Usage
There are two ways you can use Raster Loader:
* Using the CLI by running `carto` in your terminal
* Using Raster Loader as a Python library (`import raster_loader`)
### CLI
After installing Raster Loader, you can run the CLI by typing `carto` in your terminal.
Currently, Raster Loader supports uploading raster data to [BigQuery](https://cloud.google.com/bigquery).
Accessing BigQuery with Raster Loader requires the
`GOOGLE_APPLICATION_CREDENTIALS` environment variable to be set to the path of a JSON
file containing your BigQuery credentials. See the
[GCP documentation](https://cloud.google.com/docs/authentication/provide-credentials-adc#local-key)
for more information.
Two commands are available:
#### Uploading to BigQuery
`carto bigquery upload` loads raster data from a local file to a BigQuery table.
At a minimum, the `carto bigquery upload` command requires a `file_path` to a local
raster file that can be [read by GDAL](https://gdal.org/drivers/raster/index.html) and processed with [rasterio](https://rasterio.readthedocs.io/en/latest/). It also requires
the `project` (the [GCP project name](https://cloud.google.com/resource-manager/docs/creating-managing-projects))
and `dataset` (the [BigQuery dataset name](https://cloud.google.com/bigquery/docs/datasets-intro))
parameters. There are also additional parameters, such as `table` ([BigQuery table
name](https://cloud.google.com/bigquery/docs/tables-intro)) and `overwrite` (to
overwrite existing data).
For example:
``` bash
carto bigquery upload \
--file_path /path/to/my/raster/file.tif \
--project my-gcp-project \
--dataset my-bigquery-dataset \
--table my-bigquery-table \
--overwrite
```
This command uploads the TIFF file from `/path/to/my/raster/file.tif` to a BigQuery
project named `my-gcp-project`, a dataset named `my-bigquery-dataset`, and a table
named `my-bigquery-table`. If the table already contains data, this data will be
overwritten because the `--overwrite` flag is set.
#### Inspecting a raster file on BigQuery
Use the `carto bigquery describe` command to retrieve information about a raster file
stored in a BigQuery table.
At a minimum, this command requires a
[GCP project name](https://cloud.google.com/resource-manager/docs/creating-managing-projects),
a [BigQuery dataset name](https://cloud.google.com/bigquery/docs/datasets-intro), and a
[BigQuery table name](https://cloud.google.com/bigquery/docs/tables-intro).
For example:
``` bash
carto bigquery describe \
--project my-gcp-project \
--dataset my-bigquery-dataset \
--table my-bigquery-table
```
### Using Raster Loader as a Python library
After installing Raster Loader, you can import the package into your Python project. For
example:
``` python
from raster_loader import rasterio_to_bigquery, bigquery_to_records
```
Currently, Raster Loader supports uploading raster data to [BigQuery](https://cloud.google.com/bigquery). Accessing BigQuery with Raster Loader requires the
`GOOGLE_APPLICATION_CREDENTIALS` environment variable to be set to the path of a JSON
file containing your BigQuery credentials. See the
[GCP documentation](https://cloud.google.com/docs/authentication/provide-credentials-adc#local-key)
for more information.
You can use Raster Loader to upload a local raster file to an existing
BigQuery table using the `rasterio_to_bigquery()` function:
``` python
rasterio_to_bigquery(
file_path = 'path/to/raster.tif',
project_id = 'my-project',
dataset_id = 'my_dataset',
table_id = 'my_table',
)
```
This function returns `True` if the upload was successful.
You can also access and inspect a raster file from a BigQuery table using the
`bigquery_to_records()` function:
``` python
records_df = bigquery_to_records(
project_id = 'my-project',
dataset_id = 'my_dataset',
table_id = 'my_table',
)
```
This function returns a DataFrame with some samples from the raster table on BigQuery
(10 rows by default).
## Development
See [CONTRIBUTING.md](CONTRIBUTING.md) for information on how to contribute to this
project.
[ROADMAP.md](ROADMAP.md) contains a list of features and improvements planned for future
versions of Raster Loader.
Raw data
{
"_id": null,
"home_page": "https://github.com/cartodb/raster-loader",
"name": "raster-loader",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "carto, raster, gis, data warehouse, bigquery, snowflake",
"author": "CARTO",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/04/81/e84c8fef5881b5d0873f6d7dcb9476944e3777f7ad8d90717c2b27e5c2e3/raster-loader-0.6.1.tar.gz",
"platform": null,
"description": "# raster-loader\n\n[](https://badge.fury.io/py/raster-loader)\n[](https://pypistats.org/packages/raster-loader)\n[](https://github.com/cartodb/raster-loader/actions)\n[](https://raster-loader.readthedocs.io/en/latest/?badge=latest)\n\nPython library for loading GIS raster data to standard cloud-based data warehouses that\ndon't natively support raster data.\n\nRaster Loader is currently tested on Python 3.8, 3.9, 3.10, and 3.11.\n\n## Documentation\n\nThe Raster Loader documentation is available at [raster-loader.readthedocs.io](https://raster-loader.readthedocs.io).\n\n## Install\n\n```bash\npip install -U raster-loader\n\npip install -U raster-loader\"[bigquery]\"\npip install -U raster-loader\"[snowflake]\"\n```\n\n### Installing from source\n\n```bash\ngit clone https://github.com/cartodb/raster-loader\ncd raster-loader\npip install .\n```\n\n## Usage\n\nThere are two ways you can use Raster Loader:\n\n* Using the CLI by running `carto` in your terminal\n* Using Raster Loader as a Python library (`import raster_loader`)\n\n### CLI\n\nAfter installing Raster Loader, you can run the CLI by typing `carto` in your terminal.\n\nCurrently, Raster Loader supports uploading raster data to [BigQuery](https://cloud.google.com/bigquery).\nAccessing BigQuery with Raster Loader requires the\n`GOOGLE_APPLICATION_CREDENTIALS` environment variable to be set to the path of a JSON\nfile containing your BigQuery credentials. See the\n[GCP documentation](https://cloud.google.com/docs/authentication/provide-credentials-adc#local-key)\nfor more information.\n\nTwo commands are available:\n\n#### Uploading to BigQuery\n\n`carto bigquery upload` loads raster data from a local file to a BigQuery table.\nAt a minimum, the `carto bigquery upload` command requires a `file_path` to a local\nraster file that can be [read by GDAL](https://gdal.org/drivers/raster/index.html) and processed with [rasterio](https://rasterio.readthedocs.io/en/latest/). It also requires\nthe `project` (the [GCP project name](https://cloud.google.com/resource-manager/docs/creating-managing-projects))\nand `dataset` (the [BigQuery dataset name](https://cloud.google.com/bigquery/docs/datasets-intro))\nparameters. There are also additional parameters, such as `table` ([BigQuery table\nname](https://cloud.google.com/bigquery/docs/tables-intro)) and `overwrite` (to\noverwrite existing data).\n\nFor example:\n\n``` bash\n\ncarto bigquery upload \\\n --file_path /path/to/my/raster/file.tif \\\n --project my-gcp-project \\\n --dataset my-bigquery-dataset \\\n --table my-bigquery-table \\\n --overwrite\n\n```\n\nThis command uploads the TIFF file from `/path/to/my/raster/file.tif` to a BigQuery\nproject named `my-gcp-project`, a dataset named `my-bigquery-dataset`, and a table\nnamed `my-bigquery-table`. If the table already contains data, this data will be\noverwritten because the `--overwrite` flag is set.\n\n#### Inspecting a raster file on BigQuery\n\nUse the `carto bigquery describe` command to retrieve information about a raster file\nstored in a BigQuery table.\n\nAt a minimum, this command requires a\n[GCP project name](https://cloud.google.com/resource-manager/docs/creating-managing-projects),\na [BigQuery dataset name](https://cloud.google.com/bigquery/docs/datasets-intro), and a\n[BigQuery table name](https://cloud.google.com/bigquery/docs/tables-intro).\n\nFor example:\n\n``` bash\ncarto bigquery describe \\\n --project my-gcp-project \\\n --dataset my-bigquery-dataset \\\n --table my-bigquery-table\n```\n\n### Using Raster Loader as a Python library\n\nAfter installing Raster Loader, you can import the package into your Python project. For\nexample:\n\n``` python\nfrom raster_loader import rasterio_to_bigquery, bigquery_to_records\n```\n\nCurrently, Raster Loader supports uploading raster data to [BigQuery](https://cloud.google.com/bigquery). Accessing BigQuery with Raster Loader requires the\n`GOOGLE_APPLICATION_CREDENTIALS` environment variable to be set to the path of a JSON\nfile containing your BigQuery credentials. See the\n[GCP documentation](https://cloud.google.com/docs/authentication/provide-credentials-adc#local-key)\nfor more information.\n\nYou can use Raster Loader to upload a local raster file to an existing\nBigQuery table using the `rasterio_to_bigquery()` function:\n\n``` python\nrasterio_to_bigquery(\n file_path = 'path/to/raster.tif',\n project_id = 'my-project',\n dataset_id = 'my_dataset',\n table_id = 'my_table',\n)\n```\n\nThis function returns `True` if the upload was successful.\n\nYou can also access and inspect a raster file from a BigQuery table using the\n`bigquery_to_records()` function:\n\n``` python\nrecords_df = bigquery_to_records(\n project_id = 'my-project',\n dataset_id = 'my_dataset',\n table_id = 'my_table',\n)\n```\n\nThis function returns a DataFrame with some samples from the raster table on BigQuery\n(10 rows by default).\n\n## Development\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for information on how to contribute to this\nproject.\n\n[ROADMAP.md](ROADMAP.md) contains a list of features and improvements planned for future\nversions of Raster Loader.\n",
"bugtrack_url": null,
"license": "BSD 3-Clause",
"summary": "Python library for loading GIS raster data to standard cloud-based data warehouses that don't natively support raster data.",
"version": "0.6.1",
"project_urls": {
"Homepage": "https://github.com/cartodb/raster-loader"
},
"split_keywords": [
"carto",
" raster",
" gis",
" data warehouse",
" bigquery",
" snowflake"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "0ab518a3b628fb4037af88690ffac5e4ade525701f4f1a8a519d074c312a977f",
"md5": "f41aff4427b68b1efe4b1d1bf3b6bb79",
"sha256": "4ba91d37172e25260c95836268ce495dba7d1dd64111d83a385a303d1a456bcb"
},
"downloads": -1,
"filename": "raster_loader-0.6.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f41aff4427b68b1efe4b1d1bf3b6bb79",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 31347,
"upload_time": "2024-04-02T16:10:03",
"upload_time_iso_8601": "2024-04-02T16:10:03.548831Z",
"url": "https://files.pythonhosted.org/packages/0a/b5/18a3b628fb4037af88690ffac5e4ade525701f4f1a8a519d074c312a977f/raster_loader-0.6.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "0481e84c8fef5881b5d0873f6d7dcb9476944e3777f7ad8d90717c2b27e5c2e3",
"md5": "ffa2db93e82a9687336a0130447e140b",
"sha256": "15d255bc978df8a1e8f4d9c3a297bf566d1e06a7e45853bdd3a553bc1eb5028d"
},
"downloads": -1,
"filename": "raster-loader-0.6.1.tar.gz",
"has_sig": false,
"md5_digest": "ffa2db93e82a9687336a0130447e140b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 2129380,
"upload_time": "2024-04-02T16:10:05",
"upload_time_iso_8601": "2024-04-02T16:10:05.018064Z",
"url": "https://files.pythonhosted.org/packages/04/81/e84c8fef5881b5d0873f6d7dcb9476944e3777f7ad8d90717c2b27e5c2e3/raster-loader-0.6.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-02 16:10:05",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "cartodb",
"github_project": "raster-loader",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"requirements": [],
"lcname": "raster-loader"
}
JSON
