# bcdata
[](https://github.com/smnorris/bcdata/actions?query=workflow%3Atests)
[](https://pypi.python.org/pypi/bcdata/)
Python and command line tools for quick access to DataBC geo-data available via WFS/WCS.
There is a [wealth of British Columbia geographic information available as open
data](https://catalogue.data.gov.bc.ca/dataset?download_audience=Public),
but direct file download urls are not available and the syntax to accesss WFS via `ogr2ogr` and/or `curl/wget` can be awkward.
This tool attempts to simplify downloads of BC geographic data and smoothly integrate with PostGIS and Python GIS tools like `geopandas`, `fiona` and `rasterio`. As noted above, the module is an interface only to geographic data [avialable via WFS/WCS](https://bcgov.github.io/data-publication/pages/tips_tricks_webservices.html) - to work with other BC open data, use the [bcdata R package](https://github.com/bcgov/bcdata).
Note that this tool depends on BC Open Geospatial Web Services API at [openmaps.gov.bc.ca](https://openmaps.gov.bc.ca/geo/pub/ows?service=WFS&request=Getcapabilities) and the [BC Data Catalogue API](https://catalogue.data.gov.bc.ca/dataset/bc-data-catalogue-api). If downloads are failing, consult the [BCGov Data Systems and Services uptime monitor](https://uptime.com/statuspage/bcgov-dss).
**Disclaimer**
*It is the user's responsibility to check the licensing for any downloads, data are generally licensed as [OGL-BC](http://www2.gov.bc.ca/gov/content/governments/about-the-bc-government/databc/open-data/open-government-license-bc)*
## Dependencies and installation
`bcdata` requires `gdal`. If working with an OS such as linux or MacOS, with `gdal` already installed to a known location, install with `pip`:
$ pip install bcdata
Alternatively, `conda` can be used to install/manage the required dependencies as per the [GeoPandas guide](https://geopandas.org/en/stable/getting_started/install.html):
$ conda create --name bcdataenv
$ conda activate bcdataenv
$ conda config --env --add channels conda-forge
$ conda config --env --set channel_priority strict
$ conda install python=3 geopandas
$ conda install rasterio
$ pip install bcdata
### Configuration
#### Default PostgreSQL database
The default target database connection (used by `bc2pg`) can be set via the `DATABASE_URL` environment variable (the password parameter should not be required if using a [.pgpass file](https://www.postgresql.org/docs/current/libpq-pgpass.html))
Linux/Mac: `export DATABASE_URL=postgresql://{username}:{password}@{hostname}:{port}/{database}`
Windows: `SET DATABASE_URL=postgresql://{username}:{password}@{hostname}:{port}/{database}`
#### Layer list / layer schema cache
To reduce the volume of requests, information about data requested is cached locally.
Schemas of individual layers that have previously been requested are cached with the cache file name matching the object/table name.
Location of the cache defaults to `~/.bcdata` but can be modified by setting the `BCDATA_CACHE` environment variable:
`export BCDATA_CACHE=/path/to/bcdata_cache`
The cache will automatically refresh after 30 days - force a cache refresh by deleting the files in the cache or the entire cache folder.
## Usage
Typical usage will involve a manual search of the [DataBC Catalogue](https://catalogue.data.gov.bc.ca/dataset?download_audience=Public) to find a layer of interest. Once a dataset of interest is found, note the key with which to retreive it. This can be either the `id`/`package name` (the last portion of the url) or the `Object Name` (Under `Object Description`).
For example, for [BC Airports]( https://catalogue.data.gov.bc.ca/dataset/bc-airports), either of these keys will work:
- id/package name: `bc-airports`
- object name: `WHSE_IMAGERY_AND_BASE_MAPS.GSR_AIRPORTS_SVW`
Note that some packages [may have more than one layer](https://catalogue.data.gov.bc.ca/dataset/forest-development-units) - if you request a package like this, `bcdata` will prompt you with a list of valid object/table names to use instead of the package name.
### Python module
```python
import bcdata
# get a feature as geojson
geojson = bcdata.get_data(
'bc-airports',
query="AIRPORT_NAME='Terrace (Northwest Regional) Airport'"
)
geojson
{'type': 'FeatureCollection', 'features': [{'type': 'Feature', 'id': 'WHSE_IMAGERY_AND_BASE_MAPS.GSR_AIRPORTS_SVW.fid-f0cdbe4_16811fe142b_-6f34', 'geometry': {'type': 'Point', ...
# optionally, load data as a geopandas GeoDataFrame
gdf = bcdata.get_data(
'bc-airports',
query="AIRPORT_NAME='Terrace (Northwest Regional) Airport'",
as_gdf=True
)
gdf.head()
AERODROME_STATUS AIRCRAFT_ACCESS_IND AIRPORT_NAME ... TC_LID_CODE WEBSITE_URL geometry
0 Certified Y Terrace (Northwest Regional) Airport ... None None POINT (-128.5783333 54.46861111)
```
### CLI
Commands available via the bcdata command line interface are documented with the --help option
```
$ bcdata --help
Usage: bcdata [OPTIONS] COMMAND [ARGS]...
Options:
--version Show the version and exit.
--help Show this message and exit.
Commands:
bc2pg Load a DataBC WFS layer to a postgres db
cat Write DataBC features to stdout as GeoJSON feature objects.
dem Dump BC DEM to TIFF
dump Write DataBC features to stdout as GeoJSON feature collection.
info Print basic metadata about a DataBC WFS layer as JSON.
list List DataBC layers available via WFS
```
#### bc2pg
```
$ bcdata bc2pg --help
Usage: bcdata bc2pg [OPTIONS] DATASET
Load a DataBC WFS layer to a postgres db
$ bcdata bc2pg whse_imagery_and_base_maps.gsr_airports_svw
Options:
-db, --db_url TEXT Target database url, defaults to
$DATABASE_URL environment variable if set
--table TEXT Destination table name
--schema TEXT Destination schema name
--geometry_type TEXT Spatial type of geometry column
--query TEXT A valid CQL or ECQL query
--bounds TEXT Bounds: "left bottom right top" or "[left,
bottom, right, top]". Coordinates are BC
Albers (default) or --bounds_crs
--bounds-crs, --bounds_crs TEXT
CRS of provided bounds
-c, --count INTEGER Total number of features to load
-k, --primary_key TEXT Primary key of dataset
-s, --sortby TEXT Name of sort field
-e, --schema_only Create empty table from catalogue schema
-a, --append Append to existing table
-r, --refresh Truncate from existing table before load
-t, --no_timestamp Do not log download to bcdata.log
-v, --verbose Increase verbosity.
-q, --quiet Decrease verbosity.
--help Show this message and exit.
```
#### cat
```
$ bcdata cat --help
Usage: bcdata cat [OPTIONS] DATASET
Write DataBC features to stdout as GeoJSON feature objects.
Options:
--query TEXT A valid CQL or ECQL query
-c, --count INTEGER Number of features to request and dump
--bounds TEXT Bounds: "left bottom right top" or "[left,
bottom, right, top]". Coordinates are BC
Albers (default) or --bounds_crs
--bounds-crs, --bounds_crs TEXT
CRS of provided bounds
--indent INTEGER Indentation level for JSON output
--compact / --not-compact Use compact separators (',', ':').
--dst-crs, --dst_crs TEXT Destination CRS
-s, --sortby TEXT Name of sort field
-l, --lowercase Write column/properties names as lowercase
-m, --promote-to-multi Promote features to multipart
-v, --verbose Increase verbosity.
-q, --quiet Decrease verbosity.
--help Show this message and exit.
```
#### dem
```
$ bcdata dem --help
Usage: bcdata dem [OPTIONS]
Dump BC DEM to TIFF
Options:
-o, --out_file TEXT Output file
--bounds TEXT Bounds: "left bottom right top" or "[left,
bottom, right, top]". Coordinates are BC
Albers (default) or --bounds_crs [required]
--dst-crs TEXT CRS of output file
--bounds-crs TEXT CRS of provided bounds
-r, --resolution INTEGER
-a, --align Align provided bounds to provincial standard
-i, --interpolation [nearest|bilinear|bicubic]
-v, --verbose Increase verbosity.
-q, --quiet Decrease verbosity.
--help Show this message and exit.
```
#### dump
```
$ bcdata dump --help
Usage: bcdata dump [OPTIONS] DATASET
Write DataBC features to stdout as GeoJSON feature collection.
$ bcdata dump bc-airports
$ bcdata dump bc-airports --query "AIRPORT_NAME='Victoria Harbour (Shoal Point) Heliport'"
$ bcdata dump bc-airports --bounds xmin ymin xmax ymax
It can also be combined to read bounds of a feature dataset using Fiona:
$ bcdata dump bc-airports --bounds $(fio info aoi.shp --bounds)
Options:
--query TEXT A valid CQL or ECQL query
-c, --count INTEGER Number of features to request and dump
--bounds TEXT Bounds: "left bottom right top" or "[left,
bottom, right, top]". Coordinates are BC
Albers (default) or --bounds_crs
--bounds-crs, --bounds_crs TEXT
CRS of provided bounds
-s, --sortby TEXT Name of sort field
-l, --lowercase Write column/properties names as lowercase
-m, --promote-to-multi Promote features to multipart
-v, --verbose Increase verbosity.
-q, --quiet Decrease verbosity.
--help Show this message and exit.
```
#### info
```
$ bcdata info --help
Usage: bcdata info [OPTIONS] DATASET
Print basic metadata about a DataBC WFS layer as JSON.
Optionally print a single metadata item as a string.
Options:
--indent INTEGER Indentation level for JSON output
--count Print the count of features.
--name Print the table name of the dateset.
--description Print the description of the dateset.
-v, --verbose Increase verbosity.
-q, --quiet Decrease verbosity.
--help Show this message and exit.
```
#### list
```
$ bcdata list --help
Usage: bcdata list [OPTIONS]
List DataBC layers available via WFS
Options:
-r, --refresh Refresh the cached list
--help Show this message and exit.
```
#### CLI notes
Note that `bc2pg` creates `bcdata.log` and logs the most recent download date for each table downloaded.
Disable with the switch `--no_timestamp` if you do not wish to create this table.
Example of a record in `bcdata.log`:
```
mydb=# select * from bcdata.log;
table_name | latest_download
---------------------------------------------+-------------------------------
whse_imagery_and_base_maps.gsr_airports_svw | 2021-02-17 11:50:34.044481-08
```
#### CLI examples
Search the data listing for airports:
$ bcdata list | grep AIRPORTS
WHSE_IMAGERY_AND_BASE_MAPS.GSR_AIRPORTS_SVW
Describe a dataset. Note that if we know the id of a dataset, we can use that rather than the object name:
$ bcdata info bc-airports --indent 2
{
"name": "WHSE_IMAGERY_AND_BASE_MAPS.GSR_AIRPORTS_SVW",
"count": 455,
"schema": {
"properties": {
"CUSTODIAN_ORG_DESCRIPTION": "string",
"BUSINESS_CATEGORY_CLASS": "string",
"BUSINESS_CATEGORY_DESCRIPTION": "string",
"OCCUPANT_TYPE_DESCRIPTION": "string",
...etc...
},
"geometry": "GeometryCollection",
"geometry_column": "SHAPE"
}
}
The JSON output can be manipulated with [jq](https://stedolan.github.io/jq/). For example, to show only the fields available in the dataset:
$ bcdata info bc-airports | jq '.schema'
{
"CUSTODIAN_ORG_DESCRIPTION": "string",
"BUSINESS_CATEGORY_CLASS": "string",
"BUSINESS_CATEGORY_DESCRIPTION": "string",
"OCCUPANT_TYPE_DESCRIPTION": "string",
etc...
}
Dump data to geojson ([`EPSG:4326` only](https://tools.ietf.org/html/rfc7946#section-4)):
$ bcdata dump bc-airports > bc-airports.geojson
Get a single feature and send it to geojsonio (requires [geojson-cli](https://github.com/mapbox/geojsonio-cli)). Note the double quotes required around a CQL FILTER provided to the `--query` option.
$ bcdata dump \
WHSE_IMAGERY_AND_BASE_MAPS.GSR_AIRPORTS_SVW \
--query "AIRPORT_NAME='Terrace (Northwest Regional) Airport'" \
| geojsonio
Save a layer to a geopackage in BC Albers:
$ bcdata cat bc-airports --dst-crs EPSG:3005 \
| fio collect \
| fio load -f GPKG --dst-crs EPSG:3005 airports.gpkg
Note that this will not work if the source data has mixed geometry types.
Load to parquet on s3:
$ bcdata dump bc-airports \
| ogr2ogr -f Parquet /vsis3/$BUCKET/airports.parquet -s_srs EPSG:4326 -t_srs EPSG:3005 /vsistdin/
Load data to postgres and run a spatial query:
$ bcdata bc2pg bc-airports \
--db_url postgresql://postgres:postgres@localhost:5432/postgis
$ bcdata bc2pg WHSE_LEGAL_ADMIN_BOUNDARIES.ABMS_MUNICIPALITIES_SP \
--db_url postgresql://postgres:postgres@localhost:5432/postgis
$ psql -c \
"SELECT airport_name
FROM whse_imagery_and_base_maps.gsr_airports_svw a
INNER JOIN whse_legal_admin_boundaries.abms_municipalities_sp m
ON ST_Intersects(a.geom, m.geom)
WHERE admin_area_name LIKE '%Victoria%'"
airport_name
------------------------------------------------------------------
Victoria Harbour (Camel Point) Heliport
Victoria Inner Harbour Airport (Victoria Harbour Water Airport)
Victoria Harbour (Shoal Point) Heliport
(3 rows)
## Projections / CRS
**CLI**
`bcdata dump` returns GeoJSON in WGS84 (`EPSG:4326`).
`bcdata cat` provides the `--dst-crs` option, use any CRS the WFS server supports.
`bcdata bc2pg` loads data to PostgreSQL in BC Albers (`EPSG:3005`).
**Python module**
`bcdata.get_data()` defaults to `EPSG:4236` but any CRS can be specified (that the server will accept).
## Development and testing
`bc2pg` tests require database `postgresql://postgres@localhost:5432/test_bcdata` exists and has PostGIS installed:
psql -c "create database test_bcdata"
psql test_bcdata -c "create extension postgis"
Create virtualenv and install `bcdata` in development mode:
$ git clone git@github.com:smnorris/bcdata.git
$ cd bcdata
$ python -m venv .venv
$ source .venv/bin/activate
(.venv)$ pip install -e .[test]
(.venv)$ py.test
## Other implementations
- [bcdata R package](https://github.com/bcgov/bcdata) - official BC Open Data R client
- [OWSLib](https://github.com/geopython/OWSLib) - used internally by `bcdata`
- [pgsql-ogr-fdw](https://github.com/pramsey/pgsql-ogr-fdw) - read WFS data directly into with postgres foreign data wrapper
- general purpose command line tools [(`ogr2ogr`/`curl`/`wget`)](ogr2ogr.md)
Raw data
{
"_id": null,
"home_page": null,
"name": "bcdata",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": null,
"author": null,
"author_email": "Simon Norris <snorris@hillcrestgeo.ca>",
"download_url": "https://files.pythonhosted.org/packages/08/b1/30647150a8d8bfca1224ff375c3c4ff0ea21069c207bc47f3245ffab5756/bcdata-0.14.0.tar.gz",
"platform": null,
"description": "# bcdata\n\n[](https://github.com/smnorris/bcdata/actions?query=workflow%3Atests)\n[](https://pypi.python.org/pypi/bcdata/)\n\nPython and command line tools for quick access to DataBC geo-data available via WFS/WCS.\n\nThere is a [wealth of British Columbia geographic information available as open\ndata](https://catalogue.data.gov.bc.ca/dataset?download_audience=Public),\nbut direct file download urls are not available and the syntax to accesss WFS via `ogr2ogr` and/or `curl/wget` can be awkward.\n\nThis tool attempts to simplify downloads of BC geographic data and smoothly integrate with PostGIS and Python GIS tools like `geopandas`, `fiona` and `rasterio`. As noted above, the module is an interface only to geographic data [avialable via WFS/WCS](https://bcgov.github.io/data-publication/pages/tips_tricks_webservices.html) - to work with other BC open data, use the [bcdata R package](https://github.com/bcgov/bcdata).\n\nNote that this tool depends on BC Open Geospatial Web Services API at [openmaps.gov.bc.ca](https://openmaps.gov.bc.ca/geo/pub/ows?service=WFS&request=Getcapabilities) and the [BC Data Catalogue API](https://catalogue.data.gov.bc.ca/dataset/bc-data-catalogue-api). If downloads are failing, consult the [BCGov Data Systems and Services uptime monitor](https://uptime.com/statuspage/bcgov-dss).\n\n**Disclaimer** \n*It is the user's responsibility to check the licensing for any downloads, data are generally licensed as [OGL-BC](http://www2.gov.bc.ca/gov/content/governments/about-the-bc-government/databc/open-data/open-government-license-bc)*\n\n## Dependencies and installation\n\n`bcdata` requires `gdal`. If working with an OS such as linux or MacOS, with `gdal` already installed to a known location, install with `pip`:\n\n $ pip install bcdata\n\nAlternatively, `conda` can be used to install/manage the required dependencies as per the [GeoPandas guide](https://geopandas.org/en/stable/getting_started/install.html):\n\n $ conda create --name bcdataenv\n $ conda activate bcdataenv\n $ conda config --env --add channels conda-forge\n $ conda config --env --set channel_priority strict\n $ conda install python=3 geopandas\n $ conda install rasterio\n $ pip install bcdata\n\n\n### Configuration\n\n#### Default PostgreSQL database\n\nThe default target database connection (used by `bc2pg`) can be set via the `DATABASE_URL` environment variable (the password parameter should not be required if using a [.pgpass file](https://www.postgresql.org/docs/current/libpq-pgpass.html))\n\nLinux/Mac: `export DATABASE_URL=postgresql://{username}:{password}@{hostname}:{port}/{database}`\n\nWindows: `SET DATABASE_URL=postgresql://{username}:{password}@{hostname}:{port}/{database}`\n\n\n#### Layer list / layer schema cache\n\nTo reduce the volume of requests, information about data requested is cached locally. \nSchemas of individual layers that have previously been requested are cached with the cache file name matching the object/table name.\nLocation of the cache defaults to `~/.bcdata` but can be modified by setting the `BCDATA_CACHE` environment variable:\n\n`export BCDATA_CACHE=/path/to/bcdata_cache`\n\nThe cache will automatically refresh after 30 days - force a cache refresh by deleting the files in the cache or the entire cache folder.\n\n## Usage\n\nTypical usage will involve a manual search of the [DataBC Catalogue](https://catalogue.data.gov.bc.ca/dataset?download_audience=Public) to find a layer of interest. Once a dataset of interest is found, note the key with which to retreive it. This can be either the `id`/`package name` (the last portion of the url) or the `Object Name` (Under `Object Description`).\n\nFor example, for [BC Airports]( https://catalogue.data.gov.bc.ca/dataset/bc-airports), either of these keys will work:\n\n- id/package name: `bc-airports`\n- object name: `WHSE_IMAGERY_AND_BASE_MAPS.GSR_AIRPORTS_SVW`\n\nNote that some packages [may have more than one layer](https://catalogue.data.gov.bc.ca/dataset/forest-development-units) - if you request a package like this, `bcdata` will prompt you with a list of valid object/table names to use instead of the package name.\n\n\n### Python module\n\n```python\nimport bcdata\n\n# get a feature as geojson\ngeojson = bcdata.get_data(\n 'bc-airports',\n query=\"AIRPORT_NAME='Terrace (Northwest Regional) Airport'\"\n)\ngeojson\n{'type': 'FeatureCollection', 'features': [{'type': 'Feature', 'id': 'WHSE_IMAGERY_AND_BASE_MAPS.GSR_AIRPORTS_SVW.fid-f0cdbe4_16811fe142b_-6f34', 'geometry': {'type': 'Point', ...\n\n# optionally, load data as a geopandas GeoDataFrame\ngdf = bcdata.get_data(\n 'bc-airports',\n query=\"AIRPORT_NAME='Terrace (Northwest Regional) Airport'\",\n as_gdf=True\n)\ngdf.head()\nAERODROME_STATUS AIRCRAFT_ACCESS_IND AIRPORT_NAME ... TC_LID_CODE WEBSITE_URL geometry\n0 Certified Y Terrace (Northwest Regional) Airport ... None None POINT (-128.5783333 54.46861111)\n```\n\n### CLI\n\nCommands available via the bcdata command line interface are documented with the --help option\n\n```\n\n$ bcdata --help\n\nUsage: bcdata [OPTIONS] COMMAND [ARGS]...\n\nOptions:\n --version Show the version and exit.\n --help Show this message and exit.\n\nCommands:\n bc2pg Load a DataBC WFS layer to a postgres db\n cat Write DataBC features to stdout as GeoJSON feature objects.\n dem Dump BC DEM to TIFF\n dump Write DataBC features to stdout as GeoJSON feature collection.\n info Print basic metadata about a DataBC WFS layer as JSON.\n list List DataBC layers available via WFS\n```\n\n#### bc2pg\n\n```\n$ bcdata bc2pg --help\n\nUsage: bcdata bc2pg [OPTIONS] DATASET\n\n Load a DataBC WFS layer to a postgres db\n\n $ bcdata bc2pg whse_imagery_and_base_maps.gsr_airports_svw\n\nOptions:\n -db, --db_url TEXT Target database url, defaults to\n $DATABASE_URL environment variable if set\n --table TEXT Destination table name\n --schema TEXT Destination schema name\n --geometry_type TEXT Spatial type of geometry column\n --query TEXT A valid CQL or ECQL query\n --bounds TEXT Bounds: \"left bottom right top\" or \"[left,\n bottom, right, top]\". Coordinates are BC\n Albers (default) or --bounds_crs\n --bounds-crs, --bounds_crs TEXT\n CRS of provided bounds\n -c, --count INTEGER Total number of features to load\n -k, --primary_key TEXT Primary key of dataset\n -s, --sortby TEXT Name of sort field\n -e, --schema_only Create empty table from catalogue schema\n -a, --append Append to existing table\n -r, --refresh Truncate from existing table before load\n -t, --no_timestamp Do not log download to bcdata.log\n -v, --verbose Increase verbosity.\n -q, --quiet Decrease verbosity.\n --help Show this message and exit.\n```\n\n#### cat\n\n```\n$ bcdata cat --help\n\nUsage: bcdata cat [OPTIONS] DATASET\n\n Write DataBC features to stdout as GeoJSON feature objects.\n\nOptions:\n --query TEXT A valid CQL or ECQL query\n -c, --count INTEGER Number of features to request and dump\n --bounds TEXT Bounds: \"left bottom right top\" or \"[left,\n bottom, right, top]\". Coordinates are BC\n Albers (default) or --bounds_crs\n --bounds-crs, --bounds_crs TEXT\n CRS of provided bounds\n --indent INTEGER Indentation level for JSON output\n --compact / --not-compact Use compact separators (',', ':').\n --dst-crs, --dst_crs TEXT Destination CRS\n -s, --sortby TEXT Name of sort field\n -l, --lowercase Write column/properties names as lowercase\n -m, --promote-to-multi Promote features to multipart\n -v, --verbose Increase verbosity.\n -q, --quiet Decrease verbosity.\n --help Show this message and exit.\n```\n\n#### dem \n\n```\n$ bcdata dem --help\n\nUsage: bcdata dem [OPTIONS]\n\n Dump BC DEM to TIFF\n\nOptions:\n -o, --out_file TEXT Output file\n --bounds TEXT Bounds: \"left bottom right top\" or \"[left,\n bottom, right, top]\". Coordinates are BC\n Albers (default) or --bounds_crs [required]\n --dst-crs TEXT CRS of output file\n --bounds-crs TEXT CRS of provided bounds\n -r, --resolution INTEGER\n -a, --align Align provided bounds to provincial standard\n -i, --interpolation [nearest|bilinear|bicubic]\n -v, --verbose Increase verbosity.\n -q, --quiet Decrease verbosity.\n --help Show this message and exit.\n```\n\n#### dump \n\n```\n$ bcdata dump --help\n\nUsage: bcdata dump [OPTIONS] DATASET\n\n Write DataBC features to stdout as GeoJSON feature collection.\n\n $ bcdata dump bc-airports\n $ bcdata dump bc-airports --query \"AIRPORT_NAME='Victoria Harbour (Shoal Point) Heliport'\"\n $ bcdata dump bc-airports --bounds xmin ymin xmax ymax\n\n It can also be combined to read bounds of a feature dataset using Fiona: \b\n $ bcdata dump bc-airports --bounds $(fio info aoi.shp --bounds)\n\nOptions:\n --query TEXT A valid CQL or ECQL query\n -c, --count INTEGER Number of features to request and dump\n --bounds TEXT Bounds: \"left bottom right top\" or \"[left,\n bottom, right, top]\". Coordinates are BC\n Albers (default) or --bounds_crs\n --bounds-crs, --bounds_crs TEXT\n CRS of provided bounds\n -s, --sortby TEXT Name of sort field\n -l, --lowercase Write column/properties names as lowercase\n -m, --promote-to-multi Promote features to multipart\n -v, --verbose Increase verbosity.\n -q, --quiet Decrease verbosity.\n --help Show this message and exit.\n```\n\n#### info\n\n```\n$ bcdata info --help\n\nUsage: bcdata info [OPTIONS] DATASET\n\n Print basic metadata about a DataBC WFS layer as JSON.\n\n Optionally print a single metadata item as a string.\n\nOptions:\n --indent INTEGER Indentation level for JSON output\n --count Print the count of features.\n --name Print the table name of the dateset.\n --description Print the description of the dateset.\n -v, --verbose Increase verbosity.\n -q, --quiet Decrease verbosity.\n --help Show this message and exit.\n```\n\n#### list\n\n```\n$ bcdata list --help\n\nUsage: bcdata list [OPTIONS]\n\n List DataBC layers available via WFS\n\nOptions:\n -r, --refresh Refresh the cached list\n --help Show this message and exit.\n```\n\n#### CLI notes\n\nNote that `bc2pg` creates `bcdata.log` and logs the most recent download date for each table downloaded.\nDisable with the switch `--no_timestamp` if you do not wish to create this table.\n\nExample of a record in `bcdata.log`:\n\n```\nmydb=# select * from bcdata.log;\n table_name | latest_download\n---------------------------------------------+-------------------------------\n whse_imagery_and_base_maps.gsr_airports_svw | 2021-02-17 11:50:34.044481-08\n```\n\n\n#### CLI examples\n\nSearch the data listing for airports:\n\n $ bcdata list | grep AIRPORTS\n WHSE_IMAGERY_AND_BASE_MAPS.GSR_AIRPORTS_SVW\n\nDescribe a dataset. Note that if we know the id of a dataset, we can use that rather than the object name:\n\n $ bcdata info bc-airports --indent 2\n {\n \"name\": \"WHSE_IMAGERY_AND_BASE_MAPS.GSR_AIRPORTS_SVW\",\n \"count\": 455,\n \"schema\": {\n \"properties\": {\n \"CUSTODIAN_ORG_DESCRIPTION\": \"string\",\n \"BUSINESS_CATEGORY_CLASS\": \"string\",\n \"BUSINESS_CATEGORY_DESCRIPTION\": \"string\",\n \"OCCUPANT_TYPE_DESCRIPTION\": \"string\",\n ...etc...\n },\n \"geometry\": \"GeometryCollection\",\n \"geometry_column\": \"SHAPE\"\n }\n }\n\nThe JSON output can be manipulated with [jq](https://stedolan.github.io/jq/). For example, to show only the fields available in the dataset:\n\n $ bcdata info bc-airports | jq '.schema'\n {\n \"CUSTODIAN_ORG_DESCRIPTION\": \"string\",\n \"BUSINESS_CATEGORY_CLASS\": \"string\",\n \"BUSINESS_CATEGORY_DESCRIPTION\": \"string\",\n \"OCCUPANT_TYPE_DESCRIPTION\": \"string\",\n etc...\n }\n\nDump data to geojson ([`EPSG:4326` only](https://tools.ietf.org/html/rfc7946#section-4)):\n\n $ bcdata dump bc-airports > bc-airports.geojson\n\nGet a single feature and send it to geojsonio (requires [geojson-cli](https://github.com/mapbox/geojsonio-cli)). Note the double quotes required around a CQL FILTER provided to the `--query` option.\n\n $ bcdata dump \\\n WHSE_IMAGERY_AND_BASE_MAPS.GSR_AIRPORTS_SVW \\\n --query \"AIRPORT_NAME='Terrace (Northwest Regional) Airport'\" \\\n | geojsonio\n\nSave a layer to a geopackage in BC Albers:\n\n $ bcdata cat bc-airports --dst-crs EPSG:3005 \\\n | fio collect \\\n | fio load -f GPKG --dst-crs EPSG:3005 airports.gpkg\n\nNote that this will not work if the source data has mixed geometry types.\n\nLoad to parquet on s3:\n\n $ bcdata dump bc-airports \\\n | ogr2ogr -f Parquet /vsis3/$BUCKET/airports.parquet -s_srs EPSG:4326 -t_srs EPSG:3005 /vsistdin/\n\nLoad data to postgres and run a spatial query:\n\n $ bcdata bc2pg bc-airports \\\n --db_url postgresql://postgres:postgres@localhost:5432/postgis\n\n $ bcdata bc2pg WHSE_LEGAL_ADMIN_BOUNDARIES.ABMS_MUNICIPALITIES_SP \\\n --db_url postgresql://postgres:postgres@localhost:5432/postgis\n\n $ psql -c \\\n \"SELECT airport_name\n FROM whse_imagery_and_base_maps.gsr_airports_svw a\n INNER JOIN whse_legal_admin_boundaries.abms_municipalities_sp m\n ON ST_Intersects(a.geom, m.geom)\n WHERE admin_area_name LIKE '%Victoria%'\"\n airport_name\n ------------------------------------------------------------------\n Victoria Harbour (Camel Point) Heliport\n Victoria Inner Harbour Airport\u00a0(Victoria Harbour Water Airport)\n Victoria Harbour (Shoal Point) Heliport\n (3 rows)\n\n## Projections / CRS\n\n**CLI**\n\n`bcdata dump` returns GeoJSON in WGS84 (`EPSG:4326`).\n\n`bcdata cat` provides the `--dst-crs` option, use any CRS the WFS server supports.\n\n`bcdata bc2pg` loads data to PostgreSQL in BC Albers (`EPSG:3005`).\n\n\n**Python module**\n\n`bcdata.get_data()` defaults to `EPSG:4236` but any CRS can be specified (that the server will accept).\n\n\n## Development and testing\n\n`bc2pg` tests require database `postgresql://postgres@localhost:5432/test_bcdata` exists and has PostGIS installed:\n\n psql -c \"create database test_bcdata\"\n psql test_bcdata -c \"create extension postgis\"\n\nCreate virtualenv and install `bcdata` in development mode:\n\n $ git clone git@github.com:smnorris/bcdata.git\n $ cd bcdata\n $ python -m venv .venv\n $ source .venv/bin/activate\n (.venv)$ pip install -e .[test]\n (.venv)$ py.test\n\n\n## Other implementations\n- [bcdata R package](https://github.com/bcgov/bcdata) - official BC Open Data R client\n- [OWSLib](https://github.com/geopython/OWSLib) - used internally by `bcdata`\n- [pgsql-ogr-fdw](https://github.com/pramsey/pgsql-ogr-fdw) - read WFS data directly into with postgres foreign data wrapper\n- general purpose command line tools [(`ogr2ogr`/`curl`/`wget`)](ogr2ogr.md) \n",
"bugtrack_url": null,
"license": null,
"summary": "Download open data, monitor and report on changes",
"version": "0.14.0",
"project_urls": {
"Homepage": "https://github.com/smnorris/bcdata",
"Issues": "https://github.com/smnorris/bcdata"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "664be5e9e4115bdcdc778ccaf0b892663d7ae27e783223339a82ffc75e18c145",
"md5": "0c749a4b0c7262ce5583e067e480365e",
"sha256": "940c2a3c45d5f66f1d181957e15bec7a1d84bc430b13d8adeb92e17be1c9c81b"
},
"downloads": -1,
"filename": "bcdata-0.14.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0c749a4b0c7262ce5583e067e480365e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 25246,
"upload_time": "2024-12-17T17:55:37",
"upload_time_iso_8601": "2024-12-17T17:55:37.675148Z",
"url": "https://files.pythonhosted.org/packages/66/4b/e5e9e4115bdcdc778ccaf0b892663d7ae27e783223339a82ffc75e18c145/bcdata-0.14.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "08b130647150a8d8bfca1224ff375c3c4ff0ea21069c207bc47f3245ffab5756",
"md5": "74b6038dd296f6c25febc614b97bc063",
"sha256": "1b4bba62da945a054b70391a6d13c0fd1afe4b3526bfa9054693819b2c30497a"
},
"downloads": -1,
"filename": "bcdata-0.14.0.tar.gz",
"has_sig": false,
"md5_digest": "74b6038dd296f6c25febc614b97bc063",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 34494,
"upload_time": "2024-12-17T17:55:40",
"upload_time_iso_8601": "2024-12-17T17:55:40.563200Z",
"url": "https://files.pythonhosted.org/packages/08/b1/30647150a8d8bfca1224ff375c3c4ff0ea21069c207bc47f3245ffab5756/bcdata-0.14.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-17 17:55:40",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "smnorris",
"github_project": "bcdata",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "geoalchemy2",
"specs": [
[
">=",
"0.15"
]
]
},
{
"name": "geopandas",
"specs": [
[
">=",
"1.0"
]
]
},
{
"name": "owslib",
"specs": [
[
">=",
"0.31"
]
]
},
{
"name": "psycopg2-binary",
"specs": [
[
">=",
"2.9"
]
]
},
{
"name": "rasterio",
"specs": [
[
">=",
"1.3"
]
]
},
{
"name": "requests",
"specs": [
[
">=",
"2.32"
]
]
},
{
"name": "sqlalchemy",
"specs": [
[
">=",
"2.0"
]
]
},
{
"name": "stamina",
"specs": [
[
">=",
"24.2"
]
]
}
],
"lcname": "bcdata"
}
JSON
