# Python SpatiaFi API
Python library for interacting with the SpatiaFi API.
Also included is `gdal_auth` which is a CLI tool to help with GCP authentication for GDAL.
## Quickstart
### Install the Package
```shell
pip install spatiafi
```
### Get Authenticated Session
```python
from spatiafi import get_session
session = get_session()
# The `session` object works just like `requests` but will automatically
# refresh the authentication token when it expires
params = {"item_id": "wildfire-risk-current-global-v1.0"}
url = "https://api.spatiafi.com/api/info"
response = session.get(url, params=params)
```
### Using a proxy
If you're using the client in a constrained environment you may need to pass in
`proxies` to the `get_session` function:
```python
from spatiafi import get_session
proxies = {
"http": "http://fake-proxy.example.com:443",
"https": "http://fake-proxy.example.com:443",
}
session = get_session(proxies=proxies)
params = {"item_id": "wildfire-risk-current-global-v1.0"}
url = "https://api.spatiafi.com/api/info"
response = session.get(url, params=params)
```
### Get help with `gdal_auth`
```shell
gdal_auth --help
```
---
## GDAL Authentication
`gdal_auth` is a CLI tool to help with GCP authentication for GDAL.
This command can be used in three ways:
1. Set the environment variables in a file that can be sourced to set the
environment variables:
```
gdal_auth --file
source /tmp/gdal_auth.env
```
2. Print a command that can be run to set the environment
variables once:
```
gdal_auth --line
```
Running this command will output a line that you will need to copy and run.
This will set the GDAL environment variables for anything run afterward in
the same terminal session. (e.g. run the command and then _in the command line_
run another command like `qgis`)
3. Print instructions for setting up aliases. These aliases allow `gdal_*`
commands to be run as normal and authentication will be handled
automatically:
```
gdal_auth --alias
```
**Note**: for all options except `--alias`, the authentication will eventually
expire. This is because the tokens generated by the Google Cloud SDK expire
after 1 hour. The aliases will automatically refresh the authentication
tokens when they expire.
---
## Mini-Batch (Async Queue)
The `AsyncQueue` class is a helper class for running many (up to ~1 million) API queries in parallel.
To use it, you must:
1. Create a task function
2. Create an `AsyncQueue` object
3. Enqueue tasks
4. Fetch results
**tl;dr:** See [`tests/test_async_queue.py`]( tests/test_async_queue.py) for an example.
### Create a AsyncQueue Task
A valid AsyncQueue task must:
* Be an async function
* Take a single argument
* Take an optional *async* session argument (if not provided, an async session will be created)
* Return a single, serializable object (a `dict` is recommended)
If your task function requires multiple arguments, you can:
* Use a wrapper function or closure (may not work on Windows or 'spawn' multiprocessing)
* Create a new function using `functools.partial` (as shown in [`tests/test_async_queue.py`]( tests/test_async_queue.py))
* Pass a tuple as the argument and unpack it in the task function e.g.
```python
async def task(args, session=None):
arg1, arg2, arg3 = args
...
with AsyncQueue(task) as async_queue:
async_queue.enqueue((arg1, arg2, arg3))
```
#### Example Task Function
```python
from spatiafi.async_queue import AsyncQueue
from spatiafi.session import get_async_session
async def get_point(point, session=None):
"""
Get a point from the SpatiaFI API.
"""
# Unpack the `point` tuple because we can only pass
# a single argument to the task function.
lon, lat = point
# Create an async session if one is not provided.
if session is None:
session = await get_async_session()
# Create the url.
url = (
"https://api.spatiafi.com/api/point/" + str(lon) + "," + str(lat)
)
params = {"item_id": "wildfire-risk-current-global-v1.0"}
r = await session.get(url, params=params)
# We want to raise for all errors except 400 (bad request)
if not (r.status_code == 200 or r.status_code == 400):
r.raise_for_status()
return r.json()
```
### Create an AsyncQueue and Enqueue Tasks
`AsyncQueue` takes a task function as an argument, and launches multiple instances of that task in parallel.
The `AsyncQueue.enqueue` method takes a _single argument_ is used to add tasks to the queue.
The `AsyncQueue.results` property will return a list of results in the order they were enqueued.
When starting the `AsyncQueue`, it is **highly recommended** that you specify the number of workers/CPUs to use
using the `n_cores` argument. The default is to use the minimum of 4 and the number of CPUs on the machine.
This queue is designed to be used with the `with` statement. Entering the `with` statement will start the
subprocess and event loop. Exiting the `with` statement will wait for all tasks to finish and then stop the
event loop and subprocess.
For example:
```python
from spatiafi.async_queue import AsyncQueue
with AsyncQueue(get_point) as async_queue:
for _, row in df.iterrows():
async_queue.enqueue((row["lon"], row["lat"]))
results = async_queue.results
```
Alternatively, you can use the `start` and `stop` methods:
```python
from spatiafi.async_queue import AsyncQueue
async_queue = AsyncQueue(get_point)
async_queue.start()
for _, row in df.iterrows():
async_queue.enqueue((row["lon"], row["lat"]))
async_queue.stop()
results = async_queue.results
```
---
## Development
### Use a Virtual Environment
Development should be done in a virtual environment.
It is recommended to use the virtual environment manager built into PyCharm.
To create a new virtual environment:
* Open the project in PyCharm and select `File > Settings > Project: spfi-api > Python Interpreter`.
* In the top right corner of the window, click the gear icon and select `Add Interpreter > Add Local Interpreter...`
### Mark `src` as a Source Root
In PyCharm, mark the `src` folder as a source root. This will allow you to import modules from the `src` folder without using relative imports.
Right-click on the `src` folder and select `Mark Directory as > Sources Root`.
### Bootstrap the Development Environment
Run `./scripts/bootstrap_dev.sh` to install the package and development dependencies.
This will also set up access to our private PyPI server, generate the first `requirements.txt` (if required),
and install `pre-commit` hooks.
**Protip:** This script can be run at any time if you're afraid you've messed up your environment.
### Running the tests
Tests can be run locally via the `scripts/test.sh` script:
```
./scripts/test.sh
```
All additional arguments to that script will be passed to PyTest which allows
you to do things such as run a single test:
```
./scripts/test.sh -k test_async_queue
```
### Manage Dependencies in `setup.cfg`
Dependencies are managed in `setup.cfg` using the `install_requires` and `extras_require` sections.
To add a new dependency:
1. Install the package in the virtual environment with `pip install <package_name>`
(Hint: use the terminal built in to PyCharm)
2. Run `pip show <package_name>` to get the package name and version
3. Add the package name *and version* to `setup.cfg` in the `install_requires` section.
Use the [compatible release](https://www.python.org/dev/peps/pep-0440/#compatible-release) syntax
`package_name ~=version`.
**DO NOT** add the package to the `requirements.txt` file. This file is automatically generated by
`scripts/gen_requirements.sh`.
If the dependency is only needed for development, add it to the `dev` section of `extras_require` in `setup.cfg`.
### Building Docker Images Locally
**tl;dr:** run `./scripts/build_docker.sh`.
We need to inject a GCP access token into the Docker build to access private PyPI packages.
This requires using BuildKit (enabled by default in recent versions of Docker), and passing the token as a build
argument.
## `pre-commit` Hooks
This project uses `pre-commit` to run a series of checks before each git commit.
To install the `pre-commit` hooks, run `pre-commit install` in the virtual environment.
(This is done automatically by `./scripts/bootstrap_dev.sh`)
To format all your code manually, run `pre-commit run --all-files`.
**Note:** If your code does not pass the `pre-commit` checks, automatic builds may fail.
### Use `pip-sync` to Update Dependencies
To update local dependencies, run `pip-sync` in the virtual environment.
This will make sure your virtual environment is in sync with the `requirements.txt` file,
including uninstalling any packages that are not in the `requirements.txt` file.
### Versions
The project uses [semantic versioning](https://semver.org/).
Package versions are automatically generated from git tags.
Create your first tag with `git tag 0.1.0` and push it with `git push --tags`
## Installation
**tl;dr:** `./scripts/install_package.sh`
For development, it is recommended to install the package in editable mode with `pip install -e .[dev]`.
Raw data
{
"_id": null,
"home_page": "https://github.com/climateengine/py-spatiafi",
"name": "spatiafi",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "",
"author": "Climate Engine Team",
"author_email": "admin@climateengine.com",
"download_url": "https://files.pythonhosted.org/packages/e3/c8/2d7c986599dbd8c61f32c1ccacb75e348c37091892ba590b5a13c290c5e2/spatiafi-1.8.0.tar.gz",
"platform": null,
"description": "# Python SpatiaFi API\nPython library for interacting with the SpatiaFi API.\n\nAlso included is `gdal_auth` which is a CLI tool to help with GCP authentication for GDAL.\n\n## Quickstart\n\n### Install the Package\n```shell\npip install spatiafi\n```\n\n\n### Get Authenticated Session\n```python\nfrom spatiafi import get_session\n\nsession = get_session()\n# The `session` object works just like `requests` but will automatically\n# refresh the authentication token when it expires\n\nparams = {\"item_id\": \"wildfire-risk-current-global-v1.0\"}\nurl = \"https://api.spatiafi.com/api/info\"\n\nresponse = session.get(url, params=params)\n```\n\n### Using a proxy\n\nIf you're using the client in a constrained environment you may need to pass in\n`proxies` to the `get_session` function:\n\n```python\nfrom spatiafi import get_session\n\nproxies = {\n \"http\": \"http://fake-proxy.example.com:443\",\n \"https\": \"http://fake-proxy.example.com:443\",\n}\n\nsession = get_session(proxies=proxies)\n\nparams = {\"item_id\": \"wildfire-risk-current-global-v1.0\"}\nurl = \"https://api.spatiafi.com/api/info\"\n\nresponse = session.get(url, params=params)\n```\n\n\n### Get help with `gdal_auth`\n```shell\ngdal_auth --help\n```\n\n---\n\n## GDAL Authentication\n\n`gdal_auth` is a CLI tool to help with GCP authentication for GDAL.\n\nThis command can be used in three ways:\n\n1. Set the environment variables in a file that can be sourced to set the\nenvironment variables:\n ```\n gdal_auth --file\n source /tmp/gdal_auth.env\n ```\n\n2. Print a command that can be run to set the environment\nvariables once:\n ```\n gdal_auth --line\n ```\n Running this command will output a line that you will need to copy and run.\n This will set the GDAL environment variables for anything run afterward in\n the same terminal session. (e.g. run the command and then _in the command line_\n run another command like `qgis`)\n\n\n3. Print instructions for setting up aliases. These aliases allow `gdal_*`\ncommands to be run as normal and authentication will be handled\nautomatically:\n ```\n gdal_auth --alias\n ```\n\n**Note**: for all options except `--alias`, the authentication will eventually\nexpire. This is because the tokens generated by the Google Cloud SDK expire\nafter 1 hour. The aliases will automatically refresh the authentication\ntokens when they expire.\n\n---\n\n## Mini-Batch (Async Queue)\n\nThe `AsyncQueue` class is a helper class for running many (up to ~1 million) API queries in parallel.\nTo use it, you must:\n 1. Create a task function\n 2. Create an `AsyncQueue` object\n 3. Enqueue tasks\n 4. Fetch results\n\n**tl;dr:** See [`tests/test_async_queue.py`]( tests/test_async_queue.py) for an example.\n\n### Create a AsyncQueue Task\n\nA valid AsyncQueue task must:\n * Be an async function\n * Take a single argument\n * Take an optional *async* session argument (if not provided, an async session will be created)\n * Return a single, serializable object (a `dict` is recommended)\n\nIf your task function requires multiple arguments, you can:\n * Use a wrapper function or closure (may not work on Windows or 'spawn' multiprocessing)\n * Create a new function using `functools.partial` (as shown in [`tests/test_async_queue.py`]( tests/test_async_queue.py))\n * Pass a tuple as the argument and unpack it in the task function e.g.\n ```python\n async def task(args, session=None):\n arg1, arg2, arg3 = args\n ...\n\n with AsyncQueue(task) as async_queue:\n async_queue.enqueue((arg1, arg2, arg3))\n ```\n\n#### Example Task Function\n```python\nfrom spatiafi.async_queue import AsyncQueue\nfrom spatiafi.session import get_async_session\n\n\nasync def get_point(point, session=None):\n \"\"\"\n Get a point from the SpatiaFI API.\n \"\"\"\n\n # Unpack the `point` tuple because we can only pass\n # a single argument to the task function.\n lon, lat = point\n\n # Create an async session if one is not provided.\n if session is None:\n session = await get_async_session()\n\n # Create the url.\n url = (\n \"https://api.spatiafi.com/api/point/\" + str(lon) + \",\" + str(lat)\n )\n\n params = {\"item_id\": \"wildfire-risk-current-global-v1.0\"}\n\n r = await session.get(url, params=params)\n\n # We want to raise for all errors except 400 (bad request)\n if not (r.status_code == 200 or r.status_code == 400):\n r.raise_for_status()\n\n return r.json()\n```\n\n### Create an AsyncQueue and Enqueue Tasks\n\n`AsyncQueue` takes a task function as an argument, and launches multiple instances of that task in parallel.\nThe `AsyncQueue.enqueue` method takes a _single argument_ is used to add tasks to the queue.\nThe `AsyncQueue.results` property will return a list of results in the order they were enqueued.\n\nWhen starting the `AsyncQueue`, it is **highly recommended** that you specify the number of workers/CPUs to use\nusing the `n_cores` argument. The default is to use the minimum of 4 and the number of CPUs on the machine.\n\nThis queue is designed to be used with the `with` statement. Entering the `with` statement will start the\nsubprocess and event loop. Exiting the `with` statement will wait for all tasks to finish and then stop the\nevent loop and subprocess.\n\nFor example:\n```python\nfrom spatiafi.async_queue import AsyncQueue\n\nwith AsyncQueue(get_point) as async_queue:\n for _, row in df.iterrows():\n async_queue.enqueue((row[\"lon\"], row[\"lat\"]))\n\nresults = async_queue.results\n```\n\nAlternatively, you can use the `start` and `stop` methods:\n```python\nfrom spatiafi.async_queue import AsyncQueue\n\nasync_queue = AsyncQueue(get_point)\nasync_queue.start()\n\nfor _, row in df.iterrows():\n async_queue.enqueue((row[\"lon\"], row[\"lat\"]))\n\nasync_queue.stop()\nresults = async_queue.results\n```\n\n---\n\n## Development\n\n### Use a Virtual Environment\nDevelopment should be done in a virtual environment.\nIt is recommended to use the virtual environment manager built into PyCharm.\nTo create a new virtual environment:\n * Open the project in PyCharm and select `File > Settings > Project: spfi-api > Python Interpreter`.\n * In the top right corner of the window, click the gear icon and select `Add Interpreter > Add Local Interpreter...`\n\n\n### Mark `src` as a Source Root\nIn PyCharm, mark the `src` folder as a source root. This will allow you to import modules from the `src` folder without using relative imports.\nRight-click on the `src` folder and select `Mark Directory as > Sources Root`.\n\n\n### Bootstrap the Development Environment\nRun `./scripts/bootstrap_dev.sh` to install the package and development dependencies.\n\nThis will also set up access to our private PyPI server, generate the first `requirements.txt` (if required),\nand install `pre-commit` hooks.\n\n**Protip:** This script can be run at any time if you're afraid you've messed up your environment.\n\n\n### Running the tests\nTests can be run locally via the `scripts/test.sh` script:\n\n```\n./scripts/test.sh\n```\n\nAll additional arguments to that script will be passed to PyTest which allows\nyou to do things such as run a single test:\n\n```\n./scripts/test.sh -k test_async_queue\n```\n\n\n### Manage Dependencies in `setup.cfg`\nDependencies are managed in `setup.cfg` using the `install_requires` and `extras_require` sections.\n\nTo add a new dependency:\n 1. Install the package in the virtual environment with `pip install <package_name>`\n (Hint: use the terminal built in to PyCharm)\n 2. Run `pip show <package_name>` to get the package name and version\n 3. Add the package name *and version* to `setup.cfg` in the `install_requires` section.\n Use the [compatible release](https://www.python.org/dev/peps/pep-0440/#compatible-release) syntax\n `package_name ~=version`.\n\n**DO NOT** add the package to the `requirements.txt` file. This file is automatically generated by\n`scripts/gen_requirements.sh`.\n\nIf the dependency is only needed for development, add it to the `dev` section of `extras_require` in `setup.cfg`.\n\n\n### Building Docker Images Locally\n**tl;dr:** run `./scripts/build_docker.sh`.\n\nWe need to inject a GCP access token into the Docker build to access private PyPI packages.\nThis requires using BuildKit (enabled by default in recent versions of Docker), and passing the token as a build\nargument.\n\n\n## `pre-commit` Hooks\nThis project uses `pre-commit` to run a series of checks before each git commit.\nTo install the `pre-commit` hooks, run `pre-commit install` in the virtual environment.\n(This is done automatically by `./scripts/bootstrap_dev.sh`)\n\nTo format all your code manually, run `pre-commit run --all-files`.\n\n**Note:** If your code does not pass the `pre-commit` checks, automatic builds may fail.\n\n\n### Use `pip-sync` to Update Dependencies\nTo update local dependencies, run `pip-sync` in the virtual environment.\nThis will make sure your virtual environment is in sync with the `requirements.txt` file,\nincluding uninstalling any packages that are not in the `requirements.txt` file.\n\n\n### Versions\nThe project uses [semantic versioning](https://semver.org/).\n\nPackage versions are automatically generated from git tags.\nCreate your first tag with `git tag 0.1.0` and push it with `git push --tags`\n\n\n## Installation\n**tl;dr:** `./scripts/install_package.sh`\n\nFor development, it is recommended to install the package in editable mode with `pip install -e .[dev]`.\n",
"bugtrack_url": null,
"license": "",
"summary": "Python library for interacting with the SpatiaFi API",
"version": "1.8.0",
"project_urls": {
"Homepage": "https://github.com/climateengine/py-spatiafi"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "bc16acaf8bbbcc9f04640fc61989842bbf3040034f456214fb5bb866c61f127d",
"md5": "53db931ac2ba01e8382586436f61d452",
"sha256": "645d1b4b74064962325b32f9429ce9bd1582e797ffd5f3796ee453bb4e08713f"
},
"downloads": -1,
"filename": "spatiafi-1.8.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "53db931ac2ba01e8382586436f61d452",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 20579,
"upload_time": "2023-10-17T16:46:40",
"upload_time_iso_8601": "2023-10-17T16:46:40.936327Z",
"url": "https://files.pythonhosted.org/packages/bc/16/acaf8bbbcc9f04640fc61989842bbf3040034f456214fb5bb866c61f127d/spatiafi-1.8.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e3c82d7c986599dbd8c61f32c1ccacb75e348c37091892ba590b5a13c290c5e2",
"md5": "6bc018e62596a454e948e610806a42a3",
"sha256": "8fcc81c80334c889747144292290ba3c4a71d38fb66b9d8ad072deb82b80c380"
},
"downloads": -1,
"filename": "spatiafi-1.8.0.tar.gz",
"has_sig": false,
"md5_digest": "6bc018e62596a454e948e610806a42a3",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 32842,
"upload_time": "2023-10-17T16:46:42",
"upload_time_iso_8601": "2023-10-17T16:46:42.317053Z",
"url": "https://files.pythonhosted.org/packages/e3/c8/2d7c986599dbd8c61f32c1ccacb75e348c37091892ba590b5a13c290c5e2/spatiafi-1.8.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-10-17 16:46:42",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "climateengine",
"github_project": "py-spatiafi",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "anyio",
"specs": [
[
"==",
"3.7.0"
]
]
},
{
"name": "authlib",
"specs": [
[
"==",
"1.2.0"
]
]
},
{
"name": "cachetools",
"specs": [
[
"==",
"5.3.1"
]
]
},
{
"name": "certifi",
"specs": [
[
"==",
"2023.5.7"
]
]
},
{
"name": "cffi",
"specs": [
[
"==",
"1.15.1"
]
]
},
{
"name": "charset-normalizer",
"specs": [
[
"==",
"3.1.0"
]
]
},
{
"name": "click",
"specs": [
[
"==",
"8.1.3"
]
]
},
{
"name": "cryptography",
"specs": [
[
"==",
"41.0.1"
]
]
},
{
"name": "exceptiongroup",
"specs": [
[
"==",
"1.1.1"
]
]
},
{
"name": "google-api-core",
"specs": [
[
"==",
"2.11.1"
]
]
},
{
"name": "google-auth",
"specs": [
[
"==",
"2.17.3"
]
]
},
{
"name": "google-auth-oauthlib",
"specs": [
[
"==",
"1.0.0"
]
]
},
{
"name": "googleapis-common-protos",
"specs": [
[
"==",
"1.60.0"
]
]
},
{
"name": "h11",
"specs": [
[
"==",
"0.14.0"
]
]
},
{
"name": "httpcore",
"specs": [
[
"==",
"0.17.2"
]
]
},
{
"name": "httpx",
"specs": [
[
"==",
"0.24.1"
]
]
},
{
"name": "idna",
"specs": [
[
"==",
"3.4"
]
]
},
{
"name": "numpy",
"specs": [
[
"==",
"1.25.2"
]
]
},
{
"name": "oauthlib",
"specs": [
[
"==",
"3.2.2"
]
]
},
{
"name": "pandas",
"specs": [
[
"==",
"2.1.0"
]
]
},
{
"name": "platformdirs",
"specs": [
[
"==",
"3.9.1"
]
]
},
{
"name": "protobuf",
"specs": [
[
"==",
"4.24.2"
]
]
},
{
"name": "pyasn1",
"specs": [
[
"==",
"0.5.0"
]
]
},
{
"name": "pyasn1-modules",
"specs": [
[
"==",
"0.3.0"
]
]
},
{
"name": "pycparser",
"specs": [
[
"==",
"2.21"
]
]
},
{
"name": "python-dateutil",
"specs": [
[
"==",
"2.8.2"
]
]
},
{
"name": "pytz",
"specs": [
[
"==",
"2023.3.post1"
]
]
},
{
"name": "requests",
"specs": [
[
"==",
"2.31.0"
]
]
},
{
"name": "requests-oauthlib",
"specs": [
[
"==",
"1.3.1"
]
]
},
{
"name": "rsa",
"specs": [
[
"==",
"4.9"
]
]
},
{
"name": "six",
"specs": [
[
"==",
"1.16.0"
]
]
},
{
"name": "sniffio",
"specs": [
[
"==",
"1.3.0"
]
]
},
{
"name": "tzdata",
"specs": [
[
"==",
"2023.3"
]
]
},
{
"name": "urllib3",
"specs": [
[
"==",
"2.0.3"
]
]
}
],
"lcname": "spatiafi"
}
JSON
