# nowcasting_datamodel
<!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->
[](#contributors-)
<!-- ALL-CONTRIBUTORS-BADGE:END -->
Datamodel for the nowcasting project
The data model has been made using `sqlalchemy` with a mirrored model in `pydantic`.
⚠️ Database tables are currently made automatically,
but in the future there should be a migration process
Future: The data model could be moved, to be a more modular solution.
## nowcasting_datamodel
### models.py
All models are in `nowcasting_datamodel.models.py`.
The diagram below shows how the different tables are connected.
### connection.py
`nowcasting_datamodel.connection.py` contains a connection class which can be used to make a sqlalchemy session.
```python
from nowcasting_datamodel.connection import DatabaseConnection
# make connection object
db_connection = DatabaseConnection(url='sqlite:///test.db')
# make sessions
with db_connection.get_session() as session:
# do something with the database
pass
```
### 👓 read.py
`nowcasting_datamodel.read.py` contains functions to read the database.
The idea is that these are easy to use functions that query the database in an efficient and easy way.
- get_latest_forecast: Get the latest `Forecast` for a specific GSP.
- get_all_gsp_ids_latest_forecast: Get the latest `Forecast` for all GSPs.
- get_forecast_values: Gets the latest `ForecastValue` for a specific GSP
- get_latest_national_forecast: Returns the latest national forecast
- get_location: Gets a `Location` object
```python
from nowcasting_datamodel.connection import DatabaseConnection
from nowcasting_datamodel.read import get_latest_forecast
# make connection object
db_connection = DatabaseConnection(url='sqlite:///test.db')
# make sessions
with db_connection.get_session() as session:
f = get_latest_forecast(session=session, gsp_id=1)
```
### 💾 save.py
`nowcasting_datamodel.save.py` has one functions to save a list of `Forecast` to the database
### 🇬🇧 national.py
`nowcasting_datamodel.fake.py` has a useful function for adding up forecasts for all GSPs into a national Forecast.
### fake.py
`nowcasting_datamodel.fake.py`
Functions used to make fake model data.
## 🩺 Testing
Tests are run by using the following command
```bash
docker stop $(docker ps -a -q)
docker-compose -f test-docker-compose.yml build
docker-compose -f test-docker-compose.yml run tests
```
These sets up `postgres` in a docker container and runs the tests in another docker container.
This slightly more complicated testing framework is needed (compared to running `pytest`)
as some queries can not be fully tested on a `sqlite` database
### Mac M1 users
An upstream builds issue of libgp may cause the following error:
`sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SCRAM authentication requires libpq version 10 or above`
As suggested in this [thread](https://stackoverflow.com/questions/62807717/how-can-i-solve-postgresql-scram-authentication-problem), a temporary fix is to set the env variable `DOCKER_DEFAULT_PLATFORM=linux/amd64` prior to building the test images - although this reportedly comes with performance penalties.
## 🛠️ infrastructure
`.github/workflows` contains a number of CI actions
1. linters.yaml: Runs linting checks on the code
2. release.yaml: Make and pushes docker files on a new code release
3. test-docker.yaml': Runs tests on every push
The docker file is in the folder `infrastructure/docker/`
The version is bumped automatically for any push to `main`.
## Environmental Variables
- DB_URL: The database url which the forecasts will be saved too
## Contributors ✨
Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):
<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->
<!-- prettier-ignore-start -->
<!-- markdownlint-disable -->
<table>
<tbody>
<tr>
<td align="center" valign="top" width="14.28%"><a href="http://lostcoding.com"><img src="https://avatars.githubusercontent.com/u/20285369?v=4?s=100" width="100px;" alt="Brandon Ly"/><br /><sub><b>Brandon Ly</b></sub></a><br /><a href="https://github.com/openclimatefix/nowcasting_datamodel/commits?author=branberry" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/lucasc896"><img src="https://avatars.githubusercontent.com/u/1273006?v=4?s=100" width="100px;" alt="Chris Lucas"/><br /><sub><b>Chris Lucas</b></sub></a><br /><a href="https://github.com/openclimatefix/nowcasting_datamodel/commits?author=lucasc896" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/dfulu"><img src="https://avatars.githubusercontent.com/u/41546094?v=4?s=100" width="100px;" alt="James Fulton"/><br /><sub><b>James Fulton</b></sub></a><br /><a href="https://github.com/openclimatefix/nowcasting_datamodel/commits?author=dfulu" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/roshnaeem"><img src="https://avatars.githubusercontent.com/u/47316899?v=4?s=100" width="100px;" alt="Rosheen Naeem"/><br /><sub><b>Rosheen Naeem</b></sub></a><br /><a href="https://github.com/openclimatefix/nowcasting_datamodel/commits?author=roshnaeem" title="Code">💻</a></td>
</tr>
</tbody>
</table>
<!-- markdownlint-restore -->
<!-- prettier-ignore-end -->
<!-- ALL-CONTRIBUTORS-LIST:END -->
This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!
Raw data
{
"_id": null,
"home_page": "https://github.com/openclimatefix/nowcasting_datamodel",
"name": "nowcasting-datamodel",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "SQL, Datamodel",
"author": "Peter Dudfield",
"author_email": "peter@openclimatefix.org",
"download_url": "https://files.pythonhosted.org/packages/1a/02/651db39aec3f112caabee26a0f3b9c7cc6b73d1abeb5d1b710a87be083ca/nowcasting_datamodel-1.5.38.tar.gz",
"platform": null,
"description": "# nowcasting_datamodel\n<!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->\n[](#contributors-)\n<!-- ALL-CONTRIBUTORS-BADGE:END -->\nDatamodel for the nowcasting project\n\n\nThe data model has been made using `sqlalchemy` with a mirrored model in `pydantic`.\n\n\u26a0\ufe0f Database tables are currently made automatically,\nbut in the future there should be a migration process\n\nFuture: The data model could be moved, to be a more modular solution.\n\n## nowcasting_datamodel\n\n### models.py\nAll models are in `nowcasting_datamodel.models.py`.\nThe diagram below shows how the different tables are connected.\n\n\n\n\n### connection.py\n\n`nowcasting_datamodel.connection.py` contains a connection class which can be used to make a sqlalchemy session.\n```python\nfrom nowcasting_datamodel.connection import DatabaseConnection\n\n# make connection object\ndb_connection = DatabaseConnection(url='sqlite:///test.db')\n\n# make sessions\nwith db_connection.get_session() as session:\n\n # do something with the database\n pass\n```\n\n### \ud83d\udc53 read.py\n\n`nowcasting_datamodel.read.py` contains functions to read the database.\nThe idea is that these are easy to use functions that query the database in an efficient and easy way.\n\n - get_latest_forecast: Get the latest `Forecast` for a specific GSP.\n - get_all_gsp_ids_latest_forecast: Get the latest `Forecast` for all GSPs.\n - get_forecast_values: Gets the latest `ForecastValue` for a specific GSP\n - get_latest_national_forecast: Returns the latest national forecast\n - get_location: Gets a `Location` object\n\n```python\nfrom nowcasting_datamodel.connection import DatabaseConnection\nfrom nowcasting_datamodel.read import get_latest_forecast\n\n# make connection object\ndb_connection = DatabaseConnection(url='sqlite:///test.db')\n\n# make sessions\nwith db_connection.get_session() as session:\n f = get_latest_forecast(session=session, gsp_id=1)\n```\n\n### \ud83d\udcbe save.py\n`nowcasting_datamodel.save.py` has one functions to save a list of `Forecast` to the database\n\n### \ud83c\uddec\ud83c\udde7 national.py\n`nowcasting_datamodel.fake.py` has a useful function for adding up forecasts for all GSPs into a national Forecast.\n\n### fake.py\n`nowcasting_datamodel.fake.py`\n\nFunctions used to make fake model data.\n\n\n## \ud83e\ude7a Testing\n\nTests are run by using the following command\n```bash\ndocker stop $(docker ps -a -q)\ndocker-compose -f test-docker-compose.yml build\ndocker-compose -f test-docker-compose.yml run tests\n```\n\nThese sets up `postgres` in a docker container and runs the tests in another docker container.\nThis slightly more complicated testing framework is needed (compared to running `pytest`)\nas some queries can not be fully tested on a `sqlite` database\n\n### Mac M1 users\nAn upstream builds issue of libgp may cause the following error:\n\n`sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SCRAM authentication requires libpq version 10 or above`\n\nAs suggested in this [thread](https://stackoverflow.com/questions/62807717/how-can-i-solve-postgresql-scram-authentication-problem), a temporary fix is to set the env variable `DOCKER_DEFAULT_PLATFORM=linux/amd64` prior to building the test images - although this reportedly comes with performance penalties.\n\n## \ud83d\udee0\ufe0f infrastructure\n\n`.github/workflows` contains a number of CI actions\n1. linters.yaml: Runs linting checks on the code\n2. release.yaml: Make and pushes docker files on a new code release\n3. test-docker.yaml': Runs tests on every push\n\nThe docker file is in the folder `infrastructure/docker/`\n\nThe version is bumped automatically for any push to `main`.\n\n## Environmental Variables\n\n- DB_URL: The database url which the forecasts will be saved too\n\n## Contributors \u2728\n\nThanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):\n\n<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->\n<!-- prettier-ignore-start -->\n<!-- markdownlint-disable -->\n<table>\n <tbody>\n <tr>\n <td align=\"center\" valign=\"top\" width=\"14.28%\"><a href=\"http://lostcoding.com\"><img src=\"https://avatars.githubusercontent.com/u/20285369?v=4?s=100\" width=\"100px;\" alt=\"Brandon Ly\"/><br /><sub><b>Brandon Ly</b></sub></a><br /><a href=\"https://github.com/openclimatefix/nowcasting_datamodel/commits?author=branberry\" title=\"Code\">\ud83d\udcbb</a></td>\n <td align=\"center\" valign=\"top\" width=\"14.28%\"><a href=\"https://github.com/lucasc896\"><img src=\"https://avatars.githubusercontent.com/u/1273006?v=4?s=100\" width=\"100px;\" alt=\"Chris Lucas\"/><br /><sub><b>Chris Lucas</b></sub></a><br /><a href=\"https://github.com/openclimatefix/nowcasting_datamodel/commits?author=lucasc896\" title=\"Code\">\ud83d\udcbb</a></td>\n <td align=\"center\" valign=\"top\" width=\"14.28%\"><a href=\"https://github.com/dfulu\"><img src=\"https://avatars.githubusercontent.com/u/41546094?v=4?s=100\" width=\"100px;\" alt=\"James Fulton\"/><br /><sub><b>James Fulton</b></sub></a><br /><a href=\"https://github.com/openclimatefix/nowcasting_datamodel/commits?author=dfulu\" title=\"Code\">\ud83d\udcbb</a></td>\n <td align=\"center\" valign=\"top\" width=\"14.28%\"><a href=\"https://github.com/roshnaeem\"><img src=\"https://avatars.githubusercontent.com/u/47316899?v=4?s=100\" width=\"100px;\" alt=\"Rosheen Naeem\"/><br /><sub><b>Rosheen Naeem</b></sub></a><br /><a href=\"https://github.com/openclimatefix/nowcasting_datamodel/commits?author=roshnaeem\" title=\"Code\">\ud83d\udcbb</a></td>\n </tr>\n </tbody>\n</table>\n\n<!-- markdownlint-restore -->\n<!-- prettier-ignore-end -->\n\n<!-- ALL-CONTRIBUTORS-LIST:END -->\n\nThis project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Data Model for the OCF nowcasting project",
"version": "1.5.38",
"project_urls": {
"Homepage": "https://github.com/openclimatefix/nowcasting_datamodel"
},
"split_keywords": [
"sql",
" datamodel"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "46e3e576c5489a5a7fd6ca693847691b4bd3bceb219a34e64f233c69d93546ba",
"md5": "3e22b8f215a05618b37949e2ea2b3a95",
"sha256": "1e266d6ef7339283c706b0832c774ced580e8ec6837a95216d68dac388b4f51d"
},
"downloads": -1,
"filename": "nowcasting_datamodel-1.5.38-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3e22b8f215a05618b37949e2ea2b3a95",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 43577,
"upload_time": "2024-04-08T08:39:05",
"upload_time_iso_8601": "2024-04-08T08:39:05.021229Z",
"url": "https://files.pythonhosted.org/packages/46/e3/e576c5489a5a7fd6ca693847691b4bd3bceb219a34e64f233c69d93546ba/nowcasting_datamodel-1.5.38-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1a02651db39aec3f112caabee26a0f3b9c7cc6b73d1abeb5d1b710a87be083ca",
"md5": "3dd16f5b1aae13f7ec09fbe526ca80e3",
"sha256": "dbfe2c86dd5578f960b360cc9f6efe41696962f5c7ca0689ca12e060e55dbbdd"
},
"downloads": -1,
"filename": "nowcasting_datamodel-1.5.38.tar.gz",
"has_sig": false,
"md5_digest": "3dd16f5b1aae13f7ec09fbe526ca80e3",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 214493,
"upload_time": "2024-04-08T08:39:08",
"upload_time_iso_8601": "2024-04-08T08:39:08.797706Z",
"url": "https://files.pythonhosted.org/packages/1a/02/651db39aec3f112caabee26a0f3b9c7cc6b73d1abeb5d1b710a87be083ca/nowcasting_datamodel-1.5.38.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-08 08:39:08",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "openclimatefix",
"github_project": "nowcasting_datamodel",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"requirements": [
{
"name": "pydantic",
"specs": [
[
"==",
"2.5.3"
]
]
},
{
"name": "pandas",
"specs": []
},
{
"name": "sqlalchemy",
"specs": []
},
{
"name": "psycopg2-binary",
"specs": []
},
{
"name": "numpy",
"specs": []
},
{
"name": "alembic",
"specs": []
},
{
"name": "click",
"specs": []
},
{
"name": "freezegun",
"specs": []
},
{
"name": "structlog",
"specs": []
}
],
"lcname": "nowcasting-datamodel"
}
JSON
