# ERT Storage Server
[](https://img.shields.io/pypi/pyversions/ert-storage)
[](https://pepy.tech/project/ert-storage)
[](https://img.shields.io/github/commit-activity/m/equinor/ert-storage)
[](https://img.shields.io/github/contributors-anon/equinor/ert-storage)
This is the permanent storage solution for the
[ERT](https://github.com/equinor/ert) project. It is written in Python 3.6+
using FastAPI, Pydantic, SQLAlchemy and Azure Blob Storage.
# Development Environment
This section describes how to use ERT Storage for testing or development purposes. It uses an in-memory SQLite database with no Azure Blob Storage.
First, install ERT Storage into a Python virtual environment:
``` sh
# Clone this repo
git clone https://github.com/equinor/ert-storage
cd ert-storage
# Create the virtual environment (assuming you have Python 3.6+)
python3 -m venv my-venv
# Enable the environment
source my-venv/bin/activate # sh/bash/zsh
# Install ERT Storage into the environment from current working directory
pip install -e .
```
Then start ERT Storage simply by using the new `ert-storage` command. This command will tell you that you are using an in-memory SQLite database. Now, using your browser, navigate to http://localhost:8000 to bring up the Swagger documentation page. Pages of note are:
- http://localhost:8000/doc Swagger
- http://localhost:8000/redoc ReDoc (alternative to Swagger)
# Persisting data
Specifying the `ERT_STORAGE_DATABASE_URL` environment variable it is possible to
specify a non-in-memory database. Refer to [SQLAlchemy
Documentation](https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls)
for all possible database URLs. However, only SQLite and PostgreSQL with default
SQLAlchemy drivers are officially supported at this time.
## SQLite
To persist a SQLite database (note the number of forward slashes `/`):
``` sh
export ERT_STORAGE_DATABASE_URL="sqlite:///ert.db" # Use ert.db in current working directory
export ERT_STORAGE_DATABASE_URL="sqlite:////tmp/ert.db" # Use /tmp/ert.db
```
NOTE: SQLite does not support the same features as PostgreSQL, and as such
migrations (via `alembic`) are not supported and some data may be stored or
fetched in an inefficient manner. You will have to delete your data whenever we
update the database schema, which happens relatively often.
## PostgreSQL
We use PostgreSQL in production and we use `alembic` for migrations. The following will set everything up.
Set `ERT_STORAGE_DATABASE_URL`
``` sh
# Example postgres settings
HOST="localhost"
PORT=5432 # default PostgreSQL port
DATA="ert_store"
USER="my-usrnam"
PASS="my-passwd"
# The connection string is then:
export ERT_STORAGE_DATABASE_URL="postgresql://${USER}:${PASS}@${HOST}:${PORT}/${DATA}"
# or
export ERT_STORAGE_DATABASE_URL="postgresql://my-usrnam:my-passwd@localhost:5432/ert_store"
```
Then install a compatible SQLAlchemy PostgreSQL driver, such as
[`psycopg2`](https://pypi.org/project/psycopg2/). We provide the extras
requirement `postgres` which installs the necessary requirements.
``` sh
pip install -e .[postgres]
```
The `psycopg2` package requires the postgres client development headers to be
installed on your machine. The
[`psycopg2-binary`](https://pypi.org/project/psycopg2-binary/) is a precompiled
version of this package that might be compatible with your OS and PostgreSQL
version.
### Database setup: Container
You can use either Docker or `podman` to easily start a postgres instance as a
container. On modern Linux distributions it is easier to use the `podman`
project.
[See the docker.io page for postgres for more information](https://hub.docker.com/_/postgres/)
The following instructions are for `podman`. Replacing `podman` with `docker` will
``` sh
# Start the container in the background, with password 'password', forwarding the port 5432 to the host.
podman run --name ert-storage-pg -e POSTGRES_PASSWORD=password -p 5432:5432 -d docker.io/postgres
# Useful commands
podman stop ert-storage-pg
podman start ert-storage-pg
podman kill ert-storage-pg # SIGTERM
podman rm ert-storage-pg # Remove the container entirely
# Use with ERT Storage
export ERT_STORAGE_DATABASE_URL="postgresql://postgres:password@localhost:5432/postgres"
```
To reset the database, do:
``` sh
podman exec ert-storage-pg sh -c "dropdb -U postgres postgres; createdb -U postgres postgres"
ert-storage alembic upgrade head
```
### Database setup: Linux service
First, install the dependencies:
``` sh
# RHEL / CentOS / Fedora
sudo yum install postgresql-server postgresql-devel
# Ubuntu
sudo apt install postgresql postgresql-client
```
Then, initialize the database and configure a user with a password.
``` sh
# Initialise stuff in /var (OPTIONAL)
sudo -i -u postgres -- initdb
# Create superuser
sudo -i -u postgres -- createuser -s ert_user
# Give yourself a password (default is to disallow password login)
sudo -i -u postgres -- psql -c "alter user ert_user with password '12345';"
# Create a database with you as its owner
createdb -U ert_user ert_db
# Set the database URL so that ERT Storage can connect to it
# You will need to redo this every time you close your shell
export ERT_STORAGE_DATABASE_URL="postgresql://ert_user:12345@localhost:5432/ert_db"
```
To recreate the database, do:
``` sh
dropdb ert_db && createdb -O ert_user ert_db
ert-storage alembic upgrade head
```
### Database setup: Local user setup on RHEL 7
First, download the required version of postgresql. For instance, for RHEL-7 you can check [here](https://yum.postgresql.org/13/redhat/rhel-7-x86_64/repoview/postgresql13-server.html).
Extract and copy postgresql to a custom directory; eg.:
```sh
rpm2cpio postgresql-<version>-x86_64.rpm | cpio -idv
```
Append the postgres bin path to your actual PATH; eg.
```sh
export PATH=$PATH:/data/install/postgres/usr/pgsql-<version>/bin
```
In the directory where you want to host the database run
```sh
cd <path-to-your-project>
initdb -D ~/dbdir
```
where dbdir represents the directory name where the database is hosted.
Open the config file `~/dbdir/postgresql.conf` and find the entry `unix_socket_directories` and set it to
unix_socket_directories = '/tmp/'
Start the server via
```sh
postgres -D ~/dbdir -k /tmp
```
in other terminal, whenever you want to make use of the database, set
```sh
export PGHOST=/tmp
```
Now everything should be good to go. The logs can be found in `~/dbdir/log/` directory.
To create a user and a database, you can use the following commands:
```sh
createuser -h localhost -p 5432 -s ert_user
createdb -h localhost -U ert_user ert_db
```
### After setup
Once a PostgreSQL server is running, you need to apply alembic migrations.
`ert-storage` provides a wrapper around the `alembic` command which
automatically selects the correct alembic configuration.
``` sh
# Upgrade to latest migration
ert-storage alembic upgrade head
```
# Azure Blob Storage
ERT Storage supports Azure Blob Storage for storing opaque data. This feature is invisible to the user. Install the `azure` extras with `pip install ert-storage[azure]`.
Set the `ERT_STORAGE_AZURE_CONNECTION_STRING` environment variable to the connection string found in
your Azure configuration to have ERT Storage use Azure Blob Storage.
## Local instance for development
It is possible to use [Azurite](https://github.com/Azure/Azurite), an open source Azure Storage implementation, for testing and development purposes.
As a container:
``` sh
# Podman
podman run --name ert-storage-az -p 10000:10000 -p 10001:10001 -d mcr.microsoft.com/azure-storage/azurite
# Let ERT Storage know about Azurite
export ERT_STORAGE_AZURE_CONNECTION_STRING="DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;"
```
# Development
For development, install the `test` extras `pip install ert-storage[test]`,
which installs `black`, `pytest` and `mypy`. Run tests using `pytest`.
All integration tests must be able to pass when using a non-empty production
database. In practice that means that each test needs to create a unique
"experiment".
During test debugging it might be helpful to have the tests persist data in the
database. The following environment variable tells ERT Storage not to rollback the data after each test.
``` sh
export ERT_STORAGE_NO_ROLLBACK=1
```
Raw data
{
"_id": null,
"home_page": "https://github.com/equinor/ert-storage",
"name": "ert-storage",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "",
"author": "Equinor ASA",
"author_email": "fg_sib-scout@equinor.com",
"download_url": "",
"platform": null,
"description": "# ERT Storage Server\n\n[](https://img.shields.io/pypi/pyversions/ert-storage)\n[](https://pepy.tech/project/ert-storage)\n[](https://img.shields.io/github/commit-activity/m/equinor/ert-storage)\n[](https://img.shields.io/github/contributors-anon/equinor/ert-storage)\n\nThis is the permanent storage solution for the\n[ERT](https://github.com/equinor/ert) project. It is written in Python 3.6+\nusing FastAPI, Pydantic, SQLAlchemy and Azure Blob Storage.\n\n# Development Environment\n\nThis section describes how to use ERT Storage for testing or development purposes. It uses an in-memory SQLite database with no Azure Blob Storage.\n\nFirst, install ERT Storage into a Python virtual environment:\n\n``` sh\n# Clone this repo\ngit clone https://github.com/equinor/ert-storage\ncd ert-storage\n\n# Create the virtual environment (assuming you have Python 3.6+)\npython3 -m venv my-venv\n\n# Enable the environment\nsource my-venv/bin/activate # sh/bash/zsh\n\n# Install ERT Storage into the environment from current working directory\npip install -e .\n```\n\nThen start ERT Storage simply by using the new `ert-storage` command. This command will tell you that you are using an in-memory SQLite database. Now, using your browser, navigate to http://localhost:8000 to bring up the Swagger documentation page. Pages of note are:\n\n- http://localhost:8000/doc Swagger\n- http://localhost:8000/redoc ReDoc (alternative to Swagger)\n\n# Persisting data\n\nSpecifying the `ERT_STORAGE_DATABASE_URL` environment variable it is possible to\nspecify a non-in-memory database. Refer to [SQLAlchemy\nDocumentation](https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls)\nfor all possible database URLs. However, only SQLite and PostgreSQL with default\nSQLAlchemy drivers are officially supported at this time.\n\n## SQLite\nTo persist a SQLite database (note the number of forward slashes `/`):\n\n``` sh\nexport ERT_STORAGE_DATABASE_URL=\"sqlite:///ert.db\" # Use ert.db in current working directory\nexport ERT_STORAGE_DATABASE_URL=\"sqlite:////tmp/ert.db\" # Use /tmp/ert.db\n```\n\nNOTE: SQLite does not support the same features as PostgreSQL, and as such\nmigrations (via `alembic`) are not supported and some data may be stored or\nfetched in an inefficient manner. You will have to delete your data whenever we\nupdate the database schema, which happens relatively often.\n\n## PostgreSQL\nWe use PostgreSQL in production and we use `alembic` for migrations. The following will set everything up.\n\n\nSet `ERT_STORAGE_DATABASE_URL`\n``` sh\n# Example postgres settings\nHOST=\"localhost\"\nPORT=5432 # default PostgreSQL port\nDATA=\"ert_store\"\nUSER=\"my-usrnam\"\nPASS=\"my-passwd\"\n\n# The connection string is then:\nexport ERT_STORAGE_DATABASE_URL=\"postgresql://${USER}:${PASS}@${HOST}:${PORT}/${DATA}\"\n\n# or\nexport ERT_STORAGE_DATABASE_URL=\"postgresql://my-usrnam:my-passwd@localhost:5432/ert_store\"\n```\n\nThen install a compatible SQLAlchemy PostgreSQL driver, such as\n[`psycopg2`](https://pypi.org/project/psycopg2/). We provide the extras\nrequirement `postgres` which installs the necessary requirements.\n\n``` sh\npip install -e .[postgres]\n```\n\nThe `psycopg2` package requires the postgres client development headers to be\ninstalled on your machine. The\n[`psycopg2-binary`](https://pypi.org/project/psycopg2-binary/) is a precompiled\nversion of this package that might be compatible with your OS and PostgreSQL\nversion.\n\n### Database setup: Container\nYou can use either Docker or `podman` to easily start a postgres instance as a\ncontainer. On modern Linux distributions it is easier to use the `podman`\nproject.\n\n[See the docker.io page for postgres for more information](https://hub.docker.com/_/postgres/)\n\nThe following instructions are for `podman`. Replacing `podman` with `docker` will \n\n``` sh\n# Start the container in the background, with password 'password', forwarding the port 5432 to the host.\npodman run --name ert-storage-pg -e POSTGRES_PASSWORD=password -p 5432:5432 -d docker.io/postgres\n\n# Useful commands\npodman stop ert-storage-pg\npodman start ert-storage-pg\npodman kill ert-storage-pg # SIGTERM\npodman rm ert-storage-pg # Remove the container entirely\n\n# Use with ERT Storage\nexport ERT_STORAGE_DATABASE_URL=\"postgresql://postgres:password@localhost:5432/postgres\"\n\n```\n\nTo reset the database, do:\n``` sh\npodman exec ert-storage-pg sh -c \"dropdb -U postgres postgres; createdb -U postgres postgres\"\nert-storage alembic upgrade head\n```\n\n### Database setup: Linux service\n\nFirst, install the dependencies:\n\n``` sh\n# RHEL / CentOS / Fedora\nsudo yum install postgresql-server postgresql-devel\n\n# Ubuntu\nsudo apt install postgresql postgresql-client\n```\n\nThen, initialize the database and configure a user with a password.\n\n``` sh\n# Initialise stuff in /var (OPTIONAL)\nsudo -i -u postgres -- initdb\n\n# Create superuser\nsudo -i -u postgres -- createuser -s ert_user\n\n# Give yourself a password (default is to disallow password login)\nsudo -i -u postgres -- psql -c \"alter user ert_user with password '12345';\"\n\n# Create a database with you as its owner\ncreatedb -U ert_user ert_db\n\n# Set the database URL so that ERT Storage can connect to it\n# You will need to redo this every time you close your shell\nexport ERT_STORAGE_DATABASE_URL=\"postgresql://ert_user:12345@localhost:5432/ert_db\"\n```\n\nTo recreate the database, do:\n\n``` sh\ndropdb ert_db && createdb -O ert_user ert_db\nert-storage alembic upgrade head\n```\n\n### Database setup: Local user setup on RHEL 7\n\nFirst, download the required version of postgresql. For instance, for RHEL-7 you can check [here](https://yum.postgresql.org/13/redhat/rhel-7-x86_64/repoview/postgresql13-server.html). \nExtract and copy postgresql to a custom directory; eg.:\n\n```sh\nrpm2cpio postgresql-<version>-x86_64.rpm | cpio -idv\n```\n\nAppend the postgres bin path to your actual PATH; eg. \n\n```sh\nexport PATH=$PATH:/data/install/postgres/usr/pgsql-<version>/bin\n```\n\nIn the directory where you want to host the database run\n\n```sh\ncd <path-to-your-project>\ninitdb -D ~/dbdir\n```\n\nwhere dbdir represents the directory name where the database is hosted.\n\nOpen the config file `~/dbdir/postgresql.conf` and find the entry `unix_socket_directories` and set it to\nunix_socket_directories = '/tmp/'\n\nStart the server via \n\n```sh\npostgres -D ~/dbdir -k /tmp\n```\n\nin other terminal, whenever you want to make use of the database, set\n\n```sh\nexport PGHOST=/tmp\n```\n\nNow everything should be good to go. The logs can be found in `~/dbdir/log/` directory.\nTo create a user and a database, you can use the following commands:\n\n```sh\ncreateuser -h localhost -p 5432 -s ert_user\ncreatedb -h localhost -U ert_user ert_db\n```\n\n\n### After setup\nOnce a PostgreSQL server is running, you need to apply alembic migrations.\n`ert-storage` provides a wrapper around the `alembic` command which\nautomatically selects the correct alembic configuration.\n\n``` sh\n# Upgrade to latest migration\nert-storage alembic upgrade head\n```\n\n# Azure Blob Storage\nERT Storage supports Azure Blob Storage for storing opaque data. This feature is invisible to the user. Install the `azure` extras with `pip install ert-storage[azure]`.\n\nSet the `ERT_STORAGE_AZURE_CONNECTION_STRING` environment variable to the connection string found in\nyour Azure configuration to have ERT Storage use Azure Blob Storage.\n\n## Local instance for development\nIt is possible to use [Azurite](https://github.com/Azure/Azurite), an open source Azure Storage implementation, for testing and development purposes.\n\nAs a container:\n``` sh\n# Podman\npodman run --name ert-storage-az -p 10000:10000 -p 10001:10001 -d mcr.microsoft.com/azure-storage/azurite\n\n# Let ERT Storage know about Azurite\nexport ERT_STORAGE_AZURE_CONNECTION_STRING=\"DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;\"\n```\n\n# Development\nFor development, install the `test` extras `pip install ert-storage[test]`,\nwhich installs `black`, `pytest` and `mypy`. Run tests using `pytest`.\n\nAll integration tests must be able to pass when using a non-empty production\ndatabase. In practice that means that each test needs to create a unique\n\"experiment\".\n\nDuring test debugging it might be helpful to have the tests persist data in the\ndatabase. The following environment variable tells ERT Storage not to rollback the data after each test.\n\n``` sh\nexport ERT_STORAGE_NO_ROLLBACK=1\n```\n",
"bugtrack_url": null,
"license": "GPLv3",
"summary": "Storage service for ERT",
"version": "0.3.20",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "74b66e75b5adf35b9e7f1e41ced488958896330d378a3c486e8786a0c01a126c",
"md5": "c7f8e6afeda07d65d87bb2a2b012aad9",
"sha256": "724e8dbba1fb7e092adb54f07f1c3de9f4a35cf5b32ef95e84910a88a7458d50"
},
"downloads": -1,
"filename": "ert_storage-0.3.20-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c7f8e6afeda07d65d87bb2a2b012aad9",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 76531,
"upload_time": "2023-03-17T09:42:07",
"upload_time_iso_8601": "2023-03-17T09:42:07.055077Z",
"url": "https://files.pythonhosted.org/packages/74/b6/6e75b5adf35b9e7f1e41ced488958896330d378a3c486e8786a0c01a126c/ert_storage-0.3.20-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-03-17 09:42:07",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "equinor",
"github_project": "ert-storage",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "ert-storage"
}
JSON
