# Domus TDD API
A Python and SPARQL based Thing Description Directory API compliant to:
https://www.w3.org/TR/wot-discovery/
To learn more about the routes and function of this server, see
the [API documentation](doc/api.md).
## Motivation
According to [World History Encyclopedia](https://www.worldhistory.org/article/77/the-roman-domus/) the
> Roman domus was much more than a place of dwelling for a Roman familia. It also served as a place of business and a religious center for worship. The size of a domus could range from a very small house to a luxurious mansion. In some cases, one domus took up an entire city-block, while more commonly, there were up to 8 domus per insula (city-block). All domus were free-standing structures. Some were constructed like modern-day townhouses with common walls between them, while others were detached.
In the same way the Domus TDD API does not just offer a standards-conformant interface as specified in the link above, but also allows for flexible and scalable deployment, and has the possibility for extensions to carry out a bit more than just store & retrieve operations.
## Configuration
The TDD API can be configured using two methods.
1. Using environment variables prefixing the variable names by `TDD__`
(to avoid conflicts)
```
export TDD__SPARQLENDPOINT_URL="http://my-new-sparql.endpoint/address"
export TDD__CHECK_SCHEMA=True
```
2. Editing the `config.toml` file using the direct name of the variable
```
SPARQLENDPOINT_URL="http://my-new-sparql.endpoint/address"
CHECK_SCHEMA=True
```
Those two configurations can be mixed with a priority as follows:
1. If the value is set in an environment variable, use the environment variable value
2. If not, and it is set in the `config.toml`, use the `config.toml` value
3. If not, use the default value
The `config.toml` file can also be used to define FLask server configuration (c.f.
[documentation](https://flask.palletsprojects.com/en/2.1.x/config/#builtin-configuration-values)).
- The **SPARQLENDPOINT_URL** variable is mandatory
### Configuration variables
| Variable name | default value | description |
| ----------------------------- | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [TDD__]TD_REPO_URL | http://localhost:5000 | The URL to access the TDD API server |
| [TDD__]SPARQLENDPOINT_URL | http://localhost:3030/things | The SPARQL endpoint URL |
| [TDD__]CHECK_SCHEMA | False | Define if TDD API will check the TDs regarding to the JSON-Schema and SHACL shapes |
| [TDD__]MAX_TTL | None | Integer, maximum time-to-live (in seconds) that a TD will be kept on the server (unlimited if None) |
| [TDD__]MANDATE_TTL | False | Boolean value, if set to True, it will only upload TDs having a time-to-live (ttl) value. The server will send a 400 HTTP code if the TD does not contain one. |
| [TDD__]LIMIT_BATCH_TDS | 25 | Default limit of returned TDs by batch (used for pagination) |
| [TDD__]ENDPOINT_TYPE | None | Special configuration to workaround SPARQL endpoints which do not follow the SPARQL standard. Possible values: `GRAPHDB` or `VIRTUOSO` |
| [TDD__]TD_JSONSCHEMA | ./tdd/data/td-json-schema-validation.json | The path to the file containing JSON-Schema to validate the TDs |
| [TDD__]TD_ONTOLOGY | ./tdd/data/td.ttl | The path to the file containing the TD OWL Ontology (only used for SHACL validation) |
| [TDD__]TD_SHACL_VALIDATOR | ./tdd/data/td-validation.ttl | The path to the file containing the SHACL shapes (only used for SHACL validation) |
| [TDD__]PERIOD_CLEAR_EXPIRE_TD | 3600 | The number of seconds between each clearing of expired TDs (0 to disable clearing expired TD) |
| [TDD__]OVERWRITE_DISCOVERY | False | Use custom discovery context (for offline purposes) |
## Deploy to develop on the API
### Create and activate a virtual environment
Create the [virtual environment](https://docs.python.org/3/library/venv.html)
```bash
python3 -m venv .venv
```
Activate the virtual environment (in each new terminal where you need the libraries
or the project)
```bash
source .venv/bin/activate
```
Install the project and it dependencies in the virtual environment by running:
```bash
pip install -e .[dev]
```
Install the JavaScript dependencies (the project relies on jsonld.js for JSON-LD framing)
```bash
npm ci
```
### Deploy a SPARQL endpoint
The TDD API relies on a SPARQL endpoint as database connection.
You need to set up one before you run the project.
In the [SPARQL endpoint documentation](doc/sparql-endpoints/README.md) we provide
you with guidelines on how to set-up your SPARQL endpoint.
### Run the flask server
First, set up your configuration (the SPARQL endpoint URL) (see [configuration](#configuration))
if your SPARQL endpoint URL is not the default http://localhost:3030/things.
Then run the flask server at the root of this project in your python virtual environment.
```bash
flask run
```
## Import data using script
To import the TDs from a directory to your SPARQL endpoint using the proxy api, run:
```bash
python scripts/import_all_plugfest.py /path/to/TDs/directory <WOT API URL>/things
```
To import without any JSON-SCHEMA validation, change the `check-schema` param as
follows:
```bash
python scripts/import_all_plugfest.py /path/to/TDs/directory <WOT API URL>/things?check-schema=false
```
## Deploy locally with docker-compose
For a quick launch of the SPARQL endpoint and TDD API with docker-compose:
```bash
chmod a+rwx fuseki-docker/configuration
chmod a+rwx fuseki-docker/databases
docker-compose build # builds api and sparqlendpoint
docker-compose up # runs api and sparqlendpoint
```
If you want to deploy only the TDD API using docker-compose and use an
existing SPARQL endpoint then you should edit the `config.toml` file with the
appropriate `SPARQLENDPOINT_URL` value (see [configuration](#configuration)).
Then run only the api image.
If the api image is already built you do not have to rebuild, relaunching it
will use the new config.
```bash
docker-compose build api # builds the api image
docker-compose run api # runs the api
```
## Deploy production
If you want to deploy production without using docker or docker-compose you can use
the following commands:
```bash
pip install .[prod]
gunicorn -b 0.0.0.0:5000 app:app
```
You can change the `-b` parameter if you want to deploy only for localhost
access, or public access, or change the port deployment.
In this example we use the configuration using the environment variables but you can edit
the `config.toml` file instead if you prefer.
## Code quality
A few quality tests (formatting, linting, testing) have been developed to keep
this code maintainable.
Some tests have been developed to test the API's behaviour in python.
We invite you to modify the tests if you change the behaviour and add
new tests if you develop new features.
You can do it with tox if you have it installed:
```bash
tox -p all
```
Or manually if you do not have tox installed:
```bash
black .
flake8
pytests tests
```
## Plugin
To use a specific plugin you can juste pip install the module and relaunch your
TDD-API server. The new plugins routes and transformers will then be available.
### Develop your own plugin
You can develop your own plugin to add features to your TDD-API server.
To do so you can create a new project and follow the instructions defined in the
[Plugin Documentation](doc/plugin.md) to add it to the TDD-API.
### Installing a plugin from pypi
Plugins are python packages. If the plugin you want to add is on Pypi, you
can simply `pip install` it.
## Publishing domus-tdd-api to pypi.org
To publish the package on pypi:
- bump the package version in setup.py
- tag the commit with the version
- run `npm ci && npm run build` to recreate the javascript bundles
- remove existing local dist and .egg folders `rm -rf build dist .egg .egg-info`
- build python package `python3 setup.py sdist bdist_wheel`
- install twine `pip install twine`
- verify package with twine `twine check dist/*`
- upload package to pypi `twine upload --skip-existing dist/*`
Raw data
{
"_id": null,
"home_page": "https://github.com/eclipse-thingweb/domus-tdd-api",
"name": "domus-tdd-api",
"maintainer": "Eclipse Thingweb Project",
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "web of things, thing description directory, wot, tdd",
"author": "Eclipse Thingweb Project",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/18/04/80a1d132f1113cea57e9b03072367a1dcd3d7a4d1561ba9eed8fd90e05a9/domus-tdd-api-1.0.0.tar.gz",
"platform": null,
"description": "# Domus TDD API\n\nA Python and SPARQL based Thing Description Directory API compliant to:\nhttps://www.w3.org/TR/wot-discovery/\n\nTo learn more about the routes and function of this server, see\nthe [API documentation](doc/api.md).\n\n## Motivation\n\nAccording to [World History Encyclopedia](https://www.worldhistory.org/article/77/the-roman-domus/) the\n\n> Roman domus was much more than a place of dwelling for a Roman familia. It also served as a place of business and a religious center for worship. The size of a domus could range from a very small house to a luxurious mansion. In some cases, one domus took up an entire city-block, while more commonly, there were up to 8 domus per insula (city-block). All domus were free-standing structures. Some were constructed like modern-day townhouses with common walls between them, while others were detached.\n\nIn the same way the Domus TDD API does not just offer a standards-conformant interface as specified in the link above, but also allows for flexible and scalable deployment, and has the possibility for extensions to carry out a bit more than just store & retrieve operations.\n\n## Configuration\n\nThe TDD API can be configured using two methods.\n\n1. Using environment variables prefixing the variable names by `TDD__`\n (to avoid conflicts)\n\n ```\n export TDD__SPARQLENDPOINT_URL=\"http://my-new-sparql.endpoint/address\"\n export TDD__CHECK_SCHEMA=True\n ```\n\n2. Editing the `config.toml` file using the direct name of the variable\n\n ```\n SPARQLENDPOINT_URL=\"http://my-new-sparql.endpoint/address\"\n CHECK_SCHEMA=True\n ```\n\nThose two configurations can be mixed with a priority as follows:\n\n1. If the value is set in an environment variable, use the environment variable value\n2. If not, and it is set in the `config.toml`, use the `config.toml` value\n3. If not, use the default value\n\nThe `config.toml` file can also be used to define FLask server configuration (c.f.\n[documentation](https://flask.palletsprojects.com/en/2.1.x/config/#builtin-configuration-values)).\n\n- The **SPARQLENDPOINT_URL** variable is mandatory\n\n### Configuration variables\n\n| Variable name | default value | description |\n| ----------------------------- | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| [TDD__]TD_REPO_URL | http://localhost:5000 | The URL to access the TDD API server |\n| [TDD__]SPARQLENDPOINT_URL | http://localhost:3030/things | The SPARQL endpoint URL |\n| [TDD__]CHECK_SCHEMA | False | Define if TDD API will check the TDs regarding to the JSON-Schema and SHACL shapes |\n| [TDD__]MAX_TTL | None | Integer, maximum time-to-live (in seconds) that a TD will be kept on the server (unlimited if None) |\n| [TDD__]MANDATE_TTL | False | Boolean value, if set to True, it will only upload TDs having a time-to-live (ttl) value. The server will send a 400 HTTP code if the TD does not contain one. |\n| [TDD__]LIMIT_BATCH_TDS | 25 | Default limit of returned TDs by batch (used for pagination) |\n| [TDD__]ENDPOINT_TYPE | None | Special configuration to workaround SPARQL endpoints which do not follow the SPARQL standard. Possible values: `GRAPHDB` or `VIRTUOSO` |\n| [TDD__]TD_JSONSCHEMA | ./tdd/data/td-json-schema-validation.json | The path to the file containing JSON-Schema to validate the TDs |\n| [TDD__]TD_ONTOLOGY | ./tdd/data/td.ttl | The path to the file containing the TD OWL Ontology (only used for SHACL validation) |\n| [TDD__]TD_SHACL_VALIDATOR | ./tdd/data/td-validation.ttl | The path to the file containing the SHACL shapes (only used for SHACL validation) |\n| [TDD__]PERIOD_CLEAR_EXPIRE_TD | 3600 | The number of seconds between each clearing of expired TDs (0 to disable clearing expired TD) |\n| [TDD__]OVERWRITE_DISCOVERY | False | Use custom discovery context (for offline purposes) |\n\n## Deploy to develop on the API\n\n### Create and activate a virtual environment\n\nCreate the [virtual environment](https://docs.python.org/3/library/venv.html)\n\n```bash\npython3 -m venv .venv\n```\n\nActivate the virtual environment (in each new terminal where you need the libraries\nor the project)\n\n```bash\nsource .venv/bin/activate\n```\n\nInstall the project and it dependencies in the virtual environment by running:\n\n```bash\npip install -e .[dev]\n```\n\nInstall the JavaScript dependencies (the project relies on jsonld.js for JSON-LD framing)\n\n```bash\nnpm ci\n```\n\n### Deploy a SPARQL endpoint\n\nThe TDD API relies on a SPARQL endpoint as database connection.\nYou need to set up one before you run the project.\n\nIn the [SPARQL endpoint documentation](doc/sparql-endpoints/README.md) we provide\nyou with guidelines on how to set-up your SPARQL endpoint.\n\n### Run the flask server\n\nFirst, set up your configuration (the SPARQL endpoint URL) (see [configuration](#configuration))\nif your SPARQL endpoint URL is not the default http://localhost:3030/things.\n\nThen run the flask server at the root of this project in your python virtual environment.\n\n```bash\nflask run\n```\n\n## Import data using script\n\nTo import the TDs from a directory to your SPARQL endpoint using the proxy api, run:\n\n```bash\npython scripts/import_all_plugfest.py /path/to/TDs/directory <WOT API URL>/things\n```\n\nTo import without any JSON-SCHEMA validation, change the `check-schema` param as\nfollows:\n\n```bash\npython scripts/import_all_plugfest.py /path/to/TDs/directory <WOT API URL>/things?check-schema=false\n```\n\n## Deploy locally with docker-compose\n\nFor a quick launch of the SPARQL endpoint and TDD API with docker-compose:\n\n```bash\nchmod a+rwx fuseki-docker/configuration\nchmod a+rwx fuseki-docker/databases\ndocker-compose build # builds api and sparqlendpoint\ndocker-compose up # runs api and sparqlendpoint\n```\n\nIf you want to deploy only the TDD API using docker-compose and use an\nexisting SPARQL endpoint then you should edit the `config.toml` file with the\nappropriate `SPARQLENDPOINT_URL` value (see [configuration](#configuration)).\nThen run only the api image.\nIf the api image is already built you do not have to rebuild, relaunching it\nwill use the new config.\n\n```bash\ndocker-compose build api # builds the api image\ndocker-compose run api # runs the api\n```\n\n## Deploy production\n\nIf you want to deploy production without using docker or docker-compose you can use\nthe following commands:\n\n```bash\npip install .[prod]\ngunicorn -b 0.0.0.0:5000 app:app\n```\n\nYou can change the `-b` parameter if you want to deploy only for localhost\naccess, or public access, or change the port deployment.\n\nIn this example we use the configuration using the environment variables but you can edit\nthe `config.toml` file instead if you prefer.\n\n## Code quality\n\nA few quality tests (formatting, linting, testing) have been developed to keep\nthis code maintainable.\nSome tests have been developed to test the API's behaviour in python.\nWe invite you to modify the tests if you change the behaviour and add\nnew tests if you develop new features.\n\nYou can do it with tox if you have it installed:\n\n```bash\ntox -p all\n```\n\nOr manually if you do not have tox installed:\n\n```bash\nblack .\nflake8\npytests tests\n```\n\n## Plugin\n\nTo use a specific plugin you can juste pip install the module and relaunch your\nTDD-API server. The new plugins routes and transformers will then be available.\n\n### Develop your own plugin\n\nYou can develop your own plugin to add features to your TDD-API server.\nTo do so you can create a new project and follow the instructions defined in the\n[Plugin Documentation](doc/plugin.md) to add it to the TDD-API.\n\n### Installing a plugin from pypi\n\nPlugins are python packages. If the plugin you want to add is on Pypi, you\ncan simply `pip install` it.\n\n## Publishing domus-tdd-api to pypi.org\n\nTo publish the package on pypi:\n\n- bump the package version in setup.py\n- tag the commit with the version\n- run `npm ci && npm run build` to recreate the javascript bundles\n- remove existing local dist and .egg folders `rm -rf build dist .egg .egg-info`\n- build python package `python3 setup.py sdist bdist_wheel`\n- install twine `pip install twine`\n- verify package with twine `twine check dist/*`\n- upload package to pypi `twine upload --skip-existing dist/*`\n",
"bugtrack_url": null,
"license": null,
"summary": "A modular and semantic Thing Description Directory",
"version": "1.0.0",
"project_urls": {
"Homepage": "https://github.com/eclipse-thingweb/domus-tdd-api"
},
"split_keywords": [
"web of things",
" thing description directory",
" wot",
" tdd"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "3e56de1b1406b4ed42284ae0175b327773b1018a70123a311badb72be993ee79",
"md5": "66a6b5c8e0c23ca6e56469baba0bcb3e",
"sha256": "ec862794319c494c9428af9ef66ee568bc3e1495cfb2a281dfb4c99706642a7e"
},
"downloads": -1,
"filename": "domus_tdd_api-1.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "66a6b5c8e0c23ca6e56469baba0bcb3e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 118901,
"upload_time": "2024-04-26T15:54:23",
"upload_time_iso_8601": "2024-04-26T15:54:23.868826Z",
"url": "https://files.pythonhosted.org/packages/3e/56/de1b1406b4ed42284ae0175b327773b1018a70123a311badb72be993ee79/domus_tdd_api-1.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "180480a1d132f1113cea57e9b03072367a1dcd3d7a4d1561ba9eed8fd90e05a9",
"md5": "15c932897c11f68bd33148ad353ad4eb",
"sha256": "3ec329d31045c0f0e6352a2e0a24aa939fe0da765306f04a51f4c2e8b91769b9"
},
"downloads": -1,
"filename": "domus-tdd-api-1.0.0.tar.gz",
"has_sig": false,
"md5_digest": "15c932897c11f68bd33148ad353ad4eb",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 1001165,
"upload_time": "2024-04-26T15:54:26",
"upload_time_iso_8601": "2024-04-26T15:54:26.420798Z",
"url": "https://files.pythonhosted.org/packages/18/04/80a1d132f1113cea57e9b03072367a1dcd3d7a4d1561ba9eed8fd90e05a9/domus-tdd-api-1.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-26 15:54:26",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "eclipse-thingweb",
"github_project": "domus-tdd-api",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "domus-tdd-api"
}
JSON
