# Graph Validation Tests
[](https://pypi.python.org/pypi/graph-validation-test-runners)
[](https://opensource.org/licenses/MIT)
[](https://github.com/TranslatorSRI/graph-validation-test-runners/actions/workflows/test.yml)
This repository provides the implementation of Translator knowledge graph validation test runners within the new 2023 Testing Infrastructure. The current package currently contains two such test runners:
- **StandardsValidationTest:** is a wrapper of the [Translator reasoner-validator package](https://github.com/NCATSTranslator/reasoner-validator) which certifies that knowledge graph data access is TRAPI compliant and the graph semantic content is Biolink Model compliant.
- **OneHopTest:** is a slimmed down excerpt of "One Hop" knowledge graph navigation unit test code from the legacy [SRI_Testing test harness](https://github.com/TranslatorSRI/SRI_testing), code which validates that single hop TRAPI lookup queries on a Translator knowledge graph, meet basic expectation of input test edge data recovery in the output, using several diverse kinds of templated TRAPI queries. Unlike **SRI_Testing**, the **graph-validation-test-runners** TestRunners uses test data directly from the new [NCATS Translator Tests](https://github.com/NCATSTranslator/Tests) repository.
Programmatically, the command line or programmatic parameters to each kind of test are identical, but the underlying Test Cases (derived from the source Test Assets) is the same.
## Usage
The **standards_validation_test_runner** and **one_hop_test_runner* may be run directly from the command line or programmatically, from within a Python script.
### Installation
The **graph-validation-test-runners** module can be installed from pypi and used as part of the Translator-wide automated testing.
_Note: Requires 3.9 <= Python release < 3.12_
#### From Pypi
From within your target working directory:
- Create a python virtual environment: `python -m venv venv`
- Activate your environment: `. ./venv/bin/activate`
- Install dependencies: `pip install graph-validation-test-runners`
then proceed with [command line execution](#cli) or [script level execution](#programmatic-level-execution).
#### From GitHub
You can also check out the project from GitHub. If you do that, the installation process will be slightly different, since the project itself uses [Poetry](https://python-poetry.org/) for dependency management - the following instructions assume that you've [installed Poetry on your system](https://python-poetry.org/docs/#installation).
- Check out the code: `git checkout https://github.com/TranslatorSRI/graph-validation-test-runners.git`
- Create a Poetry shell: `poetry shell`
- Install dependencies: `poetry install`
then proceed with [command line execution](#cli) or [script level execution](#programmatic-level-execution).
### CLI
Within a command line terminal, type:
```shell
$ standards_validation_test --help
```
or
```shell
$ one_hop_test --help
```
should give usage instructions as follows (where <tool name> is either 'standards_validation_test_runner' or 'one_hop_test_runner'):
```shell
usage: <tool name> [-h] [--components COMPONENTS] [--environment {dev,ci,test,prod}] --subject_id SUBJECT_ID --predicate_id PREDICATE_ID
--object_id OBJECT_ID [--trapi_version TRAPI_VERSION] [--biolink_version BIOLINK_VERSION]
[--log_level {ERROR,WARNING,INFO,DEBUG}]
Translator TRAPI and Biolink Model Validation of Knowledge Graphs
options:
-h, --help show this help message and exit
--components COMPONENTS
Names Translator components to be tested taken from the Translator Testing Model 'ComponentEnum'
(may be a comma separated string of such names; default: run the test against the 'ars')
--environment {dev,ci,test,prod}
Translator execution environment of the Translator Component targeted for testing.
--subject_id SUBJECT_ID
Statement object concept CURIE
--predicate_id PREDICATE_ID
Statement Biolink Predicate identifier
--object_id OBJECT_ID
Statement object concept CURIE
--trapi_version TRAPI_VERSION
TRAPI version expected for knowledge graph access (default: use current default release)
--biolink_version BIOLINK_VERSION
Biolink Model version expected for knowledge graph access (default: use current default release)
```
### Programmatic Level Execution
### Standards Validation Test
To run TRAPI and Biolink Model validation tests validating query outputs from a knowledge graph TRAPI component:
```python
from typing import Dict
import asyncio
from translator_testing_model.datamodel.pydanticmodel import ComponentEnum
from standards_validation_test_runner import run_standards_validation_tests
test_data = {
# One test edge (asset)
"subject_id": "DRUGBANK:DB01592",
"subject_category": "biolink:SmallMolecule",
"predicate_id": "biolink:has_side_effect",
"object_id": "MONDO:0011426",
"object_category": "biolink:Disease",
"components": ["arax", "molepro"]
# "environment": environment, # Optional[TestEnvEnum] = None; default: 'TestEnvEnum.ci' if not given
# "trapi_version": trapi_version, # Optional[str] = None; latest community release if not given
# "biolink_version": biolink_version, # Optional[str] = None; current Biolink Toolkit default if not given
# "runner_settings": asset.test_runner_settings, # Optional[List[str]] = None
}
results: Dict = asyncio.run(run_standards_validation_tests(**test_data))
print(results)
```
### One Hop Test
To run "One Hop" knowledge graph navigation tests validating query outputs from a knowledge graph TRAPI component:
```python
from typing import Dict
import asyncio
from one_hop_test_runner import run_one_hop_tests
test_data = {
# One test edge (asset)
"subject_id": "DRUGBANK:DB01592",
"subject_category": "biolink:SmallMolecule",
"predicate_id": "biolink:has_side_effect",
"object_id": "MONDO:0011426",
"object_category": "biolink:Disease",
"components": ["arax", "molepro"]
#
# "environment": environment, # Optional[TestEnvEnum] = None; default: 'TestEnvEnum.ci' if not given
# "trapi_version": trapi_version, # Optional[str] = None; latest community release if not given
# "biolink_version": biolink_version, # Optional[str] = None; current Biolink Toolkit default if not given
# "runner_settings": asset.test_runner_settings, # Optional[List[str]] = None
}
results: Dict = asyncio.run(run_one_hop_tests(**test_data))
print(results)
```
The above wrapper method runs all related TestCases derived from the specified TestAsset (i.e. subject_id, etc.) without any special test parameters. If more fine-grained testing is desired, a subset of the underlying TRAPI queries can be run directly, something like this (here, we ignore the TestCases 'by_subject', 'inverse_by_new_subject' and 'by_object', and specify the 'strict_validation' parameter of True to Biolink Model validation, as understood by the **reasoner-validator** code running behind the scenes):
```python
from typing import Dict
import asyncio
from standards_validation_test_runner import StandardsValidationTest
from graph_validation_tests.utils.unit_test_templates import (
# by_subject,
# inverse_by_new_subject,
# by_object,
raise_subject_entity,
raise_object_entity,
raise_object_by_subject,
raise_predicate_by_subject
)
test_data = {
# One test edge (asset)
"subject_id": "DRUGBANK:DB01592",
"subject_category": "biolink:SmallMolecule",
"predicate_id": "biolink:has_side_effect",
"object_id": "MONDO:0011426",
"object_category": "biolink:Disease",
"components": ["arax", "molepro"],
"environment": "test",
"trapi_version": "1.5.0-beta",
"biolink_version": "4.1.6",
"runner_settings": "Inferred"
}
trapi_generators = [
# by_subject,
# inverse_by_new_subject,
# by_object,
raise_subject_entity,
raise_object_entity,
raise_object_by_subject,
raise_predicate_by_subject
]
# A test runner specific parameter passed through
kwargs = {
"strict_validation": True
}
results: Dict = asyncio.run(StandardsValidationTest.run_tests(
**test_data, trapi_generators=trapi_generators, **kwargs)
)
```
Note that the trapi_generation variables - defined in the **graph_validation_test.utils.unit_test_templates** module - are all simply Python functions returning TRAPI JSON messages to send to the target components. In principle, if one understands what those functions are doing, you could write your own methods to do other kinds of TRAPI queries whose output can then be validated against the specified TRAPI and Biolink Model releases.
### Sample Output
This is a sample of what the JSON output from test runs currently looks like (this sample came from a OneHopTest run).
```json
{
"pks": {
"molepro": "molepro"
},
"results": {
"TestAsset_1-by_subject": {
"molepro": {
"status": "FAILED",
"messages": {
"error": {
"error.trapi.response.knowledge_graph.missing_expected_edge": {
"global": {
"TestAsset_1|(CHEBI:58579#biolink:SmallMolecule)-[biolink:is_active_metabolite_of]->(UniProtKB:Q9NQ88#biolink:Protein)": null
}
}
}
}
}
},
"TestAsset_1-inverse_by_new_subject": {
"molepro": {
"status": "FAILED",
"messages": {
"critical": {
"critical.trapi.request.invalid": {
"global": {
"predicate 'biolink:is_active_metabolite_of'": [
{
"context": "inverse_by_new_subject",
"reason": "is an unknown or has no inverse?"
}
]
}
}
}
}
}
}
}
}
```
## Releases
A [full change log](CHANGELOG.md) is provided documenting each release, but we summarize key release limitations here:
### v0.0.\* Releases
- This initial code release only supports testing of [Translator SmartAPI Registry](https://smart-api.info/registry/translator) catalogued components which are TRAPI implementations for Translator Autonomous Relay Agent (ARA) and Knowledge Providers (KP), but _not_ direct testing of the Translator Autonomous Relay System (ARS) or Translator user interface (UI)
Raw data
{
"_id": null,
"home_page": "https://github.com/TranslatorSRI",
"name": "graph-validation-test-runners",
"maintainer": "Richard Bruskiewich",
"docs_url": null,
"requires_python": "<3.12,>=3.9",
"maintainer_email": "richard.bruskiewich@delphinai.com",
"keywords": "NCATS, Biomedical Data Translator, Translator, ReasonerAPI, TRAPI, validation, Biolink Model",
"author": "Richard Bruskiewich",
"author_email": "richard.bruskiewich@delphinai.com",
"download_url": "https://files.pythonhosted.org/packages/d3/8e/b44eef8c7bd2d47061cd6b896c908799d61a73a2f73c2d33ecc82a2ed57a/graph_validation_test_runners-0.1.3.tar.gz",
"platform": null,
"description": "# Graph Validation Tests\n\n\n[](https://pypi.python.org/pypi/graph-validation-test-runners)\n[](https://opensource.org/licenses/MIT)\n[](https://github.com/TranslatorSRI/graph-validation-test-runners/actions/workflows/test.yml)\n\n\nThis repository provides the implementation of Translator knowledge graph validation test runners within the new 2023 Testing Infrastructure. The current package currently contains two such test runners:\n\n- **StandardsValidationTest:** is a wrapper of the [Translator reasoner-validator package](https://github.com/NCATSTranslator/reasoner-validator) which certifies that knowledge graph data access is TRAPI compliant and the graph semantic content is Biolink Model compliant.\n- **OneHopTest:** is a slimmed down excerpt of \"One Hop\" knowledge graph navigation unit test code from the legacy [SRI_Testing test harness](https://github.com/TranslatorSRI/SRI_testing), code which validates that single hop TRAPI lookup queries on a Translator knowledge graph, meet basic expectation of input test edge data recovery in the output, using several diverse kinds of templated TRAPI queries. Unlike **SRI_Testing**, the **graph-validation-test-runners** TestRunners uses test data directly from the new [NCATS Translator Tests](https://github.com/NCATSTranslator/Tests) repository. \n\nProgrammatically, the command line or programmatic parameters to each kind of test are identical, but the underlying Test Cases (derived from the source Test Assets) is the same.\n\n## Usage\n\nThe **standards_validation_test_runner** and **one_hop_test_runner* may be run directly from the command line or programmatically, from within a Python script.\n\n### Installation\n\nThe **graph-validation-test-runners** module can be installed from pypi and used as part of the Translator-wide automated testing.\n\n_Note: Requires 3.9 <= Python release < 3.12_\n\n#### From Pypi\n\nFrom within your target working directory:\n\n- Create a python virtual environment: `python -m venv venv`\n- Activate your environment: `. ./venv/bin/activate`\n- Install dependencies: `pip install graph-validation-test-runners`\n\nthen proceed with [command line execution](#cli) or [script level execution](#programmatic-level-execution).\n\n#### From GitHub\n\nYou can also check out the project from GitHub. If you do that, the installation process will be slightly different, since the project itself uses [Poetry](https://python-poetry.org/) for dependency management - the following instructions assume that you've [installed Poetry on your system](https://python-poetry.org/docs/#installation).\n\n- Check out the code: `git checkout https://github.com/TranslatorSRI/graph-validation-test-runners.git`\n- Create a Poetry shell: `poetry shell`\n- Install dependencies: `poetry install`\n\nthen proceed with [command line execution](#cli) or [script level execution](#programmatic-level-execution).\n\n### CLI\n\nWithin a command line terminal, type:\n\n```shell\n$ standards_validation_test --help\n```\nor\n\n```shell\n$ one_hop_test --help\n```\n\nshould give usage instructions as follows (where <tool name> is either 'standards_validation_test_runner' or 'one_hop_test_runner'):\n\n```shell\nusage: <tool name> [-h] [--components COMPONENTS] [--environment {dev,ci,test,prod}] --subject_id SUBJECT_ID --predicate_id PREDICATE_ID\n --object_id OBJECT_ID [--trapi_version TRAPI_VERSION] [--biolink_version BIOLINK_VERSION]\n [--log_level {ERROR,WARNING,INFO,DEBUG}]\n\nTranslator TRAPI and Biolink Model Validation of Knowledge Graphs\n\noptions:\n -h, --help show this help message and exit\n --components COMPONENTS\n Names Translator components to be tested taken from the Translator Testing Model 'ComponentEnum' \n (may be a comma separated string of such names; default: run the test against the 'ars')\n --environment {dev,ci,test,prod}\n Translator execution environment of the Translator Component targeted for testing.\n --subject_id SUBJECT_ID\n Statement object concept CURIE\n --predicate_id PREDICATE_ID\n Statement Biolink Predicate identifier\n --object_id OBJECT_ID\n Statement object concept CURIE\n --trapi_version TRAPI_VERSION\n TRAPI version expected for knowledge graph access (default: use current default release)\n --biolink_version BIOLINK_VERSION\n Biolink Model version expected for knowledge graph access (default: use current default release)\n```\n\n### Programmatic Level Execution\n\n### Standards Validation Test\n\nTo run TRAPI and Biolink Model validation tests validating query outputs from a knowledge graph TRAPI component:\n\n```python\nfrom typing import Dict\nimport asyncio\nfrom translator_testing_model.datamodel.pydanticmodel import ComponentEnum\nfrom standards_validation_test_runner import run_standards_validation_tests\n\ntest_data = {\n # One test edge (asset)\n \"subject_id\": \"DRUGBANK:DB01592\",\n \"subject_category\": \"biolink:SmallMolecule\",\n \"predicate_id\": \"biolink:has_side_effect\",\n \"object_id\": \"MONDO:0011426\",\n \"object_category\": \"biolink:Disease\",\n \"components\": [\"arax\", \"molepro\"]\n # \"environment\": environment, # Optional[TestEnvEnum] = None; default: 'TestEnvEnum.ci' if not given\n # \"trapi_version\": trapi_version, # Optional[str] = None; latest community release if not given\n # \"biolink_version\": biolink_version, # Optional[str] = None; current Biolink Toolkit default if not given\n # \"runner_settings\": asset.test_runner_settings, # Optional[List[str]] = None\n}\nresults: Dict = asyncio.run(run_standards_validation_tests(**test_data))\nprint(results)\n```\n\n### One Hop Test\n\nTo run \"One Hop\" knowledge graph navigation tests validating query outputs from a knowledge graph TRAPI component:\n\n```python\nfrom typing import Dict\nimport asyncio\nfrom one_hop_test_runner import run_one_hop_tests\n\ntest_data = {\n # One test edge (asset)\n \"subject_id\": \"DRUGBANK:DB01592\",\n \"subject_category\": \"biolink:SmallMolecule\",\n \"predicate_id\": \"biolink:has_side_effect\",\n \"object_id\": \"MONDO:0011426\",\n \"object_category\": \"biolink:Disease\",\n \"components\": [\"arax\", \"molepro\"]\n #\n # \"environment\": environment, # Optional[TestEnvEnum] = None; default: 'TestEnvEnum.ci' if not given\n # \"trapi_version\": trapi_version, # Optional[str] = None; latest community release if not given\n # \"biolink_version\": biolink_version, # Optional[str] = None; current Biolink Toolkit default if not given\n # \"runner_settings\": asset.test_runner_settings, # Optional[List[str]] = None\n}\nresults: Dict = asyncio.run(run_one_hop_tests(**test_data))\nprint(results)\n```\n\nThe above wrapper method runs all related TestCases derived from the specified TestAsset (i.e. subject_id, etc.) without any special test parameters. If more fine-grained testing is desired, a subset of the underlying TRAPI queries can be run directly, something like this (here, we ignore the TestCases 'by_subject', 'inverse_by_new_subject' and 'by_object', and specify the 'strict_validation' parameter of True to Biolink Model validation, as understood by the **reasoner-validator** code running behind the scenes):\n\n```python\nfrom typing import Dict\nimport asyncio\nfrom standards_validation_test_runner import StandardsValidationTest\nfrom graph_validation_tests.utils.unit_test_templates import (\n # by_subject,\n # inverse_by_new_subject,\n # by_object,\n raise_subject_entity,\n raise_object_entity,\n raise_object_by_subject,\n raise_predicate_by_subject\n)\n\ntest_data = {\n # One test edge (asset)\n \"subject_id\": \"DRUGBANK:DB01592\",\n \"subject_category\": \"biolink:SmallMolecule\",\n \"predicate_id\": \"biolink:has_side_effect\",\n \"object_id\": \"MONDO:0011426\",\n \"object_category\": \"biolink:Disease\",\n \"components\": [\"arax\", \"molepro\"],\n \"environment\": \"test\",\n \"trapi_version\": \"1.5.0-beta\",\n \"biolink_version\": \"4.1.6\",\n \"runner_settings\": \"Inferred\"\n}\ntrapi_generators = [\n # by_subject,\n # inverse_by_new_subject,\n # by_object,\n raise_subject_entity,\n raise_object_entity,\n raise_object_by_subject,\n raise_predicate_by_subject\n]\n\n# A test runner specific parameter passed through\nkwargs = {\n \"strict_validation\": True\n}\nresults: Dict = asyncio.run(StandardsValidationTest.run_tests(\n **test_data, trapi_generators=trapi_generators, **kwargs)\n)\n```\n\nNote that the trapi_generation variables - defined in the **graph_validation_test.utils.unit_test_templates** module - are all simply Python functions returning TRAPI JSON messages to send to the target components. In principle, if one understands what those functions are doing, you could write your own methods to do other kinds of TRAPI queries whose output can then be validated against the specified TRAPI and Biolink Model releases.\n\n### Sample Output\n\nThis is a sample of what the JSON output from test runs currently looks like (this sample came from a OneHopTest run).\n\n```json\n{ \n \"pks\": {\n \"molepro\": \"molepro\"\n },\n \"results\": {\n \"TestAsset_1-by_subject\": {\n \"molepro\": {\n \"status\": \"FAILED\",\n \"messages\": {\n \"error\": {\n \"error.trapi.response.knowledge_graph.missing_expected_edge\": {\n \"global\": {\n \"TestAsset_1|(CHEBI:58579#biolink:SmallMolecule)-[biolink:is_active_metabolite_of]->(UniProtKB:Q9NQ88#biolink:Protein)\": null\n }\n }\n }\n }\n }\n },\n \"TestAsset_1-inverse_by_new_subject\": {\n \"molepro\": {\n \"status\": \"FAILED\",\n \"messages\": {\n \"critical\": {\n \"critical.trapi.request.invalid\": {\n \"global\": {\n \"predicate 'biolink:is_active_metabolite_of'\": [\n {\n \"context\": \"inverse_by_new_subject\",\n \"reason\": \"is an unknown or has no inverse?\"\n }\n ]\n }\n }\n }\n }\n }\n }\n }\n}\n```\n\n## Releases\n\nA [full change log](CHANGELOG.md) is provided documenting each release, but we summarize key release limitations here:\n\n### v0.0.\\* Releases\n\n- This initial code release only supports testing of [Translator SmartAPI Registry](https://smart-api.info/registry/translator) catalogued components which are TRAPI implementations for Translator Autonomous Relay Agent (ARA) and Knowledge Providers (KP), but _not_ direct testing of the Translator Autonomous Relay System (ARS) or Translator user interface (UI)\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Validation of Translator Knowledge Graphs - TRAPI, Biolink Model and One Hop navigation",
"version": "0.1.3",
"project_urls": {
"Bug Tracker": "https://github.com/TranslatorSRI/graph-validation-test-runners/issues",
"Change Log": "https://github.com/TranslatorSRI/graph-validation-test-runners/blob/master/CHANGELOG.md",
"Documentation": "https://github.com/TranslatorSRI/graph-validation-test-runners/blob/main/README.md",
"Homepage": "https://github.com/TranslatorSRI",
"Repository": "https://github.com/TranslatorSRI/graph-validation-test-runners"
},
"split_keywords": [
"ncats",
" biomedical data translator",
" translator",
" reasonerapi",
" trapi",
" validation",
" biolink model"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "cbea923d016c65508647d44560bcab64aceac168552660e9d3ff7d230abfa71c",
"md5": "502ac45dade424f3b0b9ed0c3a24b921",
"sha256": "d35a1864df7d8f5b29d7865b49c31a9429c80bfeec45d9d25c7544eeabc4f710"
},
"downloads": -1,
"filename": "graph_validation_test_runners-0.1.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "502ac45dade424f3b0b9ed0c3a24b921",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.12,>=3.9",
"size": 37704,
"upload_time": "2024-05-18T09:22:58",
"upload_time_iso_8601": "2024-05-18T09:22:58.476418Z",
"url": "https://files.pythonhosted.org/packages/cb/ea/923d016c65508647d44560bcab64aceac168552660e9d3ff7d230abfa71c/graph_validation_test_runners-0.1.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d38eb44eef8c7bd2d47061cd6b896c908799d61a73a2f73c2d33ecc82a2ed57a",
"md5": "8df00785e4ceb1c183bb417f88dbf84e",
"sha256": "c9fe52c507e17a2d2fa17754ee1885576ab62cd8b6aaaf7ab4465d549148b094"
},
"downloads": -1,
"filename": "graph_validation_test_runners-0.1.3.tar.gz",
"has_sig": false,
"md5_digest": "8df00785e4ceb1c183bb417f88dbf84e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.12,>=3.9",
"size": 47935,
"upload_time": "2024-05-18T09:23:00",
"upload_time_iso_8601": "2024-05-18T09:23:00.018579Z",
"url": "https://files.pythonhosted.org/packages/d3/8e/b44eef8c7bd2d47061cd6b896c908799d61a73a2f73c2d33ecc82a2ed57a/graph_validation_test_runners-0.1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-18 09:23:00",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "TranslatorSRI",
"github_project": "graph-validation-test-runners",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "graph-validation-test-runners"
}
JSON
