# Schematic
[](https://actions-badge.atrox.dev/Sage-Bionetworks/schematic/goto?ref=develop) [](https://sage-schematic.readthedocs.io/en/develop/?badge=develop) [](https://badge.fury.io/py/schematicpy)
# Table of contents
- [Introduction](#introduction)
- [Installation](#installation)
- [Installation Requirements](#installation-requirements)
- [Installation guide for data curator app](#installation-guide-for-data-curator-app)
- [Installation guide for developers/contributors](#installation-guide-for-developerscontributors)
- [Other Contribution Guidelines](#other-contribution-guidelines)
- [Update readthedocs documentation](#update-readthedocs-documentation)
- [Command Line Usage](#command-line-usage)
- [Testing](#testing)
- [Updating Synapse test resources](#updating-synapse-test-resources)
- [Code Style](#code-style)
- [Contributors](#contributors)
# Introduction
SCHEMATIC is an acronym for _Schema Engine for Manifest Ingress and Curation_. The Python based infrastructure provides a _novel_ schema-based, metadata ingress ecosystem, that is meant to streamline the process of biomedical dataset annotation, metadata validation and submission to a data repository for various data contributors.
# Installation
## Installation Requirements
* Python version 3.9.0≤x<3.11.0
* You need to be a registered and certified user on [`synapse.org`](https://www.synapse.org/)
Note: Our credential policy for Google credentials in order to create Google sheet files from Schematic, see tutorial ['HERE'](https://scribehow.com/shared/Get_Credentials_for_Google_Drive_and_Google_Sheets_APIs_to_use_with_schematicpy__yqfcJz_rQVeyTcg0KQCINA). If you plan to use `config.yml`, please ensure that the path of `schematic_service_account_creds.json` is indicated there (see `google_sheets > service_account_creds` section)
## Installation guide for data curator app
Create and activate a virtual environment within which you can install the package:
```
python3 -m venv .venv
source .venv/bin/activate
```
Note: Python 3 has a built-in support for virtual environment [venv](https://docs.python.org/3/library/venv.html#module-venv) so you no longer need to install virtualenv.
Install and update the package using [pip](https://pip.pypa.io/en/stable/quickstart/):
```
python3 -m pip install schematicpy
```
If you run into error: Failed building wheel for numpy, the error might be able to resolve by upgrading pip. Please try to upgrade pip by:
```
pip3 install --upgrade pip
```
## Installation guide for developers/contributors
When contributing to this repository, please first discuss the change you wish to make via issue, email, or any other method with the owners of this repository before making a change.
Please note we have a [code of conduct](CODE_OF_CONDUCT.md), please follow it in all your interactions with the project.
### Development environment setup
1. Clone the `schematic` package repository.
```
git clone https://github.com/Sage-Bionetworks/schematic.git
```
2. Install `poetry` (version 1.3.0 or later) using either the [official installer](https://python-poetry.org/docs/#installing-with-the-official-installer) or [pipx](https://python-poetry.org/docs/#installing-with-pipx). If you have an older installation of Poetry, we recommend uninstalling it first.
3. Start the virtual environment by doing:
```
poetry shell
```
4. Install the dependencies by doing:
```
poetry install
```
This command will install the dependencies based on what we specify in poetry.lock. If this step is taking a long time, try to go back to step 2 and check your version of poetry. Alternatively, you could also try deleting the lock file and regenerate it by doing `poetry install` (Please note this method should be used as a last resort because this would force other developers to change their development environment)
If you want to install the API you will need to install those dependencies as well:
```
poetry install --extras "api"
```
If you want to install the uwsgi:
```
poetry install --extras "api"
```
5. Fill in credential files:
*Note*: If you won't interact with Synapse, please ignore this section.
There are two main configuration files that need to be edited:
config.yml
and [synapseConfig](https://raw.githubusercontent.com/Sage-Bionetworks/synapsePythonClient/v2.3.0-rc/synapseclient/.synapseConfig)
<strong>Configure .synapseConfig File</strong>
Download a copy of the ``.synapseConfig`` file, open the file in the
editor of your choice and edit the `username` and `authtoken` attribute under the `authentication` section
*Note*: You could also visit [configparser](https://docs.python.org/3/library/configparser.html#module-configparser>) doc to see the format that `.synapseConfig` must have. For instance:
>[authentication]<br> username = ABC <br> authtoken = abc
<strong>Configure config.yml File</strong>
There are some defaults in schematic that can be configured. These fields are in ``config_example.yml``:
```text
# This is an example config for Schematic.
# All listed values are those that are the default if a config is not used.
# Save this as config.yml, this will be gitignored.
# Remove any fields in the config you don't want to change
# Change the values of any fields you do want to change
# This describes where assets such as manifests are stored
asset_store:
# This is when assets are stored in a synapse project
synapse:
# Synapse ID of the file view listing all project data assets.
master_fileview_id: "syn23643253"
# Path to the synapse config file, either absolute or relative to this file
config: ".synapseConfig"
# Base name that manifest files will be saved as
manifest_basename: "synapse_storage_manifest"
# This describes information about manifests as it relates to generation and validation
manifest:
# Location where manifests will saved to
manifest_folder: "manifests"
# Title or title prefix given to generated manifest(s)
title: "example"
# Data types of manifests to be generated or data type (singular) to validate manifest against
data_type:
- "Biospecimen"
- "Patient"
# Describes the location of your schema
model:
# Location of your schema jsonld, it must be a path relative to this file or absolute
location: "tests/data/example.model.jsonld"
# This section is for using google sheets with Schematic
google_sheets:
# The Synapse id of the Google service account credentials.
service_acct_creds_synapse_id: "syn25171627"
# Path to the synapse config file, either absolute or relative to this file
service_acct_creds: "schematic_service_account_creds.json"
# When doing google sheet validation (regex match) with the validation rules.
# true is alerting the user and not allowing entry of bad values.
# false is warning but allowing the entry on to the sheet.
strict_validation: true
```
If you want to change any of these copy ``config_example.yml`` to ``config.yml``, change any fields you want to, and remove any fields you don't.
For example if you wanted to change the folder where manifests are downloaded your config should look like:
```text
manifest:
manifest_folder: "my_manifest_folder_path"
```
_Note_: `config.yml` is ignored by git.
_Note_: Paths can be specified relative to the `config.yml` file or as absolute paths.
6. Login to Synapse by using the command line
On the CLI in your virtual environment, run the following command:
```
synapse login -u <synapse username> -p <synapse password> --rememberMe
```
7. Obtain Google credential Files
Running `schematic init` is no longer supported due to security concerns. To obtain `schematic_service_account_creds.json`, please follow the instructions [here](https://scribehow.com/shared/Enable_Google_Drive_and_Google_Sheets_APIs_for_project__yqfcJz_rQVeyTcg0KQCINA).
> As v22.12.1 version of schematic, using `token` mode of authentication (in other words, using `token.pickle` and `credentials.json`) is no longer supported due to Google's decision to move away from using OAuth out-of-band (OOB) flow. Click [here](https://developers.google.com/identity/protocols/oauth2/resources/oob-migration) to learn more.
*Notes*: Use the ``schematic_service_account_creds.json`` file for the service
account mode of authentication (*for Google services/APIs*). Service accounts
are special Google accounts that can be used by applications to access Google APIs
programmatically via OAuth2.0, with the advantage being that they do not require
human authorization.
*Background*: schematic uses Google’s API to generate google sheet templates that users fill in to provide (meta)data.
Most Google sheet functionality could be authenticated with service account. However, more complex Google sheet functionality
requires token-based authentication. As browser support that requires the token-based authentication diminishes, we are hoping to deprecate
token-based authentication and keep only service account authentication in the future.
8. Set up pre-commit hooks
This repository is configured to utilize pre-commit hooks as part of the development process. To enable these hooks, please run the following command and look for the following success message:
```
$ pre-commit install
pre-commit installed at .git/hooks/pre-commit
```
### Development process instruction
For new features, bugs, enhancements
1. Pull the latest code from [develop branch in the upstream repo](https://github.com/Sage-Bionetworks/schematic)
2. Checkout a new branch develop-<feature/fix-name> from the develop branch
3. Do development on branch develop-<feature/fix-name>
a. may need to ensure that schematic poetry toml and lock files are compatible with your local environment
4. Add changed files for tracking and commit changes using [best practices](https://www.perforce.com/blog/vcs/git-best-practices-git-commit)
5. Have granular commits: not “too many” file changes, and not hundreds of code lines of changes
6. Commits with work in progress are encouraged:
a. add WIP to the beginning of the commit message for “Work In Progress” commits
7. Keep commit messages descriptive but less than a page long, see best practices
8. Push code to develop-<feature/fix-name> in upstream repo
9. Branch out off develop-<feature/fix-name> if needed to work on multiple features associated with the same code base
10. After feature work is complete and before creating a PR to the develop branch in upstream
a. ensure that code runs locally
b. test for logical correctness locally
c. wait for git workflow to complete (e.g. tests are run) on github
11. Create a PR from develop-<feature/fix-name> into the develop branch of the upstream repo
12. Request a code review on the PR
13. Once code is approved merge in the develop branch
14. Delete the develop-<feature/fix-name> branch
*Note*: Make sure you have the latest version of the `develop` branch on your local machine.
## Installation Guide - Docker
1. Install docker from https://www.docker.com/ . <br>
2. Identify docker image of interest from [Schematic DockerHub](https://hub.docker.com/r/sagebionetworks/schematic/tags) <br>
Ex `docker pull sagebionetworks/schematic:latest` from the CLI or, run `docker compose up` after cloning the schematic github repo <br>
in this case, `sagebionetworks/schematic:latest` is the name of the image chosen
3. Run Schematic Command with `docker run <flags> <schematic command and args>`. <br>
<t> - For more information on flags for `docker run` and what they do, visit the [Docker Documentation](https://docs.docker.com/engine/reference/commandline/run/) <br>
<t> - These example commands assume that you have navigated to the directory you want to run schematic from. To specify your working directory, use `$(pwd)` on MacOS/Linux or `%cd%` on Windows. <br>
<t> - If not using the latest image, then the full name should be specified: ie `sagebionetworks/schematic:commit-e611e4a` <br>
<t> - If using local image created by `docker compose up`, then the docker image name should be changed: i.e. `schematic_schematic` <br>
<t> - Using the `--name` flag sets the name of the container running locally on your machine <br>
### Example For REST API <br>
#### Use file path of `config.yml` to run API endpoints:
```
docker run --rm -p 3001:3001 \
-v $(pwd):/schematic -w /schematic --name schematic \
-e SCHEMATIC_CONFIG=/schematic/config.yml \
-e GE_HOME=/usr/src/app/great_expectations/ \
sagebionetworks/schematic \
python /usr/src/app/run_api.py
```
#### Use content of `config.yml` and `schematic_service_account_creds.json`as an environment variable to run API endpoints:
1. save content of `config.yml` as to environment variable `SCHEMATIC_CONFIG_CONTENT` by doing: `export SCHEMATIC_CONFIG_CONTENT=$(cat /path/to/config.yml)`
2. Similarly, save the content of `schematic_service_account_creds.json` as `SERVICE_ACCOUNT_CREDS` by doing: `export SERVICE_ACCOUNT_CREDS=$(cat /path/to/schematic_service_account_creds.json)`
3. Pass `SCHEMATIC_CONFIG_CONTENT` and `schematic_service_account_creds` as environment variables by using `docker run`
```
docker run --rm -p 3001:3001 \
-v $(pwd):/schematic -w /schematic --name schematic \
-e GE_HOME=/usr/src/app/great_expectations/ \
-e SCHEMATIC_CONFIG_CONTENT=$SCHEMATIC_CONFIG_CONTENT \
-e SERVICE_ACCOUNT_CREDS=$SERVICE_ACCOUNT_CREDS \
sagebionetworks/schematic \
python /usr/src/app/run_api.py
```
### Example For Schematic on mac/linux <br>
To run example below, first clone schematic into your home directory `git clone https://github.com/sage-bionetworks/schematic ~/schematic` <br>
Then update .synapseConfig with your credentials
```
docker run \
-v ~/schematic:/schematic \
-w /schematic \
-e SCHEMATIC_CONFIG=/schematic/config.yml \
-e GE_HOME=/usr/src/app/great_expectations/ \
sagebionetworks/schematic schematic model \
-c /schematic/config.yml validate \
-mp /schematic/tests/data/mock_manifests/Valid_Test_Manifest.csv \
-dt MockComponent \
-js /schematic/tests/data/example.model.jsonld
```
### Example For Schematic on Windows <br>
```
docker run -v %cd%:/schematic \
-w /schematic \
-e GE_HOME=/usr/src/app/great_expectations/ \
sagebionetworks/schematic \
schematic model \
-c config.yml validate -mp tests/data/mock_manifests/inValid_Test_Manifest.csv -dt MockComponent -js /schematic/data/example.model.jsonld
```
# Other Contribution Guidelines
## Updating readthedocs documentation
1. `cd docs`
2. After making relevant changes, you could run the `make html` command to re-generate the `build` folder.
3. Please contact the dev team to publish your updates
*Other helpful resources*:
1. [Getting started with Sphinx](https://haha.readthedocs.io/en/latest/intro/getting-started-with-sphinx.html)
2. [Installing Sphinx](https://haha.readthedocs.io/en/latest/intro/getting-started-with-sphinx.html)
## Update toml file and lock file
If you install external libraries by using `poetry add <name of library>`, please make sure that you include `pyproject.toml` and `poetry.lock` file in your commit.
## Reporting bugs or feature requests
You can **create bug and feature requests** through [Sage Bionetwork's FAIR Data service desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/5/group/8). Providing enough details to the developers to verify and troubleshoot your issue is paramount:
- **Provide a clear and descriptive title as well as a concise summary** of the issue to identify the problem.
- **Describe the exact steps which reproduce the problem** in as many details as possible.
- **Describe the behavior you observed after following the steps** and point out what exactly is the problem with that behavior.
- **Explain which behavior you expected to see** instead and why.
- **Provide screenshots of the expected or actual behaviour** where applicable.
# Command Line Usage
Please visit more documentation [here](https://sage-schematic.readthedocs.io/en/develop/cli_reference.html)
# Testing
All code added to the client must have tests. The Python client uses pytest to run tests. The test code is located in the [tests](https://github.com/Sage-Bionetworks/schematic/tree/develop-docs-update/tests) subdirectory.
You can run the test suite in the following way:
```
pytest -vs tests/
```
## Updating Synapse test resources
1. Duplicate the entity being updated (or folder if applicable).
2. Edit the duplicates (_e.g._ annotations, contents, name).
3. Update the test suite in your branch to use these duplicates, including the expected values in the test assertions.
4. Open a PR as per the usual process (see above).
5. Once the PR is merged, leave the original copies on Synapse to maintain support for feature branches that were forked from `develop` before your update.
- If the old copies are problematic and need to be removed immediately (_e.g._ contain sensitive data), proceed with the deletion and alert the other contributors that they need to merge the latest `develop` branch into their feature branches for their tests to work.
# Code style
* Please consult the [Google Python style guide](http://google.github.io/styleguide/pyguide.html) prior to contributing code to this project.
* Be consistent and follow existing code conventions and spirit.
# Contributors
Main contributors and developers:
- [Milen Nikolov](https://github.com/milen-sage)
- [Mialy DeFelice](https://github.com/mialy-defelice)
- [Sujay Patil](https://github.com/sujaypatil96)
- [Bruno Grande](https://github.com/BrunoGrandePhD)
- [Robert Allaway](https://github.com/allaway)
- [Gianna Jordan](https://github.com/giajordan)
- [Lingling Peng](https://github.com/linglp)
Raw data
{
"_id": null,
"home_page": "https://github.com/Sage-Bionetworks/schematic",
"name": "schematicpy",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.11,>=3.9.0",
"maintainer_email": null,
"keywords": "schema, metadata, validation, data model, linked data",
"author": "Milen Nikolov",
"author_email": "milen.nikolov@sagebase.org",
"download_url": "https://files.pythonhosted.org/packages/fb/8b/881a00418000f1d40dc02306f093668bd30e96313afe1a0725cc0d97b168/schematicpy-24.4.1.tar.gz",
"platform": null,
"description": "# Schematic\n[](https://actions-badge.atrox.dev/Sage-Bionetworks/schematic/goto?ref=develop) [](https://sage-schematic.readthedocs.io/en/develop/?badge=develop) [](https://badge.fury.io/py/schematicpy)\n\n# Table of contents\n- [Introduction](#introduction)\n- [Installation](#installation)\n - [Installation Requirements](#installation-requirements)\n - [Installation guide for data curator app](#installation-guide-for-data-curator-app)\n - [Installation guide for developers/contributors](#installation-guide-for-developerscontributors)\n- [Other Contribution Guidelines](#other-contribution-guidelines)\n - [Update readthedocs documentation](#update-readthedocs-documentation)\n- [Command Line Usage](#command-line-usage)\n- [Testing](#testing)\n - [Updating Synapse test resources](#updating-synapse-test-resources)\n- [Code Style](#code-style)\n- [Contributors](#contributors)\n\n# Introduction\nSCHEMATIC is an acronym for _Schema Engine for Manifest Ingress and Curation_. The Python based infrastructure provides a _novel_ schema-based, metadata ingress ecosystem, that is meant to streamline the process of biomedical dataset annotation, metadata validation and submission to a data repository for various data contributors.\n\n# Installation\n## Installation Requirements\n* Python version 3.9.0\u2264x<3.11.0\n* You need to be a registered and certified user on [`synapse.org`](https://www.synapse.org/)\n\nNote: Our credential policy for Google credentials in order to create Google sheet files from Schematic, see tutorial ['HERE'](https://scribehow.com/shared/Get_Credentials_for_Google_Drive_and_Google_Sheets_APIs_to_use_with_schematicpy__yqfcJz_rQVeyTcg0KQCINA). If you plan to use `config.yml`, please ensure that the path of `schematic_service_account_creds.json` is indicated there (see `google_sheets > service_account_creds` section)\n\n\n## Installation guide for data curator app\n\nCreate and activate a virtual environment within which you can install the package:\n\n```\npython3 -m venv .venv\nsource .venv/bin/activate\n```\n\nNote: Python 3 has a built-in support for virtual environment [venv](https://docs.python.org/3/library/venv.html#module-venv) so you no longer need to install virtualenv.\n\nInstall and update the package using [pip](https://pip.pypa.io/en/stable/quickstart/):\n\n```\npython3 -m pip install schematicpy\n```\n\nIf you run into error: Failed building wheel for numpy, the error might be able to resolve by upgrading pip. Please try to upgrade pip by:\n\n```\npip3 install --upgrade pip\n```\n\n## Installation guide for developers/contributors \n\nWhen contributing to this repository, please first discuss the change you wish to make via issue, email, or any other method with the owners of this repository before making a change.\n\nPlease note we have a [code of conduct](CODE_OF_CONDUCT.md), please follow it in all your interactions with the project.\n\n### Development environment setup\n1. Clone the `schematic` package repository.\n```\ngit clone https://github.com/Sage-Bionetworks/schematic.git\n```\n2. Install `poetry` (version 1.3.0 or later) using either the [official installer](https://python-poetry.org/docs/#installing-with-the-official-installer) or [pipx](https://python-poetry.org/docs/#installing-with-pipx). If you have an older installation of Poetry, we recommend uninstalling it first. \n\n3. Start the virtual environment by doing: \n```\npoetry shell\n```\n4. Install the dependencies by doing: \n```\npoetry install\n```\nThis command will install the dependencies based on what we specify in poetry.lock. If this step is taking a long time, try to go back to step 2 and check your version of poetry. Alternatively, you could also try deleting the lock file and regenerate it by doing `poetry install` (Please note this method should be used as a last resort because this would force other developers to change their development environment)\n\nIf you want to install the API you will need to install those dependencies as well:\n\n```\npoetry install --extras \"api\"\n```\n\nIf you want to install the uwsgi:\n\n```\npoetry install --extras \"api\"\n```\n\n5. Fill in credential files: \n*Note*: If you won't interact with Synapse, please ignore this section.\n\nThere are two main configuration files that need to be edited:\nconfig.yml\nand [synapseConfig](https://raw.githubusercontent.com/Sage-Bionetworks/synapsePythonClient/v2.3.0-rc/synapseclient/.synapseConfig)\n\n<strong>Configure .synapseConfig File</strong>\n\nDownload a copy of the ``.synapseConfig`` file, open the file in the\neditor of your choice and edit the `username` and `authtoken` attribute under the `authentication` section \n\n*Note*: You could also visit [configparser](https://docs.python.org/3/library/configparser.html#module-configparser>) doc to see the format that `.synapseConfig` must have. For instance:\n>[authentication]<br> username = ABC <br> authtoken = abc\n\n<strong>Configure config.yml File</strong>\n\nThere are some defaults in schematic that can be configured. These fields are in ``config_example.yml``:\n\n```text\n\n# This is an example config for Schematic.\n# All listed values are those that are the default if a config is not used.\n# Save this as config.yml, this will be gitignored.\n# Remove any fields in the config you don't want to change\n# Change the values of any fields you do want to change\n\n\n# This describes where assets such as manifests are stored\nasset_store:\n # This is when assets are stored in a synapse project\n synapse:\n # Synapse ID of the file view listing all project data assets.\n master_fileview_id: \"syn23643253\"\n # Path to the synapse config file, either absolute or relative to this file\n config: \".synapseConfig\"\n # Base name that manifest files will be saved as\n manifest_basename: \"synapse_storage_manifest\"\n\n# This describes information about manifests as it relates to generation and validation\nmanifest:\n # Location where manifests will saved to\n manifest_folder: \"manifests\"\n # Title or title prefix given to generated manifest(s)\n title: \"example\"\n # Data types of manifests to be generated or data type (singular) to validate manifest against\n data_type:\n - \"Biospecimen\"\n - \"Patient\"\n\n# Describes the location of your schema\nmodel:\n # Location of your schema jsonld, it must be a path relative to this file or absolute\n location: \"tests/data/example.model.jsonld\"\n\n# This section is for using google sheets with Schematic\ngoogle_sheets:\n # The Synapse id of the Google service account credentials.\n service_acct_creds_synapse_id: \"syn25171627\"\n # Path to the synapse config file, either absolute or relative to this file\n service_acct_creds: \"schematic_service_account_creds.json\"\n # When doing google sheet validation (regex match) with the validation rules.\n # true is alerting the user and not allowing entry of bad values.\n # false is warning but allowing the entry on to the sheet.\n strict_validation: true\n```\n\nIf you want to change any of these copy ``config_example.yml`` to ``config.yml``, change any fields you want to, and remove any fields you don't.\n\nFor example if you wanted to change the folder where manifests are downloaded your config should look like:\n\n```text\n\nmanifest:\n manifest_folder: \"my_manifest_folder_path\"\n```\n\n_Note_: `config.yml` is ignored by git.\n\n_Note_: Paths can be specified relative to the `config.yml` file or as absolute paths.\n\n6. Login to Synapse by using the command line\nOn the CLI in your virtual environment, run the following command: \n```\nsynapse login -u <synapse username> -p <synapse password> --rememberMe\n```\n\n7. Obtain Google credential Files\nRunning `schematic init` is no longer supported due to security concerns. To obtain `schematic_service_account_creds.json`, please follow the instructions [here](https://scribehow.com/shared/Enable_Google_Drive_and_Google_Sheets_APIs_for_project__yqfcJz_rQVeyTcg0KQCINA). \n\n> As v22.12.1 version of schematic, using `token` mode of authentication (in other words, using `token.pickle` and `credentials.json`) is no longer supported due to Google's decision to move away from using OAuth out-of-band (OOB) flow. Click [here](https://developers.google.com/identity/protocols/oauth2/resources/oob-migration) to learn more. \n\n*Notes*: Use the ``schematic_service_account_creds.json`` file for the service\naccount mode of authentication (*for Google services/APIs*). Service accounts \nare special Google accounts that can be used by applications to access Google APIs \nprogrammatically via OAuth2.0, with the advantage being that they do not require \nhuman authorization. \n\n*Background*: schematic uses Google\u2019s API to generate google sheet templates that users fill in to provide (meta)data.\nMost Google sheet functionality could be authenticated with service account. However, more complex Google sheet functionality\nrequires token-based authentication. As browser support that requires the token-based authentication diminishes, we are hoping to deprecate\ntoken-based authentication and keep only service account authentication in the future. \n\n8. Set up pre-commit hooks\n\nThis repository is configured to utilize pre-commit hooks as part of the development process. To enable these hooks, please run the following command and look for the following success message:\n```\n$ pre-commit install\npre-commit installed at .git/hooks/pre-commit\n```\n\n### Development process instruction\n\nFor new features, bugs, enhancements\n\n1. Pull the latest code from [develop branch in the upstream repo](https://github.com/Sage-Bionetworks/schematic)\n2. Checkout a new branch develop-<feature/fix-name> from the develop branch\n3. Do development on branch develop-<feature/fix-name>\n a. may need to ensure that schematic poetry toml and lock files are compatible with your local environment\n4. Add changed files for tracking and commit changes using [best practices](https://www.perforce.com/blog/vcs/git-best-practices-git-commit)\n5. Have granular commits: not \u201ctoo many\u201d file changes, and not hundreds of code lines of changes\n6. Commits with work in progress are encouraged:\n a. add WIP to the beginning of the commit message for \u201cWork In Progress\u201d commits\n7. Keep commit messages descriptive but less than a page long, see best practices\n8. Push code to develop-<feature/fix-name> in upstream repo\n9. Branch out off develop-<feature/fix-name> if needed to work on multiple features associated with the same code base\n10. After feature work is complete and before creating a PR to the develop branch in upstream\n a. ensure that code runs locally\n b. test for logical correctness locally\n c. wait for git workflow to complete (e.g. tests are run) on github\n11. Create a PR from develop-<feature/fix-name> into the develop branch of the upstream repo\n12. Request a code review on the PR\n13. Once code is approved merge in the develop branch\n14. Delete the develop-<feature/fix-name> branch\n\n*Note*: Make sure you have the latest version of the `develop` branch on your local machine.\n\n## Installation Guide - Docker \n\n1. Install docker from https://www.docker.com/ . <br>\n2. Identify docker image of interest from [Schematic DockerHub](https://hub.docker.com/r/sagebionetworks/schematic/tags) <br>\n Ex `docker pull sagebionetworks/schematic:latest` from the CLI or, run `docker compose up` after cloning the schematic github repo <br>\n in this case, `sagebionetworks/schematic:latest` is the name of the image chosen\n3. Run Schematic Command with `docker run <flags> <schematic command and args>`. <br>\n<t> - For more information on flags for `docker run` and what they do, visit the [Docker Documentation](https://docs.docker.com/engine/reference/commandline/run/) <br>\n<t> - These example commands assume that you have navigated to the directory you want to run schematic from. To specify your working directory, use `$(pwd)` on MacOS/Linux or `%cd%` on Windows. <br>\n<t> - If not using the latest image, then the full name should be specified: ie `sagebionetworks/schematic:commit-e611e4a` <br>\n<t> - If using local image created by `docker compose up`, then the docker image name should be changed: i.e. `schematic_schematic` <br>\n<t> - Using the `--name` flag sets the name of the container running locally on your machine <br>\n\n### Example For REST API <br>\n\n#### Use file path of `config.yml` to run API endpoints: \n```\ndocker run --rm -p 3001:3001 \\\n -v $(pwd):/schematic -w /schematic --name schematic \\\n -e SCHEMATIC_CONFIG=/schematic/config.yml \\\n -e GE_HOME=/usr/src/app/great_expectations/ \\\n sagebionetworks/schematic \\\n python /usr/src/app/run_api.py\n``` \n\n#### Use content of `config.yml` and `schematic_service_account_creds.json`as an environment variable to run API endpoints: \n1. save content of `config.yml` as to environment variable `SCHEMATIC_CONFIG_CONTENT` by doing: `export SCHEMATIC_CONFIG_CONTENT=$(cat /path/to/config.yml)`\n\n2. Similarly, save the content of `schematic_service_account_creds.json` as `SERVICE_ACCOUNT_CREDS` by doing: `export SERVICE_ACCOUNT_CREDS=$(cat /path/to/schematic_service_account_creds.json)`\n\n3. Pass `SCHEMATIC_CONFIG_CONTENT` and `schematic_service_account_creds` as environment variables by using `docker run`\n\n```\ndocker run --rm -p 3001:3001 \\\n -v $(pwd):/schematic -w /schematic --name schematic \\\n -e GE_HOME=/usr/src/app/great_expectations/ \\\n -e SCHEMATIC_CONFIG_CONTENT=$SCHEMATIC_CONFIG_CONTENT \\\n -e SERVICE_ACCOUNT_CREDS=$SERVICE_ACCOUNT_CREDS \\\n sagebionetworks/schematic \\\n python /usr/src/app/run_api.py\n``` \n\n\n### Example For Schematic on mac/linux <br>\nTo run example below, first clone schematic into your home directory `git clone https://github.com/sage-bionetworks/schematic ~/schematic` <br>\nThen update .synapseConfig with your credentials\n```\ndocker run \\\n -v ~/schematic:/schematic \\\n -w /schematic \\\n -e SCHEMATIC_CONFIG=/schematic/config.yml \\\n -e GE_HOME=/usr/src/app/great_expectations/ \\\n sagebionetworks/schematic schematic model \\\n -c /schematic/config.yml validate \\\n -mp /schematic/tests/data/mock_manifests/Valid_Test_Manifest.csv \\\n -dt MockComponent \\\n -js /schematic/tests/data/example.model.jsonld\n``` \n\n### Example For Schematic on Windows <br>\n```\ndocker run -v %cd%:/schematic \\\n -w /schematic \\\n -e GE_HOME=/usr/src/app/great_expectations/ \\\n sagebionetworks/schematic \\\n schematic model \\\n -c config.yml validate -mp tests/data/mock_manifests/inValid_Test_Manifest.csv -dt MockComponent -js /schematic/data/example.model.jsonld\n```\n\n# Other Contribution Guidelines\n## Updating readthedocs documentation\n1. `cd docs`\n2. After making relevant changes, you could run the `make html` command to re-generate the `build` folder.\n3. Please contact the dev team to publish your updates\n\n*Other helpful resources*:\n\n1. [Getting started with Sphinx](https://haha.readthedocs.io/en/latest/intro/getting-started-with-sphinx.html)\n2. [Installing Sphinx](https://haha.readthedocs.io/en/latest/intro/getting-started-with-sphinx.html)\n\n## Update toml file and lock file\nIf you install external libraries by using `poetry add <name of library>`, please make sure that you include `pyproject.toml` and `poetry.lock` file in your commit.\n\n## Reporting bugs or feature requests\nYou can **create bug and feature requests** through [Sage Bionetwork's FAIR Data service desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/5/group/8). Providing enough details to the developers to verify and troubleshoot your issue is paramount:\n- **Provide a clear and descriptive title as well as a concise summary** of the issue to identify the problem.\n- **Describe the exact steps which reproduce the problem** in as many details as possible.\n- **Describe the behavior you observed after following the steps** and point out what exactly is the problem with that behavior.\n- **Explain which behavior you expected to see** instead and why.\n- **Provide screenshots of the expected or actual behaviour** where applicable.\n\n# Command Line Usage\nPlease visit more documentation [here](https://sage-schematic.readthedocs.io/en/develop/cli_reference.html)\n\n\n\n# Testing \n\nAll code added to the client must have tests. The Python client uses pytest to run tests. The test code is located in the [tests](https://github.com/Sage-Bionetworks/schematic/tree/develop-docs-update/tests) subdirectory.\n\nYou can run the test suite in the following way:\n\n```\npytest -vs tests/\n```\n\n## Updating Synapse test resources\n\n1. Duplicate the entity being updated (or folder if applicable).\n2. Edit the duplicates (_e.g._ annotations, contents, name).\n3. Update the test suite in your branch to use these duplicates, including the expected values in the test assertions.\n4. Open a PR as per the usual process (see above).\n5. Once the PR is merged, leave the original copies on Synapse to maintain support for feature branches that were forked from `develop` before your update.\n - If the old copies are problematic and need to be removed immediately (_e.g._ contain sensitive data), proceed with the deletion and alert the other contributors that they need to merge the latest `develop` branch into their feature branches for their tests to work.\n\n# Code style\n\n* Please consult the [Google Python style guide](http://google.github.io/styleguide/pyguide.html) prior to contributing code to this project.\n* Be consistent and follow existing code conventions and spirit.\n\n\n# Contributors\n\nMain contributors and developers:\n\n- [Milen Nikolov](https://github.com/milen-sage)\n- [Mialy DeFelice](https://github.com/mialy-defelice)\n- [Sujay Patil](https://github.com/sujaypatil96)\n- [Bruno Grande](https://github.com/BrunoGrandePhD)\n- [Robert Allaway](https://github.com/allaway)\n- [Gianna Jordan](https://github.com/giajordan)\n- [Lingling Peng](https://github.com/linglp)\n",
"bugtrack_url": null,
"license": null,
"summary": "Package for biomedical data model and metadata ingress management",
"version": "24.4.1",
"project_urls": {
"Documentation": "https://github.com/Sage-Bionetworks/schematic",
"Homepage": "https://github.com/Sage-Bionetworks/schematic",
"Repository": "https://github.com/Sage-Bionetworks/schematic"
},
"split_keywords": [
"schema",
" metadata",
" validation",
" data model",
" linked data"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "55e0a4bba3d2610816746e3ed4336e9a76a889ce5f2144da2fb88f25f7abd66d",
"md5": "63de1600e8f34283ed06cd633fb64ed0",
"sha256": "d31426243b1eb0d127f0979e79a056850369066c3d40df158496abe8716a5e6c"
},
"downloads": -1,
"filename": "schematicpy-24.4.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "63de1600e8f34283ed06cd633fb64ed0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.11,>=3.9.0",
"size": 413965,
"upload_time": "2024-04-25T22:44:16",
"upload_time_iso_8601": "2024-04-25T22:44:16.368407Z",
"url": "https://files.pythonhosted.org/packages/55/e0/a4bba3d2610816746e3ed4336e9a76a889ce5f2144da2fb88f25f7abd66d/schematicpy-24.4.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "fb8b881a00418000f1d40dc02306f093668bd30e96313afe1a0725cc0d97b168",
"md5": "a26ff2211d386b105ae4256fe76929fa",
"sha256": "be77fb812bf2340c1ecddbf16e35c41c274ea7e7b1f08a9d48bae9f8ed1ef7e4"
},
"downloads": -1,
"filename": "schematicpy-24.4.1.tar.gz",
"has_sig": false,
"md5_digest": "a26ff2211d386b105ae4256fe76929fa",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.11,>=3.9.0",
"size": 396110,
"upload_time": "2024-04-25T22:44:18",
"upload_time_iso_8601": "2024-04-25T22:44:18.773300Z",
"url": "https://files.pythonhosted.org/packages/fb/8b/881a00418000f1d40dc02306f093668bd30e96313afe1a0725cc0d97b168/schematicpy-24.4.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-25 22:44:18",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Sage-Bionetworks",
"github_project": "schematic",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "schematicpy"
}
JSON
