[](https://github.com/BlueBrain/morphology-workflows/releases)
[](https://github.com/BlueBrain/morphology-workflows/actions)
[](https://codecov.io/github/BlueBrain/morphology-workflows?branch=main)
[](https://github.com/BlueBrain/morphology-workflows/blob/main/LICENSE.txt)
[](https://morphology-workflows.readthedocs.io/)
[](https://doi.org/10.5281/zenodo.10809196)
# Morphology Workflows
This project contains several workflows for processing morphologies:
- **Initialize:** create the initial configuration files that the user can then update according to
its needs.
- **Fetch:** download morphologies from online database (like Allen, NeuroMorpho or MouseLight).
- **Placeholders:** compute the placeholders for a given region and mtype set.
- **Curate:** from raw morphologies, ensures that morphologies can be used with the rest of
BBP codes (click on the image for more details).
[](autoapi/tasks/workflows/index.html#tasks.workflows.Curate)
- **Annotate:** create various annotations on morphologies needed by specific BBP codes (click on
the image for more details).
[](autoapi/tasks/workflows/index.html#tasks.workflows.Annotate)
- **Repair:** process morphologies to correct for artifacts of in-vitro reconstruction (click on
the image for more details).
[](autoapi/tasks/workflows/index.html#tasks.workflows.Repair)
In a nutshell, the user provides a list of morphologies in a ``.csv`` file, with their names and
paths and a ``luigi.cfg`` configuration file. Each workflow is run independently and creates an
output folder, with one subfolder per task. In each, there will be a ``report.csv`` and a ``data``
folder containing the output files of the task if any. In the ``report.csv`` file, columns contain
paths to these files, additional information, error messages if the task failed on that
morphologies, as well as a flag ``is_valid``, used in subsequent tasks to filter valid morphologies.
At the end of each workflow, another ``report.csv`` file is created, with the main output columns of
each tasks, and a ``report.pdf`` containing a human readable summary of the result of the workflow.
Usually, the user should run the ``Curate`` workflow, then the ``Annotate`` workflow and finally the
``Repair`` workflow.
The complete documentation can be found here:
* stable: https://morphology-workflows.readthedocs.io/en/stable/
* latest: https://morphology-workflows.readthedocs.io/en/latest/
## Installation
This should be installed using pip:
```bash
pip install morphology-workflows
```
## Usage
### Create inputs for the workflows using the **Initialize** workflow
Usually, user should create a new project folder using this workflow. It will create the files
required to run the other workflows. For example, the command:
```bash
morphology-workflows Initialize --source-database NeuroMorpho
```
will create the inputs to download the morphologies from the NeuroMorpho database and the run the
other workflows on these morphologies.
If the user wants to provide arbitrary morphologies to the **Curate** / **Annotate** / **Repair**
workflows, the following command can be used:
```bash
morphology-workflows Initialize --input-dir <path-to-the-morphologies>
```
in order to create the dataset required by these workflows.
Once the **Initialize** workflows is complete, the user can update the outputs according to its
needs (see the next sections for details about each workflow).
### Create inputs for the **Fetch** workflow
This workflow helps fetching morphologies from online databases. This workflow only needs a
configuration file, which depends on the source from which the morphologies are fetched.
The possible sources are:
* NeuroMorpho
* MouseLight
* Allen
For each of them, the configuration file should be a JSON file containing a list of objects like
the following examples:
* NeuroMorpho:
```JSON
[
{
"species": "mouse",
"brain_region": "neocortex",
"cell_type": "interneuron",
"nb_morphologies": 10
}
]
```
* MouseLight:
```JSON
[
{
"brain_region": "neocortex",
"nb_morphologies": 10,
"seed": 0
}
]
```
* Allen:
```JSON
[
{
"species": "Mus musculus",
"brain_region": "VISli",
"cell_type": "pyramidal",
"nb_morphologies": 10,
"seed": 0
}
]
```
In these examples, the ``seed`` attribute is optional and is only used to sample which morphologies
are fetched among those which pass the filter.
Each `JSON` object in the list will give a set of morphologies to fetch, depending on the given
filters.
Note that all attributes are optional, so it's possible to pass an empty object to fetch all the
morphologies from the database, though it is not recommended.
### Create inputs for the **Curate**, **Annotate** and **Repair** workflows
The **Annotate** and **Repair** workflows should usually be run after the **Curate** workflow since
their inputs should be the outputs of the **Curate** workflow. But it is still possible to run them
on arbitrary inputs (though the morphologies must be valid, as the ones processed by the **Curate**
workflow).
The inputs should consist in:
* a directory containing the input morphologies.
* a CSV file with the following columns:
1. ``morph_path``: the path to the morphology file.
2. ``morph_name``: the name of the morphology.
3. ``mtype``: the morphology-type of the morphology (this column is optional).
Any other column is kept into the results but not used in the workflows.
Note that the column names should not contain spaces between commas and names.
* a ``luigi.cfg`` file containing the configuration for all the tasks of the workflow.
* an optional ``logging.conf`` file containing the logging configuration. If you prefer default logging
behavior, remove this file and comment line in ``logging_conf_file = logging.conf`` in ``luigi.cfg``.
The [examples](https://github.com/BlueBrain/morphology-workflows/tree/main/examples) folder contains
examples for the ``luigi.cfg`` and ``logging.conf`` files.
These inputs can be created using the **Fetch** workflow that downloads morphlogies from online
databases and create the dataset in the correct format.
### Run the workflows
These workflows are based on the ``luigi`` library but can be run via the command line interface.
For example, you can run the ``Curate`` workflow with the following command:
```bash
morphology-workflows Curate
```
> **NOTE:** This command must be executed from a directory containing a ``luigi.cfg`` file.
> An example of such file is given in the ``examples`` directory.
By default, a local scheduler is used but it is also possible to use a Luigi's master scheduler
using the `-m / --master-scheduler` trigger:
```bash
morphology-workflows -m Curate
```
Once the **Curate** workflow has run, the **Annotate** and **Repair** workflows can be run
directly, they will just take the results of the **Curate** workflow as input:
```bash
morphology-workflows Annotate
morphology-workflows Repair
```
Note that it is also possible to run the 3 workflows directly because each workflow depends on each other:
```bash
morphology-workflows Repair
```
More details can be found in the command line interface section of the documentation or by running
the commands:
```bash
morphology-workflows --help
morphology-workflows <workflow> --help
```
## Results
Each workflow will create several new directories, one for each sub-step. These new directories can
be nested into a global result directory for each workflow, using the ``result_path`` parameter.
These directories contain intermediate data so it is possible to understand why a morphology
could not be validated at a given step.
The invalid morphologies should be manually fixed before being processed again by the workflows.
The main workflows (**Curate**, **Annotate** and **Repair**) will also create a final CSV file
which contains most of the relevant data of the workflow (main sub-step results and final
morphology paths). Finally, the **Repair** workflow can generate morphology releases that contain
the final morphologies that could be validated and automatically fixed by the workflow. Usually
these morphologies are the most relevant ones for later use.
The morphologies marked as invalid may be fixed manually before rerunning the workflow on them.
## Examples
The [examples](https://github.com/BlueBrain/morphology-workflows/tree/main/examples) folder contains
a simple example that will fetch and process a set of morphologies.
A ``dataset.csv`` file is provided which is taken as input for the workflows. A ``luigi.cfg`` file
is also provided to give a default configuration for the workflows.
This example can simply be run using the following command:
```bash
./run_curation.sh
```
This script will create a new directory ``out_curated`` which will contain the report and all the
results.
## Citation
When you use this software, we kindly ask you to cite the following DOI:
[](https://doi.org/10.5281/zenodo.10809196)
## Funding & Acknowledgment
The development of this software was supported by funding to the Blue Brain Project, a research
center of the École polytechnique fédérale de Lausanne (EPFL), from the Swiss government’s ETH Board
of the Swiss Federal Institutes of Technology.
For license and authors, see `LICENSE.txt` and `AUTHORS.md` respectively.
Copyright © 2021-2022 Blue Brain Project/EPFL
Raw data
{
"_id": null,
"home_page": "https://morphology-workflows.readthedocs.io",
"name": "morphology-workflows",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": null,
"author": "Blue Brain Project, EPFL",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/65/23/86b650c7b11e03b083296e98a4a9decadabf4a8542325ae2d84e52a86820/morphology_workflows-0.11.0.tar.gz",
"platform": null,
"description": "\n\n[](https://github.com/BlueBrain/morphology-workflows/releases)\n[](https://github.com/BlueBrain/morphology-workflows/actions)\n[](https://codecov.io/github/BlueBrain/morphology-workflows?branch=main)\n[](https://github.com/BlueBrain/morphology-workflows/blob/main/LICENSE.txt)\n[](https://morphology-workflows.readthedocs.io/)\n[](https://doi.org/10.5281/zenodo.10809196)\n\n\n# Morphology Workflows\n\nThis project contains several workflows for processing morphologies:\n- **Initialize:** create the initial configuration files that the user can then update according to\n its needs.\n- **Fetch:** download morphologies from online database (like Allen, NeuroMorpho or MouseLight).\n- **Placeholders:** compute the placeholders for a given region and mtype set.\n- **Curate:** from raw morphologies, ensures that morphologies can be used with the rest of\n BBP codes (click on the image for more details).\n [](autoapi/tasks/workflows/index.html#tasks.workflows.Curate)\n- **Annotate:** create various annotations on morphologies needed by specific BBP codes (click on\n the image for more details).\n [](autoapi/tasks/workflows/index.html#tasks.workflows.Annotate)\n- **Repair:** process morphologies to correct for artifacts of in-vitro reconstruction (click on\n the image for more details).\n [](autoapi/tasks/workflows/index.html#tasks.workflows.Repair)\n\n\nIn a nutshell, the user provides a list of morphologies in a ``.csv`` file, with their names and\npaths and a ``luigi.cfg`` configuration file. Each workflow is run independently and creates an\noutput folder, with one subfolder per task. In each, there will be a ``report.csv`` and a ``data``\nfolder containing the output files of the task if any. In the ``report.csv`` file, columns contain\npaths to these files, additional information, error messages if the task failed on that\nmorphologies, as well as a flag ``is_valid``, used in subsequent tasks to filter valid morphologies.\nAt the end of each workflow, another ``report.csv`` file is created, with the main output columns of\neach tasks, and a ``report.pdf`` containing a human readable summary of the result of the workflow.\n\nUsually, the user should run the ``Curate`` workflow, then the ``Annotate`` workflow and finally the\n``Repair`` workflow.\n\nThe complete documentation can be found here:\n * stable: https://morphology-workflows.readthedocs.io/en/stable/\n * latest: https://morphology-workflows.readthedocs.io/en/latest/\n\n\n## Installation\n\nThis should be installed using pip:\n\n```bash\npip install morphology-workflows\n```\n\n\n## Usage\n\n### Create inputs for the workflows using the **Initialize** workflow\n\nUsually, user should create a new project folder using this workflow. It will create the files\nrequired to run the other workflows. For example, the command:\n\n```bash\nmorphology-workflows Initialize --source-database NeuroMorpho\n```\n\nwill create the inputs to download the morphologies from the NeuroMorpho database and the run the\nother workflows on these morphologies.\n\nIf the user wants to provide arbitrary morphologies to the **Curate** / **Annotate** / **Repair**\nworkflows, the following command can be used:\n\n```bash\nmorphology-workflows Initialize --input-dir <path-to-the-morphologies>\n```\n\nin order to create the dataset required by these workflows.\n\nOnce the **Initialize** workflows is complete, the user can update the outputs according to its\nneeds (see the next sections for details about each workflow).\n\n### Create inputs for the **Fetch** workflow\n\nThis workflow helps fetching morphologies from online databases. This workflow only needs a\nconfiguration file, which depends on the source from which the morphologies are fetched.\n\nThe possible sources are:\n\n* NeuroMorpho\n* MouseLight\n* Allen\n\nFor each of them, the configuration file should be a JSON file containing a list of objects like\nthe following examples:\n\n* NeuroMorpho:\n ```JSON\n [\n {\n \"species\": \"mouse\",\n \"brain_region\": \"neocortex\",\n \"cell_type\": \"interneuron\",\n \"nb_morphologies\": 10\n }\n ]\n ```\n\n* MouseLight:\n ```JSON\n [\n {\n \"brain_region\": \"neocortex\",\n \"nb_morphologies\": 10,\n \"seed\": 0\n }\n ]\n ```\n\n* Allen:\n ```JSON\n [\n {\n \"species\": \"Mus musculus\",\n \"brain_region\": \"VISli\",\n \"cell_type\": \"pyramidal\",\n \"nb_morphologies\": 10,\n \"seed\": 0\n }\n ]\n ```\n\nIn these examples, the ``seed`` attribute is optional and is only used to sample which morphologies\nare fetched among those which pass the filter.\n\nEach `JSON` object in the list will give a set of morphologies to fetch, depending on the given\nfilters.\nNote that all attributes are optional, so it's possible to pass an empty object to fetch all the\nmorphologies from the database, though it is not recommended.\n\n### Create inputs for the **Curate**, **Annotate** and **Repair** workflows\n\nThe **Annotate** and **Repair** workflows should usually be run after the **Curate** workflow since\ntheir inputs should be the outputs of the **Curate** workflow. But it is still possible to run them\non arbitrary inputs (though the morphologies must be valid, as the ones processed by the **Curate**\nworkflow).\n\nThe inputs should consist in:\n\n* a directory containing the input morphologies.\n* a CSV file with the following columns:\n 1. ``morph_path``: the path to the morphology file.\n 2. ``morph_name``: the name of the morphology.\n 3. ``mtype``: the morphology-type of the morphology (this column is optional).\n\n Any other column is kept into the results but not used in the workflows.\n\n Note that the column names should not contain spaces between commas and names.\n* a ``luigi.cfg`` file containing the configuration for all the tasks of the workflow.\n* an optional ``logging.conf`` file containing the logging configuration. If you prefer default logging\n behavior, remove this file and comment line in ``logging_conf_file = logging.conf`` in ``luigi.cfg``.\n\nThe [examples](https://github.com/BlueBrain/morphology-workflows/tree/main/examples) folder contains\nexamples for the ``luigi.cfg`` and ``logging.conf`` files.\n\nThese inputs can be created using the **Fetch** workflow that downloads morphlogies from online\ndatabases and create the dataset in the correct format.\n\n\n### Run the workflows\n\nThese workflows are based on the ``luigi`` library but can be run via the command line interface.\nFor example, you can run the ``Curate`` workflow with the following command:\n\n```bash\nmorphology-workflows Curate\n```\n\n> **NOTE:** This command must be executed from a directory containing a ``luigi.cfg`` file.\n> An example of such file is given in the ``examples`` directory.\n\nBy default, a local scheduler is used but it is also possible to use a Luigi's master scheduler\nusing the `-m / --master-scheduler` trigger:\n\n```bash\nmorphology-workflows -m Curate\n```\n\nOnce the **Curate** workflow has run, the **Annotate** and **Repair** workflows can be run\ndirectly, they will just take the results of the **Curate** workflow as input:\n\n```bash\nmorphology-workflows Annotate\nmorphology-workflows Repair\n```\n\nNote that it is also possible to run the 3 workflows directly because each workflow depends on each other:\n```bash\nmorphology-workflows Repair\n```\n\nMore details can be found in the command line interface section of the documentation or by running\nthe commands:\n\n```bash\nmorphology-workflows --help\nmorphology-workflows <workflow> --help\n```\n\n\n## Results\n\nEach workflow will create several new directories, one for each sub-step. These new directories can\nbe nested into a global result directory for each workflow, using the ``result_path`` parameter.\nThese directories contain intermediate data so it is possible to understand why a morphology\ncould not be validated at a given step.\nThe invalid morphologies should be manually fixed before being processed again by the workflows.\n\nThe main workflows (**Curate**, **Annotate** and **Repair**) will also create a final CSV file\nwhich contains most of the relevant data of the workflow (main sub-step results and final\nmorphology paths). Finally, the **Repair** workflow can generate morphology releases that contain\nthe final morphologies that could be validated and automatically fixed by the workflow. Usually\nthese morphologies are the most relevant ones for later use.\n\nThe morphologies marked as invalid may be fixed manually before rerunning the workflow on them.\n\n\n## Examples\n\nThe [examples](https://github.com/BlueBrain/morphology-workflows/tree/main/examples) folder contains\na simple example that will fetch and process a set of morphologies.\nA ``dataset.csv`` file is provided which is taken as input for the workflows. A ``luigi.cfg`` file\nis also provided to give a default configuration for the workflows.\nThis example can simply be run using the following command:\n\n```bash\n./run_curation.sh\n```\n\nThis script will create a new directory ``out_curated`` which will contain the report and all the\nresults.\n\n## Citation\n\nWhen you use this software, we kindly ask you to cite the following DOI:\n\n[](https://doi.org/10.5281/zenodo.10809196)\n\n## Funding & Acknowledgment\n\nThe development of this software was supported by funding to the Blue Brain Project, a research\ncenter of the \u00c9cole polytechnique f\u00e9d\u00e9rale de Lausanne (EPFL), from the Swiss government\u2019s ETH Board\nof the Swiss Federal Institutes of Technology.\n\nFor license and authors, see `LICENSE.txt` and `AUTHORS.md` respectively.\n\nCopyright \u00a9 2021-2022 Blue Brain Project/EPFL\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "Workflows used for morphology processing.",
"version": "0.11.0",
"project_urls": {
"Homepage": "https://morphology-workflows.readthedocs.io",
"Source": "https://github.com/BlueBrain/morphology-workflows",
"Tracker": "https://github.com/BlueBrain/morphology-workflows/issues"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "eba653bd5fc090de1fe1e6c3cfd5372f57d672d110851954c5187f227f524492",
"md5": "2c142cf0f0d1dd0e22583e1fa7fc7a33",
"sha256": "c5331d52d87de74105ac50eec07a5f43d9b30894986c2e9ea1112b05e8ca1b5b"
},
"downloads": -1,
"filename": "morphology_workflows-0.11.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "2c142cf0f0d1dd0e22583e1fa7fc7a33",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 58690,
"upload_time": "2024-04-26T11:25:18",
"upload_time_iso_8601": "2024-04-26T11:25:18.504661Z",
"url": "https://files.pythonhosted.org/packages/eb/a6/53bd5fc090de1fe1e6c3cfd5372f57d672d110851954c5187f227f524492/morphology_workflows-0.11.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "652386b650c7b11e03b083296e98a4a9decadabf4a8542325ae2d84e52a86820",
"md5": "753cabb2c1b51c409e44b08de7e307db",
"sha256": "4ff093bf58258120ef57d589d135ab754f128b27e5d15ccc7c39a7cb6d613948"
},
"downloads": -1,
"filename": "morphology_workflows-0.11.0.tar.gz",
"has_sig": false,
"md5_digest": "753cabb2c1b51c409e44b08de7e307db",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 1688926,
"upload_time": "2024-04-26T11:25:20",
"upload_time_iso_8601": "2024-04-26T11:25:20.449865Z",
"url": "https://files.pythonhosted.org/packages/65/23/86b650c7b11e03b083296e98a4a9decadabf4a8542325ae2d84e52a86820/morphology_workflows-0.11.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-26 11:25:20",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "BlueBrain",
"github_project": "morphology-workflows",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"tox": true,
"lcname": "morphology-workflows"
}
JSON
