# Blue Brain Atlas Pipeline
## Table of Contents
- [**Introduction**](#introduction)
- [**Installation**](#installation)
- [Git repository](#git-repository)
- [Singularity image on BB5](#singularity-image-on-bb5)
- [Docker image](#docker-image)
- [**Run the pipeline**](#run-the-pipeline)
- [Running the reference Atlas pipeline](#running-the-reference-atlas-pipeline)
- [Customize a pipeline rule](#customize-a-pipeline-rule)
- [Useful Snakemake options](#useful-snakemake-options)
- [**Blue Brain Atlas Pipeline**](#blue-brain-atlas-pipeline-1)
- [Rules and modules](#rules-and-modules)
- [Configuration](#configuration)
- [Additional information](#additional-information)
- [**Appendix**](#appendix)
- [Placement hints data catalog json format](#placement-hints-data-catalog-json-format)
- [**Acknowledgment**](#funding--acknowledgment)
## Introduction
The Blue Brain Atlas Pipeline (BBAP) is a set of processing modules that generate new data such as:
- Annotation volume, brain region hierarchy, direction vectors, orientations and placement hints for selected brain regions,
- Cell density volumes for several cell types,
- CellComposition summary of the brain regions.
To view the command for creating the Atlas as it is pushed to Nexus and consumed by OBP
(the "reference" Atlas), see the below section [Running the Reference Atlas Pipeline](#running-the-reference-atlas-pipeline).
## Installation
The Blue Brain Atlas Pipeline (BBAP) can be installed in three different ways:
- via this [Git repository](#git-repository),
- via a [Singularity image](#singularity-image-on-bb5) (recommended),
- via a [Docker image](#docker-image).
For computation time reason and ease of installation, it is recommended to run the pipeline
on the BB5 cluster via the Singularity image described [hereafter](#singularity-image-on-bb5).
You can log in to the cluster with
`ssh -l <your-Gaspar-username> bbpv1.epfl.ch`
and your Gaspar password, or via the [OpenOnDemand service](https://bbpteam.epfl.ch/project/spaces/display/SDKB/JupyterHub+on+BB5).
Once the installation step is completed, go to [Run the pipeline](#run-the-pipeline) for the instructions to run the pipeline.
### Git repository
The BBAP can be installed directly from the `setup.py` file available in this repository:
1. `git clone https://github.com/BlueBrain/bbp-atlas-pipeline.git`
2. `pip install blue_brain_atlas_pipeline/`
3. `cd blue_brain_atlas_pipeline`
#### Dependencies
Each package run as part of the pipeline is considered a pipeline dependency:
- [token-fetch](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_nexus_token_fetch)
- [nexusforge](https://github.com/BlueBrain/nexus-forge)
- [bba-datafetch](https://bbpteam.epfl.ch/project/spaces/display/BBKG/bba-data-fetch)
- [atlas-direction-vectors](https://github.com/BlueBrain/atlas-direction-vectors)
- [atlas-splitter](https://github.com/BlueBrain/atlas-splitter)
- [atlas-placement-hints](https://github.com/BlueBrain/atlas-placement-hints)
- [atlas-densities](https://github.com/BlueBrain/atlas-densities)
- [parcellationexport](https://bbpteam.epfl.ch/project/spaces/display/BBKG/parcellationexport)
- [bba-data-integrity-check](https://bbpteam.epfl.ch/project/spaces/display/BBKG/bba-data-check)
- [bba-data-push](https://bbpteam.epfl.ch/project/spaces/display/BBKG/bba-data-push)
On BB5, most packages are available also as modules:
```
module load unstable \
snakemake \
py-token-fetch \
py-nexusforge \
py-bba-datafetch \
py-atlas-building-tools \
py-bba-webexporter \
py-data-integrity-check \
py-bba-data-push
```
Or they can be installed following the ‘Installation’ section in their Confluence documentation page.
Now you can go to [Run the pipeline](#run-the-pipeline) for the instructions to run the pipeline.
### Singularity image on BB5
A Singularity image (created from the [Docker image](#docker-image)) is available on BB5 in:
`/gpfs/bbp.cscs.ch/data/project/proj84/atlas_singularity_images/`
The folder contains
- `blue_brain_atlas_pipeline_dev.sif`: development image regularly updated,
- `blue_brain_atlas_pipeline_<tag>.sif`: production image corresponding to a repository [tag](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/tags)
(such as `v0.5.2`).
One can spawn the corresponding container with (example with dev)
1. `module load unstable singularityce`
2. `singularity shell /gpfs/bbp.cscs.ch/data/project/proj84/atlas_singularity_images/blue_brain_atlas_pipeline_dev.sif`
and run the following commands to copy the pipeline files in a path (e.g `$HOME`) where snakemake can write:
3. `cp -r /pipeline/blue_brain_atlas_pipeline $HOME`
4. `cd blue_brain_atlas_pipeline`
Now you can go to [Run the pipeline](#run-the-pipeline) for the instructions to run the pipeline.
### Docker image
A [Docker](https://docs.docker.com/reference) image containing all the pipeline dependencies is available in the Git [registry](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/container_registry/159):
`bbpgitlab.epfl.ch:5050/dke/apps/blue_brain_atlas_pipeline:<tag>`
where `<tag>` = `dev` or a repository tag.
It can be pulled and run with
1. `docker login bbpgitlab.epfl.ch:5050 -u <your-Gaspar-username> -p <your-Gaspar-password>`
2. `docker pull bbpgitlab.epfl.ch:5050/dke/apps/blue_brain_atlas_pipeline:<tag>`
3. `docker run -it bbpgitlab.epfl.ch:5050/dke/apps/blue_brain_atlas_pipeline:<tag> bash`
4. `cd blue_brain_atlas_pipeline`
or converted into an **Apptainer** image with
`apptainer pull --docker-login docker://bbpgitlab.epfl.ch:5050/dke/apps/blue_brain_atlas_pipeline:<tag>`
A benchmark of the resources to provision as required by the different pipeline steps
is available [here](#profiling).
Now you can go to [Run the pipeline](#run-the-pipeline) for the instructions to run the pipeline.
## Run the pipeline
Once the pipeline environment is [installed](#installation), from the root directory execute
`export PYTHONPATH=.:$PYTHONPATH`
and the general command to run the pipeline is available:
```
bbp-atlas --target-rule <target_rule> --snakemake-options '<options>'
```
where
- `<target_rule>` represents the target action to execute.
- `<options>` represents the snakemake options.
A set of most common options is available [here](#useful-snakemake-options).
The option `--cores <number_of_cores>` is mandatory unless the `--dryrun` option is used,
and must be provided as last option.
_Note_: If running multicore on a BB5 node, the step
`transplant_mtypes_densities_from_probability_map` may exceed the available memory and
cause a node failure. Therefore, it is recommended to use a maximum of 70 cores.
A benchmark of the resources required by the different pipeline steps is available [here](#profiling).
### Running the reference Atlas pipeline
The command that is used to run the version of the Blue Brain Atlas Pipeline that is
pushed to Nexus and then used for OBP is as follows:
```
bbp-atlas --target-rule push_atlas_datasets --user-config-path customize_pipeline/user_config.json --snakemake-options '--config NEXUS_REGISTRATION=True --cores all'
```
Note that unless you have special permissions, the `push_...` rules are expected to fail
because only some users have write access to Nexus.
The main entities generated by the pipeline are stored under the paths and names defined
in the config file located at `$HOME/blue_brain_atlas_pipeline/rules_config_dir_templates/push_dataset_config_template.yaml`.
#### AtlasRelease
The following command:
```
bbp-atlas --target-rule push_atlas_release --snakemake-options '--config NEXUS_REGISTRATION=False --cores 1'
```
will generate (locally, without registering in Nexus) the following AtlasRelease (see the prod [AtlasRelease entity](https://bbp.epfl.ch/nexus/v1/resources/bbp/atlas/_/https:%2F%2Fbbp.epfl.ch%2Fneurosciencegraph%2Fdata%2F4906ab85-694f-469d-962f-c0174e901885)) datasets:
- parcellationVolume: annotation volume nrrd file generated at the location defined in the config under `GeneratedDatasetPath.VolumetricFile.annotation_ccfv3_l23split_barrelsplit`
- parcellationOntology: brain region hierarchy generated at the location defined in the config under `HierarchyJson.hierarchy_ccfv3_l23split_barrelsplit`
- directionVector: direction vector volume generated at the location defined in the config under `GeneratedDatasetPath.VolumetricFile.direction_vectors_ccfv3`
- cellOrientationField: orientation field volume generated at the location defined in the config under `GeneratedDatasetPath.VolumetricFile.cell_orientations`
- hemisphereVolume: orientation field volume generated at the location defined in the config under `GeneratedDatasetPath.VolumetricFile.hemispheres`
- placementHintsDataCatalog: json catalog of placement hints volumes generated at the location `WORKING_DIR/ph_catalog_distribution.json`
This catalog has the format described in the [Appendix](#placement-hints-data-catalog-json-format) and groups the placement hints by regions and layers.
The set of actual placement hints nrrd files are generated at the location defined in the config under `GeneratedDatasetPath.VolumetricFile.placement_hints`
#### CellComposition
The following command:
```
bbp-atlas --target-rule push_cellComposition --snakemake-options '--config NEXUS_REGISTRATION=False --cores 1'
```
will generate (locally, without registering in Nexus) the following CellComposition (see the prod [CellComposition entity](https://bbp.epfl.ch/nexus/v1/resources/bbp/atlasdatasetrelease/_/https:%2F%2Fbbp.epfl.ch%2Fneurosciencegraph%2Fdata%2Fcellcompositions%2F54818e46-cf8c-4bd6-9b68-34dffbc8a68c)) datasets:
- cellCompositionVolume: json file generated at the location `WORKING_DIR/cellCompositionVolume_payload.json`,
containing the ids of selected ME-type density nrrd files registered in Nexus,
grouped by M-type and E-type.
The whole set of ME-type densities is generated at the location defined in the config under `GeneratedDatasetPath.VolumetricFile.mtypes_densities_probability_map_transplant`
- cellCompositionSummary: json file generated at the location `WORKING_DIR/cellCompositionSummary_payload.json`,
containing the values of the ME-type densities in the cellCompositionVolume,
grouped by regions
***
**NOTE for versions < v1.0.0**
The selected ME-type densities that enter the [CellCompositionVolume](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/blob/develop/cellCompVolume_payload.py?ref_type=heads#L16)
are those having a `layer` in their Nexus Resource property `brainLocation`, plus the two
`Generic{Inhibitory,Excitatory}NeuronMType`-`Generic{Inhibitory,Excitatory}NeuronEType`.
For an ME-type density Resource to be registered by the pipeline with such a `layer` property,
the [Nexus Ontology Class of its M-type](https://bbpteam.epfl.ch/project/spaces/display/BBKG/METypes+Registration#METypesRegistration-Checkexistingcelltypes)
must have the `hasLayerLocationPhenotype` attribute:
```
res = forge.resolve("<M-type label>", scope="ontology", target="CellType", strategy="EXACT_MATCH")
res_layer = res.hasLayerLocationPhenotype
```
***
#### Miscellanea
The rules in the previous commands trigger many intermediate dependent rules as described [here](#blue-brain-atlas-pipeline-1).
The pipeline consumes a configuration file described [here](#configuration), by default
named `config.yaml` and located in the directory from which the pipeline is run.
A specific config file can be provided via the `--configfile` option:
```
bbp-atlas --target-rule <target_rule> --snakemake-options '--configfile <config_file_path>'
```
**NOTE**
To run the pipeline skipping the generation of datasets already available (in
case a previous run failed at an intermediate step for instance), the [option](#useful-snakemake-options)
`--rerun-trigger mtime` can be used as in the following command:
```
bbp-atlas --target-rule <target_rule> --snakemake-options '--rerun-trigger mtime --cores 1'
```
Such an option allows to skip the execution of the pipeline steps whose output
files exist and have a modification time (`mtime`) more recent than any of their
input files.
***
### Customize a pipeline rule
It is possible to customize a pipeline rule that generates a (set of) volumetric file
(`.nrrd`) in order to change the values of a specific region of the volume (and leave
the rest of the volume unchanged).
The customization happens via the configuration file [`customize_pipeline/user_config.json`](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/blob/develop/customize_pipeline/user_config.json)
with the following structure:
- `rule`: name of the rule to customize from the default pipeline;
- `brainRegion`: ID of the brain region to customize;
- `CLI`:
- `command`: CLI to execute in order to produce the volumetric file with the desired values for the brain region of interest;
- `args`: CLI arguments that can reference variables between curly brackets (see below);
- `output_dir`: path of the folder where the volumetric file(s) is generated by the CLI;
- `container`: URL of the Docker image to use in order to spawn a container where the CLI will be executed. This parameter
is optional: if not provided, the CLI will be executed in the same environment of the default pipeline (in such a case,
the user must ensure that the provided CLI is defined therein).
_Note_: the Snakemake option `--use-singularity` must be provided for the configuration parameter `container` to be considered.
The CLI `args` can reference one or more variables which points to files generated by
pipeline rules executed before the rule to customize. The list of variables is available
in [customize_pipeline/available_vars.yaml](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/blob/develop/customize_pipeline/available_vars.yaml).
#### Filename convention
The user must ensure that the files generated by the provided CLI have the same names as
the files generated by the rule to customize.
For example, the rule `direction_vectors_placeholder_ccfv3`
in the sample configuration generates one output file `direction_vectors_ccfv3.nrrd`.
The `placement_hints` rule generates seven volumetric files: `[PH]y.nrrd` and
`[PH]layer_n.nrrd` where n = 1, ..., 6. The mapping between each nrrd file and the layer
it refers to for each region is available in [this dictionary](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/blob/develop/metadata/PH_layers_regions_map.json),
which the user needs to extend with the `"region acronym": {"layer ID", "layer label"}`
of its customized region. A layer is considered associated to a region if the
corresponding layer ID appears in the [regions-to-layers mapping](#brain-region-layers) for that region
or for at least one of that region's offspring.
#### Customized pipeline
Once the [configuration file](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/blob/develop/customize_pipeline/user_config.json)
is ready, the customized pipeline can be run with the following command:
```
bbp-atlas --target-rule <target_rule> --user-config-path customize_pipeline/user_config.json --snakemake-options '<options>'
```
When a rule is customized as described above, the pipeline will run
1. the default rule to generate the default output file(s),
2. the CLI provided in the configuration file to produce the corresponding
region-specific output file(s),
3. a merge step to override the specific region in the default file(s) (step 1) with the
values of that region from the region-specific file(s) (step 2).
#### Integration
In case a user wants to request the integration of the customized version of a dataset:
1. Open a Merge Request (MR) in this repository including the updated `user_config.json`
and any additional input [metadata](#metadata) required.
2. The MR is then reviewed and, if approved, a new Atlas pipeline dev image is produced accordingly.
3. The new pipeline is run and the new datasets are registered in Nexus staging for wider tests.
4. When the new version of the datasets is validated, a new tag of the Atlas pipeline is
cut and the corresponding image is used to register the datasets in Nexus prod.
##### Metadata
Some pipeline steps require metadata as input, which are fetched from Nexus.
Currently, the files available in the [`metadata`](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/tree/develop/metadata)
directory are automatically synchronized with their Nexus versions.
If you want to update/add one metadata file, make sure to update/add also the
corresponding documentation file in the [`metadata/docs`](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/tree/develop/metadata/docs)
directory, keeping the current naming convention (`probability_map_*{.csv,.txt}`).
#### Direction-vectors and placement-hints
If you want to add creation of your region's direction-vectors, placement-hints, or other NRRD files that need to be merged with files containing data from other regions, you should add them using the instructions in [Customize a pipeline rule](#customize-a-pipeline-rule) into `customize_pipeline/user_config.json` instead of the `snakefile` directly.
### Useful Snakemake options
Snakemake being a command-line tool, it comes with a multitude of optional arguments to
execute, debug, and visualize workflows. Here is a selection of the most used:
- `--cores <number_of_cores>`, `-c <number_of_cores>` → Specify the number of cores
snakemake can use.
- `--dry-run`, `-n` → Perform a dry run (execute nothing but print the list of rules
that would be executed).
- `--rerun-trigger mtime` → Use only the modification time (`mtime`) of the existing
output files to determine which rules to execute.
- `--forcerun <some_rule>` → Force a given rule to be re-executed (overwrite the output
if it already exists).
- `--list`, `-l` → Print a list of all the available rules from the snakefile.
Every Snakemake command line argument is listed and described in the [Snakemake](https://snakemake.readthedocs.io/en/stable/) official documentation page.
## Blue Brain Atlas Pipeline
Its workflow consists of the following steps:
1. Fetch the required datasets from Nexus. These input data consist of the [original AIBS ccfv3 brain parcellation](https://bbp.epfl.ch/nexus/web/bbp/atlas/resources/https%3A%2F%2Fbbp.epfl.ch%2Fneurosciencegraph%2Fdata%2F025eef5f-2a9a-4119-b53f-338452c72f2a),
the [AIBS Mouse CCF Atlas regions hierarchy file](https://bbp.epfl.ch/nexus/web/neurosciencegraph/datamodels/resources/http%3A%2F%2Fbbp.epfl.ch%2Fneurosciencegraph%2Fontologies%2Fmba) and a series of Nissl and ISH volumes
as described in the documentation page [Allen Mouse CCF Compatible Data](https://bbpteam.epfl.ch/project/spaces/display/BBKG/Allen+Mouse+CCF+Compatible+Data).
2. The fetched datasets are then fed to the [Snakemake](https://snakemake.readthedocs.io/en/stable/) rules, and under the hood consumed by atlas modules to generate products.
3. Each product can (optionally) be pushed into Nexus with a set of metadata automatically filled up and be visualised in
the [Blue Brain Atlas](https://bbpteam.epfl.ch/documentation/#:~:text=Visualize-,Blue%20Brain%20Atlas,-Morphology%20visualization).
This workflow is illustrated on the following diagram containing the directed acyclic graph (DAG)
of the Snakemake rules of the BBAP:
A more detailed DAG listing the input and output files for each step is available [here](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/blob/develop/doc/source/figures/dag_push_atlas_fg.svg).
### Rules and modules
In this document, a “module” is a CLI encapsulated inside one of the components of the pipeline.
Such component is called a “rule”. This terminology comes from SnakeMake, where a “rule”
can leverage one or more modules and where a module can be used by one of more rules,
usually using a different set of arguments.
You can find more information on rules in the [SnakeMake documentation](https://snakemake.readthedocs.io/en/stable/).
To only visualize the command that a given rule will execute without running it, one can
use the `--dryrun` option as in the following command:
```
bbp-atlas --target-rule <target_rule> --snakemake-options '--dryrun'
```
The documentation of each command is available in the corresponding pipeline [dependency](#dependencies).
#### Profiling
A detailed profiling of the most resource-intensive rules (sorted by execution order)
is available in the following table, corresponding to a single core of an
Intel Xeon Gold 6140 CPU (BB5 node).
Some rules can exploit multiple cores, in which case a second entry for such rules
appears in the table along with the number of cores ("--cores n") used for the
profiling.
The **total multicore wall clock time** required by the two final rules `push_atlas_release`
(which depends on direction vectors, orientation field, placement hints) and
`push_cellComposition` (which depends on all the density generation rules) is
respectively **1 h** (with an RSS peak of 10 GB) and **4 h** (with an RSS peak of 8 GB).
| Rule name | wall clock time [s] | wall clock time [h:m:s] | max [RSS](https://en.wikipedia.org/wiki/Resident_set_size) [MB] | max [VMS](https://en.wikipedia.org/wiki/Virtual_memory) [MB] | max [USS](https://en.wikipedia.org/wiki/Unique_set_size) [MB] | max [PSS](https://en.wikipedia.org/wiki/Proportional_set_size) [MB] | I/O in [B] | I/O out [B] | average CPU load [%] | CPU time [s] |
|-------------------------------------------------------------|--------------------:|------------------------:|----------------------------------------------------------------:|-------------------------------------------------------------:|--------------------------------------------------------------:|--------------------------------------------------------------------:|-----------:|------------:|---------------------:|-------------:|
| direction_vectors_default_ccfv3 | 352.1527 | 0:05:52 | 3345.09 | 4503.41 | 3309.88 | 3321.13 | 0.07 | 0.00 | 98.53 | 347.43 |
| direction_vectors_isocortex_ccfv3 | 376.2279 | 0:06:16 | 5438.86 | 6049.21 | 5401.71 | 5412.96 | 0.00 | 0.00 | 92.23 | 347.29 |
| orientation_field | 248.3647 | 0:04:08 | 8423.66 | 9010.55 | 8388.29 | 8399.54 | 0.00 | 0.00 | 91.73 | 228.17 |
| split_isocortex_layer_23_ccfv3 | 147.5079 | 0:02:27 | 1945.17 | 2977.68 | 1866.67 | 1877.88 | 0.83 | 0.00 | 92.70 | 137.05 |
| create_leaves_only_hierarchy_annotation_ccfv3 | 46.3272 | 0:00:46 | 5897.06 | 6542.66 | 5818.63 | 5830.39 | 0.04 | 0.00 | 36.30 | 17.06 |
| split_barrel_ccfv3_l23split | 141.6368 | 0:02:21 | 715.49 | 2420.59 | 679.82 | 691.35 | 0.06 | 0.00 | 96.49 | 137.23 |
| validate_annotation_v3 | 4.8913 | 0:00:04 | 908.17 | 1749.66 | 865.54 | 876.86 | 0.03 | 0.00 | 71.09 | 3.81 |
| placement_hints | 924.7421 | 0:15:24 | 6600.64 | 7225.64 | 6524.41 | 6537.06 | 7.41 | 0.00 | 99.36 | 919.29 |
| create_hemispheres_ccfv3 | 6.3128 | 0:00:06 | 547.75 | 1233.48 | 514.01 | 524.86 | 0.00 | 0.00 | 68.63 | 4.67 |
| export_brain_region | 27482.2978 | 7:38:02 | 2649.79 | 3214.40 | 3497.71 | 3509.16 | 0.40 | 0.00 | 99.68 | 27395.00 |
| export_brain_region (`--cores 70`) | 1162.2325 | 0:19:22 | 143397.54 | 193202.21 | 115383.76 | 115749.06 | 0.04 | 0.00 | 4307.04 | 50067.62 |
| combine_v2_annotations | 17.5364 | 0:00:17 | 1030.33 | 1581.99 | 1002.29 | 1015.24 | 1.09 | 0.00 | 85.00 | 15.44 |
| direction_vectors_isocortex_ccfv2 | 299.9846 | 0:04:59 | 5483.91 | 6050.02 | 5454.11 | 5467.01 | 0.04 | 0.00 | 95.15 | 285.99 |
| split_isocortex_layer_23_ccfv2 | 153.1193 | 0:02:33 | 1999.78 | 2979.68 | 1937.86 | 1950.75 | 0.00 | 0.00 | 88.49 | 135.96 |
| create_leaves_only_hierarchy_annotation_ccfv2 | 31.2190 | 0:00:31 | 6027.33 | 6836.57 | 6276.27 | 6289.61 | 0.04 | 0.00 | 51.64 | 16.60 |
| split_barrel_ccfv2_l23split | 112.8031 | 0:01:52 | 737.26 | 2421.75 | 706.36 | 719.25 | 0.06 | 0.00 | 93.59 | 106.04 |
| validate_annotation_v2 | 2.3923 | 0:00:02 | 1116.59 | 1736.83 | 1076.46 | 1089.53 | 0.03 | 0.00 | 66.48 | 2.12 |
| cell_density_correctednissl | 55.1323 | 0:00:55 | 2867.18 | 3418.92 | 2839.32 | 2852.28 | 0.00 | 0.00 | 82.24 | 45.88 |
| validate_cell_density | 4.8925 | 0:00:04 | 1252.33 | 2355.65 | 1767.52 | 1780.59 | 0.00 | 0.00 | 76.66 | 4.27 |
| combine_markers | 555.3407 | 0:09:15 | 5147.12 | 5697.71 | 5119.06 | 5132.01 | 0.00 | 0.00 | 94.58 | 525.61 |
| glia_cell_densities_correctednissl | 221.8078 | 0:03:41 | 7373.72 | 8061.20 | 7298.36 | 7311.31 | 0.00 | 0.00 | 87.87 | 195.19 |
| validate_neuron_glia_cell_densities | 17.4026 | 0:00:17 | 3845.63 | 4698.86 | 4109.74 | 4122.81 | 0.00 | 0.00 | 89.61 | 16.00 |
| average_densities_correctednissl | 2886.3829 | 0:48:06 | 3889.09 | 5342.21 | 3841.21 | 3854.18 | 0.00 | 0.00 | 99.22 | 2864.22 |
| fit_average_densities_correctednissl | 2185.0088 | 0:36:25 | 5605.01 | 7106.13 | 5410.92 | 5423.89 | 0.00 | 0.00 | 99.52 | 2174.67 |
| inhibitory_neuron_densities_linprog_correctednissl | 2859.1158 | 0:47:39 | 4799.59 | 18207.76 | 4771.89 | 4784.86 | 0.00 | 0.00 | 99.16 | 2834.94 |
| compute_lamp5_density | 53.7274 | 0:00:53 | 3652.39 | 4176.77 | 4964.55 | 4977.10 | 0.00 | 0.00 | 83.84 | 45.36 |
| create_mtypes_densities_from_probability_map | 31310.7357 | 8:41:50 | 31178.52 | 32289.28 | 31150.59 | 31163.56 | 0.00 | 0.00 | 99.69 | 31209.90 |
| create_mtypes_densities_from_probability_map (`--cores 70`) | 6276.3961 | 1:44:36 | 394077.36 | 2285360.80 | 63690.27 | 82778.46 | 1.04 | 0.00 | 5992.08 | 376431.41 |
| excitatory_split | 222.6994 | 0:03:42 | 3437.98 | 3989.99 | 3410.12 | 3423.07 | 0.00 | 0.00 | 87.40 | 195.18 |
| create_cellCompositionVolume_payload | 414.9835 | 0:06:54 | 0 | 0 | 0 | 0 | 0.00 | 0.00 | 0.25 | 0 |
| create_cellCompositionSummary_payload | 1205.1099 | 0:20:05 | 2414.04 | 5000.99 | 2230.68 | 2308.75 | 3.73 | 0.00 | 84.15 | 1030.68 |
| create_cellCompositionSummary_payload (`--cores 70`) | 206.0507 | 0:03:26 | 21817.64 | 67112.53 | 12911.70 | 13023.87 | 0.00 | 0.00 | 635.94 | 1314.64 |
#### Fetch rules
The rules starting with "fetch_" are used to download a given file from Nexus.
The IDs of the corresponding Nexus Resource (containing a description of the file to
fetch) are listed in the [nexus_ids.json](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/blob/develop/nexus_ids.json) (the explicit link between a fetch
rule and the corresponding Resource ID lays in the `nexus_id` parameter of the rule
definition in the [snakefile](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/blob/develop/snakefile)).
_Note_: the rule "fetch_genes_correctednissl" is not linked to a specific Resource, it's
used just to trigger the execution of a set of single "fetch_gene_" rules needed by the
"fit-average-densities" step.
In order to run the pipeline with a different version of a fetched file, one can just
execute the corresponding fetch rule and subsequently replace the downloaded file with
the desired version, by keeping the same name of the originally fetched file.
The `--rerun-trigger mtime` [option](#useful-snakemake-options) may be useful here.
### Configuration
The configuration of the pipeline is provided in the `config.yaml` file. The most important
variables that a user can customize are:
- `WORKING_DIR`: the output directory of the pipeline files,
- `NEXUS_IDS_FILE`: the json file containing the Ids of the Nexus Resources to fetch,
- `FORGE_CONFIG`: the configuration file (yaml) to instantiate nexus-forge,
- `NEW_ATLAS`: boolean flag to trigger the creation of a brand-new atlas release,
- `RESOLUTION`: resolution (in μm) of the input volumetric files to be consumed by the pipeline (default to 25),
- `NEXUS_REGISTRATION`: boolean flag to trigger data registration in Nexus
- `RESOURCE_TAG`: string to use as tag of the data registered in Nexus
- `IS_PROD_ENV`: boolean flag to indicate whether the target Nexus environment is production or not (staging),
- `NEXUS_DESTINATION_ORG`/`NEXUS_DESTINATION_PROJ`: Nexus organization/project where register the pipeline products,
- `DISPLAY_HELP`: boolean flag to display every rule of the snakefile with its descriptions.
It is possible to override the config variables at runtime using the snakemake argument `--config`:
`--config <VAR_NAME>=<VALUE>`
### Additional information
The release notes are available [here](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/blob/develop/doc/release_notes.md).
More information about The Blue Brain Atlas Pipeline (BBAP) are available in its [confluence documentation](https://bbpteam.epfl.ch/project/spaces/display/BBKG/Atlas+Pipeline).
This space contains several documentation pages describing:
The Allen Mouse CCF Compatible Data : [https://bbpteam.epfl.ch/project/spaces/display/BBKG/Allen+Mouse+CCF+Compatible+Data](https://bbpteam.epfl.ch/project/spaces/display/BBKG/Allen+Mouse+CCF+Compatible+Data)
The Atlas Modules : [https://bbpteam.epfl.ch/project/spaces/display/BBKG/Atlas+Modules](https://bbpteam.epfl.ch/project/spaces/display/BBKG/Atlas+Modules)
## Appendix
### Brain region layers
Some brain areas have a subdivision in layers.
The mapping adopted in the BBP between a brain region and the layers it belongs to is
provided in [this dictionary](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/blob/develop/metadata/regions_layers_map.json),
where the keys are brain region IDs and the layers are identified with
[Uberon](https://www.ebi.ac.uk/ols4/ontologies/uberon) classes.
One layer - "Neocortex layer 6a" - is not present in the Uberon ontology and is defined
as follows:
```
<https://bbp.epfl.ch/ontologies/core/bmo/neocortex_layer_6a> rdf:type owl:Class ;
rdfs:subClassOf <http://purl.obolibrary.org/obo/UBERON_0002301> ;
rdfs:label "L6a"^^xsd:string ;
<http://www.w3.org/2004/02/skos/core#definition> "Neocortex layer 6a."^^xsd:string ;
<http://www.w3.org/2004/02/skos/core#altLabel> "layer 6a"^^xsd:string ;
<http://www.w3.org/2004/02/skos/core#altLabel> "neocortex layer 6a"^^xsd:string ;
<http://www.w3.org/2004/02/skos/core#prefLabel> "L6a"^^xsd:string ;
<http://www.w3.org/2004/02/skos/core#notation> "L6a"^^xsd:string .
```
### Placement hints data catalog json format
```json
{
"placementHints": [
{
"@id": "https://bbp.epfl.ch/data/bbp/atlas/f1049c1b-f1af-4d33-acd9-099f05c56bbf",
"_rev": 13,
"distribution": {
"atLocation": {
"location": "file:///gpfs/bbp.cscs.ch/data/project/proj39/nexus/bbp/atlas/9/b/1/3/3/7/7/9/%5BPH%5Dlayer_1.nrrd"
},
"name": "[PH]layer_1.nrrd"
},
"regions": {
"Isocortex": {
"@id": "http://api.brain-map.org/api/v2/data/Structure/315",
"hasLeafRegionPart": [
"PL1",
"..."
],
"layer": {
"@id": "http://purl.obolibrary.org/obo/UBERON_0005390",
"label": "L1"
}
},
"Hippocampal formation": {
"@id": "http://api.brain-map.org/api/v2/data/Structure/1089",
"hasLeafRegionPart": [
"CA1sp",
"..."
],
"layer": {
"@id": "http://purl.obolibrary.org/obo/UBERON_0002313",
"label": "SP"
}
},
"...": {}
}
},
{
"@id": "https://bbp.epfl.ch/data/bbp/atlas/74ba22b1-39ee-486d-ab3c-cb960d006a5d",
"_rev": 13,
"distribution": {
"atLocation": {
"location": "file:///gpfs/bbp.cscs.ch/data/project/proj39/nexus/bbp/atlas/a/9/3/0/d/e/a/8/%5BPH%5Dlayer_2.nrrd"
},
"name": "[PH]layer_2.nrrd"
},
"regions": {
"Isocortex": {
"@id": "http://api.brain-map.org/api/v2/data/Structure/315",
"hasLeafRegionPart": [
"AUDp2",
"..."
],
"layer": {
"@id": "http://purl.obolibrary.org/obo/UBERON_0005391",
"label": "L2"
}
},
"Hippocampal formation": {
"@id": "http://api.brain-map.org/api/v2/data/Structure/1089",
"hasLeafRegionPart": [
"CA1so",
"..."
],
"layer": {
"@id": "http://purl.obolibrary.org/obo/UBERON_0005371",
"label": "SO"
}
},
"...": {}
}
},
"..."
],
"voxelDistanceToRegionBottom": {
"@id": "https://bbp.epfl.ch/data/bbp/atlas/59a2bca3-d8b6-43b1-870e-a0c19a020175",
"_rev": 13,
"distribution": {
"atLocation": {
"location": "file:///gpfs/bbp.cscs.ch/data/project/proj39/nexus/bbp/atlas/3/9/e/b/6/d/8/b/%5BPH%5Dy.nrrd"
},
"name": "[PH]y.nrrd"
}
}
}
```
## Funding & Acknowledgment
The development of this software was supported by funding to the Blue Brain Project, a
research center of the École polytechnique fédérale de Lausanne (EPFL), from the Swiss
government’s ETH Board of the Swiss Federal Institutes of Technology.
Copyright © 2020-2024 Blue Brain Project/EPFL
Raw data
{
"_id": null,
"home_page": "https://github.com/BlueBrain/bbp-atlas-pipeline",
"name": "bba-pipeline",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": null,
"author": "Blue Brain Project, EPFL",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/37/fc/f5a2e20b1f07082234f2b482dfa54bdcf17931ec7bd757bc7ac330008baa/bba_pipeline-1.3.1.tar.gz",
"platform": null,
"description": "# Blue Brain Atlas Pipeline\n\n## Table of Contents\n- [**Introduction**](#introduction)\n- [**Installation**](#installation)\n - [Git repository](#git-repository)\n - [Singularity image on BB5](#singularity-image-on-bb5)\n - [Docker image](#docker-image)\n- [**Run the pipeline**](#run-the-pipeline)\n - [Running the reference Atlas pipeline](#running-the-reference-atlas-pipeline)\n - [Customize a pipeline rule](#customize-a-pipeline-rule)\n - [Useful Snakemake options](#useful-snakemake-options)\n- [**Blue Brain Atlas Pipeline**](#blue-brain-atlas-pipeline-1)\n - [Rules and modules](#rules-and-modules)\n - [Configuration](#configuration)\n - [Additional information](#additional-information)\n- [**Appendix**](#appendix)\n - [Placement hints data catalog json format](#placement-hints-data-catalog-json-format)\n- [**Acknowledgment**](#funding--acknowledgment)\n\n\n## Introduction\n\nThe Blue Brain Atlas Pipeline (BBAP) is a set of processing modules that generate new data such as:\n\n- Annotation volume, brain region hierarchy, direction vectors, orientations and placement hints for selected brain regions,\n- Cell density volumes for several cell types,\n- CellComposition summary of the brain regions.\n\nTo view the command for creating the Atlas as it is pushed to Nexus and consumed by OBP \n(the \"reference\" Atlas), see the below section [Running the Reference Atlas Pipeline](#running-the-reference-atlas-pipeline).\n\n\n## Installation\n\nThe Blue Brain Atlas Pipeline (BBAP) can be installed in three different ways:\n- via this [Git repository](#git-repository),\n- via a [Singularity image](#singularity-image-on-bb5) (recommended),\n- via a [Docker image](#docker-image).\n\nFor computation time reason and ease of installation, it is recommended to run the pipeline \non the BB5 cluster via the Singularity image described [hereafter](#singularity-image-on-bb5).\nYou can log in to the cluster with \n`ssh -l <your-Gaspar-username> bbpv1.epfl.ch` \nand your Gaspar password, or via the [OpenOnDemand service](https://bbpteam.epfl.ch/project/spaces/display/SDKB/JupyterHub+on+BB5).\n\nOnce the installation step is completed, go to [Run the pipeline](#run-the-pipeline) for the instructions to run the pipeline.\n\n### Git repository\nThe BBAP can be installed directly from the `setup.py` file available in this repository:\n\n1. `git clone https://github.com/BlueBrain/bbp-atlas-pipeline.git`\n2. `pip install blue_brain_atlas_pipeline/`\n3. `cd blue_brain_atlas_pipeline`\n\n#### Dependencies\nEach package run as part of the pipeline is considered a pipeline dependency:\n\n- [token-fetch](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_nexus_token_fetch)\n- [nexusforge](https://github.com/BlueBrain/nexus-forge)\n- [bba-datafetch](https://bbpteam.epfl.ch/project/spaces/display/BBKG/bba-data-fetch)\n- [atlas-direction-vectors](https://github.com/BlueBrain/atlas-direction-vectors)\n- [atlas-splitter](https://github.com/BlueBrain/atlas-splitter)\n- [atlas-placement-hints](https://github.com/BlueBrain/atlas-placement-hints)\n- [atlas-densities](https://github.com/BlueBrain/atlas-densities)\n- [parcellationexport](https://bbpteam.epfl.ch/project/spaces/display/BBKG/parcellationexport)\n- [bba-data-integrity-check](https://bbpteam.epfl.ch/project/spaces/display/BBKG/bba-data-check)\n- [bba-data-push](https://bbpteam.epfl.ch/project/spaces/display/BBKG/bba-data-push)\n\nOn BB5, most packages are available also as modules:\n```\nmodule load unstable \\\nsnakemake \\\npy-token-fetch \\\npy-nexusforge \\\npy-bba-datafetch \\\npy-atlas-building-tools \\\npy-bba-webexporter \\\npy-data-integrity-check \\\npy-bba-data-push\n```\nOr they can be installed following the \u2018Installation\u2019 section in their Confluence documentation page.\n\nNow you can go to [Run the pipeline](#run-the-pipeline) for the instructions to run the pipeline.\n\n\n### Singularity image on BB5\nA Singularity image (created from the [Docker image](#docker-image)) is available on BB5 in: \n`/gpfs/bbp.cscs.ch/data/project/proj84/atlas_singularity_images/`\n\nThe folder contains\n- `blue_brain_atlas_pipeline_dev.sif`: development image regularly updated,\n- `blue_brain_atlas_pipeline_<tag>.sif`: production image corresponding to a repository [tag](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/tags)\n (such as `v0.5.2`).\n\nOne can spawn the corresponding container with (example with dev) \n1. `module load unstable singularityce`\n2. `singularity shell /gpfs/bbp.cscs.ch/data/project/proj84/atlas_singularity_images/blue_brain_atlas_pipeline_dev.sif` \nand run the following commands to copy the pipeline files in a path (e.g `$HOME`) where snakemake can write: \n3. `cp -r /pipeline/blue_brain_atlas_pipeline $HOME` \n4. `cd blue_brain_atlas_pipeline` \n\nNow you can go to [Run the pipeline](#run-the-pipeline) for the instructions to run the pipeline.\n\n\n### Docker image\nA [Docker](https://docs.docker.com/reference) image containing all the pipeline dependencies is available in the Git [registry](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/container_registry/159): \n`bbpgitlab.epfl.ch:5050/dke/apps/blue_brain_atlas_pipeline:<tag>` \nwhere `<tag>` = `dev` or a repository tag. \n\nIt can be pulled and run with \n1. `docker login bbpgitlab.epfl.ch:5050 -u <your-Gaspar-username> -p <your-Gaspar-password>`\n2. `docker pull bbpgitlab.epfl.ch:5050/dke/apps/blue_brain_atlas_pipeline:<tag>`\n3. `docker run -it bbpgitlab.epfl.ch:5050/dke/apps/blue_brain_atlas_pipeline:<tag> bash`\n4. `cd blue_brain_atlas_pipeline`\n\nor converted into an **Apptainer** image with \n`apptainer pull --docker-login docker://bbpgitlab.epfl.ch:5050/dke/apps/blue_brain_atlas_pipeline:<tag>`\n\nA benchmark of the resources to provision as required by the different pipeline steps \nis available [here](#profiling).\n\nNow you can go to [Run the pipeline](#run-the-pipeline) for the instructions to run the pipeline.\n\n\n## Run the pipeline\n\nOnce the pipeline environment is [installed](#installation), from the root directory execute \n`export PYTHONPATH=.:$PYTHONPATH` \nand the general command to run the pipeline is available:\n```\nbbp-atlas --target-rule <target_rule> --snakemake-options '<options>'\n```\nwhere\n- `<target_rule>` represents the target action to execute.\n- `<options>` represents the snakemake options. \nA set of most common options is available [here](#useful-snakemake-options). \nThe option `--cores <number_of_cores>` is mandatory unless the `--dryrun` option is used,\nand must be provided as last option.\n\n_Note_: If running multicore on a BB5 node, the step \n`transplant_mtypes_densities_from_probability_map` may exceed the available memory and \ncause a node failure. Therefore, it is recommended to use a maximum of 70 cores.\n\nA benchmark of the resources required by the different pipeline steps is available [here](#profiling).\n\n\n### Running the reference Atlas pipeline\n\nThe command that is used to run the version of the Blue Brain Atlas Pipeline that is \npushed to Nexus and then used for OBP is as follows:\n```\nbbp-atlas --target-rule push_atlas_datasets --user-config-path customize_pipeline/user_config.json --snakemake-options '--config NEXUS_REGISTRATION=True --cores all'\n```\n\nNote that unless you have special permissions, the `push_...` rules are expected to fail\nbecause only some users have write access to Nexus.\n\nThe main entities generated by the pipeline are stored under the paths and names defined \nin the config file located at `$HOME/blue_brain_atlas_pipeline/rules_config_dir_templates/push_dataset_config_template.yaml`. \n\n#### AtlasRelease\n\nThe following command:\n```\n bbp-atlas --target-rule push_atlas_release --snakemake-options '--config NEXUS_REGISTRATION=False --cores 1'\n```\nwill generate (locally, without registering in Nexus) the following AtlasRelease (see the prod [AtlasRelease entity](https://bbp.epfl.ch/nexus/v1/resources/bbp/atlas/_/https:%2F%2Fbbp.epfl.ch%2Fneurosciencegraph%2Fdata%2F4906ab85-694f-469d-962f-c0174e901885)) datasets:\n- parcellationVolume: annotation volume nrrd file generated at the location defined in the config under `GeneratedDatasetPath.VolumetricFile.annotation_ccfv3_l23split_barrelsplit`\n- parcellationOntology: brain region hierarchy generated at the location defined in the config under `HierarchyJson.hierarchy_ccfv3_l23split_barrelsplit`\n- directionVector: direction vector volume generated at the location defined in the config under `GeneratedDatasetPath.VolumetricFile.direction_vectors_ccfv3`\n- cellOrientationField: orientation field volume generated at the location defined in the config under `GeneratedDatasetPath.VolumetricFile.cell_orientations`\n- hemisphereVolume: orientation field volume generated at the location defined in the config under `GeneratedDatasetPath.VolumetricFile.hemispheres`\n- placementHintsDataCatalog: json catalog of placement hints volumes generated at the location `WORKING_DIR/ph_catalog_distribution.json`\n This catalog has the format described in the [Appendix](#placement-hints-data-catalog-json-format) and groups the placement hints by regions and layers.\n The set of actual placement hints nrrd files are generated at the location defined in the config under `GeneratedDatasetPath.VolumetricFile.placement_hints`\n\n#### CellComposition\n\nThe following command:\n```\n bbp-atlas --target-rule push_cellComposition --snakemake-options '--config NEXUS_REGISTRATION=False --cores 1'\n```\nwill generate (locally, without registering in Nexus) the following CellComposition (see the prod [CellComposition entity](https://bbp.epfl.ch/nexus/v1/resources/bbp/atlasdatasetrelease/_/https:%2F%2Fbbp.epfl.ch%2Fneurosciencegraph%2Fdata%2Fcellcompositions%2F54818e46-cf8c-4bd6-9b68-34dffbc8a68c)) datasets:\n- cellCompositionVolume: json file generated at the location `WORKING_DIR/cellCompositionVolume_payload.json`,\n containing the ids of selected ME-type density nrrd files registered in Nexus,\n grouped by M-type and E-type. \n The whole set of ME-type densities is generated at the location defined in the config under `GeneratedDatasetPath.VolumetricFile.mtypes_densities_probability_map_transplant`\n- cellCompositionSummary: json file generated at the location `WORKING_DIR/cellCompositionSummary_payload.json`,\n containing the values of the ME-type densities in the cellCompositionVolume,\n grouped by regions\n\n***\n**NOTE for versions < v1.0.0** \nThe selected ME-type densities that enter the [CellCompositionVolume](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/blob/develop/cellCompVolume_payload.py?ref_type=heads#L16)\nare those having a `layer` in their Nexus Resource property `brainLocation`, plus the two \n`Generic{Inhibitory,Excitatory}NeuronMType`-`Generic{Inhibitory,Excitatory}NeuronEType`.\n\nFor an ME-type density Resource to be registered by the pipeline with such a `layer` property,\nthe [Nexus Ontology Class of its M-type](https://bbpteam.epfl.ch/project/spaces/display/BBKG/METypes+Registration#METypesRegistration-Checkexistingcelltypes)\nmust have the `hasLayerLocationPhenotype` attribute:\n```\nres = forge.resolve(\"<M-type label>\", scope=\"ontology\", target=\"CellType\", strategy=\"EXACT_MATCH\")\nres_layer = res.hasLayerLocationPhenotype\n```\n***\n\n#### Miscellanea\n\nThe rules in the previous commands trigger many intermediate dependent rules as described [here](#blue-brain-atlas-pipeline-1).\n\nThe pipeline consumes a configuration file described [here](#configuration), by default \nnamed `config.yaml` and located in the directory from which the pipeline is run. \nA specific config file can be provided via the `--configfile` option:\n```\nbbp-atlas --target-rule <target_rule> --snakemake-options '--configfile <config_file_path>'\n```\n\n**NOTE** \nTo run the pipeline skipping the generation of datasets already available (in \ncase a previous run failed at an intermediate step for instance), the [option](#useful-snakemake-options) \n`--rerun-trigger mtime` can be used as in the following command:\n```\nbbp-atlas --target-rule <target_rule> --snakemake-options '--rerun-trigger mtime --cores 1'\n```\nSuch an option allows to skip the execution of the pipeline steps whose output \nfiles exist and have a modification time (`mtime`) more recent than any of their\ninput files.\n***\n\n### Customize a pipeline rule\n\nIt is possible to customize a pipeline rule that generates a (set of) volumetric file \n(`.nrrd`) in order to change the values of a specific region of the volume (and leave \nthe rest of the volume unchanged).\nThe customization happens via the configuration file [`customize_pipeline/user_config.json`](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/blob/develop/customize_pipeline/user_config.json)\nwith the following structure:\n- `rule`: name of the rule to customize from the default pipeline;\n- `brainRegion`: ID of the brain region to customize;\n- `CLI`:\n - `command`: CLI to execute in order to produce the volumetric file with the desired values for the brain region of interest;\n - `args`: CLI arguments that can reference variables between curly brackets (see below);\n- `output_dir`: path of the folder where the volumetric file(s) is generated by the CLI;\n- `container`: URL of the Docker image to use in order to spawn a container where the CLI will be executed. This parameter\nis optional: if not provided, the CLI will be executed in the same environment of the default pipeline (in such a case,\nthe user must ensure that the provided CLI is defined therein).\n\n_Note_: the Snakemake option `--use-singularity` must be provided for the configuration parameter `container` to be considered.\n\nThe CLI `args` can reference one or more variables which points to files generated by \npipeline rules executed before the rule to customize. The list of variables is available\nin [customize_pipeline/available_vars.yaml](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/blob/develop/customize_pipeline/available_vars.yaml).\n\n#### Filename convention\nThe user must ensure that the files generated by the provided CLI have the same names as\nthe files generated by the rule to customize. \nFor example, the rule `direction_vectors_placeholder_ccfv3`\nin the sample configuration generates one output file `direction_vectors_ccfv3.nrrd`. \nThe `placement_hints` rule generates seven volumetric files: `[PH]y.nrrd` and \n`[PH]layer_n.nrrd` where n = 1, ..., 6. The mapping between each nrrd file and the layer\nit refers to for each region is available in [this dictionary](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/blob/develop/metadata/PH_layers_regions_map.json),\nwhich the user needs to extend with the `\"region acronym\": {\"layer ID\", \"layer label\"}` \nof its customized region. A layer is considered associated to a region if the \ncorresponding layer ID appears in the [regions-to-layers mapping](#brain-region-layers) for that region \nor for at least one of that region's offspring.\n\n#### Customized pipeline\nOnce the [configuration file](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/blob/develop/customize_pipeline/user_config.json)\nis ready, the customized pipeline can be run with the following command:\n```\nbbp-atlas --target-rule <target_rule> --user-config-path customize_pipeline/user_config.json --snakemake-options '<options>'\n```\n\nWhen a rule is customized as described above, the pipeline will run\n1. the default rule to generate the default output file(s),\n2. the CLI provided in the configuration file to produce the corresponding \nregion-specific output file(s),\n3. a merge step to override the specific region in the default file(s) (step 1) with the \nvalues of that region from the region-specific file(s) (step 2).\n\n#### Integration\nIn case a user wants to request the integration of the customized version of a dataset:\n1. Open a Merge Request (MR) in this repository including the updated `user_config.json`\nand any additional input [metadata](#metadata) required.\n2. The MR is then reviewed and, if approved, a new Atlas pipeline dev image is produced accordingly.\n3. The new pipeline is run and the new datasets are registered in Nexus staging for wider tests.\n4. When the new version of the datasets is validated, a new tag of the Atlas pipeline is\ncut and the corresponding image is used to register the datasets in Nexus prod.\n\n##### Metadata\nSome pipeline steps require metadata as input, which are fetched from Nexus. \nCurrently, the files available in the [`metadata`](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/tree/develop/metadata)\ndirectory are automatically synchronized with their Nexus versions. \nIf you want to update/add one metadata file, make sure to update/add also the \ncorresponding documentation file in the [`metadata/docs`](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/tree/develop/metadata/docs)\ndirectory, keeping the current naming convention (`probability_map_*{.csv,.txt}`).\n\n#### Direction-vectors and placement-hints\n\nIf you want to add creation of your region's direction-vectors, placement-hints, or other NRRD files that need to be merged with files containing data from other regions, you should add them using the instructions in [Customize a pipeline rule](#customize-a-pipeline-rule) into `customize_pipeline/user_config.json` instead of the `snakefile` directly.\n\n### Useful Snakemake options\n\nSnakemake being a command-line tool, it comes with a multitude of optional arguments to\nexecute, debug, and visualize workflows. Here is a selection of the most used:\n\n- `--cores <number_of_cores>`, `-c <number_of_cores>` \u2192 Specify the number of cores \nsnakemake can use.\n- `--dry-run`, `-n` \u2192 Perform a dry run (execute nothing but print the list of rules\nthat would be executed).\n- `--rerun-trigger mtime` \u2192 Use only the modification time (`mtime`) of the existing \noutput files to determine which rules to execute.\n- `--forcerun <some_rule>` \u2192 Force a given rule to be re-executed (overwrite the output \nif it already exists).\n- `--list`, `-l` \u2192 Print a list of all the available rules from the snakefile.\n\nEvery Snakemake command line argument is listed and described in the [Snakemake](https://snakemake.readthedocs.io/en/stable/) official documentation page.\n\n\n## Blue Brain Atlas Pipeline\n\nIts workflow consists of the following steps:\n1. Fetch the required datasets from Nexus. These input data consist of the [original AIBS ccfv3 brain parcellation](https://bbp.epfl.ch/nexus/web/bbp/atlas/resources/https%3A%2F%2Fbbp.epfl.ch%2Fneurosciencegraph%2Fdata%2F025eef5f-2a9a-4119-b53f-338452c72f2a), \nthe [AIBS Mouse CCF Atlas regions hierarchy file](https://bbp.epfl.ch/nexus/web/neurosciencegraph/datamodels/resources/http%3A%2F%2Fbbp.epfl.ch%2Fneurosciencegraph%2Fontologies%2Fmba) and a series of Nissl and ISH volumes \nas described in the documentation page [Allen Mouse CCF Compatible Data](https://bbpteam.epfl.ch/project/spaces/display/BBKG/Allen+Mouse+CCF+Compatible+Data).\n2. The fetched datasets are then fed to the [Snakemake](https://snakemake.readthedocs.io/en/stable/) rules, and under the hood consumed by atlas modules to generate products. \n3. Each product can (optionally) be pushed into Nexus with a set of metadata automatically filled up and be visualised in \nthe [Blue Brain Atlas](https://bbpteam.epfl.ch/documentation/#:~:text=Visualize-,Blue%20Brain%20Atlas,-Morphology%20visualization).\n\nThis workflow is illustrated on the following diagram containing the directed acyclic graph (DAG)\nof the Snakemake rules of the BBAP:\n\n\n\nA more detailed DAG listing the input and output files for each step is available [here](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/blob/develop/doc/source/figures/dag_push_atlas_fg.svg).\n\n### Rules and modules\nIn this document, a \u201cmodule\u201d is a CLI encapsulated inside one of the components of the pipeline.\nSuch component is called a \u201crule\u201d. This terminology comes from SnakeMake, where a \u201crule\u201d \ncan leverage one or more modules and where a module can be used by one of more rules,\nusually using a different set of arguments. \nYou can find more information on rules in the [SnakeMake documentation](https://snakemake.readthedocs.io/en/stable/).\n\nTo only visualize the command that a given rule will execute without running it, one can\nuse the `--dryrun` option as in the following command:\n```\n bbp-atlas --target-rule <target_rule> --snakemake-options '--dryrun'\n```\nThe documentation of each command is available in the corresponding pipeline [dependency](#dependencies).\n\n#### Profiling\nA detailed profiling of the most resource-intensive rules (sorted by execution order) \nis available in the following table, corresponding to a single core of an \nIntel Xeon Gold 6140 CPU (BB5 node). \nSome rules can exploit multiple cores, in which case a second entry for such rules \nappears in the table along with the number of cores (\"--cores n\") used for the \nprofiling.\n\nThe **total multicore wall clock time** required by the two final rules `push_atlas_release` \n(which depends on direction vectors, orientation field, placement hints) and \n`push_cellComposition` (which depends on all the density generation rules) is \nrespectively **1 h** (with an RSS peak of 10 GB) and **4 h** (with an RSS peak of 8 GB).\n\n| Rule name | wall clock time [s] | wall clock time [h:m:s] | max [RSS](https://en.wikipedia.org/wiki/Resident_set_size) [MB] | max [VMS](https://en.wikipedia.org/wiki/Virtual_memory) [MB] | max [USS](https://en.wikipedia.org/wiki/Unique_set_size) [MB] | max [PSS](https://en.wikipedia.org/wiki/Proportional_set_size) [MB] | I/O in [B] | I/O out [B] | average CPU load [%] | CPU time [s] |\n|-------------------------------------------------------------|--------------------:|------------------------:|----------------------------------------------------------------:|-------------------------------------------------------------:|--------------------------------------------------------------:|--------------------------------------------------------------------:|-----------:|------------:|---------------------:|-------------:|\n| direction_vectors_default_ccfv3 | 352.1527 | 0:05:52 | 3345.09 | 4503.41 | 3309.88 | 3321.13 | 0.07 | 0.00 | 98.53 | 347.43 |\n| direction_vectors_isocortex_ccfv3 | 376.2279 | 0:06:16 | 5438.86 | 6049.21 | 5401.71 | 5412.96 | 0.00 | 0.00 | 92.23 | 347.29 |\n| orientation_field | 248.3647 | 0:04:08 | 8423.66 | 9010.55 | 8388.29 | 8399.54 | 0.00 | 0.00 | 91.73 | 228.17 |\n| split_isocortex_layer_23_ccfv3 | 147.5079 | 0:02:27 | 1945.17 | 2977.68 | 1866.67 | 1877.88 | 0.83 | 0.00 | 92.70 | 137.05 |\n| create_leaves_only_hierarchy_annotation_ccfv3 | 46.3272 | 0:00:46 | 5897.06 | 6542.66 | 5818.63 | 5830.39 | 0.04 | 0.00 | 36.30 | 17.06 |\n| split_barrel_ccfv3_l23split | 141.6368 | 0:02:21 | 715.49 | 2420.59 | 679.82 | 691.35 | 0.06 | 0.00 | 96.49 | 137.23 |\n| validate_annotation_v3 | 4.8913 | 0:00:04 | 908.17 | 1749.66 | 865.54 | 876.86 | 0.03 | 0.00 | 71.09 | 3.81 |\n| placement_hints | 924.7421 | 0:15:24 | 6600.64 | 7225.64 | 6524.41 | 6537.06 | 7.41 | 0.00 | 99.36 | 919.29 |\n| create_hemispheres_ccfv3 | 6.3128 | 0:00:06 | 547.75 | 1233.48 | 514.01 | 524.86 | 0.00 | 0.00 | 68.63 | 4.67 |\n| export_brain_region | 27482.2978 | 7:38:02 | 2649.79 | 3214.40 | 3497.71 | 3509.16 | 0.40 | 0.00 | 99.68 | 27395.00 |\n| export_brain_region (`--cores 70`) | 1162.2325 | 0:19:22 | 143397.54 | 193202.21 | 115383.76 | \t115749.06 | 0.04 | \t0.00 | \t4307.04 | 50067.62 |\n| combine_v2_annotations | 17.5364 | 0:00:17 | 1030.33 | 1581.99 | 1002.29 | 1015.24 | 1.09 | 0.00 | 85.00 | 15.44 |\n| direction_vectors_isocortex_ccfv2 | 299.9846 | 0:04:59 | 5483.91 | 6050.02 | 5454.11 | 5467.01 | 0.04 | 0.00 | 95.15 | 285.99 |\n| split_isocortex_layer_23_ccfv2 | 153.1193 | 0:02:33 | 1999.78 | 2979.68 | 1937.86 | 1950.75 | 0.00 | 0.00 | 88.49 | 135.96 |\n| create_leaves_only_hierarchy_annotation_ccfv2 | 31.2190 | 0:00:31 | 6027.33 | 6836.57 | 6276.27 | 6289.61 | 0.04 | 0.00 | 51.64 | 16.60 |\n| split_barrel_ccfv2_l23split | 112.8031 | 0:01:52 | 737.26 | 2421.75 | 706.36 | 719.25 | 0.06 | 0.00 | 93.59 | 106.04 |\n| validate_annotation_v2 | 2.3923 | 0:00:02 | 1116.59 | 1736.83 | 1076.46 | 1089.53 | 0.03 | 0.00 | 66.48 | 2.12 |\n| cell_density_correctednissl | 55.1323 | 0:00:55 | 2867.18 | 3418.92 | 2839.32 | 2852.28 | 0.00 | 0.00 | 82.24 | 45.88 |\n| validate_cell_density | 4.8925 | 0:00:04 | 1252.33 | 2355.65 | 1767.52 | 1780.59 | 0.00 | 0.00 | 76.66 | 4.27 |\n| combine_markers | 555.3407 | 0:09:15 | 5147.12 | 5697.71 | 5119.06 | 5132.01 | 0.00 | 0.00 | 94.58 | 525.61 |\n| glia_cell_densities_correctednissl | 221.8078 | 0:03:41 | 7373.72 | 8061.20 | 7298.36 | 7311.31 | 0.00 | 0.00 | 87.87 | 195.19 |\n| validate_neuron_glia_cell_densities | 17.4026 | 0:00:17 | 3845.63 | 4698.86 | 4109.74 | 4122.81 | 0.00 | 0.00 | 89.61 | 16.00 |\n| average_densities_correctednissl | 2886.3829 | 0:48:06 | 3889.09 | 5342.21 | 3841.21 | 3854.18 | 0.00 | 0.00 | 99.22 | 2864.22 |\n| fit_average_densities_correctednissl | 2185.0088 | 0:36:25 | 5605.01 | 7106.13 | 5410.92 | 5423.89 | 0.00 | 0.00 | 99.52 | 2174.67 |\n| inhibitory_neuron_densities_linprog_correctednissl | 2859.1158 | 0:47:39 | 4799.59 | 18207.76 | 4771.89 | 4784.86 | 0.00 | 0.00 | 99.16 | 2834.94 |\n| compute_lamp5_density | 53.7274 | 0:00:53 | 3652.39 | 4176.77 | 4964.55 | 4977.10 | 0.00 | 0.00 | 83.84 | 45.36 |\n| create_mtypes_densities_from_probability_map | 31310.7357 | 8:41:50 | 31178.52 | 32289.28 | 31150.59 | 31163.56 | 0.00 | 0.00 | 99.69 | 31209.90 |\n| create_mtypes_densities_from_probability_map (`--cores 70`) | 6276.3961 | 1:44:36 | 394077.36 | 2285360.80 | 63690.27 | 82778.46 | 1.04 | \t0.00 | \t5992.08 | 376431.41 |\n| excitatory_split | 222.6994 | 0:03:42 | 3437.98 | 3989.99 | 3410.12 | 3423.07 | 0.00 | 0.00 | 87.40 | 195.18 |\n| create_cellCompositionVolume_payload | 414.9835 | 0:06:54 | 0 | 0 | 0 | 0 | 0.00 | 0.00 | 0.25 | 0 |\n| create_cellCompositionSummary_payload | 1205.1099 | 0:20:05 | 2414.04 | \t5000.99 | \t2230.68 | 2308.75 | 3.73 | \t0.00 | 84.15 | 1030.68 |\n| create_cellCompositionSummary_payload (`--cores 70`) | 206.0507 | 0:03:26 | \t21817.64 | 67112.53 | 12911.70 | 13023.87 | 0.00 | \t0.00 | 635.94 | 1314.64 |\n\n\n#### Fetch rules\nThe rules starting with \"fetch_\" are used to download a given file from Nexus. \nThe IDs of the corresponding Nexus Resource (containing a description of the file to \nfetch) are listed in the [nexus_ids.json](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/blob/develop/nexus_ids.json) (the explicit link between a fetch \nrule and the corresponding Resource ID lays in the `nexus_id` parameter of the rule \ndefinition in the [snakefile](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/blob/develop/snakefile)). \n_Note_: the rule \"fetch_genes_correctednissl\" is not linked to a specific Resource, it's \nused just to trigger the execution of a set of single \"fetch_gene_\" rules needed by the\n\"fit-average-densities\" step.\n\nIn order to run the pipeline with a different version of a fetched file, one can just\nexecute the corresponding fetch rule and subsequently replace the downloaded file with \nthe desired version, by keeping the same name of the originally fetched file. \nThe `--rerun-trigger mtime` [option](#useful-snakemake-options) may be useful here.\n\n### Configuration\n\nThe configuration of the pipeline is provided in the `config.yaml` file. The most important\nvariables that a user can customize are:\n\n- `WORKING_DIR`: the output directory of the pipeline files,\n- `NEXUS_IDS_FILE`: the json file containing the Ids of the Nexus Resources to fetch,\n- `FORGE_CONFIG`: the configuration file (yaml) to instantiate nexus-forge,\n- `NEW_ATLAS`: boolean flag to trigger the creation of a brand-new atlas release,\n- `RESOLUTION`: resolution (in \u03bcm) of the input volumetric files to be consumed by the pipeline (default to 25),\n- `NEXUS_REGISTRATION`: boolean flag to trigger data registration in Nexus\n- `RESOURCE_TAG`: string to use as tag of the data registered in Nexus\n- `IS_PROD_ENV`: boolean flag to indicate whether the target Nexus environment is production or not (staging),\n- `NEXUS_DESTINATION_ORG`/`NEXUS_DESTINATION_PROJ`: Nexus organization/project where register the pipeline products,\n- `DISPLAY_HELP`: boolean flag to display every rule of the snakefile with its descriptions.\n\nIt is possible to override the config variables at runtime using the snakemake argument `--config`: \n`--config <VAR_NAME>=<VALUE>`\n\n### Additional information\nThe release notes are available [here](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/blob/develop/doc/release_notes.md).\n\nMore information about The Blue Brain Atlas Pipeline (BBAP) are available in its [confluence documentation](https://bbpteam.epfl.ch/project/spaces/display/BBKG/Atlas+Pipeline). \nThis space contains several documentation pages describing: \nThe Allen Mouse CCF Compatible Data : [https://bbpteam.epfl.ch/project/spaces/display/BBKG/Allen+Mouse+CCF+Compatible+Data](https://bbpteam.epfl.ch/project/spaces/display/BBKG/Allen+Mouse+CCF+Compatible+Data) \nThe Atlas Modules : [https://bbpteam.epfl.ch/project/spaces/display/BBKG/Atlas+Modules](https://bbpteam.epfl.ch/project/spaces/display/BBKG/Atlas+Modules)\n\n\n## Appendix\n\n### Brain region layers\nSome brain areas have a subdivision in layers. \nThe mapping adopted in the BBP between a brain region and the layers it belongs to is \nprovided in [this dictionary](https://bbpgitlab.epfl.ch/dke/apps/blue_brain_atlas_pipeline/-/blob/develop/metadata/regions_layers_map.json),\nwhere the keys are brain region IDs and the layers are identified with \n[Uberon](https://www.ebi.ac.uk/ols4/ontologies/uberon) classes. \nOne layer - \"Neocortex layer 6a\" - is not present in the Uberon ontology and is defined \nas follows:\n```\n<https://bbp.epfl.ch/ontologies/core/bmo/neocortex_layer_6a> rdf:type owl:Class ;\n rdfs:subClassOf <http://purl.obolibrary.org/obo/UBERON_0002301> ;\n rdfs:label \"L6a\"^^xsd:string ;\n <http://www.w3.org/2004/02/skos/core#definition> \"Neocortex layer 6a.\"^^xsd:string ;\n <http://www.w3.org/2004/02/skos/core#altLabel> \"layer 6a\"^^xsd:string ;\n <http://www.w3.org/2004/02/skos/core#altLabel> \"neocortex layer 6a\"^^xsd:string ;\n <http://www.w3.org/2004/02/skos/core#prefLabel> \"L6a\"^^xsd:string ;\n <http://www.w3.org/2004/02/skos/core#notation> \"L6a\"^^xsd:string .\n```\n\n\n### Placement hints data catalog json format\n```json\n{\n \"placementHints\": [\n {\n \"@id\": \"https://bbp.epfl.ch/data/bbp/atlas/f1049c1b-f1af-4d33-acd9-099f05c56bbf\",\n \"_rev\": 13,\n \"distribution\": {\n \"atLocation\": {\n \"location\": \"file:///gpfs/bbp.cscs.ch/data/project/proj39/nexus/bbp/atlas/9/b/1/3/3/7/7/9/%5BPH%5Dlayer_1.nrrd\"\n },\n \"name\": \"[PH]layer_1.nrrd\"\n },\n \"regions\": {\n \"Isocortex\": {\n \"@id\": \"http://api.brain-map.org/api/v2/data/Structure/315\",\n \"hasLeafRegionPart\": [\n \"PL1\",\n \"...\"\n ],\n \"layer\": {\n \"@id\": \"http://purl.obolibrary.org/obo/UBERON_0005390\",\n \"label\": \"L1\"\n }\n },\n \"Hippocampal formation\": {\n \"@id\": \"http://api.brain-map.org/api/v2/data/Structure/1089\",\n \"hasLeafRegionPart\": [\n \"CA1sp\",\n \"...\"\n ],\n \"layer\": {\n \"@id\": \"http://purl.obolibrary.org/obo/UBERON_0002313\",\n \"label\": \"SP\"\n }\n },\n \"...\": {}\n }\n },\n {\n \"@id\": \"https://bbp.epfl.ch/data/bbp/atlas/74ba22b1-39ee-486d-ab3c-cb960d006a5d\",\n \"_rev\": 13,\n \"distribution\": {\n \"atLocation\": {\n \"location\": \"file:///gpfs/bbp.cscs.ch/data/project/proj39/nexus/bbp/atlas/a/9/3/0/d/e/a/8/%5BPH%5Dlayer_2.nrrd\"\n },\n \"name\": \"[PH]layer_2.nrrd\"\n },\n \"regions\": {\n \"Isocortex\": {\n \"@id\": \"http://api.brain-map.org/api/v2/data/Structure/315\",\n \"hasLeafRegionPart\": [\n \"AUDp2\",\n \"...\"\n ],\n \"layer\": {\n \"@id\": \"http://purl.obolibrary.org/obo/UBERON_0005391\",\n \"label\": \"L2\"\n }\n },\n \"Hippocampal formation\": {\n \"@id\": \"http://api.brain-map.org/api/v2/data/Structure/1089\",\n \"hasLeafRegionPart\": [\n \"CA1so\",\n \"...\"\n ],\n \"layer\": {\n \"@id\": \"http://purl.obolibrary.org/obo/UBERON_0005371\",\n \"label\": \"SO\"\n }\n },\n \"...\": {}\n }\n },\n \"...\"\n ],\n \"voxelDistanceToRegionBottom\": {\n \"@id\": \"https://bbp.epfl.ch/data/bbp/atlas/59a2bca3-d8b6-43b1-870e-a0c19a020175\",\n \"_rev\": 13,\n \"distribution\": {\n \"atLocation\": {\n \"location\": \"file:///gpfs/bbp.cscs.ch/data/project/proj39/nexus/bbp/atlas/3/9/e/b/6/d/8/b/%5BPH%5Dy.nrrd\"\n },\n \"name\": \"[PH]y.nrrd\"\n }\n }\n}\n```\n\n\n## Funding & Acknowledgment\nThe development of this software was supported by funding to the Blue Brain Project, a \nresearch center of the \u00c9cole polytechnique f\u00e9d\u00e9rale de Lausanne (EPFL), from the Swiss \ngovernment\u2019s ETH Board of the Swiss Federal Institutes of Technology.\n\nCopyright \u00a9 2020-2024 Blue Brain Project/EPFL\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Package defining a SnakeMake pipeline to create the Blue Brain Atlas datasetsand push them into Nexus.",
"version": "1.3.1",
"project_urls": {
"Homepage": "https://github.com/BlueBrain/bbp-atlas-pipeline"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a1e158925f6bf0ffbb95732c4892aefc1e77d1bde3ab4dabc357e77c56ff4fa5",
"md5": "2881444a8e2424d12d1ba30aa68e81dd",
"sha256": "c24afdf9be931f55ebf681f1d00d2562148bbc420bac730a4cd1a430e78ba39b"
},
"downloads": -1,
"filename": "bba_pipeline-1.3.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "2881444a8e2424d12d1ba30aa68e81dd",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 15949707,
"upload_time": "2024-11-01T14:46:57",
"upload_time_iso_8601": "2024-11-01T14:46:57.028849Z",
"url": "https://files.pythonhosted.org/packages/a1/e1/58925f6bf0ffbb95732c4892aefc1e77d1bde3ab4dabc357e77c56ff4fa5/bba_pipeline-1.3.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "37fcf5a2e20b1f07082234f2b482dfa54bdcf17931ec7bd757bc7ac330008baa",
"md5": "3d80deec3599b8d2b68d82b47bc3848e",
"sha256": "2778de21f7b085bc51bfc39ccf0ff06e6a8b9440d285b4c4b234eacba4d38d95"
},
"downloads": -1,
"filename": "bba_pipeline-1.3.1.tar.gz",
"has_sig": false,
"md5_digest": "3d80deec3599b8d2b68d82b47bc3848e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 17102547,
"upload_time": "2024-11-01T14:46:59",
"upload_time_iso_8601": "2024-11-01T14:46:59.804722Z",
"url": "https://files.pythonhosted.org/packages/37/fc/f5a2e20b1f07082234f2b482dfa54bdcf17931ec7bd757bc7ac330008baa/bba_pipeline-1.3.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-01 14:46:59",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "BlueBrain",
"github_project": "bbp-atlas-pipeline",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "pytest",
"specs": []
},
{
"name": "pytest-cov",
"specs": []
},
{
"name": "pyyaml",
"specs": []
},
{
"name": "nexusforge",
"specs": [
[
">=",
"0.8.2"
]
]
},
{
"name": "sphinx",
"specs": []
},
{
"name": "sphinx-bluebrain-theme",
"specs": []
},
{
"name": "myst-parser",
"specs": []
},
{
"name": "numpy",
"specs": []
},
{
"name": "voxcell",
"specs": []
},
{
"name": "click",
"specs": []
},
{
"name": "blue-cwl",
"specs": []
},
{
"name": "pipeline-validator",
"specs": [
[
">=",
"0.3.1"
]
]
},
{
"name": "python-gitlab",
"specs": []
}
],
"lcname": "bba-pipeline"
}
JSON
