|banner|
|license| |doi| |tox| |coverage| |release| |doc|
Connectome-Manipulator
======================
A connectome manipulation framework for SONATA circuits
Table of contents
-----------------
1. `Introduction`_
2. `Install`_
3. `Framework overview`_
- `Main components`_
- `Operation principle of the "connectome manipulator"`_
4. `How to run`_
- `Connectome manipulation or building`_
- `Model building`_
- `Structural comparison`_
5. `Examples`_
6. `Documentation`_
7. `How to contribute`_
8. `Citation`_
9. `Publications that use or mention Connectome-Manipulator`_
- `Scientific papers that use Connectome-Manipulator`_
- `Posters that use Connectome-Manipulator`_
10. `Funding & Acknowledgment`_
Introduction
------------
An important way to study cortical function *in silico* lies in manipulations that are impossible to perform *in vivo* and *in vitro*. The purpose of the *Connectome-Manipulator* is to provide a general framework to apply and study various types of structural manipulations of a circuit connectome. The framework allows for rapid connectome manipulations of biophysically detailed network models in `SONATA <https://github.com/AllenInstitute/sonata/blob/master/docs/SONATA_DEVELOPER_GUIDE.md>`_ format, the standard for large-scale network models. Manipulations can be targeted to entire models, specific sub-networks, or even single neurons, ranging from insertion or removal of specific motifs to complete rewiring based on stochastic connectivity models at various levels of complexity.
Important use cases include wiring a connectome from scratch in a biologically realistic way based on given features of connectivity, rewiring an existing connectome while preserving certain aspects of connectivity, and transplanting specific connectivity characteristics from one connectome to another. The resulting connectomes can be readily simulated using any simulator supporting SONATA, allowing systematic and reproducible characterization of causal effects of manipulations on network activity.
Install
-------
From `PyPI <https://pypi.org/project/connectome-manipulator>`_
(recommended):
::
pip install connectome-manipulator
Alternatively, from
`GitHub <https://github.com/BlueBrain/connectome-manipulator>`_:
::
git clone https://github.com/BlueBrain/connectome-manipulator.git
cd connectome-manipulator
pip install .
All dependencies declared in ``setup.py`` and are available from PyPI, including one optional dependency, ``mpi4py`` (v3.1.4), which is required for parallel processing, i.e., to run ``parallel-manipulator``. Another optional dependency, ``parquet-converters`` (v0.8.0 or higher), required for converting .parquet output files to SONATA must be installed separately, see instructions under https://github.com/BlueBrain/parquet-converters.
Recommended Python version: v3.10.8
❗ Compatibility notes
~~~~~~~~~~~~~~~~~~~~~~
The software famework is intended to be used on Linux/MacOS-based systems! Specifically, some dependencies, like ``libsonata``, are currently not compatible with Microsoft Windows OS.
Framework overview
------------------
Main components
~~~~~~~~~~~~~~~
The *Connectome-Manipulator* framework is illustrated in Figure 1 and
consists of the following main components:
- | **Connectome manipulator**
| As specified in the config, applies one or a sequence of manipulations to a given SONATA connectome, and writes the manipulated connectome to a new SONATA edges file. All manipulations are separately implemented in sub-modules and can be easily extended.
| Details can be found in the corresponding README file in the repository: `connectome_manipulation/README.md <connectome_manipulator/connectome_manipulation/README.md>`_
- | **Model building**
| As specified in the config, builds a model from a given connectome and writes the model to a file to be loaded and used by specific manipulations requiring a model (e.g., model-based rewiring based on connection probability model). All models are separately implemented in sub-modules and can be easily extended.
| Details can be found in the corresponding README file in the repository: `model_building/README.md <connectome_manipulator/model_building/README.md>`_
Notes:
- Some models may not even require a connectome as input.
- Some models may depend on other models as input for model
building.
- | **Structural comparator**
| As specified in the config, performs a structural comparison of the original and manipulated connectomes. Different structural parameters to compare (connection probability, synapses per connection, ...) are separately implemented in sub-modules and can be easily extended.
| Details can be found in the corresponding README file in the repository: `connectome_comparison/README.md <connectome_manipulator/connectome_comparison/README.md>`_
The structure of the respective configuration files can be found under `doc/source/config_file_structure.rst <doc/source/config_file_structure.rst>`_
ℹ️ More details can be also found in the accompanying publication (esp.
*Supplementary tables*), see `Citation`_.
|schematic|
**Figure 1:** Schematic overview of the connectome manipulation framework, consisting of the "connectome manipulator", "model building", and "structural comparator" components.
Operation principle of the "connectome manipulator"
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
As illustrated in Figure 2, the synapses of the connectome (SONATA edges) are divided into k splits targeting disjoint sets of N post-synaptic neurons (SONATA nodes), which reduces the memory consumption and facilitates parallelization on multiple computation nodes. Each split is an edge table loaded as Pandas dataframe and comprising a list of synapses together with all synapse properties, an example is shown in Figure 3. The manipulations are then applied separately to each edge table in sequence (or alternatively, in parallel), resulting in manipulated edge tables which are then written to separate .parquet files. In the end, all .parquet files are merged into one manipulated SONATA connectome file using ``parquet-converters``.
|operation|
**Figure 2:** Operation principle of the "connectome manipulator", illustrating its split-based processing architecture.
|edgetable|
**Figure 3:** Example of an edge table (Pandas dataframe), comprising a list of synapses together with all synapse properties.
..
Notes:
- Manipulations can only be applied to a single SONATA edges population at a time.
- The synapses in each loaded edge table are assumed to be sorted by post-synaptic neuron ID. Likewise, the manipulated edges tables are to be returned with synapses sorted by post-synaptic neuron ID.
- Optionally, processing can be resumed from an earlier (incomplete) run, by re-using all .parquet files that already exist instead of re-computing them.
- By default, all .parquet files will be deleted after successfull completion, i.e., after the manipulated SONATA connectome file has been generated. Optionally, these temporary .parquet files can be kept as well.
How to run
----------
::
Usage: connectome-manipulator [OPTIONS] COMMAND [ARGS]...
Connectome manipulation tools.
Options:
--version Show the version and exit.
-v, --verbose -v for INFO, -vv for DEBUG [default: 0]
--help Show this message and exit.
Commands:
build-model Extract and build models from existing connectomes.
compare-connectomes Compare connectome structure of two circuits.
manipulate-connectome Manipulate or build a circuit's connectome.
Connectome manipulation or building
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
Usage: connectome-manipulator manipulate-connectome [OPTIONS] CONFIG
Manipulate or build a circuit's connectome.
Options:
--output-dir PATH Output directory. [required]
--profile Enable profiling.
--resume Resume from exisiting .parquet files.
--keep-parquet Keep temporary parquet files.
--convert-to-sonata Convert parquet to sonata and generate circuit
config
--overwrite-edges Overwrite existing edges file
--splits INTEGER Number of blocks, overwrites value in config file
[default: 0]
--target-payload INTEGER Number of gid-gid pairs to consider for one block.
Supersedes splits when a parquet based
configuration is used [default: 20000000000]
--parallel Run using a parallel DASK job scheduler
-a, --parallel-arg TEXT Overwrite the arguments for the Dask Client with
key=value
--help Show this message and exit.
Just running serially you can do something like this:
::
connectome-manipulator -v manipulate-connectome wiring_config.json \
--output-dir PATH_TO_OUTPUT --profile --convert-to-sonata --splits 1
Running splits in parallel (with Dask) you can use the ``parallel-manipulator`` executable that will set up Dask automatically (and switch the ``--parallel`` flag by default to ``True``, too):
::
srun --nodes 10 --tasks-per-node=2 --cpus-per-task=20 --constraint=clx --mem=0 \
parallel-manipulator -v manipulate-connectome wiring_config.json \
--output-dir PATH_TO_OUTPUT --profile --convert-to-sonata --splits=100
Please note that this feature will require at least 4 MPI ranks. Dask will use 2 ranks to manage the distributed cluster. We recommend to use a high number for ``--cpus-per-task`` to create Dask workers that will be able to process a lot of data in parallel.
When processing with ``parallel-manipulator``, one may pass the flag ``--target-payload`` to determine how big the individual workload for each process should be. The default value of 20e9 was determined empirically to run on the whole mouse brain with 75 million neurons. We recommend to use this value as a starting point and scale it up or down to achieve the desired runtime characteristics.
Details on the CONFIG file structure can be found under `doc/source/config_file_structure.rst <doc/source/config_file_structure.rst>`_
❗ Notes on error handling
^^^^^^^^^^^^^^^^^^^^^^^^^^
Errors may occur for many different reasons and are not always easy to track. Most common errors are that an allocation gets "killed", either due to a time limit or due to an out-of-memory error. Here we provide a few hints on how to avoid or track errors that may occur:
- Use the "verbose" mode (``-v`` flag) which will produce a lot of log output.
- Look into the log files: there is usually one master log file and individual log files for all data splits, all of which can be found in the ``/logs`` subfolder of the output circuit folder.
- Use a small connectome to start with.
- Use a simple operation to start with, such as ``null_manipulation`` (see examples).
- Run serially to start with, before switching to parallel processing.
- Start with a single data split.
- But: In case of memory errors, use more than a single data splits, even when running serially (!), which will reduce the memory consumption as individual splits will be processed one after the other.
- When running in parallel, use ``--tasks-per-node`` in the SLURM configuration to define how many tasks (=splits) will be executed on a single node; reducing this number may reduce the risk of out-of-memory errors.
- In general: Increasing memory allocation and/or allocation time may help.
- For high performance: Allocate many nodes and use parallel processing together with a relatively large number of data splits depending on the network size (i.e., aim for a few hundered post-synaptic neurons per data split).
Model building
~~~~~~~~~~~~~~
::
Usage: connectome-manipulator build-model [OPTIONS] CONFIG
Extract and build models from existing connectomes.
Options:
--force-reextract Force re-extraction of data, in case already existing.
--force-rebuild Force model re-building, in case already existing.
--cv-folds INTEGER Optional number of cross-validation folds, overwrites
value in config file
--help Show this message and exit.
Details on the CONFIG file structure can be found under `doc/source/config_file_structure.rst <doc/source/config_file_structure.rst>`_
Structural comparison
~~~~~~~~~~~~~~~~~~~~~
::
Usage: connectome-manipulator compare-connectomes [OPTIONS] CONFIG
Compare connectome structure of two circuits.
Options:
--force-recomp-circ1 Force re-computation of 1st circuit's comparison data,
in case already existing.
--force-recomp-circ2 Force re-computation of 2nd circuit's comparison data,
in case already existing.
--help Show this message and exit.
Details on the CONFIG file structure can be found under `doc/source/config_file_structure.rst <doc/source/config_file_structure.rst>`_
Examples
--------
Examples can be found under `examples/ </examples>`_ in the repository.
Documentation
-------------
The full documentation (API reference, CONFIG file structure, ...) can be found on `Read the Docs <https://connectome-manipulator.readthedocs.io>`_.
How to contribute
-----------------
Contribution guidelines can be found in `CONTRIBUTING.md <CONTRIBUTING.md>`_ in the repository.
Citation
--------
If you use this software, we kindly ask you to cite the following publication:
Christoph Pokorny, Omar Awile, James B. Isbister, Kerem Kurban, Matthias Wolf, and Michael W. Reimann (2024). **A connectome manipulation framework for the systematic and reproducible study of structure-function relationships through simulations.** bioRxiv 2024.05.24.593860. DOI: `10.1101/2024.05.24.593860 <https://doi.org/10.1101/2024.05.24.593860>`_
::
@article{pokorny2024connectome,
author = {Pokorny, Christoph and Awile, Omar and Isbister, James B and Kurban, Kerem and Wolf, Matthias and Reimann, Michael W},
title = {A connectome manipulation framework for the systematic and reproducible study of structure--function relationships through simulations},
journal = {bioRxiv},
year = {2024},
publisher={Cold Spring Harbor Laboratory},
doi = {10.1101/2024.05.24.593860}
}
Publications that use or mention Connectome-Manipulator
-------------------------------------------------------
Scientific papers that use Connectome-Manipulator
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Michael W. Reimann, Sirio Bolaños-Puchet, Jean-Denis Courcol, Daniela Egas Santander, et al. (2024) **Modeling and Simulation of Neocortical Micro- and Mesocircuitry. Part I: Anatomy.** eLife, 13:RP99688. DOI: `10.7554/eLife.99688.1 <https://doi.org/10.7554/eLife.99688.1>`_
- James B. Isbister, András Ecker, Christoph Pokorny, Sirio Bolaños-Puchet, Daniela Egas Santander, et al. (2023) **Modeling and Simulation of Neocortical Micro- and Mesocircuitry. Part II: Physiology and Experimentation.** bioRxiv, 2023.05.17.541168. DOI: `10.1101/2023.05.17.541168 <https://doi.org/10.1101/2023.05.17.541168>`_
- Daniela Egas Santander, Christoph Pokorny, András Ecker, Jānis Lazovskis, Matteo Santoro, Jason P. Smith, Kathryn Hess, Ran Levi, and Michael W. Reimann. (2024) **Efficiency and reliability in biological neural network architectures.** bioRxiv, 2024.03.15.585196. DOI: `10.1101/2024.03.15.585196 <https://doi.org/10.1101/2024.03.15.585196>`_
Posters that use Connectome-Manipulator
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Christoph Pokorny, Sirio Bolaños-Puchet, András Ecker, James B. Isbister, Michael Gevaert, Joni Herttuainen, Henry Markram, and Michael W. Reimann. **Impact of simplified network structure on cortical activity.** Bernstein Conference, 2022, Berlin.
- Kerem Kurban, Christoph Pokorny, Julian Budd, Alberto Antonietti, Armando Romani, and Henry Markram. **Topological properties of a full-scale model of rat hippocampus CA1 and their functional implications.** Annual meeting of the Society for Neuroscience, 2022, San Diego.
- Christoph Pokorny, Omar Awile, Sirio Bolaños-Puchet, András Ecker, Daniela Egas Santander, James B. Isbister, Matthias Wolf, Henry Markram, and Michael W. Reimann. **A connectome manipulation framework for the systematic and reproducible study of the structure-function relationship through simulations.** Bernstein Conference, 2023, Berlin.
- Christoph Pokorny, Omar Awile, James B. Isbister, Kerem Kurban, Matthias Wolf, and Michael W. Reimann. **A connectome manipulation framework for the systematic and reproducible study of structure-function relationships through simulations.** FENS Forum, 2024, Vienna.
Funding & Acknowledgment
------------------------
The development of this software was supported by funding to the Blue Brain Project, a research center of the École polytechnique fédérale de Lausanne (EPFL), from the Swiss government’s ETH Board of the Swiss Federal Institutes of Technology.
Copyright (c) 2024 Blue Brain Project/EPFL
.. |license| image:: https://img.shields.io/badge/License-Apache_2.0-blue.svg
:target: https://opensource.org/licenses/Apache-2.0
:alt: License
.. |doi| image:: http://img.shields.io/badge/DOI-10.1101/2024.05.24.593860-B31B1B.svg
:target: https://doi.org/10.1101/2024.05.24.593860
:alt: DOI
.. |tox| image:: https://github.com/BlueBrain/connectome-manipulator/actions/workflows/run-tox.yml/badge.svg
:target: https://github.com/BlueBrain/connectome-manipulator/actions/workflows/run-tox.yml
:alt: Tox
.. |coverage| image:: https://codecov.io/github/BlueBrain/connectome-manipulator/coverage.svg?branch=main
:target: https://codecov.io/github/BlueBrain/connectome-manipulator
:alt: Coverage
.. |release| image:: https://img.shields.io/pypi/v/connectome-manipulator.svg
:target: https://pypi.org/project/connectome-manipulator/
:alt: Release
.. |doc| image:: https://readthedocs.org/projects/connectome-manipulator/badge/?version=latest
:target: https://connectome-manipulator.readthedocs.io
:alt: Documentation
.. substitutions
.. |banner| image:: BPP-Connectome-Manipulator-Banner.jpg
.. |schematic| image:: doc/source/images/schematic_overview.png
.. |operation| image:: doc/source/images/operation_principle.png
.. |edgetable| image:: doc/source/images/edge_table.png
Raw data
{
"_id": null,
"home_page": "https://github.com/BlueBrain/connectome-manipulator.git",
"name": "connectome-manipulator",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": null,
"author": "Blue Brain Project, EPFL",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/2e/db/f8c1824044b8825bda4499277c0416c84c93af1af937e14c85a3fc6eecc5/connectome_manipulator-1.0.2.tar.gz",
"platform": null,
"description": "|banner|\n\n|license| |doi| |tox| |coverage| |release| |doc|\n\nConnectome-Manipulator\n======================\n\nA connectome manipulation framework for SONATA circuits\n\nTable of contents\n-----------------\n\n1. `Introduction`_\n2. `Install`_\n3. `Framework overview`_\n\n - `Main components`_\n - `Operation principle of the \"connectome manipulator\"`_\n\n4. `How to run`_\n\n - `Connectome manipulation or building`_\n - `Model building`_\n - `Structural comparison`_\n\n5. `Examples`_\n6. `Documentation`_\n7. `How to contribute`_\n8. `Citation`_\n9. `Publications that use or mention Connectome-Manipulator`_\n\n - `Scientific papers that use Connectome-Manipulator`_\n - `Posters that use Connectome-Manipulator`_\n\n10. `Funding & Acknowledgment`_\n\nIntroduction\n------------\n\nAn important way to study cortical function *in silico* lies in manipulations that are impossible to perform *in vivo* and *in vitro*. The purpose of the *Connectome-Manipulator* is to provide a general framework to apply and study various types of structural manipulations of a circuit connectome. The framework allows for rapid connectome manipulations of biophysically detailed network models in `SONATA <https://github.com/AllenInstitute/sonata/blob/master/docs/SONATA_DEVELOPER_GUIDE.md>`_ format, the standard for large-scale network models. Manipulations can be targeted to entire models, specific sub-networks, or even single neurons, ranging from insertion or removal of specific motifs to complete rewiring based on stochastic connectivity models at various levels of complexity.\nImportant use cases include wiring a connectome from scratch in a biologically realistic way based on given features of connectivity, rewiring an existing connectome while preserving certain aspects of connectivity, and transplanting specific connectivity characteristics from one connectome to another. The resulting connectomes can be readily simulated using any simulator supporting SONATA, allowing systematic and reproducible characterization of causal effects of manipulations on network activity.\n\nInstall\n-------\n\nFrom `PyPI <https://pypi.org/project/connectome-manipulator>`_\n(recommended):\n\n::\n\n pip install connectome-manipulator\n\nAlternatively, from\n`GitHub <https://github.com/BlueBrain/connectome-manipulator>`_:\n\n::\n\n git clone https://github.com/BlueBrain/connectome-manipulator.git\n cd connectome-manipulator\n pip install .\n\nAll dependencies declared in ``setup.py`` and are available from PyPI, including one optional dependency, ``mpi4py`` (v3.1.4), which is required for parallel processing, i.e., to run ``parallel-manipulator``. Another optional dependency, ``parquet-converters`` (v0.8.0 or higher), required for converting .parquet output files to SONATA must be installed separately, see instructions under https://github.com/BlueBrain/parquet-converters.\n\nRecommended Python version: v3.10.8\n\n\u2757 Compatibility notes\n~~~~~~~~~~~~~~~~~~~~~~\n\nThe software famework is intended to be used on Linux/MacOS-based systems! Specifically, some dependencies, like ``libsonata``, are currently not compatible with Microsoft Windows OS.\n\nFramework overview\n------------------\n\nMain components\n~~~~~~~~~~~~~~~\n\nThe *Connectome-Manipulator* framework is illustrated in Figure 1 and\nconsists of the following main components:\n\n- | **Connectome manipulator**\n | As specified in the config, applies one or a sequence of manipulations to a given SONATA connectome, and writes the manipulated connectome to a new SONATA edges file. All manipulations are separately implemented in sub-modules and can be easily extended.\n | Details can be found in the corresponding README file in the repository: `connectome_manipulation/README.md <connectome_manipulator/connectome_manipulation/README.md>`_\n\n- | **Model building**\n | As specified in the config, builds a model from a given connectome and writes the model to a file to be loaded and used by specific manipulations requiring a model (e.g., model-based rewiring based on connection probability model). All models are separately implemented in sub-modules and can be easily extended.\n | Details can be found in the corresponding README file in the repository: `model_building/README.md <connectome_manipulator/model_building/README.md>`_\n\n Notes:\n\n - Some models may not even require a connectome as input.\n - Some models may depend on other models as input for model\n building.\n\n- | **Structural comparator**\n | As specified in the config, performs a structural comparison of the original and manipulated connectomes. Different structural parameters to compare (connection probability, synapses per connection, ...) are separately implemented in sub-modules and can be easily extended.\n | Details can be found in the corresponding README file in the repository: `connectome_comparison/README.md <connectome_manipulator/connectome_comparison/README.md>`_\n\nThe structure of the respective configuration files can be found under `doc/source/config_file_structure.rst <doc/source/config_file_structure.rst>`_\n\n\u2139\ufe0f More details can be also found in the accompanying publication (esp.\n*Supplementary tables*), see `Citation`_.\n\n|schematic|\n**Figure 1:** Schematic overview of the connectome manipulation framework, consisting of the \"connectome manipulator\", \"model building\", and \"structural comparator\" components.\n\nOperation principle of the \"connectome manipulator\"\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nAs illustrated in Figure 2, the synapses of the connectome (SONATA edges) are divided into k splits targeting disjoint sets of N post-synaptic neurons (SONATA nodes), which reduces the memory consumption and facilitates parallelization on multiple computation nodes. Each split is an edge table loaded as Pandas dataframe and comprising a list of synapses together with all synapse properties, an example is shown in Figure 3. The manipulations are then applied separately to each edge table in sequence (or alternatively, in parallel), resulting in manipulated edge tables which are then written to separate .parquet files. In the end, all .parquet files are merged into one manipulated SONATA connectome file using ``parquet-converters``.\n\n|operation|\n**Figure 2:** Operation principle of the \"connectome manipulator\", illustrating its split-based processing architecture.\n\n|edgetable|\n**Figure 3:** Example of an edge table (Pandas dataframe), comprising a list of synapses together with all synapse properties.\n\n..\n\n Notes:\n\n - Manipulations can only be applied to a single SONATA edges population at a time.\n - The synapses in each loaded edge table are assumed to be sorted by post-synaptic neuron ID. Likewise, the manipulated edges tables are to be returned with synapses sorted by post-synaptic neuron ID.\n - Optionally, processing can be resumed from an earlier (incomplete) run, by re-using all .parquet files that already exist instead of re-computing them.\n - By default, all .parquet files will be deleted after successfull completion, i.e., after the manipulated SONATA connectome file has been generated. Optionally, these temporary .parquet files can be kept as well.\n\nHow to run\n----------\n\n::\n\n Usage: connectome-manipulator [OPTIONS] COMMAND [ARGS]...\n\n Connectome manipulation tools.\n\n Options:\n --version Show the version and exit.\n -v, --verbose -v for INFO, -vv for DEBUG [default: 0]\n --help Show this message and exit.\n\n Commands:\n build-model Extract and build models from existing connectomes.\n compare-connectomes Compare connectome structure of two circuits.\n manipulate-connectome Manipulate or build a circuit's connectome.\n\nConnectome manipulation or building\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n::\n\n Usage: connectome-manipulator manipulate-connectome [OPTIONS] CONFIG\n\n Manipulate or build a circuit's connectome.\n\n Options:\n --output-dir PATH Output directory. [required]\n --profile Enable profiling.\n --resume Resume from exisiting .parquet files.\n --keep-parquet Keep temporary parquet files.\n --convert-to-sonata Convert parquet to sonata and generate circuit\n config\n --overwrite-edges Overwrite existing edges file\n --splits INTEGER Number of blocks, overwrites value in config file\n [default: 0]\n --target-payload INTEGER Number of gid-gid pairs to consider for one block.\n Supersedes splits when a parquet based\n configuration is used [default: 20000000000]\n --parallel Run using a parallel DASK job scheduler\n -a, --parallel-arg TEXT Overwrite the arguments for the Dask Client with\n key=value\n --help Show this message and exit.\n\nJust running serially you can do something like this:\n\n::\n\n connectome-manipulator -v manipulate-connectome wiring_config.json \\\n --output-dir PATH_TO_OUTPUT --profile --convert-to-sonata --splits 1\n\nRunning splits in parallel (with Dask) you can use the ``parallel-manipulator`` executable that will set up Dask automatically (and switch the ``--parallel`` flag by default to ``True``, too):\n\n::\n\n srun --nodes 10 --tasks-per-node=2 --cpus-per-task=20 --constraint=clx --mem=0 \\\n parallel-manipulator -v manipulate-connectome wiring_config.json \\\n --output-dir PATH_TO_OUTPUT --profile --convert-to-sonata --splits=100\n\nPlease note that this feature will require at least 4 MPI ranks. Dask will use 2 ranks to manage the distributed cluster. We recommend to use a high number for ``--cpus-per-task`` to create Dask workers that will be able to process a lot of data in parallel.\n\nWhen processing with ``parallel-manipulator``, one may pass the flag ``--target-payload`` to determine how big the individual workload for each process should be. The default value of 20e9 was determined empirically to run on the whole mouse brain with 75 million neurons. We recommend to use this value as a starting point and scale it up or down to achieve the desired runtime characteristics.\n\nDetails on the CONFIG file structure can be found under `doc/source/config_file_structure.rst <doc/source/config_file_structure.rst>`_\n\n\u2757 Notes on error handling\n^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nErrors may occur for many different reasons and are not always easy to track. Most common errors are that an allocation gets \"killed\", either due to a time limit or due to an out-of-memory error. Here we provide a few hints on how to avoid or track errors that may occur:\n\n- Use the \"verbose\" mode (``-v`` flag) which will produce a lot of log output.\n- Look into the log files: there is usually one master log file and individual log files for all data splits, all of which can be found in the ``/logs`` subfolder of the output circuit folder.\n- Use a small connectome to start with.\n- Use a simple operation to start with, such as ``null_manipulation`` (see examples).\n- Run serially to start with, before switching to parallel processing.\n- Start with a single data split.\n- But: In case of memory errors, use more than a single data splits, even when running serially (!), which will reduce the memory consumption as individual splits will be processed one after the other.\n- When running in parallel, use ``--tasks-per-node`` in the SLURM configuration to define how many tasks (=splits) will be executed on a single node; reducing this number may reduce the risk of out-of-memory errors.\n- In general: Increasing memory allocation and/or allocation time may help.\n- For high performance: Allocate many nodes and use parallel processing together with a relatively large number of data splits depending on the network size (i.e., aim for a few hundered post-synaptic neurons per data split).\n\nModel building\n~~~~~~~~~~~~~~\n\n::\n\n Usage: connectome-manipulator build-model [OPTIONS] CONFIG\n\n Extract and build models from existing connectomes.\n\n Options:\n --force-reextract Force re-extraction of data, in case already existing.\n --force-rebuild Force model re-building, in case already existing.\n --cv-folds INTEGER Optional number of cross-validation folds, overwrites\n value in config file\n --help Show this message and exit.\n\nDetails on the CONFIG file structure can be found under `doc/source/config_file_structure.rst <doc/source/config_file_structure.rst>`_\n\nStructural comparison\n~~~~~~~~~~~~~~~~~~~~~\n\n::\n\n Usage: connectome-manipulator compare-connectomes [OPTIONS] CONFIG\n\n Compare connectome structure of two circuits.\n\n Options:\n --force-recomp-circ1 Force re-computation of 1st circuit's comparison data,\n in case already existing.\n --force-recomp-circ2 Force re-computation of 2nd circuit's comparison data,\n in case already existing.\n --help Show this message and exit.\n\nDetails on the CONFIG file structure can be found under `doc/source/config_file_structure.rst <doc/source/config_file_structure.rst>`_\n\nExamples\n--------\n\nExamples can be found under `examples/ </examples>`_ in the repository.\n\nDocumentation\n-------------\n\nThe full documentation (API reference, CONFIG file structure, ...) can be found on `Read the Docs <https://connectome-manipulator.readthedocs.io>`_.\n\nHow to contribute\n-----------------\n\nContribution guidelines can be found in `CONTRIBUTING.md <CONTRIBUTING.md>`_ in the repository.\n\nCitation\n--------\n\nIf you use this software, we kindly ask you to cite the following publication:\n\nChristoph Pokorny, Omar Awile, James B. Isbister, Kerem Kurban, Matthias Wolf, and Michael W. Reimann (2024). **A connectome manipulation framework for the systematic and reproducible study of structure-function relationships through simulations.** bioRxiv 2024.05.24.593860. DOI: `10.1101/2024.05.24.593860 <https://doi.org/10.1101/2024.05.24.593860>`_\n\n::\n\n @article{pokorny2024connectome,\n author = {Pokorny, Christoph and Awile, Omar and Isbister, James B and Kurban, Kerem and Wolf, Matthias and Reimann, Michael W},\n title = {A connectome manipulation framework for the systematic and reproducible study of structure--function relationships through simulations},\n journal = {bioRxiv},\n year = {2024},\n publisher={Cold Spring Harbor Laboratory},\n doi = {10.1101/2024.05.24.593860}\n }\n\nPublications that use or mention Connectome-Manipulator\n-------------------------------------------------------\n\nScientific papers that use Connectome-Manipulator\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n- Michael W. Reimann, Sirio Bola\u00f1os-Puchet, Jean-Denis Courcol, Daniela Egas Santander, et al. (2024) **Modeling and Simulation of Neocortical Micro- and Mesocircuitry. Part I: Anatomy.** eLife, 13:RP99688. DOI: `10.7554/eLife.99688.1 <https://doi.org/10.7554/eLife.99688.1>`_\n\n- James B. Isbister, Andr\u00e1s Ecker, Christoph Pokorny, Sirio Bola\u00f1os-Puchet, Daniela Egas Santander, et al. (2023) **Modeling and Simulation of Neocortical Micro- and Mesocircuitry. Part II: Physiology and Experimentation.** bioRxiv, 2023.05.17.541168. DOI: `10.1101/2023.05.17.541168 <https://doi.org/10.1101/2023.05.17.541168>`_\n\n- Daniela Egas Santander, Christoph Pokorny, Andr\u00e1s Ecker, J\u0101nis Lazovskis, Matteo Santoro, Jason P. Smith, Kathryn Hess, Ran Levi, and Michael W. Reimann. (2024) **Efficiency and reliability in biological neural network architectures.** bioRxiv, 2024.03.15.585196. DOI: `10.1101/2024.03.15.585196 <https://doi.org/10.1101/2024.03.15.585196>`_\n\nPosters that use Connectome-Manipulator\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n- Christoph Pokorny, Sirio Bola\u00f1os-Puchet, Andr\u00e1s Ecker, James B. Isbister, Michael Gevaert, Joni Herttuainen, Henry Markram, and Michael W. Reimann. **Impact of simplified network structure on cortical activity.** Bernstein Conference, 2022, Berlin.\n\n- Kerem Kurban, Christoph Pokorny, Julian Budd, Alberto Antonietti, Armando Romani, and Henry Markram. **Topological properties of a full-scale model of rat hippocampus CA1 and their functional implications.** Annual meeting of the Society for Neuroscience, 2022, San Diego.\n\n- Christoph Pokorny, Omar Awile, Sirio Bola\u00f1os-Puchet, Andr\u00e1s Ecker, Daniela Egas Santander, James B. Isbister, Matthias Wolf, Henry Markram, and Michael W. Reimann. **A connectome manipulation framework for the systematic and reproducible study of the structure-function relationship through simulations.** Bernstein Conference, 2023, Berlin.\n\n- Christoph Pokorny, Omar Awile, James B. Isbister, Kerem Kurban, Matthias Wolf, and Michael W. Reimann. **A connectome manipulation framework for the systematic and reproducible study of structure-function relationships through simulations.** FENS Forum, 2024, Vienna.\n\nFunding & Acknowledgment\n------------------------\n\nThe development of this software was supported by funding to the Blue Brain Project, a research center of the \u00c9cole polytechnique f\u00e9d\u00e9rale de Lausanne (EPFL), from the Swiss government\u2019s ETH Board of the Swiss Federal Institutes of Technology.\n\nCopyright (c) 2024 Blue Brain Project/EPFL\n\n.. |license| image:: https://img.shields.io/badge/License-Apache_2.0-blue.svg\n :target: https://opensource.org/licenses/Apache-2.0\n :alt: License\n.. |doi| image:: http://img.shields.io/badge/DOI-10.1101/2024.05.24.593860-B31B1B.svg\n :target: https://doi.org/10.1101/2024.05.24.593860\n :alt: DOI\n.. |tox| image:: https://github.com/BlueBrain/connectome-manipulator/actions/workflows/run-tox.yml/badge.svg\n :target: https://github.com/BlueBrain/connectome-manipulator/actions/workflows/run-tox.yml\n :alt: Tox\n.. |coverage| image:: https://codecov.io/github/BlueBrain/connectome-manipulator/coverage.svg?branch=main\n :target: https://codecov.io/github/BlueBrain/connectome-manipulator\n :alt: Coverage\n.. |release| image:: https://img.shields.io/pypi/v/connectome-manipulator.svg\n :target: https://pypi.org/project/connectome-manipulator/\n :alt: Release\n.. |doc| image:: https://readthedocs.org/projects/connectome-manipulator/badge/?version=latest\n :target: https://connectome-manipulator.readthedocs.io\n :alt: Documentation\n\n.. substitutions\n.. |banner| image:: BPP-Connectome-Manipulator-Banner.jpg\n.. |schematic| image:: doc/source/images/schematic_overview.png\n.. |operation| image:: doc/source/images/operation_principle.png\n.. |edgetable| image:: doc/source/images/edge_table.png\n",
"bugtrack_url": null,
"license": "Apache-2",
"summary": "A connectome manipulation framework for SONATA circuits",
"version": "1.0.2",
"project_urls": {
"Homepage": "https://github.com/BlueBrain/connectome-manipulator.git"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f839c9c6ba3859651b955790a7921d1613e6da0be0571b89f7a4a7a4a3425dcf",
"md5": "b32ef6fb9e3e4018e302f5f42c6a9451",
"sha256": "3e42af86e594866da0cc0f12dd876afd44afc1de167a04519aaf669a6fce0e65"
},
"downloads": -1,
"filename": "connectome_manipulator-1.0.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "b32ef6fb9e3e4018e302f5f42c6a9451",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 149873,
"upload_time": "2024-11-05T08:08:17",
"upload_time_iso_8601": "2024-11-05T08:08:17.257794Z",
"url": "https://files.pythonhosted.org/packages/f8/39/c9c6ba3859651b955790a7921d1613e6da0be0571b89f7a4a7a4a3425dcf/connectome_manipulator-1.0.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "2edbf8c1824044b8825bda4499277c0416c84c93af1af937e14c85a3fc6eecc5",
"md5": "ec7ee5b2c02fd54a49109b73b5af8f10",
"sha256": "416b8d8c29c1df4cc015c87317dae58840765b746acfe642cac5cf8a9cb60172"
},
"downloads": -1,
"filename": "connectome_manipulator-1.0.2.tar.gz",
"has_sig": false,
"md5_digest": "ec7ee5b2c02fd54a49109b73b5af8f10",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 1112297,
"upload_time": "2024-11-05T08:08:18",
"upload_time_iso_8601": "2024-11-05T08:08:18.787365Z",
"url": "https://files.pythonhosted.org/packages/2e/db/f8c1824044b8825bda4499277c0416c84c93af1af937e14c85a3fc6eecc5/connectome_manipulator-1.0.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-05 08:08:18",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "BlueBrain",
"github_project": "connectome-manipulator",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "connectome-manipulator"
}
JSON
