| Name | swc-ephys JSON |
| Version |
0.0.0
JSON |
| download |
| home_page | |
| Summary | Manage extracellular electrophysiology analysis |
| upload_time | 2023-08-03 19:35:20 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.8.0 |
| license | BSD-3-Clause |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
> **Warning**
> **SWC Ephys is not sufficiently tested to be used in analysis. This release is only for testing. Do not use for your final analyses.**
> **Warning** **Limitations**
> - works only on SpikeGLX recordings with 1 gate, 1 trigger, 1 probe (per run, e.g. g0, t0, imec0)
> - requires standard input folder format
> - only run one subject / run at a time
> - has limited preprocessing options (`tshift`, `bandpass_filter`, `common median reference`)
> - no options to remove potentially large intermediate files
> - installation / running on HPC is a bit clunky. In future this can be simplified with SLURM jobs organised under the hood and setting up a HPC module.
> - untested!
# Features
- preprocess SpikeGLX data (`tshift`, `bandpass_filter`, `common median reference`)
- spike sorting (`kilosort2`, `kilosort2_5`, `kilosort3`)
- quality check measures on the sorting results
# Local Installation
Sorting requires a NVIDIA GPU and so is currently only available using the SWC's High-Performance Computer (HPC). However, local installation is useful for visualising the preprocessing steps prior to running the full pipeline (see 'Visualisation' below).
To install locally, clone the repository to your local machine using git.
`git clone git@github.com:neuroinformatics-unit/swc_ephys.git`
Change directory to the repo and install using
`pip install -e .`
or, to also install developer dependencies
`pip install -e .[dev]`
or if using the zsh shell
`pip install -e ".[dev]"`
After installation, the module can be imported with `import swc_ephys`.
## Running on the HPC
Currently, sorting is required to run on the SWC HPC with access to `/ceph/neuroinformatics`.
To connect and run on the HPC (e.g. from Windows, macOS or Linux terminal):
`ssh username@ssh.swc.ucl.ac.uk`
`ssh hpc-gw1`
The first time using, it is necessary to steup and install `swc_ephys`. It is strongly recommended to make a new conda environment on the HPC, before installing `swc_ephys`.
`module load miniconda`
`conda create --name swc_ephys python=3.10`
`conda activate swc_ephys`
and install swc_ephys and it's dependencies:
`mkdir ~/git-repos`
`cd ~/git-repos`
`git clone https://github.com/JoeZiminski/swc_ephys.git`
`cd swc_ephys`
`pip install -e .`
Before running, it is necessary to request use of a GPU node on the HPC to run spike sorting with KiloSort. To run preprocessing and spike sorting, create a script using the API or call from the command line interface (instructions below).
`srun -p gpu --gres=gpu:1 -n 8 --mem=40GB --pty bash -i`
`module load cuda`
`module load miniconda`
`conda activate swc_ephys`
`python my_pipeline_script.py`
# Quick Start Guide
SWC Ephys (currently) expects input data to be stored in a `rawdata` folder. A subject (e.g. mouse) data should be stored in the `rawdata` folder and contain SpikeGLX output format (example below). **Currently, only recordings with 1 gate, 1 trigger and 1 probe are supported (i.e. index 0 for all gate, trigger probe, `g0`, `t0` and `imec0`)**.
```
└── rawdata/
└── 1110925/
└── 1110925_test_shank1_g0/
└── 1110925_test_shank1_g0_imec0/
├── 1110925_test_shank1_g0_t0.imec0.ap.bin
└── 1110925_test_shank1_g0_t0.imec0.ap.meta
```
## API (script)
Example code to analyse this data in this format is below:
```
from swc_ephys.pipeline.full_pipeline import run_full_pipeline
base_path = "/ceph/neuroinformatics/neuroinformatics/scratch/ece_ephys_learning"
if __name__ == "__main__":
run_full_pipeline(
base_path=base_path,
sub_name="1110925",
run_name="1110925_test_shank1",
config_name="test",
sorter="kilosort2_5",
)
```
`base_path` is the path containing the required `rawdata` folder.
`sub_name` is the subject to run, and `run_name` is the SpikeGLX run name to run.
`configs_name` contains the name of the preprocessing / sorting settings to use (see below)
`sorter` is the name of the sorter to use (currently supported is `kilosort2`, `kilosort2_5` and `kilosort3`)
Note `run_full_pipline` must be run in the `if __name__ == "__main__"` block as it uses the `multiprocessing` module.
## Output
Output of spike sorting will be in a `derivatives` folder at the same level as the `rawdata`. The subfolder organisation of `derivatives` will match `rawdata`.
Output are the saved preprocessed data, spike sorting results as well as a list of [quality check measures](https://spikeinterface.readthedocs.io/en/latest/modules/qualitymetrics.html). For example, the full output of a sorting run with the input data as above is:
```
├── rawdata/
│ └── ...
└── derivatives/
└── 1110925/
└── 1110925_test_shank1_g0 /
└── 1110925_test_shank1_g0_imec0/
├── preprocessed/
│ ├── data_class.pkl
│ └── si_recording
├── kilosort2_5-sorting/
├── in_container_sorting/
├── sorter_output/
├── waveforms/
│ └── <spikeinterface waveforms output>
├── quality_metrics.csv
├── spikeinterface_log.json
├── spikeinterface_params.json
└── spikeinterface_recording.json
```
**preprocessed**:
- Binary-format spikeinterface recording from the final preprocessing step (`si_recording`) 2) `data_class.pkl` swc_ephys internal use.
**-sorting output (e.g. kilosort2_5-sorting, multiple sorters can be run)**:
- <u>in_container_sorting</u>: stored options used to run the sorter
- <u>sorter_output</u>: the full output of the sorter (e.g. kilosort .npy files)
- <u>waveforms</u>: spikeinterface [waveforms](https://spikeinterface.readthedocs.io/en/latest/modules/core.html#waveformextractor) output containing AP
waveforms for detected spikes
- <u>quality_metrics.csv</u>: output of spikeinterface [quality check measures](https://spikeinterface.readthedocs.io/en/latest/modules/qualitymetrics.html)
# Set Preprocessing Options
Currently supported are multiplexing correction or tshift (termed `phase shift` here), common median referencing (CMR) (termed `common_reference` here) and bandpass filtering (`bandpass_filter`). These options provide an interface to [SpikeInterface preprocessing](https://spikeinterface.readthedocs.io/en/0.13.0/modules/toolkit/plot_1_preprocessing.html) options, more will be added soon.
Preprocessing options are set in `yaml` configuration files stored in `sbi_ephys/sbi_ephys/configs/`. A default pipeline is stored in `test.yaml`.
Custom preprocessing configuration files may be passed to the `config_name` argument, by passing the full path to the `.yaml` configuration file. For example:
```
'preprocessing':
'1':
- phase_shift
- {}
'2':
- bandpass_filter
- freq_min: 300
freq_max: 6000
'3':
- common_reference
- operator: median
reference: global
'sorting':
'kilosort3':
'car': False
'freq_min': 300
```
Configuration files are structured as a dictionary where keys indicate the order to run preprocessing The values hold a list in which the first element is the name of the preprocessing step to run, and the second element a dictionary containing kwargs passed to the spikeinterface function.
# Visualise Preprocessing
Visualising preprocesing output can be run locally to inspect output of preprocessing routines. To visualise preprocessing outputs:
```
from swc_ephys.pipeline.preprocess import preprocess
from swc_ephys.pipeline.visualise import visualise
base_path = "/ceph/neuroinformatics/neuroinformatics/scratch/ece_ephys_learning"
sub_name = "1110925"
run_name = "1110925_test_shank1"
data = preprocess(base_path=base_path, sub_name=sub_name, run_name=run_name)
visualise(
data,
steps="all",
mode="map",
as_subplot=True,
channel_idx_to_show=np.arange(10, 50),
show_channel_ids=False,
time_range=(1, 2),
)
```
This will display a plot showing data from all preprocessing steps, displaying channels with idx 10 - 50, over time period 1-2. Note this requires a GUI (i.e. not run on the HPC terminal) and is best run locally.
Raw data
{
"_id": null,
"home_page": "",
"name": "swc-ephys",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8.0",
"maintainer_email": "",
"keywords": "",
"author": "",
"author_email": "Joe Ziminski <joseph.j.ziminski@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/6e/94/192ec2c67b275afa30efdbf0420aa6b95424554bfa80c5eef73719ffcb5e/swc_ephys-0.0.0.tar.gz",
"platform": null,
"description": "> **Warning** \r\n> **SWC Ephys is not sufficiently tested to be used in analysis. This release is only for testing. Do not use for your final analyses.**\r\n\r\n> **Warning** **Limitations**\r\n> - works only on SpikeGLX recordings with 1 gate, 1 trigger, 1 probe (per run, e.g. g0, t0, imec0)\r\n> - requires standard input folder format\r\n> - only run one subject / run at a time\r\n> - has limited preprocessing options (`tshift`, `bandpass_filter`, `common median reference`)\r\n> - no options to remove potentially large intermediate files\r\n> - installation / running on HPC is a bit clunky. In future this can be simplified with SLURM jobs organised under the hood and setting up a HPC module.\r\n> - untested!\r\n\r\n\r\n# Features\r\n\r\n- preprocess SpikeGLX data (`tshift`, `bandpass_filter`, `common median reference`)\r\n- spike sorting (`kilosort2`, `kilosort2_5`, `kilosort3`)\r\n- quality check measures on the sorting results\r\n\r\n# Local Installation\r\n\r\nSorting requires a NVIDIA GPU and so is currently only available using the SWC's High-Performance Computer (HPC). However, local installation is useful for visualising the preprocessing steps prior to running the full pipeline (see 'Visualisation' below).\r\n\r\nTo install locally, clone the repository to your local machine using git. \r\n\r\n`git clone git@github.com:neuroinformatics-unit/swc_ephys.git`\r\n\r\nChange directory to the repo and install using\r\n\r\n`pip install -e .`\r\n\r\nor, to also install developer dependencies\r\n\r\n`pip install -e .[dev]` \r\n\r\nor if using the zsh shell\r\n\r\n`pip install -e \".[dev]\"` \r\n\r\nAfter installation, the module can be imported with `import swc_ephys`.\r\n\r\n## Running on the HPC\r\n\r\nCurrently, sorting is required to run on the SWC HPC with access to `/ceph/neuroinformatics`. \r\n\r\nTo connect and run on the HPC (e.g. from Windows, macOS or Linux terminal):\r\n\r\n`ssh username@ssh.swc.ucl.ac.uk`\r\n\r\n`ssh hpc-gw1`\r\n\r\nThe first time using, it is necessary to steup and install `swc_ephys`. It is strongly recommended to make a new conda environment on the HPC, before installing `swc_ephys`.\r\n\r\n`module load miniconda`\r\n\r\n`conda create --name swc_ephys python=3.10`\r\n\r\n`conda activate swc_ephys`\r\n\r\nand install swc_ephys and it's dependencies:\r\n\r\n`mkdir ~/git-repos`\r\n\r\n`cd ~/git-repos`\r\n\r\n`git clone https://github.com/JoeZiminski/swc_ephys.git`\r\n\r\n`cd swc_ephys`\r\n\r\n`pip install -e .`\r\n\r\nBefore running, it is necessary to request use of a GPU node on the HPC to run spike sorting with KiloSort. To run preprocessing and spike sorting, create a script using the API or call from the command line interface (instructions below). \r\n\r\n`srun -p gpu --gres=gpu:1 -n 8 --mem=40GB --pty bash -i`\r\n\r\n`module load cuda`\r\n\r\n`module load miniconda`\r\n\r\n`conda activate swc_ephys`\r\n\r\n`python my_pipeline_script.py`\r\n\r\n# Quick Start Guide\r\n\r\nSWC Ephys (currently) expects input data to be stored in a `rawdata` folder. A subject (e.g. mouse) data should be stored in the `rawdata` folder and contain SpikeGLX output format (example below). **Currently, only recordings with 1 gate, 1 trigger and 1 probe are supported (i.e. index 0 for all gate, trigger probe, `g0`, `t0` and `imec0`)**.\r\n\r\n```\r\n\u2514\u2500\u2500 rawdata/\r\n \u2514\u2500\u2500 1110925/\r\n \u2514\u2500\u2500 1110925_test_shank1_g0/\r\n \u2514\u2500\u2500 1110925_test_shank1_g0_imec0/\r\n \u251c\u2500\u2500 1110925_test_shank1_g0_t0.imec0.ap.bin\r\n \u2514\u2500\u2500 1110925_test_shank1_g0_t0.imec0.ap.meta\r\n```\r\n\r\n\r\n## API (script)\r\n\r\nExample code to analyse this data in this format is below:\r\n\r\n```\r\nfrom swc_ephys.pipeline.full_pipeline import run_full_pipeline\r\n\r\nbase_path = \"/ceph/neuroinformatics/neuroinformatics/scratch/ece_ephys_learning\"\r\n\r\nif __name__ == \"__main__\":\r\n\r\n run_full_pipeline(\r\n base_path=base_path,\r\n sub_name=\"1110925\",\r\n run_name=\"1110925_test_shank1\",\r\n config_name=\"test\",\r\n sorter=\"kilosort2_5\",\r\n )\r\n```\r\n\r\n`base_path` is the path containing the required `rawdata` folder. \r\n\r\n`sub_name` is the subject to run, and `run_name` is the SpikeGLX run name to run. \r\n\r\n`configs_name` contains the name of the preprocessing / sorting settings to use (see below)\r\n\r\n`sorter` is the name of the sorter to use (currently supported is `kilosort2`, `kilosort2_5` and `kilosort3`)\r\n\r\nNote `run_full_pipline` must be run in the `if __name__ == \"__main__\"` block as it uses the `multiprocessing` module.\r\n\r\n## Output\r\n\r\nOutput of spike sorting will be in a `derivatives` folder at the same level as the `rawdata`. The subfolder organisation of `derivatives` will match `rawdata`. \r\n\r\nOutput are the saved preprocessed data, spike sorting results as well as a list of [quality check measures](https://spikeinterface.readthedocs.io/en/latest/modules/qualitymetrics.html). For example, the full output of a sorting run with the input data as above is:\r\n\r\n```\r\n\u251c\u2500\u2500 rawdata/\r\n\u2502 \u2514\u2500\u2500 ...\r\n\u2514\u2500\u2500 derivatives/\r\n \u2514\u2500\u2500 1110925/\r\n \u2514\u2500\u2500 1110925_test_shank1_g0 /\r\n \u2514\u2500\u2500 1110925_test_shank1_g0_imec0/\r\n \u251c\u2500\u2500 preprocessed/\r\n \u2502 \u251c\u2500\u2500 data_class.pkl\r\n \u2502 \u2514\u2500\u2500 si_recording\r\n \u251c\u2500\u2500 kilosort2_5-sorting/\r\n \u251c\u2500\u2500 in_container_sorting/\r\n \u251c\u2500\u2500 sorter_output/\r\n \u251c\u2500\u2500 waveforms/\r\n \u2502 \u2514\u2500\u2500 <spikeinterface waveforms output>\r\n \u251c\u2500\u2500 quality_metrics.csv\r\n \u251c\u2500\u2500 spikeinterface_log.json\r\n \u251c\u2500\u2500 spikeinterface_params.json\r\n \u2514\u2500\u2500 spikeinterface_recording.json\r\n```\r\n\r\n\r\n**preprocessed**: \r\n\r\n- Binary-format spikeinterface recording from the final preprocessing step (`si_recording`) 2) `data_class.pkl` swc_ephys internal use.\r\n\r\n**-sorting output (e.g. kilosort2_5-sorting, multiple sorters can be run)**: \r\n\r\n- <u>in_container_sorting</u>: stored options used to run the sorter\r\n\r\n- <u>sorter_output</u>: the full output of the sorter (e.g. kilosort .npy files)\r\n\r\n- <u>waveforms</u>: spikeinterface [waveforms](https://spikeinterface.readthedocs.io/en/latest/modules/core.html#waveformextractor) output containing AP \r\nwaveforms for detected spikes\r\n\r\n- <u>quality_metrics.csv</u>: output of spikeinterface [quality check measures](https://spikeinterface.readthedocs.io/en/latest/modules/qualitymetrics.html)\r\n\r\n# Set Preprocessing Options\r\n\r\nCurrently supported are multiplexing correction or tshift (termed `phase shift` here), common median referencing (CMR) (termed `common_reference` here) and bandpass filtering (`bandpass_filter`). These options provide an interface to [SpikeInterface preprocessing](https://spikeinterface.readthedocs.io/en/0.13.0/modules/toolkit/plot_1_preprocessing.html) options, more will be added soon.\r\n\r\nPreprocessing options are set in `yaml` configuration files stored in `sbi_ephys/sbi_ephys/configs/`. A default pipeline is stored in `test.yaml`.\r\n\r\nCustom preprocessing configuration files may be passed to the `config_name` argument, by passing the full path to the `.yaml` configuration file. For example:\r\n\r\n```\r\n'preprocessing':\r\n '1':\r\n - phase_shift\r\n - {}\r\n '2':\r\n - bandpass_filter\r\n - freq_min: 300\r\n freq_max: 6000\r\n '3':\r\n - common_reference\r\n - operator: median\r\n reference: global\r\n\r\n'sorting':\r\n 'kilosort3':\r\n 'car': False\r\n 'freq_min': 300\r\n```\r\n\r\nConfiguration files are structured as a dictionary where keys indicate the order to run preprocessing The values hold a list in which the first element is the name of the preprocessing step to run, and the second element a dictionary containing kwargs passed to the spikeinterface function.\r\n\r\n# Visualise Preprocessing\r\n\r\nVisualising preprocesing output can be run locally to inspect output of preprocessing routines. To visualise preprocessing outputs:\r\n\r\n```\r\nfrom swc_ephys.pipeline.preprocess import preprocess\r\nfrom swc_ephys.pipeline.visualise import visualise\r\n\r\nbase_path = \"/ceph/neuroinformatics/neuroinformatics/scratch/ece_ephys_learning\"\r\nsub_name = \"1110925\"\r\nrun_name = \"1110925_test_shank1\"\r\n\r\ndata = preprocess(base_path=base_path, sub_name=sub_name, run_name=run_name)\r\n\r\nvisualise(\r\n data,\r\n steps=\"all\",\r\n mode=\"map\",\r\n as_subplot=True,\r\n channel_idx_to_show=np.arange(10, 50),\r\n show_channel_ids=False,\r\n time_range=(1, 2),\r\n)\r\n```\r\n\r\nThis will display a plot showing data from all preprocessing steps, displaying channels with idx 10 - 50, over time period 1-2. Note this requires a GUI (i.e. not run on the HPC terminal) and is best run locally.\r\n\r\n\r\n",
"bugtrack_url": null,
"license": "BSD-3-Clause",
"summary": "Manage extracellular electrophysiology analysis",
"version": "0.0.0",
"project_urls": {
"bug_tracker": "https://github.com/JoeZiminski/swc_ephys/issues",
"documentation": "https://github.com/JoeZiminski/swc_ephys",
"homepage": "https://github.com/JoeZiminski/swc_ephys",
"source_code": "https://github.com/JoeZiminski/swc_ephys",
"user_support": "https://github.com/JoeZiminski/swc_ephys/issues"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "992ced18440c567d28de6178f778ce478f2b7f504c0f1f12d7b62c03d9397c0c",
"md5": "6838551434bd4a05f0ee94ece089e3a8",
"sha256": "4383e2454a36d1a30e9e9907eb242fc2cad9462f3e7e837976a1a3bee14dbb3f"
},
"downloads": -1,
"filename": "swc_ephys-0.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6838551434bd4a05f0ee94ece089e3a8",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8.0",
"size": 45134,
"upload_time": "2023-08-03T19:35:18",
"upload_time_iso_8601": "2023-08-03T19:35:18.363827Z",
"url": "https://files.pythonhosted.org/packages/99/2c/ed18440c567d28de6178f778ce478f2b7f504c0f1f12d7b62c03d9397c0c/swc_ephys-0.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6e94192ec2c67b275afa30efdbf0420aa6b95424554bfa80c5eef73719ffcb5e",
"md5": "3fc47cb1ce17be5e25bf8bf095430b9c",
"sha256": "705836ab285cdefc69cf1b43e198ee8aa4495242f0da29b1e2dbce45c1a92487"
},
"downloads": -1,
"filename": "swc_ephys-0.0.0.tar.gz",
"has_sig": false,
"md5_digest": "3fc47cb1ce17be5e25bf8bf095430b9c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8.0",
"size": 1658011,
"upload_time": "2023-08-03T19:35:20",
"upload_time_iso_8601": "2023-08-03T19:35:20.966710Z",
"url": "https://files.pythonhosted.org/packages/6e/94/192ec2c67b275afa30efdbf0420aa6b95424554bfa80c5eef73719ffcb5e/swc_ephys-0.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-03 19:35:20",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "JoeZiminski",
"github_project": "swc_ephys",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "swc-ephys"
}
