| Name | cryoemservices JSON |
| Version |
1.0.2
JSON |
| download |
| home_page | None |
| Summary | Services for CryoEM processing |
| upload_time | 2025-01-16 15:06:42 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.10 |
| license | BSD 3-Clause License Copyright (c) 2023, Diamond Light Source Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| keywords |
cryoem-services
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# cryoem-services
[](https://pypi.python.org/pypi/cryoemservices)
[](https://pypi.python.org/pypi/cryoemservices)
[](https://codecov.io/gh/DiamondLightSource/cryoem-services)
[](https://github.com/ambv/black)
Services and configuration for cryo-EM pipelines.
This package consists of a number of services to process cryo-EM micrographs,
both for single particle analysis and tomography,
using a range of commonly used cryo-EM processing software.
These services can be run independently to process data,
or as part of a wider structure for performing live analysis during microscope collection.
For live analysis, this package integrates with a package
for transferring and monitoring collected data,
[Murfey](https://github.com/DiamondLightSource/python-murfey),
and a database for storing processing outcomes,
[ISPyB](https://github.com/DiamondLightSource/ispyb-database).
To run these services the software executables being called must be installed.
These do not come with this package.
# Tomography processing
The tomography processing pipeline consists of:
- Motion correction
- CTF estimation
- Tomogram alignment
- Tomogram denoising using [Topaz](http://topaz-em.readthedocs.io)
- Segmentation using [membrain-seg](https://github.com/teamtomo/membrain-seg)
The results of this processing can be opened and continued using
[Relion 5.0](https://relion.readthedocs.io).
# Single particle analysis
The single particle analysis pipeline produces a project
that can be opened and continued using
[CCP-EM doppio](https://www.ccpem.ac.uk/docs/doppio/user_guide.html)
or [Relion](https://relion.readthedocs.io).
The processing pipeline consists of:
- Motion correction
- CTF estimation
- Particle picking
- (Optionally) Ice thickness estimation
- Particle extraction and rebatching
- 2D classification using Relion
- Automated 2D class selection using Relion
- 3D classification using Relion
- 3D Refinement and post-processing
- BFactor estimation by refinement with varying particle count
# Services currently available
The following services are provided for running the pipelines:
- Utility services:
- **ClusterSubmission**: Submits wrapper scripts to an HPC cluster
- **EMISPyB**: Inserts results into an ISPyB database
- **Images**: Creates thumbnail images for viewing processing outcomes
- **NodeCreator**: Creates Relion project files for the services run
- **ProcessRecipe**: Converts recipes into messages suitable for processing services
- Processing services:
- **BFactor**: Performs the setup for 3D refinement with varying particle count
- **CrYOLO**: Particle picking on micrographs using [crYOLO](https://cryolo.readthedocs.io)
- **CTFFind**: CTF estimation on micrographs using [CTFFIND4](https://grigoriefflab.umassmed.edu/ctffind4)
- **DenoiseSlurm**: Tomogram denoising, submitted to a slurm HPC cluster, using [Topaz](http://topaz-em.readthedocs.io)
- **Extract**: Extracts picked particles from micrographs
- **ExtractClass**: Extracts particles from a given 3D class
- **IceBreaker**: Ice thickness estimation with [IceBreaker](https://github.com/DiamondLightSource/python-icebreaker)
- **MembrainSeg**: Tomogram segmentation, submitted to a slurm HPC cluster, using [membrain-seg](https://github.com/teamtomo/membrain-seg)
- **MotionCorr**: Motion correction of micrographs using [MotionCor2](http://emcore.ucsf.edu/ucsf-software) or [Relion](https://relion.readthedocs.io), optionally submitted to a slurm HPC cluster
- **PostProcess**: Post-processing of 3D refinements using [Relion](https://relion.readthedocs.io)
- **SelectClasses**: Runs automated 2D class selection using [Relion](https://relion.readthedocs.io) and re-batches the particles from these classes
- **SelectParticles**: Creates files listing batches of extracted particles
- **TomoAlign**: Tomogram reconstruction from a list of micrographs using [imod](https://bio3d.colorado.edu/imod) and [AreTomo2](https://github.com/czimaginginstitute/AreTomo2)
- **TomoAlignSlurm**: Tomogram alignment processing submitted to a slurm HPC cluster
There are also three wrapper scripts that can be run on an HPC cluster using the ClusterSubmission service.
These perform 2D classification, 3D classification and 3D refinement
using [Relion](https://relion.readthedocs.io).
# Running services
The services in this package are run using
[zocalo](https://github.com/DiamondLightSource/python-zocalo)
and [python-workflows](https://github.com/DiamondLightSource/python-workflows).
They consume messages off a [RabbitMQ](https://www.rabbitmq.com/)
instance and processing happens in sequences defined by the recipes in the `recipes` folder.
To start a service run the `cryoemservices.service` command and specify the service name.
For example, to start a motion correction service:
```bash
$ cryoemservices.service -s MotionCorr -c config_file.yaml
```
The configuration file should contain the following:
```yaml
rabbitmq_credentials: <file with connection credentials to rabbitmq>
recipe_directory: <directory containing the recipes to run>
ispyb_credentials: <(Optionally) file with credentials for an ispyb database>
slurm_credentials:
default: <(Optionally) credentials for a slurm RestAPI>
graylog_host: <(Optionally) the name of a graylog instance>
graylog_port: <(Optionally) the port used by graylog>
```
Once started, these services will initialise and then wait for messages to be sent to them.
Messages are sent through a message broker,
currently [RabbitMQ](http://www.rabbitmq.com) is supported using pika transport in `python-workflows`.
Individual processing stages can be run by sending a dictionary of the parameters,
but the processing pipelines are designed to run through recipes.
A recipe is a specication of a series of steps to carry out,
and how these steps interact with each other.
Recipes for the current processing pipelines are provided in the `recipes` folder.
To run a recipe in python a dictionary needs to be provided consisting of
the recipe name and the parameters expected by the recipe.
The following snippet shows an example of the setup needed.
This will send a message to a running **ProcessRecipe** service which
prepares the recipe for the processing services.
```python
import workflows.transport.pika_transport as pt
example_message = {
"recipes": ["em-tomo-align"],
"parameters": {
"path_pattern": "micrograph_*.mrc",
"pix_size": "1",
...
},
}
transport = pt.PikaTransport()
transport.connect()
transport.send("processing_recipe", example_message)
```
Raw data
{
"_id": null,
"home_page": null,
"name": "cryoemservices",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "cryoem-services",
"author": null,
"author_email": "\"Diamond Light Source - Data Analysis et al.\" <dataanalysis@diamond.ac.uk>",
"download_url": "https://files.pythonhosted.org/packages/51/1d/5089c697b71ebda11898e7c02b60650b068cfdc7f8345d1495359eca9f00/cryoemservices-1.0.2.tar.gz",
"platform": null,
"description": "# cryoem-services\n\n[](https://pypi.python.org/pypi/cryoemservices)\n[](https://pypi.python.org/pypi/cryoemservices)\n[](https://codecov.io/gh/DiamondLightSource/cryoem-services)\n[](https://github.com/ambv/black)\n\nServices and configuration for cryo-EM pipelines.\n\nThis package consists of a number of services to process cryo-EM micrographs,\nboth for single particle analysis and tomography,\nusing a range of commonly used cryo-EM processing software.\nThese services can be run independently to process data,\nor as part of a wider structure for performing live analysis during microscope collection.\nFor live analysis, this package integrates with a package\nfor transferring and monitoring collected data,\n[Murfey](https://github.com/DiamondLightSource/python-murfey),\nand a database for storing processing outcomes,\n[ISPyB](https://github.com/DiamondLightSource/ispyb-database).\n\nTo run these services the software executables being called must be installed.\nThese do not come with this package.\n\n# Tomography processing\n\nThe tomography processing pipeline consists of:\n\n- Motion correction\n- CTF estimation\n- Tomogram alignment\n- Tomogram denoising using [Topaz](http://topaz-em.readthedocs.io)\n- Segmentation using [membrain-seg](https://github.com/teamtomo/membrain-seg)\n\nThe results of this processing can be opened and continued using\n[Relion 5.0](https://relion.readthedocs.io).\n\n# Single particle analysis\n\nThe single particle analysis pipeline produces a project\nthat can be opened and continued using\n[CCP-EM doppio](https://www.ccpem.ac.uk/docs/doppio/user_guide.html)\nor [Relion](https://relion.readthedocs.io).\n\nThe processing pipeline consists of:\n\n- Motion correction\n- CTF estimation\n- Particle picking\n- (Optionally) Ice thickness estimation\n- Particle extraction and rebatching\n- 2D classification using Relion\n- Automated 2D class selection using Relion\n- 3D classification using Relion\n- 3D Refinement and post-processing\n- BFactor estimation by refinement with varying particle count\n\n# Services currently available\n\nThe following services are provided for running the pipelines:\n\n- Utility services:\n - **ClusterSubmission**: Submits wrapper scripts to an HPC cluster\n - **EMISPyB**: Inserts results into an ISPyB database\n - **Images**: Creates thumbnail images for viewing processing outcomes\n - **NodeCreator**: Creates Relion project files for the services run\n - **ProcessRecipe**: Converts recipes into messages suitable for processing services\n- Processing services:\n - **BFactor**: Performs the setup for 3D refinement with varying particle count\n - **CrYOLO**: Particle picking on micrographs using [crYOLO](https://cryolo.readthedocs.io)\n - **CTFFind**: CTF estimation on micrographs using [CTFFIND4](https://grigoriefflab.umassmed.edu/ctffind4)\n - **DenoiseSlurm**: Tomogram denoising, submitted to a slurm HPC cluster, using [Topaz](http://topaz-em.readthedocs.io)\n - **Extract**: Extracts picked particles from micrographs\n - **ExtractClass**: Extracts particles from a given 3D class\n - **IceBreaker**: Ice thickness estimation with [IceBreaker](https://github.com/DiamondLightSource/python-icebreaker)\n - **MembrainSeg**: Tomogram segmentation, submitted to a slurm HPC cluster, using [membrain-seg](https://github.com/teamtomo/membrain-seg)\n - **MotionCorr**: Motion correction of micrographs using [MotionCor2](http://emcore.ucsf.edu/ucsf-software) or [Relion](https://relion.readthedocs.io), optionally submitted to a slurm HPC cluster\n - **PostProcess**: Post-processing of 3D refinements using [Relion](https://relion.readthedocs.io)\n - **SelectClasses**: Runs automated 2D class selection using [Relion](https://relion.readthedocs.io) and re-batches the particles from these classes\n - **SelectParticles**: Creates files listing batches of extracted particles\n - **TomoAlign**: Tomogram reconstruction from a list of micrographs using [imod](https://bio3d.colorado.edu/imod) and [AreTomo2](https://github.com/czimaginginstitute/AreTomo2)\n - **TomoAlignSlurm**: Tomogram alignment processing submitted to a slurm HPC cluster\n\nThere are also three wrapper scripts that can be run on an HPC cluster using the ClusterSubmission service.\nThese perform 2D classification, 3D classification and 3D refinement\nusing [Relion](https://relion.readthedocs.io).\n\n# Running services\n\nThe services in this package are run using\n[zocalo](https://github.com/DiamondLightSource/python-zocalo)\nand [python-workflows](https://github.com/DiamondLightSource/python-workflows).\nThey consume messages off a [RabbitMQ](https://www.rabbitmq.com/)\ninstance and processing happens in sequences defined by the recipes in the `recipes` folder.\nTo start a service run the `cryoemservices.service` command and specify the service name.\nFor example, to start a motion correction service:\n\n```bash\n$ cryoemservices.service -s MotionCorr -c config_file.yaml\n```\n\nThe configuration file should contain the following:\n\n```yaml\nrabbitmq_credentials: <file with connection credentials to rabbitmq>\nrecipe_directory: <directory containing the recipes to run>\nispyb_credentials: <(Optionally) file with credentials for an ispyb database>\nslurm_credentials:\n default: <(Optionally) credentials for a slurm RestAPI>\ngraylog_host: <(Optionally) the name of a graylog instance>\ngraylog_port: <(Optionally) the port used by graylog>\n```\n\nOnce started, these services will initialise and then wait for messages to be sent to them.\nMessages are sent through a message broker,\ncurrently [RabbitMQ](http://www.rabbitmq.com) is supported using pika transport in `python-workflows`.\nIndividual processing stages can be run by sending a dictionary of the parameters,\nbut the processing pipelines are designed to run through recipes.\n\nA recipe is a specication of a series of steps to carry out,\nand how these steps interact with each other.\nRecipes for the current processing pipelines are provided in the `recipes` folder.\n\nTo run a recipe in python a dictionary needs to be provided consisting of\nthe recipe name and the parameters expected by the recipe.\nThe following snippet shows an example of the setup needed.\nThis will send a message to a running **ProcessRecipe** service which\nprepares the recipe for the processing services.\n\n```python\nimport workflows.transport.pika_transport as pt\n\nexample_message = {\n \"recipes\": [\"em-tomo-align\"],\n \"parameters\": {\n \"path_pattern\": \"micrograph_*.mrc\",\n \"pix_size\": \"1\",\n ...\n },\n}\n\ntransport = pt.PikaTransport()\ntransport.connect()\ntransport.send(\"processing_recipe\", example_message)\n```\n",
"bugtrack_url": null,
"license": "BSD 3-Clause License Copyright (c) 2023, Diamond Light Source Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ",
"summary": "Services for CryoEM processing",
"version": "1.0.2",
"project_urls": {
"Bug-Tracker": "https://github.com/DiamondLightSource/cryoem-services/issues",
"GitHub": "https://github.com/DiamondLightSource/cryoem-services"
},
"split_keywords": [
"cryoem-services"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "02564b543631b1655ba83e78946099faf4ffa4dea68d4c7bb481c1b955be9e93",
"md5": "9e9f5c174073539542b0b7236facd6b9",
"sha256": "8646e692459fddc9b539745cd6d9162ca146b4e722cf90a0b6ca36153d85dd05"
},
"downloads": -1,
"filename": "cryoemservices-1.0.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "9e9f5c174073539542b0b7236facd6b9",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 161121,
"upload_time": "2025-01-16T15:06:40",
"upload_time_iso_8601": "2025-01-16T15:06:40.217100Z",
"url": "https://files.pythonhosted.org/packages/02/56/4b543631b1655ba83e78946099faf4ffa4dea68d4c7bb481c1b955be9e93/cryoemservices-1.0.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "511d5089c697b71ebda11898e7c02b60650b068cfdc7f8345d1495359eca9f00",
"md5": "3152fa206491c5452f8e5c37832fbbd2",
"sha256": "ea4389670af39a1b9b627b544a3bfde8f977b75d9799b404c4f23d44620a2f04"
},
"downloads": -1,
"filename": "cryoemservices-1.0.2.tar.gz",
"has_sig": false,
"md5_digest": "3152fa206491c5452f8e5c37832fbbd2",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 124586,
"upload_time": "2025-01-16T15:06:42",
"upload_time_iso_8601": "2025-01-16T15:06:42.467212Z",
"url": "https://files.pythonhosted.org/packages/51/1d/5089c697b71ebda11898e7c02b60650b068cfdc7f8345d1495359eca9f00/cryoemservices-1.0.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-16 15:06:42",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "DiamondLightSource",
"github_project": "cryoem-services",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "cryoemservices"
}
