# cryoem-services
Services and configuration for cryo-EM pipelines.
This package consists of a number of services to process cryo-EM micrographs,
both for single particle analysis and tomography,
using a range of commonly used cryo-EM processing software.
These services can be run independently to process data,
or as part of a wider structure for performing live analysis during microscope collection.
For live analysis, this package integrates with a package
for transferring and monitoring collected data,
[Murfey](https://github.com/DiamondLightSource/python-murfey),
and a database for storing processing outcomes,
[ISPyB](https://github.com/DiamondLightSource/ispyb-database).
To run these services the software executables being called must be installed.
These do not come with this package.
# Tomography processing
The tomography processing pipeline consists of:
- Motion correction
- CTF estimation
- Tomogram alignment
- Tomogram denoising using [Topaz](http://topaz-em.readthedocs.io)
# Single particle analysis
The single particle analysis pipeline produces a project
that can be opened and continued using
[CCP-EM doppio](https://www.ccpem.ac.uk/docs/doppio/user_guide.html)
or [Relion](https://relion.readthedocs.io).
The processing pipeline consists of:
- Motion correction
- CTF estimation
- Particle picking
- (Optionally) Ice thickness estimation
- Particle extraction and rebatching
- 2D classification using Relion
- Automated 2D class selection using Relion
- 3D classification using Relion
# Services currently available
The following services are provided for running the pipelines:
- Utility services:
- **ClusterSubmission**: Submits zocalo wrappers to an HPC cluster
- **Dispatcher**: Converts recipes into messages suitable for processing services
- **Images**: Creates thumbnail images for viewing processing outcomes
- **ISPyB**: Inserts results into an ISPyB database
- **NodeCreator**: Creates Relion project files for the services run
- Processing services:
- **CrYOLO**: Particle picking on micrographs using
[crYOLO](https://cryolo.readthedocs.io)
- **CTFFind**: CTF estimation on micrographs using
[CTFFIND4](https://grigoriefflab.umassmed.edu/ctffind4)
- **Extract**: Extracts picked particles from micrographs
- **IceBreaker**: Ice thickness estimation with
[IceBreaker](https://github.com/DiamondLightSource/python-icebreaker)
- **MotionCorr**: Motion correction of micrographs using
[MotionCor2](http://emcore.ucsf.edu/ucsf-software)
or [Relion](https://relion.readthedocs.io)
- **MotionCorrSlurm**: MotionCor2 processing submitted to a slurm HPC cluster
- **SelectClasses**: Runs automated 2D class selection using
[Relion](https://relion.readthedocs.io) and re-batches the particles from these classes
- **SelectParticles**: Creates files listing batches of extracted particles
- **TomoAlign**: Tomogram reconstruction from a list of micrographs using
[imod](https://bio3d.colorado.edu/imod) and [AreTomo](http://msg.ucsf.edu/software)
There are also two zocalo wrapper scripts that can be run on an HPC cluster.
These perform 2D and 3D classification using [Relion](https://relion.readthedocs.io).
# Running services using zocalo
The services in this package are run using
[zocalo](https://github.com/DiamondLightSource/python-zocalo)
and [python-workflows](https://github.com/DiamondLightSource/python-workflows).
To start a service run the `zocalo.service` command and specify the service name.
For example, to start a motion correction service:
```bash
$ zocalo.service -s MotionCorr
```
Once started, these services will initialise and then wait for messages to be sent to them.
Messages are sent through a message broker,
currently [RabbitMQ](http://www.rabbitmq.com) is supported using pika transport in `python-workflows`.
Individual processing stages can be run by sending a dictionary of the parameters,
but the processing pipelines are designed to run through recipes.
A recipe is a specication of a series of steps to carry out,
and how these steps interact with each other.
Recipes for the current processing pipelines are provided in the `recipes` folder.
To run a recipe in python a dictionary needs to be provided consisting of
the recipe name and the parameters expected by the recipe.
The following snippet shows an example of the setup needed.
This will send a message to a running **Dispatcher** service which
prepares the recipe for the processing services.
```python
import workflows.transport.pika_transport as pt
example_message = {
"recipes": ["em-tomo-align"],
"parameters": {
"path_pattern": "micrograph_*.mrc",
"pix_size": "1",
...
},
}
transport = pt.PikaTransport()
transport.connect()
transport.send("processing_recipe", example_message)
```
Raw data
{
"_id": null,
"home_page": null,
"name": "cryoemservices",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "cryoem-services",
"author": "Diamond Light Source - Data Analysis et al.",
"author_email": "dataanalysis@diamond.ac.uk",
"download_url": "https://files.pythonhosted.org/packages/b3/06/0ccf66d36ca967bfdddada2a2b4c0ca61b7119b0dec11a1a905f4c44d4c9/cryoemservices-0.2.0.tar.gz",
"platform": null,
"description": "# cryoem-services\nServices and configuration for cryo-EM pipelines.\n\nThis package consists of a number of services to process cryo-EM micrographs,\nboth for single particle analysis and tomography,\nusing a range of commonly used cryo-EM processing software.\nThese services can be run independently to process data,\nor as part of a wider structure for performing live analysis during microscope collection.\nFor live analysis, this package integrates with a package\nfor transferring and monitoring collected data,\n[Murfey](https://github.com/DiamondLightSource/python-murfey),\nand a database for storing processing outcomes,\n[ISPyB](https://github.com/DiamondLightSource/ispyb-database).\n\nTo run these services the software executables being called must be installed.\nThese do not come with this package.\n\n\n# Tomography processing\n\nThe tomography processing pipeline consists of:\n- Motion correction\n- CTF estimation\n- Tomogram alignment\n- Tomogram denoising using [Topaz](http://topaz-em.readthedocs.io)\n\n\n# Single particle analysis\n\nThe single particle analysis pipeline produces a project\nthat can be opened and continued using\n[CCP-EM doppio](https://www.ccpem.ac.uk/docs/doppio/user_guide.html)\nor [Relion](https://relion.readthedocs.io).\n\nThe processing pipeline consists of:\n- Motion correction\n- CTF estimation\n- Particle picking\n- (Optionally) Ice thickness estimation \n- Particle extraction and rebatching\n- 2D classification using Relion\n- Automated 2D class selection using Relion\n- 3D classification using Relion\n\n\n# Services currently available\n\nThe following services are provided for running the pipelines:\n- Utility services:\n - **ClusterSubmission**: Submits zocalo wrappers to an HPC cluster\n - **Dispatcher**: Converts recipes into messages suitable for processing services\n - **Images**: Creates thumbnail images for viewing processing outcomes\n - **ISPyB**: Inserts results into an ISPyB database\n - **NodeCreator**: Creates Relion project files for the services run\n- Processing services:\n - **CrYOLO**: Particle picking on micrographs using \n[crYOLO](https://cryolo.readthedocs.io)\n - **CTFFind**: CTF estimation on micrographs using \n[CTFFIND4](https://grigoriefflab.umassmed.edu/ctffind4)\n - **Extract**: Extracts picked particles from micrographs\n - **IceBreaker**: Ice thickness estimation with \n[IceBreaker](https://github.com/DiamondLightSource/python-icebreaker)\n - **MotionCorr**: Motion correction of micrographs using \n[MotionCor2](http://emcore.ucsf.edu/ucsf-software) \nor [Relion](https://relion.readthedocs.io)\n - **MotionCorrSlurm**: MotionCor2 processing submitted to a slurm HPC cluster\n - **SelectClasses**: Runs automated 2D class selection using \n[Relion](https://relion.readthedocs.io) and re-batches the particles from these classes\n - **SelectParticles**: Creates files listing batches of extracted particles\n - **TomoAlign**: Tomogram reconstruction from a list of micrographs using \n[imod](https://bio3d.colorado.edu/imod) and [AreTomo](http://msg.ucsf.edu/software)\n\nThere are also two zocalo wrapper scripts that can be run on an HPC cluster.\nThese perform 2D and 3D classification using [Relion](https://relion.readthedocs.io).\n\n\n# Running services using zocalo\nThe services in this package are run using\n[zocalo](https://github.com/DiamondLightSource/python-zocalo)\nand [python-workflows](https://github.com/DiamondLightSource/python-workflows).\nTo start a service run the `zocalo.service` command and specify the service name.\nFor example, to start a motion correction service:\n\n```bash\n$ zocalo.service -s MotionCorr\n```\n\nOnce started, these services will initialise and then wait for messages to be sent to them.\nMessages are sent through a message broker,\ncurrently [RabbitMQ](http://www.rabbitmq.com) is supported using pika transport in `python-workflows`.\nIndividual processing stages can be run by sending a dictionary of the parameters,\nbut the processing pipelines are designed to run through recipes.\n\nA recipe is a specication of a series of steps to carry out,\nand how these steps interact with each other.\nRecipes for the current processing pipelines are provided in the `recipes` folder.\n\nTo run a recipe in python a dictionary needs to be provided consisting of\nthe recipe name and the parameters expected by the recipe.\nThe following snippet shows an example of the setup needed.\nThis will send a message to a running **Dispatcher** service which\nprepares the recipe for the processing services.\n\n```python\nimport workflows.transport.pika_transport as pt\n\nexample_message = {\n \"recipes\": [\"em-tomo-align\"],\n \"parameters\": {\n \"path_pattern\": \"micrograph_*.mrc\",\n \"pix_size\": \"1\",\n ...\n },\n}\n\ntransport = pt.PikaTransport()\ntransport.connect()\ntransport.send(\"processing_recipe\", example_message)\n```\n\n",
"bugtrack_url": null,
"license": "BSD 3-Clause",
"summary": "Services for CryoEM processing",
"version": "0.2.0",
"project_urls": {
"Bug-Tracker": "https://github.com/DiamondLightSource/cryoem-services/issues",
"GitHub": "https://github.com/DiamondLightSource/cryoem-services"
},
"split_keywords": [
"cryoem-services"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "50cbd74b4c97e3f73c6503f62140df08712dfee151c3222206da72fc25b88892",
"md5": "2723340e4f340c4ecef997c306a2c63e",
"sha256": "614c2b89a559ebb386ce6adb2ed557277683dcc6459488aa0faceba4e04c996e"
},
"downloads": -1,
"filename": "cryoemservices-0.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "2723340e4f340c4ecef997c306a2c63e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 113943,
"upload_time": "2024-03-26T16:37:25",
"upload_time_iso_8601": "2024-03-26T16:37:25.780580Z",
"url": "https://files.pythonhosted.org/packages/50/cb/d74b4c97e3f73c6503f62140df08712dfee151c3222206da72fc25b88892/cryoemservices-0.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b3060ccf66d36ca967bfdddada2a2b4c0ca61b7119b0dec11a1a905f4c44d4c9",
"md5": "53c59dfbd7a4742f42c8fabad9e805a4",
"sha256": "5e53981b93ae12c55a0f233ce4941f0f1e1c838ac697350da244b5d813aef707"
},
"downloads": -1,
"filename": "cryoemservices-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "53c59dfbd7a4742f42c8fabad9e805a4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 87877,
"upload_time": "2024-03-26T16:37:27",
"upload_time_iso_8601": "2024-03-26T16:37:27.915461Z",
"url": "https://files.pythonhosted.org/packages/b3/06/0ccf66d36ca967bfdddada2a2b4c0ca61b7119b0dec11a1a905f4c44d4c9/cryoemservices-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-26 16:37:27",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "DiamondLightSource",
"github_project": "cryoem-services",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "cryoemservices"
}
JSON
