Asari
=====
[](https://asari.readthedocs.io/en/latest/?badge=latest)
[](https://doi.org/10.1038/s41467-023-39889-1)
Trackable and scalable Python program for high-resolution LC-MS metabolomics data preprocessing ([Li et al. Nature Communications 14.1 (2023): 4113](https://www.nature.com/articles/s41467-023-39889-1)):
- Taking advantage of high mass resolution to prioritize mass separation and alignment
- Peak detection on a composite map instead of repeated on individual samples
- Statistics guided peak dection, based on local maxima and prominence, selective use of smoothing
- Reproducible, track and backtrack between features and EICs
- Tracking peak quality, selectiviy metrics on m/z, chromatography and annotation databases
- Scalable, performance conscious, disciplined use of memory and CPU
- Transparent, JSON centric data structures, easy to chain other tools
A web server (https://asari.app) and [full pipeline](https://pypi.org/project/pcpfm/) are available now.
Install
=======
- From PyPi repository: `pip3 install asari-metabolomics`. Add `--upgrade` to update to new versions.
- Or clone from source code: https://github.com/shuzhao-li/asari . One can run it as a Python module by calling Python interpreter. GitHub repo is often ahead of PyPi versions.
- Requires Python 3.8+. Installation time ~ 5 seconds if common libraries already exist.
- One can use the web version (https://asari.app) without local installation.
Input
=====
Input data are centroied mzML files from LC-MS metabolomics.
We use ThermoRawFileParser (https://github.com/compomics/ThermoRawFileParser) to convert Thermo .RAW files to .mzML.
Msconvert in ProteoWizard (https://proteowizard.sourceforge.io/tools.shtml) can handle the conversion of most vendor data formats and .mzXML files.
MS/MS spectra are ignored by asari.
Our pipeline (https://pypi.org/project/pcpfm/) has annotation steps to use MS/MS data.
Use
===
If installed from pip, one can run `asari` as a command in a terminal, followed by a subcommand for specific tasks.
For help information:
`asari -h`
To process all mzML files under directory mydir/projectx_dir:
`asari process --mode pos --input mydir/projectx_dir`
To get statistical description on a single file (useful to understand data and parameters):
`asari analyze --input mydir/projectx_dir/file_to_analyze.mzML`
To get annotation on a tab delimited feature table:
`asari annotate --mode pos --ppm 10 --input mydir/projectx_dir/feature_table_file.tsv`
To do automatic esitmation of min peak height, add this argument:
`--autoheight True`
To output additional extraction table on a targeted list of m/z values from target_mzs.txt:
`asari extract --input mydir/projectx_dir --target target_mzs.txt`
This is useful to add QC check during data processing, e.g. the target_mzs.txt file can be spike-in controls.
To launch a dashboard in your web browser after the project is processed into directory process_result_dir:
`asari viz --input process_result_dir`
Alternative to a standalone command, to run as a module via Python interpreter, one needs to point to module location, e.g.:
`python3 -m asari.main process --mode pos --input mydir/projectx_dir`
Output
======
A typical run on disk may generatae a directory like this
rsvstudy_asari_project_427105156
├── Annotated_empricalCompounds.json
├── Feature_annotation.tsv
├── export
│ ├── _mass_grid_mapping.csv
│ ├── cmap.pickle
│ ├── full_Feature_table.tsv
│ └── unique_compound__Feature_table.tsv
├── pickle
│ ├── Blank_20210803_003.pickle
│ ├── ...
├── preferred_Feature_table.tsv
└── project.json
The recommended feature table is `preferred_Feature_table.tsv`.
All peaks are kept in `export/full_Feature_table.tsv` if they meet signal (snr) and shape standards
(part of input parameters but default values are fine for most people).
That is, if a feature is only present in one sample, it will be reported,
as we think this is important for applications like exposome and personalized medicine.
The filtering decisions are left to end users.
The `pickle` folder keeps intermediate files during processing.
They are removed after the processing by default, to save disk space.
Users can choose to keep them by specifying `--pickle True`.
Dashboard
=========
After data are processed, users can use `asari viz --input process_result_dir` to launch a dashboard to inspect data, where 'process_result_dir' refers to the result folder. The dashboard uses these files under the result folder: 'project.json', 'export/cmap.pickle', 'export/epd.pickle' and 'export/full_Feature_table.tsv'. Thus, one can move around the folder, but modification of these files is not a good idea. Please note that pickle files are for internal use, and one should not trust pickle files from other people.
Parameters
==========
Only one parameter in asari requires real attention, i.e., m/z precision is set at 5 ppm by default.
Most modern instruments are fine with 5 ppm, but one may want to change if needed.
Default ionization mode is `pos`. Change to `neg` if needed, by specifying `--mode neg` in command line.
Users can supply a custom parameter file `xyz.yaml`, via `--parameters xyz.yaml` in command line.
A template YAML file can be found at `test/parameters.yaml`.
When the above methods overlap, command line arguments take priority.
That is, commandline overwrites `xyz.yaml`, which overwrites default asari parameters in `defaul_parameters.py`.
Algorithms
==========
Basic data concepts follow https://github.com/shuzhao-li/metDataModel, organized as
├── Experiment
├── Sample
├── MassTrack
├── Peak
├── Peak
├── MassTrack
├── Peak
├── Peak
...
├── Sample
...
├── Sample
A sample here corresponds to an injection file in LC-MS experiments.
A MassTrack is an extracted chromatogram for a specific m/z measurement, governing full retention time.
Therefore, a MassTrack may include multiple mass traces, or EICs/XICs, as referred by literature.
Peak (an elution peak at specific m/z) is specific to a sample, but a feature is defined at the level of an experiment after correspondence.
Additional details:
- Use of MassTracks simplifies m/z correspondence, which results in a MassGrid
- Two modes of m/z correspondence: a clustering method for studies >= N (default 10) samples;
and a slower method based on landmark peaks and verifying mass precision.
- Chromatogram construction is based on m/z values via flexible bins and frequency counts (in lieu histograms).
- Elution peak alignment is based on LOWESS
- Use integers for RT scan numbers and intensities for computing efficiency
- Avoid mathematical curves whereas possible for computing efficiency
Selectivity is tracked for
- mSelectivity, how distinct are m/z measurements
- cSelectivity, how distinct are chromatograhic elution peaks
Step-by-step algorithms are explained in doc/README.md.
This package uses `mass2chem`, `khipu` and `JMS` for mass search and annotation functions.
Performance
===========
Asari is designed to run > 1000 samples on a laptop computer. The performance is achieved via
- Implementation of basic functions using discrete mathematics and avoiding continuous curves.
- Main intensity values of each sample are not kept in memory.
- Simple (and transparent) peak detection based on local maxima (no curve fitting until evaluation)
- Composite mass tracks greatly reduce the run cycles on peak detection
- Using Python numerical libraries and vector operations
- Alignment of mass tracks uses clustering in larger sample size
When a study has N (default 10) or fewer samples, the MassGrid assembly uses a slower algorithm to compensate statistical distribution.
If the individual files are large or the sample number is very high, it is easy to split the data and run asari separately.
One can then use `asari join` to merge the results [in progress].
Future improvement can be made by implementing some functions, e.g. chromatogram building, in C.
Docker image
============
At https://hub.docker.com/r/shuzhao/asari.
This image includes mono and ThermoRawFileParser, which converts Thermo .raw files to .mzML files.
Example use
To launch with volume mapping `$ docker run -v /Users/shuzhao/data:/home -ti shuzhao/asari`.
In the container, ThermoRawFileParser is under `/usr/local/thermo/`.
```
# mono /usr/local/thermo/ThermoRawFileParser.exe -d my_data_dir
# asari analyze --input tmp/file_008.mzML
# asari process --mode neg --input tmp --output test99
```
Links
=====
Source code: https://github.com/shuzhao-li/asari
Package Repository: https://pypi.org/project/asari-metabolomics/
Test data: https://github.com/shuzhao-li/data/tree/main/data
Notebooks to reproduce publication figures: https://github.com/shuzhao-li/data/tree/main/notebooks
How accurate are my m/z values?
===============================
The mass tracks are scaffolds to assemble data. Very close m/z values may not be distinguished on a mass track. For example, when mass tracks are constructed for 5 ppm resolution, two m/z values of 3 ppm apart will be reported on the same mass track. This leads to a situation where the m/z values are not optimal. Asari is designed for reliable information retrieval. If the data are processed under 5 ppm, the information can be retrieved by 5 ppm. The true m/z values will be recovered via annotation, if the features are resolved by LC, when asari features are matched to annotation libraries.
As discussed in the manuscript, ppm is not perfect in modeling mass resolution and is not constant for all m/z ranges. It is a practical tool we currently work with. If two compounds are not resolved by LC and their m/z values are 4 ppm apart, asari processing by 5 ppm will treat them as one feature. If the mass resolution is justified, one can run asari using, for instance, 3 ppm. The default workflow in asari does not fine-tune the m/z values, because the split m/z peaks from centroiding are difficult to distinguish from real m/z peaks. We leave the fine-tuning to annotation or targeted extraction workflow.
We thank reviewer #1 for valuable discussions on this topic.
Related projects
================
Mummichog: metabolomics pathway/network analysis
metDataModel: data models for metabolomics
mass2chem: common utilities in interpreting mass spectrometry data, annotation
khipu: a Python library for generalized, low-level annotation of MS metabolomics
JMS: Json's Metabolite Services
Raw data
{
"_id": null,
"home_page": "https://github.com/shuzhao-li/asari",
"name": "asari-metabolomics",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "metabolomics bioinformatics mass spectrometry",
"author": "Shuzhao Li",
"author_email": "shuzhao.li@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/32/f0/5bd9a5f9d4fad265cbf16d93f8313eafbe7481739f43ca423034186402cd/asari-metabolomics-1.13.1.tar.gz",
"platform": null,
"description": "Asari\n=====\n[](https://asari.readthedocs.io/en/latest/?badge=latest)\n[](https://doi.org/10.1038/s41467-023-39889-1)\n\nTrackable and scalable Python program for high-resolution LC-MS metabolomics data preprocessing ([Li et al. Nature Communications 14.1 (2023): 4113](https://www.nature.com/articles/s41467-023-39889-1)):\n\n- Taking advantage of high mass resolution to prioritize mass separation and alignment\n- Peak detection on a composite map instead of repeated on individual samples\n- Statistics guided peak dection, based on local maxima and prominence, selective use of smoothing\n- Reproducible, track and backtrack between features and EICs\n- Tracking peak quality, selectiviy metrics on m/z, chromatography and annotation databases\n- Scalable, performance conscious, disciplined use of memory and CPU \n- Transparent, JSON centric data structures, easy to chain other tools\n\nA web server (https://asari.app) and [full pipeline](https://pypi.org/project/pcpfm/) are available now.\n\nInstall\n=======\n- From PyPi repository: `pip3 install asari-metabolomics`. Add `--upgrade` to update to new versions.\n\n- Or clone from source code: https://github.com/shuzhao-li/asari . One can run it as a Python module by calling Python interpreter. GitHub repo is often ahead of PyPi versions.\n\n- Requires Python 3.8+. Installation time ~ 5 seconds if common libraries already exist.\n\n- One can use the web version (https://asari.app) without local installation.\n\nInput\n=====\nInput data are centroied mzML files from LC-MS metabolomics. \nWe use ThermoRawFileParser (https://github.com/compomics/ThermoRawFileParser) to convert Thermo .RAW files to .mzML. \nMsconvert in ProteoWizard (https://proteowizard.sourceforge.io/tools.shtml) can handle the conversion of most vendor data formats and .mzXML files.\n\nMS/MS spectra are ignored by asari. \nOur pipeline (https://pypi.org/project/pcpfm/) has annotation steps to use MS/MS data.\n\nUse \n===\nIf installed from pip, one can run `asari` as a command in a terminal, followed by a subcommand for specific tasks.\n\nFor help information:\n\n`asari -h`\n\nTo process all mzML files under directory mydir/projectx_dir:\n\n`asari process --mode pos --input mydir/projectx_dir`\n\nTo get statistical description on a single file (useful to understand data and parameters):\n\n`asari analyze --input mydir/projectx_dir/file_to_analyze.mzML`\n\nTo get annotation on a tab delimited feature table:\n\n`asari annotate --mode pos --ppm 10 --input mydir/projectx_dir/feature_table_file.tsv`\n\nTo do automatic esitmation of min peak height, add this argument:\n\n`--autoheight True`\n\nTo output additional extraction table on a targeted list of m/z values from target_mzs.txt:\n\n`asari extract --input mydir/projectx_dir --target target_mzs.txt`\n\nThis is useful to add QC check during data processing, e.g. the target_mzs.txt file can be spike-in controls.\n\nTo launch a dashboard in your web browser after the project is processed into directory process_result_dir:\n\n`asari viz --input process_result_dir`\n\nAlternative to a standalone command, to run as a module via Python interpreter, one needs to point to module location, e.g.:\n\n`python3 -m asari.main process --mode pos --input mydir/projectx_dir`\n\nOutput\n======\nA typical run on disk may generatae a directory like this\n\n rsvstudy_asari_project_427105156\n \u251c\u2500\u2500 Annotated_empricalCompounds.json\n \u251c\u2500\u2500 Feature_annotation.tsv\n \u251c\u2500\u2500 export\n \u2502\u00a0\u00a0 \u251c\u2500\u2500 _mass_grid_mapping.csv\n \u2502\u00a0\u00a0 \u251c\u2500\u2500 cmap.pickle\n \u2502\u00a0\u00a0 \u251c\u2500\u2500 full_Feature_table.tsv\n \u2502\u00a0\u00a0 \u2514\u2500\u2500 unique_compound__Feature_table.tsv\n \u251c\u2500\u2500 pickle\n \u2502\u00a0\u00a0 \u251c\u2500\u2500 Blank_20210803_003.pickle\n \u2502\u00a0\u00a0 \u251c\u2500\u2500 ...\n \u251c\u2500\u2500 preferred_Feature_table.tsv\n \u2514\u2500\u2500 project.json\n\nThe recommended feature table is `preferred_Feature_table.tsv`. \n\nAll peaks are kept in `export/full_Feature_table.tsv` if they meet signal (snr) and shape standards \n(part of input parameters but default values are fine for most people). \nThat is, if a feature is only present in one sample, it will be reported, \nas we think this is important for applications like exposome and personalized medicine. \nThe filtering decisions are left to end users.\n\nThe `pickle` folder keeps intermediate files during processing.\nThey are removed after the processing by default, to save disk space.\nUsers can choose to keep them by specifying `--pickle True`.\n\n\nDashboard\n=========\nAfter data are processed, users can use `asari viz --input process_result_dir` to launch a dashboard to inspect data, where 'process_result_dir' refers to the result folder. The dashboard uses these files under the result folder: 'project.json', 'export/cmap.pickle', 'export/epd.pickle' and 'export/full_Feature_table.tsv'. Thus, one can move around the folder, but modification of these files is not a good idea. Please note that pickle files are for internal use, and one should not trust pickle files from other people.\n \n\n\n\nParameters\n==========\nOnly one parameter in asari requires real attention, i.e., m/z precision is set at 5 ppm by default. \nMost modern instruments are fine with 5 ppm, but one may want to change if needed.\n\nDefault ionization mode is `pos`. Change to `neg` if needed, by specifying `--mode neg` in command line.\n\nUsers can supply a custom parameter file `xyz.yaml`, via `--parameters xyz.yaml` in command line.\nA template YAML file can be found at `test/parameters.yaml`.\n\nWhen the above methods overlap, command line arguments take priority.\nThat is, commandline overwrites `xyz.yaml`, which overwrites default asari parameters in `defaul_parameters.py`. \n\nAlgorithms\n==========\nBasic data concepts follow https://github.com/shuzhao-li/metDataModel, organized as\n\n \u251c\u2500\u2500 Experiment\n \u00a0\u00a0 \u251c\u2500\u2500 Sample\n \u00a0\u00a0 \u251c\u2500\u2500 MassTrack\n \u00a0\u00a0 \u251c\u2500\u2500 Peak\n \u00a0\u00a0 \u251c\u2500\u2500 Peak\n \u00a0\u00a0 \u251c\u2500\u2500 MassTrack \n \u00a0\u00a0 \u251c\u2500\u2500 Peak\n \u00a0\u00a0 \u251c\u2500\u2500 Peak\n ...\n \u00a0\u00a0 \u251c\u2500\u2500 Sample \n ...\n \u00a0\u00a0 \u251c\u2500\u2500 Sample \n\nA sample here corresponds to an injection file in LC-MS experiments. \nA MassTrack is an extracted chromatogram for a specific m/z measurement, governing full retention time.\nTherefore, a MassTrack may include multiple mass traces, or EICs/XICs, as referred by literature.\nPeak (an elution peak at specific m/z) is specific to a sample, but a feature is defined at the level of an experiment after correspondence.\n\nAdditional details:\n- Use of MassTracks simplifies m/z correspondence, which results in a MassGrid\n- Two modes of m/z correspondence: a clustering method for studies >= N (default 10) samples; \n and a slower method based on landmark peaks and verifying mass precision.\n- Chromatogram construction is based on m/z values via flexible bins and frequency counts (in lieu histograms). \n- Elution peak alignment is based on LOWESS\n- Use integers for RT scan numbers and intensities for computing efficiency\n- Avoid mathematical curves whereas possible for computing efficiency\n\nSelectivity is tracked for\n- mSelectivity, how distinct are m/z measurements \n- cSelectivity, how distinct are chromatograhic elution peaks\n\nStep-by-step algorithms are explained in doc/README.md.\n\nThis package uses `mass2chem`, `khipu` and `JMS` for mass search and annotation functions.\n\nPerformance\n===========\nAsari is designed to run > 1000 samples on a laptop computer. The performance is achieved via\n- Implementation of basic functions using discrete mathematics and avoiding continuous curves.\n- Main intensity values of each sample are not kept in memory.\n- Simple (and transparent) peak detection based on local maxima (no curve fitting until evaluation)\n- Composite mass tracks greatly reduce the run cycles on peak detection\n- Using Python numerical libraries and vector operations\n- Alignment of mass tracks uses clustering in larger sample size\n\nWhen a study has N (default 10) or fewer samples, the MassGrid assembly uses a slower algorithm to compensate statistical distribution.\n\nIf the individual files are large or the sample number is very high, it is easy to split the data and run asari separately. \nOne can then use `asari join` to merge the results [in progress].\n\nFuture improvement can be made by implementing some functions, e.g. chromatogram building, in C.\n\nDocker image\n============\nAt https://hub.docker.com/r/shuzhao/asari.\n\nThis image includes mono and ThermoRawFileParser, which converts Thermo .raw files to .mzML files.\n\nExample use\nTo launch with volume mapping `$ docker run -v /Users/shuzhao/data:/home -ti shuzhao/asari`.\n\nIn the container, ThermoRawFileParser is under `/usr/local/thermo/`.\n```\n# mono /usr/local/thermo/ThermoRawFileParser.exe -d my_data_dir\n\n# asari analyze --input tmp/file_008.mzML \n\n# asari process --mode neg --input tmp --output test99\n```\n\n\nLinks\n=====\nSource code: https://github.com/shuzhao-li/asari\n\nPackage Repository: https://pypi.org/project/asari-metabolomics/\n\nTest data: https://github.com/shuzhao-li/data/tree/main/data\n\nNotebooks to reproduce publication figures: https://github.com/shuzhao-li/data/tree/main/notebooks\n\nHow accurate are my m/z values?\n===============================\nThe mass tracks are scaffolds to assemble data. Very close m/z values may not be distinguished on a mass track. For example, when mass tracks are constructed for 5 ppm resolution, two m/z values of 3 ppm apart will be reported on the same mass track. This leads to a situation where the m/z values are not optimal. Asari is designed for reliable information retrieval. If the data are processed under 5 ppm, the information can be retrieved by 5 ppm. The true m/z values will be recovered via annotation, if the features are resolved by LC, when asari features are matched to annotation libraries.\n\nAs discussed in the manuscript, ppm is not perfect in modeling mass resolution and is not constant for all m/z ranges. It is a practical tool we currently work with. If two compounds are not resolved by LC and their m/z values are 4 ppm apart, asari processing by 5 ppm will treat them as one feature. If the mass resolution is justified, one can run asari using, for instance, 3 ppm. The default workflow in asari does not fine-tune the m/z values, because the split m/z peaks from centroiding are difficult to distinguish from real m/z peaks. We leave the fine-tuning to annotation or targeted extraction workflow.\n\nWe thank reviewer #1 for valuable discussions on this topic.\n\n\nRelated projects\n================\n\nMummichog: metabolomics pathway/network analysis\n\nmetDataModel: data models for metabolomics\n\nmass2chem: common utilities in interpreting mass spectrometry data, annotation\n\nkhipu: a Python library for generalized, low-level annotation of MS metabolomics\n\nJMS: Json's Metabolite Services\n",
"bugtrack_url": null,
"license": "BSD 3-Clause",
"summary": "LC-MS metabolomics data preprocessing",
"version": "1.13.1",
"project_urls": {
"Homepage": "https://github.com/shuzhao-li/asari"
},
"split_keywords": [
"metabolomics",
"bioinformatics",
"mass",
"spectrometry"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "9a16f55f5d6e6010c07b6199e974dfdd67fa9a26ad95aad885b6b126e6314eb7",
"md5": "32b5a226ff25985dbcb2559085a1d469",
"sha256": "8c9ffa72a84ba7332fbbb35ae67cc6395f108caf862d9d340c1b59e82fdfb215"
},
"downloads": -1,
"filename": "asari_metabolomics-1.13.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "32b5a226ff25985dbcb2559085a1d469",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 12952126,
"upload_time": "2024-03-04T19:45:43",
"upload_time_iso_8601": "2024-03-04T19:45:43.893426Z",
"url": "https://files.pythonhosted.org/packages/9a/16/f55f5d6e6010c07b6199e974dfdd67fa9a26ad95aad885b6b126e6314eb7/asari_metabolomics-1.13.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "32f05bd9a5f9d4fad265cbf16d93f8313eafbe7481739f43ca423034186402cd",
"md5": "9b7547b5a6af24fd93bc7b89240bd0da",
"sha256": "2cf8aa878495009cbf937e17c5e75a058d7073d70fbac6ec121ea4ef9e41503d"
},
"downloads": -1,
"filename": "asari-metabolomics-1.13.1.tar.gz",
"has_sig": false,
"md5_digest": "9b7547b5a6af24fd93bc7b89240bd0da",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 6323620,
"upload_time": "2024-03-04T19:45:47",
"upload_time_iso_8601": "2024-03-04T19:45:47.067973Z",
"url": "https://files.pythonhosted.org/packages/32/f0/5bd9a5f9d4fad265cbf16d93f8313eafbe7481739f43ca423034186402cd/asari-metabolomics-1.13.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-04 19:45:47",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "shuzhao-li",
"github_project": "asari",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "metDatamodel",
"specs": []
},
{
"name": "mass2chem",
"specs": []
},
{
"name": "khipu-metabolomics",
"specs": [
[
">=",
"0.7.0"
]
]
},
{
"name": "jms-metabolite-services",
"specs": [
[
">=",
"0.5.7"
]
]
},
{
"name": "pymzml",
"specs": []
},
{
"name": "numpy",
"specs": []
},
{
"name": "scipy",
"specs": []
},
{
"name": "statsmodels",
"specs": []
},
{
"name": "pyyaml",
"specs": []
},
{
"name": "panel",
"specs": [
[
">=",
"0.13.0"
]
]
},
{
"name": "holoviews",
"specs": []
},
{
"name": "hvplot",
"specs": []
},
{
"name": "seaborn",
"specs": []
},
{
"name": "matplotlib",
"specs": []
}
],
"lcname": "asari-metabolomics"
}
JSON
