| Name | metadatamapping JSON |
| Version |
1.2.2
JSON |
| download |
| home_page | |
| Summary | A python library to fetch metadata from NCBI and MetaSRA for a list of NCBI accessions and data extraction from ARCHS4 |
| upload_time | 2024-03-18 08:54:02 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.10 |
| license | MIT License Copyright (c) 2023 Daniel Malzl Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
database
metadata
biopython
bioinformatics
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# metadatamapping
A python library to fetch metadata from NCBI and MetaSRA for a list of NCBI accessions and data extraction from ARCHS4
## Installation
Simply run the following
```commandline
pip install metadatamapping
```
or clone the repository
```commandline
git clone git@github.com:dmalzl/metadatamapping.git
```
and run
```commandline
cd metadatamapping
pip install .
```
you should then be able to import the package as usual
## Example usage
NCBI sample, experiment, biosample or geo accessions can be mapped to SRA uids using the `map_accessions_to_srauids` function from the `metadata` module of the package. The call to the function as shown below invokes two processes that concurrently fetch the SRA UIDs for the accessions in batches and write the results to the outputfile `"/path/to/outputfile"`.
```python
from metadatamapping import metadata
sra_uids = metadata.map_accessions_to_srauids(
accessions,
"/path/to/outputfile",
n_processes = 2
)
```
The resulting SRA UIDs can then either be used to retrieve all associated accessions from the SRA with the `srauids_to_accessions` function from the `metadata` module like so
```python
from metadatamapping import metadata
ncbi_accessions = metadata.srauids_to_accessions(
sra_uids
)
```
or link them to BioSample UIDs and then retrieve the associated metadata with the `link_sra_to_biosample` function from the `link` module and the `biosampleuids_to_metadata` function from the metadata module
```python
from metadatamapping import metadata, link
srauids_to_biosampleuids = link.link_sra_to_biosample(
sra_uids.uid
)
biosample_metadata = metadata.biosample_uids_to_metadata(
srauids_to_biosampleuids.biosample
)
```
Finally we can retrieve normalized metadata for the samples from [MetaSRA](https://metasra.biostat.wisc.edu/) using the `metasra_from_study_id` function of the `metadata` module (note that this database might not contain data for all your samples so the function may only returns normalized metadata for some of your samples)
```python
from metadatamapping import metadata
metasra_metadata = metadata.metasra_from_study_id(
ncbi_accessions.study.unique()
)
```
While most of the biosample metadata is also found in GEO entries some of the metadata provided in GEO (e.g. treatment protocol) is GEO exclusive but may contain vital information. Because this data is not retrievable from the Entrez API, we adopted a similar approach to [`geofetch`](https://github.com/pepkit/geofetch) and download the data from the GEO FTP. An example usage would be as follows:
```python
from metadatamapping import metadata
import pandas as pd
geo_accessions = pd.DataFrame(
[
('GSM2791352', 'GSE104174'),
('GSM2771062', 'GSE103424'),
('GSM6271252', 'GSE207049'),
('GSM4329764', 'GSE145668,GSE145669'),
('GSM5064568', 'GSE166148,GSE166150')
],
columns = ['GSM', 'GSE']
)
geo_metadata = metadata.fetch_geo_metadata(
geo_accessions,
'/path/to/outputfile',
n_processes = 24
)
```
Additionally, the package provides an interface for parsing the [ARCHS4 HDF5 format](https://maayanlab.cloud/archs4/help.html) which is located in the `archs4` module and handles parsing of associated metadata with the `get_filtered_sample_metadata` function as well as extraction of expression data in the [`AnnData`](https://anndata.readthedocs.io/en/latest/generated/anndata.AnnData.html) format with the `samples` function
```python
archs4_file = "/path/to/archs4.h5"
retain_keys = [
'geo_accession', 'characteristics_ch1', 'molecule_ch1', 'readsaligned', 'relation',
'series_id', 'singlecellprobability', 'source_name_ch1', 'title'
]
archs4_metadata = archs4.get_filtered_sample_metadata(
archs4_file,
retain_keys
)
archs4_adata = archs4.samples(
archs4_file,
dataframe_indexed_by_geo_accessions,
n_processes = 2
)
```
For a full demonstration of usage please refer to the [`Snakefile`](https://github.com/dmalzl/metadatamapping/blob/main/examples/Snakefile) in the [`examples`](https://github.com/dmalzl/metadatamapping/tree/main/examples) directory which gives an overview of how the intended usage looks like.
## Entrez credentials
`metadatamapping` retrieves data from the Entrez eUtilities using the [`biopython` interface](https://biopython.org/docs/1.75/api/Bio.Entrez.html). By default the Entrez API only allows 3 requests per second if `Entrez.email` and `Entrez.api_key` are not set. This can be increased by setting these properties accordingly which also speeds up the most timeconsuming part of the pipeline which is the accession -> SRA UID mapping as this relies on eSearch which only allows for one accession at a time (maybe it also takes several but I did not test this as I expect it to be cumbersome to pull apart then). So please make sure to set the `Entrez` properties accordingly like so
```python
from Bio import Entrez
Entrez.email = "<user>@<provider>.<domain>"
Entrez.api_key = "<NCBI API key>
```
The email typically is the email associated to your NCBI account. The API key can be generated as described [here](https://support.nlm.nih.gov/knowledgebase/article/KA-05317/en-us)
Raw data
{
"_id": null,
"home_page": "",
"name": "metadatamapping",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": "",
"keywords": "database,metadata,biopython,bioinformatics",
"author": "",
"author_email": "Daniel Malzl <daniel@menchelab.com>",
"download_url": "https://files.pythonhosted.org/packages/58/dc/0a05b61cf1541b1dc1ffb9ce3ec281fdf7037f67f3662a6ff281ceff2245/metadatamapping-1.2.2.tar.gz",
"platform": null,
"description": "# metadatamapping\n\n\n\n\nA python library to fetch metadata from NCBI and MetaSRA for a list of NCBI accessions and data extraction from ARCHS4\n\n## Installation\nSimply run the following\n```commandline\npip install metadatamapping\n```\nor clone the repository \n```commandline\ngit clone git@github.com:dmalzl/metadatamapping.git\n```\nand run\n```commandline\ncd metadatamapping\npip install .\n```\nyou should then be able to import the package as usual\n\n## Example usage\nNCBI sample, experiment, biosample or geo accessions can be mapped to SRA uids using the `map_accessions_to_srauids` function from the `metadata` module of the package. The call to the function as shown below invokes two processes that concurrently fetch the SRA UIDs for the accessions in batches and write the results to the outputfile `\"/path/to/outputfile\"`.\n```python\nfrom metadatamapping import metadata\nsra_uids = metadata.map_accessions_to_srauids(\n accessions,\n \"/path/to/outputfile\",\n n_processes = 2\n)\n```\nThe resulting SRA UIDs can then either be used to retrieve all associated accessions from the SRA with the `srauids_to_accessions` function from the `metadata` module like so\n```python\nfrom metadatamapping import metadata\nncbi_accessions = metadata.srauids_to_accessions(\n sra_uids\n)\n```\nor link them to BioSample UIDs and then retrieve the associated metadata with the `link_sra_to_biosample` function from the `link` module and the `biosampleuids_to_metadata` function from the metadata module\n```python\nfrom metadatamapping import metadata, link\nsrauids_to_biosampleuids = link.link_sra_to_biosample(\n sra_uids.uid\n)\nbiosample_metadata = metadata.biosample_uids_to_metadata(\n srauids_to_biosampleuids.biosample\n)\n```\nFinally we can retrieve normalized metadata for the samples from [MetaSRA](https://metasra.biostat.wisc.edu/) using the `metasra_from_study_id` function of the `metadata` module (note that this database might not contain data for all your samples so the function may only returns normalized metadata for some of your samples)\n```python\nfrom metadatamapping import metadata\nmetasra_metadata = metadata.metasra_from_study_id(\n ncbi_accessions.study.unique()\n)\n```\nWhile most of the biosample metadata is also found in GEO entries some of the metadata provided in GEO (e.g. treatment protocol) is GEO exclusive but may contain vital information. Because this data is not retrievable from the Entrez API, we adopted a similar approach to [`geofetch`](https://github.com/pepkit/geofetch) and download the data from the GEO FTP. An example usage would be as follows:\n```python\nfrom metadatamapping import metadata\nimport pandas as pd\ngeo_accessions = pd.DataFrame(\n [\n ('GSM2791352', 'GSE104174'),\n ('GSM2771062', 'GSE103424'),\n ('GSM6271252', 'GSE207049'),\n ('GSM4329764', 'GSE145668,GSE145669'),\n ('GSM5064568', 'GSE166148,GSE166150')\n ],\n columns = ['GSM', 'GSE']\n)\n\ngeo_metadata = metadata.fetch_geo_metadata(\n geo_accessions,\n '/path/to/outputfile',\n n_processes = 24\n)\n```\nAdditionally, the package provides an interface for parsing the [ARCHS4 HDF5 format](https://maayanlab.cloud/archs4/help.html) which is located in the `archs4` module and handles parsing of associated metadata with the `get_filtered_sample_metadata` function as well as extraction of expression data in the [`AnnData`](https://anndata.readthedocs.io/en/latest/generated/anndata.AnnData.html) format with the `samples` function\n```python\narchs4_file = \"/path/to/archs4.h5\"\nretain_keys = [\n 'geo_accession', 'characteristics_ch1', 'molecule_ch1', 'readsaligned', 'relation', \n 'series_id', 'singlecellprobability', 'source_name_ch1', 'title'\n]\narchs4_metadata = archs4.get_filtered_sample_metadata(\n archs4_file,\n retain_keys\n)\narchs4_adata = archs4.samples(\n archs4_file,\n dataframe_indexed_by_geo_accessions,\n n_processes = 2\n)\n```\nFor a full demonstration of usage please refer to the [`Snakefile`](https://github.com/dmalzl/metadatamapping/blob/main/examples/Snakefile) in the [`examples`](https://github.com/dmalzl/metadatamapping/tree/main/examples) directory which gives an overview of how the intended usage looks like.\n\n## Entrez credentials\n`metadatamapping` retrieves data from the Entrez eUtilities using the [`biopython` interface](https://biopython.org/docs/1.75/api/Bio.Entrez.html). By default the Entrez API only allows 3 requests per second if `Entrez.email` and `Entrez.api_key` are not set. This can be increased by setting these properties accordingly which also speeds up the most timeconsuming part of the pipeline which is the accession -> SRA UID mapping as this relies on eSearch which only allows for one accession at a time (maybe it also takes several but I did not test this as I expect it to be cumbersome to pull apart then). So please make sure to set the `Entrez` properties accordingly like so\n```python\nfrom Bio import Entrez\nEntrez.email = \"<user>@<provider>.<domain>\"\nEntrez.api_key = \"<NCBI API key>\n```\nThe email typically is the email associated to your NCBI account. The API key can be generated as described [here](https://support.nlm.nih.gov/knowledgebase/article/KA-05317/en-us)\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2023 Daniel Malzl Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
"summary": "A python library to fetch metadata from NCBI and MetaSRA for a list of NCBI accessions and data extraction from ARCHS4",
"version": "1.2.2",
"project_urls": {
"Homepage": "https://github.com/dmalzl/metadatamapping"
},
"split_keywords": [
"database",
"metadata",
"biopython",
"bioinformatics"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "aed8974f389b1a98aa176038b755431223ed60e618e89852b655df84bf5c45e4",
"md5": "1c27c6702e32cef2caad056baa1ecc8c",
"sha256": "c8d99136ef7f074c11cc3ad7c2b07d635347842ea494feb7cca18491c939eb09"
},
"downloads": -1,
"filename": "metadatamapping-1.2.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "1c27c6702e32cef2caad056baa1ecc8c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 31647,
"upload_time": "2024-03-18T08:53:59",
"upload_time_iso_8601": "2024-03-18T08:53:59.955151Z",
"url": "https://files.pythonhosted.org/packages/ae/d8/974f389b1a98aa176038b755431223ed60e618e89852b655df84bf5c45e4/metadatamapping-1.2.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "58dc0a05b61cf1541b1dc1ffb9ce3ec281fdf7037f67f3662a6ff281ceff2245",
"md5": "3a6b512fc8d167ea6a3d5ece68ddc4f9",
"sha256": "9ba5ea69a7053e3b9a2ea4efcd2bfcd66c1932ac5d9516ba3ee8505d12ade90f"
},
"downloads": -1,
"filename": "metadatamapping-1.2.2.tar.gz",
"has_sig": false,
"md5_digest": "3a6b512fc8d167ea6a3d5ece68ddc4f9",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 28380,
"upload_time": "2024-03-18T08:54:02",
"upload_time_iso_8601": "2024-03-18T08:54:02.923804Z",
"url": "https://files.pythonhosted.org/packages/58/dc/0a05b61cf1541b1dc1ffb9ce3ec281fdf7037f67f3662a6ff281ceff2245/metadatamapping-1.2.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-18 08:54:02",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "dmalzl",
"github_project": "metadatamapping",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "metadatamapping"
}
