[](https://github.com/rpetit3/fastq-dl/releases)
[](https://anaconda.org/bioconda/fastq-dl)
[](https://gitpod.io/#https://github.com/rpetit3/fastq-dl)
[](https://doi.org/10.5281/zenodo.8051230)
# fastq-dl
Download FASTQ files from the [European Nucleotide Archive](https://www.ebi.ac.uk/ena) or the
[Sequence Read Archive](https://www.ncbi.nlm.nih.gov/sra) repositories.
## Introduction
`fastq-dl` takes an ENA/SRA accession (Study, Sample, Experiment, or Run) and queries ENA (via
[Data Warehouse API](https://www.ebi.ac.uk/ena/browse/search-rest)) to determine the associated
metadata. It then downloads FASTQ files for each Run. For Samples or Experiments with multiple
Runs, users can optionally merge the runs.
## Installation
### Bioconda
`fastq-dl` is available from [Bioconda](https://bioconda.github.io/) and I highly recommend you
go this route to for installation.
```{bash}
conda create -n fastq-dl -c conda-forge -c bioconda fastq-dl
conda activate fastq-dl
```
## Usage
```{bash}
fastq-dl --help
Usage: fastq-dl [OPTIONS]
Download FASTQ files from ENA or SRA.
╭─ Required Options ─────────────────────────────────────────────────────────────────────────────╮
│ * --accession -a TEXT ENA/SRA accession to query. (Study, Sample, Experiment, Run │
│ accession) │
│ [required] │
╰────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Additional Options ───────────────────────────────────────────────────────────────────────────╮
│ --provider [ena|sra] Specify which provider (ENA or SRA) to use. │
│ [default: ena] │
│ --group-by-experiment Group Runs by experiment accession. │
│ --group-by-sample Group Runs by sample accession. │
│ --outdir -o TEXT Directory to output downloads to. [default: ./] │
│ --prefix TEXT Prefix to use for naming log files. [default: fastq] │
│ --cpus INTEGER Total cpus used for downloading from SRA. │
│ [default: 1] │
│ --max-attempts -m INTEGER Maximum number of download attempts. [default: 10] │
│ --force Overwrite existing files if their MD5 checksums do │
│ not match. │
│ --sra-lite Set preference to SRA Lite │
│ --only-provider Only attempt download from specified provider. │
│ --only-download-metadata Skip FASTQ downloads, and retrieve only the metadata. │
│ --silent Only critical errors will be printed. │
│ --sleep -s INTEGER Minimum amount of time to sleep between retries (API │
│ query and download) │
│ [default: 10] │
│ --version -V Show the version and exit. │
│ --verbose -v Print debug related text. │
│ --help -h Show this message and exit. │
╰────────────────────────────────────────────────────────────────────────────────────────────────╯
```
*fastq-dl* requires a single ENA/SRA Study, Sample, Experiment, or Run accession and FASTQs
for all Runs that fall under the given accession will be downloaded. For example, if a Study
accession is given all Runs under that studies umbrella will be downloaded. By default,
`fastq-dl` will try to download from ENA first, then SRA.
### --accession
The accession you would like to download associated FASTQS for. Currently the following types
of accessions are accepted.
| Accession Type | Prefixes | Example |
|----------------|---------------------|------------------------------------------|
| BioProject | PRJEB, PRJNA, PRJDB | PRJEB42779, PRJNA480016, PRJDB14838 |
| Study | ERP, DRP, SRP | ERP126685, DRP009283, SRP158268 |
| BioSample | SAMD, SAME, SAMN | SAMD00258402, SAMEA7997453, SAMN06479985 |
| Sample | ERS, DRS, SRS | ERS5684710, DRS259711, SRS2024210 |
| Experiment | ERX, DRX, SRX | ERX5050800, DRX406443, SRX4563689 |
| Run | ERR, DRR, SRR | ERR5260405, DRR421224, SRR7706354 |
The accessions are using regular expressions from the [ENA Training Modules - Accession Numbers](https://ena-docs.readthedocs.io/en/latest/submit/general-guide/accessions.html#accession-numbers) section.
### --provider
`fastq-dl` gives you the option to download from ENA or SRA. the `--provider` option will
specify which provider you would like to attempt downloads from first. If a download fails
from the first provider, additional attempts will be made using the other provider.
ENA was selected as the default provider because the FASTQs are available directly without
the need for conversion.
### --only-provider
By default, `fastq-dl` will fallback on a secondary provider to attempt downloads. There
may be cases where you would prefer to disable this feature, and that is exactly the
purpose of `--only-provider`. When provided, if a FASTQ cannot be downloaded from the
original provider, no additional attempts will be made.
### --group-by-experiment & --group-by-sample
There maybe times you might want to group Run accessions based on a Experiment or Sample
accessions. This will merge FASTQs associated with a Run accession based its associated
Experiment accession (`--group-by-experiment`) or Sample accession (`--group-by-sample`).
### --sra-lite
Downloads from SRA are provided in [SRA Normalized and SRA Lite](https://www.ncbi.nlm.nih.gov/sra/docs/sra-data-formats/) formats.
SRA Normalized is the original format with full base quality scores and SRA Lite are smaller
due to simplifying the quality scores to a uniform Q30. By default the preference will be
set to SRA Normalized, if you prefer SRA Lite you can use `--sra-lite` to set the
preference to SRA Lite.
## Output Files
| Extension | Description |
|--------------------|------------------------------------------------------------------------------------------|
| `-run-info.tsv` | Tab-delimited file containing metadata for each Run downloaded |
| `-run-mergers.tsv` | Tab-delimited file merge information from `--group-by-experiment` or `--group-by-sample` |
| `.fastq.gz` | FASTQ files downloaded from ENA or SRA |
## Example Usage
#### Download FASTQs associated with a Study
Sometimes you might be reading a paper and they very kindly provided a Bioproject of all
the samples they sequenced. So, you decide you want to download FASTQs for all the samples
asscociated with the Bioproject. `fastq-dl` can help you with that!
```{bash}
fastq-dl --accession PRJNA248678 --provider SRA
fastq-dl --accession PRJNA248678
```
The above commands will download the 3 Runs that fall under Study accession [PRJNA248678](https://www.ebi.ac.uk/ena/browser/view/PRJNA248678)
from either SRA (`--provider SRA`) or ENA (without `--provider`).
#### Download FASTQs associated with an Experiment
Let's say instead of the whole Bioproject you just want a single Experiment. You can do
that as well.
```{bash}
fastq-dl --accession SRX477044
```
The above command would download the Run accessions from ENA that fall under Experiment SRX477044.
The relationship of Experiment to Run is a 1-to-many relationship, or there can be many Run accessions
associated with a single Experiment Accession (e.g. re-sequencing the same sample). Although in most
cases, it is a 1-to-1 relationship, you can use `--group-by-experiment` to merge multiple runs
associated with an Experiment accession into a single FASTQ file.
#### Download FASTQs associated with an Sample
Ok, this time you just want a single Sample, or Biosample.
```{bash}
fastq-dl --accession SRS1904245 --provider SRA
```
The above command would download the Run accessions from SRA that fall under Sample SRS1904245.
Similar to Experiment accessions, the relationship of Sample to Run is a 1-to-many relationship,
or there can be many Run accessions associated with a single Sample Accession. Although in most
cases, it is a 1-to-1 relationship, you can use `--group-by-sample` to merge multiple runs
associated with an Sample accession into a single FASTQ file.
_Warning! For some type strains (e.g. S. aureus USA300) a Biosample accession might be associated with
100s or 1000s of Run accessions. These Runs are likely associated with many different conditions and
really should not fall under a single BioSample accession. Please consider this when using
`--group-by-sample`.
#### Download FASTQs associated with a Run
Let's keep it super simple and just download a Run.
```
fastq-dl --accession SRR1178105 --provider SRA
```
The above command would download the Run SRR1178105 from SRA. Run accessions are the end of the
line (1-to-1 relationship), so you will always get the expected Run.
## Alternatives
`fastq-dl`, is a spin-off of [ena-dl](https://github.com/rpetit3/ena-dl), that has been developed for
usage with [Bactopia](https://github.com/bactopia/bactopia). With this in mind, EBI/NCBI and provide
their own tools ([enaBrowserTools](https://github.com/enasequence/enaBrowserTools) and
[SRA Toolkit](https://github.com/ncbi/sra-tools)) that offer more extensive access to their databases.
Raw data
{
"_id": null,
"home_page": "https://github.com/rpetit3/fastq-dl",
"name": "fastq-dl",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7.1,<4.0.0",
"maintainer_email": "",
"keywords": "bioinformatics,fastq,download,SRA,ENA",
"author": "Robert A. Petit III",
"author_email": "robbie.petit@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/1d/02/e589bd29da49c3f04ad38f968ee5b67c9a9f5439bfbda7b94664d8df6ebb/fastq_dl-2.0.3.tar.gz",
"platform": null,
"description": "[](https://github.com/rpetit3/fastq-dl/releases)\n[](https://anaconda.org/bioconda/fastq-dl)\n[](https://gitpod.io/#https://github.com/rpetit3/fastq-dl)\n[](https://doi.org/10.5281/zenodo.8051230)\n\n# fastq-dl\n\nDownload FASTQ files from the [European Nucleotide Archive](https://www.ebi.ac.uk/ena) or the\n[Sequence Read Archive](https://www.ncbi.nlm.nih.gov/sra) repositories.\n\n## Introduction\n\n`fastq-dl` takes an ENA/SRA accession (Study, Sample, Experiment, or Run) and queries ENA (via\n[Data Warehouse API](https://www.ebi.ac.uk/ena/browse/search-rest)) to determine the associated\nmetadata. It then downloads FASTQ files for each Run. For Samples or Experiments with multiple\nRuns, users can optionally merge the runs.\n\n## Installation\n\n### Bioconda\n\n`fastq-dl` is available from [Bioconda](https://bioconda.github.io/) and I highly recommend you\ngo this route to for installation.\n\n```{bash}\nconda create -n fastq-dl -c conda-forge -c bioconda fastq-dl\nconda activate fastq-dl \n```\n\n## Usage\n\n```{bash}\nfastq-dl --help\n\n Usage: fastq-dl [OPTIONS] \n \n Download FASTQ files from ENA or SRA. \n \n\u256d\u2500 Required Options \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 * --accession -a TEXT ENA/SRA accession to query. (Study, Sample, Experiment, Run \u2502\n\u2502 accession) \u2502\n\u2502 [required] \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n\u256d\u2500 Additional Options \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 --provider [ena|sra] Specify which provider (ENA or SRA) to use. \u2502\n\u2502 [default: ena] \u2502\n\u2502 --group-by-experiment Group Runs by experiment accession. \u2502\n\u2502 --group-by-sample Group Runs by sample accession. \u2502\n\u2502 --outdir -o TEXT Directory to output downloads to. [default: ./] \u2502\n\u2502 --prefix TEXT Prefix to use for naming log files. [default: fastq] \u2502\n\u2502 --cpus INTEGER Total cpus used for downloading from SRA. \u2502\n\u2502 [default: 1] \u2502\n\u2502 --max-attempts -m INTEGER Maximum number of download attempts. [default: 10] \u2502\n\u2502 --force Overwrite existing files if their MD5 checksums do \u2502\n\u2502 not match. \u2502\n\u2502 --sra-lite Set preference to SRA Lite \u2502\n\u2502 --only-provider Only attempt download from specified provider. \u2502\n\u2502 --only-download-metadata Skip FASTQ downloads, and retrieve only the metadata. \u2502\n\u2502 --silent Only critical errors will be printed. \u2502\n\u2502 --sleep -s INTEGER Minimum amount of time to sleep between retries (API \u2502\n\u2502 query and download) \u2502\n\u2502 [default: 10] \u2502\n\u2502 --version -V Show the version and exit. \u2502\n\u2502 --verbose -v Print debug related text. \u2502\n\u2502 --help -h Show this message and exit. \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n```\n\n*fastq-dl* requires a single ENA/SRA Study, Sample, Experiment, or Run accession and FASTQs\nfor all Runs that fall under the given accession will be downloaded. For example, if a Study\naccession is given all Runs under that studies umbrella will be downloaded. By default, \n`fastq-dl` will try to download from ENA first, then SRA.\n\n### --accession\n\nThe accession you would like to download associated FASTQS for. Currently the following types\nof accessions are accepted.\n\n| Accession Type | Prefixes | Example |\n|----------------|---------------------|------------------------------------------|\n| BioProject | PRJEB, PRJNA, PRJDB | PRJEB42779, PRJNA480016, PRJDB14838 |\n| Study | ERP, DRP, SRP | ERP126685, DRP009283, SRP158268 |\n| BioSample | SAMD, SAME, SAMN | SAMD00258402, SAMEA7997453, SAMN06479985 |\n| Sample | ERS, DRS, SRS | ERS5684710, DRS259711, SRS2024210 |\n| Experiment | ERX, DRX, SRX | ERX5050800, DRX406443, SRX4563689 |\n| Run | ERR, DRR, SRR | ERR5260405, DRR421224, SRR7706354 |\n\nThe accessions are using regular expressions from the [ENA Training Modules - Accession Numbers](https://ena-docs.readthedocs.io/en/latest/submit/general-guide/accessions.html#accession-numbers) section.\n\n### --provider\n\n`fastq-dl` gives you the option to download from ENA or SRA. the `--provider` option will\nspecify which provider you would like to attempt downloads from first. If a download fails\nfrom the first provider, additional attempts will be made using the other provider.\n\nENA was selected as the default provider because the FASTQs are available directly without\nthe need for conversion.\n\n### --only-provider\n\nBy default, `fastq-dl` will fallback on a secondary provider to attempt downloads. There\nmay be cases where you would prefer to disable this feature, and that is exactly the\npurpose of `--only-provider`. When provided, if a FASTQ cannot be downloaded from the\noriginal provider, no additional attempts will be made.\n\n### --group-by-experiment & --group-by-sample\n\nThere maybe times you might want to group Run accessions based on a Experiment or Sample\naccessions. This will merge FASTQs associated with a Run accession based its associated\nExperiment accession (`--group-by-experiment`) or Sample accession (`--group-by-sample`).\n\n### --sra-lite\n\nDownloads from SRA are provided in [SRA Normalized and SRA Lite](https://www.ncbi.nlm.nih.gov/sra/docs/sra-data-formats/) formats.\nSRA Normalized is the original format with full base quality scores and SRA Lite are smaller\ndue to simplifying the quality scores to a uniform Q30. By default the preference will be\nset to SRA Normalized, if you prefer SRA Lite you can use `--sra-lite` to set the\npreference to SRA Lite.\n\n## Output Files\n\n| Extension | Description |\n|--------------------|------------------------------------------------------------------------------------------|\n| `-run-info.tsv` | Tab-delimited file containing metadata for each Run downloaded |\n| `-run-mergers.tsv` | Tab-delimited file merge information from `--group-by-experiment` or `--group-by-sample` |\n| `.fastq.gz` | FASTQ files downloaded from ENA or SRA |\n\n## Example Usage\n\n#### Download FASTQs associated with a Study\n\nSometimes you might be reading a paper and they very kindly provided a Bioproject of all\nthe samples they sequenced. So, you decide you want to download FASTQs for all the samples\nasscociated with the Bioproject. `fastq-dl` can help you with that! \n\n```{bash}\nfastq-dl --accession PRJNA248678 --provider SRA\nfastq-dl --accession PRJNA248678\n```\n\nThe above commands will download the 3 Runs that fall under Study accession [PRJNA248678](https://www.ebi.ac.uk/ena/browser/view/PRJNA248678)\nfrom either SRA (`--provider SRA`) or ENA (without `--provider`).\n\n#### Download FASTQs associated with an Experiment\n\nLet's say instead of the whole Bioproject you just want a single Experiment. You can do\nthat as well.\n\n```{bash}\nfastq-dl --accession SRX477044\n```\n\nThe above command would download the Run accessions from ENA that fall under Experiment SRX477044.\n\nThe relationship of Experiment to Run is a 1-to-many relationship, or there can be many Run accessions\nassociated with a single Experiment Accession (e.g. re-sequencing the same sample). Although in most\ncases, it is a 1-to-1 relationship, you can use `--group-by-experiment` to merge multiple runs\nassociated with an Experiment accession into a single FASTQ file.\n\n#### Download FASTQs associated with an Sample\n\nOk, this time you just want a single Sample, or Biosample.\n\n```{bash}\nfastq-dl --accession SRS1904245 --provider SRA\n```\n\nThe above command would download the Run accessions from SRA that fall under Sample SRS1904245.\n\nSimilar to Experiment accessions, the relationship of Sample to Run is a 1-to-many relationship,\nor there can be many Run accessions associated with a single Sample Accession. Although in most\ncases, it is a 1-to-1 relationship, you can use `--group-by-sample` to merge multiple runs\nassociated with an Sample accession into a single FASTQ file.\n\n_Warning! For some type strains (e.g. S. aureus USA300) a Biosample accession might be associated with\n100s or 1000s of Run accessions. These Runs are likely associated with many different conditions and\nreally should not fall under a single BioSample accession. Please consider this when using\n`--group-by-sample`.\n\n#### Download FASTQs associated with a Run\n\nLet's keep it super simple and just download a Run.\n\n```\nfastq-dl --accession SRR1178105 --provider SRA\n```\n\nThe above command would download the Run SRR1178105 from SRA. Run accessions are the end of the\nline (1-to-1 relationship), so you will always get the expected Run.\n\n## Alternatives\n`fastq-dl`, is a spin-off of [ena-dl](https://github.com/rpetit3/ena-dl), that has been developed for\nusage with [Bactopia](https://github.com/bactopia/bactopia). With this in mind, EBI/NCBI and provide\ntheir own tools ([enaBrowserTools](https://github.com/enasequence/enaBrowserTools) and\n[SRA Toolkit](https://github.com/ncbi/sra-tools)) that offer more extensive access to their databases.\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Download FASTQ files from SRA or ENA repositories.",
"version": "2.0.3",
"project_urls": {
"Homepage": "https://github.com/rpetit3/fastq-dl",
"Repository": "https://github.com/rpetit3/fastq-dl"
},
"split_keywords": [
"bioinformatics",
"fastq",
"download",
"sra",
"ena"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1e517381207e3bc58275af18567d5f40d1161c4c68c2201242294f1cc42098d1",
"md5": "16eed95876ef1ba57390094b3cf37db5",
"sha256": "bfd5daecd07d79268bb7602985944834d032bbd3d654e5e0fa3d98d9c038b2fd"
},
"downloads": -1,
"filename": "fastq_dl-2.0.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "16eed95876ef1ba57390094b3cf37db5",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7.1,<4.0.0",
"size": 13103,
"upload_time": "2023-08-06T22:22:46",
"upload_time_iso_8601": "2023-08-06T22:22:46.035669Z",
"url": "https://files.pythonhosted.org/packages/1e/51/7381207e3bc58275af18567d5f40d1161c4c68c2201242294f1cc42098d1/fastq_dl-2.0.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1d02e589bd29da49c3f04ad38f968ee5b67c9a9f5439bfbda7b94664d8df6ebb",
"md5": "43fd836592f9500f49dbd9a32e9730ba",
"sha256": "2d521d6e4ad2d3fd5683a55b429ec4d237e454fcbe7a512225da65bb3bd2409d"
},
"downloads": -1,
"filename": "fastq_dl-2.0.3.tar.gz",
"has_sig": false,
"md5_digest": "43fd836592f9500f49dbd9a32e9730ba",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7.1,<4.0.0",
"size": 15851,
"upload_time": "2023-08-06T22:22:47",
"upload_time_iso_8601": "2023-08-06T22:22:47.603376Z",
"url": "https://files.pythonhosted.org/packages/1d/02/e589bd29da49c3f04ad38f968ee5b67c9a9f5439bfbda7b94664d8df6ebb/fastq_dl-2.0.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-06 22:22:47",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "rpetit3",
"github_project": "fastq-dl",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "fastq-dl"
}
JSON
