# pairtools
[](http://pairtools.readthedocs.org/en/latest/)
[](https://travis-ci.org/mirnylab/pairtools)
[](https://bit.ly/2UaOpAe)
[](https://doi.org/10.5281/zenodo.1490831)
## Process Hi-C pairs with pairtools
`pairtools` is a simple and fast command-line framework to process sequencing
data from a Hi-C experiment.
`pairtools` process pair-end sequence alignments and perform the following
operations:
- detect ligation junctions (a.k.a. Hi-C pairs) in aligned paired-end sequences of Hi-C DNA molecules
- sort .pairs files for downstream analyses
- detect, tag and remove PCR/optical duplicates
- generate extensive statistics of Hi-C datasets
- select Hi-C pairs given flexibly defined criteria
- restore .sam alignments from Hi-C pairs
- annotate restriction digestion sites
- get the mutated positions in Hi-C pairs
To get started:
- Visit [pairtools tutorials](https://pairtools.readthedocs.io/en/latest/examples/pairtools_walkthrough.html),
- Take a look at a [quick example](https://github.com/open2c/pairtools#quick-example),
- Check out the detailed [documentation](http://pairtools.readthedocs.io).
## Data formats
`pairtools` produce and operate on tab-separated files compliant with the
[.pairs](https://github.com/4dn-dcic/pairix/blob/master/pairs_format_specification.md)
format defined by the [4D Nucleome Consortium](https://www.4dnucleome.org/). All
pairtools properly manage file headers and keep track of the data
processing history.
Additionally, `pairtools` define the [.pairsam format](https://pairtools.readthedocs.io/en/latest/formats.html#pairsam), an extension of .pairs that includes the SAM alignments
of a sequenced Hi-C molecule. .pairsam complies with the .pairs format, and can be processed by any tool that
operates on .pairs files.
`pairtools` produces a set of additional extra columns, which describe properties of alignments, phase, mutations, restriction and complex walks.
The full list of possible extra columns is provided in the [`pairtools` format specification](https://pairtools.readthedocs.io/en/latest/formats.html#extra-columns).
## Installation
Requirements:
- Python 3.x
- Python packages `cython`, `pysam`, `bioframe`, `pyyaml`, `numpy`, `scipy`, `pandas` and `click`.
- Command-line utilities `sort` (the Unix version), `bgzip` (shipped with `samtools`) and `samtools`. If available, `pairtools` can compress outputs with `pbgzip` and `lz4`.
For the full list of recommended versions, see [requirements in the the GitHub repo](https://github.com/open2c/pairtools/blob/detect_mutations/requirements.txt).
We highly recommend using the `conda` package manager to install `pairtools` together with all its dependencies. To get it, you can either install the full [Anaconda](https://www.continuum.io/downloads) Python distribution or just the standalone [conda](http://conda.pydata.org/miniconda.html) package manager.
With `conda`, you can install `pairtools` and all of its dependencies from the [bioconda](https://bioconda.github.io/index.html) channel.
```sh
$ conda install -c conda-forge -c bioconda pairtools
```
Alternatively, install non-Python dependencies and `pairtools` with Python-only dependencies from PyPI using pip:
```sh
$ pip install numpy pysam cython
$ pip install pairtools
```
## Quick example
Setup a new test folder and download a small Hi-C dataset mapped to sacCer3 genome:
```bash
$ mkdir /tmp/test-pairtools
$ cd /tmp/test-pairtools
$ wget https://github.com/open2c/distiller-test-data/raw/master/bam/MATalpha_R1.bam
```
Additionally, we will need a .chromsizes file, a TAB-separated plain text table describing the names, sizes and the order of chromosomes in the genome assembly used during mapping:
```bash
$ wget https://raw.githubusercontent.com/open2c/distiller-test-data/master/genome/sacCer3.reduced.chrom.sizes
```
With `pairtools parse`, we can convert paired-end sequence alignments stored in .sam/.bam format into .pairs, a TAB-separated table of Hi-C ligation junctions:
```bash
$ pairtools parse -c sacCer3.reduced.chrom.sizes -o MATalpha_R1.pairs.gz --drop-sam MATalpha_R1.bam
```
Inspect the resulting table:
```bash
$ less MATalpha_R1.pairs.gz
```
## Pipelines
- We provide a simple working example of a mapping bash pipeline in /examples/.
- [distiller](https://github.com/open2c/distiller-nf) is a powerful
Hi-C data analysis workflow, based on `pairtools` and [nextflow](https://www.nextflow.io/).
## Tools
- `parse`: read .sam/.bam files produced by bwa and form Hi-C pairs
- form Hi-C pairs by reporting the outer-most mapped positions and the strand
on the either side of each molecule;
- report unmapped/multimapped (ambiguous alignments)/chimeric alignments as
chromosome "!", position 0, strand "-";
- perform upper-triangular flipping of the sides of Hi-C molecules
such that the first side has a lower sorting index than the second side;
- form hybrid pairsam output, where each line contains all available data
for one Hi-C molecule (outer-most mapped positions on the either side,
read ID, pair type, and .sam entries for each alignment);
- report .sam tags or mutations of the alignments;
- print the .sam header as #-comment lines at the start of the file.
- `parse2`: read .sam/.bam files with long paired-and or single-end reads and form Hi-C pairs from complex walks
- identify and rescue chrimeric alignments produced by singly-ligated Hi-C
molecules with a sequenced ligation junction on one of the sides;
- annotate chimeric alignments by restriction fragments and report true junctions and hops (One-Read-Based Interactions Annotation, ORBITA);
- perform intra-molecule deduplication of paired-end data when one side reads through the DNA on the other side of the read;
- report index of the pair in the complex walk;
- make combinatorial expansion of pairs produced from the same walk;
- `sort`: sort pairs files (the lexicographic order for chromosomes,
the numeric order for the positions, the lexicographic order for pair types).
- `merge`: merge sorted .pairs files
- merge sort .pairs;
- combine the .pairs headers from all input files;
- check that each .pairs file was mapped to the same reference genome index
(by checking the identity of the @SQ sam header lines).
- `select`: select pairs according to specified criteria
- select pairs entries according to the provided condition. A programmable
interface allows for arbitrarily complex queries on specific pair types,
chromosomes, positions, strands, read IDs (including matches to a
wildcard/regexp/list).
- optionally print the non-matching entries into a separate file.
- `dedup`: remove PCR duplicates from a sorted triu-flipped .pairs file
- remove PCR duplicates by finding pairs of entries with both sides mapped
to similar genomic locations (+/- N bp);
- optionally output the PCR duplicate entries into a separate file;
- detect optical duplicates from the original Illumina read ids;
- apply filtering by various properties of pairs (MAPQ; orientation; distance) together with deduplication;
- output yaml or convenient tsv deduplication stats into text file.
- NOTE: in order to remove all PCR duplicates, the input must contain \*all\*
mapped read pairs from a single experimental replicate;
- `maskasdup`: mark all pairs in a pairsam as Hi-C duplicates
- change the field pair_type to DD;
- change the pair_type tag (Yt:Z:) for all sam alignments;
- set the PCR duplicate binary flag for all sam alignments (0x400).
- `split`: split a .pairsam file into .pairs and .sam.
- `flip`: flip pairs to get an upper-triangular matrix
- `header`: manipulate the .pairs/.pairsam header
- generate new header for headerless .pairs file
- transfer header from one .pairs file to another
- set column names for the .pairs file
- validate that the header corresponds to the information stored in .pairs file
- `stats`: calculate various statistics of .pairs files
- `restrict`: identify the span of the restriction fragment forming a Hi-C junction
- `phase`: phase pairs mapped to a diploid genome
## Contributing
[Pull requests](https://akrabat.com/the-beginners-guide-to-contributing-to-a-github-project/) are welcome.
For development, clone and install in "editable" (i.e. development) mode with the `-e` option. This way you can also pull changes on the fly.
```sh
$ git clone https://github.com/open2c/pairtools.git
$ cd pairtools
$ pip install -e .
```
## Citing `pairtools`
Open2C*, Nezar Abdennur, Geoffrey Fudenberg, Ilya M. Flyamer, Aleksandra A. Galitsyna*, Anton Goloborodko*, Maxim Imakaev, Sergey V. Venev. "Pairtools: from sequencing data to chromosome contacts" bioRxiv, February 13, 2023. ; doi: https://doi.org/10.1101/2023.02.13.528389
## License
MIT
Raw data
{
"_id": null,
"home_page": "https://github.com/open2c/pairtools",
"name": "pairtools",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": "genomics, bioinformatics, Hi-C, contact",
"author": "Open2C",
"author_email": "open.chromosome.collective@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/c7/d0/dae9ee69ff357b65dc1be70554d76601d19bba274ba85d11349e669a2638/pairtools-1.1.0.tar.gz",
"platform": null,
"description": "# pairtools\n\n[](http://pairtools.readthedocs.org/en/latest/)\n[](https://travis-ci.org/mirnylab/pairtools)\n[](https://bit.ly/2UaOpAe)\n[](https://doi.org/10.5281/zenodo.1490831)\n\n## Process Hi-C pairs with pairtools\n\n`pairtools` is a simple and fast command-line framework to process sequencing\ndata from a Hi-C experiment.\n\n`pairtools` process pair-end sequence alignments and perform the following\noperations:\n\n- detect ligation junctions (a.k.a. Hi-C pairs) in aligned paired-end sequences of Hi-C DNA molecules\n- sort .pairs files for downstream analyses\n- detect, tag and remove PCR/optical duplicates \n- generate extensive statistics of Hi-C datasets\n- select Hi-C pairs given flexibly defined criteria\n- restore .sam alignments from Hi-C pairs\n- annotate restriction digestion sites\n- get the mutated positions in Hi-C pairs\n\nTo get started:\n- Visit [pairtools tutorials](https://pairtools.readthedocs.io/en/latest/examples/pairtools_walkthrough.html),\n- Take a look at a [quick example](https://github.com/open2c/pairtools#quick-example),\n- Check out the detailed [documentation](http://pairtools.readthedocs.io).\n\n## Data formats\n\n`pairtools` produce and operate on tab-separated files compliant with the\n[.pairs](https://github.com/4dn-dcic/pairix/blob/master/pairs_format_specification.md) \nformat defined by the [4D Nucleome Consortium](https://www.4dnucleome.org/). All\npairtools properly manage file headers and keep track of the data\nprocessing history.\n\nAdditionally, `pairtools` define the [.pairsam format](https://pairtools.readthedocs.io/en/latest/formats.html#pairsam), an extension of .pairs that includes the SAM alignments \nof a sequenced Hi-C molecule. .pairsam complies with the .pairs format, and can be processed by any tool that\noperates on .pairs files.\n\n`pairtools` produces a set of additional extra columns, which describe properties of alignments, phase, mutations, restriction and complex walks.\nThe full list of possible extra columns is provided in the [`pairtools` format specification](https://pairtools.readthedocs.io/en/latest/formats.html#extra-columns). \n\n## Installation\n\nRequirements:\n\n- Python 3.x\n- Python packages `cython`, `pysam`, `bioframe`, `pyyaml`, `numpy`, `scipy`, `pandas` and `click`.\n- Command-line utilities `sort` (the Unix version), `bgzip` (shipped with `samtools`) and `samtools`. If available, `pairtools` can compress outputs with `pbgzip` and `lz4`.\n\nFor the full list of recommended versions, see [requirements in the the GitHub repo](https://github.com/open2c/pairtools/blob/detect_mutations/requirements.txt). \n\nWe highly recommend using the `conda` package manager to install `pairtools` together with all its dependencies. To get it, you can either install the full [Anaconda](https://www.continuum.io/downloads) Python distribution or just the standalone [conda](http://conda.pydata.org/miniconda.html) package manager.\n\nWith `conda`, you can install `pairtools` and all of its dependencies from the [bioconda](https://bioconda.github.io/index.html) channel.\n```sh\n$ conda install -c conda-forge -c bioconda pairtools\n```\n\nAlternatively, install non-Python dependencies and `pairtools` with Python-only dependencies from PyPI using pip:\n```sh\n$ pip install numpy pysam cython\n$ pip install pairtools\n```\n\n## Quick example\n\nSetup a new test folder and download a small Hi-C dataset mapped to sacCer3 genome:\n```bash\n$ mkdir /tmp/test-pairtools\n$ cd /tmp/test-pairtools\n$ wget https://github.com/open2c/distiller-test-data/raw/master/bam/MATalpha_R1.bam\n```\n\nAdditionally, we will need a .chromsizes file, a TAB-separated plain text table describing the names, sizes and the order of chromosomes in the genome assembly used during mapping:\n```bash\n$ wget https://raw.githubusercontent.com/open2c/distiller-test-data/master/genome/sacCer3.reduced.chrom.sizes\n```\n\nWith `pairtools parse`, we can convert paired-end sequence alignments stored in .sam/.bam format into .pairs, a TAB-separated table of Hi-C ligation junctions:\n\n```bash\n$ pairtools parse -c sacCer3.reduced.chrom.sizes -o MATalpha_R1.pairs.gz --drop-sam MATalpha_R1.bam \n```\n\nInspect the resulting table:\n\n```bash\n$ less MATalpha_R1.pairs.gz\n```\n\n## Pipelines\n\n- We provide a simple working example of a mapping bash pipeline in /examples/.\n- [distiller](https://github.com/open2c/distiller-nf) is a powerful\nHi-C data analysis workflow, based on `pairtools` and [nextflow](https://www.nextflow.io/).\n\n\n## Tools\n\n- `parse`: read .sam/.bam files produced by bwa and form Hi-C pairs\n - form Hi-C pairs by reporting the outer-most mapped positions and the strand\n on the either side of each molecule;\n - report unmapped/multimapped (ambiguous alignments)/chimeric alignments as\n chromosome \"!\", position 0, strand \"-\";\n - perform upper-triangular flipping of the sides of Hi-C molecules \n such that the first side has a lower sorting index than the second side;\n - form hybrid pairsam output, where each line contains all available data \n for one Hi-C molecule (outer-most mapped positions on the either side, \n read ID, pair type, and .sam entries for each alignment);\n - report .sam tags or mutations of the alignments;\n - print the .sam header as #-comment lines at the start of the file.\n\n- `parse2`: read .sam/.bam files with long paired-and or single-end reads and form Hi-C pairs from complex walks \n - identify and rescue chrimeric alignments produced by singly-ligated Hi-C \n molecules with a sequenced ligation junction on one of the sides;\n - annotate chimeric alignments by restriction fragments and report true junctions and hops (One-Read-Based Interactions Annotation, ORBITA);\n - perform intra-molecule deduplication of paired-end data when one side reads through the DNA on the other side of the read;\n - report index of the pair in the complex walk;\n - make combinatorial expansion of pairs produced from the same walk; \n\n- `sort`: sort pairs files (the lexicographic order for chromosomes, \n the numeric order for the positions, the lexicographic order for pair types).\n\n- `merge`: merge sorted .pairs files\n - merge sort .pairs;\n - combine the .pairs headers from all input files;\n - check that each .pairs file was mapped to the same reference genome index \n (by checking the identity of the @SQ sam header lines).\n\n- `select`: select pairs according to specified criteria\n - select pairs entries according to the provided condition. A programmable\n interface allows for arbitrarily complex queries on specific pair types, \n chromosomes, positions, strands, read IDs (including matches to a\n wildcard/regexp/list).\n - optionally print the non-matching entries into a separate file.\n\n- `dedup`: remove PCR duplicates from a sorted triu-flipped .pairs file\n - remove PCR duplicates by finding pairs of entries with both sides mapped\n to similar genomic locations (+/- N bp);\n - optionally output the PCR duplicate entries into a separate file;\n - detect optical duplicates from the original Illumina read ids;\n - apply filtering by various properties of pairs (MAPQ; orientation; distance) together with deduplication; \n - output yaml or convenient tsv deduplication stats into text file.\n - NOTE: in order to remove all PCR duplicates, the input must contain \\*all\\* \n mapped read pairs from a single experimental replicate;\n\n- `maskasdup`: mark all pairs in a pairsam as Hi-C duplicates\n - change the field pair_type to DD;\n - change the pair_type tag (Yt:Z:) for all sam alignments;\n - set the PCR duplicate binary flag for all sam alignments (0x400).\n\n- `split`: split a .pairsam file into .pairs and .sam.\n\n- `flip`: flip pairs to get an upper-triangular matrix\n\n- `header`: manipulate the .pairs/.pairsam header\n - generate new header for headerless .pairs file\n - transfer header from one .pairs file to another\n - set column names for the .pairs file\n - validate that the header corresponds to the information stored in .pairs file\n\n- `stats`: calculate various statistics of .pairs files\n\n- `restrict`: identify the span of the restriction fragment forming a Hi-C junction\n\n- `phase`: phase pairs mapped to a diploid genome \n\n## Contributing\n\n[Pull requests](https://akrabat.com/the-beginners-guide-to-contributing-to-a-github-project/) are welcome.\n\nFor development, clone and install in \"editable\" (i.e. development) mode with the `-e` option. This way you can also pull changes on the fly.\n```sh\n$ git clone https://github.com/open2c/pairtools.git\n$ cd pairtools\n$ pip install -e .\n```\n\n## Citing `pairtools`\n\nOpen2C*, Nezar Abdennur, Geoffrey Fudenberg, Ilya M. Flyamer, Aleksandra A. Galitsyna*, Anton Goloborodko*, Maxim Imakaev, Sergey V. Venev. \"Pairtools: from sequencing data to chromosome contacts\" bioRxiv, February 13, 2023. ; doi: https://doi.org/10.1101/2023.02.13.528389\n\n## License\n\nMIT\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "CLI tools to process mapped Hi-C data",
"version": "1.1.0",
"project_urls": {
"Homepage": "https://github.com/open2c/pairtools"
},
"split_keywords": [
"genomics",
" bioinformatics",
" hi-c",
" contact"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "63ac65258bb981b0687eee08c4f61ff235bb91e581e4d219c7473cd51b15dc38",
"md5": "22e404edc8a8b00c79ba5a3340a1c61d",
"sha256": "7794748d3c5497bddc4e3bbbbb33f35b51aaad0a8653b966f78f8e8ccca06daa"
},
"downloads": -1,
"filename": "pairtools-1.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "22e404edc8a8b00c79ba5a3340a1c61d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 98130,
"upload_time": "2024-04-23T19:21:35",
"upload_time_iso_8601": "2024-04-23T19:21:35.321348Z",
"url": "https://files.pythonhosted.org/packages/63/ac/65258bb981b0687eee08c4f61ff235bb91e581e4d219c7473cd51b15dc38/pairtools-1.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c7d0dae9ee69ff357b65dc1be70554d76601d19bba274ba85d11349e669a2638",
"md5": "e495a972ac543456f5ba5305a02168ef",
"sha256": "3f265ebcd50fb07a36199a99c58f6b17810b48bacf2d5f3096453319c89bd522"
},
"downloads": -1,
"filename": "pairtools-1.1.0.tar.gz",
"has_sig": false,
"md5_digest": "e495a972ac543456f5ba5305a02168ef",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 1424083,
"upload_time": "2024-04-23T19:21:37",
"upload_time_iso_8601": "2024-04-23T19:21:37.283646Z",
"url": "https://files.pythonhosted.org/packages/c7/d0/dae9ee69ff357b65dc1be70554d76601d19bba274ba85d11349e669a2638/pairtools-1.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-23 19:21:37",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "open2c",
"github_project": "pairtools",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "cython",
"specs": []
},
{
"name": "numpy",
"specs": [
[
">=",
"1.10"
]
]
},
{
"name": "click",
"specs": [
[
">=",
"6.6"
]
]
},
{
"name": "scipy",
"specs": [
[
">=",
"1.7.0"
]
]
},
{
"name": "pandas",
"specs": [
[
">=",
"1.3.4"
]
]
},
{
"name": "pysam",
"specs": [
[
">=",
"0.15.0"
]
]
},
{
"name": "pyyaml",
"specs": []
},
{
"name": "bioframe",
"specs": [
[
">=",
"0.3.3"
]
]
}
],
"lcname": "pairtools"
}
JSON
