| Name | kmerdb JSON |
| Version |
0.8.10
JSON |
| download |
| home_page | https://github.com/MatthewRalston/kmerdb |
| Summary | Yet another correction to the 'yet another correction to just a k-mer counter...' |
| upload_time | 2024-10-12 23:05:45 |
| maintainer | None |
| docs_url | None |
| author | Matt Ralston |
| requires_python | >=3.12.2 |
| license | Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright 2020 Matthew Ralston Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. |
| keywords |
bioinformatics
fastq
fasta
k-mer
kmer
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# README - kmerdb
> Python CLI and module for k-mer profiles, similarities, and graph files
NOTE: Beta-stage `.bgzf` and `zlib` compatible k-mer count vectors and DeBruijn graph edge-list formats.
## Development Status
[](https://pypi.org/project/kmerdb)
[](https://github.com/MatthewRalston/kmerdb/releases)
[][pip]
[][Pythons]
[](https://dl.circleci.com/status-badge/redirect/circleci/TxK1S2m7siJCSY89Dc6s4A/Dm3xDervECRDhDYKUkgUJN/tree/main)
[](https://codecov.io/gh/MatthewRalston/kmerdb)
[][RTD]
[pip]: https://pypi.org/project/kmerdb/
[Pythons]: https://pypi.org/project/kmerdb/
[RTD]: https://kmerdb.readthedocs.io/en/latest/
[/]: https://kdb.info
[-]: ...
## Summary
`kmerdb` is a Python CLI designed for k-mer counting and k-mer graph edge-lists. It addresses the ['k-mer' problem](https://en.wikipedia.org/wiki/K-mer) (substrings of length k) in a simple and performant manner. It stores the k-mer counts in a columnar format (input checksums, total and unique k-mer counts, nullomers, mononucleotide counts) with a YAML formatted metadata header in the first block of a `bgzf` formatted file.
- [ x ] [Homepage:](https://matthewralston.github.io/kmerdb)
- [ x ] [Quick Start guide](https://matthewralston.github.io/kmerdb/quickstart)
- [ x ] `kmerdb usage subcommand_name`
- `profile` - Make k-mer count vectors/profiles, calculate unique k-mer counts, total k-mer counts, nullomer counts. Import to read/write NumPy arrays from profile object attributes.
- `graph` - Make a weighted edge list of kmer-to-kmer relationships, akin to a De Bruijn graph.
- `usage` - Display verbose input file/parameter and algorithm details of subcommands.
- `help` - Display verbose input file/parameter and algorithm details of subcommands.
- `view` - View .tsv count/frequency vectors with/without preamble.
- `header` - View YAML formatted header and aggregate counts
- `matrix` - Collate multiple profiles into a count matrix for dimensionality reduction, etc.
- `kmeans` - k-means clustering on a distance matrix via Scikit-learn or BioPython with kcluster distances
- `hierarchical` - hierarchical clustering on a distance matrix via BioPython with linkage choices
- `distance` - Distance matrices (from kmer count matrices) including SciPy distances, a Pearson correlation coefficient implemented in Cython, and Spearman rank correlation included as additional distances.
- `index` - Create an index file for the kmer profile (Delayed:)
- `shuf` - Shuffle a k-mer count vector/profile (Delayed:)
- `version` - Display kmerdb version number
- `citation` - Silence citation suggestion
- [ x ] `kmerdb subcommand -h|--help`
k-mer counts from .fa(.gz)/.fq(.gz) sequence data can be computed and stored for access to metadata and count aggregation faculties. For those familiar with `.bam`, a `view` and `header` functions are provided. This file is compatible with `zlib`.
Install with `pip install kmerdb`
Please see the [Quickstart guide](https://matthewralston.github.io/kmerdb/quickstart) for more information about the format, the library, and the project.
## Usage
```bash
# Usage --help option --debug mode
kmerdb --help # [+ --debug mode]
kmerdb usage profile
# +
# [ 3 main features: ] [ 1. - k-mer counts ]
# Create a [composite] profile of k-mer counts from sequence files. (.fasta|.fastq|.fa.gz|.fq.gz)
kmerdb profile -vv -k 8 --output-name sample_1 sample_1_rep1.fq.gz [sample_1_rep2.fq.gz]
# Creates k-mer count vector/profile in sample_1.8.kdb. This is the input to other steps, including count matrix aggregation. --minK and --maxK options can be specified to create multiple k-mer profiles at once.
<!-- # Alternatively, can also take a plain-text samplesheet.txt with one filepath on each line. -->
# De Bruijn graphs (not a main feature yet, delayed)
# Build a weighted edge list (+ node ids/counts = De Bruijn graph)
kmerdb graph -vv -k 12 example_1.fq.gz example_2.fq.gz edges_1.kdbg
# View k-mer count vector
kmerdb view -vv profile_1.8.kdb # -H for full header
# Note: zlib compatibility
#zcat profile_1.8.kdb
# View header (config.py[kdb_metadata_schema#L84])
kmerdb header -vv profile_1.8.kdb
## [ 3 main features: ] [ 2. Optional normalization, PCA/tSNE, and distance metrics ]
# K-mer count matrix - Cython Pearson coefficient of correlation [ ssxy/sqrt(ssxx*ssyy) ]
kmerdb matrix -vv from *.8.kdb | kmerdb distance pearson STDIN
#
# kmerdb matrix -vv DESeq2 *.8.kdb
# kmerdb matrix -vv PCA *.8.kdb
# kmerdb matrix -vv tSNE *.8.kdb
# # <from> just makes a k-mer count matrix from k-mer count vectors.
#
# Distances on count matrices [ SciPy ] pdists + [ Cython ] Pearson correlation, scipy Spearman and scipy correlation pdist calculations are available ]
kmerdb distance -h
#
#
# usage: kmerdb distance [-h] [-v] [--debug] [-l LOG_FILE] [--output-delimiter OUTPUT_DELIMITER] [-p PARALLEL] [--column-names COLUMN_NAMES] [--delimiter DELIMITER] [-k K]
# {braycurtis,canberra,chebyshev,cityblock,correlation,cosine,dice,euclidean,hamming,jaccard,jensenshannon,kulsinski,mahalanobis,matching,minkowski,pearson,rogerstanimotorusselrao,seuclidean,sokalmichener,sokalsneath,spearman,sqeuclidean,yule} [<kdbfile1 kdbfile2 ...|input.tsv|STDIN> ...]
# [ 3 main features: ] [ 3. Clustering: k-means and hierarchical with matplotlib ]
# Kmeans (sklearn, BioPython)
kmerdb kmeans -vv -k 4 -i dist.tsv
# BioPython Phylip tree + upgma
kmerdb hierarchical -vv -i dist.tsv
```
## Usage example
## Installation
### OSX and Linux release:
```sh
pip install kmerdb
```
#### Optional DESeq2 normalization
DESeq2 is an optional R dependency for rpy2-mediated normalization. Make sure development libraries are installed from the repository.
```
pip install -r requirements-dev.txt
```
Next, install DESeq2 via bioconductor.
```r
if (!requireNamespace("BiocManager", quietly = TRUE))
install.packages("BiocManager")
BiocManager::install("DESeq2")
```
## IUPAC support:
```bash
kmerdb profile -k $k -o output input.fa # This simply discards non-IUPAC characters.
```
IUPAC residues (ATCG+RYSWKM+BDHV) are kept throughout the k-mer counting. But non-IUPAC residues (N) and characters are trimmed from the sequences prior to k-mer counting. Non-standard IUPAC residues are counted as doublets or triplets.
## Documentation
Check out the [main webpage](https://matthewralston.github.io/kmerdb) and the [Readthedocs documentation](https://kdb.readthedocs.io/en/stable/), with examples and descriptions of the module usage.
Important features to usage that may be important may not be fully documented as the project is in beta.
For example, the IUPAC treatment is largely custom, and does the sensible thing when ambiguous bases are found in fasta files, but it could use some polishing. For example, the '`N`' residue rejection creates gaps in the k-mer profile from the real dataset by admittedly ommitting certain k-mer counts.
This is one method for counting k-mers and handling ambiguity. Fork it and play with it a bit.
Also, the parallel handling may not always be smooth, if you're trying to load dozens of 12+ mer profiles into memory. This would especially matter in the matrix command, before the matrix is generated. You can use single-core if your machine can't collate that much into main memory at once, depending on how deep the fastq dataset is. Even when handling small-ish k-mer profiles, you may bump into memory overheads rather quickly.
Besides that, I'd suggest reading the source, the differente elements of the [main page](https://matthewralston.github.io/kmerdb) or the [RTD documentation](https://kdb.readthedocs.io/en/stable/).
## Minimum-viable product Documentation
### Problem statement:
Calculate relevant metadata from k-mer profiles, manipulate count matrices and distance matrices on inter-species inter-profile distances, perform exploartory analysis using tools such as UPGMA hierarchical clustering, k-means clustering, PCA, and others.
Currently unsupported but targeted features include regression modeling, strassen multiplication, NumPy CPU multipication option, least squares and associated regression code.
Currently in-progress is the application note. I wouldn't mind ironing out a few more components of the program for the application note.
### Target audience:
Biologists and researchers using metagenomics and genomics tools for the assessment of inter-genomic distances, k-mer counts, nullomer counts, unique counts, and other metadata.
### Feature prioritization:
Features currently implemented in `kmerdb` include:
- [x] k-mer counting and profile generation
- [x] multiplexed profiles
- [x] De Bruijn graph structures
- [x] Count aggergation and normalization (DESeq2)
- [x] PCA
- [x] t-Stochastic Neighbor Embedding
- [x] Cython Pearson correlation coefficient
- [x] Other correlation coefficients (Spearman, Pearson via scipy)
- [x] hierarchical clustering
- [x] k-means clustering
- [x] data pipelining (Unix pipelines)
Features currently under consideration include:
- [ ] regression modeling for metagenomnic populations
- [ ] alternative k-mer count vector distances
- [ ] alignment using k-mers as seed regions
- [ ] De Bruijn graph traversals and contig generation using DBA (De Bruijn graph) assembly
### Success Criteria:
### Technical specifications:
### User stories:
### Development timeline:
- 2/24 - 6/24 v0.7.7+ - appmap, De Bruijn parser, other updates
- 10/23 -
### Testing and quality assurance plan:
### Feedback collection strategy:
## Development
https://matthewralston.github.io/kmerdb/developing
```bash
python setup.py test
```
## License
Created by Matthew Ralston - [Scientist, Programmer, Musician](http://matthewralston.github.io) - [Email](mailto:mralston.development@gmail.com)
Distributed under the Apache license. See `LICENSE.txt` for the copy distributed with this project. Open source software is not for everyone, and im the author and maintainer. cheers, on me. You may use and distribute this software, gratis, so long as the original LICENSE.txt is distributed along with the software. This software is distributed AS IS and provides no warranties of any kind.
```
Copyright 2020 Matthew Ralston
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
```
## Contributing
1. Fork it (<https://github.com/MatthewRalston/kmerdb/fork>)
2. Create your feature branch (`git checkout -b feature/fooBar`)
3. Commit your changes (`git commit -am 'Add some fooBar'`)
4. Push to the branch (`git push origin feature/fooBar`)
5. Create a new Pull Request
## Acknowledgements
Thanks mom and dad and my hometown, awesome hs, and the University of Delaware faculty for support and encouragement. Thanks to my former mentors, bosses, and coworkers. It's been really enjoyable anticipating what the metagenomics community might want from a tool that can handle microbial k-mers well.
Thank you to the authors of kPAL and Jellyfish for the inspiration and bit shifting trick. And thank you to others for the encouragement along the way, who shall remain nameless.
The intention is that more developers would want to add functionality to the codebase or even just utilize things downstream, but to build out directly with numpy and scipy/scikit as needed to suggest the basic infrastructure for the ML problems and modeling approaches that could be applied to such datasets. This project began under GPL v3.0 and was relicensed with Apache v2. Hopefully this project could gain some interest. I have so much fun working on this project. There's more to it than meets the eye. I'm working on a preprint, and the draft is included in some of the latest versions of the codebase, specifically .Rmd files.
More on the flip-side. It's so complex with technology these days...
<!--
Thanks of course to my fans (and haters). Yeah i see you.... but i dont.
Thanks to my former mentors BC, MR, IN, CR, and my newer bosses PJ and KL.
Thanks to the Pap lab for the first dataset that I continue to use.
Thank you to Ryan for the food and stuff. I actually made this project specifically so you and I could converse...
Thanks to Blahah for tolerating someone snooping and imitating his Ruby style.
Thanks to Erin for getting my feet wet in this new field. You are my mvp.
Thanks to Rachel for the good memories and friendship. And Sophie too. veggies n' R love.
Thanks to Yasmeen for the usual banter.
Thanks to A for the newer banter.
Thanks to Max, Robin, and Robert for the good memories in St. Louis. What's new?
Thanks to Fred for the good memories. Hope you're on soon.
Thanks to Nichole for the cookies and good memories. And your cute furballs too! Hope you're well
Thanks to S for the lessons, convos, and even embarassing moments. You're kind of awesome to me.
Thanks to a few friends I met in 2023 that reminded me I have a lot to learn about friendship, dating, and street smarts.
Thanks to them even more now that I got it xd up.
Thanks to the people of NCC for the Doordash money. It might not be much but I don't have it twisted (I do.)
Thanks to D from BCCS.
Thanks to C4H&H. I'm 'healthier' now, but I really still think I need more support than just BCCS. it's urgent.
Thanks to CT and family. Your love and support means the world to me.
Thanks to AS and family. Your support made a difference. Praying for better employment and opportunities.
And thanks to my family and friends.
Go Blue Hens
-->
Raw data
{
"_id": null,
"home_page": "https://github.com/MatthewRalston/kmerdb",
"name": "kmerdb",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.12.2",
"maintainer_email": null,
"keywords": "bioinformatics, fastq, fasta, k-mer, kmer",
"author": "Matt Ralston",
"author_email": "\"Matt Ralston <mralston.development@gmail.com>\" <mralston.development@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/a4/6f/bf93de2c0195620c641fdea10950ac989365adc9a045442354051ee84500/kmerdb-0.8.10.tar.gz",
"platform": null,
"description": "# README - kmerdb\n> Python CLI and module for k-mer profiles, similarities, and graph files\n\nNOTE: Beta-stage `.bgzf` and `zlib` compatible k-mer count vectors and DeBruijn graph edge-list formats.\n\n## Development Status\n[](https://pypi.org/project/kmerdb)\n\n[](https://github.com/MatthewRalston/kmerdb/releases)\n[][pip]\n[][Pythons]\n[](https://dl.circleci.com/status-badge/redirect/circleci/TxK1S2m7siJCSY89Dc6s4A/Dm3xDervECRDhDYKUkgUJN/tree/main)\n[](https://codecov.io/gh/MatthewRalston/kmerdb)\n[][RTD]\n\n\n[pip]: https://pypi.org/project/kmerdb/\n[Pythons]: https://pypi.org/project/kmerdb/\n[RTD]: https://kmerdb.readthedocs.io/en/latest/\n[/]: https://kdb.info\n[-]: ...\n\n\n\n## Summary \n\n`kmerdb` is a Python CLI designed for k-mer counting and k-mer graph edge-lists. It addresses the ['k-mer' problem](https://en.wikipedia.org/wiki/K-mer) (substrings of length k) in a simple and performant manner. It stores the k-mer counts in a columnar format (input checksums, total and unique k-mer counts, nullomers, mononucleotide counts) with a YAML formatted metadata header in the first block of a `bgzf` formatted file. \n\n- [ x ] [Homepage:](https://matthewralston.github.io/kmerdb)\n- [ x ] [Quick Start guide](https://matthewralston.github.io/kmerdb/quickstart)\n- [ x ] `kmerdb usage subcommand_name`\n - `profile` - Make k-mer count vectors/profiles, calculate unique k-mer counts, total k-mer counts, nullomer counts. Import to read/write NumPy arrays from profile object attributes.\n - `graph` - Make a weighted edge list of kmer-to-kmer relationships, akin to a De Bruijn graph.\n - `usage` - Display verbose input file/parameter and algorithm details of subcommands.\n - `help` - Display verbose input file/parameter and algorithm details of subcommands.\n - `view` - View .tsv count/frequency vectors with/without preamble.\n - `header` - View YAML formatted header and aggregate counts\n - `matrix` - Collate multiple profiles into a count matrix for dimensionality reduction, etc.\n - `kmeans` - k-means clustering on a distance matrix via Scikit-learn or BioPython with kcluster distances\n - `hierarchical` - hierarchical clustering on a distance matrix via BioPython with linkage choices\n - `distance` - Distance matrices (from kmer count matrices) including SciPy distances, a Pearson correlation coefficient implemented in Cython, and Spearman rank correlation included as additional distances.\n - `index` - Create an index file for the kmer profile (Delayed:)\n - `shuf` - Shuffle a k-mer count vector/profile (Delayed:)\n - `version` - Display kmerdb version number\n - `citation` - Silence citation suggestion\n- [ x ] `kmerdb subcommand -h|--help`\n\n\nk-mer counts from .fa(.gz)/.fq(.gz) sequence data can be computed and stored for access to metadata and count aggregation faculties. For those familiar with `.bam`, a `view` and `header` functions are provided. This file is compatible with `zlib`.\n\nInstall with `pip install kmerdb`\n\n\n\nPlease see the [Quickstart guide](https://matthewralston.github.io/kmerdb/quickstart) for more information about the format, the library, and the project.\n\n\n## Usage\n\n```bash\n# Usage --help option --debug mode\nkmerdb --help # [+ --debug mode]\nkmerdb usage profile\n\n\n# +\n\n# [ 3 main features: ] [ 1. - k-mer counts ]\n\n# Create a [composite] profile of k-mer counts from sequence files. (.fasta|.fastq|.fa.gz|.fq.gz)\nkmerdb profile -vv -k 8 --output-name sample_1 sample_1_rep1.fq.gz [sample_1_rep2.fq.gz]\n# Creates k-mer count vector/profile in sample_1.8.kdb. This is the input to other steps, including count matrix aggregation. --minK and --maxK options can be specified to create multiple k-mer profiles at once.\n<!-- # Alternatively, can also take a plain-text samplesheet.txt with one filepath on each line. -->\n\n# De Bruijn graphs (not a main feature yet, delayed)\n# Build a weighted edge list (+ node ids/counts = De Bruijn graph)\nkmerdb graph -vv -k 12 example_1.fq.gz example_2.fq.gz edges_1.kdbg\n\n# View k-mer count vector\nkmerdb view -vv profile_1.8.kdb # -H for full header\n\n# Note: zlib compatibility\n#zcat profile_1.8.kdb\n\n# View header (config.py[kdb_metadata_schema#L84])\nkmerdb header -vv profile_1.8.kdb\n\n## [ 3 main features: ] [ 2. Optional normalization, PCA/tSNE, and distance metrics ]\n\n# K-mer count matrix - Cython Pearson coefficient of correlation [ ssxy/sqrt(ssxx*ssyy) ]\nkmerdb matrix -vv from *.8.kdb | kmerdb distance pearson STDIN\n# \n# kmerdb matrix -vv DESeq2 *.8.kdb\n# kmerdb matrix -vv PCA *.8.kdb\n# kmerdb matrix -vv tSNE *.8.kdb\n# # <from> just makes a k-mer count matrix from k-mer count vectors.\n# \n\n# Distances on count matrices [ SciPy ] pdists + [ Cython ] Pearson correlation, scipy Spearman and scipy correlation pdist calculations are available ]\nkmerdb distance -h\n# \n#\n# usage: kmerdb distance [-h] [-v] [--debug] [-l LOG_FILE] [--output-delimiter OUTPUT_DELIMITER] [-p PARALLEL] [--column-names COLUMN_NAMES] [--delimiter DELIMITER] [-k K]\n# {braycurtis,canberra,chebyshev,cityblock,correlation,cosine,dice,euclidean,hamming,jaccard,jensenshannon,kulsinski,mahalanobis,matching,minkowski,pearson,rogerstanimotorusselrao,seuclidean,sokalmichener,sokalsneath,spearman,sqeuclidean,yule} [<kdbfile1 kdbfile2 ...|input.tsv|STDIN> ...]\n\n# [ 3 main features: ] [ 3. Clustering: k-means and hierarchical with matplotlib ]\n\n# Kmeans (sklearn, BioPython)\nkmerdb kmeans -vv -k 4 -i dist.tsv\n# BioPython Phylip tree + upgma\nkmerdb hierarchical -vv -i dist.tsv\n\n\n```\n\n\n\n\n## Usage example\n\n\n\n\n\n\n## Installation\n\n### OSX and Linux release:\n\n```sh\npip install kmerdb\n```\n\n#### Optional DESeq2 normalization\n\nDESeq2 is an optional R dependency for rpy2-mediated normalization. Make sure development libraries are installed from the repository.\n\n\n```\npip install -r requirements-dev.txt\n```\nNext, install DESeq2 via bioconductor.\n```r\nif (!requireNamespace(\"BiocManager\", quietly = TRUE))\n install.packages(\"BiocManager\")\n\nBiocManager::install(\"DESeq2\")\n```\n\n\n## IUPAC support:\n\n```bash\nkmerdb profile -k $k -o output input.fa # This simply discards non-IUPAC characters.\n```\nIUPAC residues (ATCG+RYSWKM+BDHV) are kept throughout the k-mer counting. But non-IUPAC residues (N) and characters are trimmed from the sequences prior to k-mer counting. Non-standard IUPAC residues are counted as doublets or triplets.\n\n\n## Documentation\n\nCheck out the [main webpage](https://matthewralston.github.io/kmerdb) and the [Readthedocs documentation](https://kdb.readthedocs.io/en/stable/), with examples and descriptions of the module usage.\n\nImportant features to usage that may be important may not be fully documented as the project is in beta.\n\nFor example, the IUPAC treatment is largely custom, and does the sensible thing when ambiguous bases are found in fasta files, but it could use some polishing. For example, the '`N`' residue rejection creates gaps in the k-mer profile from the real dataset by admittedly ommitting certain k-mer counts.\nThis is one method for counting k-mers and handling ambiguity. Fork it and play with it a bit.\n\nAlso, the parallel handling may not always be smooth, if you're trying to load dozens of 12+ mer profiles into memory. This would especially matter in the matrix command, before the matrix is generated. You can use single-core if your machine can't collate that much into main memory at once, depending on how deep the fastq dataset is. Even when handling small-ish k-mer profiles, you may bump into memory overheads rather quickly. \n\nBesides that, I'd suggest reading the source, the differente elements of the [main page](https://matthewralston.github.io/kmerdb) or the [RTD documentation](https://kdb.readthedocs.io/en/stable/).\n\n\n\n## Minimum-viable product Documentation\n\n### Problem statement: \n\nCalculate relevant metadata from k-mer profiles, manipulate count matrices and distance matrices on inter-species inter-profile distances, perform exploartory analysis using tools such as UPGMA hierarchical clustering, k-means clustering, PCA, and others.\n\nCurrently unsupported but targeted features include regression modeling, strassen multiplication, NumPy CPU multipication option, least squares and associated regression code.\n\nCurrently in-progress is the application note. I wouldn't mind ironing out a few more components of the program for the application note.\n\n### Target audience:\n\nBiologists and researchers using metagenomics and genomics tools for the assessment of inter-genomic distances, k-mer counts, nullomer counts, unique counts, and other metadata.\n\n### Feature prioritization:\n\nFeatures currently implemented in `kmerdb` include:\n\n\n- [x] k-mer counting and profile generation\n- [x] multiplexed profiles\n- [x] De Bruijn graph structures\n- [x] Count aggergation and normalization (DESeq2)\n- [x] PCA\n- [x] t-Stochastic Neighbor Embedding\n- [x] Cython Pearson correlation coefficient\n- [x] Other correlation coefficients (Spearman, Pearson via scipy)\n- [x] hierarchical clustering\n- [x] k-means clustering\n- [x] data pipelining (Unix pipelines)\n\nFeatures currently under consideration include: \n\n- [ ] regression modeling for metagenomnic populations\n- [ ] alternative k-mer count vector distances\n- [ ] alignment using k-mers as seed regions\n- [ ] De Bruijn graph traversals and contig generation using DBA (De Bruijn graph) assembly\n\n### Success Criteria:\n\n\n\n\n### Technical specifications:\n\n### User stories:\n\n### Development timeline:\n\n- 2/24 - 6/24 v0.7.7+ - appmap, De Bruijn parser, other updates\n\n- 10/23 - \n\n### Testing and quality assurance plan:\n\n### Feedback collection strategy:\n\n\n\n\n## Development\n\nhttps://matthewralston.github.io/kmerdb/developing\n\n```bash\npython setup.py test\n```\n\n## License\n\nCreated by Matthew Ralston - [Scientist, Programmer, Musician](http://matthewralston.github.io) - [Email](mailto:mralston.development@gmail.com)\n\nDistributed under the Apache license. See `LICENSE.txt` for the copy distributed with this project. Open source software is not for everyone, and im the author and maintainer. cheers, on me. You may use and distribute this software, gratis, so long as the original LICENSE.txt is distributed along with the software. This software is distributed AS IS and provides no warranties of any kind.\n\n```\n Copyright 2020 Matthew Ralston\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n```\n\n## Contributing\n\n1. Fork it (<https://github.com/MatthewRalston/kmerdb/fork>)\n2. Create your feature branch (`git checkout -b feature/fooBar`)\n3. Commit your changes (`git commit -am 'Add some fooBar'`)\n4. Push to the branch (`git push origin feature/fooBar`)\n5. Create a new Pull Request\n\n## Acknowledgements\n\nThanks mom and dad and my hometown, awesome hs, and the University of Delaware faculty for support and encouragement. Thanks to my former mentors, bosses, and coworkers. It's been really enjoyable anticipating what the metagenomics community might want from a tool that can handle microbial k-mers well.\n\nThank you to the authors of kPAL and Jellyfish for the inspiration and bit shifting trick. And thank you to others for the encouragement along the way, who shall remain nameless. \n\nThe intention is that more developers would want to add functionality to the codebase or even just utilize things downstream, but to build out directly with numpy and scipy/scikit as needed to suggest the basic infrastructure for the ML problems and modeling approaches that could be applied to such datasets. This project began under GPL v3.0 and was relicensed with Apache v2. Hopefully this project could gain some interest. I have so much fun working on this project. There's more to it than meets the eye. I'm working on a preprint, and the draft is included in some of the latest versions of the codebase, specifically .Rmd files.\n\nMore on the flip-side. It's so complex with technology these days...\n\n<!--\nThanks of course to my fans (and haters). Yeah i see you.... but i dont.\nThanks to my former mentors BC, MR, IN, CR, and my newer bosses PJ and KL.\nThanks to the Pap lab for the first dataset that I continue to use.\nThank you to Ryan for the food and stuff. I actually made this project specifically so you and I could converse...\nThanks to Blahah for tolerating someone snooping and imitating his Ruby style.\nThanks to Erin for getting my feet wet in this new field. You are my mvp.\nThanks to Rachel for the good memories and friendship. And Sophie too. veggies n' R love.\nThanks to Yasmeen for the usual banter.\nThanks to A for the newer banter.\nThanks to Max, Robin, and Robert for the good memories in St. Louis. What's new?\nThanks to Fred for the good memories. Hope you're on soon.\nThanks to Nichole for the cookies and good memories. And your cute furballs too! Hope you're well\nThanks to S for the lessons, convos, and even embarassing moments. You're kind of awesome to me.\nThanks to a few friends I met in 2023 that reminded me I have a lot to learn about friendship, dating, and street smarts.\nThanks to them even more now that I got it xd up.\n\nThanks to the people of NCC for the Doordash money. It might not be much but I don't have it twisted (I do.)\n\n\n\nThanks to D from BCCS.\nThanks to C4H&H. I'm 'healthier' now, but I really still think I need more support than just BCCS. it's urgent.\nThanks to CT and family. Your love and support means the world to me.\nThanks to AS and family. Your support made a difference. Praying for better employment and opportunities.\n\nAnd thanks to my family and friends.\nGo Blue Hens\n-->\n",
"bugtrack_url": null,
"license": " Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. \"License\" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. \"Licensor\" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. \"Legal Entity\" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, \"control\" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. \"You\" (or \"Your\") shall mean an individual or Legal Entity exercising permissions granted by this License. \"Source\" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. \"Object\" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. \"Work\" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). \"Derivative Works\" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. \"Contribution\" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, \"submitted\" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as \"Not a Contribution.\" \"Contributor\" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a \"NOTICE\" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets \"[]\" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same \"printed page\" as the copyright notice for easier identification within third-party archives. Copyright 2020 Matthew Ralston Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ",
"summary": "Yet another correction to the 'yet another correction to just a k-mer counter...'",
"version": "0.8.10",
"project_urls": {
"Download": "https://github.com/MatthewRalston/kmerdb/archive/v0.8.10.tar.gz",
"Github": "https://github.com/MatthewRalston/kmerdb",
"Homepage": "https://matthewralston.github.io/kmerdb",
"Issues": "https://github.com/MatthewRalston/kmerdb/issues",
"Projects": "https://github.com/MatthewRalston?tab=projects",
"QuickStart": "https://matthewralston.github.io/kmerdb/quickstart.html"
},
"split_keywords": [
"bioinformatics",
" fastq",
" fasta",
" k-mer",
" kmer"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a46fbf93de2c0195620c641fdea10950ac989365adc9a045442354051ee84500",
"md5": "7967952e0678776a0803a4e7a3793cdb",
"sha256": "3ef89bb1925e2a663d2ac4a9d353fce438ac4b2f90c7b1172e1ecd8da93c6666"
},
"downloads": -1,
"filename": "kmerdb-0.8.10.tar.gz",
"has_sig": false,
"md5_digest": "7967952e0678776a0803a4e7a3793cdb",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.12.2",
"size": 403822,
"upload_time": "2024-10-12T23:05:45",
"upload_time_iso_8601": "2024-10-12T23:05:45.602923Z",
"url": "https://files.pythonhosted.org/packages/a4/6f/bf93de2c0195620c641fdea10950ac989365adc9a045442354051ee84500/kmerdb-0.8.10.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-12 23:05:45",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "MatthewRalston",
"github_project": "kmerdb",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"circle": true,
"requirements": [],
"tox": true,
"lcname": "kmerdb"
}
