# MINT
Meaning-Informed Next-token Transformation
## Project Goals
MINT adds a transformation layer that redistributes next\-token probabilities
according to semantic token similarity based on the model's embedding space. The
aim is to produce more varied, human\-like text without sacrificing coherence.
## Installation
Create a virtual environment and install the package in editable mode:
```bash
pip install -e .
```
This installs the `mint` package and provides the `mint` command-line interface.
To use the published release from PyPI (when available), run:
```bash
pip install mint-llm
```
## CLI Usage
Run the CLI with:
```bash
mint --help
```
The `mint` command exposes several subcommands. The typical workflow is shown
below.
## Using MINT
Run the commands below to build and apply the redistribution layer:
1. **Pick** a checkpoint from the Hugging Face Hub (optional). This
uses the `mint.utils.download_checkpoint` helper to fetch and merge
sharded weights automatically.
```bash
mint pick <model_id> checkpoint/
```
2. **Extract** token embeddings from the checkpoint.
```bash
mint extract checkpoint/model.safetensors embeddings.safetensors
```
3. **Blend** the embeddings into a low-rank similarity factor.
```bash
mint blend embeddings.safetensors mint_out/ --rank 1024 --gpu 0
# → mint_out/W.safetensors (plus R.safetensors with --keep-residual)
```
Use `-r/--rank` to set factor rank (default 1024). Add `--cpu` or
`--gpu IDX` to choose the device. The optional `--sdk` flag selects an
acceleration backend—CUDA and Vulkan are supported. ZLUDA,
ROCm and Metal support will be added in a future release. Pass `--keep-residual` to also save a
sparse `R.safetensors` file.
4. **Brew** new text from the wrapped model.
```bash
mint brew model_id_or_path mint_out/ --prompt "Hello"
```
Omit `--prompt` or pass `--interactive` to read prompts from stdin.
5. **Infuse** the tested similarity matrix into a local model and save the result
to a directory.
```bash
mint infuse path/to/model mint_out/ infused-model --alpha 0.1
```
```python
from mint.wrapper import load_wrapped_model
from mint.logits import SRLogitsProcessor
model, tokenizer, layer = load_wrapped_model("model_id_or_path", "mint_out/")
processor = SRLogitsProcessor(layer)
```
See the [notebooks](notebooks/) and [`examples/quickstart.py`](examples/quickstart.py)
for a more detailed walk-through and an automated script. You can also explore
the generator interactively using the CLI.
## Additional Utilities
The CLI exposes optional commands for working with checkpoints:
- **Crush** merge sharded checkpoints referenced by an index file.
```bash
mint crush checkpoint/model.safetensors.index.json checkpoint/model.safetensors
```
- **Chop** split a `.safetensors` checkpoint into shards. Provide a shard count
or size:
```bash
mint chop model.safetensors shards/ --shards 2
mint chop model.safetensors shards/ --size-mb 500
```
## Brand-based ISVD Functions
MINT ships helper utilities implementing Brand's Incremental Singular Value Decomposition alongside Zhang et al.'s update strategy. Use `initialize_isvd`, `update_isvd`, and `final_isvd_check` from `mint.brand_svd` to maintain low-rank factors as new embedding vectors are streamed. These functions support optional weighting matrices to match the similarity metric. We continuously refine the implementation and optimize performance—see the `Brand SVD` folder for the original papers.
## Quickstart Script
Run [`examples/quickstart.py`](examples/quickstart.py) for an end-to-end
demonstration. The script mirrors the `mint` CLI commands:
`extract`, `blend` and `brew`.
Required argument:
- `--prompt` – input text to generate from.
Optional arguments default to values defined in
[`tests/utils/model_config.json`](tests/utils/model_config.json):
- `--checkpoint` – checkpoint path. If this points to a
`*.safetensors.index.json` file the required shards are downloaded and merged
automatically. If omitted `model_url` is used.
- `--model` – model identifier or path. When a model ID is provided the
checkpoint shards are fetched and merged automatically. Defaults to
`model_id` or one derived from `model_url`.
- `--embeddings` – output file for embeddings (default
`embeddings.safetensors`).
- `--similarity` – output directory for `W.safetensors` (and optionally
`R.safetensors`, default `.cache/mint`).
```bash
python examples/quickstart.py --prompt "Hello"
```
The script extracts embeddings, builds the similarity matrix and generates text
using the wrapped model.
## Examples
Practical examples are provided in the [notebooks](notebooks/) directory.
They demonstrate embedding extraction, building a similarity matrix and
brewing text from a short prompt.
## Development
Install development dependencies with:
```bash
pip install -e '.[dev]'
```
The development extras include the `vulkan` package so local tests can run
against the Vulkan backend. GitHub Actions does not provide Vulkan support,
and any Vulkan tests are skipped in CI.
Use the provided Makefile to run common tasks:
```bash
make format # check black formatting
make lint # run ruff and mypy (if configured)
make lint-fast # run ruff only, skip mypy
make test # run the pytest suite
make all # runs all checks
```
`make` commands `format`, `lint`, `lint-fast`, and `all` can also be suffixed with `-fix` (e.g. `make lint-fix` or `make all-fix`)
to attempt to automatically fix issues. `make fix` will run `all-fix`.
Tests are executed with `-Werror`, so any warnings will fail the build.
Continuous integration uses `.github/workflows/lint.yml` and
`.github/workflows/tests.yml`. Tagged releases
first run `.github/workflows/version-bump.yml`, which commits the
updated version information back to `src/mint/__init__.py` and
`CITATION.cff`. This workflow is shell-only and doesn't require a Python
environment. When that workflow completes successfully,
`.github/workflows/publish.yml` and `.github/workflows/release.yml`
build and upload the package and GitHub release. Both workflows use
`scripts/prepare_pypi_readme.py` to prepare the README for PyPI.
## Contributing
Development tasks are tracked in `todos.json`. See
[`project_proposal-MINT.md`](project_proposal-MINT.md) for the full technical
plan. Release notes are available in
[`CHANGELOG.md`](CHANGELOG.md). Feel free to open issues or pull requests to
contribute.
## Citation
```yaml
cff-version: 1.2.0
title: MINT - Meaning-Informed Next-token Transformation
message: 'If you reference this project, please cite it as below.'
type: software
authors:
- given-names: Bryan
family-names: O'Malley
email: bo122081@hotmail.com
identifiers:
- type: url
value: 'https://github.com/Reithan/MINT'
description: github repo for MINT
repository-code: 'https://github.com/Reithan/MINT'
url: 'https://github.com/Reithan/MINT'
abstract: >-
MINT adds a post-softmax decoding layer that redistributes
token log-probs according to token similarity based on
the model's embedding space. The aim is to produce more
varied, human-like text without sacrificing coherence.
keywords:
- llm
- ai
- svd
- isvd
- transformers
- safetensors
- text-generation
- chat-completion
- huggingface
commit: 75bb29a90e0988862d03020d2f7fb399ff621845
version: v0.1.4-alpha
date-released: '2025-06-19'
references:
- type: conference-paper
title: "Incremental Singular Value Decomposition of Uncertain Data with Missing Values"
authors:
- given-names: Matthew
family-names: Brand
year: 2002
doi: "10.1007/3-540-47969-4_47"
url: "https://link.springer.com/chapter/10.1007/3-540-47969-4_47"
citation: "@inproceedings{brand2002incremental,\n author = {Brand, M.},\n title = {{Incremental Singular Value Decomposition of Uncertain Data with Missing Values}},\n booktitle = {European Conference on Computer Vision (ECCV)},\n volume = {2350},\n pages = {707--720},\n year = {2002},\n doi = {10.1007/3-540-47969-4_47},\n url = {https://link.springer.com/chapter/10.1007/3-540-47969-4_47}\n}"
- type: article
title: "An answer to an open question in the incremental SVD"
authors:
- given-names: Yangwen
family-names: Zhang
year: 2022
url: "https://arxiv.org/abs/2204.05398"
citation: "@article{zhang2022answer,\n author = {Zhang, Yangwen},\n title = {{An answer to an open question in the incremental SVD}},\n journal = {arXiv preprint arXiv:2204.05398},\n year = {2022},\n url = {https://arxiv.org/abs/2204.05398}\n}"
```
Raw data
{
"_id": null,
"home_page": null,
"name": "mint-llm",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.13",
"maintainer_email": null,
"keywords": "llm, machine learning, text-generation, huggingface, hf, pytorch, tools, svd, isvd",
"author": null,
"author_email": "Bryan O'Malley <bo122081@hotmail.com>",
"download_url": "https://files.pythonhosted.org/packages/ed/d0/3df91f8bfc40f0d4a59d62287000d57dc9b10c732bd0c38b35cb049054c5/mint_llm-0.1.4.tar.gz",
"platform": null,
"description": "# MINT\nMeaning-Informed Next-token Transformation\n\n## Project Goals\nMINT adds a transformation layer that redistributes next\\-token probabilities\naccording to semantic token similarity based on the model's embedding space. The\naim is to produce more varied, human\\-like text without sacrificing coherence.\n\n## Installation\nCreate a virtual environment and install the package in editable mode:\n\n```bash\npip install -e .\n```\n\nThis installs the `mint` package and provides the `mint` command-line interface.\n\nTo use the published release from PyPI (when available), run:\n\n```bash\npip install mint-llm\n```\n\n## CLI Usage\nRun the CLI with:\n\n```bash\nmint --help\n```\n\nThe `mint` command exposes several subcommands. The typical workflow is shown\nbelow.\n\n## Using MINT\n\nRun the commands below to build and apply the redistribution layer:\n\n1. **Pick** a checkpoint from the Hugging Face Hub (optional). This\n uses the `mint.utils.download_checkpoint` helper to fetch and merge\n sharded weights automatically.\n\n ```bash\n mint pick <model_id> checkpoint/\n ```\n\n2. **Extract** token embeddings from the checkpoint.\n\n ```bash\n mint extract checkpoint/model.safetensors embeddings.safetensors\n ```\n\n3. **Blend** the embeddings into a low-rank similarity factor.\n\n ```bash\n mint blend embeddings.safetensors mint_out/ --rank 1024 --gpu 0\n # \u2192 mint_out/W.safetensors (plus R.safetensors with --keep-residual)\n ```\n Use `-r/--rank` to set factor rank (default 1024). Add `--cpu` or\n `--gpu IDX` to choose the device. The optional `--sdk` flag selects an\n acceleration backend\u2014CUDA and Vulkan are supported. ZLUDA,\n ROCm and Metal support will be added in a future release. Pass `--keep-residual` to also save a\n sparse `R.safetensors` file.\n\n4. **Brew** new text from the wrapped model.\n\n ```bash\n mint brew model_id_or_path mint_out/ --prompt \"Hello\"\n ```\n Omit `--prompt` or pass `--interactive` to read prompts from stdin.\n\n5. **Infuse** the tested similarity matrix into a local model and save the result\n to a directory.\n\n ```bash\n mint infuse path/to/model mint_out/ infused-model --alpha 0.1\n ```\n\n ```python\n from mint.wrapper import load_wrapped_model\n from mint.logits import SRLogitsProcessor\n\n model, tokenizer, layer = load_wrapped_model(\"model_id_or_path\", \"mint_out/\")\n processor = SRLogitsProcessor(layer)\n ```\n\nSee the [notebooks](notebooks/) and [`examples/quickstart.py`](examples/quickstart.py)\nfor a more detailed walk-through and an automated script. You can also explore\nthe generator interactively using the CLI.\n\n## Additional Utilities\nThe CLI exposes optional commands for working with checkpoints:\n\n- **Crush** merge sharded checkpoints referenced by an index file.\n\n ```bash\n mint crush checkpoint/model.safetensors.index.json checkpoint/model.safetensors\n ```\n\n- **Chop** split a `.safetensors` checkpoint into shards. Provide a shard count\n or size:\n\n ```bash\n mint chop model.safetensors shards/ --shards 2\n\n mint chop model.safetensors shards/ --size-mb 500\n ```\n\n## Brand-based ISVD Functions\nMINT ships helper utilities implementing Brand's Incremental Singular Value Decomposition alongside Zhang et al.'s update strategy. Use `initialize_isvd`, `update_isvd`, and `final_isvd_check` from `mint.brand_svd` to maintain low-rank factors as new embedding vectors are streamed. These functions support optional weighting matrices to match the similarity metric. We continuously refine the implementation and optimize performance\u2014see the `Brand SVD` folder for the original papers.\n\n## Quickstart Script\nRun [`examples/quickstart.py`](examples/quickstart.py) for an end-to-end\ndemonstration. The script mirrors the `mint` CLI commands:\n`extract`, `blend` and `brew`.\n\nRequired argument:\n\n- `--prompt` \u2013 input text to generate from.\n\nOptional arguments default to values defined in\n[`tests/utils/model_config.json`](tests/utils/model_config.json):\n\n- `--checkpoint` \u2013 checkpoint path. If this points to a\n `*.safetensors.index.json` file the required shards are downloaded and merged\n automatically. If omitted `model_url` is used.\n- `--model` \u2013 model identifier or path. When a model ID is provided the\n checkpoint shards are fetched and merged automatically. Defaults to\n `model_id` or one derived from `model_url`.\n- `--embeddings` \u2013 output file for embeddings (default\n`embeddings.safetensors`).\n- `--similarity` \u2013 output directory for `W.safetensors` (and optionally\n `R.safetensors`, default `.cache/mint`).\n\n```bash\npython examples/quickstart.py --prompt \"Hello\"\n```\n\nThe script extracts embeddings, builds the similarity matrix and generates text\nusing the wrapped model.\n\n## Examples\nPractical examples are provided in the [notebooks](notebooks/) directory.\nThey demonstrate embedding extraction, building a similarity matrix and\nbrewing text from a short prompt.\n\n## Development\nInstall development dependencies with:\n\n```bash\npip install -e '.[dev]'\n```\n\nThe development extras include the `vulkan` package so local tests can run\nagainst the Vulkan backend. GitHub Actions does not provide Vulkan support,\nand any Vulkan tests are skipped in CI.\n\nUse the provided Makefile to run common tasks:\n\n```bash\nmake format # check black formatting\nmake lint # run ruff and mypy (if configured)\nmake lint-fast # run ruff only, skip mypy\nmake test # run the pytest suite\nmake all # runs all checks\n```\n\n`make` commands `format`, `lint`, `lint-fast`, and `all` can also be suffixed with `-fix` (e.g. `make lint-fix` or `make all-fix`)\nto attempt to automatically fix issues. `make fix` will run `all-fix`.\n\nTests are executed with `-Werror`, so any warnings will fail the build.\n\nContinuous integration uses `.github/workflows/lint.yml` and\n`.github/workflows/tests.yml`. Tagged releases\nfirst run `.github/workflows/version-bump.yml`, which commits the\nupdated version information back to `src/mint/__init__.py` and\n`CITATION.cff`. This workflow is shell-only and doesn't require a Python\nenvironment. When that workflow completes successfully,\n`.github/workflows/publish.yml` and `.github/workflows/release.yml`\nbuild and upload the package and GitHub release. Both workflows use\n`scripts/prepare_pypi_readme.py` to prepare the README for PyPI.\n\n## Contributing\nDevelopment tasks are tracked in `todos.json`. See\n[`project_proposal-MINT.md`](project_proposal-MINT.md) for the full technical\nplan. Release notes are available in\n[`CHANGELOG.md`](CHANGELOG.md). Feel free to open issues or pull requests to\ncontribute.\n\n\n## Citation\n```yaml\ncff-version: 1.2.0\ntitle: MINT - Meaning-Informed Next-token Transformation\nmessage: 'If you reference this project, please cite it as below.'\ntype: software\nauthors:\n - given-names: Bryan\n family-names: O'Malley\n email: bo122081@hotmail.com\nidentifiers:\n - type: url\n value: 'https://github.com/Reithan/MINT'\n description: github repo for MINT\nrepository-code: 'https://github.com/Reithan/MINT'\nurl: 'https://github.com/Reithan/MINT'\nabstract: >-\n MINT adds a post-softmax decoding layer that redistributes\n token log-probs according to token similarity based on\n the model's embedding space. The aim is to produce more\n varied, human-like text without sacrificing coherence.\nkeywords:\n - llm\n - ai\n - svd\n - isvd\n - transformers\n - safetensors\n - text-generation\n - chat-completion\n - huggingface\ncommit: 75bb29a90e0988862d03020d2f7fb399ff621845\nversion: v0.1.4-alpha\ndate-released: '2025-06-19'\n\nreferences:\n - type: conference-paper\n title: \"Incremental Singular Value Decomposition of Uncertain Data with Missing Values\"\n authors:\n - given-names: Matthew\n family-names: Brand\n year: 2002\n doi: \"10.1007/3-540-47969-4_47\"\n url: \"https://link.springer.com/chapter/10.1007/3-540-47969-4_47\"\n citation: \"@inproceedings{brand2002incremental,\\n author = {Brand, M.},\\n title = {{Incremental Singular Value Decomposition of Uncertain Data with Missing Values}},\\n booktitle = {European Conference on Computer Vision (ECCV)},\\n volume = {2350},\\n pages = {707--720},\\n year = {2002},\\n doi = {10.1007/3-540-47969-4_47},\\n url = {https://link.springer.com/chapter/10.1007/3-540-47969-4_47}\\n}\"\n - type: article\n title: \"An answer to an open question in the incremental SVD\"\n authors:\n - given-names: Yangwen\n family-names: Zhang\n year: 2022\n url: \"https://arxiv.org/abs/2204.05398\"\n citation: \"@article{zhang2022answer,\\n author = {Zhang, Yangwen},\\n title = {{An answer to an open question in the incremental SVD}},\\n journal = {arXiv preprint arXiv:2204.05398},\\n year = {2022},\\n url = {https://arxiv.org/abs/2204.05398}\\n}\"\n```\n",
"bugtrack_url": null,
"license": null,
"summary": "Meaning-Informed Next-token Transformation",
"version": "0.1.4",
"project_urls": {
"pypi": "https://pypi.org/project/mint-llm/"
},
"split_keywords": [
"llm",
" machine learning",
" text-generation",
" huggingface",
" hf",
" pytorch",
" tools",
" svd",
" isvd"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "f216c2ad6381160726ea79dc512aa0520df5c4f79d7f6888543cbbd58af54694",
"md5": "7227238e42eb3fd74ddde5cff54c951b",
"sha256": "f256a5b91ea24fd4c997947e5d07176a965552cac60829e5f3ad806042cf725c"
},
"downloads": -1,
"filename": "mint_llm-0.1.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "7227238e42eb3fd74ddde5cff54c951b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.13",
"size": 31159,
"upload_time": "2025-07-10T10:33:06",
"upload_time_iso_8601": "2025-07-10T10:33:06.709234Z",
"url": "https://files.pythonhosted.org/packages/f2/16/c2ad6381160726ea79dc512aa0520df5c4f79d7f6888543cbbd58af54694/mint_llm-0.1.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "edd03df91f8bfc40f0d4a59d62287000d57dc9b10c732bd0c38b35cb049054c5",
"md5": "06ed4f6d7f7828b787ff912c0bee05e7",
"sha256": "553317eb52d8e1bde9c02d00c668a2752f683e8901d094ab1eda2640560f5b41"
},
"downloads": -1,
"filename": "mint_llm-0.1.4.tar.gz",
"has_sig": false,
"md5_digest": "06ed4f6d7f7828b787ff912c0bee05e7",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.13",
"size": 39048,
"upload_time": "2025-07-10T10:33:08",
"upload_time_iso_8601": "2025-07-10T10:33:08.077735Z",
"url": "https://files.pythonhosted.org/packages/ed/d0/3df91f8bfc40f0d4a59d62287000d57dc9b10c732bd0c38b35cb049054c5/mint_llm-0.1.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-10 10:33:08",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "mint-llm"
}
JSON