<!---
Copyright 2023 Yuan He. All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<p align="center">
<a href="https://krr-oxford.github.io/HierarchyTransformers/">
<img alt="license" src="https://raw.githubusercontent.com/KRR-Oxford/HierarchyTransformers/main/docs/assets/images/hit_logo+title.png" style="height: 5em;">
</a>
</p>
<p align="center">
<a href="https://github.com/KRR-Oxford/HierarchyTransformers/blob/main/LICENSE">
<img alt="license" src="https://img.shields.io/github/license/KRR-Oxford/HierarchyTransformers">
</a>
<a href="https://huggingface.co/Hierarchy-Transformers">
<img alt="docs" src="https://img.shields.io/badge/website-online-informational">
</a>
<a href="https://pypi.org/project/hierarchy_transformers/">
<img alt="pypi" src="https://img.shields.io/pypi/v/hierarchy_transformers">
</a>
</p>
<h4 align="center">
<p>
<a href="https://krr-oxford.github.io/HierarchyTransformers/">Project</a> |
<a href="https://huggingface.co/Hierarchy-Transformers">HuggingFace</a> |
<a href="https://arxiv.org/abs/2401.11374">arXiv</a> |
<a href="https://doi.org/10.5281/zenodo.10511042">Zenodo</a>
</p>
</h4>
<h3 align="center">
<p>Embedding hierarchies with language models.</p>
</h3>
**News** ([changelog](docs/changelog.md)) :newspaper:
- [X] Refactor code and add customised HiT trainer (**v0.1.1**).
- [X] Significant development to align with `sentence-transformers>=3.4.0.dev0` (**v0.1.0**).
- [X] Project page is now available ([click](https://krr-oxford.github.io/HierarchyTransformers/)).
- [X] Initial release (should work with `sentence-transformers<3.0.0` ) and bug fix. (**v0.0.3**)
## About
Hierarchy Transformer (HiT) is a framework that enables transformer encoder-based language models (LMs) to learn hierarchical structures in hyperbolic space. The main idea is to construct a Poincaré ball that directly circumscribes the output embedding space of LMs,leveraging the exponential expansion of hyperbolic space to organise entity embeddings hierarchically. In addition to presenting this framework (see code on [GitHub](https://github.com/KRR-Oxford/HierarchyTransformers)), we are committed to training and releasing HiT models across various hierachiies. The models and datasets will be accessible on [HuggingFace](https://huggingface.co/Hierarchy-Transformers/).
## Installation
### Main Dependencies
This repository follows a similar layout as the [`sentence-transformers`](https://www.sbert.net/index.html) library. The main model directly extends the sentence transformer architecture. We also utilise [`deeponto`](https://krr-oxford.github.io/DeepOnto/) for extracting hierarchies from source data and constructing datasets from hierarchies, and [`geoopt`](https://geoopt.readthedocs.io/en/latest/index.html) for arithmetic in hyperbolic space.
> The current release of `sentence-transformers=3.3.1` contains bugs during evaluation, which were fixed in their GitHub dev version `sentence-transformers=3.4.0.dev0`, please update the dependency manually until the official `3.4.0` is released.
### Install from PyPI
```bash
# requiring Python>=3.9
pip install hierarchy_transformers
```
### Install from GitHub
```bash
pip install git+https://github.com/KRR-Oxford/HierarchyTransformers.git
```
## Huggingface Hub
Our HiT models and datasets are released on the [HuggingFace Hub](https://huggingface.co/Hierarchy-Transformers).
### Get Started
```python
from hierarchy_transformers import HierarchyTransformer
# load the model
model = HierarchyTransformer.from_pretrained('Hierarchy-Transformers/HiT-MiniLM-L12-WordNetNoun')
# entity names to be encoded.
entity_names = ["computer", "personal computer", "fruit", "berry"]
# get the entity embeddings
entity_embeddings = model.encode(entity_names)
```
### Default Probing for Subsumption Prediction
Use the entity embeddings to predict the subsumption relationships between them.
```python
# suppose we want to compare "personal computer" and "computer", "berry" and "fruit"
child_entity_embeddings = model.encode(["personal computer", "berry"], convert_to_tensor=True)
parent_entity_embeddings = model.encode(["computer", "fruit"], convert_to_tensor=True)
# compute the hyperbolic distances and norms of entity embeddings
dists = model.manifold.dist(child_entity_embeddings, parent_entity_embeddings)
child_norms = model.manifold.dist0(child_entity_embeddings)
parent_norms = model.manifold.dist0(parent_entity_embeddings)
# use the empirical function for subsumption prediction proposed in the paper
# `centri_score_weight` and the overall threshold are determined on the validation set
subsumption_scores = - (dists + centri_score_weight * (parent_norms - child_norms))
```
### Train Your Own Models
Use the example scripts in our [repository](https://github.com/KRR-Oxford/HierarchyTransformers/tree/main/scripts) to reproduce existing models and train/evaluate your own models.
## License
Copyright 2023 Yuan He.
All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at *<http://www.apache.org/licenses/LICENSE-2.0>*
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
## Citation
If you find this repository or the released models useful, please cite our publication:
*Yuan He, Zhangdie Yuan, Jiaoyan Chen, Ian Horrocks.* **Language Models as Hierarchy Encoders.** To appear at NeurIPS 2024. /[arxiv](https://arxiv.org/abs/2401.11374)/ /[neurips](https://neurips.cc/virtual/2024/poster/95913)/
```
@article{he2024language,
title={Language Models as Hierarchy Encoders},
author={He, Yuan and Yuan, Zhangdie and Chen, Jiaoyan and Horrocks, Ian},
journal={arXiv preprint arXiv:2401.11374},
year={2024}
}
```
Raw data
{
"_id": null,
"home_page": null,
"name": "hierarchy-transformers",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": "Yuan He <yuan.he@cs.ox.ac.uk>",
"keywords": "Language Models, Transformer Encoders, Hierarchy Encoders, Hierarchy Embedding, Hyperbolic Embedding",
"author": null,
"author_email": "Yuan He <yuan.he@cs.ox.ac.uk>",
"download_url": "https://files.pythonhosted.org/packages/41/18/b976ce86502a079cb32d9785c96c989e39473377229512058b2bd3ec95be/hierarchy_transformers-0.1.1.tar.gz",
"platform": null,
"description": "<!---\nCopyright 2023 Yuan He. All rights reserved.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n-->\n\n\n<p align=\"center\">\n <a href=\"https://krr-oxford.github.io/HierarchyTransformers/\">\n <img alt=\"license\" src=\"https://raw.githubusercontent.com/KRR-Oxford/HierarchyTransformers/main/docs/assets/images/hit_logo+title.png\" style=\"height: 5em;\">\n </a>\n</p>\n\n<p align=\"center\">\n <a href=\"https://github.com/KRR-Oxford/HierarchyTransformers/blob/main/LICENSE\">\n <img alt=\"license\" src=\"https://img.shields.io/github/license/KRR-Oxford/HierarchyTransformers\">\n </a>\n <a href=\"https://huggingface.co/Hierarchy-Transformers\">\n <img alt=\"docs\" src=\"https://img.shields.io/badge/website-online-informational\">\n </a>\n <a href=\"https://pypi.org/project/hierarchy_transformers/\">\n <img alt=\"pypi\" src=\"https://img.shields.io/pypi/v/hierarchy_transformers\">\n </a>\n</p>\n\n<h4 align=\"center\">\n <p>\n <a href=\"https://krr-oxford.github.io/HierarchyTransformers/\">Project</a> |\n <a href=\"https://huggingface.co/Hierarchy-Transformers\">HuggingFace</a> |\n <a href=\"https://arxiv.org/abs/2401.11374\">arXiv</a> |\n <a href=\"https://doi.org/10.5281/zenodo.10511042\">Zenodo</a>\n </p>\n</h4>\n\n<h3 align=\"center\">\n <p>Embedding hierarchies with language models.</p>\n</h3>\n\n**News** ([changelog](docs/changelog.md)) :newspaper:\n\n- [X] Refactor code and add customised HiT trainer (**v0.1.1**).\n- [X] Significant development to align with `sentence-transformers>=3.4.0.dev0` (**v0.1.0**).\n- [X] Project page is now available ([click](https://krr-oxford.github.io/HierarchyTransformers/)).\n- [X] Initial release (should work with `sentence-transformers<3.0.0` ) and bug fix. (**v0.0.3**)\n\n\n## About\n\nHierarchy Transformer (HiT) is a framework that enables transformer encoder-based language models (LMs) to learn hierarchical structures in hyperbolic space. The main idea is to construct a Poincar\u00e9 ball that directly circumscribes the output embedding space of LMs,leveraging the exponential expansion of hyperbolic space to organise entity embeddings hierarchically. In addition to presenting this framework (see code on [GitHub](https://github.com/KRR-Oxford/HierarchyTransformers)), we are committed to training and releasing HiT models across various hierachiies. The models and datasets will be accessible on [HuggingFace](https://huggingface.co/Hierarchy-Transformers/).\n\n## Installation\n\n### Main Dependencies\n\nThis repository follows a similar layout as the [`sentence-transformers`](https://www.sbert.net/index.html) library. The main model directly extends the sentence transformer architecture. We also utilise [`deeponto`](https://krr-oxford.github.io/DeepOnto/) for extracting hierarchies from source data and constructing datasets from hierarchies, and [`geoopt`](https://geoopt.readthedocs.io/en/latest/index.html) for arithmetic in hyperbolic space.\n\n> The current release of `sentence-transformers=3.3.1` contains bugs during evaluation, which were fixed in their GitHub dev version `sentence-transformers=3.4.0.dev0`, please update the dependency manually until the official `3.4.0` is released.\n\n### Install from PyPI\n\n```bash\n# requiring Python>=3.9\npip install hierarchy_transformers\n```\n\n### Install from GitHub\n\n```bash\npip install git+https://github.com/KRR-Oxford/HierarchyTransformers.git\n```\n\n## Huggingface Hub\n\nOur HiT models and datasets are released on the [HuggingFace Hub](https://huggingface.co/Hierarchy-Transformers). \n\n### Get Started\n\n```python\nfrom hierarchy_transformers import HierarchyTransformer\n\n# load the model\nmodel = HierarchyTransformer.from_pretrained('Hierarchy-Transformers/HiT-MiniLM-L12-WordNetNoun')\n\n# entity names to be encoded.\nentity_names = [\"computer\", \"personal computer\", \"fruit\", \"berry\"]\n\n# get the entity embeddings\nentity_embeddings = model.encode(entity_names)\n```\n\n### Default Probing for Subsumption Prediction\n\nUse the entity embeddings to predict the subsumption relationships between them.\n\n```python\n# suppose we want to compare \"personal computer\" and \"computer\", \"berry\" and \"fruit\"\nchild_entity_embeddings = model.encode([\"personal computer\", \"berry\"], convert_to_tensor=True)\nparent_entity_embeddings = model.encode([\"computer\", \"fruit\"], convert_to_tensor=True)\n\n# compute the hyperbolic distances and norms of entity embeddings\ndists = model.manifold.dist(child_entity_embeddings, parent_entity_embeddings)\nchild_norms = model.manifold.dist0(child_entity_embeddings)\nparent_norms = model.manifold.dist0(parent_entity_embeddings)\n\n# use the empirical function for subsumption prediction proposed in the paper\n# `centri_score_weight` and the overall threshold are determined on the validation set\nsubsumption_scores = - (dists + centri_score_weight * (parent_norms - child_norms))\n```\n\n### Train Your Own Models\n\nUse the example scripts in our [repository](https://github.com/KRR-Oxford/HierarchyTransformers/tree/main/scripts) to reproduce existing models and train/evaluate your own models. \n\n## License\n\n\n\n Copyright 2023 Yuan He.\n All rights reserved.\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at *<http://www.apache.org/licenses/LICENSE-2.0>*\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n\n## Citation\n\nIf you find this repository or the released models useful, please cite our publication:\n\n*Yuan He, Zhangdie Yuan, Jiaoyan Chen, Ian Horrocks.* **Language Models as Hierarchy Encoders.** To appear at NeurIPS 2024. /[arxiv](https://arxiv.org/abs/2401.11374)/ /[neurips](https://neurips.cc/virtual/2024/poster/95913)/\n\n```\n@article{he2024language,\n title={Language Models as Hierarchy Encoders},\n author={He, Yuan and Yuan, Zhangdie and Chen, Jiaoyan and Horrocks, Ian},\n journal={arXiv preprint arXiv:2401.11374},\n year={2024}\n}\n```\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "Language models as hierarchy encoders.",
"version": "0.1.1",
"project_urls": {
"Homepage": "https://krr-oxford.github.io/HierarchyTransformers/",
"Repository": "https://github.com/KRR-Oxford/HierarchyTransformers"
},
"split_keywords": [
"language models",
" transformer encoders",
" hierarchy encoders",
" hierarchy embedding",
" hyperbolic embedding"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "624f7a7446a221287449c651e105ad04978675e3cc0d788a13653bb0e4afccff",
"md5": "02faa32fab6c324be90a07c42e0637f7",
"sha256": "20bad0d079da5b489484e0ae895f6f8eae34449ab40102d05462925ce3f38892"
},
"downloads": -1,
"filename": "hierarchy_transformers-0.1.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "02faa32fab6c324be90a07c42e0637f7",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 41771,
"upload_time": "2024-12-13T02:23:19",
"upload_time_iso_8601": "2024-12-13T02:23:19.059604Z",
"url": "https://files.pythonhosted.org/packages/62/4f/7a7446a221287449c651e105ad04978675e3cc0d788a13653bb0e4afccff/hierarchy_transformers-0.1.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "4118b976ce86502a079cb32d9785c96c989e39473377229512058b2bd3ec95be",
"md5": "4e4b86f774de9555426cba5b502ef5a4",
"sha256": "d1cf3a9695131006bc50666b49353d2063deac1dd868e6339ea71ea792a9e93b"
},
"downloads": -1,
"filename": "hierarchy_transformers-0.1.1.tar.gz",
"has_sig": false,
"md5_digest": "4e4b86f774de9555426cba5b502ef5a4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 28850,
"upload_time": "2024-12-13T02:23:20",
"upload_time_iso_8601": "2024-12-13T02:23:20.262061Z",
"url": "https://files.pythonhosted.org/packages/41/18/b976ce86502a079cb32d9785c96c989e39473377229512058b2bd3ec95be/hierarchy_transformers-0.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-13 02:23:20",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "KRR-Oxford",
"github_project": "HierarchyTransformers",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "sentence_transformers",
"specs": [
[
">=",
"3.4.0.dev0"
]
]
},
{
"name": "deeponto",
"specs": [
[
">=",
"0.9.2"
]
]
},
{
"name": "geoopt",
"specs": [
[
"==",
"0.5.0"
]
]
},
{
"name": "scipy",
"specs": [
[
"==",
"1.13.1"
]
]
},
{
"name": "seaborn",
"specs": []
}
],
"lcname": "hierarchy-transformers"
}
JSON
