# HybridBackend
[](https://github.com/alibaba/HybridBackend/actions/workflows/cibuild.yaml)
[](https://hybridbackend.readthedocs.io/en/latest/?badge=latest)
[](http://makeapullrequest.com)
[](https://opensource.org/licenses/Apache-2.0)
HybridBackend is a high-performance framework for training wide-and-deep
recommender systems on heterogeneous cluster.
## Features
- Memory-efficient loading of categorical data
- GPU-efficient orchestration of embedding layers
- Communication-efficient training and evaluation at scale
- Easy to use with existing AI workflows
## Usage
A minimal example:
```python
import tensorflow as tf
import hybridbackend.tensorflow as hb
ds = hb.data.Dataset.from_parquet(filenames)
ds = ds.batch(batch_size)
# ...
with tf.device('/gpu:0'):
embs = tf.nn.embedding_lookup_sparse(weights, input_ids)
# ...
```
Please see [documentation](https://hybridbackend.readthedocs.io/en/latest/) for
more information.
## Install
### Method 1: Install from PyPI
`pip install {PACKAGE}`
| `{PACKAGE}` | Dependency | Python | CUDA | GLIBC | Data Opt. | Embedding Opt. | Parallelism Opt. |
| ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- | ------ | ---- | ------ | --------- | -------------- | ---------------- |
| [hybridbackend-tf115-cu121](https://pypi.org/project/hybridbackend-tf115-cu121/) | [TensorFlow 1.15](https://github.com/NVIDIA/tensorflow) | 3.8 | 12.1 | >=2.31 | ✓ | ✓ | ✓ |
| [hybridbackend-tf115-cu100](https://pypi.org/project/hybridbackend-tf115-cu100/) | [TensorFlow 1.15](https://github.com/tensorflow/tensorflow/tree/r1.15) | 3.6 | 10.0 | >=2.27 | ✓ | ✓ | ✗ |
| [hybridbackend-tf115-cpu](https://pypi.org/project/hybridbackend-tf115-cpu/) | [TensorFlow 1.15](https://github.com/tensorflow/tensorflow/tree/r1.15) | 3.6 | - | >=2.24 | ✓ | ✗ | ✗ |
### Method 2: Build from source
See [Building Instructions](https://github.com/alibaba/HybridBackend/blob/main/BUILD.md).
We also provide built docker images for latest [DeepRec](https://github.com/alibaba/DeepRec):
`registry.cn-shanghai.aliyuncs.com/pai-dlc/hybridbackend:1.0.0-deeprec-py3.6-cu114-ubuntu18.04`
## License
HybridBackend is licensed under the [Apache 2.0 License](LICENSE).
## Community
- Please see [Contributing Guide](https://github.com/alibaba/HybridBackend/blob/main/CONTRIBUTING.md)
before your first contribution.
- Please [register as an adopter](https://github.com/alibaba/HybridBackend/blob/main/ADOPTERS.md)
if your organization is interested in adoption. We will discuss
[RoadMap](https://github.com/alibaba/HybridBackend/blob/main/ROADMAP.md) with
registered adopters in advance.
- Please cite [HybridBackend](https://ieeexplore.ieee.org/document/9835450) in your publications if it helps:
```text
@inproceedings{zhang2022picasso,
title={PICASSO: Unleashing the Potential of GPU-centric Training for Wide-and-deep Recommender Systems},
author={Zhang, Yuanxing and Chen, Langshi and Yang, Siran and Yuan, Man and Yi, Huimin and Zhang, Jie and Wang, Jiamang and Dong, Jianbo and Xu, Yunlong and Song, Yue and others},
booktitle={2022 IEEE 38th International Conference on Data Engineering (ICDE)},
year={2022},
organization={IEEE}
}
```
## Contact Us
If you would like to share your experiences with others, you are welcome to
contact us in DingTalk:
[](https://qr.dingtalk.com/action/joingroup?code=v1,k1,VouhbeuTwXYEgaLzSOE8o6VF2kTHVJ8lw5h93WbZW8o=&_dt_no_comment=1&origin=11)
Raw data
{
"_id": null,
"home_page": "https://github.com/alibaba/HybridBackend",
"name": "hybridbackend-tf115-cu121",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "deep learning,recommendation system",
"author": "Alibaba Group Holding Limited",
"author_email": "",
"download_url": "https://github.com/alibaba/HybridBackend/tags",
"platform": null,
"description": "# HybridBackend\n\n[](https://github.com/alibaba/HybridBackend/actions/workflows/cibuild.yaml)\n[](https://hybridbackend.readthedocs.io/en/latest/?badge=latest)\n[](http://makeapullrequest.com)\n[](https://opensource.org/licenses/Apache-2.0)\n\nHybridBackend is a high-performance framework for training wide-and-deep\nrecommender systems on heterogeneous cluster.\n\n## Features\n\n- Memory-efficient loading of categorical data\n- GPU-efficient orchestration of embedding layers\n- Communication-efficient training and evaluation at scale\n- Easy to use with existing AI workflows\n\n## Usage\n\nA minimal example:\n\n```python\nimport tensorflow as tf\nimport hybridbackend.tensorflow as hb\n\nds = hb.data.Dataset.from_parquet(filenames)\nds = ds.batch(batch_size)\n# ...\n\nwith tf.device('/gpu:0'):\n embs = tf.nn.embedding_lookup_sparse(weights, input_ids)\n # ...\n```\n\nPlease see [documentation](https://hybridbackend.readthedocs.io/en/latest/) for\nmore information.\n\n## Install\n\n### Method 1: Install from PyPI\n\n`pip install {PACKAGE}`\n\n| `{PACKAGE}` | Dependency | Python | CUDA | GLIBC | Data Opt. | Embedding Opt. | Parallelism Opt. |\n| ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- | ------ | ---- | ------ | --------- | -------------- | ---------------- |\n| [hybridbackend-tf115-cu121](https://pypi.org/project/hybridbackend-tf115-cu121/) | [TensorFlow 1.15](https://github.com/NVIDIA/tensorflow) | 3.8 | 12.1 | >=2.31 | ✓ | ✓ | ✓ |\n| [hybridbackend-tf115-cu100](https://pypi.org/project/hybridbackend-tf115-cu100/) | [TensorFlow 1.15](https://github.com/tensorflow/tensorflow/tree/r1.15) | 3.6 | 10.0 | >=2.27 | ✓ | ✓ | ✗ |\n| [hybridbackend-tf115-cpu](https://pypi.org/project/hybridbackend-tf115-cpu/) | [TensorFlow 1.15](https://github.com/tensorflow/tensorflow/tree/r1.15) | 3.6 | - | >=2.24 | ✓ | ✗ | ✗ |\n\n### Method 2: Build from source\n\nSee [Building Instructions](https://github.com/alibaba/HybridBackend/blob/main/BUILD.md).\n\nWe also provide built docker images for latest [DeepRec](https://github.com/alibaba/DeepRec): \n`registry.cn-shanghai.aliyuncs.com/pai-dlc/hybridbackend:1.0.0-deeprec-py3.6-cu114-ubuntu18.04`\n\n## License\n\nHybridBackend is licensed under the [Apache 2.0 License](LICENSE).\n\n## Community\n\n- Please see [Contributing Guide](https://github.com/alibaba/HybridBackend/blob/main/CONTRIBUTING.md)\n before your first contribution.\n- Please [register as an adopter](https://github.com/alibaba/HybridBackend/blob/main/ADOPTERS.md)\n if your organization is interested in adoption. We will discuss\n [RoadMap](https://github.com/alibaba/HybridBackend/blob/main/ROADMAP.md) with\n registered adopters in advance.\n- Please cite [HybridBackend](https://ieeexplore.ieee.org/document/9835450) in your publications if it helps:\n\n ```text\n @inproceedings{zhang2022picasso,\n title={PICASSO: Unleashing the Potential of GPU-centric Training for Wide-and-deep Recommender Systems},\n author={Zhang, Yuanxing and Chen, Langshi and Yang, Siran and Yuan, Man and Yi, Huimin and Zhang, Jie and Wang, Jiamang and Dong, Jianbo and Xu, Yunlong and Song, Yue and others},\n booktitle={2022 IEEE 38th International Conference on Data Engineering (ICDE)},\n year={2022},\n organization={IEEE}\n }\n ```\n\n## Contact Us\n\nIf you would like to share your experiences with others, you are welcome to\ncontact us in DingTalk:\n\n[](https://qr.dingtalk.com/action/joingroup?code=v1,k1,VouhbeuTwXYEgaLzSOE8o6VF2kTHVJ8lw5h93WbZW8o=&_dt_no_comment=1&origin=11)\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "A high-performance framework for training wide-and-deep recommender systems on heterogeneous cluster",
"version": "1.0.0",
"project_urls": {
"Bug Tracker": "https://github.com/alibaba/HybridBackend/issues",
"Documentation": "https://hybridbackend.readthedocs.io/en/latest/",
"Download": "https://github.com/alibaba/HybridBackend/tags",
"Homepage": "https://github.com/alibaba/HybridBackend",
"Source Code": "https://github.com/alibaba/HybridBackend"
},
"split_keywords": [
"deep learning",
"recommendation system"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ab37eafc53963e31530ebefe40cad57aaa54fb6a6ff74fa1742e4382b60d9c8e",
"md5": "b5b29880320ca135d9d23039a1ad9640",
"sha256": "bb1e5bb1f7235be3fcc1ab48f73c9c6f663f49203e87c734ca52fd833e6e95f6"
},
"downloads": -1,
"filename": "hybridbackend_tf115_cu121-1.0.0-cp38-cp38-manylinux_2_31_x86_64.whl",
"has_sig": false,
"md5_digest": "b5b29880320ca135d9d23039a1ad9640",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": null,
"size": 52725376,
"upload_time": "2023-07-20T09:33:04",
"upload_time_iso_8601": "2023-07-20T09:33:04.753701Z",
"url": "https://files.pythonhosted.org/packages/ab/37/eafc53963e31530ebefe40cad57aaa54fb6a6ff74fa1742e4382b60d9c8e/hybridbackend_tf115_cu121-1.0.0-cp38-cp38-manylinux_2_31_x86_64.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-20 09:33:04",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "alibaba",
"github_project": "HybridBackend",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "hybridbackend-tf115-cu121"
}
JSON
