# HybridBackend
[](https://github.com/alibaba/HybridBackend/actions/workflows/cibuild.yaml)
[](https://hybridbackend.readthedocs.io/en/latest/?badge=latest)
[](http://makeapullrequest.com)
[](https://opensource.org/licenses/Apache-2.0)
HybridBackend is a high-performance framework for training wide-and-deep
recommender systems on heterogeneous cluster.
## Features
- Memory-efficient loading of categorical data
- GPU-efficient orchestration of embedding layers
- Communication-efficient training and evaluation at scale
- Easy to use with existing AI workflows
## Usage
A minimal example:
```python
import tensorflow as tf
import hybridbackend.tensorflow as hb
ds = hb.data.Dataset.from_parquet(filenames)
ds = ds.batch(batch_size)
# ...
with tf.device('/gpu:0'):
embs = tf.nn.embedding_lookup_sparse(weights, input_ids)
# ...
```
Please see [documentation](https://hybridbackend.readthedocs.io/en/latest/) for
more information.
## Install
### Method 1: Install from PyPI
`pip install {PACKAGE}`
| `{PACKAGE}` | Dependency | Python | CUDA | GLIBC | Data Opt. | Embedding Opt. | Parallelism Opt. |
| ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- | ------ | ---- | ------ | --------- | -------------- | ---------------- |
| [hybridbackend-tf115-cu121](https://pypi.org/project/hybridbackend-tf115-cu121/) | [TensorFlow 1.15](https://github.com/NVIDIA/tensorflow) | 3.8 | 12.1 | >=2.31 | ✓ | ✓ | ✓ |
| [hybridbackend-tf115-cu100](https://pypi.org/project/hybridbackend-tf115-cu100/) | [TensorFlow 1.15](https://github.com/tensorflow/tensorflow/tree/r1.15) | 3.6 | 10.0 | >=2.27 | ✓ | ✓ | ✗ |
| [hybridbackend-tf115-cpu](https://pypi.org/project/hybridbackend-tf115-cpu/) | [TensorFlow 1.15](https://github.com/tensorflow/tensorflow/tree/r1.15) | 3.6 | - | >=2.24 | ✓ | ✗ | ✗ |
### Method 2: Build from source
See [Building Instructions](https://github.com/alibaba/HybridBackend/blob/main/BUILD.md).
We also provide built docker images for latest [DeepRec](https://github.com/alibaba/DeepRec):
`registry.cn-shanghai.aliyuncs.com/pai-dlc/hybridbackend:1.0.0-deeprec-py3.6-cu114-ubuntu18.04`
## License
HybridBackend is licensed under the [Apache 2.0 License](LICENSE).
## Community
- Please see [Contributing Guide](https://github.com/alibaba/HybridBackend/blob/main/CONTRIBUTING.md)
before your first contribution.
- Please [register as an adopter](https://github.com/alibaba/HybridBackend/blob/main/ADOPTERS.md)
if your organization is interested in adoption. We will discuss
[RoadMap](https://github.com/alibaba/HybridBackend/blob/main/ROADMAP.md) with
registered adopters in advance.
- Please cite [HybridBackend](https://ieeexplore.ieee.org/document/9835450) in your publications if it helps:
```text
@inproceedings{zhang2022picasso,
title={PICASSO: Unleashing the Potential of GPU-centric Training for Wide-and-deep Recommender Systems},
author={Zhang, Yuanxing and Chen, Langshi and Yang, Siran and Yuan, Man and Yi, Huimin and Zhang, Jie and Wang, Jiamang and Dong, Jianbo and Xu, Yunlong and Song, Yue and others},
booktitle={2022 IEEE 38th International Conference on Data Engineering (ICDE)},
year={2022},
organization={IEEE}
}
```
## Contact Us
If you would like to share your experiences with others, you are welcome to
contact us in DingTalk:
[](https://qr.dingtalk.com/action/joingroup?code=v1,k1,VouhbeuTwXYEgaLzSOE8o6VF2kTHVJ8lw5h93WbZW8o=&_dt_no_comment=1&origin=11)
Raw data
{
"_id": null,
"home_page": "https://github.com/alibaba/HybridBackend",
"name": "hybridbackend-tf115-cpu",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "deep learning,recommendation system",
"author": "Alibaba Group Holding Limited",
"author_email": "",
"download_url": "https://github.com/alibaba/HybridBackend/tags",
"platform": null,
"description": "# HybridBackend\n\n[](https://github.com/alibaba/HybridBackend/actions/workflows/cibuild.yaml)\n[](https://hybridbackend.readthedocs.io/en/latest/?badge=latest)\n[](http://makeapullrequest.com)\n[](https://opensource.org/licenses/Apache-2.0)\n\nHybridBackend is a high-performance framework for training wide-and-deep\nrecommender systems on heterogeneous cluster.\n\n## Features\n\n- Memory-efficient loading of categorical data\n- GPU-efficient orchestration of embedding layers\n- Communication-efficient training and evaluation at scale\n- Easy to use with existing AI workflows\n\n## Usage\n\nA minimal example:\n\n```python\nimport tensorflow as tf\nimport hybridbackend.tensorflow as hb\n\nds = hb.data.Dataset.from_parquet(filenames)\nds = ds.batch(batch_size)\n# ...\n\nwith tf.device('/gpu:0'):\n embs = tf.nn.embedding_lookup_sparse(weights, input_ids)\n # ...\n```\n\nPlease see [documentation](https://hybridbackend.readthedocs.io/en/latest/) for\nmore information.\n\n## Install\n\n### Method 1: Install from PyPI\n\n`pip install {PACKAGE}`\n\n| `{PACKAGE}` | Dependency | Python | CUDA | GLIBC | Data Opt. | Embedding Opt. | Parallelism Opt. |\n| ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- | ------ | ---- | ------ | --------- | -------------- | ---------------- |\n| [hybridbackend-tf115-cu121](https://pypi.org/project/hybridbackend-tf115-cu121/) | [TensorFlow 1.15](https://github.com/NVIDIA/tensorflow) | 3.8 | 12.1 | >=2.31 | ✓ | ✓ | ✓ |\n| [hybridbackend-tf115-cu100](https://pypi.org/project/hybridbackend-tf115-cu100/) | [TensorFlow 1.15](https://github.com/tensorflow/tensorflow/tree/r1.15) | 3.6 | 10.0 | >=2.27 | ✓ | ✓ | ✗ |\n| [hybridbackend-tf115-cpu](https://pypi.org/project/hybridbackend-tf115-cpu/) | [TensorFlow 1.15](https://github.com/tensorflow/tensorflow/tree/r1.15) | 3.6 | - | >=2.24 | ✓ | ✗ | ✗ |\n\n### Method 2: Build from source\n\nSee [Building Instructions](https://github.com/alibaba/HybridBackend/blob/main/BUILD.md).\n\nWe also provide built docker images for latest [DeepRec](https://github.com/alibaba/DeepRec): \n`registry.cn-shanghai.aliyuncs.com/pai-dlc/hybridbackend:1.0.0-deeprec-py3.6-cu114-ubuntu18.04`\n\n## License\n\nHybridBackend is licensed under the [Apache 2.0 License](LICENSE).\n\n## Community\n\n- Please see [Contributing Guide](https://github.com/alibaba/HybridBackend/blob/main/CONTRIBUTING.md)\n before your first contribution.\n- Please [register as an adopter](https://github.com/alibaba/HybridBackend/blob/main/ADOPTERS.md)\n if your organization is interested in adoption. We will discuss\n [RoadMap](https://github.com/alibaba/HybridBackend/blob/main/ROADMAP.md) with\n registered adopters in advance.\n- Please cite [HybridBackend](https://ieeexplore.ieee.org/document/9835450) in your publications if it helps:\n\n ```text\n @inproceedings{zhang2022picasso,\n title={PICASSO: Unleashing the Potential of GPU-centric Training for Wide-and-deep Recommender Systems},\n author={Zhang, Yuanxing and Chen, Langshi and Yang, Siran and Yuan, Man and Yi, Huimin and Zhang, Jie and Wang, Jiamang and Dong, Jianbo and Xu, Yunlong and Song, Yue and others},\n booktitle={2022 IEEE 38th International Conference on Data Engineering (ICDE)},\n year={2022},\n organization={IEEE}\n }\n ```\n\n## Contact Us\n\nIf you would like to share your experiences with others, you are welcome to\ncontact us in DingTalk:\n\n[](https://qr.dingtalk.com/action/joingroup?code=v1,k1,VouhbeuTwXYEgaLzSOE8o6VF2kTHVJ8lw5h93WbZW8o=&_dt_no_comment=1&origin=11)\n\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "A high-performance framework for training wide-and-deep recommender systems on heterogeneous cluster",
"version": "1.0.0",
"project_urls": {
"Bug Tracker": "https://github.com/alibaba/HybridBackend/issues",
"Documentation": "https://hybridbackend.readthedocs.io/en/latest/",
"Download": "https://github.com/alibaba/HybridBackend/tags",
"Homepage": "https://github.com/alibaba/HybridBackend",
"Source Code": "https://github.com/alibaba/HybridBackend"
},
"split_keywords": [
"deep learning",
"recommendation system"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "d926811b7b0061529a40ec441215a52cca6ba41607bc38909a66fa86027a4c91",
"md5": "c3f2d039bba7cd9c69a84dc544c38c53",
"sha256": "b91f5ce30279b02e3aa0ceb33071e6ee48fb09d732b3331383ad8e3b9f923e89"
},
"downloads": -1,
"filename": "hybridbackend_tf115_cpu-1.0.0-cp36-cp36m-manylinux_2_24_x86_64.whl",
"has_sig": false,
"md5_digest": "c3f2d039bba7cd9c69a84dc544c38c53",
"packagetype": "bdist_wheel",
"python_version": "cp36",
"requires_python": null,
"size": 32962272,
"upload_time": "2023-07-20T09:26:37",
"upload_time_iso_8601": "2023-07-20T09:26:37.989039Z",
"url": "https://files.pythonhosted.org/packages/d9/26/811b7b0061529a40ec441215a52cca6ba41607bc38909a66fa86027a4c91/hybridbackend_tf115_cpu-1.0.0-cp36-cp36m-manylinux_2_24_x86_64.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-20 09:26:37",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "alibaba",
"github_project": "HybridBackend",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "hybridbackend-tf115-cpu"
}
JSON
