# HybridBackend
[](https://github.com/alibaba/HybridBackend/actions/workflows/cibuild.yaml)
[](https://hybridbackend.readthedocs.io/en/latest/?badge=latest)
[](http://makeapullrequest.com)
[](https://opensource.org/licenses/Apache-2.0)
HybridBackend is a high-performance framework for training wide-and-deep
recommender systems on heterogeneous cluster.
## Features
- Memory-efficient loading of categorical data
- GPU-efficient orchestration of embedding layers
- Communication-efficient training and evaluation at scale
- Easy to use with existing AI workflows
## Usage
A minimal example:
```python
import tensorflow as tf
import hybridbackend.tensorflow as hb
ds = hb.data.Dataset.from_parquet(filenames)
ds = ds.batch(batch_size)
# ...
with tf.device('/gpu:0'):
embs = tf.nn.embedding_lookup_sparse(weights, input_ids)
# ...
```
Please see [documentation](https://hybridbackend.readthedocs.io/en/latest/) for
more information.
## Install
### Method 1: Install from PyPI
`pip install {PACKAGE}`
| `{PACKAGE}` | Dependency | Python | CUDA | GLIBC | Data Opt. | Embedding Opt. | Parallelism Opt. |
| ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- | ------ | ---- | ------ | --------- | -------------- | ---------------- |
| [hybridbackend-tf115-cu118](https://pypi.org/project/hybridbackend-tf115-cu118/) | [TensorFlow 1.15](https://github.com/NVIDIA/tensorflow) `1` | 3.8 | 11.8 | >=2.31 | ✓ | ✓ | ✓ |
| [hybridbackend-tf115-cu100](https://pypi.org/project/hybridbackend-tf115-cu100/) | [TensorFlow 1.15](https://github.com/tensorflow/tensorflow/tree/r1.15) | 3.6 | 10.0 | >=2.27 | ✓ | ✓ | ✗ |
| [hybridbackend-tf115-cpu](https://pypi.org/project/hybridbackend-tf115-cpu/) | [TensorFlow 1.15](https://github.com/tensorflow/tensorflow/tree/r1.15) | 3.6 | - | >=2.24 | ✓ | ✗ | ✗ |
| [hybridbackend-deeprec2212-cu114](https://pypi.org/project/hybridbackend-deeprec2212-cu114/) | [DeepRec 22.12](https://github.com/alibaba/DeepRec/tree/deeprec2212) `2` | 3.6 | 11.4 | >=2.27 | ✓ | ✓ | ✓ |
> `1`: Suggested docker image: `nvcr.io/nvidia/tensorflow:23.02-tf1-py3`
> `2`: Suggested docker image: `registry.cn-shanghai.aliyuncs.com/pai-dlc/tensorflow-training:deeprec2212-gpu-py36-cu114-ubuntu18.04`
### Method 2: Build from source
See [Building Instructions](https://github.com/alibaba/HybridBackend/blob/main/BUILD.md).
## License
HybridBackend is licensed under the [Apache 2.0 License](LICENSE).
## Community
- Please see [Contributing Guide](https://github.com/alibaba/HybridBackend/blob/main/CONTRIBUTING.md)
before your first contribution.
- Please [register as an adopter](https://github.com/alibaba/HybridBackend/blob/main/ADOPTERS.md)
if your organization is interested in adoption. We will discuss
[RoadMap](https://github.com/alibaba/HybridBackend/blob/main/ROADMAP.md) with
registered adopters in advance.
- Please cite [HybridBackend](https://ieeexplore.ieee.org/document/9835450) in your publications if it helps:
```text
@inproceedings{zhang2022picasso,
title={PICASSO: Unleashing the Potential of GPU-centric Training for Wide-and-deep Recommender Systems},
author={Zhang, Yuanxing and Chen, Langshi and Yang, Siran and Yuan, Man and Yi, Huimin and Zhang, Jie and Wang, Jiamang and Dong, Jianbo and Xu, Yunlong and Song, Yue and others},
booktitle={2022 IEEE 38th International Conference on Data Engineering (ICDE)},
year={2022},
organization={IEEE}
}
```
## Contact Us
If you would like to share your experiences with others, you are welcome to
contact us in DingTalk:
[](https://qr.dingtalk.com/action/joingroup?code=v1,k1,VouhbeuTwXYEgaLzSOE8o6VF2kTHVJ8lw5h93WbZW8o=&_dt_no_comment=1&origin=11)
Raw data
{
"_id": null,
"home_page": "https://github.com/alibaba/HybridBackend",
"name": "hybridbackend-deeprec2212-cu114",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "deep learning,recommendation system",
"author": "Alibaba Group Holding Limited",
"author_email": "",
"download_url": "https://github.com/alibaba/HybridBackend/tags",
"platform": null,
"description": "# HybridBackend\n\n[](https://github.com/alibaba/HybridBackend/actions/workflows/cibuild.yaml)\n[](https://hybridbackend.readthedocs.io/en/latest/?badge=latest)\n[](http://makeapullrequest.com)\n[](https://opensource.org/licenses/Apache-2.0)\n\nHybridBackend is a high-performance framework for training wide-and-deep\nrecommender systems on heterogeneous cluster.\n\n## Features\n\n- Memory-efficient loading of categorical data\n- GPU-efficient orchestration of embedding layers\n- Communication-efficient training and evaluation at scale\n- Easy to use with existing AI workflows\n\n## Usage\n\nA minimal example:\n\n```python\nimport tensorflow as tf\nimport hybridbackend.tensorflow as hb\n\nds = hb.data.Dataset.from_parquet(filenames)\nds = ds.batch(batch_size)\n# ...\n\nwith tf.device('/gpu:0'):\n embs = tf.nn.embedding_lookup_sparse(weights, input_ids)\n # ...\n```\n\nPlease see [documentation](https://hybridbackend.readthedocs.io/en/latest/) for\nmore information.\n\n## Install\n\n### Method 1: Install from PyPI\n\n`pip install {PACKAGE}`\n\n| `{PACKAGE}` | Dependency | Python | CUDA | GLIBC | Data Opt. | Embedding Opt. | Parallelism Opt. |\n| ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- | ------ | ---- | ------ | --------- | -------------- | ---------------- |\n| [hybridbackend-tf115-cu118](https://pypi.org/project/hybridbackend-tf115-cu118/) | [TensorFlow 1.15](https://github.com/NVIDIA/tensorflow) `1` | 3.8 | 11.8 | >=2.31 | ✓ | ✓ | ✓ |\n| [hybridbackend-tf115-cu100](https://pypi.org/project/hybridbackend-tf115-cu100/) | [TensorFlow 1.15](https://github.com/tensorflow/tensorflow/tree/r1.15) | 3.6 | 10.0 | >=2.27 | ✓ | ✓ | ✗ |\n| [hybridbackend-tf115-cpu](https://pypi.org/project/hybridbackend-tf115-cpu/) | [TensorFlow 1.15](https://github.com/tensorflow/tensorflow/tree/r1.15) | 3.6 | - | >=2.24 | ✓ | ✗ | ✗ |\n| [hybridbackend-deeprec2212-cu114](https://pypi.org/project/hybridbackend-deeprec2212-cu114/) | [DeepRec 22.12](https://github.com/alibaba/DeepRec/tree/deeprec2212) `2` | 3.6 | 11.4 | >=2.27 | ✓ | ✓ | ✓ |\n\n> `1`: Suggested docker image: `nvcr.io/nvidia/tensorflow:23.02-tf1-py3`\n\n> `2`: Suggested docker image: `registry.cn-shanghai.aliyuncs.com/pai-dlc/tensorflow-training:deeprec2212-gpu-py36-cu114-ubuntu18.04`\n\n### Method 2: Build from source\n\nSee [Building Instructions](https://github.com/alibaba/HybridBackend/blob/main/BUILD.md).\n\n## License\n\nHybridBackend is licensed under the [Apache 2.0 License](LICENSE).\n\n## Community\n\n- Please see [Contributing Guide](https://github.com/alibaba/HybridBackend/blob/main/CONTRIBUTING.md)\n before your first contribution.\n- Please [register as an adopter](https://github.com/alibaba/HybridBackend/blob/main/ADOPTERS.md)\n if your organization is interested in adoption. We will discuss\n [RoadMap](https://github.com/alibaba/HybridBackend/blob/main/ROADMAP.md) with\n registered adopters in advance.\n- Please cite [HybridBackend](https://ieeexplore.ieee.org/document/9835450) in your publications if it helps:\n\n ```text\n @inproceedings{zhang2022picasso,\n title={PICASSO: Unleashing the Potential of GPU-centric Training for Wide-and-deep Recommender Systems},\n author={Zhang, Yuanxing and Chen, Langshi and Yang, Siran and Yuan, Man and Yi, Huimin and Zhang, Jie and Wang, Jiamang and Dong, Jianbo and Xu, Yunlong and Song, Yue and others},\n booktitle={2022 IEEE 38th International Conference on Data Engineering (ICDE)},\n year={2022},\n organization={IEEE}\n }\n ```\n\n## Contact Us\n\nIf you would like to share your experiences with others, you are welcome to\ncontact us in DingTalk:\n\n[](https://qr.dingtalk.com/action/joingroup?code=v1,k1,VouhbeuTwXYEgaLzSOE8o6VF2kTHVJ8lw5h93WbZW8o=&_dt_no_comment=1&origin=11)\n\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "A high-performance framework for training wide-and-deep recommender systems on heterogeneous cluster",
"version": "0.8.0.dev1679289143",
"split_keywords": [
"deep learning",
"recommendation system"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "42268007a97a5586c099820848cad96916b72be3adcc53d9562f406b0c31ed2a",
"md5": "e4b6764a9c682401f965c134b3956cf8",
"sha256": "c1c1b93a8732c676d14dccd89b65b10914e5dae127241579f5977b6c20fbdc57"
},
"downloads": -1,
"filename": "hybridbackend_deeprec2212_cu114-0.8.0.dev1679289143-cp36-cp36m-manylinux_2_27_x86_64.whl",
"has_sig": false,
"md5_digest": "e4b6764a9c682401f965c134b3956cf8",
"packagetype": "bdist_wheel",
"python_version": "cp36",
"requires_python": null,
"size": 47641559,
"upload_time": "2023-03-20T12:41:01",
"upload_time_iso_8601": "2023-03-20T12:41:01.552866Z",
"url": "https://files.pythonhosted.org/packages/42/26/8007a97a5586c099820848cad96916b72be3adcc53d9562f406b0c31ed2a/hybridbackend_deeprec2212_cu114-0.8.0.dev1679289143-cp36-cp36m-manylinux_2_27_x86_64.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-03-20 12:41:01",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "alibaba",
"github_project": "HybridBackend",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "hybridbackend-deeprec2212-cu114"
}
JSON
