hybridbackend-tf115-cu116


Namehybridbackend-tf115-cu116 JSON
Version 0.7.0.dev1672506489 PyPI version JSON
download
home_pagehttps://github.com/alibaba/HybridBackend
SummaryA high-performance framework for training wide-and-deep recommender systems on heterogeneous cluster
upload_time2022-12-31 17:53:08
maintainer
docs_urlNone
authorAlibaba Group Holding Limited
requires_python
licenseApache License 2.0
keywords deep learning recommendation system
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls. 
            # HybridBackend

[![cibuild](https://github.com/alibaba/HybridBackend/actions/workflows/cibuild.yaml/badge.svg?branch=main&event=push)](https://github.com/alibaba/HybridBackend/actions/workflows/cibuild.yaml)
[![readthedocs](https://readthedocs.org/projects/hybridbackend/badge/?version=latest)](https://hybridbackend.readthedocs.io/en/latest/?badge=latest)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](http://makeapullrequest.com)
[![license](https://img.shields.io/badge/License-Apache%202.0-brightgreen.svg)](https://opensource.org/licenses/Apache-2.0)

HybridBackend is a high-performance framework for training wide-and-deep
recommender systems on heterogeneous cluster.

## Features

- Memory-efficient loading of categorical data

- GPU-efficient orchestration of embedding layers

- Communication-efficient training and evaluation at scale

- Easy to use with existing AI workflows

## Usage

A minimal example:

```python
import tensorflow as tf
import hybridbackend.tensorflow as hb

ds = hb.data.ParquetDataset(filenames, batch_size=batch_size)
ds = ds.apply(hb.data.parse())
# ...

with tf.device('/gpu:0'):
  embs = tf.nn.embedding_lookup_sparse(weights, input_ids)
  # ...
```

Please see [documentation](https://hybridbackend.readthedocs.io/en/latest/) for
more information.

## Install

### Method 1: Pull container images from [PAI DLC](https://www.aliyun.com/activity/bigdata/pai-dlc)

`docker pull registry.cn-shanghai.aliyuncs.com/pai-dlc/hybridbackend:{TAG}`

`{TAG}` | TensorFlow | Python  | CUDA | OS | Columnar Data Loading | Embedding Orchestration | Hybrid Parallelism
------- | ---------- | ------- | ---- | ----- | ------------ | ----------------------- | ------------------
`0.7-tf1.15-py3.8-cu116-ubuntu20.04` | 1.15 | 3.8 | 11.6 | Ubuntu 20.04 | ✓ | ✓ | ✓

### Method 2: Install from PyPI

`pip install {PACKAGE}`

`{PACKAGE}` | TensorFlow | Python  | CUDA | GLIBC | Columnar Data Loading | Embedding Orchestration | Hybrid Parallelism
----------- | ---------- | ------- | ---- | ----- | ------------ | ----------------------- | ------------------
[hybridbackend-tf115-cu114](https://pypi.org/project/hybridbackend-tf115-cu114/) `*` | 1.15 | 3.8 | 11.4 | >=2.31 | ✓ | ✓ | ✓
[hybridbackend-tf115-cu100](https://pypi.org/project/hybridbackend-tf115-cu100/) | 1.15 | 3.6 | 10.0 | >=2.27 | ✓ | ✓ | ✗
[hybridbackend-tf115-cpu](https://pypi.org/project/hybridbackend-tf115-cpu/) | 1.15 | 3.6 | - | >=2.24 | ✓ | ✗ | ✗

> `*` [nvidia-pyindex](https://pypi.org/project/nvidia-pyindex/) must be installed first

### Method 3: Build from source

See [Building Instructions](https://github.com/alibaba/HybridBackend/blob/main/BUILD.md).

## License

HybridBackend is licensed under the [Apache 2.0 License](LICENSE).

## Community

- Please see [Contributing Guide](https://github.com/alibaba/HybridBackend/blob/main/CONTRIBUTING.md)
before your first contribution.

- Please [register as an adopter](https://github.com/alibaba/HybridBackend/blob/main/ADOPTERS.md)
if your organization is interested in adoption. We will discuss
[RoadMap](https://github.com/alibaba/HybridBackend/blob/main/ROADMAP.md) with
registered adopters in advance.

- Please cite [HybridBackend](https://ieeexplore.ieee.org/document/9835450) in your publications if it helps:

  ```text
  @inproceedings{zhang2022picasso,
    title={PICASSO: Unleashing the Potential of GPU-centric Training for Wide-and-deep Recommender Systems},
    author={Zhang, Yuanxing and Chen, Langshi and Yang, Siran and Yuan, Man and Yi, Huimin and Zhang, Jie and Wang, Jiamang and Dong, Jianbo and Xu, Yunlong and Song, Yue and others},
    booktitle={2022 IEEE 38th International Conference on Data Engineering (ICDE)},
    year={2022},
    organization={IEEE}
  }
  ```

## Contact Us

If you would like to share your experiences with others, you are welcome to
contact us in DingTalk:

[<img src="https://github.com/alibaba/HybridBackend/raw/main/docs/images/dingtalk.png" alt="dingtalk" width="200"/>](https://h5.dingtalk.com/circle/healthCheckin.html?dtaction=os&corpId=ding14f3e2ea4b79994cadf6428847a62d4a&51951ad=a84b419&cbdbhh=qwertyuiop)

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/alibaba/HybridBackend",
    "name": "hybridbackend-tf115-cu116",
    "maintainer": "",
    "docs_url": null,
    "requires_python": "",
    "maintainer_email": "",
    "keywords": "deep learning,recommendation system",
    "author": "Alibaba Group Holding Limited",
    "author_email": "",
    "download_url": "https://github.com/alibaba/HybridBackend/tags",
    "platform": null,
    "description": "# HybridBackend\n\n[![cibuild](https://github.com/alibaba/HybridBackend/actions/workflows/cibuild.yaml/badge.svg?branch=main&event=push)](https://github.com/alibaba/HybridBackend/actions/workflows/cibuild.yaml)\n[![readthedocs](https://readthedocs.org/projects/hybridbackend/badge/?version=latest)](https://hybridbackend.readthedocs.io/en/latest/?badge=latest)\n[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](http://makeapullrequest.com)\n[![license](https://img.shields.io/badge/License-Apache%202.0-brightgreen.svg)](https://opensource.org/licenses/Apache-2.0)\n\nHybridBackend is a high-performance framework for training wide-and-deep\nrecommender systems on heterogeneous cluster.\n\n## Features\n\n- Memory-efficient loading of categorical data\n\n- GPU-efficient orchestration of embedding layers\n\n- Communication-efficient training and evaluation at scale\n\n- Easy to use with existing AI workflows\n\n## Usage\n\nA minimal example:\n\n```python\nimport tensorflow as tf\nimport hybridbackend.tensorflow as hb\n\nds = hb.data.ParquetDataset(filenames, batch_size=batch_size)\nds = ds.apply(hb.data.parse())\n# ...\n\nwith tf.device('/gpu:0'):\n  embs = tf.nn.embedding_lookup_sparse(weights, input_ids)\n  # ...\n```\n\nPlease see [documentation](https://hybridbackend.readthedocs.io/en/latest/) for\nmore information.\n\n## Install\n\n### Method 1: Pull container images from [PAI DLC](https://www.aliyun.com/activity/bigdata/pai-dlc)\n\n`docker pull registry.cn-shanghai.aliyuncs.com/pai-dlc/hybridbackend:{TAG}`\n\n`{TAG}` | TensorFlow | Python  | CUDA | OS | Columnar Data Loading | Embedding Orchestration | Hybrid Parallelism\n------- | ---------- | ------- | ---- | ----- | ------------ | ----------------------- | ------------------\n`0.7-tf1.15-py3.8-cu116-ubuntu20.04` | 1.15 | 3.8 | 11.6 | Ubuntu 20.04 | &check; | &check; | &check;\n\n### Method 2: Install from PyPI\n\n`pip install {PACKAGE}`\n\n`{PACKAGE}` | TensorFlow | Python  | CUDA | GLIBC | Columnar Data Loading | Embedding Orchestration | Hybrid Parallelism\n----------- | ---------- | ------- | ---- | ----- | ------------ | ----------------------- | ------------------\n[hybridbackend-tf115-cu114](https://pypi.org/project/hybridbackend-tf115-cu114/) `*` | 1.15 | 3.8 | 11.4 | >=2.31 | &check; | &check; | &check;\n[hybridbackend-tf115-cu100](https://pypi.org/project/hybridbackend-tf115-cu100/) | 1.15 | 3.6 | 10.0 | >=2.27 | &check; | &check; | &cross;\n[hybridbackend-tf115-cpu](https://pypi.org/project/hybridbackend-tf115-cpu/) | 1.15 | 3.6 | - | >=2.24 | &check; | &cross; | &cross;\n\n> `*` [nvidia-pyindex](https://pypi.org/project/nvidia-pyindex/) must be installed first\n\n### Method 3: Build from source\n\nSee [Building Instructions](https://github.com/alibaba/HybridBackend/blob/main/BUILD.md).\n\n## License\n\nHybridBackend is licensed under the [Apache 2.0 License](LICENSE).\n\n## Community\n\n- Please see [Contributing Guide](https://github.com/alibaba/HybridBackend/blob/main/CONTRIBUTING.md)\nbefore your first contribution.\n\n- Please [register as an adopter](https://github.com/alibaba/HybridBackend/blob/main/ADOPTERS.md)\nif your organization is interested in adoption. We will discuss\n[RoadMap](https://github.com/alibaba/HybridBackend/blob/main/ROADMAP.md) with\nregistered adopters in advance.\n\n- Please cite [HybridBackend](https://ieeexplore.ieee.org/document/9835450) in your publications if it helps:\n\n  ```text\n  @inproceedings{zhang2022picasso,\n    title={PICASSO: Unleashing the Potential of GPU-centric Training for Wide-and-deep Recommender Systems},\n    author={Zhang, Yuanxing and Chen, Langshi and Yang, Siran and Yuan, Man and Yi, Huimin and Zhang, Jie and Wang, Jiamang and Dong, Jianbo and Xu, Yunlong and Song, Yue and others},\n    booktitle={2022 IEEE 38th International Conference on Data Engineering (ICDE)},\n    year={2022},\n    organization={IEEE}\n  }\n  ```\n\n## Contact Us\n\nIf you would like to share your experiences with others, you are welcome to\ncontact us in DingTalk:\n\n[<img src=\"https://github.com/alibaba/HybridBackend/raw/main/docs/images/dingtalk.png\" alt=\"dingtalk\" width=\"200\"/>](https://h5.dingtalk.com/circle/healthCheckin.html?dtaction=os&corpId=ding14f3e2ea4b79994cadf6428847a62d4a&51951ad=a84b419&cbdbhh=qwertyuiop)\n",
    "bugtrack_url": null,
    "license": "Apache License 2.0",
    "summary": "A high-performance framework for training wide-and-deep recommender systems on heterogeneous cluster",
    "version": "0.7.0.dev1672506489",
    "split_keywords": [
        "deep learning",
        "recommendation system"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "60fdd41adcbdb757568bf5b58b5cae61",
                "sha256": "31ff35ca781ec50f158c6a9058aa909ee45aad6c30b70f679a0d3969487a671b"
            },
            "downloads": -1,
            "filename": "hybridbackend_tf115_cu116-0.7.0.dev1672506489-cp38-cp38-manylinux_2_31_x86_64.whl",
            "has_sig": false,
            "md5_digest": "60fdd41adcbdb757568bf5b58b5cae61",
            "packagetype": "bdist_wheel",
            "python_version": "cp38",
            "requires_python": null,
            "size": 62164301,
            "upload_time": "2022-12-31T17:53:08",
            "upload_time_iso_8601": "2022-12-31T17:53:08.872282Z",
            "url": "https://files.pythonhosted.org/packages/8c/41/301d331b4aa6011f90aebbf8cbf6850b098b98e29e825d51bd0c7023a455/hybridbackend_tf115_cu116-0.7.0.dev1672506489-cp38-cp38-manylinux_2_31_x86_64.whl",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2022-12-31 17:53:08",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "github_user": "alibaba",
    "github_project": "HybridBackend",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "lcname": "hybridbackend-tf115-cu116"
}
Alibaba Group Holding Limited
Elapsed time: 0.04702s