[](https://pypi.org/project/pytensor-federated)
[](https://github.com/michaelosthege/pytensor-federated/actions)
[](https://codecov.io/gh/michaelosthege/pytensor-federated)
# `pytensor-federated`
This package implements federated computing with [PyTensor](https://github.com/pymc-devs/pytensor).
Using `pytensor-federated`, differentiable cost functions can be computed on federated nodes.
Inputs and outputs are transmitted in binary via a bidirectional gRPC stream.
A client side `LogpGradOp` is provided to conveniently embed federated compute operations in PyTensor graphs such as a [PyMC](https://github.com/pymc-devs/pymc) model.
The example code implements a simple Bayesian linear regression to data that is "private" to the federated compute process.
Run each command in its own terminal:
```bash
python demo_node.py
```
```bash
python demo_model.py
```
## Architecture
`pytensor-federated` is designed to be a very generalizable framework for federated computing with gRPC, but it comes with implementations for PyTensor, and specifically for use cases of Bayesian inference.
This is reflected in the actual implementation, where the most basic gRPC service implementation -- the `ArraysToArraysService` -- is wrapped by a few implementation flavors, specifically for common use cases in Bayesian inference.
At the core, everything is built around an `ArraysToArrays` gRPC service, which takes any number of (NumPy) arrays as parameters, and returns any number of (NumPy) arrays as outputs.
The arrays can have arbitrary `dtype` or `shape`, as long as the buffer interface is supported (meaning `dtype=object` doesn't work, but `datetime` dtypes are ok).
This `ArraysToArraysService` can be used to wrap arbitrary model functions, thereby enabling to run model simulations and MCMC/optimization on different machines.
The protobuf files that specify the data types and gRPC interface can be compiled to other programming languages, such that the model implementation could be C++, while MCMC/optimization run in Python.
For the Bayesian inference or optimization use case, it helps to first understand the inputs and outputs of the undelying computation graph.
For example, parameter estimation with a differential equation model requires...
* `observations` to which the model should be fitted
* `timepoints` at which there were observations
* parameters (including initial states) `theta`, some of which are to be estimated
From `timepoints` and parameters `theta`, the `model` predicts `trajectories`.
Together with `observations`, these predictions are fed into some kind of likelihood function, which produces a scalar log-likelihood `log-likelihood` as the output.
Different sub-graphs of this example could be wrapped by an `ArraysToArraysService`:
* `[theta,] -> [log-likelihood,]`
* `[timepoints, theta] -> [trajectories,]`
* `[timepoints, observations, theta] -> [log-likelihood,]`
If the entire model is differentiable, one can even return gradients.
For example, with a linear model: `[slope, intercept] -> [LL, dLL_dslope, dLL_dintercept]`.
The role of PyTensor here is purely technical:
PyTensor is a graph computation framework that implements auto-differentiation.
Wrapping the `ArraysToArraysServiceClient` in PyTensor `Op`s simply makes it easier to build more sophisticated compute graphs.
PyTensor is also the computatation backend for PyMC, which is the most popular framework for Bayesian inference in Python.
## Installation & Contributing
```bash
conda env create -f environment.yml
```
Additional dependencies are needed to compile the [protobufs](./protobufs/):
```bash
conda install -c conda-forge protobuf
pip install --pre betterproto[compiler]
```
```bash
python protobufs/generate.py
```
Set up `pre-commit` for automated code style enforcement:
```bash
pip install pre-commit
pre-commit install
```
Raw data
{
"_id": null,
"home_page": "https://github.com/michaelosthege/pytensor-federated",
"name": "pytensor-federated",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "",
"author": "Michael Osthege",
"author_email": "michael.osthege@outlook.com",
"download_url": "https://files.pythonhosted.org/packages/a3/69/13982fda50ebe93ee660e495a45814393c6755b77e38458c5688c19bebd0/pytensor_federated-1.0.1.tar.gz",
"platform": null,
"description": "[](https://pypi.org/project/pytensor-federated)\n[](https://github.com/michaelosthege/pytensor-federated/actions)\n[](https://codecov.io/gh/michaelosthege/pytensor-federated)\n\n# `pytensor-federated`\nThis package implements federated computing with [PyTensor](https://github.com/pymc-devs/pytensor).\n\nUsing `pytensor-federated`, differentiable cost functions can be computed on federated nodes.\nInputs and outputs are transmitted in binary via a bidirectional gRPC stream.\n\nA client side `LogpGradOp` is provided to conveniently embed federated compute operations in PyTensor graphs such as a [PyMC](https://github.com/pymc-devs/pymc) model.\n\nThe example code implements a simple Bayesian linear regression to data that is \"private\" to the federated compute process.\n\nRun each command in its own terminal:\n\n```bash\npython demo_node.py\n```\n\n```bash\npython demo_model.py\n```\n\n## Architecture\n`pytensor-federated` is designed to be a very generalizable framework for federated computing with gRPC, but it comes with implementations for PyTensor, and specifically for use cases of Bayesian inference.\nThis is reflected in the actual implementation, where the most basic gRPC service implementation -- the `ArraysToArraysService` -- is wrapped by a few implementation flavors, specifically for common use cases in Bayesian inference.\n\nAt the core, everything is built around an `ArraysToArrays` gRPC service, which takes any number of (NumPy) arrays as parameters, and returns any number of (NumPy) arrays as outputs.\nThe arrays can have arbitrary `dtype` or `shape`, as long as the buffer interface is supported (meaning `dtype=object` doesn't work, but `datetime` dtypes are ok).\n\n\n\nThis `ArraysToArraysService` can be used to wrap arbitrary model functions, thereby enabling to run model simulations and MCMC/optimization on different machines.\nThe protobuf files that specify the data types and gRPC interface can be compiled to other programming languages, such that the model implementation could be C++, while MCMC/optimization run in Python.\n\n\n\nFor the Bayesian inference or optimization use case, it helps to first understand the inputs and outputs of the undelying computation graph.\nFor example, parameter estimation with a differential equation model requires...\n* `observations` to which the model should be fitted\n* `timepoints` at which there were observations\n* parameters (including initial states) `theta`, some of which are to be estimated\n\nFrom `timepoints` and parameters `theta`, the `model` predicts `trajectories`.\nTogether with `observations`, these predictions are fed into some kind of likelihood function, which produces a scalar log-likelihood `log-likelihood` as the output.\n\nDifferent sub-graphs of this example could be wrapped by an `ArraysToArraysService`:\n* `[theta,] -> [log-likelihood,]`\n* `[timepoints, theta] -> [trajectories,]`\n* `[timepoints, observations, theta] -> [log-likelihood,]`\n\n\n\n\nIf the entire model is differentiable, one can even return gradients.\nFor example, with a linear model: `[slope, intercept] -> [LL, dLL_dslope, dLL_dintercept]`.\n\nThe role of PyTensor here is purely technical:\nPyTensor is a graph computation framework that implements auto-differentiation.\nWrapping the `ArraysToArraysServiceClient` in PyTensor `Op`s simply makes it easier to build more sophisticated compute graphs.\nPyTensor is also the computatation backend for PyMC, which is the most popular framework for Bayesian inference in Python.\n\n\n## Installation & Contributing\n```bash\nconda env create -f environment.yml\n```\n\nAdditional dependencies are needed to compile the [protobufs](./protobufs/):\n\n```bash\nconda install -c conda-forge protobuf\npip install --pre betterproto[compiler]\n```\n\n```bash\npython protobufs/generate.py\n```\n\nSet up `pre-commit` for automated code style enforcement:\n\n```bash\npip install pre-commit\npre-commit install\n```\n",
"bugtrack_url": null,
"license": "GNU Affero General Public License v3",
"summary": "This package helps to reduce the amount of boilerplate code when creating Airflow DAGs from Python callables.",
"version": "1.0.1",
"project_urls": {
"Download": "https://github.com/michaelosthege/pytensor-federated/tarball/1.0.1",
"Homepage": "https://github.com/michaelosthege/pytensor-federated"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8e443152ec31ec7b347103874ae1ec1b9f25bb19670d2dd0faef68a216081887",
"md5": "ec6893b8a76cb3ade2bbcb5943768946",
"sha256": "21147c7d26260cafaa82968fecda6aa25d600bcf11329ce35987cfbf72b56bd6"
},
"downloads": -1,
"filename": "pytensor_federated-1.0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "ec6893b8a76cb3ade2bbcb5943768946",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 38857,
"upload_time": "2023-11-26T16:06:28",
"upload_time_iso_8601": "2023-11-26T16:06:28.974349Z",
"url": "https://files.pythonhosted.org/packages/8e/44/3152ec31ec7b347103874ae1ec1b9f25bb19670d2dd0faef68a216081887/pytensor_federated-1.0.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a36913982fda50ebe93ee660e495a45814393c6755b77e38458c5688c19bebd0",
"md5": "55885919a711a62b4d0c6de4683a4e2b",
"sha256": "657df91896104575e29eb0b5e6eb197f9bdab2fdc7fcac71af6888fb99d9da43"
},
"downloads": -1,
"filename": "pytensor_federated-1.0.1.tar.gz",
"has_sig": false,
"md5_digest": "55885919a711a62b4d0c6de4683a4e2b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 35315,
"upload_time": "2023-11-26T16:06:30",
"upload_time_iso_8601": "2023-11-26T16:06:30.703548Z",
"url": "https://files.pythonhosted.org/packages/a3/69/13982fda50ebe93ee660e495a45814393c6755b77e38458c5688c19bebd0/pytensor_federated-1.0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-26 16:06:30",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "michaelosthege",
"github_project": "pytensor-federated",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"requirements": [],
"lcname": "pytensor-federated"
}
JSON
