# tls-python-object (tlspyo)
:computer: :globe_with_meridians: :computer:
**A library for easy and secure transfer of python objects over network.**
[](https://github.com/MISTLab/tls-python-object/actions/workflows/python-package.yml)
[](https://tlspyo.readthedocs.io/en/latest/?badge=latest)
:rocket: [Quickstart guide](https://tlspyo.readthedocs.io/en/latest/quickstart.html)
:scroll: [API documentation](https://tlspyo.readthedocs.io/en/latest/)
`tlspyo` provides a simple API to transfer python objects in a robust and safe way via [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security), between several machines (and/or processes) called `Endpoints`.
- `Endpoints` are part of one to several groups,
- Arbitrarily many `Endpoints` connect together via a central `Relay`,
- Each `Endpoint` can *broadcast* or *produce* python objects to the desired groups.
:information_source: _Please carefully read the [Security](#security) section before using `tlspyo` anywhere other than your own secure private network._
## Quick links
- [Documentation](https://tlspyo.readthedocs.io/en/latest/)
- [Principle](#principle)
- [Example usage](#example-usage)
- [Getting started](#getting-started)
- [Installation](#installation)
- [TLS setup](#tls-setup)
- [Producer-consumer example](#a-simple-producer-consumer-example)
- [Security](#security)
- [Custom serialization](#custom-serialization)
## Principle
`tlspyo` provides two classes: `Relay` and `Endpoint`.
* The `Relay` is the center point of all communication between `Endpoints`,
* An `Endpoint` is a node in your network. It connects to the `Relay` and is part of one to several `groups`.
`Endpoints` can do a multitude of things, including:
- *broadcast* python objects to whole groups of `Endpoints`,
- *retrieve* the objects broadcast to the group(s) it is part of,
- *produce* a single object that will be consumed by a single `Endpoint` of a target group,
- *notify* the `Relay` that it is ready to consume a produced object and wait until it receives it.
By default, `tlspyo` relies on Transport Layer Security (TLS) to secure object transfers over network.
## Example usage
```python
from tlspyo import Relay, Endpoint
if __name__ == "__main__":
# Create a relay to allow connectivity between endpoints
re = Relay(
port=3000, # this must be the same on your Relay and Endpoints
password="VerySecurePassword", # must be the same on Relay and Endpoints, AND be strong
local_com_port=3001 # needs to be non-overlapping if Relays/Endpoints are on the same machine
)
# Create an Endpoint in group "producers" (arbitrary name)
prod = Endpoint(
ip_server='127.0.0.1', # IP of the Relay (here: localhost)
port=3000, # must be same port as the Relay
password="VerySecurePassword", # must be same (strong) password as the Relay
groups="producers", # this endpoint is part of the group "producers"
local_com_port=3002
)
# Create a bunch of other Endpoints in group "consumers" (arbitrary name)
cons_1 = Endpoint(
ip_server='127.0.0.1',
port=3000,
password="VerySecurePassword",
groups="consumers", # this endpoint is part of group "consumers"
local_com_port=3003
)
cons_2 = Endpoint(
ip_server='127.0.0.1',
port=3000,
password="VerySecurePassword",
groups="consumers", # this endpoint is part of group "consumers"
local_com_port=3004,
)
# Producer broadcasts an object to any and all endpoint in the destination group "consumers"
prod.broadcast("I HAVE BEEN BROADCAST", "consumers")
# Producer sends an object to the shared queue of destination group "consumers"
prod.produce("I HAVE BEEN PRODUCED", "consumers")
# Consumer 1 notifies the Relay that it wants one produced object destined for "consumers"
cons_1.notify("consumers")
# Consumer 1 is able to retrieve the broadcast AND the consumed object:
res = []
while len(res) < 2:
res += cons_1.receive_all(blocking=True)
print(f"Consumer 1 has received: {res}")
# Consumer 2 is able to retrieve only the broadcast object:
res = cons_2.receive_all(blocking=True)
print(f"Consumer 2 has received: {res}")
# Let us close everyone gracefully:
prod.stop()
cons_1.stop()
cons_2.stop()
re.stop()
```
## Getting started
:information_source: _The machine hosting your `Relay` must be visible to the machines hosting your `Endpoints` through the chosen `port`, via its public `ip_server`.
When using `tlspyo` over the Internet, this typically requires you to configure your router such that it forwards `port` to the IP of the machine hosting your `Relay` on your local network._
### Installation
From PyPI:
```bash
pip install tlspyo
```
### TLS setup:
:information_source: _You can skip this section if you do not want to use TLS.
For instance if you use `tlspyo` on your own private secure network.
When using `tlspyo` over the Internet, you should of course use TLS (read the [security](#security) section if you do not understand why)._
- **Generate TLS credentials:**
`tlspyo` makes the process of generating your TLS credentials straightforward.
:arrow_forward: On the machine that will host your `Relay`, execute the following command line:
```bash
python -m tlspyo --generate
```
This will generate two files in the `tlspyo/credentials` data directory: `key.pem` and `certificate.pem`.
:information_source: _In case you wish to customize your TLS certificate, add the `--custom` option in the previous command line._
Now, your need to retrieve your `certificate.pem` on the machines that will host your `Endpoints`
_(note: you can skip the following steps if your `Endpoints` are on the same machine as your `Relay`)._
This can be achieved via either of the following methods:
- **METHOD 1: manually copy the public certificate (more secure):**
:arrow_forward: On the machines that will host your `Endpoints`, execute:
```bash
python -m tlspyo --credentials
```
This creates and displays the target folder where you need to copy the `certificate.pem` that you generated on the machine that will host the `Relay` (the source folder was displayed when you executed `--generate`).
- **METHOD 2: transfer the public certificate via TCP (not secure):**
:warning: _This method is not secure.
In particular, a man-in-the-middle can impersonate the certificate-broadcasting server and send you a fraudulent TLS certificate.
Use with caution._
:arrow_forward: On the machine that will host your `Relay`, start a certificate-broadcasting server:
```bash
python -m tlspyo --broadcast --port=<port>
```
where `<port>` is a port through which other machines will attempt to retrieve your certificate via TCP.
:arrow_forward: On the machines that will host your `Endpoints`, execute:
```bash
python -m tlspyo --retrieve --ip=<ip> --port=<port>
```
where `<ip>` is the public IP of the certificate-broadcasting machine, and `<port>` is the same as previously.
And you are all set! :sunglasses:
_You can now stop the certificate-broadcasting server by closing the terminal where it runs._
### A Simple Producer-Consumer Example
Let us now see how to make basic usage of `tlspyo`.
In this example, we will create a `Relay` and two `Endpoints` on the same machine, and have them transfer objects via `localhost`.
The full script for this example can be found [here](https://github.com/MISTLab/tls-python-object/blob/main/examples/example_doc.py).
Import the `Relay` and `Endpoint` classes:
```python
from tlspyo import Relay, Endpoint
```
#### Relay
Every `tlspyo` application requires a central `Relay`.
The `Relay` lives on a machine that can be reached by all `Endpoints`.
Typically, you will want this machine to be accessible to your `Endpoints` via your private local network, or via the Internet through [port forwarding](https://en.wikipedia.org/wiki/Port_forwarding).
**Note however that, before you make your `Relay` visible to the Internet via, e.g., port forwarding, it is important that you read the [Security](#security) section.**
Creating a `Relay` is straightforward:
```python
# Initialize a relay to allow connectivity between endpoints
re = Relay(
port=3000, # this must be the same on your Relay and Endpoints
password="VerySecurePassword", # this must be the same on Relay and Endpoints, AND be strong
local_com_port=3001, # this needs to be non-overlapping if Relays/Endpoints live on the same machine
security="TLS" # this is the default; replace by None if you do not want to use TLS
)
```
As soon as your `Relay` is created, it is up and running.
Behind the scenes, it is now waiting for TLS connections from `Endpoints`.
This is done in a background process that listens to `port` 3000 in this example.
This process also communicates with your `Relay` via `local_com_port` 3001 in this example.
Usually, you can ignore `local_com_port` and leave it to the default, unless you use several `Endpoints/Relay` on the same machine, which we will do.
#### Endpoints
Now that our `Relay` is ready, let us create a bunch of `Endpoints`.
This is also pretty straightforward:
```python
# Initialize a producer endpoint
prod = Endpoint(
ip_server='127.0.0.1', # IP of the Relay (here: localhost)
port=3000, # must be same port as the Relay
password="VerySecurePassword", # must be same (strong) password as the Relay
groups="producers", # this endpoint is part of the group "producers"
local_com_port=3002, # must be unique
security="TLS" # this is the default; replace by None if you do not want to use TLS
)
# Initialize consumer endpoints
cons_1 = Endpoint(
ip_server='127.0.0.1',
port=3000,
password="VerySecurePassword",
groups="consumers", # this endpoint is part of group "consumers"
local_com_port=3003, # must be unique
security="TLS"
)
cons_2 = Endpoint(
ip_server='127.0.0.1',
port=3000,
password="VerySecurePassword",
groups="consumers", # this endpoint is part of group "consumers"
local_com_port=3004, # must be unique
security="TLS"
)
```
A nice thing about `tlspyo` is that all communication is handled behind the scenes.
The above calls have all launched processes in the background which handle connection and communication between `Endpoints` through the `Relay`.
Let us now send some objects from the producer to the consumers.
As you may have noticed, we created two different groups here.
We put the producer in a group that we have named "producers", and the consumers in another group that we have called "consumers".
Note that `Endpoint` can be created as being part of any number of groups (`groups` can take a list of strings).
When communicating between endpoints, you can use those groups to make sure the right endpoints receive the right objects.
There are two ways for `Endpoints` to send objects in `tlspyo`:
* **Broadcasting** is used to send an object to all endpoint in a given group.
Furthermore, when an `Endpoint` connects to the `Relay`, it receives the last object that was broadcast to each of his groups.
```python
# Producer broadcasts an object to any and all endpoint in the destination group "consumers"
prod.broadcast("I HAVE BEEN BROADCAST", "consumers")
```
* **Producing** is used to send an object to a queue (FIFO) that is shared between all `Endpoints` of a given group.
The endpoints of the receiving group must **Notify** the `Relay` to get access to an object that has been put in that shared queue.
```python
# Producer sends an object to the shared queue of destination group "consumers"
prod.produce("I HAVE BEEN PRODUCED", "consumers")
# Consumer notifies the Relay that it wants one produced object destined for "consumers"
cons_1.notify("consumers")
```
Once objects reach the consumer endpoint, they are stored in a local queue from which you can retrieve objects whenever you want. To do this, there are multiple options:
* To retrieve from the local queue in a FIFO fashion, use `pop(blocking=blocking, max_items=max_items)`.
* To retrieve the most recent item(s) in the local queue and discard the rest, use `get_last(blocking=blocking, max_items=max_items)`.
* To get all items that are currently in the local queue, use `receive_all(blocking=blocking)`.
:information_source: _Notes:_
* _All calls above return a list of objects. If no objects are returned, the result will be an empty list._
* _If `blocking` is `True`, all methods above will block until at least one item is received (default to `False`)._
* _In`pop` and `get_last`, use `max_items` to specify a maximum number of items to be returned (defaults to 1)._
Now, let our consumers retrieve their loot:
```python
# Consumer 1 is able to retrieve the broadcast AND the consumed object:
res = []
while len(res) < 2:
res += cons_1.receive_all(blocking=True)
print(f"Consumer 1 has received: {res}")
# Consumer 2 is able to retrieve only the broadcast object:
res = cons_2.receive_all(blocking=True)
print(f"Consumer 2 has received: {res}")
```
which prints:
```terminal
Consumer 1 has received: ['I HAVE BEEN BROADCAST', 'I HAVE BEEN PRODUCED']
Consumer 2 has received: ['I HAVE BEEN BROADCAST']
```
Once we are done, we can `stop` all `Endpoints`, and then the `Relay` for the sake of a graceful exit:
```python
# Let us close everyone gracefully:
prod.stop()
cons_1.stop()
cons_2.stop()
re.stop()
```
There you go! You have now sent your first object over the network using `tlspyo`.
Please check out the [API documentation](https://tlspyo.readthedocs.io/en/latest/) for more advanced usage.
## Security
### DISCLAIMER
We are doing our best to make `tlspyo` reasonably secure **when used correctly**, but we provide ABSOLUTELY NO GUARANTEE that it is in any sense.
We are a small open-source community, and we greatly appreciate your contribution to tackle any potentially unreasonable security concerns or important missing information.
Please submit a detailed issue if you are aware of any important exploit not covered in this section.
### Implementation
`tlspyo` relies on the [Twisted](https://twisted.org) framework regarding [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security) implementation and network management.
### Important to know
:warning: Objects transferred by `tlspyo` are serialized with `pickle` by default, so that you can transfer most python objects easily.
NEVER TRANSFER PICKLED OBJECTS OVER A PUBLIC NETWORK WITHOUT `tlspyo`, as this would make you vulnerable to [dangerous exploits](https://davidhamann.de/2020/04/05/exploiting-python-pickle/).
This is because unpickling untrusted pickled objects (i.e., pickled objects created by a malicious user) can lead to arbitrary code execution on your machine.
To prevent this from happening, `tlspyo` provides two interdependent layers of security:
* `Endpoints` authenticate your `Relay` via [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security), which must use [your own secret key and public certificate](#tls-setup).
This ensures your `Endpoints` are indeed talking to your `Relay` and not to some [man-in-the-middle](https://en.wikipedia.org/wiki/Man-in-the-middle_attack), **provided you keep your secret key secure**.
This also prevents anyone else from [eavesdropping](https://en.wikipedia.org/wiki/Eavesdropping) thanks to TLS encryption.
* Every object transfer is protected by a password known to both the `Relay` and the `Endpoints` (the `password` argument).
No object is deserialized without verification of the password.
This ensures that anyone posing as an endpoint will never be able to send undesired objects through your relay **unless they know your password**.
If a malicious user successfully posed as your `Relay`, your `Endpoint` would send them messages that they could decrypt, including your password (this is prevented by TLS when using your own secret key and public certificate).
If they successfully posed as your `Endpoint` they could send malicious pickled objects to your `Relay` (this is prevented by them not knowing your password).
In a nutshell, **when using `tlspyo` you want your password to be as strong as possible, and your TLS secret key to be kept... well, secret** :lock:
For safety-critical applications, we recommend you ditch `pickle` altogether and instead code a secure custom serialization protocol, on top of the TLS layer provided by `tlspyo`.
## Custom serialization
By default, `tlspyo` uses `pickle` for serialization and relies on TLS to prevent attacks.
In advanced application, you may want to use another serialization protocol instead.
For instance, you may want to transfer non-picklable objects, further optimize the security of your application, or simply use a `pickle` serialization protocol or your choice instead of your Python's default.
**In particular, in `security=None` mode (i.e., with TLS disabled) over a public network, using your own secure serialization protocol is critical.**
`tlspyo` makes this easy.
All you need to do is code your own serialization protocol following the `pickle.dumps`/`pickle.loads` signature, and pass it to the `serializer`/`deserializer` arguments of both your `Relay` and `Endpoints`.
For instance:
```python
import pickle as pkl
from tlspyo import Relay, Endpoint
# We define a custom serialization protocol based on pickle for simplicity.
# Of course, this is only for illustration.
# In practice, you may not want to use pickle here.
def my_custom_serializer(obj):
"""
Takes a python object as input and outputs a bytestring
"""
return b"header" + pkl.dumps(["TEST", pkl.dumps(obj)])
def my_custom_deserializer(bytestring):
"""
Takes a bytestring as input and outputs a python object
"""
assert len(bytestring) > len(b"header")
assert bytestring[:len(b"header")] == b"header"
bytestring = bytestring[len(b"header"):]
tmp = pkl.loads(bytestring)
assert isinstance(tmp, list)
assert len(tmp) == 2
assert tmp[0] == "TEST"
obj = pkl.loads(tmp[1])
return obj
if __name__ == '__main__':
re = Relay(
port=3000,
password="VerySecurePassword",
local_com_port=3001,
security="TLS",
serializer=my_custom_serializer,
deserializer=my_custom_deserializer
)
ep = Endpoint(
ip_server='127.0.0.1',
port=3000,
password="VerySecurePassword",
groups="group1",
local_com_port=3002,
security="TLS",
serializer=my_custom_serializer,
deserializer=my_custom_deserializer
)
```
## External links
`tlspyo` is an open-source project hosted at [Polytechnique Montreal - MISTlab](https://mistlab.ca).
We use it in various projects, ranging from parallel [meta-learning](https://github.com/Portiloop) to data transfer between multiple [learning robots](https://github.com/trackmania-rl/tmrl).
`tlspyo` relies on [Twisted](https://twisted.org) to manage network robustness and security.
## Contributing
Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are **greatly appreciated**.
If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement".
Don't forget to give the project a star! Thanks again!
1. Fork the Project
2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)
3. Commit your Changes (`git commit -m 'Add some AmazingFeature'`)
4. Push to the Branch (`git push origin feature/AmazingFeature`)
5. Open a Pull Request
## License
Distributed under the MIT License. See `LICENSE.txt` for more information.
Raw data
{
"_id": null,
"home_page": "https://github.com/MISTLab/tls-python-object",
"name": "tlspyo",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "python,tls,ssl,pickle,transfer,object,transport,twisted",
"author": "Yann Bouteiller, Milo Sobral",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/46/68/eaecd83c142abca8652b3d602c598e9d392e0f34aa7f63b52d8fa26c8d4e/tlspyo-0.3.0.tar.gz",
"platform": null,
"description": "# tls-python-object (tlspyo)\r\n\r\n:computer: :globe_with_meridians: :computer:\r\n\r\n**A library for easy and secure transfer of python objects over network.**\r\n\r\n[](https://github.com/MISTLab/tls-python-object/actions/workflows/python-package.yml)\r\n[](https://tlspyo.readthedocs.io/en/latest/?badge=latest)\r\n\r\n:rocket: [Quickstart guide](https://tlspyo.readthedocs.io/en/latest/quickstart.html)\r\n:scroll: [API documentation](https://tlspyo.readthedocs.io/en/latest/)\r\n\r\n`tlspyo` provides a simple API to transfer python objects in a robust and safe way via [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security), between several machines (and/or processes) called `Endpoints`.\r\n\r\n- `Endpoints` are part of one to several groups,\r\n- Arbitrarily many `Endpoints` connect together via a central `Relay`,\r\n- Each `Endpoint` can *broadcast* or *produce* python objects to the desired groups.\r\n\r\n:information_source: _Please carefully read the [Security](#security) section before using `tlspyo` anywhere other than your own secure private network._\r\n\r\n\r\n## Quick links\r\n\r\n- [Documentation](https://tlspyo.readthedocs.io/en/latest/)\r\n- [Principle](#principle)\r\n- [Example usage](#example-usage)\r\n- [Getting started](#getting-started)\r\n - [Installation](#installation)\r\n - [TLS setup](#tls-setup)\r\n - [Producer-consumer example](#a-simple-producer-consumer-example)\r\n- [Security](#security)\r\n- [Custom serialization](#custom-serialization)\r\n\r\n\r\n## Principle\r\n\r\n`tlspyo` provides two classes: `Relay` and `Endpoint`.\r\n\r\n* The `Relay` is the center point of all communication between `Endpoints`,\r\n* An `Endpoint` is a node in your network. It connects to the `Relay` and is part of one to several `groups`.\r\n\r\n`Endpoints` can do a multitude of things, including:\r\n- *broadcast* python objects to whole groups of `Endpoints`,\r\n- *retrieve* the objects broadcast to the group(s) it is part of,\r\n- *produce* a single object that will be consumed by a single `Endpoint` of a target group,\r\n- *notify* the `Relay` that it is ready to consume a produced object and wait until it receives it.\r\n\r\nBy default, `tlspyo` relies on Transport Layer Security (TLS) to secure object transfers over network.\r\n\r\n## Example usage\r\n\r\n```python\r\nfrom tlspyo import Relay, Endpoint\r\n\r\nif __name__ == \"__main__\":\r\n\r\n # Create a relay to allow connectivity between endpoints\r\n\r\n re = Relay(\r\n port=3000, # this must be the same on your Relay and Endpoints\r\n password=\"VerySecurePassword\", # must be the same on Relay and Endpoints, AND be strong\r\n local_com_port=3001 # needs to be non-overlapping if Relays/Endpoints are on the same machine\r\n )\r\n\r\n # Create an Endpoint in group \"producers\" (arbitrary name)\r\n\r\n prod = Endpoint(\r\n ip_server='127.0.0.1', # IP of the Relay (here: localhost)\r\n port=3000, # must be same port as the Relay\r\n password=\"VerySecurePassword\", # must be same (strong) password as the Relay\r\n groups=\"producers\", # this endpoint is part of the group \"producers\"\r\n local_com_port=3002\r\n )\r\n\r\n # Create a bunch of other Endpoints in group \"consumers\" (arbitrary name)\r\n\r\n cons_1 = Endpoint(\r\n ip_server='127.0.0.1',\r\n port=3000,\r\n password=\"VerySecurePassword\",\r\n groups=\"consumers\", # this endpoint is part of group \"consumers\"\r\n local_com_port=3003\r\n )\r\n\r\n cons_2 = Endpoint(\r\n ip_server='127.0.0.1',\r\n port=3000,\r\n password=\"VerySecurePassword\",\r\n groups=\"consumers\", # this endpoint is part of group \"consumers\"\r\n local_com_port=3004,\r\n )\r\n\r\n # Producer broadcasts an object to any and all endpoint in the destination group \"consumers\"\r\n prod.broadcast(\"I HAVE BEEN BROADCAST\", \"consumers\")\r\n\r\n # Producer sends an object to the shared queue of destination group \"consumers\"\r\n prod.produce(\"I HAVE BEEN PRODUCED\", \"consumers\")\r\n\r\n # Consumer 1 notifies the Relay that it wants one produced object destined for \"consumers\"\r\n cons_1.notify(\"consumers\")\r\n\r\n # Consumer 1 is able to retrieve the broadcast AND the consumed object:\r\n res = []\r\n while len(res) < 2:\r\n res += cons_1.receive_all(blocking=True)\r\n print(f\"Consumer 1 has received: {res}\")\r\n\r\n # Consumer 2 is able to retrieve only the broadcast object:\r\n res = cons_2.receive_all(blocking=True)\r\n print(f\"Consumer 2 has received: {res}\")\r\n\r\n # Let us close everyone gracefully:\r\n prod.stop()\r\n cons_1.stop()\r\n cons_2.stop()\r\n re.stop()\r\n```\r\n\r\n## Getting started\r\n\r\n:information_source: _The machine hosting your `Relay` must be visible to the machines hosting your `Endpoints` through the chosen `port`, via its public `ip_server`.\r\nWhen using `tlspyo` over the Internet, this typically requires you to configure your router such that it forwards `port` to the IP of the machine hosting your `Relay` on your local network._\r\n\r\n\r\n### Installation\r\nFrom PyPI:\r\n```bash\r\npip install tlspyo\r\n```\r\n\r\n### TLS setup:\r\n\r\n:information_source: _You can skip this section if you do not want to use TLS.\r\nFor instance if you use `tlspyo` on your own private secure network.\r\nWhen using `tlspyo` over the Internet, you should of course use TLS (read the [security](#security) section if you do not understand why)._\r\n\r\n- **Generate TLS credentials:**\r\n\r\n`tlspyo` makes the process of generating your TLS credentials straightforward.\r\n\r\n:arrow_forward: On the machine that will host your `Relay`, execute the following command line:\r\n```bash\r\npython -m tlspyo --generate\r\n```\r\nThis will generate two files in the `tlspyo/credentials` data directory: `key.pem` and `certificate.pem`.\r\n\r\n:information_source: _In case you wish to customize your TLS certificate, add the `--custom` option in the previous command line._\r\n\r\nNow, your need to retrieve your `certificate.pem` on the machines that will host your `Endpoints`\r\n_(note: you can skip the following steps if your `Endpoints` are on the same machine as your `Relay`)._\r\n\r\nThis can be achieved via either of the following methods:\r\n\r\n- **METHOD 1: manually copy the public certificate (more secure):**\r\n\r\n:arrow_forward: On the machines that will host your `Endpoints`, execute:\r\n```bash\r\npython -m tlspyo --credentials\r\n```\r\nThis creates and displays the target folder where you need to copy the `certificate.pem` that you generated on the machine that will host the `Relay` (the source folder was displayed when you executed `--generate`).\r\n\r\n- **METHOD 2: transfer the public certificate via TCP (not secure):**\r\n\r\n:warning: _This method is not secure.\r\nIn particular, a man-in-the-middle can impersonate the certificate-broadcasting server and send you a fraudulent TLS certificate.\r\nUse with caution._\r\n\r\n:arrow_forward: On the machine that will host your `Relay`, start a certificate-broadcasting server:\r\n```bash\r\npython -m tlspyo --broadcast --port=<port>\r\n```\r\nwhere `<port>` is a port through which other machines will attempt to retrieve your certificate via TCP.\r\n\r\n:arrow_forward: On the machines that will host your `Endpoints`, execute:\r\n```bash\r\npython -m tlspyo --retrieve --ip=<ip> --port=<port>\r\n```\r\nwhere `<ip>` is the public IP of the certificate-broadcasting machine, and `<port>` is the same as previously.\r\n\r\nAnd you are all set! :sunglasses:\r\n\r\n_You can now stop the certificate-broadcasting server by closing the terminal where it runs._\r\n\r\n\r\n\r\n\r\n\r\n\r\n### A Simple Producer-Consumer Example\r\n\r\nLet us now see how to make basic usage of `tlspyo`.\r\nIn this example, we will create a `Relay` and two `Endpoints` on the same machine, and have them transfer objects via `localhost`.\r\nThe full script for this example can be found [here](https://github.com/MISTLab/tls-python-object/blob/main/examples/example_doc.py).\r\n\r\nImport the `Relay` and `Endpoint` classes:\r\n```python\r\nfrom tlspyo import Relay, Endpoint\r\n```\r\n\r\n#### Relay\r\n\r\nEvery `tlspyo` application requires a central `Relay`.\r\n\r\nThe `Relay` lives on a machine that can be reached by all `Endpoints`.\r\nTypically, you will want this machine to be accessible to your `Endpoints` via your private local network, or via the Internet through [port forwarding](https://en.wikipedia.org/wiki/Port_forwarding).\r\n**Note however that, before you make your `Relay` visible to the Internet via, e.g., port forwarding, it is important that you read the [Security](#security) section.**\r\n\r\nCreating a `Relay` is straightforward:\r\n```python\r\n# Initialize a relay to allow connectivity between endpoints\r\n\r\nre = Relay(\r\n port=3000, # this must be the same on your Relay and Endpoints\r\n password=\"VerySecurePassword\", # this must be the same on Relay and Endpoints, AND be strong\r\n local_com_port=3001, # this needs to be non-overlapping if Relays/Endpoints live on the same machine\r\n security=\"TLS\" # this is the default; replace by None if you do not want to use TLS\r\n)\r\n```\r\nAs soon as your `Relay` is created, it is up and running.\r\nBehind the scenes, it is now waiting for TLS connections from `Endpoints`.\r\nThis is done in a background process that listens to `port` 3000 in this example.\r\nThis process also communicates with your `Relay` via `local_com_port` 3001 in this example.\r\n\r\nUsually, you can ignore `local_com_port` and leave it to the default, unless you use several `Endpoints/Relay` on the same machine, which we will do.\r\n\r\n#### Endpoints\r\nNow that our `Relay` is ready, let us create a bunch of `Endpoints`.\r\nThis is also pretty straightforward:\r\n```python\r\n# Initialize a producer endpoint\r\n\r\nprod = Endpoint(\r\n ip_server='127.0.0.1', # IP of the Relay (here: localhost)\r\n port=3000, # must be same port as the Relay\r\n password=\"VerySecurePassword\", # must be same (strong) password as the Relay\r\n groups=\"producers\", # this endpoint is part of the group \"producers\"\r\n local_com_port=3002, # must be unique\r\n security=\"TLS\" # this is the default; replace by None if you do not want to use TLS\r\n)\r\n\r\n# Initialize consumer endpoints\r\n\r\ncons_1 = Endpoint(\r\n ip_server='127.0.0.1',\r\n port=3000,\r\n password=\"VerySecurePassword\",\r\n groups=\"consumers\", # this endpoint is part of group \"consumers\"\r\n local_com_port=3003, # must be unique\r\n security=\"TLS\"\r\n) \r\n\r\ncons_2 = Endpoint(\r\n ip_server='127.0.0.1',\r\n port=3000,\r\n password=\"VerySecurePassword\",\r\n groups=\"consumers\", # this endpoint is part of group \"consumers\"\r\n local_com_port=3004, # must be unique\r\n security=\"TLS\"\r\n) \r\n```\r\n A nice thing about `tlspyo` is that all communication is handled behind the scenes.\r\n The above calls have all launched processes in the background which handle connection and communication between `Endpoints` through the `Relay`.\r\n\r\n Let us now send some objects from the producer to the consumers.\r\n As you may have noticed, we created two different groups here.\r\n We put the producer in a group that we have named \"producers\", and the consumers in another group that we have called \"consumers\".\r\n Note that `Endpoint` can be created as being part of any number of groups (`groups` can take a list of strings).\r\n When communicating between endpoints, you can use those groups to make sure the right endpoints receive the right objects.\r\n\r\n There are two ways for `Endpoints` to send objects in `tlspyo`:\r\n * **Broadcasting** is used to send an object to all endpoint in a given group.\r\nFurthermore, when an `Endpoint` connects to the `Relay`, it receives the last object that was broadcast to each of his groups.\r\n ```python\r\n # Producer broadcasts an object to any and all endpoint in the destination group \"consumers\"\r\n prod.broadcast(\"I HAVE BEEN BROADCAST\", \"consumers\")\r\n ```\r\n * **Producing** is used to send an object to a queue (FIFO) that is shared between all `Endpoints` of a given group.\r\nThe endpoints of the receiving group must **Notify** the `Relay` to get access to an object that has been put in that shared queue.\r\n\r\n ```python\r\n # Producer sends an object to the shared queue of destination group \"consumers\"\r\n prod.produce(\"I HAVE BEEN PRODUCED\", \"consumers\")\r\n\r\n # Consumer notifies the Relay that it wants one produced object destined for \"consumers\"\r\n cons_1.notify(\"consumers\")\r\n ```\r\n\r\nOnce objects reach the consumer endpoint, they are stored in a local queue from which you can retrieve objects whenever you want. To do this, there are multiple options:\r\n* To retrieve from the local queue in a FIFO fashion, use `pop(blocking=blocking, max_items=max_items)`.\r\n* To retrieve the most recent item(s) in the local queue and discard the rest, use `get_last(blocking=blocking, max_items=max_items)`.\r\n* To get all items that are currently in the local queue, use `receive_all(blocking=blocking)`. \r\n\r\n:information_source: _Notes:_\r\n* _All calls above return a list of objects. If no objects are returned, the result will be an empty list._\r\n* _If `blocking` is `True`, all methods above will block until at least one item is received (default to `False`)._\r\n* _In`pop` and `get_last`, use `max_items` to specify a maximum number of items to be returned (defaults to 1)._\r\n\r\nNow, let our consumers retrieve their loot:\r\n```python\r\n# Consumer 1 is able to retrieve the broadcast AND the consumed object:\r\n res = []\r\n while len(res) < 2:\r\n res += cons_1.receive_all(blocking=True)\r\n print(f\"Consumer 1 has received: {res}\")\r\n\r\n # Consumer 2 is able to retrieve only the broadcast object:\r\n res = cons_2.receive_all(blocking=True)\r\n print(f\"Consumer 2 has received: {res}\")\r\n```\r\n which prints:\r\n```terminal\r\nConsumer 1 has received: ['I HAVE BEEN BROADCAST', 'I HAVE BEEN PRODUCED']\r\nConsumer 2 has received: ['I HAVE BEEN BROADCAST']\r\n```\r\n\r\nOnce we are done, we can `stop` all `Endpoints`, and then the `Relay` for the sake of a graceful exit:\r\n```python\r\n# Let us close everyone gracefully:\r\n prod.stop()\r\n cons_1.stop()\r\n cons_2.stop()\r\n re.stop()\r\n```\r\n\r\nThere you go! You have now sent your first object over the network using `tlspyo`.\r\n\r\nPlease check out the [API documentation](https://tlspyo.readthedocs.io/en/latest/) for more advanced usage.\r\n\r\n## Security\r\n\r\n### DISCLAIMER\r\nWe are doing our best to make `tlspyo` reasonably secure **when used correctly**, but we provide ABSOLUTELY NO GUARANTEE that it is in any sense.\r\nWe are a small open-source community, and we greatly appreciate your contribution to tackle any potentially unreasonable security concerns or important missing information.\r\nPlease submit a detailed issue if you are aware of any important exploit not covered in this section.\r\n\r\n### Implementation\r\n\r\n`tlspyo` relies on the [Twisted](https://twisted.org) framework regarding [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security) implementation and network management.\r\n\r\n### Important to know\r\n\r\n:warning: Objects transferred by `tlspyo` are serialized with `pickle` by default, so that you can transfer most python objects easily.\r\n\r\nNEVER TRANSFER PICKLED OBJECTS OVER A PUBLIC NETWORK WITHOUT `tlspyo`, as this would make you vulnerable to [dangerous exploits](https://davidhamann.de/2020/04/05/exploiting-python-pickle/).\r\nThis is because unpickling untrusted pickled objects (i.e., pickled objects created by a malicious user) can lead to arbitrary code execution on your machine.\r\n\r\nTo prevent this from happening, `tlspyo` provides two interdependent layers of security:\r\n* `Endpoints` authenticate your `Relay` via [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security), which must use [your own secret key and public certificate](#tls-setup).\r\nThis ensures your `Endpoints` are indeed talking to your `Relay` and not to some [man-in-the-middle](https://en.wikipedia.org/wiki/Man-in-the-middle_attack), **provided you keep your secret key secure**.\r\nThis also prevents anyone else from [eavesdropping](https://en.wikipedia.org/wiki/Eavesdropping) thanks to TLS encryption.\r\n* Every object transfer is protected by a password known to both the `Relay` and the `Endpoints` (the `password` argument).\r\nNo object is deserialized without verification of the password.\r\nThis ensures that anyone posing as an endpoint will never be able to send undesired objects through your relay **unless they know your password**.\r\n\r\nIf a malicious user successfully posed as your `Relay`, your `Endpoint` would send them messages that they could decrypt, including your password (this is prevented by TLS when using your own secret key and public certificate).\r\nIf they successfully posed as your `Endpoint` they could send malicious pickled objects to your `Relay` (this is prevented by them not knowing your password).\r\n\r\nIn a nutshell, **when using `tlspyo` you want your password to be as strong as possible, and your TLS secret key to be kept... well, secret** :lock:\r\n\r\nFor safety-critical applications, we recommend you ditch `pickle` altogether and instead code a secure custom serialization protocol, on top of the TLS layer provided by `tlspyo`.\r\n\r\n## Custom serialization\r\n\r\nBy default, `tlspyo` uses `pickle` for serialization and relies on TLS to prevent attacks.\r\n\r\nIn advanced application, you may want to use another serialization protocol instead.\r\nFor instance, you may want to transfer non-picklable objects, further optimize the security of your application, or simply use a `pickle` serialization protocol or your choice instead of your Python's default.\r\n\r\n**In particular, in `security=None` mode (i.e., with TLS disabled) over a public network, using your own secure serialization protocol is critical.**\r\n\r\n`tlspyo` makes this easy.\r\nAll you need to do is code your own serialization protocol following the `pickle.dumps`/`pickle.loads` signature, and pass it to the `serializer`/`deserializer` arguments of both your `Relay` and `Endpoints`.\r\n\r\nFor instance:\r\n```python\r\nimport pickle as pkl\r\nfrom tlspyo import Relay, Endpoint\r\n\r\n# We define a custom serialization protocol based on pickle for simplicity.\r\n# Of course, this is only for illustration.\r\n# In practice, you may not want to use pickle here.\r\n\r\n\r\ndef my_custom_serializer(obj):\r\n \"\"\"\r\n Takes a python object as input and outputs a bytestring\r\n \"\"\"\r\n return b\"header\" + pkl.dumps([\"TEST\", pkl.dumps(obj)])\r\n\r\n\r\ndef my_custom_deserializer(bytestring):\r\n \"\"\"\r\n Takes a bytestring as input and outputs a python object\r\n \"\"\"\r\n assert len(bytestring) > len(b\"header\")\r\n assert bytestring[:len(b\"header\")] == b\"header\"\r\n bytestring = bytestring[len(b\"header\"):]\r\n tmp = pkl.loads(bytestring)\r\n assert isinstance(tmp, list)\r\n assert len(tmp) == 2\r\n assert tmp[0] == \"TEST\"\r\n obj = pkl.loads(tmp[1])\r\n return obj\r\n\r\n\r\nif __name__ == '__main__':\r\n\r\n re = Relay(\r\n port=3000,\r\n password=\"VerySecurePassword\",\r\n local_com_port=3001,\r\n security=\"TLS\",\r\n serializer=my_custom_serializer,\r\n deserializer=my_custom_deserializer\r\n )\r\n\r\n ep = Endpoint(\r\n ip_server='127.0.0.1',\r\n port=3000,\r\n password=\"VerySecurePassword\",\r\n groups=\"group1\",\r\n local_com_port=3002,\r\n security=\"TLS\",\r\n serializer=my_custom_serializer,\r\n deserializer=my_custom_deserializer\r\n)\r\n```\r\n\r\n## External links\r\n\r\n`tlspyo` is an open-source project hosted at [Polytechnique Montreal - MISTlab](https://mistlab.ca).\r\nWe use it in various projects, ranging from parallel [meta-learning](https://github.com/Portiloop) to data transfer between multiple [learning robots](https://github.com/trackmania-rl/tmrl).\r\n\r\n`tlspyo` relies on [Twisted](https://twisted.org) to manage network robustness and security.\r\n\r\n## Contributing\r\n\r\nContributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are **greatly appreciated**.\r\n\r\nIf you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag \"enhancement\".\r\nDon't forget to give the project a star! Thanks again!\r\n\r\n1. Fork the Project\r\n2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)\r\n3. Commit your Changes (`git commit -m 'Add some AmazingFeature'`)\r\n4. Push to the Branch (`git push origin feature/AmazingFeature`)\r\n5. Open a Pull Request\r\n\r\n\r\n## License\r\n\r\nDistributed under the MIT License. See `LICENSE.txt` for more information.\r\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Secure transport of python objects using TLS encryption",
"version": "0.3.0",
"project_urls": {
"Download": "https://github.com/MISTLab/tls-python-object/archive/refs/tags/v0.3.0.tar.gz",
"Homepage": "https://github.com/MISTLab/tls-python-object"
},
"split_keywords": [
"python",
"tls",
"ssl",
"pickle",
"transfer",
"object",
"transport",
"twisted"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4668eaecd83c142abca8652b3d602c598e9d392e0f34aa7f63b52d8fa26c8d4e",
"md5": "ac6ff71635c7d37425acc3ac9d72519e",
"sha256": "fe8ee0323338ab70bd386bd5fb4c582c3a68be0a533128f7f03c36677c7c673c"
},
"downloads": -1,
"filename": "tlspyo-0.3.0.tar.gz",
"has_sig": false,
"md5_digest": "ac6ff71635c7d37425acc3ac9d72519e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 33387,
"upload_time": "2024-01-06T04:31:47",
"upload_time_iso_8601": "2024-01-06T04:31:47.864367Z",
"url": "https://files.pythonhosted.org/packages/46/68/eaecd83c142abca8652b3d602c598e9d392e0f34aa7f63b52d8fa26c8d4e/tlspyo-0.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-06 04:31:47",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "MISTLab",
"github_project": "tls-python-object",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "tlspyo"
}
JSON
