| Name | netfl JSON |
| Version |
1.3.0
JSON |
| download |
| home_page | None |
| Summary | NetFL is a framework for running Federated Learning (FL) experiments in simulated IoT and Fog/Edge computing environments, with support for heterogeneous devices and configurable network conditions. |
| upload_time | 2025-10-19 15:11:12 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.9 |
| license | Apache License 2.0 |
| keywords |
edge-computing
federated-learning
fog-computing
iot
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# NetFL
**NetFL** is a framework for executing _Federated Learning_ (FL) experiments in **simulated IoT and Fog/Edge computing environments**.
It enables the modeling of **heterogeneous and resource-constrained scenarios**, incorporating factors such as computational disparities among devices, limited bandwidth, latency, packet loss, and diverse network topologies.
Using its **native abstractions for tasks, devices, and networks**, NetFL allows researchers to configure and execute FL experiments in a **declarative and reproducible** manner, providing realistic evaluations of algorithms under non-ideal, real-world conditions.
Under the hood, NetFL leverages [Fogbed](https://github.com/larsid/fogbed) for distributed network emulation and [Flower](https://github.com/adap/flower) for federated learning orchestration. These libraries provide robust foundations for virtualization and FL training, and NetFL integrates and extends them into a **unified framework designed specifically for FL research in IoT and Edge Computing**.
## Installation
> **Requirements**: Ubuntu 22.04 LTS or later, Python 3.9 or higher.
### 1. Set up Containernet
Refer to the [Containernet documentation](https://github.com/containernet/containernet) for further details.
Install Ansible:
```
sudo apt-get install ansible
```
Clone the Containernet repository:
```
git clone https://github.com/containernet/containernet.git
```
Run the installation playbook:
```
sudo ansible-playbook -i "localhost," -c local containernet/ansible/install.yml
```
Create and activate a virtual environment:
```
python3 -m venv venv
```
```
source venv/bin/activate
```
> **Note:** The virtual environment **must be activated** before installing or using any Python packages, including Containernet and NetFL.
Install Containernet into the active virtual environment:
```
pip install containernet/.
```
### 2. Install NetFL
While the virtual environment is still active, run:
```
pip install netfl
```
## Running and Understanding a NetFL Experiment
NetFL experiments are designed to be modular and declarative, making it easy to set up federated learning scenarios. The steps below describe how to set up and run an experiment using **NetFL**. The example uses the **MNIST** dataset. You can find more examples in the [examples](./examples/) folder:
### 1. Define the Task
The `Task` class encapsulates the dataset, model, partitioning strategy, and training configuration. You can inherit from it and override methods to specify:
- **Dataset information**: Source, input/label keys, data types
- **Partitioning**: How data is split among clients (IID, non-IID, etc.)
- **Preprocessing**: Any transformations applied to the data
- **Model**: The model architecture and optimizer
- **Aggregation strategy**: Federated averaging or custom strategies
- **Training configs**: Batch size, epochs, number of clients, etc.
> After implementing the task, export an `FLTask` class that extends it for use by NetFL.
```py
from typing import Any
import tensorflow as tf
from keras import models, optimizers
from flwr.server.strategy import Strategy, FedAvg
from netfl.core.task import Task, Dataset, DatasetInfo, DatasetPartitioner, TrainConfigs
from netfl.core.models import cnn3
from netfl.core.partitioners import IidPartitioner
class MNIST(Task):
def dataset_info(self) -> DatasetInfo:
return DatasetInfo(
huggingface_path="ylecun/mnist",
input_key="image",
label_key="label",
input_dtype=tf.float32,
label_dtype=tf.int32,
)
def dataset_partitioner(self) -> DatasetPartitioner:
return IidPartitioner()
def preprocess_dataset(self, dataset: Dataset, training: bool) -> Dataset:
return Dataset(x=tf.divide(dataset.x, 255.0), y=dataset.y)
def model(self) -> models.Model:
return cnn3(
input_shape=(28, 28, 1),
output_classes=10,
optimizer=optimizers.SGD(learning_rate=0.01),
)
def aggregation_strategy(self) -> tuple[type[Strategy], dict[str, Any]]:
return FedAvg, {}
def train_configs(self) -> TrainConfigs:
return TrainConfigs(
batch_size=16,
epochs=2,
num_clients=4,
num_partitions=4,
num_rounds=10,
seed_data=42,
shuffle_data=True,
)
class FLTask(MNIST):
pass
```
### 2. Build the Experiment
NetFL uses resource classes to model the infrastructure. You can create heterogeneous environments by varying these parameters, simulating real-world IoT and edge scenarios:
- `NetworkResource`: Defines network links between clusters/devices
- `DeviceResource`: Represents a server or client device (CPU, memory, bandwidth)
- `ClusterResource`: Groups devices into clusters (cloud, edge, etc.)
Use `NetflExperiment` to assemble the task and resources:
1. Create the network, device, and cluster resources
2. Instantiate the experiment with a name, task, and cluster resources
3. Create server and client devices
4. Assign devices to clusters
5. Register remote workers (for distributed execution)
6. Link clusters with network resources to define topology
> When the worker host `cpu_clock` is set to `BASE_COMPUTE_UNIT`, all resource `cpu_clock` values are interpreted in Docker CPU units (e.g., millicores) instead of GHz.
```py
from netfl.core.experiment import NetflExperiment
from netfl.utils.resources import (
WorkerHostResource,
NetworkResource,
DeviceResource,
ClusterResource,
ClusterResourceType,
BASE_COMPUTE_UNIT,
)
from task import FLTask
task = FLTask()
num_clients = task.train_configs().num_clients
worker_host_resource = WorkerHostResource(cpu_clock=BASE_COMPUTE_UNIT)
server_resource = DeviceResource(
name="server",
cpu_cores=1,
cpu_clock=1.0,
memory=1024,
network_resource=NetworkResource(bw=1000),
worker_host_resource=worker_host_resource,
)
client_a_resource = DeviceResource(
name="client_a",
cpu_cores=1,
cpu_clock=0.5,
memory=512,
network_resource=NetworkResource(bw=100),
worker_host_resource=worker_host_resource,
)
client_b_resource = DeviceResource(
name="client_b",
cpu_cores=1,
cpu_clock=0.25,
memory=512,
network_resource=NetworkResource(bw=50),
worker_host_resource=worker_host_resource,
)
cloud_resource = ClusterResource(
name="cloud",
type=ClusterResourceType.CLOUD,
device_resources=[server_resource],
)
edge_0_resource = ClusterResource(
name="edge_0",
type=ClusterResourceType.EDGE,
device_resources=(num_clients // 2) * [client_a_resource],
)
edge_1_resource = ClusterResource(
name="edge_1",
type=ClusterResourceType.EDGE,
device_resources=(num_clients // 2) * [client_b_resource],
)
exp = NetflExperiment(
name="mnist-exp",
task=task,
cluster_resources=[cloud_resource, edge_0_resource, edge_1_resource],
)
server = exp.create_server(server_resource)
edge_0_clients = exp.create_clients(client_a_resource, edge_0_resource.num_devices)
edge_1_clients = exp.create_clients(client_b_resource, edge_1_resource.num_devices)
cloud = exp.create_cluster(cloud_resource)
edge_0 = exp.create_cluster(edge_0_resource)
edge_1 = exp.create_cluster(edge_1_resource)
exp.add_to_cluster(server, cloud)
for client in edge_0_clients:
exp.add_to_cluster(client, edge_0)
for client in edge_1_clients:
exp.add_to_cluster(client, edge_1)
worker = exp.register_remote_worker("127.0.0.1")
worker.add_cluster(cloud)
worker.add_cluster(edge_0)
worker.add_cluster(edge_1)
worker.create_cluster_link(cloud, edge_0, NetworkResource(bw=10))
worker.create_cluster_link(cloud, edge_1, NetworkResource(bw=20))
try:
exp.start()
except Exception as ex:
print(ex)
finally:
exp.stop()
```
### 3. Run the Experiment
Start the required worker(s), and then run your experiment script. Refer to the [Fogbed documentation](https://larsid.github.io/fogbed/distributed_emulation) for detailed instructions on starting workers.
For example:
```
RunWorker -p=5000
```
```
python3 experiment.py
```
## Running a NetFL Experiment without a Customized Network Topology Using Docker Compose
### 1. Clone the repository
```
git clone https://github.com/larsid/netfl.git
```
### 2. Create the Task
In the project root directory, create or modify a **NetFL Task** and name the file `task.py`. Refer to the examples in the `examples` folder for guidance on task creation.
### 3. Create the Infrastructure
Use Docker Compose to set up the infrastructure, including the server and clients:
```
docker compose up -d
```
### 4. View Training Results
To check the server logs, run:
```
docker logs server
```
Training logs are also stored in the `logs/` folder within the project root directory.
### 5. Shut Down the Infrastructure
To stop and remove all running containers, use the following command:
```
docker compose down
```
## More information
- [NetFL on GitHub](https://github.com/larsid/netfl)
- [NetFL on PyPI](https://pypi.org/project/netfl)
- [NetFL on Docker Hub](https://hub.docker.com/r/netfl/netfl)
## License
NetFL is licensed under the Apache License 2.0. See [LICENSE](./LICENSE) for details.
Raw data
{
"_id": null,
"home_page": null,
"name": "netfl",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "edge-computing, federated-learning, fog-computing, iot",
"author": null,
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/e7/c8/0705d892cce4666a4e045bd564a936f144a1e40c6959e01a1dafee68d94c/netfl-1.3.0.tar.gz",
"platform": null,
"description": "# NetFL\n\n**NetFL** is a framework for executing _Federated Learning_ (FL) experiments in **simulated IoT and Fog/Edge computing environments**.\nIt enables the modeling of **heterogeneous and resource-constrained scenarios**, incorporating factors such as computational disparities among devices, limited bandwidth, latency, packet loss, and diverse network topologies.\n\nUsing its **native abstractions for tasks, devices, and networks**, NetFL allows researchers to configure and execute FL experiments in a **declarative and reproducible** manner, providing realistic evaluations of algorithms under non-ideal, real-world conditions.\n\nUnder the hood, NetFL leverages [Fogbed](https://github.com/larsid/fogbed) for distributed network emulation and [Flower](https://github.com/adap/flower) for federated learning orchestration. These libraries provide robust foundations for virtualization and FL training, and NetFL integrates and extends them into a **unified framework designed specifically for FL research in IoT and Edge Computing**.\n\n## Installation\n\n> **Requirements**: Ubuntu 22.04 LTS or later, Python 3.9 or higher.\n\n### 1. Set up Containernet\n\nRefer to the [Containernet documentation](https://github.com/containernet/containernet) for further details.\n\nInstall Ansible:\n\n```\nsudo apt-get install ansible\n```\n\nClone the Containernet repository:\n\n```\ngit clone https://github.com/containernet/containernet.git\n```\n\nRun the installation playbook:\n\n```\nsudo ansible-playbook -i \"localhost,\" -c local containernet/ansible/install.yml\n```\n\nCreate and activate a virtual environment:\n\n```\npython3 -m venv venv\n```\n\n```\nsource venv/bin/activate\n```\n\n> **Note:** The virtual environment **must be activated** before installing or using any Python packages, including Containernet and NetFL.\n\nInstall Containernet into the active virtual environment:\n\n```\npip install containernet/.\n```\n\n### 2. Install NetFL\n\nWhile the virtual environment is still active, run:\n\n```\npip install netfl\n```\n\n## Running and Understanding a NetFL Experiment\n\nNetFL experiments are designed to be modular and declarative, making it easy to set up federated learning scenarios. The steps below describe how to set up and run an experiment using **NetFL**. The example uses the **MNIST** dataset. You can find more examples in the [examples](./examples/) folder:\n\n### 1. Define the Task\n\nThe `Task` class encapsulates the dataset, model, partitioning strategy, and training configuration. You can inherit from it and override methods to specify:\n\n- **Dataset information**: Source, input/label keys, data types\n- **Partitioning**: How data is split among clients (IID, non-IID, etc.)\n- **Preprocessing**: Any transformations applied to the data\n- **Model**: The model architecture and optimizer\n- **Aggregation strategy**: Federated averaging or custom strategies\n- **Training configs**: Batch size, epochs, number of clients, etc.\n\n> After implementing the task, export an `FLTask` class that extends it for use by NetFL.\n\n```py\nfrom typing import Any\n\nimport tensorflow as tf\nfrom keras import models, optimizers\nfrom flwr.server.strategy import Strategy, FedAvg\n\nfrom netfl.core.task import Task, Dataset, DatasetInfo, DatasetPartitioner, TrainConfigs\nfrom netfl.core.models import cnn3\nfrom netfl.core.partitioners import IidPartitioner\n\n\nclass MNIST(Task):\n def dataset_info(self) -> DatasetInfo:\n return DatasetInfo(\n huggingface_path=\"ylecun/mnist\",\n input_key=\"image\",\n label_key=\"label\",\n input_dtype=tf.float32,\n label_dtype=tf.int32,\n )\n\n def dataset_partitioner(self) -> DatasetPartitioner:\n return IidPartitioner()\n\n def preprocess_dataset(self, dataset: Dataset, training: bool) -> Dataset:\n return Dataset(x=tf.divide(dataset.x, 255.0), y=dataset.y)\n\n def model(self) -> models.Model:\n return cnn3(\n input_shape=(28, 28, 1),\n output_classes=10,\n optimizer=optimizers.SGD(learning_rate=0.01),\n )\n\n def aggregation_strategy(self) -> tuple[type[Strategy], dict[str, Any]]:\n return FedAvg, {}\n\n def train_configs(self) -> TrainConfigs:\n return TrainConfigs(\n batch_size=16,\n epochs=2,\n num_clients=4,\n num_partitions=4,\n num_rounds=10,\n seed_data=42,\n shuffle_data=True,\n )\n\n\nclass FLTask(MNIST):\n pass\n\n```\n\n### 2. Build the Experiment\n\nNetFL uses resource classes to model the infrastructure. You can create heterogeneous environments by varying these parameters, simulating real-world IoT and edge scenarios:\n\n- `NetworkResource`: Defines network links between clusters/devices\n- `DeviceResource`: Represents a server or client device (CPU, memory, bandwidth)\n- `ClusterResource`: Groups devices into clusters (cloud, edge, etc.)\n\nUse `NetflExperiment` to assemble the task and resources:\n\n1. Create the network, device, and cluster resources\n2. Instantiate the experiment with a name, task, and cluster resources\n3. Create server and client devices\n4. Assign devices to clusters\n5. Register remote workers (for distributed execution)\n6. Link clusters with network resources to define topology\n\n> When the worker host `cpu_clock` is set to `BASE_COMPUTE_UNIT`, all resource `cpu_clock` values are interpreted in Docker CPU units (e.g., millicores) instead of GHz.\n\n\n\n```py\nfrom netfl.core.experiment import NetflExperiment\nfrom netfl.utils.resources import (\n WorkerHostResource,\n NetworkResource,\n DeviceResource,\n ClusterResource,\n ClusterResourceType,\n BASE_COMPUTE_UNIT,\n)\n\nfrom task import FLTask\n\n\ntask = FLTask()\nnum_clients = task.train_configs().num_clients\n\nworker_host_resource = WorkerHostResource(cpu_clock=BASE_COMPUTE_UNIT)\n\nserver_resource = DeviceResource(\n name=\"server\",\n cpu_cores=1,\n cpu_clock=1.0,\n memory=1024,\n network_resource=NetworkResource(bw=1000),\n worker_host_resource=worker_host_resource,\n)\n\nclient_a_resource = DeviceResource(\n name=\"client_a\",\n cpu_cores=1,\n cpu_clock=0.5,\n memory=512,\n network_resource=NetworkResource(bw=100),\n worker_host_resource=worker_host_resource,\n)\n\nclient_b_resource = DeviceResource(\n name=\"client_b\",\n cpu_cores=1,\n cpu_clock=0.25,\n memory=512,\n network_resource=NetworkResource(bw=50),\n worker_host_resource=worker_host_resource,\n)\n\ncloud_resource = ClusterResource(\n name=\"cloud\",\n type=ClusterResourceType.CLOUD,\n device_resources=[server_resource],\n)\n\nedge_0_resource = ClusterResource(\n name=\"edge_0\",\n type=ClusterResourceType.EDGE,\n device_resources=(num_clients // 2) * [client_a_resource],\n)\n\nedge_1_resource = ClusterResource(\n name=\"edge_1\",\n type=ClusterResourceType.EDGE,\n device_resources=(num_clients // 2) * [client_b_resource],\n)\n\nexp = NetflExperiment(\n name=\"mnist-exp\",\n task=task,\n cluster_resources=[cloud_resource, edge_0_resource, edge_1_resource],\n)\n\nserver = exp.create_server(server_resource)\nedge_0_clients = exp.create_clients(client_a_resource, edge_0_resource.num_devices)\nedge_1_clients = exp.create_clients(client_b_resource, edge_1_resource.num_devices)\n\ncloud = exp.create_cluster(cloud_resource)\nedge_0 = exp.create_cluster(edge_0_resource)\nedge_1 = exp.create_cluster(edge_1_resource)\n\nexp.add_to_cluster(server, cloud)\n\nfor client in edge_0_clients:\n exp.add_to_cluster(client, edge_0)\nfor client in edge_1_clients:\n exp.add_to_cluster(client, edge_1)\n\nworker = exp.register_remote_worker(\"127.0.0.1\")\nworker.add_cluster(cloud)\nworker.add_cluster(edge_0)\nworker.add_cluster(edge_1)\nworker.create_cluster_link(cloud, edge_0, NetworkResource(bw=10))\nworker.create_cluster_link(cloud, edge_1, NetworkResource(bw=20))\n\ntry:\n exp.start()\nexcept Exception as ex:\n print(ex)\nfinally:\n exp.stop()\n\n```\n\n### 3. Run the Experiment\n\nStart the required worker(s), and then run your experiment script. Refer to the [Fogbed documentation](https://larsid.github.io/fogbed/distributed_emulation) for detailed instructions on starting workers.\n\nFor example:\n\n```\nRunWorker -p=5000\n```\n\n```\npython3 experiment.py\n```\n\n## Running a NetFL Experiment without a Customized Network Topology Using Docker Compose\n\n### 1. Clone the repository\n\n```\ngit clone https://github.com/larsid/netfl.git\n```\n\n### 2. Create the Task\n\nIn the project root directory, create or modify a **NetFL Task** and name the file `task.py`. Refer to the examples in the `examples` folder for guidance on task creation.\n\n### 3. Create the Infrastructure\n\nUse Docker Compose to set up the infrastructure, including the server and clients:\n\n```\ndocker compose up -d\n```\n\n### 4. View Training Results\n\nTo check the server logs, run:\n\n```\ndocker logs server\n```\n\nTraining logs are also stored in the `logs/` folder within the project root directory.\n\n### 5. Shut Down the Infrastructure\n\nTo stop and remove all running containers, use the following command:\n\n```\ndocker compose down\n```\n\n## More information\n\n- [NetFL on GitHub](https://github.com/larsid/netfl)\n- [NetFL on PyPI](https://pypi.org/project/netfl)\n- [NetFL on Docker Hub](https://hub.docker.com/r/netfl/netfl)\n\n## License\n\nNetFL is licensed under the Apache License 2.0. See [LICENSE](./LICENSE) for details.\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "NetFL is a framework for running Federated Learning (FL) experiments in simulated IoT and Fog/Edge computing environments, with support for heterogeneous devices and configurable network conditions.",
"version": "1.3.0",
"project_urls": {
"Homepage": "https://github.com/larsid/netfl"
},
"split_keywords": [
"edge-computing",
" federated-learning",
" fog-computing",
" iot"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "2b77d709a2602f8598dc4e2eb6aea4003a4eae50d5ba4bfa295401a3312df6d2",
"md5": "2a6f0e9e00e1014b34fbf1740062f566",
"sha256": "dc7986f73fc658c7b2c364175263b5b7cd67473a3cb9693d0f7b50cd46de6388"
},
"downloads": -1,
"filename": "netfl-1.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "2a6f0e9e00e1014b34fbf1740062f566",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 25703,
"upload_time": "2025-10-19T15:11:11",
"upload_time_iso_8601": "2025-10-19T15:11:11.017882Z",
"url": "https://files.pythonhosted.org/packages/2b/77/d709a2602f8598dc4e2eb6aea4003a4eae50d5ba4bfa295401a3312df6d2/netfl-1.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "e7c80705d892cce4666a4e045bd564a936f144a1e40c6959e01a1dafee68d94c",
"md5": "94e9462ef3128530d4661ae413f29138",
"sha256": "6be36bb6aff4298d0174e998fd6a144f9420f92de6f922f91f766bc58f4e498d"
},
"downloads": -1,
"filename": "netfl-1.3.0.tar.gz",
"has_sig": false,
"md5_digest": "94e9462ef3128530d4661ae413f29138",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 127752,
"upload_time": "2025-10-19T15:11:12",
"upload_time_iso_8601": "2025-10-19T15:11:12.257844Z",
"url": "https://files.pythonhosted.org/packages/e7/c8/0705d892cce4666a4e045bd564a936f144a1e40c6959e01a1dafee68d94c/netfl-1.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-10-19 15:11:12",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "larsid",
"github_project": "netfl",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "netfl"
}
