# Wandb Carbs
[](https://badge.fury.io/py/wandb-carbs)
[](LICENSE)
Wandb Carbs is a Python package that integrates the [CARBS](https://github.com/imbue-ai/carbs) (Cost-Aware Bayesian Search) hyperparameter optimization library with [Weights & Biases](https://wandb.ai/) (W&B) sweeps. It enables cost-aware Bayesian parameter search using W&B's sweep functionality, allowing for efficient hyperparameter optimization across multiple parallel agents while storing all state within W&B.
## Table of Contents
- [Features](#features)
- [Installation](#installation)
- [Usage](#usage)
- [How It Works](#how-it-works)
- [Creating a Sweep](#creating-a-sweep)
- [Running an Experiment](#running-an-experiment)
- [Examples](#examples)
- [Notes](#notes)
- [References](#references)
- [License](#license)
## Features
- **Cost-Aware Bayesian Optimization**: Optimize hyperparameters considering both performance and computational cost.
- **Seamless W&B Integration**: Leverage W&B sweeps for distributed hyperparameter optimization with state stored in W&B.
- **Parallel Execution**: Support multiple parallel agents running as part of the sweep.
- **Automatic State Management**: Each agent initializes with observations from completed runs and generates suggestions for incomplete runs.
- **Customizable Parameter Spaces**: Support for `LogSpace`, `LogitSpace`, and `LinearSpace` parameter spaces.
## Installation
You can install Wandb Carbs via pip:
```bash
pip install wandb_carbs
```
## Usage
### Prerequisites
- An account on [Weights & Biases](https://wandb.ai/).
- Basic understanding of [CARBS](https://github.com/imbue-ai/carbs) for Bayesian optimization.
- Familiarity with W&B sweeps.
### How It Works
CARBS performs cost-aware Bayesian parameter search but doesn't natively integrate with W&B sweeps. Wandb Carbs bridges this gap by:
- **Creating a W&B Sweep**: Converts CARBS parameter definitions into a W&B sweep configuration.
- **Initializing Agents**: Each sweep agent run creates a new CARBS instance.
- **State Initialization**: Agents initialize with observations from completed runs in the sweep and generate suggestions for incomplete runs.
- **Parameter Suggestion**: Generates a suggestion for the current run and updates the run's configuration with the suggested parameters.
- **Recording Results**: After the run completes, it stores the CARBS objective and cost in the run's metadata.
This setup allows multiple parallel agents to run as part of the sweep while storing all the state in W&B.
### Creating a Sweep
First, define your parameter spaces using CARBS:
```python
from carbs import Param, LogSpace, LinearSpace
# Define parameter spaces
params = [
Param(name='learning_rate', space=LogSpace(min=1e-5, max=1e-1)),
Param(name='batch_size', space=LinearSpace(min=16, max=128, is_integer=True)),
]
```
Use the `create_sweep` function from Wandb Carbs to create a W&B sweep:
```python
from wandb_carbs import create_sweep
sweep_id = create_sweep(
sweep_name='My CARBS Sweep',
wandb_entity='your_wandb_entity',
wandb_project='your_wandb_project',
carbs_spaces=params
)
```
This function converts the CARBS parameters into a W&B sweep configuration and returns a `sweep_id` that you can use to manage the sweep.
### Running an Experiment
In your training script, integrate Wandb Carbs:
```python
import wandb
from wandb_carbs import WandbCarbs
from carbs import CARBS
def train():
# Initialize W&B run
wandb.init()
# Initialize CARBS
carbs = CARBS(params=params)
# Initialize Wandb Carbs
wandb_carbs = WandbCarbs(carbs=carbs)
# Get hyperparameters suggestion
suggestion = wandb_carbs.suggest()
# Your training code here
# The suggested parameters are in wandb.config
model = build_model(wandb.config)
performance = evaluate_model(model)
# Record observation
objective = performance['accuracy'] # The metric you aim to optimize
cost = performance['training_time'] # Computational cost measure
wandb_carbs.record_observation(objective=objective, cost=cost)
# Finish the run
wandb.finish()
if __name__ == '__main__':
train()
```
**Note:** The suggested hyperparameters are automatically stored in `wandb.config` by the W&B agent; therefore, you don't need to manually update `wandb.config` with the suggestion.
## Examples
### Full Example
```python
import wandb
from wandb_carbs import WandbCarbs, create_sweep
from carbs import CARBS, Param, LogSpace, LinearSpace
# Define parameter spaces
params = [
Param(name='learning_rate', space=LogSpace(min=1e-5, max=1e-1)),
Param(name='batch_size', space=LinearSpace(min=16, max=128, is_integer=True)),
]
# Create a sweep
sweep_id = create_sweep(
sweep_name='My CARBS Sweep',
wandb_entity='your_wandb_entity',
wandb_project='your_wandb_project',
carbs_spaces=params
)
def train():
# Initialize W&B run
wandb.init()
# Initialize CARBS
carbs = CARBS(params=params)
# Initialize Wandb Carbs
wandb_carbs = WandbCarbs(carbs=carbs)
# Get hyperparameters suggestion
suggestion = wandb_carbs.suggest()
# Your training code here
# The suggested parameters are in wandb.config
model = build_model(wandb.config)
performance = evaluate_model(model)
# Record observation
objective = performance['accuracy']
cost = performance['training_time']
wandb_carbs.record_observation(objective=objective, cost=cost)
# Finish the run
wandb.finish()
if __name__ == '__main__':
train()
```
### Managing the Sweep
- Use the `sweep_id` returned by `create_sweep` to monitor and manage your sweep in the W&B dashboard.
- Ensure that all agents (parallel runs) are correctly configured to run the `train` function.
## Notes
- Replace `your_wandb_entity` and `your_wandb_project` with your actual W&B entity and project names.
- The `objective` should be the metric you aim to optimize (e.g., accuracy, F1 score).
- The `cost` can be any measure of computational cost, like training time, memory usage, or FLOPs.
- Ensure that your `build_model` and `evaluate_model` functions use the parameters from `wandb.config`.
## References
- **CARBS**: For more detailed instructions on CARBS, visit the [CARBS GitHub repository](https://github.com/imbue-ai/carbs).
- **Weights & Biases**: Learn more about W&B sweeps [here](https://docs.wandb.ai/guides/sweeps).
## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
Raw data
{
"_id": null,
"home_page": "https://daveey.github.io",
"name": "wandb_carbs",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.10",
"maintainer_email": null,
"keywords": "bayesian-optimization, bayesian-search, cost-aware, pareto-region, carbs, wandb, weights-and-biases",
"author": "David Bloomin",
"author_email": "daveey@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/c7/87/ef4efe397efbf106388fdd8811661aac1448ff7361137db1476a50c2809e/wandb_carbs-0.0.7.tar.gz",
"platform": null,
"description": "# Wandb Carbs\n\n[](https://badge.fury.io/py/wandb-carbs)\n[](LICENSE)\n\nWandb Carbs is a Python package that integrates the [CARBS](https://github.com/imbue-ai/carbs) (Cost-Aware Bayesian Search) hyperparameter optimization library with [Weights & Biases](https://wandb.ai/) (W&B) sweeps. It enables cost-aware Bayesian parameter search using W&B's sweep functionality, allowing for efficient hyperparameter optimization across multiple parallel agents while storing all state within W&B.\n\n## Table of Contents\n\n- [Features](#features)\n- [Installation](#installation)\n- [Usage](#usage)\n - [How It Works](#how-it-works)\n - [Creating a Sweep](#creating-a-sweep)\n - [Running an Experiment](#running-an-experiment)\n- [Examples](#examples)\n- [Notes](#notes)\n- [References](#references)\n- [License](#license)\n\n## Features\n\n- **Cost-Aware Bayesian Optimization**: Optimize hyperparameters considering both performance and computational cost.\n- **Seamless W&B Integration**: Leverage W&B sweeps for distributed hyperparameter optimization with state stored in W&B.\n- **Parallel Execution**: Support multiple parallel agents running as part of the sweep.\n- **Automatic State Management**: Each agent initializes with observations from completed runs and generates suggestions for incomplete runs.\n- **Customizable Parameter Spaces**: Support for `LogSpace`, `LogitSpace`, and `LinearSpace` parameter spaces.\n\n## Installation\n\nYou can install Wandb Carbs via pip:\n\n```bash\npip install wandb_carbs\n```\n\n## Usage\n\n### Prerequisites\n\n- An account on [Weights & Biases](https://wandb.ai/).\n- Basic understanding of [CARBS](https://github.com/imbue-ai/carbs) for Bayesian optimization.\n- Familiarity with W&B sweeps.\n\n### How It Works\n\nCARBS performs cost-aware Bayesian parameter search but doesn't natively integrate with W&B sweeps. Wandb Carbs bridges this gap by:\n\n- **Creating a W&B Sweep**: Converts CARBS parameter definitions into a W&B sweep configuration.\n- **Initializing Agents**: Each sweep agent run creates a new CARBS instance.\n- **State Initialization**: Agents initialize with observations from completed runs in the sweep and generate suggestions for incomplete runs.\n- **Parameter Suggestion**: Generates a suggestion for the current run and updates the run's configuration with the suggested parameters.\n- **Recording Results**: After the run completes, it stores the CARBS objective and cost in the run's metadata.\n\nThis setup allows multiple parallel agents to run as part of the sweep while storing all the state in W&B.\n\n### Creating a Sweep\n\nFirst, define your parameter spaces using CARBS:\n\n```python\nfrom carbs import Param, LogSpace, LinearSpace\n\n# Define parameter spaces\nparams = [\n Param(name='learning_rate', space=LogSpace(min=1e-5, max=1e-1)),\n Param(name='batch_size', space=LinearSpace(min=16, max=128, is_integer=True)),\n]\n```\n\nUse the `create_sweep` function from Wandb Carbs to create a W&B sweep:\n\n```python\nfrom wandb_carbs import create_sweep\n\nsweep_id = create_sweep(\n sweep_name='My CARBS Sweep',\n wandb_entity='your_wandb_entity',\n wandb_project='your_wandb_project',\n carbs_spaces=params\n)\n```\n\nThis function converts the CARBS parameters into a W&B sweep configuration and returns a `sweep_id` that you can use to manage the sweep.\n\n### Running an Experiment\n\nIn your training script, integrate Wandb Carbs:\n\n```python\nimport wandb\nfrom wandb_carbs import WandbCarbs\nfrom carbs import CARBS\n\ndef train():\n # Initialize W&B run\n wandb.init()\n\n # Initialize CARBS\n carbs = CARBS(params=params)\n\n # Initialize Wandb Carbs\n wandb_carbs = WandbCarbs(carbs=carbs)\n\n # Get hyperparameters suggestion\n suggestion = wandb_carbs.suggest()\n\n # Your training code here\n # The suggested parameters are in wandb.config\n model = build_model(wandb.config)\n performance = evaluate_model(model)\n\n # Record observation\n objective = performance['accuracy'] # The metric you aim to optimize\n cost = performance['training_time'] # Computational cost measure\n wandb_carbs.record_observation(objective=objective, cost=cost)\n\n # Finish the run\n wandb.finish()\n\nif __name__ == '__main__':\n train()\n```\n\n**Note:** The suggested hyperparameters are automatically stored in `wandb.config` by the W&B agent; therefore, you don't need to manually update `wandb.config` with the suggestion.\n\n## Examples\n\n### Full Example\n\n```python\nimport wandb\nfrom wandb_carbs import WandbCarbs, create_sweep\nfrom carbs import CARBS, Param, LogSpace, LinearSpace\n\n# Define parameter spaces\nparams = [\n Param(name='learning_rate', space=LogSpace(min=1e-5, max=1e-1)),\n Param(name='batch_size', space=LinearSpace(min=16, max=128, is_integer=True)),\n]\n\n# Create a sweep\nsweep_id = create_sweep(\n sweep_name='My CARBS Sweep',\n wandb_entity='your_wandb_entity',\n wandb_project='your_wandb_project',\n carbs_spaces=params\n)\n\ndef train():\n # Initialize W&B run\n wandb.init()\n\n # Initialize CARBS\n carbs = CARBS(params=params)\n\n # Initialize Wandb Carbs\n wandb_carbs = WandbCarbs(carbs=carbs)\n\n # Get hyperparameters suggestion\n suggestion = wandb_carbs.suggest()\n\n # Your training code here\n # The suggested parameters are in wandb.config\n model = build_model(wandb.config)\n performance = evaluate_model(model)\n\n # Record observation\n objective = performance['accuracy']\n cost = performance['training_time']\n wandb_carbs.record_observation(objective=objective, cost=cost)\n\n # Finish the run\n wandb.finish()\n\nif __name__ == '__main__':\n train()\n```\n\n### Managing the Sweep\n\n- Use the `sweep_id` returned by `create_sweep` to monitor and manage your sweep in the W&B dashboard.\n- Ensure that all agents (parallel runs) are correctly configured to run the `train` function.\n\n## Notes\n\n- Replace `your_wandb_entity` and `your_wandb_project` with your actual W&B entity and project names.\n- The `objective` should be the metric you aim to optimize (e.g., accuracy, F1 score).\n- The `cost` can be any measure of computational cost, like training time, memory usage, or FLOPs.\n- Ensure that your `build_model` and `evaluate_model` functions use the parameters from `wandb.config`.\n\n## References\n\n- **CARBS**: For more detailed instructions on CARBS, visit the [CARBS GitHub repository](https://github.com/imbue-ai/carbs).\n- **Weights & Biases**: Learn more about W&B sweeps [here](https://docs.wandb.ai/guides/sweeps).\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Integrates Imbue's Cost Aware pareto-Region Bayesian Search (CARBS) with Weights and Biases (WanDB)",
"version": "0.0.7",
"project_urls": {
"Homepage": "https://daveey.github.io",
"Repository": "https://github.com/Metta-AI/wandb_carbs"
},
"split_keywords": [
"bayesian-optimization",
" bayesian-search",
" cost-aware",
" pareto-region",
" carbs",
" wandb",
" weights-and-biases"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e99adbfee44503279aea1cdb1346ffd342b8b0bc48852a390ac481812263ff0b",
"md5": "827de2822dfcbf5b882f17b230ab8b23",
"sha256": "f720ff161b02dd76a332e014c50d653d593fdb51edae73a8966a9287367aeec1"
},
"downloads": -1,
"filename": "wandb_carbs-0.0.7-py3-none-any.whl",
"has_sig": false,
"md5_digest": "827de2822dfcbf5b882f17b230ab8b23",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.10",
"size": 9700,
"upload_time": "2024-10-08T01:28:08",
"upload_time_iso_8601": "2024-10-08T01:28:08.188509Z",
"url": "https://files.pythonhosted.org/packages/e9/9a/dbfee44503279aea1cdb1346ffd342b8b0bc48852a390ac481812263ff0b/wandb_carbs-0.0.7-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c787ef4efe397efbf106388fdd8811661aac1448ff7361137db1476a50c2809e",
"md5": "31ae3bb70edb6f85755d1ab8d675a184",
"sha256": "b9347d80840a27b4d809643089df325c14ebadf14c2041ff464f9be660669af9"
},
"downloads": -1,
"filename": "wandb_carbs-0.0.7.tar.gz",
"has_sig": false,
"md5_digest": "31ae3bb70edb6f85755d1ab8d675a184",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.10",
"size": 6138,
"upload_time": "2024-10-08T01:28:09",
"upload_time_iso_8601": "2024-10-08T01:28:09.794836Z",
"url": "https://files.pythonhosted.org/packages/c7/87/ef4efe397efbf106388fdd8811661aac1448ff7361137db1476a50c2809e/wandb_carbs-0.0.7.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-08 01:28:09",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Metta-AI",
"github_project": "wandb_carbs",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "wandb_carbs"
}
JSON
