# experiments_csv - experiment tracking via CSV files
[](https://badge.fury.io/py/experiments_csv)
## Motivation
You run an experiments with various input parameters. You check various input combinations and run the experiment on each combination.
Suddenly, during one of the runs, the program crashes. You have to restart the experiment for the current and future inputs, but you do not want to repeat it for all previous inputs.
## Solution
`experiments_csv` saves all the experiment input and output data into a CSV file. When you restart the experiment, it reads the CSV file and notes all the input combinations for which the experiment already completed. It then automatically skips these input combinations.
## Installation
Basic installation:
```
pip install experiments-csv
```
Installation with plotting ability:
```
pip install experiments-csv[plotting]
```
## Usage
See the demo programs in the [examples](examples/) folder for usage examples. In detail, you should:
1. Write a function for running a single instance of the experiment.
The function may take any number of arguments as inputs.
It should return a dict with any number of arguments as outputs.
For example:
```
def fair_division_algorithm(num_of_agents:int, num_of_items:int, threshold:float, criterion:str):
# ...
# Run the fair division algorithm with the given parameters
# ...
return {
"runtime": runtime,
"max_value": max(values),
"min_value": min(values)
}
```
2. Decide what ranges you want for your input parameters, for example:
```
input_ranges = {
"num_of_agents": [2, 3, 4],
"num_of_items": range(2,10),
"threshold": [0.5],
"criterion": ["proportionality", "envy-freeness"]
}
```
3. Create an Experiment object. It takes as arguments the folder for the experiment results, the experiment file name, and the folder for backups:
```
import experiments_csv
ex = experiments_csv.Experiment("results/", "results.csv", "results/backups")
```
4. Run the experiment:
```
ex.run(fair_division_algorithm, input_ranges)
```
This loops over all combinations of inputs in the ranges you passed (the Cartesian product of the ranges in input_ranges), calls the single-instance function, and records the results in the given CSV file. The CSV file will have a column for every input and output (in the example: num_of_agents, num_of_items, threshold, criterion, runtime, min_value, max_value), and a single row for every run.
If the experiment stops abruptly due to an error, you can re-run the same code, and it will not repeat the experiments with the combinations of arguments it already completed - it will only run the experiments for the combinations not done yet.
If you do want to restart the experiment from scratch, either manually delete the CSV file, or use a new CSV file with a different name, or do
```
ex.clear_previous_results()
```
To log the inputs and outputs during run, you can do:
```
ex.logger.setLevel(logging.INFO)
```
## Plotting
To plot the results, you can use the `plot_results` function, for example:
```
from matplotlib import pyplot as plt
experiments_csv.plot_results(plt, "results.csv", filter={"algorithm": "abc"}, xcolumn="size", ycolumn="runtime", zcolumn="bits")
```
See the [demo programs](demo/) for usage examples.
Raw data
{
"_id": null,
"home_page": "https://github.com/erelsgl/experiments_csv",
"name": "experiments-csv",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "experiments",
"author": "Erel Segal-Halevi",
"author_email": "erelsgl@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/8c/52/b28e67c770d53c7b1298d965e3efb2af3cf4e4c02db81f5ce78e8ab16781/experiments_csv-0.5.3.tar.gz",
"platform": null,
"description": "# experiments_csv - experiment tracking via CSV files\r\n[](https://badge.fury.io/py/experiments_csv)\r\n\r\n## Motivation\r\nYou run an experiments with various input parameters. You check various input combinations and run the experiment on each combination.\r\n\r\nSuddenly, during one of the runs, the program crashes. You have to restart the experiment for the current and future inputs, but you do not want to repeat it for all previous inputs.\r\n\r\n## Solution\r\n`experiments_csv` saves all the experiment input and output data into a CSV file. When you restart the experiment, it reads the CSV file and notes all the input combinations for which the experiment already completed. It then automatically skips these input combinations.\r\n\r\n## Installation\r\n\r\nBasic installation:\r\n\r\n```\r\n pip install experiments-csv\r\n```\r\n\r\nInstallation with plotting ability:\r\n\r\n```\r\n pip install experiments-csv[plotting]\r\n```\r\n\r\n\r\n\r\n## Usage\r\nSee the demo programs in the [examples](examples/) folder for usage examples. In detail, you should:\r\n\r\n1. Write a function for running a single instance of the experiment.\r\nThe function may take any number of arguments as inputs.\r\nIt should return a dict with any number of arguments as outputs.\r\nFor example:\r\n\r\n```\r\n def fair_division_algorithm(num_of_agents:int, num_of_items:int, threshold:float, criterion:str):\r\n # ...\r\n # Run the fair division algorithm with the given parameters\r\n # ...\r\n return {\r\n \"runtime\": runtime,\r\n \"max_value\": max(values),\r\n \"min_value\": min(values)\r\n }\r\n```\r\n\r\n2. Decide what ranges you want for your input parameters, for example:\r\n\r\n```\r\n input_ranges = {\r\n \"num_of_agents\": [2, 3, 4],\r\n \"num_of_items\": range(2,10),\r\n \"threshold\": [0.5],\r\n \"criterion\": [\"proportionality\", \"envy-freeness\"]\r\n }\r\n```\r\n\r\n3. Create an Experiment object. It takes as arguments the folder for the experiment results, the experiment file name, and the folder for backups:\r\n\r\n```\r\n import experiments_csv\r\n ex = experiments_csv.Experiment(\"results/\", \"results.csv\", \"results/backups\")\r\n```\r\n\r\n4. Run the experiment:\r\n\r\n```\r\n ex.run(fair_division_algorithm, input_ranges)\r\n```\r\n\r\nThis loops over all combinations of inputs in the ranges you passed (the Cartesian product of the ranges in input_ranges), calls the single-instance function, and records the results in the given CSV file. The CSV file will have a column for every input and output (in the example: num_of_agents, num_of_items, threshold, criterion, runtime, min_value, max_value), and a single row for every run.\r\n\r\nIf the experiment stops abruptly due to an error, you can re-run the same code, and it will not repeat the experiments with the combinations of arguments it already completed - it will only run the experiments for the combinations not done yet.\r\n\r\nIf you do want to restart the experiment from scratch, either manually delete the CSV file, or use a new CSV file with a different name, or do\r\n\r\n```\r\n ex.clear_previous_results()\r\n```\r\n\r\nTo log the inputs and outputs during run, you can do:\r\n```\r\n ex.logger.setLevel(logging.INFO)\r\n```\r\n\r\n## Plotting\r\n\r\nTo plot the results, you can use the `plot_results` function, for example:\r\n\r\n```\r\n from matplotlib import pyplot as plt\r\n experiments_csv.plot_results(plt, \"results.csv\", filter={\"algorithm\": \"abc\"}, xcolumn=\"size\", ycolumn=\"runtime\", zcolumn=\"bits\")\r\n```\r\n\r\nSee the [demo programs](demo/) for usage examples.\r\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Simple framework for running simulation experiments and recording them in a CSV file",
"version": "0.5.3",
"project_urls": {
"Bug Reports": "https://github.com/erelsgl/experiments_csv/issues",
"Homepage": "https://github.com/erelsgl/experiments_csv",
"Source Code": "https://github.com/erelsgl/experiments_csv"
},
"split_keywords": [
"experiments"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "5cd78e87ba925e25686bcbde1ce2c95041531ced1623536cec47ad07e535c989",
"md5": "1a9faff77488a04a498d0d379d0226cc",
"sha256": "ec4a0e7f5720198ed194179f5ac984abce73047d9ef9ad81ddca4ad6b872a757"
},
"downloads": -1,
"filename": "experiments_csv-0.5.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "1a9faff77488a04a498d0d379d0226cc",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 10621,
"upload_time": "2023-07-23T14:16:04",
"upload_time_iso_8601": "2023-07-23T14:16:04.064582Z",
"url": "https://files.pythonhosted.org/packages/5c/d7/8e87ba925e25686bcbde1ce2c95041531ced1623536cec47ad07e535c989/experiments_csv-0.5.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "8c52b28e67c770d53c7b1298d965e3efb2af3cf4e4c02db81f5ce78e8ab16781",
"md5": "8e99a47fc75c32629ed84e4a3da4fc9c",
"sha256": "7934734750af74d4183a6943959b3f7dab05f327ea273f71a881dbc9d5997df0"
},
"downloads": -1,
"filename": "experiments_csv-0.5.3.tar.gz",
"has_sig": false,
"md5_digest": "8e99a47fc75c32629ed84e4a3da4fc9c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 11636,
"upload_time": "2023-07-23T14:16:05",
"upload_time_iso_8601": "2023-07-23T14:16:05.827771Z",
"url": "https://files.pythonhosted.org/packages/8c/52/b28e67c770d53c7b1298d965e3efb2af3cf4e4c02db81f5ce78e8ab16781/experiments_csv-0.5.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-23 14:16:05",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "erelsgl",
"github_project": "experiments_csv",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "experiments-csv"
}
JSON
