| Name | autotuning_methodology JSON |
| Version |
1.0.0
JSON |
| download |
| home_page | None |
| Summary | Software package easing implementation of the guidelines of the 2024 paper 'A Methodology for Comparing Auto-Tuning Optimization Algorithms' (https://doi.org/10.1016/j.future.2024.05.021). The DOI of this software is https://doi.org/10.5281/zenodo.11243974. |
| upload_time | 2024-05-31 10:32:03 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.9 |
| license | None |
| keywords |
autotuning
auto-tuning
methodology
scientific
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
|
# Autotuning Methodology Software Package
<p align="center">
<img width="25%" src="https://autotuningassociation.github.io/autotuning_methodology/_static/logo_autotuning_methodology.svg" />
</p>
[](https://github.com/autotuningassociation/autotuning_methodology/actions/workflows/build-test-python-package.yml)
[](https://autotuningassociation.github.io/autotuning_methodology/)
[](https://pypi.org/project/autotuning_methodology/)
[](https://pypi.org/project/autotuning_methodology/)
This repository contains the software package accompanying [the paper "A Methodology for Comparing Auto-Tuning Optimization Algorithms"](https://doi.org/10.1016/j.future.2024.05.021).
It makes the guidelines in the methodology easy to apply: simply specify the `.json` file, run `autotuning_visualize [path_to_json]` and wait for the results!
### Limitations & Future Work
Currently, the stable releases of this software package are compatible with [Kernel Tuner](https://github.com/KernelTuner/kernel_tuner) and [KTT](https://github.com/HiPerCoRe/KTT), as in the paper. We plan to soon extend this to support more frameworks.
## Installation
The package can be installed with `pip install autotuning_methodology`.
Alternatively, it can be installed by cloning this repository and running `pip install .` in the root of the cloned project.
Python >= 3.9 is supported.
## Notable features
- Official software by the authors of the methodology-defining paper.
- Supports [BAT benchmark suite](https://github.com/NTNU-HPC-Lab/BAT), [KTT](https://github.com/HiPerCoRe/KTT), and [Kernel Tuner](https://github.com/KernelTuner/kernel_tuner).
- Split executer and visualizer to allow running the algorithms on a cluster and visualize locally.
- Caching built-in to avoid duplicate executions.
- Planned support for T1 input and T4 output files.
- Notebook / interactive window mode; if enabled, plots are shown in the notebook / window instead of written to a folder.
<img width="674" alt="example run in interactive window" src="https://user-images.githubusercontent.com/6725103/232880006-70a05b0e-a4e4-4cc7-bea9-473959c474c2.png">
<img width="483" alt="example run in interactive window 2" src="https://user-images.githubusercontent.com/6725103/232881244-d432ea8e-801a-44b1-9acb-b98cc1b740ac.png">
## Usage
### Entry points
There are two entry points defined: `autotuning_experiment` and `autotuning_visualize`. Both take one argument: the path to an experiment file (see below).
### Input files
To get started, all you need is an experiments file. This is a `json` file that describes the details of your comparison: which algorithms to use, which programs to tune on which devices, the graphs to output and so on.
You can find the API and an example `experiments.json` in the [documentation](https://autotuningassociation.github.io/autotuning_methodology/modules.html).
### File references
As we are dealing with input and output files, file references matter.
When calling the entrypoints, we are already providing the path to an experiments file.
File references in experiments files are relative to the location of the experiment file itself.
File references in tuning scripts are relative to the location of the tuning script itself. Tuning scripts need to have the global literals `file_path_results` and `file_path_metadata` for this package to know where to get the results.
Plots outputted by this package are placed in a folder called `generated_plots` relative to the current working directory.
## Pipeline
The below schematics show the pipeline implemented by this tool as described in the paper.
<!-- <img width="100%" alt="flowchart performance curve generation" src="https://autotuningassociation.github.io/autotuning_methodology/_static/flowchart_performance_curve_generation.svg"> -->
The first flowchart shows the tranformation of raw, stochastic optimization algorithm data to a performance curve.
<!-- <img width="100%" alt="flowchart output generation" src="docs/source/flowchart_output_generation.svg"> -->
The second flowchart shows the adaption of performance curves of various optimization algorithms and search spaces to the desired output.
## Contributing
### Setup
If you're looking to contribute to this package: welcome!
Start out by installing with `pip install -e .[dev]` (this installs the package in editable mode alongside the development dependencies).
During development, unit and integration tests can be ran with `pytest`.
[Black](https://pypi.org/project/black/) is used as a formatter, and [Ruff](https://pypi.org/project/ruff/) is used as a linter to check the formatting, import sorting et cetera.
When using Visual Studio Code, use the `settings.json` found in `.vscode` to automatically have the correct linting, formatting and sorting during development.
In addition, install the extensions recommended by us by searching for `@recommended:workspace` in the extensions tab for a better development experience.
### Documentation
The documentation can be found [here](https://autotuningassociation.github.io/autotuning_methodology/).
Locally, the documentation can be build with `make clean html` from the `docs` folder, but the package must have been installed in editable mode with `pip install -e .`.
Upon pushing to main or publishing a version, this documentation will be built and published to the GitHub Pages.
The Docstring format used is Google. Type hints are to be included in the function signature and therefor omitted from the docstring. In Visual Studio Code, the `autoDocstring` extension can be used to automatically infer docstrings. When referrring to functions and parameters in the docstring outside of their definition, use double backquotes to be compatible with both MarkDown and ReStructuredText, e.g.: *"skip_draws_check: skips checking that each value in ``draws`` is in the ``dist``."*.
### Tests
Before contributing a pull request, please run `nox` and ensure it has no errors. This will test against all Python versions explicitely supported by this package, and will check whether the correct formatting has been applied.
Upon submitting a pull request or pushing to main, these same checks will be ran remotely via GitHub Actions.
### Publishing
For publising the package to PyPI (the Python Package Index), we use [Flit](https://flit.pypa.io) and the [to-pypi-using-flit](https://github.com/AsifArmanRahman/to-pypi-using-flit/tree/v1/) GitHub Action to automate this.
[Semantic version numbering](https://semver.org) is used as follows: `MAJOR.Minor.patch`.
`MAJOR` version for incompatible API changes.
`Minor` version for functionality in a backward compatible manner.
`patch` version for backward compatible bug fixes.
In addition, [PEP 440](https://peps.python.org/pep-0440/) is adhered to, specifically for [pre-release versioning](https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/#id62).
Raw data
{
"_id": null,
"home_page": null,
"name": "autotuning_methodology",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "autotuning, auto-tuning, methodology, scientific",
"author": null,
"author_email": "Floris-Jan Willemsen <fjwillemsen97@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/70/00/c07c6a010f703bffcd0771e640e543adae00879f50433770891322e0da51/autotuning_methodology-1.0.0.tar.gz",
"platform": null,
"description": "# Autotuning Methodology Software Package\n\n<p align=\"center\">\n <img width=\"25%\" src=\"https://autotuningassociation.github.io/autotuning_methodology/_static/logo_autotuning_methodology.svg\" />\n</p>\n\n\n[](https://github.com/autotuningassociation/autotuning_methodology/actions/workflows/build-test-python-package.yml)\n[](https://autotuningassociation.github.io/autotuning_methodology/)\n[](https://pypi.org/project/autotuning_methodology/)\n[](https://pypi.org/project/autotuning_methodology/)\n\n\n\n \nThis repository contains the software package accompanying [the paper \"A Methodology for Comparing Auto-Tuning Optimization Algorithms\"](https://doi.org/10.1016/j.future.2024.05.021). \nIt makes the guidelines in the methodology easy to apply: simply specify the `.json` file, run `autotuning_visualize [path_to_json]` and wait for the results!\n\n### Limitations & Future Work\nCurrently, the stable releases of this software package are compatible with [Kernel Tuner](https://github.com/KernelTuner/kernel_tuner) and [KTT](https://github.com/HiPerCoRe/KTT), as in the paper. We plan to soon extend this to support more frameworks. \n\n## Installation\nThe package can be installed with `pip install autotuning_methodology`. \nAlternatively, it can be installed by cloning this repository and running `pip install .` in the root of the cloned project. \nPython >= 3.9 is supported. \n\n## Notable features\n- Official software by the authors of the methodology-defining paper. \n- Supports [BAT benchmark suite](https://github.com/NTNU-HPC-Lab/BAT), [KTT](https://github.com/HiPerCoRe/KTT), and [Kernel Tuner](https://github.com/KernelTuner/kernel_tuner).\n- Split executer and visualizer to allow running the algorithms on a cluster and visualize locally. \n- Caching built-in to avoid duplicate executions. \n- Planned support for T1 input and T4 output files.\n- Notebook / interactive window mode; if enabled, plots are shown in the notebook / window instead of written to a folder. \n\n<img width=\"674\" alt=\"example run in interactive window\" src=\"https://user-images.githubusercontent.com/6725103/232880006-70a05b0e-a4e4-4cc7-bea9-473959c474c2.png\">\n<img width=\"483\" alt=\"example run in interactive window 2\" src=\"https://user-images.githubusercontent.com/6725103/232881244-d432ea8e-801a-44b1-9acb-b98cc1b740ac.png\">\n\n## Usage\n\n### Entry points\nThere are two entry points defined: `autotuning_experiment` and `autotuning_visualize`. Both take one argument: the path to an experiment file (see below). \n\n### Input files\nTo get started, all you need is an experiments file. This is a `json` file that describes the details of your comparison: which algorithms to use, which programs to tune on which devices, the graphs to output and so on. \nYou can find the API and an example `experiments.json` in the [documentation](https://autotuningassociation.github.io/autotuning_methodology/modules.html). \n\n### File references\nAs we are dealing with input and output files, file references matter. \nWhen calling the entrypoints, we are already providing the path to an experiments file. \nFile references in experiments files are relative to the location of the experiment file itself. \nFile references in tuning scripts are relative to the location of the tuning script itself. Tuning scripts need to have the global literals `file_path_results` and `file_path_metadata` for this package to know where to get the results. \nPlots outputted by this package are placed in a folder called `generated_plots` relative to the current working directory. \n\n\n## Pipeline\nThe below schematics show the pipeline implemented by this tool as described in the paper. \n\n<!-- <img width=\"100%\" alt=\"flowchart performance curve generation\" src=\"https://autotuningassociation.github.io/autotuning_methodology/_static/flowchart_performance_curve_generation.svg\"> -->\n\nThe first flowchart shows the tranformation of raw, stochastic optimization algorithm data to a performance curve. \n\n<!-- <img width=\"100%\" alt=\"flowchart output generation\" src=\"docs/source/flowchart_output_generation.svg\"> -->\n\nThe second flowchart shows the adaption of performance curves of various optimization algorithms and search spaces to the desired output.\n\n\n## Contributing\n\n### Setup\nIf you're looking to contribute to this package: welcome!\nStart out by installing with `pip install -e .[dev]` (this installs the package in editable mode alongside the development dependencies). \nDuring development, unit and integration tests can be ran with `pytest`. \n[Black](https://pypi.org/project/black/) is used as a formatter, and [Ruff](https://pypi.org/project/ruff/) is used as a linter to check the formatting, import sorting et cetera. \nWhen using Visual Studio Code, use the `settings.json` found in `.vscode` to automatically have the correct linting, formatting and sorting during development. \nIn addition, install the extensions recommended by us by searching for `@recommended:workspace` in the extensions tab for a better development experience. \n\n### Documentation\nThe documentation can be found [here](https://autotuningassociation.github.io/autotuning_methodology/). \nLocally, the documentation can be build with `make clean html` from the `docs` folder, but the package must have been installed in editable mode with `pip install -e .`. \nUpon pushing to main or publishing a version, this documentation will be built and published to the GitHub Pages. \nThe Docstring format used is Google. Type hints are to be included in the function signature and therefor omitted from the docstring. In Visual Studio Code, the `autoDocstring` extension can be used to automatically infer docstrings. When referrring to functions and parameters in the docstring outside of their definition, use double backquotes to be compatible with both MarkDown and ReStructuredText, e.g.: *\"skip_draws_check: skips checking that each value in ``draws`` is in the ``dist``.\"*.\n\n### Tests\nBefore contributing a pull request, please run `nox` and ensure it has no errors. This will test against all Python versions explicitely supported by this package, and will check whether the correct formatting has been applied.\nUpon submitting a pull request or pushing to main, these same checks will be ran remotely via GitHub Actions. \n\n### Publishing\nFor publising the package to PyPI (the Python Package Index), we use [Flit](https://flit.pypa.io) and the [to-pypi-using-flit](https://github.com/AsifArmanRahman/to-pypi-using-flit/tree/v1/) GitHub Action to automate this. \n\n[Semantic version numbering](https://semver.org) is used as follows: `MAJOR.Minor.patch`. \n`MAJOR` version for incompatible API changes.\n`Minor` version for functionality in a backward compatible manner.\n`patch` version for backward compatible bug fixes. \nIn addition, [PEP 440](https://peps.python.org/pep-0440/) is adhered to, specifically for [pre-release versioning](https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/#id62). \n",
"bugtrack_url": null,
"license": null,
"summary": "Software package easing implementation of the guidelines of the 2024 paper 'A Methodology for Comparing Auto-Tuning Optimization Algorithms' (https://doi.org/10.1016/j.future.2024.05.021). The DOI of this software is https://doi.org/10.5281/zenodo.11243974.",
"version": "1.0.0",
"project_urls": {
"Bug Tracker": "https://github.com/fjwillemsen/autotuning_methodology/issues",
"Documentation": "https://fjwillemsen.github.io/autotuning_methodology/",
"Repository": "https://github.com/fjwillemsen/autotuning_methodology"
},
"split_keywords": [
"autotuning",
" auto-tuning",
" methodology",
" scientific"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "fbec0a595e4ce5844cc64d0d0b8fbdacb1f776b63e212c3f9f070278275ad874",
"md5": "90ff4a9a5355c6acc27bdb87d59687a0",
"sha256": "f242c782dfd03757e6aba9261733565140b771f2a1305132df615a1094e0858f"
},
"downloads": -1,
"filename": "autotuning_methodology-1.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "90ff4a9a5355c6acc27bdb87d59687a0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 51162,
"upload_time": "2024-05-31T10:32:01",
"upload_time_iso_8601": "2024-05-31T10:32:01.801074Z",
"url": "https://files.pythonhosted.org/packages/fb/ec/0a595e4ce5844cc64d0d0b8fbdacb1f776b63e212c3f9f070278275ad874/autotuning_methodology-1.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "7000c07c6a010f703bffcd0771e640e543adae00879f50433770891322e0da51",
"md5": "45284075e050a70a90d03dada74e1c10",
"sha256": "0bb6e3b3f2489cbb6b8adc9ddacef7eaae9902849295652fdbf30556ebeab8f2"
},
"downloads": -1,
"filename": "autotuning_methodology-1.0.0.tar.gz",
"has_sig": false,
"md5_digest": "45284075e050a70a90d03dada74e1c10",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 3670591,
"upload_time": "2024-05-31T10:32:03",
"upload_time_iso_8601": "2024-05-31T10:32:03.266999Z",
"url": "https://files.pythonhosted.org/packages/70/00/c07c6a010f703bffcd0771e640e543adae00879f50433770891322e0da51/autotuning_methodology-1.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-31 10:32:03",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "fjwillemsen",
"github_project": "autotuning_methodology",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"lcname": "autotuning_methodology"
}
