# museval
[](https://github.com/sigsep/sigsep-mus-eval/actions?query=workflow%3ACI+branch%3Amaster+event%3Apush)
[](https://pypi.python.org/pypi/museval)
[](https://pypi.python.org/pypi/museval)
A python package to evaluate source separation results using the [MUSDB18](https://sigsep.github.io/musdb) dataset. This package was part of the [MUS task](https://sisec.inria.fr/home/2018-professionally-produced-music-recordings/) of the [Signal Separation Evaluation Campaign (SISEC)](https://sisec.inria.fr/).
### BSSEval v4
The BSSEval metrics, as implemented in the [MATLAB toolboxes](http://bass-db.gforge.inria.fr/bss_eval/) and their re-implementation in [mir_eval](http://craffel.github.io/mir_eval/#module-mir_eval.separation) are widely used in the audio separation literature. One particularity of BSSEval is to compute the metrics after optimally matching the estimates to the true sources through linear distortion filters. This allows the criteria to be robust to some linear mismatches. Apart from the optional evaluation for all possible permutations of the sources, this matching is the reason for most of the computation cost of BSSEval, especially considering it is done for each evaluation window when the metrics are computed on a framewise basis.
For this package, we enabled the option of having _time invariant_ distortion filters, instead of necessarily taking them as varying over time as done in the previous versions of BSS eval. First, enabling this option _significantly reduces_ the computational cost for evaluation because matching needs to be done only once for the whole signal. Second, it introduces much more dynamics in the evaluation, because time-varying matching filters turn out to over-estimate performance. Third, this makes matching more robust, because true sources are not silent throughout the whole recording, while they often were for short windows.
## Installation
### Package installation
You can install the `museval` parsing package using pip:
```bash
pip install museval
```
## Usage
The purpose of this package is to evaluate source separation results and write out validated `json` files. We want to encourage users to use this evaluation output format as the standardized way to share source separation results. `museval` is designed to work in conjuction with the [musdb](https://github.com/sigsep/sigsep-mus-db) tools and the MUSDB18 dataset (however, `museval` can also be used without `musdb`).
### Separate MUSDB18 tracks and Evaluate on-the-fly
- If you want to perform evaluation while processing your source separation results, you can make use `musdb` track objects.
Here is an example for such a function separating the mixture into a __vocals__ and __accompaniment__ track:
```python
import musdb
import museval
def estimate_and_evaluate(track):
# assume mix as estimates
estimates = {
'vocals': track.audio,
'accompaniment': track.audio
}
# Evaluate using museval
scores = museval.eval_mus_track(
track, estimates, output_dir="path/to/json"
)
# print nicely formatted and aggregated scores
print(scores)
mus = musdb.DB()
for track in mus:
estimate_and_evaluate(track)
```
Make sure `output_dir` is set. `museval` will recreate the `musdb` file structure in that folder and write the evaluation results to this folder.
### Evaluate MUSDB18 tracks later
If you have already computed your estimates, we provide you with an easy-to-use function to process evaluation results afterwards.
Simply use the `museval.eval_mus_dir` to evaluate your `estimates_dir` and write the results into the `output_dir`. For convenience, the `eval_mus_dir` function accepts all parameters of the `musdb.run()`.
```python
import musdb
import museval
# initiate musdb
mus = musdb.DB()
# evaluate an existing estimate folder with wav files
museval.eval_mus_dir(
dataset=mus, # instance of musdb
estimates_dir=..., # path to estimate folder
output_dir=..., # set a folder to write eval json files
ext='wav
)
```
### Aggregate and Analyze Scores
Scores for each track can also be aggregated in a pandas DataFrame for easier analysis or the creation of boxplots.
To aggregate multiple tracks in a DataFrame, create `museval.EvalStore()` object and add the track scores successively.
```python
results = museval.EvalStore(frames_agg='median', tracks_agg='median')
for track in tracks:
# ...
results.add_track(museval.eval_mus_track(track, estimates))
```
You may also add scores that have been computed beforehand through `museval.eval_mus_dir`:
```python
results = museval.EvalStore(frames_agg='median', tracks_agg='median')
results.add_eval_dir(
path=...# path to the output_dir for eval_mus_dir
)
```
When all tracks have been added, the aggregated scores can be shown using `print(results)` and results may be saved as a pandas DataFrame `results.save('my_method.pandas')`.
To compare multiple methods, create a `museval.MethodStore()` object add the results
```python
methods = museval.MethodStore()
methods.add_evalstore(results, name="XZY")
```
To compare against participants from [SiSEC MUS 2018](https://github.com/sigsep/sigsep-mus-2018), we provide a convenient method to load the existing scores on demand using `methods.add_sisec18()`. For the creation of plots and statistical significance tests we refer to our [list of examples](/examples).
#### Commandline tool
We provide a command line wrapper of `eval_mus_dir` by calling the `museval` command line tool. The following example is equivalent to the code example above:
```
museval --musdb path/to/musdb -o path/to/output_dir path/to/estimate_dir
```
:bulb: you use the `--is-wav` flag to use the decoded wav _musdb_ dataset.
### Using Docker for Evaluation
If you don't want to set up a Python environment to run the evaluation, we would recommend to use [Docker](http://docker.com). Assuming you have already computed your estimates and installed docker in your machine, you just need to run the following two lines in your terminal:
#### 1. Pull Docker Container
Pull our precompiled `sigsep-mus-eval` image from [dockerhub](https://hub.docker.com/r/faroit/sigsep-mus-eval/):
```
docker pull faroit/sigsep-mus-eval
```
#### 2. Run evaluation
To run the evaluation inside of the docker, three absolute paths are required:
* `estimatesdir` will stand here for the absolute path to the estimates directory. (For instance `/home/faroit/dev/mymethod/musdboutput`)
* `musdbdir` will stand here for the absolute path to the root folder of musdb. (For instance `/home/faroit/dev/data/musdb18`)
* `outputdir` will stand here for the absolute path to the output directory. (For instance `/home/faroit/dev/mymethod/scores`)
We just mount these directories into the docker container using the `-v` flags and start the docker instance:
```
docker run --rm -v estimatesdir:/est -v musdbdir:/mus -v outputdir:/out faroit/sigsep-mus-eval --musdb /mus -o /out /est
```
In the line above, replace `estimatesdir`, `musdbdir` and `outputdir` by the absolute paths for your setting. Please note that docker requires absolute paths so you have to rely on your command line environment to convert relative paths to absolute paths (e.g. by using `$HOME/` on Unix).
:warning: `museval` requires a significant amount of memory for the evaluation. Evaluating all five targets for _MUSDB18_ may require more than 4GB of RAM. It is recommended to adjust your Docker preferences, because the docker container might just quit if its out of memory.
## How to contribute
_museval_ is a community focused project, we therefore encourage the community to submit bug-fixes and requests for technical support through [github issues](https://github.com/sigsep/sigsep-mus-eval/issues/new). For more details of how to contribute, please follow our [`CONTRIBUTING.md`](CONTRIBUTING.md).
## References
A. If you use the `museval` in the context of source separation evaluation comparing a method it to other methods of [SiSEC 2018](http://sisec18.unmix.app/), please cite
```
@InProceedings{SiSEC18,
author="St{\"o}ter, Fabian-Robert and Liutkus, Antoine and Ito, Nobutaka",
title="The 2018 Signal Separation Evaluation Campaign",
booktitle="Latent Variable Analysis and Signal Separation:
14th International Conference, LVA/ICA 2018, Surrey, UK",
year="2018",
pages="293--305"
}
```
B. if you use the software for any other purpose, you can cite the software release
[](https://doi.org/10.5281/zenodo.3376621)
Raw data
{
"_id": null,
"home_page": "https://github.com/sigsep/sigsep-mus-eval",
"name": "museval",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "",
"author": "Fabian-Robert Stoeter",
"author_email": "mail@faroit.com",
"download_url": "https://files.pythonhosted.org/packages/10/ff/30917f3fb1ae02371183a105120c96056ac5c6b0bfabdfc4ce5a0dfc3e4b/museval-0.4.1.tar.gz",
"platform": null,
"description": "# museval\n\n[](https://github.com/sigsep/sigsep-mus-eval/actions?query=workflow%3ACI+branch%3Amaster+event%3Apush)\n[](https://pypi.python.org/pypi/museval)\n[](https://pypi.python.org/pypi/museval)\n\nA python package to evaluate source separation results using the [MUSDB18](https://sigsep.github.io/musdb) dataset. This package was part of the [MUS task](https://sisec.inria.fr/home/2018-professionally-produced-music-recordings/) of the [Signal Separation Evaluation Campaign (SISEC)](https://sisec.inria.fr/).\n\n### BSSEval v4\n\nThe BSSEval metrics, as implemented in the [MATLAB toolboxes](http://bass-db.gforge.inria.fr/bss_eval/) and their re-implementation in [mir_eval](http://craffel.github.io/mir_eval/#module-mir_eval.separation) are widely used in the audio separation literature. One particularity of BSSEval is to compute the metrics after optimally matching the estimates to the true sources through linear distortion filters. This allows the criteria to be robust to some linear mismatches. Apart from the optional evaluation for all possible permutations of the sources, this matching is the reason for most of the computation cost of BSSEval, especially considering it is done for each evaluation window when the metrics are computed on a framewise basis.\n\nFor this package, we enabled the option of having _time invariant_ distortion filters, instead of necessarily taking them as varying over time as done in the previous versions of BSS eval. First, enabling this option _significantly reduces_ the computational cost for evaluation because matching needs to be done only once for the whole signal. Second, it introduces much more dynamics in the evaluation, because time-varying matching filters turn out to over-estimate performance. Third, this makes matching more robust, because true sources are not silent throughout the whole recording, while they often were for short windows.\n\n## Installation\n\n### Package installation\n\nYou can install the `museval` parsing package using pip:\n\n```bash\npip install museval\n```\n\n## Usage\n\nThe purpose of this package is to evaluate source separation results and write out validated `json` files. We want to encourage users to use this evaluation output format as the standardized way to share source separation results. `museval` is designed to work in conjuction with the [musdb](https://github.com/sigsep/sigsep-mus-db) tools and the MUSDB18 dataset (however, `museval` can also be used without `musdb`).\n\n### Separate MUSDB18 tracks and Evaluate on-the-fly\n\n- If you want to perform evaluation while processing your source separation results, you can make use `musdb` track objects.\nHere is an example for such a function separating the mixture into a __vocals__ and __accompaniment__ track:\n\n```python\nimport musdb\nimport museval\n\ndef estimate_and_evaluate(track):\n # assume mix as estimates\n estimates = {\n 'vocals': track.audio,\n 'accompaniment': track.audio\n }\n\n # Evaluate using museval\n scores = museval.eval_mus_track(\n track, estimates, output_dir=\"path/to/json\"\n )\n\n # print nicely formatted and aggregated scores\n print(scores)\n\nmus = musdb.DB()\nfor track in mus:\n estimate_and_evaluate(track)\n\n```\n\nMake sure `output_dir` is set. `museval` will recreate the `musdb` file structure in that folder and write the evaluation results to this folder.\n\n### Evaluate MUSDB18 tracks later\n\nIf you have already computed your estimates, we provide you with an easy-to-use function to process evaluation results afterwards.\n\nSimply use the `museval.eval_mus_dir` to evaluate your `estimates_dir` and write the results into the `output_dir`. For convenience, the `eval_mus_dir` function accepts all parameters of the `musdb.run()`.\n\n```python\nimport musdb\nimport museval\n\n# initiate musdb\nmus = musdb.DB()\n\n# evaluate an existing estimate folder with wav files\nmuseval.eval_mus_dir(\n dataset=mus, # instance of musdb\n estimates_dir=..., # path to estimate folder\n output_dir=..., # set a folder to write eval json files\n ext='wav\n)\n```\n\n### Aggregate and Analyze Scores\n\nScores for each track can also be aggregated in a pandas DataFrame for easier analysis or the creation of boxplots.\nTo aggregate multiple tracks in a DataFrame, create `museval.EvalStore()` object and add the track scores successively.\n\n```python\nresults = museval.EvalStore(frames_agg='median', tracks_agg='median')\nfor track in tracks:\n # ...\n results.add_track(museval.eval_mus_track(track, estimates))\n```\n\nYou may also add scores that have been computed beforehand through `museval.eval_mus_dir`:\n```python\nresults = museval.EvalStore(frames_agg='median', tracks_agg='median')\nresults.add_eval_dir(\n path=...# path to the output_dir for eval_mus_dir\n)\n```\n\nWhen all tracks have been added, the aggregated scores can be shown using `print(results)` and results may be saved as a pandas DataFrame `results.save('my_method.pandas')`.\n\nTo compare multiple methods, create a `museval.MethodStore()` object add the results\n\n```python\nmethods = museval.MethodStore()\nmethods.add_evalstore(results, name=\"XZY\")\n```\n\nTo compare against participants from [SiSEC MUS 2018](https://github.com/sigsep/sigsep-mus-2018), we provide a convenient method to load the existing scores on demand using `methods.add_sisec18()`. For the creation of plots and statistical significance tests we refer to our [list of examples](/examples).\n\n#### Commandline tool\n\nWe provide a command line wrapper of `eval_mus_dir` by calling the `museval` command line tool. The following example is equivalent to the code example above:\n\n```\nmuseval --musdb path/to/musdb -o path/to/output_dir path/to/estimate_dir\n```\n\n:bulb: you use the `--is-wav` flag to use the decoded wav _musdb_ dataset.\n\n### Using Docker for Evaluation\n\nIf you don't want to set up a Python environment to run the evaluation, we would recommend to use [Docker](http://docker.com). Assuming you have already computed your estimates and installed docker in your machine, you just need to run the following two lines in your terminal:\n\n#### 1. Pull Docker Container\n\nPull our precompiled `sigsep-mus-eval` image from [dockerhub](https://hub.docker.com/r/faroit/sigsep-mus-eval/):\n\n```\ndocker pull faroit/sigsep-mus-eval\n```\n\n#### 2. Run evaluation\n\nTo run the evaluation inside of the docker, three absolute paths are required:\n\n* `estimatesdir` will stand here for the absolute path to the estimates directory. (For instance `/home/faroit/dev/mymethod/musdboutput`)\n* `musdbdir` will stand here for the absolute path to the root folder of musdb. (For instance `/home/faroit/dev/data/musdb18`)\n* `outputdir` will stand here for the absolute path to the output directory. (For instance `/home/faroit/dev/mymethod/scores`)\n\nWe just mount these directories into the docker container using the `-v` flags and start the docker instance:\n\n```\ndocker run --rm -v estimatesdir:/est -v musdbdir:/mus -v outputdir:/out faroit/sigsep-mus-eval --musdb /mus -o /out /est\n```\n\nIn the line above, replace `estimatesdir`, `musdbdir` and `outputdir` by the absolute paths for your setting. Please note that docker requires absolute paths so you have to rely on your command line environment to convert relative paths to absolute paths (e.g. by using `$HOME/` on Unix).\n\n:warning: `museval` requires a significant amount of memory for the evaluation. Evaluating all five targets for _MUSDB18_ may require more than 4GB of RAM. It is recommended to adjust your Docker preferences, because the docker container might just quit if its out of memory.\n\n## How to contribute\n\n_museval_ is a community focused project, we therefore encourage the community to submit bug-fixes and requests for technical support through [github issues](https://github.com/sigsep/sigsep-mus-eval/issues/new). For more details of how to contribute, please follow our [`CONTRIBUTING.md`](CONTRIBUTING.md). \n\n## References\n\nA. If you use the `museval` in the context of source separation evaluation comparing a method it to other methods of [SiSEC 2018](http://sisec18.unmix.app/), please cite\n\n```\n@InProceedings{SiSEC18,\n author=\"St{\\\"o}ter, Fabian-Robert and Liutkus, Antoine and Ito, Nobutaka\",\n title=\"The 2018 Signal Separation Evaluation Campaign\",\n booktitle=\"Latent Variable Analysis and Signal Separation:\n 14th International Conference, LVA/ICA 2018, Surrey, UK\",\n year=\"2018\",\n pages=\"293--305\"\n}\n```\n\nB. if you use the software for any other purpose, you can cite the software release\n\n[](https://doi.org/10.5281/zenodo.3376621)\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Evaluation tools for the SIGSEP MUS database",
"version": "0.4.1",
"project_urls": {
"Homepage": "https://github.com/sigsep/sigsep-mus-eval"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "3f232a4fc9f10f4f889da61c082e92092ff86b3c00f2eda72953293d0d708794",
"md5": "2942984124bc412ba5d7529e05fa04e4",
"sha256": "4b5320bc8aff68b218ea0571959da0c1e2f11aaf78a0264b659e8ac55d98d0f8"
},
"downloads": -1,
"filename": "museval-0.4.1-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "2942984124bc412ba5d7529e05fa04e4",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": null,
"size": 20333,
"upload_time": "2023-05-24T11:56:28",
"upload_time_iso_8601": "2023-05-24T11:56:28.665216Z",
"url": "https://files.pythonhosted.org/packages/3f/23/2a4fc9f10f4f889da61c082e92092ff86b3c00f2eda72953293d0d708794/museval-0.4.1-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "10ff30917f3fb1ae02371183a105120c96056ac5c6b0bfabdfc4ce5a0dfc3e4b",
"md5": "a542f01e46c3e347e37a6f2057d25b91",
"sha256": "24d2140c8595fd171674a5aed40f837c9880a0443d82e1a6dbaa99f26bf6086e"
},
"downloads": -1,
"filename": "museval-0.4.1.tar.gz",
"has_sig": false,
"md5_digest": "a542f01e46c3e347e37a6f2057d25b91",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 24391,
"upload_time": "2023-05-24T11:56:31",
"upload_time_iso_8601": "2023-05-24T11:56:31.124206Z",
"url": "https://files.pythonhosted.org/packages/10/ff/30917f3fb1ae02371183a105120c96056ac5c6b0bfabdfc4ce5a0dfc3e4b/museval-0.4.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-05-24 11:56:31",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "sigsep",
"github_project": "sigsep-mus-eval",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "museval"
}
JSON
