<h1 align="center">Reproducible Subjective Evaluation (ReSEval)</h1>
<div align="center">
[](https://pypi.python.org/pypi/reseval)
[](https://opensource.org/licenses/MIT)
[](https://www.python.org/downloads/)
[](https://pepy.tech/project/reseval)
</div>
ReSEval is a framework for quickly building subjective evaluation and
annotation tasks that are deployed on crowdworker platforms like
[Amazon Mechanical Turk](https://www.mturk.com/). ReSEval currently supports
A/B, ABX, MOS, MUSHRA, and Word Selection tests on audio, image, text, and video data.
<h3 align="center">
While our code is free to use, performing crowdsourced subjective
evaluation is not free.<br/>We are not responsible for costs incurred
while using our code.
</h3>
### Citation
If you use ReSEval in an academic publication, please cite
[our paper](https://www.maxrmorrison.com/pdfs/morrison2022reproducible.pdf).
### IEEE
M. Morrison, B. Tang, G. Tan, and B. Pardo, "Reproducible Subjective Evaluation," ICLR Workshop on ML Evaluation Standards, April 2022.
### BibTex
```
@inproceedings{morrison2022reproducible,
title={Reproducible Subjective Evaluation},
author={Morrison, Max and Tang, Brian and Tan, Gefei and Pardo, Bryan},
booktitle={ICLR Workshop on ML Evaluation Standards},
month={April},
year={2022}
}
```
## Table of contents
- [Installation](#installation)
* [Deploying locally](#deploying-locally)
- [Configuration](#configuration)
- [Adding files](#adding-files)
* [AB](#ab)
* [ABX](#abx)
* [MOS](#mos)
* [MUSHRA](#mushra)
* [WordSelect](#wordselect)
- [Credentials](#credentials)
* [Heroku](#heroku)
* [Amazon Web Services](#amazon-web-services)
* [Amazon Mechanical Turk](#amazon-mechanical-turk)
- [Usage](#usage)
* [Command-line interface](#command-line-interface)
* [Create](#create)
* [Monitor](#monitor)
* [Results](#results)
* [Pay](#pay)
* [Destroy](#destroy)
* [Extend](#extend)
* [Application programming interface](#application-programming-interface)
- [Advanced usage](#advanced-usage)
* [CLI](#cli)
* [API](#api)
- [Additional monitoring](#additional-monitoring)
## Installation
First, install the Python module. ReSEval requires Python 3.9 or higher.
`pip install reseval`
Next, [download Node.js](https://nodejs.org/en/). You can check that your installation is correct by running `node --version`. ReSEval uses Node.js version 18.16.1 and is not guaranteed to work on all versions. If needed, Linux and OS X users can use `n` to change their version of Node.js, and Windows users can use [NVM for Windows](https://github.com/coreybutler/nvm-windows).
```
# Linux or OS X
sudo npm install -g n
sudo n 18.16.1
# Windows
# Must be run with administrator privileges
nvm install 18.16.1
nvm use 18.16.1
```
**Note** - You must restart your terminal after changing versions of node for the change to take effect
### Deploying locally
To be able to preview your subjective evaluation locally, you must
[setup a local MySQL database server](https://dev.mysql.com/doc/mysql-getting-started/en/)
and create a user with database creation privileges.
```bash
# Linux installation
sudo apt install mysql-server
# MySQL setup
sudo mysql_secure_installation
# Login to MySQL as root
sudo mysql -u root
```
```sql
-- Create a user (change new_user and new_password)
mysql> CREATE USER 'new_user'@'localhost' IDENTIFIED BY 'new_password';
-- Give user database creation privileges
mysql> GRANT ALL PRIVILEGES ON * . * TO 'new_user'@'localhost';
```
Run the following to store the username and password in
`reseval.CACHE / '.env'`.
```bash
python -m reseval.credentials \
--mysql_local_user <mysql_user> \
--mysql_local_password <mysql_local_password>
```
The `.env` file is used to set local environment variables and is not pushed to
GitHub or uploaded to any remote storage.
## Configuration
All configuration is performed in a YAML configuration file. See `examples/*.yaml` for examples and documentation of parameters.
## Adding files
The files to be evaluated must be organized in a directory structure according
to the type of test being run. The directory structures for each test are as
follows. Examples of valid directories of evaluation files can be found in
`examples/`.
### AB
```
ab
├── <condition-1>
│ ├── <file-0>
│ ├── <file-1>
│ ├── <file-2>
│ ├── ...
├── <condition-2>
│ ├── <file-0>
│ ├── <file-1>
│ ├── <file-2>
│ ├── ...
```
### ABX
```
abx
└── reference
│ ├── <file-0>
│ ├── <file-1>
│ ├── <file-2>
│ ├── ...
├── <condition-1>
│ ├── <file-0>
│ ├── <file-1>
│ ├── <file-2>
│ ├── ...
├── <condition-2>
│ ├── <file-0>
│ ├── <file-1>
│ ├── <file-2>
│ ├── ...
```
### MOS
```
mos
├── <condition-0>
│ ├── <file-0>
│ ├── <file-1>
│ ├── <file-2>
│ ├── ...
├── <condition-1>
│ ├── <file-0>
│ ├── <file-1>
│ ├── <file-2>
│ ├── ...
├── <condition-2>
│ ├── <file-0>
│ ├── <file-1>
│ ├── <file-2>
│ ├── ...
├── <condition-3>
│ ├── <file-0>
│ ├── <file-1>
│ ├── <file-2>
│ ├── ...
└── ...
```
### MUSHRA
```
mushra
├── <condition-0>
│ ├── <file-0>
│ ├── <file-1>
│ ├── <file-2>
│ ├── ...
├── <condition-1>
│ ├── <file-0>
│ ├── <file-1>
│ ├── <file-2>
│ ├── ...
├── <condition-2>
│ ├── <file-0>
│ ├── <file-1>
│ ├── <file-2>
│ ├── ...
├── <condition-3>
│ ├── <file-0>
│ ├── <file-1>
│ ├── <file-2>
│ ├── ...
└── ...
```
### WordSelect
```
wordselect
├── <file-0>
├── <words-0>
├── <file-1>
├── <words-1>
├── <file-2>
├── <words-2>
├── ...
```
`<words-x>` is `<file-x>` with `-words.txt` extension.
## Credentials
API keys are required to use the third-party services that ReSEval depends on.
These are not required for local development. Do not share these API keys.
### Amazon Web Services
Sign up for an AWS account. Go to `Security Credentials`. Under `Access keys`, click `Create New Access Key`.
<p align="center">
<img src="docs/images/aws-00.png" width="400" alt="AWS API key instructions">
<img src="docs/images/aws-01.png" width="400" alt="AWS API key instructions">
</p>
Note that this gives you a root access key. You can alternatively use the Identity & Access Management (IAM) system to setup more restrictive permissions for a user.
If you have never used AWS Elastic Beanstalk, one more step is required. Elastic Beanstalk instances created from Python (as opposed to the AWS console) do not instantiate the default instance profile IAM role. The solution is to either create (and then delete) an Elastic Beanstalk instance from the console, or to create the required IAM role manually. The IAM role is called `aws-elasticbeanstalk-ec2-role` and contains three permissions: (1) `AWSElasticBeanstalkWebTier`, (2) `AWSElasticBeanstalkMulticontainerDocker`, and (3) `AWSElasticBeanstalkWorkerTier`.
### Amazon Mechanical Turk
Follow the instructions [here](https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMechanicalTurkGettingStartedGuide/SetUp.html) for setting up MTurk and connecting it to your AWS account.
### Heroku (Optional if you don't want to use AWS)
Sign up for a Heroku account. Go to `Account Settings`. At the bottom of the page in the `API Key` section is a `Reveal` button.
<p align="center">
<img src="docs/images/heroku-00.png" width="400" alt="Heroku API key instructions">
<img src="docs/images/heroku-01.png" width="400" alt="Heroku API key instructions">
</p>
You will also need to enable billing. You can do so [here](https://heroku.com/verify).
## Usage
Once you have your configuration file and a properly formatted directory of evaluation files, you are ready to deploy a subjective evaluation. Example configuration files and corresponding evaluation files can be found in `examples/`.
If you are not deploying locally, add your API keys.
```bash
# AWS credentials
python -m reseval.credentials \
--aws_api_key <aws_api_key> \
--aws_api_secret_key <aws_api_secret_key>
# (Optional) Heroku credentials
python -m reseval.credentials \
--heroku_api_key <heroku_api_key>
```
API keys are saved in `reseval.CACHE / '.keys'`. The `.keys` file is used to set local environment variables and is not pushed to GitHub or uploaded to any remote storage.
### Command-line interface
Arguments for the following command-line interfaces are as follows, unless otherwise specified.
- `<config>` - The configuration file
- `<directory>` - The directory of evaluation files
- `<name>` - The name of the evaluation given in the configuration file
#### Create
Create a subjective evaluation either locally, in remote development mode (e.g., MTurk Sandbox), or in production mode.
**Note** - `reseval.create` is not currently thread-safe. Wait until the first call has finished before calling it again. See [this GitHub issue](https://github.com/reseval/reseval/issues/5).
```bash
# Local development
python -m reseval.create <config> <directory> --local
# Remote development
python -m reseval.create <config> <directory>
# Production
python -m reseval.create <config> <directory> --production
```
#### Monitor
```bash
# Monitor all subjective evaluations
python -m reseval.monitor
# Monitor one subjective evaluation
# The name of the evaluation can be found in its configuration file
python -m reseval.monitor --name <name>
```
**Note** - By default, the monitor updates once every minute. You can update the monitor more or less often by providing an update interval in seconds.
```bash
# Update the monitor once every ten seconds
python -m reseval.monitor --interval 10
```
#### Results
```bash
# Get the results of a subjective evaluation.
# Results are stored in <directory>/<name>.
# <directory> defaults to the current directory.
python -m reseval.results <name> --directory <directory>
```
#### Pay
```bash
# Pay participants
python -m reseval.pay <name>
```
#### Destroy
```bash
# Destroy the compute resources of a subjective evaluation (e.g., any cloud
# storage, databases, or servers)
python -m reseval.destroy <name>
# Destroy a subjective evaluation even if it is still active.
# Participants who have taken the test so far will be paid.
python -m reseval.destroy <name> --force
```
#### Extend
```bash
# Add <participants> additional participants to a finished evaluation
python -m reseval.extend <name> <participants>
```
### Application programming interface
Documentation for our API can be found [here](https://reseval.github.io/reseval/html/index.html).
## Advanced usage
Once you feel comfortable with using ReSEval step-by-step from the
command-line and after you have added your credentials with
`reseval.credentials`, you can use the CLI or API to run your evaluation with
only a single command.
### CLI
```bash
# Local development
python -m reseval <config> <directory> --local
# Remote development
python -m reseval <config> <directory>
# Production
python -m reseval <config> <directory> --production
```
### API
```python
import reseval
# Local development
reseval.run(config, directory, local=True)
# Remote development
reseval.run(config, directory)
# Production
reseval.run(config, directory, production=True)
```
## Additional monitoring
### AWS S3
To monitor, edit, or delete AWS S3 storage buckets, or see storage costs, use the
[AWS S3 console](https://s3.console.aws.amazon.com/s3).
### AWS Elastic Beanstalk
To monitor, edit, or delete the server compute, use the
[AWS Elastic Beanstalk console](https://console.aws.amazon.com/elasticbeanstalk).
### AWS Relational Database Service
To monitor, edit, or delete the database, use the
[AWS RDS console](https://console.aws.amazon.com/rds).
### MTurk
HITs not created on the MTurk dashboard are not visible on the MTurk dashboard.
You can use the MTurk CLI to monitor, edit, or delete HITs. MTurk costs appear
on the [AWS billing dashboard](https://console.aws.amazon.com/billing/home)
at the end of the billing period.
### Heroku
To monitor, edit, or delete Heroku databases and servers, use the
[Heroku application dashboard](https://dashboard.heroku.com/apps). You can see
any costs on the
[billing dashboard](https://dashboard.heroku.com/account/billing).
Raw data
{
"_id": null,
"home_page": "https://github.com/reseval/reseval",
"name": "reseval",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "annotation,audio,ab,abx,crowdsourcing,evaluation,image,mos,mushra,speech,subjective",
"author": "Max Morrison, Brian Tang, Gefei Tan, Bryan Pardo",
"author_email": "maxrmorrison@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/a5/16/4bfab6771f6c6fce6f710cc881a98e995efc6c6bb1a688bc369db31dcf17/reseval-0.1.6.tar.gz",
"platform": null,
"description": "<h1 align=\"center\">Reproducible Subjective Evaluation (ReSEval)</h1>\n<div align=\"center\">\n\n[](https://pypi.python.org/pypi/reseval)\n[](https://opensource.org/licenses/MIT)\n[](https://www.python.org/downloads/)\n[](https://pepy.tech/project/reseval)\n\n</div>\n\nReSEval is a framework for quickly building subjective evaluation and\nannotation tasks that are deployed on crowdworker platforms like\n[Amazon Mechanical Turk](https://www.mturk.com/). ReSEval currently supports\nA/B, ABX, MOS, MUSHRA, and Word Selection tests on audio, image, text, and video data.\n\n<h3 align=\"center\">\n While our code is free to use, performing crowdsourced subjective\n evaluation is not free.<br/>We are not responsible for costs incurred\n while using our code.\n</h3>\n\n\n### Citation\n\nIf you use ReSEval in an academic publication, please cite\n[our paper](https://www.maxrmorrison.com/pdfs/morrison2022reproducible.pdf).\n\n\n### IEEE\n\nM. Morrison, B. Tang, G. Tan, and B. Pardo, \"Reproducible Subjective Evaluation,\" ICLR Workshop on ML Evaluation Standards, April 2022.\n\n\n### BibTex\n\n```\n@inproceedings{morrison2022reproducible,\n title={Reproducible Subjective Evaluation},\n author={Morrison, Max and Tang, Brian and Tan, Gefei and Pardo, Bryan},\n booktitle={ICLR Workshop on ML Evaluation Standards},\n month={April},\n year={2022}\n}\n```\n\n\n## Table of contents\n- [Installation](#installation)\n * [Deploying locally](#deploying-locally)\n- [Configuration](#configuration)\n- [Adding files](#adding-files)\n * [AB](#ab)\n * [ABX](#abx)\n * [MOS](#mos)\n * [MUSHRA](#mushra)\n * [WordSelect](#wordselect)\n- [Credentials](#credentials)\n * [Heroku](#heroku)\n * [Amazon Web Services](#amazon-web-services)\n * [Amazon Mechanical Turk](#amazon-mechanical-turk)\n- [Usage](#usage)\n * [Command-line interface](#command-line-interface)\n * [Create](#create)\n * [Monitor](#monitor)\n * [Results](#results)\n * [Pay](#pay)\n * [Destroy](#destroy)\n * [Extend](#extend)\n * [Application programming interface](#application-programming-interface)\n- [Advanced usage](#advanced-usage)\n * [CLI](#cli)\n * [API](#api)\n- [Additional monitoring](#additional-monitoring)\n\n\n## Installation\n\nFirst, install the Python module. ReSEval requires Python 3.9 or higher.\n\n`pip install reseval`\n\nNext, [download Node.js](https://nodejs.org/en/). You can check that your installation is correct by running `node --version`. ReSEval uses Node.js version 18.16.1 and is not guaranteed to work on all versions. If needed, Linux and OS X users can use `n` to change their version of Node.js, and Windows users can use [NVM for Windows](https://github.com/coreybutler/nvm-windows).\n\n```\n# Linux or OS X\nsudo npm install -g n\nsudo n 18.16.1\n\n# Windows\n# Must be run with administrator privileges\nnvm install 18.16.1\nnvm use 18.16.1\n```\n\n**Note** - You must restart your terminal after changing versions of node for the change to take effect\n\n### Deploying locally\n\nTo be able to preview your subjective evaluation locally, you must\n[setup a local MySQL database server](https://dev.mysql.com/doc/mysql-getting-started/en/)\nand create a user with database creation privileges.\n\n```bash\n# Linux installation\nsudo apt install mysql-server\n\n# MySQL setup\nsudo mysql_secure_installation\n\n# Login to MySQL as root\nsudo mysql -u root\n```\n\n```sql\n-- Create a user (change new_user and new_password)\nmysql> CREATE USER 'new_user'@'localhost' IDENTIFIED BY 'new_password';\n\n-- Give user database creation privileges\nmysql> GRANT ALL PRIVILEGES ON * . * TO 'new_user'@'localhost';\n```\n\nRun the following to store the username and password in\n`reseval.CACHE / '.env'`.\n\n```bash\npython -m reseval.credentials \\\n --mysql_local_user <mysql_user> \\\n --mysql_local_password <mysql_local_password>\n```\n\nThe `.env` file is used to set local environment variables and is not pushed to\nGitHub or uploaded to any remote storage.\n\n\n## Configuration\n\nAll configuration is performed in a YAML configuration file. See `examples/*.yaml` for examples and documentation of parameters.\n\n\n## Adding files\n\nThe files to be evaluated must be organized in a directory structure according\nto the type of test being run. The directory structures for each test are as\nfollows. Examples of valid directories of evaluation files can be found in\n`examples/`.\n\n\n### AB\n\n```\nab\n\u251c\u2500\u2500 <condition-1>\n\u2502 \u251c\u2500\u2500 <file-0>\n\u2502 \u251c\u2500\u2500 <file-1>\n\u2502 \u251c\u2500\u2500 <file-2>\n\u2502 \u251c\u2500\u2500 ...\n\u251c\u2500\u2500 <condition-2>\n\u2502 \u251c\u2500\u2500 <file-0>\n\u2502 \u251c\u2500\u2500 <file-1>\n\u2502 \u251c\u2500\u2500 <file-2>\n\u2502 \u251c\u2500\u2500 ...\n```\n\n\n### ABX\n\n```\nabx\n\u2514\u2500\u2500 reference\n\u2502 \u251c\u2500\u2500 <file-0>\n\u2502 \u251c\u2500\u2500 <file-1>\n\u2502 \u251c\u2500\u2500 <file-2>\n\u2502 \u251c\u2500\u2500 ...\n\u251c\u2500\u2500 <condition-1>\n\u2502 \u251c\u2500\u2500 <file-0>\n\u2502 \u251c\u2500\u2500 <file-1>\n\u2502 \u251c\u2500\u2500 <file-2>\n\u2502 \u251c\u2500\u2500 ...\n\u251c\u2500\u2500 <condition-2>\n\u2502 \u251c\u2500\u2500 <file-0>\n\u2502 \u251c\u2500\u2500 <file-1>\n\u2502 \u251c\u2500\u2500 <file-2>\n\u2502 \u251c\u2500\u2500 ...\n```\n\n\n### MOS\n\n```\nmos\n\u251c\u2500\u2500 <condition-0>\n\u2502 \u251c\u2500\u2500 <file-0>\n\u2502 \u251c\u2500\u2500 <file-1>\n\u2502 \u251c\u2500\u2500 <file-2>\n\u2502 \u251c\u2500\u2500 ...\n\u251c\u2500\u2500 <condition-1>\n\u2502 \u251c\u2500\u2500 <file-0>\n\u2502 \u251c\u2500\u2500 <file-1>\n\u2502 \u251c\u2500\u2500 <file-2>\n\u2502 \u251c\u2500\u2500 ...\n\u251c\u2500\u2500 <condition-2>\n\u2502 \u251c\u2500\u2500 <file-0>\n\u2502 \u251c\u2500\u2500 <file-1>\n\u2502 \u251c\u2500\u2500 <file-2>\n\u2502 \u251c\u2500\u2500 ...\n\u251c\u2500\u2500 <condition-3>\n\u2502 \u251c\u2500\u2500 <file-0>\n\u2502 \u251c\u2500\u2500 <file-1>\n\u2502 \u251c\u2500\u2500 <file-2>\n\u2502 \u251c\u2500\u2500 ...\n\u2514\u2500\u2500 ...\n```\n\n\n### MUSHRA\n\n```\nmushra\n\u251c\u2500\u2500 <condition-0>\n\u2502 \u251c\u2500\u2500 <file-0>\n\u2502 \u251c\u2500\u2500 <file-1>\n\u2502 \u251c\u2500\u2500 <file-2>\n\u2502 \u251c\u2500\u2500 ...\n\u251c\u2500\u2500 <condition-1>\n\u2502 \u251c\u2500\u2500 <file-0>\n\u2502 \u251c\u2500\u2500 <file-1>\n\u2502 \u251c\u2500\u2500 <file-2>\n\u2502 \u251c\u2500\u2500 ...\n\u251c\u2500\u2500 <condition-2>\n\u2502 \u251c\u2500\u2500 <file-0>\n\u2502 \u251c\u2500\u2500 <file-1>\n\u2502 \u251c\u2500\u2500 <file-2>\n\u2502 \u251c\u2500\u2500 ...\n\u251c\u2500\u2500 <condition-3>\n\u2502 \u251c\u2500\u2500 <file-0>\n\u2502 \u251c\u2500\u2500 <file-1>\n\u2502 \u251c\u2500\u2500 <file-2>\n\u2502 \u251c\u2500\u2500 ...\n\u2514\u2500\u2500 ...\n```\n\n\n### WordSelect\n\n```\nwordselect\n\u251c\u2500\u2500 <file-0>\n\u251c\u2500\u2500 <words-0>\n\u251c\u2500\u2500 <file-1>\n\u251c\u2500\u2500 <words-1>\n\u251c\u2500\u2500 <file-2>\n\u251c\u2500\u2500 <words-2>\n\u251c\u2500\u2500 ...\n```\n\n`<words-x>` is `<file-x>` with `-words.txt` extension.\n\n\n## Credentials\n\nAPI keys are required to use the third-party services that ReSEval depends on.\nThese are not required for local development. Do not share these API keys.\n\n\n### Amazon Web Services\n\nSign up for an AWS account. Go to `Security Credentials`. Under `Access keys`, click `Create New Access Key`.\n\n<p align=\"center\">\n <img src=\"docs/images/aws-00.png\" width=\"400\" alt=\"AWS API key instructions\">\n <img src=\"docs/images/aws-01.png\" width=\"400\" alt=\"AWS API key instructions\">\n</p>\n\nNote that this gives you a root access key. You can alternatively use the Identity & Access Management (IAM) system to setup more restrictive permissions for a user.\n\nIf you have never used AWS Elastic Beanstalk, one more step is required. Elastic Beanstalk instances created from Python (as opposed to the AWS console) do not instantiate the default instance profile IAM role. The solution is to either create (and then delete) an Elastic Beanstalk instance from the console, or to create the required IAM role manually. The IAM role is called `aws-elasticbeanstalk-ec2-role` and contains three permissions: (1) `AWSElasticBeanstalkWebTier`, (2) `AWSElasticBeanstalkMulticontainerDocker`, and (3) `AWSElasticBeanstalkWorkerTier`.\n\n\n### Amazon Mechanical Turk\n\nFollow the instructions [here](https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMechanicalTurkGettingStartedGuide/SetUp.html) for setting up MTurk and connecting it to your AWS account.\n\n\n### Heroku (Optional if you don't want to use AWS)\n\nSign up for a Heroku account. Go to `Account Settings`. At the bottom of the page in the `API Key` section is a `Reveal` button.\n\n<p align=\"center\">\n <img src=\"docs/images/heroku-00.png\" width=\"400\" alt=\"Heroku API key instructions\">\n <img src=\"docs/images/heroku-01.png\" width=\"400\" alt=\"Heroku API key instructions\">\n</p>\n\nYou will also need to enable billing. You can do so [here](https://heroku.com/verify).\n\n\n## Usage\n\nOnce you have your configuration file and a properly formatted directory of evaluation files, you are ready to deploy a subjective evaluation. Example configuration files and corresponding evaluation files can be found in `examples/`.\n\nIf you are not deploying locally, add your API keys.\n\n```bash\n# AWS credentials\npython -m reseval.credentials \\\n --aws_api_key <aws_api_key> \\\n --aws_api_secret_key <aws_api_secret_key>\n\n# (Optional) Heroku credentials\npython -m reseval.credentials \\\n --heroku_api_key <heroku_api_key>\n```\n\n API keys are saved in `reseval.CACHE / '.keys'`. The `.keys` file is used to set local environment variables and is not pushed to GitHub or uploaded to any remote storage.\n\n\n### Command-line interface\n\n Arguments for the following command-line interfaces are as follows, unless otherwise specified.\n\n- `<config>` - The configuration file\n- `<directory>` - The directory of evaluation files\n- `<name>` - The name of the evaluation given in the configuration file\n\n\n#### Create\n\nCreate a subjective evaluation either locally, in remote development mode (e.g., MTurk Sandbox), or in production mode.\n\n**Note** - `reseval.create` is not currently thread-safe. Wait until the first call has finished before calling it again. See [this GitHub issue](https://github.com/reseval/reseval/issues/5).\n\n```bash\n# Local development\npython -m reseval.create <config> <directory> --local\n\n# Remote development\npython -m reseval.create <config> <directory>\n\n# Production\npython -m reseval.create <config> <directory> --production\n```\n\n\n#### Monitor\n\n```bash\n# Monitor all subjective evaluations\npython -m reseval.monitor\n\n# Monitor one subjective evaluation\n# The name of the evaluation can be found in its configuration file\npython -m reseval.monitor --name <name>\n```\n\n**Note** - By default, the monitor updates once every minute. You can update the monitor more or less often by providing an update interval in seconds.\n\n```bash\n# Update the monitor once every ten seconds\npython -m reseval.monitor --interval 10\n```\n\n\n#### Results\n\n```bash\n# Get the results of a subjective evaluation.\n# Results are stored in <directory>/<name>.\n# <directory> defaults to the current directory.\npython -m reseval.results <name> --directory <directory>\n```\n\n\n#### Pay\n\n```bash\n# Pay participants\npython -m reseval.pay <name>\n```\n\n\n#### Destroy\n\n```bash\n# Destroy the compute resources of a subjective evaluation (e.g., any cloud\n# storage, databases, or servers)\npython -m reseval.destroy <name>\n\n# Destroy a subjective evaluation even if it is still active.\n# Participants who have taken the test so far will be paid.\npython -m reseval.destroy <name> --force\n```\n\n\n#### Extend\n\n```bash\n# Add <participants> additional participants to a finished evaluation\npython -m reseval.extend <name> <participants>\n```\n\n\n### Application programming interface\n\nDocumentation for our API can be found [here](https://reseval.github.io/reseval/html/index.html).\n\n\n## Advanced usage\n\nOnce you feel comfortable with using ReSEval step-by-step from the\ncommand-line and after you have added your credentials with\n`reseval.credentials`, you can use the CLI or API to run your evaluation with\nonly a single command.\n\n\n### CLI\n\n```bash\n# Local development\npython -m reseval <config> <directory> --local\n\n# Remote development\npython -m reseval <config> <directory>\n\n# Production\npython -m reseval <config> <directory> --production\n```\n\n\n### API\n\n```python\nimport reseval\n\n# Local development\nreseval.run(config, directory, local=True)\n\n# Remote development\nreseval.run(config, directory)\n\n# Production\nreseval.run(config, directory, production=True)\n```\n\n\n## Additional monitoring\n\n### AWS S3\n\nTo monitor, edit, or delete AWS S3 storage buckets, or see storage costs, use the\n[AWS S3 console](https://s3.console.aws.amazon.com/s3).\n\n\n### AWS Elastic Beanstalk\n\nTo monitor, edit, or delete the server compute, use the\n[AWS Elastic Beanstalk console](https://console.aws.amazon.com/elasticbeanstalk).\n\n\n### AWS Relational Database Service\n\nTo monitor, edit, or delete the database, use the\n[AWS RDS console](https://console.aws.amazon.com/rds).\n\n\n### MTurk\n\nHITs not created on the MTurk dashboard are not visible on the MTurk dashboard.\nYou can use the MTurk CLI to monitor, edit, or delete HITs. MTurk costs appear\non the [AWS billing dashboard](https://console.aws.amazon.com/billing/home)\nat the end of the billing period.\n\n\n### Heroku\n\nTo monitor, edit, or delete Heroku databases and servers, use the\n[Heroku application dashboard](https://dashboard.heroku.com/apps). You can see\nany costs on the\n[billing dashboard](https://dashboard.heroku.com/account/billing).\n",
"bugtrack_url": null,
"license": "",
"summary": "Reproducible Subjective Evaluation",
"version": "0.1.6",
"project_urls": {
"Homepage": "https://github.com/reseval/reseval"
},
"split_keywords": [
"annotation",
"audio",
"ab",
"abx",
"crowdsourcing",
"evaluation",
"image",
"mos",
"mushra",
"speech",
"subjective"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "87d5b3f2c34bf0940e4b9c13661d11e454d909d6b37af5e742ca77ab0e317b66",
"md5": "19b49490eaa75c42d085f2244da50cec",
"sha256": "2a798acaeecccdec974454e9bc48fa02d92ac665dc86c6e10b59908a93c96833"
},
"downloads": -1,
"filename": "reseval-0.1.6-py3-none-any.whl",
"has_sig": false,
"md5_digest": "19b49490eaa75c42d085f2244da50cec",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 3842863,
"upload_time": "2024-03-03T05:32:13",
"upload_time_iso_8601": "2024-03-03T05:32:13.224923Z",
"url": "https://files.pythonhosted.org/packages/87/d5/b3f2c34bf0940e4b9c13661d11e454d909d6b37af5e742ca77ab0e317b66/reseval-0.1.6-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a5164bfab6771f6c6fce6f710cc881a98e995efc6c6bb1a688bc369db31dcf17",
"md5": "bc453241ad6fcb1966b60f749205e01e",
"sha256": "a09c06d9ac9738e818bc7813ccc509fa4a1965dcd52b8189e83db69ed2eea2d6"
},
"downloads": -1,
"filename": "reseval-0.1.6.tar.gz",
"has_sig": false,
"md5_digest": "bc453241ad6fcb1966b60f749205e01e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 3813111,
"upload_time": "2024-03-03T05:32:16",
"upload_time_iso_8601": "2024-03-03T05:32:16.387484Z",
"url": "https://files.pythonhosted.org/packages/a5/16/4bfab6771f6c6fce6f710cc881a98e995efc6c6bb1a688bc369db31dcf17/reseval-0.1.6.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-03 05:32:16",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "reseval",
"github_project": "reseval",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "reseval"
}
JSON
