# Autograder.io Command-Line Interface
A command-line tool for managing assignments on Autograder.io.
We also recommend Amir Kamil's [autograder-tools](https://gitlab.eecs.umich.edu/akamil/autograder-tools/tree/master) as a complimentary collection of applications.
## Quickstart
### Install
While this tool is usable in its current state, things may change between now and our first official release.
To install the latest development release, pass the `--pre` flag to pip as below:
```
pip install --pre autograder-cli
```
### Obtain API Token
Visit https://autograder.io/web/__apitoken__ and sign in.
Save the file you are prompted to download as `.agtoken` in your home directory or the directory.
Check that you've authenticated correctly with the following command:
```
ag http get /api/users/current/
```
This command should return your user information.
**IMPORTANT**: If you are using your own deployment of Autograder.io, you will need to specify the base URL of that deployment for every command you run.
Specify the base URL with the --base_url flag, e.g.:
```
ag --base_url https://your.url.com http get /api/users/current/
```
You may want to alias `ag --base_url https://your.url.com` in your shell profile for convenience.
### Configure Autocomplete in VSCode
1. Install the [VSCode YAML plugin](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml) (published by RedHat).
2. Generate the Autograder.io CLI JSON Schema:
```
ag write-schema
```
This will create a file called `autograder_io_cli_schema.json` in the directory the above command was run in.
3. Add the following to your VSCode settings.json:
```
"yaml.schemas": {
"/path/to//autograder_io_cli_schema.json": ["agproject.yml", "*.agproject.yml"]
}
```
This will cause the YAML plugin to recognize `agproject.yml` and `*.agproject.yml` files as using the Autograder.io CLI schema.
Pull requests are welcome that add instructions for setting up autocomplete on other editors.
### Common Usage
#### New Project From Scratch
Use the following command to create a project config file with default values.
Replace `'My Course' Fall 2025` with the name, term, and year of your course.
Replace `'My Project'` with the name of your assignment.
The course you identify by name, semester, and year must already exist before you can save the project.
This only creates the config file locally, it does not save any changes to Autograder.io.
```
ag project init 'My Course' Fall 2025 'My Project'
```
This will create a configuration file `agproject.yml`.
You can choose a different name with the `--config_file` flag.
This CLI attempts to detect your local timezone.
You can change the `timezone` field in the config file if you wish to use a different timezone.
See [Save a Project](Save-a-Project) to save your configured project
#### Download an Existing Project
This command creates a config file for an existing project and downloads instructor files associated with that project.
For example:
```
ag project load 'My Course' Fall 2025 'My Project' myproject.agproject.yml
```
will download the specified project and save its configuration to the file `myproject.agproject.yml`.
Instructor files for that project will be saved in the same directory as the config file.
That is, if you specify `../some/path/myproject.agproject.yml`, the instructor files will be saved to `../some/path/`.
#### Save a Project
To save your configured project to Autograder.io, run the following command from the same directory as the config file:
```
ag project save
```
You can specify a different config file with the `--config_file` flag.
## FAQ and Tips
### Repeating Test Cases
The config format provides a "repeat" mechanism for defining groups of similar test cases in a compact way.
For example:
```
project:
# Project information and settings
# ...
test_suites:
- name: Python Unit Tests
test_cases:
# $test_label is a user-chosen placeholder.
# The leading $ is not required but helps readability.
- name: Test $test_label
cmd: python3 -m unittest -k $test_name
return_code:
expected: zero
feedback:
normal: pass/fail+timeout
final_graded_submission: pass/fail+timeout
# Define substitutions for our placeholders
repeat:
- $test_label: Normal 1
$test_name: normal1
# Specify specific values for this test case.
_override:
return_code:
points: 2
- $test_label: Edge 1
$test_name: edge1
_override:
return_code:
points: 3
```
Saving this configuration will create two test cases with the same feedback and expected return code settings, but with different names, commands, and point values.
### Default Values
Since there is not a perfect one-to-one mapping between CLI project configuration options and API fields, there are some cases where the CLI and API have different default values.
Removing a field from the config file and then saving the project will reset that value to its CLI default.
The `ag project init` command creates a config file with all fields present and set to their CLI defaults.
In contrast, the `ag project load` command creates a config file that only contains non-default values.
### Feedback Presets
The config file supports several presets for test case feedback settings.
You can also define your own feedback presets under the `feedback_presets` key (for single command tests or the commands in multi-command tests) and the `feedback_presets_test_suite_setup` key (for test suite setup commands).
The config file created by `ag project init` contains definitions for several built-in presets.
Do not edit the contents presets.
However, once you are familiar with the CLI's built-in presets, you may remove them from the config file.
The CLI will recognize them even if they are not present in the config file.
### Sandbox Images
The CLI does not yet support building sandbox images.
Please build your images through the Autograder.io website.
In test suite definitions in the config file, you may specify the name of any sandbox image you've built for the course on Autograder.io.
### Deleting Entries
The CLI currently does NOT support deleting entries such as instructor files, expected student files, or test cases.
Removing these entries from the config file and saving the project will NOT delete those entries.
If you need to delete those entires, please do so through the Autograder.io website.
## Versioning
This package uses calendar versioning following [Python conventions](https://packaging.python.org/en/latest/discussions/versioning/), with version numbers of the form `yy.mm.X`, where `X` is for minor versions.
For example: `24.8.0` corresponds to August 2024.
We also make use of pre-release tags such as `.devX`.
The year and month of the release specify the earliest version of the Autograder.io API this package is compatible with.
However, the minor version number **does not correspond** to Autograder.io's minor version numbers.
Also note that backwards-incompatible changes to the Autograder.io API may make future versions of that API incompatible with earlier versions of the CLI.
## Dev Setup
### Clone the Repository
```
git clone --recursive git@github.com:eecs-autograder/autograder-cli.git
```
If you omitted the `--recursive` flag, initialize the submodule with:
```
git submodule update --init
```
### Install Dependencies
Create and activate a virtual environment:
```
python3.10 -m venv venv
source venv/bin/activate
```
Install package dependencies and install the autograder-cli as a local editable package:
```
pip install pip-tools
./dev_scripts/install_deps.sh
```
Install [dyff](https://github.com/homeport/dyff) for comparing yaml files in test cases:
```
curl --silent --location https://git.io/JYfAY | bash
```
### Build the Local autograder-server Stack
Build and start the stack:
```
./dev_scripts/local_stack.sh build
./dev_scripts/local_stack.sh up -d
```
`./dev_scripts/local_stack.sh` is an alias for a docker-compose command.
Generate the gpg secrets for the autograder-server stack:
```
python -m pip install Django==3.1 python-gnupg
cd tests/local_stack/autograder-server && python3 generate_secrets.py
```
[Running the tests](Tests) will finish preparing the stack by applying migrations and clearing the database.
### Linters
Run isort, black, pycodestyle, pydocstyle, and pyright to check for style, formatting, and type issues:
```
./dev_scripts/lint.sh
```
Python code should be formatted using isort and black.
### Tests
This project uses pytest as its test runner.
Most of the test cases are currently "roundtrip" tests that save and load a configuration.
To generate a new roundtrip test, run:
```
./dev_scripts/new_roundtrip_test.sh {test name}
```
The test name can include directories (e.g., ag_test_suite/setup_cmd).
This will initialize a roundtrip test in tests/roundtrip/{test name}.test.
Roundtrip tests consist of the following steps:
1. Save the project found in `{test name}/project.create.yml`.
2. Load that project and compare the loaded version with `{test name}/project.create.expected.yml`.
3. Save the project found in `{test name}/project.update.yml`. (this is intended to be the same project that was created in step one, but with some fields changed)
4. Load that project and compare the loaded version with `{test name}/project.update.expected.yml`.
When testing deadline formats (e.g., fixed cutoff, relative cutoff), you can specify which format to load deadlines into in the file `{test name}/deadline_cutoff_preference`.
In the `.expected.yml` files, if you haven't set any values in `project.settings`, you will need to set `project.settings` to an empty dictionary.
We haven't made the `new_roundtrip_test.sh` script make this change because it serves as a way to have new test cases fail until they are edited.
### The HTTPClient
The `HTTPClient` class is a starting point for sending custom requests in Python applications.
```
import json
from ag_contrib.http_client import HTTPClient, check_response_status
client = HTTPClient.make_default()
response = client.get('/api/users/current/')
check_response_status(response)
print(json.dumps(response.json(), indent=4))
```
#### HTTP Client Command-Line Interface
This library provides a minimal command-line interface for sending custom HTTP requests to the Autograder.io API.
Run `ag http --help` for more details.
Raw data
{
"_id": null,
"home_page": null,
"name": "autograder-cli",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "autograder, autograder.io",
"author": null,
"author_email": "James Perretta <jameslp@umich.edu>",
"download_url": "https://files.pythonhosted.org/packages/ad/51/d6baba17250e2be6ff3475bf0a0d1a41b58dfce2884dac4b4f9d1ad03b4f/autograder_cli-24.8.0.dev1.tar.gz",
"platform": null,
"description": "# Autograder.io Command-Line Interface\nA command-line tool for managing assignments on Autograder.io.\n\nWe also recommend Amir Kamil's [autograder-tools](https://gitlab.eecs.umich.edu/akamil/autograder-tools/tree/master) as a complimentary collection of applications.\n\n## Quickstart\n### Install\nWhile this tool is usable in its current state, things may change between now and our first official release.\nTo install the latest development release, pass the `--pre` flag to pip as below:\n```\npip install --pre autograder-cli\n```\n\n### Obtain API Token\nVisit https://autograder.io/web/__apitoken__ and sign in.\nSave the file you are prompted to download as `.agtoken` in your home directory or the directory.\n\nCheck that you've authenticated correctly with the following command:\n```\nag http get /api/users/current/\n```\nThis command should return your user information.\n\n**IMPORTANT**: If you are using your own deployment of Autograder.io, you will need to specify the base URL of that deployment for every command you run.\nSpecify the base URL with the --base_url flag, e.g.:\n```\nag --base_url https://your.url.com http get /api/users/current/\n```\nYou may want to alias `ag --base_url https://your.url.com` in your shell profile for convenience.\n\n### Configure Autocomplete in VSCode\n1. Install the [VSCode YAML plugin](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml) (published by RedHat).\n2. Generate the Autograder.io CLI JSON Schema:\n```\nag write-schema\n```\nThis will create a file called `autograder_io_cli_schema.json` in the directory the above command was run in.\n3. Add the following to your VSCode settings.json:\n```\n \"yaml.schemas\": {\n \"/path/to//autograder_io_cli_schema.json\": [\"agproject.yml\", \"*.agproject.yml\"]\n }\n```\nThis will cause the YAML plugin to recognize `agproject.yml` and `*.agproject.yml` files as using the Autograder.io CLI schema.\n\nPull requests are welcome that add instructions for setting up autocomplete on other editors.\n\n### Common Usage\n#### New Project From Scratch\nUse the following command to create a project config file with default values.\nReplace `'My Course' Fall 2025` with the name, term, and year of your course.\nReplace `'My Project'` with the name of your assignment.\nThe course you identify by name, semester, and year must already exist before you can save the project.\nThis only creates the config file locally, it does not save any changes to Autograder.io.\n```\nag project init 'My Course' Fall 2025 'My Project'\n```\n\nThis will create a configuration file `agproject.yml`.\nYou can choose a different name with the `--config_file` flag.\n\nThis CLI attempts to detect your local timezone.\nYou can change the `timezone` field in the config file if you wish to use a different timezone.\n\nSee [Save a Project](Save-a-Project) to save your configured project\n\n#### Download an Existing Project\nThis command creates a config file for an existing project and downloads instructor files associated with that project.\nFor example:\n```\nag project load 'My Course' Fall 2025 'My Project' myproject.agproject.yml\n```\nwill download the specified project and save its configuration to the file `myproject.agproject.yml`.\nInstructor files for that project will be saved in the same directory as the config file.\nThat is, if you specify `../some/path/myproject.agproject.yml`, the instructor files will be saved to `../some/path/`.\n\n#### Save a Project\nTo save your configured project to Autograder.io, run the following command from the same directory as the config file:\n```\nag project save\n```\n\nYou can specify a different config file with the `--config_file` flag.\n\n## FAQ and Tips\n### Repeating Test Cases\nThe config format provides a \"repeat\" mechanism for defining groups of similar test cases in a compact way.\nFor example:\n```\nproject:\n # Project information and settings\n # ...\n test_suites:\n - name: Python Unit Tests\n test_cases:\n # $test_label is a user-chosen placeholder.\n # The leading $ is not required but helps readability.\n - name: Test $test_label\n cmd: python3 -m unittest -k $test_name\n return_code:\n expected: zero\n feedback:\n normal: pass/fail+timeout\n final_graded_submission: pass/fail+timeout\n # Define substitutions for our placeholders\n repeat:\n - $test_label: Normal 1\n $test_name: normal1\n # Specify specific values for this test case.\n _override:\n return_code:\n points: 2\n - $test_label: Edge 1\n $test_name: edge1\n _override:\n return_code:\n points: 3\n```\n\nSaving this configuration will create two test cases with the same feedback and expected return code settings, but with different names, commands, and point values.\n\n### Default Values\nSince there is not a perfect one-to-one mapping between CLI project configuration options and API fields, there are some cases where the CLI and API have different default values.\nRemoving a field from the config file and then saving the project will reset that value to its CLI default.\n\nThe `ag project init` command creates a config file with all fields present and set to their CLI defaults.\nIn contrast, the `ag project load` command creates a config file that only contains non-default values.\n\n### Feedback Presets\nThe config file supports several presets for test case feedback settings.\nYou can also define your own feedback presets under the `feedback_presets` key (for single command tests or the commands in multi-command tests) and the `feedback_presets_test_suite_setup` key (for test suite setup commands).\nThe config file created by `ag project init` contains definitions for several built-in presets.\nDo not edit the contents presets.\nHowever, once you are familiar with the CLI's built-in presets, you may remove them from the config file.\nThe CLI will recognize them even if they are not present in the config file.\n\n### Sandbox Images\nThe CLI does not yet support building sandbox images.\nPlease build your images through the Autograder.io website.\nIn test suite definitions in the config file, you may specify the name of any sandbox image you've built for the course on Autograder.io.\n\n### Deleting Entries\nThe CLI currently does NOT support deleting entries such as instructor files, expected student files, or test cases.\nRemoving these entries from the config file and saving the project will NOT delete those entries.\nIf you need to delete those entires, please do so through the Autograder.io website.\n\n## Versioning\nThis package uses calendar versioning following [Python conventions](https://packaging.python.org/en/latest/discussions/versioning/), with version numbers of the form `yy.mm.X`, where `X` is for minor versions.\nFor example: `24.8.0` corresponds to August 2024.\nWe also make use of pre-release tags such as `.devX`.\n\nThe year and month of the release specify the earliest version of the Autograder.io API this package is compatible with.\nHowever, the minor version number **does not correspond** to Autograder.io's minor version numbers.\nAlso note that backwards-incompatible changes to the Autograder.io API may make future versions of that API incompatible with earlier versions of the CLI.\n\n## Dev Setup\n### Clone the Repository\n```\ngit clone --recursive git@github.com:eecs-autograder/autograder-cli.git\n```\n\nIf you omitted the `--recursive` flag, initialize the submodule with:\n```\ngit submodule update --init\n```\n\n### Install Dependencies\nCreate and activate a virtual environment:\n```\npython3.10 -m venv venv\nsource venv/bin/activate\n```\n\nInstall package dependencies and install the autograder-cli as a local editable package:\n```\npip install pip-tools\n./dev_scripts/install_deps.sh\n```\n\nInstall [dyff](https://github.com/homeport/dyff) for comparing yaml files in test cases:\n```\ncurl --silent --location https://git.io/JYfAY | bash\n```\n\n### Build the Local autograder-server Stack\nBuild and start the stack:\n```\n./dev_scripts/local_stack.sh build\n./dev_scripts/local_stack.sh up -d\n```\n`./dev_scripts/local_stack.sh` is an alias for a docker-compose command.\n\nGenerate the gpg secrets for the autograder-server stack:\n```\npython -m pip install Django==3.1 python-gnupg\ncd tests/local_stack/autograder-server && python3 generate_secrets.py\n```\n\n[Running the tests](Tests) will finish preparing the stack by applying migrations and clearing the database.\n\n### Linters\nRun isort, black, pycodestyle, pydocstyle, and pyright to check for style, formatting, and type issues:\n```\n./dev_scripts/lint.sh\n```\nPython code should be formatted using isort and black.\n\n### Tests\nThis project uses pytest as its test runner.\nMost of the test cases are currently \"roundtrip\" tests that save and load a configuration.\nTo generate a new roundtrip test, run:\n```\n./dev_scripts/new_roundtrip_test.sh {test name}\n```\n\nThe test name can include directories (e.g., ag_test_suite/setup_cmd).\nThis will initialize a roundtrip test in tests/roundtrip/{test name}.test.\nRoundtrip tests consist of the following steps:\n1. Save the project found in `{test name}/project.create.yml`.\n2. Load that project and compare the loaded version with `{test name}/project.create.expected.yml`.\n3. Save the project found in `{test name}/project.update.yml`. (this is intended to be the same project that was created in step one, but with some fields changed)\n4. Load that project and compare the loaded version with `{test name}/project.update.expected.yml`.\n\nWhen testing deadline formats (e.g., fixed cutoff, relative cutoff), you can specify which format to load deadlines into in the file `{test name}/deadline_cutoff_preference`.\n\nIn the `.expected.yml` files, if you haven't set any values in `project.settings`, you will need to set `project.settings` to an empty dictionary.\nWe haven't made the `new_roundtrip_test.sh` script make this change because it serves as a way to have new test cases fail until they are edited.\n\n### The HTTPClient\nThe `HTTPClient` class is a starting point for sending custom requests in Python applications.\n```\nimport json\nfrom ag_contrib.http_client import HTTPClient, check_response_status\n\nclient = HTTPClient.make_default()\nresponse = client.get('/api/users/current/')\ncheck_response_status(response)\nprint(json.dumps(response.json(), indent=4))\n```\n\n#### HTTP Client Command-Line Interface\nThis library provides a minimal command-line interface for sending custom HTTP requests to the Autograder.io API.\nRun `ag http --help` for more details.\n",
"bugtrack_url": null,
"license": null,
"summary": "A command-line http client and other utilities for using the autograder.io API.",
"version": "24.8.0.dev1",
"project_urls": {
"Homepage": "https://github.com/eecs-autograder/autograder-cli",
"Issues": "https://github.com/eecs-autograder/autograder-cli/issues"
},
"split_keywords": [
"autograder",
" autograder.io"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "04969a663f3559f67f6bf417a163e226760726910dae3b5324f2e1eea8d428ea",
"md5": "c3ff08e128301f142190a68e9471c55a",
"sha256": "0aec92f883e77f0254904eba31e25887aa32e31934aa1ab633f246e7eff04859"
},
"downloads": -1,
"filename": "autograder_cli-24.8.0.dev1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c3ff08e128301f142190a68e9471c55a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 63761,
"upload_time": "2025-01-08T22:35:12",
"upload_time_iso_8601": "2025-01-08T22:35:12.718380Z",
"url": "https://files.pythonhosted.org/packages/04/96/9a663f3559f67f6bf417a163e226760726910dae3b5324f2e1eea8d428ea/autograder_cli-24.8.0.dev1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ad51d6baba17250e2be6ff3475bf0a0d1a41b58dfce2884dac4b4f9d1ad03b4f",
"md5": "0bdf918b7db98f814018f920904f1caa",
"sha256": "75ac9e46f35e56c55430d7eace4cbda806636b72430e9185fd4ae0b43c38fc46"
},
"downloads": -1,
"filename": "autograder_cli-24.8.0.dev1.tar.gz",
"has_sig": false,
"md5_digest": "0bdf918b7db98f814018f920904f1caa",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 98699,
"upload_time": "2025-01-08T22:35:13",
"upload_time_iso_8601": "2025-01-08T22:35:13.957637Z",
"url": "https://files.pythonhosted.org/packages/ad/51/d6baba17250e2be6ff3475bf0a0d1a41b58dfce2884dac4b4f9d1ad03b4f/autograder_cli-24.8.0.dev1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-08 22:35:13",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "eecs-autograder",
"github_project": "autograder-cli",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "annotated-types",
"specs": [
[
"==",
"0.6.0"
]
]
},
{
"name": "certifi",
"specs": [
[
"==",
"2023.11.17"
]
]
},
{
"name": "charset-normalizer",
"specs": [
[
"==",
"3.3.2"
]
]
},
{
"name": "dnspython",
"specs": [
[
"==",
"2.4.2"
]
]
},
{
"name": "email-validator",
"specs": [
[
"==",
"2.1.0.post1"
]
]
},
{
"name": "idna",
"specs": [
[
"==",
"3.6"
]
]
},
{
"name": "pydantic",
"specs": [
[
"==",
"2.10.3"
]
]
},
{
"name": "pydantic-core",
"specs": [
[
"==",
"2.27.1"
]
]
},
{
"name": "python-dateutil",
"specs": [
[
"==",
"2.9.0.post0"
]
]
},
{
"name": "pyyaml",
"specs": [
[
"==",
"6.0.1"
]
]
},
{
"name": "requests",
"specs": [
[
"==",
"2.31.0"
]
]
},
{
"name": "six",
"specs": [
[
"==",
"1.16.0"
]
]
},
{
"name": "typing-extensions",
"specs": [
[
"==",
"4.12.2"
]
]
},
{
"name": "tzlocal",
"specs": [
[
"==",
"5.2"
]
]
},
{
"name": "urllib3",
"specs": [
[
"==",
"2.1.0"
]
]
}
],
"lcname": "autograder-cli"
}
JSON
