| Name | turnt JSON |
| Version |
1.12.0
JSON |
| download |
| home_page | https://github.com/cucapra/turnt |
| Summary | Turnt is a simple expect-style testing tool for command-line |
| upload_time | 2024-07-12 13:51:20 |
| maintainer | None |
| docs_url | None |
| author | Adrian Sampson |
| requires_python | >=3.6 |
| license | None |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
Tiny Unified Runner N' Tester (Turnt)
=====================================
Turnt is a simple snapshot testing tool inspired by [Cram][] and [LLVM's lit][lit].
It's good for testing things that translate text files to other text files, like compilers.
The idea is that each test is one input file, and you want to run a command and check that it still matches the saved output file.
To use it:
1. *Configure.*
Decide what command you want to test.
Make a `turnt.toml` config file and put `command = "mycmd {filename}"` in it to pass each test file as an argument to `mycmd`.
2. *Create a test.*
Just write an input file next to your `turnt.toml`.
We'll call it `foo.t`.
3. *Take a snapshot.*
Run `turnt --save foo.t` to execute `mycmd foo.t` and save the standard output into `foo.out`.
You might want to take a look at this output to make sure it's what you expect.
Then you check both the input `foo.t` and output `foo.out` into version control.
4. *Test your work.*
Now that you have a test in place, keep working.
Use `turnt *.t` to run all your tests and confirm that the output still matches.
If there's a mismatch, you can do `turnt --diff` to see the changes.
(Or if you're confident, try `turnt --save` followed by `git diff`.)
Turnt's philosophy is to minimize the effort it takes to write new tests so you can quickly build up lots of them.
You don't write any custom logic to check results; you just record the complete "golden" output for each test.
Compared to ordinary unit testing, "snapshot" tests incur the mental effort of manually inspecting diffs when things change.
In return, it's easier to expand test coverage.
Snapshots also act as a crude form of documentation because every test is a complete, valid input to your program.
[cram]: https://bitheap.org/cram/
[lit]: https://llvm.org/docs/CommandGuide/lit.html
Install
-------
This is a Python 3 tool.
Install it with [pip][]:
$ pip install --user turnt
Or, if you want to work on Turnt, you can install [Flit][], clone this repository, and type this to get a "live" installation with a symlink:
$ flit install --symlink --user
[pip]: https://pip.pypa.io/
[flit]: https://flit.readthedocs.io/
Configuring
-----------
Turnt looks for a configuration file called `turnt.toml` in any of the ancestor directories of your test file.
It can be alongside the test file or in any containing directory.
It's a [TOML][] file that looks like something this:
command = "mycmd {args} < {filename}"
return_code = 42
output.txt = "result.txt"
### `command`
Set `command` to a shell command to run on a given test file.
This is the only setting that is truly required.
The command is a [template][str.format]; Turnt will fill in these values:
- `{filename}`: The path to the test file, relative to the working directory.
- `{base}`: The basename of the test file, with the extension removed.
- `{args}`: Some extra arguments that the test or user provides.
(See the `ARGS:` override and the `--args` command-line option below.)
The working directory for the command is the location of the `turnt.toml` configuration file, if any.
If there's no configuration file, then it's the location of the test file itself.
### `return_code`
By default, Turnt expects the test command to succeed, i.e., exit with status code 0.
Set `return_code` to a different status if you expect failure.
### `output`
By default, Turnt captures the standard output stream from your test command.
If your command produces other output files "on the side" or you want to capture the standard error stream, you can configure the `output` table.
`output` is a mapping from *snapshot extensions* to *collected filenames*.
For example, this TOML configuration:
output.txt = "result.txt"
means that running the command will produce a file called `result.txt`, and we want to save that file in a snapshot called `<test-name>.txt`.
In place of a filename, use `-` to indicate the command's standard output and `2` to indicate its standard error.
The default behaves like this configuration:
output.out = "-"
which captures stdout and saves it in `<test-name>.out`.
Defining `output` in `turnt.toml` disables this default behavior; you can include it explicitly if you want it alongside other outputs.
### `binary`
By default, Turnt looks inside test files for overrides (see below).
This won't work if your test inputs are binary (non-text) files (Turnt will warn you and proceed with no overrides).
Set `binary = true` to suppress this search for overrides altogether.
### `out_dir`
Optionally put expected test outputs in a different directory.
By default, this is `.`, i.e., expected output files go "next to" test files.
Set this to a different directory, relative to the test file, to keep them somewhere else.
### `todo`
Mark a test as an allowed failure.
You might add a test that exposes a bug that you haven't fixed yet, for example, but you want your CI to stay "green."
Set `todo = true` to allow the test to fail; Turnt will mark it with `# todo` in the TAP output.
Per-Test Overrides
------------------
Sometimes you need to alter the setup for a specific test file.
Turnt looks for some overrides embedded in the test file itself: for example, you might put them in a comment at the top of a test program.
Put these things into your test file to override the configuration:
- `CMD: <command>` overrides `command` from the configuration.
- `ARGS: <arguments>` adds arguments to a configured command.
Turnt puts this string in where the command uses `{args}`.
- `OUT: <ext> <filename>` overrides `output` from the configuration.
You can specify multiple files this way: one line per file.
- `RETURN: <code>` overrides the expected exit status.
- `TODO: <true|false>` can mark a test as a "todo" (allowed failure).
Multiple Environments
---------------------
Turnt is mostly about running one command on many input files.
Sometimes, however, you need to run several commands on each file.
This can be especially useful for *[differential testing][dt]:*
when you want to check that multiple things behave the same way by checking that they produce the same input when you give them the same input.
You can create multiple environments in your configuration file under the `envs` table.
The table maps environment *names* to sets of configuration options that are just like the [top-level configuration described above](#configuring).
For example:
[envs.baseline]
command = "interp -g {filename}"
[envs.optimized]
command = "compile -O3 {filename} ; ./a.out"
Each environment can have the full complement of configuration options described above:
for example, `command`, `output`, and `return_code`.
When you run `turnt` on some files, it will run all the environments on all the files.
There is also a Boolean `default` flag to turn off an environment by default; that way, you can only use it by asking for it with the `-e` flag (see below).
This example is a differential testing setup because both environments share the same (default) snapshot file:
a test `foo.t` will look for its stdout snapshot in `foo.out`.
If the two environments don't match, at least one will fail.
If you want a more standard (non-differential) setup, just set the `output` configuration differently for the two environments, like this:
[envs.interp]
command = "interp -g {filename}"
output.res = "-"
[envs.profile]
command = "profile {filename}"
output.prof = "-"
[dt]: https://en.wikipedia.org/wiki/Differential_testing
Directory Tests
---------------
A Turnt test is usually just a single input file, but you can also organize multiple related files into a directory.
Use the directory the same way as you would a single file:
pass its path to the `turnt` command, and the path will appear as the `{filename}` for the configured command.
So you might configure your test command like this:
command = "mycmd {filename}/test.c"
if you want each test directory to contain a file called `test.c`.
Turnt puts snapshots *inside* the test directory instead of adjacent to it.
It names them `out.<extension>` in that directory.
There are two configuration options just for dealing with directory tests:
- `out_base`.
The basename for output files in directory tests: by default, `out`.
- `opts_file`.
The filename to read inside of a directory test to search for embedded overrides.
In our above example, you could set this to `test.c` to look for `ARGS:` and friends in that file, or `opts.txt` to look for them in a separate file on the side.
[toml]: https://github.com/toml-lang/toml
[str.format]: https://docs.python.org/3/library/string.html#formatstrings
Command-Line Interface
----------------------
The most common `turnt` command-line options you'll need while running and updating tests are:
- `--save`: Bless the current output from each test as the "correct" output, saving it to the output file that you'll want to check into version control.
- `--diff`: Show diffs between the actual and expected output for each test.
(The diff goes to stderr while the [TAP][] results remain on stdout.)
You also might enjoy:
- `--parallel` or `-j`: Run your tests faster using parallel threads.
These options are useful when working with one specific test file:
- `--verbose` or `-v`: Disable Turnt's default behavior where it will suppress test commands' stderr output. The result is more helpful but harder to read.
- `--print` or `-p`: Instead of checking test results, just run the command and show the output directly. This can be useful (especially in combination with `-v`) when iterating on a test interactively.
- `--args` or `-a`: Override the `{args}` string in the test command.
These options lets you switch between different test environments:
- `--env` or `-e`: Give the name of a configured [environment](#multiple-environments) to run. Use this multiple times to run multiple environments. By default, Turnt runs all the configured environments for every test. Use the special name `default` for the top-level, unnamed test environment.
- `--config` or `-c`: Look for this config filename instead of the default `turnt.toml`.
TAP
---
Turnt outputs results in the machine-readable [TAP][] format.
To make the output more pleasant to read, you can pipe it into a tool like [tap-difflet][], [tap-dot][], or [faucet][]:
$ npm install -g tap-difflet
$ turnt *.t | tap-difflet
[tap]: http://testanything.org
[tap-difflet]: https://github.com/namuol/tap-difflet
[tap-dot]: https://github.com/scottcorgan/tap-dot
[faucet]: https://github.com/substack/faucet
Credits
-------
Turnt is by [Adrian Sampson][adrian] and [Alexa VanHattum][alexa].
We made it to test various research compilers in [Capra][].
The license is [MIT][].
[adrian]: https://www.cs.cornell.edu/~asampson/
[alexa]: https://www.cs.cornell.edu/~avh/
[capra]: https://capra.cs.cornell.edu
[mit]: https://opensource.org/licenses/MIT
Raw data
{
"_id": null,
"home_page": "https://github.com/cucapra/turnt",
"name": "turnt",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": null,
"keywords": null,
"author": "Adrian Sampson",
"author_email": "asampson@cs.cornell.edu",
"download_url": "https://files.pythonhosted.org/packages/6b/ff/9e3653777373cc13565401685faaa95c941c463e54c8d53abd81e3656ebb/turnt-1.12.0.tar.gz",
"platform": null,
"description": "Tiny Unified Runner N' Tester (Turnt)\n=====================================\n\nTurnt is a simple snapshot testing tool inspired by [Cram][] and [LLVM's lit][lit].\nIt's good for testing things that translate text files to other text files, like compilers.\nThe idea is that each test is one input file, and you want to run a command and check that it still matches the saved output file.\n\nTo use it:\n\n1. *Configure.*\n Decide what command you want to test.\n Make a `turnt.toml` config file and put `command = \"mycmd {filename}\"` in it to pass each test file as an argument to `mycmd`.\n2. *Create a test.*\n Just write an input file next to your `turnt.toml`.\n We'll call it `foo.t`.\n3. *Take a snapshot.*\n Run `turnt --save foo.t` to execute `mycmd foo.t` and save the standard output into `foo.out`.\n You might want to take a look at this output to make sure it's what you expect.\n Then you check both the input `foo.t` and output `foo.out` into version control.\n4. *Test your work.*\n Now that you have a test in place, keep working.\n Use `turnt *.t` to run all your tests and confirm that the output still matches.\n If there's a mismatch, you can do `turnt --diff` to see the changes.\n (Or if you're confident, try `turnt --save` followed by `git diff`.)\n\nTurnt's philosophy is to minimize the effort it takes to write new tests so you can quickly build up lots of them.\nYou don't write any custom logic to check results; you just record the complete \"golden\" output for each test.\n\nCompared to ordinary unit testing, \"snapshot\" tests incur the mental effort of manually inspecting diffs when things change.\nIn return, it's easier to expand test coverage.\nSnapshots also act as a crude form of documentation because every test is a complete, valid input to your program.\n\n[cram]: https://bitheap.org/cram/\n[lit]: https://llvm.org/docs/CommandGuide/lit.html\n\n\nInstall\n-------\n\nThis is a Python 3 tool.\nInstall it with [pip][]:\n\n $ pip install --user turnt\n\nOr, if you want to work on Turnt, you can install [Flit][], clone this repository, and type this to get a \"live\" installation with a symlink:\n\n $ flit install --symlink --user\n\n[pip]: https://pip.pypa.io/\n[flit]: https://flit.readthedocs.io/\n\n\nConfiguring\n-----------\n\nTurnt looks for a configuration file called `turnt.toml` in any of the ancestor directories of your test file.\nIt can be alongside the test file or in any containing directory.\nIt's a [TOML][] file that looks like something this:\n\n command = \"mycmd {args} < {filename}\"\n return_code = 42\n output.txt = \"result.txt\"\n\n### `command`\n\nSet `command` to a shell command to run on a given test file.\nThis is the only setting that is truly required.\n\nThe command is a [template][str.format]; Turnt will fill in these values:\n\n- `{filename}`: The path to the test file, relative to the working directory.\n- `{base}`: The basename of the test file, with the extension removed.\n- `{args}`: Some extra arguments that the test or user provides.\n (See the `ARGS:` override and the `--args` command-line option below.)\n\nThe working directory for the command is the location of the `turnt.toml` configuration file, if any.\nIf there's no configuration file, then it's the location of the test file itself.\n\n### `return_code`\n\nBy default, Turnt expects the test command to succeed, i.e., exit with status code 0.\nSet `return_code` to a different status if you expect failure.\n\n### `output`\n\nBy default, Turnt captures the standard output stream from your test command.\nIf your command produces other output files \"on the side\" or you want to capture the standard error stream, you can configure the `output` table.\n\n`output` is a mapping from *snapshot extensions* to *collected filenames*.\nFor example, this TOML configuration:\n\n output.txt = \"result.txt\"\n\nmeans that running the command will produce a file called `result.txt`, and we want to save that file in a snapshot called `<test-name>.txt`.\n\nIn place of a filename, use `-` to indicate the command's standard output and `2` to indicate its standard error.\nThe default behaves like this configuration:\n\n output.out = \"-\"\n\nwhich captures stdout and saves it in `<test-name>.out`.\nDefining `output` in `turnt.toml` disables this default behavior; you can include it explicitly if you want it alongside other outputs.\n\n### `binary`\n\nBy default, Turnt looks inside test files for overrides (see below).\nThis won't work if your test inputs are binary (non-text) files (Turnt will warn you and proceed with no overrides).\nSet `binary = true` to suppress this search for overrides altogether.\n\n### `out_dir`\n\nOptionally put expected test outputs in a different directory.\nBy default, this is `.`, i.e., expected output files go \"next to\" test files.\nSet this to a different directory, relative to the test file, to keep them somewhere else.\n\n### `todo`\n\nMark a test as an allowed failure.\nYou might add a test that exposes a bug that you haven't fixed yet, for example, but you want your CI to stay \"green.\"\nSet `todo = true` to allow the test to fail; Turnt will mark it with `# todo` in the TAP output.\n\n\nPer-Test Overrides\n------------------\n\nSometimes you need to alter the setup for a specific test file.\nTurnt looks for some overrides embedded in the test file itself: for example, you might put them in a comment at the top of a test program.\n\nPut these things into your test file to override the configuration:\n\n- `CMD: <command>` overrides `command` from the configuration.\n- `ARGS: <arguments>` adds arguments to a configured command.\n Turnt puts this string in where the command uses `{args}`.\n- `OUT: <ext> <filename>` overrides `output` from the configuration.\n You can specify multiple files this way: one line per file.\n- `RETURN: <code>` overrides the expected exit status.\n- `TODO: <true|false>` can mark a test as a \"todo\" (allowed failure).\n\n\nMultiple Environments\n---------------------\n\nTurnt is mostly about running one command on many input files.\nSometimes, however, you need to run several commands on each file.\nThis can be especially useful for *[differential testing][dt]:*\nwhen you want to check that multiple things behave the same way by checking that they produce the same input when you give them the same input.\n\nYou can create multiple environments in your configuration file under the `envs` table.\nThe table maps environment *names* to sets of configuration options that are just like the [top-level configuration described above](#configuring).\nFor example:\n\n [envs.baseline]\n command = \"interp -g {filename}\"\n\n [envs.optimized]\n command = \"compile -O3 {filename} ; ./a.out\"\n\nEach environment can have the full complement of configuration options described above:\nfor example, `command`, `output`, and `return_code`.\nWhen you run `turnt` on some files, it will run all the environments on all the files.\nThere is also a Boolean `default` flag to turn off an environment by default; that way, you can only use it by asking for it with the `-e` flag (see below).\n\nThis example is a differential testing setup because both environments share the same (default) snapshot file:\na test `foo.t` will look for its stdout snapshot in `foo.out`.\nIf the two environments don't match, at least one will fail.\nIf you want a more standard (non-differential) setup, just set the `output` configuration differently for the two environments, like this:\n\n [envs.interp]\n command = \"interp -g {filename}\"\n output.res = \"-\"\n\n [envs.profile]\n command = \"profile {filename}\"\n output.prof = \"-\"\n\n[dt]: https://en.wikipedia.org/wiki/Differential_testing\n\n\nDirectory Tests\n---------------\n\nA Turnt test is usually just a single input file, but you can also organize multiple related files into a directory.\nUse the directory the same way as you would a single file:\npass its path to the `turnt` command, and the path will appear as the `{filename}` for the configured command.\nSo you might configure your test command like this:\n\n command = \"mycmd {filename}/test.c\"\n\nif you want each test directory to contain a file called `test.c`.\n\nTurnt puts snapshots *inside* the test directory instead of adjacent to it.\nIt names them `out.<extension>` in that directory.\n\nThere are two configuration options just for dealing with directory tests:\n\n- `out_base`.\n The basename for output files in directory tests: by default, `out`.\n- `opts_file`.\n The filename to read inside of a directory test to search for embedded overrides.\n In our above example, you could set this to `test.c` to look for `ARGS:` and friends in that file, or `opts.txt` to look for them in a separate file on the side.\n\n[toml]: https://github.com/toml-lang/toml\n[str.format]: https://docs.python.org/3/library/string.html#formatstrings\n\n\nCommand-Line Interface\n----------------------\n\nThe most common `turnt` command-line options you'll need while running and updating tests are:\n\n- `--save`: Bless the current output from each test as the \"correct\" output, saving it to the output file that you'll want to check into version control.\n- `--diff`: Show diffs between the actual and expected output for each test.\n (The diff goes to stderr while the [TAP][] results remain on stdout.)\n\nYou also might enjoy:\n\n- `--parallel` or `-j`: Run your tests faster using parallel threads.\n\nThese options are useful when working with one specific test file:\n\n- `--verbose` or `-v`: Disable Turnt's default behavior where it will suppress test commands' stderr output. The result is more helpful but harder to read.\n- `--print` or `-p`: Instead of checking test results, just run the command and show the output directly. This can be useful (especially in combination with `-v`) when iterating on a test interactively.\n- `--args` or `-a`: Override the `{args}` string in the test command.\n\nThese options lets you switch between different test environments:\n\n- `--env` or `-e`: Give the name of a configured [environment](#multiple-environments) to run. Use this multiple times to run multiple environments. By default, Turnt runs all the configured environments for every test. Use the special name `default` for the top-level, unnamed test environment.\n- `--config` or `-c`: Look for this config filename instead of the default `turnt.toml`.\n\n\nTAP\n---\n\nTurnt outputs results in the machine-readable [TAP][] format.\nTo make the output more pleasant to read, you can pipe it into a tool like [tap-difflet][], [tap-dot][], or [faucet][]:\n\n $ npm install -g tap-difflet\n $ turnt *.t | tap-difflet\n\n[tap]: http://testanything.org\n[tap-difflet]: https://github.com/namuol/tap-difflet\n[tap-dot]: https://github.com/scottcorgan/tap-dot\n[faucet]: https://github.com/substack/faucet\n\n\nCredits\n-------\n\nTurnt is by [Adrian Sampson][adrian] and [Alexa VanHattum][alexa].\nWe made it to test various research compilers in [Capra][].\nThe license is [MIT][].\n\n[adrian]: https://www.cs.cornell.edu/~asampson/\n[alexa]: https://www.cs.cornell.edu/~avh/\n[capra]: https://capra.cs.cornell.edu\n[mit]: https://opensource.org/licenses/MIT\n",
"bugtrack_url": null,
"license": null,
"summary": "Turnt is a simple expect-style testing tool for command-line",
"version": "1.12.0",
"project_urls": {
"Homepage": "https://github.com/cucapra/turnt"
},
"split_keywords": [],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "2ffefe48f721fc4ae86ec5198ddce1f463e3550d954c287feccd5e6bcc2fb110",
"md5": "e3f92120f6c072a20419d331103a67e9",
"sha256": "dc3fb21df0127ccad3f4efae1cb6147c255c53187080145b9fad889d68daba45"
},
"downloads": -1,
"filename": "turnt-1.12.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e3f92120f6c072a20419d331103a67e9",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 13160,
"upload_time": "2024-07-12T13:51:19",
"upload_time_iso_8601": "2024-07-12T13:51:19.688878Z",
"url": "https://files.pythonhosted.org/packages/2f/fe/fe48f721fc4ae86ec5198ddce1f463e3550d954c287feccd5e6bcc2fb110/turnt-1.12.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "6bff9e3653777373cc13565401685faaa95c941c463e54c8d53abd81e3656ebb",
"md5": "4198c764fa1a60c9a002856eb20e9523",
"sha256": "e0aedca86c0a12b19b67e77155ad3abfc6887eba552c2dbdddddbd413fafb01c"
},
"downloads": -1,
"filename": "turnt-1.12.0.tar.gz",
"has_sig": false,
"md5_digest": "4198c764fa1a60c9a002856eb20e9523",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 19829,
"upload_time": "2024-07-12T13:51:20",
"upload_time_iso_8601": "2024-07-12T13:51:20.931539Z",
"url": "https://files.pythonhosted.org/packages/6b/ff/9e3653777373cc13565401685faaa95c941c463e54c8d53abd81e3656ebb/turnt-1.12.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-07-12 13:51:20",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "cucapra",
"github_project": "turnt",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "turnt"
}
