======================
Cram: It's test time
======================
Cram is a functional testing framework for command line applications.
Cram tests look like snippets of interactive shell sessions. Cram runs
each command and compares the command output in the test with the
command's actual output.
Here's a snippet from `Cram's own test suite`_::
The $PYTHON environment variable should be set when running this test
from Python.
$ [ -n "$PYTHON" ] || PYTHON="`which python`"
$ [ -n "$PYTHONPATH" ] || PYTHONPATH="$TESTDIR/.." && export PYTHONPATH
$ if [ -n "$COVERAGE" ]; then
> coverage erase
> alias cram="`which coverage` run --branch -a $TESTDIR/../scripts/cram"
> else
> alias cram="$PYTHON $TESTDIR/../scripts/cram"
> fi
$ command -v md5 > /dev/null || alias md5=md5sum
Usage:
$ cram -h
[Uu]sage: cram \[OPTIONS\] TESTS\.\.\. (re)
[Oo]ptions: (re)
-h, --help show this help message and exit
-V, --version show version information and exit
-q, --quiet don't print diffs
-v, --verbose show filenames and test status
-i, --interactive interactively merge changed test output
-d, --debug write script output directly to the terminal
-y, --yes answer yes to all questions
-n, --no answer no to all questions
-E, --preserve-env don't reset common environment variables
--keep-tmpdir keep temporary directories
--shell=PATH shell to use for running tests (default: /bin/sh)
--shell-opts=OPTS arguments to invoke shell with
--indent=NUM number of spaces to use for indentation (default: 2)
--xunit-file=PATH path to write xUnit XML output
The format in a nutshell:
* Cram tests use the ``.t`` file extension.
* Lines beginning with two spaces, a dollar sign, and a space are run
in the shell.
* Lines beginning with two spaces, a greater than sign, and a space
allow multi-line commands.
* All other lines beginning with two spaces are considered command
output.
* Output lines ending with a space and the keyword ``(re)`` are
matched as `Perl-compatible regular expressions`_.
* Lines ending with a space and the keyword ``(glob)`` are matched
with a glob-like syntax. The only special characters supported are
``*`` and ``?``. Both characters can be escaped using ``\``, and the
backslash can be escaped itself.
* Output lines ending with either of the above keywords are always
first matched literally with actual command output.
* Lines ending with a space and the keyword ``(no-eol)`` will match
actual output that doesn't end in a newline.
* Actual output lines containing unprintable characters are escaped
and suffixed with a space and the keyword ``(esc)``. Lines matching
unprintable output must also contain the keyword.
* Anything else is a comment.
.. _Cram's own test suite: https://bitbucket.org/brodie/cram/src/0.6/tests/cram.t
.. _Perl-compatible regular expressions: https://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expressions
Download
--------
* `cram-0.7.tar.gz`_ (32 KB, requires Python 2.4-2.7 or Python 3.1 or newer)
.. _cram-0.7.tar.gz: https://bitheap.org/cram/cram-0.7.tar.gz
Installation
------------
Install Cram using make::
$ wget https://bitheap.org/cram/cram-0.7.tar.gz
$ tar zxvf cram-0.7.tar.gz
$ cd cram-0.7
$ make install
Usage
-----
Cram will print a dot for each passing test. If a test fails, a
`unified context diff`_ is printed showing the test's expected output
and the actual output. Skipped tests (empty tests and tests that exit
with return code ``80``) are marked with ``s`` instead of a dot.
For example, if we run Cram on `its own example tests`_::
.s.!
--- examples/fail.t
+++ examples/fail.t.err
@@ -3,21 +3,22 @@
$ echo 1
1
$ echo 1
- 2
+ 1
$ echo 1
1
Invalid regex:
$ echo 1
- +++ (re)
+ 1
Offset regular expression:
$ printf 'foo\nbar\nbaz\n\n1\nA\n@\n'
foo
+ bar
baz
\d (re)
[A-Z] (re)
- #
+ @
s.
# Ran 6 tests, 2 skipped, 1 failed.
Cram will also write the test with its actual output to
``examples/fail.t.err``, allowing you to use other diff tools. This
file is automatically removed the next time the test passes.
When you're first writing a test, you might just write the commands
and run the test to see what happens. If you run Cram with ``-i`` or
``--interactive``, you'll be prompted to merge the actual output back
into the test. This makes it easy to quickly prototype new tests.
You can specify a default set of options by creating a ``.cramrc``
file. For example::
[cram]
verbose = True
indent = 4
Is the same as invoking Cram with ``--verbose`` and ``--indent=4``.
To change what configuration file Cram loads, you can set the
``CRAMRC`` environment variable. You can also specify command line
options in the ``CRAM`` environment variable.
Note that the following environment variables are reset before tests
are run:
* ``TMPDIR``, ``TEMP``, and ``TMP`` are set to the test runner's
``tmp`` directory.
* ``LANG``, ``LC_ALL``, and ``LANGUAGE`` are set to ``C``.
* ``TZ`` is set to ``GMT``.
* ``COLUMNS`` is set to ``80``. (Note: When using ``--shell=zsh``,
this cannot be reset. It will reflect the actual terminal's width.)
* ``CDPATH`` and ``GREP_OPTIONS`` are set to an empty string.
Cram also provides the following environment variables to tests:
* ``CRAMTMP``, set to the test runner's temporary directory.
* ``TESTDIR``, set to the directory containing the test file.
* ``TESTFILE``, set to the basename of the current test file.
* ``TESTSHELL``, set to the value specified by ``--shell``.
Also note that care should be taken with commands that close the test
shell's ``stdin``. For example, if you're trying to invoke ``ssh`` in
a test, try adding the ``-n`` option to prevent it from closing
``stdin``. Similarly, if you invoke a daemon process that inherits
``stdout`` and fails to close it, it may cause Cram to hang while
waiting for the test shell's ``stdout`` to be fully closed.
.. _unified context diff: https://en.wikipedia.org/wiki/Diff#Unified_format
.. _its own example tests: https://bitbucket.org/brodie/cram/src/default/examples/
Development
-----------
Download the official development repository using Mercurial_::
hg clone https://bitbucket.org/brodie/cram
Or Git_::
git clone https://github.com/brodie/cram.git
Test Cram using Cram::
pip install -r requirements.txt
make test
Visit Bitbucket_ or GitHub_ if you'd like to fork the project, watch
for new changes, or report issues.
.. _Mercurial: http://mercurial.selenic.com/
.. _Git: http://git-scm.com/
.. _coverage.py: http://nedbatchelder.com/code/coverage/
.. _Bitbucket: https://bitbucket.org/brodie/cram
.. _GitHub: https://github.com/brodie/cram
Raw data
{
"_id": null,
"home_page": "https://bitheap.org/cram/",
"name": "cram",
"maintainer": "",
"docs_url": null,
"requires_python": null,
"maintainer_email": "",
"keywords": "automatic functional test framework",
"author": "Brodie Rao",
"author_email": "brodie@bitheap.org",
"download_url": "https://files.pythonhosted.org/packages/38/85/5a8a3397b2ccb2ffa3ba871f76a4d72c16531e43d0e58fc89a0f2983adbd/cram-0.7.tar.gz",
"platform": "UNKNOWN",
"description": "======================\r\n Cram: It's test time\r\n======================\r\n\r\nCram is a functional testing framework for command line applications.\r\nCram tests look like snippets of interactive shell sessions. Cram runs\r\neach command and compares the command output in the test with the\r\ncommand's actual output.\r\n\r\nHere's a snippet from `Cram's own test suite`_::\r\n\r\n The $PYTHON environment variable should be set when running this test\r\n from Python.\r\n\r\n $ [ -n \"$PYTHON\" ] || PYTHON=\"`which python`\"\r\n $ [ -n \"$PYTHONPATH\" ] || PYTHONPATH=\"$TESTDIR/..\" && export PYTHONPATH\r\n $ if [ -n \"$COVERAGE\" ]; then\r\n > coverage erase\r\n > alias cram=\"`which coverage` run --branch -a $TESTDIR/../scripts/cram\"\r\n > else\r\n > alias cram=\"$PYTHON $TESTDIR/../scripts/cram\"\r\n > fi\r\n $ command -v md5 > /dev/null || alias md5=md5sum\r\n\r\n Usage:\r\n\r\n $ cram -h\r\n [Uu]sage: cram \\[OPTIONS\\] TESTS\\.\\.\\. (re)\r\n\r\n [Oo]ptions: (re)\r\n -h, --help show this help message and exit\r\n -V, --version show version information and exit\r\n -q, --quiet don't print diffs\r\n -v, --verbose show filenames and test status\r\n -i, --interactive interactively merge changed test output\r\n -d, --debug write script output directly to the terminal\r\n -y, --yes answer yes to all questions\r\n -n, --no answer no to all questions\r\n -E, --preserve-env don't reset common environment variables\r\n --keep-tmpdir keep temporary directories\r\n --shell=PATH shell to use for running tests (default: /bin/sh)\r\n --shell-opts=OPTS arguments to invoke shell with\r\n --indent=NUM number of spaces to use for indentation (default: 2)\r\n --xunit-file=PATH path to write xUnit XML output\r\n\r\nThe format in a nutshell:\r\n\r\n* Cram tests use the ``.t`` file extension.\r\n\r\n* Lines beginning with two spaces, a dollar sign, and a space are run\r\n in the shell.\r\n\r\n* Lines beginning with two spaces, a greater than sign, and a space\r\n allow multi-line commands.\r\n\r\n* All other lines beginning with two spaces are considered command\r\n output.\r\n\r\n* Output lines ending with a space and the keyword ``(re)`` are\r\n matched as `Perl-compatible regular expressions`_.\r\n\r\n* Lines ending with a space and the keyword ``(glob)`` are matched\r\n with a glob-like syntax. The only special characters supported are\r\n ``*`` and ``?``. Both characters can be escaped using ``\\``, and the\r\n backslash can be escaped itself.\r\n\r\n* Output lines ending with either of the above keywords are always\r\n first matched literally with actual command output.\r\n\r\n* Lines ending with a space and the keyword ``(no-eol)`` will match\r\n actual output that doesn't end in a newline.\r\n\r\n* Actual output lines containing unprintable characters are escaped\r\n and suffixed with a space and the keyword ``(esc)``. Lines matching\r\n unprintable output must also contain the keyword.\r\n\r\n* Anything else is a comment.\r\n\r\n.. _Cram's own test suite: https://bitbucket.org/brodie/cram/src/0.6/tests/cram.t\r\n.. _Perl-compatible regular expressions: https://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expressions\r\n\r\n\r\nDownload\r\n--------\r\n\r\n* `cram-0.7.tar.gz`_ (32 KB, requires Python 2.4-2.7 or Python 3.1 or newer)\r\n\r\n.. _cram-0.7.tar.gz: https://bitheap.org/cram/cram-0.7.tar.gz\r\n\r\n\r\nInstallation\r\n------------\r\n\r\nInstall Cram using make::\r\n\r\n $ wget https://bitheap.org/cram/cram-0.7.tar.gz\r\n $ tar zxvf cram-0.7.tar.gz\r\n $ cd cram-0.7\r\n $ make install\r\n\r\n\r\nUsage\r\n-----\r\n\r\nCram will print a dot for each passing test. If a test fails, a\r\n`unified context diff`_ is printed showing the test's expected output\r\nand the actual output. Skipped tests (empty tests and tests that exit\r\nwith return code ``80``) are marked with ``s`` instead of a dot.\r\n\r\nFor example, if we run Cram on `its own example tests`_::\r\n\r\n .s.!\r\n --- examples/fail.t\r\n +++ examples/fail.t.err\r\n @@ -3,21 +3,22 @@\r\n $ echo 1\r\n 1\r\n $ echo 1\r\n - 2\r\n + 1\r\n $ echo 1\r\n 1\r\n\r\n Invalid regex:\r\n\r\n $ echo 1\r\n - +++ (re)\r\n + 1\r\n\r\n Offset regular expression:\r\n\r\n $ printf 'foo\\nbar\\nbaz\\n\\n1\\nA\\n@\\n'\r\n foo\r\n + bar\r\n baz\r\n\r\n \\d (re)\r\n [A-Z] (re)\r\n - #\r\n + @\r\n s.\r\n # Ran 6 tests, 2 skipped, 1 failed.\r\n\r\nCram will also write the test with its actual output to\r\n``examples/fail.t.err``, allowing you to use other diff tools. This\r\nfile is automatically removed the next time the test passes.\r\n\r\nWhen you're first writing a test, you might just write the commands\r\nand run the test to see what happens. If you run Cram with ``-i`` or\r\n``--interactive``, you'll be prompted to merge the actual output back\r\ninto the test. This makes it easy to quickly prototype new tests.\r\n\r\nYou can specify a default set of options by creating a ``.cramrc``\r\nfile. For example::\r\n\r\n [cram]\r\n verbose = True\r\n indent = 4\r\n\r\nIs the same as invoking Cram with ``--verbose`` and ``--indent=4``.\r\n\r\nTo change what configuration file Cram loads, you can set the\r\n``CRAMRC`` environment variable. You can also specify command line\r\noptions in the ``CRAM`` environment variable.\r\n\r\nNote that the following environment variables are reset before tests\r\nare run:\r\n\r\n* ``TMPDIR``, ``TEMP``, and ``TMP`` are set to the test runner's\r\n ``tmp`` directory.\r\n\r\n* ``LANG``, ``LC_ALL``, and ``LANGUAGE`` are set to ``C``.\r\n\r\n* ``TZ`` is set to ``GMT``.\r\n\r\n* ``COLUMNS`` is set to ``80``. (Note: When using ``--shell=zsh``,\r\n this cannot be reset. It will reflect the actual terminal's width.)\r\n\r\n* ``CDPATH`` and ``GREP_OPTIONS`` are set to an empty string.\r\n\r\nCram also provides the following environment variables to tests:\r\n\r\n* ``CRAMTMP``, set to the test runner's temporary directory.\r\n\r\n* ``TESTDIR``, set to the directory containing the test file.\r\n\r\n* ``TESTFILE``, set to the basename of the current test file.\r\n\r\n* ``TESTSHELL``, set to the value specified by ``--shell``.\r\n\r\nAlso note that care should be taken with commands that close the test\r\nshell's ``stdin``. For example, if you're trying to invoke ``ssh`` in\r\na test, try adding the ``-n`` option to prevent it from closing\r\n``stdin``. Similarly, if you invoke a daemon process that inherits\r\n``stdout`` and fails to close it, it may cause Cram to hang while\r\nwaiting for the test shell's ``stdout`` to be fully closed.\r\n\r\n.. _unified context diff: https://en.wikipedia.org/wiki/Diff#Unified_format\r\n.. _its own example tests: https://bitbucket.org/brodie/cram/src/default/examples/\r\n\r\n\r\nDevelopment\r\n-----------\r\n\r\nDownload the official development repository using Mercurial_::\r\n\r\n hg clone https://bitbucket.org/brodie/cram\r\n\r\nOr Git_::\r\n\r\n git clone https://github.com/brodie/cram.git\r\n\r\nTest Cram using Cram::\r\n\r\n pip install -r requirements.txt\r\n make test\r\n\r\nVisit Bitbucket_ or GitHub_ if you'd like to fork the project, watch\r\nfor new changes, or report issues.\r\n\r\n.. _Mercurial: http://mercurial.selenic.com/\r\n.. _Git: http://git-scm.com/\r\n.. _coverage.py: http://nedbatchelder.com/code/coverage/\r\n.. _Bitbucket: https://bitbucket.org/brodie/cram\r\n.. _GitHub: https://github.com/brodie/cram",
"bugtrack_url": null,
"license": "GNU GPLv2 or any later version",
"summary": "A simple testing framework for command line applications",
"version": "0.7",
"project_urls": {
"Download": "https://bitheap.org/cram/cram-0.7.tar.gz",
"Homepage": "https://bitheap.org/cram/"
},
"split_keywords": [
"automatic",
"functional",
"test",
"framework"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a209c119252f12b35c21b030c01d38d292936660c167e725ce07517fae2b01a0",
"md5": "cce1d748831f0b46a462bcb7000a0a5f",
"sha256": "008e4e8b4d325cf040964b5f62460535b004a7bc816d54f8527a4d299edfe4a3"
},
"downloads": -1,
"filename": "cram-0.7-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "cce1d748831f0b46a462bcb7000a0a5f",
"packagetype": "bdist_wheel",
"python_version": "3.5",
"requires_python": null,
"size": 22141,
"upload_time": "2016-02-24T10:29:55",
"upload_time_iso_8601": "2016-02-24T10:29:55.022847Z",
"url": "https://files.pythonhosted.org/packages/a2/09/c119252f12b35c21b030c01d38d292936660c167e725ce07517fae2b01a0/cram-0.7-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "38855a8a3397b2ccb2ffa3ba871f76a4d72c16531e43d0e58fc89a0f2983adbd",
"md5": "2ea37ada5190526b9bcaac5e4099221c",
"sha256": "7da7445af2ce15b90aad5ec4792f857cef5786d71f14377e9eb994d8b8337f2f"
},
"downloads": -1,
"filename": "cram-0.7.tar.gz",
"has_sig": false,
"md5_digest": "2ea37ada5190526b9bcaac5e4099221c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 33527,
"upload_time": "2016-02-24T10:29:44",
"upload_time_iso_8601": "2016-02-24T10:29:44.838806Z",
"url": "https://files.pythonhosted.org/packages/38/85/5a8a3397b2ccb2ffa3ba871f76a4d72c16531e43d0e58fc89a0f2983adbd/cram-0.7.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2016-02-24 10:29:44",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "cram"
}
JSON