StatusChecker
=============
.. contents::
:local:
Introduction
------------
StatusChecker is a tool for validating that executed `Robot Framework`_
test cases have expected statuses and log messages. It is mainly useful
for Robot Framework test library developers who want to use Robot
Framework to also test their libraries. StatusChecker 1.3 and newer are
compatible both with Python 2 and Python 3.
StatusChecker project is hosted at GitHub_ and downloads are at
PyPI_.
.. _Robot Framework: http://robotframework.org
.. _GitHub: https://github.com/robotframework/statuschecker
.. _PyPI: https://pypi.python.org/pypi/robotstatuschecker
.. _pip: http://pip-installer.org
Installation instructions
-------------------------
The easiest way to install StatusChecker is by using pip_::
pip install robotstatuschecker
Alternatively you can get the code by cloning the project from
GitHub_ or downloading the source distribution from PyPI_ and
extracting it. After that you can install the tool with::
python setup.py install
Usage
-----
From the command line::
python -m robotstatuschecker infile [outfile]
Programmatically:
.. sourcecode:: python
from robotstatuschecker import process_output
process_output('infile.xml', 'outfile.xml')
If an output file is not given, the input file is edited in place.
Defining expected test status
-----------------------------
By default, all test cases are expected to *PASS* and have no
message. Changing the expected status to *FAIL* is done by having
the word ``FAIL`` (in uppercase) somewhere in the test case
documentation. The expected error message must then follow
the ``FAIL`` marker.
For robotframework version 4 you can also change the expected status
to *SKIP* by adding the word ``SKIP`` in the test case documentation.
Like Fail, the expected skip message must follow the word ``SKIP``.
If a test documentation contains the words ``FAIL`` and ``SKIP``, ``SKIP``
will be ignored and the expected status will be *FAIL*.
If a test is expected to *PASS* with a certain message, the word
``PASS`` must be added to its documentation explicitly and the
expected message given after that.
If a message check should happen in test setup or teardown, that check
must be prefixed with ``SETUP`` or ``TEARDOWN`` word.
The expected message can also be specified as a regular expression by
prefixing it with ``REGEXP:``. The specified regular expression
must match the error message fully. Having spaces between the status,
the message and the possible regular expression prefix is optional.
An alternative to using regular expressions is using glob patterns where
``*`` matches anything (including newline) and ``?`` matches any single
character. This is can be accomplished by starting the expected message
with ``GLOB:``.
Finally, it is possible to test that the message starts with something
by prefixing the expected message with ``STARTS:``.
The following examples illustrate different ways to define test
statuses and messages:
.. sourcecode:: robotframework
*** Test Cases ***
Simple failure
[Documentation] FAIL Expected error message
Steps
Check in test setup is done by SETUP marker
[Documentation] LOG SETUP This first log message in test setup
[Setup] Test specific setup
Steps
Exclude documentation before marker
[Documentation] This text is ignored FAIL Expected error message
Steps
Regexp example
[Documentation] FAIL REGEXP: (IOError|OSError): .*
Steps
Glob example
[Documentation] FAIL GLOB: ??Error: *
Steps
Start example
[Documentation] FAIL STARTS: IOError:
Steps
Passing without message
Steps
Passing with message
[Documentation] PASS Expected message
Steps
Defining expected log messages
------------------------------
The expected keyword log messages can also be defined in the test case
documentation using a syntax such as::
LOG x.y:z LEVEL Actual message
The part before the colon specifies the keyword to check. For
example, ``1`` means first keyword, ``1.2`` is the second child
keyword of the first keyword, and so on.
The part after the colon species the message. For example, ``1:2``
means the second message of the first keyword and ``1.2:3`` is
the third message of the second child keyword of the first keyword.
The message index is optional and defaults to ``1``.
The message index also supports wildcard ``*``. For example ``1:*``
matches any message of the first keyword.
Message level is specified before the actual message, and it can be
any of the valid log levels in capital letters. If the level is not
given it defaults to ``INFO``. Starting from 1.4 release also
``ERROR`` level is supported. The message level also supports wildcard
``ANY`` which will match all log levels.
Possible leading and trailing whitespace is ignored both in the expected
and in the actual log message.
This syntax can be used multiple times to test multiple messages. It
also works together with specifying the expected error message with
``FAIL``, but it that case ``FAIL`` and the expected error must
be first.
It is also possible to give the message as a regular expression or glob
pattern or to give just the start of the message. This is accomplished
by prefixing the message with ``REGEXP:``, ``GLOB:`` or ``STARTS:``,
respectively, exactly like when `defining expected test status`_.
Finally, to check that a keyword does not have a certain message, it
is possible to use ``NONE`` in the place of the message.
.. sourcecode:: robotframework
*** Test cases ***
Simple example
[Documentation] LOG 1 Hello, world!
Steps
Nested keywords
[Documentation] LOG 2.1 1st child of 2nd kw
Steps
Message index
[Documentation] LOG 2:2 2nd msg of 2nd kw
Steps
Nested and index
[Documentation] LOG 3.1:2 2nd msg of 3rd kw's 1st child
Steps
Log levels
[Documentation] LOG 2 DEBUG Debug-level message
... LOG 1.2:3 WARN Warning
Steps
Multiple messages
[Documentation] LOG 1 First tested message
... LOG 1.2 Second tested message
... LOG 2.2.1 DEBUG Third tested message
Steps
Status and log
[Documentation] FAIL Expected error message
... LOG 1.2 Expected log message
Steps
Regexp message
[Documentation] LOG 1 REGEXP: (Hello|Hi) world!
Steps
Glob message
[Documentation] LOG 1 GLOB: * world!
Steps
Start of the message
[Documentation] LOG 1 STARTS: Hello w
Steps
No message
[Documentation] LOG 1:1 Test that we have only 1 msg
... LOG 1:2 NONE
Steps
Count Messages
[Documentation] LOG 4 COUNT: 2 # Fourth keyword should have excatly 2 messages.
Steps
Raw data
{
"_id": null,
"home_page": "https://github.com/robotframework/statuschecker",
"name": "robotstatuschecker",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6,<4.0",
"maintainer_email": "",
"keywords": "robotframework testing testautomation atdd",
"author": "Robot Framework Developers",
"author_email": "robotframework@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/6d/53/8cedf0cf652c31abed7c2a16e0aad40387839dcfbd0b69ecef9c005f346a/robotstatuschecker-3.0.1.tar.gz",
"platform": "any",
"description": "StatusChecker\n=============\n\n.. contents::\n :local:\n\nIntroduction\n------------\n\nStatusChecker is a tool for validating that executed `Robot Framework`_\ntest cases have expected statuses and log messages. It is mainly useful\nfor Robot Framework test library developers who want to use Robot\nFramework to also test their libraries. StatusChecker 1.3 and newer are\ncompatible both with Python 2 and Python 3.\n\nStatusChecker project is hosted at GitHub_ and downloads are at\nPyPI_.\n\n.. _Robot Framework: http://robotframework.org\n.. _GitHub: https://github.com/robotframework/statuschecker\n.. _PyPI: https://pypi.python.org/pypi/robotstatuschecker\n.. _pip: http://pip-installer.org\n\nInstallation instructions\n-------------------------\n\nThe easiest way to install StatusChecker is by using pip_::\n\n pip install robotstatuschecker\n\nAlternatively you can get the code by cloning the project from\nGitHub_ or downloading the source distribution from PyPI_ and\nextracting it. After that you can install the tool with::\n\n python setup.py install\n\nUsage\n-----\n\nFrom the command line::\n\n python -m robotstatuschecker infile [outfile]\n\nProgrammatically:\n\n.. sourcecode:: python\n\n from robotstatuschecker import process_output\n\n process_output('infile.xml', 'outfile.xml')\n\nIf an output file is not given, the input file is edited in place.\n\nDefining expected test status\n-----------------------------\n\nBy default, all test cases are expected to *PASS* and have no\nmessage. Changing the expected status to *FAIL* is done by having\nthe word ``FAIL`` (in uppercase) somewhere in the test case\ndocumentation. The expected error message must then follow\nthe ``FAIL`` marker.\n\nFor robotframework version 4 you can also change the expected status\nto *SKIP* by adding the word ``SKIP`` in the test case documentation.\nLike Fail, the expected skip message must follow the word ``SKIP``.\nIf a test documentation contains the words ``FAIL`` and ``SKIP``, ``SKIP``\nwill be ignored and the expected status will be *FAIL*.\n\nIf a test is expected to *PASS* with a certain message, the word\n``PASS`` must be added to its documentation explicitly and the\nexpected message given after that.\n\nIf a message check should happen in test setup or teardown, that check\nmust be prefixed with ``SETUP`` or ``TEARDOWN`` word.\n\nThe expected message can also be specified as a regular expression by\nprefixing it with ``REGEXP:``. The specified regular expression\nmust match the error message fully. Having spaces between the status,\nthe message and the possible regular expression prefix is optional.\n\nAn alternative to using regular expressions is using glob patterns where\n``*`` matches anything (including newline) and ``?`` matches any single\ncharacter. This is can be accomplished by starting the expected message\nwith ``GLOB:``.\n\nFinally, it is possible to test that the message starts with something\nby prefixing the expected message with ``STARTS:``.\n\nThe following examples illustrate different ways to define test\nstatuses and messages:\n\n.. sourcecode:: robotframework\n\n *** Test Cases ***\n Simple failure\n [Documentation] FAIL Expected error message\n Steps\n\n Check in test setup is done by SETUP marker\n [Documentation] LOG SETUP This first log message in test setup\n [Setup] Test specific setup\n Steps\n\n Exclude documentation before marker\n [Documentation] This text is ignored FAIL Expected error message\n Steps\n\n Regexp example\n [Documentation] FAIL REGEXP: (IOError|OSError): .*\n Steps\n\n Glob example\n [Documentation] FAIL GLOB: ??Error: *\n Steps\n\n Start example\n [Documentation] FAIL STARTS: IOError:\n Steps\n\n Passing without message\n Steps\n\n Passing with message\n [Documentation] PASS Expected message\n Steps\n\nDefining expected log messages\n------------------------------\n\nThe expected keyword log messages can also be defined in the test case\ndocumentation using a syntax such as::\n\n LOG x.y:z LEVEL Actual message\n\nThe part before the colon specifies the keyword to check. For\nexample, ``1`` means first keyword, ``1.2`` is the second child\nkeyword of the first keyword, and so on.\n\nThe part after the colon species the message. For example, ``1:2``\nmeans the second message of the first keyword and ``1.2:3`` is\nthe third message of the second child keyword of the first keyword.\nThe message index is optional and defaults to ``1``.\nThe message index also supports wildcard ``*``. For example ``1:*``\nmatches any message of the first keyword.\n\nMessage level is specified before the actual message, and it can be\nany of the valid log levels in capital letters. If the level is not\ngiven it defaults to ``INFO``. Starting from 1.4 release also\n``ERROR`` level is supported. The message level also supports wildcard\n``ANY`` which will match all log levels.\n\nPossible leading and trailing whitespace is ignored both in the expected\nand in the actual log message.\n\nThis syntax can be used multiple times to test multiple messages. It\nalso works together with specifying the expected error message with\n``FAIL``, but it that case ``FAIL`` and the expected error must\nbe first.\n\nIt is also possible to give the message as a regular expression or glob\npattern or to give just the start of the message. This is accomplished\nby prefixing the message with ``REGEXP:``, ``GLOB:`` or ``STARTS:``,\nrespectively, exactly like when `defining expected test status`_.\n\nFinally, to check that a keyword does not have a certain message, it\nis possible to use ``NONE`` in the place of the message.\n\n.. sourcecode:: robotframework\n\n *** Test cases ***\n Simple example\n [Documentation] LOG 1 Hello, world!\n Steps\n\n Nested keywords\n [Documentation] LOG 2.1 1st child of 2nd kw\n Steps\n\n Message index\n [Documentation] LOG 2:2 2nd msg of 2nd kw\n Steps\n\n Nested and index\n [Documentation] LOG 3.1:2 2nd msg of 3rd kw's 1st child\n Steps\n\n Log levels\n [Documentation] LOG 2 DEBUG Debug-level message\n ... LOG 1.2:3 WARN Warning\n Steps\n\n Multiple messages\n [Documentation] LOG 1 First tested message\n ... LOG 1.2 Second tested message\n ... LOG 2.2.1 DEBUG Third tested message\n Steps\n\n Status and log\n [Documentation] FAIL Expected error message\n ... LOG 1.2 Expected log message\n Steps\n\n Regexp message\n [Documentation] LOG 1 REGEXP: (Hello|Hi) world!\n Steps\n\n Glob message\n [Documentation] LOG 1 GLOB: * world!\n Steps\n\n Start of the message\n [Documentation] LOG 1 STARTS: Hello w\n Steps\n\n No message\n [Documentation] LOG 1:1 Test that we have only 1 msg\n ... LOG 1:2 NONE\n Steps\n\n Count Messages\n [Documentation] LOG 4 COUNT: 2 # Fourth keyword should have excatly 2 messages.\n Steps\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "A tool for checking that Robot Framework test cases have expected statuses and log messages.",
"version": "3.0.1",
"project_urls": {
"Download": "https://pypi.python.org/pypi/robotstatuschecker",
"Homepage": "https://github.com/robotframework/statuschecker"
},
"split_keywords": [
"robotframework",
"testing",
"testautomation",
"atdd"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a3acd16430ce9077fe1351c68ff4cd3591a72eb93261ebbcdb9185d4210ea5b4",
"md5": "fc4fc2babd93f0fb7d78ce19eba57fd6",
"sha256": "eeebc1bde9638076d97a47e69832ca6fbfc0c1d5502993172fe1203f506beacd"
},
"downloads": -1,
"filename": "robotstatuschecker-3.0.1-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "fc4fc2babd93f0fb7d78ce19eba57fd6",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.6,<4.0",
"size": 7314,
"upload_time": "2023-06-12T20:50:46",
"upload_time_iso_8601": "2023-06-12T20:50:46.766103Z",
"url": "https://files.pythonhosted.org/packages/a3/ac/d16430ce9077fe1351c68ff4cd3591a72eb93261ebbcdb9185d4210ea5b4/robotstatuschecker-3.0.1-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6d538cedf0cf652c31abed7c2a16e0aad40387839dcfbd0b69ecef9c005f346a",
"md5": "2b919f2f58ecedae8ab7614c8cbc27b2",
"sha256": "4a761a0d59c0a764d479364855f045aba371ecf94a69a43e6f378bd0592a9112"
},
"downloads": -1,
"filename": "robotstatuschecker-3.0.1.tar.gz",
"has_sig": false,
"md5_digest": "2b919f2f58ecedae8ab7614c8cbc27b2",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6,<4.0",
"size": 7613,
"upload_time": "2023-06-12T20:50:48",
"upload_time_iso_8601": "2023-06-12T20:50:48.949241Z",
"url": "https://files.pythonhosted.org/packages/6d/53/8cedf0cf652c31abed7c2a16e0aad40387839dcfbd0b69ecef9c005f346a/robotstatuschecker-3.0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-06-12 20:50:48",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "robotframework",
"github_project": "statuschecker",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "robotstatuschecker"
}
JSON
