# robotframework-stopwatch
Robot Framework listener extension for agile perfomance testing.
Evaluate test case execution against its statistical average run time & user-configured timedelta in.
## Why?
While Robot Framework does provide an ability to setup time limits (e.g. timeouts) as a means of failing tests/suites depending on their time of execution, such approach is somewhat set in stone. Meaning that you have to manually inspect runtimes of your tests and decide upon much needed time limits. When you have single master suite, or your testing production is not quite large - this works alright. But when your number of tests is 300+, managing each timeout individually is rather straining. Add to that multiple layers of configurations such as different environments, testing tools, or downright infrastructure inconsistencies that do occur and you end up quite strained with the idea of reliable perfomance testing. **This extension is an attempt to provide a reliable tool for flexible, integrated perfomance testing.**
## Overview
As mentioned above, *robotframework-stopwatch is not quite a library, but extension of built-in Robot Framework Listener.*
When first integrated, the ``StopwatchListener`` will initialise an empty logfile (by default: ``./stopwatch_log.json``) to store required run data for every individual test. After first run, taking into consideration the test succeeded, average runtime for this test will be set to this run time of execution. Afterwards, each succesful test runtime will be used to update the last average runtime value for the test. The user-preconfigured ``timedelta`` is then used to evaluate all future test runs status. Failed tests, whether due to timedelta exceeded or natural test fails, have no effect upon test average runtime. Extension also provides a log data divider ``environment`` to catalog same tests based on different setups.
When test with established average runtime fails due to exceeded time limit, test is marked as failed with a following message output:
```
Stopwatch! Test runtime exceeded the acceptable time limit.
```
## Installation
```shell
pip install robotframework-stopwatch
```
## Usage
**robotframework-stopwatch** relies on a single base class ``StopwatchListener``, that comes with following attributes:
- ``logfile (str)``: The path to the log file. By default: ``./stopwatch_log.json``
- ``environment (str, optional)`` The marker/divider used to separate test run data across different configurations.
- ``timedelta (str, optional)``: The time delta value used for evaluating test runtimes.
Defaults to ``"10s"``. Time formats include: ``"5s"``, ``"2 min"``, ``"1 hour"``.
There are two ways to implement robotframework-stopwatch.
#### Listener approach
When custom listener is already in-use, *or robotframework-stopwatch is needed on a global level* (every suite, every test), create or modify the listener:
```python
from robot.stopwatch import StopwatchListener
class CustomListener(StopwatchListener):
def __init__(self, logfile, environment, timedelta):
```
This won't interfere with your custon listener functionality.
#### Command-line approach
When there is a need to set separate timedelta for different suites/tests, or setting up a listener is not required, it is possible to pass ``StopwatchListener`` and its command line arguments directly from terminal:
```shell
robot --listener robot.stopwatch.StopwatchListener:"logfile":"environment":"timedelta" some_test.robot
```
### Concurecy
As of now, robotframework-stopwatch supports concurent read/write, making it possible to share a logfile data among multiple agents, instances, test runners. To incorporate concurent logging do rely on the most suitable tactics of shared file storage for your organisation (e.g. Azure Shared Storage).
## Plans
1. Interface for setting up test run data log in a remote database.
2. Robust ``timedelta`` time formats.
3. Robust ``logfile`` handling.
Raw data
{
"_id": null,
"home_page": null,
"name": "robotframework-stopwatch",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "robotframework, listener, robot framework, automation, performance testing",
"author": null,
"author_email": "extraorange <extraorangeio@pm.me>",
"download_url": "https://files.pythonhosted.org/packages/00/ff/258ee7243c4e5bee82f1f93b94ee42cdacee6fcbf53a83a8210f75f25e5f/robotframework_stopwatch-0.2.10.tar.gz",
"platform": null,
"description": "# robotframework-stopwatch\n\n\nRobot Framework listener extension for agile perfomance testing.\n\nEvaluate test case execution against its statistical average run time & user-configured timedelta in.\n\n## Why?\nWhile Robot Framework does provide an ability to setup time limits (e.g. timeouts) as a means of failing tests/suites depending on their time of execution, such approach is somewhat set in stone. Meaning that you have to manually inspect runtimes of your tests and decide upon much needed time limits. When you have single master suite, or your testing production is not quite large - this works alright. But when your number of tests is 300+, managing each timeout individually is rather straining. Add to that multiple layers of configurations such as different environments, testing tools, or downright infrastructure inconsistencies that do occur and you end up quite strained with the idea of reliable perfomance testing. **This extension is an attempt to provide a reliable tool for flexible, integrated perfomance testing.**\n\n## Overview\nAs mentioned above, *robotframework-stopwatch is not quite a library, but extension of built-in Robot Framework Listener.*\n\nWhen first integrated, the ``StopwatchListener`` will initialise an empty logfile (by default: ``./stopwatch_log.json``) to store required run data for every individual test. After first run, taking into consideration the test succeeded, average runtime for this test will be set to this run time of execution. Afterwards, each succesful test runtime will be used to update the last average runtime value for the test. The user-preconfigured ``timedelta`` is then used to evaluate all future test runs status. Failed tests, whether due to timedelta exceeded or natural test fails, have no effect upon test average runtime. Extension also provides a log data divider ``environment`` to catalog same tests based on different setups.\n\nWhen test with established average runtime fails due to exceeded time limit, test is marked as failed with a following message output:\n\n```\nStopwatch! Test runtime exceeded the acceptable time limit.\n```\n\n## Installation\n\n```shell\npip install robotframework-stopwatch\n```\n\n## Usage\n**robotframework-stopwatch** relies on a single base class ``StopwatchListener``, that comes with following attributes:\n\n- ``logfile (str)``: The path to the log file. By default: ``./stopwatch_log.json``\n\n- ``environment (str, optional)`` The marker/divider used to separate test run data across different configurations.\n\n- ``timedelta (str, optional)``: The time delta value used for evaluating test runtimes.\n Defaults to ``\"10s\"``. Time formats include: ``\"5s\"``, ``\"2 min\"``, ``\"1 hour\"``.\n\nThere are two ways to implement robotframework-stopwatch.\n\n#### Listener approach\nWhen custom listener is already in-use, *or robotframework-stopwatch is needed on a global level* (every suite, every test), create or modify the listener:\n\n```python\nfrom robot.stopwatch import StopwatchListener\n\nclass CustomListener(StopwatchListener):\n def __init__(self, logfile, environment, timedelta):\n```\n\nThis won't interfere with your custon listener functionality.\n\n#### Command-line approach\nWhen there is a need to set separate timedelta for different suites/tests, or setting up a listener is not required, it is possible to pass ``StopwatchListener`` and its command line arguments directly from terminal:\n\n```shell\nrobot --listener robot.stopwatch.StopwatchListener:\"logfile\":\"environment\":\"timedelta\" some_test.robot\n```\n### Concurecy\n\nAs of now, robotframework-stopwatch supports concurent read/write, making it possible to share a logfile data among multiple agents, instances, test runners. To incorporate concurent logging do rely on the most suitable tactics of shared file storage for your organisation (e.g. Azure Shared Storage).\n\n## Plans\n1. Interface for setting up test run data log in a remote database. \n2. Robust ``timedelta`` time formats.\n3. Robust ``logfile`` handling.\n",
"bugtrack_url": null,
"license": "GPL-3.0-only",
"summary": "Evaluate test case execution against its statistical average run time & user-configured timedelta.",
"version": "0.2.10",
"project_urls": {
"Documentation": "https://github.com/yourusername/robotframework-stopwatch#readme",
"Homepage": "https://github.com/extraorange/robotframework-stopwatch"
},
"split_keywords": [
"robotframework",
" listener",
" robot framework",
" automation",
" performance testing"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a1963cace8fe68f0ef86bb5bc2082dfbe99dd665cbd3398796e8ccf8f1ac2451",
"md5": "1f143630b5a1d06fecd4fa479dd025d4",
"sha256": "793e1803fef7acbb722de21f743d45135c80980acfa1dff9adbcd2abb589e6a8"
},
"downloads": -1,
"filename": "robotframework_stopwatch-0.2.10-py3-none-any.whl",
"has_sig": false,
"md5_digest": "1f143630b5a1d06fecd4fa479dd025d4",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 17594,
"upload_time": "2024-10-02T17:34:27",
"upload_time_iso_8601": "2024-10-02T17:34:27.776204Z",
"url": "https://files.pythonhosted.org/packages/a1/96/3cace8fe68f0ef86bb5bc2082dfbe99dd665cbd3398796e8ccf8f1ac2451/robotframework_stopwatch-0.2.10-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "00ff258ee7243c4e5bee82f1f93b94ee42cdacee6fcbf53a83a8210f75f25e5f",
"md5": "9938ffda30bae6ee27105c24d79d055e",
"sha256": "d6a4258b5fe0b9aaf200bd739c362571839340177af5fb0fc7359bc4263e478f"
},
"downloads": -1,
"filename": "robotframework_stopwatch-0.2.10.tar.gz",
"has_sig": false,
"md5_digest": "9938ffda30bae6ee27105c24d79d055e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 17251,
"upload_time": "2024-10-02T17:34:29",
"upload_time_iso_8601": "2024-10-02T17:34:29.279190Z",
"url": "https://files.pythonhosted.org/packages/00/ff/258ee7243c4e5bee82f1f93b94ee42cdacee6fcbf53a83a8210f75f25e5f/robotframework_stopwatch-0.2.10.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-02 17:34:29",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "yourusername",
"github_project": "robotframework-stopwatch#readme",
"github_not_found": true,
"lcname": "robotframework-stopwatch"
}
JSON
