Welcome to PySys!
=================
PySys is a powerful cross-platform framework for writing great system/integration tests.
It provides a comprehensive package of methods to make all the common system/integration testing operations a breeze,
with all the flexibility and power of the Python language at your fingertips. PySys is a framework that gives you a
single unified way to control the selection, ordering and reporting of every type of test including system
correctness, performance, soak/robustness testing, unit testing and manual testing.
Whatever language the application you're testing is written in, and whatever platforms it needs to run on,
PySys can help!
Key features include:
- A comprehensive library of assertion methods appropriate for system-level
testing, such as checking for error/success messages in log files and
comparing the contents of output files.
- A comprehensive library of methods to automate platform-independent process
starting, orchestration, and cleanup, for both Windows and Unix-based
systems. Includes common operations such as:
* dynamically picking a free TCP/IP port,
* waiting until a server is running on a specified port,
* waiting until a file contains a specified message,
* powerful primitives for pre-processing text files (e.g. configuration input files, or output logs)
* aborting early with a clear failure messages if an error is detected in a log file
- Support for executing tests in parallel to significantly speed up execution
time, with a flexible mechanism for controlling execution order.
- Ability to cycle a test many times (and in parallel) to reproduce rare race
conditions.
- Support for executing the same test in several modes during your test
run (for example against different web browsers, databases, or for writing
parameterized tests). Python expressions give the power to easily create
complex and dynamic lists of modes that combine multi-dimensional sets of parameters.
- A performance monitoring framework for recording and aggregating latency,
throughput and other performance metrics.
- A process memory monitoring framework to monitor memory usage when soak
testing your application.
- A pluggable "writers" framework for recording test outcomes in any format. Includes
a test output archiver, a writer for the ubiquitous JUnit/Ant(TM) XML file format,
and built-in support for running tests
under CI providers such as GitHub(R) Actions and Travis CI(R).
- Integrated support for running PyUnit tests and doctests, in case your
application is also written in Python.
- Integrated support for executing manual/interactively driven test cases.
- Test categorization and selective include/exclude execution, using per-test
classification groups.
- Support for Windows, Linux and macOS.
PySys was created by Moray Grieve. The maintainer is now Ben Spiller.
This is a community project so we welcome your contributions, whether
enhancement issues or GitHub pull requests!
Project Links
=============
.. image:: https://img.shields.io/pypi/v/PySys
:target: https://pypi.org/project/PySys/
.. image:: https://img.shields.io/badge/license-LGPL-blue
:target: https://pysys-test.github.io/pysys-test/license.html
.. image:: https://github.com/pysys-test/pysys-test/actions/workflows/pysys-test.yml/badge.svg
:target: https://github.com/pysys-test/pysys-test/actions/workflows/pysys-test.yml
.. image:: https://codecov.io/gh/pysys-test/pysys-test/branch/master/graph/badge.svg
:target: https://codecov.io/gh/pysys-test/pysys-test
- Documentation: https://pysys-test.github.io/pysys-test
- Stack Overflow tag for questions: https://stackoverflow.com/questions/ask?tags=pysys
- Bug/enhancement issue tracker: https://github.com/pysys-test/pysys-test/issues
- Source repository and sample projects: https://github.com/pysys-test
.. inclusion-marker-section-start-installation
Installation
============
PySys can be installed into any Python version from 3.8 to 3.12.
The best way to install PySys is using the standard ``pip`` installer which
downloads and install the binary package for the current PySys
release, by executing::
> python -m pip install PySys
Alternatively, you can download the binary ``.whl`` package from
https://github.com/pysys-test/pysys-test/releases and use
``python -m pip install PySys-<VERSION>.whl`` instead.
Make sure you have an up-to-date pip using ``python -m pip install --upgrade pip``.
See https://packaging.python.org/tutorials/installing-packages for
more information about using ``pip``.
Windows
-------
On Windows, pip will automatically install the
`pywin32 <https://pypi.org/project/pywin32/>`_ and
`colorama <https://pypi.org/project/colorama/>`_
libraries that PySys depends upon.
The executable launcher script ``pysys.py`` is installed into the ``Scripts\``
directory of the Python installation, e.g. ``c:\Python\Scripts\pysys.py``.
To allow easy invocation of PySys from any test directory you may wish to add
the Scripts directory to your ``PATH`` or copy the script to a location that is
already on ``PATH``. Alternatively you can run PySys using ``python -m pysys``.
Unix
----
The executable launcher script ``pysys.py`` is installed into Python's binary
directory, e.g. ``/usr/local/bin``, and hence should be on the current user's
``PATH`` automatically; if not, just add it. Alternatively you can run PySys
using ``python -m pysys``.
Those wishing to use the manual tester should ensure they have
installed the tcl/tk libraries on the host machine and are using a Python
version that was compiled with tcl/tk support.
.. inclusion-marker-section-start-getting-started
Getting Started
===============
After installation, to see the available options to the pysys.py script use::
> pysys.py --help
The script has four main commands:
- ``makeproject`` to create your top-level testing project configuration file,
- ``make`` to create individual testcases,
- ``run`` to execute them, and
- ``clean`` to delete testcase output after execution.
For detailed information, see the ``--help`` command line.
To get started, create a new directory to hold your tests. Then run the
``makeproject`` command from that directory to add a ``pysysproject.xml``
file which will hold default settings for your tests::
> mkdir test
> cd test
> pysys.py makeproject
Then to create your first test, run::
> pysys.py make MyApplication_001
This will create a ``MyApplication_001`` subdirectory with a ``pysystest.py`` file that contains both "descriptor"
metadata about the test such as its title, and a Python class where you can add the logic to ``execute`` your test,
and to ``validate`` that the results are as expected.
To run your testcases, simply execute::
> pysys.py run
To give a flavour for what's possible, here's a system test for checking the behaviour of a server application
called MyServer, which shows of the most common PySys methods:
.. code-block:: python
__pysys_title__ = r""" MyServer startup - basic sanity test (+ demo of PySys basics) """
__pysys_purpose__ = r""" To demonstrate that MyServer can startup and response to basic requests.
"""
class PySysTest(pysys.basetest.BaseTest):
def execute(self):
# Ask PySys to allocate a free TCP port to start the server on (this allows running many tests in
# parallel without clashes)
serverPort = self.getNextAvailableTCPPort()
# A common system testing task is pre-processing a file, for example to substitute in required
# testing parameters
self.copy(self.input+'/myserverconfig.json', self.output+'/', mappers=[
lambda line: line.replace('@SERVER_PORT@', str(serverPort)),
])
# Start the server application we're testing (as a background process)
# self.project provides access to properties in pysysproject.xml, such as appHome which is the
# location of the application we're testing
server = self.startProcess(
command = self.project.appHome+'/my_server.%s'%('bat' if IS_WINDOWS else 'sh'),
arguments = ['--configfile', self.output+'/myserverconfig.json', ],
environs = self.createEnvirons(addToExePath=os.path.dirname(PYTHON_EXE)),
stdouterr = 'my_server', displayName = 'my_server<port %s>'%serverPort, background = True,
)
# Wait for the server to start by polling for a grep regular expression. The errorExpr/process
# arguments ensure we abort with a really informative message if the server fails to start
self.waitForGrep('my_server.out', 'Started MyServer .*on port .*', errorExpr=[' (ERROR|FATAL) '], process=server)
# Run a test tool (in this case, written in Python) from this test's Input/ directory.
self.startPython([self.input+'/httpget.py', f'http://localhost:{serverPort}/data/myfile.json'],
stdouterr='httpget_myfile')
def validate(self):
# This method is called after execute() to perform validation of the results by checking the
# contents of files in the test's output directory. Note that during test development you can
# re-run validate() without waiting for a full execute() run using "pysys run --validateOnly".
# It's good practice to check for unexpected errors and warnings so they don't go unnoticed
self.assertGrep('my_server.out', ' (ERROR|FATAL|WARN) .*', contains=False)
# Checking for exception stack traces is also a good idea; and joining them into a single line with a mapper will
# give a more descriptive error if the test fails
self.assertGrep('my_server.out', r'Traceback [(]most recent call last[)]', contains=False,
mappers=[pysys.mappers.JoinLines.PythonTraceback()])
self.assertThat('message == expected',
message=pysys.utils.fileutils.loadJSON(self.output+'/httpget_myfile.out')['message'],
expected="Hello world!",
)
self.logFileContents('my_server.out')
If you're curious about any of the functionality demonstrated above, there's lots of helpful information on these
methods and further examples in the documentation:
- `pysys.basetest.BaseTest.getNextAvailableTCPPort()`
- `pysys.basetest.BaseTest.copy()`
- `pysys.basetest.BaseTest.startProcess()` (+ `pysys.basetest.BaseTest.createEnvirons()` and `pysys.basetest.BaseTest.startPython()`)
- `pysys.basetest.BaseTest.waitForGrep()`
- `pysys.basetest.BaseTest.assertGrep()`
- `pysys.basetest.BaseTest.assertThat()`
- `pysys.basetest.BaseTest.logFileContents()`
- `pysys.mappers`
Now take a look at `pysys.basetest` to begin exploring more of the powerful functionality
PySys provides to help you implement your own ``pysystest.py`` system tests.
The sample projects under https://github.com/pysys-test are a great starting point for learning more about PySys, and
for creating your first project.
.. inclusion-marker-section-start-license
License
=======
PySys System Test Framework
Copyright (C) 2006-2023 M.B. Grieve
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
Raw data
{
"_id": null,
"home_page": "https://pysys-test.github.io/pysys-test",
"name": "PySys",
"maintainer": "Ben Spiller",
"docs_url": null,
"requires_python": ">=3, <4",
"maintainer_email": "pysys-dev@googlegroups.com",
"keywords": "testing,qa,system testing,integration testing,unit testing",
"author": "Moray Grieve, Ben Spiller",
"author_email": "pysys-dev@googlegroups.com",
"download_url": "",
"platform": "Operating System :: Microsoft :: Windows",
"description": "Welcome to PySys!\n=================\n\nPySys is a powerful cross-platform framework for writing great system/integration tests. \n\nIt provides a comprehensive package of methods to make all the common system/integration testing operations a breeze, \nwith all the flexibility and power of the Python language at your fingertips. PySys is a framework that gives you a \nsingle unified way to control the selection, ordering and reporting of every type of test including system \ncorrectness, performance, soak/robustness testing, unit testing and manual testing.\n\nWhatever language the application you're testing is written in, and whatever platforms it needs to run on, \nPySys can help!\n\nKey features include:\n\n- A comprehensive library of assertion methods appropriate for system-level \n testing, such as checking for error/success messages in log files and \n comparing the contents of output files.\n- A comprehensive library of methods to automate platform-independent process \n starting, orchestration, and cleanup, for both Windows and Unix-based \n systems. Includes common operations such as:\n\n * dynamically picking a free TCP/IP port, \n * waiting until a server is running on a specified port,\n * waiting until a file contains a specified message, \n * powerful primitives for pre-processing text files (e.g. configuration input files, or output logs)\n * aborting early with a clear failure messages if an error is detected in a log file\n\n- Support for executing tests in parallel to significantly speed up execution \n time, with a flexible mechanism for controlling execution order.\n- Ability to cycle a test many times (and in parallel) to reproduce rare race \n conditions. \n- Support for executing the same test in several modes during your test \n run (for example against different web browsers, databases, or for writing \n parameterized tests). Python expressions give the power to easily create \n complex and dynamic lists of modes that combine multi-dimensional sets of parameters. \n- A performance monitoring framework for recording and aggregating latency, \n throughput and other performance metrics.\n- A process memory monitoring framework to monitor memory usage when soak \n testing your application.\n- A pluggable \"writers\" framework for recording test outcomes in any format. Includes \n a test output archiver, a writer for the ubiquitous JUnit/Ant(TM) XML file format, \n and built-in support for running tests \n under CI providers such as GitHub(R) Actions and Travis CI(R).\n- Integrated support for running PyUnit tests and doctests, in case your \n application is also written in Python.\n- Integrated support for executing manual/interactively driven test cases.\n- Test categorization and selective include/exclude execution, using per-test \n classification groups.\n- Support for Windows, Linux and macOS. \n\nPySys was created by Moray Grieve. The maintainer is now Ben Spiller. \nThis is a community project so we welcome your contributions, whether \nenhancement issues or GitHub pull requests! \n\nProject Links\n=============\n.. image:: https://img.shields.io/pypi/v/PySys\n\t:target: https://pypi.org/project/PySys/\n\n.. image:: https://img.shields.io/badge/license-LGPL-blue\n\t:target: https://pysys-test.github.io/pysys-test/license.html\n\n.. image:: https://github.com/pysys-test/pysys-test/actions/workflows/pysys-test.yml/badge.svg\n\t:target: https://github.com/pysys-test/pysys-test/actions/workflows/pysys-test.yml\n\n.. image:: https://codecov.io/gh/pysys-test/pysys-test/branch/master/graph/badge.svg\n\t:target: https://codecov.io/gh/pysys-test/pysys-test\n\n- Documentation: https://pysys-test.github.io/pysys-test\n- Stack Overflow tag for questions: https://stackoverflow.com/questions/ask?tags=pysys\n- Bug/enhancement issue tracker: https://github.com/pysys-test/pysys-test/issues\n- Source repository and sample projects: https://github.com/pysys-test\n\n.. inclusion-marker-section-start-installation\n\nInstallation\n============\n\nPySys can be installed into any Python version from 3.8 to 3.12. \n\nThe best way to install PySys is using the standard ``pip`` installer which \ndownloads and install the binary package for the current PySys \nrelease, by executing::\n\n\t> python -m pip install PySys\n\nAlternatively, you can download the binary ``.whl`` package from \nhttps://github.com/pysys-test/pysys-test/releases and use \n``python -m pip install PySys-<VERSION>.whl`` instead. \n\nMake sure you have an up-to-date pip using ``python -m pip install --upgrade pip``.\nSee https://packaging.python.org/tutorials/installing-packages for \nmore information about using ``pip``.\n\nWindows\n-------\nOn Windows, pip will automatically install the \n`pywin32 <https://pypi.org/project/pywin32/>`_ and \n`colorama <https://pypi.org/project/colorama/>`_ \nlibraries that PySys depends upon.\n\nThe executable launcher script ``pysys.py`` is installed into the ``Scripts\\`` \ndirectory of the Python installation, e.g. ``c:\\Python\\Scripts\\pysys.py``. \nTo allow easy invocation of PySys from any test directory you may wish to add \nthe Scripts directory to your ``PATH`` or copy the script to a location that is \nalready on ``PATH``. Alternatively you can run PySys using ``python -m pysys``.\n\n\nUnix\n----\nThe executable launcher script ``pysys.py`` is installed into Python's binary \ndirectory, e.g. ``/usr/local/bin``, and hence should be on the current user's \n``PATH`` automatically; if not, just add it. Alternatively you can run PySys \nusing ``python -m pysys``.\n\nThose wishing to use the manual tester should ensure they have \ninstalled the tcl/tk libraries on the host machine and are using a Python \nversion that was compiled with tcl/tk support.\n\n.. inclusion-marker-section-start-getting-started\n\nGetting Started\n===============\nAfter installation, to see the available options to the pysys.py script use::\n\n\t> pysys.py --help\n \nThe script has four main commands: \n\n- ``makeproject`` to create your top-level testing project configuration file, \n- ``make`` to create individual testcases, \n- ``run`` to execute them, and \n- ``clean`` to delete testcase output after execution.\n\nFor detailed information, see the ``--help`` command line. \n\nTo get started, create a new directory to hold your tests. Then run the \n``makeproject`` command from that directory to add a ``pysysproject.xml`` \nfile which will hold default settings for your tests::\n\n\t> mkdir test\n\t> cd test\n\t> pysys.py makeproject\n\nThen to create your first test, run::\n\n\t> pysys.py make MyApplication_001\n\nThis will create a ``MyApplication_001`` subdirectory with a ``pysystest.py`` file that contains both \"descriptor\" \nmetadata about the test such as its title, and a Python class where you can add the logic to ``execute`` your test, \nand to ``validate`` that the results are as expected. \n\nTo run your testcases, simply execute::\n\n\t> pysys.py run\n\nTo give a flavour for what's possible, here's a system test for checking the behaviour of a server application \ncalled MyServer, which shows of the most common PySys methods:\n\n.. code-block:: python\n\n __pysys_title__ = r\"\"\" MyServer startup - basic sanity test (+ demo of PySys basics) \"\"\"\n \n __pysys_purpose__ = r\"\"\" To demonstrate that MyServer can startup and response to basic requests. \n \"\"\"\n\n class PySysTest(pysys.basetest.BaseTest):\n def execute(self):\n # Ask PySys to allocate a free TCP port to start the server on (this allows running many tests in \n # parallel without clashes)\n serverPort = self.getNextAvailableTCPPort()\n \n # A common system testing task is pre-processing a file, for example to substitute in required \n # testing parameters\n self.copy(self.input+'/myserverconfig.json', self.output+'/', mappers=[\n lambda line: line.replace('@SERVER_PORT@', str(serverPort)),\n ])\n \n # Start the server application we're testing (as a background process)\n # self.project provides access to properties in pysysproject.xml, such as appHome which is the \n # location of the application we're testing\n server = self.startProcess(\n command = self.project.appHome+'/my_server.%s'%('bat' if IS_WINDOWS else 'sh'), \n arguments = ['--configfile', self.output+'/myserverconfig.json', ], \n environs = self.createEnvirons(addToExePath=os.path.dirname(PYTHON_EXE)),\n stdouterr = 'my_server', displayName = 'my_server<port %s>'%serverPort, background = True,\n )\n \n # Wait for the server to start by polling for a grep regular expression. The errorExpr/process \n # arguments ensure we abort with a really informative message if the server fails to start\n self.waitForGrep('my_server.out', 'Started MyServer .*on port .*', errorExpr=[' (ERROR|FATAL) '], process=server) \n \n # Run a test tool (in this case, written in Python) from this test's Input/ directory.\n self.startPython([self.input+'/httpget.py', f'http://localhost:{serverPort}/data/myfile.json'], \n stdouterr='httpget_myfile')\n \n def validate(self):\n # This method is called after execute() to perform validation of the results by checking the \n # contents of files in the test's output directory. Note that during test development you can \n # re-run validate() without waiting for a full execute() run using \"pysys run --validateOnly\". \n \n # It's good practice to check for unexpected errors and warnings so they don't go unnoticed\n self.assertGrep('my_server.out', ' (ERROR|FATAL|WARN) .*', contains=False)\n \n # Checking for exception stack traces is also a good idea; and joining them into a single line with a mapper will \n # give a more descriptive error if the test fails\n self.assertGrep('my_server.out', r'Traceback [(]most recent call last[)]', contains=False, \n mappers=[pysys.mappers.JoinLines.PythonTraceback()])\n \n self.assertThat('message == expected', \n message=pysys.utils.fileutils.loadJSON(self.output+'/httpget_myfile.out')['message'], \n expected=\"Hello world!\", \n )\n \n self.logFileContents('my_server.out')\n\nIf you're curious about any of the functionality demonstrated above, there's lots of helpful information on these \nmethods and further examples in the documentation:\n\n- `pysys.basetest.BaseTest.getNextAvailableTCPPort()`\n- `pysys.basetest.BaseTest.copy()`\n- `pysys.basetest.BaseTest.startProcess()` (+ `pysys.basetest.BaseTest.createEnvirons()` and `pysys.basetest.BaseTest.startPython()`)\n- `pysys.basetest.BaseTest.waitForGrep()`\n- `pysys.basetest.BaseTest.assertGrep()`\n- `pysys.basetest.BaseTest.assertThat()`\n- `pysys.basetest.BaseTest.logFileContents()`\n- `pysys.mappers`\n\nNow take a look at `pysys.basetest` to begin exploring more of the powerful functionality \nPySys provides to help you implement your own ``pysystest.py`` system tests. \n\nThe sample projects under https://github.com/pysys-test are a great starting point for learning more about PySys, and \nfor creating your first project. \n\n.. inclusion-marker-section-start-license\n\nLicense\n=======\n\nPySys System Test Framework\n\nCopyright (C) 2006-2023 M.B. Grieve\n\nThis library is free software; you can redistribute it and/or\nmodify it under the terms of the GNU Lesser General Public\nLicense as published by the Free Software Foundation; either\nversion 2.1 of the License, or (at your option) any later version.\n\nThis library is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU\nLesser General Public License for more details.\n",
"bugtrack_url": null,
"license": "GNU Lesser General Public License",
"summary": "Python System Test Framework",
"version": "2.2",
"project_urls": {
"Ask a Question": "https://stackoverflow.com/questions/ask?tags=pysys",
"Change Log": "https://pysys-test.github.io/pysys-test/ChangeLog.html",
"Homepage": "https://pysys-test.github.io/pysys-test",
"Repository": "https://github.com/pysys-test/pysys-test",
"Sample": "https://github.com/pysys-test/sample-getting-started",
"Tracker": "https://github.com/pysys-test/pysys-test/issues"
},
"split_keywords": [
"testing",
"qa",
"system testing",
"integration testing",
"unit testing"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c09c8dca2e09bf0c78a274f22dbbe6a4531f36f6081341b11c9958b0a0643dad",
"md5": "70959ac379994c4712c2a0e6c9b65c4d",
"sha256": "968b9a681695c2c22d73baae2465238b15b54c355c46e61118e4b25c161a5d1e"
},
"downloads": -1,
"filename": "PySys-2.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "70959ac379994c4712c2a0e6c9b65c4d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3, <4",
"size": 458177,
"upload_time": "2023-10-19T14:18:27",
"upload_time_iso_8601": "2023-10-19T14:18:27.892276Z",
"url": "https://files.pythonhosted.org/packages/c0/9c/8dca2e09bf0c78a274f22dbbe6a4531f36f6081341b11c9958b0a0643dad/PySys-2.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-10-19 14:18:27",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "pysys-test",
"github_project": "pysys-test",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pysys"
}
JSON
