.. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.5811772.svg
:target: https://doi.org/10.5281/zenodo.5811772
:alt: 10.5281/zenodo.5811772
.. image:: https://github.com/astropy/pytest-arraydiff/workflows/CI/badge.svg
:target: https://github.com/astropy/pytest-arraydiff/actions
:alt: CI Status
.. image:: https://img.shields.io/pypi/v/pytest-arraydiff.svg
:target: https://pypi.org/project/pytest-arraydiff
:alt: PyPI Status
About
-----
This is a `py.test <http://pytest.org>`__ plugin to facilitate the
generation and comparison of data arrays produced during tests, in particular
in cases where the arrays are too large to conveniently hard-code them
in the tests (e.g. ``np.testing.assert_allclose(x, [1, 2, 3])``).
The basic idea is that you can write a test that generates a Numpy array (or
other related objects depending on the format, e.g. pandas DataFrame).
You can then either run the
tests in a mode to **generate** reference files from the arrays, or you can run
the tests in **comparison** mode, which will compare the results of the tests to
the reference ones within some tolerance.
At the moment, the supported file formats for the reference files are:
- A plain text-based format (based on Numpy ``loadtxt`` output)
- The FITS format (requires `astropy <http://www.astropy.org>`__). With this
format, tests can return either a Numpy array for a FITS HDU object.
- A pandas HDF5 format using the pandas HDFStore
For more information on how to write tests to do this, see the **Using**
section below.
Installing
----------
This plugin is compatible with Python 2.7, and 3.5 and later, and
requires `pytest <http://pytest.org>`__ and
`numpy <http://www.numpy.org>`__ to be installed.
To install, you can do::
pip install pytest-arraydiff
You can check that the plugin is registered with pytest by doing::
py.test --version
which will show a list of plugins::
This is pytest version 2.7.1, imported from ...
setuptools registered plugins:
pytest-arraydiff-0.1 at ...
Using
-----
To use, you simply need to mark the function where you want to compare
arrays using ``@pytest.mark.array_compare``, and make sure that the
function returns a plain Numpy array::
python
import pytest
import numpy as np
@pytest.mark.array_compare
def test_succeeds():
return np.arange(3 * 5 * 4).reshape((3, 5, 4))
To generate the reference data files, run the tests with the
``--arraydiff-generate-path`` option with the name of the directory
where the generated files should be placed::
py.test --arraydiff-generate-path=reference
If the directory does not exist, it will be created. The directory will
be interpreted as being relative to where you are running ``py.test``.
Make sure you manually check the reference arrays to ensure they are
correct.
Once you are happy with the generated data files, you should move them
to a sub-directory called ``reference`` relative to the test files (this
name is configurable, see below). You can also generate the baseline
arrays directly in the right directory.
You can then run the tests simply with::
py.test --arraydiff
and the tests will pass if the arrays are the same. If you omit the
``--arraydiff`` option, the tests will run but will only check that the
code runs without checking the output arrays.
Options
-------
The ``@pytest.mark.array_compare`` marker take an argument to specify
the format to use for the reference files:
.. code:: python
@pytest.mark.array_compare(file_format='text')
def test_array():
...
The default file format can also be specified using the
``--arraydiff-default-format=<format>`` flag when running ``py.test``,
and ``<format>`` should be either ``fits`` or ``text``.
The supported formats at this time are ``text`` and ``fits``, and
contributions for other formats are welcome. The default format is
``text``.
Additional arguments are the relative and absolute tolerances for floating
point values (which default to 1e-7 and 0, respectively):
.. code:: python
@pytest.mark.array_compare(rtol=20, atol=0.1)
def test_array():
...
You can also pass keyword arguments to the writers using the
``write_kwargs``. For the ``text`` format, these arguments are passed to
``savetxt`` while for the ``fits`` format they are passed to Astropy's
``fits.writeto`` function.
.. code:: python
@pytest.mark.array_compare(file_format='fits', write_kwargs={'output_verify': 'silentfix'})
def test_array():
...
Other options include the name of the reference directory (which
defaults to ``reference`` ) and the filename for the reference file
(which defaults to the name of the test with a format-dependent
extension).
.. code:: python
@pytest.mark.array_compare(reference_dir='baseline_arrays',
filename='other_name.fits')
def test_array():
...
The reference directory in the decorator above will be interpreted as
being relative to the test file. Note that the baseline directory can
also be a URL (which should start with ``http://`` or ``https://`` and
end in a slash).
Finally, you can also set a custom baseline directory globally when
running tests by running ``py.test`` with::
py.test --arraydiff --arraydiff-reference-path=baseline_arrays
This directory will be interpreted as being relative to where the tests
are run. In addition, if both this option and the ``reference_dir``
option in the ``array_compare`` decorator are used, the one in the
decorator takes precedence.
Test failure example
--------------------
If the arrays produced by the tests are correct, then the test will
pass, but if they are not, the test will fail with a message similar to
the following::
E AssertionError:
E
E a: /var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/tmpbvjkzt_q/test_to_mask_rect-mode_subpixels-subpixels_18.txt
E b: /var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/tmpbvjkzt_q/reference-test_to_mask_rect-mode_subpixels-subpixels_18.txt
E
E Not equal to tolerance rtol=1e-07, atol=0
E
E (mismatch 47.22222222222222%)
E x: array([[ 0. , 0. , 0. , 0. , 0.404012, 0.55 ,
E 0.023765, 0. , 0. ],
E [ 0. , 0. , 0. , 0.112037, 1.028704, 1.1 ,...
E y: array([[ 0. , 0. , 0. , 0. , 0.367284, 0.5 ,
E 0.021605, 0. , 0. ],
E [ 0. , 0. , 0. , 0.101852, 0.935185, 1. ,...
The file paths included in the exception are then available for
inspection.
Running the tests for pytest-arraydiff
--------------------------------------
If you are contributing some changes and want to run the tests, first
install the latest version of the plugin then do::
cd tests
py.test --arraydiff
The reason for having to install the plugin first is to ensure that the
plugin is correctly loaded as part of the test suite.
Raw data
{
"_id": null,
"home_page": "https://github.com/astropy/pytest-arraydiff",
"name": "pytest-arraydiff",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "",
"author": "The Astropy Developers",
"author_email": "astropy.team@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/1e/8a/c43e892759d10d134763b6cccf853e66a18cd956616bebd4b7d782471534/pytest-arraydiff-0.6.1.tar.gz",
"platform": null,
"description": ".. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.5811772.svg\n :target: https://doi.org/10.5281/zenodo.5811772\n :alt: 10.5281/zenodo.5811772\n\n.. image:: https://github.com/astropy/pytest-arraydiff/workflows/CI/badge.svg\n :target: https://github.com/astropy/pytest-arraydiff/actions\n :alt: CI Status\n\n.. image:: https://img.shields.io/pypi/v/pytest-arraydiff.svg\n :target: https://pypi.org/project/pytest-arraydiff\n :alt: PyPI Status\n\nAbout\n-----\n\nThis is a `py.test <http://pytest.org>`__ plugin to facilitate the\ngeneration and comparison of data arrays produced during tests, in particular\nin cases where the arrays are too large to conveniently hard-code them\nin the tests (e.g. ``np.testing.assert_allclose(x, [1, 2, 3])``).\n\nThe basic idea is that you can write a test that generates a Numpy array (or\nother related objects depending on the format, e.g. pandas DataFrame).\nYou can then either run the\ntests in a mode to **generate** reference files from the arrays, or you can run\nthe tests in **comparison** mode, which will compare the results of the tests to\nthe reference ones within some tolerance.\n\nAt the moment, the supported file formats for the reference files are:\n\n- A plain text-based format (based on Numpy ``loadtxt`` output)\n- The FITS format (requires `astropy <http://www.astropy.org>`__). With this\n format, tests can return either a Numpy array for a FITS HDU object.\n- A pandas HDF5 format using the pandas HDFStore\n\nFor more information on how to write tests to do this, see the **Using**\nsection below.\n\nInstalling\n----------\n\nThis plugin is compatible with Python 2.7, and 3.5 and later, and\nrequires `pytest <http://pytest.org>`__ and\n`numpy <http://www.numpy.org>`__ to be installed.\n\nTo install, you can do::\n\n pip install pytest-arraydiff\n\nYou can check that the plugin is registered with pytest by doing::\n\n py.test --version\n\nwhich will show a list of plugins::\n\n This is pytest version 2.7.1, imported from ...\n setuptools registered plugins:\n pytest-arraydiff-0.1 at ...\n\nUsing\n-----\n\nTo use, you simply need to mark the function where you want to compare\narrays using ``@pytest.mark.array_compare``, and make sure that the\nfunction returns a plain Numpy array::\n\n python\n import pytest\n import numpy as np\n\n @pytest.mark.array_compare\n def test_succeeds():\n return np.arange(3 * 5 * 4).reshape((3, 5, 4))\n\nTo generate the reference data files, run the tests with the\n``--arraydiff-generate-path`` option with the name of the directory\nwhere the generated files should be placed::\n\n py.test --arraydiff-generate-path=reference\n\nIf the directory does not exist, it will be created. The directory will\nbe interpreted as being relative to where you are running ``py.test``.\nMake sure you manually check the reference arrays to ensure they are\ncorrect.\n\nOnce you are happy with the generated data files, you should move them\nto a sub-directory called ``reference`` relative to the test files (this\nname is configurable, see below). You can also generate the baseline\narrays directly in the right directory.\n\nYou can then run the tests simply with::\n\n py.test --arraydiff\n\nand the tests will pass if the arrays are the same. If you omit the\n``--arraydiff`` option, the tests will run but will only check that the\ncode runs without checking the output arrays.\n\nOptions\n-------\n\nThe ``@pytest.mark.array_compare`` marker take an argument to specify\nthe format to use for the reference files:\n\n.. code:: python\n\n @pytest.mark.array_compare(file_format='text')\n def test_array():\n ...\n\nThe default file format can also be specified using the\n``--arraydiff-default-format=<format>`` flag when running ``py.test``,\nand ``<format>`` should be either ``fits`` or ``text``.\n\nThe supported formats at this time are ``text`` and ``fits``, and\ncontributions for other formats are welcome. The default format is\n``text``.\n\nAdditional arguments are the relative and absolute tolerances for floating\npoint values (which default to 1e-7 and 0, respectively):\n\n.. code:: python\n\n @pytest.mark.array_compare(rtol=20, atol=0.1)\n def test_array():\n ...\n\nYou can also pass keyword arguments to the writers using the\n``write_kwargs``. For the ``text`` format, these arguments are passed to\n``savetxt`` while for the ``fits`` format they are passed to Astropy's\n``fits.writeto`` function.\n\n.. code:: python\n\n @pytest.mark.array_compare(file_format='fits', write_kwargs={'output_verify': 'silentfix'})\n def test_array():\n ...\n\nOther options include the name of the reference directory (which\ndefaults to ``reference`` ) and the filename for the reference file\n(which defaults to the name of the test with a format-dependent\nextension).\n\n.. code:: python\n\n @pytest.mark.array_compare(reference_dir='baseline_arrays',\n filename='other_name.fits')\n def test_array():\n ...\n\nThe reference directory in the decorator above will be interpreted as\nbeing relative to the test file. Note that the baseline directory can\nalso be a URL (which should start with ``http://`` or ``https://`` and\nend in a slash).\n\nFinally, you can also set a custom baseline directory globally when\nrunning tests by running ``py.test`` with::\n\n py.test --arraydiff --arraydiff-reference-path=baseline_arrays\n\nThis directory will be interpreted as being relative to where the tests\nare run. In addition, if both this option and the ``reference_dir``\noption in the ``array_compare`` decorator are used, the one in the\ndecorator takes precedence.\n\nTest failure example\n--------------------\n\nIf the arrays produced by the tests are correct, then the test will\npass, but if they are not, the test will fail with a message similar to\nthe following::\n\n E AssertionError:\n E\n E a: /var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/tmpbvjkzt_q/test_to_mask_rect-mode_subpixels-subpixels_18.txt\n E b: /var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/tmpbvjkzt_q/reference-test_to_mask_rect-mode_subpixels-subpixels_18.txt\n E\n E Not equal to tolerance rtol=1e-07, atol=0\n E\n E (mismatch 47.22222222222222%)\n E x: array([[ 0. , 0. , 0. , 0. , 0.404012, 0.55 ,\n E 0.023765, 0. , 0. ],\n E [ 0. , 0. , 0. , 0.112037, 1.028704, 1.1 ,...\n E y: array([[ 0. , 0. , 0. , 0. , 0.367284, 0.5 ,\n E 0.021605, 0. , 0. ],\n E [ 0. , 0. , 0. , 0.101852, 0.935185, 1. ,...\n\nThe file paths included in the exception are then available for\ninspection.\n\nRunning the tests for pytest-arraydiff\n--------------------------------------\n\nIf you are contributing some changes and want to run the tests, first\ninstall the latest version of the plugin then do::\n\n cd tests\n py.test --arraydiff\n\nThe reason for having to install the plugin first is to ensure that the\nplugin is correctly loaded as part of the test suite.\n",
"bugtrack_url": null,
"license": "BSD",
"summary": "pytest plugin to help with comparing array output from tests",
"version": "0.6.1",
"project_urls": {
"Homepage": "https://github.com/astropy/pytest-arraydiff"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "cf2a2f8efb1ef8048aec7b44f37bc3fecb8c2c22e3238781d3e93c58d9c89268",
"md5": "ce1e3d6ea9ab45a7c2f36c8fd5e98cf4",
"sha256": "64be1cc8e79874203eca80b1959134b8bb7a47b41cf7631310ba7fe6e5840694"
},
"downloads": -1,
"filename": "pytest_arraydiff-0.6.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "ce1e3d6ea9ab45a7c2f36c8fd5e98cf4",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 10042,
"upload_time": "2023-11-27T10:52:16",
"upload_time_iso_8601": "2023-11-27T10:52:16.382799Z",
"url": "https://files.pythonhosted.org/packages/cf/2a/2f8efb1ef8048aec7b44f37bc3fecb8c2c22e3238781d3e93c58d9c89268/pytest_arraydiff-0.6.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1e8ac43e892759d10d134763b6cccf853e66a18cd956616bebd4b7d782471534",
"md5": "ee1a683fa6a312d86dd8bd9342744f78",
"sha256": "2937b1450fc935620f24709d87d40c67e055a043d7b8541a25fdfa994dda67de"
},
"downloads": -1,
"filename": "pytest-arraydiff-0.6.1.tar.gz",
"has_sig": false,
"md5_digest": "ee1a683fa6a312d86dd8bd9342744f78",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 16907,
"upload_time": "2023-11-27T10:52:18",
"upload_time_iso_8601": "2023-11-27T10:52:18.175956Z",
"url": "https://files.pythonhosted.org/packages/1e/8a/c43e892759d10d134763b6cccf853e66a18cd956616bebd4b7d782471534/pytest-arraydiff-0.6.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-27 10:52:18",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "astropy",
"github_project": "pytest-arraydiff",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "pytest-arraydiff"
}
JSON
