# Booktest
booktest is review-driven testing tool that combines Jupyterbook style data science
development with traditional regression testing. Booktest is developed by
[Lumoa.me](https://lumoa.me), the actionable feedback analytics platform.
booktest is designed to tackle a common problem with the data science
RnD work flows and regression testing:
* Data science produces results such as probability estimates, which can be
good or bad, but not really right or wrong as in the traditional software engineering.
* Because the DS results are not strictly right or wrong, it's very difficult to use assertions
for quality assurance and preventing regression.
* For example, you cannot really say that accuracy 0.84 is correct, while the
accuracy 0.83 is incorrect, especially if you have other measurements (log likelihood)
giving conflicting results. Neither evaluating a topic model as correct or incorrect
is non-sensical. In practice, most data science applications require an expert
review.
* This less ambigious quality also creates need for a better visibility of how
the system behaves. One typically wants to print out edge cases and their diagnostics
to see the behavior, see intermediate steps and see the results for different data sets .
* There is also the problem of the data science data being big and the intermediate
results being computationally expensive.
* Jupyter notebook deals with this problem by keeping the state in memory between runs, while
traditional unittests tend to lose the program state between runs. This leads to very slow
test runs, slow iteration speed and low productivity.
* While the Jupyter Notebook provides good visibility to results required by the expert review and
powerful caching functionality: it fails short on a) often requiring copy-pasting production code to
make results visible, b) it doesn't support automated regression testing and c) expert review requires
expensive full review even if nothing changed.
booktest solves this problem setting by delivering on 3 main points:
* Focus on the results and analytic as in Jupyter notebook by allowing user to print
the results as MD files.
* Keep the intermediate results cached either in memory or in filesystem by
having two level cache.
* Instead of doing strict assertions, do testing by comparing old results with
new results.
As such, booktest does snapshot testing, and it stores the snapshots in filesystem and in Git.
Additional benefit of this approach is that you can trace the result development in Git.
# Getting started guide
You can find getting started guide [here](getting-started.md)
# Workflows, coverage and CI
You can find guide on common workflows, coverage measurements and
continuous integratio [here](workflows.md)
# Examples
Examples are found in the [test example directory](test/examples).
Example results are visible in the [book index](books/index.md).
There is also [separate example project](https://github.com/lumoa-oss/booktest-example)
# API reference
API reference is generated under [docs](docs) directory. Main classes are:
* [TestCaseRun](docs/testcaserun.py.md), which provides API for tests
* [TestBook](docs/testbook.py.md), which provide a base class for test suite object
* [TestSuite](docs/testsuite.py.md), which provide a base class for test suite object
* [Tests](docs/tests.py.md), which manages CLI interface
# Developing booktest
Development guide is available [here](development.md)
Raw data
{
"_id": null,
"home_page": "https://github.com/lumoa-oss/booktest",
"name": "booktest",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.8",
"maintainer_email": null,
"keywords": null,
"author": "Antti Rauhala",
"author_email": "antti@lumoa.me",
"download_url": "https://files.pythonhosted.org/packages/0c/f1/3b0921ee0ee61f7c1524d79c05c514abbd017f010f938365714be63c8dfb/booktest-0.3.30.tar.gz",
"platform": null,
"description": "# Booktest\n\nbooktest is review-driven testing tool that combines Jupyterbook style data science\ndevelopment with traditional regression testing. Booktest is developed by \n[Lumoa.me](https://lumoa.me), the actionable feedback analytics platform.\n\nbooktest is designed to tackle a common problem with the data science\nRnD work flows and regression testing: \n\n * Data science produces results such as probability estimates, which can be \n good or bad, but not really right or wrong as in the traditional software engineering. \n * Because the DS results are not strictly right or wrong, it's very difficult to use assertions \n for quality assurance and preventing regression.\n * For example, you cannot really say that accuracy 0.84 is correct, while the \n accuracy 0.83 is incorrect, especially if you have other measurements (log likelihood)\n giving conflicting results. Neither evaluating a topic model as correct or incorrect\n is non-sensical. In practice, most data science applications require an expert \n review.\n * This less ambigious quality also creates need for a better visibility of how\n the system behaves. One typically wants to print out edge cases and their diagnostics\n to see the behavior, see intermediate steps and see the results for different data sets . \n * There is also the problem of the data science data being big and the intermediate \n results being computationally expensive. \n * Jupyter notebook deals with this problem by keeping the state in memory between runs, while \n traditional unittests tend to lose the program state between runs. This leads to very slow \n test runs, slow iteration speed and low productivity.\n * While the Jupyter Notebook provides good visibility to results required by the expert review and\n powerful caching functionality: it fails short on a) often requiring copy-pasting production code to \n make results visible, b) it doesn't support automated regression testing and c) expert review requires\n expensive full review even if nothing changed.\n\nbooktest solves this problem setting by delivering on 3 main points:\n\n * Focus on the results and analytic as in Jupyter notebook by allowing user to print\n the results as MD files. \n * Keep the intermediate results cached either in memory or in filesystem by\n having two level cache.\n * Instead of doing strict assertions, do testing by comparing old results with \n new results.\n\nAs such, booktest does snapshot testing, and it stores the snapshots in filesystem and in Git. \nAdditional benefit of this approach is that you can trace the result development in Git.\n\n# Getting started guide\n\nYou can find getting started guide [here](getting-started.md)\n\n# Workflows, coverage and CI\n\nYou can find guide on common workflows, coverage measurements and \ncontinuous integratio [here](workflows.md)\n\n# Examples\n\nExamples are found in the [test example directory](test/examples). \n\nExample results are visible in the [book index](books/index.md).\n\nThere is also [separate example project](https://github.com/lumoa-oss/booktest-example)\n\n\n# API reference\n\nAPI reference is generated under [docs](docs) directory. Main classes are:\n\n * [TestCaseRun](docs/testcaserun.py.md), which provides API for tests\n * [TestBook](docs/testbook.py.md), which provide a base class for test suite object\n * [TestSuite](docs/testsuite.py.md), which provide a base class for test suite object\n * [Tests](docs/tests.py.md), which manages CLI interface\n\n# Developing booktest\n\nDevelopment guide is available [here](development.md)\n\n",
"bugtrack_url": null,
"license": null,
"summary": "Booktest is a snapshot testing library for review driven testing.",
"version": "0.3.30",
"project_urls": {
"Homepage": "https://github.com/lumoa-oss/booktest",
"Repository": "https://github.com/lumoa-oss/booktest"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "606c25809802ed15a0f65758be5d6a603b217c7e1952a7b661d36d1283f8df6d",
"md5": "e49650e7d9d764f5af65dcb527d60998",
"sha256": "8afb1862c81b7d48d8e0e1c8fae8cad678791e6239dc7b0a80c80555ec2035fd"
},
"downloads": -1,
"filename": "booktest-0.3.30-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e49650e7d9d764f5af65dcb527d60998",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.8",
"size": 43237,
"upload_time": "2024-05-23T11:53:27",
"upload_time_iso_8601": "2024-05-23T11:53:27.339779Z",
"url": "https://files.pythonhosted.org/packages/60/6c/25809802ed15a0f65758be5d6a603b217c7e1952a7b661d36d1283f8df6d/booktest-0.3.30-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "0cf13b0921ee0ee61f7c1524d79c05c514abbd017f010f938365714be63c8dfb",
"md5": "ca1ae037aa943c5143bdb6a93390e942",
"sha256": "9770e7b0f7a5131631ebcb1a182aefd60534f0a6fa42c9a07dfbfd1aefd8f242"
},
"downloads": -1,
"filename": "booktest-0.3.30.tar.gz",
"has_sig": false,
"md5_digest": "ca1ae037aa943c5143bdb6a93390e942",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.8",
"size": 32945,
"upload_time": "2024-05-23T11:53:29",
"upload_time_iso_8601": "2024-05-23T11:53:29.250512Z",
"url": "https://files.pythonhosted.org/packages/0c/f1/3b0921ee0ee61f7c1524d79c05c514abbd017f010f938365714be63c8dfb/booktest-0.3.30.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-23 11:53:29",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "lumoa-oss",
"github_project": "booktest",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"lcname": "booktest"
}
JSON
