<h1 align="center">pytest-history</h1>
<p align="center">
Pytest plugin to keep history of your pytest runs
</p>
<p align="center">
<a href="https://github.com/Nicoretti/one-piece/python/pytest-history">
<img src="https://img.shields.io/github/checks-status/nicoretti/crc/master" alt="Checks Master">
</a>
<a href="https://opensource.org/licenses/MIT">
<img src="https://img.shields.io/pypi/l/pytest-history" alt="License">
</a>
<a href="https://pypi.org/project/pytest-history/">
<img src="https://img.shields.io/pypi/pyversions/pytest-history" alt="Supported Python Versions">
</a>
<a href="https://pypi.org/project/pytest-history/">
<img src="https://img.shields.io/pypi/v/pytest-history" alt="PyPi Package">
</a>
</p>
## Overview
`pytest-history` enables the tracking of test statuses and other metadata across multiple test runs, providing
additional insights into test behavior.
Initially, this plugin was developed specifically to identify potentially flaky tests (approximately 200) within a test
suite containing over 1000 tests, where various tests exhibited inconsistent behavior by failing on alternate runs.
## Purpose
- **Tracking Test History:** Capturing and storing historical test results, encompassing pass, fail, and other pertinent
metadata.
- **Identifying Flaky Tests:** Enabling the identification of flaky tests by scrutinizing historical data, detecting
irregularities or recurring patterns in test outcomes.
- **Facilitating Debugging:** Offering developers and testers insights into test stability, thereby assisting in
debugging efforts and enhancing overall test reliability.
## Usage
1. Install the plugin using `pip install pytest-history`.
2. Utilize the historical data stored in the `.test-results.db` SQLite database using either the `pytest-history` CLI or an SQLite client.
## Configuration Options
### 1. INI File Setting
Users can configure the database file location by adding the following setting in their INI file:
```ini
# pytest.ini
[pytest]
pytest-history = /path/to/history.db
```
```toml
# pyproject.toml
[tool.pytest.ini_options]
pytest-history = "history.db"
```
### 2. Environment Variable
Another option to configure the database file location is through an environment variable:
Setting the environment variable in a Unix-based system:
```bash
export PYTEST_HISTORY_DB=/path/to/history.db
```
Setting the environment variable in a Windows system:
```cmd
set PYTEST_HISTORY_DB=C:\path\to\history.db
```
### 3. Command-line Parameter
Users can also specify the database file location via a command-line parameter:
```shell
pytest --history-db /path/to/history.db tests/
```
The database file path specified should be writable by the user running pytest commands.
When multiple configuration methods are used simultaneously, the command-line parameter takes precedence over the environment variable, and both take precedence over the INI file setting.
## CLI
The `pytest-history` command utilizes a SQLite database (default: `.test-results.db`) to provide analysis and information about past test runs and their results. It offers two main subcommands: `list` and `flakes`.
### Subcommands
#### `list`
- `pytest-history list` offers insights into historical data and provides two additional subcommands: `results` and `runs`.
##### subcommand `results`
- `pytest-history list results <id>`: Lists historic test results for a specific test run identified by its ID.
##### subcommand `runs`
- `pytest-history list runs`: Lists historic test runs.
#### `flakes`
- `pytest-history flakes`: Lists all flaky tests identified in the test suite.
### Usage
```bash
pytest-history [options] <subcommand> [suboptions]
options
-h, --help: Show the help message and exit.
--db DB: Specify the database to be used for analyzing data (default: .test-results.db).
```
### Examples
List historic test runs
```bash
pytest-history list runs
```
List historic test results for a specific run
```bash
pytest-history list results <id>
```
List all flaky tests
```bash
pytest-history flakes
```
## Example Queries
Example: To find flaky tests between two distinct test runs, execute the following SQL query:
```sql
SELECT t1.testcase, t1.test_run, t2.test_run, t1.outcome, t2.outcome
FROM "test.results" t1
JOIN "test.results" t2 on t1.testcase = t2.testcase AND (t1.test_run = 1 AND t2.test_run = 2)
WHERE (t1.outcome = 'passed' AND t2.outcome = 'failed')
OR (t1.outcome = 'failed' AND t2.outcome = 'passed')
GROUP BY t1.testcase
ORDER BY t1.testcase;
```
Raw data
{
"_id": null,
"home_page": "https://github.com/Nicoretti/one-piece/tree/grand-line/python/pytest-history",
"name": "pytest-history",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8,<4.0",
"maintainer_email": "",
"keywords": "pytest,sqlite,sql,history",
"author": "Nicola Coretti",
"author_email": "nico.coretti@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/cc/7f/41fb61b2ff9b35d2b4987f74c0bb17f80ec3daed21597ca38adc8598615f/pytest_history-0.3.0.tar.gz",
"platform": null,
"description": "<h1 align=\"center\">pytest-history</h1>\n<p align=\"center\">\nPytest plugin to keep history of your pytest runs\n</p>\n\n<p align=\"center\">\n\n<a href=\"https://github.com/Nicoretti/one-piece/python/pytest-history\">\n <img src=\"https://img.shields.io/github/checks-status/nicoretti/crc/master\" alt=\"Checks Master\">\n</a>\n<a href=\"https://opensource.org/licenses/MIT\">\n <img src=\"https://img.shields.io/pypi/l/pytest-history\" alt=\"License\">\n</a>\n<a href=\"https://pypi.org/project/pytest-history/\">\n <img src=\"https://img.shields.io/pypi/pyversions/pytest-history\" alt=\"Supported Python Versions\">\n</a>\n<a href=\"https://pypi.org/project/pytest-history/\">\n <img src=\"https://img.shields.io/pypi/v/pytest-history\" alt=\"PyPi Package\">\n</a>\n</p>\n\n## Overview\n\n`pytest-history` enables the tracking of test statuses and other metadata across multiple test runs, providing\nadditional insights into test behavior.\n\nInitially, this plugin was developed specifically to identify potentially flaky tests (approximately 200) within a test\nsuite containing over 1000 tests, where various tests exhibited inconsistent behavior by failing on alternate runs.\n\n## Purpose\n\n- **Tracking Test History:** Capturing and storing historical test results, encompassing pass, fail, and other pertinent\n metadata.\n- **Identifying Flaky Tests:** Enabling the identification of flaky tests by scrutinizing historical data, detecting\n irregularities or recurring patterns in test outcomes.\n- **Facilitating Debugging:** Offering developers and testers insights into test stability, thereby assisting in\n debugging efforts and enhancing overall test reliability.\n\n## Usage\n\n1. Install the plugin using `pip install pytest-history`.\n2. Utilize the historical data stored in the `.test-results.db` SQLite database using either the `pytest-history` CLI or an SQLite client.\n\n## Configuration Options\n### 1. INI File Setting\nUsers can configure the database file location by adding the following setting in their INI file:\n\n```ini\n# pytest.ini\n[pytest]\npytest-history = /path/to/history.db\n```\n\n```toml\n# pyproject.toml\n[tool.pytest.ini_options]\npytest-history = \"history.db\"\n```\n\n\n### 2. Environment Variable\nAnother option to configure the database file location is through an environment variable:\n\nSetting the environment variable in a Unix-based system:\n```bash\nexport PYTEST_HISTORY_DB=/path/to/history.db\n```\n\nSetting the environment variable in a Windows system:\n```cmd\nset PYTEST_HISTORY_DB=C:\\path\\to\\history.db\n```\n\n### 3. Command-line Parameter\nUsers can also specify the database file location via a command-line parameter:\n\n```shell\npytest --history-db /path/to/history.db tests/\n```\n\n\nThe database file path specified should be writable by the user running pytest commands.\nWhen multiple configuration methods are used simultaneously, the command-line parameter takes precedence over the environment variable, and both take precedence over the INI file setting.\n\n\n## CLI\nThe `pytest-history` command utilizes a SQLite database (default: `.test-results.db`) to provide analysis and information about past test runs and their results. It offers two main subcommands: `list` and `flakes`.\n\n### Subcommands\n\n#### `list`\n- `pytest-history list` offers insights into historical data and provides two additional subcommands: `results` and `runs`.\n \n##### subcommand `results`\n- `pytest-history list results <id>`: Lists historic test results for a specific test run identified by its ID.\n \n##### subcommand `runs`\n- `pytest-history list runs`: Lists historic test runs.\n\n#### `flakes`\n- `pytest-history flakes`: Lists all flaky tests identified in the test suite.\n\n### Usage\n\n```bash\npytest-history [options] <subcommand> [suboptions]\n\noptions\n -h, --help: Show the help message and exit.\n --db DB: Specify the database to be used for analyzing data (default: .test-results.db).\n```\n\n### Examples\n\nList historic test runs\n```bash\npytest-history list runs\n```\n\nList historic test results for a specific run\n```bash\npytest-history list results <id>\n```\n\nList all flaky tests\n```bash\npytest-history flakes\n```\n\n## Example Queries\n\nExample: To find flaky tests between two distinct test runs, execute the following SQL query:\n\n```sql\nSELECT t1.testcase, t1.test_run, t2.test_run, t1.outcome, t2.outcome\nFROM \"test.results\" t1\n JOIN \"test.results\" t2 on t1.testcase = t2.testcase AND (t1.test_run = 1 AND t2.test_run = 2)\nWHERE (t1.outcome = 'passed' AND t2.outcome = 'failed')\n OR (t1.outcome = 'failed' AND t2.outcome = 'passed')\nGROUP BY t1.testcase\nORDER BY t1.testcase;\n```\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Pytest plugin to keep a history of your pytest runs",
"version": "0.3.0",
"project_urls": {
"Homepage": "https://github.com/Nicoretti/one-piece/tree/grand-line/python/pytest-history",
"Repository": "https://github.com/Nicoretti/one-piece/tree/grand-line/python/pytest-history"
},
"split_keywords": [
"pytest",
"sqlite",
"sql",
"history"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c278f53d98a0d17ebaa5a1c0a0bf9b37b9e2181f575d2a122811865c1e1eae38",
"md5": "777478915a866d53e485aaa2184b63d8",
"sha256": "5c1ed89432827632817a36b97bd897d32eecddbbede1edba37fef4e67da91136"
},
"downloads": -1,
"filename": "pytest_history-0.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "777478915a866d53e485aaa2184b63d8",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8,<4.0",
"size": 7142,
"upload_time": "2024-01-14T08:51:36",
"upload_time_iso_8601": "2024-01-14T08:51:36.603150Z",
"url": "https://files.pythonhosted.org/packages/c2/78/f53d98a0d17ebaa5a1c0a0bf9b37b9e2181f575d2a122811865c1e1eae38/pytest_history-0.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "cc7f41fb61b2ff9b35d2b4987f74c0bb17f80ec3daed21597ca38adc8598615f",
"md5": "a76bbb9cc882e94e8680d3f52ef08c72",
"sha256": "0dbdd60bd8e86f0242c802eed7ff2dcde3e0335a961b307a0e513f8a573a9f82"
},
"downloads": -1,
"filename": "pytest_history-0.3.0.tar.gz",
"has_sig": false,
"md5_digest": "a76bbb9cc882e94e8680d3f52ef08c72",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8,<4.0",
"size": 6325,
"upload_time": "2024-01-14T08:51:38",
"upload_time_iso_8601": "2024-01-14T08:51:38.542800Z",
"url": "https://files.pythonhosted.org/packages/cc/7f/41fb61b2ff9b35d2b4987f74c0bb17f80ec3daed21597ca38adc8598615f/pytest_history-0.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-14 08:51:38",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Nicoretti",
"github_project": "one-piece",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "pytest-history"
}
JSON
