| Name | pytest-scrutinize JSON |
| Version |
0.1.5
JSON |
| download |
| home_page | None |
| Summary | Scrutinize your pytest test suites for slow fixtures, tests and more. |
| upload_time | 2024-08-19 12:32:29 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.9 |
| license | MIT License |
| keywords |
pytest
timing
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# pytest-scrutinize
    
Big test suites for large projects can be a pain to optimize. `pytest-scrutinize` helps you
profile your test runs by exporting *detailed* timings as JSON for the following things:
- Tests
- [Fixture setup/teardowns](#fixture-setup-and-teardown)
- [Django SQL queries](#django-sql-queries)
- [pytest-xdist](https://pypi.org/project/pytest-xdist/) worker boot times
- [Arbitrary functions](#record-additional-functions-)
- [Garbage collections](#garbage-collection)
- Pytest setup/collection times
All data is associated with the currently executing test or fixture. As an example, you can
use this to find all the Django SQL queries executed within a given fixture across your
entire test suite.
## Installation:
Install with pip [from PyPI](https://pypi.org/project/pytest-scrutinize/)
```
pip install pytest-scrutinize
```
## Usage:
Run your test suite with the `--scrutinize` flag, passing a file path to write to:
```
pytest --scrutinize=test-timings.jsonl.gz
```
## Analysing the results
A tool to help with analysing this data is not included yet, however it can be quickly explored
with [DuckDB](https://duckdb.org/). For example, to find the top 10 fixtures by total duration
along with the number of tests that where executed:
```sql
select name,
to_microseconds(sum(runtime.as_microseconds)::bigint) as duration,
count(distinct test_id) as test_count
from 'test-timings.jsonl.gz'
where type = 'fixture'
group by all
order by duration desc
limit 10;
```
Or the tests with the highest number of duplicated SQL queries executed as part of it or
any fixture it depends on:
```sql
select test_id,
sum(count) as duplicate_queries,
count(distinct sql_hash) as unique_queries,
FROM (SELECT test_id, fixture_name, sql_hash, COUNT(*) AS count
from 'test-timings.jsonl.gz'
where type = 'django-sql'
GROUP BY all
HAVING count > 1)
group by all
order by duplicate_queries desc limit 10;
```
## Data captured:
The resulting file will contain newline-delimited JSON objects. The Pydantic models for these
can be [found here](./pytest_scrutinize/data.py).
All events captured contain a `meta` structure that contains the `xdist` worker (if any), the
absolute time the timing was taken and the Python thread name that the timing was captured in.
<details>
<summary>Meta example</summary>
```json
{
"meta": {
"worker": "gw0",
"recorded_at": "2024-08-17T22:02:44.956924Z",
"thread_id": 3806124,
"thread_name": "MainThread"
}
}
```
</details>
All durations are expressed with the same structure, containing the duration in different formats:
nanoseconds, microseconds, ISO 8601 and text
<details>
<summary>Duration example</summary>
```json
{
"runtime": {
"as_nanoseconds": 60708,
"as_microseconds": 60,
"as_iso": "PT0.00006S",
"as_text": "60 microseconds"
}
}
```
</details>
### Fixture setup and teardown
Pytest fixtures can be simple functions, or context managers that can clean up resources after a
test has finished. `pytest-scrutinize` records both the setup _and_ teardown times for all fixtures,
allowing you to precisely locate performance bottlenecks:
```python
@pytest.fixture
def slow_teardown():
yield
time.sleep(1)
```
<details>
<summary>Example</summary>
```json
{
"meta": {
"worker": "master",
"recorded_at": "2024-08-17T21:23:54.736177Z",
"thread_name": "MainThread"
},
"type": "fixture",
"name": "pytest_django.plugin._django_set_urlconf",
"short_name": "_django_set_urlconf",
"test_id": "tests/test_plugin.py::test_all[normal]",
"scope": "function",
"setup": {
"as_nanoseconds": 5792,
"as_microseconds": 5,
"as_iso": "PT0.000005S",
"as_text": "5 microseconds"
},
"teardown": {
"as_nanoseconds": 2167,
"as_microseconds": 2,
"as_iso": "PT0.000002S",
"as_text": "2 microseconds"
},
"runtime": {
"as_nanoseconds": 7959,
"as_microseconds": 7,
"as_iso": "PT0.000007S",
"as_text": "7 microseconds"
}
}
```
</details>
### Django SQL queries
Information on Django SQL queries can be captured with the `--scrutinize-django-sql` flag. By
default, the hash of the SQL query is captured (allowing you to count duplicate queries), but
the raw SQL can also be captured:
```shell
# Log the hashes of the executed SQL queries
pytest --scrutinize=test-timings.jsonl.gz --scrutinize-django-sql
# Log raw SQL queries. Warning: May produce very large files!
pytest --scrutinize=test-timings.jsonl.gz --scrutinize-django-sql=query
```
<details>
<summary>Example</summary>
```json
{
"meta": {
"worker": "master",
"recorded_at": "2024-08-17T22:02:47.218492Z",
"thread_name": "MainThread"
},
"name": "django_sql",
"test_id": "test_django.py::test_case",
"fixture_name": "test_django.teardown_fixture",
"runtime": {
"as_nanoseconds": 18375,
"as_microseconds": 18,
"as_iso": "PT0.000018S",
"as_text": "18 microseconds"
},
"type": "django-sql",
"sql_hash": "be0beb84a58eab3bdc1fc4214f90abe9e937e5cc7f54008e02ab81d51533bc16",
"sql": "INSERT INTO \"django_app_dummymodel\" (\"foo\") VALUES (%s) RETURNING \"django_app_dummymodel\".\"id\""
}
```
</details>
### Record additional functions
Any arbitrary Python function can be captured by passing a comma-separated string of paths to
`--scrutinize-func`:
```shell
# Record all boto3 clients that are created, along with their timings:
pytest --scrutinize=test-timings.jsonl.gz --scrutinize-func=botocore.session.Session.create_client
```
<details>
<summary>Example</summary>
```json
{
"meta": {
"worker": "gw0",
"recorded_at": "2024-08-17T22:02:44.296938Z",
"thread_name": "MainThread"
},
"name": "urllib.parse.parse_qs",
"test_id": "test_mock.py::test_case",
"fixture_name": "test_mock.teardown_fixture",
"runtime": {
"as_nanoseconds": 2916,
"as_microseconds": 2,
"as_iso": "PT0.000002S",
"as_text": "2 microseconds"
},
"type": "mock"
}
```
</details>
### Garbage collection
Garbage collection events can be captured with the `--scrutinize-gc` flag. Every GC is captured,
along with the total time and number of objects collected. This can be used to find tests that
generate significant GC pressure by creating lots of circular-referenced objects:
```shell
pytest --scrutinize=test-timings.jsonl.gz --scrutinize-gc
```
<details>
<summary>Example</summary>
```json
{
"meta": {
"worker": "gw0",
"recorded_at": "2024-08-17T22:02:44.962665Z",
"thread_name": "MainThread"
},
"type": "gc",
"runtime": {
"as_nanoseconds": 5404333,
"as_microseconds": 5404,
"as_iso": "PT0.005404S",
"as_text": "5404 microseconds"
},
"collected_count": 279,
"generation": 2
}
```
</details>
Raw data
{
"_id": null,
"home_page": null,
"name": "pytest-scrutinize",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "pytest, timing",
"author": null,
"author_email": "Thomas Forbes <tom@tomforb.es>",
"download_url": "https://files.pythonhosted.org/packages/15/2a/073f554203c1dded9a17057adb8d2a1abb23f0ce9b8141102f529c61a6f7/pytest_scrutinize-0.1.5.tar.gz",
"platform": null,
"description": "# pytest-scrutinize\n\n    \n\nBig test suites for large projects can be a pain to optimize. `pytest-scrutinize` helps you\nprofile your test runs by exporting *detailed* timings as JSON for the following things:\n\n- Tests\n- [Fixture setup/teardowns](#fixture-setup-and-teardown)\n- [Django SQL queries](#django-sql-queries)\n- [pytest-xdist](https://pypi.org/project/pytest-xdist/) worker boot times\n- [Arbitrary functions](#record-additional-functions-)\n- [Garbage collections](#garbage-collection)\n- Pytest setup/collection times\n\nAll data is associated with the currently executing test or fixture. As an example, you can\nuse this to find all the Django SQL queries executed within a given fixture across your\nentire test suite.\n\n## Installation:\n\nInstall with pip [from PyPI](https://pypi.org/project/pytest-scrutinize/)\n\n```\npip install pytest-scrutinize\n```\n\n## Usage:\n\nRun your test suite with the `--scrutinize` flag, passing a file path to write to:\n\n```\npytest --scrutinize=test-timings.jsonl.gz\n```\n\n## Analysing the results\n\n\nA tool to help with analysing this data is not included yet, however it can be quickly explored \nwith [DuckDB](https://duckdb.org/). For example, to find the top 10 fixtures by total duration \nalong with the number of tests that where executed:\n\n```sql\nselect name,\n to_microseconds(sum(runtime.as_microseconds)::bigint) as duration,\n count(distinct test_id) as test_count\nfrom 'test-timings.jsonl.gz'\nwhere type = 'fixture'\ngroup by all\norder by duration desc\nlimit 10;\n```\n\nOr the tests with the highest number of duplicated SQL queries executed as part of it or \nany fixture it depends on:\n\n```sql\nselect test_id,\n sum(count) as duplicate_queries,\n count(distinct sql_hash) as unique_queries,\nFROM (SELECT test_id, fixture_name, sql_hash, COUNT(*) AS count\n from 'test-timings.jsonl.gz'\n where type = 'django-sql'\n GROUP BY all\n HAVING count > 1)\ngroup by all\norder by duplicate_queries desc limit 10;\n```\n\n## Data captured:\n\nThe resulting file will contain newline-delimited JSON objects. The Pydantic models for these \ncan be [found here](./pytest_scrutinize/data.py).\n\nAll events captured contain a `meta` structure that contains the `xdist` worker (if any), the \nabsolute time the timing was taken and the Python thread name that the timing was captured in.\n\n<details>\n<summary>Meta example</summary>\n\n```json\n{\n \"meta\": {\n \"worker\": \"gw0\",\n \"recorded_at\": \"2024-08-17T22:02:44.956924Z\",\n \"thread_id\": 3806124,\n \"thread_name\": \"MainThread\"\n }\n}\n```\n\n</details>\n\nAll durations are expressed with the same structure, containing the duration in different formats: \nnanoseconds, microseconds, ISO 8601 and text\n\n<details>\n<summary>Duration example</summary>\n\n```json\n{\n \"runtime\": {\n \"as_nanoseconds\": 60708,\n \"as_microseconds\": 60,\n \"as_iso\": \"PT0.00006S\",\n \"as_text\": \"60 microseconds\"\n }\n}\n```\n\n</details>\n\n### Fixture setup and teardown\n\nPytest fixtures can be simple functions, or context managers that can clean up resources after a\ntest has finished. `pytest-scrutinize` records both the setup _and_ teardown times for all fixtures,\nallowing you to precisely locate performance bottlenecks:\n\n```python\n@pytest.fixture\ndef slow_teardown():\n yield\n time.sleep(1)\n```\n\n<details>\n<summary>Example</summary>\n\n```json\n{\n \"meta\": {\n \"worker\": \"master\",\n \"recorded_at\": \"2024-08-17T21:23:54.736177Z\",\n \"thread_name\": \"MainThread\"\n },\n \"type\": \"fixture\",\n \"name\": \"pytest_django.plugin._django_set_urlconf\",\n \"short_name\": \"_django_set_urlconf\",\n \"test_id\": \"tests/test_plugin.py::test_all[normal]\",\n \"scope\": \"function\",\n \"setup\": {\n \"as_nanoseconds\": 5792,\n \"as_microseconds\": 5,\n \"as_iso\": \"PT0.000005S\",\n \"as_text\": \"5 microseconds\"\n },\n \"teardown\": {\n \"as_nanoseconds\": 2167,\n \"as_microseconds\": 2,\n \"as_iso\": \"PT0.000002S\",\n \"as_text\": \"2 microseconds\"\n },\n \"runtime\": {\n \"as_nanoseconds\": 7959,\n \"as_microseconds\": 7,\n \"as_iso\": \"PT0.000007S\",\n \"as_text\": \"7 microseconds\"\n }\n}\n```\n\n</details>\n\n### Django SQL queries\n\nInformation on Django SQL queries can be captured with the `--scrutinize-django-sql` flag. By\ndefault, the hash of the SQL query is captured (allowing you to count duplicate queries), but\nthe raw SQL can also be captured:\n\n```shell\n# Log the hashes of the executed SQL queries\npytest --scrutinize=test-timings.jsonl.gz --scrutinize-django-sql\n# Log raw SQL queries. Warning: May produce very large files!\npytest --scrutinize=test-timings.jsonl.gz --scrutinize-django-sql=query\n```\n\n<details>\n<summary>Example</summary>\n\n```json\n{\n \"meta\": {\n \"worker\": \"master\",\n \"recorded_at\": \"2024-08-17T22:02:47.218492Z\",\n \"thread_name\": \"MainThread\"\n },\n \"name\": \"django_sql\",\n \"test_id\": \"test_django.py::test_case\",\n \"fixture_name\": \"test_django.teardown_fixture\",\n \"runtime\": {\n \"as_nanoseconds\": 18375,\n \"as_microseconds\": 18,\n \"as_iso\": \"PT0.000018S\",\n \"as_text\": \"18 microseconds\"\n },\n \"type\": \"django-sql\",\n \"sql_hash\": \"be0beb84a58eab3bdc1fc4214f90abe9e937e5cc7f54008e02ab81d51533bc16\",\n \"sql\": \"INSERT INTO \\\"django_app_dummymodel\\\" (\\\"foo\\\") VALUES (%s) RETURNING \\\"django_app_dummymodel\\\".\\\"id\\\"\"\n}\n```\n\n</details>\n\n### Record additional functions\n\nAny arbitrary Python function can be captured by passing a comma-separated string of paths to\n`--scrutinize-func`:\n\n```shell\n# Record all boto3 clients that are created, along with their timings:\npytest --scrutinize=test-timings.jsonl.gz --scrutinize-func=botocore.session.Session.create_client\n```\n\n<details>\n<summary>Example</summary>\n\n```json\n{\n \"meta\": {\n \"worker\": \"gw0\",\n \"recorded_at\": \"2024-08-17T22:02:44.296938Z\",\n \"thread_name\": \"MainThread\"\n },\n \"name\": \"urllib.parse.parse_qs\",\n \"test_id\": \"test_mock.py::test_case\",\n \"fixture_name\": \"test_mock.teardown_fixture\",\n \"runtime\": {\n \"as_nanoseconds\": 2916,\n \"as_microseconds\": 2,\n \"as_iso\": \"PT0.000002S\",\n \"as_text\": \"2 microseconds\"\n },\n \"type\": \"mock\"\n}\n```\n\n</details>\n\n### Garbage collection\n\nGarbage collection events can be captured with the `--scrutinize-gc` flag. Every GC is captured,\nalong with the total time and number of objects collected. This can be used to find tests that\ngenerate significant GC pressure by creating lots of circular-referenced objects:\n\n```shell\npytest --scrutinize=test-timings.jsonl.gz --scrutinize-gc\n```\n\n<details>\n<summary>Example</summary>\n\n```json\n{\n \"meta\": {\n \"worker\": \"gw0\",\n \"recorded_at\": \"2024-08-17T22:02:44.962665Z\",\n \"thread_name\": \"MainThread\"\n },\n \"type\": \"gc\",\n \"runtime\": {\n \"as_nanoseconds\": 5404333,\n \"as_microseconds\": 5404,\n \"as_iso\": \"PT0.005404S\",\n \"as_text\": \"5404 microseconds\"\n },\n \"collected_count\": 279,\n \"generation\": 2\n}\n```\n\n</details>",
"bugtrack_url": null,
"license": "MIT License",
"summary": "Scrutinize your pytest test suites for slow fixtures, tests and more.",
"version": "0.1.5",
"project_urls": {
"Homepage": "https://github.com/orf/pytest-scrutinize/",
"Issues": "https://github.com/orf/pytest-scrutinize/issues",
"Repository": "https://github.com/orf/pytest-scrutinize.git"
},
"split_keywords": [
"pytest",
" timing"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f307d60b404a250dd630778bf253924b0d73e21de5d490f7d3e4342ed9179da3",
"md5": "d698c1834b07058000225072e0a2acaf",
"sha256": "2790d1270228c7890eee1e43bffa655eacbb473d505b7ca075547ed80615c909"
},
"downloads": -1,
"filename": "pytest_scrutinize-0.1.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d698c1834b07058000225072e0a2acaf",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 12557,
"upload_time": "2024-08-19T12:32:28",
"upload_time_iso_8601": "2024-08-19T12:32:28.199614Z",
"url": "https://files.pythonhosted.org/packages/f3/07/d60b404a250dd630778bf253924b0d73e21de5d490f7d3e4342ed9179da3/pytest_scrutinize-0.1.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "152a073f554203c1dded9a17057adb8d2a1abb23f0ce9b8141102f529c61a6f7",
"md5": "2dd8d6219c1393e30c4632a4a7c12efd",
"sha256": "56a383fca3414b1be73829dba628741f2d78248f0395815c32de7e11a5cf9690"
},
"downloads": -1,
"filename": "pytest_scrutinize-0.1.5.tar.gz",
"has_sig": false,
"md5_digest": "2dd8d6219c1393e30c4632a4a7c12efd",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 23309,
"upload_time": "2024-08-19T12:32:29",
"upload_time_iso_8601": "2024-08-19T12:32:29.146445Z",
"url": "https://files.pythonhosted.org/packages/15/2a/073f554203c1dded9a17057adb8d2a1abb23f0ce9b8141102f529c61a6f7/pytest_scrutinize-0.1.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-19 12:32:29",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "orf",
"github_project": "pytest-scrutinize",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pytest-scrutinize"
}
