| Name | cython-dev-tools JSON |
| Version |
0.6.7
JSON |
| download |
| home_page | |
| Summary | Cython development toolkit (debugger, profiler, coverage, unit tests) |
| upload_time | 2023-05-14 12:40:00 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.6 |
| license | MIT License Copyright (c) 2021 Aleksandr Vedeneev Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
cython
debugger
profiler
coverage
unit tests
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Cython Dev Tools: toolkit for efficient Cython development
## Key features
- Keeping all development tools in one place with minimal efforts
- Built-in Cython debugger (including low-level cdef / C code, based on GDB)
- Cython code unit testing with coverage reports
- Cython code unit test with cdef class mocks / side-effects
- Line-profiler
- Easy running .pyx files by entry-point functions
- Cython annotations and index file for all project
- Cython project boilerplate
- Cython snippets for low-level C-code and debugging tricks
- Makefile for running command shortcuts, also don't let you forget to build changed files (for example `make test`, `make test-debug p=path/to_test.py`)
## Requirements
- Python 3.6+ (including debug version for CyGDB)
- GDB 7+ (tested with ver 10 and 13)
- Cython 0.29
## Getting started
```bash
pip install cython-dev-tools
# Create a new directory for a project
mkdir init_project
cd init_project
# Initialize cython tools project files
cytool initialize . --include-samples --include-boilerplate --boilerplate-name=cytoolz
# The initialize commands will setup a simple package with core code and tests at ./cytools dir
# also at `cy_tools_sample` directory you can find samples
```
## Initialize command
Cython tools require each project root to be initialized, the folder `.cython_dev_tools`
will be created at the project root. There you can find all temporary files, like annotations,
coverage data, debug files, etc.
The main purpose of initialization also is keeping the all project paths relative to
its project root, because most of Cython utils are sensitive to relative paths, and
require appropriate project structure.
To get more help run:
```bash
cytool initialize --help
```
## Building cython code
Cython tools does the building automatically even if the module code doesn't use any
automatic `pyximport`, the compiled `.c` source and modules `.so|.dll` will be placed
near each `.pyx` file in the project.
Building debug version of project is a mandatory for cython tools functions:
```
cytool build --debug
```
**IMPORTANT:** If you have the `setup.py` that somehow compiles Cython code the `cytool`
will gracefully use it, but you will have to add new code/modules for compilation manually.
## Debugging
Cython debugging is a hard fork of CyGDB, however I refactored its core to make it simply
work.
The **critical** requirement for functioning of the debugger is having a `python-dbg` as
the interpreter, that runs/compiles cython. There is no Anaconda python version with
debug symbols, so I ended with Linux built-in Python 3.9d `sudo apt-get install python-dbg`
and PipEnv.
To figure out whether your modules compiled with debug info check the `d` in the file name,
like this: cy_memory_unsafe.cpython-39**d**-x86_64-linux-gnu.so, python interpreter should
be like `python3.6-dbg` or `python3.6d`.
### Launching debug session
```
# cytool build --debug (required)
#
# Both .py / .pyx entry points are suported
#
# Can be called by file path relative to project root
cytool debug cy_tools_samples/debugging/abort.pyx@main
# Also by package name with breakpoint at line 24 (only Cython!)
cytool debug cy_tools_samples.debugging.assertions@main:24
# Can contain default breakpoint at line 24 (only Cython!)
cytool debug cy_tools_samples/debugging/assertions.pyx@main:24
# Can break at segmentation fault too (at line 13 at the entry module)
cytool debug cy_tools_samples/debugging/segfault.pyx@main -b 13
# Can break at class method
cytool debug cy_tools_samples/cy_memory_unsafe.pyx@main:TestClass.hello_add
# Many breakpoints allowed
cytool debug cy_tools_samples/debugging/segfault.pyx@main -b 13 -b 21
```
### More help
```
cytool debug --help
```
[CyGDB commands](https://cython.readthedocs.io/en/latest/src/userguide/debugging.html#using-the-debugger)
Check the `cy_tools_samples/debugging/` for more tricks on how to set breakpoints,
c-style (not python!) asserts and debug them.
### Troubleshooting the debugger
This functionality is still under development, so expect bugs everywhere. However, there are some
common issues with the Cython debugger:
1. Make sure that you are using **debug** version of python
2. Make sure that the current build is debug, try to force rebuild
```
cytool build --debug --force
```
3. Ramp up verbosity
```
cytool -vvvv debug cy_tools_samples/debugging/abort.pyx@main --cygdb-verbosity=3
```
4. Don't hesitate to fill an issue on GitHub!
## Unit Testing
Running unit tests on Cython low-level cdef functions or even classes is also possible task, but requires some
code preparation.
To run unit tests on low-level code you must do the following steps:
1. Implement unit test class in `test_cython_.pyx` file:
```python
import unittest
# cdef-classes require cimport and .pxd file!
from bp_cython.bp_cython_class cimport SomeCythonClass
class CythonTestCase(unittest.TestCase):
# IMPORTANT: in some reason, NoseTest doesn't recognize this module as a test
def test_cytoolzz_class_init(self):
cdef double * ptr = NULL # You can also use all low-level C-like stuff
c = SomeCythonClass(4, 3)
# c.square() is cdef method and not accessible via Python
self.assertEqual(12, c.square())
```
2. Implement python file which includes `test_cython_.pyx` (don't use `pyximport`!) But it's literally 1 line!
```python
from my_proj.tests.test_cython_ import *
```
3. Re-build project in debug mode and run test suite. Only `pytest` works for me, BUT you still have to write `unittest.TestCase` classes,
`pytest` fancy fixtures are not supported for Cython tests.
```
> cytool build --debug
> cytool tests .
# Or alternatively via make (no extra build step required)
> make tests
```
[See Cython code with tests examples](https://github.com/alexveden/cython-dev-tools/tree/main/src/cython_dev_tools/_boilerplate_package/bp_cython)
## Unit Test Mocks for Cython cdef classes
Sometimes it's crucial to have fast cdef code, but occasionally it could have undetermined functions like random or datetime.now(), so I found an approach
for implementing unit test mocks for Cython class methods. Unfortunately it's not so elegant as Python one, and require implementing special class. But in
some cases it's better than nothing.
[Cython UnitTest Mocks example](https://github.com/alexveden/cython-dev-tools/tree/main/src/cython_dev_tools/_boilerplate_package/bp_cython/tests/test_bp_cython_class_mock_.pyx)
## Code coverage
Install the boilerplate project `cytool initialize --include-boilerplate`, this package
contains unit-tests, so you can play with it.
This command will run all tests in project root and open coverage report in the browser.
```
# More help
# cytool cover --help
# cytool build --debug (required)
cytool cover . --browser
```
## Annotate
For developing high performance Cython code it's crucial to run annotations to see
potential bottlenecks. Cython tools provides this functionality, you can build one file or
all files in the folder/project.
This command will run all tests in project root and open coverage report in the browser.
```
# More help
# cytool annotate --help
cytool annotate . --browser
cytool annotate cy_tools_samples/debugging/segfault.pyx --browser
```
## Running
A simple command for running the Cython code by entry point
```
# More help
# cytool run --help
cytool run cy_tools_samples/debugging/segfault.pyx@main
cytool run cy_tools_samples.debugging.segfault@main
cytool run python/works/too.py
```
## Line Profiler
Line profiler uses https://github.com/pyutils/line_profiler project, and uses the similar
logic as `%lprun` Jupyter magic
```
# More help
# cytool lprun --help
# cytool build --debug (required)
# Profile approx_pi2"(10)" -- equals to launching this function as approx_pi2(n=10)
cytool lprun cy_tools_samples/profiler/cy_module.pyx@approx_pi2"(10)"
# Profile extra functions in the entry-point module
cytool lprun cy_tools_samples/profiler/cy_module.pyx@approx_pi2"(10)" -f recip_square2
# Profile class mehtod in entry-module
cytool lprun cy_tools_samples/profiler/py_module.py@approx_pi2"(10)" -f SQ.recip_square2
# Profile function in another module
cytool lprun cy_tools_samples/profiler/py_module.py@approx_pi2"(10)" -f cy_tools_samples/profiler/cy_module.pyx@recip_square2
# Profile entire module
cytool lprun cy_tools_samples/profiler/cy_module.pyx@approx_pi2"(10)" -m cy_tools_samples/profiler/cy_module.pyx
```
## Cleanup
Cleanup all compilation junk
```
# For more help
# cytool clean --help
cytool clean
```
## Templates
Creates Cython cdef module or class using boilerplate templates relative to project root.
Optionally it can include tests in new package directory.
```
# For more help
# cytool template --help
cytool template subpkg.my_template
```
Raw data
{
"_id": null,
"home_page": "",
"name": "cython-dev-tools",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": "",
"keywords": "cython,debugger,profiler,coverage,unit tests",
"author": "",
"author_email": "Alex Veden <i@alexveden.com>",
"download_url": "https://files.pythonhosted.org/packages/b2/e3/fc67a878ee590805fab4283a0848608ffd749b3766c35a40dd732fe8b928/cython-dev-tools-0.6.7.tar.gz",
"platform": null,
"description": "# Cython Dev Tools: toolkit for efficient Cython development\n\n## Key features\n- Keeping all development tools in one place with minimal efforts\n- Built-in Cython debugger (including low-level cdef / C code, based on GDB)\n- Cython code unit testing with coverage reports\n- Cython code unit test with cdef class mocks / side-effects\n- Line-profiler\n- Easy running .pyx files by entry-point functions\n- Cython annotations and index file for all project\n- Cython project boilerplate\n- Cython snippets for low-level C-code and debugging tricks\n- Makefile for running command shortcuts, also don't let you forget to build changed files (for example `make test`, `make test-debug p=path/to_test.py`)\n\n## Requirements\n- Python 3.6+ (including debug version for CyGDB)\n- GDB 7+ (tested with ver 10 and 13)\n- Cython 0.29\n\n## Getting started\n```bash\npip install cython-dev-tools\n\n# Create a new directory for a project\nmkdir init_project\ncd init_project\n\n# Initialize cython tools project files\ncytool initialize . --include-samples --include-boilerplate --boilerplate-name=cytoolz \n\n# The initialize commands will setup a simple package with core code and tests at ./cytools dir\n# also at `cy_tools_sample` directory you can find samples\n```\n\n## Initialize command\nCython tools require each project root to be initialized, the folder `.cython_dev_tools` \nwill be created at the project root. There you can find all temporary files, like annotations,\ncoverage data, debug files, etc.\n\nThe main purpose of initialization also is keeping the all project paths relative to \nits project root, because most of Cython utils are sensitive to relative paths, and\nrequire appropriate project structure.\n\nTo get more help run:\n```bash\ncytool initialize --help\n```\n\n## Building cython code\nCython tools does the building automatically even if the module code doesn't use any\nautomatic `pyximport`, the compiled `.c` source and modules `.so|.dll` will be placed\nnear each `.pyx` file in the project.\n\nBuilding debug version of project is a mandatory for cython tools functions:\n```\ncytool build --debug\n```\n\n**IMPORTANT:** If you have the `setup.py` that somehow compiles Cython code the `cytool`\nwill gracefully use it, but you will have to add new code/modules for compilation manually.\n\n## Debugging\nCython debugging is a hard fork of CyGDB, however I refactored its core to make it simply \nwork. \n\nThe **critical** requirement for functioning of the debugger is having a `python-dbg` as \nthe interpreter, that runs/compiles cython. There is no Anaconda python version with \ndebug symbols, so I ended with Linux built-in Python 3.9d `sudo apt-get install python-dbg`\nand PipEnv. \n\nTo figure out whether your modules compiled with debug info check the `d` in the file name, \nlike this: cy_memory_unsafe.cpython-39**d**-x86_64-linux-gnu.so, python interpreter should\nbe like `python3.6-dbg` or `python3.6d`.\n\n\n### Launching debug session\n```\n# cytool build --debug (required)\n#\n# Both .py / .pyx entry points are suported \n#\n\n# Can be called by file path relative to project root \ncytool debug cy_tools_samples/debugging/abort.pyx@main\n\n# Also by package name with breakpoint at line 24 (only Cython!)\ncytool debug cy_tools_samples.debugging.assertions@main:24\n\n# Can contain default breakpoint at line 24 (only Cython!)\ncytool debug cy_tools_samples/debugging/assertions.pyx@main:24\n\n# Can break at segmentation fault too (at line 13 at the entry module)\ncytool debug cy_tools_samples/debugging/segfault.pyx@main -b 13\n\n# Can break at class method\ncytool debug cy_tools_samples/cy_memory_unsafe.pyx@main:TestClass.hello_add\n\n# Many breakpoints allowed\ncytool debug cy_tools_samples/debugging/segfault.pyx@main -b 13 -b 21\n```\n\n### More help\n```\ncytool debug --help\n```\n[CyGDB commands](https://cython.readthedocs.io/en/latest/src/userguide/debugging.html#using-the-debugger)\n\nCheck the `cy_tools_samples/debugging/` for more tricks on how to set breakpoints,\nc-style (not python!) asserts and debug them.\n\n### Troubleshooting the debugger\nThis functionality is still under development, so expect bugs everywhere. However, there are some\ncommon issues with the Cython debugger: \n\n1. Make sure that you are using **debug** version of python\n2. Make sure that the current build is debug, try to force rebuild\n```\ncytool build --debug --force\n```\n3. Ramp up verbosity\n```\ncytool -vvvv debug cy_tools_samples/debugging/abort.pyx@main --cygdb-verbosity=3\n```\n4. Don't hesitate to fill an issue on GitHub!\n\n## Unit Testing\nRunning unit tests on Cython low-level cdef functions or even classes is also possible task, but requires some \ncode preparation.\nTo run unit tests on low-level code you must do the following steps:\n1. Implement unit test class in `test_cython_.pyx` file:\n```python\nimport unittest\n# cdef-classes require cimport and .pxd file!\nfrom bp_cython.bp_cython_class cimport SomeCythonClass\n\nclass CythonTestCase(unittest.TestCase):\n # IMPORTANT: in some reason, NoseTest doesn't recognize this module as a test\n def test_cytoolzz_class_init(self):\n cdef double * ptr = NULL # You can also use all low-level C-like stuff\n c = SomeCythonClass(4, 3)\n # c.square() is cdef method and not accessible via Python\n self.assertEqual(12, c.square()) \n```\n2. Implement python file which includes `test_cython_.pyx` (don't use `pyximport`!) But it's literally 1 line!\n```python\nfrom my_proj.tests.test_cython_ import *\n```\n3. Re-build project in debug mode and run test suite. Only `pytest` works for me, BUT you still have to write `unittest.TestCase` classes, \n`pytest` fancy fixtures are not supported for Cython tests.\n```\n> cytool build --debug \n> cytool tests .\n\n# Or alternatively via make (no extra build step required)\n> make tests\n```\n[See Cython code with tests examples](https://github.com/alexveden/cython-dev-tools/tree/main/src/cython_dev_tools/_boilerplate_package/bp_cython)\n\n## Unit Test Mocks for Cython cdef classes \nSometimes it's crucial to have fast cdef code, but occasionally it could have undetermined functions like random or datetime.now(), so I found an approach\nfor implementing unit test mocks for Cython class methods. Unfortunately it's not so elegant as Python one, and require implementing special class. But in \nsome cases it's better than nothing.\n\n[Cython UnitTest Mocks example](https://github.com/alexveden/cython-dev-tools/tree/main/src/cython_dev_tools/_boilerplate_package/bp_cython/tests/test_bp_cython_class_mock_.pyx)\n\n## Code coverage\nInstall the boilerplate project `cytool initialize --include-boilerplate`, this package\ncontains unit-tests, so you can play with it.\n\nThis command will run all tests in project root and open coverage report in the browser.\n```\n# More help\n# cytool cover --help\n# cytool build --debug (required)\ncytool cover . --browser\n```\n\n## Annotate\nFor developing high performance Cython code it's crucial to run annotations to see\npotential bottlenecks. Cython tools provides this functionality, you can build one file or\nall files in the folder/project.\n\nThis command will run all tests in project root and open coverage report in the browser.\n```\n# More help\n# cytool annotate --help\ncytool annotate . --browser\ncytool annotate cy_tools_samples/debugging/segfault.pyx --browser\n```\n\n## Running\nA simple command for running the Cython code by entry point\n```\n# More help\n# cytool run --help\n \ncytool run cy_tools_samples/debugging/segfault.pyx@main\ncytool run cy_tools_samples.debugging.segfault@main\ncytool run python/works/too.py\n```\n\n## Line Profiler\nLine profiler uses https://github.com/pyutils/line_profiler project, and uses the similar\nlogic as `%lprun` Jupyter magic\n```\n# More help\n# cytool lprun --help\n# cytool build --debug (required)\n \n# Profile approx_pi2\"(10)\" -- equals to launching this function as approx_pi2(n=10) \ncytool lprun cy_tools_samples/profiler/cy_module.pyx@approx_pi2\"(10)\"\n\n# Profile extra functions in the entry-point module \ncytool lprun cy_tools_samples/profiler/cy_module.pyx@approx_pi2\"(10)\" -f recip_square2\n\n# Profile class mehtod in entry-module\ncytool lprun cy_tools_samples/profiler/py_module.py@approx_pi2\"(10)\" -f SQ.recip_square2\n\n# Profile function in another module\ncytool lprun cy_tools_samples/profiler/py_module.py@approx_pi2\"(10)\" -f cy_tools_samples/profiler/cy_module.pyx@recip_square2\n\n# Profile entire module\ncytool lprun cy_tools_samples/profiler/cy_module.pyx@approx_pi2\"(10)\" -m cy_tools_samples/profiler/cy_module.pyx\n\n```\n\n## Cleanup\nCleanup all compilation junk \n```\n# For more help\n# cytool clean --help\ncytool clean\n```\n\n## Templates\nCreates Cython cdef module or class using boilerplate templates relative to project root.\nOptionally it can include tests in new package directory.\n\n```\n# For more help\n# cytool template --help\ncytool template subpkg.my_template\n```\n\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2021 Aleksandr Vedeneev Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.",
"summary": "Cython development toolkit (debugger, profiler, coverage, unit tests)",
"version": "0.6.7",
"project_urls": {
"Homepage": "https://github.com/alexveden/cython-dev-tools"
},
"split_keywords": [
"cython",
"debugger",
"profiler",
"coverage",
"unit tests"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "237dcc41bcb818d54b8807e21057c7308b2677bf80abb48efdf4c548f87bf721",
"md5": "611970ec393d2d61c5a7e9a6e49880b5",
"sha256": "ace9b002e731752192fd8b9cab06fe3c66faad5b9183e38c340366389533fbf2"
},
"downloads": -1,
"filename": "cython_dev_tools-0.6.7-py3-none-any.whl",
"has_sig": false,
"md5_digest": "611970ec393d2d61c5a7e9a6e49880b5",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 109866,
"upload_time": "2023-05-14T12:39:57",
"upload_time_iso_8601": "2023-05-14T12:39:57.884704Z",
"url": "https://files.pythonhosted.org/packages/23/7d/cc41bcb818d54b8807e21057c7308b2677bf80abb48efdf4c548f87bf721/cython_dev_tools-0.6.7-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b2e3fc67a878ee590805fab4283a0848608ffd749b3766c35a40dd732fe8b928",
"md5": "68ed4e7ad34ccfcc56082646a0754579",
"sha256": "fb17754734939034c2dbc8ec51a1318ff2b619c1a02dc786213a8e5860f53ca9"
},
"downloads": -1,
"filename": "cython-dev-tools-0.6.7.tar.gz",
"has_sig": false,
"md5_digest": "68ed4e7ad34ccfcc56082646a0754579",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 107332,
"upload_time": "2023-05-14T12:40:00",
"upload_time_iso_8601": "2023-05-14T12:40:00.689044Z",
"url": "https://files.pythonhosted.org/packages/b2/e3/fc67a878ee590805fab4283a0848608ffd749b3766c35a40dd732fe8b928/cython-dev-tools-0.6.7.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-05-14 12:40:00",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "alexveden",
"github_project": "cython-dev-tools",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "cython-dev-tools"
}
