*******************
Introduction
*******************
``pymemtrace`` provides tools for tracking and understanding Python memory usage at different levels, at different
granularities and with different runtime costs.
Full documentation: https://pymemtrace.readthedocs.io
pymemtrace Tools
======================
The tools provided by ``pymemtrace``:
* ``process`` is a very lightweight way of logging the total memory usage at regular time intervals.
It can plot memory over time with plotting programs such as ``gnuplot``.
See `some process examples <https://pymemtrace.readthedocs.io/en/latest/examples/process.html>`_
* ``cPyMemTrace`` is a memory tracer written in C that can report total memory usage for every function call/return for
both C and Python sections.
See some `cPyMemTrace examples <https://pymemtrace.readthedocs.io/en/latest/examples/c_py_mem_trace.html>`_
and a `technical note on cPyMemTrace <https://pymemtrace.readthedocs.io/en/latest/tech_notes/cPyMemTrace.html>`_.
* DTrace: Here are a number of D scripts that can trace the low level ``malloc()`` and ``free()`` system calls and
report how much memory was allocated and by whom.
See some `DTrace examples <https://pymemtrace.readthedocs.io/en/latest/examples/dtrace.html>`_
and a `technical note on DTrace <https://pymemtrace.readthedocs.io/en/latest/tech_notes/dtrace.html>`_.
* ``trace_malloc`` is a convenience wrapper around the Python standard library `tracemalloc` module.
This can report Python memory usage by module and line compensating for the cost of ``tracemalloc``.
This can take memory snapshots before and after code blocks and show the change on memory caused by that code.
See some `trace_malloc examples <https://pymemtrace.readthedocs.io/en/latest/examples/trace_malloc.html>`_
* ``debug_malloc_stats`` is a wrapper around the ``sys._debugmallocstats`` function that can take snapshots of
memory before and after code execution and report the significant differences of the Python small object allocator.
See some `debug_malloc_stats examples <https://pymemtrace.readthedocs.io/en/latest/examples/debug_malloc_stats.html>`_
Tool Characteristics
======================
Each tool can be characterised by:
- *Memory Granularity*: In how much detail is a memory change is observed.
An example of *coarse* memory granularity is measuring the
`Resident Set Size <https://en.wikipedia.org/wiki/Resident_set_size>`_ which is normally in chunks of 4096 bytes.
An example of *fine* memory granularity is recording every ``malloc()`` and ``free()``.
- *Execution Granularity*: In how much code detail is the memory change observed.
An example of *coarse* execution granularity is measuring the memory usage every second.
An example of *fine* execution granularity is recording the memory usage every Python line.
- *Memory Cost*: How much extra memory the tool needs.
- *Execution Cost*: How much the execution time is increased.
Clearly there are trade-offs between these depending on the problem you are trying to solve.
.. list-table:: **Tool Characteristics**
:widths: 15 30 30 30 30
:header-rows: 1
* - Tool
- Memory Granularity
- Execution Granularity
- Memory Cost
- Execution Cost
* - ``process``
- RSS (total Python and C memory).
- Regular time intervals.
- Near zero.
- Near zero.
* - ``cPyMemTrace``
- RSS (total Python and C memory).
- Per Python line, Python function and C function call.
- Near zero.
- x10 to x20.
* - DTrace
- Every ``malloc()`` and ``free()``.
- Per function call and return.
- Minimal.
- x90 to x100.
* - ``trace_malloc``
- Every Python object.
- Per Python line, per function call.
- Significant but compensated.
- x900 for small objects, x6 for large objects.
* - ``debug_malloc_stats``
- Python memory pool.
- Snapshots the CPython memory pool either side of a block of code.
- Minimal.
- x2000+ for small objects, x12 for large objects.
Package Metadata
=========================
.. image:: https://img.shields.io/pypi/v/pymemtrace.svg
:target: https://pypi.python.org/pypi/pymemtrace
.. image:: https://img.shields.io/travis/paulross/pymemtrace.svg
:target: https://travis-ci.org/paulross/pymemtrace
.. image:: https://readthedocs.org/projects/pymemtrace/badge/?version=latest
:target: https://pymemtrace.readthedocs.io/en/latest/?badge=latest
:alt: Documentation Status
.. image:: https://pyup.io/repos/github/paulross/pymemtrace/shield.svg
:target: https://pyup.io/repos/github/paulross/pymemtrace/
:alt: Updates
Licence
-----------------------
Python memory tracing.
* Free software: MIT license
* Documentation: https://pymemtrace.readthedocs.io.
* Project: https://github.com/paulross/pymemtrace.
Credits
-----------------
Phil Smith (AHL) with whom a casual lunch time chat lead to the creation of an earlier, but quite different
implementation, of ``cPyMemTrace`` in pure Python.
This package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.
.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage
=======
History
=======
0.2.0 (2024-11-17)
---------------------
* cPyMemTrace:
* Add P/T, stack depth and python version to log file name, example:
"20241107_195847_62264_P_0_PY3.13.0b3.log"
* Add stacking of trace/profile functions with linked list of tTraceFileWrapperLinkedList.
* Add an option to log to a specific file.
* Add an API write_to_log() to inject text into the log file.
* Add an optional message to the log file in cPyMemTrace.
* Add Python API to get log file being written to by cPyMemTrace.
* Bug fixes in cPyMemTrace.c
* Safety fix for file path name lengths.
* Fix for log files where '#' was being concatenated.
0.1.7 (2024-09-12)
------------------
* Minor fix for a single test.
0.1.6 (2024-09-11)
------------------
* Add support for Python versions 3.12, 3.13. Now supports Python versions 3.7, 3.8, 3.9, 3.10, 3.11, 3.12, 3.13.
0.1.5 (2023-06-21)
------------------
* Add support for Python versions 3.10, 3.11. Now supports Python versions 3.7, 3.8, 3.9, 3.10, 3.11.
0.1.4 (2022-03-19)
------------------
* Fix Linux build.
0.1.3 (2022-03-17)
------------------
* Fix some tests.
0.1.2 (2022-03-17)
------------------
* Fix source distribution that had missing headers.
0.1.1 (2020-11-17)
------------------
* Add cPyMemTrace the C level profiler.
* Add DTrace scripts for low level tracing.
* Add debug_malloc_stats the wrapper around sys._debugmallocstats.
* Add process from the TotalDepth project.
* Add redirect_stdout for debug_malloc_stats.
* Add trace_malloc, a wrapper around the tracemalloc module.
* Includes extensive documentation and performance measurement.
* First release on PyPI.
0.1.0 (2017-12-04)
------------------
* Initial idea and implementation, never released.
Raw data
{
"_id": null,
"home_page": "https://github.com/paulross/pymemtrace",
"name": "pymemtrace",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "pymemtrace",
"author": "Paul Ross",
"author_email": "apaulross@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/95/d3/e049b9e1f8c1b61e221b8fc45f4743e366632720ca59578ff74f4a35b288/pymemtrace-0.2.0.tar.gz",
"platform": null,
"description": "*******************\nIntroduction\n*******************\n\n\n``pymemtrace`` provides tools for tracking and understanding Python memory usage at different levels, at different\ngranularities and with different runtime costs.\n\nFull documentation: https://pymemtrace.readthedocs.io\n\npymemtrace Tools\n======================\n\nThe tools provided by ``pymemtrace``:\n\n* ``process`` is a very lightweight way of logging the total memory usage at regular time intervals.\n It can plot memory over time with plotting programs such as ``gnuplot``.\n See `some process examples <https://pymemtrace.readthedocs.io/en/latest/examples/process.html>`_\n* ``cPyMemTrace`` is a memory tracer written in C that can report total memory usage for every function call/return for\n both C and Python sections.\n See some `cPyMemTrace examples <https://pymemtrace.readthedocs.io/en/latest/examples/c_py_mem_trace.html>`_\n and a `technical note on cPyMemTrace <https://pymemtrace.readthedocs.io/en/latest/tech_notes/cPyMemTrace.html>`_.\n* DTrace: Here are a number of D scripts that can trace the low level ``malloc()`` and ``free()`` system calls and\n report how much memory was allocated and by whom.\n See some `DTrace examples <https://pymemtrace.readthedocs.io/en/latest/examples/dtrace.html>`_\n and a `technical note on DTrace <https://pymemtrace.readthedocs.io/en/latest/tech_notes/dtrace.html>`_.\n* ``trace_malloc`` is a convenience wrapper around the Python standard library `tracemalloc` module.\n This can report Python memory usage by module and line compensating for the cost of ``tracemalloc``.\n This can take memory snapshots before and after code blocks and show the change on memory caused by that code.\n See some `trace_malloc examples <https://pymemtrace.readthedocs.io/en/latest/examples/trace_malloc.html>`_\n* ``debug_malloc_stats`` is a wrapper around the ``sys._debugmallocstats`` function that can take snapshots of\n memory before and after code execution and report the significant differences of the Python small object allocator.\n See some `debug_malloc_stats examples <https://pymemtrace.readthedocs.io/en/latest/examples/debug_malloc_stats.html>`_\n\n\nTool Characteristics\n======================\n\nEach tool can be characterised by:\n\n- *Memory Granularity*: In how much detail is a memory change is observed.\n An example of *coarse* memory granularity is measuring the\n `Resident Set Size <https://en.wikipedia.org/wiki/Resident_set_size>`_ which is normally in chunks of 4096 bytes.\n An example of *fine* memory granularity is recording every ``malloc()`` and ``free()``.\n- *Execution Granularity*: In how much code detail is the memory change observed.\n An example of *coarse* execution granularity is measuring the memory usage every second.\n An example of *fine* execution granularity is recording the memory usage every Python line.\n- *Memory Cost*: How much extra memory the tool needs.\n- *Execution Cost*: How much the execution time is increased.\n\nClearly there are trade-offs between these depending on the problem you are trying to solve.\n\n.. list-table:: **Tool Characteristics**\n :widths: 15 30 30 30 30\n :header-rows: 1\n\n * - Tool\n - Memory Granularity\n - Execution Granularity\n - Memory Cost\n - Execution Cost\n * - ``process``\n - RSS (total Python and C memory).\n - Regular time intervals.\n - Near zero.\n - Near zero.\n * - ``cPyMemTrace``\n - RSS (total Python and C memory).\n - Per Python line, Python function and C function call.\n - Near zero.\n - x10 to x20.\n * - DTrace\n - Every ``malloc()`` and ``free()``.\n - Per function call and return.\n - Minimal.\n - x90 to x100.\n * - ``trace_malloc``\n - Every Python object.\n - Per Python line, per function call.\n - Significant but compensated.\n - x900 for small objects, x6 for large objects.\n * - ``debug_malloc_stats``\n - Python memory pool.\n - Snapshots the CPython memory pool either side of a block of code.\n - Minimal.\n - x2000+ for small objects, x12 for large objects.\n\nPackage Metadata\n=========================\n\n.. image:: https://img.shields.io/pypi/v/pymemtrace.svg\n :target: https://pypi.python.org/pypi/pymemtrace\n\n.. image:: https://img.shields.io/travis/paulross/pymemtrace.svg\n :target: https://travis-ci.org/paulross/pymemtrace\n\n.. image:: https://readthedocs.org/projects/pymemtrace/badge/?version=latest\n :target: https://pymemtrace.readthedocs.io/en/latest/?badge=latest\n :alt: Documentation Status\n\n.. image:: https://pyup.io/repos/github/paulross/pymemtrace/shield.svg\n :target: https://pyup.io/repos/github/paulross/pymemtrace/\n :alt: Updates\n \n\nLicence\n-----------------------\n\nPython memory tracing.\n\n* Free software: MIT license\n* Documentation: https://pymemtrace.readthedocs.io.\n* Project: https://github.com/paulross/pymemtrace.\n\nCredits\n-----------------\n\nPhil Smith (AHL) with whom a casual lunch time chat lead to the creation of an earlier, but quite different\nimplementation, of ``cPyMemTrace`` in pure Python.\n\nThis package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.\n\n.. _Cookiecutter: https://github.com/audreyr/cookiecutter\n.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage\n\n\n\n=======\nHistory\n=======\n\n0.2.0 (2024-11-17)\n---------------------\n\n* cPyMemTrace:\n * Add P/T, stack depth and python version to log file name, example:\n \"20241107_195847_62264_P_0_PY3.13.0b3.log\"\n * Add stacking of trace/profile functions with linked list of tTraceFileWrapperLinkedList.\n * Add an option to log to a specific file.\n * Add an API write_to_log() to inject text into the log file.\n * Add an optional message to the log file in cPyMemTrace.\n * Add Python API to get log file being written to by cPyMemTrace.\n * Bug fixes in cPyMemTrace.c\n * Safety fix for file path name lengths.\n * Fix for log files where '#' was being concatenated.\n\n0.1.7 (2024-09-12)\n------------------\n\n* Minor fix for a single test.\n\n0.1.6 (2024-09-11)\n------------------\n\n* Add support for Python versions 3.12, 3.13. Now supports Python versions 3.7, 3.8, 3.9, 3.10, 3.11, 3.12, 3.13.\n\n0.1.5 (2023-06-21)\n------------------\n\n* Add support for Python versions 3.10, 3.11. Now supports Python versions 3.7, 3.8, 3.9, 3.10, 3.11.\n\n0.1.4 (2022-03-19)\n------------------\n\n* Fix Linux build.\n\n0.1.3 (2022-03-17)\n------------------\n\n* Fix some tests.\n\n0.1.2 (2022-03-17)\n------------------\n\n* Fix source distribution that had missing headers.\n\n0.1.1 (2020-11-17)\n------------------\n\n* Add cPyMemTrace the C level profiler.\n* Add DTrace scripts for low level tracing.\n* Add debug_malloc_stats the wrapper around sys._debugmallocstats.\n* Add process from the TotalDepth project.\n* Add redirect_stdout for debug_malloc_stats.\n* Add trace_malloc, a wrapper around the tracemalloc module.\n* Includes extensive documentation and performance measurement.\n* First release on PyPI.\n\n0.1.0 (2017-12-04)\n------------------\n\n* Initial idea and implementation, never released.\n",
"bugtrack_url": null,
"license": "MIT license",
"summary": "Python memory tracing.",
"version": "0.2.0",
"project_urls": {
"Homepage": "https://github.com/paulross/pymemtrace"
},
"split_keywords": [
"pymemtrace"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "accb20afafc78f7ee48a92986c349794b8f7e60007932be18f66fdf72b385419",
"md5": "667f791a6a3b7c060f6baba805d931e7",
"sha256": "e79ca3be49aaec471a3a0420202c9b75c4d9f07c440ab1ee90d32ad36f0f9382"
},
"downloads": -1,
"filename": "pymemtrace-0.2.0-cp310-cp310-macosx_10_9_universal2.whl",
"has_sig": false,
"md5_digest": "667f791a6a3b7c060f6baba805d931e7",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": null,
"size": 90015,
"upload_time": "2024-11-17T19:02:25",
"upload_time_iso_8601": "2024-11-17T19:02:25.808241Z",
"url": "https://files.pythonhosted.org/packages/ac/cb/20afafc78f7ee48a92986c349794b8f7e60007932be18f66fdf72b385419/pymemtrace-0.2.0-cp310-cp310-macosx_10_9_universal2.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "60b6406ee890d485ee6ed7ffa40c54c5cc182374eb8a5275d6e82c63b3401ff4",
"md5": "1817bb579d1dac1a53baf721af7890c9",
"sha256": "27d2df543860eb30dde9ba30d2d540101a3fdca20ad3cc88fc2014442dd67fe4"
},
"downloads": -1,
"filename": "pymemtrace-0.2.0-cp311-cp311-macosx_10_9_universal2.whl",
"has_sig": false,
"md5_digest": "1817bb579d1dac1a53baf721af7890c9",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": null,
"size": 90281,
"upload_time": "2024-11-17T19:02:27",
"upload_time_iso_8601": "2024-11-17T19:02:27.961860Z",
"url": "https://files.pythonhosted.org/packages/60/b6/406ee890d485ee6ed7ffa40c54c5cc182374eb8a5275d6e82c63b3401ff4/pymemtrace-0.2.0-cp311-cp311-macosx_10_9_universal2.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f119204e6da714d1ae30135393ccf8700b1bfbff7cf640bd28ed4c8f87ddb557",
"md5": "e012deb8d087dd47a1384d37e36c5d64",
"sha256": "499da61026e588ef44f25a9a977796be44dbe8402155efacfbdb52773c896d70"
},
"downloads": -1,
"filename": "pymemtrace-0.2.0-cp312-cp312-macosx_10_9_universal2.whl",
"has_sig": false,
"md5_digest": "e012deb8d087dd47a1384d37e36c5d64",
"packagetype": "bdist_wheel",
"python_version": "cp312",
"requires_python": null,
"size": 91425,
"upload_time": "2024-11-17T19:02:29",
"upload_time_iso_8601": "2024-11-17T19:02:29.812072Z",
"url": "https://files.pythonhosted.org/packages/f1/19/204e6da714d1ae30135393ccf8700b1bfbff7cf640bd28ed4c8f87ddb557/pymemtrace-0.2.0-cp312-cp312-macosx_10_9_universal2.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "31854dd464af110a4ea5850ad9284de78213f18da363d4f3444fabf2ccf8a5e4",
"md5": "d570867789d70d2b6ef7d569613cc0e3",
"sha256": "91bc7fb7805d403506afd2aad370c0b731ab2140b010657800cd870cd7afd35d"
},
"downloads": -1,
"filename": "pymemtrace-0.2.0-cp313-cp313-macosx_10_13_universal2.whl",
"has_sig": false,
"md5_digest": "d570867789d70d2b6ef7d569613cc0e3",
"packagetype": "bdist_wheel",
"python_version": "cp313",
"requires_python": null,
"size": 91418,
"upload_time": "2024-11-17T19:02:31",
"upload_time_iso_8601": "2024-11-17T19:02:31.517497Z",
"url": "https://files.pythonhosted.org/packages/31/85/4dd464af110a4ea5850ad9284de78213f18da363d4f3444fabf2ccf8a5e4/pymemtrace-0.2.0-cp313-cp313-macosx_10_13_universal2.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f1dc16bec9b57a317153dff557306924295bd19c8b7d356dfdb6f4e5988023e9",
"md5": "b81162476f3e83ef3cf0726b5e70f4d2",
"sha256": "d5b3efd27e5bccb32ea4ea872affdc320e30f464314d8ae1d792332c003667e6"
},
"downloads": -1,
"filename": "pymemtrace-0.2.0-cp37-cp37m-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "b81162476f3e83ef3cf0726b5e70f4d2",
"packagetype": "bdist_wheel",
"python_version": "cp37",
"requires_python": null,
"size": 72304,
"upload_time": "2024-11-17T19:02:33",
"upload_time_iso_8601": "2024-11-17T19:02:33.060277Z",
"url": "https://files.pythonhosted.org/packages/f1/dc/16bec9b57a317153dff557306924295bd19c8b7d356dfdb6f4e5988023e9/pymemtrace-0.2.0-cp37-cp37m-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b36e8c5a9744d597d014d42cefdc901921d3d403c9772d6fdf216bf08baa53d2",
"md5": "71d66ecc7fed59a238b4c57cc07947f6",
"sha256": "e8873a415d02c2a2a630f56c8f1fe0ce1cdad47b8f819645d98e5662cfb28bb5"
},
"downloads": -1,
"filename": "pymemtrace-0.2.0-cp38-cp38-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "71d66ecc7fed59a238b4c57cc07947f6",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": null,
"size": 72813,
"upload_time": "2024-11-17T19:02:34",
"upload_time_iso_8601": "2024-11-17T19:02:34.116697Z",
"url": "https://files.pythonhosted.org/packages/b3/6e/8c5a9744d597d014d42cefdc901921d3d403c9772d6fdf216bf08baa53d2/pymemtrace-0.2.0-cp38-cp38-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d5a03b2962369ea8b96fc6aea76390e1abc3006b6f2a88697956656a6e71b980",
"md5": "616b05f64bb9f1fc2320e75b0771126d",
"sha256": "870d4bec1d15d0235a1e4f1b4d7130565af13d344a6f5c502c76cbcaca780c9b"
},
"downloads": -1,
"filename": "pymemtrace-0.2.0-cp39-cp39-macosx_10_9_universal2.whl",
"has_sig": false,
"md5_digest": "616b05f64bb9f1fc2320e75b0771126d",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": null,
"size": 89934,
"upload_time": "2024-11-17T19:02:35",
"upload_time_iso_8601": "2024-11-17T19:02:35.858388Z",
"url": "https://files.pythonhosted.org/packages/d5/a0/3b2962369ea8b96fc6aea76390e1abc3006b6f2a88697956656a6e71b980/pymemtrace-0.2.0-cp39-cp39-macosx_10_9_universal2.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "95d3e049b9e1f8c1b61e221b8fc45f4743e366632720ca59578ff74f4a35b288",
"md5": "f931bfaf53dc7b86fa28feb41079d5cc",
"sha256": "d0a8c3b1131f5e6f19ca6215c1f924e95504a488991cf849a8466c6feb351f5a"
},
"downloads": -1,
"filename": "pymemtrace-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "f931bfaf53dc7b86fa28feb41079d5cc",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 377091,
"upload_time": "2024-11-17T19:02:36",
"upload_time_iso_8601": "2024-11-17T19:02:36.956289Z",
"url": "https://files.pythonhosted.org/packages/95/d3/e049b9e1f8c1b61e221b8fc45f4743e366632720ca59578ff74f4a35b288/pymemtrace-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-17 19:02:36",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "paulross",
"github_project": "pymemtrace",
"travis_ci": true,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "psutil",
"specs": []
},
{
"name": "pytest",
"specs": []
},
{
"name": "pytest-runner",
"specs": []
},
{
"name": "setuptools",
"specs": []
},
{
"name": "Sphinx",
"specs": []
}
],
"tox": true,
"lcname": "pymemtrace"
}
JSON
