*******************
Introduction
*******************
``pymemtrace`` provides tools for tracking and understanding Python memory usage at different levels, at different
granularities and with different runtime costs.
Full documentation: https://pymemtrace.readthedocs.io
pymemtrace Tools
======================
The tools provided by ``pymemtrace``:
* ``process`` is a very lightweight way of logging the total memory usage at regular time intervals.
It can plot memory over time with plotting programs such as ``gnuplot``.
See `some process examples <https://pymemtrace.readthedocs.io/en/latest/examples/process.html>`_
* ``cPyMemTrace`` is a memory tracer written in C that can report total memory usage for every function call/return for
both C and Python sections.
See some `cPyMemTrace examples <https://pymemtrace.readthedocs.io/en/latest/examples/c_py_mem_trace.html>`_
and a `technical note on cPyMemTrace <https://pymemtrace.readthedocs.io/en/latest/tech_notes/cPyMemTrace.html>`_.
* DTrace: Here are a number of D scripts that can trace the low level ``malloc()`` and ``free()`` system calls and
report how much memory was allocated and by whom.
See some `DTrace examples <https://pymemtrace.readthedocs.io/en/latest/examples/dtrace.html>`_
and a `technical note on DTrace <https://pymemtrace.readthedocs.io/en/latest/tech_notes/dtrace.html>`_.
* ``trace_malloc`` is a convenience wrapper around the Python standard library `tracemalloc` module.
This can report Python memory usage by module and line compensating for the cost of ``tracemalloc``.
This can take memory snapshots before and after code blocks and show the change on memory caused by that code.
See some `trace_malloc examples <https://pymemtrace.readthedocs.io/en/latest/examples/trace_malloc.html>`_
* ``debug_malloc_stats`` is a wrapper around the ``sys._debugmallocstats`` function that can take snapshots of
memory before and after code execution and report the significant differences of the Python small object allocator.
See some `debug_malloc_stats examples <https://pymemtrace.readthedocs.io/en/latest/examples/debug_malloc_stats.html>`_
Tool Characteristics
======================
Each tool can be characterised by:
- *Memory Granularity*: In how much detail is a memory change is observed.
An example of *coarse* memory granularity is measuring the
`Resident Set Size <https://en.wikipedia.org/wiki/Resident_set_size>`_ which is normally in chunks of 4096 bytes.
An example of *fine* memory granularity is recording every ``malloc()`` and ``free()``.
- *Execution Granularity*: In how much code detail is the memory change observed.
An example of *coarse* execution granularity is measuring the memory usage every second.
An example of *fine* execution granularity is recording the memory usage every Python line.
- *Memory Cost*: How much extra memory the tool needs.
- *Execution Cost*: How much the execution time is increased.
Clearly there are trade-offs between these depending on the problem you are trying to solve.
.. list-table:: **Tool Characteristics**
:widths: 15 30 30 30 30
:header-rows: 1
* - Tool
- Memory Granularity
- Execution Granularity
- Memory Cost
- Execution Cost
* - ``process``
- RSS (total Python and C memory).
- Regular time intervals.
- Near zero.
- Near zero.
* - ``cPyMemTrace``
- RSS (total Python and C memory).
- Per Python line, Python function and C function call.
- Near zero.
- x10 to x20.
* - DTrace
- Every ``malloc()`` and ``free()``.
- Per function call and return.
- Minimal.
- x90 to x100.
* - ``trace_malloc``
- Every Python object.
- Per Python line, per function call.
- Significant but compensated.
- x900 for small objects, x6 for large objects.
* - ``debug_malloc_stats``
- Python memory pool.
- Snapshots the CPython memory pool either side of a block of code.
- Minimal.
- x2000+ for small objects, x12 for large objects.
Package Metadata
=========================
.. image:: https://img.shields.io/pypi/v/pymemtrace.svg
:target: https://pypi.python.org/pypi/pymemtrace
.. image:: https://img.shields.io/travis/paulross/pymemtrace.svg
:target: https://travis-ci.org/paulross/pymemtrace
.. image:: https://readthedocs.org/projects/pymemtrace/badge/?version=latest
:target: https://pymemtrace.readthedocs.io/en/latest/?badge=latest
:alt: Documentation Status
.. image:: https://pyup.io/repos/github/paulross/pymemtrace/shield.svg
:target: https://pyup.io/repos/github/paulross/pymemtrace/
:alt: Updates
Licence
-----------------------
Python memory tracing.
* Free software: MIT license
* Documentation: https://pymemtrace.readthedocs.io.
* Project: https://github.com/paulross/pymemtrace.
Credits
-----------------
Phil Smith (AHL) with whom a casual lunch time chat lead to the creation of an earlier, but quite different
implementation, of ``cPyMemTrace`` in pure Python.
This package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.
.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage
=======
History
=======
0.1.5 (2023-06-21)
------------------
* Add support for Python versions 3.10, 3.11. Now supports Python versions 3.7, 3.8, 3.9, 3.10, 3.11.
0.1.4 (2022-03-19)
------------------
* Fix Linux build.
0.1.3 (2022-03-17)
------------------
* Fix some tests.
0.1.2 (2022-03-17)
------------------
* Fix source distribution that had missing headers.
0.1.1 (2020-11-17)
------------------
* Add cPyMemTrace the C level profiler.
* Add DTrace scripts for low level tracing.
* Add debug_malloc_stats the wrapper around sys._debugmallocstats.
* Add process from the TotalDepth project.
* Add redirect_stdout for debug_malloc_stats.
* Add trace_malloc, a wrapper around the tracemalloc module.
* Includes extensive documentation and performance measurement.
* First release on PyPI.
0.1.0 (2017-12-04)
------------------
* Initial idea and implementation, never released.
Raw data
{
"_id": null,
"home_page": "https://github.com/paulross/pymemtrace",
"name": "pymemtrace",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "pymemtrace",
"author": "Paul Ross",
"author_email": "apaulross@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/a4/8e/0c3645874cc529f2dd51e033dd7741d13ea206539c18653ee320255bbad9/pymemtrace-0.1.5.tar.gz",
"platform": null,
"description": "*******************\nIntroduction\n*******************\n\n\n``pymemtrace`` provides tools for tracking and understanding Python memory usage at different levels, at different\ngranularities and with different runtime costs.\n\nFull documentation: https://pymemtrace.readthedocs.io\n\npymemtrace Tools\n======================\n\nThe tools provided by ``pymemtrace``:\n\n* ``process`` is a very lightweight way of logging the total memory usage at regular time intervals.\n It can plot memory over time with plotting programs such as ``gnuplot``.\n See `some process examples <https://pymemtrace.readthedocs.io/en/latest/examples/process.html>`_\n* ``cPyMemTrace`` is a memory tracer written in C that can report total memory usage for every function call/return for\n both C and Python sections.\n See some `cPyMemTrace examples <https://pymemtrace.readthedocs.io/en/latest/examples/c_py_mem_trace.html>`_\n and a `technical note on cPyMemTrace <https://pymemtrace.readthedocs.io/en/latest/tech_notes/cPyMemTrace.html>`_.\n* DTrace: Here are a number of D scripts that can trace the low level ``malloc()`` and ``free()`` system calls and\n report how much memory was allocated and by whom.\n See some `DTrace examples <https://pymemtrace.readthedocs.io/en/latest/examples/dtrace.html>`_\n and a `technical note on DTrace <https://pymemtrace.readthedocs.io/en/latest/tech_notes/dtrace.html>`_.\n* ``trace_malloc`` is a convenience wrapper around the Python standard library `tracemalloc` module.\n This can report Python memory usage by module and line compensating for the cost of ``tracemalloc``.\n This can take memory snapshots before and after code blocks and show the change on memory caused by that code.\n See some `trace_malloc examples <https://pymemtrace.readthedocs.io/en/latest/examples/trace_malloc.html>`_\n* ``debug_malloc_stats`` is a wrapper around the ``sys._debugmallocstats`` function that can take snapshots of\n memory before and after code execution and report the significant differences of the Python small object allocator.\n See some `debug_malloc_stats examples <https://pymemtrace.readthedocs.io/en/latest/examples/debug_malloc_stats.html>`_\n\n\nTool Characteristics\n======================\n\nEach tool can be characterised by:\n\n- *Memory Granularity*: In how much detail is a memory change is observed.\n An example of *coarse* memory granularity is measuring the\n `Resident Set Size <https://en.wikipedia.org/wiki/Resident_set_size>`_ which is normally in chunks of 4096 bytes.\n An example of *fine* memory granularity is recording every ``malloc()`` and ``free()``.\n- *Execution Granularity*: In how much code detail is the memory change observed.\n An example of *coarse* execution granularity is measuring the memory usage every second.\n An example of *fine* execution granularity is recording the memory usage every Python line.\n- *Memory Cost*: How much extra memory the tool needs.\n- *Execution Cost*: How much the execution time is increased.\n\nClearly there are trade-offs between these depending on the problem you are trying to solve.\n\n.. list-table:: **Tool Characteristics**\n :widths: 15 30 30 30 30\n :header-rows: 1\n\n * - Tool\n - Memory Granularity\n - Execution Granularity\n - Memory Cost\n - Execution Cost\n * - ``process``\n - RSS (total Python and C memory).\n - Regular time intervals.\n - Near zero.\n - Near zero.\n * - ``cPyMemTrace``\n - RSS (total Python and C memory).\n - Per Python line, Python function and C function call.\n - Near zero.\n - x10 to x20.\n * - DTrace\n - Every ``malloc()`` and ``free()``.\n - Per function call and return.\n - Minimal.\n - x90 to x100.\n * - ``trace_malloc``\n - Every Python object.\n - Per Python line, per function call.\n - Significant but compensated.\n - x900 for small objects, x6 for large objects.\n * - ``debug_malloc_stats``\n - Python memory pool.\n - Snapshots the CPython memory pool either side of a block of code.\n - Minimal.\n - x2000+ for small objects, x12 for large objects.\n\nPackage Metadata\n=========================\n\n.. image:: https://img.shields.io/pypi/v/pymemtrace.svg\n :target: https://pypi.python.org/pypi/pymemtrace\n\n.. image:: https://img.shields.io/travis/paulross/pymemtrace.svg\n :target: https://travis-ci.org/paulross/pymemtrace\n\n.. image:: https://readthedocs.org/projects/pymemtrace/badge/?version=latest\n :target: https://pymemtrace.readthedocs.io/en/latest/?badge=latest\n :alt: Documentation Status\n\n.. image:: https://pyup.io/repos/github/paulross/pymemtrace/shield.svg\n :target: https://pyup.io/repos/github/paulross/pymemtrace/\n :alt: Updates\n \n\nLicence\n-----------------------\n\nPython memory tracing.\n\n* Free software: MIT license\n* Documentation: https://pymemtrace.readthedocs.io.\n* Project: https://github.com/paulross/pymemtrace.\n\nCredits\n-----------------\n\nPhil Smith (AHL) with whom a casual lunch time chat lead to the creation of an earlier, but quite different\nimplementation, of ``cPyMemTrace`` in pure Python.\n\nThis package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.\n\n.. _Cookiecutter: https://github.com/audreyr/cookiecutter\n.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage\n\n\n\n=======\nHistory\n=======\n\n0.1.5 (2023-06-21)\n------------------\n\n* Add support for Python versions 3.10, 3.11. Now supports Python versions 3.7, 3.8, 3.9, 3.10, 3.11.\n\n0.1.4 (2022-03-19)\n------------------\n\n* Fix Linux build.\n\n0.1.3 (2022-03-17)\n------------------\n\n* Fix some tests.\n\n0.1.2 (2022-03-17)\n------------------\n\n* Fix source distribution that had missing headers.\n\n0.1.1 (2020-11-17)\n------------------\n\n* Add cPyMemTrace the C level profiler.\n* Add DTrace scripts for low level tracing.\n* Add debug_malloc_stats the wrapper around sys._debugmallocstats.\n* Add process from the TotalDepth project.\n* Add redirect_stdout for debug_malloc_stats.\n* Add trace_malloc, a wrapper around the tracemalloc module.\n* Includes extensive documentation and performance measurement.\n* First release on PyPI.\n\n0.1.0 (2017-12-04)\n------------------\n\n* Initial idea and implementation, never released.\n",
"bugtrack_url": null,
"license": "MIT license",
"summary": "Python memory tracing.",
"version": "0.1.5",
"project_urls": {
"Homepage": "https://github.com/paulross/pymemtrace"
},
"split_keywords": [
"pymemtrace"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "59f3153b235ece0874007bd6be68ff8b81178a70fb0d06428837285b3df4a7c8",
"md5": "d66b7b53d46a05b4205c4779dac67206",
"sha256": "3ff239bf8d944751f579e9955dda56730733beae9212acf21077fc82e1a3d90a"
},
"downloads": -1,
"filename": "pymemtrace-0.1.5-cp310-cp310-macosx_10_9_universal2.whl",
"has_sig": false,
"md5_digest": "d66b7b53d46a05b4205c4779dac67206",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": null,
"size": 60573,
"upload_time": "2023-06-21T12:14:21",
"upload_time_iso_8601": "2023-06-21T12:14:21.549066Z",
"url": "https://files.pythonhosted.org/packages/59/f3/153b235ece0874007bd6be68ff8b81178a70fb0d06428837285b3df4a7c8/pymemtrace-0.1.5-cp310-cp310-macosx_10_9_universal2.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9524a268a133ffd6f740617bf082cb453277484fa32b726db85d0ead39195c5e",
"md5": "037ac448f5c2ff9ccb0d2ae2009b6a3e",
"sha256": "3750640326efaed60556a0aabb1cb434c0714341c71feccbcce5fa1699428556"
},
"downloads": -1,
"filename": "pymemtrace-0.1.5-cp311-cp311-macosx_10_9_universal2.whl",
"has_sig": false,
"md5_digest": "037ac448f5c2ff9ccb0d2ae2009b6a3e",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": null,
"size": 60594,
"upload_time": "2023-06-21T12:14:23",
"upload_time_iso_8601": "2023-06-21T12:14:23.845616Z",
"url": "https://files.pythonhosted.org/packages/95/24/a268a133ffd6f740617bf082cb453277484fa32b726db85d0ead39195c5e/pymemtrace-0.1.5-cp311-cp311-macosx_10_9_universal2.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "12cb972e8356c294c4ab64e57c9458659b08af55d43519d7722c6534ce16b6e5",
"md5": "57c4298957393d90326b8c38b9aafdf6",
"sha256": "176bee8fd7cf61aa14a5e0a9f8d1e6b21fad6cdb06b1b8644554c92b9eb2c177"
},
"downloads": -1,
"filename": "pymemtrace-0.1.5-cp37-cp37m-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "57c4298957393d90326b8c38b9aafdf6",
"packagetype": "bdist_wheel",
"python_version": "cp37",
"requires_python": null,
"size": 60322,
"upload_time": "2023-06-21T12:14:25",
"upload_time_iso_8601": "2023-06-21T12:14:25.565773Z",
"url": "https://files.pythonhosted.org/packages/12/cb/972e8356c294c4ab64e57c9458659b08af55d43519d7722c6534ce16b6e5/pymemtrace-0.1.5-cp37-cp37m-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "2208c5570c08a2a80afb6c1e9bb5e2e6b91c7cbfaa47e6d22f728ad744f68683",
"md5": "5d35e1ec06c90f03404681886c3a3d14",
"sha256": "fae21cc8bece6cbce24b127c557fbff09b04d5b4e495ba81462cf651a91388a6"
},
"downloads": -1,
"filename": "pymemtrace-0.1.5-cp38-cp38-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "5d35e1ec06c90f03404681886c3a3d14",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": null,
"size": 60553,
"upload_time": "2023-06-21T12:14:27",
"upload_time_iso_8601": "2023-06-21T12:14:27.477535Z",
"url": "https://files.pythonhosted.org/packages/22/08/c5570c08a2a80afb6c1e9bb5e2e6b91c7cbfaa47e6d22f728ad744f68683/pymemtrace-0.1.5-cp38-cp38-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "370c50edaeac21bb9553f857c132aebceb0e30b9dccdd285d28e90a9a154a9ed",
"md5": "bbc22c3a2c98063ff1c1d3047e8e3654",
"sha256": "d04b3ce2153d591d5be63d2c674b98c06919467bda81e0150438f683c3c1bd2e"
},
"downloads": -1,
"filename": "pymemtrace-0.1.5-cp39-cp39-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "bbc22c3a2c98063ff1c1d3047e8e3654",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": null,
"size": 60552,
"upload_time": "2023-06-21T12:14:29",
"upload_time_iso_8601": "2023-06-21T12:14:29.345515Z",
"url": "https://files.pythonhosted.org/packages/37/0c/50edaeac21bb9553f857c132aebceb0e30b9dccdd285d28e90a9a154a9ed/pymemtrace-0.1.5-cp39-cp39-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a48e0c3645874cc529f2dd51e033dd7741d13ea206539c18653ee320255bbad9",
"md5": "039bffdc6edc63b1b385bd404f43a265",
"sha256": "9ea84032cb3821225acc494d3cad133e176a24a4b7f00fc4e07b7bf8185e7611"
},
"downloads": -1,
"filename": "pymemtrace-0.1.5.tar.gz",
"has_sig": false,
"md5_digest": "039bffdc6edc63b1b385bd404f43a265",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 376578,
"upload_time": "2023-06-21T12:14:31",
"upload_time_iso_8601": "2023-06-21T12:14:31.279375Z",
"url": "https://files.pythonhosted.org/packages/a4/8e/0c3645874cc529f2dd51e033dd7741d13ea206539c18653ee320255bbad9/pymemtrace-0.1.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-06-21 12:14:31",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "paulross",
"github_project": "pymemtrace",
"travis_ci": true,
"coveralls": false,
"github_actions": false,
"requirements": [],
"tox": true,
"lcname": "pymemtrace"
}
JSON
