[](https://pypi.org/project/mprofile/)
[](https://travis-ci.org/timpalpant/mprofile)
[](https://pypistats.org/packages/mprofile)
[](https://github.com/psf/black)
# mprofile
A low-overhead sampling memory profiler for Python, derived from [heapprof](https://github.com/humu/heapprof), with an interface similar to [tracemalloc](https://pytracemalloc.readthedocs.io).
mprofile attempts to give results comparable to tracemalloc, but uses statistical sampling to lower memory and CPU overhead. The sampling algorithm is the one used by [tcmalloc](https://github.com/gperftools/gperftools) and Golang heap profilers.
## Installation & usage
1. Install the profiler package using PyPI:
```shell
pip3 install mprofile
```
2. Enable the profiler in your application, get a snapshot of (sampled) memory usage:
```python
import mprofile
mprofile.start(sample_rate=128 * 1024)
snap = mprofile.take_snapshot()
```
See the [tracemalloc](https://docs.python.org/3/library/tracemalloc.html) for API documentation. The API and objects returned by mprofile are compatible.
## Compatibility
mprofile is compatible with Python >= 3.4.
It can also be used with earlier versions of Python, but you must build CPython from source and apply the [pytracemalloc patches](https://pytracemalloc.readthedocs.io/install.html#manual-installation).
## Benchmarks
We are primarily interested in profiling the memory usage of webservers, so used the `tornado_http` benchmark from pyperformance to estimate overhead.
mprofile has similar performance to tracemalloc when comprehensively tracing all allocations, but when statistical sampling is used, the overhead is significantly reduced.
In addition, mprofile interns call stacks in a tree data structure that reduces memory overhead of storing the traces.
With the recommended setting of `sample_rate=128kB`, we observe ~5% slow down in the `tornado_http` benchmark.
TODO: Run the full [pyperformance](https://pyperformance.readthedocs.io) suite of benchmarks.
### Baseline
```
Python 2.7.16, no profiling:
tornado_http: Mean +- std dev: 664 ms +- 30 ms
Maximum resident set size (kbytes): 39176
```
### tracemalloc
```
Python 2.7.16, tracemallocframes=128:
tornado_http: Mean +- std dev: 1.74 sec +- 0.04 sec
Maximum resident set size (kbytes): 43752
# Saving only one frame in each stack trace rather than full call stacks.
Python 2.7.16, tracemallocframes=1:
tornado_http: Mean +- std dev: 960 ms +- 30 ms
Maximum resident set size (kbytes): 40000
```
### mprofile
```
Python 2.7.16, mprofileframes=128, mprofilerate=1 (i.e. tracemalloc):
tornado_http: Mean +- std dev: 1.78 sec +- 0.05 sec
Maximum resident set size (kbytes): 40588
Python 2.7.16, mprofileframes=128, mprofilerate=1024:
tornado_http: Mean +- std dev: 888 ms +- 28 ms
Maximum resident set size (kbytes): 39752
Python 2.7.16, mprofileframes=128, mprofilerate=128 * 1024:
tornado_http: Mean +- std dev: 700 ms +- 26 ms
Maximum resident set size (kbytes): 39388
# Saving only one frame in each stack trace rather than full call stacks.
Python 2.7.16, mprofileframes=1, mprofilerate=1 (i.e. tracemalloc):
tornado_http: Mean +- std dev: 890 ms +- 19 ms
Maximum resident set size (kbytes): 40152
Python 2.7.16, mprofileframes=1, mprofilerate=1024:
tornado_http: Mean +- std dev: 738 ms +- 24 ms
Maximum resident set size (kbytes): 39568
Python 2.7.16, mprofileframes=1, mprofilerate=128 * 1024:
tornado_http: Mean +- std dev: 678 ms +- 22 ms
Maximum resident set size (kbytes): 39328
```
## Developer notes
Run the unit tests:
```
bazel test --test_output=streamed //src:profiler_test
```
Run the benchmarks:
```
bazel test -c opt --test_output=streamed //src:profiler_bench
```
Run the end-to-end (Python) tests:
```
bazel test --config asan --test_output=streamed //test:*
```
Run tests with ASAN and UBSAN:
```
bazel test --config asan --test_output=streamed //src:* //test:*
```
# Contributing
Pull requests and issues are welcomed!
# License
mprofile is released under the [MIT License](https://opensource.org/licenses/MIT) and incorporates code from [heapprof](https://github.com/humu/heapprof), which is also released under the MIT license.
Raw data
{
"_id": null,
"home_page": "http://github.com/timpalpant/mprofile",
"name": "mprofile",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "profiling performance",
"author": "Timothy Palpant",
"author_email": "tim@palpant.us",
"download_url": "https://files.pythonhosted.org/packages/f9/06/a2a99bd3caf8daa6ce17154a19a2e05412496379f49cf28f654033072d4c/mprofile-0.0.15.tar.gz",
"platform": "Mac OS X",
"description": "[](https://pypi.org/project/mprofile/)\n[](https://travis-ci.org/timpalpant/mprofile)\n\n\n[](https://pypistats.org/packages/mprofile)\n[](https://github.com/psf/black)\n\n# mprofile\n\nA low-overhead sampling memory profiler for Python, derived from [heapprof](https://github.com/humu/heapprof), with an interface similar to [tracemalloc](https://pytracemalloc.readthedocs.io).\nmprofile attempts to give results comparable to tracemalloc, but uses statistical sampling to lower memory and CPU overhead. The sampling algorithm is the one used by [tcmalloc](https://github.com/gperftools/gperftools) and Golang heap profilers.\n\n## Installation & usage\n\n1. Install the profiler package using PyPI:\n\n ```shell\n pip3 install mprofile\n ```\n\n2. Enable the profiler in your application, get a snapshot of (sampled) memory usage:\n\n ```python\n import mprofile\n\n mprofile.start(sample_rate=128 * 1024)\n snap = mprofile.take_snapshot()\n ```\n\nSee the [tracemalloc](https://docs.python.org/3/library/tracemalloc.html) for API documentation. The API and objects returned by mprofile are compatible.\n\n## Compatibility\n\nmprofile is compatible with Python >= 3.4.\nIt can also be used with earlier versions of Python, but you must build CPython from source and apply the [pytracemalloc patches](https://pytracemalloc.readthedocs.io/install.html#manual-installation).\n\n## Benchmarks\n\nWe are primarily interested in profiling the memory usage of webservers, so used the `tornado_http` benchmark from pyperformance to estimate overhead.\nmprofile has similar performance to tracemalloc when comprehensively tracing all allocations, but when statistical sampling is used, the overhead is significantly reduced.\nIn addition, mprofile interns call stacks in a tree data structure that reduces memory overhead of storing the traces.\n\nWith the recommended setting of `sample_rate=128kB`, we observe ~5% slow down in the `tornado_http` benchmark.\n\nTODO: Run the full [pyperformance](https://pyperformance.readthedocs.io) suite of benchmarks.\n\n### Baseline\n```\nPython 2.7.16, no profiling:\ntornado_http: Mean +- std dev: 664 ms +- 30 ms\nMaximum resident set size (kbytes): 39176\n```\n\n### tracemalloc\n```\nPython 2.7.16, tracemallocframes=128:\ntornado_http: Mean +- std dev: 1.74 sec +- 0.04 sec\nMaximum resident set size (kbytes): 43752\n\n# Saving only one frame in each stack trace rather than full call stacks.\nPython 2.7.16, tracemallocframes=1:\ntornado_http: Mean +- std dev: 960 ms +- 30 ms\nMaximum resident set size (kbytes): 40000\n```\n\n### mprofile\n```\nPython 2.7.16, mprofileframes=128, mprofilerate=1 (i.e. tracemalloc):\ntornado_http: Mean +- std dev: 1.78 sec +- 0.05 sec\nMaximum resident set size (kbytes): 40588\n\nPython 2.7.16, mprofileframes=128, mprofilerate=1024:\ntornado_http: Mean +- std dev: 888 ms +- 28 ms\nMaximum resident set size (kbytes): 39752\n\nPython 2.7.16, mprofileframes=128, mprofilerate=128 * 1024:\ntornado_http: Mean +- std dev: 700 ms +- 26 ms\nMaximum resident set size (kbytes): 39388\n\n# Saving only one frame in each stack trace rather than full call stacks.\nPython 2.7.16, mprofileframes=1, mprofilerate=1 (i.e. tracemalloc):\ntornado_http: Mean +- std dev: 890 ms +- 19 ms\nMaximum resident set size (kbytes): 40152\n\nPython 2.7.16, mprofileframes=1, mprofilerate=1024:\ntornado_http: Mean +- std dev: 738 ms +- 24 ms\nMaximum resident set size (kbytes): 39568\n\nPython 2.7.16, mprofileframes=1, mprofilerate=128 * 1024:\ntornado_http: Mean +- std dev: 678 ms +- 22 ms\nMaximum resident set size (kbytes): 39328\n```\n\n## Developer notes\n\nRun the unit tests:\n```\nbazel test --test_output=streamed //src:profiler_test\n```\n\nRun the benchmarks:\n```\nbazel test -c opt --test_output=streamed //src:profiler_bench\n```\n\nRun the end-to-end (Python) tests:\n```\nbazel test --config asan --test_output=streamed //test:*\n```\n\nRun tests with ASAN and UBSAN:\n```\nbazel test --config asan --test_output=streamed //src:* //test:*\n```\n\n# Contributing\n\nPull requests and issues are welcomed!\n\n# License\n\nmprofile is released under the [MIT License](https://opensource.org/licenses/MIT) and incorporates code from [heapprof](https://github.com/humu/heapprof), which is also released under the MIT license.\n\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A low-overhead memory profiler.",
"version": "0.0.15",
"split_keywords": [
"profiling",
"performance"
],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "3c2d5b145b26f85cccdd9304c1c3e200",
"sha256": "31c893321be93910e80052d0e7dfc19c8083db153cd8b2bb31498dcd920a7da9"
},
"downloads": -1,
"filename": "mprofile-0.0.15-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "3c2d5b145b26f85cccdd9304c1c3e200",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": null,
"size": 833844,
"upload_time": "2023-01-02T16:30:44",
"upload_time_iso_8601": "2023-01-02T16:30:44.681919Z",
"url": "https://files.pythonhosted.org/packages/67/c9/909a46d1573198685be8fb3590396e02bc2cb195668c5b11ea7d86bb21c9/mprofile-0.0.15-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"md5": "6344b2925219c21b02e62af286b385b2",
"sha256": "e33edf1d469723d33184e14a757aa3b202f54afb991da1d0b7f7fba5c49aef27"
},
"downloads": -1,
"filename": "mprofile-0.0.15-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "6344b2925219c21b02e62af286b385b2",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": null,
"size": 835407,
"upload_time": "2023-01-02T16:30:46",
"upload_time_iso_8601": "2023-01-02T16:30:46.218280Z",
"url": "https://files.pythonhosted.org/packages/69/50/e859b298acda9e01fe018826d8961d6ac3b4a32b499ded1c9cc49793c131/mprofile-0.0.15-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"md5": "03e2c3b16171f62080d0e81a4170afbf",
"sha256": "c544275c4d1373696987bedb516a36e56600ef43ab0920cf3fb53f18344190a6"
},
"downloads": -1,
"filename": "mprofile-0.0.15-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "03e2c3b16171f62080d0e81a4170afbf",
"packagetype": "bdist_wheel",
"python_version": "cp36",
"requires_python": null,
"size": 832802,
"upload_time": "2023-01-02T16:30:47",
"upload_time_iso_8601": "2023-01-02T16:30:47.881365Z",
"url": "https://files.pythonhosted.org/packages/57/c5/a82e119d6f3fe374a224c1dfc39081402fb78bc1ae95dd1d779162961c12/mprofile-0.0.15-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"md5": "cf8f4e62dc0e61af1a6fdaf433774a90",
"sha256": "2bd8efc8d898d28181c404bfe304f13fd87ab8a439b6034235b5ea0f25670127"
},
"downloads": -1,
"filename": "mprofile-0.0.15-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "cf8f4e62dc0e61af1a6fdaf433774a90",
"packagetype": "bdist_wheel",
"python_version": "cp37",
"requires_python": null,
"size": 832662,
"upload_time": "2023-01-02T16:30:49",
"upload_time_iso_8601": "2023-01-02T16:30:49.618880Z",
"url": "https://files.pythonhosted.org/packages/6a/2f/3ea66c1fda13de4efbaab2d8522a7700112595e2103d4be22d303502ede4/mprofile-0.0.15-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"md5": "677471ab74ba6356733fc8d097516a0e",
"sha256": "88218635ec951f1e449e23e96e0bf40b43a2ae3ca3b5c9f2506eb992515fca52"
},
"downloads": -1,
"filename": "mprofile-0.0.15-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "677471ab74ba6356733fc8d097516a0e",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": null,
"size": 833580,
"upload_time": "2023-01-02T16:30:51",
"upload_time_iso_8601": "2023-01-02T16:30:51.338730Z",
"url": "https://files.pythonhosted.org/packages/71/b5/1db1430c5aeb79aae47927cb2a82f551e5d12ddbd7e5f23b95a93aa8bf95/mprofile-0.0.15-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"md5": "7832a6c7083a830eb821bc0f0eeb0d44",
"sha256": "6f5d4d7fad4005abae0f16f76ee234cffdad7cb9d763a103672b5ea6cd3f732d"
},
"downloads": -1,
"filename": "mprofile-0.0.15-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "7832a6c7083a830eb821bc0f0eeb0d44",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": null,
"size": 833201,
"upload_time": "2023-01-02T16:30:53",
"upload_time_iso_8601": "2023-01-02T16:30:53.228385Z",
"url": "https://files.pythonhosted.org/packages/37/79/d3cf2183fcae42197b576ce4fecfbe29226e3862318d2471f8b42e08ada2/mprofile-0.0.15-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"md5": "8ccfa9c642d8648adf30aa7fd3d2f9bd",
"sha256": "fb21d99dd99bc4672f057ac53bc6bd2ddc2139a18b83bc4565a0303ebd5b7efd"
},
"downloads": -1,
"filename": "mprofile-0.0.15.tar.gz",
"has_sig": false,
"md5_digest": "8ccfa9c642d8648adf30aa7fd3d2f9bd",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 126132,
"upload_time": "2023-01-02T16:30:34",
"upload_time_iso_8601": "2023-01-02T16:30:34.208671Z",
"url": "https://files.pythonhosted.org/packages/f9/06/a2a99bd3caf8daa6ce17154a19a2e05412496379f49cf28f654033072d4c/mprofile-0.0.15.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-01-02 16:30:34",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "timpalpant",
"github_project": "mprofile",
"travis_ci": true,
"coveralls": false,
"github_actions": false,
"lcname": "mprofile"
}
JSON
