DAMO: Data Access Monitoring Operator
=====================================
`damo` is a user space tool for [DAMON](https://damonitor.github.io). Using
this, you can monitor the data access patterns of your system or workloads and
make data access-aware memory management optimizations.
Official Repositories
---------------------
Official git repos for `damo` are hosted on kernel.org and GitHub:
- kernel.org: https://git.kernel.org/pub/scm/linux/kernel/git/sj/damo.git
- GitHub: https://github.com/damonitor/damo
_Note_ that the repo under awslabs GitHub organization
(https://github.com/awslabs/damo)
[was](https://lore.kernel.org/20240813232158.83903-1-sj@kernel.org) one of the
official repos. It is not an official repo since 2024-09-05.
Demo Video
==========
Please click the below thumbnail to show the short demo video.
[](
http://www.youtube.com/watch?v=l63eqbVBZRY
"DAMON: a demo for the Kernel Summit 2020")
Getting Started
===============
[](https://repology.org/project/damo/versions)
Follow below instructions and commands to monitor and visualize the access
pattern of your workload.
$ # ensure DAMON is enabled on your kernel
$ # install damo using packaging systems listed above,
$ # or cloning the source repo and updating $PATH.
$ sudo damo start $(pidof <your workload>)
$ sudo damo report access
$ sudo damo record ongoing
$ sudo damo report heatmaps --heatmap stdout --stdout_colorset emotion
The second and last commands will show the access pattern of your workload,
like below:
FAQs
====
How can I ensure DAMON is enabled on my kernel?
-----------------------------------------------
Please refer to 'Install'
[section](https://damonitor.github.io/#install) of the project webpage.
`damo` aims to support any Linux kernel that enables DAMON, regardless of the
version and downstream changes. Of course, `damo` will be unable to support
some of such kernels in some special cases. That said, please reach out to the
development community for discussions.
Where can I get more detailed usage?
------------------------------------
The below sections provide quick introductions for `damo`'s major features.
For more detailed usage, please refer to [USAGE.md](https://github.com/damonitor/damo/blob/v2.6.0/USAGE.md) file.
What does the version numbers mean?
-----------------------------------
Nothing at all but indicate which version is more fresh. A higher version
number means it is more recently released.
Will the features of `damo` be supported forever?
-------------------------------------------------
We try our best to make `damo` stable and doesn't introduce regressions to
users. However, nothing goes forever. Sometimes, some features will be
deprecated. Some features will have longer support more than others.
In short, features that documented on [USAGE.md](https://github.com/damonitor/damo/blob/v2.6.0/USAGE.md) and not explicitly
marked as experimental, the feature will be better supported, and provides at
least three months of deprecation grace period. Within the grace period, users
can ask extension of the support.
Detailed features deprecation process is documented at
[FEATURES_DEPRECATION_PROCESS.md](FEATURES_DEPRECATION_PROCESS.md). Schedules
and status of deprecations are updated on
[FEATURES_DEPRECATION_SCHEDULE.md](FEATURES_DEPRECATION_SCHEDULE.md).
How can I participate in the development of `damo`?
---------------------------------------------------
Please refer to
[CONTRIBUTING](https://github.com/damonitor/damo/blob/next/CONTRIBUTING) file.
Quick Intro for Major Features
==============================
Below are quick introductions for `damo`'s major features.
For more detailed usage, please refer to [USAGE.md](https://github.com/damonitor/damo/blob/v2.6.0/USAGE.md) file.
Snapshot Data Access Pattern
----------------------------
Below commands repeatedly get a snapshot of the access pattern of a program for
every second.
$ git clone https://github.com/sjp38/masim
$ cd masim; make; ./masim ./configs/zigzag.cfg --silent &
$ sudo damo start --target_pid $(pidof masim)
$ while :; do sudo damo report access; sleep 1; done
The first two lines of the commands get an artificial memory access generator
program and run it in the background. It will repeatedly access two 100
MiB-sized memory regions one by one. You can substitute this with your real
workload.
The third line asks ``damo`` to start monitoring the access pattern of the
process. Finally, the last line retries a snapshot of the monitoring results
every second and show on terminal.
Recording Data Access Patterns
------------------------------
Below commands record memory access patterns of a program and save the
monitoring results in `damon.data` file.
$ git clone https://github.com/sjp38/masim
$ cd masim; make; ./masim ./configs/zigzag.cfg --silent &
$ sudo damo record -o damon.data $(pidof masim)
The first two lines of the commands get an artificial memory access generator
program and run it in the background. It will repeatedly access two 100
MiB-sized memory regions one by one. You can substitute this with your real
workload. The last line asks ``damo`` to record the access pattern in
``damon.data`` file.
Visualizing Recorded Patterns
-----------------------------
Below three commands visualize the recorded access patterns into three
image files.
$ damo report heatmap
$ damo report wss --range 0 101 1
$ damo report wss --range 0 101 1 --sortby time --plot
- ``access_pattern_heatmap.png`` will show the data access pattern in a
heatmap, which shows when (x-axis) what memory region (y-axis) is how
frequently accessed (color).
- ``wss_dist.png`` will show the distribution of the working set size.
- ``wss_chron_change.png`` will show how the working set size has
chronologically changed.
You can show the images on a web page [1]. Those made with other realistic
workloads are also available [2,3,4].
[1] https://damonitor.github.io/doc/html/latest/admin-guide/mm/damon/start.html#visualizing-recorded-patterns<br>
[2] https://damonitor.github.io/test/result/visual/latest/rec.heatmap.1.png.html<br>
[3] https://damonitor.github.io/test/result/visual/latest/rec.wss_sz.png.html<br>
[4] https://damonitor.github.io/test/result/visual/latest/rec.wss_time.png.html
Data Access Pattern Aware Memory Management
-------------------------------------------
Below command makes every memory region of size >=4K that hasn't accessed for
>=60 seconds in your workload to be swapped out. By doing this, you can make
your workload more memory efficient with only modest performance overhead.
$ sudo damo start --damos_access_rate 0 0 --damos_sz_region 4K max \
--damos_age 60s max --damos_action pageout \
<pid of your workload>
Holistic Memory Usage Monitoring
--------------------------------
You can also show access pattern heatmap, memory footprints, and hotspot
functions of the system and/or workloads all at once in live, like below.
Raw data
{
"_id": null,
"home_page": "https://github.com/damonitor/damo",
"name": "damo",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": null,
"keywords": null,
"author": "SeongJae Park",
"author_email": "sj@kernel.org",
"download_url": "https://files.pythonhosted.org/packages/b9/a9/e72bdb020a7871e1d98f59a86d5f9b0a373b6f55999c9d8b7d19a63c83b9/damo-2.6.0.tar.gz",
"platform": null,
"description": "DAMO: Data Access Monitoring Operator\n=====================================\n\n`damo` is a user space tool for [DAMON](https://damonitor.github.io). Using\nthis, you can monitor the data access patterns of your system or workloads and\nmake data access-aware memory management optimizations.\n\n\n\nOfficial Repositories\n---------------------\n\nOfficial git repos for `damo` are hosted on kernel.org and GitHub:\n\n- kernel.org: https://git.kernel.org/pub/scm/linux/kernel/git/sj/damo.git\n- GitHub: https://github.com/damonitor/damo\n\n_Note_ that the repo under awslabs GitHub organization\n(https://github.com/awslabs/damo)\n[was](https://lore.kernel.org/20240813232158.83903-1-sj@kernel.org) one of the\nofficial repos. It is not an official repo since 2024-09-05.\n\n\nDemo Video\n==========\n\nPlease click the below thumbnail to show the short demo video.\n\n[](\nhttp://www.youtube.com/watch?v=l63eqbVBZRY\n\"DAMON: a demo for the Kernel Summit 2020\")\n\n\nGetting Started\n===============\n\n[](https://repology.org/project/damo/versions)\n\nFollow below instructions and commands to monitor and visualize the access\npattern of your workload.\n\n $ # ensure DAMON is enabled on your kernel\n $ # install damo using packaging systems listed above,\n $ # or cloning the source repo and updating $PATH.\n $ sudo damo start $(pidof <your workload>)\n $ sudo damo report access\n $ sudo damo record ongoing\n $ sudo damo report heatmaps --heatmap stdout --stdout_colorset emotion\n\nThe second and last commands will show the access pattern of your workload,\nlike below:\n\n\n\n\n\nFAQs\n====\n\nHow can I ensure DAMON is enabled on my kernel?\n-----------------------------------------------\n\nPlease refer to 'Install'\n[section](https://damonitor.github.io/#install) of the project webpage.\n\n`damo` aims to support any Linux kernel that enables DAMON, regardless of the\nversion and downstream changes. Of course, `damo` will be unable to support\nsome of such kernels in some special cases. That said, please reach out to the\ndevelopment community for discussions.\n\n\nWhere can I get more detailed usage?\n------------------------------------\n\nThe below sections provide quick introductions for `damo`'s major features.\nFor more detailed usage, please refer to [USAGE.md](https://github.com/damonitor/damo/blob/v2.6.0/USAGE.md) file.\n\n\nWhat does the version numbers mean?\n-----------------------------------\n\nNothing at all but indicate which version is more fresh. A higher version\nnumber means it is more recently released.\n\n\nWill the features of `damo` be supported forever?\n-------------------------------------------------\n\nWe try our best to make `damo` stable and doesn't introduce regressions to\nusers. However, nothing goes forever. Sometimes, some features will be\ndeprecated. Some features will have longer support more than others.\n\nIn short, features that documented on [USAGE.md](https://github.com/damonitor/damo/blob/v2.6.0/USAGE.md) and not explicitly\nmarked as experimental, the feature will be better supported, and provides at\nleast three months of deprecation grace period. Within the grace period, users\ncan ask extension of the support.\n\nDetailed features deprecation process is documented at\n[FEATURES_DEPRECATION_PROCESS.md](FEATURES_DEPRECATION_PROCESS.md). Schedules\nand status of deprecations are updated on\n[FEATURES_DEPRECATION_SCHEDULE.md](FEATURES_DEPRECATION_SCHEDULE.md).\n\n\nHow can I participate in the development of `damo`?\n---------------------------------------------------\n\nPlease refer to\n[CONTRIBUTING](https://github.com/damonitor/damo/blob/next/CONTRIBUTING) file.\n\n\nQuick Intro for Major Features\n==============================\n\nBelow are quick introductions for `damo`'s major features.\nFor more detailed usage, please refer to [USAGE.md](https://github.com/damonitor/damo/blob/v2.6.0/USAGE.md) file.\n\n\nSnapshot Data Access Pattern\n----------------------------\n\nBelow commands repeatedly get a snapshot of the access pattern of a program for\nevery second.\n\n $ git clone https://github.com/sjp38/masim\n $ cd masim; make; ./masim ./configs/zigzag.cfg --silent &\n $ sudo damo start --target_pid $(pidof masim)\n $ while :; do sudo damo report access; sleep 1; done\n\nThe first two lines of the commands get an artificial memory access generator\nprogram and run it in the background. It will repeatedly access two 100\nMiB-sized memory regions one by one. You can substitute this with your real\nworkload.\n\nThe third line asks ``damo`` to start monitoring the access pattern of the\nprocess. Finally, the last line retries a snapshot of the monitoring results\nevery second and show on terminal.\n\n\nRecording Data Access Patterns\n------------------------------\n\nBelow commands record memory access patterns of a program and save the\nmonitoring results in `damon.data` file.\n\n $ git clone https://github.com/sjp38/masim\n $ cd masim; make; ./masim ./configs/zigzag.cfg --silent &\n $ sudo damo record -o damon.data $(pidof masim)\n\nThe first two lines of the commands get an artificial memory access generator\nprogram and run it in the background. It will repeatedly access two 100\nMiB-sized memory regions one by one. You can substitute this with your real\nworkload. The last line asks ``damo`` to record the access pattern in\n``damon.data`` file.\n\n\nVisualizing Recorded Patterns\n-----------------------------\n\nBelow three commands visualize the recorded access patterns into three\nimage files.\n\n $ damo report heatmap\n $ damo report wss --range 0 101 1\n $ damo report wss --range 0 101 1 --sortby time --plot\n\n- ``access_pattern_heatmap.png`` will show the data access pattern in a\n heatmap, which shows when (x-axis) what memory region (y-axis) is how\n frequently accessed (color).\n- ``wss_dist.png`` will show the distribution of the working set size.\n- ``wss_chron_change.png`` will show how the working set size has\n chronologically changed.\n\nYou can show the images on a web page [1]. Those made with other realistic\nworkloads are also available [2,3,4].\n\n[1] https://damonitor.github.io/doc/html/latest/admin-guide/mm/damon/start.html#visualizing-recorded-patterns<br>\n[2] https://damonitor.github.io/test/result/visual/latest/rec.heatmap.1.png.html<br>\n[3] https://damonitor.github.io/test/result/visual/latest/rec.wss_sz.png.html<br>\n[4] https://damonitor.github.io/test/result/visual/latest/rec.wss_time.png.html\n\n\nData Access Pattern Aware Memory Management\n-------------------------------------------\n\nBelow command makes every memory region of size >=4K that hasn't accessed for\n>=60 seconds in your workload to be swapped out. By doing this, you can make\nyour workload more memory efficient with only modest performance overhead.\n\n $ sudo damo start --damos_access_rate 0 0 --damos_sz_region 4K max \\\n --damos_age 60s max --damos_action pageout \\\n <pid of your workload>\n\nHolistic Memory Usage Monitoring\n--------------------------------\n\nYou can also show access pattern heatmap, memory footprints, and hotspot\nfunctions of the system and/or workloads all at once in live, like below.\n\n\n",
"bugtrack_url": null,
"license": null,
"summary": "DAMON user-space tool",
"version": "2.6.0",
"project_urls": {
"Bug Tracker": "https://github.com/damonitor/damo/issues",
"DAMON": "https://damonitor.github.io",
"Homepage": "https://github.com/damonitor/damo"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1a25a6d005b1fa16a9de69529c14b763623682f7fa3a3970c1ed2b426798af01",
"md5": "bd55a8ea2593ef972627a6ed261aa1ac",
"sha256": "0d490a08e835544e9bd1e800db9eec250e6f84cc3de207e929f7c4c4f70e5e88"
},
"downloads": -1,
"filename": "damo-2.6.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "bd55a8ea2593ef972627a6ed261aa1ac",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 102801,
"upload_time": "2024-12-17T05:04:32",
"upload_time_iso_8601": "2024-12-17T05:04:32.767554Z",
"url": "https://files.pythonhosted.org/packages/1a/25/a6d005b1fa16a9de69529c14b763623682f7fa3a3970c1ed2b426798af01/damo-2.6.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b9a9e72bdb020a7871e1d98f59a86d5f9b0a373b6f55999c9d8b7d19a63c83b9",
"md5": "0ad39cfeb7e57743475ec5e436309708",
"sha256": "2f2c76fbb6de4095073da8e93031fc1ed274de3cf6099aeb4b291f793ed955d4"
},
"downloads": -1,
"filename": "damo-2.6.0.tar.gz",
"has_sig": false,
"md5_digest": "0ad39cfeb7e57743475ec5e436309708",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 84942,
"upload_time": "2024-12-17T05:04:34",
"upload_time_iso_8601": "2024-12-17T05:04:34.992634Z",
"url": "https://files.pythonhosted.org/packages/b9/a9/e72bdb020a7871e1d98f59a86d5f9b0a373b6f55999c9d8b7d19a63c83b9/damo-2.6.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-17 05:04:34",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "damonitor",
"github_project": "damo",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "damo"
}
JSON
