[](https://github.com/hariharan-devarajan/dlio-profiler/actions/workflows/ci.yml)
[](https://coveralls.io/github/hariharan-devarajan/dlio-profiler?branch=dev)
[](https://dlio-profiler.readthedocs.io/en/latest/?badge=latest)
# DLIO Profiler v0.0.6
A multi-level profiler for capturing application functions and low-level system I/O calls from deep learning workloads.
Requirements for profiler
1. Python > 3.7
2. pybind11
Requirements for analyzer
1. bokeh>=2.4.2
2. pybind11
3. [zindex_py](https://github.com/hariharan-devarajan/zindex.git)
4. pandas>=2.0.3
5. dask>=2023.5.0
6. distributed
7. numpy>=1.24.3
8. pyarrow>=12.0.1
9. rich>=13.6.0
10. python-intervals>=1.10.0.post1
11. matplotlib>=3.7.3
## Build DLIO Profiler with pip
Users can easily install DLIO profiler using pip. This is the way most python packages are installed.
This method would work for both native python environments and conda environments.
### From source
```bash
git clone git@github.com:hariharan-devarajan/dlio-profiler.git
cd dlio-profiler
# You can skip this for installing the dev branch.
# for latest stable version use master branch.
git checkout tags/<Release> -b <Release>
pip install .
```
### From Github
```bash
DLP_VERSION=dev
pip install git+https://github.com/hariharan-devarajan/dlio-profiler.git@${DLP_VERSION}
```
For more build instructions check [here](https://dlio-profiler.readthedocs.io/en/latest/build.html)
Usage
```
from dlio_profiler.logger import dlio_logger, fn_interceptor
log_inst = dlio_logger.initialize_log(logfile=None, data_dir=None, process_id=-1)
dlio_log = fn_interceptor("COMPUTE")
# Example of using function decorators
@dlio_log.log
def log_events(index):
sleep(1)
# Example of function spawning and implicit I/O calls
def posix_calls(val):
index, is_spawn = val
path = f"{cwd}/data/demofile{index}.txt"
f = open(path, "w+")
f.write("Now the file has more content!")
f.close()
if is_spawn:
print(f"Calling spawn on {index} with pid {os.getpid()}")
log_inst.finalize() # This need to be called to correctly finalize DLIO Profiler.
else:
print(f"Not calling spawn on {index} with pid {os.getpid()}")
# NPZ calls internally calls POSIX calls.
def npz_calls(index):
# print(f"{cwd}/data/demofile2.npz")
path = f"{cwd}/data/demofile{index}.npz"
if os.path.exists(path):
os.remove(path)
records = np.random.randint(255, size=(8, 8, 1024), dtype=np.uint8)
record_labels = [0] * 1024
np.savez(path, x=records, y=record_labels)
def main():
log_events(0)
npz_calls(1)
with get_context('spawn').Pool(1, initializer=init) as pool:
pool.map(posix_calls, ((2, True),))
log_inst.finalize()
if __name__ == "__main__":
main()
```
For this example, as the DLIO_PROFILER_CPP_INIT do not pass log file or data dir, we need to set ``DLIO_PROFILER_LOG_FILE`` and ``DLIO_PROFILER_DATA_DIR``.
By default the DLIO Profiler mode is set to FUNCTION.
Example of running this configurations are:
```
# the process id, app_name and .pfw will be appended by the profiler for each app and process.
# name of final log file is ~/log_file-<APP_NAME>-<PID>.pfw
DLIO_PROFILER_LOG_FILE=~/log_file
# Colon separated paths for including for profiler
DLIO_PROFILER_DATA_DIR=/dev/shm/:/p/gpfs1/$USER/dataset:$PWD/data
# Enable profiler
DLIO_PROFILER_ENABLE=1
```
For more example check [Examples](https://dlio-profiler.readthedocs.io/en/latest/examples.html).
Raw data
{
"_id": null,
"home_page": "https://github.com/hariharan-devarajan/dlio-profiler",
"name": "dlio-profiler-py",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": "profiler, deep learning, I/O, benchmark, NPZ, pytorch benchmark, tensorflow benchmark",
"author": "Hariharan Devarajan (Hari)",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/13/7c/65449d155bef8689954e8d36e3f00817c167598d3a47fd96a37f4658548e/dlio_profiler_py-0.0.6.tar.gz",
"platform": null,
"description": "[](https://github.com/hariharan-devarajan/dlio-profiler/actions/workflows/ci.yml)\n[](https://coveralls.io/github/hariharan-devarajan/dlio-profiler?branch=dev)\n[](https://dlio-profiler.readthedocs.io/en/latest/?badge=latest)\n\n# DLIO Profiler v0.0.6\nA multi-level profiler for capturing application functions and low-level system I/O calls from deep learning workloads.\n\nRequirements for profiler\n1. Python > 3.7\n2. pybind11\n\n\nRequirements for analyzer\n1. bokeh>=2.4.2\n2. pybind11\n3. [zindex_py](https://github.com/hariharan-devarajan/zindex.git)\n4. pandas>=2.0.3\n5. dask>=2023.5.0\n6. distributed\n7. numpy>=1.24.3\n8. pyarrow>=12.0.1\n9. rich>=13.6.0\n10. python-intervals>=1.10.0.post1\n11. matplotlib>=3.7.3\n\n## Build DLIO Profiler with pip\n\nUsers can easily install DLIO profiler using pip. This is the way most python packages are installed.\nThis method would work for both native python environments and conda environments.\n\n### From source\n\n```bash\n git clone git@github.com:hariharan-devarajan/dlio-profiler.git\n cd dlio-profiler\n # You can skip this for installing the dev branch.\n # for latest stable version use master branch.\n git checkout tags/<Release> -b <Release>\n pip install .\n```\n\n### From Github\n\n```bash\nDLP_VERSION=dev\npip install git+https://github.com/hariharan-devarajan/dlio-profiler.git@${DLP_VERSION}\n```\n\nFor more build instructions check [here](https://dlio-profiler.readthedocs.io/en/latest/build.html)\n\nUsage\n\n```\n from dlio_profiler.logger import dlio_logger, fn_interceptor\n log_inst = dlio_logger.initialize_log(logfile=None, data_dir=None, process_id=-1)\n dlio_log = fn_interceptor(\"COMPUTE\")\n\n # Example of using function decorators\n @dlio_log.log\n def log_events(index):\n sleep(1)\n\n # Example of function spawning and implicit I/O calls\n def posix_calls(val):\n index, is_spawn = val\n path = f\"{cwd}/data/demofile{index}.txt\"\n f = open(path, \"w+\")\n f.write(\"Now the file has more content!\")\n f.close()\n if is_spawn:\n print(f\"Calling spawn on {index} with pid {os.getpid()}\")\n log_inst.finalize() # This need to be called to correctly finalize DLIO Profiler.\n else:\n print(f\"Not calling spawn on {index} with pid {os.getpid()}\")\n\n # NPZ calls internally calls POSIX calls.\n def npz_calls(index):\n # print(f\"{cwd}/data/demofile2.npz\")\n path = f\"{cwd}/data/demofile{index}.npz\"\n if os.path.exists(path):\n os.remove(path)\n records = np.random.randint(255, size=(8, 8, 1024), dtype=np.uint8)\n record_labels = [0] * 1024\n np.savez(path, x=records, y=record_labels)\n\n def main():\n log_events(0)\n npz_calls(1)\n with get_context('spawn').Pool(1, initializer=init) as pool:\n pool.map(posix_calls, ((2, True),))\n log_inst.finalize()\n\n\n if __name__ == \"__main__\":\n main()\n\n```\n\nFor this example, as the DLIO_PROFILER_CPP_INIT do not pass log file or data dir, we need to set ``DLIO_PROFILER_LOG_FILE`` and ``DLIO_PROFILER_DATA_DIR``.\nBy default the DLIO Profiler mode is set to FUNCTION.\nExample of running this configurations are:\n\n```\n\n # the process id, app_name and .pfw will be appended by the profiler for each app and process.\n # name of final log file is ~/log_file-<APP_NAME>-<PID>.pfw\n DLIO_PROFILER_LOG_FILE=~/log_file\n # Colon separated paths for including for profiler\n DLIO_PROFILER_DATA_DIR=/dev/shm/:/p/gpfs1/$USER/dataset:$PWD/data\n # Enable profiler\n DLIO_PROFILER_ENABLE=1\n```\n\nFor more example check [Examples](https://dlio-profiler.readthedocs.io/en/latest/examples.html).\n\n",
"bugtrack_url": null,
"license": null,
"summary": "I/O profiler for deep learning python apps. Specifically for dlio_benchmark.",
"version": "0.0.6",
"project_urls": {
"Bug Reports": "https://github.com/hariharan-devarajan/dlio-profiler/issues",
"Homepage": "https://github.com/hariharan-devarajan/dlio-profiler",
"Source": "https://github.com/hariharan-devarajan/dlio-profiler"
},
"split_keywords": [
"profiler",
" deep learning",
" i/o",
" benchmark",
" npz",
" pytorch benchmark",
" tensorflow benchmark"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1acebb7c75a19c79bef590a8059e4ae6185804a98c779f5778a25f428ede17d1",
"md5": "0b8e9d5b20560a8a214635e3b54d8d67",
"sha256": "093c8e94b77b12f4a7d33b4f472c26a956384d04e8815054a7ff04c880a4966c"
},
"downloads": -1,
"filename": "dlio_profiler_py-0.0.6-cp310-cp310-manylinux_2_31_x86_64.whl",
"has_sig": false,
"md5_digest": "0b8e9d5b20560a8a214635e3b54d8d67",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.7",
"size": 2027687,
"upload_time": "2024-06-18T00:35:50",
"upload_time_iso_8601": "2024-06-18T00:35:50.452261Z",
"url": "https://files.pythonhosted.org/packages/1a/ce/bb7c75a19c79bef590a8059e4ae6185804a98c779f5778a25f428ede17d1/dlio_profiler_py-0.0.6-cp310-cp310-manylinux_2_31_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "be9fc22e80da02ad64b9e67f85b50b125ab02a9a632f0d2f8bc841b3c83f51a5",
"md5": "a79a98c0cbfaebdd708d0446393d1198",
"sha256": "9df1d615b89d28ccd9024ac3d266d8720d0fb54a7f9803b9f21eb4fd67c28947"
},
"downloads": -1,
"filename": "dlio_profiler_py-0.0.6-cp310-cp310-manylinux_2_34_x86_64.whl",
"has_sig": false,
"md5_digest": "a79a98c0cbfaebdd708d0446393d1198",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.7",
"size": 2122919,
"upload_time": "2024-06-18T00:35:08",
"upload_time_iso_8601": "2024-06-18T00:35:08.018206Z",
"url": "https://files.pythonhosted.org/packages/be/9f/c22e80da02ad64b9e67f85b50b125ab02a9a632f0d2f8bc841b3c83f51a5/dlio_profiler_py-0.0.6-cp310-cp310-manylinux_2_34_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "195dee26ade20899e69314a796f98a6d816243e18b5e01ceb923aef83c37e9b6",
"md5": "89e30f6e08875cac5948574c780fbf5b",
"sha256": "57ca4796124fb3f9962580955dc4caa24ee5f666fc2e755ad3edc874ce025aa2"
},
"downloads": -1,
"filename": "dlio_profiler_py-0.0.6-cp311-cp311-manylinux_2_31_x86_64.whl",
"has_sig": false,
"md5_digest": "89e30f6e08875cac5948574c780fbf5b",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.7",
"size": 2029465,
"upload_time": "2024-06-18T00:35:32",
"upload_time_iso_8601": "2024-06-18T00:35:32.656561Z",
"url": "https://files.pythonhosted.org/packages/19/5d/ee26ade20899e69314a796f98a6d816243e18b5e01ceb923aef83c37e9b6/dlio_profiler_py-0.0.6-cp311-cp311-manylinux_2_31_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "60c2d6cd5099e782e6e3ac685fe71798f1a8f6c2303060300ea164716d50c4d4",
"md5": "a32d200ec9f42b9f55bbef6dc61f552b",
"sha256": "c515b21d33836d5c677632a1f072d92f83c98e653cb16b7e047f9bd931ef9b18"
},
"downloads": -1,
"filename": "dlio_profiler_py-0.0.6-cp311-cp311-manylinux_2_34_x86_64.whl",
"has_sig": false,
"md5_digest": "a32d200ec9f42b9f55bbef6dc61f552b",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.7",
"size": 2126465,
"upload_time": "2024-06-18T00:35:32",
"upload_time_iso_8601": "2024-06-18T00:35:32.643440Z",
"url": "https://files.pythonhosted.org/packages/60/c2/d6cd5099e782e6e3ac685fe71798f1a8f6c2303060300ea164716d50c4d4/dlio_profiler_py-0.0.6-cp311-cp311-manylinux_2_34_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d7f2e371d91c5120249586f9938e65bd426ac18838fe8bb5a92a68de48d181f3",
"md5": "3f89abf31ff155df8c3209ec42a7212b",
"sha256": "36fe02ec57c4df8614abb1ae2444af4050c7893db26b4b217c963e31df7814e5"
},
"downloads": -1,
"filename": "dlio_profiler_py-0.0.6-cp37-cp37m-manylinux_2_31_x86_64.whl",
"has_sig": false,
"md5_digest": "3f89abf31ff155df8c3209ec42a7212b",
"packagetype": "bdist_wheel",
"python_version": "cp37",
"requires_python": ">=3.7",
"size": 2031324,
"upload_time": "2024-06-18T00:35:33",
"upload_time_iso_8601": "2024-06-18T00:35:33.610934Z",
"url": "https://files.pythonhosted.org/packages/d7/f2/e371d91c5120249586f9938e65bd426ac18838fe8bb5a92a68de48d181f3/dlio_profiler_py-0.0.6-cp37-cp37m-manylinux_2_31_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ea7b0b6b6e492483f9c85d30eab525c33e3eaee284ff79c7ac4ac629522d0a3f",
"md5": "16b4e83c5fb79b02f117f91a974783cf",
"sha256": "0df6b593e9b07b4f63a8a1824810c8c3ee4ca39916aa2a1e08015c6faf1ca964"
},
"downloads": -1,
"filename": "dlio_profiler_py-0.0.6-cp37-cp37m-manylinux_2_34_x86_64.whl",
"has_sig": false,
"md5_digest": "16b4e83c5fb79b02f117f91a974783cf",
"packagetype": "bdist_wheel",
"python_version": "cp37",
"requires_python": ">=3.7",
"size": 2122795,
"upload_time": "2024-06-18T00:35:26",
"upload_time_iso_8601": "2024-06-18T00:35:26.274409Z",
"url": "https://files.pythonhosted.org/packages/ea/7b/0b6b6e492483f9c85d30eab525c33e3eaee284ff79c7ac4ac629522d0a3f/dlio_profiler_py-0.0.6-cp37-cp37m-manylinux_2_34_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "87ba2a2209330733261c51916f9bd2ee891eea38c5c0127446ad54b9b147de13",
"md5": "13467f6bf8fdad2a85348be99c567bd9",
"sha256": "ee895db9e948173bdb5df645582b11c0950f0f4dc44a2def0fe3368ba520bd95"
},
"downloads": -1,
"filename": "dlio_profiler_py-0.0.6-cp38-cp38-manylinux_2_31_x86_64.whl",
"has_sig": false,
"md5_digest": "13467f6bf8fdad2a85348be99c567bd9",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.7",
"size": 2027766,
"upload_time": "2024-06-18T00:36:46",
"upload_time_iso_8601": "2024-06-18T00:36:46.373716Z",
"url": "https://files.pythonhosted.org/packages/87/ba/2a2209330733261c51916f9bd2ee891eea38c5c0127446ad54b9b147de13/dlio_profiler_py-0.0.6-cp38-cp38-manylinux_2_31_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d32eda17ce3ba05e2c1d58a85d69f43ebcd3c605395f87f071686aacb35be21f",
"md5": "145190ba353546bce1ce9ecc52adf924",
"sha256": "24c95f965311c641cd6257f1d660622abbd818194c4551e91a33107485ca360d"
},
"downloads": -1,
"filename": "dlio_profiler_py-0.0.6-cp38-cp38-manylinux_2_34_x86_64.whl",
"has_sig": false,
"md5_digest": "145190ba353546bce1ce9ecc52adf924",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.7",
"size": 2123235,
"upload_time": "2024-06-18T00:36:05",
"upload_time_iso_8601": "2024-06-18T00:36:05.836694Z",
"url": "https://files.pythonhosted.org/packages/d3/2e/da17ce3ba05e2c1d58a85d69f43ebcd3c605395f87f071686aacb35be21f/dlio_profiler_py-0.0.6-cp38-cp38-manylinux_2_34_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e445f19e144661f87835c39cc52aef78277bb7a15686f6edf3e42b5f516dd89d",
"md5": "31c976fda5009414e59d89d79100104c",
"sha256": "a50eaa922df60d5ed41f04509af6a4318156051c44891d576f570cf4683d11f3"
},
"downloads": -1,
"filename": "dlio_profiler_py-0.0.6-cp39-cp39-manylinux_2_31_x86_64.whl",
"has_sig": false,
"md5_digest": "31c976fda5009414e59d89d79100104c",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.7",
"size": 2028455,
"upload_time": "2024-06-18T00:36:51",
"upload_time_iso_8601": "2024-06-18T00:36:51.711083Z",
"url": "https://files.pythonhosted.org/packages/e4/45/f19e144661f87835c39cc52aef78277bb7a15686f6edf3e42b5f516dd89d/dlio_profiler_py-0.0.6-cp39-cp39-manylinux_2_31_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "8c70c98e736ca67d2d48d75da26e43604f0e0eb688c84401d560ec4fa9657f7b",
"md5": "924c95993b5a5858f2c1b8afec78ecb3",
"sha256": "8d9f6f64ecfc407f6f3e3592c98c106971d95c5403d4766b2b328c7932726638"
},
"downloads": -1,
"filename": "dlio_profiler_py-0.0.6-cp39-cp39-manylinux_2_34_x86_64.whl",
"has_sig": false,
"md5_digest": "924c95993b5a5858f2c1b8afec78ecb3",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.7",
"size": 2123676,
"upload_time": "2024-06-18T00:35:41",
"upload_time_iso_8601": "2024-06-18T00:35:41.092265Z",
"url": "https://files.pythonhosted.org/packages/8c/70/c98e736ca67d2d48d75da26e43604f0e0eb688c84401d560ec4fa9657f7b/dlio_profiler_py-0.0.6-cp39-cp39-manylinux_2_34_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "137c65449d155bef8689954e8d36e3f00817c167598d3a47fd96a37f4658548e",
"md5": "71e31e0281c9c40618ca1b46cdd8c960",
"sha256": "4053b4e116c3a57fe497211bb391fd087e7f38c59709658e16edf52697c0162f"
},
"downloads": -1,
"filename": "dlio_profiler_py-0.0.6.tar.gz",
"has_sig": false,
"md5_digest": "71e31e0281c9c40618ca1b46cdd8c960",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 42203,
"upload_time": "2024-06-18T00:35:38",
"upload_time_iso_8601": "2024-06-18T00:35:38.558139Z",
"url": "https://files.pythonhosted.org/packages/13/7c/65449d155bef8689954e8d36e3f00817c167598d3a47fd96a37f4658548e/dlio_profiler_py-0.0.6.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-06-18 00:35:38",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "hariharan-devarajan",
"github_project": "dlio-profiler",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "dlio-profiler-py"
}
JSON
