| Name | mpiprof JSON |
| Version |
0.2.1
JSON |
| download |
| home_page | None |
| Summary | Profiler for MPI python programs |
| upload_time | 2025-08-26 14:48:59 |
| maintainer | None |
| docs_url | None |
| author | Your Name |
| requires_python | >=3.8 |
| license | MIT |
| keywords |
mpi
profiling
cprofile
mpi4py
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
mpiprof.py: Profiler for MPI python programs
============================================
Enables:
1. using cProfile on each launched MPI process.
2. measuring the communication overhead of your MPI setup.
mpiprof provides two complementary pieces:
1. A drop-in wrapper for cProfile that you can invoke as a module,
``mpiexec -n 4 python3 -m mpiprof your_script.py arg1 arg2``.
This writes a separate .pstats file for each rank.
2. An optional ``MPIProfiler`` class that wraps a ``mpi4py.MPI.Comm`` and
records basic timing/count statistics of MPI calls and where they came from.
You can write the results to disk by calling ``write_statistics()``.
Installation
------------
- From pypi::
pip install mpiprof
- From a local checkout, by cloning this repository::
git clone https://github.com/JohannesBuchner/mpiprof.py
cd mpiprof.py/
pip install .
Requirements
------------
- Python 3.8+
- mpi4py
- An MPI runtime if you plan to launch under mpiexec/srun/etc.
Usage: per-rank cProfile runner
-------------------------------
Run your script under mpiexec (or srun) using the module form:
- ``mpiexec -n 4 python3 -m mpiprof your_script.py arg1 arg2``
This writes one profile file per rank named ``mpiprof.<rank>.pstats``
in the current working directory.
Options:
- ``-o / --outfile`` sets the output path or pattern (default:
``mpiprof.{rank}.pstats``). The literal substring ``{rank}`` will be
replaced with the MPI rank.
Signal handling:
- mpiprof installs SIGINT and SIGTERM handlers to dump the profile
before exiting, so Ctrl-C or a clean termination should still produce
output. If the MPI launcher escalates to SIGKILL immediately, no tool
can save a profile.
Usage: MPIProfiler for mpi4py
-----------------------------
The ``MPIProfiler`` wraps a communicator and measures wall-clock time
and call counts for common blocking MPI calls. It is intentionally
lightweight and safe to leave in production runs (low overhead on the
measured calls). Nonblocking calls are counted but not timed precisely
(the time recorded is call overhead, not the transfer time).
Example:
.. code-block:: python
from mpi4py import MPI
from mpiprof import MPIProfiler
comm = MPIProfiler(MPI.COMM_WORLD)
# Your MPI code as usual, using comm instead of MPI.COMM_WORLD:
rank = comm.Get_rank()
size = comm.Get_size()
data = rank
# Simple collective
total = comm.allreduce(data, op=MPI.SUM)
# Point-to-point
if rank == 0:
comm.send(b"hello", dest=1, tag=0)
elif rank == 1:
msg = comm.recv(source=0, tag=0)
# Write stats at the end (one file per rank)
comm.write_statistics() # default name: mpiprof.stats.<rank>.json
Notes:
- The wrapper exposes common methods (e.g., ``send``, ``recv``, ``bcast``,
``reduce``, ``allreduce``, ``gather``, ``scatter``, ``barrier``) and
forwards any other attributes to the underlying communicator. It tries
to be case-insensitive to match mpi4py idioms (both ``Send`` and
``send`` are supported).
- For nonblocking operations (``Isend``, ``Irecv``), the wrapper records
the call count but cannot attribute data transfer time unless you also
wrap and time ``Wait``/``Waitall``. A simple ``wait`` wrapper is
provided to time individual requests returned by the wrapper.
Rank detection
--------------
The runner tries to detect the rank via common environment variables:
- ``OMPI_COMM_WORLD_RANK``, ``PMIX_RANK`` (Open MPI)
- ``PMI_RANK`` (MPICH, Intel MPI)
- ``MV2_COMM_WORLD_RANK`` (MVAPICH2)
- ``SLURM_PROCID`` (scheduler fallback)
- default: 0 if none found
Output files
------------
- Runner: ``mpiprof.<rank>.pstats`` (or the pattern you set with ``-o``).
You can analyze it with ``pstats`` or tools like ``snakeviz``:
- ``python3 -m pstats mpiprof.0.pstats``
- ``snakeviz mpiprof.0.pstats``
- MPIProfiler: ``MPIprofile.<rank>.out`` with operation counts and
total wall-clock time per operation. Example::
Function: scatter
Call stack:
surveymcmc.py:466
surveymcmc.py:431 sampler.advance_one_step(False)
mcaeis.py:107 local_coords = self.comm.scatter(chunks, root=0)
Number of calls: 2
Duration During Call: 98.990435s
Duration Before Call: 0.000014s
Function: scatter
Call stack:
surveymcmc.py:466
surveymcmc.py:448 sampler.advance_one_step()
mcaeis.py:75 self._init()
mcaeis.py:55 self.log_probs = self._evaluate_log_probs(self.coords)
mcaeis.py:64 local_coords = self.comm.scatter(chunks, root=0)
Number of calls: 39
Duration During Call: 2.228181s
Duration Before Call: 0.000025s
Total MPI Time: 388.922329s
Total Non-MPI Time: 206.503627s
Results are sorted (descending) by duration during call.
Limitations
-----------
- The runner cannot save profiles if the process is killed by SIGKILL.
- MPIProfiler’s accounting for nonblocking calls is approximate unless
you consistently call ``wait``/``waitall`` on the requests returned
by the wrapper’s nonblocking methods.
License
-------
MIT
Raw data
{
"_id": null,
"home_page": null,
"name": "mpiprof",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "MPI, profiling, cProfile, mpi4py",
"author": "Your Name",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/8c/6f/5d5234bb582a2ef68e50927edfad659ae7cfcb5af791abfd306b68894984/mpiprof-0.2.1.tar.gz",
"platform": null,
"description": "mpiprof.py: Profiler for MPI python programs\n============================================\n\nEnables:\n\n1. using cProfile on each launched MPI process.\n2. measuring the communication overhead of your MPI setup.\n\nmpiprof provides two complementary pieces:\n\n1. A drop-in wrapper for cProfile that you can invoke as a module,\n ``mpiexec -n 4 python3 -m mpiprof your_script.py arg1 arg2``.\n This writes a separate .pstats file for each rank.\n\n2. An optional ``MPIProfiler`` class that wraps a ``mpi4py.MPI.Comm`` and\n records basic timing/count statistics of MPI calls and where they came from.\n You can write the results to disk by calling ``write_statistics()``.\n\nInstallation\n------------\n\n- From pypi::\n\n pip install mpiprof\n\n- From a local checkout, by cloning this repository::\n\n git clone https://github.com/JohannesBuchner/mpiprof.py\n cd mpiprof.py/\n pip install .\n\nRequirements\n------------\n- Python 3.8+\n- mpi4py\n- An MPI runtime if you plan to launch under mpiexec/srun/etc.\n\nUsage: per-rank cProfile runner\n-------------------------------\nRun your script under mpiexec (or srun) using the module form:\n\n- ``mpiexec -n 4 python3 -m mpiprof your_script.py arg1 arg2``\n\nThis writes one profile file per rank named ``mpiprof.<rank>.pstats``\nin the current working directory.\n\nOptions:\n\n- ``-o / --outfile`` sets the output path or pattern (default:\n ``mpiprof.{rank}.pstats``). The literal substring ``{rank}`` will be\n replaced with the MPI rank.\n\nSignal handling:\n\n- mpiprof installs SIGINT and SIGTERM handlers to dump the profile\n before exiting, so Ctrl-C or a clean termination should still produce\n output. If the MPI launcher escalates to SIGKILL immediately, no tool\n can save a profile.\n\nUsage: MPIProfiler for mpi4py\n-----------------------------\nThe ``MPIProfiler`` wraps a communicator and measures wall-clock time\nand call counts for common blocking MPI calls. It is intentionally\nlightweight and safe to leave in production runs (low overhead on the\nmeasured calls). Nonblocking calls are counted but not timed precisely\n(the time recorded is call overhead, not the transfer time).\n\nExample:\n\n.. code-block:: python\n\n from mpi4py import MPI\n from mpiprof import MPIProfiler\n\n comm = MPIProfiler(MPI.COMM_WORLD)\n\n # Your MPI code as usual, using comm instead of MPI.COMM_WORLD:\n rank = comm.Get_rank()\n size = comm.Get_size()\n data = rank\n\n # Simple collective\n total = comm.allreduce(data, op=MPI.SUM)\n\n # Point-to-point\n if rank == 0:\n comm.send(b\"hello\", dest=1, tag=0)\n elif rank == 1:\n msg = comm.recv(source=0, tag=0)\n\n # Write stats at the end (one file per rank)\n comm.write_statistics() # default name: mpiprof.stats.<rank>.json\n\nNotes:\n\n- The wrapper exposes common methods (e.g., ``send``, ``recv``, ``bcast``,\n ``reduce``, ``allreduce``, ``gather``, ``scatter``, ``barrier``) and\n forwards any other attributes to the underlying communicator. It tries\n to be case-insensitive to match mpi4py idioms (both ``Send`` and\n ``send`` are supported).\n\n- For nonblocking operations (``Isend``, ``Irecv``), the wrapper records\n the call count but cannot attribute data transfer time unless you also\n wrap and time ``Wait``/``Waitall``. A simple ``wait`` wrapper is\n provided to time individual requests returned by the wrapper.\n\nRank detection\n--------------\nThe runner tries to detect the rank via common environment variables:\n\n- ``OMPI_COMM_WORLD_RANK``, ``PMIX_RANK`` (Open MPI)\n- ``PMI_RANK`` (MPICH, Intel MPI)\n- ``MV2_COMM_WORLD_RANK`` (MVAPICH2)\n- ``SLURM_PROCID`` (scheduler fallback)\n- default: 0 if none found\n\nOutput files\n------------\n- Runner: ``mpiprof.<rank>.pstats`` (or the pattern you set with ``-o``).\n You can analyze it with ``pstats`` or tools like ``snakeviz``:\n\n - ``python3 -m pstats mpiprof.0.pstats``\n - ``snakeviz mpiprof.0.pstats``\n\n- MPIProfiler: ``MPIprofile.<rank>.out`` with operation counts and\n total wall-clock time per operation. Example::\n\n Function: scatter\n Call stack:\n surveymcmc.py:466\n surveymcmc.py:431 sampler.advance_one_step(False)\n mcaeis.py:107 local_coords = self.comm.scatter(chunks, root=0)\n Number of calls: 2\n Duration During Call: 98.990435s\n Duration Before Call: 0.000014s\n\n Function: scatter\n Call stack:\n surveymcmc.py:466\n surveymcmc.py:448 sampler.advance_one_step()\n mcaeis.py:75 self._init()\n mcaeis.py:55 self.log_probs = self._evaluate_log_probs(self.coords)\n mcaeis.py:64 local_coords = self.comm.scatter(chunks, root=0)\n Number of calls: 39\n Duration During Call: 2.228181s\n Duration Before Call: 0.000025s\n \n Total MPI Time: 388.922329s\n Total Non-MPI Time: 206.503627s\n\nResults are sorted (descending) by duration during call.\n\nLimitations\n-----------\n- The runner cannot save profiles if the process is killed by SIGKILL.\n- MPIProfiler\u2019s accounting for nonblocking calls is approximate unless\n you consistently call ``wait``/``waitall`` on the requests returned\n by the wrapper\u2019s nonblocking methods.\n\nLicense\n-------\nMIT\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Profiler for MPI python programs",
"version": "0.2.1",
"project_urls": {
"Homepage": "https://github.com/JohannesBuchner/mpiprof.py",
"Repository": "https://github.com/JohannesBuchner/mpiprof.py"
},
"split_keywords": [
"mpi",
" profiling",
" cprofile",
" mpi4py"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "c4e14a90b4fdc6c3c9c1c11d17f85f2595c0fd199ff2d6a993991567cbb59ac8",
"md5": "3f74f898beac40c6f760b922a0137324",
"sha256": "093cfe8e1919520cb1b83310ff5666ffa77ad499bdb5d281bbe3759d42aedbe5"
},
"downloads": -1,
"filename": "mpiprof-0.2.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3f74f898beac40c6f760b922a0137324",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 6581,
"upload_time": "2025-08-26T14:48:58",
"upload_time_iso_8601": "2025-08-26T14:48:58.507650Z",
"url": "https://files.pythonhosted.org/packages/c4/e1/4a90b4fdc6c3c9c1c11d17f85f2595c0fd199ff2d6a993991567cbb59ac8/mpiprof-0.2.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "8c6f5d5234bb582a2ef68e50927edfad659ae7cfcb5af791abfd306b68894984",
"md5": "16b5c10aeb7e790989b88f891902d5a9",
"sha256": "dd9bdaa9054ba9093bd8c6091b73d42504c60fb8b687490d48227dc50058e541"
},
"downloads": -1,
"filename": "mpiprof-0.2.1.tar.gz",
"has_sig": false,
"md5_digest": "16b5c10aeb7e790989b88f891902d5a9",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 6168,
"upload_time": "2025-08-26T14:48:59",
"upload_time_iso_8601": "2025-08-26T14:48:59.474447Z",
"url": "https://files.pythonhosted.org/packages/8c/6f/5d5234bb582a2ef68e50927edfad659ae7cfcb5af791abfd306b68894984/mpiprof-0.2.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-26 14:48:59",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "JohannesBuchner",
"github_project": "mpiprof.py",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "mpiprof"
}
