**A visual "grep" of mmap. This is a large HPC Fortran solver codebase. Each function/subroutine is a circle, size proportional to the number os lines of code, aggregated in files, then folders. If the function contains a pragma "$!ACC", the circle is shown in red. The whole figure shows what parts of the code ported to GPU using OpenACC pragmas**
**Global callgraph of a large HPC code, using the pyvis backend and a barnes-hut layout algorithm. The large inner structures of the code are emerging. In red, the NUMERICS parts, in blue the Lagrangian solver, in Green the boundary conditions.**
# Marauder's map
## About
Marauder's map is a python helper tool to create visual representations of the internal structure of python and Fortran packages.
Just like Harry Potter's Marauder's map, which can be asked different things and will adapt its response to the request, the Marauder's map is an automated solution developed to study software geography.
The online documentation is available on [the internal Cerfacs forge](http://opentea.pg.cerfacs.fr/maraudersmap/) (Credentials needed). Soon the package will be updated on Pypi with documentation on readthedocs.
---
## Installation
This is an open-source python package available on the Cerfacs Gitlab Forge [here](https://gitlab.com/cerfacs/maraudersmap.)
Soon, it will be released on Pypi.
## Basic usage:
The installation add the command `mmap` to your terminal:
```
>mmap
Usage: mmap [OPTIONS] COMMAND [ARGS]...
Package maraudersmap v0.0.0
--------------- Marauders map --------------------
You are now using the Command line interface of Marauders map package,
a set of tools created at CERFACS (https://cerfacs.fr). It creates
callgraphs for python and fortran projects.
This is a python package currently installed in your python environment.
Usage: For all static callgraphs, use `>mmap anew` before others
commands to create your project. Then use `>mmap
imports/cgstat/cgstat_ftn` to generate a JSON file storong the
callgraph. Finally use `>mmap show mycallgraph.json` to visualize
your data. The mmap.yml input file is the same for all commands, to
ensure homogeneity btw analysis.
For the python dynamic callgraph use >mmap `cgdyn --new` to create your
monitoring function.
Free ad.: if you just want to scan a Python module structure, try :
https://pypi.org/project/code2flow/. Awsom' stuff.
Options:
--help Show this message and exit.
Commands:
anew Create a default input file.
cgdyn Python : dynamic callgraph from the eXecution of a function.
cgstat Python : static callgraph - all functions and methods.
cgstat-ftn Fortran: static callgraph - all functions and subroutines.
imports Python : static callgraph of modules - only imports.
show Visualize mmap callgraphs stored in a JSON_DATA file
```
Now, write the following command to create a default .yml input file that you will then edit. This is the starting point of using Marauder's map.
```
mmap anew
````
An input file has been created: *mmap_in.yml* which looks like this:
```yaml
# the path where your sources are stored
path : /Users/desplats/TEST/flinter/Gitlab/Nek5000/core.py
# name of the package
package: Nek5000
# blacklist of unwanted nodes in the graph
remove_patterns :
- "matplotlib*"
- "numpy*"
- "tkinter*"
- "json*"
- "PIL*"
soften_patterns :
- "*BaseForm*"
# If rules is selected, then color_rules will be applied, otherwise chose size or ccn for example
color_by: "ccn"
# coloring rules to apply (last color prevail)
color_rules :
acquisition_canvas: "#EEEEBB"
utils: "grey"
constants: "grey"
base_canvas_object: "#EE8866"
popups: "#222255"
forms: "#77AADD"
###########################################################
# Static graphs
showparents: False # add missing parent nodes
remove_hyperconnect: 5 #remove leaves with more than 5 parents
"mmap_in.yml" 34L, 840B
```
You need to edit the path towards your sources, name the package, and set the "colored_by" option to either "ccn" (coloration by complexity), "size" (coloration by size, in number of lines), or "rules" (customized coloration, with the rules to edit under).
Free ad.: if you just want to scan a Python module structure, try : https://pypi.org/project/code2flow/. Aw som' stuff.
## Complete workflow
Here follows the data flow of marauder's map command, showing you all the possibilities at hand.
```bash
mmap_in.yml
|
|
|
+------------------+-----------+-----------+--------+
| | | |
| +----+---+ | |
| |treefile| | |
| +----+---+ | +---v---+
+----+---+ | | |imports|
|treefunc| v | +---+---+
+----+---+ file_tree.json | |
| | | v
| | |import_graphs.js
v | | | |
func_tree.json-------------------+---|||------+ | | |
| | | | | | |
| | | | | | |
| | | +v---v---v+ |
| +------------+ | | |callgraph| |
| |regexp-input| | | +----+----+ |
| +-------+----+ | | | |
| | | | | |
+------+ | +-----+ v | | | |
| grep <------+-------->score<-rules.yml| | | |
+---+--+ +--+--+ | | | |
| | | | v |
| score<-------+ | | callgraph.json |
v | | | | |
graphical stats<---+ | | | |
output | | | | |
v | | | |
func_tree_score.json | | | |
| +------+ | | |
+---+ | | +-v-----v-+
| | +--------+ |showgraph|
| | | +----+----+
+v--v--v-+ |
|showtree| v
+---+----+ graphical
| output
v
graphical
output
```
## Acknowledgements
Marauder's Map is a service created thanks to the [EXCELLERAT Center Of Excellence](https://www.excellerat.eu/wp/) (drifting from the flinter service). and is continued as part of the [EXCELLERAT P2 Center Of Excellence](https://www.excellerat.eu/).
It is also feeded by the community of the [COEC Center Of Excellence](https://coec-project.eu/). These projects ware funded by the European community.
Raw data
{
"_id": null,
"home_page": "https://gitlab.com/cerfacs/maraudersmap",
"name": "maraudersmap",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": "findimports graph",
"author": "CoopTeam-CERFACS",
"author_email": "coop@cerfacs.com",
"download_url": "https://files.pythonhosted.org/packages/67/3f/bbbff8949971e56688025d27e88f10fd7d39fb8dde9c06c4509d9e27692c/maraudersmap-0.3.0.tar.gz",
"platform": null,
"description": "\n\n**A visual \"grep\" of mmap. This is a large HPC Fortran solver codebase. Each function/subroutine is a circle, size proportional to the number os lines of code, aggregated in files, then folders. If the function contains a pragma \"$!ACC\", the circle is shown in red. The whole figure shows what parts of the code ported to GPU using OpenACC pragmas**\n\n\n**Global callgraph of a large HPC code, using the pyvis backend and a barnes-hut layout algorithm. The large inner structures of the code are emerging. In red, the NUMERICS parts, in blue the Lagrangian solver, in Green the boundary conditions.**\n\n\n# Marauder's map\n\n## About\n\nMarauder's map is a python helper tool to create visual representations of the internal structure of python and Fortran packages.\nJust like Harry Potter's Marauder's map, which can be asked different things and will adapt its response to the request, the Marauder's map is an automated solution developed to study software geography.\n\nThe online documentation is available on [the internal Cerfacs forge](http://opentea.pg.cerfacs.fr/maraudersmap/) (Credentials needed). Soon the package will be updated on Pypi with documentation on readthedocs.\n\n\n---\n\n## Installation\n\nThis is an open-source python package available on the Cerfacs Gitlab Forge [here](https://gitlab.com/cerfacs/maraudersmap.)\nSoon, it will be released on Pypi.\n\n\n## Basic usage:\n\nThe installation add the command `mmap` to your terminal:\n\n```\n>mmap\nUsage: mmap [OPTIONS] COMMAND [ARGS]...\n\n Package maraudersmap v0.0.0\n\n --------------- Marauders map --------------------\n\n You are now using the Command line interface of Marauders map package,\n a set of tools created at CERFACS (https://cerfacs.fr). It creates\n callgraphs for python and fortran projects.\n\n This is a python package currently installed in your python environment.\n\n Usage: For all static callgraphs, use `>mmap anew` before others\n commands to create your project. Then use `>mmap\n imports/cgstat/cgstat_ftn` to generate a JSON file storong the\n callgraph. Finally use `>mmap show mycallgraph.json` to visualize\n your data. The mmap.yml input file is the same for all commands, to\n ensure homogeneity btw analysis.\n\n For the python dynamic callgraph use >mmap `cgdyn --new` to create your\n monitoring function.\n\n Free ad.: if you just want to scan a Python module structure, try :\n https://pypi.org/project/code2flow/. Awsom' stuff.\n\nOptions:\n --help Show this message and exit.\n\nCommands:\n anew Create a default input file.\n cgdyn Python : dynamic callgraph from the eXecution of a function.\n cgstat Python : static callgraph - all functions and methods.\n cgstat-ftn Fortran: static callgraph - all functions and subroutines.\n imports Python : static callgraph of modules - only imports.\n show Visualize mmap callgraphs stored in a JSON_DATA file\n```\n\nNow, write the following command to create a default .yml input file that you will then edit. This is the starting point of using Marauder's map.\n\n```\nmmap anew\n````\n\nAn input file has been created: *mmap_in.yml* which looks like this: \n\n\n```yaml\n# the path where your sources are stored\npath : /Users/desplats/TEST/flinter/Gitlab/Nek5000/core.py\n# name of the package\npackage: Nek5000\n\n# blacklist of unwanted nodes in the graph\nremove_patterns :\n - \"matplotlib*\"\n - \"numpy*\"\n - \"tkinter*\"\n - \"json*\"\n - \"PIL*\"\n\nsoften_patterns :\n - \"*BaseForm*\"\n\n# If rules is selected, then color_rules will be applied, otherwise chose size or ccn for example\ncolor_by: \"ccn\"\n\n# coloring rules to apply (last color prevail)\ncolor_rules :\n acquisition_canvas: \"#EEEEBB\"\n utils: \"grey\"\n constants: \"grey\"\n base_canvas_object: \"#EE8866\"\n popups: \"#222255\"\n forms: \"#77AADD\"\n\n###########################################################\n# Static graphs\nshowparents: False # add missing parent nodes\nremove_hyperconnect: 5 #remove leaves with more than 5 parents\n\n\"mmap_in.yml\" 34L, 840B\n```\n\nYou need to edit the path towards your sources, name the package, and set the \"colored_by\" option to either \"ccn\" (coloration by complexity), \"size\" (coloration by size, in number of lines), or \"rules\" (customized coloration, with the rules to edit under). \n\n\nFree ad.: if you just want to scan a Python module structure, try : https://pypi.org/project/code2flow/. Aw som' stuff.\n \n\n## Complete workflow\n\nHere follows the data flow of marauder's map command, showing you all the possibilities at hand.\n \n\n```bash\n mmap_in.yml\n |\n |\n |\n +------------------+-----------+-----------+--------+\n | | | |\n | +----+---+ | |\n | |treefile| | |\n | +----+---+ | +---v---+\n +----+---+ | | |imports|\n |treefunc| v | +---+---+\n +----+---+ file_tree.json | |\n | | | v\n | | |import_graphs.js\n v | | | |\n func_tree.json-------------------+---|||------+ | | |\n | | | | | | |\n | | | | | | |\n | | | +v---v---v+ |\n | +------------+ | | |callgraph| |\n | |regexp-input| | | +----+----+ |\n | +-------+----+ | | | |\n | | | | | |\n+------+ | +-----+ v | | | |\n| grep <------+-------->score<-rules.yml| | | |\n+---+--+ +--+--+ | | | |\n | | | | v |\n | score<-------+ | | callgraph.json |\n v | | | | |\ngraphical stats<---+ | | | |\n output | | | | |\n v | | | |\n func_tree_score.json | | | |\n | +------+ | | |\n +---+ | | +-v-----v-+\n | | +--------+ |showgraph|\n | | | +----+----+\n +v--v--v-+ |\n |showtree| v\n +---+----+ graphical\n | output\n v\n graphical\n output\n```\n\n\n## Acknowledgements\n\nMarauder's Map is a service created thanks to the [EXCELLERAT Center Of Excellence](https://www.excellerat.eu/wp/) (drifting from the flinter service). and is continued as part of the [EXCELLERAT P2 Center Of Excellence](https://www.excellerat.eu/).\n\nIt is also feeded by the community of the [COEC Center Of Excellence](https://coec-project.eu/). These projects ware funded by the European community.\n\n\n\n\n",
"bugtrack_url": null,
"license": null,
"summary": "Fast generation of Inports graph",
"version": "0.3.0",
"project_urls": {
"Bug Tracker": "https://gitlab.com/cerfacs/maraudersmap/-/issues",
"Documentation": "https://maraudersmap.readthedocs.io/en/latest/",
"Homepage": "https://gitlab.com/cerfacs/maraudersmap"
},
"split_keywords": [
"findimports",
"graph"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "b68c10f654f72108c683a6cb3759f1a40fc6c2cd55c153497d6db7c02c634234",
"md5": "015eec71294bfef94828e25760212b62",
"sha256": "7628bf88d16273a8c308029c8fd4b50ef95a198464675f797d76fafe7ee69cc9"
},
"downloads": -1,
"filename": "maraudersmap-0.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "015eec71294bfef94828e25760212b62",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 58699,
"upload_time": "2024-07-29T08:49:25",
"upload_time_iso_8601": "2024-07-29T08:49:25.225674Z",
"url": "https://files.pythonhosted.org/packages/b6/8c/10f654f72108c683a6cb3759f1a40fc6c2cd55c153497d6db7c02c634234/maraudersmap-0.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "673fbbbff8949971e56688025d27e88f10fd7d39fb8dde9c06c4509d9e27692c",
"md5": "7ca5bb32cb86fabb7b6c365566882dda",
"sha256": "610dbca13a653a0911d5687220fae3be9dd93c87fb20973a46b7499de8efc658"
},
"downloads": -1,
"filename": "maraudersmap-0.3.0.tar.gz",
"has_sig": false,
"md5_digest": "7ca5bb32cb86fabb7b6c365566882dda",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 53554,
"upload_time": "2024-07-29T08:49:26",
"upload_time_iso_8601": "2024-07-29T08:49:26.593694Z",
"url": "https://files.pythonhosted.org/packages/67/3f/bbbff8949971e56688025d27e88f10fd7d39fb8dde9c06c4509d9e27692c/maraudersmap-0.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-07-29 08:49:26",
"github": false,
"gitlab": true,
"bitbucket": false,
"codeberg": false,
"gitlab_user": "cerfacs",
"gitlab_project": "maraudersmap",
"lcname": "maraudersmap"
}
JSON