# Introduction
This project offers a high level python library to perform code analysis with CPG and Joern [server](https://docs.joern.io/server). Several API methods including integration with [NetworkX](https://networkx.org/documentation/stable/index.html) and [PyTorch Geometric](https://pytorch-geometric.readthedocs.io/en/latest/index.html) are offered to perform code analysis and research on complex code bases in a pythonic manner from cli and from notebooks.
[](https://github.com/appthreat/joern-lib/actions/workflows/pythonpublish.yml)
[](https://pepy.tech/project/joern-lib)
[](https://discord.gg/tmmtjCEHNV)
```shell
pip install joern-lib
# To install the optional science pack, clone this repo and use poetry > 1.5 to install the science group
poetry install --with science # cpu
poetry install --with science-cu117 # cuda 11.7
poetry install --with science-cu118 # cuda 11.8
```
## Notebook support
The repository includes docker compose configuration to interactively query the joern server with polynote notebooks.
## Usage
Run joern server and polynote locally.
```
git clone https://github.com/appthreat/joern-lib.git
# Edit docker-compose.yml to set sources directory
docker compose up -d
# podman-compose up --build
```
Navigate to http://localhost:8192 for an interactive polynote notebook. You could open one of the sample notebooks from the [contrib](./contrib/polynote/notebooks/joern-notes/) directory to learn about Joern server and this library.
### Common steps
Refer to the [API documentation](https://github.com/AppThreat/joern-lib/tree/main/api_docs/joern_lib) for programmatic usage.
```
python -m asyncio
```
Execute single query
```
from joern_lib import client, workspace, utils
from joern_lib.detectors import common as cpg
connection = await client.get("http://localhost:9000", "http://localhost:7072", "admin", "admin")
# connection = await client.get("http://localhost:9000")
res = await client.q(connection, "val a=1");
# {'response': 'a: Int = 1\n'}
```
Execute bulk query
```
res = await client.bulk_query(connection, ["val a=1", "val b=2", "val c=a+b"]);
# [{'response': 'a: Int = 1\n'}, {'response': 'b: Int = 2\n'}, {'response': 'c: Int = 3\n'}]
```
### Workspace
List workspaces
```
res = await workspace.ls(connection)
```
Get workspace path
```
res = await workspace.get_path(connection)
# /workspace (Response would be parsed)
```
Check if cpg exists
```
await workspace.cpg_exists(connection, "NodeGoat")
```
Import code for analysis
```
res = await workspace.import_code(connection, "/app", "NodeGoat")
# True
```
Import an existing CPG for analysis
```
res = await workspace.import_cpg(connection, "/app/sandbox/crAPI/cpg_out/crAPI-python-cpg.bin.zip", "crAPI-python")
```
Create a CPG with a remote cpggen server
```
res = await workspace.create_cpg(connection, "/app/sandbox/crAPI", out_dir="/app/sandbox/crAPI/cpg_out", languages="python", project_name="crAPI-python")
```
### CPG core
List files
```
res = await cpg.list_files(connection)
# list of files
```
Print call tree
```
res = await cpg.get_call_tree(connection, "com.example.vulnspring.WebController.issue:java.lang.String(org.springframework.ui.Model,java.lang.String)")
utils.print_tree(res)
```
### Java specific
```
from joern_lib.detectors import java
```
List http routes
```
await java.list_http_routes(connection)
```
### JavaScript specific
```
from joern_lib.detectors import js
```
List http routes
```
await js.list_http_routes(connection)
```
Name of the variable containing express()
```
await js.get_express_appvar(connection)
```
List of require statements
```
await js.list_requires(connection)
```
List of import statements
```
await js.list_imports(connection)
```
List of NoSQL DB collection names
```
await js.list_nosql_collections(connection)
```
Get HTTP sources
```
await js.get_http_sources(connection)
await js.get_http_sinks(connection)
```
### AWS
Requires TypeScript project
```
await js.list_aws_modules(connection)
```
## Troubleshooting
### No response from server
If Joern server stops responding after a while restart docker.
```
docker compose down
docker compose up -d
```
### Websockets connection closed error
Adding asyncio.sleep(0) seems to fix such errors.
```
# Workaround to fix websockets.exceptions.ConnectionClosedError
await asyncio.sleep(0)
```
Alternatively, use the sync api.
### pygraphviz refuses to install
```
pygraphviz/graphviz_wrap.c:2711:10: fatal error: graphviz/cgraph.h: No such file or directory
2711 | #include "graphviz/cgraph.h"
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1
```
Install `graphviz-devel` or `graphviz-dev` package for your OS. See [here](https://github.com/pygraphviz/pygraphviz/issues/155)
Raw data
{
"_id": null,
"home_page": "https://github.com/AppThreat/joern-lib",
"name": "joern-lib",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8.1,<3.12",
"maintainer_email": "",
"keywords": "joern,code analysis,static analysis",
"author": "Team AppThreat",
"author_email": "cloud@appthreat.com",
"download_url": "https://files.pythonhosted.org/packages/a1/3f/fc8a7b88565a728979be24bca961932caaab38460f745cc802188ca6cb6b/joern_lib-0.12.0.tar.gz",
"platform": null,
"description": "# Introduction\n\nThis project offers a high level python library to perform code analysis with CPG and Joern [server](https://docs.joern.io/server). Several API methods including integration with [NetworkX](https://networkx.org/documentation/stable/index.html) and [PyTorch Geometric](https://pytorch-geometric.readthedocs.io/en/latest/index.html) are offered to perform code analysis and research on complex code bases in a pythonic manner from cli and from notebooks.\n\n[](https://github.com/appthreat/joern-lib/actions/workflows/pythonpublish.yml)\n[](https://pepy.tech/project/joern-lib)\n[](https://discord.gg/tmmtjCEHNV)\n\n```shell\npip install joern-lib\n# To install the optional science pack, clone this repo and use poetry > 1.5 to install the science group\npoetry install --with science # cpu\npoetry install --with science-cu117 # cuda 11.7\npoetry install --with science-cu118 # cuda 11.8\n```\n\n## Notebook support\n\nThe repository includes docker compose configuration to interactively query the joern server with polynote notebooks.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n## Usage\n\nRun joern server and polynote locally.\n\n```\ngit clone https://github.com/appthreat/joern-lib.git\n# Edit docker-compose.yml to set sources directory\ndocker compose up -d\n\n# podman-compose up --build\n```\n\nNavigate to http://localhost:8192 for an interactive polynote notebook. You could open one of the sample notebooks from the [contrib](./contrib/polynote/notebooks/joern-notes/) directory to learn about Joern server and this library.\n\n### Common steps\n\nRefer to the [API documentation](https://github.com/AppThreat/joern-lib/tree/main/api_docs/joern_lib) for programmatic usage.\n\n```\npython -m asyncio\n```\n\nExecute single query\n\n```\nfrom joern_lib import client, workspace, utils\nfrom joern_lib.detectors import common as cpg\n\nconnection = await client.get(\"http://localhost:9000\", \"http://localhost:7072\", \"admin\", \"admin\")\n\n# connection = await client.get(\"http://localhost:9000\")\n\nres = await client.q(connection, \"val a=1\");\n\n# {'response': 'a: Int = 1\\n'}\n```\n\nExecute bulk query\n\n```\nres = await client.bulk_query(connection, [\"val a=1\", \"val b=2\", \"val c=a+b\"]);\n# [{'response': 'a: Int = 1\\n'}, {'response': 'b: Int = 2\\n'}, {'response': 'c: Int = 3\\n'}]\n```\n\n### Workspace\n\nList workspaces\n\n```\nres = await workspace.ls(connection)\n```\n\nGet workspace path\n\n```\nres = await workspace.get_path(connection)\n# /workspace (Response would be parsed)\n```\n\nCheck if cpg exists\n\n```\nawait workspace.cpg_exists(connection, \"NodeGoat\")\n```\n\nImport code for analysis\n\n```\nres = await workspace.import_code(connection, \"/app\", \"NodeGoat\")\n# True\n```\n\nImport an existing CPG for analysis\n\n```\nres = await workspace.import_cpg(connection, \"/app/sandbox/crAPI/cpg_out/crAPI-python-cpg.bin.zip\", \"crAPI-python\")\n```\n\nCreate a CPG with a remote cpggen server\n\n```\nres = await workspace.create_cpg(connection, \"/app/sandbox/crAPI\", out_dir=\"/app/sandbox/crAPI/cpg_out\", languages=\"python\", project_name=\"crAPI-python\")\n```\n\n### CPG core\n\nList files\n\n```\nres = await cpg.list_files(connection)\n# list of files\n```\n\nPrint call tree\n\n```\nres = await cpg.get_call_tree(connection, \"com.example.vulnspring.WebController.issue:java.lang.String(org.springframework.ui.Model,java.lang.String)\")\nutils.print_tree(res)\n```\n\n### Java specific\n\n```\nfrom joern_lib.detectors import java\n```\n\nList http routes\n\n```\nawait java.list_http_routes(connection)\n```\n\n### JavaScript specific\n\n```\nfrom joern_lib.detectors import js\n```\n\nList http routes\n\n```\nawait js.list_http_routes(connection)\n```\n\nName of the variable containing express()\n\n```\nawait js.get_express_appvar(connection)\n```\n\nList of require statements\n\n```\nawait js.list_requires(connection)\n```\n\nList of import statements\n\n```\nawait js.list_imports(connection)\n```\n\nList of NoSQL DB collection names\n\n```\nawait js.list_nosql_collections(connection)\n```\n\nGet HTTP sources\n\n```\nawait js.get_http_sources(connection)\nawait js.get_http_sinks(connection)\n```\n\n### AWS\n\nRequires TypeScript project\n\n```\nawait js.list_aws_modules(connection)\n```\n\n## Troubleshooting\n\n### No response from server\n\nIf Joern server stops responding after a while restart docker.\n\n```\ndocker compose down\ndocker compose up -d\n```\n\n### Websockets connection closed error\n\nAdding asyncio.sleep(0) seems to fix such errors.\n\n```\n# Workaround to fix websockets.exceptions.ConnectionClosedError\nawait asyncio.sleep(0)\n```\n\nAlternatively, use the sync api.\n\n### pygraphviz refuses to install\n\n```\npygraphviz/graphviz_wrap.c:2711:10: fatal error: graphviz/cgraph.h: No such file or directory\n 2711 | #include \"graphviz/cgraph.h\"\n | ^~~~~~~~~~~~~~~~~~~\n compilation terminated.\n error: command '/usr/bin/gcc' failed with exit code 1\n```\n\nInstall `graphviz-devel` or `graphviz-dev` package for your OS. See [here](https://github.com/pygraphviz/pygraphviz/issues/155)\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Python library for code analysis with CPG and Joern",
"version": "0.12.0",
"project_urls": {
"Homepage": "https://github.com/AppThreat/joern-lib",
"Repository": "https://github.com/AppThreat/joern-lib"
},
"split_keywords": [
"joern",
"code analysis",
"static analysis"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a27a6d7de16bc91aeb82b86f4eba778d26f6763aa33fa2ffb6b1935684adbe06",
"md5": "5437ec8dbea05bc9b48aad050e87704a",
"sha256": "1674d22112c01eeec983948fc6dfe9d455d2831d0cfcd0a22970185a1635d041"
},
"downloads": -1,
"filename": "joern_lib-0.12.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5437ec8dbea05bc9b48aad050e87704a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8.1,<3.12",
"size": 30097,
"upload_time": "2023-06-06T18:48:11",
"upload_time_iso_8601": "2023-06-06T18:48:11.610304Z",
"url": "https://files.pythonhosted.org/packages/a2/7a/6d7de16bc91aeb82b86f4eba778d26f6763aa33fa2ffb6b1935684adbe06/joern_lib-0.12.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a13ffc8a7b88565a728979be24bca961932caaab38460f745cc802188ca6cb6b",
"md5": "77d0493587b659aa5fb9f27ce44c3674",
"sha256": "d62e53aedbfd369371305905bcc83ff749bae8c95c3d6c5ce2fc737fdd4aa0c5"
},
"downloads": -1,
"filename": "joern_lib-0.12.0.tar.gz",
"has_sig": false,
"md5_digest": "77d0493587b659aa5fb9f27ce44c3674",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8.1,<3.12",
"size": 28310,
"upload_time": "2023-06-06T18:48:12",
"upload_time_iso_8601": "2023-06-06T18:48:12.869799Z",
"url": "https://files.pythonhosted.org/packages/a1/3f/fc8a7b88565a728979be24bca961932caaab38460f745cc802188ca6cb6b/joern_lib-0.12.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-06-06 18:48:12",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "AppThreat",
"github_project": "joern-lib",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "joern-lib"
}
JSON
