# Intro
[](https://badge.fury.io/py/libsa4py)
[](https://codecov.io/gh/saltudelft/libsa4py)
`LibSA4Py` is a static analysis library for Python, which extracts type hints and features for training ML-based type inference models.
- [Requirements](#requirements)
- [Quick Installation](#quick-installation)
- [Usage](#usage)
- [Processing projects](#processing-projects)
- [Merging projects](#merging-projects)
- [JSON Output](#json-output)
# Requirements
- Python 3.7 or newer (Python 3.8 is recommended)
- [Watchman](https://facebook.github.io/watchman/) (for running [pyre](https://pyre-check.org/)) [**Optional**]
- MacOS or Linux systems
# Quick Installation
```
git clone https://github.com/saltudelft/libsa4py.git
cd libsa4py && pip install .
```
# Usage
## Processing projects
Given Python repositories, run the following command to process source code files and generate JSON-formatted outputs:
```
libsa4py process --p $REPOS_PATH --o $OUTPUT_PATH --d $DUPLICATE_PATH --j $WORKERS_COUNT --l $LIMIT --c --no-nlp --pyre
```
Description:
- `--p $REPOS_PATH`: The path to the Python corpus or dataset.
- `--o $OUTPUT_PATH`: Path to store processed projects.
- `--d $DUPLICATE_PATH`: Path to duplicate files of the given dataset (i.e. jsonl.gz file produced by the [CD4Py](https://github.com/saltudelft/CD4Py) tool). [**Optional**]
- `--s`: Path to the CSV file for splitting the given dataset. [**Optional**]
- `--j $WORKERS_COUNT`: Number of workers for processing projects. [**Optional**, default=no. of available CPU cores]
- `--l $LIMIT`: Number of projects to be processed. [**Optional**]
- `--c`: Whether to ignore processed projects. [**Optional**, default=False]
- `--no-nlp`: Whether to apply standard NLP techniques to extracted identifiers. [**Optional**, default=True]
- `--pyre`: Whether to run `pyre` to infer the types of variables for given projects. [**Optional**, default=False]
- `--tc`: Whether to type-check type annotations in projects. [**Optional**, default=False]
## Merging projects
To merge all the processed JSON-formatted projects into a single dataframe, run the following command:
```
libsa4py merge --o $OUTPUT_PATH --l $LIMIT
```
Description:
- `--o $OUTPUT_PATH`: Path to the processed projects, used in the previous processing step.
- `--l $LIMIT`: Number of projects to be merged. [**Optional**]
## Applying types
To apply Pyre's inferred types to projects, run the following command:
```
libsa4py apply --p $REPOS_PATH --o $OUTPUT_PATH
```
Description:
- `--p $REPOS_PATH`: The path to the Python corpus or dataset.
- `--o $OUTPUT_PATH`: Path to the processed projects, used in the previous processing step.
# JSON Output
After processing each project, a JSON-formatted file is produced, which is described [here](https://github.com/saltudelft/light-sa-type-inf/blob/master/JSONOutput.md).
Raw data
{
"_id": null,
"home_page": "https://github.com/saltudelft/libsa4py",
"name": "libsa4py",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "libsa4py static analysis features type hints type inference machine learning python pipeline light-weight",
"author": "Amir M. Mir (TU Delft)",
"author_email": "mir-am@hotmail.com",
"download_url": "https://files.pythonhosted.org/packages/c8/43/bed5e39db0ff9b18354c2764be54155e668a8acf0aca5b216bdbe2a92943/libsa4py-0.4.0.tar.gz",
"platform": null,
"description": "# Intro\n[](https://badge.fury.io/py/libsa4py) \n\n[](https://codecov.io/gh/saltudelft/libsa4py)\n\n`LibSA4Py` is a static analysis library for Python, which extracts type hints and features for training ML-based type inference models.\n\n- [Requirements](#requirements)\n- [Quick Installation](#quick-installation)\n- [Usage](#usage)\n - [Processing projects](#processing-projects)\n - [Merging projects](#merging-projects)\n- [JSON Output](#json-output)\n\n# Requirements\n\n- Python 3.7 or newer (Python 3.8 is recommended)\n- [Watchman](https://facebook.github.io/watchman/) (for running [pyre](https://pyre-check.org/)) [**Optional**]\n- MacOS or Linux systems\n\n# Quick Installation\n\n```\ngit clone https://github.com/saltudelft/libsa4py.git\ncd libsa4py && pip install .\n```\n\n# Usage\n## Processing projects\nGiven Python repositories, run the following command to process source code files and generate JSON-formatted outputs:\n```\nlibsa4py process --p $REPOS_PATH --o $OUTPUT_PATH --d $DUPLICATE_PATH --j $WORKERS_COUNT --l $LIMIT --c --no-nlp --pyre\n```\n\nDescription:\n- `--p $REPOS_PATH`: The path to the Python corpus or dataset.\n- `--o $OUTPUT_PATH`: Path to store processed projects.\n- `--d $DUPLICATE_PATH`: Path to duplicate files of the given dataset (i.e. jsonl.gz file produced by the [CD4Py](https://github.com/saltudelft/CD4Py) tool). [**Optional**]\n- `--s`: Path to the CSV file for splitting the given dataset. [**Optional**]\n- `--j $WORKERS_COUNT`: Number of workers for processing projects. [**Optional**, default=no. of available CPU cores]\n- `--l $LIMIT`: Number of projects to be processed. [**Optional**]\n- `--c`: Whether to ignore processed projects. [**Optional**, default=False]\n- `--no-nlp`: Whether to apply standard NLP techniques to extracted identifiers. [**Optional**, default=True]\n- `--pyre`: Whether to run `pyre` to infer the types of variables for given projects. [**Optional**, default=False]\n- `--tc`: Whether to type-check type annotations in projects. [**Optional**, default=False]\n\n## Merging projects\nTo merge all the processed JSON-formatted projects into a single dataframe, run the following command:\n```\nlibsa4py merge --o $OUTPUT_PATH --l $LIMIT\n```\n\nDescription:\n- `--o $OUTPUT_PATH`: Path to the processed projects, used in the previous processing step.\n- `--l $LIMIT`: Number of projects to be merged. [**Optional**]\n\n## Applying types\nTo apply Pyre's inferred types to projects, run the following command:\n```\nlibsa4py apply --p $REPOS_PATH --o $OUTPUT_PATH\n```\n\nDescription:\n- `--p $REPOS_PATH`: The path to the Python corpus or dataset.\n- `--o $OUTPUT_PATH`: Path to the processed projects, used in the previous processing step.\n\n# JSON Output\nAfter processing each project, a JSON-formatted file is produced, which is described [here](https://github.com/saltudelft/light-sa-type-inf/blob/master/JSONOutput.md).\n",
"bugtrack_url": null,
"license": "",
"summary": "LibSA4Py: Light-weight static analysis for extracting type hints and features",
"version": "0.4.0",
"project_urls": {
"Homepage": "https://github.com/saltudelft/libsa4py"
},
"split_keywords": [
"libsa4py",
"static",
"analysis",
"features",
"type",
"hints",
"type",
"inference",
"machine",
"learning",
"python",
"pipeline",
"light-weight"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "60b5d68becae77e0e7c3417af0ea3a03ecfaa6d24af643e1712130bba4eee697",
"md5": "b09e7152f00bcbeae78ad6364daed973",
"sha256": "2c89559d5bcbedf75741b17c0c60ee2eb8ecf3d8273c54e61f19641ad94d160d"
},
"downloads": -1,
"filename": "libsa4py-0.4.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "b09e7152f00bcbeae78ad6364daed973",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 41797,
"upload_time": "2023-05-08T11:46:44",
"upload_time_iso_8601": "2023-05-08T11:46:44.619167Z",
"url": "https://files.pythonhosted.org/packages/60/b5/d68becae77e0e7c3417af0ea3a03ecfaa6d24af643e1712130bba4eee697/libsa4py-0.4.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c843bed5e39db0ff9b18354c2764be54155e668a8acf0aca5b216bdbe2a92943",
"md5": "ab5defaa45c52a10acfaef05f7723344",
"sha256": "953f4b2e2204af2bd5901f88779c0ed41287ba43e5f26096377b3507a0161163"
},
"downloads": -1,
"filename": "libsa4py-0.4.0.tar.gz",
"has_sig": false,
"md5_digest": "ab5defaa45c52a10acfaef05f7723344",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 88450,
"upload_time": "2023-05-08T11:46:46",
"upload_time_iso_8601": "2023-05-08T11:46:46.399443Z",
"url": "https://files.pythonhosted.org/packages/c8/43/bed5e39db0ff9b18354c2764be54155e668a8acf0aca5b216bdbe2a92943/libsa4py-0.4.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-05-08 11:46:46",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "saltudelft",
"github_project": "libsa4py",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "libcst",
"specs": [
[
">=",
"0.3.18"
]
]
},
{
"name": "numpy",
"specs": []
},
{
"name": "pandas",
"specs": []
},
{
"name": "nltk",
"specs": []
},
{
"name": "joblib",
"specs": []
},
{
"name": "tqdm",
"specs": []
},
{
"name": "docstring_parser",
"specs": []
},
{
"name": "dpu_utils",
"specs": []
},
{
"name": "pyre-check",
"specs": []
},
{
"name": "toml",
"specs": []
},
{
"name": "mypy",
"specs": []
}
],
"lcname": "libsa4py"
}
JSON
