scantist-karby


Namescantist-karby JSON
Version 0.0.2 PyPI version JSON
download
home_page
Summaryscan the provided projects by using snyk, whitesource or scantist, and convert the result into scantist's format
upload_time2021-04-08 11:25:33
maintainer
docs_urlNone
authorscantist
requires_python
license
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # Karby

## Prerequisite

1. install `pipenv` according to any tutorial.
2. enter `Karby` main folder, execute `pipenv shell`, run `pipenv install` to install the dependencies.
3. There is a filed named `.env.example`, copy one and change it to `.env` . This is where we put all the system environment to. Every time you initiate the `pipenv shell`, the system will read those environment variables. All the credentials, tokens or password should not be directly placed in code, but here in `.env`

## Running

now you should be able to run the script.

The entrance is `sca_tool_scan.py`. the full command example is

```
python sca_tool_scan.py [snyk|scantist|whitesource] [upload|cmd] <any/git/url> -name <this is optional> -output <this is optional>
```

> you may not be able to run this because you don't have snyk credentials in your environment
>
> for parameter details, you should run `python sca_tool_scan.py -h`



## Basic Idea

**Motivation**

Basically, vulnerability detection tools are aim to find all the components in a given project as well as their vulnerabilities. But different tools are so different in using. `karby` is a project aims to collect all of these vulnerability detection tools and simplify their usage. 

**Input and output**

The input should be a single project URL. It could be a git hub URL(for upload scan) or local project directory(for cmd scan).

The output should be a CSV file in the format of `Scantist`. You could see the example in `karby/report_format_example` . There should be 2 files, the format is `<tool_name>-<component or issue>-<project name>.csv`.

- **component file**: there are 10 fields in this report, but actually only 2 of them matters: `Library` and `Version`. Just leave the rest empty
- **issue file**: there are 10 fields in this report, but actually only 3 of them matters: `Library` `Library Version` and `Public ID`. `Public ID` is the CVE public Id, if it is not provided, you can also left it empty. 	

**Scan types**

I defined 2 types of scan: `cmd` and `upload`. They are also called: `ci scan` and `airgap scan`. 

- `cmd` scan need the user to provide the local **well-build** project directory, and trigger the scan locally in command line. Then, you can use any method to collect the result. You can download the report by url or just parse the output from command line. We don't care, but keep it simple and guarantee the final format(Scantist format)
- `upload` scan need the user to provide the github url to trigger the scan. Or say, trigger by calling APIs given by that tool. This method do not require local build, so we call this **un-build ** project scan as `airgap scan`. Not all tools support this `upload` scan method. Need an example? You can refer to snyk api scan. 


## Scan Tools

### Whitesource   

**Mode avaliable**
- `cmd`: support
- `upload`: support (this will not trigger a new scan, but find the project online by given name)

**Prerequisite** 
1. One of the following Java versions:
   - Java JDK 8
   - Java JRE 8
   - Java JDK 11
2. Depending on your project type, ensure that the relevant package managers are installed. You can refer to "Prerequisites" section in the following link for more details.
   - https://whitesource.atlassian.net/wiki/spaces/WD/pages/1140852201/Getting+Started+with+the+Unified+Agent

**Setup**
1. Download _wss-unified-agent.jar_ from the following links:
   - https://unified-agent.s3.amazonaws.com/wss-unified-agent.jar

**Environment Variables**
- `WHITESOURCE_API_KEY`: WhiteSource API key, a unique identifier of your organization. Should be a __256-bit hex number__. 
- `WHITESOURCE_USER_KEY`: WhiteSource User Key. Can be generated from the Profile page in WhiteSource account. In order to get a scan report, the user key must be generated by __administrators__.  Should be a __256-bit hex number__. 
- `WHITESOURCE_PRODUCT_NAME`: Name of your product. 
- `PATH_TO_WHITESOURCE_JAR` (`cmd` mode only): Path to the Whitesource local agent .jar file (e.g. /path/to/jar/wss-unified-agent.jar)
- `PATH_TO_WHITESOURCE_CFG` (`cmd` mode only, optional): Path to the Whitesource local agent .config file (e.g. /path/to/cfg/wss-unified-agent.cfg). We put a prepared one inside folde `resources`. If you need any special modification, go to `resources/wss-unified-agent.config`

### snyk

**Mode avaliable**
- `cmd`: support
- `upload`: support 

**Prerequisite**
1. install snyk command line tool in the local environment using
    ```
    npm install -g snyk
    ``` 
    > https://support.snyk.io/hc/en-us/articles/360003812538-Install-the-Snyk-CLI#UUID-b346a6dc-682a-471f-bdc2-a82d8e5f8b6e
2. see if snyk is successfully installed
    ```
    snyk version
   ```

**Environment Variables**
- `SNYK_USR_TOKEN`: can be retrieved from personal setting: https://app.snyk.io/manage/integrations
- `SNYK_ORG_ID`: can be retrieved from personal setting
- `SNYK_INTEGRATION_ID`: can be retrieved from personal setting

### scantist

**Mode avaliable**
- `cmd`: support
- `upload`: not support 


**Prerequisite**
1. install scantist command line tool in the local environment using
    ```
    pip install scantist-command-tool
    ``` 
**Environment Variables**
- `SCANTIST_EMAIL`: scantist account email
- `SCANTIST_PSW`: scantist account password
- `SCANTIST_BASEURL`: target base url, default to "https://api-staging.scantist.io/"


            

Raw data

            {
    "_id": null,
    "home_page": "",
    "name": "scantist-karby",
    "maintainer": "",
    "docs_url": null,
    "requires_python": "",
    "maintainer_email": "",
    "keywords": "",
    "author": "scantist",
    "author_email": "lida@scantist.com",
    "download_url": "https://files.pythonhosted.org/packages/95/16/83615d472c8d73e8fb2c61941cf1b74c4e96ae492e661b26d23614bc9e89/scantist-karby-0.0.2.tar.gz",
    "platform": "",
    "description": "# Karby\n\n## Prerequisite\n\n1. install `pipenv` according to any tutorial.\n2. enter `Karby` main folder, execute `pipenv shell`, run `pipenv install` to install the dependencies.\n3. There is a filed named `.env.example`, copy one and change it to `.env` . This is where we put all the system environment to. Every time you initiate the `pipenv shell`, the system will read those environment variables. All the credentials, tokens or password should not be directly placed in code, but here in `.env`\n\n## Running\n\nnow you should be able to run the script.\n\nThe entrance is `sca_tool_scan.py`. the full command example is\n\n```\npython sca_tool_scan.py [snyk|scantist|whitesource] [upload|cmd] <any/git/url> -name <this is optional> -output <this is optional>\n```\n\n> you may not be able to run this because you don't have snyk credentials in your environment\n>\n> for parameter details, you should run `python sca_tool_scan.py -h`\n\n\n\n## Basic Idea\n\n**Motivation**\n\nBasically, vulnerability detection tools are aim to find all the components in a given project as well as their vulnerabilities. But different tools are so different in using. `karby` is a project aims to collect all of these vulnerability detection tools and simplify their usage. \n\n**Input and output**\n\nThe input should be a single project URL. It could be a git hub URL(for upload scan) or local project directory(for cmd scan).\n\nThe output should be a CSV file in the format of `Scantist`. You could see the example in `karby/report_format_example` . There should be 2 files, the format is `<tool_name>-<component or issue>-<project name>.csv`.\n\n- **component file**: there are 10 fields in this report, but actually only 2 of them matters: `Library` and `Version`. Just leave the rest empty\n- **issue file**: there are 10 fields in this report, but actually only 3 of them matters: `Library` `Library Version` and `Public ID`. `Public ID` is the CVE public Id, if it is not provided, you can also left it empty. \t\n\n**Scan types**\n\nI defined 2 types of scan: `cmd` and `upload`. They are also called: `ci scan` and `airgap scan`. \n\n- `cmd` scan need the user to provide the local **well-build** project directory, and trigger the scan locally in command line. Then, you can use any method to collect the result. You can download the report by url or just parse the output from command line. We don't care, but keep it simple and guarantee the final format(Scantist format)\n- `upload` scan need the user to provide the github url to trigger the scan. Or say, trigger by calling APIs given by that tool. This method do not require local build, so we call this **un-build ** project scan as `airgap scan`. Not all tools support this `upload` scan method. Need an example? You can refer to snyk api scan. \n\n\n## Scan Tools\n\n### Whitesource   \n\n**Mode avaliable**\n- `cmd`: support\n- `upload`: support (this will not trigger a new scan, but find the project online by given name)\n\n**Prerequisite** \n1. One of the following Java versions:\n   - Java JDK 8\n   - Java JRE 8\n   - Java JDK 11\n2. Depending on your project type, ensure that the relevant package managers are installed. You can refer to \"Prerequisites\" section in the following link for more details.\n   - https://whitesource.atlassian.net/wiki/spaces/WD/pages/1140852201/Getting+Started+with+the+Unified+Agent\n\n**Setup**\n1. Download _wss-unified-agent.jar_ from the following links:\n   - https://unified-agent.s3.amazonaws.com/wss-unified-agent.jar\n\n**Environment Variables**\n- `WHITESOURCE_API_KEY`: WhiteSource API key, a unique identifier of your organization. Should be a __256-bit hex number__. \n- `WHITESOURCE_USER_KEY`: WhiteSource User Key. Can be generated from the Profile page in WhiteSource account. In order to get a scan report, the user key must be generated by __administrators__.  Should be a __256-bit hex number__. \n- `WHITESOURCE_PRODUCT_NAME`: Name of your product. \n- `PATH_TO_WHITESOURCE_JAR` (`cmd` mode only): Path to the Whitesource local agent .jar file (e.g. /path/to/jar/wss-unified-agent.jar)\n- `PATH_TO_WHITESOURCE_CFG` (`cmd` mode only, optional): Path to the Whitesource local agent .config file (e.g. /path/to/cfg/wss-unified-agent.cfg). We put a prepared one inside folde `resources`. If you need any special modification, go to `resources/wss-unified-agent.config`\n\n### snyk\n\n**Mode avaliable**\n- `cmd`: support\n- `upload`: support \n\n**Prerequisite**\n1. install snyk command line tool in the local environment using\n    ```\n    npm install -g snyk\n    ``` \n    > https://support.snyk.io/hc/en-us/articles/360003812538-Install-the-Snyk-CLI#UUID-b346a6dc-682a-471f-bdc2-a82d8e5f8b6e\n2. see if snyk is successfully installed\n    ```\n    snyk version\n   ```\n\n**Environment Variables**\n- `SNYK_USR_TOKEN`: can be retrieved from personal setting: https://app.snyk.io/manage/integrations\n- `SNYK_ORG_ID`: can be retrieved from personal setting\n- `SNYK_INTEGRATION_ID`: can be retrieved from personal setting\n\n### scantist\n\n**Mode avaliable**\n- `cmd`: support\n- `upload`: not support \n\n\n**Prerequisite**\n1. install scantist command line tool in the local environment using\n    ```\n    pip install scantist-command-tool\n    ``` \n**Environment Variables**\n- `SCANTIST_EMAIL`: scantist account email\n- `SCANTIST_PSW`: scantist account password\n- `SCANTIST_BASEURL`: target base url, default to \"https://api-staging.scantist.io/\"\n\n",
    "bugtrack_url": null,
    "license": "",
    "summary": "scan the provided projects by using snyk, whitesource or scantist, and convert the result into scantist's format",
    "version": "0.0.2",
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "7be70fed598e07a0756880aee36cf442",
                "sha256": "b625793549befc733c3950160f40f035a3907b2cece1adb6aceef30b67144aba"
            },
            "downloads": -1,
            "filename": "scantist_karby-0.0.2-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "7be70fed598e07a0756880aee36cf442",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 25737,
            "upload_time": "2021-04-08T11:25:32",
            "upload_time_iso_8601": "2021-04-08T11:25:32.153230Z",
            "url": "https://files.pythonhosted.org/packages/41/0e/c893d3db7d59832d1b1adafe313a01051656ecec6d93aaab17ee88de9bd8/scantist_karby-0.0.2-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "f426c807b139a569466c953f77269678",
                "sha256": "a683b63282d21a3a5bc78505ba05e05d27fc7950588531be1cd632f328c78854"
            },
            "downloads": -1,
            "filename": "scantist-karby-0.0.2.tar.gz",
            "has_sig": false,
            "md5_digest": "f426c807b139a569466c953f77269678",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 14719,
            "upload_time": "2021-04-08T11:25:33",
            "upload_time_iso_8601": "2021-04-08T11:25:33.425581Z",
            "url": "https://files.pythonhosted.org/packages/95/16/83615d472c8d73e8fb2c61941cf1b74c4e96ae492e661b26d23614bc9e89/scantist-karby-0.0.2.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2021-04-08 11:25:33",
    "github": false,
    "gitlab": false,
    "bitbucket": false,
    "lcname": "scantist-karby"
}
        
Elapsed time: 0.24573s