#  [](https://scorecard.dev/viewer/?uri=github.com/TrustSource/ts-deepscan) 
# TrustSource DeepScan
Repository scanner for the identification of *_effective licenses, copyright information and identiifcation of encryption algorithms_*.
## What it does?
DeepScan takes the URL of a git repository as input, clones the content and scans all files contained (see below) for license indicators and copyright comments. All *findings* will be returned in a hirarchical structure (including references) and the cloned repo will be deleted afterwards.
We provided also a hosted version with a nice UI to review the findings, that you can find [here](https://deepscan.trustsource.io "Link to DeepScan Service"). The hosted version is available for free through the Web-UI. It is also part of the subscription based [TrustSource](https://app.trustsource.io)-Service. This version allows in addition the cloning and scanning of private repositories and (new!) the chained processing of complete SBOMs. It has been set up to support many and large repositories. If you are interested in using it from within your CI/CD pipeline, we offer API-subscriptions.
To learn more about TrustSource - the only process focussed Open Source Compliance tool, which supports all aspects of the open source compliance tool chain - visit our website at https://www.trustsource.io.
## What it covers?
Currently, the tool searches all kind of source files for comments. It covers a wide range of programming languages:
* AppleScript
* Assembly
* Batch
* C
* Clif
* Clojure
* CMake
* CSharp
* Dart
* Elixir
* Fortran
* GLSLF
* Go
* Haskell
* HTML
* Flex
* Java
* JavaScript
* Kotlin
* Lisp
* Matlab
* Markdown
* MySQL
* NinjaBuild
* ObjectiveC
* Perl
* Python
* R
* Ruby
* Rust
* Shader
* Shell
* SQL
* Swift
* SWIG
* TypeScript
* Yacc
* Yaml
In addition, it assesses all LICENSE, COPYING and README named files. Text, respectively comments in source files, will be analyzed for copyright statements and licenses will be verified against the original liecense text using a similarity analysis approach. The tool compiles results in a JSON file.
**PLEASE NOTE: The tool will ignore all kind of image, audio or video files.**
## How it works?
You may decide upon what to include in the scan. You may want to search for license indications only or also for copyright information. The latter will increase the operations time drastically.
The scanning has several phases. It seeks for SPDX license identifiers as well as compares license texts using a similarity analysis.
For the copyright scaning it has several routines but mainly builds on [ScanCode](https://github.com/nexB/scancode-toolkit "ScanCode Repository") (credits to [Philippe](https://github.com/nexB/scancode-toolkit/commits?author=pombredanne])).
## Where to learn more?
DeepScan is the CLI-version of the [TrustSource DeepScan](https://www.trustsource.io/deepscan "DeepScan Webservice UI") service. TrustSource DeepScan is a free repository scanning service operated by [TrustSource](https://www.trustsource.io "TrustSource Website"). To learn more about it and how it works, see the [TrustSource Knowledgebase](https://support.trustsource.io). It is the "Repo-Scanner" part of the OpenChain Reference Tooling. See [here](https://support.trustsource.io/hc/en-us/articles/360012782880-Architecture-Overview "Link to article") for a complete description of the reference model.
# Getting started / set up
The tool is realized in Python 3, we recommend to provide at least Python 3.7. To install it you may use pip:
```
pip install ts-deepscan
```
Currently, we do not support any alterantive installations, but we are planning to provide homebrew shortly.
To execute a scan, make sure the machine you are using to perform the scan has access to the internet, so that deepscan will be able to laod the latest Update of license data. This requires https (443). We regularly update the license texts. To provide this service, we kindly thank the [SPDX](https://spdx.org)-team. They do the heavy lifting on updating the licenses.
```
# clone a repo
git clone https://github.com/trustsource/ts-deepscan
# execute a scan
ts-deepscan scan -o results.json ./ts-deepscan
```
If you omit the -o parameter, the tool will use standard out as default. For further options please see the next section.
In the first run, it will download the license data from this repository. Occasionlaly we update the file. Original data is provided by SPDX org.
# TrustSource DeepScan CLI usage and configuration options
DeepScan may be used to scan a complete directory or a single file depending on the path argument. Pleaase note! we have split the scanning and uploading into two steps. thus you may scan and export the results locally instead of uploading the findigs directly to TrustSource. The formats supported are SPDX and CycloneDX.
## CLI help
To list all available commands:
```
ts-deepscan --help
```
To get help to particular commands:
```
ts-deepscan <COMMAND> --help
```
## Selecting the scan target
To scan a particular file:
```
ts-deepscan scan ./ts-deepscan/LICENSE
```
To scan a complete directory:
```
ts-deepscan scan ./ts-deepscan
```
To set a timeout for each file (not the whole scan) in seconds (Default: 60 seconds. Unlimited: -1):
```
ts-deepscan scan --timeout 120 ./ts-deepscan
```
You may use a piped command in combination with ```xargs``` to select only files matching a particular pattern:
```
find packages -type f -print | grep "linkedin" | xargs ts-deepscan scan -o somefile.json
```
The above command will check the directory `packages`and filter the list of file names for `linkedin`. This is then converted by xargs into parameters and added to the ts-deepscan command.
## Switching copyright collection on/off
The default is to scan for license indicators only. If you also want to invest into the costly search for copyright infromation, you must inlcude the *--include-copyright* parameter, e.g.:
```
ts-deepscan scan --include-copyright ./ts-deepscan
```
## Switching detection of used cryptographic algorithms on/off
If you want to turn on the search for used cryptographic algorithms in source code files (based on ScanOSS detection algorithms), you must inlcude the *--include-crypto* parameter, e.g.:
```
ts-deepscan scan --include-crypto ./ts-deepscan
```
## Selecting the output target
Default output will be stdout (console). To write in a file instead, use the -o option, e.g.:
```
ts-deepscan scan --include-copyright -o result.json ./ts-deepscan
```
## Exclude files from analysis
If you want to exclude files from analysis while scanning directories, you can specify one or more ignore file patterns using the *--ignore-pattern* option, e.g. the following command
```
ts-deepscan scan --ignore-pattern "*.pyc" -o result.json ./ts-deepscan
```
scans the directory 'ts-deepscan' and exludes all Python compiled files with an extension "pyc" from the analysis.
To add several ignore patterns in parallel, add several '--ignore-pattern' key and value pairs. Please note, that it will not be necessary to exclude binaries or multi-media files. DeepScan will identify and skip these automatically.
## Number of parallel jobs
The default is to use the maximal number of availbale CPU cores (cpu_count - 1) on Unix based systems and one core on Windows (due to slow process forking), if you want to specify the number of parallel jobs manually, you can use *-j* option, e.g.
```
ts-deepscan scan -j 4 -o result.json ./ts-deepscan
```
## Enable YARA analyser
To enable the YARA analyser you can use the *--enable-yara* option together with the *--yara-rules* option specifying the YARA rules location. The location can be either a file or a directory with files, the scanner loads all files with *.yar* extension:
```
ts-deepscan scan --enable-yara --yara-rules <YOUR RULES SET> -o result.json ./ts-deepscan
```
## Uploading scan results
For uploading previously stored scan results (scan executed with the '-o' option) to the TrustSource app use the following command:
```
ts-deepscan upload --module-name ts-deepscan --api-key <YOUR_API_KEY> result.json
```
Note: before uploading scan, a project has to be created in the TrustSource app as well as an API key has to be generated in the TrustSource Account settings
# Next steps
On our roadmap we see two capabilities being of relevance:
### a) Identify re-use of code
Already now, each file assessed is hashed, so that we do not need to assess the same file several times. This feature can be used to identify multiple appearances of the same file across different repositories. The subscription based implementation already has this capability. However, to resolve this reuirement, additional services are required. These we have provided in the context of the managed solution, e.g. storage of results, indexing, query, search and reply. DeepScan is set to support this scenario but will not further be extended on this.
### b) Identification of malicious pattern
One of the next steps will be to add support for the identification of malicious or weak programming patterns to help protecting the inbound supply chain.
### c) Treatment of linked repositories
In GitHub and other git derivatives it is possible to link a repository into your repository. The current version is treating these repositories as local code, which leads to difficulties with deep linking the original files. If there is higher demand to resolve this, we could invest into that direction.
### d) Identify license deltas
Given we identified a license text, with a similarity lower than 90%, it is possible that the original license text has been amended. Sometimes authors do add a clause or remove one. In a next version, we plan to outline these deltas. However, our analysis did show, that deviations typically require manual intervention for further analysis. That is, why we skipped activity in that direction for now.
In case you have specific need for any of the cases mentioned above, feel free to reach out and let us know about your case. If you have additional ideas, feel free to open a feature request in the issues section.
# Contribution, Contact and Support
Feel free to reach out to the [TrustSource Team](https://support.trustsource.io/hc/en-us/requests/new "TrustSource Knowledgebase") by dropping us a message or providing [issues](/org/ts-deepscan/issues). We'ld love to hear your feedback to learn and improve.
Contributions are welcome. Just clone, create your branch and send a pull request. Please make sure to agree to the [contribution agreement](/org/ContributionAgreeemnt.md "Contribution Agreement") and the [coding guidelines](/org/CodingGuidelines.md "Coding Guidelines").
If you like the tool and want to support our further work, feel free to support us with donations or sign an API-usage agreement.
Thank you & best regards!
Raw data
{
"_id": null,
"home_page": null,
"name": "ts-deepscan",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.13,>=3.10",
"maintainer_email": null,
"keywords": "DeepScan, TrustSource",
"author": "EACG GmbH",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/5c/0f/e3ae3be04413d41d04704fe9caf6ac805ad8e1bdeb216c9c0e8b03230003/ts_deepscan-2.3.5.tar.gz",
"platform": null,
"description": "\n#  [](https://scorecard.dev/viewer/?uri=github.com/TrustSource/ts-deepscan) \n\n\n# TrustSource DeepScan \nRepository scanner for the identification of *_effective licenses, copyright information and identiifcation of encryption algorithms_*. \n\n## What it does? \nDeepScan takes the URL of a git repository as input, clones the content and scans all files contained (see below) for license indicators and copyright comments. All *findings* will be returned in a hirarchical structure (including references) and the cloned repo will be deleted afterwards. \n\nWe provided also a hosted version with a nice UI to review the findings, that you can find [here](https://deepscan.trustsource.io \"Link to DeepScan Service\"). The hosted version is available for free through the Web-UI. It is also part of the subscription based [TrustSource](https://app.trustsource.io)-Service. This version allows in addition the cloning and scanning of private repositories and (new!) the chained processing of complete SBOMs. It has been set up to support many and large repositories. If you are interested in using it from within your CI/CD pipeline, we offer API-subscriptions.\n\nTo learn more about TrustSource - the only process focussed Open Source Compliance tool, which supports all aspects of the open source compliance tool chain - visit our website at https://www.trustsource.io.\n\n## What it covers? \nCurrently, the tool searches all kind of source files for comments. It covers a wide range of programming languages:\n* AppleScript\n* Assembly\n* Batch\n* C\n* Clif \n* Clojure \n* CMake \n* CSharp \n* Dart\n* Elixir \n* Fortran \n* GLSLF\n* Go\n* Haskell \n* HTML\n* Flex \n* Java \n* JavaScript \n* Kotlin \n* Lisp \n* Matlab \n* Markdown \n* MySQL \n* NinjaBuild \n* ObjectiveC \n* Perl \n* Python \n* R\n* Ruby \n* Rust \n* Shader\n* Shell \n* SQL \n* Swift \n* SWIG \n* TypeScript \n* Yacc \n* Yaml\n\nIn addition, it assesses all LICENSE, COPYING and README named files. Text, respectively comments in source files, will be analyzed for copyright statements and licenses will be verified against the original liecense text using a similarity analysis approach. The tool compiles results in a JSON file. \n\n**PLEASE NOTE: The tool will ignore all kind of image, audio or video files.**\n\n## How it works? \nYou may decide upon what to include in the scan. You may want to search for license indications only or also for copyright information. The latter will increase the operations time drastically. \nThe scanning has several phases. It seeks for SPDX license identifiers as well as compares license texts using a similarity analysis.\nFor the copyright scaning it has several routines but mainly builds on [ScanCode](https://github.com/nexB/scancode-toolkit \"ScanCode Repository\") (credits to [Philippe](https://github.com/nexB/scancode-toolkit/commits?author=pombredanne])).\n\n## Where to learn more? \nDeepScan is the CLI-version of the [TrustSource DeepScan](https://www.trustsource.io/deepscan \"DeepScan Webservice UI\") service. TrustSource DeepScan is a free repository scanning service operated by [TrustSource](https://www.trustsource.io \"TrustSource Website\"). To learn more about it and how it works, see the [TrustSource Knowledgebase](https://support.trustsource.io). It is the \"Repo-Scanner\" part of the OpenChain Reference Tooling. See [here](https://support.trustsource.io/hc/en-us/articles/360012782880-Architecture-Overview \"Link to article\") for a complete description of the reference model. \n\n# Getting started / set up\nThe tool is realized in Python 3, we recommend to provide at least Python 3.7. To install it you may use pip:\n```\npip install ts-deepscan\n```\nCurrently, we do not support any alterantive installations, but we are planning to provide homebrew shortly.\nTo execute a scan, make sure the machine you are using to perform the scan has access to the internet, so that deepscan will be able to laod the latest Update of license data. This requires https (443). We regularly update the license texts. To provide this service, we kindly thank the [SPDX](https://spdx.org)-team. They do the heavy lifting on updating the licenses. \n```\n# clone a repo\ngit clone https://github.com/trustsource/ts-deepscan\n\n# execute a scan\nts-deepscan scan -o results.json ./ts-deepscan\n```\nIf you omit the -o parameter, the tool will use standard out as default. For further options please see the next section.\nIn the first run, it will download the license data from this repository. Occasionlaly we update the file. Original data is provided by SPDX org.\n\n# TrustSource DeepScan CLI usage and configuration options\nDeepScan may be used to scan a complete directory or a single file depending on the path argument. Pleaase note! we have split the scanning and uploading into two steps. thus you may scan and export the results locally instead of uploading the findigs directly to TrustSource. The formats supported are SPDX and CycloneDX.\n\n## CLI help\n\nTo list all available commands:\n\n```\nts-deepscan --help\n```\n\nTo get help to particular commands:\n\n```\nts-deepscan <COMMAND> --help\n```\n\n## Selecting the scan target\nTo scan a particular file:\n```\nts-deepscan scan ./ts-deepscan/LICENSE\n```\nTo scan a complete directory:\n```\nts-deepscan scan ./ts-deepscan\n```\nTo set a timeout for each file (not the whole scan) in seconds (Default: 60 seconds. Unlimited: -1):\n```\nts-deepscan scan --timeout 120 ./ts-deepscan\n```\nYou may use a piped command in combination with ```xargs``` to select only files matching a particular pattern:\n```\nfind packages -type f -print | grep \"linkedin\" | xargs ts-deepscan scan -o somefile.json\n```\nThe above command will check the directory `packages`and filter the list of file names for `linkedin`. This is then converted by xargs into parameters and added to the ts-deepscan command. \n\n## Switching copyright collection on/off\nThe default is to scan for license indicators only. If you also want to invest into the costly search for copyright infromation, you must inlcude the *--include-copyright* parameter, e.g.: \n```\nts-deepscan scan --include-copyright ./ts-deepscan\n```\n\n## Switching detection of used cryptographic algorithms on/off\nIf you want to turn on the search for used cryptographic algorithms in source code files (based on ScanOSS detection algorithms), you must inlcude the *--include-crypto* parameter, e.g.: \n```\nts-deepscan scan --include-crypto ./ts-deepscan\n```\n\n## Selecting the output target\nDefault output will be stdout (console). To write in a file instead, use the -o option, e.g.:\n```\nts-deepscan scan --include-copyright -o result.json ./ts-deepscan\n```\n\n## Exclude files from analysis\nIf you want to exclude files from analysis while scanning directories, you can specify one or more ignore file patterns using the *--ignore-pattern* option, e.g. the following command \n```\nts-deepscan scan --ignore-pattern \"*.pyc\" -o result.json ./ts-deepscan\n```\nscans the directory 'ts-deepscan' and exludes all Python compiled files with an extension \"pyc\" from the analysis.\nTo add several ignore patterns in parallel, add several '--ignore-pattern' key and value pairs. Please note, that it will not be necessary to exclude binaries or multi-media files. DeepScan will identify and skip these automatically.\n\n## Number of parallel jobs\nThe default is to use the maximal number of availbale CPU cores (cpu_count - 1) on Unix based systems and one core on Windows (due to slow process forking), if you want to specify the number of parallel jobs manually, you can use *-j* option, e.g.\n\n```\nts-deepscan scan -j 4 -o result.json ./ts-deepscan\n```\n## Enable YARA analyser\nTo enable the YARA analyser you can use the *--enable-yara* option together with the *--yara-rules* option specifying the YARA rules location. The location can be either a file or a directory with files, the scanner loads all files with *.yar* extension: \n\n```\nts-deepscan scan --enable-yara --yara-rules <YOUR RULES SET> -o result.json ./ts-deepscan\n```\n\n## Uploading scan results\nFor uploading previously stored scan results (scan executed with the '-o' option) to the TrustSource app use the following command: \n\n```\nts-deepscan upload --module-name ts-deepscan --api-key <YOUR_API_KEY> result.json\n```\n\nNote: before uploading scan, a project has to be created in the TrustSource app as well as an API key has to be generated in the TrustSource Account settings \n\n# Next steps\nOn our roadmap we see two capabilities being of relevance:\n\n### a) Identify re-use of code\nAlready now, each file assessed is hashed, so that we do not need to assess the same file several times. This feature can be used to identify multiple appearances of the same file across different repositories. The subscription based implementation already has this capability. However, to resolve this reuirement, additional services are required. These we have provided in the context of the managed solution, e.g. storage of results, indexing, query, search and reply. DeepScan is set to support this scenario but will not further be extended on this. \n\n### b) Identification of malicious pattern\nOne of the next steps will be to add support for the identification of malicious or weak programming patterns to help protecting the inbound supply chain. \n\n### c) Treatment of linked repositories\nIn GitHub and other git derivatives it is possible to link a repository into your repository. The current version is treating these repositories as local code, which leads to difficulties with deep linking the original files. If there is higher demand to resolve this, we could invest into that direction. \n\n### d) Identify license deltas\nGiven we identified a license text, with a similarity lower than 90%, it is possible that the original license text has been amended. Sometimes authors do add a clause or remove one. In a next version, we plan to outline these deltas. However, our analysis did show, that deviations typically require manual intervention for further analysis. That is, why we skipped activity in that direction for now.\n\nIn case you have specific need for any of the cases mentioned above, feel free to reach out and let us know about your case. If you have additional ideas, feel free to open a feature request in the issues section. \n\n# Contribution, Contact and Support\nFeel free to reach out to the [TrustSource Team](https://support.trustsource.io/hc/en-us/requests/new \"TrustSource Knowledgebase\") by dropping us a message or providing [issues](/org/ts-deepscan/issues). We'ld love to hear your feedback to learn and improve.\nContributions are welcome. Just clone, create your branch and send a pull request. Please make sure to agree to the [contribution agreement](/org/ContributionAgreeemnt.md \"Contribution Agreement\") and the [coding guidelines](/org/CodingGuidelines.md \"Coding Guidelines\").\n\nIf you like the tool and want to support our further work, feel free to support us with donations or sign an API-usage agreement.\nThank you & best regards!\n",
"bugtrack_url": null,
"license": null,
"summary": "Repository scanner for the identification of effective licenses and copyright information.",
"version": "2.3.5",
"project_urls": {
"Homepage": "https://github.com/TrustSource/ts-deepscan.git"
},
"split_keywords": [
"deepscan",
" trustsource"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "76d2bcdeeca225de16c814feb98324c0d0bafb4830b2f43cb3064602ecd83629",
"md5": "3dad051566659e9ae0830687962f132e",
"sha256": "53dd65bbfae6a69ff0d935e579927f0d76c846d7895d1272c04015c11f56152e"
},
"downloads": -1,
"filename": "ts_deepscan-2.3.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3dad051566659e9ae0830687962f132e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.13,>=3.10",
"size": 680750,
"upload_time": "2025-08-27T15:40:09",
"upload_time_iso_8601": "2025-08-27T15:40:09.128550Z",
"url": "https://files.pythonhosted.org/packages/76/d2/bcdeeca225de16c814feb98324c0d0bafb4830b2f43cb3064602ecd83629/ts_deepscan-2.3.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "5c0fe3ae3be04413d41d04704fe9caf6ac805ad8e1bdeb216c9c0e8b03230003",
"md5": "a5d4082caf67ac4a0e63d16038e1f880",
"sha256": "dbcd32474d31fd7da77959595a36c119241729c66e909d49c6d08a4ba889d3a6"
},
"downloads": -1,
"filename": "ts_deepscan-2.3.5.tar.gz",
"has_sig": false,
"md5_digest": "a5d4082caf67ac4a0e63d16038e1f880",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.13,>=3.10",
"size": 1139650,
"upload_time": "2025-08-27T15:40:10",
"upload_time_iso_8601": "2025-08-27T15:40:10.870845Z",
"url": "https://files.pythonhosted.org/packages/5c/0f/e3ae3be04413d41d04704fe9caf6ac805ad8e1bdeb216c9c0e8b03230003/ts_deepscan-2.3.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-27 15:40:10",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "TrustSource",
"github_project": "ts-deepscan",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "spacy",
"specs": [
[
"~=",
"3.8.4"
]
]
},
{
"name": "nltk",
"specs": [
[
"~=",
"3.9"
]
]
},
{
"name": "text-unidecode",
"specs": [
[
"~=",
"1.3"
]
]
},
{
"name": "requests",
"specs": [
[
"~=",
"2.32.3"
]
]
},
{
"name": "click",
"specs": [
[
"==",
"8.1.7"
]
]
},
{
"name": "osadl_matrix",
"specs": [
[
"==",
"2024.5.22.10535"
]
]
},
{
"name": "pyminr",
"specs": [
[
"~=",
"0.1.1"
]
]
},
{
"name": "scancode-toolkit",
"specs": [
[
"~=",
"32.3.2"
]
]
},
{
"name": "dataclasses-json",
"specs": [
[
"~=",
"0.6.7"
]
]
},
{
"name": "gitignore-parser",
"specs": [
[
"~=",
"0.1.11"
]
]
},
{
"name": "tqdm",
"specs": [
[
"~=",
"4.67.0"
]
]
},
{
"name": "scanoss",
"specs": [
[
"~=",
"1.20.4"
]
]
},
{
"name": "yara-python",
"specs": [
[
"~=",
"4.5.0"
]
]
}
],
"lcname": "ts-deepscan"
}
JSON
