# ACT Workers
## Introduction
This repository contains workers for the [ACT platform](https://github.com/mnemonic-no/act-platform).
The source code the workers are available on [github](https://github.com/mnemonic-no/act-workers).
# Changelog
## 2.0.0
* Configuration is moved from `~/.config/actworkers/actworkers.ini` to `~/.config/act/act.ini`
* The old scio worker is removed and the new scio-worker (act-scio2) is renamed to act-scio
# Setup
To use the workers, install from PyPi:
```bash
sudo pip3 install act-workers
```
This will install scripts for all workers:
* act-argus-case
* act-attack
* act-country-regions
* act-fact-chain-helper (see usage below)
* act-feed
* act-ip-filter
* act-misp-feeds
* act-mnemonic-pdns
* act-scio
* act-search-graph
* act-shadowserver-asn
* act-uploader
* act-url-shorter-unpack
* act-ta-helper (see usage below)
* act-veris
* act-vt
# Origins
All workers support the optional arguments --origin-name and --origin-id. If they are not specified, facts will be added with the origin of the user performing the upload to the platform.
For managing origins, use the tool `act-origin` in `act-admin` package on pypi.
# Access mode
By default, all facts will have access-mode "RoleBased", which means that the user needs access to the organization specified when creating the facts.
The access mode can be explicit set with `--access-mode`, e.g. like this, to set all facts to Public access mode:
```
--access-mode Public
```
There are also some workers, e.g. `act-scio` and `act-mnemoninc-pdns` that will set access-mode based on the input document (TLP green/white -> access-mode=Public), unless you explicit configures it to not do so.
# Organization
All workers support the optional arguments `--organization` If they are not specified, facts will be added with the organization of the origin or the user performing the upload to the platform (if not set by the origin.
# Worker usage
To print facts to stdout:
```bash
$ act-country-regions
{"type": "memberOf", "value": "", "accessMode": "Public", "sourceObject": {"type": "country", "value": "Afghanistan"}, "destinationObject": {"type": "subRegion", "value": "Southern Asia"}, "bidirectionalBinding": false}
{"type": "memberOf", "value": "", "accessMode": "Public", "sourceObject": {"type": "subRegion", "value": "Southern Asia"}, "destinationObject": {"type": "region", "value": "Asia"}, "bidirectionalBinding": false}
(...)
```
Or print facts as text representation:
```bash
$ act-country-regions --output-format str
(country/Afghanistan) -[memberOf]-> (subRegion/Southern Asia)
(subRegion/Southern Asia) -[memberOf]-> (region/Asia)
(...)
```
To add facts directly to the platform, include the act-baseurl and user-id options:
```bash
$ act-country-regions --act-baseurl http://localhost:8888 --user-id 1
```
# Configuration
All workers support options specified as command line arguments, environment variables and in a configuration file.
A utility to show and start with a default ini file is also included:
```bash
act-worker-config --help
usage: ACT worker config [-h] {show,user,system}
positional arguments:
{show,user,system}
optional arguments:
-h, --help show this help message and exit
show - Print default config
user - Copy default config to /home/fredrikb/.config/actworkers/actworkers.ini
system - Copy default config to /etc/actworkers.ini
```
You can see the default options in [act/workers/etc/actworkers.ini](act/workers/etc/actworkers.ini).
The configuration presedence are (from lowest to highest):
1. Defaults (shown in --help for each worker)
2. INI file
3. Environment variable
4. Command line argument
## INI-file
Arguments are parsed in two phases. First, it will look for the argument --config argument
which can be used to specify an alternative location for the ini file. If not --config argument
is given it will look for an ini file in the following locations:
/etc/<CONFIG_FILE_NAME>
~/.config/<CONFIG_ID>/<CONFIG_FILE_NAME> (or directory specified by $XDG_CONFIG_HOME)
The ini file contains a "[DEFAULT]" section that will be used for all workers.
In addition there are separate sections for each worker which you can use to configure
worker-specific options, and override default options.
## Environment variables
The configuration step will also look for environment variables in uppercase and
with "-" replaced with "_". For the example for the option "cert-file" it will look for the
enviornment variable "$CERT_FILE".
## Requirements
All workers requires python version >= 3.5 and the act-api library:
* [act-api](https://github.com/mnemonic-no/act-api-python) (act-api on [pypi](https://pypi.org/project/act-api/))
In addition some of the libraries might have additional requirements. See requirements.txt for a full list of all requirements.
# Proxy configuration
Workers will honor the `proxy-string` option on the command line when connecting to external APIs. However, if you need to
use the proxy to connect to the ACT platform (--act-baseurl), you will need to add the "--proxy-platform" switch:
```bash
echo -n www.mnemonic.no | act-vt --proxy-string <PROXY> --user-id <USER-ID> --act-baseurl <ACT-HOST> --proxy-platform
```
# Local development
Use pip to install in [local development mode](https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs). act-workers (and act-api) uses namespacing, so it is not compatible with using `setup.py install` or `setup.py develop`.
In repository, run:
```bash
pip3 install --user -e .
```
# search
A worker to run graph queries is also included. A sample search config is inscluded in `etc/searc_jobs.ini`:
```bash
act-search-graph etc/search_jobs.ini
```
# act-feed
`act-feed` can be used to download feed bundles from a remote uri. Feed uri must have a file, manifest.json, that lists all bundle files that can be download by the feed worker:
```json
{
"bundles": {
"000bd93b8ce1fa2acfa448fa916083d76e64b008e336d6f54ad89f12f4232dcf.gz": 1636355543,
"0598e898979bc851d58b440a3ea248fe64d9df4d2ab1665da3be73c6d13df411.gz": 1636681082,
"0c03f87d142345a1bbea0867c145608e70ab2c0ea8282b5e06833f5f2b0f5f04.gz": 1636366321,
},
"updated": 1636699081
}
```
A local cache will be stored stored to keep track of the last bundle download, so bundles will not be downloaded multiple times.
You can either let act-feed download files to a local directory:
```bash
act-feed --feed-uri https://act.mnemonic.no/feed --dump-dir [path]
```
output all facts to stdout:
```bash
act-feed --feed-uri https://act.mnemonic.no/feed
```
or upload facts directory to the platform
```bash
act-feed --feed-uri https://act.mnemonic.no/feed --act-baseurl [platform UR] --user-id [USERID]
```
# act-fact-chain-helper
The act-fact-chain-helper is a utility that can be used to add facts chains based on known start and end nodes. This can be usefull when adding
information from reports where do not have all the information availiable to you.
## example
We know from a report that the threat actor APT 1 uses the tool Mimikatz
```bash
$ act-fact-chain-helper --output-format str --start threatActor/apt1 --end tool/mimikatz --avoid mentions --include content
(incident/[placeholder[1194f1ba4ea8ded28250a0193327654e5ee305bce03741a3e6f183f03b5c6b35]]) -[attributedTo]-> (threatActor/apt1)
(event/[placeholder[1194f1ba4ea8ded28250a0193327654e5ee305bce03741a3e6f183f03b5c6b35]]) -[attributedTo]-> (incident/[placeholder[1194f1ba4ea8ded28250a0193327654e5ee305bce03741a3e6f183f03b5c6b35]])
(content/[placeholder[1194f1ba4ea8ded28250a0193327654e5ee305bce03741a3e6f183f03b5c6b35]]) -[observedIn]-> (event/[placeholder[1194f1ba4ea8ded28250a0193327654e5ee305bce03741a3e6f183f03b5c6b35]])
(content/[placeholder[1194f1ba4ea8ded28250a0193327654e5ee305bce03741a3e6f183f03b5c6b35]]) -[classifiedAs]-> (tool/mimikatz)
```
In this case we do need to hint at the tool that it should include the "content" object and avoid "mention" facts.
The source and destination is interchangeable. In the example above, the fact chain created by swapping the source and destination is identical.
by adding the ```--act-baseurl``` option, the data will be stored to the platform.
The tool works by finding the shortest path though the datamodel. The options --avoid and --include will modify the cost of traversing certain nodes or edges. This tool is considered experimental.l.
# act-ta-helper
You can use `act-ta-helper` to create facts, including placeholders, for typical scenarios where you have some
information of TA activity, but not all.
This worker should not be used if any of objects listed as placeholders below are known.
```bash
--ta THREAT-ACTOR Threat Actor Name (Required)
--ta-located-in COUNTRY Country where threat actor is located
--campaign CAMPAIGN Campaign
--techniques TECHNIQUES List of techniques (commaseparated).
Support technique IDs, e.g. T1002 and name
e.g "Valid Accounts"
--tools TOOLS List of tools (commaseparated)
--sectors SECTORS List of Sectors (commaseparated)
--target-countries COUNTRIES Target Countries (commaseparated)
```
You can run the tool with the options above, and add `--output-format str` to se a suggested list of facts.
You can then add `--act-baseurl` and `--user-id` to add the facts to a platform instance.
You can add all options on a single command line, like this example:
```bash
act-ta-helper \
--output-format str \
--ta HAFNIUM \
--sectors pharmaceuticals,education,defense,non-profit \
--tools PsExec,Procdump,7-Zip,Nishang,PowerCat,WinRar,SIMPLESEESHARP,SPORTSBALL,ChinaChopper,ASPXSPY,Covenant \
--target-countries "Denmark,United States of America" \
--campaign "Operation Exchange Marauder" \
--ta-located-in China \
--techniques T1588,T1003,T1190,T1560,T1583,T1071,T1114,T1567,T1136,T1021
```
The following facts/placeholders will be created based on the options (placeholders are marked with [*]):
## `ta-located-in`
Use this if you know the threat actor name and country where TA is located, but organization is unknown.
```
threatActor -[attributedTo]-> organization[*] -[locatedIn]-> country
```
## `campaign`
Use this if you know the threat actor and campaign name, but incident is unknown.
```
threatActor -[attributedTo]-> incident[*] -[attributedTo]-> campaign
```
## `techniques`
Use this if you know the threat actor and technique, but event and incident is unknown.
```
technique <-[classifiedAs]- event[*] -[attributedTo]-> incident[*] -[attributedTo]-> threatActor
```
## `tools`
Use if tool and threat actor is known, but incident, content and event is unknown.
```
threatActor <-[attributedTo]- incident[*] <-[observedIn]- content[*] -[classifiedAs]-> tool
```
## `sectors`
Use this if threat actor and target sector is known, but organization and incident is unknown.
```
threatActor -[attributedTo]-> incident[*] -[targets]-> organization[*] -[memberOf]-> sector
```
## `target-countries`
Use this if threat actor and target country is known, but organization and incident is unknown.
```
threatActor -[attributedTo]-> incident[*] -[targets]-> organization[*] -[locatedIn]-> country
```
Raw data
{
"_id": null,
"home_page": "https://github.com/mnemonic-no/act-workers",
"name": "act-workers",
"maintainer": "",
"docs_url": null,
"requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <4",
"maintainer_email": "",
"keywords": "ACT,mnemonic",
"author": "mnemonic AS",
"author_email": "opensource@mnemonic.no",
"download_url": "https://files.pythonhosted.org/packages/64/82/f8321050a6edea2366a8f08d18c073161bb6c5caa4d4075abaa0f1034a21/act-workers-2.1.13.tar.gz",
"platform": null,
"description": "# ACT Workers\n\n## Introduction\n\nThis repository contains workers for the [ACT platform](https://github.com/mnemonic-no/act-platform).\n\nThe source code the workers are available on [github](https://github.com/mnemonic-no/act-workers).\n\n# Changelog\n\n## 2.0.0\n\n* Configuration is moved from `~/.config/actworkers/actworkers.ini` to `~/.config/act/act.ini`\n* The old scio worker is removed and the new scio-worker (act-scio2) is renamed to act-scio\n\n# Setup\n\nTo use the workers, install from PyPi:\n\n```bash\nsudo pip3 install act-workers\n```\n\nThis will install scripts for all workers:\n\n* act-argus-case\n* act-attack\n* act-country-regions\n* act-fact-chain-helper (see usage below)\n* act-feed\n* act-ip-filter\n* act-misp-feeds\n* act-mnemonic-pdns\n* act-scio\n* act-search-graph\n* act-shadowserver-asn\n* act-uploader\n* act-url-shorter-unpack\n* act-ta-helper (see usage below)\n* act-veris\n* act-vt\n\n# Origins\n\nAll workers support the optional arguments --origin-name and --origin-id. If they are not specified, facts will be added with the origin of the user performing the upload to the platform.\n\nFor managing origins, use the tool `act-origin` in `act-admin` package on pypi.\n\n# Access mode\nBy default, all facts will have access-mode \"RoleBased\", which means that the user needs access to the organization specified when creating the facts.\n\nThe access mode can be explicit set with `--access-mode`, e.g. like this, to set all facts to Public access mode:\n\n```\n--access-mode Public\n```\n\nThere are also some workers, e.g. `act-scio` and `act-mnemoninc-pdns` that will set access-mode based on the input document (TLP green/white -> access-mode=Public), unless you explicit configures it to not do so.\n\n# Organization\n\nAll workers support the optional arguments `--organization` If they are not specified, facts will be added with the organization of the origin or the user performing the upload to the platform (if not set by the origin.\n\n\n# Worker usage\n\nTo print facts to stdout:\n\n```bash\n$ act-country-regions\n{\"type\": \"memberOf\", \"value\": \"\", \"accessMode\": \"Public\", \"sourceObject\": {\"type\": \"country\", \"value\": \"Afghanistan\"}, \"destinationObject\": {\"type\": \"subRegion\", \"value\": \"Southern Asia\"}, \"bidirectionalBinding\": false}\n{\"type\": \"memberOf\", \"value\": \"\", \"accessMode\": \"Public\", \"sourceObject\": {\"type\": \"subRegion\", \"value\": \"Southern Asia\"}, \"destinationObject\": {\"type\": \"region\", \"value\": \"Asia\"}, \"bidirectionalBinding\": false}\n(...)\n```\n\nOr print facts as text representation:\n\n```bash\n$ act-country-regions --output-format str\n(country/Afghanistan) -[memberOf]-> (subRegion/Southern Asia)\n(subRegion/Southern Asia) -[memberOf]-> (region/Asia)\n(...)\n```\n\nTo add facts directly to the platform, include the act-baseurl and user-id options:\n\n```bash\n$ act-country-regions --act-baseurl http://localhost:8888 --user-id 1\n```\n\n# Configuration\n\nAll workers support options specified as command line arguments, environment variables and in a configuration file.\n\nA utility to show and start with a default ini file is also included:\n\n```bash\nact-worker-config --help\nusage: ACT worker config [-h] {show,user,system}\n\npositional arguments:\n {show,user,system}\n\noptional arguments:\n -h, --help show this help message and exit\n\n show - Print default config\n\n user - Copy default config to /home/fredrikb/.config/actworkers/actworkers.ini\n\n system - Copy default config to /etc/actworkers.ini\n```\n\nYou can see the default options in [act/workers/etc/actworkers.ini](act/workers/etc/actworkers.ini).\n\nThe configuration presedence are (from lowest to highest):\n1. Defaults (shown in --help for each worker)\n2. INI file\n3. Environment variable\n4. Command line argument\n\n## INI-file\nArguments are parsed in two phases. First, it will look for the argument --config argument\nwhich can be used to specify an alternative location for the ini file. If not --config argument\nis given it will look for an ini file in the following locations:\n\n /etc/<CONFIG_FILE_NAME>\n ~/.config/<CONFIG_ID>/<CONFIG_FILE_NAME> (or directory specified by $XDG_CONFIG_HOME)\n\nThe ini file contains a \"[DEFAULT]\" section that will be used for all workers.\nIn addition there are separate sections for each worker which you can use to configure\nworker-specific options, and override default options.\n\n## Environment variables\n\nThe configuration step will also look for environment variables in uppercase and\nwith \"-\" replaced with \"_\". For the example for the option \"cert-file\" it will look for the\nenviornment variable \"$CERT_FILE\".\n\n## Requirements\n\nAll workers requires python version >= 3.5 and the act-api library:\n\n* [act-api](https://github.com/mnemonic-no/act-api-python) (act-api on [pypi](https://pypi.org/project/act-api/))\n\nIn addition some of the libraries might have additional requirements. See requirements.txt for a full list of all requirements.\n\n# Proxy configuration\n\nWorkers will honor the `proxy-string` option on the command line when connecting to external APIs. However, if you need to\nuse the proxy to connect to the ACT platform (--act-baseurl), you will need to add the \"--proxy-platform\" switch:\n\n```bash\necho -n www.mnemonic.no | act-vt --proxy-string <PROXY> --user-id <USER-ID> --act-baseurl <ACT-HOST> --proxy-platform\n```\n\n# Local development\n\nUse pip to install in [local development mode](https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs). act-workers (and act-api) uses namespacing, so it is not compatible with using `setup.py install` or `setup.py develop`.\n\nIn repository, run:\n\n```bash\npip3 install --user -e .\n```\n# search\nA worker to run graph queries is also included. A sample search config is inscluded in `etc/searc_jobs.ini`:\n\n```bash\nact-search-graph etc/search_jobs.ini\n```\n\n# act-feed\n\n`act-feed` can be used to download feed bundles from a remote uri. Feed uri must have a file, manifest.json, that lists all bundle files that can be download by the feed worker:\n\n```json\n{\n \"bundles\": {\n \"000bd93b8ce1fa2acfa448fa916083d76e64b008e336d6f54ad89f12f4232dcf.gz\": 1636355543,\n \"0598e898979bc851d58b440a3ea248fe64d9df4d2ab1665da3be73c6d13df411.gz\": 1636681082,\n \"0c03f87d142345a1bbea0867c145608e70ab2c0ea8282b5e06833f5f2b0f5f04.gz\": 1636366321,\n },\n \"updated\": 1636699081\n}\n```\n\nA local cache will be stored stored to keep track of the last bundle download, so bundles will not be downloaded multiple times.\n\nYou can either let act-feed download files to a local directory:\n\n```bash\nact-feed --feed-uri https://act.mnemonic.no/feed --dump-dir [path]\n```\n\noutput all facts to stdout:\n\n```bash\nact-feed --feed-uri https://act.mnemonic.no/feed \n```\n\nor upload facts directory to the platform\n\n```bash\nact-feed --feed-uri https://act.mnemonic.no/feed --act-baseurl [platform UR] --user-id [USERID]\n```\n\n# act-fact-chain-helper\n\nThe act-fact-chain-helper is a utility that can be used to add facts chains based on known start and end nodes. This can be usefull when adding\ninformation from reports where do not have all the information availiable to you.\n\n## example\n\nWe know from a report that the threat actor APT 1 uses the tool Mimikatz\n\n```bash\n$ act-fact-chain-helper --output-format str --start threatActor/apt1 --end tool/mimikatz --avoid mentions --include content\n(incident/[placeholder[1194f1ba4ea8ded28250a0193327654e5ee305bce03741a3e6f183f03b5c6b35]]) -[attributedTo]-> (threatActor/apt1)\n(event/[placeholder[1194f1ba4ea8ded28250a0193327654e5ee305bce03741a3e6f183f03b5c6b35]]) -[attributedTo]-> (incident/[placeholder[1194f1ba4ea8ded28250a0193327654e5ee305bce03741a3e6f183f03b5c6b35]])\n(content/[placeholder[1194f1ba4ea8ded28250a0193327654e5ee305bce03741a3e6f183f03b5c6b35]]) -[observedIn]-> (event/[placeholder[1194f1ba4ea8ded28250a0193327654e5ee305bce03741a3e6f183f03b5c6b35]])\n(content/[placeholder[1194f1ba4ea8ded28250a0193327654e5ee305bce03741a3e6f183f03b5c6b35]]) -[classifiedAs]-> (tool/mimikatz)\n```\n\nIn this case we do need to hint at the tool that it should include the \"content\" object and avoid \"mention\" facts.\n\nThe source and destination is interchangeable. In the example above, the fact chain created by swapping the source and destination is identical.\n\nby adding the ```--act-baseurl``` option, the data will be stored to the platform.\n\nThe tool works by finding the shortest path though the datamodel. The options --avoid and --include will modify the cost of traversing certain nodes or edges. This tool is considered experimental.l.\n\n# act-ta-helper\n\nYou can use `act-ta-helper` to create facts, including placeholders, for typical scenarios where you have some\ninformation of TA activity, but not all.\n\nThis worker should not be used if any of objects listed as placeholders below are known.\n\n```bash\n --ta THREAT-ACTOR Threat Actor Name (Required)\n --ta-located-in COUNTRY Country where threat actor is located\n --campaign CAMPAIGN Campaign\n --techniques TECHNIQUES List of techniques (commaseparated).\n Support technique IDs, e.g. T1002 and name\n e.g \"Valid Accounts\"\n --tools TOOLS List of tools (commaseparated)\n --sectors SECTORS List of Sectors (commaseparated)\n --target-countries COUNTRIES Target Countries (commaseparated)\n```\n\nYou can run the tool with the options above, and add `--output-format str` to se a suggested list of facts.\nYou can then add `--act-baseurl` and `--user-id` to add the facts to a platform instance.\n\n\nYou can add all options on a single command line, like this example:\n```bash\nact-ta-helper \\\n --output-format str \\\n --ta HAFNIUM \\\n --sectors pharmaceuticals,education,defense,non-profit \\\n --tools PsExec,Procdump,7-Zip,Nishang,PowerCat,WinRar,SIMPLESEESHARP,SPORTSBALL,ChinaChopper,ASPXSPY,Covenant \\\n --target-countries \"Denmark,United States of America\" \\\n --campaign \"Operation Exchange Marauder\" \\\n --ta-located-in China \\\n --techniques T1588,T1003,T1190,T1560,T1583,T1071,T1114,T1567,T1136,T1021\n```\n\nThe following facts/placeholders will be created based on the options (placeholders are marked with [*]):\n\n## `ta-located-in`\n\nUse this if you know the threat actor name and country where TA is located, but organization is unknown.\n\n```\nthreatActor -[attributedTo]-> organization[*] -[locatedIn]-> country\n```\n\n## `campaign`\n\nUse this if you know the threat actor and campaign name, but incident is unknown.\n\n```\nthreatActor -[attributedTo]-> incident[*] -[attributedTo]-> campaign\n```\n\n## `techniques`\n\nUse this if you know the threat actor and technique, but event and incident is unknown.\n\n\n```\ntechnique <-[classifiedAs]- event[*] -[attributedTo]-> incident[*] -[attributedTo]-> threatActor\n\n```\n\n## `tools`\n\nUse if tool and threat actor is known, but incident, content and event is unknown.\n\n```\nthreatActor <-[attributedTo]- incident[*] <-[observedIn]- content[*] -[classifiedAs]-> tool\n```\n\n## `sectors`\n\nUse this if threat actor and target sector is known, but organization and incident is unknown.\n\n```\nthreatActor -[attributedTo]-> incident[*] -[targets]-> organization[*] -[memberOf]-> sector\n```\n\n## `target-countries`\n\nUse this if threat actor and target country is known, but organization and incident is unknown.\n\n```\nthreatActor -[attributedTo]-> incident[*] -[targets]-> organization[*] -[locatedIn]-> country\n```\n",
"bugtrack_url": null,
"license": "ISC",
"summary": "ACT workers",
"version": "2.1.13",
"split_keywords": [
"act",
"mnemonic"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "6482f8321050a6edea2366a8f08d18c073161bb6c5caa4d4075abaa0f1034a21",
"md5": "1aa08f46164f99ec8a797e2a7e56245c",
"sha256": "9348e2df967de6a5efa57402ded0b2624d62ae0298b7f7c5e9b600244332588d"
},
"downloads": -1,
"filename": "act-workers-2.1.13.tar.gz",
"has_sig": false,
"md5_digest": "1aa08f46164f99ec8a797e2a7e56245c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <4",
"size": 82986,
"upload_time": "2023-04-24T07:19:49",
"upload_time_iso_8601": "2023-04-24T07:19:49.723880Z",
"url": "https://files.pythonhosted.org/packages/64/82/f8321050a6edea2366a8f08d18c073161bb6c5caa4d4075abaa0f1034a21/act-workers-2.1.13.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-04-24 07:19:49",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "mnemonic-no",
"github_project": "act-workers",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "act-workers"
}
JSON
