# Assemblyline 4 - Service Base
This repository provides the base service functionality for Assemblyline 4 services.
## Creating a new Assemblyline service
### Service file structure
An Assemblyline service has the following file structure:
```text
assemblyline-service-<service name>
│
├── Dockerfile
├── <service name>.py
└── service_manifest.yml
```
This is overview of what each of these does:
- `Dockerfile` ─ Build file for the service container, see _Dockerfile_ section below for more details
- `<service name>.py` ─ Contains main service code
- `service_manifest.yml` ─ Service definition file, see _Service manifest_ section below for more details
### Service manifest
Every service must have a `service_manifest.yml` file in its root directory. The manifest file presents essential information about the service to the Assemblyline core system, information the system must have before it can run the service.
The diagram below shows all the elements that the manifest file can contain, including a brief description of each.
```yaml
# Name of the service
name: ResultSample
# Version of the service
version: 1
# Description of the service
description: >
ALv4 Result example service
This service provides examples of how to:
- define your service manifest
- use the different section types
- use tags
- use heuristics to score sections
- use the att&ck matrix
- use the updater framework
- define submission parameters
- define service configuration parameters
# Regex defining the types of files the service accepts and rejects
accepts: .*
rejects: empty|metadata/.*
# At which stage the service should run (one of: FILTER, EXTRACT, CORE, SECONDARY, POST)
# NOTE: Stages are executed in the order defined in the list
stage: CORE
# Which category the service is part of (one of: Antivirus, Dynamic Analysis, External, Extraction, Filtering, Networking, Static Analysis)
category: Static Analysis
# Does the service require access to the file to perform its task
# If set to false, the service will only have access to the file metadata (e.g. Hashes, size, type, ...)
file_required: true
# Maximum execution time the service has before it's considered to be timed out
timeout: 60
# Does the service force the caching of results to be disabled
# (only use for service that will always provided different results each run)
disable_cache: false
# is the service enabled by default
enabled: true
# does the service make APIs call to other product not part of the assemblyline infrastructure (e.g. VirusTotal, ...)
is_external: false
# Number of concurrent services allowed to run at the same time
licence_count: 0
# service configuration block (dictionary of config variables)
# NOTE: The key names can be anything and the value can be of any types
config:
str_config: value1
int_config: 1
list_config: [1, 2, 3, 4]
bool_config: false
# submission params block: a list of submission param object that define parameters
# that the user can change about the service for each of its scans
# supported types: bool, int, str, list
submission_params:
- default: ""
name: password
type: str
value: ""
- default: false
name: extra_work
type: bool
value: false
# Service heuristic blocks: List of heuristics object that define the different heuristics used in the service
heuristics:
- description: This the first Heuristic for ResultSample service.
filetype: pdf
heur_id: AL_RESULTSAMPLE_1
name: Masks has PDF
score: 100
attack_id: T1001
- description: This is second Heuristic for ResultSample service.
filetype: exe
heur_id: AL_RESULTSAMPLE_2
name: Drops an exe
score: 1000
- description: This is third Heuristic for ResultSample service.
filetype: exe
heur_id: AL_RESULTSAMPLE_3
name: Extraction information
score: 0
# Docker configuration block which defines:
# - the name of the docker container that will be created
# - cpu and ram allocation by the container
docker_config:
image: cccs/assemblyline-service-resultsample:latest
cpu_cores: 1.0
ram_mb_min: 128
ram_mb: 256
# Update configuration block
update_config:
# list of source object from where to fetch files for update and what will be the name of those files on disk
sources:
- uri: https://file-examples.com/wp-content/uploads/2017/02/zip_2MB.zip
name: sample_2mb_file
- uri: https://file-examples.com/wp-content/uploads/2017/02/zip_5MB.zip
name: sample_5mb_file
# intervale in seconds at which the updater runs
update_interval_seconds: 300
# Should the downloaded files be used to create signatures in the system
generates_signatures: false
```
### Dockerfile
A Dockerfile is required to build the service container that will be executed in the system.
The following items must be set for all services:
- All services must be based on the `cccs/assemblyline-v4-service-base:latest` image
- An environment variable must be set for the service path
- Install any service requirements
- Copy the service code into `/opt/al/al_service/`
```dockerfile
FROM cccs/assemblyline-v4-service-base:latest
# Set the service path
ENV SERVICE_PATH result_sample.ResultSample
# By default, the base service container as the assemblyline user as the running user
# switch to root to perform installation of dependancies
USER root
# See that we all these operations in one line to reduce
# the number of container layers and size of the container
RUN apt-get update && apt-get install -y my_debian_apt_dependency_package && rm -rf /var/lib/apt/lists/*
RUN pip install --no-cache-dir --user my_pip_dependency && rm -rf ~/.cache/pip
# Change to the assemblyline user to make sure your service does not run as root
USER assemblyline
# Copy the service code in the service directory
WORKDIR /opt/al_service
COPY assemblyline_result_sample_service .
```
## Testing an Assemblyline service
To test an Assemblyline service in standalone mode, the [run_service_once.py](https://github.com/CybercentreCanada/assemblyline-v4-service/src/master/dev/run_service_once.py) script can be used to run a single task through the service for testing. That script does not require that you have a working version of Assemblyline installed, all you need are the Assemblyline python libraries.
### Setting up dev environment
**NOTE:** The following environment setup has only been tested on Ubuntu 20.04.
1. Install required packages
```
sudo apt-get install build-essential libffi-dev python3.7 python3.7-dev python3-pip automake autoconf libtool
```
2. Install Assemblyline v4 service package
```
pip install --no-cache-dir --user assemblyline-v4-service
```
3. Add your service development directory path (ie. `/home/ubuntu/assemblyline-v4-service`) to the PYTHONPATH environment variable
### Using the `run_service_once.py` script
#### Steps
1. Ensure the current working directory is the root of the service directory of the service to be run
```shell
cd assemblyline-service-<service name>
```
2. From a terminal, run the `run_service_once` script, where `<service path>` is the path to the service module and `<file path>` is the path of the file to be processed
```shell
python3.11 -m assemblyline_v4_service.dev.run_service_once <service path> <file path>
```
3. The output of the service (`result.json` and extracted/supplementary files) will be located in a directory where the input file is located
#### Example of running the ResultSample service
1. Change working directory to root of the service:
```shell
cd assemblyline_result_sample_service
```
2. From a terminal, run the `run_service_once` script
```shell
python3.11 -m assemblyline_v4_service.dev.run_service_once assemblyline_result_sample_service.result_sample.ResultSample /home/ubuntu/testfile.doc
```
3. The `results.json` and any extracted/supplementary files will be outputted to `/home/ubuntu/testfile_resultsample`
Raw data
{
"_id": null,
"home_page": "https://github.com/CybercentreCanada/assemblyline-v4-service/",
"name": "assemblyline-v4-service",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "assemblyline automated malware analysis gc canada cse-cst cse cst cyber cccs",
"author": "CCCS Assemblyline development team",
"author_email": "assemblyline@cyber.gc.ca",
"download_url": null,
"platform": null,
"description": "# Assemblyline 4 - Service Base\n\nThis repository provides the base service functionality for Assemblyline 4 services.\n\n## Creating a new Assemblyline service\n\n### Service file structure\n\nAn Assemblyline service has the following file structure:\n\n```text\nassemblyline-service-<service name>\n\u2502\n\u251c\u2500\u2500 Dockerfile\n\u251c\u2500\u2500 <service name>.py\n\u2514\u2500\u2500 service_manifest.yml\n```\n\nThis is overview of what each of these does:\n\n- `Dockerfile` \u2500 Build file for the service container, see _Dockerfile_ section below for more details\n- `<service name>.py` \u2500 Contains main service code\n- `service_manifest.yml` \u2500 Service definition file, see _Service manifest_ section below for more details\n\n\n### Service manifest\n\nEvery service must have a `service_manifest.yml` file in its root directory. The manifest file presents essential information about the service to the Assemblyline core system, information the system must have before it can run the service.\n\nThe diagram below shows all the elements that the manifest file can contain, including a brief description of each.\n\n```yaml\n# Name of the service\nname: ResultSample\n# Version of the service\nversion: 1\n# Description of the service\ndescription: >\n ALv4 Result example service\n\n This service provides examples of how to:\n - define your service manifest\n - use the different section types\n - use tags\n - use heuristics to score sections\n - use the att&ck matrix\n - use the updater framework\n - define submission parameters\n - define service configuration parameters\n\n# Regex defining the types of files the service accepts and rejects\naccepts: .*\nrejects: empty|metadata/.*\n\n# At which stage the service should run (one of: FILTER, EXTRACT, CORE, SECONDARY, POST)\n# NOTE: Stages are executed in the order defined in the list\nstage: CORE\n# Which category the service is part of (one of: Antivirus, Dynamic Analysis, External, Extraction, Filtering, Networking, Static Analysis)\ncategory: Static Analysis\n\n# Does the service require access to the file to perform its task\n# If set to false, the service will only have access to the file metadata (e.g. Hashes, size, type, ...)\nfile_required: true\n# Maximum execution time the service has before it's considered to be timed out\ntimeout: 60\n# Does the service force the caching of results to be disabled\n# (only use for service that will always provided different results each run)\ndisable_cache: false\n\n# is the service enabled by default\nenabled: true\n# does the service make APIs call to other product not part of the assemblyline infrastructure (e.g. VirusTotal, ...)\nis_external: false\n# Number of concurrent services allowed to run at the same time\nlicence_count: 0\n\n# service configuration block (dictionary of config variables)\n# NOTE: The key names can be anything and the value can be of any types\nconfig:\n str_config: value1\n int_config: 1\n list_config: [1, 2, 3, 4]\n bool_config: false\n\n# submission params block: a list of submission param object that define parameters\n# that the user can change about the service for each of its scans\n# supported types: bool, int, str, list\nsubmission_params:\n - default: \"\"\n name: password\n type: str\n value: \"\"\n - default: false\n name: extra_work\n type: bool\n value: false\n\n# Service heuristic blocks: List of heuristics object that define the different heuristics used in the service\nheuristics:\n - description: This the first Heuristic for ResultSample service.\n filetype: pdf\n heur_id: AL_RESULTSAMPLE_1\n name: Masks has PDF\n score: 100\n attack_id: T1001\n - description: This is second Heuristic for ResultSample service.\n filetype: exe\n heur_id: AL_RESULTSAMPLE_2\n name: Drops an exe\n score: 1000\n - description: This is third Heuristic for ResultSample service.\n filetype: exe\n heur_id: AL_RESULTSAMPLE_3\n name: Extraction information\n score: 0\n\n# Docker configuration block which defines:\n# - the name of the docker container that will be created\n# - cpu and ram allocation by the container\ndocker_config:\n image: cccs/assemblyline-service-resultsample:latest\n cpu_cores: 1.0\n ram_mb_min: 128\n ram_mb: 256\n\n# Update configuration block\nupdate_config:\n # list of source object from where to fetch files for update and what will be the name of those files on disk\n sources:\n - uri: https://file-examples.com/wp-content/uploads/2017/02/zip_2MB.zip\n name: sample_2mb_file\n - uri: https://file-examples.com/wp-content/uploads/2017/02/zip_5MB.zip\n name: sample_5mb_file\n # intervale in seconds at which the updater runs\n update_interval_seconds: 300\n # Should the downloaded files be used to create signatures in the system\n generates_signatures: false\n```\n\n### Dockerfile\n\nA Dockerfile is required to build the service container that will be executed in the system.\n\nThe following items must be set for all services:\n\n- All services must be based on the `cccs/assemblyline-v4-service-base:latest` image\n- An environment variable must be set for the service path\n- Install any service requirements\n- Copy the service code into `/opt/al/al_service/`\n\n```dockerfile\nFROM cccs/assemblyline-v4-service-base:latest\n\n# Set the service path\nENV SERVICE_PATH result_sample.ResultSample\n\n# By default, the base service container as the assemblyline user as the running user\n# switch to root to perform installation of dependancies\nUSER root\n\n# See that we all these operations in one line to reduce\n# the number of container layers and size of the container\nRUN apt-get update && apt-get install -y my_debian_apt_dependency_package && rm -rf /var/lib/apt/lists/*\nRUN pip install --no-cache-dir --user my_pip_dependency && rm -rf ~/.cache/pip\n\n# Change to the assemblyline user to make sure your service does not run as root\nUSER assemblyline\n\n# Copy the service code in the service directory\nWORKDIR /opt/al_service\nCOPY assemblyline_result_sample_service .\n```\n\n## Testing an Assemblyline service\n\nTo test an Assemblyline service in standalone mode, the [run_service_once.py](https://github.com/CybercentreCanada/assemblyline-v4-service/src/master/dev/run_service_once.py) script can be used to run a single task through the service for testing. That script does not require that you have a working version of Assemblyline installed, all you need are the Assemblyline python libraries.\n\n### Setting up dev environment\n\n**NOTE:** The following environment setup has only been tested on Ubuntu 20.04.\n\n1. Install required packages\n\n ```\n sudo apt-get install build-essential libffi-dev python3.7 python3.7-dev python3-pip automake autoconf libtool\n ```\n\n2. Install Assemblyline v4 service package\n\n ```\n pip install --no-cache-dir --user assemblyline-v4-service\n ```\n\n3. Add your service development directory path (ie. `/home/ubuntu/assemblyline-v4-service`) to the PYTHONPATH environment variable\n\n### Using the `run_service_once.py` script\n\n#### Steps\n\n1. Ensure the current working directory is the root of the service directory of the service to be run\n\n ```shell\n cd assemblyline-service-<service name>\n ```\n\n2. From a terminal, run the `run_service_once` script, where `<service path>` is the path to the service module and `<file path>` is the path of the file to be processed\n\n ```shell\n python3.11 -m assemblyline_v4_service.dev.run_service_once <service path> <file path>\n ```\n\n\n3. The output of the service (`result.json` and extracted/supplementary files) will be located in a directory where the input file is located\n\n#### Example of running the ResultSample service\n\n1. Change working directory to root of the service:\n\n ```shell\n cd assemblyline_result_sample_service\n ```\n\n2. From a terminal, run the `run_service_once` script\n\n ```shell\n python3.11 -m assemblyline_v4_service.dev.run_service_once assemblyline_result_sample_service.result_sample.ResultSample /home/ubuntu/testfile.doc\n ```\n\n3. The `results.json` and any extracted/supplementary files will be outputted to `/home/ubuntu/testfile_resultsample`\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Assemblyline 4 - Service base",
"version": "4.5.0.57",
"project_urls": {
"Homepage": "https://github.com/CybercentreCanada/assemblyline-v4-service/"
},
"split_keywords": [
"assemblyline",
"automated",
"malware",
"analysis",
"gc",
"canada",
"cse-cst",
"cse",
"cst",
"cyber",
"cccs"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "90ac279ee60582d135be35818b0052b2d477869c0b169099627c90fc4f195d5b",
"md5": "5b726181836c0a25f71e043b671abf3b",
"sha256": "5c012b57aa2d73175e64bbf574f751c282c9d7cb740c828dfea3fe60a05f4f4f"
},
"downloads": -1,
"filename": "assemblyline_v4_service-4.5.0.57-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5b726181836c0a25f71e043b671abf3b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 88432,
"upload_time": "2024-10-24T17:37:14",
"upload_time_iso_8601": "2024-10-24T17:37:14.235538Z",
"url": "https://files.pythonhosted.org/packages/90/ac/279ee60582d135be35818b0052b2d477869c0b169099627c90fc4f195d5b/assemblyline_v4_service-4.5.0.57-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-24 17:37:14",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "CybercentreCanada",
"github_project": "assemblyline-v4-service",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "assemblyline-v4-service"
}
JSON
