pepperize.cdk-autoscaling-gitlab-runner


Namepepperize.cdk-autoscaling-gitlab-runner JSON
Version 0.2.111 PyPI version JSON
download
home_pagehttps://github.com/pepperize/cdk-autoscaling-gitlab-runner.git
SummaryAWS CDK GitLab Runner autoscaling on EC2 instances using docker+machine executor.
upload_time2022-06-30 02:44:56
maintainer
docs_urlNone
authorPatrick Florek<patrick.florek@gmail.com>
requires_python~=3.7
licenseMIT
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            [![GitHub](https://img.shields.io/github/license/pepperize/cdk-autoscaling-gitlab-runner?style=flat-square)](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/LICENSE)
[![npm (scoped)](https://img.shields.io/npm/v/@pepperize/cdk-autoscaling-gitlab-runner?style=flat-square)](https://www.npmjs.com/package/@pepperize/cdk-autoscaling-gitlab-runner)
[![PyPI](https://img.shields.io/pypi/v/pepperize.cdk-autoscaling-gitlab-runner?style=flat-square)](https://pypi.org/project/pepperize.cdk-autoscaling-gitlab-runner/)
[![Nuget](https://img.shields.io/nuget/v/Pepperize.CDK.AutoscalingGitlabRunner?style=flat-square)](https://www.nuget.org/packages/Pepperize.CDK.AutoscalingGitlabRunner/)
[![Sonatype Nexus (Releases)](https://img.shields.io/nexus/r/com.pepperize/cdk-autoscaling-gitlab-runner?server=https%3A%2F%2Fs01.oss.sonatype.org%2F&style=flat-square)](https://s01.oss.sonatype.org/content/repositories/releases/com/pepperize/cdk-autoscaling-gitlab-runner/)
[![GitHub Workflow Status (branch)](https://img.shields.io/github/workflow/status/pepperize/cdk-autoscaling-gitlab-runner/release/main?label=release&style=flat-square)](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/actions/workflows/release.yml)
[![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/pepperize/cdk-autoscaling-gitlab-runner?sort=semver&style=flat-square)](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/releases)

# AWS CDK GitLab Runner autoscaling on EC2

This project provides a CDK construct to [execute jobs on auto-scaled EC2 instances](https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/index.html) using the [Docker Machine](https://docs.gitlab.com/runner/executors/docker_machine.html) executor.

> Running out of [Runner minutes](https://about.gitlab.com/pricing/),
> using [Docker-in-Docker (dind)](https://docs.gitlab.com/ee/ci/docker/using_docker_build.html),
> speed up jobs with [shared S3 Cache](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching),
> cross compiling/building environment [multiarch](https://hub.docker.com/r/multiarch/qemu-user-static/),
> cost effective [autoscaling on EC2](https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/#the-runnersmachine-section),
> deploy directly from AWS accounts (without [AWS Access Key](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys)),
> running on [Spot instances](https://aws.amazon.com/ec2/spot/),
> having a bigger [build log size](https://docs.gitlab.com/runner/configuration/advanced-configuration.html)

## Install

### TypeScript

```shell
npm install @pepperize/cdk-autoscaling-gitlab-runner
```

or

```shell
yarn add @pepperize/cdk-autoscaling-gitlab-runner
```

### Python

```shell
pip install pepperize.cdk-autoscaling-gitlab-runner
```

### C# / .Net

```
dotnet add package Pepperize.CDK.AutoscalingGitlabRunner
```

### Java

```xml
<dependency>
  <groupId>com.pepperize</groupId>
  <artifactId>cdk-autoscaling-gitlab-runner</artifactId>
  <version>${cdkAutoscalingGitlabRunner.version}</version>
</dependency>
```

## Quickstart

1. **Create a new AWS CDK App** in TypeScript with [projen](https://github.com/projen/projen)

   ```shell
   mkdir gitlab-runner
   cd gitlab-runner
   git init
   npx projen new awscdk-app-ts
   ```
2. **Configure your project in `.projenrc.js`**

   * Add `deps: ["@pepperize/cdk-autoscaling-gitlab-runner"],`
3. **Update project files and install dependencies**

   ```shell
   npx projen
   ```
4. **Register a new runner**

   [Registering runners](https://docs.gitlab.com/runner/register/):

   * For a [shared runner](https://docs.gitlab.com/ee/ci/runners/#shared-runners), go to the GitLab Admin Area and click **Overview > Runners**
   * For a [group runner](https://docs.gitlab.com/ee/ci/runners/index.html#group-runners), go to **Settings > CI/CD** and expand the **Runners** section
   * For a [project runner](https://docs.gitlab.com/ee/ci/runners/index.html#specific-runners), go to **Settings > CI/CD** and expand the **Runners** section

   *Optionally enable: **Run untagged jobs** [x]
   Indicates whether this runner can pick jobs without tags*

   See also *[Registration token vs. Authentication token](https://docs.gitlab.com/ee/api/runners.html#registration-and-authentication-tokens)*
5. **Retrieve a new runner authentication token**

   [Register a new runner](https://docs.gitlab.com/ee/api/runners.html#register-a-new-runner)

   ```shell
   curl --request POST "https://gitlab.com/api/v4/runners" --form "token=<your register token>" --form "description=gitlab-runner" --form "tag_list=pepperize,docker,production"
   ```
6. **Store runner authentication token in SSM ParameterStore**

   [Create a String parameter](https://docs.aws.amazon.com/systems-manager/latest/userguide/param-create-cli.html#param-create-cli-string)

   ```shell
   aws ssm put-parameter --name "/gitlab-runner/token" --value "<your runner authentication token>" --type "String"
   ```
7. **Add to your `main.ts`**

   ```typescript
   import { Vpc } from "@aws-cdk/aws-ec2";
   import { App, Stack } from "@aws-cdk/core";
   import { GitlabRunnerAutoscaling } from "@pepperize/cdk-autoscaling-gitlab-runner";

   const app = new App();
   const stack = new Stack(app, "GitLabRunnerStack");
   const vpc = Vpc.fromLookup(app, "ExistingVpc", {
     vpcId: "<your vpc id>",
   });
   const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
     parameterName: "/gitlab-runner/token",
   });
   new GitlabRunnerAutoscaling(stack, "GitlabRunner", {
     network: {
       vpc: vpc,
     },
     runners: [
       {
         token: token,
         configuration: {
           // optionally configure your runner
         },
       },
     ],
   });
   ```
8. **Create service linked role**

   *(If requesting spot instances, default: true)*

   ```sh
   aws iam create-service-linked-role --aws-service-name spot.amazonaws.com
   ```
9. **Configure the AWS CLI**

   * [AWSume](https://awsu.me/)
   * [Configuring the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html)
   * [AWS Single Sign-On](https://docs.aws.amazon.com/singlesignon/latest/userguide/what-is.html)
10. **Deploy the GitLab Runner**

    ```shell
    npm run deploy
    ```

## Example

### Custom cache bucket

By default, an AWS S3 Bucket is created as GitLab Runner's distributed cache.
It's encrypted and public access is blocked.
A custom S3 Bucket can be configured:

```typescript
const cache = new Bucket(this, "Cache", {
  // Your custom bucket
});
const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
  parameterName: "/gitlab-runner/token",
});

new GitlabRunnerAutoscaling(this, "Runner", {
  runners: [
    {
      token: token,
    },
  ],
  cache: { bucket: cache },
});
```

See [example](https://github.com/pepperize/cdk-autoscaling-gitlab-runner-example/blob/main/src/cache.ts),
[GitlabRunnerAutoscalingCacheProps](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/API.md#gitlabrunnerautoscalingcacheprops-)

### Custom EC2 key pair

By default, the [amazonec2](https://gitlab.com/gitlab-org/ci-cd/docker-machine/-/blob/main/drivers/amazonec2/amazonec2.go) driver will create an EC2 key pair for each runner. To use custom ssh credentials provide a SecretsManager Secret with the private and public key file:

```shell
aws secretsmanager create-secret --name CustomEC2KeyPair --secret-string "{\"theKeyPairName\":\"<the private key>\",\"theKeyPairName.pub\":\"<the public key>\"}"
```

```typescript
const keyPair = Secret.fromSecretNameV2(stack, "Secret", "CustomEC2KeyPair")

new GitlabRunnerAutoscaling(this, "Runner", {
  runners: [
    {
      keyPair: keyPair,
       configuration: {
        machine: {
          machineOptions: {
            keypairName: "theKeyPairName",
          },
        },
       },
    },
  ],
  cache: { bucket: cache },
});
```

### Configure Docker Machine

By default, docker machine is configured to run privileged with `CAP_SYS_ADMIN` to support [Docker-in-Docker using the OverlayFS driver](https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#use-the-overlayfs-driver)
and cross compiling/building with [multiarch](https://hub.docker.com/r/multiarch/qemu-user-static/).

See [runners.docker section](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnersdocker-section)
in [Advanced configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html)

```typescript
import { GitlabRunnerAutoscaling } from "@pepperize/cdk-autoscaling-gitlab-runner";
import { StringParameter } from "aws-cdk-lib/aws-ssm";

const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
  parameterName: "/gitlab-runner/token",
});

new GitlabRunnerAutoscaling(this, "Runner", {
  runners: [
    {
      token: token,
      configuration: {
        environment: [], // Reset the OverlayFS driver for every project
        docker: {
          capAdd: [], // Remove the CAP_SYS_ADMIN
          privileged: false, // Run unprivileged
        },
        machine: {
          idleCount: 2, // Number of idle machine
          idleTime: 3000, // Waiting time in idle state
          maxBuilds: 1, // Max builds before instance is removed
        },
      },
    },
  ],
});
```

See [example](https://github.com/pepperize/cdk-autoscaling-gitlab-runner-example/blob/main/src/docker-machine.ts),
[DockerConfiguration](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/API.md#dockerconfiguration-)

### Bigger instance type

By default, t3.nano is used for the manager/coordinator and t3.micro instances will be spawned.
For bigger projects, for example with [webpack](https://webpack.js.org/), this won't be enough memory.

```typescript
const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
  parameterName: "/gitlab-runner/token",
});

new GitlabRunnerAutoscaling(this, "Runner", {
  manager: {
    instanceType: InstanceType.of(InstanceClass.T3, InstanceSize.SMALL),
  },
  runners: [
    {
      instanceType: InstanceType.of(InstanceClass.T3, InstanceSize.LARGE),
      token: token,
      configuration: {
        // optionally configure your runner
      },
    },
  ],
});
```

> You may have to disable or configure [Spot instances](#spot-instances)

See [example](https://github.com/pepperize/cdk-autoscaling-gitlab-runner-example/blob/main/src/instance-type.ts),
[GitlabRunnerAutoscalingManagerProps](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/API.md#gitlabrunnerautoscalingmanagerprops-),
[GitlabRunnerAutoscalingJobRunnerProps](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/API.md#gitlabrunnerautoscalingjobrunnerprops-)

### Different machine image

By default, the latest [Amazon 2 Linux](https://aws.amazon.com/amazon-linux-2/) will be used for the manager/coordinator.
The manager/coordinator instance's cloud init scripts requires [yum](https://access.redhat.com/solutions/9934) is installed, any RHEL flavor should work.
The requested runner instances by default using Ubuntu 20.04, any OS implemented by the [Docker Machine provisioner](https://gitlab.com/gitlab-org/ci-cd/docker-machine/-/tree/main/libmachine/provision) should work.

```typescript
const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
  parameterName: "/gitlab-runner/token",
});

new GitlabRunnerAutoscaling(this, "Runner", {
  manager: {
    machineImage: MachineImage.genericLinux(managerAmiMap),
  },
  runners: [
    {
      machineImage: MachineImage.genericLinux(runnerAmiMap),
      token: token,
      configuration: {
        // optionally configure your runner
      },
    },
  ],
});
```

See [example](https://github.com/pepperize/cdk-autoscaling-gitlab-runner-example/blob/main/src/machine-image.ts),
[GitlabRunnerAutoscalingManagerProps](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/API.md#gitlabrunnerautoscalingmanagerprops-),
[GitlabRunnerAutoscalingJobRunnerProps](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/API.md#gitlabrunnerautoscalingjobrunnerprops-)

### Multiple runners configuration

Each runner defines one `[[runners]]` section in the [configuration file](https://docs.gitlab.com/runner/configuration/).
Use [Specific runners](https://docs.gitlab.com/ee/ci/runners/runners_scope.html#specific-runners) when you want to use runners for specific projects.

```typescript
const privilegedRole = new Role(this, "PrivilegedRunnersRole", {
  // role 1
});

const restrictedRole = new Role(this, "RestrictedRunnersRole", {
  // role 2
});

const token1 = StringParameter.fromStringParameterAttributes(stack, "Token", {
  parameterName: "/gitlab-runner/token1",
});

const token2 = StringParameter.fromStringParameterAttributes(stack, "Token", {
  parameterName: "/gitlab-runner/token2",
});

new GitlabRunnerAutoscaling(this, "Runner", {
  runners: [
    {
      token: token1,
      configuration: {
        name: "privileged-runner",
      },
      role: privilegedRole,
    },
    {
      token: token2,
      configuration: {
        name: "restricted-runner",
        docker: {
          privileged: false, // Run unprivileged
        },
      },
      role: restrictedRole,
    },
  ],
});
```

See [example](https://github.com/pepperize/cdk-autoscaling-gitlab-runner-example/blob/main/src/machine-image.ts),
[GitlabRunnerAutoscalingProps](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/API.md#@pepperize/cdk-autoscaling-gitlab-runner.GitlabRunnerAutoscalingProps)

### Spot instances

By default, EC2 Spot Instances are requested.

```typescript
const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
  parameterName: "/gitlab-runner/token",
});

new GitlabRunnerAutoscaling(this, "Runner", {
  runners: [
    {
      token: token,
      configuration: {
        machine: {
          machineOptions: {
            requestSpotInstance: false,
            spotPrice: 0.5,
          },
        },
      },
    },
  ],
});
```

See [example](https://github.com/pepperize/cdk-autoscaling-gitlab-runner-example/blob/main/src/on-demand-instances.ts),
[EC2 spot price](https://aws.amazon.com/de/ec2/spot/pricing/),
[MachineConfiguration](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/API.md#machineconfiguration-),
[MachineOptions](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/API.md#machineoptions-),
[Advanced configuration - runners.machine.autoscaling](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnersmachineautoscaling-sections)

### Cross-Compile with Multiarch

To build binaries of different architectures can also use [Multiarch](https://wiki.debian.org/Multiarch)

```typescript
const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
  parameterName: "/gitlab-runner/token",
});

new GitlabRunnerAutoscaling(this, "Runner", {
  runners: [
    {
      token: token,
      configuration: {
        docker: {
          privileged: true,
        },
      },
    },
  ],
});
```

Configure your [.gitlab-ci.yml](https://docs.gitlab.com/ee/ci/yaml/) file

```yaml
build:
  image: multiarch/debian-debootstrap:armhf-buster
  services:
    - docker:stable-dind
    - name: multiarch/qemu-user-static:register
      command:
        - "--reset"
  script:
    - make build
```

See [multiarch/qemu-user-static](https://hub.docker.com/r/multiarch/qemu-user-static)

### Running on AWS Graviton

To run your jobs on [AWS Graviton](https://aws.amazon.com/ec2/graviton/) you have to provide an AMI for arm64 architecture.

```typescript
const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
  parameterName: "/gitlab-runner/token",
});

new GitlabRunnerAutoscaling(this, "Runner", {
  runners: [
    {
      token: token,
      configuration: {
        instanceType: InstanceType.of(InstanceClass.M6G, InstanceSize.LARGE),
        machineImage: MachineImage.genericLinux({
          [this.region]: new LookupMachineImage({
            name: "ubuntu/images/hvm-ssd/ubuntu-focal-20.04-*-server-*",
            owners: ["099720109477"],
            filters: {
              architecture: [InstanceArchitecture.ARM_64],
              "image-type": ["machine"],
              state: ["available"],
              "root-device-type": ["ebs"],
              "virtualization-type": ["hvm"],
            },
          }).getImage(this).imageId,
        }),
      },
    },
  ],
});
```

See [Ubuntu Amazon EC2 AMI Locator](https://cloud-images.ubuntu.com/locator/ec2/)

### Custom runner's role

To deploy from within your GitLab Runner Instances, you may pass a Role with the IAM Policies attached.

```typescript
const role = new Role(this, "RunnersRole", {
  assumedBy: new ServicePrincipal("ec2.amazonaws.com", {}),
  inlinePolicies: {},
});
const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
  parameterName: "/gitlab-runner/token",
});

new GitlabRunnerAutoscaling(this, "Runner", {
  runners: [
    {
      role: role,
      token: token,
      configuration: {
        // optionally configure your runner
      },
    },
  ],
});
```

See [example](https://github.com/pepperize/cdk-autoscaling-gitlab-runner-example/blob/main/src/runner-role.ts),
[GitlabRunnerAutoscalingProps](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/API.md#@pepperize/cdk-autoscaling-gitlab-runner.GitlabRunnerAutoscalingProps)

### Vpc

If no existing Vpc is passed, a cheap [VPC](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-ec2.Vpc.html) with a NatInstance (t3.nano) and a single AZ will be created.

```typescript
const natInstanceProvider = aws_ec2.NatProvider.instance({
  instanceType: aws_ec2.InstanceType.of(InstanceClass.T3, InstanceSize.NANO), // using a cheaper gateway (not scalable)
});
const vpc = new Vpc(this, "Vpc", {
  // Your custom vpc, i.e.:
  natGatewayProvider: natInstanceProvider,
  maxAzs: 1,
});

const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
  parameterName: "/gitlab-runner/token",
});

new GitlabRunnerAutoscaling(this, "Runner", {
  runners: [
    {
      token: token,
      configuration: {
        // optionally configure your runner
      },
    },
  ],
  network: { vpc: vpc },
});
```

See [example](https://github.com/pepperize/cdk-autoscaling-gitlab-runner-example/blob/main/src/vpc.ts),
[GitlabRunnerAutoscalingProps](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/API.md#@pepperize/cdk-autoscaling-gitlab-runner.GitlabRunnerAutoscalingProps)

### Zero config

Deploys the [Autoscaling GitLab Runner on AWS EC2](https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/) with the default settings mentioned above.

Happy with the presets?

```typescript
const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
  parameterName: "/gitlab-runner/token",
});

new GitlabRunnerAutoscaling(this, "Runner", {
  runners: [
    {
      token: token,
      configuration: {
        // optionally configure your runner
      },
    },
  ],
});
```

See [example](https://github.com/pepperize/cdk-autoscaling-gitlab-runner-example/blob/main/src/zero-config.ts),
[GitlabRunnerAutoscalingProps](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/API.md#@pepperize/cdk-autoscaling-gitlab-runner.GitlabRunnerAutoscalingProps)

## Projen

This project uses [projen](https://github.com/projen/projen) to maintain project configuration through code. Thus, the synthesized files with projen should never be manually edited (in fact, projen enforces that).

To modify the project setup, you should interact with rich strongly-typed
class [AwsCdkTypeScriptApp](https://github.com/projen/projen/blob/master/API.md#projen-awscdktypescriptapp) and
execute `npx projen` to update project configuration files.

> In simple words, developers can only modify `.projenrc.js` file for configuration/maintenance and files under `/src` directory for development.

See also [Create and Publish CDK Constructs Using projen and jsii](https://github.com/seeebiii/projen-test).



            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/pepperize/cdk-autoscaling-gitlab-runner.git",
    "name": "pepperize.cdk-autoscaling-gitlab-runner",
    "maintainer": "",
    "docs_url": null,
    "requires_python": "~=3.7",
    "maintainer_email": "",
    "keywords": "",
    "author": "Patrick Florek<patrick.florek@gmail.com>",
    "author_email": "",
    "download_url": "https://files.pythonhosted.org/packages/ed/7d/f345c045a59dc9b135dec53eb41500b414187e22b0f8d40a108d71b45d16/pepperize.cdk-autoscaling-gitlab-runner-0.2.111.tar.gz",
    "platform": null,
    "description": "[![GitHub](https://img.shields.io/github/license/pepperize/cdk-autoscaling-gitlab-runner?style=flat-square)](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/LICENSE)\n[![npm (scoped)](https://img.shields.io/npm/v/@pepperize/cdk-autoscaling-gitlab-runner?style=flat-square)](https://www.npmjs.com/package/@pepperize/cdk-autoscaling-gitlab-runner)\n[![PyPI](https://img.shields.io/pypi/v/pepperize.cdk-autoscaling-gitlab-runner?style=flat-square)](https://pypi.org/project/pepperize.cdk-autoscaling-gitlab-runner/)\n[![Nuget](https://img.shields.io/nuget/v/Pepperize.CDK.AutoscalingGitlabRunner?style=flat-square)](https://www.nuget.org/packages/Pepperize.CDK.AutoscalingGitlabRunner/)\n[![Sonatype Nexus (Releases)](https://img.shields.io/nexus/r/com.pepperize/cdk-autoscaling-gitlab-runner?server=https%3A%2F%2Fs01.oss.sonatype.org%2F&style=flat-square)](https://s01.oss.sonatype.org/content/repositories/releases/com/pepperize/cdk-autoscaling-gitlab-runner/)\n[![GitHub Workflow Status (branch)](https://img.shields.io/github/workflow/status/pepperize/cdk-autoscaling-gitlab-runner/release/main?label=release&style=flat-square)](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/actions/workflows/release.yml)\n[![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/pepperize/cdk-autoscaling-gitlab-runner?sort=semver&style=flat-square)](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/releases)\n\n# AWS CDK GitLab Runner autoscaling on EC2\n\nThis project provides a CDK construct to [execute jobs on auto-scaled EC2 instances](https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/index.html) using the [Docker Machine](https://docs.gitlab.com/runner/executors/docker_machine.html) executor.\n\n> Running out of [Runner minutes](https://about.gitlab.com/pricing/),\n> using [Docker-in-Docker (dind)](https://docs.gitlab.com/ee/ci/docker/using_docker_build.html),\n> speed up jobs with [shared S3 Cache](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching),\n> cross compiling/building environment [multiarch](https://hub.docker.com/r/multiarch/qemu-user-static/),\n> cost effective [autoscaling on EC2](https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/#the-runnersmachine-section),\n> deploy directly from AWS accounts (without [AWS Access Key](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys)),\n> running on [Spot instances](https://aws.amazon.com/ec2/spot/),\n> having a bigger [build log size](https://docs.gitlab.com/runner/configuration/advanced-configuration.html)\n\n## Install\n\n### TypeScript\n\n```shell\nnpm install @pepperize/cdk-autoscaling-gitlab-runner\n```\n\nor\n\n```shell\nyarn add @pepperize/cdk-autoscaling-gitlab-runner\n```\n\n### Python\n\n```shell\npip install pepperize.cdk-autoscaling-gitlab-runner\n```\n\n### C# / .Net\n\n```\ndotnet add package Pepperize.CDK.AutoscalingGitlabRunner\n```\n\n### Java\n\n```xml\n<dependency>\n  <groupId>com.pepperize</groupId>\n  <artifactId>cdk-autoscaling-gitlab-runner</artifactId>\n  <version>${cdkAutoscalingGitlabRunner.version}</version>\n</dependency>\n```\n\n## Quickstart\n\n1. **Create a new AWS CDK App** in TypeScript with [projen](https://github.com/projen/projen)\n\n   ```shell\n   mkdir gitlab-runner\n   cd gitlab-runner\n   git init\n   npx projen new awscdk-app-ts\n   ```\n2. **Configure your project in `.projenrc.js`**\n\n   * Add `deps: [\"@pepperize/cdk-autoscaling-gitlab-runner\"],`\n3. **Update project files and install dependencies**\n\n   ```shell\n   npx projen\n   ```\n4. **Register a new runner**\n\n   [Registering runners](https://docs.gitlab.com/runner/register/):\n\n   * For a [shared runner](https://docs.gitlab.com/ee/ci/runners/#shared-runners), go to the GitLab Admin Area and click **Overview > Runners**\n   * For a [group runner](https://docs.gitlab.com/ee/ci/runners/index.html#group-runners), go to **Settings > CI/CD** and expand the **Runners** section\n   * For a [project runner](https://docs.gitlab.com/ee/ci/runners/index.html#specific-runners), go to **Settings > CI/CD** and expand the **Runners** section\n\n   *Optionally enable: **Run untagged jobs** [x]\n   Indicates whether this runner can pick jobs without tags*\n\n   See also *[Registration token vs. Authentication token](https://docs.gitlab.com/ee/api/runners.html#registration-and-authentication-tokens)*\n5. **Retrieve a new runner authentication token**\n\n   [Register a new runner](https://docs.gitlab.com/ee/api/runners.html#register-a-new-runner)\n\n   ```shell\n   curl --request POST \"https://gitlab.com/api/v4/runners\" --form \"token=<your register token>\" --form \"description=gitlab-runner\" --form \"tag_list=pepperize,docker,production\"\n   ```\n6. **Store runner authentication token in SSM ParameterStore**\n\n   [Create a String parameter](https://docs.aws.amazon.com/systems-manager/latest/userguide/param-create-cli.html#param-create-cli-string)\n\n   ```shell\n   aws ssm put-parameter --name \"/gitlab-runner/token\" --value \"<your runner authentication token>\" --type \"String\"\n   ```\n7. **Add to your `main.ts`**\n\n   ```typescript\n   import { Vpc } from \"@aws-cdk/aws-ec2\";\n   import { App, Stack } from \"@aws-cdk/core\";\n   import { GitlabRunnerAutoscaling } from \"@pepperize/cdk-autoscaling-gitlab-runner\";\n\n   const app = new App();\n   const stack = new Stack(app, \"GitLabRunnerStack\");\n   const vpc = Vpc.fromLookup(app, \"ExistingVpc\", {\n     vpcId: \"<your vpc id>\",\n   });\n   const token = StringParameter.fromStringParameterAttributes(stack, \"Token\", {\n     parameterName: \"/gitlab-runner/token\",\n   });\n   new GitlabRunnerAutoscaling(stack, \"GitlabRunner\", {\n     network: {\n       vpc: vpc,\n     },\n     runners: [\n       {\n         token: token,\n         configuration: {\n           // optionally configure your runner\n         },\n       },\n     ],\n   });\n   ```\n8. **Create service linked role**\n\n   *(If requesting spot instances, default: true)*\n\n   ```sh\n   aws iam create-service-linked-role --aws-service-name spot.amazonaws.com\n   ```\n9. **Configure the AWS CLI**\n\n   * [AWSume](https://awsu.me/)\n   * [Configuring the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html)\n   * [AWS Single Sign-On](https://docs.aws.amazon.com/singlesignon/latest/userguide/what-is.html)\n10. **Deploy the GitLab Runner**\n\n    ```shell\n    npm run deploy\n    ```\n\n## Example\n\n### Custom cache bucket\n\nBy default, an AWS S3 Bucket is created as GitLab Runner's distributed cache.\nIt's encrypted and public access is blocked.\nA custom S3 Bucket can be configured:\n\n```typescript\nconst cache = new Bucket(this, \"Cache\", {\n  // Your custom bucket\n});\nconst token = StringParameter.fromStringParameterAttributes(stack, \"Token\", {\n  parameterName: \"/gitlab-runner/token\",\n});\n\nnew GitlabRunnerAutoscaling(this, \"Runner\", {\n  runners: [\n    {\n      token: token,\n    },\n  ],\n  cache: { bucket: cache },\n});\n```\n\nSee [example](https://github.com/pepperize/cdk-autoscaling-gitlab-runner-example/blob/main/src/cache.ts),\n[GitlabRunnerAutoscalingCacheProps](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/API.md#gitlabrunnerautoscalingcacheprops-)\n\n### Custom EC2 key pair\n\nBy default, the [amazonec2](https://gitlab.com/gitlab-org/ci-cd/docker-machine/-/blob/main/drivers/amazonec2/amazonec2.go) driver will create an EC2 key pair for each runner. To use custom ssh credentials provide a SecretsManager Secret with the private and public key file:\n\n```shell\naws secretsmanager create-secret --name CustomEC2KeyPair --secret-string \"{\\\"theKeyPairName\\\":\\\"<the private key>\\\",\\\"theKeyPairName.pub\\\":\\\"<the public key>\\\"}\"\n```\n\n```typescript\nconst keyPair = Secret.fromSecretNameV2(stack, \"Secret\", \"CustomEC2KeyPair\")\n\nnew GitlabRunnerAutoscaling(this, \"Runner\", {\n  runners: [\n    {\n      keyPair: keyPair,\n       configuration: {\n        machine: {\n          machineOptions: {\n            keypairName: \"theKeyPairName\",\n          },\n        },\n       },\n    },\n  ],\n  cache: { bucket: cache },\n});\n```\n\n### Configure Docker Machine\n\nBy default, docker machine is configured to run privileged with `CAP_SYS_ADMIN` to support [Docker-in-Docker using the OverlayFS driver](https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#use-the-overlayfs-driver)\nand cross compiling/building with [multiarch](https://hub.docker.com/r/multiarch/qemu-user-static/).\n\nSee [runners.docker section](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnersdocker-section)\nin [Advanced configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html)\n\n```typescript\nimport { GitlabRunnerAutoscaling } from \"@pepperize/cdk-autoscaling-gitlab-runner\";\nimport { StringParameter } from \"aws-cdk-lib/aws-ssm\";\n\nconst token = StringParameter.fromStringParameterAttributes(stack, \"Token\", {\n  parameterName: \"/gitlab-runner/token\",\n});\n\nnew GitlabRunnerAutoscaling(this, \"Runner\", {\n  runners: [\n    {\n      token: token,\n      configuration: {\n        environment: [], // Reset the OverlayFS driver for every project\n        docker: {\n          capAdd: [], // Remove the CAP_SYS_ADMIN\n          privileged: false, // Run unprivileged\n        },\n        machine: {\n          idleCount: 2, // Number of idle machine\n          idleTime: 3000, // Waiting time in idle state\n          maxBuilds: 1, // Max builds before instance is removed\n        },\n      },\n    },\n  ],\n});\n```\n\nSee [example](https://github.com/pepperize/cdk-autoscaling-gitlab-runner-example/blob/main/src/docker-machine.ts),\n[DockerConfiguration](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/API.md#dockerconfiguration-)\n\n### Bigger instance type\n\nBy default, t3.nano is used for the manager/coordinator and t3.micro instances will be spawned.\nFor bigger projects, for example with [webpack](https://webpack.js.org/), this won't be enough memory.\n\n```typescript\nconst token = StringParameter.fromStringParameterAttributes(stack, \"Token\", {\n  parameterName: \"/gitlab-runner/token\",\n});\n\nnew GitlabRunnerAutoscaling(this, \"Runner\", {\n  manager: {\n    instanceType: InstanceType.of(InstanceClass.T3, InstanceSize.SMALL),\n  },\n  runners: [\n    {\n      instanceType: InstanceType.of(InstanceClass.T3, InstanceSize.LARGE),\n      token: token,\n      configuration: {\n        // optionally configure your runner\n      },\n    },\n  ],\n});\n```\n\n> You may have to disable or configure [Spot instances](#spot-instances)\n\nSee [example](https://github.com/pepperize/cdk-autoscaling-gitlab-runner-example/blob/main/src/instance-type.ts),\n[GitlabRunnerAutoscalingManagerProps](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/API.md#gitlabrunnerautoscalingmanagerprops-),\n[GitlabRunnerAutoscalingJobRunnerProps](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/API.md#gitlabrunnerautoscalingjobrunnerprops-)\n\n### Different machine image\n\nBy default, the latest [Amazon 2 Linux](https://aws.amazon.com/amazon-linux-2/) will be used for the manager/coordinator.\nThe manager/coordinator instance's cloud init scripts requires [yum](https://access.redhat.com/solutions/9934) is installed, any RHEL flavor should work.\nThe requested runner instances by default using Ubuntu 20.04, any OS implemented by the [Docker Machine provisioner](https://gitlab.com/gitlab-org/ci-cd/docker-machine/-/tree/main/libmachine/provision) should work.\n\n```typescript\nconst token = StringParameter.fromStringParameterAttributes(stack, \"Token\", {\n  parameterName: \"/gitlab-runner/token\",\n});\n\nnew GitlabRunnerAutoscaling(this, \"Runner\", {\n  manager: {\n    machineImage: MachineImage.genericLinux(managerAmiMap),\n  },\n  runners: [\n    {\n      machineImage: MachineImage.genericLinux(runnerAmiMap),\n      token: token,\n      configuration: {\n        // optionally configure your runner\n      },\n    },\n  ],\n});\n```\n\nSee [example](https://github.com/pepperize/cdk-autoscaling-gitlab-runner-example/blob/main/src/machine-image.ts),\n[GitlabRunnerAutoscalingManagerProps](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/API.md#gitlabrunnerautoscalingmanagerprops-),\n[GitlabRunnerAutoscalingJobRunnerProps](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/API.md#gitlabrunnerautoscalingjobrunnerprops-)\n\n### Multiple runners configuration\n\nEach runner defines one `[[runners]]` section in the [configuration file](https://docs.gitlab.com/runner/configuration/).\nUse [Specific runners](https://docs.gitlab.com/ee/ci/runners/runners_scope.html#specific-runners) when you want to use runners for specific projects.\n\n```typescript\nconst privilegedRole = new Role(this, \"PrivilegedRunnersRole\", {\n  // role 1\n});\n\nconst restrictedRole = new Role(this, \"RestrictedRunnersRole\", {\n  // role 2\n});\n\nconst token1 = StringParameter.fromStringParameterAttributes(stack, \"Token\", {\n  parameterName: \"/gitlab-runner/token1\",\n});\n\nconst token2 = StringParameter.fromStringParameterAttributes(stack, \"Token\", {\n  parameterName: \"/gitlab-runner/token2\",\n});\n\nnew GitlabRunnerAutoscaling(this, \"Runner\", {\n  runners: [\n    {\n      token: token1,\n      configuration: {\n        name: \"privileged-runner\",\n      },\n      role: privilegedRole,\n    },\n    {\n      token: token2,\n      configuration: {\n        name: \"restricted-runner\",\n        docker: {\n          privileged: false, // Run unprivileged\n        },\n      },\n      role: restrictedRole,\n    },\n  ],\n});\n```\n\nSee [example](https://github.com/pepperize/cdk-autoscaling-gitlab-runner-example/blob/main/src/machine-image.ts),\n[GitlabRunnerAutoscalingProps](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/API.md#@pepperize/cdk-autoscaling-gitlab-runner.GitlabRunnerAutoscalingProps)\n\n### Spot instances\n\nBy default, EC2 Spot Instances are requested.\n\n```typescript\nconst token = StringParameter.fromStringParameterAttributes(stack, \"Token\", {\n  parameterName: \"/gitlab-runner/token\",\n});\n\nnew GitlabRunnerAutoscaling(this, \"Runner\", {\n  runners: [\n    {\n      token: token,\n      configuration: {\n        machine: {\n          machineOptions: {\n            requestSpotInstance: false,\n            spotPrice: 0.5,\n          },\n        },\n      },\n    },\n  ],\n});\n```\n\nSee [example](https://github.com/pepperize/cdk-autoscaling-gitlab-runner-example/blob/main/src/on-demand-instances.ts),\n[EC2 spot price](https://aws.amazon.com/de/ec2/spot/pricing/),\n[MachineConfiguration](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/API.md#machineconfiguration-),\n[MachineOptions](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/API.md#machineoptions-),\n[Advanced configuration - runners.machine.autoscaling](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnersmachineautoscaling-sections)\n\n### Cross-Compile with Multiarch\n\nTo build binaries of different architectures can also use [Multiarch](https://wiki.debian.org/Multiarch)\n\n```typescript\nconst token = StringParameter.fromStringParameterAttributes(stack, \"Token\", {\n  parameterName: \"/gitlab-runner/token\",\n});\n\nnew GitlabRunnerAutoscaling(this, \"Runner\", {\n  runners: [\n    {\n      token: token,\n      configuration: {\n        docker: {\n          privileged: true,\n        },\n      },\n    },\n  ],\n});\n```\n\nConfigure your [.gitlab-ci.yml](https://docs.gitlab.com/ee/ci/yaml/) file\n\n```yaml\nbuild:\n  image: multiarch/debian-debootstrap:armhf-buster\n  services:\n    - docker:stable-dind\n    - name: multiarch/qemu-user-static:register\n      command:\n        - \"--reset\"\n  script:\n    - make build\n```\n\nSee [multiarch/qemu-user-static](https://hub.docker.com/r/multiarch/qemu-user-static)\n\n### Running on AWS Graviton\n\nTo run your jobs on [AWS Graviton](https://aws.amazon.com/ec2/graviton/) you have to provide an AMI for arm64 architecture.\n\n```typescript\nconst token = StringParameter.fromStringParameterAttributes(stack, \"Token\", {\n  parameterName: \"/gitlab-runner/token\",\n});\n\nnew GitlabRunnerAutoscaling(this, \"Runner\", {\n  runners: [\n    {\n      token: token,\n      configuration: {\n        instanceType: InstanceType.of(InstanceClass.M6G, InstanceSize.LARGE),\n        machineImage: MachineImage.genericLinux({\n          [this.region]: new LookupMachineImage({\n            name: \"ubuntu/images/hvm-ssd/ubuntu-focal-20.04-*-server-*\",\n            owners: [\"099720109477\"],\n            filters: {\n              architecture: [InstanceArchitecture.ARM_64],\n              \"image-type\": [\"machine\"],\n              state: [\"available\"],\n              \"root-device-type\": [\"ebs\"],\n              \"virtualization-type\": [\"hvm\"],\n            },\n          }).getImage(this).imageId,\n        }),\n      },\n    },\n  ],\n});\n```\n\nSee [Ubuntu Amazon EC2 AMI Locator](https://cloud-images.ubuntu.com/locator/ec2/)\n\n### Custom runner's role\n\nTo deploy from within your GitLab Runner Instances, you may pass a Role with the IAM Policies attached.\n\n```typescript\nconst role = new Role(this, \"RunnersRole\", {\n  assumedBy: new ServicePrincipal(\"ec2.amazonaws.com\", {}),\n  inlinePolicies: {},\n});\nconst token = StringParameter.fromStringParameterAttributes(stack, \"Token\", {\n  parameterName: \"/gitlab-runner/token\",\n});\n\nnew GitlabRunnerAutoscaling(this, \"Runner\", {\n  runners: [\n    {\n      role: role,\n      token: token,\n      configuration: {\n        // optionally configure your runner\n      },\n    },\n  ],\n});\n```\n\nSee [example](https://github.com/pepperize/cdk-autoscaling-gitlab-runner-example/blob/main/src/runner-role.ts),\n[GitlabRunnerAutoscalingProps](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/API.md#@pepperize/cdk-autoscaling-gitlab-runner.GitlabRunnerAutoscalingProps)\n\n### Vpc\n\nIf no existing Vpc is passed, a cheap [VPC](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-ec2.Vpc.html) with a NatInstance (t3.nano) and a single AZ will be created.\n\n```typescript\nconst natInstanceProvider = aws_ec2.NatProvider.instance({\n  instanceType: aws_ec2.InstanceType.of(InstanceClass.T3, InstanceSize.NANO), // using a cheaper gateway (not scalable)\n});\nconst vpc = new Vpc(this, \"Vpc\", {\n  // Your custom vpc, i.e.:\n  natGatewayProvider: natInstanceProvider,\n  maxAzs: 1,\n});\n\nconst token = StringParameter.fromStringParameterAttributes(stack, \"Token\", {\n  parameterName: \"/gitlab-runner/token\",\n});\n\nnew GitlabRunnerAutoscaling(this, \"Runner\", {\n  runners: [\n    {\n      token: token,\n      configuration: {\n        // optionally configure your runner\n      },\n    },\n  ],\n  network: { vpc: vpc },\n});\n```\n\nSee [example](https://github.com/pepperize/cdk-autoscaling-gitlab-runner-example/blob/main/src/vpc.ts),\n[GitlabRunnerAutoscalingProps](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/API.md#@pepperize/cdk-autoscaling-gitlab-runner.GitlabRunnerAutoscalingProps)\n\n### Zero config\n\nDeploys the [Autoscaling GitLab Runner on AWS EC2](https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/) with the default settings mentioned above.\n\nHappy with the presets?\n\n```typescript\nconst token = StringParameter.fromStringParameterAttributes(stack, \"Token\", {\n  parameterName: \"/gitlab-runner/token\",\n});\n\nnew GitlabRunnerAutoscaling(this, \"Runner\", {\n  runners: [\n    {\n      token: token,\n      configuration: {\n        // optionally configure your runner\n      },\n    },\n  ],\n});\n```\n\nSee [example](https://github.com/pepperize/cdk-autoscaling-gitlab-runner-example/blob/main/src/zero-config.ts),\n[GitlabRunnerAutoscalingProps](https://github.com/pepperize/cdk-autoscaling-gitlab-runner/blob/main/API.md#@pepperize/cdk-autoscaling-gitlab-runner.GitlabRunnerAutoscalingProps)\n\n## Projen\n\nThis project uses [projen](https://github.com/projen/projen) to maintain project configuration through code. Thus, the synthesized files with projen should never be manually edited (in fact, projen enforces that).\n\nTo modify the project setup, you should interact with rich strongly-typed\nclass [AwsCdkTypeScriptApp](https://github.com/projen/projen/blob/master/API.md#projen-awscdktypescriptapp) and\nexecute `npx projen` to update project configuration files.\n\n> In simple words, developers can only modify `.projenrc.js` file for configuration/maintenance and files under `/src` directory for development.\n\nSee also [Create and Publish CDK Constructs Using projen and jsii](https://github.com/seeebiii/projen-test).\n\n\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "AWS CDK GitLab Runner autoscaling on EC2 instances using docker+machine executor.",
    "version": "0.2.111",
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "49107d9f27202395e9539c33e1376e79",
                "sha256": "b18099ef86878ff6322a576117ae6b806163c67a88efb851d5f9c8784f59707f"
            },
            "downloads": -1,
            "filename": "pepperize.cdk_autoscaling_gitlab_runner-0.2.111-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "49107d9f27202395e9539c33e1376e79",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": "~=3.7",
            "size": 173231,
            "upload_time": "2022-06-30T02:44:53",
            "upload_time_iso_8601": "2022-06-30T02:44:53.332108Z",
            "url": "https://files.pythonhosted.org/packages/5d/d0/e3eda09455b4b729ac804ba6a5bb334e342e16403a6cd197d20bf0472b39/pepperize.cdk_autoscaling_gitlab_runner-0.2.111-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "2926252b7ac15770c4033f72c49eb100",
                "sha256": "bda1bb9d6dd380b8484aaacd420a4cd24053de3b541e45e84439f2e2422e0cf1"
            },
            "downloads": -1,
            "filename": "pepperize.cdk-autoscaling-gitlab-runner-0.2.111.tar.gz",
            "has_sig": false,
            "md5_digest": "2926252b7ac15770c4033f72c49eb100",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": "~=3.7",
            "size": 173253,
            "upload_time": "2022-06-30T02:44:56",
            "upload_time_iso_8601": "2022-06-30T02:44:56.409070Z",
            "url": "https://files.pythonhosted.org/packages/ed/7d/f345c045a59dc9b135dec53eb41500b414187e22b0f8d40a108d71b45d16/pepperize.cdk-autoscaling-gitlab-runner-0.2.111.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2022-06-30 02:44:56",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "github_user": "pepperize",
    "github_project": "cdk-autoscaling-gitlab-runner.git",
    "lcname": "pepperize.cdk-autoscaling-gitlab-runner"
}
        
Elapsed time: 0.71796s