# gitlab-ci-generator
This project builds a dynamic gitlab ci file that is intended to be used for monorepos. The concept is to simplify the amount of work that is needed to manage the monorepo.
## Installation
You can download and run directly with python or install using:
```
pip install gitlab-ci-generator
```
This installs like an executable and you can simply run:
```
gitlab-ci-generator
```
## Usage
The project takes the following parameters:
| Name | Parameter | Description | Required |
| ------------- | ----------------- | -------------------------------------------------------------------------------------------- | -------- |
| Input File | --inputfile/-f | The yaml file that will be used to render the gitlab ci file. | Yes |
| Output File | --outputfile/-o | The yaml file that is rendered using the jinja2 template with data from the input yaml file. | No |
| Template File | --templatefile/-t | A jinja2 file that can be used to override default templating | No |
| Help | -h | Displays help info | No |
This program will load the input yaml file into two distinct dictionaries that are available to the template:
* pipeline_info
* jobs
To run without specifying an output file you can write stdout to a redirect file like so:
```
gitlab-ci-generator --inputfile ./job-dependency.yml 1> /tmp/test.out
```
### pipeline_info
This section is gets loaded as is into a dictionary so you can easily extend it for your needs. The built-in template uses the following:
```
pipeline_info:
shared_includes: <<-- Used by the template to build the include: section of the gitlab ci file.
- name: rules <<-- List of includes and can be of type local or project. The rules example here is used in the template in concert with the share_reference_rules: section to define dynamic rules for jobs.
local: "example/example-config/rules.yml"
shared_variables: << -- Used by the template to build the variables: section of the gitlab ci under each job.
- name: MYVAR <-- The variable name to include
value: MYVAR_VALUE <-- The value string or a variable for the value
shared_reference_rules: <<-- Used by the template to build the rules: section of the gitlab ci under each job.
- rule_name: standard-rule-if <- The rule_source and rule_name will create a !reference[rule_source, rule_name] value under the rules section of each job.
rule_source: .job_changes_rules
only_changes: true
final_job: <<-- Optional section that can be used to specify a final trigger job to run at the end of your pipeline.
name: <<-- The name of the final job.
stage: <<-- The stage that you want the final job to run in.
trigger_job_info: <<-- Info around the final job.
project: <<-- gitlab project location for final job.
ref: <<-- Final job branch reference
file: <<-- Path to the yml file to be triggered.
rules: <<-- Custom rules for the final job.
- rule_name: <<-- The rule_source and rule_name will create a !reference[rule_source, rule_name] value under the rules section of each job.
- rule_source: <<-- Points the rule_name in the !reference
only_changes: true <<-- True will run this job only if changes occur.
```
### jobs
This section is used to define the hierarchy for the gitlab jobs. This section is not extensible and should be structured as follows:
```
jobs: <<-- List of jobs and dependencies that will build the appropriate changes/needs logic in the generated gitlab ci file.
- name: example-1 <<-- name: will be the job name in the generated gitlab file and is required.
folder: example/example-folder-1 <<-- folder: The subfolder in the gitlab repo that will be monitored for changes and contains the sub-repo in the monorepo.
gitlab_yml_file: .gitlab-example.yml <<-- gitlab_yml_file: is the pipeline file that will be triggered in the folder: that is specified
dependent_jobs: <<-- dependent_jobs: is a list of dependent jobs and can have dependent_jobs on its jobs. In other words, this can go multiple levels deep with dependencies.
- name: example-2
folder: example/example-folder-2
gitlab_yml_file: .gitlab-example.yml
```
## Sample Input yaml File
This sample can also be found in the gitlab repo for this project at the location [example/example-config/job-dependency.yml](https://gitlab.com/gary.schaetz/public/gitlab-ci-generator/-/blob/main/example/example-config/job-dependency.yml)
```
---
pipeline_info:
shared_includes:
- name: rules
local: "example/example-config/rules.yml"
shared_variables:
- name: MYVAR
value: MYVAR_VALUE
shared_reference_rules:
- rule_name: standard-rule-if
rule_source: .job_changes_rules
only_changes: true
- rule_name: manual-rule-if
rule_source: .job_changes_rules
when: manual
only_changes: true
- rule_name: schedule-rule-if
rule_source: .job_changes_rules
only_changes: false
final_job:
name: cut-tag
stage: post-deploy
trigger_job_info:
project: gary.schaetz/private/pipelines
ref: main
file: /path/to/child-pipeline.yml
rules:
- rule_name: standard-rule-if
rule_source: .job_changes_rules
only_changes: true
jobs:
- name: example-1
folder: example/example-folder-1
gitlab_yml_file: .gitlab-example.yml
dependent_jobs:
- name: example-2
folder: example/example-folder-2
gitlab_yml_file: .gitlab-example.yml
```
### Sample Output
Using the jinja template [here](https://gitlab.com/gary.schaetz/public/gitlab-ci-generator/-/blob/main/src/gitlab_ci_generator_package/templates/gitlab-template.jinja) and the data from [example/example-config/job-dependency.yml](https://gitlab.com/gary.schaetz/public/gitlab-ci-generator/-/blob/main/example/example-config/job-dependency.yml)
```
include:
- local: 'example/example-config/rules.yml'
stages:
- build
- post-deploy
example-1:
stage: build
variables:
PROJECT_SUBDIR: example/example-folder-1
PROJECT_JOB_NAME: example-1
PROJECT_SUBDIR_LAST_FOLDER: example-folder-1
MYVAR: MYVAR_VALUE
rules:
- if: !reference [.job_changes_rules,standard-rule-if]
changes:
- example/example-folder-1/**/*
- if: !reference [.job_changes_rules,manual-rule-if]
when: manual
changes:
- example/example-folder-1/**/*
- if: !reference [.job_changes_rules,schedule-rule-if]
trigger:
include: example/example-folder-1/.gitlab-example.yml
strategy: depend
forward:
pipeline_variables: true
example-2:
stage: build
variables:
PROJECT_SUBDIR: example/example-folder-2
PROJECT_JOB_NAME: example-2
PROJECT_SUBDIR_LAST_FOLDER: example-folder-2
MYVAR: MYVAR_VALUE
rules:
- if: !reference [.job_changes_rules,standard-rule-if]
changes:
- example/example-folder-2/**/*
- example/example-folder-1/**/*
- if: !reference [.job_changes_rules,manual-rule-if]
when: manual
changes:
- example/example-folder-2/**/*
- example/example-folder-1/**/*
- if: !reference [.job_changes_rules,schedule-rule-if]
needs:
- job: example-1
optional: true
trigger:
include: example/example-folder-2/.gitlab-example.yml
strategy: depend
forward:
pipeline_variables: true
cut-tag:
stage: post-deploy
rules:
- if: !reference [.job_changes_rules,standard-rule-if]
changes:
- example/example-folder-1/**/*
- example/example-folder-2/**/*
trigger:
include:
- project: 'gary.schaetz/private/pipelines'
ref: 'main'
file: '/path/to/child-pipeline.yml'
strategy: depend
forward:
pipeline_variables: true
```
## How to use in your pipeline
A variable is passed into your pipeline that you can see in the previous section called PROJECT_SUBDIR. Use this in before scripts to cd into the active project.
Two other variables are exposed:
* PROJECT_JOB_NAME which contains the value from the name tag in your input yaml
* PROJECT_SUBDIR_LAST_FOLDER the last folder in PROJECT_SUBDIR
Here is an example of a pipeline that will use this program in one job to generate the ci file and then trigger it in the next job. Notice how we set PARENT_PIPELINE_SOURCE variable so that you can use this in your downstream rules as a trigger overrides the original CI_PIPELINE_SOURCE.
```
include:
- project: "gary.schaetz/private/pipelines"
ref: main
file: 'templates/rules/rules.yml'
stages:
- build_config
- run
generate-pipeline:
image: gschaetz/base-alpine:latest
stage: build_config
tags:
- docker
- self-managed
- ubuntu-worker
rules:
- !reference [.job_rules, standard-rule]
before_script:
- pip install gitlab-ci-generator
script:
- gitlab-ci-generator -f job-dependency.yml -o generated-ci.yml
artifacts:
paths:
- generated-ci.yml
trigger-child-pipeline:
stage: run
rules:
- if: !reference [.job_changes_rules,standard-rule-if]
changes:
- "*/**/*"
trigger:
include:
- artifact: generated-ci.yml
job: generate-pipeline
strategy: depend
variables:
PARENT_PIPELINE_ID: $CI_PIPELINE_ID
PARENT_PIPELINE_SOURCE: $CI_PIPELINE_SOURCE
```
## License
MIT License
Raw data
{
"_id": null,
"home_page": "https://gitlab.com/gary.schaetz/public/gitlab-ci-generator",
"name": "gitlab-ci-generator",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "gitlab,template,generator,gitlab-ci.yml,dynamic,pipeline",
"author": "Gary Schaetz",
"author_email": "gary@schaetzkc.com",
"download_url": "https://files.pythonhosted.org/packages/aa/ea/4f52c294fac77e583f9bc60708a5bb46e991dc3589435e8078847e7cdca5/gitlab-ci-generator-1.0.11.tar.gz",
"platform": null,
"description": "# gitlab-ci-generator\nThis project builds a dynamic gitlab ci file that is intended to be used for monorepos. The concept is to simplify the amount of work that is needed to manage the monorepo.\n\n## Installation\nYou can download and run directly with python or install using:\n```\npip install gitlab-ci-generator\n```\nThis installs like an executable and you can simply run: \n```\ngitlab-ci-generator\n```\n\n## Usage\nThe project takes the following parameters:\n| Name | Parameter | Description | Required |\n| ------------- | ----------------- | -------------------------------------------------------------------------------------------- | -------- |\n| Input File | --inputfile/-f | The yaml file that will be used to render the gitlab ci file. | Yes |\n| Output File | --outputfile/-o | The yaml file that is rendered using the jinja2 template with data from the input yaml file. | No |\n| Template File | --templatefile/-t | A jinja2 file that can be used to override default templating | No |\n| Help | -h | Displays help info | No |\n\nThis program will load the input yaml file into two distinct dictionaries that are available to the template:\n* pipeline_info\n* jobs\n\nTo run without specifying an output file you can write stdout to a redirect file like so: \n```\ngitlab-ci-generator --inputfile ./job-dependency.yml 1> /tmp/test.out\n```\n\n### pipeline_info\nThis section is gets loaded as is into a dictionary so you can easily extend it for your needs. The built-in template uses the following:\n```\npipeline_info:\n shared_includes: <<-- Used by the template to build the include: section of the gitlab ci file.\n - name: rules <<-- List of includes and can be of type local or project. The rules example here is used in the template in concert with the share_reference_rules: section to define dynamic rules for jobs.\n local: \"example/example-config/rules.yml\" \n shared_variables: << -- Used by the template to build the variables: section of the gitlab ci under each job.\n - name: MYVAR <-- The variable name to include\n value: MYVAR_VALUE <-- The value string or a variable for the value \n shared_reference_rules: <<-- Used by the template to build the rules: section of the gitlab ci under each job.\n - rule_name: standard-rule-if <- The rule_source and rule_name will create a !reference[rule_source, rule_name] value under the rules section of each job.\n rule_source: .job_changes_rules\n only_changes: true\n final_job: <<-- Optional section that can be used to specify a final trigger job to run at the end of your pipeline.\n name: <<-- The name of the final job.\n stage: <<-- The stage that you want the final job to run in.\n trigger_job_info: <<-- Info around the final job.\n project: <<-- gitlab project location for final job.\n ref: <<-- Final job branch reference\n file: <<-- Path to the yml file to be triggered.\n rules: <<-- Custom rules for the final job.\n - rule_name: <<-- The rule_source and rule_name will create a !reference[rule_source, rule_name] value under the rules section of each job.\n - rule_source: <<-- Points the rule_name in the !reference\n only_changes: true <<-- True will run this job only if changes occur.\n```\n\n### jobs\nThis section is used to define the hierarchy for the gitlab jobs. This section is not extensible and should be structured as follows:\n```\njobs: <<-- List of jobs and dependencies that will build the appropriate changes/needs logic in the generated gitlab ci file.\n - name: example-1 <<-- name: will be the job name in the generated gitlab file and is required. \n folder: example/example-folder-1 <<-- folder: The subfolder in the gitlab repo that will be monitored for changes and contains the sub-repo in the monorepo.\n gitlab_yml_file: .gitlab-example.yml <<-- gitlab_yml_file: is the pipeline file that will be triggered in the folder: that is specified\n dependent_jobs: <<-- dependent_jobs: is a list of dependent jobs and can have dependent_jobs on its jobs. In other words, this can go multiple levels deep with dependencies. \n - name: example-2\n folder: example/example-folder-2\n gitlab_yml_file: .gitlab-example.yml\n```\n\n## Sample Input yaml File\nThis sample can also be found in the gitlab repo for this project at the location [example/example-config/job-dependency.yml](https://gitlab.com/gary.schaetz/public/gitlab-ci-generator/-/blob/main/example/example-config/job-dependency.yml)\n```\n---\npipeline_info:\n shared_includes:\n - name: rules\n local: \"example/example-config/rules.yml\"\n shared_variables:\n - name: MYVAR\n value: MYVAR_VALUE\n shared_reference_rules:\n - rule_name: standard-rule-if\n rule_source: .job_changes_rules\n only_changes: true\n - rule_name: manual-rule-if\n rule_source: .job_changes_rules\n when: manual\n only_changes: true\n - rule_name: schedule-rule-if\n rule_source: .job_changes_rules\n only_changes: false\n final_job:\n name: cut-tag\n stage: post-deploy\n trigger_job_info:\n project: gary.schaetz/private/pipelines\n ref: main\n file: /path/to/child-pipeline.yml\n rules:\n - rule_name: standard-rule-if\n rule_source: .job_changes_rules\n only_changes: true\njobs:\n - name: example-1\n folder: example/example-folder-1\n gitlab_yml_file: .gitlab-example.yml\n dependent_jobs:\n - name: example-2\n folder: example/example-folder-2\n gitlab_yml_file: .gitlab-example.yml\n```\n\n### Sample Output\nUsing the jinja template [here](https://gitlab.com/gary.schaetz/public/gitlab-ci-generator/-/blob/main/src/gitlab_ci_generator_package/templates/gitlab-template.jinja) and the data from [example/example-config/job-dependency.yml](https://gitlab.com/gary.schaetz/public/gitlab-ci-generator/-/blob/main/example/example-config/job-dependency.yml)\n```\ninclude:\n - local: 'example/example-config/rules.yml'\n\nstages:\n - build\n - post-deploy\n\n\nexample-1:\n stage: build\n variables:\n PROJECT_SUBDIR: example/example-folder-1\n PROJECT_JOB_NAME: example-1\n PROJECT_SUBDIR_LAST_FOLDER: example-folder-1\n MYVAR: MYVAR_VALUE\n rules:\n - if: !reference [.job_changes_rules,standard-rule-if]\n changes:\n - example/example-folder-1/**/*\n - if: !reference [.job_changes_rules,manual-rule-if]\n when: manual\n changes:\n - example/example-folder-1/**/*\n - if: !reference [.job_changes_rules,schedule-rule-if]\n trigger:\n include: example/example-folder-1/.gitlab-example.yml \n strategy: depend\n forward:\n pipeline_variables: true\n \n\nexample-2:\n stage: build\n variables:\n PROJECT_SUBDIR: example/example-folder-2\n PROJECT_JOB_NAME: example-2\n PROJECT_SUBDIR_LAST_FOLDER: example-folder-2\n MYVAR: MYVAR_VALUE\n rules:\n - if: !reference [.job_changes_rules,standard-rule-if]\n changes:\n - example/example-folder-2/**/*\n - example/example-folder-1/**/*\n - if: !reference [.job_changes_rules,manual-rule-if]\n when: manual\n changes:\n - example/example-folder-2/**/*\n - example/example-folder-1/**/*\n - if: !reference [.job_changes_rules,schedule-rule-if]\n needs: \n - job: example-1\n optional: true \n trigger:\n include: example/example-folder-2/.gitlab-example.yml \n strategy: depend\n forward:\n pipeline_variables: true\n \ncut-tag:\n stage: post-deploy\n rules:\n - if: !reference [.job_changes_rules,standard-rule-if]\n changes:\n - example/example-folder-1/**/*\n - example/example-folder-2/**/*\n trigger:\n include: \n - project: 'gary.schaetz/private/pipelines'\n ref: 'main'\n file: '/path/to/child-pipeline.yml' \n strategy: depend\n forward:\n pipeline_variables: true\n```\n\n## How to use in your pipeline\nA variable is passed into your pipeline that you can see in the previous section called PROJECT_SUBDIR. Use this in before scripts to cd into the active project. \n\nTwo other variables are exposed:\n* PROJECT_JOB_NAME which contains the value from the name tag in your input yaml\n* PROJECT_SUBDIR_LAST_FOLDER the last folder in PROJECT_SUBDIR\n\nHere is an example of a pipeline that will use this program in one job to generate the ci file and then trigger it in the next job. Notice how we set PARENT_PIPELINE_SOURCE variable so that you can use this in your downstream rules as a trigger overrides the original CI_PIPELINE_SOURCE.\n```\ninclude: \n - project: \"gary.schaetz/private/pipelines\"\n ref: main\n file: 'templates/rules/rules.yml'\n\nstages:\n - build_config\n - run\n\ngenerate-pipeline:\n image: gschaetz/base-alpine:latest\n stage: build_config\n tags: \n - docker \n - self-managed \n - ubuntu-worker\n rules:\n - !reference [.job_rules, standard-rule] \n before_script:\n - pip install gitlab-ci-generator\n script:\n - gitlab-ci-generator -f job-dependency.yml -o generated-ci.yml\n artifacts:\n paths:\n - generated-ci.yml\n\ntrigger-child-pipeline:\n stage: run\n rules:\n - if: !reference [.job_changes_rules,standard-rule-if]\n changes:\n - \"*/**/*\"\n trigger:\n include:\n - artifact: generated-ci.yml\n job: generate-pipeline\n strategy: depend\n variables:\n PARENT_PIPELINE_ID: $CI_PIPELINE_ID\n PARENT_PIPELINE_SOURCE: $CI_PIPELINE_SOURCE\n```\n\n## License\nMIT License\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Generates a mono-repo ci file.",
"version": "1.0.11",
"project_urls": {
"Homepage": "https://gitlab.com/gary.schaetz/public/gitlab-ci-generator"
},
"split_keywords": [
"gitlab",
"template",
"generator",
"gitlab-ci.yml",
"dynamic",
"pipeline"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "aaea4f52c294fac77e583f9bc60708a5bb46e991dc3589435e8078847e7cdca5",
"md5": "d8839c7befcd4518c45be56afafe561a",
"sha256": "0ec1600088cf37c7613044860bc85cf5692801df956b21d647d7a91d71d351a1"
},
"downloads": -1,
"filename": "gitlab-ci-generator-1.0.11.tar.gz",
"has_sig": false,
"md5_digest": "d8839c7befcd4518c45be56afafe561a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 6818,
"upload_time": "2023-08-14T22:04:50",
"upload_time_iso_8601": "2023-08-14T22:04:50.219214Z",
"url": "https://files.pythonhosted.org/packages/aa/ea/4f52c294fac77e583f9bc60708a5bb46e991dc3589435e8078847e7cdca5/gitlab-ci-generator-1.0.11.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-14 22:04:50",
"github": false,
"gitlab": true,
"bitbucket": false,
"codeberg": false,
"gitlab_user": "gary.schaetz",
"gitlab_project": "public",
"lcname": "gitlab-ci-generator"
}
JSON