# T4-CICD Project
This project is a CI/CD system designed to work on repository of any programming languages. It has been tested on **Python**, **Java**, and **JavaScript** repository.
Pypi package: [t4-cicd](https://pypi.org/project/t4-cicd/)
## About This README
This README is divided into two main sections:
1. **User Section**:
- For those who want to use the application, this section provides setup instructions, requirements for external services, and basic usage commands.
2. **Developer Section**:
- For contributors or developers working on this project, this section includes instructions for setting up the development environment, managing dependencies, running tests, and generating documentation.
# User Setup Instructions For Using The Application
The `t4-cicd` CLI application requires the following external services:
- MongoDB service
- Docker Engine service
- AWS S3 Service (user must use their own AWS account)
## Setting Up the Docker Engine Service
Install Docker Desktop from the [official website](https://www.docker.com/products/docker-desktop/). Docker Engine is included in the Docker Desktop installation.
On macOS or Windows:
Ensure Docker Desktop is open and running before using the `t4-cicd` CLI application.
On Linux:
Install Docker Engine directly via your package manager and ensure the Docker service (dockerd) is running.
The `t4-cicd` CLI uses the Docker Engine to perform its operations.
## Setting Up the MongoDB Service
You can use a local or remote MongoDB Service. You will need to save the URL to the MongoDB service
into a `.env` file in your project folder, or as an environment variable in your shell configuration file i.e., `~/.bashrc` or `~/.zshrc`.
The example below shows how to set up the `MONGO_DB_URL` for a local and remote MongoDB service.
```shell
# Local
export MONGO_DB_URL="mongodb://localhost:27017/"
# Remote, replace the <db_username>, <db_password> and <cluster_address>
export MONGO_DB_URL="mongodb+srv://<db_username>:<db_password>@<cluster_address>/"
```
Follow these instructions for setting up local MongoDB service.
Download the MongoDB Community Server from
https://www.mongodb.com/try/download/community-kubernetes-operator
Follow the instructions for self-managed deployments
https://www.mongodb.com/docs/manual/administration/install-community/
To view the database, use MongoDB Compass (comes together with MongoDB installation)
## Setting Up the AWS S3 Service
S3 bucket names must be globally unique across all AWS accounts. In order for `Upload Artifact` feature to work, user/developer needs to specify a unique name in their configuration file `artifact_upload_path: "<UNIQUE_BUCKET_NAME>"`.
```yml
global:
pipeline_name: "cicd_pipeline"
docker: #...
artifact_upload_path: <UNIQUE_BUCKET_NAME>
```
You need to copy and paste the following AWS credentials into `~/.aws/credentials` file.
The credentials you provided must be able to create a bucket and upload into S3.
```shell
aws_access_key_id=<your_id>
aws_secret_access_key=<your_secret_access_key>
aws_session_token=<your_session_token> #if applicable
```
You also need to set up the target region in your `.env` file or set it as environment variable.
```shell
DEFAULT_S3_LOC=<AWS_REGION>
# example set to us-west-2
DEFAULT_S3_LOC='us-west-2'
```
## Installing the Program
You can install directly from PyPI using pip. It is recommended to install it under a virtual environment. See the section under Developer set up for how to activate a virtual environment.
```shell
pip install t4-cicd
```
## Basic usage
There are 2 options for running the application:
1. Run the command in a Git repository’s root directory to target that repository.
2. Run the command in an empty directory and specify the target repository, which can be either local or remote.
**Note**: The target repository must contain a `.cicd-pipelines` folder with a `pipelines.yml` file. These files will be used as the default configuration if not explicitly specified in the command.
Check [here](https://drive.google.com/file/d/1Mj3mnk20G5_Zj8X6bcFSnPRqY0stA-zu/view?usp=sharing) for the syntax required to write the YML file, check [here](https://drive.google.com/file/d/17vtjnLadMs-ItmI6e787_bvmH_j2YPVa/view?usp=sharing) for sample pipeline configuration.
The main commands of the CLI you can run are listed below. Use the --help flag for detailed information about each command.
```shell
# Checking pipeline configuration file,
# only work within a git repository
cid config
# Setting up repository
cid config set-repo
# Getting repository info
cid config get-repo
# Override pipeline configuration value in Datastore (MongoDB)
# only work within a git repository
cid config override
# Running a pipeline run
cid pipeline run
# Retrieve pipeline report
cid pipeline report
```
## Caveats - Current Limitation of the program
- The program will install three packages with name **cli**, **controller** and **util**. Users have to ensure there is no naming conflict on their python packages installed, as well as naming conflict in the repository that they are working on. Installing in a virtual environment is recommended for testing.
- When saving data into the Datastore, the current id for user's session will be used to identify the user. The recommendation is to run the program in an user account instead of the root account, so that the user's session detail will be uniquely identified and stored.
- The program need to have access right to write to the parent folder where the command is executed to write the debug log. For example, if command is executed in directory `/temp/t4-cicd`, the program need to write a debug log at `/temp` folder.
## Errors and Debug Information
- When errors occur, the program stdout and stderr will give a brief summary of the error message.
- For error details to help in debugging, you can look for the `debug.log` file created:
- By default, a `debug.log` file will be created at the parent directory where the command was executed.
- i.e., command is executed in directory `/temp/t4-cicd`, a `debug.log` will be at `/temp`
# Developer Setup Instructions
First, follow the [User Setup Instructions](#User-Setup-Instructions-For-Using-The-Application) to set the Docker Engine service, MongoDB service and AWS S3 service.
## Installation Instruction from Project GitHub Folder
Reference for Poetry and Click [here](https://medium.com/@chinsj/develop-and-deploy-cli-tool-on-python-with-poetry-and-click-ab62f4341c45)
## Prerequisites:
1. Installed Python 3.12 from official [website](https://www.python.org/downloads/)
2. Installed pipx and poetry
```shell
# Install pipx
python -m pip install --user pipx
python -m pipx ensurepath
# Then go to your environment variable on windows/OS, check the path added and
# update lowercase to uppercase if necessary
# restart your window/OS.
# Installation of poetry using pipx
pipx install poetry
```
3. Activated Virtual Environment
```shell
# Navigate to the project directory
# First create your virtual environment using venv,
# Give it a name like myenv, if you use other name, add it to the .gitignore file
# venv come with standard python package
python -m venv myenv
# Activate your virtual environment. For bash terminal
source <your_venv_dir>/Scripts/activate
# Example when you name venv as myenv
# For Windows
source myenv/Scripts/activate
# For Mac / Unix
source myenv/bin/activate
```
The alternative way to activate the virtual environment is to run `poetry shell`.
This is less recommended as the downside of this approach is any command not starting with poetry
will run using your background environment package instead from the virtual environment.
```sh
poetry shell
```
## Installation, Lint & Test
```shell
# This will install all dependencies, and the project itself into your venv
poetry install
# Run pylint for src directory, --fail-under score to be 9.5
poetry run pylint ./src --fail-under=9.5
# Run pytest with coverage, note current passing coverage score is set at 50%
poetry run pytest
# Run pydoctor to generate the API documentation. This will generate a folder name 'apidocs/'
# in the repository.
# This required Administrative right. To link the file
poetry run pydoctor
# To test if you can run the command
poetry run cid --help
# Or
cid --help
```
## Development - Managing Dependencies
[Reference](https://python-poetry.org/docs/managing-dependencies/#installing-group-dependencies) for dependencies management.
Please check the pyproject.toml file to ensure the dependency is added/removed
```shell
# Adding new dependency for general
poetry add <dependency/lib/package>
# Adding new dependency for development environment only
poetry add <dependency/lib/package> --group dev
# Remove dependencies
poetry remove <dependency/lib/package>
# Remove dependencies from development environment
poetry remove <dependency/lib/package> --group dev
```
## Development - Using Logging
### Steps:
- Use the `get_logger()` function from `util.common_utils`
- Provide arguments to change the logging level and/or directory
- By default, a `debug.log` file will be created at a parent directory where you run the command. You can change its location.
i.e., if you run the command under directory `/temp/t4-cicd`, a `debug.log` will be at `/temp`
## Development Documentation:
Check out the list of documents in this github repository, in `dev-docs/design-docs` folder for the following:
- CLI Documentation
- Component Design
- Configuration File Documentation
- Data Store Documentation
- System Design
- Testing Documentation
For API Documentations, you can look for the artifact generated by the last GitHub workflow action step, with name api-documentations.
The backup API Documentations are available in the following share location [here](https://drive.google.com/file/d/16LzQ7oikFJ35t3c1kQeRPyio3Ls2h4Jc/view?usp=sharing).
Look for index.html to start.
## Contributions
Please fork the repository, create a branch for your changes, and submit a pull request.
Raw data
{
"_id": null,
"home_page": null,
"name": "t4-cicd",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.12",
"maintainer_email": null,
"keywords": null,
"author": "sjchin, jgautama, peihsuan-lin, akshayrao",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/b6/c6/733caa0386ee73b8ad0bb14ed141cccc766dec3f37ccad27f6bf3f551a3b/t4_cicd-0.2.1.tar.gz",
"platform": null,
"description": "# T4-CICD Project\n\nThis project is a CI/CD system designed to work on repository of any programming languages. It has been tested on **Python**, **Java**, and **JavaScript** repository.\n\nPypi package: [t4-cicd](https://pypi.org/project/t4-cicd/)\n## About This README\n\nThis README is divided into two main sections:\n\n1. **User Section**:\n\n - For those who want to use the application, this section provides setup instructions, requirements for external services, and basic usage commands.\n\n2. **Developer Section**:\n - For contributors or developers working on this project, this section includes instructions for setting up the development environment, managing dependencies, running tests, and generating documentation.\n\n# User Setup Instructions For Using The Application\n\nThe `t4-cicd` CLI application requires the following external services:\n\n- MongoDB service\n- Docker Engine service\n- AWS S3 Service (user must use their own AWS account)\n\n## Setting Up the Docker Engine Service\n\nInstall Docker Desktop from the [official website](https://www.docker.com/products/docker-desktop/). Docker Engine is included in the Docker Desktop installation.\n\nOn macOS or Windows:\nEnsure Docker Desktop is open and running before using the `t4-cicd` CLI application.\n\nOn Linux:\nInstall Docker Engine directly via your package manager and ensure the Docker service (dockerd) is running.\n\nThe `t4-cicd` CLI uses the Docker Engine to perform its operations.\n\n## Setting Up the MongoDB Service\n\nYou can use a local or remote MongoDB Service. You will need to save the URL to the MongoDB service\ninto a `.env` file in your project folder, or as an environment variable in your shell configuration file i.e., `~/.bashrc` or `~/.zshrc`.\n\nThe example below shows how to set up the `MONGO_DB_URL` for a local and remote MongoDB service.\n\n```shell\n# Local\nexport MONGO_DB_URL=\"mongodb://localhost:27017/\"\n\n# Remote, replace the <db_username>, <db_password> and <cluster_address>\nexport MONGO_DB_URL=\"mongodb+srv://<db_username>:<db_password>@<cluster_address>/\"\n```\n\nFollow these instructions for setting up local MongoDB service.\nDownload the MongoDB Community Server from\nhttps://www.mongodb.com/try/download/community-kubernetes-operator\n\nFollow the instructions for self-managed deployments\nhttps://www.mongodb.com/docs/manual/administration/install-community/\n\nTo view the database, use MongoDB Compass (comes together with MongoDB installation)\n\n## Setting Up the AWS S3 Service\n\nS3 bucket names must be globally unique across all AWS accounts. In order for `Upload Artifact` feature to work, user/developer needs to specify a unique name in their configuration file `artifact_upload_path: \"<UNIQUE_BUCKET_NAME>\"`.\n\n```yml\nglobal:\n pipeline_name: \"cicd_pipeline\"\n docker: #...\n artifact_upload_path: <UNIQUE_BUCKET_NAME>\n```\n\nYou need to copy and paste the following AWS credentials into `~/.aws/credentials` file.\nThe credentials you provided must be able to create a bucket and upload into S3.\n\n```shell\naws_access_key_id=<your_id>\naws_secret_access_key=<your_secret_access_key>\naws_session_token=<your_session_token> #if applicable\n```\n\nYou also need to set up the target region in your `.env` file or set it as environment variable.\n\n```shell\nDEFAULT_S3_LOC=<AWS_REGION>\n# example set to us-west-2\nDEFAULT_S3_LOC='us-west-2'\n```\n\n## Installing the Program\n\nYou can install directly from PyPI using pip. It is recommended to install it under a virtual environment. See the section under Developer set up for how to activate a virtual environment.\n\n```shell\npip install t4-cicd\n```\n\n## Basic usage\n\nThere are 2 options for running the application:\n\n1. Run the command in a Git repository\u2019s root directory to target that repository.\n2. Run the command in an empty directory and specify the target repository, which can be either local or remote.\n\n**Note**: The target repository must contain a `.cicd-pipelines` folder with a `pipelines.yml` file. These files will be used as the default configuration if not explicitly specified in the command.\n\nCheck [here](https://drive.google.com/file/d/1Mj3mnk20G5_Zj8X6bcFSnPRqY0stA-zu/view?usp=sharing) for the syntax required to write the YML file, check [here](https://drive.google.com/file/d/17vtjnLadMs-ItmI6e787_bvmH_j2YPVa/view?usp=sharing) for sample pipeline configuration.\n\nThe main commands of the CLI you can run are listed below. Use the --help flag for detailed information about each command.\n\n```shell\n# Checking pipeline configuration file,\n# only work within a git repository\ncid config\n\n# Setting up repository\ncid config set-repo\n\n# Getting repository info\ncid config get-repo\n\n# Override pipeline configuration value in Datastore (MongoDB)\n# only work within a git repository\ncid config override\n\n# Running a pipeline run\ncid pipeline run\n\n# Retrieve pipeline report\ncid pipeline report\n```\n\n## Caveats - Current Limitation of the program\n\n- The program will install three packages with name **cli**, **controller** and **util**. Users have to ensure there is no naming conflict on their python packages installed, as well as naming conflict in the repository that they are working on. Installing in a virtual environment is recommended for testing.\n\n- When saving data into the Datastore, the current id for user's session will be used to identify the user. The recommendation is to run the program in an user account instead of the root account, so that the user's session detail will be uniquely identified and stored.\n\n- The program need to have access right to write to the parent folder where the command is executed to write the debug log. For example, if command is executed in directory `/temp/t4-cicd`, the program need to write a debug log at `/temp` folder.\n\n## Errors and Debug Information\n\n- When errors occur, the program stdout and stderr will give a brief summary of the error message.\n\n- For error details to help in debugging, you can look for the `debug.log` file created:\n - By default, a `debug.log` file will be created at the parent directory where the command was executed.\n - i.e., command is executed in directory `/temp/t4-cicd`, a `debug.log` will be at `/temp`\n\n# Developer Setup Instructions\n\nFirst, follow the [User Setup Instructions](#User-Setup-Instructions-For-Using-The-Application) to set the Docker Engine service, MongoDB service and AWS S3 service.\n\n## Installation Instruction from Project GitHub Folder\n\nReference for Poetry and Click [here](https://medium.com/@chinsj/develop-and-deploy-cli-tool-on-python-with-poetry-and-click-ab62f4341c45)\n\n## Prerequisites:\n\n1. Installed Python 3.12 from official [website](https://www.python.org/downloads/)\n2. Installed pipx and poetry\n\n```shell\n# Install pipx\npython -m pip install --user pipx\npython -m pipx ensurepath\n\n# Then go to your environment variable on windows/OS, check the path added and\n# update lowercase to uppercase if necessary\n# restart your window/OS.\n\n# Installation of poetry using pipx\npipx install poetry\n```\n\n3. Activated Virtual Environment\n\n```shell\n# Navigate to the project directory\n# First create your virtual environment using venv,\n# Give it a name like myenv, if you use other name, add it to the .gitignore file\n# venv come with standard python package\npython -m venv myenv\n\n# Activate your virtual environment. For bash terminal\nsource <your_venv_dir>/Scripts/activate\n\n# Example when you name venv as myenv\n# For Windows\nsource myenv/Scripts/activate\n# For Mac / Unix\nsource myenv/bin/activate\n```\n\nThe alternative way to activate the virtual environment is to run `poetry shell`.\nThis is less recommended as the downside of this approach is any command not starting with poetry\nwill run using your background environment package instead from the virtual environment.\n\n```sh\npoetry shell\n```\n\n## Installation, Lint & Test\n\n```shell\n# This will install all dependencies, and the project itself into your venv\npoetry install\n\n# Run pylint for src directory, --fail-under score to be 9.5\npoetry run pylint ./src --fail-under=9.5\n\n# Run pytest with coverage, note current passing coverage score is set at 50%\npoetry run pytest\n\n# Run pydoctor to generate the API documentation. This will generate a folder name 'apidocs/'\n# in the repository.\n# This required Administrative right. To link the file\npoetry run pydoctor\n\n# To test if you can run the command\npoetry run cid --help\n# Or\ncid --help\n```\n\n## Development - Managing Dependencies\n\n[Reference](https://python-poetry.org/docs/managing-dependencies/#installing-group-dependencies) for dependencies management.\n\nPlease check the pyproject.toml file to ensure the dependency is added/removed\n\n```shell\n# Adding new dependency for general\npoetry add <dependency/lib/package>\n\n# Adding new dependency for development environment only\npoetry add <dependency/lib/package> --group dev\n\n# Remove dependencies\npoetry remove <dependency/lib/package>\n# Remove dependencies from development environment\npoetry remove <dependency/lib/package> --group dev\n```\n\n## Development - Using Logging\n\n### Steps:\n\n- Use the `get_logger()` function from `util.common_utils`\n- Provide arguments to change the logging level and/or directory\n- By default, a `debug.log` file will be created at a parent directory where you run the command. You can change its location.\n i.e., if you run the command under directory `/temp/t4-cicd`, a `debug.log` will be at `/temp`\n\n## Development Documentation:\n\nCheck out the list of documents in this github repository, in `dev-docs/design-docs` folder for the following:\n\n- CLI Documentation\n- Component Design\n- Configuration File Documentation\n- Data Store Documentation\n- System Design\n- Testing Documentation\n\nFor API Documentations, you can look for the artifact generated by the last GitHub workflow action step, with name api-documentations.\nThe backup API Documentations are available in the following share location [here](https://drive.google.com/file/d/16LzQ7oikFJ35t3c1kQeRPyio3Ls2h4Jc/view?usp=sharing).\nLook for index.html to start.\n\n## Contributions\n\nPlease fork the repository, create a branch for your changes, and submit a pull request.\n\n",
"bugtrack_url": null,
"license": "LICENSE",
"summary": null,
"version": "0.2.1",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "581932ca6ea2075ba7b1328a38f9710c84ce36c7378b53f84347e2744f3e915d",
"md5": "a562a34107cc536d47f764aa6198033c",
"sha256": "422d3313cfa63510beda991636936a5f7e5965db13ce6365993c8e26dc5da1b8"
},
"downloads": -1,
"filename": "t4_cicd-0.2.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a562a34107cc536d47f764aa6198033c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.12",
"size": 56725,
"upload_time": "2024-12-01T03:41:41",
"upload_time_iso_8601": "2024-12-01T03:41:41.165933Z",
"url": "https://files.pythonhosted.org/packages/58/19/32ca6ea2075ba7b1328a38f9710c84ce36c7378b53f84347e2744f3e915d/t4_cicd-0.2.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b6c6733caa0386ee73b8ad0bb14ed141cccc766dec3f37ccad27f6bf3f551a3b",
"md5": "86d3b000f952265c9b139ac3d7eee2da",
"sha256": "fb0e11357968d044b4f6f6af3c6203c40ff1e86b9f89d7f377c2a73734ce5bab"
},
"downloads": -1,
"filename": "t4_cicd-0.2.1.tar.gz",
"has_sig": false,
"md5_digest": "86d3b000f952265c9b139ac3d7eee2da",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.12",
"size": 53365,
"upload_time": "2024-12-01T03:41:49",
"upload_time_iso_8601": "2024-12-01T03:41:49.549584Z",
"url": "https://files.pythonhosted.org/packages/b6/c6/733caa0386ee73b8ad0bb14ed141cccc766dec3f37ccad27f6bf3f551a3b/t4_cicd-0.2.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-01 03:41:49",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "t4-cicd"
}
JSON