<div align="center">
# Harvey
The lightweight Docker Compose deployment platform.
[](https://github.com/Justintime50/harvey/actions)
[](https://coveralls.io/github/Justintime50/harvey?branch=main)
[](https://pypi.org/project/harvey-cd/)
[](LICENSE)
<img src="https://raw.githubusercontent.com/justintime50/assets/main/src/harvey/showcase.png" alt="Showcase">
</div>
> Harvey is unstable and rapidly changing. Although used in the wild, it's not completely documented and interfaces are changing frequently. Keep an eye on the project as it continues to mature.
## Why Docker Compose Deployments
I've long been a fan of the simplicity of a `docker-compose` file and its usage. Deploying with systems such as Rancher or a self-hosted GitLab seemed too daunting with uneccessary overhead for simple projects. Why can't I use the same setup in production for spinning up projects as I do during development? Skip the environment variable injection and key stores, configuring production machines, and separate workflow configuration by having Harvey spin up my project by [using Docker Compose in production](https://docs.docker.com/compose/production/), just like I do locally.
## How it Works
Harvey receives a webhook from GitHub, pulls in the changes, and deploys them. If you have Slack enabled, Harvey can send you the deployment summary along with running healthchecks to ensure the deployment was successful.
1. GitHub webhook fires and is received by Harvey stating that a new commit hit an enabled repo on an allowed branch to be deployed
2. Harvey pulls in your changes and builds your Docker image locally
3. Next, Harvey spins up the new Docker container and tears down the old one once it's up and running
4. (Optional) Harvey will then run a container healthcheck to ensure your container is up and running
5. (Optional) Harvey can send a message via Slack to notify users the status of the deployment
## Install
Because Harvey interacts directly with the Docker daemon (using sockets), to build and orchestrate Docker images and containers, Harvey cannot run in a Docker container itself and must be run on your bare-metal OS.
```bash
# Install Harvey via GitHub
git clone https://github.com/Justintime50/harvey.git
cp .env-example .env
make install
```
1. Install Docker & login
1. Ensure you've added your ssh key to the ssh agent: `ssh-add` followed by your password
1. Setup enviornment variables as needed in the `.env` file
1. Enable GitHub webhooks for all your repositories you want to use Harvey with (point them to `http://example.com:5000/deploy`, send the payload as JSON)
- You could alternatively setup a GitHub Action or other CI flow to only trigger a webhook event once tests pass for instance (must include all the details as if it were generated by GitHub, see [Workflow Webhook](https://github.com/distributhor/workflow-webhook) for an example on how to do this.)
## Usage
```bash
# Run locally for development (runs via Flask)
make run
# Run in production (runs via uWSGI)
make prod
# Spin up the optional reverse proxy (adjust the URLs in the docker-compose files)
docker compose up -d # local
docker compose -f docker-compose.yml -f docker-compose-prod.yml up -d # prod
```
### Things to Know
- Harvey will shallow clone your project to the most recent commit
- Harvey expects the container name to match the GitHub repository name exactly, otherwise the healthcheck will fail
- Harvey does not handle renamed or transferred repos for you. If you rename a repo, you may need to intervene manually to shut down the old container, remove it from Harvey, and startup the new one initially on your own
- Initial deployments are not gracefully handled. Because Harvey requires no configuration for a project, it assumes everything is already setup on the server. This means that on an initial deploy, you will need to set environment variables on your server, migrate databases, and whatever else may be required, at which point you may need to redeploy the project for the changes to take affect
- Harvey automatically rotates log files and keeps them for 2 weeks before purging them
- Because we use threads, you cannot kill an ongoing deployment because you cannot reliably kill a thread
### Harvey Configuration
#### Configuration Criteria
- Each repo either needs a `.harvey.yml` file in the root directory stored in git (which will be used whenever a GitHub webhook fires) or a `data` key passed into the webhook delivered to Harvey (via something like GitHub Actions). This can be accomplished by using something like [workflow-webhook](https://github.com/distributhor/workflow-webhook) or another homegrown solution (requires the entire webhook payload from GitHub. Harvey will always fallback to the `.harvey.yml` file if there is no `data` key present)
- You can specify one of `deploy` or `pull` as the `deployment_type` to run (`deploy` is the default)
- This file must follow proper JSON standards (start and end with `{ }`, contain commas after each item, no trailing commas, and be surrounded by quotes)
- Optional: `prod_compose: true` json can be passed to instruct Harvey to use a prod `docker-compose` file in addition to the base compose file. This will run the equivelant of the following when deploying: `docker-compose -f docker-compose.yml -f docker-compose-prod.yml`.
#### .harvey.yaml Example
```yml
deployment_type: deploy
prod_compose: true
healthcheck:
- container_name_1
- container_name_2
```
#### GitHub Action Example
```yml
deploy:
needs: ["test", "lint"]
runs-on: ubuntu-latest
steps:
- name: Deploy to Harvey
if: github.ref == 'refs/heads/main'
uses: distributhor/workflow-webhook@v2
env:
webhook_type: "json-extended"
webhook_url: ${{ secrets.WEBHOOK_URL }}
webhook_secret: ${{ secrets.WEBHOOK_SECRET }}
data: '{ "deployment_type": "deploy", "prod_compose" : true, "healthcheck": ["container_name_1", "container_name_2"] }'
```
Harvey's entrypoint (eg: `http://127.0.0.1:5000/deploy`) accepts a webhook from GitHub. If you'd like to simulate a GitHub webhook, simply pass a JSON file like the following example to the Harvey webhook endpoint:
```json
{
"ref": "refs/heads/main",
"repository": {
"name": "justinpaulhammond",
"full_name": "Justintime50/justinpaulhammond",
"url": "https://github.com/Justintime50/justinpaulhammond",
"ssh_url": "git@github.com:Justintime50/justinpaulhammond.git",
"owner": {
"name": "Justintime50"
}
},
"commits": [
{
"id": "1",
"author": {
"name": "Justintime50"
}
}
],
"data": {
"deployment_type": "deploy"
}
}
```
### App Configuration
```
Environment Variables:
USE_SLACK Set to "true" to send slack messages. Default: False
SLACK_CHANNEL The Slack channel to send messages to
SLACK_BOT_TOKEN The Slackbot token to use to authenticate each request to Slack
WEBHOOK_SECRET The Webhook secret required by GitHub (if enabled, leave blank to ignore) to secure your webhooks. Default: None
HOST The host Harvey will run on. Default: 127.0.0.1
PORT The port Harvey will run on. Default: 5000
LOG_LEVEL The logging level used for the entire application. Default: INFO
ALLOWED_BRANCHES A comma separated list of branch names that are allowed to trigger deployments from a webhook event. Default: "main,master"
PAGINATION_LIMIT The number of records to return via API. Default: 20
OPERATION_TIMEOUT The number of seconds any given operation (git command, deploy pipeline) can take before timing out. Default: 300
DEPLOY_ON_TAG A boolean specifying if a tag pushed will trigger a deploy. Default: True
HARVEY_PATH The path where Harvey will store projects, logs, and the SQLite databases. Default: ~/harvey
SENTRY_URL The URL authorized to receive sentry alerts.
```
### API
The Harvey API can be secured via Basic Auth by setting the `WEBHOOK_SECRET` env var (this secret is used for authenticating a webhook came from GitHub in addition to securing the remaining endpoints).
The following endpoints are available to interact with Harvey:
- `/deployments` (GET) - Retrieve a list of deployments
- `/deployments/{deployment_id}` (GET) - Retrieve the details of a single deployment
- `/deploy` (POST) - Deploy a project with data from a GitHub webhook
- `/projects` (GET) - Retrieve a list of projects
- `/projects/{project_name}/lock` (PUT) - Locks the deployments of a project
- `/projects/{project_name}/unlock` (PUT) - Unlocks the deployments of a project
- `/projects/{project_name}/webhook` (GET) - Retrieves the current webhook of a project
- `/locks` (GET) - Retrieve a list of locks
- `/locks/{project_name}` (GET) - Retrieve the lock status of a project
- `/threads` (GET) - Retrieves a list of threads (named after projects) running via Harvey. A thread indicates an ongoing deployment
#### Example API Calls
```bash
# Retrieve a list of deployments
curl -X GET http://127.0.0.1:5000/deployments
# Retrieve a deployment from Harvey using the full repo name
curl -X GET http://127.0.0.1:5000/deployments/justintime50-justinpaulhammond
```
## Development
```bash
# Get a comprehensive list of development tools
make help
```
### Releasing
1. Update the version in `Config.harvey_version`
1. Update the version in `setup.py`
Raw data
{
"_id": null,
"home_page": "http://github.com/justintime50/harvey",
"name": "harvey-cd",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7, <4",
"maintainer_email": "",
"keywords": "",
"author": "Justintime50",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/50/f7/a8dead4c29a9cb8557163b97f73a7b2a3c4a3cd6f568c039d8764967b395/harvey-cd-0.24.0.tar.gz",
"platform": null,
"description": "<div align=\"center\">\n\n# Harvey\n\nThe lightweight Docker Compose deployment platform.\n\n[](https://github.com/Justintime50/harvey/actions)\n[](https://coveralls.io/github/Justintime50/harvey?branch=main)\n[](https://pypi.org/project/harvey-cd/)\n[](LICENSE)\n\n<img src=\"https://raw.githubusercontent.com/justintime50/assets/main/src/harvey/showcase.png\" alt=\"Showcase\">\n\n</div>\n\n> Harvey is unstable and rapidly changing. Although used in the wild, it's not completely documented and interfaces are changing frequently. Keep an eye on the project as it continues to mature.\n\n## Why Docker Compose Deployments\n\nI've long been a fan of the simplicity of a `docker-compose` file and its usage. Deploying with systems such as Rancher or a self-hosted GitLab seemed too daunting with uneccessary overhead for simple projects. Why can't I use the same setup in production for spinning up projects as I do during development? Skip the environment variable injection and key stores, configuring production machines, and separate workflow configuration by having Harvey spin up my project by [using Docker Compose in production](https://docs.docker.com/compose/production/), just like I do locally.\n\n## How it Works\n\nHarvey receives a webhook from GitHub, pulls in the changes, and deploys them. If you have Slack enabled, Harvey can send you the deployment summary along with running healthchecks to ensure the deployment was successful.\n\n1. GitHub webhook fires and is received by Harvey stating that a new commit hit an enabled repo on an allowed branch to be deployed\n2. Harvey pulls in your changes and builds your Docker image locally\n3. Next, Harvey spins up the new Docker container and tears down the old one once it's up and running\n4. (Optional) Harvey will then run a container healthcheck to ensure your container is up and running\n5. (Optional) Harvey can send a message via Slack to notify users the status of the deployment\n\n## Install\n\nBecause Harvey interacts directly with the Docker daemon (using sockets), to build and orchestrate Docker images and containers, Harvey cannot run in a Docker container itself and must be run on your bare-metal OS.\n\n```bash\n# Install Harvey via GitHub\ngit clone https://github.com/Justintime50/harvey.git\ncp .env-example .env\nmake install\n```\n\n1. Install Docker & login\n1. Ensure you've added your ssh key to the ssh agent: `ssh-add` followed by your password\n1. Setup enviornment variables as needed in the `.env` file\n1. Enable GitHub webhooks for all your repositories you want to use Harvey with (point them to `http://example.com:5000/deploy`, send the payload as JSON)\n - You could alternatively setup a GitHub Action or other CI flow to only trigger a webhook event once tests pass for instance (must include all the details as if it were generated by GitHub, see [Workflow Webhook](https://github.com/distributhor/workflow-webhook) for an example on how to do this.)\n\n## Usage\n\n```bash\n# Run locally for development (runs via Flask)\nmake run\n\n# Run in production (runs via uWSGI)\nmake prod\n\n# Spin up the optional reverse proxy (adjust the URLs in the docker-compose files)\ndocker compose up -d # local\ndocker compose -f docker-compose.yml -f docker-compose-prod.yml up -d # prod\n```\n\n### Things to Know\n\n- Harvey will shallow clone your project to the most recent commit\n- Harvey expects the container name to match the GitHub repository name exactly, otherwise the healthcheck will fail\n- Harvey does not handle renamed or transferred repos for you. If you rename a repo, you may need to intervene manually to shut down the old container, remove it from Harvey, and startup the new one initially on your own\n- Initial deployments are not gracefully handled. Because Harvey requires no configuration for a project, it assumes everything is already setup on the server. This means that on an initial deploy, you will need to set environment variables on your server, migrate databases, and whatever else may be required, at which point you may need to redeploy the project for the changes to take affect\n- Harvey automatically rotates log files and keeps them for 2 weeks before purging them\n- Because we use threads, you cannot kill an ongoing deployment because you cannot reliably kill a thread\n\n### Harvey Configuration\n\n#### Configuration Criteria\n\n- Each repo either needs a `.harvey.yml` file in the root directory stored in git (which will be used whenever a GitHub webhook fires) or a `data` key passed into the webhook delivered to Harvey (via something like GitHub Actions). This can be accomplished by using something like [workflow-webhook](https://github.com/distributhor/workflow-webhook) or another homegrown solution (requires the entire webhook payload from GitHub. Harvey will always fallback to the `.harvey.yml` file if there is no `data` key present)\n- You can specify one of `deploy` or `pull` as the `deployment_type` to run (`deploy` is the default)\n- This file must follow proper JSON standards (start and end with `{ }`, contain commas after each item, no trailing commas, and be surrounded by quotes)\n- Optional: `prod_compose: true` json can be passed to instruct Harvey to use a prod `docker-compose` file in addition to the base compose file. This will run the equivelant of the following when deploying: `docker-compose -f docker-compose.yml -f docker-compose-prod.yml`.\n\n#### .harvey.yaml Example\n\n```yml\ndeployment_type: deploy\nprod_compose: true\nhealthcheck:\n - container_name_1\n - container_name_2\n```\n\n#### GitHub Action Example\n\n```yml\ndeploy:\n needs: [\"test\", \"lint\"]\n runs-on: ubuntu-latest\n steps:\n - name: Deploy to Harvey\n if: github.ref == 'refs/heads/main'\n uses: distributhor/workflow-webhook@v2\n env:\n webhook_type: \"json-extended\"\n webhook_url: ${{ secrets.WEBHOOK_URL }}\n webhook_secret: ${{ secrets.WEBHOOK_SECRET }}\n data: '{ \"deployment_type\": \"deploy\", \"prod_compose\" : true, \"healthcheck\": [\"container_name_1\", \"container_name_2\"] }'\n```\n\nHarvey's entrypoint (eg: `http://127.0.0.1:5000/deploy`) accepts a webhook from GitHub. If you'd like to simulate a GitHub webhook, simply pass a JSON file like the following example to the Harvey webhook endpoint:\n\n```json\n{\n \"ref\": \"refs/heads/main\",\n \"repository\": {\n \"name\": \"justinpaulhammond\",\n \"full_name\": \"Justintime50/justinpaulhammond\",\n \"url\": \"https://github.com/Justintime50/justinpaulhammond\",\n \"ssh_url\": \"git@github.com:Justintime50/justinpaulhammond.git\",\n \"owner\": {\n \"name\": \"Justintime50\"\n }\n },\n \"commits\": [\n {\n \"id\": \"1\",\n \"author\": {\n \"name\": \"Justintime50\"\n }\n }\n ],\n \"data\": {\n \"deployment_type\": \"deploy\"\n }\n}\n```\n\n### App Configuration\n\n```\nEnvironment Variables:\n USE_SLACK Set to \"true\" to send slack messages. Default: False\n SLACK_CHANNEL The Slack channel to send messages to\n SLACK_BOT_TOKEN The Slackbot token to use to authenticate each request to Slack\n WEBHOOK_SECRET The Webhook secret required by GitHub (if enabled, leave blank to ignore) to secure your webhooks. Default: None\n HOST The host Harvey will run on. Default: 127.0.0.1\n PORT The port Harvey will run on. Default: 5000\n LOG_LEVEL The logging level used for the entire application. Default: INFO\n ALLOWED_BRANCHES A comma separated list of branch names that are allowed to trigger deployments from a webhook event. Default: \"main,master\"\n PAGINATION_LIMIT The number of records to return via API. Default: 20\n OPERATION_TIMEOUT The number of seconds any given operation (git command, deploy pipeline) can take before timing out. Default: 300\n DEPLOY_ON_TAG A boolean specifying if a tag pushed will trigger a deploy. Default: True\n HARVEY_PATH The path where Harvey will store projects, logs, and the SQLite databases. Default: ~/harvey\n SENTRY_URL The URL authorized to receive sentry alerts.\n```\n\n### API\n\nThe Harvey API can be secured via Basic Auth by setting the `WEBHOOK_SECRET` env var (this secret is used for authenticating a webhook came from GitHub in addition to securing the remaining endpoints).\n\nThe following endpoints are available to interact with Harvey:\n\n- `/deployments` (GET) - Retrieve a list of deployments\n- `/deployments/{deployment_id}` (GET) - Retrieve the details of a single deployment\n- `/deploy` (POST) - Deploy a project with data from a GitHub webhook\n- `/projects` (GET) - Retrieve a list of projects\n- `/projects/{project_name}/lock` (PUT) - Locks the deployments of a project\n- `/projects/{project_name}/unlock` (PUT) - Unlocks the deployments of a project\n- `/projects/{project_name}/webhook` (GET) - Retrieves the current webhook of a project\n- `/locks` (GET) - Retrieve a list of locks\n- `/locks/{project_name}` (GET) - Retrieve the lock status of a project\n- `/threads` (GET) - Retrieves a list of threads (named after projects) running via Harvey. A thread indicates an ongoing deployment\n\n#### Example API Calls\n\n```bash\n# Retrieve a list of deployments\ncurl -X GET http://127.0.0.1:5000/deployments\n\n# Retrieve a deployment from Harvey using the full repo name\ncurl -X GET http://127.0.0.1:5000/deployments/justintime50-justinpaulhammond\n```\n\n## Development\n\n```bash\n# Get a comprehensive list of development tools\nmake help\n```\n\n### Releasing\n\n1. Update the version in `Config.harvey_version`\n1. Update the version in `setup.py`\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "The lightweight Docker Compose deployment platform.",
"version": "0.24.0",
"project_urls": {
"Homepage": "http://github.com/justintime50/harvey"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2a22961a57727c90a49f9413774863adbff1f725ac87ee1e5b2cd4e65c8fbb85",
"md5": "093ffe3ca8818f8a1b852f339918d883",
"sha256": "0715c836dc7cf46ee0320978150e327dddca7ef2c02d1d25fa4b5ab57bd371bc"
},
"downloads": -1,
"filename": "harvey_cd-0.24.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "093ffe3ca8818f8a1b852f339918d883",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7, <4",
"size": 26316,
"upload_time": "2023-06-20T22:26:50",
"upload_time_iso_8601": "2023-06-20T22:26:50.718672Z",
"url": "https://files.pythonhosted.org/packages/2a/22/961a57727c90a49f9413774863adbff1f725ac87ee1e5b2cd4e65c8fbb85/harvey_cd-0.24.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "50f7a8dead4c29a9cb8557163b97f73a7b2a3c4a3cd6f568c039d8764967b395",
"md5": "d4d79f25c3a758eb8ff9e7430ddf932d",
"sha256": "59f50f7b09dc70937c58d2dcde92411d16ec21e9d6059c6352845bc22596a098"
},
"downloads": -1,
"filename": "harvey-cd-0.24.0.tar.gz",
"has_sig": false,
"md5_digest": "d4d79f25c3a758eb8ff9e7430ddf932d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7, <4",
"size": 28602,
"upload_time": "2023-06-20T22:26:52",
"upload_time_iso_8601": "2023-06-20T22:26:52.152242Z",
"url": "https://files.pythonhosted.org/packages/50/f7/a8dead4c29a9cb8557163b97f73a7b2a3c4a3cd6f568c039d8764967b395/harvey-cd-0.24.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-06-20 22:26:52",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "justintime50",
"github_project": "harvey",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"lcname": "harvey-cd"
}
JSON
