# dcrx-api
[](https://pypi.org/project/dcrx-api/)
[](https://github.com/scorbettUM/dcrx-api/blob/main/LICENSE)
[](https://github.com/scorbettUM/dcrx-api/blob/main/CODE_OF_CONDUCT.md)
[](https://pypi.org/project/dcrx-api/)
DCRX-API is a RESTful implementation of the dcrx library, allowing you to create Docker images on request! DCRX-API extends dcrx, providing integrations with the Docker API and dcrx that allow images to be built by making REST requests.
DCRX-API also includes basic JWT authorization and user management (no-RBAC), and is intended to be deployed as an internal-facing service. dcrx also provides OpenAPI documentation to make getting started intuitive, and features integrations with SQLite, Postgres, and MySQL for storing and managing users.
# Setup and Installation
DCRX-API is available both as a Docker image and as a PyPi package. For local use, we recommend using PyPi, Python 3.10+, and a virtual environment. To install, run:
```bash
python -m venv ~/.dcrx && \
source ~/.dcrx/bin/activate && \
pip install dcrx-api
```
To get the Docker image, run:
```
docker pull adalundhe/dcrx-api:latest
```
# Getting Started
DCRX-API requires a slew of environmental variables in order to run correctly. These include:
```
DCRX_API_JOB_TIMEOUT=10m # The maximum amount of time before an image build + push job times out and is cancelled. Default is 10 minutes.
DCRX_API_MAX_MEMORY_PERCENT_USAGE=50 # The percent of system memory DCRX-API and Docker image builds created by DCRX-API can use. If exceeded, new jobs will be rejected and requests will return a 400 error. Default is 50 percent.
DCRX_API_JOB_PRUNE_INTERVAL='1s' # How often dcrx-api checks for done, failed, or cancelled jobs to remove from in-memory storage.
DCRX_API_JOB_MAX_AGE='1m'= # Time before done, failed, or cancelled jobs are pruned from in-memory storage.
DCRX_API_JOB_MAX_PENDING_WAIT=10m # The maximum amount of time a job can remain in PENDING status. Jobs that exceed this threshold are timed out and will fail.
DCRX_API_JOB_MAX_PENDING=100 # The maximum number of pending jobs that a single instance of DCRX-API allows. Default is 100.
DCRX_API_JOB_POOL_SIZE=10 # Maximum number of concurrent builds. Default is 10.
DCRX_API_JOB_WORKERS=4 # Number of workers to use per-job. Default is the number os OS threads.
DCRX_API_TOKEN_EXPIRATION='15m' # Time for JWT authorization token to expire. Default is 15 minutes.
DCRX_API_SECRET_KEY=testingthis # Initial secret used to hash user passwords.
DCRX_API_AUTH_ALGORITHM=HS256 # Algorithm to use for hashing user passwords.
DCRX_API_DATABASE_TRANSACTION_RETRIES=3 # Number of times to retry a failed transaction. Default is 3.
DCRX_API_DATABASE_TYPE=sqlite # Type of database to use. Default is sqlite and options are mysql, asyncpg, and sqlite.
DCRX_API_DATABASE_USER=admin # Username used for database connection authentication
DCRX_API_DATABASE_PASSWORD=test1234 # Password used for database connection authentrication
DCRX_API_DATABASE_NAME=dcrx # Name of database to use for storing users. Default is dcrx.
DCRX_API_DATABASE_URI=sqlite+aiosqlite:///dcrx # URL/URI of SQL database for storing users and job metadata.
DCRX_API_DATABASE_PORT=3369 # Port of SQL database for storing users and job metadata.
```
Prior to starting the server, we recommend seeding the database with an initial user. To do so, run the command:
```bash
dcrx-api database initialize --username $USERNAME --password $PASSWORD --first-name $FIRST_NAME --last-name $LAST_NAME --email $EMAIL
```
If you've chosen to run the Docker image, this step is taken care of for you, with the default user being:
```
username: admin
password: dcrxadmin1*
first-name: admin
last-name: user
email: admin@admin.com
```
Next run:
```bash
dcrx-api server run
```
and navigate to `localhost:2277/docs`, where you'll find dcrx-api's OpenAPI documentation.
# Authenticating and Building an Image
From here, we'll need to authenticate using the initial user we created above. Open the tab for `/users/login`, and for the `username` and `password` fields enter the username and password of the initial user, then click `Execute`. A `200` status response containg the initial user's username and id should return.
Before creating and pushing any images, we need to create a Registry. Navigate to the `/registry/create` tab. We'll need to provide a `registry_name` (unique shortand reference of our choosing), `registry_uri` (HTTP url to the registry), `registry_user`, and `registry_password:
```json
{
"registry_name": "hello-world",
"registry_uri": "https://docker.io/v1/testreg/hello-world",
"registry_user": "testuser",
"registry_password": "testpassword1*"
}
```
Submit the request. DCRX-API will take the information, encrypt the registry password, and store it.
Next navigate to the `/jobs/images/create` tab. As before, we'll need to fill out some data. Go ahead and copy paste the below into the input for the request body:
```json
{
"name": "testreg/hello-world",
"tag": "latest",
"registry": {
"registry_name": "hello-world"
},
"layers": [
{
"layer_type": "stage",
"base": "python",
"tag": "3.11-slim"
},
{
"layer_type": "entrypoint",
"command": [
"echo",
"Hello world!"
]
}
]
}
```
A dcrx-api image build request requires, at-minimum, a `name`, `tag`, `registry_name`, and one or more `layer` objects, which will be processed and build in-order of appearance. Note that each layer object almost exactly corresponds to its `dcrx` call. The above corresponds exactly to:
```python
from dcrx import Image
hello_world = Image('hello-world')
hello_world.stage(
'python',
'3.11-slim'
).entrypoint([
'echo',
'Hello world!'
])
```
Go ahead and submit the request via the `Execute` button. This will start a job to build the specified image and push it to the repository specified in the request.
Note that dcrx-api doesn't wait for the image to be built, instead returning a `JobMetadata` response with a status code of `200`, including the `job_id`. Building images is time-intensive and could potentially bog down a server, so dcrx-api builds and pushes images in the background. We can use the `job_id` of a Job to make further `GET` requests to the `/jobs/images/{job_id}/get` endpoint to monitor and check how our job is progressing. If we need to cancel a job for any reason, we can simple make a `DELETE` request to `/jobs/images/{job_id}/cancel`.
# Running the Docker Image
To run the Docker image, we recommend first creating a `.env` file with the required environment variables noted above. Then run the image:
```bash
docker run -p 2277:2277 --env-file .env --privileged dcrx-api:latest
```
# Deploying via Helm
DCRX-API has a minimal helm chart to help you scale up when ready. To begin, first create a Kuberenetes secret as below:
```
kubectl create secret generic dcrx-api-secrets --from-literal="DCRX_API_SECRET_KEY=$DCRX_API_SECRET_KEY" --from-literal="DOCKER_REGISTRY_USERNAME=$DOCKER_REGISTRY_USERNAME" --from-literal="DOCKER_REGISTRY_PASSWORD=$DOCKER_REGISTRY_PASSWORD"
```
Then install the provided helm chart:
```
helm install dcrx-api helm/dcrx-api/ --values helm/dcrx-api/values.yml
```
The chart creates three replica dcrx-api pods, a LoadBalancer service, and Nginx ingress to get you up and running in your Kubernetes cluster!
# Notes and Gotchas
1. Does dcrx-api require a volume to run? - Yes. While dcrx-api builds images in-memory, Docker still (frustratingly) requires that a physical file be present in the build directory. This means that, if running dcrx-api in Docker or via Kubernetes, you will need to have a volume with the correct permissions mounted.
2. Does the dcrx-api Docker image require root? - Currently yes. The dcrx-api image is based on the latest version of Docker's `dind` image, which runs as and requires root privlidges. We're working on getting the rootless version of `dind` working though!
Raw data
{
"_id": null,
"home_page": "https://github.com/scorbettUM/dcrx-api",
"name": "dcrx-api",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": "",
"keywords": "pypi,cicd,python,setup,docker,infra,devops,fastapi",
"author": "Sean Corbett",
"author_email": "sean.corbett@umontana.edu",
"download_url": "https://files.pythonhosted.org/packages/8a/e5/094fd972f55b8e441e0945efb1aa5171f4f92fdbb626a2f452bf098b3136/dcrx-api-0.4.1.tar.gz",
"platform": null,
"description": "# dcrx-api\n[](https://pypi.org/project/dcrx-api/)\n[](https://github.com/scorbettUM/dcrx-api/blob/main/LICENSE)\n[](https://github.com/scorbettUM/dcrx-api/blob/main/CODE_OF_CONDUCT.md)\n[](https://pypi.org/project/dcrx-api/)\n\nDCRX-API is a RESTful implementation of the dcrx library, allowing you to create Docker images on request! DCRX-API extends dcrx, providing integrations with the Docker API and dcrx that allow images to be built by making REST requests.\n\nDCRX-API also includes basic JWT authorization and user management (no-RBAC), and is intended to be deployed as an internal-facing service. dcrx also provides OpenAPI documentation to make getting started intuitive, and features integrations with SQLite, Postgres, and MySQL for storing and managing users.\n\n\n# Setup and Installation\n\nDCRX-API is available both as a Docker image and as a PyPi package. For local use, we recommend using PyPi, Python 3.10+, and a virtual environment. To install, run:\n\n```bash\npython -m venv ~/.dcrx && \\\nsource ~/.dcrx/bin/activate && \\\npip install dcrx-api\n```\n\nTo get the Docker image, run:\n\n```\ndocker pull adalundhe/dcrx-api:latest\n```\n\n\n# Getting Started\n\nDCRX-API requires a slew of environmental variables in order to run correctly. These include:\n\n```\nDCRX_API_JOB_TIMEOUT=10m # The maximum amount of time before an image build + push job times out and is cancelled. Default is 10 minutes.\n\nDCRX_API_MAX_MEMORY_PERCENT_USAGE=50 # The percent of system memory DCRX-API and Docker image builds created by DCRX-API can use. If exceeded, new jobs will be rejected and requests will return a 400 error. Default is 50 percent.\n\nDCRX_API_JOB_PRUNE_INTERVAL='1s' # How often dcrx-api checks for done, failed, or cancelled jobs to remove from in-memory storage.\n\nDCRX_API_JOB_MAX_AGE='1m'= # Time before done, failed, or cancelled jobs are pruned from in-memory storage.\n\nDCRX_API_JOB_MAX_PENDING_WAIT=10m # The maximum amount of time a job can remain in PENDING status. Jobs that exceed this threshold are timed out and will fail.\n\nDCRX_API_JOB_MAX_PENDING=100 # The maximum number of pending jobs that a single instance of DCRX-API allows. Default is 100.\n\nDCRX_API_JOB_POOL_SIZE=10 # Maximum number of concurrent builds. Default is 10.\n\nDCRX_API_JOB_WORKERS=4 # Number of workers to use per-job. Default is the number os OS threads.\n\nDCRX_API_TOKEN_EXPIRATION='15m' # Time for JWT authorization token to expire. Default is 15 minutes.\n\nDCRX_API_SECRET_KEY=testingthis # Initial secret used to hash user passwords.\n\nDCRX_API_AUTH_ALGORITHM=HS256 # Algorithm to use for hashing user passwords.\n\nDCRX_API_DATABASE_TRANSACTION_RETRIES=3 # Number of times to retry a failed transaction. Default is 3.\n\nDCRX_API_DATABASE_TYPE=sqlite # Type of database to use. Default is sqlite and options are mysql, asyncpg, and sqlite.\n\nDCRX_API_DATABASE_USER=admin # Username used for database connection authentication\n\nDCRX_API_DATABASE_PASSWORD=test1234 # Password used for database connection authentrication\n\nDCRX_API_DATABASE_NAME=dcrx # Name of database to use for storing users. Default is dcrx.\n\nDCRX_API_DATABASE_URI=sqlite+aiosqlite:///dcrx # URL/URI of SQL database for storing users and job metadata.\n\nDCRX_API_DATABASE_PORT=3369 # Port of SQL database for storing users and job metadata.\n\n```\n\nPrior to starting the server, we recommend seeding the database with an initial user. To do so, run the command:\n\n```bash\ndcrx-api database initialize --username $USERNAME --password $PASSWORD --first-name $FIRST_NAME --last-name $LAST_NAME --email $EMAIL\n```\n\nIf you've chosen to run the Docker image, this step is taken care of for you, with the default user being:\n\n```\nusername: admin\npassword: dcrxadmin1*\nfirst-name: admin\nlast-name: user\nemail: admin@admin.com\n```\n\nNext run:\n\n```bash\ndcrx-api server run\n```\n\nand navigate to `localhost:2277/docs`, where you'll find dcrx-api's OpenAPI documentation.\n\n\n# Authenticating and Building an Image\n\nFrom here, we'll need to authenticate using the initial user we created above. Open the tab for `/users/login`, and for the `username` and `password` fields enter the username and password of the initial user, then click `Execute`. A `200` status response containg the initial user's username and id should return.\n\nBefore creating and pushing any images, we need to create a Registry. Navigate to the `/registry/create` tab. We'll need to provide a `registry_name` (unique shortand reference of our choosing), `registry_uri` (HTTP url to the registry), `registry_user`, and `registry_password:\n\n```json\n{\n \"registry_name\": \"hello-world\",\n \"registry_uri\": \"https://docker.io/v1/testreg/hello-world\",\n \"registry_user\": \"testuser\",\n \"registry_password\": \"testpassword1*\"\n}\n```\n\nSubmit the request. DCRX-API will take the information, encrypt the registry password, and store it.\n\nNext navigate to the `/jobs/images/create` tab. As before, we'll need to fill out some data. Go ahead and copy paste the below into the input for the request body:\n\n```json\n{\n \"name\": \"testreg/hello-world\",\n \"tag\": \"latest\",\n \"registry\": {\n \"registry_name\": \"hello-world\"\n },\n \"layers\": [\n {\n \"layer_type\": \"stage\",\n \"base\": \"python\",\n \"tag\": \"3.11-slim\"\n },\n {\n \"layer_type\": \"entrypoint\",\n \"command\": [\n \"echo\",\n \"Hello world!\"\n ]\n }\n ]\n}\n```\n\nA dcrx-api image build request requires, at-minimum, a `name`, `tag`, `registry_name`, and one or more `layer` objects, which will be processed and build in-order of appearance. Note that each layer object almost exactly corresponds to its `dcrx` call. The above corresponds exactly to:\n\n```python\n\nfrom dcrx import Image\n\nhello_world = Image('hello-world')\n\nhello_world.stage(\n 'python',\n '3.11-slim'\n).entrypoint([\n 'echo',\n 'Hello world!'\n])\n```\n\nGo ahead and submit the request via the `Execute` button. This will start a job to build the specified image and push it to the repository specified in the request.\n\nNote that dcrx-api doesn't wait for the image to be built, instead returning a `JobMetadata` response with a status code of `200`, including the `job_id`. Building images is time-intensive and could potentially bog down a server, so dcrx-api builds and pushes images in the background. We can use the `job_id` of a Job to make further `GET` requests to the `/jobs/images/{job_id}/get` endpoint to monitor and check how our job is progressing. If we need to cancel a job for any reason, we can simple make a `DELETE` request to `/jobs/images/{job_id}/cancel`.\n\n\n# Running the Docker Image\n\nTo run the Docker image, we recommend first creating a `.env` file with the required environment variables noted above. Then run the image:\n\n```bash\ndocker run -p 2277:2277 --env-file .env --privileged dcrx-api:latest\n```\n\n# Deploying via Helm\n\nDCRX-API has a minimal helm chart to help you scale up when ready. To begin, first create a Kuberenetes secret as below:\n\n```\nkubectl create secret generic dcrx-api-secrets --from-literal=\"DCRX_API_SECRET_KEY=$DCRX_API_SECRET_KEY\" --from-literal=\"DOCKER_REGISTRY_USERNAME=$DOCKER_REGISTRY_USERNAME\" --from-literal=\"DOCKER_REGISTRY_PASSWORD=$DOCKER_REGISTRY_PASSWORD\"\n\n```\n\nThen install the provided helm chart:\n\n```\nhelm install dcrx-api helm/dcrx-api/ --values helm/dcrx-api/values.yml\n```\n\nThe chart creates three replica dcrx-api pods, a LoadBalancer service, and Nginx ingress to get you up and running in your Kubernetes cluster!\n\n\n# Notes and Gotchas\n\n1. Does dcrx-api require a volume to run? - Yes. While dcrx-api builds images in-memory, Docker still (frustratingly) requires that a physical file be present in the build directory. This means that, if running dcrx-api in Docker or via Kubernetes, you will need to have a volume with the correct permissions mounted.\n\n2. Does the dcrx-api Docker image require root? - Currently yes. The dcrx-api image is based on the latest version of Docker's `dind` image, which runs as and requires root privlidges. We're working on getting the rootless version of `dind` working though!\n",
"bugtrack_url": null,
"license": "",
"summary": "A RESTful implementation of the DCRX Docker library.",
"version": "0.4.1",
"project_urls": {
"Homepage": "https://github.com/scorbettUM/dcrx-api"
},
"split_keywords": [
"pypi",
"cicd",
"python",
"setup",
"docker",
"infra",
"devops",
"fastapi"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "7f13e8e4a0edcbba35586987628b87db7e81af9b93075f47918d24c623716f34",
"md5": "7e7453b91432d025344ebcb07d93fe57",
"sha256": "8aa035abe2f815f1cde68c4c3401b086125858910fbef43eda187ca2a0b58889"
},
"downloads": -1,
"filename": "dcrx_api-0.4.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "7e7453b91432d025344ebcb07d93fe57",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 56696,
"upload_time": "2023-06-14T00:54:42",
"upload_time_iso_8601": "2023-06-14T00:54:42.287331Z",
"url": "https://files.pythonhosted.org/packages/7f/13/e8e4a0edcbba35586987628b87db7e81af9b93075f47918d24c623716f34/dcrx_api-0.4.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "8ae5094fd972f55b8e441e0945efb1aa5171f4f92fdbb626a2f452bf098b3136",
"md5": "21a2a1d639c28d6c2c7ace224ce7cdc7",
"sha256": "49521f5e76a219365efa2e41c4fce780e95f93976a5c1615c9ac3970ef8a83a9"
},
"downloads": -1,
"filename": "dcrx-api-0.4.1.tar.gz",
"has_sig": false,
"md5_digest": "21a2a1d639c28d6c2c7ace224ce7cdc7",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 36169,
"upload_time": "2023-06-14T00:54:43",
"upload_time_iso_8601": "2023-06-14T00:54:43.892309Z",
"url": "https://files.pythonhosted.org/packages/8a/e5/094fd972f55b8e441e0945efb1aa5171f4f92fdbb626a2f452bf098b3136/dcrx-api-0.4.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-06-14 00:54:43",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "scorbettUM",
"github_project": "dcrx-api",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "dcrx-api"
}
JSON
