## Remote Docker on AWS for local development
[](https://badge.fury.io/py/remote-docker-aws)
Use docker to develop services, but without the overhead of running docker on your machine! This is a development tool that you should use if your machine is low performance, or if you are running many docker services.
### Why is this useful?
Frees up your local machine for useful tasks
such as running your code editor, browser, and email, leaving running Docker to a dedicated server instance.
The result is that your local machine functions faster, uses up less disk space, and consumes less power.
MacOS users will also see noticeable speed improvements since Docker on Linux (which is
what the remote hosts runs) is much more performant.
The downsides:
- SSH tunnel communication is slower than local communication. However using an AWS region with low ping makes the latency unnoticeable. Find the region fastest for you using [this site](https://www.cloudping.info/)
- Some more setup required to get everything configured properly and running (tunneling ports, syncing file changes)
- Running the ec2 instance incurs an additional cost over running locally, although a t3.medium instance in Canada only costs just under 5 cents/hour
How it works: two processes are run, a sync and a tunnel process.
- The sync process keeps local and remote files in sync so that the docker process run remotely can use docker volumes transparently
- The tunnel process forwards ports needed so your local system can communicate with docker, plus additional ports as required, such as port 443 for browser communication
## Setup
1. First login to your AWS account and [create access keys to access AWS through the CLI](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html)
You will need the following IAM policies:
- AmazonEC2FullAccess
- AWSCloudFormationFullAccess
And now in your terminal:
```bash
# Replace josh with your name
# You will need to setup an AWS account if you don't have one
# and create access key credentials
aws configure --profile josh
export AWS_PROFILE=josh
```
1. Install pre-requisites
Have [Homebrew](https://brew.sh/) (Available on both macOS and Linux now!)
Have [pipx](https://github.com/pipxproject/pipx)
```bash
pipx install remote-docker-aws
pipx install unison-gitignore
# Install unison sync utility
brew install unison
# Install file-watcher driver for unison
# On MacOS:
brew install autozimu/homebrew-formulas/unison-fsmonitor
# Or, on Linux since the above formula doesn't work:
brew install eugenmayer/dockersync/unox
```
1. Generate and upload a keypair to AWS
```bash
# Note: bash users can use `rd` instead of `remote-docker-aws`. zsh users cannot since zsh aliases `rd` to `rmdir` (!)
remote-docker-aws create-keypair
```
1. Create the ec2 instance
```bash
remote-docker-aws create
```
## Daily Running
1. Start the remote-docker ec2 instance
```bash
remote-docker-aws start
```
This will automatically switch the docker context for you. If you want to switch
back to the default agent run `docker context use default`
1. In one terminal start the tunnel so that the ports you need to connect to are exposed
```bash
remote-docker-aws tunnel
# Usually it's preferable just to forward the ports to same port
# so eg. with mysql on docker exposing port 3306 and nginx on docker exposing port 80:
remote-docker-aws tunnel -l 80:80 -l 3306:3306
# You can forward remote ports as needed with the "-r" option:
# which can be used so the docker instance can access services running locally (eg. webpack)
remote-docker-aws tunnel -r 8080:8080
```
1. In another terminal sync file changes to the remote instance:
```bash
# Add any more paths you need to sync here, or add them to the config file
# You will need to sync directories that are mounted as volumes by docker
remote-docker-aws sync ~/blog
# If watched directories are supplied in ~/.remote-docker.config.json
# then simply call:
remote-docker-aws sync
```
1. Develop and code! All services should be accessible and usable as usual (eg: `docker ps`, `docker-compose up`, etc.)
as long as you are running `remote-docker-aws tunnel` and are forwarding the ports you need
1. When you're done for the day don't forget to stop the instance to save money:
```bash
remote-docker-aws stop
```
## Config File
Looks for a config file at the path `~/.remote-docker.config.json` by default,
which can be overriden by passing `--config-path`. The config file is not necessary
and CLI usage is possible without it as long as AWS_PROFILE and AWS_REGION environment variables are set
An example `.remote-docker.config.json` file:
```json
{
"key_path": "~/.ssh/id_rsa_remote_docker",
"sync_ignore_patterns_git": [
"**/*.idea/",
"**/*.git/",
"**/*~",
"**/*.sw[pon]"
],
"profiles": {
"blog": {
"sync_ignore_patterns_git": [
"**/notes/"
],
"remote_port_forwards": {
"local-webpack-app": {"8080": "8080"}
},
"local_port_forwards": {
"blog_app": {"443": "443", "80": "8000"},
"blog_db": {"3306": "3306"}
},
"watched_directories": [
"~/.aws",
"~/blog"
]
}
},
"default_profile": "blog"
}
```
```bash
Usage: remote-docker-aws [OPTIONS] COMMAND [ARGS]...
Options:
--profile TEXT Name of the remote-docker-aws profile to use
--config-path TEXT Path of the remote-docker-aws JSON config
```
The current configurable values are:
#### `aws_region` (takes precedence over `AWS_REGION` and `.aws/config`)
- The region to create the instance in
#### `instance_type`
- Type of ec2 instance, defaults to: `t3.medium`
#### `key_path`
- defaults to: `~/.ssh/id_rsa_remote_docker`
#### `local_port_forwards`
- defaults to: `{}`
- Object containing label -> port mapping objects for opening the ports on the remote host.
A mapping of `"local_port_forwards": {"my_app": {"80": "8080"}}` will open port 80 on your local machine
and point it to port 8080 of the remote-docker instance (which ostensibly a container is listening on).
The name doesn't do anything except help legibility.
#### `remote_port_forwards`
- defaults to: `{}`
- Similar to `local_port_forwards` except will open the port on the remote instance.
This is useful to have frontend webpack apps accessible on the remote host
#### `sync_ignore_patterns_git`
- defaults to: `[]`
- use `.gitignore` syntax, and make sure to use the directory wildcard as needed
#### `user_id`
- defaults to `None`
- Used to uniquely identify the instance, this is useful if multiple remote-docker agents
will be created in the same AWS account
#### `watched_directories`
- defaults to: `[]`
- list of paths to watch by `remote-docker-aws sync`
#### `volume_size`
- defaults to: `30` (GB)
- Size of the ec2 volume.
---
Profiles are a way to organize and override settings for different projects.
Values nested in a profile override the values defined outside a profile,
except for lists and dictionaries which are merged with the values outside the profile
## Cost
A t3.medium instance on ca-central-1 currently costs $0.046 /hour. [See current prices](https://aws.amazon.com/ec2/pricing/on-demand/)
Nothing else used should incur any cost with reasonable usage
## Notes
- See `remote-docker-aws --help` for more information on the commands available
- The unison version running on the server and running locally have to
match. If one of them updates to a newer version, you should update the other.
This can be done locally via `brew upgrade unison`, and to update the remote unison version:
`rd ssh` then `brew upgrade unison`
Raw data
{
"_id": null,
"home_page": "https://github.com/lime-green/remote-docker-aws",
"name": "remote-docker-aws",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6, <4",
"maintainer_email": "",
"keywords": "docker,aws,development,macos,linux",
"author": "Josh DM",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/27/36/667909622eb3fd3bc9180db2226bc13b630ce86a60a30f2f524e9bd5503a/remote-docker-aws-3.0.1.tar.gz",
"platform": null,
"description": "## Remote Docker on AWS for local development\n[](https://badge.fury.io/py/remote-docker-aws)\n\n\nUse docker to develop services, but without the overhead of running docker on your machine! This is a development tool that you should use if your machine is low performance, or if you are running many docker services.\n\n### Why is this useful?\n\nFrees up your local machine for useful tasks\nsuch as running your code editor, browser, and email, leaving running Docker to a dedicated server instance.\nThe result is that your local machine functions faster, uses up less disk space, and consumes less power.\nMacOS users will also see noticeable speed improvements since Docker on Linux (which is\nwhat the remote hosts runs) is much more performant.\n\nThe downsides:\n- SSH tunnel communication is slower than local communication. However using an AWS region with low ping makes the latency unnoticeable. Find the region fastest for you using [this site](https://www.cloudping.info/)\n- Some more setup required to get everything configured properly and running (tunneling ports, syncing file changes)\n- Running the ec2 instance incurs an additional cost over running locally, although a t3.medium instance in Canada only costs just under 5 cents/hour\n\nHow it works: two processes are run, a sync and a tunnel process. \n- The sync process keeps local and remote files in sync so that the docker process run remotely can use docker volumes transparently\n- The tunnel process forwards ports needed so your local system can communicate with docker, plus additional ports as required, such as port 443 for browser communication\n\n## Setup\n1. First login to your AWS account and [create access keys to access AWS through the CLI](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html)\n\n You will need the following IAM policies:\n - AmazonEC2FullAccess\n - AWSCloudFormationFullAccess\n\n And now in your terminal:\n\n ```bash\n # Replace josh with your name\n # You will need to setup an AWS account if you don't have one\n # and create access key credentials\n\n aws configure --profile josh\n export AWS_PROFILE=josh\n ```\n\n1. Install pre-requisites\n\n Have [Homebrew](https://brew.sh/) (Available on both macOS and Linux now!)\n\n Have [pipx](https://github.com/pipxproject/pipx)\n\n ```bash\n pipx install remote-docker-aws\n pipx install unison-gitignore\n\n # Install unison sync utility\n brew install unison\n\n # Install file-watcher driver for unison\n # On MacOS:\n brew install autozimu/homebrew-formulas/unison-fsmonitor\n\n # Or, on Linux since the above formula doesn't work:\n brew install eugenmayer/dockersync/unox\n ```\n\n1. Generate and upload a keypair to AWS\n\n ```bash\n # Note: bash users can use `rd` instead of `remote-docker-aws`. zsh users cannot since zsh aliases `rd` to `rmdir` (!)\n remote-docker-aws create-keypair\n ```\n\n1. Create the ec2 instance\n\n ```bash\n remote-docker-aws create\n ```\n\n## Daily Running\n\n1. Start the remote-docker ec2 instance\n ```bash\n remote-docker-aws start\n ```\n This will automatically switch the docker context for you. If you want to switch\n back to the default agent run `docker context use default`\n\n1. In one terminal start the tunnel so that the ports you need to connect to are exposed\n ```bash\n remote-docker-aws tunnel\n\n # Usually it's preferable just to forward the ports to same port\n # so eg. with mysql on docker exposing port 3306 and nginx on docker exposing port 80:\n remote-docker-aws tunnel -l 80:80 -l 3306:3306\n\n # You can forward remote ports as needed with the \"-r\" option:\n # which can be used so the docker instance can access services running locally (eg. webpack)\n remote-docker-aws tunnel -r 8080:8080\n ```\n\n1. In another terminal sync file changes to the remote instance:\n ```bash\n # Add any more paths you need to sync here, or add them to the config file\n # You will need to sync directories that are mounted as volumes by docker\n remote-docker-aws sync ~/blog\n\n # If watched directories are supplied in ~/.remote-docker.config.json\n # then simply call:\n remote-docker-aws sync\n ```\n\n1. Develop and code! All services should be accessible and usable as usual (eg: `docker ps`, `docker-compose up`, etc.)\nas long as you are running `remote-docker-aws tunnel` and are forwarding the ports you need\n\n1. When you're done for the day don't forget to stop the instance to save money:\n ```bash\n remote-docker-aws stop\n ```\n\n## Config File\nLooks for a config file at the path `~/.remote-docker.config.json` by default,\nwhich can be overriden by passing `--config-path`. The config file is not necessary\nand CLI usage is possible without it as long as AWS_PROFILE and AWS_REGION environment variables are set\n\nAn example `.remote-docker.config.json` file:\n```json\n{\n \"key_path\": \"~/.ssh/id_rsa_remote_docker\",\n \"sync_ignore_patterns_git\": [\n \"**/*.idea/\",\n \"**/*.git/\",\n \"**/*~\",\n \"**/*.sw[pon]\"\n ],\n \"profiles\": {\n \"blog\": {\n \"sync_ignore_patterns_git\": [\n \"**/notes/\"\n ],\n \"remote_port_forwards\": {\n \"local-webpack-app\": {\"8080\": \"8080\"}\n },\n \"local_port_forwards\": {\n \"blog_app\": {\"443\": \"443\", \"80\": \"8000\"},\n \"blog_db\": {\"3306\": \"3306\"}\n },\n \"watched_directories\": [\n \"~/.aws\",\n \"~/blog\"\n ]\n }\n },\n \"default_profile\": \"blog\"\n}\n```\n\n```bash\nUsage: remote-docker-aws [OPTIONS] COMMAND [ARGS]...\n\nOptions:\n --profile TEXT Name of the remote-docker-aws profile to use\n --config-path TEXT Path of the remote-docker-aws JSON config\n```\n\nThe current configurable values are:\n#### `aws_region` (takes precedence over `AWS_REGION` and `.aws/config`)\n- The region to create the instance in\n\n#### `instance_type`\n- Type of ec2 instance, defaults to: `t3.medium`\n\n#### `key_path`\n - defaults to: `~/.ssh/id_rsa_remote_docker`\n\n#### `local_port_forwards`\n - defaults to: `{}`\n - Object containing label -> port mapping objects for opening the ports on the remote host.\n A mapping of `\"local_port_forwards\": {\"my_app\": {\"80\": \"8080\"}}` will open port 80 on your local machine\n and point it to port 8080 of the remote-docker instance (which ostensibly a container is listening on).\n The name doesn't do anything except help legibility.\n\n#### `remote_port_forwards`\n - defaults to: `{}`\n - Similar to `local_port_forwards` except will open the port on the remote instance.\n\n This is useful to have frontend webpack apps accessible on the remote host\n\n#### `sync_ignore_patterns_git`\n - defaults to: `[]`\n - use `.gitignore` syntax, and make sure to use the directory wildcard as needed\n\n#### `user_id`\n - defaults to `None`\n - Used to uniquely identify the instance, this is useful if multiple remote-docker agents\n will be created in the same AWS account\n\n#### `watched_directories`\n - defaults to: `[]`\n - list of paths to watch by `remote-docker-aws sync`\n\n#### `volume_size`\n - defaults to: `30` (GB)\n - Size of the ec2 volume.\n\n---\n\nProfiles are a way to organize and override settings for different projects.\nValues nested in a profile override the values defined outside a profile,\nexcept for lists and dictionaries which are merged with the values outside the profile\n\n\n## Cost\nA t3.medium instance on ca-central-1 currently costs $0.046 /hour. [See current prices](https://aws.amazon.com/ec2/pricing/on-demand/)\n\nNothing else used should incur any cost with reasonable usage\n\n## Notes\n- See `remote-docker-aws --help` for more information on the commands available\n- The unison version running on the server and running locally have to\nmatch. If one of them updates to a newer version, you should update the other.\nThis can be done locally via `brew upgrade unison`, and to update the remote unison version:\n`rd ssh` then `brew upgrade unison`\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Client to control a remote-docker agent",
"version": "3.0.1",
"split_keywords": [
"docker",
"aws",
"development",
"macos",
"linux"
],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "350e805e195e941523c1800f138a3d00",
"sha256": "aa1638a527776b853c3f2dfd55a643e2266fd99e9206369c5560b859933be232"
},
"downloads": -1,
"filename": "remote_docker_aws-3.0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "350e805e195e941523c1800f138a3d00",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6, <4",
"size": 18583,
"upload_time": "2023-01-01T17:09:07",
"upload_time_iso_8601": "2023-01-01T17:09:07.536801Z",
"url": "https://files.pythonhosted.org/packages/9f/b7/d66f0e4e7237f13b4b3350ac0313557020edaa00777d5972207a55315bbd/remote_docker_aws-3.0.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"md5": "d45f281f0ee318e4cf0cfddea7d369fe",
"sha256": "64469d58548aaf90091380efde55b2e7808ad4627525f26e8e0b4320575f3062"
},
"downloads": -1,
"filename": "remote-docker-aws-3.0.1.tar.gz",
"has_sig": false,
"md5_digest": "d45f281f0ee318e4cf0cfddea7d369fe",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6, <4",
"size": 25177,
"upload_time": "2023-01-01T17:09:08",
"upload_time_iso_8601": "2023-01-01T17:09:08.773663Z",
"url": "https://files.pythonhosted.org/packages/27/36/667909622eb3fd3bc9180db2226bc13b630ce86a60a30f2f524e9bd5503a/remote-docker-aws-3.0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-01-01 17:09:08",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "lime-green",
"github_project": "remote-docker-aws",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "remote-docker-aws"
}
JSON
