How to control Docker for AWS remotely

tl;dr: Docker for AWS enables you to set up a Docker Swarm cluster on AWS without much effort. However, managing the cluster from your local computer is not so easy. Using an SSH tunnel can help.

Thanks to Docker for AWS, setting up a Docker Swarm cluster on AWS can be done in a few clicks in minutes. The provided AWS CloudFormation template first asks for some values and then takes care of everything else automatically.

The requested values include the number of managers and workers, but also the SSH key to be used. Docker for AWS uses this to configure access to the individual EC2 instances. As external access is only allowed to the managers, you must first connect to one of them, before being able to connect to the workers from there.

Log in using SSH

This makes sense from a security perspective, and it works without any problems at first glance. When you log in to one of the managers using SSH and execute the following command, the manager does as expected and starts three instances of Nginx and distributes them in the cluster:

$ docker service create -p 80:80 -p 443:443 --replicas 3 nginx

Things become more difficult if you want to do the same with a private image, because then authentication is missing. Theoretically, you can solve this by logging in to Docker:

$ docker login

Anyway, this doesn't leave a good feeling, as personal access data, which are only needed every now and then, remain on the server.

Next, if you try to build an image based on a Dockerfile, you will notice that the build context is no longer the local machine's one, and hence the build won't work. Of course, this shouldn't come as a surprise, as you have switched to a different system using SSH.

Remote access doesn't work

As a way out, it seems to be advisable to address the manager's Docker endpoint directly from the local machine. The assumption is that all you need to do is to set the local environment variable DOCKER_HOST to the appropriate value. Then, authentication, should be possible by using certificates. But which certificate should be used?

As it turns out, Docker for AWS configures the managers in a way that their respective Docker endpoint can't be addressed over the network at all. Only local connections via /var/run/docker.sock will be accepted. This prevents any remote access.

SSH to the rescue

The solution is to use SSH with local port forwarding to open a tunnel that connects the /var/run/docker.sock path from one of the managers to a port on the local machine. To do this, use the -L flag for SSH. You also have to tell SSH that it should only open the tunnel but not execute a command. This can be done by using the -N flag. Finally, you need to specify the SSH key to use by providing the -i flag:

$ ssh -i aws.pem -N -L 2375:/var/run/docker.sock docker@$IPADDRESS

This opens the connection and allows local access to Docker. To do this, set the environment variable DOCKER_HOST to the value tcp://localhost:2375. All other environment variables that may have been set by docker-machine must be removed using unset:

$ unset DOCKER_TLS_VERIFY
$ unset DOCKER_CERT_PATH
$ unset DOCKER_MACHINE_NAME
$ export DOCKER_HOST=tcp://localhost:2375

For example, if you now execute the command

$ docker ps

the local Docker client accesses one of the Docker managers running on AWS using the SSH tunnel. The tunnel remains open until the ssh command is canceled by pressing <Ctrl>+<C>. Now the docker build command uses the local build context and works as expected.

Authentication for Docker for AWS

However, access to private images is still problematic, for example when using docker service create. Since the manager only assigns the task of downloading the appropriate image to one of the workers instead of doing it on its own, the worker fails because of authentication.

Fortunately, there is the --with-registry-auth flag, which forwards the authentication of the local Docker client to the remote Docker daemon. This way the command docker service create now finally works as expected.

Twitter Facebook LinkedIn

Golo Roden

Founder, CTO, and managing partner

Since we want to deliver elegant yet simple solutions of high quality for complex problems, we care about details with love: things are done when they are done, and we give them the time they need to mature.