AWS Container Services on Private Subnets Tutorial

Introduction

For various reasons, you may wish to use Amazon's EC2 Container Service to create and manage clusters on a private subnet. The method to do this is not immediately obvious, because instances managed by the ECS service need to be able to reach ECS management.

First a Few Definitions

ECS: Amazon's EC2 Container Service - basically it can be used to create and manage clusters (auto-scaling groups) of Docker instances along with tasks to run on those instances.
Internet Gateway: This gateway provides hosts access to and from the Internet. However it is less full featured than you may be thinking, it's not really similar to an internet gateway in a classic networking sense. In practice you can think of it as simply maintaing classic, static 1-to-1 NAT tables - mapping your private IPs to your public IPs. Note - this does nothing for any instances you have which do not have a public IP (auto-assigned or elastic).
Autoscaling Group: This is the basic building block Amazon provides for you to setup clusters of servers / instances. The most common use case is a set of servers providing content or services sitting behind a load balancer. Autoscaling Groups can be created and managed manually or by other AWS services (such as ECS).
NAT Gateway: The NAT Gateway allows outbound (IPV4) Internet access from hosts without public IP addresses or which reside on a Private Subnet. It does this by doing NAT/PAT on the hosts sending traffic to the internet.
Public and Private Subnets: The distinction between public and private subnets is in some ways left to the user. In general, however, public subnets are considered to be those which are associated with a routing table that has a default route to an Internet Gateway.

The Problem

In a nutshell - when you use ECS to create a new cluster, it assumes this cluster will be on a Public Subnet. If you choose a Private subnet as the target network for deployment, the resulting instances won't register back to ECS (ECS will continue to indicate zero running instances). This is because the container instances need to establish a connection back to the ECS management service, and this connection is over the Internet. On a private subnet they have no way to do this.

The Solution

There is a way around this catch - you can use AWS's NAT Gateway. This will allow hosts on your private subnets outbound Internet access, so that they can contact the ECS management service.

You'll need a few key ingredients to make this work:

1. You need a NAT gateway (shown in the upper left)

2. You need a route table for the private subnet that has that NAT gateway as it's default (0/0) route.

3. You need to make sure outbound calls from this subnet aren't blocked by a security group

A Few Additional Notes

• The NAT Gateway must be on the Public Subnet. This is because it's going to be forwarding traffic to the Internet (and the Private Subnet doesn't have access to the Internet).

• The Load Balancer shown here is load balancing and forwarding traffic to hosts on another network (a Private Network in this case). Occasionally folks have the impression that the target hosts must reside on the same network as the load balancer, this is not the case.