TDD but for Networking

Feb 25, 2022

Tired of spending precious time trying to debug why one AWS resource can’t talk to another? Add VPC Reachability Analyzer to your toolkit, mix it up with some Test-Driven Development (TDD) workflow and take back some of your sanity!

Write the Test Create the Path, Watch it Fail…

Create the path to analyze before your networking config is fully configured then see helpful error messages like the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

{
  "destination": {
    "arn": "arn:aws:ec2:xx-xxxx-x:xxxxxxxx:vpc-peering-connection/pcx-xxxxxxxxx",
    "id": "pcx-xxxxxxx"
  },
  "explanationCode": "NO_ROUTE_TO_DESTINATION",
  "routeTable": {
    "arn": "arn:aws:ec2:xx-xxxx-x:xxxxxxxx:route-table/rtb-xxxxxxxxxxx",
    "id": "rtb-xxxxxxxxxxxxxx"
  },
  "vpc": {
    "arn": "arn:aws:ec2:xx-xxxx-x:xxxxxxxx:vpc/vpc-xxxxxxxxxxxx",
    "id": "vpc-xxxxxxxxxxxxxx"
  }
}

Write the Code Configure the Network, Watch it Pass…

Once you’ve fixed the errors above and re-run the analysis, you’ll get the following report:

All is well

From here, you can “refactor” your networking config to make it more restrictive as needed. With every change, you can re-run the analysis (NOTE: $0.10/run) to ensure it hasn’t regressed. If this TDD workflow sounds like it could be useful to you, read on!

A Tale of Two Resources

We’ll start by defining two resources that need to talk to each other. Let’s say we want two instances in two different subnets to talk to each other.

CloudFormation, I know. At the time of writing this article, Terraform didn’t yet support Reachability Analyzer resources so I had to use CF.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95

---
Description: Deceptively simple project

Resources:
  Instance1:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref Instance1AmiId
      KeyName: !Ref Instance1KeyName
      NetworkInterfaces:
        - AssociatePublicIpAddress: "true"
          DeviceIndex: "0"
          GroupSet:
            - !Ref Instance1SecurityGroup
          SubnetId: !Ref Instance1SubnetId
      Tags:
        - Key: Name
          Value: !Join [ '-', [ !Ref AWS::StackName, !Ref Stage, 'Instance1' ] ]

  Instance1SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: !Join [ '-', [ !Ref AWS::StackName, !Ref Stage, 'Instance1-sg' ] ]
      VpcId: !Ref Instance1VpcId
      SecurityGroupEgress:
        - CidrIp: "0.0.0.0/0"
          FromPort: -1
          ToPort: -1
          IpProtocol: -1

  Instance2:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref Instance2AmiId
      KeyName: !Ref Instance2KeyName
      NetworkInterfaces:
        - AssociatePublicIpAddress: "true"
          DeviceIndex: "0"
          GroupSet:
            - !Ref Instance2SecurityGroup
          SubnetId: !Ref Instance2SubnetId
      Tags:
        - Key: Name
          Value: !Join [ '-', [ !Ref AWS::StackName, !Ref Stage, 'Instance2' ] ]

  Instance2SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: !Join [ '-', [ !Ref AWS::StackName, !Ref Stage, 'Instance2-sg' ] ]
      VpcId: !Ref Instance2VpcId
      SecurityGroupEgress:
        - CidrIp: "0.0.0.0/0"
          FromPort: -1
          ToPort: -1
          IpProtocol: -1

  Analyzer:
    Type: AWS::EC2::NetworkInsightsPath
    Properties:
      Source: !Ref Instance1
      Destination: !Ref Instance2
      DestinationPort: 8080
      Protocol: tcp
      Tags:
        - Key: Name
          Value: !Join [ '-', [ !Ref AWS::StackName, !Ref Stage, 'analyzer' ] ]

Parameters:
  Stage:
    Description: The stage (dev, staging, production) of the stack
    Type: String
  Instance1VpcId:
    Description: The VPC where Instance1 is to be deployed
    Type: String
  Instance1AmiId:
    Description: The AMI to use for Instance1
    Type: String
  Instance1KeyName:
    Description: The name (not the ID) of the ssh pubkey to inject into Instance1
    Type: String
  Instance1SubnetId:
    Description: The subnet to place Instance1 in
    Type: String
  Instance2VpcId:
    Description: The VPC where Instance2 is to be deployed
    Type: String
  Instance2AmiId:
    Description: The AMI to use for Instance2
    Type: String
  Instance2KeyName:
    Description: The name (not the ID) of the ssh pubkey to inject into Instance2
    Type: String
  Instance2SubnetId:
    Description: The subnet to place Instance2 in
    Type: String

Let’s create the stack:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

export AWS_REGION=your-preferred-region
export STAGE=dev

export INSTANCE1_VPC_ID=your-vpc-id
export INSTANCE1_AMI_ID=your-ami-id
export INSTANCE1_KEY_NAME=your-key-name
export INSTANCE1_SUBNET_ID=your-subnet-id

export INSTANCE2_VPC_ID=your-vpc-id
export INSTANCE2_AMI_ID=your-ami-id
export INSTANCE2_KEY_NAME=your-key-name
export INSTANCE2_SUBNET_ID=your-subnet-id

aws cloudformation create-stack --stack-name DeceptivelySimpleProject \
    --template-body file://_includes/code-snippets/tdd-networking/s01_first_pass.yml \
    --parameters \
        ParameterKey=Stage,ParameterValue=$STAGE \
        ParameterKey=Instance1VpcId,ParameterValue=$INSTANCE1_VPC_ID \
        ParameterKey=Instance1AmiId,ParameterValue=$INSTANCE1_AMI_ID \
        ParameterKey=Instance1KeyName,ParameterValue=$INSTANCE1_KEY_NAME \
        ParameterKey=Instance1SubnetId,ParameterValue=$INSTANCE1_SUBNET_ID \
        ParameterKey=Instance2VpcId,ParameterValue=$INSTANCE2_VPC_ID \
        ParameterKey=Instance2AmiId,ParameterValue=$INSTANCE2_AMI_ID \
        ParameterKey=Instance2KeyName,ParameterValue=$INSTANCE2_KEY_NAME \
        ParameterKey=Instance2SubnetId,ParameterValue=$INSTANCE2_SUBNET_ID

Head on over to the CloudFormation console to see the progress:

Stack Events

Next, go to the VPC console and to the Reachability Analyzer view:

Menu

Then hit the Analyze Path button

Analyze Path

You’re going to need to hit refresh for the new analysis to show up.

Refresh

Keep hitting that refresh button until the status is no longer Pending.

When you expand that Details section, you’ll get a more detailed explanation of why the instance can’t reach the other.

Fix the Security Group

So let’s fix Instance2’s security group so that it allows traffic from Instance1’s security group:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103

---
Description: Deceptively simple project

Resources:
  Instance1:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref Instance1AmiId
      KeyName: !Ref Instance1KeyName
      NetworkInterfaces:
        - AssociatePublicIpAddress: "true"
          DeviceIndex: "0"
          GroupSet:
            - !Ref Instance1SecurityGroup
          SubnetId: !Ref Instance1SubnetId
      Tags:
        - Key: Name
          Value: !Join [ '-', [ !Ref AWS::StackName, !Ref Stage, 'Instance1' ] ]

  Instance1SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: !Join [ '-', [ !Ref AWS::StackName, !Ref Stage, 'Instance1-sg' ] ]
      VpcId: !Ref Instance1VpcId
      SecurityGroupEgress:
        - CidrIp: "0.0.0.0/0"
          FromPort: -1
          ToPort: -1
          IpProtocol: -1

  Instance2:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref Instance2AmiId
      KeyName: !Ref Instance2KeyName
      NetworkInterfaces:
        - AssociatePublicIpAddress: "true"
          DeviceIndex: "0"
          GroupSet:
            - !Ref Instance2SecurityGroup
          SubnetId: !Ref Instance2SubnetId
      Tags:
        - Key: Name
          Value: !Join [ '-', [ !Ref AWS::StackName, !Ref Stage, 'Instance2' ] ]

  Instance2SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: !Join [ '-', [ !Ref AWS::StackName, !Ref Stage, 'Instance2-sg' ] ]
      VpcId: !Ref Instance2VpcId
      SecurityGroupEgress:
        - CidrIp: "0.0.0.0/0"
          FromPort: -1
          ToPort: -1
          IpProtocol: -1
      SecurityGroupIngress:
      #
      # Add this missing security group rule
      #
      - IpProtocol: tcp
        FromPort: 8080
        ToPort: 8080
        SourceSecurityGroupId: !GetAtt Instance1SecurityGroup.GroupId

  Analyzer:
    Type: AWS::EC2::NetworkInsightsPath
    Properties:
      Source: !Ref Instance1
      Destination: !Ref Instance2
      DestinationPort: 8080
      Protocol: tcp
      Tags:
        - Key: Name
          Value: !Join [ '-', [ !Ref AWS::StackName, !Ref Stage, 'analyzer' ] ]

Parameters:
  Stage:
    Description: The stage (dev, staging, production) of the stack
    Type: String
  Instance1VpcId:
    Description: The VPC where Instance1 is to be deployed
    Type: String
  Instance1AmiId:
    Description: The AMI to use for Instance1
    Type: String
  Instance1KeyName:
    Description: The name (not the ID) of the ssh pubkey to inject into Instance1
    Type: String
  Instance1SubnetId:
    Description: The subnet to place Instance1 in
    Type: String
  Instance2VpcId:
    Description: The VPC where Instance2 is to be deployed
    Type: String
  Instance2AmiId:
    Description: The AMI to use for Instance2
    Type: String
  Instance2KeyName:
    Description: The name (not the ID) of the ssh pubkey to inject into Instance2
    Type: String
  Instance2SubnetId:
    Description: The subnet to place Instance2 in
    Type: String

Take note of lines 60 to 63 where we add an ingress rule allowing incoming traffic from Instance1’s security group.

Watch it Pass!

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

aws cloudformation update-stack --stack-name DeceptivelySimpleProject \
    --template-body file://_includes/code-snippets/tdd-networking/s02_fix_sg.yml \
    --parameters \
        ParameterKey=Stage,ParameterValue=$STAGE \
        ParameterKey=Instance1VpcId,ParameterValue=$INSTANCE1_VPC_ID \
        ParameterKey=Instance1AmiId,ParameterValue=$INSTANCE1_AMI_ID \
        ParameterKey=Instance1KeyName,ParameterValue=$INSTANCE1_KEY_NAME \
        ParameterKey=Instance1SubnetId,ParameterValue=$INSTANCE1_SUBNET_ID \
        ParameterKey=Instance2VpcId,ParameterValue=$INSTANCE2_VPC_ID \
        ParameterKey=Instance2AmiId,ParameterValue=$INSTANCE2_AMI_ID \
        ParameterKey=Instance2KeyName,ParameterValue=$INSTANCE2_KEY_NAME \
        ParameterKey=Instance2SubnetId,ParameterValue=$INSTANCE2_SUBNET_ID

After you update the stack and analyze the path again, you should see another analysis in the Analysis section, this time showing that the connection is reachable:

And if you look at the details further down the page, you’ll get a listing of all the various resources in the path.

At this point, you’re basically done but it would also be prudent to see which parts of this path can be adjusted to make it more secure. The process will be pretty much the same: make minor adjustments, re-run the analysis, see it fail, make more adjustments. The number of iterations you take will depend on your budget because each analysis will cost you $0.10 as per Amazon’s pricing page. I’ll leave that exercise up to you.

Experiment with the above stack by changing the VPC and subnet of the instances then use the Reachability Analyzer to guide you in configuring your network properly. Happy hunting!