Tired of spending precious time trying to debug why one AWS resource can’t talk to another? Add VPC Reachability Analyzer to your toolkit, mix it up with some Test-Driven Development (TDD) workflow and take back some of your sanity!
Write the Test Create the Path, Watch it Fail…
Create the path to analyze before your networking config is fully configured then see helpful error messages like the following:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{
"destination": {
"arn": "arn:aws:ec2:xx-xxxx-x:xxxxxxxx:vpc-peering-connection/pcx-xxxxxxxxx",
"id": "pcx-xxxxxxx"
},
"explanationCode": "NO_ROUTE_TO_DESTINATION",
"routeTable": {
"arn": "arn:aws:ec2:xx-xxxx-x:xxxxxxxx:route-table/rtb-xxxxxxxxxxx",
"id": "rtb-xxxxxxxxxxxxxx"
},
"vpc": {
"arn": "arn:aws:ec2:xx-xxxx-x:xxxxxxxx:vpc/vpc-xxxxxxxxxxxx",
"id": "vpc-xxxxxxxxxxxxxx"
}
}
Write the Code Configure the Network, Watch it Pass…
Once you’ve fixed the errors above and re-run the analysis, you’ll get the following report:
From here, you can “refactor” your networking config to make it more restrictive as needed. With every change, you can re-run the analysis (NOTE: $0.10/run) to ensure it hasn’t regressed. If this TDD workflow sounds like it could be useful to you, read on!
A Tale of Two Resources
We’ll start by defining two resources that need to talk to each other. Let’s say we want two instances in two different subnets to talk to each other.
CloudFormation, I know. At the time of writing this article, Terraform didn’t yet support Reachability Analyzer resources so I had to use CF.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
---
Description: Deceptively simple project
Resources:
Instance1:
Type: AWS::EC2::Instance
Properties:
ImageId: !Ref Instance1AmiId
KeyName: !Ref Instance1KeyName
NetworkInterfaces:
- AssociatePublicIpAddress: "true"
DeviceIndex: "0"
GroupSet:
- !Ref Instance1SecurityGroup
SubnetId: !Ref Instance1SubnetId
Tags:
- Key: Name
Value: !Join [ '-', [ !Ref AWS::StackName, !Ref Stage, 'Instance1' ] ]
Instance1SecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: !Join [ '-', [ !Ref AWS::StackName, !Ref Stage, 'Instance1-sg' ] ]
VpcId: !Ref Instance1VpcId
SecurityGroupEgress:
- CidrIp: "0.0.0.0/0"
FromPort: -1
ToPort: -1
IpProtocol: -1
Instance2:
Type: AWS::EC2::Instance
Properties:
ImageId: !Ref Instance2AmiId
KeyName: !Ref Instance2KeyName
NetworkInterfaces:
- AssociatePublicIpAddress: "true"
DeviceIndex: "0"
GroupSet:
- !Ref Instance2SecurityGroup
SubnetId: !Ref Instance2SubnetId
Tags:
- Key: Name
Value: !Join [ '-', [ !Ref AWS::StackName, !Ref Stage, 'Instance2' ] ]
Instance2SecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: !Join [ '-', [ !Ref AWS::StackName, !Ref Stage, 'Instance2-sg' ] ]
VpcId: !Ref Instance2VpcId
SecurityGroupEgress:
- CidrIp: "0.0.0.0/0"
FromPort: -1
ToPort: -1
IpProtocol: -1
Analyzer:
Type: AWS::EC2::NetworkInsightsPath
Properties:
Source: !Ref Instance1
Destination: !Ref Instance2
DestinationPort: 8080
Protocol: tcp
Tags:
- Key: Name
Value: !Join [ '-', [ !Ref AWS::StackName, !Ref Stage, 'analyzer' ] ]
Parameters:
Stage:
Description: The stage (dev, staging, production) of the stack
Type: String
Instance1VpcId:
Description: The VPC where Instance1 is to be deployed
Type: String
Instance1AmiId:
Description: The AMI to use for Instance1
Type: String
Instance1KeyName:
Description: The name (not the ID) of the ssh pubkey to inject into Instance1
Type: String
Instance1SubnetId:
Description: The subnet to place Instance1 in
Type: String
Instance2VpcId:
Description: The VPC where Instance2 is to be deployed
Type: String
Instance2AmiId:
Description: The AMI to use for Instance2
Type: String
Instance2KeyName:
Description: The name (not the ID) of the ssh pubkey to inject into Instance2
Type: String
Instance2SubnetId:
Description: The subnet to place Instance2 in
Type: String
Let’s create the stack:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
export AWS_REGION=your-preferred-region
export STAGE=dev
export INSTANCE1_VPC_ID=your-vpc-id
export INSTANCE1_AMI_ID=your-ami-id
export INSTANCE1_KEY_NAME=your-key-name
export INSTANCE1_SUBNET_ID=your-subnet-id
export INSTANCE2_VPC_ID=your-vpc-id
export INSTANCE2_AMI_ID=your-ami-id
export INSTANCE2_KEY_NAME=your-key-name
export INSTANCE2_SUBNET_ID=your-subnet-id
aws cloudformation create-stack --stack-name DeceptivelySimpleProject \
--template-body file://_includes/code-snippets/tdd-networking/s01_first_pass.yml \
--parameters \
ParameterKey=Stage,ParameterValue=$STAGE \
ParameterKey=Instance1VpcId,ParameterValue=$INSTANCE1_VPC_ID \
ParameterKey=Instance1AmiId,ParameterValue=$INSTANCE1_AMI_ID \
ParameterKey=Instance1KeyName,ParameterValue=$INSTANCE1_KEY_NAME \
ParameterKey=Instance1SubnetId,ParameterValue=$INSTANCE1_SUBNET_ID \
ParameterKey=Instance2VpcId,ParameterValue=$INSTANCE2_VPC_ID \
ParameterKey=Instance2AmiId,ParameterValue=$INSTANCE2_AMI_ID \
ParameterKey=Instance2KeyName,ParameterValue=$INSTANCE2_KEY_NAME \
ParameterKey=Instance2SubnetId,ParameterValue=$INSTANCE2_SUBNET_ID
Head on over to the CloudFormation console to see the progress:
Next, go to the VPC console and to the Reachability Analyzer view:
Then hit the Analyze Path button
You’re going to need to hit refresh for the new analysis to show up.
Keep hitting that refresh button until the status is no longer Pending.
When you expand that Details section, you’ll get a more detailed explanation of why the instance can’t reach the other.
Fix the Security Group
So let’s fix Instance2’s security group so that it allows traffic from Instance1’s security group:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
---
Description: Deceptively simple project
Resources:
Instance1:
Type: AWS::EC2::Instance
Properties:
ImageId: !Ref Instance1AmiId
KeyName: !Ref Instance1KeyName
NetworkInterfaces:
- AssociatePublicIpAddress: "true"
DeviceIndex: "0"
GroupSet:
- !Ref Instance1SecurityGroup
SubnetId: !Ref Instance1SubnetId
Tags:
- Key: Name
Value: !Join [ '-', [ !Ref AWS::StackName, !Ref Stage, 'Instance1' ] ]
Instance1SecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: !Join [ '-', [ !Ref AWS::StackName, !Ref Stage, 'Instance1-sg' ] ]
VpcId: !Ref Instance1VpcId
SecurityGroupEgress:
- CidrIp: "0.0.0.0/0"
FromPort: -1
ToPort: -1
IpProtocol: -1
Instance2:
Type: AWS::EC2::Instance
Properties:
ImageId: !Ref Instance2AmiId
KeyName: !Ref Instance2KeyName
NetworkInterfaces:
- AssociatePublicIpAddress: "true"
DeviceIndex: "0"
GroupSet:
- !Ref Instance2SecurityGroup
SubnetId: !Ref Instance2SubnetId
Tags:
- Key: Name
Value: !Join [ '-', [ !Ref AWS::StackName, !Ref Stage, 'Instance2' ] ]
Instance2SecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: !Join [ '-', [ !Ref AWS::StackName, !Ref Stage, 'Instance2-sg' ] ]
VpcId: !Ref Instance2VpcId
SecurityGroupEgress:
- CidrIp: "0.0.0.0/0"
FromPort: -1
ToPort: -1
IpProtocol: -1
SecurityGroupIngress:
#
# Add this missing security group rule
#
- IpProtocol: tcp
FromPort: 8080
ToPort: 8080
SourceSecurityGroupId: !GetAtt Instance1SecurityGroup.GroupId
Analyzer:
Type: AWS::EC2::NetworkInsightsPath
Properties:
Source: !Ref Instance1
Destination: !Ref Instance2
DestinationPort: 8080
Protocol: tcp
Tags:
- Key: Name
Value: !Join [ '-', [ !Ref AWS::StackName, !Ref Stage, 'analyzer' ] ]
Parameters:
Stage:
Description: The stage (dev, staging, production) of the stack
Type: String
Instance1VpcId:
Description: The VPC where Instance1 is to be deployed
Type: String
Instance1AmiId:
Description: The AMI to use for Instance1
Type: String
Instance1KeyName:
Description: The name (not the ID) of the ssh pubkey to inject into Instance1
Type: String
Instance1SubnetId:
Description: The subnet to place Instance1 in
Type: String
Instance2VpcId:
Description: The VPC where Instance2 is to be deployed
Type: String
Instance2AmiId:
Description: The AMI to use for Instance2
Type: String
Instance2KeyName:
Description: The name (not the ID) of the ssh pubkey to inject into Instance2
Type: String
Instance2SubnetId:
Description: The subnet to place Instance2 in
Type: String
Take note of lines 60 to 63 where we add an ingress rule allowing incoming traffic from Instance1’s security group.
Watch it Pass!
1
2
3
4
5
6
7
8
9
10
11
12
aws cloudformation update-stack --stack-name DeceptivelySimpleProject \
--template-body file://_includes/code-snippets/tdd-networking/s02_fix_sg.yml \
--parameters \
ParameterKey=Stage,ParameterValue=$STAGE \
ParameterKey=Instance1VpcId,ParameterValue=$INSTANCE1_VPC_ID \
ParameterKey=Instance1AmiId,ParameterValue=$INSTANCE1_AMI_ID \
ParameterKey=Instance1KeyName,ParameterValue=$INSTANCE1_KEY_NAME \
ParameterKey=Instance1SubnetId,ParameterValue=$INSTANCE1_SUBNET_ID \
ParameterKey=Instance2VpcId,ParameterValue=$INSTANCE2_VPC_ID \
ParameterKey=Instance2AmiId,ParameterValue=$INSTANCE2_AMI_ID \
ParameterKey=Instance2KeyName,ParameterValue=$INSTANCE2_KEY_NAME \
ParameterKey=Instance2SubnetId,ParameterValue=$INSTANCE2_SUBNET_ID
After you update the stack and analyze the path again, you should see another analysis in the Analysis section, this time showing that the connection is reachable:
And if you look at the details further down the page, you’ll get a listing of all the various resources in the path.
At this point, you’re basically done but it would also be prudent to see which parts of this path can be adjusted to make it more secure. The process will be pretty much the same: make minor adjustments, re-run the analysis, see it fail, make more adjustments. The number of iterations you take will depend on your budget because each analysis will cost you $0.10 as per Amazon’s pricing page. I’ll leave that exercise up to you.
Experiment with the above stack by changing the VPC and subnet of the instances then use the Reachability Analyzer to guide you in configuring your network properly. Happy hunting!