AWS Traffic Mirroring
Earlier this year, AWS made available a new feature named "Traffic Mirroring" to their customers. This feature is available on any compute workload that is built using the new AWS Nitro System (link contains supported EC2 instance types). It is a really interesting feature, and as such I've wanted to try it out, as network traffic inspection and collection is notoriously challenging in the AWS cloud environment.
The following blog post introduces us to traffic mirroring with an example of setting it up in the AWS console:
First thing, it is important to identify an instance that will be best suited for the job. As a start, we will try to build a system that will capture traffic from one or more traffic sources. For this, a storage-optimized EC2 instance, the I3en, will be suitable for two reasons:
- Enhanced networking offers an adapter that can scale from 25Gbps to 100GBps
- Large high-speed local ephemral hardware storage that provides high-speed, multi-terabyte landing zone for pcap archives
The cost involved with the smallest version of this instance type can still be over $100 per month. Therefore, for experimentation, a T3 instance will allow for use of the feature, with significantly lower cost. That said, capture with a T3 instance is not recommended for production, if a lot of bandwidth is being monitored.
This feature will require the use of the newer Elastic Network Adapter. Support for this exists in the latest AMI Linux and FreeBSD images, but not in older ones. Likewise, only the newer "Nitro" instance types can support traffic mirroring, so if monitroing an existing instance is desired, the instance will need to be upgraded if it isn't on a newer instance type.
One thing to keep in mind: This will not be compatible with any "Free Tier" service levels
Additionally, one or more traffic generators will also need to be set up. In this example, we will use simple T3.micro AMI Linux instances to create traffic to be captured.
Assuming that you already have sufficient knowledge about launching EC2 instances, create a new instance with the following profile:
- AMI: Amazon Linux 2 (HVM)
- Instance Type: i3en.large (or, t3.micro if you want to test and are willing to accept packet loss and memory constraints)
- Subnet: Make sure you pick a specific subnet - both the capture system and its traffic sources need to reside in the same subnet. In the example, we will use us-east-2a.
- Filesystem: If an i3en instance was chosen, then you should have two storage mounts: an 8GB EBS, and a 1250GB SSD ephemeral volume. If you didn&qops;t pick this type, you may want to manually increase the size of the EBS volume.
Then, for the system we want to monitor experimentally, create a new t3.micro or other instance type you wish, and make sure that it is within the same Subnet as the capture system.
Launch both instances after creating them, and after a bit of waiting, they should pop up in the instances view in the console:
Once they come online, it is helpful to log into both of them and update the system, and reboot them with the following commands:
sudo yum update -y && sudo reboot
Doing the above after creating new AMI-based EC2 instances is often a good idea, as the AMI images won't ever keep up to date with the latest upstream packages.
One more important configuration will be to ensure that the Network security group for the EC2 instance receiving the mirrored traffic allows UDP traffic inbound on port
4789. This is how all mirrored traffic will be delivered.
The next few steps will be best done in a separate tab, so you may refer back to the instance configurations that were just set up. Open the VPC Service Panel in a new tab, and then proceed to navigate to the Network Interfaces section of the Network & Security section of the EC2 left-hand menu bar. Once open, this displays all of the network interfaces that are allocated to running instances. The VPC Mirror Target needs one of these assigned to be the destination of mirrored packets. This section where be where we can identify the network adapter id of the instance we need to send this mirrored traffic to.
The EC2 instance that will be used to capture & process the packets is the i3en.large instance configured in the prior steps. Looking back at the instance list, EC2 instance id i-0b7bced0417c4776d is the i3en.large EC2. Using that information, we can look up the instance id in this Network Interfaces section, and identify that the associated network interface id is eni-03eca954c33e87b41. While here, it will be important to record the network interface of the *t3a.micro instance that will have its traffic mirrored. In this case, the interface id is eni-0bc0f33a0d890fa6e**. Note that these are globally unique values, so if you are following along, your id values will differ.
Using this information, navigate back to the tab with the VPC Services opened up, and scroll the left-hand menu all the way to the bottom, and then click on the the Mirror Targets option from the Traffic Mirroring section. Next, we want to click the "Create traffic mirror target" button to begin creating a brand new mirror target. This item will merely create a new destination for mirrored traffic to go, while the other three options in the same Traffic Mirroring section of the VPC services menu.
The Name tag and Description fields will be helpful for referencing the target later on, so use something simple, like:
- Name tag:
- Description: Mirror Target for ckane traffic mirroring example
Under the Choose Target section, make sure Target Type is Network Interface, and click in the Select target text box to list the network interfaces available. Find the one matching the eni-## value identified earlier. If needed, start typing in some of the id value, and the list will shrink to the matching interfaces. Once selected, click Create. This will create a new mirror target.
Next, we will want to create a VPC mirror filter. For more advanced setups, this will become helpful as it can minimize the amount of traffic that is mirrored, by limiting mirroring to traffic that can be analyzed. For instance, if the instances you wish to mirror accept inbound TLS traffic, it often doesn't make sense to capture all TLS-encrypted traffic, since it would be mathematically impossible to decrypt the traffic after capture.
Click the Mirror Filters link, and then click the Create traffic mirror filter button. This pops up a new form, similar to the Mirror Target that allows us to configure the type of traffic that will be mirrored. For the purpose of this example, we will simply mirror all traffic. Additionally, there is a checkbox that allows us to mirror the Amazon DNS traffic, which we also want to enable.
Under Inbound Rules, click the Add Rule button and enter the following into the rule entry:
- Number: 100
- Rule Action: accept
- Protocol: All protocols
- Source CIDR Block: 0.0.0.0/0
- Destination CIDR Block: 0.0.0.0/0
- Description: Mirror all traffic inbound
Under Outbound Rules, click the Add Rule button and enter the following into the rule entry (nearly identical to above):
- Number: 100
- Rule Action: accept
- Protocol: All protocols
- Source CIDR Block: 0.0.0.0/0
- Destination CIDR Block: 0.0.0.0/0
- Description: Mirror all traffic outbound
Both of these are necessary, as traffic consists of packets going back and forth between two peers on a network, and therefore we need to mirror inbound and outbound packets to capture the full context of communications.
Once all of this is filled in, and a Description and Name tag have been specified, click the Create button. In this example, I have given it the name
VPC Traffic Mirroring never really begins until a Mirror Session has been started. This enables traffic mirroring to be something that is initiated opportunistically by another event or trigger (say… an AWS Guard Duty alert). A classic approach, however, is to simply mirror all traffic to/from a particular EC2 instance (say, one that hosts a perimeter application). This is the approach that will be used in this example.
Click on the Mirror Sessions link, to open the appropriate service, and click the Create traffic mirror session button. This entry form will allow us to select another network interface to mirror into our newly-created mirror target, filtering the traffic using the mirror filter that was just created.
Choose some descriptive values for Name tag and Description. Then, for the Mirror Source use the network id of the instance being monitored (the one configured as a t3a.micro instance earlier). In this example, it is
eni-0bc0f33a0d890fa6e. For the Mirror Target, choose the Mirror target that was created earlier - the Name tag will be available and can be searched on by typing in parts of its name. For Session Number, this will just be left as
1, the default, and VNI - optional will also be left at its default, to assign a random VNI. For Packet length, also leave the default (to capture the full packets). Finally, for the Filter, select the Mirror filter that was just created. Again, the Name tag can be used to search, if needed. Click Create and the traffic mirroring will now be active.
However, nothing will be apparent until we start using the i3en.large instance to begin capturing packets.
Next, we should be able to SSH into both of the EC2 instances. On the capture instance (the i3en.large), run the following command:
sudo tcpdump -i eth0 -n -vv udp port 4789
And on the other instance (the t3a.micro) execute a few commands to generate some SSH traffic.
If everything is working correctly,
tcpdump should be reporting traffic capture similar to below:
05:03:20.014858 IP (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto UDP (17), length 102) 172.31.13.41.65508 > 172.31.11.66.4789: [no cksum] VXLAN, flags [I] (0x08), vni 655012 IP (tos 0x28, ttl 37, id 51753, offset 0, flags [DF], proto TCP (6), length 52) 126.96.36.199.54373 > 172.31.13.41.ssh: Flags [.], cksum 0x4d7f (correct), seq 500, ack 10237, win 501, options [nop,nop,TS val 3761450126 ecr 3688232271], length 0 05:03:21.167740 IP (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto UDP (17), length 170) 172.31.13.41.65508 > 172.31.11.66.4789: [no cksum] VXLAN, flags [I] (0x08), vni 655012 IP (tos 0x28, ttl 37, id 51754, offset 0, flags [DF], proto TCP (6), length 120) 188.8.131.52.54373 > 172.31.13.41.ssh: Flags [P.], cksum 0x90ce (correct), seq 500:568, ack 10237, win 501, options [nop,nop,TS val 3761451279 ecr 3688232271], length 68 05:03:21.168017 IP (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto UDP (17), length 170) 172.31.13.41.65508 > 172.31.11.66.4789: [no cksum] VXLAN, flags [I] (0x08), vni 655012 IP (tos 0x10, ttl 255, id 62435, offset 0, flags [DF], proto TCP (6), length 120) 172.31.13.41.ssh > 184.108.40.206.54373: Flags [P.], cksum 0xb76b (correct), seq 10237:10305, ack 568, win 297, options [nop,nop,TS val 3688233449 ecr 3761451279], length 68 05:03:21.193217 IP (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto UDP (17), length 102) 172.31.13.41.65508 > 172.31.11.66.4789: [no cksum] VXLAN, flags [I] (0x08), vni 655012 IP (tos 0x28, ttl 37, id 51755, offset 0, flags [DF], proto TCP (6), length 52) 220.127.116.11.54373 > 172.31.13.41.ssh: Flags [.], cksum 0x43c3 (correct), seq 568, ack 10305, win 501, options [nop,nop,TS val 3761451304 ecr 3688233449], length 0
tcpdump appears to be parsing the traffic within the UDP packets, to report the encapsulated traffic under the hood. This encapsulation is the VXLAN standard, which is documented in RFC 7348. Unfortunately, this encapsulation will prove unhelpful when analysis of the network traffic is needed later on using tools like Zeek or Suricata.
The quickest method to record a PCAP of network traffic is to use a command similar to the following, and run it for a while. This will create a new file named
ckane-blog-ex-dump.pcap and all traffic mirrored will be written into it.
sudo tcpdump -i eth0 -w ckane-blog-ex-dump.pcap udp port 4789
While the above is running, switch to the t3a.micro instance, and run a command in the console that regularly sends traffic to the terminal, such as
top or something similar. After a few minutes of waiting, close that program, and fetch something using unencrypted HTTP. For example, to retrieve the HTTP Slashdot front page:
Then, after confirming download with the HTML displayed on screen, switch back to the running
tcpdump and use CTRL-C to kill the process. This will save the traffic to disk in the file named
ckane-blog-ex-dump.pcap and gracefully end traffic capture. As mentioned earlier, the traffic will be encapsulated, meaning that
tcpdump filters and many other network traffic analysis tools will be unable to easily navigate the data stream. In order to do this, the traffic encapsulation must be removed.
We can test out that it is unable to identify the encapsulated traffic using the following command:
tcpdump -n -r ckane-blog-ex-dump.pcap tcp port 80
The above command should not report any packets.
Luckily, I came across a tool that can be used to do just this:
On the traffic capture system, I first need to install some dev tools, and clone the repository:
sudo yum install -y git gcc git clone https://github.com/theclam/stripe
Next, I switch into the
stripe folder, and build the program:
cd stripe && make stripe && cd ..
This should create a new binary
Next, to remove the encapsulation, run the following:
./stripe/stripe -r ckane-blog-ex-dump.pcap -w ckane-blog-ex-dump-decap.pcap
This should create another PCAP file named
ckane-blog-ex-dump-decap.pcap which contains all of the mirrored traffic, without the VXLAN encapsulation. So, if we attempt the
tcpdump query again, it will discover the HTTP traffic to
tcpdump -n -r ckane-blog-ex-dump-decap.pcap tcp port 80
Here is some output, demonstrating that the mirrored traffic is now the top-level content of the new PCAP file (the encapsulated version can subsequently be deleted):
05:24:39.431250 IP 172.31.13.41.46100 > 18.104.22.168.http: Flags [S], seq 4046623720, win 26883, options [mss 1460,sackOK,TS val 2904978120 ecr 0,nop,wscale 7], length 0 05:24:39.485907 IP 22.214.171.124.http > 172.31.13.41.46100: Flags [S.], seq 1559328546, ack 4046623721, win 28960, options [mss 1460,sackOK,TS val 487254419 ecr 2904978120,nop,wscale 8], length 0 05:24:39.485989 IP 172.31.13.41.46100 > 126.96.36.199.http: Flags [.], ack 1, win 211, options [nop,nop,TS val 2904978175 ecr 487254419], length 0 05:24:39.486031 IP 172.31.13.41.46100 > 188.8.131.52.http: Flags [P.], seq 1:81, ack 1, win 211, options [nop,nop,TS val 2904978175 ecr 487254419], length 80: HTTP: GET / HTTP/1.1 05:24:39.540751 IP 184.108.40.206.http > 172.31.13.41.46100: Flags [.], ack 81, win 114, options [nop,nop,TS val 487254474 ecr 2904978175], length 0 05:24:39.540810 IP 220.127.116.11.http > 172.31.13.41.46100: Flags [P.], seq 1:404, ack 81, win 114, options [nop,nop,TS val 487254474 ecr 2904978175], length 403: HTTP: HTTP/1.1 301 Moved Permanently 05:24:39.540853 IP 172.31.13.41.46100 > 18.104.22.168.http: Flags [.], ack 404, win 219, options [nop,nop,TS val 2904978229 ecr 487254474], length 0 05:24:39.540954 IP 172.31.13.41.46100 > 22.214.171.124.http: Flags [F.], seq 81, ack 404, win 219, options [nop,nop,TS val 2904978230 ecr 487254474], length 0 05:24:39.595697 IP 126.96.36.199.http > 172.31.13.41.46100: Flags [F.], seq 404, ack 82, win 114, options [nop,nop,TS val 487254529 ecr 2904978230], length 0 05:24:39.595753 IP 172.31.13.41.46100 > 188.8.131.52.http: Flags [.], ack 405, win 219, options [nop,nop,TS val 2904978284 ecr 487254529], length 0
This write-up was intended to quickly hammer through some of the AWS-specific details that enable the traffic mirroring capability. It is important to remember that this feature is only available on AWS Nitro-enabled infrastructure, so older infrastructure needs to be updated to the latest instance types in order to be supported by this capability.
Remember to perform the following cleanup steps, when done testing, in order to minimize additional costs:
- Delete the Mirror Session and Mirror Target - these cost hourly
- Stop the EC2 instances (especially the i3en.large, which can accrue costs quickly)
The following documentation discusses this feature in more detail, and was helpful in putting these examples together:
The experiment discussed here will be used to build upon for future posts on this topic.