NAT Packet Analysis Using Wireshark (by Tony Fortunato)
Life's a Pitch (by Mike Le)

Packet Capture in a Virtual Environment (By Jim MacLeod)

Spend enough time around virtualization and it becomes clear: these were tools built for server folks, and the networking was added on as a necessary evil to move data. The focus within VM network configuration is simplicity rather than actual control, and critical monitoring stats are next to non-existent. Fortunately, things have gotten better: Open vSwitch supports port mirroring, and the feature was added to VMware in version 5. Grab your favorite packet sniffer and read on to learn about sniffing VM networks.

Scenario 1: Inter-VM in the same server

The simplest VM scenario is a single server, and already there’s traffic that’s traditionally been hard to sniff. Packets between VMs in the same server almost never leave the box, which means that a tap or a physical switch span port isn’t going to work.

Fortunately, there are two straightforward solutions. First, the virtual switch itself may support a span port. That means that it should be possible to designate a NIC on the vswitch as the target for the sniffed traffic. This gives two choices: either designate a vNIC on a VM on the server, or designate a pNIC to forward the packets to an external sniffer.

The advantage of using a VM on the server as a packet collector is primarily convenience: there’s no additional hardware, and there’s no problems with what to do with the “dead” packets once they hit the wire. The downside is that you’re creating additional traffic on the vswitch, and potentially additional disk usage if you’re storing the packets straight to disk. Normally we think of sniffing as being a passive action, but capturing to disk can have a (usually minor) negative impact on the whole VM server. It’s just not as easy to scale I/O as it is CPU or RAM.

The other straightforward solution is to capture nonpromiscuously within the VM. This has the advantage of de-facto creating a capture filter on the VM of interest, if you’re doing packet capture to diagnose a specific problem. As a general solution, it scales poorly: a running packet capture on every VM “just in case” will be difficult to manage.

Sniffing on a linux-based VM is relatively straightforward using tcpdump, but you’ll get better results if you use the “-w” flag to tell tcpdump to write the captured packets to a pcap file. Then, open the file with your network analyzer of choice – I like OmniPeek for obvious reasons – and the results will be a lot easier to analyze than the normal text output.

As a quick note, there is one case where packets between VMs may be forwarded outside the box. Most vswitches operate only at layer 2, so routing between VMs will likely be forwarded to the default gateway, routed between VLANs, then forwarded back into the box. However, this may or may not happen in your environment.

Scenario 2: Encapsulated overlay traffic

In a coordinated VM environment, using a distributed virtual switch between VM servers, traffic is commonly encapsulated, using the physical server addresses in the outer layer, hiding the VM addresses inside. The idea behind this is to create more efficient networking in the VM realm by sharing the vswitch FIB among all VM servers, so forwarding can occur without standard learning & flooding, and without Spanning Tree. It also means that the VM infrastructure can essentially manage its own VLAN-like separation, without any physical switch configuration changes.

Packet capture in this environment faces two challenges. First, there’s the sometimes non-trivial question of which physical server is hosting the VM you’re diagnosing. Second, there’s the problem of setting appropriate capture filters. Because the traffic is encapsulated, you’ll get all traffic between the two VM servers, not just for the specific VM you want. Given that lots of VM server farms are running at 10G, this can be a non-trivial problem.

Once again, the approach I like is to sniff from the vswitch. It bypasses the difficulty of extracting the interesting packets at capture time. However, if the intention is to add packet capture to the infrastructure – to be able to sniff quickly as needed – then an external device is a better solution than a capture VM on every server. Rather than fighting the vswitch, use its overlay capabilities to forward the captured traffic to a dedicated port, or to export with RSPAN or ERSPAN through the physical network to the capture hardware.

If you find yourself doing this kind of analysis on a regular basis, virtual taps are emerging as a solution to coordinate with the VM infrastructure to manage and simplify the port mirroring.

The Cloud

Cloud is the most extreme example of VM infrastructure that’s hostile to network analysis, but there are still ways to pull packets from this environment. The first thing is to stop thinking about it as something mysterious, and realize that you’re primarily looking at highly automated VM infrastructure management.

Essentially, sniffing in a cloud comes down to one question: do you run the network? If you’ve got VMs deployed at RackSpace or Amazon AWS, you’re not going to be able to touch any virtual or physical switches, so local capture is your only choice. There’s a caveat here: you can very likely only do this if you’re using an Infrastructure-as-a-Service (IaaS) solution. PaaS and SaaS don’t typically give OS-level access, so you’re probably out of luck sniffing the server, but don’t forget that you can still sniff on the client side.

The good news is that the automation surrounding VM deployment in cloud means that you should have an opportunity to wrap some scripts around your packet capture agent, with the possibility of remotely controlling that capture on-demand. (Yes, WildPackets has software to help do this on Windows.)

If you do run the network in your cloud environment, then your sniffing choices fall back to the previous scenario. The cloud solution you’re using may or may not have support for port mirroring from the vswitch, but you may have an opportunity to change that. For example, OpenStack has support for multiple different vswitches, including Open vSwitch, which has supported port mirroring since at least 2011.

Do you have a favorite way to sniff in the VM world? Feel free to leave a comment.

Jimm Wp_logo Author Profile - Jim MacLeod is a Product Manager at WildPackets. He has been in the networking industry since 1994, and started doing protocol analysis in 1996. His experience includes positions in firewall and VPN setup and policy analysis, log management, Internet filtering, anti-spam, intrusion detection, network monitoring and control, and of course packet sniffing.