LoveMyTool - Open Community for Network Management and Monitoring

How to Capture Every Packet and Why it Matters (by Chris Greer)

When analyzing an application problem, it is critical to capture traffic in the right location, with the right method (SPAN or TAP), on the right server, using the right analyzer – one that won't drop traffic.

If the stars somehow align and this is all possible, the final piece of the 'capture' puzzle is collecting traffic at the right time. The problem needs to be in progress while the capture is running. This may be the most difficult part of the process, since many application problems are intermittent, and cannot be reproduced on-demand.

To address this problem, many engineers choose to use a Network Recorder to collect every packet on a connection over a long period of time, allowing them to analyze problems that have occurred in the past. However, this analysis tool can only capture traffic that has been properly sent to it. As has been repeatedly discussed on this site, a packet can be captured in three basic ways – using a SPAN/Mirror port on a switch or a network tap. The SPAN/Mirror port or tap must be able to handle the traffic stream being monitored, which can be tricky in a high-throughput environment.

When capturing, make sure that the traffic level is well under the threshold for the capture method, and that a capture device such as the Network Time Machine is used, which can capture at line-rate in high-traffic environments, 24/7.

If a packet is lost by the capture method, meaning the SPAN or Tap, this will appear as packet loss to the analyzer. Time may be spent chasing a false alarm such as packet loss, when the issue all along was with packets not making it to the analyzer. Considering these things ahead of time and taking appropriate action will save tons of time when an application problem strikes.

Capture once. Capture smart!

Author Profile - Chris Greer is a Network Analyst for Packet Pioneer. Chris has many years of experience in analyzing and troubleshooting networks. He regularly assists companies in tracking down the source of network and application performance problems using a variety of protocol analysis and monitoring tools including Wireshark and ClearSight Analyzer. Chris also delivers training and develops technical content for several analysis vendors. He can be contacted at chris (at) packetpioneer (dot) com.

Batteries in my TAP, in my Network, What are you thinking? (by The Oldcommguy)

There is a new trend of putting batteries in network access devices that I am really not sure of, so here are my thoughts as well as some very sobering and REAL documented issues!

 First let’s talk about the reality of batteries, ALL BATTERIES – Especially All Rechargeable's!!

• They need a charge cycle thus a charge and discharge circuit.
  • This is VERY important! As no charging or overcharging can cause a rupture, fire, etc.
  • YOU CANNOT JUST LET a battery ( Lithium, NiCd or NiMH) just sit in a device without expecting failure, sooner or later.
  • You need some notification of the condition of the batteries charge state
  • You need to know that it needs replacing or a serious rupture and/or fire can occur.
    • If it ruptures you can have serious caustic situation in the air and in the chemical spill
    • The older a battery gets the higher potential for a failure and rupture
      • United States Consumer Product Safety Commission (CPSC)
        • When burning, a battery/cell can reach temperatures of 700C+
          • That will easily burn through aluminum and almost anything else
          • During and after burning there will be not only hazardous/poisonous gasses but a dangerous caustic spill!

SSL Key Management (by Jason Wells)

If you’re doing analysis on network captures, you’ll eventually run into the need to view encrypted data. This is especially true if you’re troubleshooting a secure web application like a shopping cart or online banking. If you’re doing this analysis in Wireshark, you’re somewhat caught between a rock and a hard place - you have to either:

• Give your private keys over to the engineers, and let them install it locally in their copy of Wireshark in order to decrypt data
• Have one location or virtual machine where the keys are installed on the machine’s copy of Wireshark, and have users share use of the machine

In the first case, you now face a situation where the very private keys used on your website - the whole reason your site is secure in the first place - now exist as multiple copies in multiple locations. Are you going to rely on policy that they be deleted? Or only limit access to one support engineer?

In the second case, you face the normal difficulties with trying to use one shared machine to operate a network tool. In addition, you must now secure this machine (likely a simple workstation) from intrusion to a degree necessary to protect one of your most vital assets - getting your SSL keys leaked is bad, bad news.

Network and Application Root Cause Analysis – Part 2 (by Chris Greer)

A few years ago, “Prove it’s not the network” was all that a Network Engineer had to do to get a problem off his back. If he could simply show that throughput was good end to end, latency was low, and there was no packet loss, he could throw a performance problem over the wall to the server and application people.

Today, it’s not quite that simple.

Network Engineers have access to tools which give them visibility into the packet level of a problem. This detail in visibility often requires them to work hand in hand with application people all the way through to problem resolution. In order to find performance problems in applications, the network guys have had to take the TCP bull by the horns and simply take ownership of the transport layer.

What does that mean?

First, it means that “Prove it’s not the network” isn’t enough, as we have already mentioned. But it also means that analyzing TCP windows, TCP flags, and slow server responses has fallen on their shoulders. Since this is the case today, let’s look at a quick list of transport layer issues that the Network Engineer should watch for when analyzing a slow application.

Packet Capture in a Virtual Environment (By Jim MacLeod)

Spend enough time around virtualization and it becomes clear: these were tools built for server folks, and the networking was added on as a necessary evil to move data. The focus within VM network configuration is simplicity rather than actual control, and critical monitoring stats are next to non-existent. Fortunately, things have gotten better: Open vSwitch supports port mirroring, and the feature was added to VMware in version 5. Grab your favorite packet sniffer and read on to learn about sniffing VM networks.

Scenario 1: Inter-VM in the same server

The simplest VM scenario is a single server, and already there’s traffic that’s traditionally been hard to sniff. Packets between VMs in the same server almost never leave the box, which means that a tap or a physical switch span port isn’t going to work.

Fortunately, there are two straightforward solutions. First, the virtual switch itself may support a span port. That means that it should be possible to designate a NIC on the vswitch as the target for the sniffed traffic. This gives two choices: either designate a vNIC on a VM on the server, or designate a pNIC to forward the packets to an external sniffer.

Three Ways that your Network is Vulnerable (by Jim MacLeod)

Finding the loopholes and soft spots in a network can be difficult for anyone, be they network administrator or security auditor. Network speeds are growing with widespread adoption of 10G and initial deployments of 40G, plus increasing cross-connectivity that is difficult to police with enough granular policy enforcement. Application complexity is likewise growing, with increasing horizontal scalability of applications through separation of systems and large-scale virtualization. Both of these trends combine to increase the initial attack surface and expand the potential exploitability of historically small flaws.

In short: networks of services are getting more complex, and that complexity leads to vulnerabilities which are more serious than they would have been. Given that these vulnerabilities exist, it’s important that you find and fix them, rather than an attacker.

Below are three straightforward ways to find the holes in your system before someone else does.

The Cure for Forklift Sniffing: Multi-Segment Analysis (By Jim MacLeod)

Packet analysis is a great method for finding network problems. However, while a single point of capture can verify that a problem exists, it’s often not enough to determine the location. For too long, troubleshooting has relied on reckoning whether the problem is towards the server or towards the client, then moving to a different capture point and hoping to observe the issue again. If network upgrades which require new hardware are “forklift upgrades,” then moving the packet capture point is “forklift sniffing” – and it’s clearly not a great methodology for packet analysis. What’s needed for any network larger than 1 switch or wireless access point is the ability to capture at multiple points simultaneously.

Forklift Sniffing

Single-point packet capture has the advantage of appearing to be relatively simple. It’s an on-demand activity: when there’s a network problem, pull the packet sniffer out of mothballs, plug in somewhere, and start capturing. That assumes that you have somewhere to plug in to capture. Today’s switched networks make packet capture an interesting challenge, but most of them support SPAN. So, plug in the sniffer, then log in to the switch to enable SPAN on your port, and hope that the problem is still happening. Analyze the data, and then see if you can determine what the problem is. You look at TCP to determine whether there are missing data packets (problem is closer to the server) or whether there are missing or delayed ACK packets (problem is closer to the client). Either way, the problem isn’t where you’re connected, so unplug, move to a different location, and try again.

Do you know who is in your Home? (by Tim O'Neill)

Do you know who is in your home?

Then why do you not know who is in your network and applications and what data they have access to?

In our modern day networks we pass trillions of bits, bytes and frames of data through our networks ever minute, 24 hours a day and 365 days a year. Most of the data we pass has some degree of confidentially either for ourselves or for our company. Almost anyone can tell you who is in their homes, then why do most of our network managers not know who is in their network, doing what and using what applications. Are users sticking to your network security policies and procedures? Are they leaving connections up while they go for a break or lunch or even worse until tomorrow? Are some users getting into applications and data that they are not supposed to have access to?...the list can go on and on and with the tremendous amount of data that is passed, the huge amount of internet and intranet pages browsed, the number of records accessed….etc keeping up with these answers and even just a basic network efficiency view seems like an impossible task……well it can be daunting but with the correct technology all these questions can be answered and tracked with more ease then one would imagine and can allow us to make use of even the most basic of technologies to get extremely clear views of our networks, applications and users.

Do not loose your company's important Data to a Thief!

I would like you to consider my seven important points about REAL Visualization and the value of TAP and Filtering technology -  

First – anyone one that wants to perform real network monitoring and analysis must have access to all the REAL data that flows, the good, bad and ugly in relative time! The only real way to achieve is to have TAP access to your network flows, Why TAPs – well you might want to read an older article that is used by many universities - http://www.lovemytool.com/blog/2007/08/span-ports-or-t.html  , to reveal the truth about TAP versus SPAN and the other under achieving methods of acquiring network information.

VoIP Call Quality Tracking (by Patrick Hubbard)

Once unproven technology deployed by only the largest organizations, Voice over Internet Protocol (VoIP) is becoming an everyday responsibility of many IT and network administrators. Small and medium-sized businesses in particular can reduce their telecommunications costs and increase flexibility by converting an unwieldy TelCo/PBX problem into just another service on the network. The biggest remaining challenge for VoIP administrators regardless of the number of extensions is assuring high call quality.

While there has been significant progress made in improving the quality of service for VoIP calls, quality is still generally lower than that provided by Public Switched Telephone Network (PSTN) networks. Corporate IT and accounting are drawn to VoIP based on the economics, but to most users, a phone is a phone. They expect the quality of PSTN. As a result, network administrators looking for successful user adoption of a VoIP system must keep a close eye on the actual call quality users experience.

Five Reasons to Move to the Pcapng Capture Format (by Jason Walls)

The pcap capture file format has been the universal packet capture format since the early days of computer networking. Almost all capture tools support the pcap format. And while vendors have created new formats over the years, most tools support conversion into the pcap format.

While pcap continues to be used today, it does have some limitations that make other formats more attractive. A new format called “pcapng” has been under development for a number of years. The goal of pcapng is to address some of the shortfalls of pcap and create a flexible format for the future. With the adoption of pcapng as the default format for WireShark 1.8, there appears to be a growing push to support pcapng.

CloudShark’s recent 1.6 release also pushes ahead with improved pcapng support. The addition of pcapng allows CloudShark to unify pcapng’s per packet comment capability with CloudShark’s annotation capabilities.

So, what are some of the advantages of pcapng and why should you make the switch? Are there any limitations that need to be addressed before we’ll see large scale adoption of the new format?

Where did Pcapng come from?