LoveMyTool - Open Community for Network Management and Monitoring

How to Capture Every Packet and Why it Matters (by Chris Greer)

When analyzing an application problem, it is critical to capture traffic in the right location, with the right method (SPAN or TAP), on the right server, using the right analyzer – one that won't drop traffic.

If the stars somehow align and this is all possible, the final piece of the 'capture' puzzle is collecting traffic at the right time. The problem needs to be in progress while the capture is running. This may be the most difficult part of the process, since many application problems are intermittent, and cannot be reproduced on-demand.

To address this problem, many engineers choose to use a Network Recorder to collect every packet on a connection over a long period of time, allowing them to analyze problems that have occurred in the past. However, this analysis tool can only capture traffic that has been properly sent to it. As has been repeatedly discussed on this site, a packet can be captured in three basic ways – using a SPAN/Mirror port on a switch or a network tap. The SPAN/Mirror port or tap must be able to handle the traffic stream being monitored, which can be tricky in a high-throughput environment.

When capturing, make sure that the traffic level is well under the threshold for the capture method, and that a capture device such as the Network Time Machine is used, which can capture at line-rate in high-traffic environments, 24/7.

If a packet is lost by the capture method, meaning the SPAN or Tap, this will appear as packet loss to the analyzer. Time may be spent chasing a false alarm such as packet loss, when the issue all along was with packets not making it to the analyzer. Considering these things ahead of time and taking appropriate action will save tons of time when an application problem strikes.

Capture once. Capture smart!

Author Profile - Chris Greer is a Network Analyst for Packet Pioneer. Chris has many years of experience in analyzing and troubleshooting networks. He regularly assists companies in tracking down the source of network and application performance problems using a variety of protocol analysis and monitoring tools including Wireshark and ClearSight Analyzer. Chris also delivers training and develops technical content for several analysis vendors. He can be contacted at chris (at) packetpioneer (dot) com.

Role of Packet Capture in Network Security (by Jim MacLeod)

While working on my yard this weekend, I started thinking about the tools that I was using. My favorite is probably the weed whacker. While it’s intended for up-close trimming, its design gives it a great deal of versatility. I can use it to trim, edge, weed, mow, or even dig small holes. However, I recognize that it’s not my only tool, and the best results with the least effort will come from using it in combination with other purpose-built devices. Using a weed whacker to mow the lawn is time consuming and requires more effort than pushing the lawn mower, especially since the weed whacker only covers a small area at once, and forces me to choose how deep to go.

Those of us who love packets tend to feel similarly about our packet capture. We know that professional grade tools can monitor networks 24x7, providing statistical information about protocol and node usage, as well as deep dives for captured traffic once we’ve identified what we need to analyze. However, other purpose built tools are better at certain things. Firewall logs show what traffic was forwarded or blocked. Intrusion Detection Systems (IDS) classify traffic based on patterns that have been seen in malicious activity. While we can gather the same information with packet capture, it takes more work to get to the point of finding what needs to be examined, and what can be ignored.

Network and Application Root Cause Analysis – Part 2 (by Chris Greer)

A few years ago, “Prove it’s not the network” was all that a Network Engineer had to do to get a problem off his back. If he could simply show that throughput was good end to end, latency was low, and there was no packet loss, he could throw a performance problem over the wall to the server and application people.

Today, it’s not quite that simple.

Network Engineers have access to tools which give them visibility into the packet level of a problem. This detail in visibility often requires them to work hand in hand with application people all the way through to problem resolution. In order to find performance problems in applications, the network guys have had to take the TCP bull by the horns and simply take ownership of the transport layer.

What does that mean?

First, it means that “Prove it’s not the network” isn’t enough, as we have already mentioned. But it also means that analyzing TCP windows, TCP flags, and slow server responses has fallen on their shoulders. Since this is the case today, let’s look at a quick list of transport layer issues that the Network Engineer should watch for when analyzing a slow application.

What Makes Your Network Weak (By Jim MacLeod)

Your network is going to be attacked. Get used to that idea. At least one of those attacks will succeed, and there’s not a lot you can do to stop it. However, what you can do is delay the inevitable breach and attempt to control what the attacker will be able to do once he or she gets in, and you can proactively prepare your response.

If you don’t believe that you’re vulnerable, take a minute to scan your favorite news feed for security breaches. If you think you’re too big, look at the large examples out there like the White House, large banks, health care systems, and large corporations who presumably have the talent and resources to stop attackers in their tracks. If you think you’re too small to be a target, I recently heard of a friend – an expert in network forensics – whose computer got hijacked by a drive-by download on a web site. Fortunately, the PC was recoverable, but it required pulling the hard drive and mounting it on another PC to clean. Small sites are also attractive to attackers as pivot points to launch larger attacks and as storage points for illegal material, which leaves you as an inadvertent partner of the attack.

Network Data Storage Best Practices (by Spencer Greene)

How long should I store packet captures?  How much storage should I provision to monitor a 10Gbps link?  When is NetFlow enough, and when do I need to capture at the packet level?

These are questions network operations managers everywhere are asking, because unfortunately best practices for network data retention policies are hard to find.  Whereas CIOs now generally have retention policies for customer data, internal emails and other kinds of files, and DBAs generally know how to implement those policies, the right retention policy for network capture data is less obvious.

The good news is that there are IT shops out there that are ahead of the curve and have figured a lot of this out.

Background

To begin with, it’s important to clarify for your own organization what the goals are for network history.  Some common answers include:

1. Respond faster to difficult network issues
2. Establish root cause and long-term resolution
3. Contain cyber-security breaches
4. Optimize network configuration
5. Plan network upgrades

Wireshark in the Large Enterprise - Sharkfest 2012 (by Hansang Bae)

Presenter Profile - Hansang Bae currently leads the Network/Application Performance Engineering Team with direct responsibility for Packet Capture Infrastructure at Citi. He brings a unique perspective with his broad knowledge of protocol analysis in a complex enterprise infrastructure.

As one of his colleagues puts it, “Hansang is one of the most outstanding individuals in the industry. His technical expertise is second to none. He is also an amazing leader, mentor, and a highly ethical, sincere person. Hansang possesses the unique ability to explain highly complex topics to a broad audience, which earned him the trust and respect from every person he encountered within Citigroup. ”

Editor's note - This presentation was recorded at Sharkfest 2012 - UC Berkely, CA - June 24th-27th.

Security Threats: The Who, the How, the Why (By Jim MacLeod)

As networks grow more complex, the process of securing and managing endpoints, applications and confidential information has become more challenging than ever before. Last year alone saw a dizzying array of security incidents and breaches, including Symantec, SONY, Fox News, and others. Security is becoming more and more essential to business operations, but traditional defenses aren't working to the degree that companies require. Meeting the new security service levels starts with understanding each type of threat to establish a unified security program.

While every attack is unique in its origin, execution, and range, defending against cyber attacks today can be abstracted down to three of the classic five W’s: Who, How/What and Why. The other two W’s, Where and When, are constant for network defense: your network, 24/7/365.

This blog post will introduce the elements of these categories, then present a deep dive on the Who category.

Insurance For Your Network? (by Jay Botelho)

Insurance is a way of life, right? We insure our lives, our cars, our homes, and even mundane things like our cell phones. In short, we insure those things that are dear to us and that we cannot do without. So how about your network? If you’re a network engineer the network is pretty dear to you, and your company certainly cannot do without it, so it seems like a good candidate for insurance too. But what kind of insurance do you need? What is it that you need to protect most about your network?

Last year was labelled the “Year of the Hack,” by IT security professionals, with an onslaught of breaches that were financially and politically motivated. This year seems to be following in those same footsteps, with the latest mega-breach of Global Payments in April 2012 with 1.5 million accounts compromised. And on the political front, just do a Google search on cyber crime and see how worried everyone is about cyber criminals targeting the Summer Olympics in London.

Current protections are not enough. Although current security systems certainly protect against a significant number of attacks, they obviously don’t protect against all attacks. So let’s buy some insurance!

Is Enabling SNMP Worth the Security Risks? (by Chris Greer)

Does SNMP open a hole? Well, kindof. Is it worth it? If it's configured right, YES!

The data that can be collected from switches and routers via SNMP for monitoring and analysis is completely invaluable. Information such as port utilization, device connectivity, errors, packet drops, discards, and other critical network health statistics are available with SNMP, but only if it is enabled!
Most switches and routers these days (even the cheaper home networking ones) support SNMP.

However there is still the idea floating around out there that enabling it will make a network vulnerable to security attacks. Also, some published material out there in cyberspace makes the claim that enabling SNMP will make your network unsecure. It is true that enabling SNMP on switches and routers will open a door into their management, however, there are a couple ways that we can lock up SNMP pretty tight and get the full benefits of having monitoring on the network.

1.   Community string with access list.

Network visibility and inspection (by Tim Nichols)

The world of IT and networking is changing, and it’s changing fast. Not only are networks getting much faster but IT and data are going virtual, applications are migrating to the web, people are bringing their own toys to work and the cyber crooks are playing a much smarter game than they’ve ever played before. The days of organizations deploying relatively simple point solutions on COTS (commercial off the shelf) hardware have gone for good. It’s time to face the facts: A single dropped packet could mean the difference between seeing the Stuxnet worm and seeing nothing!!! 

Too Much to see - too fast  - where is it!

The real problem that organizations face is one of network visibility. It’s inconceivable to imagine airport security letting passengers skip the metal detector when the queue gets a bit long, yet there are thousands of systems in the ground that are currently dropping millions of packets daily from capture and even inspection. Today there are glaring holes in the armor that are exposing critical data and systems to the criminal fraternity.

Here is a list of three visibility pre-requisites that decision-makers responsible for procuring network security, network monitoring or any type of network instrumentation must be looking for:

 - Absolute proof of continuous 100 percent accurate packet capture and inspection at line rate at 1Gbps, 10Gbps, 40Gbps AND 100Gbps.

100Gig is just around the corner and any technology decisions, particularly expansive long term integrations, must take into account the demands of tomorrow’s network as well as today’s. Continuous 100 percent packet capture and inspection at high speeds is a problem that, as any expert will attest, can only be solved in hardware. There’s a compelling argument that suggests we are about to enter a new age of purpose built, ultra high performance network monitoring and recording systems that are designed from the ground up to capture, analyze (inspect) and record packets for the purposes of network visibility and loss mitigation.