When the Shark Bites! (by Mike Canney)

 

Let me just start off by saying that I have used almost every commercial capture-to-disk appliance on the market.  Ranging from a 2 TB appliance that I built with open source software to the 96 TB commercial products.  Some of the biggest annoyances have always been mining the data out of this huge ring buffer.  The other annoyance has been with the vendor claims.  Capture boxes that have multiple 10 Gb interfaces and “claim” that they support 10 Gbps, sustained line rate.  As we know, it typically is an over committing marketing department that makes these claims and not the engineers that design these products. So I guess we can cut the a little slack.

The 10 Gigabit myth: 

As I have mentioned, a number of vendors claim that they handle 10 Gbps with ease, but how do you know unless you test them?  As Tim O’Neil says, “Don't trust them, test them”!  That is exactly what we did.  

Riverbed Cascade Shark:

I recently was able to get my hands on a Cascade Shark Appliance.  I suspect many of you have seen my reviews on Cascade Pilot and know how much I love this product for slicing and dicing extremely large trace files.  Pilot has truly changed my life when it comes to mining out data from large packet traces.  Naturally, I was very excited to see if the Shark Appliance could live up to my huge expectations.  

The first test we ran against the Shark was to see if it could indeed capture at line rate, sustained 10Gbps.  The test included hammering the Shark (no pun intended) with a Smartbits (Test Center) with a 10Gbps stream.  This test mimicked real world application traffic (not a 64 byte slamming of the box).  I was very impressed that the Shark held up without dropping a single packet for over 48 hours (the length of the test).  The Shark operated per the manufacture claims.  

We then decided to connect the second 10 Gb interface and run a 20 Gbps test to see how the box would react, knowing that it is only rated for 10 Gbps.  This is where I became really impressed.  No, it couldn’t survive without dropping packets at 20 Gbps, but the first interface held strong at 10 Gbps while the second interface started dropping packets at about 5 Gbps (total traffic = 15 Gbps).  So, the first interface, even though the box was way overloaded, dropped zero packets.  This was really cool. The the first interfaced retained it’s packet capture integrity and only the second interface suffered.  Granted, you wouldn’t want to deploy it this way into production, but it is nice to know that you can stretch beyond the 10 Gbps rating.

Data Mining:

In my opinion, this is the secret sauce of the Shark Appliance.  With most packet to disk capture appliances on the market, the most excruciating part of the analysis is just getting the data that you need to analyze out of the capture appliance and into your protocol analyzer.  

My typical check list and work flow goes as such:

1.)  Know when the problem occurred.  Check.

2.) Know the ip addresses of the systems in question.  Check.

3.) Analyzer zoomed in on that specific time period that the problem occurred. Check.

4.) Push the button to get the data from the ring buffer.  Check.

5.) Go to lunch because it’s going to take forever before the trace file is mined, loads and is transferred to my workstation.  Check.

With the Shark appliance, number 5 now looks like this:

5.) Take a trace clip of the data from numbers 1-3.   This makes a much smaller trace out of the huge ring buffer.  Push the button to send the data to Wireshark. Start analyzing the packets in around 30 seconds.

I am not sure how the folks at Riverbed are indexing their traces, but it is night and day faster than anyone else.  I cannot begin to tell you how much time this saves especially when on a high pressure conference call with people breathing down your neck wanting to know what the problem is.  Riverbed Cascade, keep it coming!

 

    

Author Profile: Mike Canney is the President of getpackets.com, specializing in providing application and network performance consulting services. 
 Over the past 23 years Mike has helped 100's of companies identify and resolve their application and network performance issues. Mike has also developed coursework and taught thousands of engineers how to identify, remediate, and prevent network and application issues by analyzing traffic flows at the packet level. Mike can be contacted at canney (at) getpackets (dot) com