Network Troubleshooting with Open Source Tools (by Mike Pennacchi)

Instructor Profile - Mike Pennacchi is one of the world's leading experts on pinpointing the causes of network and application performance issues. Unlike most companies that only identify the cause, Mike’s company, Network Protocol Specialists in Seattle, Washington, works with companies to solve performance issues. When Mike is not analyzing networks, he can be found lecturing or offering training on the topic for Server and Network Admins. During November 2007, Mike was appointed as the Lead Network Engineer for InteropNet which is the mission critical show network for Interop New York 2007, leading a team of networking professionals, volunteers and vendors support personnel.

"Improve your network monitoring and analysis success potential! This is a must for every serious network manager, VERY COOL!"

-- Oldcommguy

Interop is approaching rapidly and we are excited with the classes we will be presenting this year. In addition to our long running Ethernet and Application Analysis classes, we are teaching another class on how to use Open Source tools (GNU) that are free and readily available to help monitor, analyze and troubleshoot today’s complex networks.

This will be the second year that we are presenting this class in Las Vegas. Based on the success of last year, we have added a whole new dimension to help you take advantage of more powerful and dynamic tools.

“Don’t miss this class” says Oldcommguy!

The idea for a class on how to use Open Source Tools successfully came from our own consulting experiences where we find that many of the customers we have met are not employing any type of comprehensive network management solutions within their networks. Sadly, in many cases they are simply waiting for the users to notify them when problem occur.

While this reactive approach works, it does not help the credibility of the network department and their ability to detect problems and resolve them “before” they impact the business. We believe that it is possible to assemble a set of free tools that are readily available and could be easily deployed by anyone. In this year’s class, we will share our findings with you.

Being that our business is based on packet capture and analysis we started with Wireshark as the first tool in the line-up. In the class we go through how to download, install and configure the product. After getting it up and running we go through a number of trace files that are provided on a CD that is available to all of the class attendees.

The idea is to show that while analyzing packet traces can be overwhelming, it is possible to pick out some serious problems very quickly. In addition to covering the Wireshark analyzer, we also go over the command line version, tshark.

In the class we will show you how to use tshark to create a “ring buffer” of packet traces, how an analyzer can be setup on a key link and capture all of the traffic going across this link when needed. While this type of capture cannot keep up with full line rate gigabit connections, it is possible to capture several days’ worth of traffic traversing a T-1 link going out to the Internet.

The “ring buffer” is similar to a deep capture but in manageable small pieces and having this is essential in addressing a very common situation - a client reporting that things seemed slow at 3pm yesterday. The network administrator can go back to the trace files for 3pm yesterday and review the traffic loads and the health of the connections during that time period.

The next tool that we cover is Iperf. This is a small executable that fills a very large need on the network, the need to be able to measure end-to-end throughput. By running Iperf on computers at each end of the network, we can determine the maximum throughput that can be achieved over that link.

On the other hand, too often equipment is installed and configured, but no attention is paid to whether it is operating properly. A simple PING packet may be used to test this connection. Unfortunately PING is not enough to determine if the link will really be able to support the expected speeds.

Many years ago I was the network administrator at a semi-truck plant. Every truck that we produced was sent out to a dyno, speed test emulator, which located at the back of the plant. With the dyno the truck was run up to freeway speeds and the brakes were tested. This ensured that when they took it out into real traffic that they would not run into serious problems, such as stopping.

Running Iperf on a network after a change is like putting the network on the dyno. If we can get 93 megabits out of a 100 megabit network, we are in good shape. If we only get 828 kilobits out of that network, we are most likely looking at a duplex mismatch somewhere. A PING packet is not going to tell us that.

These two tools, tshark and Iperf will satisfy the basic needs of being able to capture packets and measure the available throughput, missing is the ability to measure what is going on when it comes to bandwidth. For this task we include in our class MRTG (Multi-Router Traffic Grapher). To run MRTG one needs a set of Perl scripts that will be run every 5 minutes to test and track the utilization of the routers and switch links.

MRTG may not be the best tool but it meets some of the important constraints of many networking shops, it is easy to configure and it runs under Microsoft Windows. Our rationale in using MRTG is that something is better than nothing and most shops do not even know what the utilization patterns are for most of their WAN and LAN links. MRTG is the simplest tool to get this important information.

Another part of the class is to go through the installation process and how to configure MRTG to begin monitoring utilization levels. The great thing about MRTG is that it can be used to monitor any SNMP value. Over the years we have proven its ability to monitor current connections on web servers, number of retransmissions, temperature, even the number of ICMP Redirect packets sent by a router.

On several of our consulting engagements, we have used this tool to establish a baseline before we begin optimizing the network traffic. A Baseline allows us to present a before and after view of the network, so it is clear what we have, or have not done.

NEW This Year !

This year we are not only going to teach you about the various tools discussed above, but a whole new set of tools to help network managers become even more successful. One of the major challenges today is that many of the really cool tools are not available for the Windows platform. Making the jump to Linux is a big problem for many organizations.

Fortunately a very good friend of mine, Doug Spindler introduced me to VMWare. After years of telling me that I needed to start using it, he finally flew up here to Seattle to our offices and made me install it. I am now a VM junky and my experiences will be shared with you to help you be a more successful network manager with many more tools at your fingertips to help you in your monitoring, analysis and troubleshooting efforts.

For the first time in this year’s class, we will teach you about a solution we have developed using VM access and based on the Ubuntu operating system. This new solution will allow you to expand your set of tools beyond those that run under the Windows operating system.

After many successful network consulting tours, we believe we have a set of tools that will provide you with a versatile and dynamic monitoring and analysis solution that should be the minimum required for every companies network success, from the smallest to the largest!

The new tools we will be including this year are:

Cacti – A GUI based graphing tool used for monitoring utilization and other SNMP based counters.

nTop – When given proper tapped access into a network , nTop allows us to see the distribution of traffic , as well as who is using the most of the pipe.

Nagios – Host and service monitoring tool. Nagios will provides notifications when host and services are down.

Oreon – While I am getting better at Linux, I couldn’t stand configuring Nagios using the text configuration files. This tool provides a great web front end to the Nagios configuration.

SmokePing – This PING graphing tool sends 20 pings every 5 minutes to selected hosts. The distribution of response times are graphed out over time to show how the latency changes over time.

TCPTraceRoute – Traceroute for TCP packets. This allows us to see if a firewall or ISP is blocking our TCP traffic.

Editor's Note - Mike recently posted a tutorial on SmokePing and Cacti on our OSTU (Open Source Tools University) series.

By rolling all of this up into a VM solution, we have accomplished two things:

First, all of the applications have be installed and configured, without the need to understand Linux. Most of these applications are configured and viewed using a web browser. This means that other than assigning a static IP address to the VM, there is no need to dig around in the Linux file system.

Second, this can be run on top of any operating system that support VMWare. When out on troubleshooting jobs or just out of the office or main site, we can run this VM on our laptops to monitor the network, while we are still running our Windows tools.

If you are going to be in Las Vegas for Interop this year and are looking for a class to attend on Friday, we welcome you to stop by and check out our troubleshooting class. If not, please don’t hesitate to contact us, we are happy to help you get a monitoring and analysis solution setup on your network as well as train you on how to monitor, analyze and troubleshoot your network successfully.