LoveMyTool - Open Community for Network Management and Monitoring

How many errors are acceptable on switch and router ports? (by Chris Greer)

Quick answer? Nada, Zero, Zilch! (Sure there are exceptions – but you wanted a quick answer, right?)

I recall reading a whitepaper one time which gave target average utilization and error levels for Ethernet ports. In the report, it mentioned that a target level for traffic should be 40% on average. I was ok with that number, seemed normal to me. The next figure given was for errors (like packet drops, FCS Errors, late collisions, etc…) and the acceptable figure for a healthy network was 1% of port utilization. So according to the whitepaper, if the utilization rate was at 40%, then .4 percent of that traffic could be errors, and all would be ok.

HUH?

So to put some real numbers to that, a 1Gbps network interface with 40% utilization would be 400Mbps. If the average packet size was around 350 bytes, this would mean that for any given second, there would be just over 142,000 packets flying by. So if 1% of these had errors, that would be 1,420 packets per second. As we have discussed on Love My Tool in the past, when a packet has an error, it is dropped by the next switch or router port it crosses. Since Ethernet is not responsible for retransmission (that’s TCP’s job), the switch or router does nothing to report this issue, other than add a counter to its error log. This means that for every packet drop, discard, FCS error, late collision, or long/short frame, we have to wait for TCP to timeout and retransmit, which takes time. These errors have a huge impact on network and application performance, and can really undermine the business.

How can we find them?

To put it simply, look at our routers and switch ports for errors. Does this mean that we have to comb through every router and switch, doing a “show interface” on all ports? If we don’t have a tool to do it for us – yes, we do. That can get a bit tedious, so some tool manufacturers have an SNMP based interface scanner that can comb your whole enterprise for errors, presenting them front and center so you can resolve them. The Fluke Networks OptiView XG is one such tool. It scans all known switches and routers, and if it finds Ethernet Errors, packet discards, or discovers excessive utilization, it will alert the front page, displaying the switch and port number.

Cool huh?
Don’t let errors get you down. Find the little buggers and resolve them, before they impact critical business applications.

Author Profile - Chris Greer is a Network Analyst for Packet Pioneer. Chris has many years of experience in analyzing and troubleshooting networks. He regularly assists companies in tracking down the source of network and application performance problems using a variety of protocol analysis and monitoring tools including Wireshark. When he isn’t hunting down problems at the packet level, he can be found teaching various analysis workshops at Interop and other industry trade shows. Chris also delivers training and develops technical content for several analysis vendors. He can be contacted at chris (at) packetpioneer (dot) com.

Understanding Packet Path Critical to Analyzing Application Problems (by Chris Greer)

Close your eyes and imagine that you are a packet (I know it sounds nerdy, but stick with me here). You’re leaving the NIC at the client end, and you are on your merry way to the application server at the other end of the connection. What is the first switch and port number you enter into? Which port are you forwarded out of? Which switch is next? Which routers are involved? What if you hit a roadblock, where do you get re-routed to?

Now this is probably the most important question – Are you sure you know the path betwen client and server?

Most network people know their systems pretty well, and may be confident about the path of a packet between clients and servers. But with moves, adds, changes, upgrades, and servers being made every day, be careful about assuming that a path from A to B is well documented and understood. Be especially careful about this assumption when installing analysis gear to capture a trace file and analyze the traffic. After all, there is nothing worse than getting a great analyzer installed where we assume traffic to be, starting the capture, reproducing the issue, then digging through the traffic – all to find out that what we’re in the wrong place.

How to Find the Path

There are some great tools out there at display the path a packet takes from client to server at layer 3. But one of my favorites – the OptiView XG – also shows the layer 2 path through the switches as well. It does this by using a feature called Path Analysis.

The Importance of Every Packet (by Chris Greer)

When analyzing a packet capture, nothing can be more frustrating than realizing that for whatever reason, packets are missing. Sometimes, this realization only strikes after spending hours looking through retransmissions, out-of-order, or previous packet lost warnings. Most analyzers will show these alerts in a trace file automatically, and there really isn’t a way for the analyzer to tell the difference between a packet that is really lost, and one that simply didn’t make it to the capture buffer.

This means that it’s up to the analyst to decide if packet loss is really there, or if the analyzer simply missed something. So this leads to two questions: (1) How does this happen in the first place, and (2) How can we tell if a retransmission warning is real, or just evidence of missing data?

How packets can be dropped on the way to the analyzer.

As has been repeatedly discussed on this site, a packet can be captured in three basic ways – using a SPAN/Mirror port on a switch, a tap, or on a hub. Often, the assumption is made that if we just use one of these methods, slap Wireshark on a laptop and hit capture, that we will get every packet. But, as we have seen, this assumption is not true. Several things must come together for a packet to make it to the analyzer. The SPAN/Mirror port, tap, or hub must be able to handle the traffic stream being monitored, which can be tricky in a high-throughput environment. The capture method itself can be faultly, as we have seen in several SPAN sessions where traffic is low, but the switch is busy with other business. Also, the analyzer itself must be able to handle the incoming traffic load, which is very tricky in a high-throughput environment. Remember, just because the port on your laptop says 1Gbps, the actual rate that it can capture at is much lower, depending on the packet driver used.

So when capturing, make sure that your capture method is well under it’s traffic threshold, and that you’re using a capture device that can capture at line-rate in high-traffic environments. (I wouldn’t go near a SPAN or Tap with a laptop on a port that exceeds 75Mbps, just as a rule of thumb).

Analyzing slow applications? Capture on both sides of the connection (by Chris Greer).

When firing up Wireshark (or any other packet analyzer for that matter) to capture traffic on a slow application, one of the first questions that usually comes up is –

Should I capture on the client side or server side?

For years now, this question has been debated among analysts and continues to be a matter of preference, experience, and all too often, timing. Sometimes we don’t have the time or gear to properly capture on the server side, so we are left to the client end.

So which one should we capture on? Well, as a packet geek who wants as much information as possible about a problem, I say BOTH! The best case scenario is to setup a capture on the client end, as well as the server end, simultaneously capturing traffic while the problem is occurring. At times, this may be the only way to pinpoint the real underlying problem.

How can we capture on both sides?

Slow Applications? Don’t Overlook Packet Loss (by Chris Greer)

Sometimes when we are tracking down the cause of performance problems, we can get lost in the nitty gritty details. Especially when dealing with a multi-tiered application, a complex array of load balancers, application accelerators, or the many virtualization technologies, troubleshooting a slow application can be a daunting prospect. 

But never fear. When we are looking for that golden nugget, the ultimate needle in the data center haystack which will resolve our priority-10 performance problem, we may overlook some of the heavy hitting speed-killers which are easy to spot with the right tool. So next time a problem strikes, resist the urge to pour through user privileges in the domain or dig through the back end SQL data for the bad call, and check out a few of the basics. One of which is packet loss.

Packet Loss – Really? In Today’s Networks?

The Power of Pilot - Tips and Tricks with Cace Pilot (by Chris Greer)

These days there are several ways to find out where users are going on the web and what they are doing. Any number of tools can give this data in a somewhat readable format. What is alarming though is how many environments have absolutely no edge monitoring tools in place. Whether we support large corporate enterprise networks, or small business environments, it is important to have monitoring tools in place that show is the truth - what users are doing, where they are going, and in some cases, why things are slow.

Pilot is a tool from Cace Technologies (now Riverbed) that makes looking at network traffic easy. In fact, for where it is priced in the market, I'd say it's the biggest bang for your buck when it comes to traffic analysis tools. It works alongside Wireshark to deliver information to the user in a simple, easy to read, get to the root of the problem format. In this article series, I'm going to go through some of the features of Pilot and show why it is a great tool to have around. As always in these articles, my goal is to show the power of analysis and monitoring, and how great tools are just a click away at a great price.

What Was Great Just Got a Whole Lot Better! – New Fluke Networks OptiView XG (by Chris Greer)

 

 

Not gonna lie, this new tool has has some pretty neat stuff built into it.

The new OptiView XG is packed with some great features, including everything from 10 Gig lossless capture and analysis to probing into virtual machines and pulling out useful monitoring data. For the purposes of this article, I’m not going to go into every single new feature – the folks in marketing do a great job at that! But I do want to write about some of the heavy-hitter new features from the perspective of a packet guy, things that I think are a huge help in the world of network and application analysis.

Application Infrastructure Analysis

This is a great feature for troubleshooting slow apps. The OptiView XG will track the path, port to port, of every device between itself and the application that is being tested (Path Analysis). It’ll look for port errors as well as utilization along the path, testing to see if the network could be responsible for poor application performance (dropping packets or congested links). It then fires up a TCP connection to the server and times the response, making sure there aren’t any transport layer issues that could be slowing things down. Last it will check server resources, looking for excessively utilized CPU, memory, disk, current processes, or VM Memory usage. The test will also optionally allow the server interface to be monitored for utilization over time, watching for spikes in activity.

Why it’s cool

Come On People ... TEST! (by Chris Greer)

Ok, I’ll explain the title by starting off with a story.

I received a call from a company about having an intermittent network outage. It was a major company that was brought to its knees for two days, almost completely unable to do business during that time. They called and I was ready to jump on a plane any minute once they said the word GO. A few days later, after receiving no response, I gave them a follow up call to check and see how things were. They said the problem was fixed, and that they had found an old router plugged into their core switch. After removing it the problem went away, leaving them feeling confident that they found the solution.

I had about a hundred questions flood into my head. How long was the router there? Did the problem start occurring only after it was connected? What type of traffic was the network experiencing during the “outage”. Were any users able to access and use applications?

Most importantly…

How do you know that you fixed it for sure? Now you may get the title. Come on people…. TEST!

Problems that are “fixed” but not tested and understood are doomed to repeat themselves.

This is not an isolated story. When a problem strikes, people may start scrambling to find a fix and cross their fingers after it is applied. Instead of truly testing that the root cause was resolved and understanding why the problem caused the symptoms in the first place, network techs may rely on perception from the users instead of factual results from testing. If a problem all of a sudden goes away, how do we know it won’t come back? How can we setup monitoring tools that will tell us when the problem occurs again instead of hearing it from the users?

All these questions have solutions. I’m not writing this article to sell the reader on one tool or another. But I am selling the reader on testing, analyzing, fixing, then testing again. This process will truly address the root cause of network problems and will help them go away for good.

If you have any specific network issues you would like to see discussed on Love My Tool, please comment below!


Continue reading other LoveMyTool posts by Chris Greer »

Wireless Overkill Can Cause Poor Performance (by Chris Greer)

 

When wireless networks were first implemented, one of the primary goals was to provide adequate coverage to the environment. Users wanted the ability to browse the web, check email, and access applications, without the tether of a wired connection. These apps, along with many other tasks an average user needs to perform, don’t generate much overall bandwidth on the network. Unless files are being backed up or downloaded, the average user only will generate 300Kbps when driving basic applications. Since bandwidth demand is low, when wireless networks emerged, they did not need to provide maximum performance in order to keep the average user happy. 

Things have changed.

Today, mobile, high-bandwidth applications such as voice and video have emerged on the network. These apps have driven the need to improve wireless efficiency and performance. Don’t be fooled by the sticker on the AP which boasts of up to 54Mbps. At best, because of the CSMA/CA method used by wireless, along with all the overhead, we can only expect half of the advertised speed in actual throughput. If there is significant interference, or if the wireless environment is not optimized for maximum throughput, bandwidth greedy applications will work poorly, or sometimes, not at all.

Recently, in one environment, a wireless video application required a minimum of 3Mbps of throughput. Wireless coverage was not a problem, as there were several access points to adequately cover the floor. The access points were configured for channels 1, 6, and 11 in the 802.11b/g range, but in any one spot, a client could detect 10 or more access points.

DNS and DHCP – Monitoring The Services We Can’t Live Without (by Chris Greer)

A monitoring tool for critical network services should be a no-brainer for every environment. After all, if these go down, or if they are operating slowly, things just won’t work as they should. Users may complain that the network is slow, or down, when in fact the real root cause is with DNS or DHCP.

For these reasons, it is important to have a solid DNS and DHCP monitoring tool that frequently checks these services for availability and performance. If anything looks out of sorts, we can be aware of it and react, hopefully before the users start to call in.

There are several tools out there that can help with this test, ranging from dedicated server tools, to features within a larger platform. Some are open source, others are vendor supported. No matter what we choose to monitor these services, it’s just important to HAVE!