LoveMyTool - Open Community for Network Management and Monitoring

Moving beyond "It's not the network" (by Chris Greer)

Ok, we get it. It's not the network.

Over the past few years, network engineers have slowly been winning the battle of blame. Even though most people still say "The network is slow" when there is a performance problem, many engineers now have the tools to prove otherwise. Whether they use SNMP to check their infrastructure devices, flow tools to watch for utilization spikes, packet analysis engines to watch-dog applications, or automated performance analyzers with built in alarms, most environments have some type of monitoring in place to help the engineer prove his case.

However, the problem now becomes - what next?

How to Capture Every Packet and Why it Matters (by Chris Greer)

When analyzing an application problem, it is critical to capture traffic in the right location, with the right method (SPAN or TAP), on the right server, using the right analyzer – one that won't drop traffic.

If the stars somehow align and this is all possible, the final piece of the 'capture' puzzle is collecting traffic at the right time. The problem needs to be in progress while the capture is running. This may be the most difficult part of the process, since many application problems are intermittent, and cannot be reproduced on-demand.

To address this problem, many engineers choose to use a Network Recorder to collect every packet on a connection over a long period of time, allowing them to analyze problems that have occurred in the past. However, this analysis tool can only capture traffic that has been properly sent to it. As has been repeatedly discussed on this site, a packet can be captured in three basic ways – using a SPAN/Mirror port on a switch or a network tap. The SPAN/Mirror port or tap must be able to handle the traffic stream being monitored, which can be tricky in a high-throughput environment.

When capturing, make sure that the traffic level is well under the threshold for the capture method, and that a capture device such as the Network Time Machine is used, which can capture at line-rate in high-traffic environments, 24/7.

If a packet is lost by the capture method, meaning the SPAN or Tap, this will appear as packet loss to the analyzer. Time may be spent chasing a false alarm such as packet loss, when the issue all along was with packets not making it to the analyzer. Considering these things ahead of time and taking appropriate action will save tons of time when an application problem strikes.

Capture once. Capture smart!

Author Profile - Chris Greer is a Network Analyst for Packet Pioneer. Chris has many years of experience in analyzing and troubleshooting networks. He regularly assists companies in tracking down the source of network and application performance problems using a variety of protocol analysis and monitoring tools including Wireshark and ClearSight Analyzer. Chris also delivers training and develops technical content for several analysis vendors. He can be contacted at chris (at) packetpioneer (dot) com.

Network and Application Root Cause Analysis – Part 2 (by Chris Greer)

A few years ago, “Prove it’s not the network” was all that a Network Engineer had to do to get a problem off his back. If he could simply show that throughput was good end to end, latency was low, and there was no packet loss, he could throw a performance problem over the wall to the server and application people.

Today, it’s not quite that simple.

Network Engineers have access to tools which give them visibility into the packet level of a problem. This detail in visibility often requires them to work hand in hand with application people all the way through to problem resolution. In order to find performance problems in applications, the network guys have had to take the TCP bull by the horns and simply take ownership of the transport layer.

What does that mean?

First, it means that “Prove it’s not the network” isn’t enough, as we have already mentioned. But it also means that analyzing TCP windows, TCP flags, and slow server responses has fallen on their shoulders. Since this is the case today, let’s look at a quick list of transport layer issues that the Network Engineer should watch for when analyzing a slow application.

Working during the last week in December? Here's a list to keep you busy (by Chris Greer)

So you drew the short straw and you're on deck to support the network from December 24th-Jan 1st.

First - Sorry, that's a drag.

But, it's also a great time to do some much needed network baselining, documentation, and troubleshooting. So while you are sitting at work recovering from your egg-nog hangover, take advantage of the fact that there is minimal traffic on the network, server use is low, and most of the users are gone. In a word:

BASELINE!

It’s time to get a fresh baseline. Not only because of the lower network usage, but because it’s a great way to go into the new year, when new problems, expansions, upgrades, and rollouts are bound to occur. Who really has time for a baseline once things go into full swing in 2013?

Troubleshooting is Faster with Real Visibility - OptiView XG v9 - (by Chris Greer)

One report shows that two-thirds of network engineers regularly troubleshoot problems in an unfamiliar infrastructure, with unclear visibility, old documentation, and scattered connections. Not only does this lengthen the time it takes to resolve network problems, but it also involves a whole lot of guesswork, which can prevent getting to the root cause of the issue. This means that it may stick around for awhile.

With so many people in this situation, Fluke Networks came up with some handy visualization tools, which are available in the new OptiView XG Network Analysis Tablet v9 software. This version includes two information packed features, the new Network Navigator function and the improved Path Analysis tool.

Network Navigator
Since visualizing the network is such an obstacle for troubleshooting, an as-built, usable network map would be great for getting a handle on how things are connected and configured. Using CDP and LLDP data, The Network Navigator allows network engineers to simply connect to a root switch and instantly see all connected switches and hosts, with additional drill down data available per connection.

This data is critical when troubleshooting problems because it instantly gives network engineers an as-built map, allowing them to see what is really there, rather than following cables and relying on old documentation.

PoE Challenges and How to Troubleshoot Them (by Chris Greer)

Now more than ever, IT Organizations are making use of Power over Ethernet (PoE) to power end devices such as VoIP Phones, Wireless access points, and even NetBooks. These are often critical business systems that drive important communication services. As the demand for PoE goes up, so does the demand for the amount of power provided. Video phones, security cameras, alarm systems, and 802.11n AP’s have driven the need for high power standards, delivering up to 25.5W to these end devices.

Since PoE is driving such critical systems, it is very important that during the installation and deployment stage it is load tested and verified before powered devices are connected. In most environments, testing PoE during deployment is limited to connecting an AP to the switch and seeing if it powers up. Often this is done only for a port or two, and does not take the cable infrastructure, powered device type, nor power standard into consideration. For example, I can walk around with an 802.3af AP and see if a PoE switch will light it up on different connections, but there are several questions that this test does not answer. Is the switch providing standards-based power? How much power is really provided? Is this whole circuit going to support PoE Plus, the next generation of power? Will it support 25.5W?

We see that turning on an LED on an AP doesn’t necessarily properly test PoE. Nor does it check the common challenges experienced with PoE, including:

1.    Incompatible Technologies – Vendor Proprietary vs. Standards Compliant.
2.    PoE Power Budget Limitations
3.    Cabling is too long, or incorrect for PoE Type. 802.3at requires at least 5e
4.    Bad cabling – Opens, shorts, length issues.
5.    Misconfiguration on the PSE – PoE is not always on by default. Not exactly Plug and Play.

The interesting part about all of these installation headaches is that they can all be detected and resolved during roll out. All we need is the proper tool.

What's slow? Server, Client, or Network? Find out faster than ever. (by Chris Greer)

Introducing the new 1 Click NetTest – Fluke Netwoks OptiView XG v8

Is the problem in the server, the client, or the network? Classic question, but with more complicated networks and applications than ever before, the answer is still tough to find. Since upgrading links to 10Gbps isn’t making things 10 times faster, many IT organizations are being required to dig deeper in the real underlying cause of network and application performance problems. To do so requires Network Engineers to take on another layer of responsibility and run application and SNMP tests against the server environment, proving that a problem either is or is not rooted in the network.

To help Network Engineers get to the root of the slowness problem faster than ever, Fluke Networks has launched a great new test for the OptiView XG v8 – the 1 Click NetTest.

More AP’s Are Not Always Better! (by Chris Greer)

Too Many Cooks Spoil The Soup.

Wireless overkill. That is what happens when there are way too many AP’s in a certain area. Or, when the AP’s are configured to scream at the loudest decibel possible and their voices overlap by miles. Well, not miles, but you know what I mean.

When we are rolling out more wireless infrastructure to support the endless growth of mobile devices – think tablets, phones, scanners, and other devices – it is often the assumption that more devices requiring more bandwidth (voice/video) with more critical traffic being delivered will require more AP’s. Well, that may be true in some cases. But in several environments, more AP’s doesn’t mean that we will get better wireless service.

Before changing anything on the wireless environment, know what you have, and what you need. Using a tool like the Fluke Networks Aircheck, we can quickly see what AP’s are in an area, and how much signal can be detected in a given area. This is an important step to take before deploying more AP’s, because it may be that our coverage is fine for the number of users we will realistically support. While doing this analysis, be sure to look at the interference levels to monitor not only the signal strength, but the quality. We can have great signal strength in an area, but if we have a lot of interference, then throughput will be low, and this will affect performance.

Is Enabling SNMP Worth the Security Risks? (by Chris Greer)

Does SNMP open a hole? Well, kindof. Is it worth it? If it's configured right, YES!

The data that can be collected from switches and routers via SNMP for monitoring and analysis is completely invaluable. Information such as port utilization, device connectivity, errors, packet drops, discards, and other critical network health statistics are available with SNMP, but only if it is enabled!
Most switches and routers these days (even the cheaper home networking ones) support SNMP.

However there is still the idea floating around out there that enabling it will make a network vulnerable to security attacks. Also, some published material out there in cyberspace makes the claim that enabling SNMP will make your network unsecure. It is true that enabling SNMP on switches and routers will open a door into their management, however, there are a couple ways that we can lock up SNMP pretty tight and get the full benefits of having monitoring on the network.

1.   Community string with access list.

Fluke Networks’ New LinkRunner AT – Six Essential Tests in under 10 Seconds! (by Chris Greer)

Fluke Networks just announced the LinkRunner AT Network Auto-Tester, targeted for network technicians who need to quickly troubleshoot or verify connections quickly. There are six essential tests that the LinkRunner AT will execute with the touch of a button, including DNS/DHCP Server availability and performance, TCP connectivity to key servers, and Switch Port identification including a PoE test. All these critical tests run in less than 10 seconds on most connections.

What’s so great about all of that? Well, think about it. No need to lug around the laptop, fire up the DNS performance tester, TCP Tester, and other software needed to accomplish thorough testing results. In fact, very few laptops if any can do a Power over Ethernet, Cable Test, or an auto-negotiation test, so there are tests on the LinkRunner AT that cannot be easily reproduced. Also, time is of the essence when a problem strikes in most networking environments, due to the fact that network downtime or slowness impacts productivity. Sharing information between IT departments and getting to the root of connectivity problems quickly is a top priority. The LinkRunner AT is designed to do just that.

The LinkRunner AT features a full color VGA display and can store up to 50 test results for documentation and sharing information with escalation teams. Training on the tool is also a snap, getting techs up and running in no time. In fact, the tool is so easy to use, that one customer case study said that regular company employees were able to use it to troubleshoot issues when IT Staff were out of the office! Wouldn’t that be nice?

This tool is fast, compact, great battery life, and easy to use. What more are we looking for in a tool for network techs?

For more information about the Fluke Networks’ LinkRunner AT, please visit www.flukenetworks.com/linkrunnerat.

Author Profile - Chris Greer is a Network Analyst for Packet Pioneer. Chris has many years of experience in analyzing and troubleshooting networks. He regularly assists companies in tracking down the source of network and application performance problems using a variety of protocol analysis and monitoring tools including Wireshark. When he isn’t hunting down problems at the packet level, he can be found teaching various analysis workshops at Interop and other industry trade shows. Chris also delivers training and develops technical content for several analysis vendors. He can be contacted at chris (at) packetpioneer (dot) com.