LoveMyTool - Open Community for Network Management and Monitoring

Application Performance Monitoring is all about the Users (by Jay Botelho)

Whether your users are accessing email from a local server, browsing the web, or working with applications running in a virtual server (local or as-a-service), it’s imperative to constantly track and monitor these events for all users on your network. In other words, you need to be performing Application Performance Monitoring (APM). And this doesn’t need to be complex – it can be as simple as noticing which IP conversations and what activities are “normal” for each user. By uncovering and recording this information, you’ll have everything you need to quickly determine when the user experience heads south.

A key metric to monitor as part of this “normal” behavior is Application Response Time, a quantitative measurement determining when applications are experiencing poor performance. Although quantitative, measurements of application response time can be made in different ways, and from different measurement points, leading to ambiguity as to exactly what is being measured. But in most cases application response time will give you a very good idea about the overall user experience, and that’s the primary goal in APM. It’s when it comes to the next step, determining the root cause of the problem, when the details of how and where the measurements are made really come into play.

SSL Key Management (by Jason Wells)

If you’re doing analysis on network captures, you’ll eventually run into the need to view encrypted data. This is especially true if you’re troubleshooting a secure web application like a shopping cart or online banking. If you’re doing this analysis in Wireshark, you’re somewhat caught between a rock and a hard place - you have to either:

• Give your private keys over to the engineers, and let them install it locally in their copy of Wireshark in order to decrypt data
• Have one location or virtual machine where the keys are installed on the machine’s copy of Wireshark, and have users share use of the machine

In the first case, you now face a situation where the very private keys used on your website - the whole reason your site is secure in the first place - now exist as multiple copies in multiple locations. Are you going to rely on policy that they be deleted? Or only limit access to one support engineer?

In the second case, you face the normal difficulties with trying to use one shared machine to operate a network tool. In addition, you must now secure this machine (likely a simple workstation) from intrusion to a degree necessary to protect one of your most vital assets - getting your SSL keys leaked is bad, bad news.

Pinpoint Network Bottlenecks (by Jim MacLeod)

Theory of Constraints (TOC), as popularized by the business novel The Goal in 1984, and recently resurrected for DevOps and IT in The Phoenix Project, holds that any given system is limited in throughput by only a few key bottlenecks. Improvements anywhere else won’t speed things up, but improvements at the bottleneck will have a dramatic impact on the whole system.

On a network, there are two kinds of bottlenecks: bandwidth and latency. While these concepts are familiar to most of us, I’d like to highlight them as a base for the techniques in this post. Bandwidth is the ability to move a large amount. I like to think of bandwidth as a cargo ship: lots of containers, lots of capacity, but it takes days to make the journey. Latency is more like a courier: get a small package there as fast as possible. While there’s some overlap between the two – for example, airplanes are fast and have lots of storage – the two concepts have different effects on your data, and different data has a different mix of demands.

Latency

Latency is the slowness of moving data. It’s measured in time, and adds up from end to end. Once you lose the time at one location, it’s impossible to make up for it elsewhere. That’s why finding a bottleneck is so important: a single device which adds significant latency slows down the entire trip.

Packet Capture in a Virtual Environment (By Jim MacLeod)

Spend enough time around virtualization and it becomes clear: these were tools built for server folks, and the networking was added on as a necessary evil to move data. The focus within VM network configuration is simplicity rather than actual control, and critical monitoring stats are next to non-existent. Fortunately, things have gotten better: Open vSwitch supports port mirroring, and the feature was added to VMware in version 5. Grab your favorite packet sniffer and read on to learn about sniffing VM networks.

Scenario 1: Inter-VM in the same server

The simplest VM scenario is a single server, and already there’s traffic that’s traditionally been hard to sniff. Packets between VMs in the same server almost never leave the box, which means that a tap or a physical switch span port isn’t going to work.

Fortunately, there are two straightforward solutions. First, the virtual switch itself may support a span port. That means that it should be possible to designate a NIC on the vswitch as the target for the sniffed traffic. This gives two choices: either designate a vNIC on a VM on the server, or designate a pNIC to forward the packets to an external sniffer.

Packet Payloads Important for Troubleshooting End User Experience (by Jay Botelho)

Often times you are faced with a problem that’s reported by your network monitoring solution, but you simply cannot identify the root cause. Based on NetFlow analysis you can isolate the flow(s) in question, maybe even look deeper into overall latency performance and configuration settings like QoS, but even with that there’s still just not enough information to determine the root cause. To be more specific, let’s say you have an end user who is reporting problems with a slow-moving application. You have exhausted your flow-based analysis and confirmed that overall latencies are excessive, but you can’t see why. So, what’s next? This is where packet payloads come in.

What Is a Payload?

In aerospace/military jargon, where the term originated, the payload is the carrying capacity of a particular delivery vehicle. For example, the payload of the space shuttle includes astronauts, their experiments, and equipment to be delivered to the space station. In networking, although the scale is quite different, a payload is very analogous. It is the data that is being carried within a packet or other transmission unit over the network. Simply put, the payload is the bits of meaningful data that get delivered to the end user sans the delivery data that makes an application work. Without payloads there is no useful information to communicate, kind of like sending empty envelopes via snail mail.

Working during the last week in December? Here's a list to keep you busy (by Chris Greer)

So you drew the short straw and you're on deck to support the network from December 24th-Jan 1st.

First - Sorry, that's a drag.

But, it's also a great time to do some much needed network baselining, documentation, and troubleshooting. So while you are sitting at work recovering from your egg-nog hangover, take advantage of the fact that there is minimal traffic on the network, server use is low, and most of the users are gone. In a word:

BASELINE!

It’s time to get a fresh baseline. Not only because of the lower network usage, but because it’s a great way to go into the new year, when new problems, expansions, upgrades, and rollouts are bound to occur. Who really has time for a baseline once things go into full swing in 2013?

Remote IT Support Provides Multi-Function Efficiency (by Glenn Gray)

Multi-tasking is a way of life for IT pros. Without the ability to effectively multi-task, system administrators can quickly end up buried under a stack of help desk tickets. Just like the human ability to multi-task, time-saving system administration tools include features that allow IT pros to perform multiple tasks without having to switch between software consoles. To be clear, there are many tools on the market that perform only one task and perform it well, but there is a clear movement in the industry toward function aggregation in system administration tools that promises to cut down on the software bloat seen on just about any IT pro’s workstation.

Using single-function system administration tools increases the amount of time needed to complete your daily tasks as an IT pro. It also increases the time it takes to train yourself and your staff to become proficient as system administrators. So, if you are looking for a quick boost in productivity for you and your staff, look to admin tools that let you perform multiple functions from one interface or include other time-saving features.

The Cure for Forklift Sniffing: Multi-Segment Analysis (By Jim MacLeod)

Packet analysis is a great method for finding network problems. However, while a single point of capture can verify that a problem exists, it’s often not enough to determine the location. For too long, troubleshooting has relied on reckoning whether the problem is towards the server or towards the client, then moving to a different capture point and hoping to observe the issue again. If network upgrades which require new hardware are “forklift upgrades,” then moving the packet capture point is “forklift sniffing” – and it’s clearly not a great methodology for packet analysis. What’s needed for any network larger than 1 switch or wireless access point is the ability to capture at multiple points simultaneously.

Forklift Sniffing

Single-point packet capture has the advantage of appearing to be relatively simple. It’s an on-demand activity: when there’s a network problem, pull the packet sniffer out of mothballs, plug in somewhere, and start capturing. That assumes that you have somewhere to plug in to capture. Today’s switched networks make packet capture an interesting challenge, but most of them support SPAN. So, plug in the sniffer, then log in to the switch to enable SPAN on your port, and hope that the problem is still happening. Analyze the data, and then see if you can determine what the problem is. You look at TCP to determine whether there are missing data packets (problem is closer to the server) or whether there are missing or delayed ACK packets (problem is closer to the client). Either way, the problem isn’t where you’re connected, so unplug, move to a different location, and try again.

IPv6 Adoption Challenges (by Jim MacLeod)

So far there have been two World IPv6 Days, and two of the five Regional Internet Registries (NICs) have reached IPv4 exhaustion, so everyone has deployed IPv6. Wait, they haven’t? Here’s why.

IPv6 isn’t Compatible

IPv4 is the primary protocol of the Internet. The Internet changed the way we interact with data, with the world around us, and with each other. It is a primary utility as real as water and power.

The primary problem with IPv6 is that it is not directly compatible with IPv4. Transition mechanisms are required to enable IPv6-IPv4 communication, adding an additional level of complication over the protocol transition It is a similar problem to adopting hydrogen-powered cars: people won’t buy them until there are refueling stations, and the stations won’t get built until enough people buy them. However, the problem is more complicated with IPv6: imagine if hydrogen-powered cars couldn’t use the same roads as gas-powered cars. “Transition mechanisms” would consist of car-hauling trucks or changing cars partway through the trip. Even if such mechanisms were automated, there would still be hesitation about adoption.

Network Monitoring and Analysis Strategy for the Cloud (by Jim MacLeod)

The cloud is a growing trend in outsourced computing power. On the upside, services can scale horizontally to handle load – but the cost increases with the number of servers. On the downside, there’s less visibility into application and network internals. The paradox is that, while it costs more money to run more servers, it’s more difficult to get the data necessary to troubleshoot and optimize to reduce the need for more servers.

While there are tools to allow packet capture in virtual environments, they’re generally not available in a cloud. Cloud providers won’t give you access to virtual taps, because public cloud multitenancy will expose data from multiple cloud customers. Fortunately, there are situations where cloud packet-level analysis is still possible, by focusing directly on the endpoints.

The goal of packet-level analysis in the cloud generally focuses on two things:

1. End-User Experience
2. Application efficiency of cloud-hosted services