When an Upgrade Results in a Downgrade (by Tony Fortunato)
LMTV LIVE | Cyber Security for SCADA Networks (with David Thomason of Thomason Technologies)

The Need For Speed – The IT Top Gun (by Keith Bromley)

The Need For Speed – The IT Top Gun

Being able to deliver a rapid response to problems is a core need for the modern enterprise. While many network “issues” can suffer a delay in response time, true problems like security breaches and network outages cannot. This where an “IT Top Gun” can really shine with fast response times to anomalies and problems that eliminate issues from the onset, or better yet, prevent them from happening.

For instance, in the case of a suspected security breach, you need know if you were actually breached or not, where the breach occurred, and what was compromised. According to the 2016 Verizon Data Breach Investigations Report (DBIR), almost 68% of breaches happen over the course of several days, so a rapid response to security threats can definitely help minimize the cost of a breach. The faster you isolate the attack vector, the faster you can limit the amount of damage to your business. However, according to the Ponemon Institute’s 2015 Cost of Cyber Crime Study, it is actually taking businesses longer to resolve cyber-attacks. It now takes an average of 46 days to resolve a cyber-attack, which is one day longer than it took last year. This also represents an increase of 30% over the last six year period and results in a corresponding increase in the cost for a cyber-attack.


The situation is also serious for network outages. According to the 2016 Cost of Data Center Outages study conducted by the Ponemon Institute, the average cost of a data center outage is $740,357 and lasts for about 95 minutes. This results in a cost of $7,793 per minute of downtime. A rapid response is needed in this situation as well to control costs and keep your mean time to repair as short as possible. 

So how does one become an IT/Network Top Gun?

The starting point is network visibility, every bit, byte and packet. You need clear visibility into what is and is not, happening on your network. Visibility involves creating an architecture that enables proper full data access through taps and bypass switches. Then you need to be able to “filter” that data with a network packet broker (NPB) to send to the correct network security, management, performance listening devices.

Basically, the NPB lets you do the following:

  • Filter data so you can remove extraneous information, to focus and send only the necessary data to the different monitoring tools for analysis
  • Remove packet clutter by getting rid of duplicate and unnecessary data
    • Make sure that you do not have duplicate packets in your network before filtering. Different routes of access cause applications to fail!
    • Usually one will have duplicate packets when using SPAN as network visibility access, use a TAP!
  • Perform packet slicing to reduce unnecessary information (like MPLS) that your monitoring tools don’t like
    • Also to keep only the parts of the packets you need for analysis – This is a Security issue as well as a storage issue!
    • Do not allow your corporate secure data to be seen!

Once you have the right data, you need to optimize your processes and procedures. A visibility architecture helps you here as well. Once you’ve installed taps and an NPB(s), many of your Change Board approval requirements can be simplified, if not eliminated, because your troubleshooting activities no longer require any changes to live network data. By eliminating this step, you can save a minimum of hours (and usually days or weeks of time) in your troubleshooting activities. Ixia has several case studies documenting this time savings.

With the old process (not using a visibility architecture), once Change Board approval was granted, IT often had a crash cart with a selection of tools that could be used. Since these tools are a shared resource, another common problem was that the tools would need to be configured and calibrated before use because someone had changed the settings and then not reset the tool when done. For emergency situations, this would often have been another source of delay and confusion for IT while trying to resolve a crisis situation.

The use of “floating filters” within the NPB are another time saving activity. Floating filters are typically debug filters that are created and attached to forensic monitoring tool(s) but the filter is not active – it’s in standby mode. However, should you need it, it available and can be connected in less than a minute to an incoming data stream. In addition, you can connect the floating filters remotely with the management system when needed. This gives you 24/7/365 diagnosis capabilities from remote locations and allows you to conduct lightning fast troubleshooting activities. One example is that you could filter data and then send it immediately to a sniffer to start a debug process for a troublesome network link. Another example would be that if you suspect you have an application issue, specific application data can be directed to a filter in standby mode and then sent to a waiting application performance monitoring (APM) tool.

Automated response capabilities are a third useful feature set of a proper visibility architecture. This is a proactive approach used to efficiently minimize security threats and dramatically decrease the MTTR for your network. Faster responses to problems result in a shorter mean time to diagnosis and a corresponding faster MTTR .If automation is implemented correctly within an NPB, the packet broker lets you maximize the capabilities of your monitoring tools without changing your processes. Basically, a proper implementation of automation lets the packet broker conform to how you need to use it, not the other way around.

Well-designed NPBs should have a RESTful interface. This interface allows you to connect the NPB up to a network management system (NMS), security information and event management (SIEM), or an orchestration/provisioning system. Once the systems are interconnected, the Centralized system can send commands to the NPB that control the NPB’s actions. Again, this creates a rapid response capability.

For example, you have a SIEM tool that spots some anomalous traffic on your network. The SIEM can send a message to the packet broker instructing it to capture the anomalous traffic and send it to an intrusion prevention system (IPS) to analyze the anomaly. An IPS analyses the packet captures to determine if they are threats or not. Since you have previously connected your NPB inline after your bypass switch, the SIEM can instruct the packet broker to divert an anomalous packet stream to a honeypot. From this honeypot, you can now control the system access that the intruder has and begin to understand where the threat is coming from (e.g., who sent the threat and where they are located geographically), the threat attack used to gain entry into your system, and the nature of the attack the intruder had intended (e.g., defacement of the website, crashing of the website, theft of corporate intellectual property, etc.).

Once you have your architecture in place, you’ll be able to reap the financial benefits. Ixia has found several customer implementation that have resulted in the reduction of MTTR figures by up to 80%. This reduces downtime costs. In addition, security breach decreases costs can be decreased as well. Finally, less outages and better network performance increases customer quality of experience (QoE) and reduced SLA penalties.

Once you combine the security architecture with the visibility architecture, you will equip yourself with the necessary tools to properly visualize and diagnose the problems on your network. You can find an overview of visibility architectures and security fabrics here. There is also a whitepaper that delves into the rapid response discussion in more detail.

KeithAuthor:Keith Bromley is a product marketing manager for Ixia, Inc., with more than 20 years of industry experience in marketing and engineering. Keith is responsible for marketing activities for Ixia’s network monitoring switch solutions. As a spokesperson for the industry, Keith is a subject matter expert on network monitoring, management systems, unified communications, IP telephony, SIP, wireless and wireline infrastructure. Keith joined Ixia in 2013 and has written many industry whitepapers covering topics on network monitoring, network visibility, IP telephony drivers, SIP, unified communications, as well as discussions around ROI and TCO for IP solutions. Prior to Ixia, Keith worked for several national and international Hi-Tech companies including NEC, ShoreTel, DSC, Metro-Optix, Cisco Systems and Ericsson, for whom he was industry liaison to several technical standards bodies. He holds a Bachelor of Science in Electrical Engineering. 

Keith has other popular articles on WWW.Lovemytool.com - and on Ixia.com