LoveMyTool - Open Community for Network Management and Monitoring

How many errors are acceptable on switch and router ports? (by Chris Greer)

Quick answer? Nada, Zero, Zilch! (Sure there are exceptions – but you wanted a quick answer, right?)

I recall reading a whitepaper one time which gave target average utilization and error levels for Ethernet ports. In the report, it mentioned that a target level for traffic should be 40% on average. I was ok with that number, seemed normal to me. The next figure given was for errors (like packet drops, FCS Errors, late collisions, etc…) and the acceptable figure for a healthy network was 1% of port utilization. So according to the whitepaper, if the utilization rate was at 40%, then .4 percent of that traffic could be errors, and all would be ok.

HUH?

So to put some real numbers to that, a 1Gbps network interface with 40% utilization would be 400Mbps. If the average packet size was around 350 bytes, this would mean that for any given second, there would be just over 142,000 packets flying by. So if 1% of these had errors, that would be 1,420 packets per second. As we have discussed on Love My Tool in the past, when a packet has an error, it is dropped by the next switch or router port it crosses. Since Ethernet is not responsible for retransmission (that’s TCP’s job), the switch or router does nothing to report this issue, other than add a counter to its error log. This means that for every packet drop, discard, FCS error, late collision, or long/short frame, we have to wait for TCP to timeout and retransmit, which takes time. These errors have a huge impact on network and application performance, and can really undermine the business.

How can we find them?

To put it simply, look at our routers and switch ports for errors. Does this mean that we have to comb through every router and switch, doing a “show interface” on all ports? If we don’t have a tool to do it for us – yes, we do. That can get a bit tedious, so some tool manufacturers have an SNMP based interface scanner that can comb your whole enterprise for errors, presenting them front and center so you can resolve them. The Fluke Networks OptiView XG is one such tool. It scans all known switches and routers, and if it finds Ethernet Errors, packet discards, or discovers excessive utilization, it will alert the front page, displaying the switch and port number.

Cool huh?
Don’t let errors get you down. Find the little buggers and resolve them, before they impact critical business applications.

Author Profile - Chris Greer is a Network Analyst for Packet Pioneer. Chris has many years of experience in analyzing and troubleshooting networks. He regularly assists companies in tracking down the source of network and application performance problems using a variety of protocol analysis and monitoring tools including Wireshark. When he isn’t hunting down problems at the packet level, he can be found teaching various analysis workshops at Interop and other industry trade shows. Chris also delivers training and develops technical content for several analysis vendors. He can be contacted at chris (at) packetpioneer (dot) com.

Lawful Interception: What is it and why is it important? (by Jay Botelho)

On the heels of the year’s much discussed email phishing campaigns, cyber attacks, and data breaches came one of the biggest security conferences of 2011: ISS World, Kuala Lumpur. ISS World presents methodologies as well as tools that help bridge the chasm between how intercepted data is lawfully gathered to using this information for actionable intelligence. Attendees of the conference commonly include law enforcement, intelligence and interior security analysts, as well as telecom operators who perform, manage, and address the lawful interception of network data for electronic criminal investigations. Top sessions for 2011’s conference included “Top Ten Internet Challenges Facing Law Enforcement and The Intelligence Community” and “Guru Panel on Best Practices for Deploying Intelligence Probes and DPI for Lawful Interception”.

Aside from its predominance at ISS World, Lawful Interception has been a hot topic since its inception in 1968, back in the Plain Old Telephone Service (POTS) era, with the Omnibus Criminal Control and Safe Streets Act of 1968 (this pertains only to the US). Today, with human rights groups and occasionally governments fearing that the technology used in Lawful Interception has the potential ionto be detrimental to both individuals and countries, i.e. this technology can easily be misused by totalitarian regimes, and for a cyber-arms race, the use of Lawful Interception by law enforcement agencies has been hotly contested.

So, then, what is Lawful Interception?

Analyzing slow applications? Capture on both sides of the connection (by Chris Greer).

When firing up Wireshark (or any other packet analyzer for that matter) to capture traffic on a slow application, one of the first questions that usually comes up is –

Should I capture on the client side or server side?

For years now, this question has been debated among analysts and continues to be a matter of preference, experience, and all too often, timing. Sometimes we don’t have the time or gear to properly capture on the server side, so we are left to the client end.

So which one should we capture on? Well, as a packet geek who wants as much information as possible about a problem, I say BOTH! The best case scenario is to setup a capture on the client end, as well as the server end, simultaneously capturing traffic while the problem is occurring. At times, this may be the only way to pinpoint the real underlying problem.

How can we capture on both sides?

Slow Applications? Don’t Overlook Packet Loss (by Chris Greer)

Sometimes when we are tracking down the cause of performance problems, we can get lost in the nitty gritty details. Especially when dealing with a multi-tiered application, a complex array of load balancers, application accelerators, or the many virtualization technologies, troubleshooting a slow application can be a daunting prospect. 

But never fear. When we are looking for that golden nugget, the ultimate needle in the data center haystack which will resolve our priority-10 performance problem, we may overlook some of the heavy hitting speed-killers which are easy to spot with the right tool. So next time a problem strikes, resist the urge to pour through user privileges in the domain or dig through the back end SQL data for the bad call, and check out a few of the basics. One of which is packet loss.

Packet Loss – Really? In Today’s Networks?

Real Testing – A Real Solutions Full Validation! (by The Oldcommguy)

Chris, Tony and I are always saying “Test Them, Don’t Trust Them” and Anue Systems took that to heart and did just that!

Most companies sell on their specifications alone, most if not all have of them have never been tested! What and Why do you believe them?

The Tolly Group and other test verification labs, test products to assure you, the customers that you are getting what the vendors say they are selling to you. Don’t Trust Them, Test Them!

I am writing about this because it is important to point out companies that focus on customer satisfaction as well as assuring the customer that they are getting a top quality product.

One must first decide on the parameters that are required for their solution and the solutions capability for future growth.

Then they must decide on the operational, deployment and operational difficulty level that they can handle.

In my humble opinion, Anue Systems deserves recognition for all their efforts and that is why I wrote this article. I also write about products that I have tested or worked with a customer that has done the testing and review.

As a consultant and a very avid supporter of deep network visualization access I would certainly use the Anue solution. This may seem as biased and yes it is – when I like a product and company and they have shown ethical and technically responsible behavior then I love to tell you about them! This is one of those solutions!

Anue Systems in their effort to prove that their specification and capabilities are one and the same did just that. No façade just real testing to prove that they do what they say and that buyers can be confident investing in this carrier grade network access technology solution, the Net Tool Optimizer. They wanted to again prove that they are a carrier class device and have done it with this Tolly Test along with their NEBS 3 compliance certification. One cannot be a carrier class solution without NEBs 3 compliance and fully technology verification.

Anue went to the Tolly Group and asked that they test for the real specifications required to validate the Anue NTO claims and Tolly did a great job!

Come On People ... TEST! (by Chris Greer)

Ok, I’ll explain the title by starting off with a story.

I received a call from a company about having an intermittent network outage. It was a major company that was brought to its knees for two days, almost completely unable to do business during that time. They called and I was ready to jump on a plane any minute once they said the word GO. A few days later, after receiving no response, I gave them a follow up call to check and see how things were. They said the problem was fixed, and that they had found an old router plugged into their core switch. After removing it the problem went away, leaving them feeling confident that they found the solution.

I had about a hundred questions flood into my head. How long was the router there? Did the problem start occurring only after it was connected? What type of traffic was the network experiencing during the “outage”. Were any users able to access and use applications?

Most importantly…

How do you know that you fixed it for sure? Now you may get the title. Come on people…. TEST!

Problems that are “fixed” but not tested and understood are doomed to repeat themselves.

This is not an isolated story. When a problem strikes, people may start scrambling to find a fix and cross their fingers after it is applied. Instead of truly testing that the root cause was resolved and understanding why the problem caused the symptoms in the first place, network techs may rely on perception from the users instead of factual results from testing. If a problem all of a sudden goes away, how do we know it won’t come back? How can we setup monitoring tools that will tell us when the problem occurs again instead of hearing it from the users?

All these questions have solutions. I’m not writing this article to sell the reader on one tool or another. But I am selling the reader on testing, analyzing, fixing, then testing again. This process will truly address the root cause of network problems and will help them go away for good.

If you have any specific network issues you would like to see discussed on Love My Tool, please comment below!


Continue reading other LoveMyTool posts by Chris Greer »

The Cloud - A Reality Check (by Owen O'Neill)

Cloud Computing and the Data Capture Infrastructure 

In its simplest definition Cloud Computing is “on-demand access to virtualized IT resources that are housed outside of your own data center, shared by others, simple to use, paid for via subscription, and accessed over the Web.” ¹    

 
Organizations considering a shift to this technology, despite the obvious appeal of shifting significant capital expenses to the operating expenses side of the ledger, must examine a wealth of issues related to the capture and monitoring of database and application traffic.  Troubleshooting, security, performance monitoring, capacity planning and compliance in the cloud are some of the new challenges this technology creates.  Getting access to the network traffic to perform this analysis is the job of the Data Capture Infrastructure (DCI).  The DCI is a system designed to copy traffic running through a computer network and send that traffic to probes or analyzers that can improve network efficiency or security.

Looking beyond the hype and promises of reduced costs and increased efficiencies, the network administrators, operations and security engineers, managers and other decision makers responsible for ensuring reliability and security in the enterprise environment have fundamental questions to answer.

• Do requirements for troubleshooting network operation issues decrease?
• Who is responsible for ensuring application and database performance?
• Will the cloud provider ensure security monitoring and regulatory compliance?
• Are changes in Data Capture Infrastructure (network monitoring access systems) required?
 

Is “the Cloud” Really That New or Different?

WOW!! Finally Someone Actually Verified Their Claims (by Tim - The Oldcommguy)

Tolly Group Certifies GigaStor 10 GB as Fastest FDX Long-Term Capture Device

The FIRST fully tested Network Packet Recorder Certified at 10 Gb FDX Wire Speeds to Help Companies Validate and Manage the Delivery and Performance of Critical Applications

Remember my motto – Do Not Trust Them, Test Them! Here is a great example of Success!

Well I hope that Network Instruments testing will get everyone thinking that important specifications on paper are meaningless! If a customer has an important requirement – Do not trust marketing hype and specmanship, Verify and Test or rely on a trusted verification vendor!

There are 3 types of companies, in my opinion, that when it comes to marketing their products –

First – Ethical Companies like Network Instruments that have products that MUST meet certain criteria to support the market and customer needs. These companies go out and verify the real requirements of their high tech products.

Second – Companies that are ethical and market their technology with focus on proof in service to their customer’s needs. They focus on truth and satisfying customers and fulfilling their needs with onsite verification and continued support.

Third - companies that market their products by marketing hype, usually spending most of their time coming up with unverifiable specifications and outlandish claims. They are the ones in to just get an order at all costs and move on.

Steganography - The Technology for Hiding (by The Oldcommguy)

What did you say?  Bless you? 

I have decided to write several articles on the Cyber Forensics World for the LMT Readers. 

The first article will be on Steganography or How to hide in Plain Sight –

 Steganography comes from the Greek word steganos or hidden/concealed writing and dates back to Greek times but did not become readily known until 1499 when Johannes Trithemius (German) wrote his book Steganographia which turned out to be a thesis on cryptography and steganography disguised as a book on magic.

 Today, Steganography gives one the ability to hide/conceal something like data, documents or pictures (the payload) inside other written material or pictures.

It is rapidly becoming a preferred method of passing/delivering criminal and or terrorist written material and pictures without easy recognition.

 It is in essence a form of magic with words, numbers and pictures as well as physical objects and of late data networking.

 There are several methods of data networking Steganography –

Wireshark Quick Tip: Setting a Capture Filter (by Chris Greer)

 

 

Have you ever had to dig through huge trace files looking for traffic from just one address? Sure, it’s a piece of cake to set a display filter and look for the device that way, but in some situations, it’s just easier to set a capture filter from the get-go. A capture filter is a filter that is set in Wireshark before the capture has begun. It allows only traffic that meets the filter to be allowed into the capture buffer. They are easy to set, but very easy to over-look. Sometimes the best use of a capture filter is to capture everything except a certain protocol or address.

Before going much further into this topic, I’d like to stress that at the beginning of any application analysis, I’m not a huge fan of capture filters, just because you may inadvertently filter out traffic that would otherwise help to solve a problem by setting a capture filter. For example, what if a filter was set just for a certain IP address, when in reality, that address or station had nothing to do with the underlying problem. So be careful only to set capture filters for traffic that you are sure you isolate and capture.