??Don’t Network Engineers have enough problems with GOOD packets?? (by Tim O'Neill)

You want to save bad packets and review them to the bit level?

I hear a lot of ideas and I love most of them as they lead to better products, better statistics and information. This was recently asked of me - I want to capture and save every packet – good and bad/invalided packets for later review. In reality to capture every Bit, Nibble and Octet including the InterFrame Gap!

Now this is a huge task!

By – Tim O’Neill – The Oldcommguy™

Special Thanks to Maureen Rozenhart of Anue Systems for her support!

Some initial thoughts!

I agree that in today’s world of Compliance, Security..Etc one should and needs to save lots of data, maybe even all the data for a long time, maybe even for years.

I do not agree with saving the InterFrame Gap (IFG) and bad packets. I believe that one should just be able to save the bad frame statistics and only save the real data frames for later review, evidence and loss mitigation.

Why would one save the Bad Frame Statistics? The only reason for reviewing Bad Frames, in my humble opinion, is for hardware developers that want to see how their firmware and drivers are functioning in a real world environment. There are certain security reasons where a hardware analyzer is needed but this is a subject for future discussions.

Everyone needs to do baselining on a regular basis to make sure that the application layers (layer 4 up) are running on a solid foundation. The Foundation of a data network are layers 3 and down. If one is getting lots of bad frames and thus lots of retransmissions the network and thus the application efficiency is going to be lower.  Bad Frame counts and Retransmissions are only a few of the important items in a Base Line Analysis but are essential in any analysis.

It is the old sage – Do not try to build a 5 story network on a one story foundation. So Baselining is an essential element for day to day and maybe even hour to hour in a good monitoring plan.

What is needed -

So what tools do you need to carry out this VERY demanding task of Capturing every bit, being able to Review in a filtered and focused format (with relevant and comprehensive statistics) and be able to Analyze and Look at all the data at the bit level…..

1)         Capturing large amounts of data for historical and diagnostic view – I recommend the reasonably priced GigaStor from Network Instruments with its full capture capability.

2)         To Filter, Focus and Review the data - I recommend the Anue Systems Net Tool Optimizer. It is reasonably priced and has a full and comprehensive GUI that anyone can use effectively to look at any data characteristics.

3)         To look at the data, every bit of it - I recommend the Absolute Analysis Investigator. The AAI is a full bandwidth hardware based analyzer that even sees the IFC and all the bad frames/bits, which we cover in more detail below.

a.      If one needs a higher level focused analyzer I suggest the Network Instruments Observer.

There are other products that can capture, filter and analyze but those I recommended above are my favorites.

Please remember that to carry out this huge task out one must have REAL Access to the network, through a TAP where every bit, nibble and octet is passed regardless!

My favorite TAPs are from Network Instruments and Network Critical, depending on the final need.

What is a Bad Frame and What types are there?

There are different types of Bad Frames – Here are the main ones that should always be included in a Baseline Analysis, alongside your data for full relevance.

Short Frame (Runt) – Any frame under 64 bytes (512 bits).

Long Frame (Frame to long) – Usually any frame that has more than 1,518 bytes or one that has exceeded the MTU (Maximum Transmission Unit typically 1500 bytes in Ethernet).

Symbol Error – this is the number symbols (bytes) received that the system could not decode correctly.

Fragments – This is where there is a count of the fragments sent over the network. Fragmentation occurs when a packet is too large to be sent in a network link. When the packet/frame is too large, the original packet is split into smaller packets that the network can handle; each fragment contains enough information to allow the destination device to reassemble them back to their original state/size at the conclusion of the transmission. Fragmentation typically is initiated by a router. Once a packet is fragmented, it will only be reassembled by/at the destination device.

Collisions – A Collision is indicated when two devices detect/sense that the network is idle and try to send packets at exactly the same time (within one round-trip delay). Because only one device can transmit at a time, both devices must stop sending and attempt to retransmit. 

There are 4 collision types –

1.     Single - packets sent after one collision after wait period

2.     Multiple - packets sent after multiple collisions

3.     Late - packets aborted during sending because of collisions after 64 bytes

4.     Excessive - packets not sent because of too many collisions

Alignment Error - Frames are made up of a whole number of octets (FCS). If a frame arrives and part of an octet is detected missing (number of bits not divisible by 8), this generates a Frame Check Sequence (FCS) error; this is labeled as a Alignment or Offset Error.

Frame Check Sequence error (FCS) – Similar to the CRC error, an FCS error indicates that the frame received did not have an integral number of octets.  The CRC below will tell if any of the bits in the frame have been corrupted in transit.  

Errored Frame Check Sequence - A Cyclic redundancy Check (CRC) is the polynomial remainder calculated mathematical image of the Frame to confirm total frame integrity. It is a combination of FCS and Alignment errors. These errors below indicate how the packets were received with which error - 

• A bad FCS and an integral (whole) number of octets (FCS Errors)
• A bad FCS and a non-integral (not divisible by 8) number of octets (Alignment Errors)

The CRC is a 4-byte polynomial mathematical code checksum remainder value. The CRC is calculated by the source station.  The CRC calculation is on all the bits in the frame from the Destination MAC Address through the Pad fields (that is, all fields except the preamble, start frame delimiter, and frame check sequence). The source station stores the value in CRC field of the transmitted frame. When the frame is received by the destination station, an identical check is performed and compared to the transmitted value. If the calculated value does not match the value in this field, the destination station assumes an error has occurred during transmission. The frame is usually discarded by the Network Interface card (NIC). If one has more than 3-5% CRC/Alignment errors then this should be considered excessive and analysis should be performed to identify the offending NIC/physical layer device.

Note - There are other Physical Layer issues – Jams, Late or Early collisions, Dribbles, Jabbers, Giants…etc. that are not relevant to this discussion but can be important in lower layer analysis.

Most modern data filtering devices can acquire and save these errors and more as shown in the current Anue Systems Net Tool Optimizer GUI below.

Many more statistical events can be captured and shared when the RFC 2665 dot 3 and RFC 1757 MIBs are employed.

 Anue's Net Tool Optimizer easy to use and powerful statistical view of errored frames!

Now if you did save every bit, nibble and octet and you need to look at bad frames what do you need.

Well you need a hardware analyzer like the Absolute Analysis Investigator and you need access to the full, data stream to see bad frames. See picture below -  

Absolute Analysis Investigator  - Bit by Bit view including Bad CRC recognition and bad frame!

Let’s talk about saving all frames for historical review/analysis –

Is it practical to capture everything, every bit that flows – depending on one’s needs and finances, yes it is possible.

If one wanted to capture all the bits and frames in a small network with say five 10G links and one 1G link….my example

The minimum speed in this case is 1G and five 10G and they want to capture for future analysis every packet (actually every bit including InterFrame Gaps) – good and bad.

OK let’s start with capturing every packet – some basic math…

One 1G circuits = 2 Gb/Second max to save

Five 10G circuits = 100 Gb/Second max to save

This will require at least five 10G full bandwidth of 20G each capture devices and one 1G, actually 2G capability capture tool. This calculates out to 102Gbits per second capture rate.

Most if not all capture devices will only capture good Ethernet frames which are defined by the RFC as legal frames, this does not include errored, short, bad CRC…etc frames (we discussed above) because almost every NIC will drop the bad frames, especially if one is using SPAN or VACL access, the Switch and Router will drop all the bad packets for you.

Now let’s say you can afford –

1) The cost of and support of five 10G and one 1G capture engines with many Terabytes of storage. The size would depend on the time you need to keep the data.

2) That you have the time to review 6120 Gb/minute or 367,200 Gb/hour or for 24 hours 8812.8 Tb or 1101.600 TB and at an average of 512B per frame with 12B for IFG that is 2.102 trillion frames a day or let’s put that into some easily handled numbers – It is Way toooooo MUCH!!!.

Most Network Analyst’s say that to review one packet takes at least 30 seconds. So analyzing trillions of frames is absurd. So what does a network engineer do to handle this mass of data and review it effective and in a timely manner.

Ok, so this is where filtering devices come into play, like the Anue Systems Net Tool Optimizer to filter out just what you need and forget about trying to review every packet. One can save all the packets but with the new advanced filtering technology one can easily review only the suspect data or a specific application flow or conversation while using the most advance MIB’s for keeping an eye on the fundamentals of your networks efficiency and stability during the time in question.

Statistics and Focus is the Key where saving every bit, nibble and bytes may not be the best this but may be a necessity for your success.

In conclusion -

You can capture everything on your network, analyze and review the data to the bit level if you have the correct tools and the skills. Is Capturing everything needed in every network to the bit level, Yes to a point based on one’s absolute need for review and potential for mitigation of an incident. Do you need to see every bit and be able to review and analyze to the bit level No that is not for everyone. Most Network Engineers analysis is focused on good frames with statistics on the bad stuff is more than sufficient, economical, and reasonable. Focusing on Good frames seems to keep most engineers quite busy. Does everyone need TAP access to the Network, ABSOLUTELY YES! Does everyone need Advanced Filtering technology to review with a focus; Yes I think so because technology can work for you and focus on your analysis and monitoring needs to make your analysis quicker, easier and more successful.

Be sure to check out my favored vendors:

Anue Systems, Net Tool Optimizer at www.anuesystems.com

Absolute Analysis Investigator at www.absoluteanalysis.com

Network Instruments GigaStor, Observer and TAPs at www.networkinstruments.com

Network Critical TAP systems at www.networkcritical.com  

Be sure to check out all our super sponsors - They all have great solutions -The ones above are the ones that I have used, that is all the difference and I am sure the others are super also!

Be sure to read my upcoming article on today’s easy to use data filtering technology devices.

I wish each of you Great Success with Less Stress – Oldcommguy™

Editor Profile  Tim O’Neill is an independent technology consultant. He has over 40 years experience working in the WAN, Analog, ISDN, ATM and LAN test market. Tim has worked with companies like Navtel, Network General, Ganymede and ClearSight Networks and is now helping companies get technology verification and market validation. Tim is also the Chief Contributing Editor for LoveMyTool.com, a website designed to help network managers gain access to valuable information and real world solution stories. Tim is a patent holding, published and degreed engineer, who has seen this technology grow from Teletype (current loop) data analysis to today’s 10 Gigabit LAN’s focused on business applications with heavy compliance demands. Tim is heavily involved with helping Law Enforcement gain network and analysis knowledge and a regular speaker and trainer. 

Tim can be reached at tim (at) oldcommguy (dot) com

**Please note – All Rights Reserved on the original material – Copyright 2010