Network Janitor: So Simple, Even My 13th Year Old Niece Did It (by Tony Fortunato)
Global IP Traffic to Quintuple in 2013 (by Michael Patterson)

Improving Network Recording and Storage Efficiency with Taps, Aggregation and Filtering



Most large computer networks are improving their storage infrastructure to meet compliance and security standards.  The deployment and resources required to improve security are under constant assessment. Business and technology are both driving the requirements for improved storage solutions.




Providing adequate storage solutions throughout a network is not always technically or financially feasible.  While storage costs per terabyte continue to decline, methods to efficiently manage storage facilities must evolve. 




Figure 1 - Typical Network Diagram With Probe Deployment using Port Mirror




Network recording has become commonplace for security, compliance and network analysis purposes.  Frequent backups of databases and other high volume traffic can drastically decrease the amount of available storage.  Backing up databases across a network for offsite storage requires adequate bandwidth and is often done so that all data is stored at another location. 



Recording traffic as it moves from one location to another is conveniently done at egress points of a local network.  Recording of database transfers is often not necessary for compliance or security purposes on a local recording device.  The database is fully transferred to another location, and this becomes the backup copy.  Network data, VOIP conversations, web and application traffic is what is necessary to be recorded.  Eliminating only database backups at the egress recording location has been problematic until the development of filtering TAPs.  


Filtering TAPs allow network engineers the ability to make identical copies of network traffic.  This is accomplished by placing a TAP inline between network devices, such as a router and a firewall.  As traffic flows through the TAP, a copy of the network traffic is made.  The copy is typically sent to analysis, security or recording appliances.  In this case, the recording appliance will receive the traffic.  


The advantage of a filtering tap allows the engineer to eliminate only the database backups from being recorded.  This is typically done by filtering traffic coming from the IP address of the database server, with a specific port or port range.  The combination of IP address and TCP or UDP port information, uniquely identifies the database backup traffic and eliminates it from being recorded.  The database backup is still sent to the remote location and is available for business continuity purposes.  The database backup is not recorded by the recording appliance, allowing the appliance to store a significant amount of additional traffic before archiving or other storage media is required.




Tapping the Link

A network TAP (Test Access Point) makes a copy of information in a network connection.  The TAP is designed so that it does not become a point of failure in the network.  TAPs are designed so that traffic on the network link should continue to flow, even if the TAP loses power.  TAPs also minimize latency between the network link and the monitor port on the TAP.  TAPs provide additional features that make analysis more convenient.  


TAPs will aggregate duplex traffic onto a single output port, while providing buffering capability to handle traffic utilization surges.  TAPs can also provide identical copies of traffic so that multiple tools all see the same data.  These “regeneration” TAPs are deployed when redundant probes or security tools need to have 24X7 visibility to a network segment.  The failure of one security device does not create an issue, since the other security device sees the same network data from the TAP.




Figure 2 – Common Network TAP Locations




By combining several Data Capture technologies, we can craft a solution to exclude backup traffic from our Network recorder while ensuring complete visibility of the network to our other analytical tools.  In Figure 3, SPAN ports and tapped links are brought into a filtering and aggregating device that combines the various streams of data and filters out the backup traffic. 



The backup traffic can be identified in several ways.  For example, bidirectional traffic between server IPs and the IP address of the backup server could be considered backup-related traffic.  In addition, the backup agents on target PCs could communicate with the backup server using a specific network port; this network port could be filtered from the data stream being sent to the network recorder.





Figure 3 – Aggregation Solution With Backup Data Filter At Network Recorder





Filtering & Aggregation TAPs can be deployed to significantly increase the amount of information that recording appliances store by eliminating database backups from the recording stream while still providing full visibility to other tools on the network.  This method can be successfully deployed to increase recording times, achieve compliance and provide additional information for network analysis and troubleshooting. 



The use of aggregation in addition to filtering can reduce the number of probes or analysis tools needed.  This can provide significant cost savings while still providing full network visibility.




Tim crofton

Author Profile:Tim Crofton is currently the Product Manager of Datacom SystemsPrior to Datacom, Tim worked as a Manager for NEC Unified Systems.  Tim hold and MS in Computer Science from SUNY IT and plays a mean set of bagpipes. 








Rob Buckland


Author Profile: Rob Buckland 

is currently a Sales Engineer at Datacom Systems.  Prior to Datacom, Rob worked as a Senior Enterprise Management Specialist for Universal Healthcare Systems.  Rob holds a Ms in Management Science from SUNY IT and is a soccer fanatic.




Screen shot 2010-08-30 at 11.01.34 AM


Since 1992Datacom Systems has been providing a full product line for passive test and monitoring access and traffic visibility into network links, enabling customers to access critical data from anywhere in their network.  With tens of thousands of systems installed globally, Datacom Systems provides best of breed data capture infrastructure for all major troubleshooting, security, and application monitoring tools.