In the packet world I hear a lot of concerns that when bandwidth increases, the ability to analyze is negatively impacted. When I present or work on-site, I remind network technicians that network monitoring is the basis of developing a baseline of your network health which can be accomplished many ways. You can start old school and get statistics directly from your network equipment, like good old SNMP/RMON, api's, telnet or ssh scripts and have the device report back to a centralized management system.
Regardless how you get the data, the important part of the process is to develop a process or procedure of how to interpret the data and what to do when you find an anomaly. Here's a recent example, I was reviewing some of the traffic reports from a client's router when I noticed a considerable amount of traffic on their standby link. I followed up with a simple packet trace and noticed a bunch of SSH login attempts.
When I showed the client what was happening, he was confused as to how that could happen because: - as part of their standard router configuration, ssh is blocked from all WAN ports
- since its a backup link, he didn't think the backup port was live 'on the net' until the primary failed I went digging around the router configuration and for some reason this router interface was not included in the firewall rules. I spot checked about a dozen other routers and found another 4 more with the same misconfiguration which we quickly fixed. I explained that, in this case, just because you aren't actively using the link, the interface is still active 'on the net' with a valid IP address and just because you aren't going out, doesn't mean no one can get in. I took it a step further and before making my firewall change, I typed the public IP of the backup link and showed them that I could log in to that router. I went to the network management system and showed him the traffic on the backup port before and after the firewall change.
The client asked how I 'knew' what to look for and how to fix the problem. I honestly explained that I didn't 'know' or 'expect' anything, but the pattern on the backup link compared to the other routers looked 'odd', so I simply investigated as to why. 8 out 10 times this exercise doesn't turn up anything earth shattering and can be simply explained. The moral of the story is that having monitoring is good, but having a process or methodology to review the data is better.
