I can not tell you how many times I have been involved troubleshooting an issue where the root cause was something physical. When I say physical, I mean that the answer wasnt in a packet, log or managment screen but anything from cabling, heat or other issue.
I have met so many network analysts that are very good at configuring equipment and interpeting statistics but find many of these same people unaware of the physical aspect of network managment. "Physical" typically refers to things like power consumption, heat dissipation and rack space but since I am called in to troubleshoot, install or to assist in network clean ups, I have a few more to add.
For example; one customer was telling me about a bad switch what would randomly turn off. I asked some questions and walked over to the area where he had the switch located. I explained that the issue was probably related to insufficient air flow causing the switch to overheat. He countered that the replacement switch was 'fine' and I pointed out that the old switch was fanless where the new had 4 fans. Then I explained the merit of setting up monitoring or SNMP traps that may have helped identify this possible issue.
In another scenario a customer had network equipment in what I would call, "an environmentally challenging place", specifcally a warehouse. While we were working during their maintenance window, I explained that they should inspect and probably 'clean out' the network equipment to ensure optimal operation.
I have added some examples of what I'm talking about.
This photo is the most common issue I see out there.
The switch fiber connectors are exposed to the dust and dirt in the environment causing contamination. I have seen similar issues with Fiber patch cables.
Unfortunately dirty or contaminted connectors may not show up an issue immediately when in use.
Just as a FYI, I recently discovered Fluke Networks has a Fiber Optic Cleaning Kit and a "One Click Cleaner" that you could use to easily clean these connectors.
Please refrain from breathing/fogging fibre connectors with your mouth and then polishing it with your shirt. NOT A GOOD IDEA!!
Keeping the dust caps on in the first place is the best solution.
During this troubleshooting/maintenance exercise I showed the customer that some of the Ethernet ports are showing signs of oxidization which can cause the ports to get errors, shorts or eventually fail.
I explained that there are ruggedized versions of equipment from many vendors, like ruggedcom for this kind of environment.
In many cases, a cabinet with filtration and enviromental controls would help.
I have used Dielectric Grease on connectors to prevent this from occuring when they are exposed to the elements.
In this last example, I showed a customer the amount of fine dust that was covering the switch circuit board like a nice comfy thermal blanket. They mentioned that they noticed high temperature alerts from this switch but thought the room was too warm and never considered it was just the switch. At one point they thought there was something wrong with the management system that was reporting the temperature alerts and simply ignored the alerts.
I gently dragged my finger across the board and there was at least 1/8 of a inch of dust as well prehistoric-sized dust bunnies that could choke a fan.
I took the switch outside and a simple blast of compressed air took care of it.
Unfortunately many people who have network equipment in an office building falsely believe this is only a concern in factories and warehouses. In many cases, network equipment is located in corporate wiring closets that attracts as much dust.
I've walked into wiring closests and found janitor equipment blocking cabinet vents, cleaning rags hanging from fibre cables like its somebody's laundry room, and a ladder propped up against an electrical shut off switch. If you don't believe me, look at the last photo below.
The moral of the story is to get out every so often and physically inspect your network equipment and environment where they are located. If it is not physically possible due to geographical constraits have someone in that remote office take a photo. If the room is important enough consider a webcam.