In part one, in the interest of saving your time, I explained why the common practice of attempting to locate sources of noise and interference usually yields little fruit. Part two continues in this vein, discussing how to approach troubleshooting and detecting interference and noise that is having a negative effect in your wireless network.
Following a logical troubleshooting procedure limits the downtime needed to test a wireless issue and will help you arrive at the solution sooner. So first, think about and note the following. By answering the questions below, you will find one area remains for which to focus your attention.
Is the problem affecting the whole network, or a specific tower, sector or link? Or maybe a small group of customers in a single area are affected?
Are you aware of any recent configuration or equipment changes made to the affected portion of the network? If so, what are they?
What’s going on in the environment? What is the adjacent competition doing, if anything? Are you aware of any new construction that could be presenting a new obstacle within the coverage area or link experiencing the issue?
Is there a certain time that the problem occurs or started to occur? Is there a pattern to its occurrence or is it more random in nature?
Next, you will want to gather and analyze all the data at your disposal from the area identified above before assuming the system is experiencing interference and needs to be taken offline to conduct spectrum testing. Developing a test plan based on factual data will greatly reduce your scheduled maintenance window’s period, or prevent it altogether.
- Baseline analysis: check RSS, SNR, scan, latency and performance baseline data (log files, MRTG, Solarwinds, etc.). Note the average value for each, as well as the dates and times any dramatic changes in these values occurred.
- Changes: check with colleagues to determine if any changes in hardware or software configuration have occurred since the times noted in step #1 above.
- Customer feedback (placed at step 3, but you may want to move it further down on the list, depending on your situation): If you’re a WISP operator, you are already fully aware of how slippery the slope can get when gaining customers’ perspectives. However this should not be overlooked, especially when the problem is affecting multiple customers. Correlating affected customers’ stories can quickly eliminate false information while solidifying the facts. Particular attention should be given to asking customers questions regarding the use of wireless equipment within their own home and immediate vicinity.
- Check radio configuration: access the affected radio(s) using the connection method with which you are familiar (telnet, HTTP). Check the obvious areas such as firmware version, power level, frequency, IP configuration (address, gateway, DNS, SNMP, SNTP, etc.), modulation schemes (if applicable), SSIDs, authentication (static, RADIUS, TACACS, keys), encryption, QoS, VLANs, etc. Sometimes a simple misconfiguration can masquerade as a wireless issue.
- Check radio operation and monitoring tools’ data: link activity (TCP/IP stats such as Rx and Tx packets), errors (discarded and retransmitted packets), conduct wireless performance and ping tests and collect current RSS and SNR readings; compare these to the baseline in step 1.
- Spectrum scanning: requires taking the production system offline for the time it takes to perform the scans. Using your radio’s built-in spectrum analyzer tool is the most practical and unobtrusive means of sampling what’s going on in the RF environment. If the radio does not include this tool but the installation consists of a RF cable run into an enclosure or customer premises, a stand-alone spectrum analyzer (Anritsu, Agilent) can be directly attached to the antenna or with suitable adapters (compensating for any loss added to the path between the radio and antenna). The benefit of using a stand-alone spectrum analyzer is its accuracy and flexibility compared with the software-based, built-in spectrum analyzer tool on your wireless radios.
- Hardware: inspect all connections for corrosion, water ingress, tearing, bending, cracking, etc. If suspect components are found, begin swapping out the easiest pieces (RF jumpers, connectors, band pass filters, lightning arrestors, PoEs), retesting as per step 5 and 6 above after each change. Last to try swapping is the radio itself; verify the correct configuration and firmware is loaded on the swap unit prior to the exchange.
- Parallel system testing: at this point there are only a couple of components that remain to be tested; the tower or customer RF cable (LDF4-50, LMR400) and antenna. Since these items are not easily swappable, parallel system testing makes sense. It is best to conduct this testing using the same equipment as is installed in the production system. However, many operators and field services make use of a pre-configured ‘test rig’ that is relatively easy to deploy to save time. In this case, calculations should be done to compensate for the loss or gain in the path as a result of testing with dissimilar equipment before comparing to a baseline.
Also, the active system’s radio(s) should be disabled during testing using the parallel system to prevent masking the results. If possible, it is highly recommended to perform tests in both vertical and horizontal polarities and at different elevations. If the parallel test system shows similar characteristics to the affected installed system, noise mitigation will be necessary. Make sure you have recorded all stats, scans and other test results so as to determine the course of action that will best reduce the effects of noise or interference as well as minimize downtime.
In part three I will show you how to put the troubleshooting process above to the test by using an example to identify, troubleshoot and mitigate problems stemming from changes in the noise floor.
Author Profile: Tim Preston is a Senior Network and Systems Analyst with experience dating back to 1998. He started on the front lines of technical support for a large northern Ontario Internet service provider while earning his diploma in Computer Programming and Network Analysis. After being hired by a major wireless broadband radio manufacturer, Tim moved to Toronto in 2001. In 2009 Tim started Haven IT Consulting. Examples of work he has done for his clients include providing management and troubleshooting services to wireless ISPs, interconnecting retail outlets for an equipment supplier, and providing technical auditing, network design, operations advice, and technical support for various local businesses and network solutions providers.