Greetings:
We have a client site that is driving us nuts! We have had more equipment failure issues at that one location than at all other locations we manage combined. We have come to the conclusion that there is some type of RF interference occurring that is causing these failures, but are at a loss how to prove that and track down the source.
This client site is a new facility. Incoming power is lightning protected by the power company at the meter, and each breaker panel has surge suppression breakers. Incoming telco lines are lightning and surge protected by the telco at the demarc. The building has excellent grounding. All telephone, network, and computer equipment are on UPSes. All network infrastructure is certified CAT6.
We have had the power company monitor line quality and they see no blips that would explain out observed problems. The local telco consistently claims the problem is on the customer side of the demarc -- even when it effects telco equipment.
The site has channelized T1 voice, an ATM encapsulated FR WAN, and ADSL local Internet access. We have this identical setup for this client at about a dozen other locations, and none have any problems.
The most frequent observed problem is the ADSL line dropping due to excessive noise. The problem seems to occur most at about 0130, 0730, 1530, and 2230 local time -- but not every day and not always at those times (about 20 to 25 failures per week). ADSL failures can last from 5 minutes to hours. We get frequent voice and FR circuits failures -- but of shorter duration (usually less than 5 minutes) and far less frequently (2 or 3 times a week). The voice and FR failures appear to occur concurrently (physically separate circuits), but not at the same time as the ADSL failure. I should also add that there never appears to be audible noise on the lines, even during failures.
Equipment failures is what really leads to the conclusion this is probably an RF issue. Among the stuff we have had fail are:
3 DSL routers (cisco 8x7)
1 edge router (cisco 28xx)
1 FR router (cisco 36xx)
1 patch panel
1 telco smart jack (ATM/FR circuit)
1 PBX T1 card
1 patch panel (all jacks went open on the same pair)
6+ NICs
I should add that the failures started before the building was occupied -- while we were still testing the basic infrastructure. In fact, we have observed problems from day one of the networks being turned on by the telcos.
I should also add some other points:
-- We have observed failures when the building had zero power, except for the UPS battery power in the server room, so we don't think that we are getting power spikes from anything within the building.
-- The building only operates 0600 to 1800, so many failures are occurring after hours.
-- There are no RF sources in the building.
-- We are not near an airport.
-- The building is steel framed and sided -- and a pretty good RF shield -- cell phone reception goes from 5 of 5 bars to 1 or 2 bars as soon as you walk in the door.
Given what I have described, would you think this is an RF interference problem?
RF problem or not, how would you track down this problem?
We are to the point of bringing in a consulting EE, but I am not sure that most would be equipped to solve this problem; so, what should we look for in a potential consulting engineer?
Apologizes for the dissertation! Any and all helpful suggestions welcome.
Thanks!
Jon Kibler