I recently had an encounter at my day job that left me staggered. There was a network outage at one of our locations. Our WAN is a VLAN leased from a wireless ISP, or WISP. We had some power outages over the weekend in the area, so when the network was down on Monday morning, we assumed that was the cause. I put in a call to our contact at the WISP, who was aware of the outage, and was assured that someone was in the area, had a job to complete, and then would be able to investigate our outage later that day. I provided my direct number for him to pass on for the technicians to notify me of their impending arrival at the site, said thanks, and began the long wait for my phone to ring.
Well, the day passed quickly, and it was productive for me, but I did not hear from the WISP techs. The school day ended, meaning that shortly there would be nobody at the site in question, and they still hadn’t been in touch. Whether their first job took longer than they thought, or they couldn’t make it for some other reason, I would have appreciated a phone call to update me.
The next morning I called the teacher at the offline site to hear that no, nobody from the WISP had been onsite the previous day. I then called the WISP. This is when things start to get weird. Now, I should say that the gentleman I spoke with on the phone was courteous and helpful. I started to get confused when he said yes, the techs had been on site and the problem still wasn’t fixed. I said they couldn’t have been on site, nobody called me and nobody at either location saw anyone. How could they have climbed a tower and been on a rooftop with nobody knowing? He promised to look into it and call me back.
A short while later, my phone rang, and it was my man from the WISP, with good news. A service van was 30 minutes out. I finished up what I was doing, packed up my gear and hit the road to the site that was offline. I arrived and waited. And waited. And waited. An hour later, I called the school that is the upstream site. The secretary told me that the techs were there, out on the tower. I wondered… Why? The outage is here. The NOC said they could access the upstream radio, so what are they doing? Keep in mind I, the member of the customer’s IT team, still haven’t heard from them, despite leaving my number twice.
When they did show up, I was informed of the real problem. The link was installed with not nearly enough height for the radios at either end of the link. The WISP monitoring shows that the RSL was -86 dBm before it failed. Going back to last summer, it was steady at -78 dBm. How in the world was it working at all? There are 3 locations supported by this connection. Moving from a service provider into a customer role, I dislike not knowing what our links are doing. The image below shows the terrain profile of the link in question.
There is a 30 meter (100 feet) difference between ground elevations at each end of the link. I’m not sure what the antenna height was at the tower, but the lower end was only about 6 meters (20 feet) above ground level. I truly don’t understand how this link passed any data! The techs were at the upstream location raising the antenna as high as possible.
The building directly beside ours has it’s connection to the same tower, with the radio mounted at about 68′. I was told by the techs that it’s RSL is -57 dBm. Based on that, who would think it was OK to install a link of the exact same distance at 20 dB lower RSL? On top of that, I’ve heard that the WISP told us if we wanted more bandwidth, the antenna would have to be raised. Yeah, no @%$*! Obviously they were running our link at a low modulation with a wider channel than necessary, in order to get us the speed dictated by our service agreement. Apparently the tech from the previous day had been at both sites, replaced the radio at the remote (down) end, and spent four hours trying to align the antenna to get a signal level good enough for the link to work.
When installing a point-to-point link, one should ALWAYS know exactly what the expected receive signal level is. This is based on hardware, frequency band and link distance. Many vendors have built web-based tools for their customers to do link budget calculations. Some of these tools will even calculate Fresnel zone clearance for you based on Google’s terrain data. Here are some examples:
- Ubiquiti Airlink (Google Chrome recommended)
- Mimosa Design Tool (account required; allows custom radios)
- Cambium Link Planner (download requires account)
- Ligowave LinkCalc (account required)
Radio Mobile is a good free tool for planning, if your vendor of choice can’t supply you with something. It requires some time to learn and use correctly.
The new, good tech said there was a previous agreement allowing us onto the neighbor’s tower. He also said he had no flarging clue why it wasn’t put there years ago, because the agreement is at least 2 years old. I was impressed with this guy; He and his partner moved the radio over to the top of the tower and ran a new cable into our server room. Problem solved!
There was at least five things wrong with the way the WISP and it’s employees handled this outage:
- Nobody called me on the first day, to say that they were on their way, that they were on site, that they had been on site, or that the outage was not fixed;
- Nobody called me the next day to describe the events of the previous day, or that there was another crew en route;
- Nobody called to let me know the crew had arrived at the far end of the link;
- NOBODY CALLED!;
- The link was installed and left to operate for years sub-optimally, even though the WISP’s monitoring told them that the RSL was much lower than calculated, and nothing was done until it dropped completely. This is the big one.
The lack of communication wasted about a day of my time. Customers like to be kept up-to-date, even if all you have to say is “Sorry, we don’t have a time to repair yet. We’ll keep you posted.”
The lack of professional integrity wasted about 24 man-hours of the WISP’s time, and caused a two-day outage for three of our locations. If you’re going to do a job, do it right the first time! Otherwise, you’ll probably be back to fix it, wasting the time and money of you, your company, and your customer.
I’m stepping down from my soap box now.