UBNT Infrastructure Part 8 – DFS Nightmare!

Ramblings of an IT Guy in Education

UBNT Infrastructure Part 8 – DFS Nightmare!

June 14, 2018 Infrastructure Ubiquiti 1

I hope you’ve never seen this type of email from your Unifi Controller before.

Figure 1 – AP Radar Detected

It comes with a nice alert in your controller as well. And if you get a couple that is fine, not really a big deal. However, if you keep getting them… you are in for a fun time. These alerts are known as detected radar alert, it means that your AP is using channels in the DFS range and it has detected a radar also using that same channel. If you see this wiki page you can see what channels can be impacted by DFS: https://en.wikipedia.org/wiki/List_of_WLAN_channels.

So what is actually happening… when an AP detects a radar on a DFS channel frequency, the AP will notify the clients that it is switching channels and the clients should seamlessly roam to the new channel. In theory, it shouldn’t be a big issue and your clients should barely even notice anything has changed.

However it can be anything but seamless, and for us, we were having issues with our Dell 7285s. They were struggling to roam between APs correctly, and because of that, the same devices were struggling to roam to the new channel correctly. We had also moved to wireless mirroring over the Christmas break, so even a 5-second drop of the wireless meant that our staff noticed it. We did discover that a firmware update on the 7285’s fixed this issue as it was a Dell issue and not a UniFi wireless issue.

Before I go any further I need to give a DISCLAIMER.

Disclaimer: Some of you may get annoyed that I skip over certain details in this post. While I would love to go into more detail, a lot of work was done directly with Ubiquiti for this issue. I am one of the few invited Alpha members, and that comes with signing an NDA, because of that NDA they could keep me in the loop on details. For example, a firmware they used is still in Alpha status, and to release information about it would mean I violate the NDA, and I don’t want to get sued. With that understood, Let’s go through the DFS issues that CCGS faced.

We first noticed them in Week 1 of term, there had been no prior event of them happening before staff and students returned. At first, it was only in a few places and we were able to handle it by moving some channels around and we hoped it would fix the issue.

However, it didn’t stay that way. By Week 2 we were averaging 3 DFS Detections a day and they were throughout the school. At first we looked for anything in our environment that might be causing it. And we looked at the area around us to see if there had been anything new installed near by that might be suddenly causing radar alerts.

Figure 2 – 3 Errors in 1 Day.

I also reached out to Leader Support, however, due to the nature of the issue, I was passed directly onto Ubiquiti Support.

It took two weeks before I was aware of what progress was being made. I originally thought it was because I sent a ‘strongly worded email’ about what was happening and how I was disappointed at the level of the support. I thought this was what started to get me better support, though speaking with Ubiquiti recently they informed me that the email didn’t actually get them to work harder, nor did my email convince them that things had to change. What it did do however, was get the Engineers to be more open with what they were already doing. Suddenly it changed from an email or skype message every two days to a stream of updates.

While it was good to hear about what they had been doing, it was frustrating that I needed to send that ‘nasty email’ to get told of their progress. That being said, finding out the level of support I was getting impressed me. I had had an issue with Aruba 3 years ago, where one of their engineers told me the problem with the AP was that it had more than 25 clients on it, an AP they had recommended for a section of a Library.

But back to the Ubiquiti DFS issue. At one point I had 8 engineers and some VPs from Ubiquiti in a Skype channel chat trying to help me find a solution. It was at this point that I came to fully appreciate an issue Ubiquiti had. You see DFS Detections are not something you can ignore, in fact ignoring them can get you in a lot of trouble legally, resulting in fines for the company. This is because some of the users of DFS channels like Airports don’t like it when someone’s wireless infrastructure interferes with those critical systems.

A series of things really helped show Ubiquiti that we were getting false positive DFS detections. First, there was our B Block, I have mentioned it before that we had wrapped the rooms in Cooper mesh to allow for multiple AP’s per room (I am going to post more on that soon). We were getting detections on the bottom level on Channel 100, but then not on the top level where the same channel was in use.

Figure 3 – B Block is wrapped!

Also, we made the decision to allow Ubiquiti Engineers direct access to our site via VPN and Cloud Connect controller. We also setup the AP logging to go directly to Ubiquiti so they could troubleshoot the issues directly. They were also able to see similar types of issues in their test lab you can see in Figure 4 and 5.

Ubiquiti then kicked things into high gear. First, they brought a workaround into effect, this workaround meant that when a DFS detection happened it would not kick clients off or change the channel on the AP. It triggers the DFS alert and notified Ubiquiti it had been detected, but didn’t affect my users at all. This workaround was only applied on AP’s that were getting the false positive detections, and even when it happened Ubiquiti were checking each event to make sure it still matched the false positive steps as before.

Then they turned the hand to fix the issue. First, they sent me the following picture of the Faraday cage they had built to create a test environment matching ours so they could test the setup.

Figure 4 – UBNT Faraday Cage

Figure 5 – Inside the Cage

As you can see in Figure 5, they loaded the cage with EDU AP’s (our AP’s that were having the issue) and did a range of testing. I especially like the 2 AP’s hanging from their safety wire. They also sent me 4 Ubiquiti SHD AP’s (https://unifi-shd.ubnt.com) that contain a Spectral Security Radio, this allows them to get real-time analysis of our environment from different locations without impacting my wireless.

After two months of the workaround I was informed, Ubiquiti believed they had found a solution, though the fix was still in pre-alpha. And they didn’t want to potentially cause other issues by loading that firmware and was I okay with waiting another few weeks before they released the firmware to me. Two weeks later I got the news, the firmware was out.

We did a staggered rollout of the new firmware. At first, we loaded it on a quarter of the AP’s, making sure to get ones that had experienced a high level of DFS detections. The workaround was then removed from these AP’s and they were left for a week to see what happened.

And we got no DFS detections on those APs. So we moved to the next quarter, rolling it out to half our fleet of APs in total, again the workaround removed and this time 2 weeks was allowed to wait and see. Again no DFS detections.

Finally, we rolled it out to the last half of the fleet, no matter what model of AP they were all upgraded to this new firmware and now none of our AP’s had the workaround on them anymore. And it has been that way for over a month now without any DFS Detection. Problem solved!

And as I round up the last UBNT Infrastructure post as I was calling them, it is now June after installing Ubiquiti in December and January. And yes as I have posted on this site we had some issues, but they were generally overcome of given a work around pretty quickly. And right now, if I was picking my platform again I would choose Ubiquiti again.

For me it still comes down to 2 reasons. The first reason is that Ubiquiti is meeting our needs, we are getting strong stable networking, our wireless (now) is stable and I have seen a couple of weird bugs over the last few weeks they haven’t been massive issues that have heavily impacted users. And the second reason is cost, for us it has allowed us to save a large sum of money, and for a school this has to be near the top of our list. Every dollar we save is money that can go back into the classroom. And no student or teacher has ever come to me saying something like “Wow… I am so happy you are running {insert wireless vendor here}”. But I do get staff saying thanks for making our wireless stable. Other than that they don’t care.

And that brings me to the end of my UBNT Infrastructure posts, it is highly likely I will do more posts about Ubiquiti, but they won’t be about my rollout but general posts. My next post though will be about our Vivi (https://www.vivi.io) rollout and how it has gone. And how it has helped changed the way teachers are working in our environment.

One Response

  1. crackpot says:

    Ꮤow that was ѕtrange. I just wrote an incredibly long
    comment but after I ϲlicked submit my comment dіdn’t show up.
    Grrrr… well I’m not writing all that oᴠer again. Anywaʏ, just wanted
    to say gгeat blog!

Leave a Reply

Your email address will not be published. Required fields are marked *