UBNT Infrastructure Part 6 – Routing table filling up
About two weeks after the core had been installed, I had gotten around and managed to install most of the other 45 switches, I then came across another issue. The biggest issue I could find is that I was getting bad network performance, and I was seeing that every now and then an EdgeSwitch would go ‘down’ for about thirty seconds or more.
Though to be honest I am writing about issues I was experiencing in January, now were in May, and I was working 15-20 hour days during January to get all the work done so I am a little foggy about what issue caused what. So I am sorry about that, I’ll do my best though.
I tried at first to figure out the cause of the issue myself, I thought it might be one switch or another causing the issue, but I wasn’t really able to figure out the exact cause. At first, I had read about some issues relating to the then current firmware of 1.7.3, an issue that effected uplink performances. I had been installing switches and upgrading them to 1.7.3 as step 1 of installing them, so I spent one night downgrading all my switches to 1.7.1 (online recommended as most stable current firmware). It did fix the issue slightly but didn’t get rid of it completely.
This lead me to contact the Leader Support Team, I mentioned them in my why we choose Ubiquiti post. They are a great bunch of Australian based support personnel that have undertaken a range of the Ubiquiti Certificates (I couldn’t tell you exactly which ones) but they are very knowledgeable, and while they could find some issues they also reached out to Ubiquiti themselves to get some guidance on how best to proceed. So after some chatting, I was put in touch with a Ubiquiti Engineer, UBNT-Jordan (https://community.ubnt.com/t5/user/viewprofilepage/user-id/312298). Jordan connected to my network via Team Viewer and started going through all the switches and reviewing config and checking logs.
His first thought was plan was to change the ARP Timeout to 50, as he saw that the ARP table was filling up. An example of what he found was this entry *Total Entry Count Current / Peak ………….. 481 / 493*. And while this did help, again the issue was fully resolved.
At the time we had an EdgeSwitch ES-48-500W acting as the core (we also tried the ES-XG-16), and there was the problem. As good as these switches are, our network and its VLAN routing were just too much for them. So a decision as made, I really needed something more powerful, like an EdgeMax Router. I suggested let’s just get the EdgeRouter Infinity… I’ll be honest I was getting a little frustrated at this point as I had made that suggestion a few days earlier and been ignored. I believe they were worried about the cost of adding another device to our order, though even adding an EdgeRouter Infinity to my order I was still well under the next vendor. The next issue though was getting our hands on an EdgeRouter Infinity.
At this point, it was 9 am Thursday the 25th of January, in Australia that was the day before Australia Day and a public holiday. Then on Monday, all staff would be back onsite. I really needed to get everything done before that. It was at this point I looped Ben Klason from Leader back into the chat telling him I needed an EdgeRouter Infinity onsite ASAP, and off he went doing his best to find me one. About half an hour later I got a call back saying he had one and would get it to me as soon as he could, I even offered to jump in my car and go get it.
At this point, I feel it is worth reminding people that the school I work for, Central Coast Grammar School is located on the Central Coast of NSW. That is around 80 kilometres from the Sydney CBD, and it was the day before a public holiday. It was now 9:30 am and I was curious what our next step would be. At this point I then got a text saying it would be here soon, Ben had not only found one but got one of his Sydney resellers to Uber it up to us on the Central Coast.
By 11 am the EdgeRouter was onsite and in my hands. I organised with Jordan at UBNT to work with him at 6 am the next morning for the EdgeRouter to be installed. We decided against causing an outage at 11 am in the morning to install it as we did have users on site and we weren’t sure if the outage would be less than half an hour. That afternoon I racked it up and connected it to our network. I upgraded its firmware and gave it an IP address ready for it to become the centre of our network.
The next morning, bright and early I was on site again and ready to work with Jordan. He connected in and then started loading our VLAN’s and default gates on the EdgeRouter, I didn’t even attempt this myself as I had never worked on the EdgeRouter’s before, their CLI not the same as EdgeMax switches, I wanted to know it was configured and working correctly. By 8:30 am everything was done and my network was now running with the EdgeRouter Infinity as my core.
I spent another half hour doing checks and reviewing the Routing Tables, and everything looked fine. Even my network had settled down and was running smoothly.