Ghosts in our 6 New Ubiquity Pros - provision issues.

Mailman List Mirror (Read Only)
21

We have all APs set with static addresses. EdgeMax only hands out IPs to
clients using the APs.

This happens when people are using the APs and when no one is even in the
building at 2am when there are no clients connected. It can happen to one
then 5 hours later it happens again...then doesn't happen again for 12
hours. Totally random no interval.

It is nice to know that others have no issues with these UniFi AP Pros.
They seem to be fine except for the 2 mins or so they randomly drop link
and reboot themselves. All are on APC UPSes and other devices in the same
switch , like voip phones, never drop the ports.

They are all new, delivered in various batches over time. We checked and
all are the latest versions.

Bob Evans

22

Thank You Charles,
Been on NANOG a while - all the basic stuff we know well. Like, cables,
cluster occurrences etc. Looking for the UniFi specific experience. Its
not the switches, power, cables, ports show no CRC issues etc.

We even setup another network with just 2 and it happens randomly - so its
some code or something. Think I'm going to let one of the guys here login
the the controller and see if we missed a setting in the latest code.
NANOGs real good at having someone with specific targeted knowledge
appear.

Thank You
Bob Evans
CTO

23

Only have 1 Pro on my network and it hasn't given me any issues, several of the original AP and AP-LR as well without issues.

What is the uptime on the AP? You should be able to ssh into the APs using the controller username and password. It is a linux base so 'uptime' will tell you. You can also check for ethernet errors using 'ip -s link' on the AP side.

24

Have you done a network analysis for viruses or bridge loops? This could be a broadcast storm caused by either of those network faults.

-mel

25

I've got a bunch of regular UAPs spread out over multiple customers with various network setups including ERLs as routers, CenturyLink POS modems of various generations, Dink routers, etc.

My controller is hosted off-site in Tacoma in our data center.

Some issues I've run into, particularly on the consumer devices like the older CenturyLink/Qwest modems...

1) Broken MTU clamping/fixing on PPPoE links, causing the UAPs to have problems making a connection to the remote controller.

Worked around by messing with the MSS using iptables on specifically the tcp/8080 and tcp/8443 port on the controller end.

Other devices, had to make sure to disable the firewall feature on modem, in order to get it to stop eating ICMP packets (and thus breaking pmtu).

2) Faulty DNS server daemons on the routers. The UAPs would have issues randomly resolving the controller's IP address from hostname. Have this problem time to time with anyone using the built in DNS servers on the CenturyLink/Qwest modems.

Resolved this issue by statically defining IP and DNS servers on the UAPs (DNS server set to 8.8.8.8). Also had to disable the firewall on one of the routers to get it to not intercept/mangle DNS packets.

These two issues alone have caused me major issues with the devices randomly being unable to get new configurations or download firmware updates.

On network switches connected to the UAPs, make sure that you've got the port set to whatever the switches' version of cisco 'portfast' is.

In the Site Settings under the Unifi controller, disable "Enable connectivity monitor and wireless uplink" and see if the problem eases up. If you need to use the uplink monitor, manually set the IP you want to check with, and make sure the UAPs can actually ping said IP.

I'm the head mod for /r/Ubiquiti, so feel free to bounce things off of me privately with your Unifi setup, and I'll be happy to give you a hand. I can also direct you to the unofficial Ubnt IRC channel where you can get a bunch more opinions.

26

Thank You Charles,
Been on NANOG a while - all the basic stuff we know well. Like, cables,
cluster occurrences etc. Looking for the UniFi specific experience. Its
not the switches, power, cables, ports show no CRC issues etc.

Sure. I've seen you around. Always good to check the basics, start at layer 1 and work up. That doesn't change, no matter how experienced a crew is. :slight_smile:

We even setup another network with just 2 and it happens randomly - so its
some code or something.

Wait... same controller? Or a different controller? Because if you can replicate across access points and controllers then you've probably found a bug. Well presuming you aren't fate sharing with anything else (like switches).

Very weird.

   Think I'm going to let one of the guys here login

the the controller and see if we missed a setting in the latest code.
NANOGs real good at having someone with specific targeted knowledge
appear.

Yes it sure is.

27

<snipped comments about much cpe sadness>

These two issues alone have caused me major issues with the devices
randomly being unable to get new configurations or download firmware
updates.

Question. Once they have connected and are "happy", do they drop off (re provision) like Bob is mentioning?
I'm still not entirely sure what is meant by "re provision". I've not seen it answered in the thread.

I'm the head mod for /r/Ubiquiti, so feel free to bounce things off of
me privately with your Unifi setup,

Didn't know that sub reddit existed. Awesome.

28

<snipped comments about much cpe sadness>

These two issues alone have caused me major issues with the devices
randomly being unable to get new configurations or download firmware
updates.

Question. Once they have connected and are "happy", do they drop off (re
provision) like Bob is mentioning?
I'm still not entirely sure what is meant by "re provision". I've not
seen it answered in the thread.

Reprovisioning with Unifi happens any time you make a configuration change. The next time the device does it's check-in (don't remember how often it checks in, but its at least once a min), the UAP will get a copy of its updated configuration, load it, then activate the changes (and reboot if necessary).

If the device never goes out of provisioning state, then it hasn't managed to pull its configuration or firmware properly and will likely keep trying.

When the device is having complete connection issues, it will show up as Disconnected rather then Provisioning in the controller.

Useful thing I've done - when a device is randomly having issues with provisioning, I'll setup the remote syslog option in the config, and have it remote log to my controller's syslog. Usually, it will dump exactly the reason why its failing the provision to syslog, making it easier to diagnose.

I'm the head mod for /r/Ubiquiti, so feel free to bounce things off of
me privately with your Unifi setup,

Didn't know that sub reddit existed. Awesome.

Its not as busy as the forums, but there's sometimes good info there. There's also the IRC channel as well, which has a mix of users and some Ubnt employees.

29

This is very helpful information.
We will be implementing these steps.
Thank You
Bob Evans
CTO

30

re-provisioning is to go to the controller find its config and reboot.
Thank You
Bob Evans
CTO

31

Great details !
Going to implement now.
Thank You
Bob Evans
CTO

32

Mell,
God idea , but , yes we did - no loops all are spokes - we know cabling
and setup our switches and routers to syslog those events.
Thank You
Bob Evans
CTO