I'm currently at a loss trying to figure out what is going on this past week and wondering if anyone else is seeing this?
Every hour on the hour (11:00am, 12:00pm, 1:00pm etc..) we have dozens of endpoints from different IP addresses go UNREACHABLE and drop calls if they are on one. They then become "Reachable" again after about 10 to 40 seconds. These phones have been perfect for months or years.
I've tried playing with NAT settings, using STUN, even opening up the firewall to one of their IPs and nothing seems to help. I'm starting to wonder if it's a Comcast issue with their newer modems but they won't bother fixing it.
I've been battling this for days and have noticed a few things.
- It only appears to be happening to tenants that are using Comcast.
- All tenants that are using Comcast that have their modem in 'bridge mode' and are using their own firewall are not having the issues (as far as I can tell).
- I set up a brand new Thirdlane MTE server and added a second account to some of the affected extensions and they would go UNREACHABLE on both servers at the exact same time.
- It doesn't appear to affect any of our Polycom extensions, but we don't have too many. Almost all extensions are Yealinks ranging from 7 year old models to brand new models with all different firmwares. It is happening to a Cisco SPA-112 as well though so it doesnt seem to be Yealink specific.
- I've had reboot their Comcast modems, and call Comcast. No luck.
Any thoughts or ideas would be very appreciated as I'm losing my mind trying to figure this out!
Quick update on this ongoing
Quick update on this ongoing issue.
It appears to be only affecting Comcast customers with Cisco DPC-39XX modems. What's odd is that there hasn't been a firmware update on them in over a year. We've forced Comcast to come out and swap modems for 3 clients so far and the problem has stopped for them. The other clients who can't accept the downtime of having a modem swapped we 'hope' to have solved by putting them on a VPN tunnel to our Thirdlane MTE. Some on older phones that don't support built in VPN we plan on installing a cheap router and putting their modem in bridge mode. We are working with Comcast but they require a ridiculous amount of steps for them to escalate it to their network team. (Documented support calls, Ticket #'s, and diagnostics run for every customer having the issue, we did this for 3 clients so far and each call took over an hour and resulted in multiple modem reboots/downtime)
We experienced something similar to this with Spectrum a few weeks ago. However, it was only for about half a day or so. Spectrum admitted they had a network-wide U.S.-based DNS issue which blocked all of our hosted customers on Spectrum from trying to reach our Chicago datacenter completely.
A simple test we perform in these situations is to configure a phone in our lab on the affected customer's tenant and see if the issue persists on that phone. If not, it's somewhere in the network. I would also recommend opening a ticket with Comcast on behalf of the customer to report the issue, could very well be in the modem or Comcast network.