Hi all,
So, having a... frustrating issue going on. Long wall of text ahead as I explain.
1 x CenturyLink/Lumen fiber in Boise
1 x CenturyLink/Lumen fiber in Cheyenne
1 x Comcast biz fiber in Denver
IPsec VPN tunnels between all three sites, w/ OSPF for routing failover (which unfortunately doesn't help in this situation).
Two days ago, Cheyenne to Denver (.196) traffic (both tcp and udp) were an issue initially. Failed over to routing Cheyenne VPN through Boise while we opened ticket with CL.
Yesterday, Boise to Denver (.196) traffic started having exact same issue.
Tests from another CL fiber in Boise (my own circuit, with legacy IP space and BGP) to Denver (.196) did not show same issues. Path appeared clean.
Traceroutes from Office Boise to Denver (.196) had a noticeable difference from Personal Boise to Denver (.196):
Office Boise -> Denver (.196)
I can confirm this issue exists at several sites in the Denver area with this same IPSEC issue, all routing between Level3/Lumen and Comcast.
I was told by one customer that it resolved late yesterday afternoon but I haven't been able to confirm that.
Mike
We have also seen the same behavior of intermittent customer complaint followed by issue resolving spontaneously. Our end customer has a tunnel to a supplier on Comcast with a return path via Lumen. The first ticket was opened on 9/23, has cleared and returned a few times. End customer worked with Cisco TAC and symptom is up to 30% ESP packet drop for periods from 5 minutes to 1 hour when the problem is active.
I have multiple clients with the same problem involving tunnels terminated on Lumen to terminations on Comcast. The problems started on Monday morning, 10/11/21, all in the Denver metro area. I was able to change a tunnel from IPSec-ESP to IPSec-HA and the problem went away and then changing back to ESP and the loss came back. We are also seeing problem with IPSec remote VPN clients from Comcast networks terminated to firewalls with Lumen interfaces having drops and the tunnel is unusable.
We have opened tickets at Lumen, but they get closed because a traceroute shows no drops and they say the problem is with the “application”.