Help with broken routing in MTL (AS577 174 11647) Bell, Cogent, Me

We noticed random traffic originating from AS577 stopped getting to our ASN via Cogent in Montreal at around 00:24 this morning. Based on the pattern I am guessing a broken next hop on one leg of a larger link ? e.g. IP traffic leaving AS577 towards IP addresses in one of my /19s will 100% work for certain IPs and 100% fail for others all within that same /19.

e.g. from 142.112.93.72 (an IP originating in AS577) to one of the IPs below in my 64.7.128.0/19

64.7.134.1 100% fail
64.7.134.2 100% pass
64.7.134.3 fail
64.7.134.4 fail
64.7.134.5 fail
64.7.134.6 fail
64.7.134.7 100% pass
64.7.134.8 100% fail
64.7.134.9 fail
64.7.134.10 pass

Its just packets towards my ASN that are failing. Packets TO 577 via 174 all work fine.

Traceroutes good and bad all look like this

% traceroute -s 142.112.93.72 -q1 -I 64.7.134.2
traceroute to 64.7.134.2 (64.7.134.2) from 142.112.93.72, 64 hops max, 48 byte packets
1 lns5-ottawa23_lo0.net.bell.ca (64.230.11.188) 10.507 ms
2 64.230.98.68 (64.230.98.68) 17.539 ms
3 tcore4-montreal02_1-9-0-0.net.bell.ca (64.230.79.127) 18.150 ms
4 cr02-mtrlpq02ho6_bundle-ether1.net.bell.ca (142.124.127.251) 13.787 ms
5 bx1-montrealgz_et-0-0-5.net.bell.ca (64.230.26.135) 12.237 ms
* 6 **
7 be2089.ccr21.ymq01.atlas.cogentco.com (154.54.45.113) 13.337 ms
8 be3259.ccr31.yyz02.atlas.cogentco.com (154.54.41.205) 17.969 ms
9 sentex.demarc.cogentco.com (38.104.158.78) 17.126 ms
10 agas-core1-em0.sentex.ca (67.43.129.251) 39.960 ms
11 agas-lns1c-c.sentex.ca (64.7.129.19) 32.171 ms
12 orbiting.anysphere.com (64.7.134.2) 49.336 ms

% traceroute -s 142.112.93.72 -q1 -I 64.7.134.1
traceroute to 64.7.134.1 (64.7.134.1) from 142.112.93.72, 64 hops max, 48 byte packets
1 lns5-ottawa23_lo0.net.bell.ca (64.230.11.188) 10.229 ms
2 64.230.98.66 (64.230.98.66) 13.346 ms
3 tcore3-montreal02_hu2-4-0-2.net.bell.ca (64.230.78.252) 12.660 ms
4 cr01-mtrlpq02ho5_bundle-ether1.net.bell.ca (142.124.127.233) 12.423 ms
5 bx1-montrealgz_et-0-0-1.net.bell.ca (64.230.26.133) 12.225 ms
*6 **
7 *
8 *
9 *
10 *
11 *

Not sure if hop 6 is a Bell device or Cogent device. I have a ticket open with Cogent, but they dont see any issues in their network. I am not a direct customer of Bell, so I cant open a ticket with them. Was hoping someone from AS577 would see this and take a look.

 \-\-\-Mike

Looks to be fixed as of ~ 14:55 Eastern.

 \-\-\-Mike