Connectivity issue between Verizon and Amazon EC2

Mailman List Mirror (Read Only)
1

I'm short some important details on this one, but hopefully can fill in
more shortly.

We're seeing poor performance (very slow download speeds -- <
100KB/sec) to certain EC2 instances via our Verizon hosted circuits.
The issue is reproducible on both our production Gigabit circuit as
well as a consumer grade Verizion FIOS line.

Speeds are normal (10MB/sec plus) via non-Verizon circuits we've
tested.

Source IP's are in the 198.102.62.0/24 range and destination on the EC2
side is 54.197.239.228. I'm not sure in which availability zone the
latter IP sits, but hope to find out shortly.

MTR traceroute details are as follows:

Host Loss% Snt Drop Avg Best Wrst StDev
1. 198.102.62.253 0.0% 526 0 0.2 0.2 0.5 0.0
2. 152.179.250.141 0.0% 526 0 14.1 7.0 19.4 3.6
3. 140.222.225.135 37.5% 526 197 7.7 6.8 35.8 1.9
4. 129.250.8.85 0.0% 526 0 8.1 7.4 11.7 0.3
5. 129.250.2.229 10.3% 525 54 11.4 7.1 85.7 9.6
6. 129.250.2.169 41.5% 525 218 63.0 45.5 130.7 10.3
7. 129.250.2.154 0.2% 525 1 59.9 44.5 69.0 4.0
8. ???
9. 54.240.229.96 7.8% 525 41 76.6 71.3 119.9 8.6
    54.240.229.104
    54.240.229.106
10. 54.240.229.2 6.9% 525 36 74.7 71.6 109.1 4.9
    54.240.229.4
    54.240.229.20
    54.240.229.8
    54.240.229.14
    54.240.228.254
    54.240.229.16
    54.240.229.10
11. 54.240.229.174 5.5% 525 29 76.0 71.7 109.0 7.3
    54.240.229.162
    54.240.229.160
    54.240.229.170
    54.240.229.172
    54.240.229.168
    54.240.229.164
12. 54.240.228.167 94.5% 525 495 76.4 71.7 126.0 11.6
    54.240.228.169
    54.240.228.165
    54.240.228.163
13. 72.21.220.108 5.1% 525 27 75.2 71.3 112.6 6.8
    205.251.244.12
    72.21.220.8
    205.251.244.64
    72.21.220.96
    205.251.244.8
    72.21.220.6
    205.251.244.4
14. 72.21.220.45 9.0% 525 47 74.0 71.6 199.5 8.5
    72.21.220.149
    72.21.220.29
    72.21.220.125
    72.21.220.37
    72.21.220.61
    72.21.220.2
    72.21.220.69
15. 72.21.222.33 10.5% 525 55 73.4 71.5 87.1 1.5
    205.251.245.65
    72.21.222.149
    72.21.222.35
    72.21.220.29
    72.21.222.131
    72.21.222.147
    72.21.220.37
16. 205.251.245.65 93.9% 525 492 73.1 72.2 76.2 1.2
    72.21.222.35
    72.21.222.131
17. ???
18. ???
19. 216.182.224.79 13.5% 524 71 77.9 72.4 101.2 5.4
    216.182.224.81
    216.182.224.95
    216.182.224.77
20. 216.182.224.81 94.1% 524 492 77.9 72.8 93.0 6.3
    216.182.224.95
    216.182.224.77
21. ???

The 140.222.225.135 shows up in the traceroutes via our Verizon
Business FIOS line as well.

Will be opening a ticket with both Verizon and AWS to assist, but
hoping someone out there can take a look or chime in. Feel free to
reply off list.

Thanks,
Ray

2

Have you tried dorking around with your MTU to see if that makes a difference?

3

Not in a position to easily test that tonight (PDT) but will do so
tomorrow.

Ray

4

I am seeing the same issue between AWS US-WEST 2 and Hurricane Electric's
Fremont 2 location (Linode). Looks to be deep within Amanzon's network
based on changes in latency in a simple trace route.

I would provide an mtr, however my network configuration is something mtr
doesn't support.

Cheers!
-Tim

> We're seeing poor performance (very slow download speeds -- <

100KB/sec) to certain EC2 instances via our Verizon hosted circuits.

Have you tried dorking around with your MTU to see if that makes a

difference?

5

Realized I sent the reply to Roland. Apologies.

Here it is in full:

6

Others appear to have repoted this. Seems like Verizon is pointing at
AWS:

https://forums.aws.amazon.com/thread.jspa?messageID=558094

Ray

7

Monkeyed with the MTU -- 500, 100, 1200 -- no real observed difference
and still seeing lots of retransmits.

Ray

8

Update on this:

- We have a ticket open with both AWS and Verizon.
- AWS has responded and felt the issue was with Verizon, but notified
  their network team and asked them to investigate further.
- Nothing back from Verizon yet (anyone here have a Verizon NOC
  contact?)

In the interim, the issuer persists.

Thanks,
Ray

9

Further update -- Verizon indicates that the issue is related to
saturation on a peering link between themselves and NTT. Verizon is
pointing to the NTT side as the source of the saturation / congestion.

We don't have a direct customer relationship with NTT so am hoping
someone on this list may be able to pass this information along or
investigate on our behalf.

Ray

10

[...]
Further update -- Verizon indicates that the issue is related to
saturation on a peering link between themselves and NTT. Verizon is
pointing to the NTT side as the source of the saturation / congestion.

So, Verizon is saying that Level3 into them is congested,
NTT into them is congested...sounds like there might
be a bit of a trend happening here. I wonder if
Verizon is having trouble provisioning sufficient
capacity to support all of their customers? If
so, it would kinda suck to be stuck being their
customer at the moment if you can't get to
places you want to reach. :frowning:

We don't have a direct customer relationship with NTT so am hoping
someone on this list may be able to pass this information along or
investigate on our behalf.

Ray

I'm sure there's NTT folks watching the thread go
past, but it's unlikely they'd be in a position to
say anything in a public forum like this one way
or the other. ^_^;

Matt

11

Is there anything to be said that adds anything to what is already a well
established situation regarding Verizon vs. much of the Internet?

-dorian

12

They (Verizon) should issue a list of those which are _not_ congested. I
guess the list would be rather short...

*SCNR*

13

Fooled me once shame on you. Fooled me twice... Dont by service from
companies that allow peering wars to happen at paying customers expense
(verzon, cogent, ...)

14

To close the loop on this one, Amazon made a change for us and shifted
their peering point from NTT Ashburn to NTT Dallas.

This helped out tremendously, and although we still see times where
things are "slower", it's at least not 100KB/sec slow. :slight_smile:

Appreciate all the responses received.

Ray

15

There's one coax coming into my domicile, and the owner of the other end
of the cable isn't ripping me off *too* much, and the service pretty much
works as advertised.

There's one pair of twisted copper coming to a punchdown block just outside,
and neither the owner of the copper or their competitors seem to be motivated
to provide a DSL package I consider really acceptable.

Now, there's at least one DSL provider that has an offering that I could
deal with if the owner of the cable was in fact unable to deliver the service,
but under these conditions I'm not feeling very motivated to change providers
to make a political statement....