AT&T. Layer 6-8 needed.

All,

  It appears at AT&T (including DSL, and my own home service via u-verse)
has unilaterally and without explanation started blocking websites.

  I have confirmed this with multiple tests. (It actually appears that
these sites are being blocked at a local-global scale -- that is, each
city/hub seems to have blackholes for the sites).

  The sites I know of I'll list below (see Reddit for a discussion), but
this is clearly and absolutely unacceptable. Please, comments on the nature
of the sites are OT.. Let's keep this thread that way. (Away from being OT,
that is).

  If any T folk are around, and have gotten wind of this (all comments /
direct emails will be off record), a reply would be appreciated.

  No ears enclosing clue will be reached via normal channels at ~950E on a
Sunday, but this is clearly a problem needing addressing, resolution, action
and, who knows - suit?

  Thanks in advance all for insight, comments,

-jamie

I have read on another list this evening that AT&T DSL in SoCal is blocking certain sites within 4chan.

J

Joel Esler wrote:

I have read on another list this evening that AT&T DSL in SoCal is blocking certain sites within 4chan.

I just tested and can confirm the blackhole is in Reno, too. One more reason to dump ATT in addition to their trial dollar-per-gig thing they're doing here.

~Seth

img.4chan.org is the biggest site - I've already received six replies on top
of the list-replies confirming (b/c they saw this problem mentioned on
sites/blogs) filtering.

technical information, traces, bgp views (esp. from singly-homed T
customers), etc, encouraged

-jamie

Perfectly reachable from AT&T in NY:

ny01-rtr#traceroute img.4chan.com

Type escape sequence to abort.
Tracing the route to img.4chan.com (208.73.210.27)

  1 12.94.163.57 8 msec 4 msec 4 msec
  2 cr1.n54ny.ip.att.net (12.122.131.238) [MPLS: Label 16370 Exp 0] 8 msec 8
msec 8 msec
  3 ggr4.n54ny.ip.att.net (12.122.131.25) 8 msec 8 msec 4 msec
  4 192.205.34.50 16 msec 4 msec 4 msec
  5 0.xe-5-0-3.XL4.NYC4.ALTER.NET (152.63.18.10) 36 msec 4 msec 8 msec
  6 0.so-6-0-0.XL2.LAX1.ALTER.NET (152.63.57.81) 76 msec 76 msec 76 msec
  7 POS7-0.GW4.LAX1.ALTER.NET (152.63.53.61) 76 msec 76 msec 76 msec
  8 oversee-gw.customer.alter.net (65.223.29.34) 76 msec 80 msec 80 msec
  9 208.73.208.10 80 msec 80 msec 76 msec
10 img.4chan.com (208.73.210.27) 76 msec 76 msec 76 msec

Are you sure this isn't just a technical/routing issue versus a blocking
issue? Seems like everyone's out to make a sensationalist story out of this
when it's unlikely that anyone's awake at AT&T on a Sunday afternoon who
could/would make such a change.

-Dave

There has been alot of customers on our network who were complaining about ACK
scan reports coming from 207.126.64.181. We had no choice but to block that
single IP until the attacks let up. It was a decision I made with the gentleman
that owns the colo facility currently hosts 4chan. There was no other way around
it. I'm sure AT&T is probably blocking it for the same reason. 4chan has been
under attack for over 3 weeks, the attacks filling up an entire GigE. If you
want to blame anyone, blame the script kiddies who pull this kind of stunt.

Regards,
Shon Elliott
Senior Network Engineer
unWired Broadband, Inc.

jamie wrote:

That host is not on any ThreatSTOP lists. (DShield, Cyber-TA,
Shadowserver, and several others).

From: jamie [mailto:j@arpa.com]
Sent: Sunday, July 26, 2009 7:48 PM
To: nanog@nanog.org
Subject: Re: AT&T. Layer 6-8 needed.

img.4chan.org is the biggest site - I've already received six replies

on

It seems like my blocking of 207.126.64.181 is pointless, because Level3 is also
blocking the entire net 207.127.64.0.

All I can say is.. oh well. Nothing we can do about it.

jamie wrote:

It should be blocked at the complaining customer port.

Not nationwide, and certainly not without announcement.

Jamie,

Unfortunately, that's not easy with wireless backbones. The customers don't have
their own "port". I also know for fact that 4chan is in the process of moving,
so what you're seeing could just be that. Them moving.

Regards,
Shon Elliott
Senior Network Engineer
unWired Broadband, Inc.

jamie wrote:

Someone just pointed out that I dumbassedly tracerouted to img.4chan.com,
which is a linkfarm.

img.4chan.org is also reachable from AT&T in NY:

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 207.126.64.182, timeout is 2 seconds:
.!..!
Success rate is 40 percent (2/5), round-trip min/avg/max = 164/196/228 ms
ny01-rtr#

Type escape sequence to abort.
Tracing the route to img.4chan.org (207.126.64.182)

  1 12.94.163.57 8 msec 4 msec 4 msec
  2 cr1.n54ny.ip.att.net (12.122.131.238) [MPLS: Label 16377 Exp 0] 8 msec 8
msec 8 msec
  3 ggr7.n54ny.ip.att.net (12.122.131.97) 8 msec 4 msec 8 msec
  4 192.205.35.10 8 msec 4 msec 4 msec
  5 cr1-tengig-0-8-3-0.NewYork.savvis.net (204.70.198.13) 4 msec 8 msec 4
msec
  6 cr1-pos-0-3-2-3.dallas.savvis.net (204.70.192.82) 48 msec 44 msec 44
msec
  7 * * *
  8 er1-te-3-1.dallasequinix.savvis.net (204.70.204.149) 48 msec 44 msec 44
msec
  9 208.175.175.22 164 msec 172 msec 120 msec
10 unknown.xeex.net (216.152.253.26) 48 msec 48 msec 48 msec

However, it's equally as unhealthy from Comcast:

--- img.4chan.org ping statistics ---
110 packets transmitted, 77 packets received, 30% packet loss
round-trip min/avg/max/stddev = 62.635/233.576/639.919/96.254 ms

So, enough with the conspiracy theories already.

Shon Elliott wrote:

Jamie,

Unfortunately, that's not easy with wireless backbones. The customers don't have
their own "port". I also know for fact that 4chan is in the process of moving,
so what you're seeing could just be that. Them moving.

This is definitely not "them moving":

traceroute: Warning: img.4chan.org has multiple addresses; using 207.126.64.182
traceroute to img.4chan.org (207.126.64.182), 30 hops max, 40 byte packets
  1 67.118.62.1 207.264 ms 258.116 ms 174.721 ms
  2 63.201.16.134 141.205 ms 46.683 ms 41.622 ms
  3 * * *
  4 * * *
  5 * * *
  6 * * *

Traceroute from an ATT DSL account.

~Seth

http://status.4chan.org/

'Wireless backbone'?

K.

I have a dozen confirmations off list in every time zone. SANS ISC is
soliciting technical reports on this; It's on the EFF's Radar.

"This is not a drill"

If any ISP of mine filtered my (where my = brick-and-mortar-corp) access to
any destination because of another customer (there are *always* technical
solutions to problems you describe, the one you implemented wouldn't even
make my list), you'd have one less customer and quite likely a Tortious
Interference claim..

And, as a (wired) backbone arch, if I ever filtered a host (btw: there are
five IPs in that /24 being filtered by T) that cut off every customer's
access to that host or group, I'd expect to not have a job anymore.

If I wanted filtered Internet, I'd sign up for Prodigy.

Check http://status.4chan.org - they're not moving anything at the moment,
and confirm the filtering.

Debate away, I'm off to bed.

I think 4chan's reaction to this will be bigger than the story itself - No
need for me to argue what will soon be in the News Cycle.

-j

SANS ISC isn't soliciting technical reports, we're interested and looking at the issue with a particular eye to 4chan's history of pulling pranks.

Then there is the blocking because of the DoS angle, which I admit, doesn't seem to fit the facts in this case.

There are AT&T people on this list, I presume, who can speak to the issue if need be.

I'd prefer the SANS ISC not get "name dropped" as if we lend credibility to this. We're looking, sure. That's it.

j

jamie wrote:

Seth,

I said it could be, not that it is. Thanks for pointing that out. However, I
believe the reason they are being blocked at AT&T is the main reason I supplied
on my first post. The DDoS attack issue is the main ticket here. It's not
because of content, or to piss people off. It's to protect their network, as any
of you would do when you got DDoSed on your own networks. It's damage control,
essentially, until they find out who is involved and block them, then they'll
likely lift the block. This ISN'T the first time this has happened. Especially
to 4chan. You can check their status page and see most of the entries revolve
around them being down because of DDoS attacks.

Regards,

Shon Elliott
Senior Network Engineer
unWired Broadband, Inc.

Seth Mattinen wrote:

I must have misinterpreted "send us something confirming the AT&T 4Chan
outage / isc.sans.org" message.

My bad.

...have you ever heard of forged packet headers? Just saying.

William

everyone who *still* refuses to block spoofing should think hard about it.

you know who you are.

-Dan

Apparently not

Back to the kids' table !