Quakecon: Network Operations Center tour

If the core of the mission is local LAN play and your Internet connection fills up

You're assuming the DDoS attack originates from outside the local network(s). I was curious as to whether they'd seen any *internal* DDoS attacks.

And again, external bandwidth doesn't matter for externally-sourced DDoS attacks. If the attacker wishes to do so, he'll completely overwhelm your transit bandwidth.

.... who gives a shit? The games play on.

No, they don't, if they require a connection across the Internet to game servers for matchmaking/auth purposes, etc.

Not that often you see a bunch of people talking about a video you're
in, especially so on NANOG. So here goes.

BYOC is around 2700 seats. Total attendance was around 11,000.

2Gbps has been saturated at some point every year we have had it.
Additional bandwidth is definitely a serious consideration going
forward. It is a lot better than the 45mbps or less we dealt with 2010
and prior, but better doesn't mean good enough. Many games these days
do depend upon online services, and forced us to look for options.
AT&T has been sponsoring since then and we do appreciate it.

We have had the potential for DDoS attacks on our minds. Our first
option in those cases is blackhole announcements to the carrier for
the targeted /32. AT&T did provide address space for us to use so the
BYOC was using public IPs, and hopefully the impact of blackholing a
single IP could be made minimal. Thankfully we have not yet been
targeted, and we can only keep hoping it stays that way.

We haven't tackled IPv6 yet since it adds complexity that our primary
focus doesn't significantly benefit from yet since most games just
don't support it. Our current table switches don't have an RA guard,
and will probably require replacement to get ones that are capable.

We also re-designed the LAN back in 2011 to break up the giant single
broadcast domain down to a subnet per table switch. This has
definitely gotten us some flack from the BYOC since it breaks their
LAN browsers, but we thought a stable network was more important with
how much games have become dependent on stable Internet connectivity.
Still trying to find a good way to provide a middle ground for
attendees on that one, but I'm sure everyone here would understand how
insane a single broadcast domain with 2000+ hosts that aren't under
your control is. We have tried to focus on latency on the LAN, however
when so many games are no longer LAN oriented Internet connectivity
became a dominant issue.

Some traffic is routed out a separate lower capacity connection to
keep saturation issues from impacting it during the event.

Squid and nginx do help with caching, and thankfully Steam migrated to
a http distribution method and allows for easy caching. Some other
services make it more difficult, but we try our best. Before Steam
changed to http distribution there were a few years they helped in
providing a local mirror but that seems to have been discontinued with
the migration to http. The cache pushed a little over 4Gbps of traffic
at peak at the event.

The core IT team which handles the network (L2 and above) is about 9
volunteers. The physical infrastructure is our IP & D team, which gets
a huge team of volunteers put together in order to get that 13 miles
of cable ready between Monday and Wednesday. The event is very
volunteer driven, like many LAN parties across the planet. We try to
reuse cable from year to year, including loading up the table runs
onto a pallet to be used in making new cables out of in future years.

I imagine I haven't answered everyone's questions, but hopefully that
fills in some of the blanks.

If this has anyone considering sponsorship interest in the event the
contact email is sponsors(at)quakecon.org. Information is also
available on the website http://www.quakecon.org/.

josh,

thanks for the more technical scoop. now i get it a bit better.

We also re-designed the LAN back in 2011 to break up the giant single
broadcast domain down to a subnet per table switch.

so it is heavily routed using L3 on the core 'switches'? makes a lot of
sense.

randy

Lots of switches will happily forward layer 3 packets.

Nick

Single core switch, the Cisco 6509 VE in the video, handles routing
between subnets. Table switches have an IP for management and
monitoring. We have some 3750Gs for additional routing in other parts
of the event.

so it is heavily routed using L3 on the core 'switches'? makes a lot
of sense.

Lots of switches will happily forward layer 3 packets.

and a lot of so-called switches will happily *route* at L3, which is i
think the point. in this case, heavily subnetting a LAN, it makes a lot
of sense.

otoh, i did not believe in the fad of using 65xxs at the bgp global
edge. while it was temporarily cheap, two years later not a lot of folk
had that many boats which needed anchoring.

randy

A juniper EX9200 is a switch and a cisco sup2t box is a router. The vendor
said it so it must be true.

As anchors, I would be hard put to make a choice between a 6500 and a 7500,
which was a fine router in its day but alas only had a useful lifetime of a
small number of years. Obsolescence happens.

The distinction between layer 2 and layer 3 capable kit is not that
important these days. What's important is whether the device's packet or
frame forwarding capabilities are a good match for the expected workload
and that the total operating cost over the depreciation period works.

Nick

isn't some of L3's edge still 7500's? I think some of 703/702's edges
are still 7500's even.

"Last Date of Support:
HW
The last date to receive service and support for the product. After
this date, all support services for the product are unavailable, and
the product becomes obsolete.
December 31, 2012"

oh .. maybe they really are all gone :slight_smile:

People still run things long after EoS, heh.

A 6500 *with a Sup2T* is OK at the edge, for now - it has decent ASICs which support critical edge features, unlike its predecessors. Myself, I'd much rather use an ASR9K or CRS (I don't know much about Juniper routers) as an edge device.

We haven't tackled IPv6 yet since it adds complexity that our primary
focus doesn't significantly benefit from yet since most games just
don't support it. Our current table switches don't have an RA guard,
and will probably require replacement to get ones that are capable.

The lack of RA-guard/DHCPv6-guard can still bite you. A client can still send rogue RAs and set up a rogue DNS-server and start hijacking traffic as AAAA is preferred over A records by most operating systems these days. IPv6 first-hop security is really underrated these days and not providing the clients with IPv6 does not exclude IPv6 as a potential attack vector.

We also re-designed the LAN back in 2011 to break up the giant single
broadcast domain down to a subnet per table switch. This has
definitely gotten us some flack from the BYOC since it breaks their
LAN browsers, but we thought a stable network was more important with
how much games have become dependent on stable Internet connectivity.
Still trying to find a good way to provide a middle ground for
attendees on that one, but I'm sure everyone here would understand how
insane a single broadcast domain with 2000+ hosts that aren't under
your control is. We have tried to focus on latency on the LAN, however
when so many games are no longer LAN oriented Internet connectivity
became a dominant issue.

At The Gathering we solved this by using ip helper-address for specific game ports and a broadcast forwarder daemon (which has been made publicly available). It sounds really ugly, but it works pretty good, just make sure to rate-limit the broadcast as it can be pretty ugly in the case of a potential loop/broadcast-storm.

Some traffic is routed out a separate lower capacity connection to
keep saturation issues from impacting it during the event.

Squid and nginx do help with caching, and thankfully Steam migrated to
a http distribution method and allows for easy caching. Some other
services make it more difficult, but we try our best. Before Steam
changed to http distribution there were a few years they helped in
providing a local mirror but that seems to have been discontinued with
the migration to http. The cache pushed a little over 4Gbps of traffic
at peak at the event.

The core IT team which handles the network (L2 and above) is about 9
volunteers. The physical infrastructure is our IP & D team, which gets
a huge team of volunteers put together in order to get that 13 miles
of cable ready between Monday and Wednesday. The event is very
volunteer driven, like many LAN parties across the planet. We try to
reuse cable from year to year, including loading up the table runs
onto a pallet to be used in making new cables out of in future years.

Thanks for the write-up, it's always cool to read how others in the "LAN-party scene" does things! :slight_smile:

I help with an event that has a pretty decent sized lan party as well. We're not just focused on the lan party, more of a rock concerts - huge arcade - panels - lan party type event.

It was a few years ago that a mincraft "griefing" team came and attacked the network internally. At the time the BYOC LAN party I think was using 3com switches on the edge. Griefers were doing MAC flooding or something that was causing the switches to fall over. And not just the switch they were connected to it was bringing down many of them. They were doing it in spurts and the people dealing with the network thought the issue was misbehaving equipment for a bit (it seemed foreign at that time that someone from the community would be doing it.)

Mind you the people running things (volunteers) are running on little sleep, had no time to build out security appliances let alone watch a bunch of logs. They're pretty smart but you know - you get a bunch of smart people together they all bicker about how to do things their way.

In the end, one of the griefers friends went and told on them, and that's how they were discovered. Badges yanked and banned for life.

Most of these cons and events run on surplus hardware. Granted, these days there is more and more higher end stuff being cast away. More and more 10 gig, Juniper, Force10 and other decent equipment coming into play.

Getting bandwidth into the events is a pain. Huge venues are meant for large corporate events not lower budget cons and festivals. Venue pricing I believe is 750-1500$ per megabit. 100 megabit = $75,000 for the weekend. One year I rememeber there being a switch with 8 vlans on it sitting outside the back door with 8 clear modems spread out all blinking away.
Geeks get creative.

These days, a random family next door gets their business class FiOS paid for the entire year (with a good TV package) in return for a weekend or two a year of it being slammed. But that isn't keeping up with demand.
I think sponsorship is in our future as far as bandwidth goes.

Internally, the hotels charge for any ports. So if you need cross connects between rooms, it's pretty expensive. And it's managed by them so running tagged traffic is a no go an other things. So out comes miles of fiber and rolls of gaffers tape every year. And miles of cat5. The lan party is fairly concentrated, but other departments all have other network needs. HD video streams outbound, voip telephones, ARTNet, etc.

It's crazy. But I guess it's a good way to keep skills sharp and learn new things.

Also, Steam and others should make a caching server solution similar to what exists in Apple OSX server.

             - Ethan

Pretty much how it works on the general Internet, too, it seems.

;>

Venue Internet is the bane of events. Crazy expensive. Almost as expensive as a laborer in Chicago to move your box from the truck to your booth. :wink:

hi ethan

Getting bandwidth into the events is a pain. Huge venues are meant for large
corporate events not lower budget cons and festivals. Venue pricing I
believe is 750-1500$ per megabit. 100 megabit = $75,000 for the weekend. One
year I rememeber there being a switch with 8 vlans on it sitting outside the
back door with 8 clear modems spread out all blinking away.

for connectivity, does the hotels and convention centers still have wifi jammers
so you cannot use your own 56Mbit wifi to get connection to the outside world ?
if possible, stick a bunch of dark mirrored-glass covered vans outside the event
for wifi access

the "expensive part" is due to labor unions that control the workers and
everything else working the capitalistic "supply" and demand model to the max.
the unions disallow you to carry your own gear from your car to the event
which is good and bad ...

i dont buy their $10 budweiser, $5 water, etc especially when no outside drinks
allowed inside the event

Geeks get creative.

good thing .... and no unions to control what we did/do ...

another ( 40yr old ) boat that has long since sailed since the days
of why we had to fight off the unions in the electronics industrt ...

pixie dust
alvin

In the US, the FCC has ruled that wifi jammers violate one or more parts of
the FCC Rules and Regs. Marriott hotels paid a USD600K fine. A quick Google
search on "FCC hotel jammer" pulls up a great many hits, of which these are
the first seven:

Jammer Enforcement | FCC.gov
https://www.fcc.gov/.../jamme
U.S. Federal Communications Commission
Federal law prohibits the operation, marketing, or sale of any type of jamming equipment, including devices that interfere with cellular and Personal …

Marriott to Pay $600K to Resolve WiFi-Blocking … - FCC
https://www.fcc.gov/.../marrio
U.S. Federal Communications Commission
Oct 3, 2014 - Hotel Operator Admits Employees Improperly Used Wi-Fi Monitoring … The complainant alleged that the Gaylord Opryland was “jamming …

WARNING: Wi-Fi Blocking is Prohibited | FCC.gov
https://www.fcc.gov/.../warnin
U.S. Federal Communications Commission
Jan 27, 2015 - which hotels and other commercial establishments block wireless … into this kind of unlawful activity by the operator of a resort hotel and …

FCC warns hotels against blocking guests' wi-fi
www.consumeraffairs.com/.../fcc-warns-hotels-against-blocking-guests-...
Jan 28, 2015 - Hotels, miffed by guests who used their own wi-fi hotspots instead of paying ... It's illegal to jam legal radio transmissions of any kind, FCC vows tough enforcement ... Some had argued that jamming wi-fi and cellphone calls is ...

Hotels ask FCC for permission to block guests' personal Wi ...
www.pcworld.com/.../hotel-group-asks-fcc-for-permission-to-...
PC World
Dec 22, 2014 - Marriott argued some hotspot blocking may be justified, as long as the hotel isn't using illegal signal jammers. Unlicensed Wi-Fi hotspots ...

FCC fines Marriott $600,000 for blocking guests' Wi-Fi ...
www.cnn.com/2014/10/03/travel/marriott-fcc-wi-fi-fine/
CNN
Oct 4, 2014 - It's the first time the FCC has investigated a hotel property for ... sense, where someone uses a jammer device to block wireless signals. Instead ...

How This Hotel Made Sure Your Wi-Fi Hotspot Sucked ...
readwrite.com/2014/.../marriott-nashville-opryland-jams-wifi-internet-wt...
Oct 4, 2014 - Caught by FCC for Wi-Fi jamming, Marriott's still not sorry.

I travel quite a bit worldwide, and I've never run into this. I run my portable AP on 5GHz, FWIW.

The WiFi jammers have an interesting MO. They don't throw up static on the
frequency, that would also block their own wifi. They spoof
de-authentication packets. I've been looking for a way to detect this kind
of jamming because my WiFi sucks and I live next to three hotels, what you
get for living in downtown Atlanta.

Sure - I'm saying, I don't see this anywhere, is it possible most of this activity is on 2.4GHz and not 5GHz?

hi mr bugs :slight_smile:

The WiFi jammers have an interesting MO. They don't throw up static on the
frequency, that would also block their own wifi. They spoof
de-authentication packets. I've been looking for a way to detect this kind
of jamming because my WiFi sucks and I live next to three hotels, what you
get for living in downtown Atlanta.

i forgot if kismet showed signal strengths of the wifi ap's ...
"stronger" signal wins over weaker signal strengths

might not be a jamming issue ?? kismet and tcpdump might be able to show
you the packets you're looking for ?

what happens if you put up a properly designed wire mess around the exterior
windows of your house/condo/aptr??

i'd wag/blindly say the area is probably full of rogue wifi ap's floating around
where evergbody is trying to wardrive each other and pick up un-suspecting
traveling visitor's login and passwd info ... signals bouncing off steel/concrete
is not ez to filter out what should be random background white noise if
you're sitting next to the radiating source ..

pixie dust
alvin
# DDoS-Mitigator.net
# DDoS-Simulator.net