Interesting new spam technique - getting a lot more popular.

Mailman List Mirror (Read Only)
1

http://thespamdiaries.blogspot.com/2006/02/new-host-cloaking-technique-used-by.html

Does seem to have potential, because at least one large webhost says
they got bit hard by this (when they asked me to unblock one of their
/24s) - and I've been seeing the same type of spam for quite some time
[pizzabox cpanel servers running exim, but emitting porn spam with
completely different hostnames and qmail headers]

<quote>

So this brings us today to a new technique reported by Stephen
Satchell of satchell.net last Thursday. It reads almost like a mystery
novel, involving cloaking, promiscuous interfaces, stolen IP
addresses, and tunneling. It gets a little tricky, so follow the
bouncing ball:

    * The spammer obtains a dedicated server at the victim service
provider. The server shares a subnet with other customers.

    * A spamware daemon is installed on the dedicated server, to keep
the network interface in promiscuous mode

    * The daemon determines which IP addresses on the local subnet are
not in use. It also determines the addresses of the network routers.
One or more unused IP addresses are commandeered for use by the
spammer.

    * The perp server sends unrequested ARP responses to only the
gateway routers, so that the routers never have to ask for a layer-3
to layer-2 association -- it's alway in the ARP cache of the routers.
Nobody else sees this traffic in an EtherSwitch fabric, so ARPWATCH
and its kin are defeated. Pings and traceroutes also fail with "host
unreachable.". The daemon then only has to watch on the NIC, in
promiscuous mode, for TCP packets to the hijacked address on port 80,
and pass them down the tunnel to the remote Web server.

    * Finally, GRE and IPIP tunneling is used to connect the stolen IP
addresses to the spammer's real servers hosted elsewhere.

The end result is that the spammer has created a server at an IP
address which not even the owners of the network are aware of.

There are a number of ways you can protect your own network from from
this exploit:

    * Give each customer their own subnet.

    * Null-route unused IP addresses in your network space, or
otherwise make sure that there's a legitimate server somewhere that
will claim them.

    * Monitor your local network for interfaces transmitting ARP
responses they shouldn't be.

</quote>

2

how about just mac security on switch ports? limit the number of mac's at
each port to 1 or some number 'valid' ?

3

That was not my advice btw - just forwarding on what I saw.

What you say does seem like a "must do" all right - but putting ARP
filters in is actually a reasonable idea.

4

That was not my advice btw - just forwarding on what I saw.

oh,. apologies, i did cut the message down quite a bit :frowning: I understood you
were quoting from the spamdiaries website, I apologize to the other
listeners (readers?) if it confused the issue.

What you say does seem like a "must do" all right - but putting ARP
filters in is actually a reasonable idea.

Atleast it'd trim down the 'problem' to the single customer subnet, I
assume that dedicated hosting folks don't just drop machines behind a
switch on one big flat subnet? That's probably a naive assumption though
:frowning: Perhaps this is clue #12 that that is a 'less than good' option? :slight_smile:

5

Given the people who complained, and their traditionally spammer
infested nature I wouldnt be surprised at all to find that they've put
all their hosts on a flat subnet

Various /24s of theirs keep getting complimentary upgrades from our
filters after reaching a certain limit - based on a X IPs blocked per
/24, Y /24s per /16 metric .. when that limit is reached, we
automatically upgrade the blocks to cover infested /24s.

6

[...]

I assume that dedicated hosting folks don't just drop machines
behind a switch on one big flat subnet? That's probably a naive
assumption though

I've long been a proponent of a per-customer VLAN or L3 interface,
depending on what the topology allows for, but I'm afraid we're in the
minority.

From what I've seen, the overwhelming majority of "dedicated hosters"

do precisely what the article alludes to -- placing hundreds (if not
thousands!) of disparate hosts on the same broadcast domain, with no
safeguards in place to prevent ARP spoofing, IP hijacking, and other
forms of malfeasance...

-a

7

[...]
> I assume that dedicated hosting folks don't just drop machines
> behind a switch on one big flat subnet? That's probably a naive
> assumption though

I've long been a proponent of a per-customer VLAN or L3 interface,
depending on what the topology allows for, but I'm afraid we're in the
minority.

doh :frowning:

>From what I've seen, the overwhelming majority of "dedicated hosters"
do precisely what the article alludes to -- placing hundreds (if not
thousands!) of disparate hosts on the same broadcast domain, with no
safeguards in place to prevent ARP spoofing, IP hijacking, and other
forms of malfeasance...

is it really that hard to make your foudry/extreme/cisco l3 switch vlan
and subnet??? Is this a education thing or a laziness thing? Is this
perhaps covered in a 'bcp' (not even an official IETF thing, just a
hosters bible sort of thing) ?

8

This problem is fixed by following the BCP regarding spoof filtering, if needed, doing the IP source filtering at the switchport instead of at the router level. Treat your colo customers the same way you would residential customers with the same security level.

Whatever the customer himself can change, control. IP spoof filtering, and if your platform supports it, even rewrite the MAC address so it's local to the access cable and not used in your aggregation network (some DSLAM vendors do this, for instance). I haven't seen any switch vendors that does this yet, unfortunately.

9

Date: Wed, 14 Jun 2006 04:46:31 +0000 (GMT)
From: Christopher L. Morrow

is it really that hard to make your foudry/extreme/cisco l3 switch vlan
and subnet???

Of course not.

Is this a education thing or a laziness thing?

Both.

Eddy

10

Simple: Subnets are hard, customers are stupid, and ARIN is not exactly a
hosters best friend.

When a hosting customer asks for 5 IPs today and 25 IPs tomorrow, it is
infinitely easier for the hosting folks to just slap them into /24s and
say "ok uhm you are now .69-.94" than to try and explain subnets, cidr,
reserving IP space in cidr sized blocks etc to the customer. Hosters are
also generally under-equipped in the paperwork and detailed documentation
department, so they tend to run their IP allocations into the ground while
attempting to explain their need for more space. CIDR allocations are
"wasteful" to them, especially when a customer needs to expand from 30 IPs
to 35 IPs and crosses a new boundry.

Incase you've never seen hoster configs, they generally look a little
something like this:

ip address 1.1.1.1 255.255.255.0
ip address 1.1.2.1 255.255.255.0 secondary
ip address 1.1.3.1 255.255.255.0 secondary
ip address 1.1.4.1 255.255.255.0 secondary
ip address 1.1.5.1 255.255.255.0 secondary
...

Anything else is quite honestly beyond 99% of hosters out there, they're
still blissfully calling these things "class c's". I've seen some truly
godawful thins configured by hosters, like chains of 3548s all linking
back to a single router interface in ways you can't even imagine.

If you made it dirt simple for them they would probably be doing something
better (I usually point folks who ask to pvlans, then take the opportunity
to make a hasty retreat while they are distracted), but otherwise they
don't see the benefit in it. Why bother configuring your router better
when you can just send your $5/hr monkey over with a redhat cd and have
them reinstall, right? :slight_smile:

11

Hi,

Just to be clear, simple L2 mac security doesn't help here.

This attack (arp spoofing on a shared subnet) does not involve more than
one mac per switch port. Nor are there any changes in switch port / mac
associations.

You need to watch at the higher layers (arp, ip).

Cheers

12

is it really that hard to make your foudry/extreme/cisco l3 switch vlan
and subnet??? Is this a education thing or a laziness thing? Is this
perhaps covered in a 'bcp' (not even an official IETF thing, just a
hosters bible sort of thing) ?

Subnets aren't exactly good for address space usage.

For Cisco kit, there are numerous nerd knobs that can be deployed that would
seemingly mitigate this spam technique.

In short, IP Source Guard ("stop malicious people from using IP addresses
that weren't assigned to them"), Port Security ("limit # of mac addresses on
a given port to X") and Dynamic ARP Inspection ("discard bogus arp
packets").

Combined with things like Private VLANs (allow different customers to share
the same subnet but restrict them being able to talk/see one another), there
are ways of securing things.

Of course, just like everything its up to folks to deploy them. Many of
these knobs aren't safe or practical for "default" settings.

I'm sure other vendors have similar features also.

Yes, these have been presented on numerous times within Cisco forums (e.g.
Networkers) as best practice & are typically very well attended.
Not necessarily by the all the folk that need to, I guess. :frowning:

cheers,

lincoln.

13

And in some cases even a nasty fincancial thing. Billing customers extra
datatraffic due to a large amount of broadcast traffic (especially when
running badly configured Win32 servers) inside a single /23 or even /22
in one large VLAN is sadly still the case for some hosters.

14

* Christopher L. Morrow:

15

* Christopher L. Morrow:

is it really that hard to make your foudry/extreme/cisco l3 switch vlan
and subnet??? Is this a education thing or a laziness thing?

You need those L3 switches before you can do this. Obviously, L2 gear
is much cheaper, and will work equally well until it is attacked.

Even with L3 switches, adressing it is a bit tricky. Not all vendors
support point-to-point adressing on Ethernet interfaces (certainly
some host software doesn't). There are universal subscriber gateways
that simply override all network configuration on the host, but they
aren't marketed at datacenters AFAIK. After all, who would think that
a datacenter needs a network security policy similar to that of a
hotel offering Internet access in its rooms?

16

Only if you also filter _OUTGOING_ traffic, by port, to allow only the
destination IPs that the customer equipment should be seeing.

Filtering the ingress direction (customer equipment -> your network)
does not help (until _everyone_ does it), since the spammer only needs
to _receive_ traffic with the hijacked IP, not send it (that can be
done from the other corner of the spammer's triangle route).

17

As a hoster with many customers on large shared VLANs perhaps I can add a bit...

Simple: Subnets are hard, customers are stupid, and ARIN is not exactly a
hosters best friend.

When a hosting customer asks for 5 IPs today and 25 IPs tomorrow, it is
infinitely easier for the hosting folks to just slap them into /24s and
say "ok uhm you are now .69-.94" than to try and explain subnets, cidr,
reserving IP space in cidr sized blocks etc to the customer. Hosters are
also generally under-equipped in the paperwork and detailed documentation
department, so they tend to run their IP allocations into the ground while
attempting to explain their need for more space. CIDR allocations are
"wasteful" to them, especially when a customer needs to expand from 30 IPs
to 35 IPs and crosses a new boundry.

I'm not sure documentation is really THAT much of it. I mean, we have subnets behind firewalls and manage subnet allocations there. It is a hassle compared to the simplicity of large shared VLANs, but we're certainly capable of tracking it. The problem is more for the customers with only a few servers or even just a single server. They tend to have no idea about future IP requirements. Ask any hoster CEO who isn't a hands-on networking person what his expectations are for near-future IPs and he'll generally give you some wild answer like "up to 50,000" because he wouldn't be CEO if he didn't expect his business to explode overnight. :slight_smile: In general, customers with larger firewalled solutions (and thus a private subnet and private behind-firewall VLAN) tend to have a better handle on this.

Also, have a requirement that I must be able to accept a customer server plugging into my network anywhere. If a customer buys 1 server today, then another server 6 months from now, that second server may be on a different floor of the datacenter, or even a few miles away in a nearby datacenter. A few months after setting these servers up, the customer might want to migrate a single IP from one server to another. So, since I must carry every VLAN everywhere, that can get tricky (not impossible, but messy) when you have thousands of customers, with say 2-5 VLANs each. With MSTP the spanning tree limit of a 6500 is 50,000 logical interfaces (ports*vlans). At 5000 VLANs that would mean I'm only allowed to use 10 all-vlan-carrying tagged ports on a 6500, which wouldn't make for a very efficient core.

There is also strong demand among web hosting customers to scatter sites across multiple /24's due to search engine optimization. This is trivial to satisfy in the large shared subnet model.

Finally, as we all know, there is the efficiency issue. Of course, as long as ARIN lets people get away with it, it isn't that big of a deal (other than being good netizens) so I won't go into this one much.

Incase you've never seen hoster configs, they generally look a little
something like this:

ip address 1.1.1.1 255.255.255.0
ip address 1.1.2.1 255.255.255.0 secondary
ip address 1.1.3.1 255.255.255.0 secondary
ip address 1.1.4.1 255.255.255.0 secondary
ip address 1.1.5.1 255.255.255.0 secondary

Yep, the ip address section typically looks like that, plus all the usual stuff like GLBP, policy routing, etc...

Anything else is quite honestly beyond 99% of hosters out there, they're
still blissfully calling these things "class c's". I've seen some truly
godawful thins configured by hosters, like chains of 3548s all linking
back to a single router interface in ways you can't even imagine.

I can't speak for other hosters, but for me it is more about different customer bases (some customers have no idea what their requirements are) and different business requirements. I think we are quite aware of the issues either way, and know exactly what the advantages and disadvantages are. If anything, I'd say we're very much experts in the field of large layer 2 networks. Oh, and there are no chains of 3548's here. All 6500s, each one directly attached to at least 2 cores.

If you made it dirt simple for them they would probably be doing something
better (I usually point folks who ask to pvlans, then take the opportunity
to make a hasty retreat while they are distracted), but otherwise they
don't see the benefit in it. Why bother configuring your router better
when you can just send your $5/hr monkey over with a redhat cd and have
them reinstall, right? :slight_smile:

We use pvlans successfully (though it has been a long bug-ridden journey). This mitigates just about all of the disadvantages of the big shared VLAN while maintaining all the advantages. The one disadvantage that pvlans does not mitigate is unused IPs. Thus, I have been lobbying Cisco since 2002 to fix all the pvlan bugs (almost done there!) and for the ability to add bulk static arps and stop even supporting dynamic ARPing for customer IPs (no real progress on this front). For now we have to settle with detecting unused address hijacks. Oh, and for those of you out there saying static arps are already implemented, you try putting 30,000+ in your config and see what happens...

As for monkeys, there is certainly a business case for simplifying. If I can do some hard back-end planning and setup one time to make the future everyday tasks easier, I see that as an advantage. People ask how do I move a single secondary IP between servers that got set up in different datacenters in the same city. I respond just change them on the servers, click "clear arp" on the web interface, see it ping, then update the documentation to record the change. If a monkey can figure out how to handle adds/moves/changes without danger of breaking anything else on the network then I see that as doing something right. Not everything should require a network engineer to accomplish.

Oh, and you will only hear me say "class C" in cases where I am responding to a customer (or salesperson) who asked about a "class c". If they use the terminology, I'll use it back at them to avoid confusion.

18

Note that if you're reading this list, you have already identified
yourself as a non-typical hoster. Go read WHT or GFY for 10 minutes for an
example of typical hosters, and if you're not a drooling idiot in need of
a brain transplant afterwards consider yourself lucky. :slight_smile: And don't
forget, there are hundreds of hosting networks like the ones I described,
a lot of whom are in the 1 - 30Gbps traffic range, with absolutely no clue
how to do better.

19

And let me tell you.. inheriting a network like that, knowing a better way to do it, will make you want to put a gun in your mouth. Two /19's worth of address space in VLAN1 (not just in one vlan, but in vlan *1*. Cisco nerds are slapping foreheads or spitting Coke right now.)

Trying to migrate customers to their own vlan when they've been alloted IPs, willy nilly, across one of the bajillion /24's secondaried on the vlan interface drives me into an entire new dimension of pissed off.

Don't even get me started on allocation and traffic accounting.

- billn

20

Bill Nash wrote:

Trying to migrate customers to their own vlan when they've been alloted
IPs, willy nilly, across one of the bajillion /24's secondaried on the
vlan interface drives me into an entire new dimension of pissed off.

Unless I am missing something obvious, it seems like rfc 3069 (sub/super
vlans) provides an easy (interim?) solution to this dilemma.