Intrustion attempts from Amazon EC2 IPs

Hi there,

Have any of you recently noticed a lot of ssh scanning coming from amazons EC "cloud" IP blocks?

Today alone I've seen approx 4m attempts from EC2 IPs on just 20 nodes on our network.

Has anyone any experience with Amazons abuse people?

Thanks,

Paul

Paul Kelly
Technical Director
Blacknight Internet Solutions ltd
Hosting, Colocation, Dedicated servers
IP Transit Services
Tel: +353 (0) 59 9183072
Lo-call: 1850 929 929
DDI: +353 (0) 59 9183091

e-mail: paul@blacknight.ie
web: http://www.blacknight.ie

Blacknight Internet Solutions Ltd,
Unit 12A,Barrowside Business Park,
Sleaty Road,
Graiguecullen,
Carlow,
Ireland

Company No.: 370845

Well, there's spam originating from there, and some cracked scripts generating
part of it. So ok, someone's found that it makes a handy platform for ssh port
probes and such as well.

  srs

Have any of you recently noticed a lot of ssh scanning coming from amazons EC "cloud" IP blocks?

Today alone I've seen approx 4m attempts from EC2 IPs on just 20 nodes on our network.

That's not too surprising, since any unix-like system that gets compromised can make a handy platform for extending the hacker's collection.

Has anyone any experience with Amazons abuse people?

Yeah, if you can call them that. There is no abuse coming from Amazon's EC2 cluster. I got the impression the only thing Amazon considers abuse is use of their servers and not paying the bill. If you're a paying customer, you can do whatever you like.

Hi there,

Have any of you recently noticed a lot of ssh scanning coming from amazons EC "cloud" IP blocks?

Today alone I've seen approx 4m attempts from EC2 IPs on just 20 nodes on our network.

Has anyone any experience with Amazons abuse people?

You are aware of the NANOG thread (68 messages) on spam coming from the Amazon cloud - thread title is
"amazonaws.com?" and it started May 23rd ? I would scan it as it deals with responses
(or lack thereof) from Amazon.

Regards
Marshall

jlewis@lewis.org (Jon Lewis) writes:

> Has anyone any experience with Amazons abuse people?

Yeah, if you can call them that. There is no abuse coming from Amazon's
EC2 cluster. I got the impression the only thing Amazon considers abuse
is use of their servers and not paying the bill. If you're a paying
customer, you can do whatever you like.

it seems that amazon has succeeded where google and microsoft failed. with
e-mail only services like hotmail and gmail, it was still possible to treat
an IP address as having a reputation, and to therefore blackhole hotmail
and gmail (and other free e-mail services) due to the spam emanating from
them, even though they are shared IP addresses and also emit much non-spam
traffic.

since EC2 (and eventually google app engine) are used for server side, and
commerce, the mere fact that probes and spam and ddos comes from these
shared IP addresses won't be sufficient grounds to reject all traffic from
them. i await with interest the final result: will most IP reputation
services simply whitelist EC2 and GAE and similar, and grit their teeth at
their inability to react to abuse from those IP addresses?

this is the end of an era. since the day i started the first RBL i have
had to listen to operators of shared IP addresses whine at me about how
they had many non-spamming customers and it wasn't fair that i blackholed
them simply because they couldn't stop it all. we went for many years
trying to find the equilibrium point between making sure IP address owners
were doing everything they could do (no pink contracts, fully staffed abuse
desk with the power to suspend or disconnect customers pending management's
later review, etc) while lots of other whiners said "vixie's gone soft on
spam, he's letting UUNET get away with murder, let's lynch him!"

with EC2, it's game-over for the IP reputation industry, other than
possibly lists of dynamic IP blocks (modems, DSL, etc) from which SMTP
ought not come. but for the wider IP address space, we now return to
content based filtering, and i predict a mighty increase in the number of
pink contracts in colo rooms. (the silver lining is, this could reduce
pressure on BGP piracy/injection.)

as randy bush often says, "it's just business." amazon has solid business
reasons for creating EC2 and there's no way it could be profitable if they
can't scale the user base, and there's no way to scale the user base if
they have to police it at the application or "intent" level. so, i'm not
whining, just pointing out that this is a sea change, the end of an era.

I agree that it's going to be difficult to deal with this on an IP
reputation basis (at least using IPv4 :-), but not certain that
means that a total lack of policing will stand long-term. The
litmus test will likely be subsequent to the first large scale P2P
service appearing in the EC2 cloud and distributing quantities
of copyright material...

/John

Hi, Paul

I was discussing this on an e-commerce practitioners list earlier today, and argued basically that, from an abuse point of view, EC2 is the same as any other bad neighborhood, and that operators needing to make impact fast, will treat it as they do any other bad neighborhood.

Best wishes
Andy

hi andy.

> with EC2, it's game-over for the IP reputation industry,

I was discussing this on an e-commerce practitioners list earlier today, and
argued basically that, from an abuse point of view, EC2 is the same as any
other bad neighborhood, and that operators needing to make impact fast, will
treat it as they do any other bad neighborhood.

i wish i agreed. a bad neighborhood that's mostly access customers or mostly
small businesses can be dealt with by address. but if it's mostly services
and most of those are things your own customers want to reach and many of
those are large, then the leverage is on the wrong end of the stick.

if we lived in an ipv6 world, such that every EC2/GAE customer had its own
dedicated IP address, not to be reused within a five year span, then blocking
by IP address would remain practical, even though blocking by IP prefix or ASN
is still ruled out. but in an ipv4 world where IP addresses are too precious
to dedicate or retire on a per-customer basis, i don't see any large eyeball
network subscribing to any IP reputation service who lists any part of EC2's
address space.

the problem with this model change is deeper than "we'll all get more spam".
in http://www.vix.com/personalcolo/ i wrote that:

  If you're an Internet user in a bad neighborhood -- as evidenced by
  your mail not getting through to a lot of people, who then tell you
  that they're blocking all mail from your ISP since there's effectively
  no abuse desk -- but you're unable/uninterested in operating your own
  secure computer in some remote facility, then you'll need to locate a
  provider who can offer you a suite of services like e-mail and web
  hosting, who does not also offer those services to spammers and script
  kiddies.
  ...
  It's worth pointing out that a "better neighborhood" might also have
  as its customers people whose content is objectionable to you, for
  example, it might also host a lot of web sites offering politics, or
  pornography, or alternative lifestyles, or alternative energy, or who
  knows what-all. Don't worry about this. Some of the neighborhoods on
  the Internet whose reputations are strongest, are the ones with the
  most diverse customer bases. The point is, don't let your local cable
  or DSL spam-haven offer you an e-mail account, or web publishing
  services, or anything else that they can't afford to support. As a
  rule of thumb, $40 per month is not enough money to pay for an abuse
  desk; and without a strong, well trained abuse desk, the neighborhood
  will be "bad".

among the distinctions being blurred by the EC2/GAE model, there is no longer
going to be a competitive advantage for companies with fully funded abuse
desks. if i'm right AOL/COX/Comcast cannot afford to blackhole EC2/GAE or to
subscribe to any IP reputation service who blackholes EC2/GAE, then the level
of inbound abuse these networks will treat as inevitable is going to rise to
the point where the effective difference between the IP reputation of an ISP
who signs pink contracts and/or has no abuse desk vs. an ISP who keeps out the
bad guys and fully funds their abuse desk will be approximately Nil. without
the ability to differentiate on this basis, a new lowest common denominator
will be found as "good" ISPs are driven out of the margin by "bad" ISP's.

jcurran's point that amazon may be forced to police itself if it becomes home
to P2P networks hosting DMCA-taggable content is interesting. this could mean
that amazon will have to re-price EC2 to include some policing costs, just to
protect its executives and shareholders. the devil will be in the details --
if this is the path we all go down together, then amazon will still have to
control its costs, and that'll mean picking the smallest possible list of
things they'll police, and i don't think SSH port knocking or botnet C&C or
open proxies will make *that* list, because they can manage those underlying
risks at lower cost on the back end than on the front end.

so in addition to ending an era, EC2/GAE/similar are beginning a new one in
which the debate about the definition of acceptable use becomes multilateral
rather than just a series of bilateral or unilateral agreements and actions.
that is the other "silver lining" in all this. if distributed computing is
a necessary utility then it may become a public utility and so EC2/GAE could
spawn an "internet public utilities commission" at the state or federal level.
and while i wouldn't like to see FCC-style "morality policing" of content, i
think that if big companies are going to create public nuisances for what are
perfectly valid business reasons, they should either pre-regulate or expect
to be post-regulated.

paul

Paul Vixie wrote:

with EC2, it's game-over for the IP reputation industry, other than
possibly lists of dynamic IP blocks (modems, DSL, etc) from which SMTP
ought not come. but for the wider IP address space, we now return to
content based filtering, and i predict a mighty increase in the number of
pink contracts in colo rooms. (the silver lining is, this could reduce
pressure on BGP piracy/injection.)

I'm not sure that shared resources are impossibly tied to anonymity, at
least when connectivity goes through a single entity. That entity is
motivated to increase usage, to help its customers expose their own
reputation (good or bad), and to host more complex services where this
concern comes up.

AWS already tracks VM instances and their internal IP allocations. They
recently added "elastic IPs," which are assigned to a customer rather
than a specific instance. To the rest of the world, they're static IPs.
AWS could expose rwhois for those elastic IPs, or delegate from
different shared and elastic blocks. Folks who care about establishing
trust would choose elastic IPs.

And while tracking NAT state for every connection would be painful, a
few thoughtful choices could go a long way -- Pareto principle or even
95/5. For example, track instances w/more than 50 open outbound
connections to dport 25; those trying to transmit a packet with a
spoofed source address (ever); and count or rate-limit SYNs per internal
instance IP.

I could also see AWS allowing customers to translate all outgoing
traffic to single customer-specific elastic IP, or even requiring it in
order to generate certain traffic profiles (quantity, velocity,
protocol, content).

There's big design considerations here - points of egress/translation,
EC2 availability zones - but they aren't insurmountable. Since the IP
is already allocated to the customer, AWS could allow them to set a
reverse DNS entry under their domain (and forward would match).

Though GAE's shared architecture creates a bit more of a challenge, it's
still not impossible. As it happens, GAE doesn't currently support many
of the features that are most useful to abusers (like raw sockets), and
may never. So the problems that prevent identifying a source entity
also prevent abuse in the first place.

Anyway, Amazon and Google are motivated and innovative, so I wouldn't
write it off.

Troy

I have to agree with Andy. There's simple math involved of how much
good mail versus how much bad mail is coming from a network, and very
few ISPs seem shy about blocking IPs or netblocks that cross those
thresholds.

Even if Paul is somehow correct about this becoming game changing for
the concept of IP reputation, good people (non-spammers) using the EC2
platform are going to run into a lot of delivery pain, as existing ISP
and blacklist reputation mechanisms have yet to give EC2 users a free
pass, from what I've observed so far.

I'm not going to pretend I manage inbound mail service for
thousands-to-millions of users (as most of the participants of other
lists like SPAM-L are fond of imagining themselves), but I know enough
about how IP reputation systems work at ISPs to know that if I did
manage inbound mail for such a userbase, the EC2 IPs would be blocked
repeatedly and often, and there would come a point where the blocks
escalate to /24s and larger, and there would come a point where the
blocks are removed slower and less often.

How the EC2 space is managed is not really new or exciting, as far as
outbound mail goes.

Regards,
Al Iverson

I think that era ended when RBLs became so difficult to maintain
without hour-by-hour effort. The sad thing is that there is no
inverse-RBLs and MTAs that work straight forward with them. In
todays, and tomorrows, world there needs to be a process of vetting
inbound SMTP connections. I would much rather deal with a process
that requires "proof of need to participate" rather than having wide
open doors and windows and only being able to use a fly swatter. The
problem with an maintaining an inverse-RBL is that it would require a
centralized and reliable entity, not just one or two joe schmoes

-Jim P.

From: Troy Davis <troy@yort.com>
...
AWS already tracks VM instances and their internal IP allocations. They
recently added "elastic IPs," which are assigned to a customer rather
than a specific instance. To the rest of the world, they're static IPs.

abusers don't have specific identities. they will find out the minimum level
of identity-checking they have to spoof, and spoof that. stolen credit cards,
throwaway domains, free e-mail accounts, and so on. before they get disco'd
they already have their next instance set up and ready to go. the game is to
live in the time margin of the ISP's reaction time, so that each fake identity
gets a predictable amount of time and resources before it's stopped/abandoned.

this is why during my time running MAPS, i focused on fully funded abuse desks
with the power to suspend or disconnect in real time, 24x7, pending management
review. warning policies or management approval increased the guaranteed
minimum useful lifetime of a fake hosting customer identity to the point where
there was no benefit in sending that ISP complaints at all. some ISPs went to
extreme lengths to tie fake identities together so as to increase the up-front
costs of serial abusers, but this inevitably raised their overall costs and
also their acquisition costs for non-abusive customers, and the only thing that
kept those increased costs from making these ISPs noncompetitive was that their
reputation would be better, and a better reputation had an offsetting benefit.

given that an static IP's reputation has inertia, and it takes days or weeks
or sometimes years for a "bad IP" to get its reputation cleaned up enough for
it to be reused, there's a window here where the pool of IP's EC2 can churn
through if it assigned them statically to potentially abusive customers may
not be large enough to also accomodate the new non-abusive load during the
period they want that churn-pool to cover. and they'll have clean-up costs
in resetting the reputation of previously abused IP's, with a natural tendancy
of IP reputation services to think that amazon, as a large company, is doing
the absolute minimum work nec'y to prevent serial abuse, such that inertia for
EC2 addresses is likely to be effectively higher than for non-EC2 addresses.

...
Anyway, Amazon and Google are motivated and innovative, so I wouldn't write
it off.

Troy

amazon and google are also large and profitable, and they know how to manage
their risks and costs to the maximum benefit of their shareholders and their
customers. this is a variation on "good, fast, or cheap: choose two". for
something like EC2 to be a financial success, it has to scale, and the trade-
offs that make scale possible also create dark corners and loopholes in which
abusers can thrive. reputation systems have generally not scaled well, but
they'll still be possible, based on content, domain name, digital signatures,
webs of trust, that kind of thing. just not IP addresses like in the old days.

paul

Even assuming Amazon will do as bad a job of policing EC2 as Paul suspects they will, I'm not at all convinced that customers would miss EC2 more than they'd miss mail from Hotmail or GMail.

Paul has said in the past that he refuses e-mail from the various free webmail services. If that works for him, great, but I suspect the typical e-mail service customer wouldn't consider the resulting spam savings worth the potential downside. If I did that on my own servers, I'd probably miss out on most of the e-mail I care most about receiving, since my friends and relatives seem to like free webmail services. Given the number of legitimate free webmail users out there, and the number of people who like getting mail from them, I suspect any service provider who tried to block them would end up with a lot of angry former customers.

Likewise, anybody blocking EC2 would miss out on whatever bad stuff might be coming out of EC2, but would miss out on being able to access services hosted there as well. Would they miss it more than they'd miss their friends on GMail? That seems far from guaranteed.

So yeah, if big shared services that include important stuff aren't being adequately policed, that's probably a problem for IP address reputation services. But that's not really a new problem being introduced by EC2.

-Steve

scg@gibbard.org (Steve Gibbard) writes:

...
So yeah, if big shared services that include important stuff aren't being
adequately policed, that's probably a problem for IP address reputation
services. But that's not really a new problem being introduced by EC2.

this may be an argument that the process i speak of has already begun, or
it may dovetail with my observation, since i know that some big ISP's have
been in blackhole wars with some big freemail providers already, whereas i
do not expect any blackhole wars involving EC2 or similar. it's one thing
to reject connections from some segment of the eyeball population on the
basis that "more ought to be done to keep abuse from coming from there" but
quite another thing to reject connection from a global e-commerce network
just because some shared IP's are sometimes used abusively.

the interpretive choice between "IP repututation systems were already doomed"
and "IP reputation services are now doomed" does not puzzle me overmuch.

Jim Popovitch wrote:

there is no inverse-RBLs and MTAs that work straight forward with
them.

  # Reject messages from senders listed in these DNSBLs
  # except for dnswl whitelust
  drop condition = ${if isip4{$sender_host_address}}
          message = blocked because $sender_host_address is \
                          in blacklist at $dnslist_domain: $dnslist_text
          !dnslists = list.dnswl.org
          dnslists = dialups.mail-abuse.org \
                          : rbl-plus.mail-abuse.org \
                          : qil.mail-abuse.com
          logwrite = REJECT because $sender_host_address listed in
$dnslist_domain

note list.dnswl.org

randy

SMTP blocks, when most of what's on EC2 doesnt actually originate
email? Access to it would be over http which isnt firewalled. Or
maybe ssh gets firewalled off.

Death by a thousand access lists. Ouch.

This simply means there must be a lot more effort - from their
upstreams, and from their peers (not in a "network sense" as much as
"large network operators who are of a sufficient size to talk to
amazon and ensure that they're heard". To convince them that some
filtering at their end, and implementation of abuse handling best
practices would be a good idea.

--srs

And it's even more significant in that large enterprise customers and others will have explicitly *whitelisted* the IP blocks associated with these services. While static IPs are available for EC2 and for the other services, as you point out, it can't last in an IPv4 world. Later on, as the technologies mature and standards emerge, we'll see automagic arbitraging of jobs/loads/tasks between clouds, so things will be even more diffuse in terms of pinpointing the actual sources of undesirable traffic or other antisocial behavior (e.g., a spam engine may be resident in one cloud and making use of network resources/proxies in another cloud, that kind of thing; 'botnets in the sky', as it were).

This is far different from free email Google or Hotmail - these cloud services (EC2, Mosso, Slicehost, Terremark's Enterprise Cloud, Telstra's new service, AppEngine, et.al.) are where many popular new Internet applications will live, and, even more significantly, where an increasing amount large-scale enterprise computing (like banking, pharma, government, and so forth) will take place.

I foresee interesting times ahead.

We're golden!

DSJ

Seems to me that blocking outgoing messages to 22/TCP should be easy enough. I'm sure there's some convoluted case where might be needed, but my guess is that losing those few customers would be worth the return in "trust". Not that the case where this is legitimate is very small - we're talking about a web app connecting to SSH servers that are outside the administrative control of the owner of the web app, as if they were in the same administrative control it would be trivial to run it on alternative ports.

Same goes for SMTP, but provide mail relays that let you send messages only from domains you have registered with EC2 - should be easy enough to validate ownership - scan whois for email addresses, and send them "Person X has asked to send mail from this domain, please pass this message on to them. $verification_url".

Sure there's other bad things that people are going to use this service for, but these seem to be the obvious ones that are easy to limit without big disruptions.

Do 'normal' web hosting providers allow customer created scripts to create TCP sessions out to arbitrary things?

- --
Nathan Ward

Doesn't PHP provide a fair amount of TCP functionality that can be used
simply by uploading the code you need to your shared web hosting account?

-brandon