SBC in my area (Dallas) went from wide open to outbound 25 blocked by
default/opened on request. I think doing the same thing with port 22 would
hardly be an undue burden on users, and would help keep botnets in check.
That's the problem isn't it? Who decides what can and cant go through. I think the tier approach is better, a basic user account where everything is blocked and a Sysadmin type account where everything is open. If the price is different enough then only people who are going to use those extra ports will actually pay for it.
Might as well do TCP 20, 21 and 23, too. Woah, that slope's getting slippery!
Do bots try brute force attacks on Telnet and FTP? All I see at my firewall
are SSH attacks and spam. But sure, if there's a lot of Telnet abuse block
23 too; I think it's used about as rarely by "normal" customers as SSH is.
And I'm amazed how often "slippery slope" arguments are deployed to oppose
any sort of change at all. What percentage of consumer broadband users do
you think use SSH to connect to remote servers? 1%? 0.1%? 0.01%? It seems
intuitively obvious that the number of people who will call the help desk to
unblock their SSH (which should be a ~2 minute call anyway, if not a
self-service Web page with captcha) would be an order of magnitude less than
the number of remote network users who WON'T be calling/emailing your abuse
desk to complain about bots on your network hammering their servers.
Just straight up blocking outbound ports (with the debatable exception of
port 25) seems heavy handed and too slanted toward admin convenience over
customer satisfaction. It's a slippery slope because unlike with spam,
people who are affected by brute force attacks have some degree of
complicity through either negligance or laziness. Once you decide it's
your job to protect other people's networks from their own stupidity, you
may as well do full blown deep packet inspection and look for customers
who are using your network to download kiddie porn...step by step you get
closer to losing safe harbor, or at least having to pay a lawyer to
convince otherwise.
Perhaps a more thoughtful solution would work, but would take a bit of
engineering. Ideally you would only block a certain class of
brute-forceable ports once you cross some threshold amount of brief TCP
sessions in a given period.
I would suspect an extremely low false positive rate of blocking the ports
of a user who has had, say 100 brief telnet, ssh, ftp, pop, or imap
sessions in 5 minutes.
Perhaps instead of filtering just those ports, you instead do a captive
portal where they forced to a webapp which presents them with the logs of
their issues and the suggestion they may be compromised, along with
locally reachable resources to assist in correcting the issue. But just in
case, if they feel it was an accidental situation (a perl script gone mad,
say), they could merely click a button to open them right back up.
However, three strikes and you're out until an admin reviews the case and
considers the feedback ("we run a cyber cafe, sometimes people come in
with messed up laptops") and implements a whitelisting.
Remember, YOUR customers are what matter.
Andy
Just straight up blocking outbound ports (with the debatable exception of
port 25) seems heavy handed and too slanted toward admin convenience over
customer satisfaction. It's a slippery slope because unlike with spam,
people who are affected by brute force attacks have some degree of
complicity through either negligance or laziness.
Sure, and I could* make the argument that since I have great spam filtering
inbound I don't have to care about outbound spam from my network because if
you receive it it's because of your negligence/laziness. But I think that in
the case of spam as in the case of brute force attacks it's still the
network operator's obligation to be a good netizen providing doing so places
no undue burden on his own customers or his own staff.
Blocking port 25 outbound for dynamic users until they specifically request
it be unblocked seems to me to meet the "no undue burden" test; so would
port 22 and 23. Beyond that, I'd probably be hesitant until I either started
getting a significant number of abuse reports about a certain flavor of
traffic that I had reason to believe was used by only a tiny minority of my
own users.
*but won't, ever
Blocking port 25 outbound for dynamic users until they specifically request
it be unblocked seems to me to meet the "no undue burden" test; so would
port 22 and 23. Beyond that, I'd probably be hesitant until I either started
getting a significant number of abuse reports about a certain flavor of
traffic that I had reason to believe was used by only a tiny minority of my
own users.
Sorry, I must've missed something.
Port 25 outbound (excepting ISP SMTP server) seems entirely logical to me.
Port 22 outbound? And 23? Telnet and SSH _outbound_ cause that much of a concern? I can only assume it's to stop clients exploited boxen being used to anonymise further telnet/ssh attempts - but have to admit this discussion is the first i've heard of it being done 'en masse'.
It'd frustrate me if I jacked into a friends Internet in order to do some legitimate SSH based server administration, I imagine...
Is this not 'reaching' or is there a genuine benefit in blocking these ports as well?
Mark.
The last few spam incidents I measured an outflow of about 2 messages per
second. Does anyone know how aggressive Telnet and SSH scanning is? Even
if it was greater, it's my guess there are many more hosts spewing spam than
there are running abusive telnet and SSH scans.
Frank
Frank Bulk wrote:
The last few spam incidents I measured an outflow of about 2 messages per
second. Does anyone know how aggressive Telnet and SSH scanning is? Even
if it was greater, it's my guess there are many more hosts spewing spam than
there are running abusive telnet and SSH scans.
Judging by the hits on my firewall there's a fair amount of variation
between the scanners that are doing a couple login attempts per hour, and the bot that's making thousands of login attempts with 4 or 5 connection attempts going at a time. We don't filter them till they hit a threshold.
I don't even bother to log telnet attempts anymore so I can't say much about that.
Port 22 outbound? And 23? Telnet and SSH _outbound_ cause that much of a
concern? I can only assume it's to stop clients exploited boxen being used
to anonymise further telnet/ssh attempts - but have to admit this
discussion is the first i've heard of it being done 'en masse'.
On one test machine that I leave SSH unfirewalled on, I'll see 200-4000 SSH
login attempts per day, trying to brute force it. Lets see, this morning in
an eight-minute span from one IP in Aruba 100 attempts for root; other
usernames attempted include admin, staff, sales, office, alias, stud (!),
trash, guest, test, oracle, a few personal names, apache, svn, iraf, swsoft,
gast, sirsi and nagios. And this is a relatively slow day.
Telnet I wouldn't know about, but I'm told bots will try to force it as
well.
Oh, there's plenty of names in one of my server logs too... looks almost like they've gone through a name-choosing handbook.
I can understand the logic of dropping the port, but theres some additional thought involved when looking at Port 22 - maybe i'm not well-read enough, but the bots I've seen that are doing SSH scans, etc, are not usually on Windows systems. I can figure them working on Linux, MacOS systems - but surely the vast majority of 'vulnerable' hosts are those running OS's coming from our favourite megacorp? Which typically don't come shipped with neither SSH server nor SSH client... ?
To me, at least half the users likely to be running either Linux or Mac are going to be the same users who're going to request they be allowed outbound SSH.... is the blocking of outbound SSH considered to be sufficiently useful that we're advocating it these days?
(Aren't we all just moving SSH to non-standard ports within our networks anyway?)
... Mark.
I can understand the logic of dropping the port, but theres some
additional thought involved when looking at Port 22 - maybe i'm not
well-read enough, but the bots I've seen that are doing SSH scans, etc,
are not usually on Windows systems. I can figure them working on Linux,
MacOS systems - but surely the vast majority of 'vulnerable' hosts are
those running OS's coming from our favourite megacorp? Which typically
don't come shipped with neither SSH server nor SSH client... ?
They typically don't ship with an SMTP server either. Considering that my
preferred SSH client for Windows weighs in as a single 412k .exe, I'd
imagine that bot designers are just writing their own SSH clients for
brute-forcing.
To me, at least half the users likely to be running either Linux or Mac
are going to be the same users who're going to request they be allowed
outbound SSH.... is the blocking of outbound SSH considered to be
sufficiently useful that we're advocating it these days?
Half the Mac users? You think? I know a dozen or so sysadmins who use Macs,
and about a hundred users who wouldn't know SSH from PCP; I think that's
probably a slightly skewed sample considering I'm a Mac geek who hangs
around with Mac geeks, and I'd guess the consumer users are a larger
percentage of the real-life population. I'd expect the number of folks who
want SSH unblocked to be under 1% of a consumer broadband network, and
probably closer to 0.1% or so. And again, it ought to be trivial to let your
users unblock the system, either via phone call or via self-service Web page
(though in the latter case you'd better use a captcha or something so the
bot doesn't automatically unblock itself).
.. I'm surprised botnets aren't big enough right now to do surreptitious port
scans of machines (there's 'only' 64k ports nowdays!) over timeframes measured
in weeks, from arbitrary bots (ie, not a single IP) to get a scanning footprint
to later submit in the "crack" queue.
Makes me think about Google, to be honest.
Who has more machines, botnets, or google? 
Adrian
Sorry if I wasn't more clear, but I'm not asking about inbound attempts, I'm
asking about the number of outbound attempts a host would perform.
Frank
Mark Foster wrote:
Port 22 outbound? And 23? Telnet and SSH _outbound_ cause that much of a concern? I can only assume it's to stop clients exploited boxen being used to anonymise further telnet/ssh attempts - but have to admit this discussion is the first i've heard of it being done 'en masse'.
I don't think there's much to be gained from blocking ingress 22 from customers. I don't see any SSH scans originating from my customers (though there is always the potential). I wouldn't have any issues with blocking outbound telnet though but I can't really justify it either since I don't see a real big problem with that kind of traffic originating on my network either.
Now inbound SSH and Telnet (destined for my customers) should be blocked IMHO. Doing a little prodding around our netspace I've found most SSH installs to be of a known vulnerable version or at least an old version yet to have any vulnerabilities found in. Nothing positive could come from letting them get compromised. We would of course offer a way for users to get around the block. Our current approach is to have them sign up for a static IP (another $5/month). The fee keeps everyone from automatically signing up for is as "free stuff" but still gives the legit users an inexpensive way to get what they need.
It'd frustrate me if I jacked into a friends Internet in order to do some legitimate SSH based server administration, I imagine...
Agreed but remember that people like you, I and the rest of the readers of NANOG are a teeny tiny minority on the Internet. I could pick a couple thousand of my users at random and not find one that knows what SSH is.
Is this not 'reaching' or is there a genuine benefit in blocking these ports as well?
I don't think there's much to be gained from blocking telnet & SSH from the customers to the Internet. Blocking SMTP in the same direction is critical IMHO. Blocking the same 3 ports back to the customer makes sense to me though. I think there is a real and tangible benefit to the exercise.
Justin
It varies widely. I see some extremely slow scans (1 SYN every 2-5 minutes). This is what someone on the SANS ISC page mentioned I believe.
I've also seen scans last for up to 10 minutes. The consistency of the speeds made me think that perhaps the scanning computer was on a slow link.
The worst scans are the ones that last a second or two and hit us with a SYN for every IP in our allocations. That kind of scan and its flood of packets is the one that I don't think I can stop without some sort of QoS.
I've seen coordinated scans with everything from 2 to about a dozen different hosts scanning seemingly random IPs across our network. I know it's coordinated though because together they hit every IP but never hit the same IP by more than one scanner.
I've seen scans that clearly learn where the accessible SSH daemons are, that then feed this info back to the puppet master so he can command a different compromised host (or hosts) to then handle the attacks. I've also see a scanner first scan our network and then immediately start pounding on the accessible daemons. Finally I've see the scanner stop its scan in mid-stream, pound on an accessible daemon for a while with a pre-defined set of userids and then continue on with the scans.
Clearly there's some variation in the scanning methods.
Justin
Frank Bulk wrote:
While I don't do flow monitoring today, when monitoring for outbound spam
with Wirekshark I have seen hosts systematically check all the hosts in the
block for an open SMTP port. I'm sure a lot more is going on that I don't
know. The patterns are obvious to the human observer -- too bad that such
logic isn't built into the firewall. I know there are some enterprise
security admins that subscribe to the approach that all inbound and outbound
traffic is blocked by defacto, with a few ports opened up in either
direction for known applications. Of course, port 80 becomes the port of
choice for all the undesired apps.
Frank
Dave Pooser wrote:
Half the Mac users? You think? I know a dozen or so sysadmins who use Macs,
[raises hand...]
and about a hundred users who wouldn't know SSH from PCP; I think that's
probably a slightly skewed sample considering I'm a Mac geek who hangs
around with Mac geeks, and I'd guess the consumer users are a larger
percentage of the real-life population.
I was quite surprised to see the large number of Mac laptops at NANOG 42. I didn't do a formal count but it seemed like about 1/4 to 1/3 of the laptops in use were Macs.
I'd expect the number of folks who
want SSH unblocked to be under 1% of a consumer broadband network, and
probably closer to 0.1% or so. And again, it ought to be trivial to let your
users unblock the system, either via phone call or via self-service Web page
(though in the latter case you'd better use a captcha or something so the
bot doesn't automatically unblock itself).
I'm against the slippery slope of blocking ports by default, with the possible exception of SMTP if the provider offers a well-publicized local SMTP server.
Servers that must leave ssh open to the Internet can and should consider using some form of time-out script like this one: http://www.pettingers.org/code/SSHBlack.html
I was quite surprised to see the large number of Mac laptops at NANOG 42. I didn't do a formal count but it seemed like about 1/4 to 1/3 of the laptops in use were Macs.
...You know, now that you mention it, I was also quite impressed with how many macbook pros there were in room as well. That would be good to informally track I think :
what tools(laptops) do NANOG folk tend to use?
as this data might help SW dev types to target their netTools distributions to mac platforms more quickly.
In the good ole days it seemed like 99% were PCs & maybe a couple were reinstalled with some form of unix, and the rare and incompatible couple of macs in the room were driven by freaks speaking a different language (appletalk) and hitting weirder clover keys than the rest of us.
Now, Apple seems to be the neteng laptop of choice, what the cool kids drive.
Bill
Hi,
I was quite surprised to see the large number of Mac laptops at NANOG 42. I didn't do a formal count but it seemed like about 1/4 to 1/3 of the laptops in use were Macs.
...You know, now that you mention it, I was also quite impressed with how many macbook pros there were in room as well. That would be good to informally track I think :
what tools(laptops) do NANOG folk tend to use?
Macbook Pro (all of IANA (with one recent exception) use Macs of one form or another).
as this data might help SW dev types to target their netTools distributions to mac platforms more quickly.
That would be nice.
In the good ole days it seemed like 99% were PCs & maybe a couple were reinstalled with some form of unix,
I remember the 'good old days' a bit differently -- folks were indeed using PCs (Digital HiNote Ultras and hten Sony Vaio 505TXs) but reinstallation was the norm. Maybe it was just to crowd I hung out with...
Regards,
-drc
i am moving to a macbook pro, or trying to, from a freebsd/winxp. but
why did they have to 'add value' by mucking with freebsd and breaking my
fingers? and whoever thought the mac screen was good never used my
alienware 1920x1024.
at the ipv4 econ meet on tasman last week, macs were in extreme majority.
randy