Abuse Desks

Mailman List Mirror (Read Only)
1

I noticed over the weekend that a Fail2Ban instance’s complain function wasn’t working. I fixed it. I’ve noticed a few things:

  1. Abusix likes to return RIR abuse contact information. The vast majority are LACNIC, but it also has kicked back a couple for APNIC and ARIN. When I look up the compromised IP address in Abusix via the CLI, the APNIC and ARIN ones return both ISP contact information and RIR information. When I look them up on the RIR’s whois, it just shows the ISP abuse information. Weird, but so rare it’s probably just an anomaly. However, almost everything I see in LACNIC’s region is returned with only the LACNIC abuse information when the ones I’ve checked on LACNIC’s whois list valid abuse information for that prefix. Can anyone confirm they’ve seen similar behavior out of Abusix? I reached out to them, but haven’t heard back.
  2. Digital Ocean hits my radar far more than any other entity.
  3. Azure shows up a lot less than GCP or AWS, which are about similar to each other.
  4. Around 5% respond saying it’s been addressed (or why it’s not in the event of security researchers) within a couple hours. The rest I don’t know. I’ve had a mix of small and large entities in that response.
  5. HostGator seems to have an autoresponder (due to a 1 minute response) that just indicates that you sent nothing actionable, despite the report including the relevant log file entries.
  6. Charter seems to have someone actually looking at it as it took them 16 - 17 hours to respond, but they say they don’t have enough information to act on, requesting relevant log file entries… which were provided in the initial report and are even included in their response. They request relevant log file entries with the date, time, timezone, etc. all in the body in plain text, which was delivered.
  7. The LACNIC region has about 1/3 of my reports.

Do these mirror others’ observations with security issues and how abuse desks respond?

2

Please don't use this kind of crap to send automated "we received 3 login attempts on our SSH box..waaaaaaaaa" emails.
This is why folks don't have abuse contacts that are responsive to real issues anymore.

Matt

3

Thats what SBL is for.

-Dan

4

Do you recommend that we use a DNS blacklist to check every SSH and
HTTPS connection attempt, about whether it should be filtered or not?

Ultimately if there is scanning happening from an IP address delegated
to someone, isn't their abuse@ responsible for handling the complaints?
What are "real" issues?

We have scanning happening on ssh, https, SIP, SMTP submission ports
everyday. fail2ban does a good job blocking many of these, but
ultimately should the scanning problem be ignored? Is nobody ultimately
responsible to stop these hosts from scanning?

    Mukund

5

DDoS, hijacker, botnet C&C, compromised hosts, sufficiently-hard-to-deal-with phishing, etc are all things that carry real risk to services that are otherwise well-maintained (primarily in that many of the latter lead to the former). Nothing wrong with using or monitoring fail2ban, but if you’re spamming abuse contacts in an automated fashion (a pattern of misbehavior may be different) just because of some scanning, I recommend you fire your CSO (or get one).

Matt

6

Hi Matt

DDoS, hijacker, botnet C&C, compromised hosts,
sufficiently-hard-to-deal-with phishing, etc are all things that carry
real risk to services that are otherwise well-maintained (primarily in
that many of the latter lead to the former). Nothing wrong with using
or monitoring fail2ban, but if you’re spamming abuse contacts in an
automated fashion (a pattern of misbehavior may be different) just
because of some scanning, I recommend you fire your CSO (or get one).

It a fair game, that we the victim hosts should manually scan hundreds
of reports generated due to traffic from automated bots from IP address
block, so that things are easy for abuse@ contacts?

I haven't come across a false positive report from our fail2ban
instances on various servers (which it so far emails to our internal
email address). It appears extremely unlikely for its reports to be
false postitives - its detection method by parsing logs is identical to
what a human would manually do too.

I wouldn't call emailing its reports automatically to an abuse contact
as "spamming". It is exactly what a human would do, and
programmers/sysadmins love to automate.

If an abuse report is incorrect, then it is fair to complain.

    Mukund

7

Sadly dumb kids are plentiful. If you have to nag an abuse desk every time they sell a server to a kid who’s experimenting with nmap for the first time then.... we’ll end up exactly where we are - abuse contacts are not a reliable way to get in touch with anyone, and definitely not a reliable way to do so fast or with any reasonably large network. Please don’t clog the otherwise-useful system.

If you have trouble sleeping at night, I’d recommend the “PasswordAuthentication no” option in sshd_config.

Matt

8

Sadly dumb kids are plentiful. If you have to nag an abuse desk every
time they sell a server to a kid who’s experimenting with nmap for the
first time then.... we’ll end up exactly where we are - abuse contacts
are not a reliable way to get in touch with anyone, and definitely not
a reliable way to do so fast or with any reasonably large
network. Please don’t clog the otherwise-useful system.

If you have trouble sleeping at night, I’d recommend the
“PasswordAuthentication no” option in sshd_config.

Yes we use that, and PermitRootLogin no and an AllowUsers list.

I asked in my first email, if with security practices as above and use
of fail2ban to filter attempts, should we just ignore this problem and
think that nobody is ultimately reponsible to get rid of this activity?

From our perspecive, a dumb kid's attempts look no different to a

botnet's and we cannot distinguish. We don't know what kind of
customer/end user is generating this more than the party who has the IP
block. An exploit of a vulnerability whether it is performed by a dumb
kid or a botnet has similar consequences.

If this is generally about etiqutte of emailing abuse@, look at it from
our (target's) point of view. Assume "Joe Company"'s IP addresses send
nefarious scanning queries to our hosts. If we respond to their abuse@
contact with automated reports of such activity for TCP traffic, let's
assume 10% of those reports are false-positives. Which actor is
responsible for the worse etiquette here? Joe Company, whose IP block is
reponsible for the nefarious scanning, or us who who are reporting these
attempts using a program?

We are a small company with no CFO, CTO, CSO, CXO. We have little
resources to scan every attempt. We can ignore these attempts and turn a
blind eye, or we can automate. If there's a false positive report from
us, then use the stick and that would be fair.

    Mukund

9

Sadly dumb kids are plentiful. If you have to nag an abuse desk every time they sell a server to a kid who’s experimenting with nmap for the first time then.... we’ll end up exactly where we are - abuse contacts are not a reliable way to get in touch with anyone, and definitely not a reliable way to do so fast or with any reasonably large network. Please don’t clog the otherwise-useful system.

compromised servers on your infrastructure hosting nigerian criminals look much the same as a script kiddie experimenting with nmap.

If you have trouble sleeping at night, I’d recommend the “PasswordAuthentication no” option in sshd_config.

you either care about reports of potentially compromised hosts on your infrastructure or you don't.

-Dan

10

In theory, no. In practice, unfortunately, yes.

The typical service provider has so much low-level "noise" going on that if
everyone reported everything to them, they'd semi-literally drown.
Certainly, there's no possible way they could economically handle all that
abuse reporting -- hiring all the people to examine, determine the veracity
of, and act upon the reports would cost a fortune, because you better
believe there's no One True Format for automated abuse notifications, nor
will there ever likely be one, so it's all humans, all the time.

Now, you could argue that they should clean up their network so they don't
have that volume of abuse reports coming in -- and you'd be right, in
theory. But there's a *lot* of low-level stuff that it isn't practical to
stop, in and of itself.

Consider your own reports. What is it, exactly, that you expect a provider
to do with your report of a few failed SSH login attempts to stop the
activity? Say it's a residential ISP, or an IaaS provider. They have only
a few very large hammers at their disposal -- they can (maybe) filter the
destination port, filter your destination IP, or disconnect the customer.
Any of those will very possibly result in a support call, or lost customer.
That's a very large cost you're expecting them to pay for something which
has, let's face it, cost you practically nothing.

Forcing disconnection for a port scan is also, by the way, a *great* way to
create an absolute gold-plated A+ denial-of-service: send in a
plausible-looking report of shenanigans to the ISP of someone you don't
like, and *boom* their Internet connection's cut off. WINNAH!

So what are you left with, action-wise? An ISP could keep a tally of abuse
reports by customer, and take action on whoever's at the top of the pile,
but that would then require a large and expensive army of humans to receive,
check, classify, and record all incoming abuse reports. Do *you* want to
pay $1,000/month for your home Internet connection to cover the cost of all
those extra ISP staff? Because, as I said before, there's no One True
Format for reporting abuse, and there never will be.

Not that it would work, anyway -- any sort of "threshold" system for abuse
ends up being gamed, anyway. You only need to look at how Twitter users
with an axe to grind gang up to send in malicious reports about some other
Twitter they don't like, which trips Twitter's "lots of reports => autoban"
logic, to see how that would end if you started applying it to Internet
abuse reporting.

Finally, because nobody is ever convinced by rhetoric, here's an appeal to
your self-interest: "crying wolf" is never a good idea. In the event that
you *do* have a real problem that needs to be dealt with some time in the
future, do you want to have your e-mail address, IP address, and whatever
else associated with a thousand previous GWF ("goober with firewall")
reports? Any abuse desk who has seen your hundreds of previous unactionable
reports will almost certainly round-file that important one, and then you're
*really* up the creek sans paddle. Far better to keep your powder dry and
be ready for when you actually need assistance from whoever you're
contacting.

- Matt

11

[ "you" = rhetorical "you", throughout ]

No, the reason that folks don't have responsive abuse contacts is that
they're some combination of:

  - lazy
  - cheap [1]
  - incompetent
  - unprofessional
  - actively supporting the abusers

A "we received 3 login attempts on our SSH box" complaint should be read,
investigated, and acted on. It means that something is going on that
shouldn't, and so for your own sake, as well as for the well-being of
your Internet neighbors, you should find out what that is.

That "for your own sake" clause is often overlooked. An incoming abuse
complaint is sometimes the canary in the coal mine. Ignoring it because
it appears to be trivial at first glance is extremely foolish.

The lesson of the 75-cent accounting error is now 34 years old. This would
be a really good time to learn from it.

You should also consider that -- thanks to the negligence and incompetence of
many abuse desks -- a lot of people simply don't bother reporting incidents
any more. Thus what presents to you, on the surface, as "we received 3
login attempts on our SSH box" may in fact be one isolated report of
a much larger incident. It thus requires your immediate attention, if you
want to even pretend to be a responsible, competent professional.

Incidentally, an excellent way to reduce the number of "we received 3
login attempts on our SSH box" complaints is to deal with all of them,
thus decreasing incident occurence, which will of course result in a
corresponding decrease in complaints. An even better way is to run
your operation in such a way that you detect and deal with as many
such things as possible before anybody needs to file a complaint.
After all, if they can see the traffic arriving on their side, you can
see it leaving on yours.

---rsk

[1] I note, for example, that Charter -- which is mentioned in the
original message in this thread -- currently has a market capitalization
of 116.63 billion dollars. Surely they could spare a tiny fraction of
that to appropriately staff a 24x7 multi-lingual abuse desk with senior
engineers and empower/equip them to do what needs to done. That's
what a professional operation would do.

12

“What is it, exactly, that you expect a provider to do with your report of a few failed SSH login attempts to stop the activity?.. disconnect the customer.”

Yes.

Comcast does it. My wife’s aunt and uncle had a compromised box on their network. They don’t check their e-mail, so they didn’t see the warnings from Comcast. They shut them off until the problem was resolved.

"Forcing disconnection for a port scan is also, by the way, a great way to create an absolute gold-plated A+ denial-of-service: "

Surely they have flow records showing suspicious activity from that customer. They may not confirm the specific IP being attacked, but they will see massive numbers of SSH, SMTP, SIP, etc. connections going out. It’s likely if there’s outbound activity of that nature and someone complained about it, not only were they a victim of it, but the activity is probably undesired by anyone else receiving it as well.

“cost you practically nothing.” You’re right. An insecure Internet doesn’t cost any of us anything.

“there’s no One True Format for automated abuse notifications”

So then “let’s” make one? No one can follow it if it doesn’t exist.

13

Rich,

It’s interesting that you mention “the lesson of the 75-cent accounting error” from Cliff Stoll’s The Cuckoos Egg. Because the lesson from that account is precisely that exerting a massive human-labor-intensive effort to trace every tiny abuse signal is not worth the heavy cost — in this case, the derailing of an astronomer’s career and the infliction upon humanity of irrelevant chocolate chip cookie recipes.

An even better lesson is the comparison equation of ubiquitous low-level Internet scanning activity with astronomical Cosmic Background Radiation: a fact of life and an untraceable phenomenon of the Internet universe. Imagine if astronomers emailed the IAU every time they got a tick on their QUBIC sensors.

We live in an inflationary Internet. Exhaustively policing its CBR is a waste of time. Time better spent hardening interfaces — or eliminating them using established technologies such as VPN and TLS everywhere. Any network operator getting fail2ban reports from public IPs needs to overhaul his network, not spam the rest of us with automated abuse reports.

I mean, what lazy, cheap, incompetent, unprofessional sysadmin leaves SSH ports open to the pubic Internet? :slight_smile:

-mel beckman

14

Enforcing rate limiting comes to mind. And if there is a blatant problem then very strict rate limiting to make even surfing yahoo news a pain is a good idea.

Not to mention conn tracking and limiting to allow a customer to fix their problem is much better than a plain cut-off.

The Oh my gawd!!! I’m being port scanned … pffft it’s moot. There is blatant abuse and internet fuzz coming from legitimate sec corps that believe they are making the internet better by scanning your equipment without you asking them too and hoping to sell you services, and those are all just bullshit.

15

IMO, the answer is balance.

  • Handful of SSH connection attempts against a server. Nobody got in, security hardening did it’s job. I don’t think that is worth reporting.
  • Constant brute force SSH attempts from a given source over an extended period of time, or a clear pattern of probing, yes, report that.

As much as some pound on the table and say there shouldn’t be, there is always going to be a level of background ‘cruft’ traffic between networks. Forever. An argument was made somewhere in here that “scanning” is , by itself, a problem. I disagree. There are many legitimate use cases for certain types of scans, maps, etc. It’s true that it sometimes can be difficult to distinguish between a malicious scan and an innocent one. Proposing a solution of “stop all scanning” is absolutely a baby/bathwater angle.

I would also challenge those that say “Oh well all these companies should have perfect flow logs and pay an army of engineers to analyze them for these 5 specific TCP SYNs from 2 weeks ago.” I would bet you probably couldn’t do that either.

16

Once upon a time, Mukund Sivaraman <muks@mukund.org> said:

If an abuse report is incorrect, then it is fair to complain.

The thing is: are 3 failed SSH logins from an IP legitimately "abuse"?

I've typoed IP/FQDN before and gotten an SSH response, and taken several
tries before I realized my error. Did I actually "abuse" someone's
server? I didn't get in, and it's hard to say that the server resources
I used with a few failed tries were anything more than negligible.

I've had users tripped up by fail2ban because they were trying to access
a server they don't use often and took several tries to get the password
right or had the wrong SSH key. Should that have triggered an abuse
email?

17

Perhaps some organization of Network Operators should come up with an objective standard of what constitutes “abuse” and a standard format for reporting it.

If only there was such an organization.

18

So your theory is that it is necessary for there to be a threshold of
abuse?

Is there any reason to expect that a random server is going to be able
to figure out that a large pool of a million compromised IoT devices on
a million different IP addresses is slowly probing their server for the
root password and that a SPECIFIC probe is a member of this set?

The way this stuff is trending today, you don't have a single host that
is banging on another single host for hours or days at a password per
second, which I hope we would agree would be well beyond any reasonable
threshold to consider abuse.

On the flip side, is it so much to ask that an abuse desk maybe take a
look at both the ingress and egress packet stream of their customer, to
see if there seems to be something untoward happening?

And which one of these is a less damaging strategy?

I know we're in the minority here, but policy over here at SOL hasn't
changed much in the last quarter century. If you are getting unwanted
and unsolicited traffic from us, and you contact abuse@, we're willing
to make it stop. If it didn't originate here (forged, etc) then there
isn't much to be done -- the community has been trying to encourage
BCP38 for years.

It's probably jumping the gun a bit for a single connection attempt to
result in an abuse@ message, but then again when I look at the stream
of trash addressed at SOL's IP space, maybe not. Some of it is clearly
trying to scan from large botnets.

There's also a lot of room for computers to be doing the hard work of
detecting and reporting, and helping to analyze, while letting a human
look at what's actually transpired and see if it feels problematic.

However, the general solution that seems to have been adopted by the
majority of the industry is to hire Dave Null for abuse@

... JG

19

SRonan,

If only such a standard were feasible :slight_smile:

-mel beckman

20

Joe,

Is there any reason to have a root-enabled (or any) ssh server exposed to the bare Internet? Any at all? Can you name one? I can’t. That’s basically pilot error.

-mel