Q: What do ISPs really think about security issues?

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

As some of you may know, my primary job these days is tracking down
Bad Guys (tm), identifying threats, etc.

But enough of that.

One of my primary concerns has been, unsuccessfully, engaging the
networking community.

Why is that?

This "issue" is not imagined, nor is it a scare tactic -- it has,
for lack of a better analogy, grown in proportions only proportional
to the lack of engagement from ISPs.

ISPs have really, really been absent from the discussion, for various
reasons.

Is this a topic that the NANOG community would like to discuss in
a serious manner?

I'm just curious, because I'm considering submitting a "lightning
talk" at the upcoming San Jose NANOG, just to gauge & present some
of the major issues that we are seeing that could really use your
assistance.

Any input?

- - ferg

p.s. Oh, highly recommended video short (bigger bonus: Marcus
Ranum cameo):

http://www.youtube.com/watch?v=-5zxOLZ5jXM

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

As some of you may know, my primary job these days is tracking down
Bad Guys (tm), identifying threats, etc.

But enough of that.

One of my primary concerns has been, unsuccessfully, engaging the
networking community.

Why is that?

This "issue" is not imagined, nor is it a scare tactic -- it has,
for lack of a better analogy, grown in proportions only proportional
to the lack of engagement from ISPs.

ISPs have really, really been absent from the discussion, for various
reasons.

  What discussions do you vaguely speak of? nsp-sec? nsp-sec BoF
@ NANOG? FIRST? it-scc.org? commscc.org? IT-ISAC? NCS? the ITU-D
Cybersecurity stuff? NSTAC? There's all sorts of forums that relate
to security that are defined by different criteria or self-selecting.

Is this a topic that the NANOG community would like to discuss in
a serious manner?

  I think it'd discussed at NANOG and even has it's own BoF. Without
you providing more context of what you mean, it's hard to say.

I'm just curious, because I'm considering submitting a "lightning
talk" at the upcoming San Jose NANOG, just to gauge & present some
of the major issues that we are seeing that could really use your
assistance.

  - Jared

Cool, ISPs get double damned in only 24 hours.

No matter what ISPs do or don't do, someone will think they are being excessive or not doing enough.

ISPs have been active in the many security discussions and forums that have existed over the last decade (and at least one forum I think is over two decades old). Some forums are more open, other forums are more closed.
ISPs tend to be quiet in forums that mostly exist to shout at ISPs. Likewise, LEA tend to be quiet in forums that shout at LEA, banks tend to be quiet in forums that shout at banks, and so on.

On the other hand, if you listen carefully to what ISPs need, you can be fairly effective working with ISPs. However there are practical limits
to what an ISP can do.

Paul (and the list, in the off chance my mail makes it to the list),

In defense of NetSol's practice of "frontrunning" (run whois for some wicked unlikely name, say n digits of pi, observed if unregistered, if not, then go to NetSol's retail registrar site and check that string is available, say in the .com zone, do nothing else, then run the whois again and observe if the string is still available), the following claim has been made:

begin quote:
We are protecting our customers who come to our website, check
availability of a name, and come back a few hours or a day later with
the intention to purchase to find that the name is no longer available,
as it has been taken by a taster. In such cases, the customer typically
blames the registrar. In reality, however, the search information was
sold to the taster by a registry or ISP and was not the registrar's
fault.
end quote.

The "in reality" portion of this assertion is the one I'm interested in -- the assertion that "search information was sold ... by a ... ISP".

At the last open SSAC meeting (ICANN Los Angeles, November 2007), there was considerable interest in "frontrunning", but no one could point to anything other than anecdotal "evidence" for the existence, let alone the scope of "frontrunning, and personally I thought it was like Bigfoot, a non-issue pumped up at the expense of known existing issues. Obviously, I can't tell a hawk from a handsaw.

Can anyone confirm, or deny, that some ISP sells "search information" which is sufficiently timely to support the claim above, that is, that (problematic use of the "add grace period") registration(s) by "domain tasters" can be correlated with the ISP?

Nominally, "frontrunners" are Bad Guys (tm), or at least that was the hum-of-the-SSAC room in LA last November, and also nominally, "tasters" are Bad(ish) Gals (tm), and in general, the assertion is that there are bad actors who pay ISPs for data necessary for bad actions.

Note that I assume there are "bad registrars", as we've now over 1k of the little darlings now, and some are shells for the secondary auction market and the 2pm VGRS drop, and some are shells for other, more novel forms of monitizing a registrar accreditation that do not involve offering registrations to the public.

Eric
(yes, I operate a registrar which neither frontruns nor tastes nor does bulk blind sales nor ... makes money)

Paul Ferguson wrote:

On the other hand, if you listen carefully to what ISPs need, you can be fairly effective working with ISPs. However there are practical limits
to what an ISP can do.

I'll second this point. We've had great luck working with providers globally, but only after folks (such as Sean) took us under their wing and mentored us on the processes and setups that best help ISPs. That alone would make a great *NOG presentation.

Setups that best help *ISPs*? The fun part is that there's this
fundamental disconnect even within ISPs .. their CERT guys or security
guys go talk to each other, their abuse desks go talk to each other,
their packet pushers go talk to each other .. at
nspsec/gadicon/whatever, at MAAWG, at *NOG ..

There's little or no cross pollination between these groups, if at
all. It is this kind of gap that needs to be bridged, just as much
as the gaps between ISPs and LE, ISPs and the anti phishing community
(banks etc, + the takedown vendor crowd), ISPs and the security
community etc etc needs bridging.

Leads to the kind of fun situation where a guy who does CERT/security
stuff for a very large ISP was up in front of a mostly abuse desk
audience, describing the Hotlan trojan (which compromises PCs to
script account creation and spamming through various webmail sites).

He's like "they were hitting us, Y, Z .... pity I didnt know who to
contact at Y or Z at all"

That, when people from the Y and Z abuse teams (Z being us in this
story), were in the same room as the abuse team from X (which the guy
works for). And where the X, Y and Z abuse desks know each other very
well, are in constant touch over email / IM / face to face at various
conferences etc.

Talk about fundamental disconnects .. not that I know the packet
pushers from X and Y at all (the one packet pusher I knew from X
recently got assimilated by G, so that puts paid to that ..)

    --srs

disclaimer: Names replaced by X, Y and Z solely to render this little
story fit for public consumption .. it took place at a nominally
closed meeting. It wont take you too long to arrive at reasonably
plausible guesses for X, Y and Z, so I will leave you to the guessing.
No points for the right answer, no comment either .. what I'm pointing
out is general enough that it could be any X, Y and Z companies,

Yep, and X, Y and Z could be companies in any industry. I've been at conferences where the A/V presentor didn't know the other part of the same A/V company. And conferences where the ASP presentor wasn't aware what
the other part of the same ASP was doing about the same problem. And so on.

Likewise, banks don't seem to be as concerned about identity theft as
the victimes of identity theft who call their customer service reps; anti-virus vendors doesn't seem to be as concerned about malware as the victims of malware who call their customer service reps; mail service providers don't seem to be as concerned about unsolicited messages as the victims of scams contacting their customer service reps; law enforcement agents doesn't seem to be as concerned about crimes as the victims who contact their emergency numbers.

Q: What do anti-virus companies really think about security issues?

Q: What do banks really think about security issues?

Q: What do law enforcement agencies really think about security issues?

That's why I suggested to Rob and other folks the importance of listening to what they tell you how to work their particular processes. Every large
organization has them, although often the real processes are unwritten. Once you understand how the organization works, its much easier to figure
out how to make it work for you.

Shouting at the mountain won't move the mountain out of the way, and will just leave you frustrated. But ask a native nicely for help, and you might learn about the trails and passes over the mountain.

All of it translates to

1. X more mailing lists to sign up to (lots and lots more email, great)
2. X more conferences to attend (more miles, yay, that's plat for this
year taken care of)
3. A sizeable amount of reinvention of the wheel too

Fun, isn't it?

Listening is, of course, important. As is coming in with an open mind
and without a holier than thou attitude .. especially if the attitude
is combined with the sort of URGENT!! TAKE THIS PHISHER DOWN NOW!!"
abrasiveness nobody else really appreciates.

That, by the way, is why I'm glad to see more and more organizations
holding collocated / joint meetings .. across, to use some igov jargon
(and for want of a better word) "stakeholder communities" .. banks
talking to ISPs talking to LE / regulators talking to independent
researchers etc.

--srs

All of it translates to

1. X more mailing lists to sign up to (lots and lots more email, great)
2. X more conferences to attend (more miles, yay, that's plat for this
year taken care of)
3. A sizeable amount of reinvention of the wheel too

Fun, isn't it?

To begin, I hate my inbox too. I want the same thing. And yes, I know a serious part of your inbox problem comes from me and mine--all I can offer in reparations is beer. I also dislike the fact many people are clueless, but I do like the fact clueless people are starting to get clued by, to a level, re-inventing the wheel.

This email is long, I am giving you my take. What I want to see is not necessarily your thoughts on my philosophy, but rather what YOU think should be done. What would MAKE a difference in the fighting, for you?

Suresh, you *know* I am with you and that there is nothing more important to me that information sharing and cooperation. Now let me correct that to recent times, that *used* to be the most important consideration, whether some of those in need never share back or give feedback only meant we only shared some of what we have, rather than all of it--not that we won't share.

Getting cooperation inside industries, then between them, then with academics, then with law enforcement, then with policy makers. It's been a rocky ride.. but well worth it.

The first ammendment to this was the understanding that 'diversity is good', meaning; not to get upset when others choose to double resources and not cooperate. Diversity truly is great:

   * It lets new blood in
   * It creates new political presences (not necessarily powers) that
     we need to cope with, making us less close-minded
   * Helps create and foster a community
   * Proves time and again that what we believe to be evil may have
     been bad once, but is actually pretty good in the current
     landscape--we got set in our ways and set taboos (sharing virus
     samples outside the AV world, sharing C&C information, listening
     in on bad guys, etc.)

Letting efforts run free enforces a sort of Darwinian selection as far as their methods and people, but more importantly it pushes the successful ones up to our sand box.. if only we can protect them from people like us long enough.

Naturally, diversity is not *always* good, which is the second ammendment to the thinking process.

Moving on, these subjects are in fact mainstream, no longer discussed in rants by few looney people such as us. This brought some good, and naturally some bad.. but when affecting change one has to remember people need to decide for themselves and they in turn let us be successful in protecting them. Our accomplishments aside we kept what we were working on so secret that:

   * Administrators didn't have the knowledge or tools to cope (and
     they could help)
   * Public awareness was non existent (which we are suffering from
     now)
   * Political awareness was non existent (which we are suffering
     from now)

It is not about an holier than thou attitude, it's about understanding that the Internet is truly the only functioning anarchy, and that "doing" by itself makes a difference. New people who come along and will try their own way, and a sort of non-committal Darwinian seclusion or capitalism (not necessarily monetary) will determine their success. We can't stop them so may as well help them, yes?

As to current existing mail tornados of too many places to be and to see... we get less and less over time, but it is what it is, and it is about human nature. Human nature, social structures, etc.--nuff said.

Meeting the new crowd is always good, but seeing how they not only re-invent the wheel on the how to cope, but rather in their whole thinking process, I am slightly concerned. We HAVE information sharing, we HAVE cooperations. What the Internet, and we, need, is to move to the next level, whatever that may be--of course I have my ideas about that.

That means moving from good-will based relationships to something more substantial, as the criminal side has moved on long ago to billions in revenue, R&D teams, outsourcing, and kinetic [support] operations (from fraud to throat-cutting).

We are of course limited to what we *can* do:

   * Physical world efforts (law enforcment getting better,
     conferences to bring people together)
   * Intelligence gathering

Non operational:

   * Political outreach ("there is no cyber-crime problem")
   * Awareness raising

We may have achieved a LOT on our end, but at the end of the day we have made exactly a dent in the criminals' operations, and no more. We make that dent once in a while and they move on, evolving. In retrospect we haven't made any difference on their side, and they won.

Won what, you may ask. The war? We never really fought, it is a false argument that we did, and as one of the many people who are doers out there and gave a chunk of their lives to this 'fighting' I can say that and not offend myself.

Our fighting has been (mostly) limited to getting slapped, and writing analysis about it.

What I'd like to see? Here's three items on a strategic level rather than tactical, which I can go on about forever (you know I like to hear my own voice, right? :slight_smile: )

   * People working to bridge the tech-policy gap between people like
     us and policy makers (who following Estonia *are* writing
     policy which will affect us)
   * In a situation where we don't start a war not we, but rather the
     Internet can't win--actively fight back
   * These efforts stopping to be a volunteer-based 'thing' and
     moving to people who should be doing it (not people like me)

Listening is, of course, important. As is coming in with an open mind
and without a holier than thou attitude .. especially if the attitude
is combined with the sort of URGENT!! TAKE THIS PHISHER DOWN NOW!!"
abrasiveness nobody else really appreciates.

That, by the way, is why I'm glad to see more and more organizations
holding collocated / joint meetings .. across, to use some igov jargon
(and for want of a better word) "stakeholder communities" .. banks
talking to ISPs talking to LE / regulators talking to independent
researchers etc.

Indeed!
Thing is, most stop at the talking stage, which they get off their chest and will do again 6 months from now.

The Internet is not gonna die tomorrow, it is already IPv6 in Asia. :stuck_out_tongue:

Taking a step back from security, from my niche, in which I am extremely worried--as long as people can download their pr0n and argue over Captain Kirk, I am happy. Thing is, all these millions of incidents every moment are nothing but background noise.

WE CAN'T handle them, we can just jump at big ones. As long as things remain this way, my hollistic-view self will be happy, but as the awareness decreases and the background noise increases--we will eventually be "only useless" rather than "mostly useless" in bottom line net effect on the criminals. That of course unless we understand we need to do something drastically different than what failed us so far, even if it did help us get organized.

What ISPs can do? They can do a lot more than they do now. That is also a false statement as people can always do more. ISPs may be a part of the solution, but they are not the solution. We can affect how techies work, but the business folks are the ones making the decisions and making fighting criminals make business sense is not always the best use of our time.

ISPs? Some of the best and smarted people in the world work at ISPs. Unfortunately, also some of the stupidest.

Naturally, diversity is not *always* good, which is the second ammendment
to the thinking process.

Yes, diversity is actually a good idea when everybody concerned is
aware of what the others are doing, and at least coordinate to some
extent if they are in the same space.

You aren't going to achieve some monolithic conference that will
become the go-to place for everything in this field, for sure.

It is not about an holier than thou attitude, it's about understanding
that the Internet is truly the only functioning anarchy, and that "doing"

Perhaps I ought to explain.

That remark was about at least some people / groups who routinely send
takedown notices. Arrogance coupled with a sad lack of clue at one
end (lots of tier 1 techs, often outsourced to some place with far
more customer support clue than actual abuse desk clue, employed to
send alerts, without the least idea of how to send these)

One particular vendor that saw a nigerian create a free email account
dhl@[one of our domains], and went after our registrar trying to get
the domain itself canceled. Some fun ensued when I emailed all that
to the VP of their parent company (for whom takedown services appears
to be a sideline, at best). That lot has behaved themselves for a
while I must say

Another vendor who, after being given clear escalation paths, first
kept cc'ing our upstream abuse desk, and every role account OTHER than
abuse at our domain. When they finally get enough clue hammered into
them to cc our abuse desk, they escalate to my work address within two
hours of that, demanding it be taken down.

Our abuse desk would handle tix within a business day, or even
earlier. And email about phish takes priority right after (say) LE
requests that find their way there (instead of the special POC we
already have given most LE agencies). So, escalating a manual
complaint after two hours is a bit thick, I'd say.

Anyway, that particular vendor got told to take a hike, told that we
wouldnt accept any further reports from them (and that our automated
scripts kill about 20 for every one that they report anyway), and that
we'd contact the one client they seem to send these alerts for
directly and set up something more automated, where they could send us
a list (in a standard format, and verified at their end) and we'd take
it down automatically. Of course with manual review later.

Neither of those two takedown services (especially not the one in #2)
is going to get anything like this offered to them. Not until they
actually learn to play nice with other ISPs. Which comes right back
to Sean's remark that I replied to.

Sorry for the long emails, but I do wish more takedown services (and
more abuse / security desks) would read the MAAWG abuse desk best
practice document ..

http://www.maawg.org/about/publishedDocuments/Abuse_Desk_Common_Practices.pdf

--srs

Q: What do anti-virus companies really think about security issues?

http://www.nruns.com/ps/The_Death_of_AV_Defense_in_Depth-Revisiting_Anti-Virus_Software.pdf
Of particular interest are the slides on "Vendor responses"...

Q: What do banks really think about security issues?
Q: What do law enforcement agencies really think about security issues?

In order to best answer these types of questions, I suggest you first
read Geekonomics, the dotCrime Manifesto, and Secure Programming with
Static Analysis for some background.

I see a lot of you talking about information sharing, which is great.
How much overlap is there between nspsec and the Financial ISAC? Is
FIRST the place to go to sort out these issues?

This sort of conversation came up in passing on the botnets
mailing-list only a few months ago -
http://www.mail-archive.com/botnets@whitestar.linuxbox.org/msg00924.html

I don't see any particular failure of the ISP community. We all hit
our vendors pretty hard when it comes to security issues, and we
protect and respond to customer issues better than any software vendor
that I'm aware of.

If you want to get involved in security with your local bank, attend a
local OWASP meeting. If you want to get involved with law
enforcement, attend a local Infragard meeting.

dre

Another vendor who, after being given clear escalation paths, first
kept cc'ing our upstream abuse desk, and every role account OTHER than
abuse at our domain. When they finally get enough clue hammered into
them to cc our abuse desk, they escalate to my work address within two
hours of that, demanding it be taken down.

Let me guess which one it is, the same one that called 2 minutes later and threatened to go to the Police on YOU?

Our abuse desk would handle tix within a business day, or even
earlier. And email about phish takes priority right after (say) LE
requests that find their way there (instead of the special POC we
already have given most LE agencies). So, escalating a manual
complaint after two hours is a bit thick, I'd say.

Anyway, that particular vendor got told to take a hike, told that we
wouldnt accept any further reports from them (and that our automated
scripts kill about 20 for every one that they report anyway), and that
we'd contact the one client they seem to send these alerts for
directly and set up something more automated, where they could send us
a list (in a standard format, and verified at their end) and we'd take
it down automatically. Of course with manual review later.

Their client's name starts with C? :slight_smile:

Neither of those two takedown services (especially not the one in #2)
is going to get anything like this offered to them. Not until they
actually learn to play nice with other ISPs. Which comes right back
to Sean's remark that I replied to.

Sorry for the long emails, but I do wish more takedown services (and
more abuse / security desks) would read the MAAWG abuse desk best
practice document ..

http://www.maawg.org/about/publishedDocuments/Abuse_Desk_Common_Practices.pdf

Best suggestion of the thread. Now how can we make that happen? If we can give it an easily Googable name, we may be able to mention it in the press when the occasion rises. We may be able to inform them of it (nicely) in response to abuse email. What did you find works for you?

--srs

   Gadi.

<http://www.maawg.org/about/publishedDocuments/Abuse_Desk_Common_Practic
es.pdf>

Best suggestion of the thread. Now how can we make that
happen?

Check the Introduction and Methodology paragraphs of that document.

- find two or three people who will collate information
- hold several discussions on the topic (could be mailing
  list threads.
- record all the suggestions
- rewrite it all to give the document some structure
- Publish on the web

Note that in some circumstances, it may be more useful
to publish the information on the web in Wiki format
rather than trying to create a single "document". A wiki
works better if you have more than two or three people
willing to write and collate info.

But regardless of whether you try to write a "document"
or collect your guidelines in a wiki, you need to cultivate
the document by regularly circulating it, discussing it,
seeking contributions to it, etc.

--Michael Dillon

All of it translates to

1. X more mailing lists to sign up to (lots and lots more email, great)
2. X more conferences to attend (more miles, yay, that's plat for this
year taken care of)
3. A sizeable amount of reinvention of the wheel too

Fun, isn't it?

We could just meet at the Universal Postal Union meeting, and get rid
of all those extra organizations like the ITU, IETF, NANOG, etc :slight_smile:

Although communication technology evolved to postal letters, to telegraphs, and now to the Internet most of the security problems (crimes, scams, abuse and so on) have been the same for centuries.

Is it easier to teach a technologist how to investigate, or to teach
an investigator about technology?

That, by the way, is why I'm glad to see more and more organizations
holding collocated / joint meetings .. across, to use some igov jargon
(and for want of a better word) "stakeholder communities" .. banks
talking to ISPs talking to LE / regulators talking to independent
researchers etc.

Having both shared and separate meetings and communications is important.

We can all learn alot from sharing. But its also important for organizations and people to be able to communicate just with similar
organizations and people. There is an organizational dynamic that
happens. Enabling sharing within groups is as important as sharing
between groups, but they don't and usually aren't the same.

There have been several training classes and meetings I wished I could
have attended over the years, but they were closed to only people
in law enforcement, or in banking, or in a university. While personally
frustrating, I understand why.

We could just meet at the Universal Postal Union meeting, and get rid
of all those extra organizations like the ITU, IETF, NANOG, etc :slight_smile:

The fun part is, they do take a lot of interest in this too .. the US
postal service, the various European Lapostes etc - anybody who
operates a postal bank + wire transfer system and gets to face
phishers, malware and such just like regular banks do.

That wasn't an argument for consolidation .. more like "cooperation".
And the sort of cooperation that isnt aimed at making headlines and
scoring points .. stuff like (for example) surveys of the top 10 best
and worst registrars (which dont name the very worst, and include some
very good registrars among the worst, but that's another story
altogether..).

Having both shared and separate meetings and communications is important.

We can all learn alot from sharing. But its also important for
organizations and people to be able to communicate just with similar

We're on the same page there, Sean.

srs