Repeated Blacklisting / IP reputation

Greetings,

We obtained a direct assigned IP block 69.197.64.0/18 from ARIN in 2008. This block has been cursed (for lack of a better word) since we obtained it. It seems like every customer we have added has had repeated issues with being blacklisted by DUL and the cable carriers. (AOL, AT&T, Charter, etc). I understand there is a process to getting removed, but it seems as if these IPs had been used and abused by the previous owner. We have done our best to ensure these blocks conform to RFC standards, including the proper use of reverse DNS pointers.

I can resolve the issue very easily by moving these customers over to our other direct assigned 66.254.192.0/19 block. In the last year I have done this numerous times and have had no further issues with them.

My question: Is there some way to clear the reputation of these blocks up, or start over to prevent the amount of time we are spending with each customer troubleshooting unnecessary RBL and reputation blacklisting?

I have used every opportunity to use the automated removal links from the SMTP rejections, and worked with the RBL operators directly. Most of what I get are cynical responses and promises that it will be fixed.

If there is any question, we perform inbound and outbound scanning of all e-mail, even though we know that this appears to be something more relating to the block itself.

Does anyone have any suggestions as to how we can clear this issue up? Comments on or off list welcome.

Thanks,

Tom Pipes wrote:

Greetings,

We obtained a direct assigned IP block 69.197.64.0/18 from ARIN in 2008. This block has been cursed (for lack of a better word) since we obtained it. It seems like every customer we have added has had repeated issues with being blacklisted by DUL and the cable carriers. (AOL, AT&T, Charter, etc). I understand there is a process to getting removed, but it seems as if these IPs had been used and abused by the previous owner. We have done our best to ensure these blocks conform to RFC standards, including the proper use of reverse DNS pointers.

I can resolve the issue very easily by moving these customers over to our other direct assigned 66.254.192.0/19 block. In the last year I have done this numerous times and have had no further issues with them.

My question: Is there some way to clear the reputation of these blocks up, or start over to prevent the amount of time we are spending with each customer troubleshooting unnecessary RBL and reputation blacklisting?

I have used every opportunity to use the automated removal links from the SMTP rejections, and worked with the RBL operators directly. Most of what I get are cynical responses and promises that it will be fixed.

If there is any question, we perform inbound and outbound scanning of all e-mail, even though we know that this appears to be something more relating to the block itself.

Does anyone have any suggestions as to how we can clear this issue up? Comments on or off list welcome.

Thanks,

--- Tom Pipes T6 Broadband/ Essex Telcom Inc tom.pipes@t6mail.com

Unfortunately, there is no real good way to get yourself completely delisted. We are experiencing that with a /18 we got from ARIN recently and it is basically the RBL's not updating or perhaps they are not checking the ownership of the ip's as compared to before. On some RBL's, we have IP addresses that have been listed since before the company I work for even existed. Amazing right?

Folks -

   It appears that we have a real operational problem, in that ARIN
   does indeed reissue space that has been reclaimed/returned after
   a hold-down period, and but it appears that even once they are
   removed from the actual source RBL's, there are still ISP's who
   are manually updating these and hence block traffic much longer
   than necessary.

   I'm sure there's an excellent reason why these addresses stay
   blocked, but am unable to fathom what exactly that is...
   Could some folks from the appropriate networks explain why
   this is such a problem and/or suggest additional steps that
   ARIN or the receipts should be taking to avoid this situation?

Thanks!
/John

John Curran
President and CEO
ARIN

John, its about the same situation you get when people use manually
updated bogon filters.

A much larger problem, I must admit .. having ISPs follow the maawg
best practices might help, that - and attending MAAWG sessions
(www.maawg.org -> Published Documents)

That said most of the larger players already attend MAAWG - that
leaves rural ISPs, small universities, corporate mailservers etc etc
that dont have full time postmasters, and where you're more likely to
run into this issue.

If you see actual large carriers with outdated blocklist entries after
they're removed from (say) the spamhaus pbl, then maybe MAAWG needs to
come to nanog / arin meetings .. plenty of maawg types attend those
that all that needs to be done is to free up a presentation slot or
two.

--srs

Suresh Ramasubramanian wrote:

That said most of the larger players already attend MAAWG - that
leaves rural ISPs, small universities, corporate mailservers etc etc
that dont have full time postmasters, and where you're more likely to
run into this issue.
  

I've found the opposite to hold true more often. Smaller organizations can use public blacklists for free, due to their low volume, and so have little incentive to run their own local blacklist. I've typically seen the larger organizations run their own blacklists and are much more difficult to contact for removal.

MAAWG's changing that - and most of the large ISPs are maawg members.
Things can get better, sure ...

--srs

Suresh Ramasubramanian wrote:

John, its about the same situation you get when people use manually
updated bogon filters.

A much larger problem, I must admit .. having ISPs follow the maawg
best practices might help, that - and attending MAAWG sessions
(www.maawg.org -> Published Documents)

That said most of the larger players already attend MAAWG - that
leaves rural ISPs, small universities, corporate mailservers etc etc
that dont have full time postmasters, and where you're more likely to
run into this issue.

I was always under the impression that smaller orgs were not allowed to
join the MAAWG club.

~Seth

John Curran wrote:

Folks -

   It appears that we have a real operational problem, in that ARIN
   does indeed reissue space that has been reclaimed/returned after
   a hold-down period, and but it appears that even once they are
   removed from the actual source RBL's, there are still ISP's who
   are manually updating these and hence block traffic much longer
   than necessary.

   I'm sure there's an excellent reason why these addresses stay
   blocked, but am unable to fathom what exactly that is...
   Could some folks from the appropriate networks explain why
   this is such a problem and/or suggest additional steps that
   ARIN or the receipts should be taking to avoid this situation?

I don't think there is an excellent reason, more likely inertia and no real incentive to put forth the effort to proactively remove addresses.

Many ISPs and organizations have their own private blocklists not associated with the widely known DNSBLs. Typically during or immediately after a spam run the mail administrator will manually add offending addresses or netblocks. Spamtrap hits may do this automatically. There isn't any real incentive for people to go back and remove addresses unless they're notified by their own customers that legitimate mail coming from those addresses is being blocked. Because these blocklists are individually maintained, there is no central registry or means to "clean them up" when an IP assignment changes.

To make matters worse, some organizations may simply ACL the IP space so that the TCP connection is never made in the first place (bad, looks like a network problem rather than deliberate filtering), some may drop it during SMTP with no clear indication as to the reason (less bad, as there is at least a hint that it could be filtering), and some may actually accept the mail and then silently discard it (worst).

In addition there are several DNSBLs with different policies regarding delisting. Some just time out after a period of time since abuse was detected. Some require action in the form of a delisting request. Some require a delisting request and a time period with no abuse. Some (the old SPEWS list) may not be easily reached or have well defined policies.

In meatspace, once a neighborhood winds up with a reputation of being rife with drive-by shootings, gang activity and drug dealing it may take a long time after the last of the graffiti is gone before some cab drivers will go there.

Most small to midsize networks probably have a "block it and forget it" policy. The facts that the spammer moved on, the IPs eventually got returned to the RIR and reallocated to a different network go unnoticed until the new LIR/ISP notifies those blocking the addresses that the addresses have changed hands. Ideally, the network doing the blocking will know when they started blocking an IP, look at whois, and agree that the block no longer makes sense. I'm sure some will have no idea when or why they started blocking an IP, and might be reluctant to unblock it. This assumes you can actually get in touch with someone with the access and understanding of the issues to have a conversation about their blocking. Some networks make that nearly impossible. I ran into such situations early on when trying to contact networks about their outdated bogon filters when Atlantic.net got a slice of 69/8.

This blocking (or variations of it) has been a problem for about a decade.

http://www.michnet.net/mail.archives/nanog/2001-08/msg00448.html

I don't think there is any blanket solution to this issue. Too many of the networks doing the blocking likely don't participate in any forum where the RIRs will be reach people who care and can do something.

John Curran wrote:

<snip>

  I'm sure there's an excellent reason why these addresses stay
  blocked, but am unable to fathom what exactly that is...
  Could some folks from the appropriate networks explain why
  this is such a problem and/or suggest additional steps that
  ARIN or the receipts should be taking to avoid this situation?

I don't think there is an excellent reason, more likely inertia and no real incentive to put forth the effort to proactively remove addresses.

<snip>

In addition there are several DNSBLs with different policies regarding delisting. Some just time out after a period of time since abuse was detected. Some require action in the form of a delisting request. Some require a delisting request and a time period with no abuse. Some (the old SPEWS list) may not be easily reached or have well defined policies.

In meatspace, once a neighborhood winds up with a reputation of being rife with drive-by shootings, gang activity and drug dealing it may take a long time after the last of the graffiti is gone before some cab drivers will go there.

--
Jay Hennigan - CCIE #7880
<snip>

I think this most accurately reflects the reality I see dealing with mostly enterprises and mid-to-large xSPs.

A lot of mid-range enterprises out there have legacy "free" (often meaning "subscriptions aren't enforced") DNSBLs in place that were configured years ago as a desperate attempt to reduce e-mail load, before there were well-maintained alternatives. The problem is that these services usually don't have the resources to put a lot of advanced automation and sophisticated logic into place, so delisting is a huge hassle (and some times resembles extortion).

There are some quality "free" services, such as Spamhaus (speaking personally), but they're few and far between.

I've had better luck convincing customers (or customers of customers) to stop using the poorly-maintained legacy DNSBLs than I've had getting customers delisted from such services.

YMMV.

Brian Keefer
Sr. Solutions Architect
"Defend email. Protect data."

If I'm a smaller shop with limited clue, there's 3 likely colloraries:

1) Even a smallish spam blast is big enough to cause me operational
difficulties, so I'm tempted to throw in a block to "fix" it.

2) Once the spammers have moved on, it's unlikely that I have enough customers
trying to reach the blocked address space and complaining for me to fix it, and
the people *in* that address space can't successfully complain because I've
blocked it.

3) The damage to traffic is of consequence to the remote site, but isn't a
revenue-impacting issue for *ME*.

The third point is the biggie here.

This is not actually a new problem. ISPs have been fighting this for
some time. When a dud customer spams from a given IP range and gets it
placed in various RBLs, when that customer is booted or otherwise
removed, that block will probably get reissued. The new customer then
calls up and says, "my email isn't getting through." All it takes is a
little investigation and the cause becomes clear. In my experience,
there is absolutely no way to deal with this other than contacting the
companies your customer is trying to email one by one. Not all of them
will respond to you but when they are slow or do not act at all, quite
often if the recipient on the other end calls them up and says, "WTF?"
it generates more action.

Sadly, I do not foresee this problem getting any easier.

Best practices for the public or subscription RBLs should be to place
a TTL on the entry of no more than, say, 90 days or thereabouts. Best
practices for manual entry should be to either keep a list of what and
when or periodically to simply blow the whole list away and start anew
to get rid of stale entries. Of course, that is probably an unreal
expectation.

-Wayne

The difference/issue here is that it's easy for you when turning down or turning up a customer to check the IP space being revoked/assigned in the various popular public DNSBLs, sparing your customers the headache of being assigned blacklisted IPs. Until your next customer starts using the space and can't send us email, you have no way of knowing that we null routed the subnet on our MX cluster.

Seth Mattinen wrote:

I was always under the impression that smaller orgs were not allowed to
join the MAAWG club.

They're allowed. At $4k/year minimum, up to $25K/year.

By the way, among the members...

Experian CheetahMail
ExactTarget, Inc
Responsys, Inc.
Vertical Response, Inc
Yesmail

there is a fundamental disconnect here. the IP space is neutral.
it has no bias toward or against social behaviours. its a tool.
the actual/real target here are the people who are using these tools
to be antisocial. blacklisting IP space is always reactive and
should only beused in emergency and as a -TEMPORARY- expedient.

IMHO of course., YMMV.

--bill

John Curran wrote:

> It seems simple and obvious that ARIN, RIPE, et. al. should
> determine the blacklist state of a reclaimed IP group and ensure
> that the IP group is usable before re-allocating it.
>
> When IPs are reclaimed, first check to see if the reclaimed IPs are
> on any readily checked RBL or private blacklist of major ISPs,
> corporations, universities, etc. If so, work with those groups to
> get the blocks removed *prior* to reissuing the IPs to a new
> entity. Before releasing the IPs to a new entity, double check that
> they are not being blocked (that any promises to remove them from
> a blacklist were actually fulfilled). Hold the IPs until you have
> determined that they aren't overly encumbered with prior blacklist
> blocks due to poor behavior of the previous entity. (The same
> should be done before allocating out of a new IP block, such as
> when you release the first set of IPs in a new /8.)

In this case, it's not the RBL's that are the issue; the address
block in question isn't on them. It's the ISP's and other firms
using manual copies rather than actually following best practices.

It's not that hard to make a list of the major ISPs, corporations, universities (entities with a large number of users), find willing contacts inside each organization (individual or role addresses you can email, and see if the email bounces, and who will reply if the email is received) and run some automated tests to see if the IPs are being blocked. In your follow-up email to me, you said you check "dozens" of RBLs - that is clearly insufficient - probably by an order of magnitude - of the entities you should check with. The number should be "hundreds". A reasonably cluefull intern can provide you with a suitable list in short order, probably less than 1 day, and find suitable contacts inside each organization in a similar time frame - it might take a week total to build a list of ~500 entities and associated email addresses. Because of employee turn-over the list will need to be updated, ~1-10 old addresses purged and replaced with new ones on a monthly basis.

> Why isn't this being done now?
>
> Issuing reclaimed IPs is a lot like selling a used car, except that
> the buyer has no way to "examine" the state of the IPs you will
> issue them beforehand. Therefore it's up to you (ARIN, RIPE, et.
> al.) to ensure that they are "just as good" as any other IP block.
> It is shoddy business to take someone's money and then sneakily
> give them tainted (used) goods and expect them to deal with
> cleaning up the mess that the prior owner made, especially when you
> charge the same rate for untainted goods!

Not applicable in this case, as noted above.

What do you mean, "not applicable"? You take the money and issue IPs. There is no way for the "buyer" to know before hand if the IPs are "tainted" (used) or new. It is up to you (ARIN) to ensure that the goods (IPs) are suitable for the intended use. My analogy is entirely applicable, and I'm amazed you think otherwise.

So, back to the question: could someone explain why they've got
copies of the RBL's in their network which don't get updated on any
reasonable refresh interval? (weekly? monthly?)

The "why" really isn't at issue - it happens and it's going to keep happening. The question is what are you (ARIN) going to do about it?

Give me the serenity to accept the things I cannot change,
The courage to change the things I can,
And the wisdom to know the difference.

You (ARIN et. al.) don't have any ability to change the why. What you can change is how you go about determining if an IP block is suitable for reallocation or not, and what steps you take to repair IP blocks that aren't suitable for reallocation.

jc - posted to NANOG since John indicated that he thought his reply to me was going to NANOG as well.

Jason Bertoch wrote:

Suresh Ramasubramanian wrote:

That said most of the larger players already attend MAAWG - that
leaves rural ISPs, small universities, corporate mailservers etc etc
that dont have full time postmasters, and where you're more likely to
run into this issue.
  

I've found the opposite to hold true more often. Smaller organizations can use public blacklists for free, due to their low volume, and so have little incentive to run their own local blacklist. I've typically seen the larger organizations run their own blacklists and are much more difficult to contact for removal.

Take for example GoDaddy's hosted email service. They are using a local, outdated copy of SORBS that has one of my personal servers listed in it. It was an open proxy for about week nearly 3 years ago and still they have it listed. The upside is that I've demonstrated GoDaddy's email incompetence to potential customers and gotten them to switch to our own mail services. Their loss, my gain.

Justin

Wayne E. Bouchard wrote:

Best practices for the public or subscription RBLs should be to place
a TTL on the entry of no more than, say, 90 days or thereabouts. Best
practices for manual entry should be to either keep a list of what and
when or periodically to simply blow the whole list away and start anew
to get rid of stale entries. Of course, that is probably an unreal
expectation.

I've had to implement something similar for my RTBH trigger router. After manually-adding nearly 20,000 static routes of hosts that scanned for open proxies or attacked SSH daemons on my network I had to trim the block list considerably because many of my older PEs couldn't handle that many routes without problems. I already named each static with a reason for the block(SSH, Telnet, Proxy-scan, etc) but ended up prepending a date to that string as well: 20090908-SSH-Scan. That way I can parse the config later on and create config to negate everything that's older than 3-4 months. If one of those old IPs is still trying to get to me after 4 months then it will get readded the next time I process my logs entries. If they aren't trying to hit me then they'll no longer be consuming space in my RIB.

Justin

Seth Mattinen wrote:

I was always under the impression that smaller orgs were not allowed to
join the MAAWG club.

I've heard that, too, but have no idea where it comes from. It's not true; there's no size requirement or anything like that.

http://www.maawg.org/ has the membership application and other info.

J.D. Falk wrote:

Seth Mattinen wrote:

I was always under the impression that smaller orgs were not allowed to
join the MAAWG club.

I've heard that, too, but have no idea where it comes from. It's not true; there's no size requirement or anything like that.

http://www.maawg.org/ has the membership application and other info.

The $4000/year minimum membership fee is a non-starter for small organizations who are already strapped for operating cash as it is. This is probably where the perception comes from.