Help with removing DNS shinkhole FP from Charter/Spectrum

Mailman List Mirror (Read Only)
1

Looking for some help/advice. Spectrum is sinkholing my company’s domain, validin[.]com, to 127.0.0.54. The sinkhole responses come from their recursive DNS servers, 209.18.47.61 and 209.18.47.62, which are defaults for and in use by many of their customers and are only reachable from within the Spectrum network. I’ve had 4 people over the last week (think: customers, prospects, etc) who use Charter/Spectrum tell me that they have difficulty accessing my website as a result of this sinkhole behavior. This behavior is causing reputational harm to my company.

I’ve personally confirmed this behavior from the Spectrum network (I am also a customer) using dig to test their DNS servers:

$ dig +short @209.18.47.61 [validin.com](http://validin.com)
127.0.0.54

$ dig +short @209.18.47.62 [validin.com](http://validin.com)
127.0.0.54

Using Cloudflare/Google/etc works correctly:

$ dig +short @[1.1.1.1](http://1.1.1.1) [validin.com](http://validin.com)
137.184.54.107
157.245.112.183

$ dig +short @[8.8.8.8](http://8.8.8.8) [validin.com](http://validin.com)
157.245.112.183
137.184.54.107

I suspect my domain was blocklisted last year when a threat researcher included my domain name in a blog post about a threat they were investigating and cited my company as the source for their data. Someone scraped that post, and my company’s domain was accidentally added to two Alient Vault OTX pulses and at least one collection on Virus Total. I removed the domain via false positive reporting from everything I could. However, it appears that being added to Spectrum’s DNS sinkhole list is effectively permanent and there’s no clear path for false positive remediation.

I’ve tried the official Spectrum support lines for months to no avail, and recently tried reaching out on Twitter, but have had no success there either. I’m clearly not able to find the right people through these routes, as none of the people I reach understand the difference between a DNS sinkhole and an IP block list and don’t appear to be aware that DNS blocklisting is a separate behavior from their opt-in content filtering via Security Shield.

So, if someone could please help me find the team or individual responsible for Spectrum’s DNS sinkhole behavior, I would be exceptionally grateful. :slight_smile:

As I mentioned, this is causing reputation harm, so switching my own DNS servers is not sufficient. People who need to reach me, can’t. So, I would appreciate any other help or advice you have,

Kenneth

2

Howdy,

If you can't reach a technical POC, use the legal one. Your lawyer can
find the appropriate recipient and write a cease-and-desist letter for
you. After that, it's -their- lawyer's problem to track down the
correct technical people.

Incidentally, for folks who choose to interdict DNS: whatever your
reasons, pointing the DNS to a loopback IP is bad practice. Really bad
practice. Minimum good practice points it to a web site you control
which provides enough information to get delisted. And provides you
with a test point where you can collect information about what you've
caused to be interdicted.

Regards,
Bill Herrin

3

I notice from MXToolbox.com that your domain’s IP address is on the UCEPROTECTL3 blacklist.

This is a notoriously evil blacklist that charges people for removal. This may be why Spectrum is blackholing your domain. Most respectable ISPs won’t use it. But Spectrum…

There is no delisting procedure without making a “donation” to the UCEPROTECT3 black sparrow account. They’re famous for blacklisting large swaths of IP addresses that catch up innocent parties that have never spammed a flea.

-mel

4

Hi Mel,

I appreciate the suggestion. During my earlier research, I’d noticed that as well. However, the DNS block includes all validin.com subdomains, covering those on completely different ASNs. It also does NOT affect other domains that resolve to the exact same IP addresses (e.g., validin.net). So, I’m inclined to think it’s not that simple, unfortunately.

I’d considered switching domains, but that doesn’t guarantee the problem wouldn’t just reappear again, and it’d impact the search engine ranking we’ve built up. We rely 100% on inbound, so that’d be a big set back.

Warm regards,

Kenneth

5

It appears that William Herrin <bill@herrin.us> said:

Looking for some help/advice. Spectrum is sinkholing my company's domain, validin[.]com, to 127.0.0.54.

Howdy,

If you can't reach a technical POC, use the legal one. Your lawyer can
find the appropriate recipient and write a cease-and-desist letter for
you. After that, it's -their- lawyer's problem to track down the
correct technical people.

No, that is terrible advice. In the immortal acronym of Laura Atkins, TWSD.

The only response to a letter like that is "we run our network to
serve our customers and manage it the way we think is best" and you
know what, they're right. It is absolutely legal to block traffic you
think is malicious, even if you are wrong, and there is case law.

Having said that, I suspect the least bad alternative if you can't
find an out of band contact is to get some of the Spectrum customers
who can't reach you to complain. They're customers, you aren't.

R's,
John

6

It appears that William Herrin <bill@herrin.us> said:
>If you can't reach a technical POC, use the legal one. Your lawyer can

The only response to a letter like that is "we run our network to
serve our customers and manage it the way we think is best" and you
know what, they're right.

Hi John,

Respectfully, you're mistaken. Look up "tortious interference."

Operators have considerable legal leeway to block traffic for cause,
or even by mistake if corrected upon notification, but a lawyer who
blows off a cease-and-desist letter without investigating it with the
tech staff has committed malpractice. The lawyer doesn't want to
commit malpractice. You write the lawyer via certified mail, he's
going to talk to the tech staff and you're going to get a response. At
that point, you have an open communication pathway to get things
fixed. Which was the problem to be solved.

Having said that, I suspect the least bad alternative if you can't
find an out of band contact is to get some of the Spectrum customers
who can't reach you to complain. They're customers, you aren't.

My results going through the support front-door at large companies for
oddball problems have been less than stellar. Has your experience
truly been different?

Regards,
Bill Herrin

7

“We checked the website you are trying to access for malicious and
spear-phishing content and found it likely to be unsafe.”

perhaps charter thinks there's a reason to not permit folks to access
a possibly dangerous site?
(it's also possible it just got cough up amongst some other stuff in
the hosting provider's space, nothing jumps out in passive-dns
lokoups.)

8

Respectfully, you're mistaken. Look up "tortious interference."

I'm familiar with it.

But I am also familar with many cases were spammers have sued network operators claiming that they're falsely defamed, so the operator has to deliver their mail. They have without exception lost. If you can find actual cases where a court forced an operator to deliver a third party's traffic I would like to hear about it.*

43 USC 230(c)(A) provides extremely broad protection for "good faith" blocking, which means that a complaint would have to show that the blocking was malicious rather than merited or accidental. In this case it seems probably accidental, but for all I know there might have been bad traffic to merit a block.

Here's one of the cases where a spammer lost:

https://jl.ly/Email/holomaxx.html
https://jl.ly/Email/holo4.html

And here's one where the judge rejected tortious interference:

https://jl.ly/Email/spamarrest.html

My results going through the support front-door at large companies for
oddball problems have been less than stellar. Has your experience
truly been different?

No, it's terrible, and Spectrum is particularly bad. I am now in month three of trying to get them to route a /24 to my host that belongs to one of my users, and their responses can be summarized as very complex exegeses of "duh?"

But bogus lawyer letters will just make things worse.

R's,
John

* - let's stay away for now from the Texas and Florida social network common carrier laws which are a whole other can of s*

9

Hi John,

I'll try not to belabor it, but accidental that isn't corrected upon
formal legal notification becomes negligent and negligent has more or
less the same legal status as malicious.

The spammers lost because the networks published a terms of use
document that the spammers unambiguously violated. Even though it
interfered with the spammer's business, the block was merited so the
preponderance of the evidence fell in favor of the service provider.

Regards,
Bill Herrin

10

Bill is absolutely correct. The spammers lost their case because they were demonstrably spammers. We’ve had accidental black hole cases with US providers that removed the block once they received a C&D. If they don’t have iron clad proof in hand. (More than just a few complaints and no traffic analysis), it’s just the least risky response.

That doesn’t work well with overseas providers, though, because they’re essentially immune to U.S. litigation unless the plaintiff has deep pockets.

-mel

11

Hi Bill,

I’m not sure where you saw that message, but I got this message via email after I submitted an unblock request with Spectrum Shield:

We have reviewed your request to unblock validin.com. This site was not found to be blocked by Spectrum Shield and should be accessible from your browser.

Thank you,

Spectrum

My company’s domain got caught up in some lazy copy/pasting from this blog post last year that cited my company as a source for the data. Someone copy/pasted the whole page, which included my company’s domain name, and that made it to a few AV OTX pulses and VT collections:https://gi7w0rm.medium.com/uncovering-ddgroup-a-long-time-threat-actor-d3b3020625a4

I’ve cleaned up everything I could from that botched blocklist aggregation. However, there’s no correction process for Spectrum’s DNS sinkhole, and I’m not even sure that’s how our domain got mixed up there. The support staff I’ve spoken with have denied the existence of DNS sinkholing at Spectrum, and demonstrated they lack the basic technical sophistication needed to understand the concept. They’ve each ultimately told me that each affected customer would need to reach out to the Spectrum customer service, which would then help that customer change their DNS settings to another DNS provider. Of course, the last thing I’d want to do with a potential customer is ask them to go through that painful process. I also have no idea how many potential users or customers can’t reach me and simply give up without letting me know.

Lastly, I AM a Spectrum customer. My home internet service is Spectrum. If it weren’t for that, I’d be truly SOL because support would just ignore me. But, they they claim the issue is resolved from their perspective because I can simply change my DNS settings.

But back to the topic: someone mentioned to me that Spectrum may not be the direct providers for the DNS services they provide to their customers. If anyone knows anything about how I might discover and reach out to the people responsible, please let me know. :slight_smile:

Regards,

Kenneth

12

Howdy,

That was Christopher, not me. But you should check the talos link I
sent you privately. Also https://ipcheck.proofpoint.com/. Whatever
they're detecting, it didn't happen last year.

Regards,
Bill Herrin

13

Bill is absolutely correct. The spammers lost their case because they were demonstrably spammers.

No, really they did not. I read the decisions. Have you? Hint: under CAN SPAM a great deal of spam is completely legal so it didn't matter.

We’ve had accidental black hole cases with *US* providers that removed the block once they received a C&D. If they don’t have iron clad proof in hand. (More than just a few complaints and no traffic analysis), it’s just the least risky response.

I will believe that there are people that cave in response to threats like this, but again, there is no case law to support it.

R's,
John

14

I'm not sure where you saw that message, but I got this message via email
after I submitted an unblock request with Spectrum Shield:

We have reviewed your request to unblock validin.com. This site was not

found to be blocked by Spectrum Shield and should be accessible from your
browser.

Sigh.

I've cleaned up everything I could from that botched blocklist aggregation.
However, there's no correction process for Spectrum's DNS sinkhole, and I'm
not even sure that's how our domain got mixed up there. The support staff
I've spoken with have denied the existence of DNS sinkholing at Spectrum,
and demonstrated they lack the basic technical sophistication needed to
understand the concept.

Yeah, that's the problem. And given stuff like this link below, I wouldn't expect their legal department to be any better. Clearly there is someone somewhere who is competent because their network mostly works, but damned if I know how to find them.

R's,
John

15

However, there's no correction process for Spectrum's DNS sinkhole
But back to the topic: someone mentioned to me that Spectrum may not be the direct providers for the DNS services they provide to their customers. If anyone knows anything about how I might discover and reach out to the people responsible, please let me know.

I suspect what’s happened is an incorrect assumption that DNS is even the issue here. Because you mentioned Spectrum Shield, I suspect it is not.

Spectrum Shield (The Benefits of Spectrum Security Shield – Spectrum Resources) is a customer-managed security protection service built into their gateways (I assume you can turn it off). The malware and content detection engine behind that is very likely run by CujoAI (https://cujo.com/) and it does not use DNS query/response exchanges as the control mechanism (in part to counter-act DNS-changing malware or malware using its own DoH channel for example).

You should contact Charter/Spectrum to have them investigate what their system might be blocking this content.

Comcast (where I work) runs a similar system (Use Xfinity xFi Advanced Security - Xfinity Support) and maintains a site to report these sorts of issues (Report a website blocked by Xfinity xFi Advanced Security - Xfinity Support).

Jason

16

Hi Jason,

I suspect what’s happened is an incorrect assumption that DNS is even the issue here. Because you mentioned Spectrum Shield, I suspect it is not.

I appreciate the response and links. However, I’ve been told repeatedly by Spectrum that they’re not blocking with Spectrum Shield. Despite these assurances, I’ve filled out a removal request through their published removal process several times, and the response I received stated that we’re not being blocked. This check agrees with that:
https://www.spectrum.net/support/forms/verify_url_security

“Security Shield Is Not Blocking This Site
The URL provided is not being blocked by Spectrum Security Shield
The URL you entered should be accessible.”

Further, checking Spectrum DNS servers on the Spectrum network show that my company’s main domain and all subdomains resolve to 127.0.0.54. So, if CujoAI/Spectrum Shield are not using DNS query responses to control access, then it’s not CujoAI/Spectrum Shield that is responsible for the incorrect DNS response. Using a different recursive resolve, I can resolve our domains just fine. I can also resolve other domains that point to the same IPs as the sinkholed domain just fine. However, many people use the Spectrum default DNS servers and cannot access my website because of this.

You should contact Charter/Spectrum to have them investigate what their system might be blocking this content.

I have tried, for months, including spending many hours on chat and phone support, to reach someone within Spectrum support who is capable of both understanding and directing me to someone who can fix the problem, but it hasn’t happened yet. I’ve asked to talk to someone on the DNS team and was given a flat “No.” I’ve posted here hoping that someone in the ISP-connected world knows SOMEONE at Spectrum, Akamai, or whichever company is actually responsible for the Spectrum DNS servers who can provide a remediation path.

Regards,

Kenneth

17

Validin, made an interesting observation on this. I am also a Spectrum residential customer, none of their equipment, run my own DNS server (pihole).

My DHCP Assigned DNS servers are

2001:1998:f00:1::1
2001:1998:f00:2::1

bash-3.2$ dig -x 2001:1998:f00:1::1 +short
dns-cac-lb-01.rr.com.
bash-3.2$ dig -x 2001:1998:f00:2::1 +short
dns-cac-lb-02.rr.com.
bash-3.2$

bash-3.2$ dig dns-cac-lb-01.rr.com +short
209.18.47.61
bash-3.2$ dig dns-cac-lb-02.rr.com +short
209.18.47.62
bash-3.2$

bash-3.2$ dig @209.18.47.61 validin.com +short
157.245.112.183
137.184.54.107
bash-3.2$ dig @209.18.47.62 validin.com +short
157.245.112.183
137.184.54.107
bash-3.2$

bash-3.2$ dig @2001:1998:f00:1::1 validin.com +short
127.0.0.54
bash-3.2$

bash-3.2$ dig @2001:1998:f00:2::1 validin.com +short
127.0.0.54
bash-3.2$

Same servers on V4 were returning correct info, but on V6 were not.

However, a few minutes later :

bash-3.2$ dig @2001:1998:f00:1::1 validin.com +short
157.245.112.183
137.184.54.107
bash-3.2$ dig @2001:1998:f00:2::1 validin.com +short
157.245.112.183
137.184.54.107
bash-3.2$

Deltas :

bash-3.2$ dig @2001:1998:f00:1::1 validin.com

; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42329
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;validin.com. IN A

;; ANSWER SECTION:
validin.com. 60 IN A 127.0.0.54

;; Query time: 37 msec
;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1)
;; WHEN: Tue Apr 23 13:50:03 EDT 2024
;; MSG SIZE rcvd: 45

bash-3.2$

bash-3.2$ dig @2001:1998:f00:1::1 validin.com

; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9667
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;validin.com. IN A

;; ANSWER SECTION:
validin.com. 600 IN A 157.245.112.183
validin.com. 600 IN A 137.184.54.107

;; Query time: 157 msec
;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1)
;; WHEN: Tue Apr 23 14:19:20 EDT 2024
;; MSG SIZE rcvd: 72

bash-3.2$

Seems like quite possibly they are intermittently caching bunk data from something.

18

Tom,

Thank you for this! It is very interesting that the behavior is intermittent. A friend of mine who tested it this weekend saw correct answers on IPv6 and incorrect answers on IPv4.

Kenneth

19

Hi Kenneth,

We have been working internally and with our third-party domain reputation source to get your domain removed from their malware list.

Jim

20

Thank you, Jim. Who is the vendor responsible?

Kenneth