Securing Greenfield Service Provider Clients

Dear Nanog;

Hope everyone is getting ready for a good weekend. I’m working on a greenfield service provider network and I’m running into a security challenge. I hope the great minds here can help.

Since the majority of traffic is SSL/TLS, encrypted malicious content can pass through even an “NGFW” device without detection and classification.

Without setting up SSL encrypt/decrypt through a MITM setup and handing certificates out to every client, is there any other software/hardware that can perform DPI and/or ssl analysis so I can prevent encrypted malicious content from being downloaded to my users?

Have experience with Palo and Firepower but even these need the MITM approach. I appreciate any advice anyone can provide.

Best,

CJ

Matt Harris​

Infrastructure Lead Engineer

816‑256‑5446

Direct

Looking for something?

Helpdesk Portal

Email Support

Billing Portal

We build and deliver end‑to‑end IT solutions.

DNS filtering might be an easier option to get most of the bad stuff with services like 9.9.9.9 and 1.1.1.2. Paid options like dnsfilter.com will give you better control. Cloudflare Gateway might also be an option.

CJ,

Dear Nanog;

Hope everyone is getting ready for a good weekend.� I�m working on a greenfield service provider network and I�m running into a security challenge.� I hope the great minds here can help.

Since the majority of traffic is SSL/TLS, encrypted malicious content can pass through even an �NGFW� device without detection and classification.

Without setting up SSL encrypt/decrypt through a MITM setup and handing certificates out to every client, is there any other software/hardware that can perform DPI and/or ssl analysis so I can prevent encrypted malicious content from being downloaded to my users?

Have experience with Palo and Firepower but even these need the MITM approach.� I appreciate any advice anyone can provide.

I think this most likely needs to develop into a bigger discussion, but TLS introspection will (and must, otherwise we would have big problems ) rely on a MITM setup.

DNS- and reputation-based filtering was already mentioned, there is also this work on detecting malware aspects by TLS anomalies:

I'm not aware whether there are service provider network-grade tools for this available though.

Thanks,
Matthias

Agreed DNS/IP reputation is still about the best. Then move on with everything else we should be doing.

Decrypting the content would bring us to the next problem. Malware is commonly encrypted to prevent AntiVirus from pattern matching or hash matching.

Decrypting the content always struck me as something that is better suited for spotting exfiltration. Searching for known clear text similar to “FBI Classified” or a watermark in documents sounded like an attainable goal from SSL decryption.

Are you really suggesting decrypting customer traffic? In most parts of the world that act falls in one of two categories: it is either required by law or it is illegal.

Offer your customers a good virus scanner to install instead.

Regards

Baldur

fre. 9. okt. 2020 21.27 skrev Christopher J. Wolff <cjwolff@nola.gov>:

If you search for this phrase

  During 2020 more than fifty percent of new malware campaigns will use various forms of encryption and obfuscation to conceal delivery, and to conceal ongoing communications, including data exfiltration.

you will find lots of vendors of decryption have the phrase from Gartner mentioned prominently on their web site.

I don’t think TLS decryption would be viable in our university environment.

Your email address indicates that you are in a government environment and if so you might have more control over devices and could have a better chance of making decryption work.
On the other hand if you have more control over devices a better choice might be to spend your resources on implementing whitelisting rather than decryption.

Keep in mind that if you implement decryption your decryption device is in scope for PCI and subject to the various PCI duding and logging requirements.

Attackers abuse Google DNS over HTTPS to download malware

https://www.bleepingcomputer.com/news/security/attackers-abuse-google-dns-over-https-to-download-malware/

More general and as focused on decryption but I recommend you watch these sessions from RSA conferences.

https://www.youtube.com/watch?v=d90Ov6QM1jE

https://www.youtube.com/watch?v=qzI-N0p9hFk

And also the NIST draft on Zero Trust Architecture. The document is mainly about Zero Trust but does briefly mention decryption.

https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-207.pdf

https://csrc.nist.gov/publications/detail/sp/800-207/final

Dear Mr. Curtis and Nanog;

Thank you for your responses. Yes, I am investigating the feasibility of public internet access to help with Digital Divide issues in light of the COVID-19 pandemic as well as the challenges of security in this public application.

It’s relatively straightforward to segment East-West traffic; however, I’m not so sure about the case of North-South. I need to address this issue somehow in my assessment of risks in public networks.

I do not want to decrypt SSL traffic. But I would like to be able to have some black box with a subscription at the network edge prevent malware from being downloaded through the network.

My question was whether this is even possible in a public context. Secure DNS services would go a long way toward this goal.

Is it fair to say that an NGFW must decrypt SSL traffic in order to fully categorize for IPS/IDS prevention?

Thank you,
CJ

No. That was kind of the point of SSL.

Dear Mr. Curtis and Nanog;

Thank you for your responses. Yes, I am investigating the feasibility of public internet access to help with Digital Divide issues in light of the COVID-19 pandemic as well as the challenges of security in this public application.

It’s relatively straightforward to segment East-West traffic; however, I’m not so sure about the case of North-South. I need to address this issue somehow in my assessment of risks in public networks.

I do not want to decrypt SSL traffic. But I would like to be able to have some black box with a subscription at the network edge prevent malware from being downloaded through the network.

My question was whether this is even possible in a public context. Secure DNS services would go a long way toward this goal.

Is it fair to say that an NGFW must decrypt SSL traffic in order to fully categorize for IPS/IDS prevention?

Thank you,
CJ

Just my humble opinion, many network security devices in the middle decrease the overall network security. Especially if they fall into the category of NGFW, they do too much and end up blowing themselves up.

https://www.google.com/amp/s/www.zdnet.com/google-amp/article/us-cyber-command-says-foreign-hackers-will-attempt-to-exploit-new-pan-os-security-bug/

https://www.google.com/amp/s/www.bleepingcomputer.com/news/security/cisco-patches-asa-ftd-firewall-flaw-actively-exploited-by-hackers/amp/

Also, they are insanely priced and market to people based on fear.

IPS / IDS only works if you have a full time team of folks willing to tune it. And, it is never worth it. Been the same way for 20 years. I was recently involved in an outage with an IPS rule taking an entire site off line. The fix was to stop doing IPS.

The fact is, most modern systems (win10, iOS, Android) are very secure from a network stack and do not benefit from network based tools. Even windows Vista has a local firewall on by default. The real hacks that happen in the wild are phishing … and no network based thing is going to stop that. You do see occasional nsa tools turned into wannacry style worms, but those only proliferate when SMB is enabled, and that is easily blocked with a router acl, and is a best practice below.

For public internet access, please keep it simple. Please do not waste tax payer money on security snake oil. As mentioned, free dns services like 1.1.1.3 and https://cleanbrowsing.org go a long way

Simple router ACLS are also good to shutdown back trafffic, take a hint from Comcast

https://www.xfinity.com/support/articles/list-of-blocked-ports

Regards,
CB

Is it fair to say that an NGFW *must* decrypt SSL traffic in order to
fully categorize for IPS/IDS prevention?

well, not really. aside from damage, it will not 'protect' you against
more modern transports, such as quic, which were designed to keep the
net open.

randy

If this is really greenfield, consider taking a tenant approach to your egress traffic handling, you mentioned a “black box with subscription”, then consider making that blackbox/traffic path be only available to whatever tenant subscribes to the service, and if they want the SSL/MITM decryption, then their local IT team (or yours) can handle the certificate management and risk of doing such a thing.

Then all you need to worry about is managing the egress per tenant, which can all be maintained separately from whatever services you’re wrapping up into that security service package.

Keep in mind your 80/20s :slight_smile:

-Garrett

Dear Mr. Curtis and Nanog;

Thank you for your responses. Yes, I am investigating the feasibility of public internet access to help with Digital Divide issues in light of the COVID-19 pandemic as well as the challenges of security in this public application.

It’s relatively straightforward to segment East-West traffic; however, I’m not so sure about the case of North-South. I need to address this issue somehow in my assessment of risks in public networks.

I do *not* want to decrypt SSL traffic. But I would *like* to be able to have some black box with a subscription at the network edge prevent malware from being downloaded through the network.

My question was whether this is even possible in a public context. Secure DNS services would go a long way toward this goal.

Is it fair to say that an NGFW *must* decrypt SSL traffic in order to fully categorize for IPS/IDS prevention?

Thank you,
CJ

Just my humble opinion, many network security devices in the middle decrease the overall network security.

  Yes. And in more ways than by being compromised themselves as indicated in the links you provide below. (Remember that boxes that decrypt TLS are in scope for PCI).

In addition most middle boxes are stateful devices that can be affected by DoS attacks that create state on the middle boxes.

In the CIA principle Availability is supposed to be equally important.

So if a middle box is affected by a DoS attack and normal traffic is dropped so that Availability of a service is affected that should also be considered a security failure or a decrease in overall network security.

Unfortunately in most instances the CIA principle is really applied as the CIa principle where Availability is not equal and is always secondary to C and I. Even to the point where if a primary protection is in place but a a secondary protection fails in a way that affects availability the secondary protection is not bypassed to restore Availability.

Especially if they fall into the category of NGFW, they do too much and end up blowing themselves up.

US Cyber Command says foreign hackers will attempt to exploit new PAN-OS security bug | ZDNET

Cisco patches ASA/FTD firewall flaw actively exploited by hackers

Also, they are insanely priced and market to people based on fear.

IPS / IDS only works if you have a full time team of folks willing to tune it. And, it is never worth it. Been the same way for 20 years. I was recently involved in an outage with an IPS rule taking an entire site off line. The fix was to stop doing IPS.

The fact is, most modern systems (win10, iOS, Android) are very secure from a network stack and do not benefit from network based tools. Even windows Vista has a local firewall on by default. The real hacks that happen in the wild are phishing ... and no network based thing is going to stop that. You do see occasional nsa tools turned into wannacry style worms, but those only proliferate when SMB is enabled, and that is easily blocked with a router acl, and is a best practice below.

For public internet access, please keep it simple. Please do not waste tax payer money on security snake oil. As mentioned, free dns services like 1.1.1.3 and https://cleanbrowsing.org go a long way

Simple router ACLS are also good to shutdown back trafffic, take a hint from Comcast

Blocked Internet Ports List - Xfinity Support

Regards,
CB

Get Outlook for iO

From: Curtis, Bruce <bruce.curtis@ndsu.edu>
Sent: Friday, October 9, 2020 5:23:45 PM
To: Christopher J. Wolff <cjwolff@nola.gov>
Cc: nanog@nanog.org <nanog@nanog.org>
Subject: Re: Securing Greenfield Service Provider Clients

EMAIL FROM EXTERNAL SENDER: DO NOT click links, or open attachments, if sender is unknown, or the message seems suspicious in any way. DO NOT provide your user ID or password. If you believe that this is a phishing attempt please forward this message to phishing@nola.gov

If you search for this phrase

        During 2020 more than fifty percent of new malware campaigns will use various forms of encryption and obfuscation to conceal delivery, and to conceal ongoing communications, including data exfiltration.

you will find lots of vendors of decryption have the phrase from Gartner mentioned prominently on their web site.

I don’t think TLS decryption would be viable in our university environment.

Your email address indicates that you are in a government environment and if so you might have more control over devices and could have a better chance of making decryption work.
On the other hand if you have more control over devices a better choice might be to spend your resources on implementing whitelisting rather than decryption.

Keep in mind that if you implement decryption your decryption device is in scope for PCI and subject to the various PCI duding and logging requirements.

Attackers abuse Google DNS over HTTPS to download malware

Attackers abuse Google DNS over HTTPS to download malware

More general and as focused on decryption but I recommend you watch these sessions from RSA conferences.

https://www.youtube.com/watch?v=d90Ov6QM1jE

https://www.youtube.com/watch?v=qzI-N0p9hFk

And also the NIST draft on Zero Trust Architecture. The document is mainly about Zero Trust but does briefly mention decryption.

https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-207.pdf

SP 800-207, Zero Trust Architecture | CSRC

>
> Dear Nanog;
>
> Hope everyone is getting ready for a good weekend. I’m working on a greenfield service provider network and I’m running into a security challenge. I hope the great minds here can help.
>
> Since the majority of traffic is SSL/TLS, encrypted malicious content can pass through even an “NGFW” device without detection and classification.
>
> Without setting up SSL encrypt/decrypt through a MITM setup and handing certificates out to every client, is there any other software/hardware that can perform DPI and/or ssl analysis so I can prevent encrypted malicious content from being downloaded to my users?
>
> Have experience with Palo and Firepower but even these need the MITM approach. I appreciate any advice anyone can provide.
>
> Best,
> CJ

Bruce Curtis
Network Engineer / Information Technology
NORTH DAKOTA STATE UNIVERSITY
phone: 701.231.8527
bruce.curtis@ndsu.edu

Bruce Curtis
Network Engineer / Information Technology
NORTH DAKOTA STATE UNIVERSITY
phone: 701.231.8527
bruce.curtis@ndsu.edu

Dear Mr. Curtis and Nanog;

Thank you for your responses. Yes, I am investigating the feasibility of public internet access to help with Digital Divide issues in light of the COVID-19 pandemic as well as the challenges of security in this public application.

It’s relatively straightforward to segment East-West traffic; however, I’m not so sure about the case of North-South. I need to address this issue somehow in my assessment of risks in public networks.

I do *not* want to decrypt SSL traffic. But I would *like* to be able to have some black box with a subscription at the network edge prevent malware from being downloaded through the network.

My question was whether this is even possible in a public context. Secure DNS services would go a long way toward this goal.

Is it fair to say that an NGFW *must* decrypt SSL traffic in order to fully categorize for IPS/IDS prevention?

Another thing to keep in mind is that NGFW/IPS depend on blacklisting to block malware.

Even if you did install certificates on all devices to enable TLS decryption (and bring the decryption device in scope for PCI) that is not a guarantee that the NGFW/IPS can block an amount of malware worthy of the investment.

"By 2017, around 96 percent of all malware files detected and blocked by Windows Defender were detected only once on a single computer and never seen again.”

https://cybersecurityventures.com/the-devastating-effect-of-polymorphic-malware/

"For many years, the viewpoint on malware protection has been inclined towards investing in traditional security methods such as firewalls, antivirus as well as IPS. However, when it comes to protection against polymorphic malware, these solutions do not work properly.”

While blacklisting, either in a middle box or on the host, will not stop malware that is changed to have a different signature every time it is downloaded whitelisting on the end host might stop it.
In the example where whitelisting will stop malware but blacklisting will not you are better off spending your limited resources on whitelisting.

This is from 2014 but indicates the beginning of the trend to shortening times between malware morphing had started.

Insights from one year of tracking a polymorphic threat (another example of malware that a middle box would not stop)

https://www.microsoft.com/security/blog/2019/11/26/insights-from-one-year-of-tracking-a-polymorphic-threat/

Thank you,
CJ

Get Outlook for iOS
From: Curtis, Bruce <bruce.curtis@ndsu.edu>
Sent: Friday, October 9, 2020 5:23:45 PM
To: Christopher J. Wolff <cjwolff@nola.gov>
Cc: nanog@nanog.org <nanog@nanog.org>
Subject: Re: Securing Greenfield Service Provider Clients

EMAIL FROM EXTERNAL SENDER: DO NOT click links, or open attachments, if sender is unknown, or the message seems suspicious in any way. DO NOT provide your user ID or password. If you believe that this is a phishing attempt please forward this message to phishing@nola.gov

If you search for this phrase

        During 2020 more than fifty percent of new malware campaigns will use various forms of encryption and obfuscation to conceal delivery, and to conceal ongoing communications, including data exfiltration.

you will find lots of vendors of decryption have the phrase from Gartner mentioned prominently on their web site.

I don’t think TLS decryption would be viable in our university environment.

Your email address indicates that you are in a government environment and if so you might have more control over devices and could have a better chance of making decryption work.
On the other hand if you have more control over devices a better choice might be to spend your resources on implementing whitelisting rather than decryption.

Keep in mind that if you implement decryption your decryption device is in scope for PCI and subject to the various PCI duding and logging requirements.

Attackers abuse Google DNS over HTTPS to download malware

Attackers abuse Google DNS over HTTPS to download malware

More general and as focused on decryption but I recommend you watch these sessions from RSA conferences.

https://www.youtube.com/watch?v=d90Ov6QM1jE

https://www.youtube.com/watch?v=qzI-N0p9hFk

And also the NIST draft on Zero Trust Architecture. The document is mainly about Zero Trust but does briefly mention decryption.

https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-207.pdf

SP 800-207, Zero Trust Architecture | CSRC

>
> Dear Nanog;
>
> Hope everyone is getting ready for a good weekend. I’m working on a greenfield service provider network and I’m running into a security challenge. I hope the great minds here can help.
>
> Since the majority of traffic is SSL/TLS, encrypted malicious content can pass through even an “NGFW” device without detection and classification.
>
> Without setting up SSL encrypt/decrypt through a MITM setup and handing certificates out to every client, is there any other software/hardware that can perform DPI and/or ssl analysis so I can prevent encrypted malicious content from being downloaded to my users?
>
> Have experience with Palo and Firepower but even these need the MITM approach. I appreciate any advice anyone can provide.
>
> Best,
> CJ

Bruce Curtis
Network Engineer / Information Technology
NORTH DAKOTA STATE UNIVERSITY
phone: 701.231.8527
bruce.curtis@ndsu.edu

Bruce Curtis
Network Engineer / Information Technology
NORTH DAKOTA STATE UNIVERSITY
phone: 701.231.8527
bruce.curtis@ndsu.edu