Hello NANOG community. I was directed here by our network administrator
since she is on vacation. Luckily, I minored in Computer Science so I have
some familiarity.
We have a small satellite campus of around 170 devices that share one
external IPv4 and IPv6 address via NAT for internet traffic. Internal
traffic is over an MPLS.
We're having problems where viruses are getting through Firefox, and we
think it's because our Palo Alto firewall is set to bypass filtering for
IPv6. Unfortunately, the network admin couldn't give me the password since
a local consultant set it up, and it seems they went out of business. I
need to think outside the box.
Is there some kind of NAT-based IPv6 firewall I can setup on the router
that can help block viruses? I figure that's the right place to start since
all the traffic gets funneled there. We have a Cisco Catalyst as a
router. Or, ideally, is there an easy way to turn off IPv6 completely? I
really don't see a need for it, any legitimate service should have an IPv4
address.
I'd really appreciate your advice. I plan to drive out there tomorrow,
where I can get the exact model numbers and stuff.
You emailed the wrong list to say this "Or, ideally, is there an easy way
to turn off IPv6 completely? I
really don't see a need for it, any legitimate service should have an IPv4
address."
Turning off IPv6 is not the right solution, nor will it magically fix your
issues.
Fix the Palo Alto, either hire another consultant or just erase it and
start over. Although even PA's Layer7 inspection won't catch everything and
you should have antivirus/antimailware software on the end user computers.
Hard to know where to begin with this one, but let me take a shot at it.
1. My top priority would be to get into that Palo Alto firewall. Get Palo Alto on the phone and figure out password recovery with them. Since you don’t have the password it is possible that firewall is compromised. Do not be surprised if you have to jump through some hoops with Palo Alto to prove that you own it and what has happened. Remember their job is to keep people out of your network. They are probably also going to want you to be current on support. If you have to pay to get current on support, do it. You need that help right now badly.
You could ask Palo Alto how to block the v6 while you are at it or even better set up a rules that mirror your v4 protection. I cannot stress enough how big a security issue it is to not have access to your firewall and not know who does.
2. There are lots of ways to shut off ipv6 but my suggestion would be to just secure the Palo Alto firewall, to say that any legitimate service should have a ipv4 address is not quite true now and will definitely not be true in the near future.
3. Just about any kind of firewall or router CPE device can block or firewall ipv4 and ipv6 as long as its firmware is fairly recent. However, you would most likely have to replace the Palo Alto with it. You DO NOT WANT THEM BOTH INLINE! Most likely they are both configured to do ipv4 NAT out of the box and that will not work correctly to have them both inline together. While it is possible to set up that sort of thing to work correctly, it’s a bad idea and pretty advanced configuration for a temporary network admin. The interaction of one firewall fronting another can be very difficult to troubleshoot without a deep understanding of what is going on. Referring back to item 1, you are probably going to need to get the configuration of the current firewall if you seek to replace it (there will be rules in the Palo Alto that you would want to replicate if you are going to replace it).
4. Cisco Catalyst as the router.....there could be a lot of things going on in there. The Catalyst is primarily a switch with routing functionality. It can definitely block ipv6 if configured to do so but we would need to know a lot more about its current configuration to give you the best way to do that. It could just be a service providers switch on your premise in which case you can't do much with it. Again, much easier to accomplish Item 1 with Palo Alto and let your firewall do what it is supposed to do.
On another note, using a firewall to stop viruses is probably not going to work in general (unless the firewall has some additional malware detection engine).
Here is the issue in a nutshell. A firewall primarily controls where people can connect to and from on a network. The problem with that is that a lot of malware is received from sites that your users intended to go to. People click on links without knowing where they go and people go to less than reputable web sites (or reputable sites that we recently compromised). If you, by default, allow your users to access the Internet with a browser they are vulnerable to malware. Even with malware detection capability you are still vulnerable to signatures and attacks that are not yet able to be detected.
Even if filtering was enabled on your Palo Alto for ipv6 it would not help at this point because you have no idea what signatures it is using to filter with and when the last time those were updated I doubt your v4 filtering is of much use either at this point. URL filtering is largely a big game of whack a mole that you will lose eventually. Malware filtering is based on one or both of the following methods.
1. You filter URLs known to be bad players (you are vulnerable until your protection vendor realizes they are bad players).
2. You filter based on adaptive detection of code that looks suspicious. This is a bit better but still vulnerable because the bad guys are always innovating to pass through these devices.
My recommendation would be network malware detection (possibly through a firewall add-on) as well as good virus/malware detection on the client computers. Sometimes the malware is easier to detect at the client because it reveals itself by trying to access unauthorized memory, processes, or storage.
Do you have any actual evidence (device logs, tcpdump, netflow, etc) that
support that train of thought?
Remember that your Palo Alto isn't stopping 100% of the icky stuff on the
IPv4 side either - the sad truth is that most commercial security software
is only able to identify and block between 30% and 70% of the crap that's
out in the wild. There's also BYOD issues where a laptop comes in and infects
all your systems from behind the firewall (as Marcus Ranum says: "Crunchy on
the outside, soft and chewy inside").
In any case,your first two actions should be to recover the password for the
Palo Alto, and make sure it has updated pattern definitions in effect on both
IPv4 and IPv6 connections.
And your third should be to re-examine your vendor rules of engagement, to
ensure your deliverables include things like passwords and update support
so you're not stuck if your vendor goes belly up..
So your network admin didn't bother to get the login/enable password for a device that is an integral part of your network? That's... a very big lapse in their responsibilities.
I had a consultant recently in CO try to pull the same stunt on me for one of the companies I consult for - stalling, giving bullshit reasons, etc on why they couldn't just hand over the administrative passwords to the actual IT people in the company.
Why were we demanding admin access? Because the company we were paying to maintain it we suspected weren't doing their job. We figured they knew exactly why we were asking for the information, and were buying time.
IIRC, we were right about the condition of the firewall, switches, etc.
Anyways, moral of the story, don't let a consultant hold any and all the keys to the castle for exactly the situation you have right now.
Not to belabor the point, because it will likely be made frequently in
responses, but every legitimate service _should_ have both IPv4 and IPv6
addresses.
Get Palo Alto on the horn, and get access to that box. Get it configured
properly.
I won't hammer you since you're just trying to solve a problem, but v6 is
not a second class citizen. You must consider v4 and v6 for these types of
issues, and making one or the other 'go away' is simply collecting some
tech debt that you'll have to eventually pay off.
We're having problems where viruses are getting through Firefox, and we
think it's because our Palo Alto firewall is set to bypass filtering for
IPv6.
Do you have any actual evidence (device logs, tcpdump, netflow, etc) that
support that train of thought?
Remember that your Palo Alto isn't stopping 100% of the icky stuff on the
IPv4 side either - the sad truth is that most commercial security software
is only able to identify and block between 30% and 70% of the crap that's
out in the wild.
That is only the percentage that it identifies from what it can see. It most likely can not see viruses in encrypted traffic.
" • A forecast that 70% of global Internet traffic will be encrypted in 2016, with many networks exceeding 80%”
"In the fourth quarter of 2015 nearly 65 percent of all web connections that Dell observed were encrypted, leading to a lot more under-the-radar attacks, according to the company. Gartner has predicted that 50 percent of all network attacks will take advantage of SSL/TLS by 2017."
This article mentions how difficult is it for Sandboxes to detect malware.
That is a good point. In order for your PCs to be compromised via ipv6, they would have to be able to establish ipv6 connectivity to each other or to an internet location.
If your network is not configured to support ipv6 it will probably only be possible for your clients to communicate with each other via ipv6 on the local LAN meaning they could only be infecting each other. In order for your clients to be receiving traffic from the Internet via ipv6 would probably require routing and ipv6 configuration support that it sounds like your network does not have. If your firewall is passing v6 traffic, it must understand it enough to forward it across interfaces.
At this point it does not much matter whether the transport layer is v4 or v6 because this problem is higher up the protocol stack. Setting up your firewall to bypass v6 (i.e. just pass it) was a huge tactical error (might be why your consultant is out of business and a bit hard for me to understand. If you want v6 then you would apply the same policies that you do to v4 traffic and if you don't want v6 you would just tell the firewall to drop it.
I think it is much more probable that you are receiving malware via ipv4 or even executable attachments that the out of control firewall is not detecting.
I can tell you that we use the most current versions of Checkpoint firewalls with all of the malware bells and whistles (megabucks) and they are not still 100% effective all of the time. We stop thousands of hacking and malware attempts per hour but it only takes one to become a big pain to deal with.
I would go through the password recovery options on the PaloAlto.
as a next gen firewall you need to ensure you are getting all the latets rulesets
and detection code through - check your subscription with them
once you've sorted out access you can look at the policies and ensure that
the IPv6 AV filtering rules match that for IPv4 - fairly easy with their interface.
(check your codebase version for feature abilities....once again, you may need to
deal with PA to ensure your codebase is current. these things get OLD quickly
as for NAT for IOV6. nope. and turning it off ISNT the answer (yes, its an answer...just
the wrong one! )
Hard to know where to begin with this one, but let me take a shot at it.
1. My top priority would be to get into that Palo Alto firewall. Get Palo
Alto on the phone and figure out password recovery with them. Since you
don’t have the password it is possible that firewall is compromised. Do not
be surprised if you have to jump through some hoops with Palo Alto to prove
that you own it and what has happened. Remember their job is to keep people
out of your network. They are probably also going to want you to be current
on support. If you have to pay to get current on support, do it. You need
that help right now badly.
You could ask Palo Alto how to block the v6 while you are at it or even
better set up a rules that mirror your v4 protection. I cannot stress
enough how big a security issue it is to not have access to your firewall
and not know who does.
2. There are lots of ways to shut off ipv6 but my suggestion would be to
just secure the Palo Alto firewall,
Right. But how long is it going to take to secure the Palo Alto firewall?
If the central Cisco Catalyst really is an IPv6 router, doing a
conf t
ipv6 access-list denyIPv6
deny ipv6 any any
interface [whatever connects to the ISP]
ipv6 traffic-filter denyIPv6 in
ipv6 traffic-filter denyIPv6 out
end
would be a quick fix for the firewall not doing any ipv6 filtering.
It could also break ipv6 enabled web sites or even internal
connectivity, so it'd be better to get someone on the phone w/ Cisco
tech support and have Cisco figure out the best way to block IPv6 for
you.
... to say that any legitimate service
should have a ipv4 address is not quite true now and will definitely not be
true in the near future.
True. But they're in "stop the bleeding" mode and disabling ipv6 is
just a temp work-around until the firewall is fixed.
Did you get the impression that this person asking for help was going to be able to set that up? I didn't (if he was he would probably already know what an ACL is). I do not know if the Catalyst he is looking at is his or his service providers edge devices (or maybe the consultants didn't give them access to that either), I don't know that that Catalyst is the primary router for their network (could be an L2 switch behind the firewall). I also doubt the problem stems from ipv6 as much as it comes from having an out of control firewall. Given what I am hearing about this network I am kind of doubting that it is really ipv6 enabled in any case so your fix prevents ipv6 traffic that is probably not even being routed in the first place. In my opinion not having control of your own firewall is the five alarm emergency in that network right now.
If the network is ipv6 enabled, blocking all ipv6 traffic at that router is probably not a good idea without knowing more. If it is not ipv6 enabled then it will have no effect on the reported issue (malware).
Did you get the impression that this person asking for help was going to be
able to set that up?
Yes, I think the OP could create & apply the acl. Which is why I said
it could break their network & suggested they get Cisco tech support
on the phone to figure out how to safely turn off IPv6.
I'm also giving them the benefit of the doubt that IPv6 really is the
malware infection vector.
I didn't (if he was he would probably already know
what an ACL is). I do not know if the Catalyst he is looking at is his or
his service providers edge devices (or maybe the consultants didn't give
them access to that either), I don't know that that Catalyst is the primary
router for their network (could be an L2 switch behind the firewall). I
also doubt the problem stems from ipv6 as much as it comes from having an
out of control firewall. Given what I am hearing about this network I am
kind of doubting that it is really ipv6 enabled in any case so your fix
prevents ipv6 traffic that is probably not even being routed in the first
place. In my opinion not having control of your own firewall is the five
alarm emergency in that network right now.
Maybe I wasn't clear that the call to Cisco tech support should be a
parallel effort?
If the network is ipv6 enabled, blocking all ipv6 traffic at that router is
probably not a good idea without knowing more.
Which is why I suggested getting Cisco tech support involved. A
mailing list is not where they should be going for help right now.
You need layer-7 firewalls for this. NAT-based "firewalls"
(pseudo-firewalls, really) are layer-4 only. Those will not help you
block typical viruses, as people will usually get infected from
connecting to a compromised Website, or from an e-mail attachments. And
even more, if connections are encrypted, an L7 firewall will not be able
to do anything (whether IPv4 or v6) unless... better not open a can of
worms.
They will just help you block *some* attack vectors, though: those that
rely on starting connections to your hosts from the outside.
My guess is that, with regard to e-mail attachments and compromised
Websites, IPv4 hosts are still attacked more than IPv6 ones, so, even if
you turn off IPv6 you will still get attacked through IPv4.
Everything else has been already said by others: fixing the Palo Alto is
still your best bet.
and a bit hard for me to understand. If you want v6 then you would apply the same policies that you do to v4 traffic and if you don't want v6 you would just tell the firewall to drop it.
)